Weighted Multiple Choice Questions

I’ve been mulling this blog over for quite a few weeks and thinking not only about how best to bring together a number of various ideas I’ve read and seen, but also to allow people to enact these theories by creating some free weighted multiple choice quizzes on Tes for anyone to use. Topics include:

• Mean from frequency tables 
• Place value and rounding 
• Percentage increase and decrease
• Percentages of amounts

I am hoping what follows is an explanation and justification for how the use of weighted multiple-choice questions, used as low stakes tests, can help improve student learning.

Weighted multiple choice questions are similar to multiple choice questions in that they include distractors (misconceptions), but they also allow students to indicate their confidence levels in their answers. In Mark McCourt’s Book ‘Teaching for Mastery’ I read about a model proposed by Sparck, Bjork, Bjork (2016) where student’ indicate on a triangle like the example below. Unlike multiple choice questions, guessing is greatly discouraged, since choosing an incorrect alternative or choosing any point on the line between the two incorrect alternatives, results in a major loss of points. While I’m not wholly behind quantifying learning (grading/scoring) as it can distract students from learning, I think in this case this does serve a positive purpose. The value of the points can be changed but should serve enough to tap into the hyper correction effect (discussed later). Also, the student has the option of choosing a point in the middle of the triangle to indicate they don’t know and gain a score worth zero. These are easy to self-mark and this is important since as Dylan Wiliam puts it … “the best person to mark a test is the person who just took it”[1]. Scores can be totalled up and be used in multiple ways and could be kept by the student to monitor their progress - maybe in an ipsative way (read here for my blog on ipsative assessment).

Cambridge maths espresso have produced an excellent summary of the benefits of confidence assessments. Essentially, to summarise:

Putting a self-assessment value of confidence on a question improves performance (Koivula, Hassmen & Hunt, 2001).

Most people tend to overestimate their own abilities, but this effect is more pronounced amongst those with the least skills (Dunning & Kruger, 2008)

Importantly, this tells us that using confidence assessments will benefit those learners who lack the knowledge required most. This is crucial as often worksheets and quizzes are planned and created with the most able in mind due to our natural bias and curse of the knowledge. In addition though, confidence assessments have powerful effects on mathematical thinking for both affirmation and the hyper correction effect.

The hyper correction effect refers to the finding that suggests errors committed with high confidence are more likely to be corrected than low confidence errors. The feeling of surprise a pupil has when discovering that something they firmly thought to be true was a misconception leads them to better correct the original problem and be far more likely to remember that correction for future. Also, correct high question-based confidence levels have been linked to greater retention (Hassman, Hunt & Dybeck 2003).

When considering confidence ratings it was concluded by Sparck et al. that confidence weighted multiple choice was significantly more effective than even standard multiple choice with confidence judgment in their final performance. Sparck’s findings clearly show that first of all low stakes testing has a positive impact on final study performance (fig.3) and that confidence weighted multiple choice is the best of these (fig. 4).

Learning is risky business. By definition, learning means doing something you don’t know how to do. It comes with risks. Where possible we should look to make it less risky by reducing the cost of failure. This is because failure can often lead to effects on student's engagement. Unfortunately, maths anxiety can be common and students’ motivations and self-efficacy play a large part in contributing to how much effort is invested. Which is of course a very important ingredient for learning.

Contrary maybe to common beliefs, testing regularly and frequently can be a successful method of reducing maths anxiety[1]. Ensuring that the testing is with low stakes (or even no stakes) material builds students feeling of being better prepared and practiced for higher stakes tests.

Take for example the following models of spaced study and testing patterns posed by Roediger and Karpicle (2006):

Which is more likely to result in improved long-term learning?

1. Study study study study - test
2. Study study study test – test
3. Study study test test -test
4. Study test test test – test

The most successful pattern for embedding learning is the fourth option. Having just one study and then three short testing sessions which are spaced, and then a final assessment, outperformers any other pattern. This testing effect not only helps assess what a student knows but it also enhances retention. When these are multiple choice questions the testing can be also be used to help identify why the other alternatives options are incorrect.

Developing effective low stakes testing therefore can be beneficial to learning. When used formatively this can be particularly beneficial for responsive teaching. John Mason is often quoted as saying that “teaching take place in time, learning takes place over time.” This testing should be spaced apart from the teaching so that the tests are assessing learning and not performance, but they are also being a tool for learning. Both these strategies (practice testing and distributed practice) are the learning techniques that Dunlosky (2013) have shown the highest effects on learning as they increase both retrieval and storage strength.

Often during test two or three there is no need for corrective teaching. Since the hyper correction effect is doing the teaching for you when the answers are revealed as the student recalls previous learning. However, being mindful that maybe some students are not attuned to this and are repeatedly either selecting ‘don’t know’ or are not clear with their understanding. One idea can be to raise their responsibility of asking for feedback and allow students to drop off their test for feedback in a box or something similar. Placing the emphasis on the learner again helps them build the internal desire knowing they are going to see this test again. Building the self-efficacy, self-regulation and motivation ultimately. It also seems possible that consistent use of the confidence-weighted format might lead to students’ becoming, in general, more appreciative of or sensitive to the benefits of retrieval practice.

Summary:

1. Confidence judgments with answers improve performance.
2. Weighted confidence judgments are even better.
3. Using a pattern of distributed {study, test, test, test} with low stakes testing has greatest impact on long term learning both storage and retrieval.

NQT Maths - Number: Teaching for Understanding (Secondary)

Wednesday 7 October 2020 - 4.00pm to 5.15pm 
First in a series of sessions reviewing the strands of maths and various pedagogical approaches. This session we will look at how to handle, the often-difficult topic of, directed number and the varying approaches that can be used to aid understanding. If students do not understand directed number, they often go onto to struggle with abstract content. We will also look at dividing fractions, a topic that is often taught by rote learning and again look at how you can help students understanding so that they can tackle novel problems.  
Click on the title link for more information or to book.

Maths Workshop - Sequencing Lessons: Estimating the Mean from Grouped Data (Secondary)

Tuesday 13 October 2020 - 3.30pm to 4.45pm
Maths Workshops are designed to help teachers think carefully about how to plan a sequence of lessons that promote researched informed techniques that support students long term learning. In this session we will look at a how to plan the order of teaching and ideas for teaching finding the mean from grouped data.
Click on the title link for more information or to book.

Both of these sessions will run as interactive Zoom sessions.