Last changed 13 April 2022 ............... Length about 1500 words (12,000 bytes).
This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/qdesign.html.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [EVSmain] [this page]
Question design: [one question] [purposes] [contingent sessions] [feedback] [whole sessions]

Question formats

(written by Steve Draper,   as part of the Interactive Lectures website)

There is a whole art to designing MCQs (multiple choice questions). Much of the literature on this is for assessment. In this context however we don't much care (as that literature does) about fairness, or discriminatory power, but instead will concentrate on what will maximise learning.

Here I just discuss possible formats for a question, without varying the purpose or difficulty. I was in part inspired by Michele Dickson of Strathclyde University. The useful tactic implied by her practice is to vary the way questions are asked about each topic.

A common type of MCQ concerns one relationship e.g. (using school chemistry as an example domain) "What is the chemical symbol for gold: Ag, Al, Au, Ar ?"

Reversing the relationship

You can equally, and additionally, ask about the same relationship in reverse: "Which metal is represented by the symbol 'Au'? Gold, silver, platinum, copper?"

Multiple types of relationship

When you have several relationships, the alternative question types multiply. Consider these 3 linked pieces of information: a photo of a gold nugget or ring; the word (name) "Gold"; and the symbol "Au". These 3 pieces of information each have a relationship with the other 2, giving 3 types of relationship; and each has 2 directions, giving 6 question types in all:

Applied to statistics this might be:

The idea is to require students to access knowledge of a topic from several different starting points. Here I exercised three kinds of link, and each kind in both directions. Exercising these different types and directions of link is not only important in itself (because understanding requires understanding all of these) but keeps the type of mental demand on the students fresh, even if you are in fact sticking on one topic.

Types of relationship to exercise / test

In the abstract there are three different classes of relationship to test:

The first is that of linking ideas or concepts to particular examples or instances of them e.g. is a whale a fish or a mammal? Another form of this is linking (engineering or maths) problems with the principle or rule that is likely to be used to solve it. However both concepts and instances are represented in more than one way, and practice at these alternative representations and their equivalences is usually an essential aspect of learning a subject. Thus concepts usually have both a technical name, and a definition or description, and testing this relationship is important. Similarly instances usually have more than one standard method of description and, although these are specific to each subject, learners need to master them all, and questions testing these equivalences are important. In teaching French language, both the spelling, the pronounciation, and the meaning of a word need to be learned. In statistics, an example data set should be represented by a graph, a table of values, as well as a description such as "bell shaped curve with long tails". In chemistry, the name "copper sulfate" should be linked to "CuSO4" and a photograph of blue crystals, and questions should test these links. (See Johnstone, A.H. (1991) "Why is science difficult to learn? Things are seldom what they seem" Journal of computer assisted learning vol.7 no.2 pp.75-83 for an argument related to this based in teaching Chemistry. See also Roy Tasker's group: http://visualizingchemistry.com/research.)

These relationships are all bidirectional, so questions can (and should) be asked in both directions e.g. both "which of these is a mammal" and "to which of these categories do dolphins belong?". Thus a subject with three standard representations for instances plus concept names and concept definitions will have five representations, and so 20 types of question (pick one of five for the question, and one of the remaining four for the response categories). Additional variations come from allowing more than one item as an answer, or asking the question in the negative e.g. "which of these is not a mammal?: mouse, platypus, porpoise?".

The problem of technical vocabulary is a general one, and suggests that the concept name-definition link should be treated especially carefully. If you ask questions that are problems (real-world cases) and ask which concept applies but use only the technical names of the concepts, then students must understand perfectly both concept and the vocabulary; and if they get it wrong you don't know which aspect they got wrong. Asking concept-case questions using not technical vocabulary but paraphrased descriptions of the concepts can separate these; and separate questions to test name-definition (i.e. concept vocabulary).

Further Response Options

The handsets do not directly allow the audience to specify more than one answer per question. However you can offer at least some combinations yourself e.g.
"Is a Black Widow:
  1. A spider
  2. An insect
  3. An arachnid
  4. (1) and (2)
  5. (2) and (3)
  6. (1) and (3)
  7. (1) and (2) and (3)
  8. None of the above

It may or may not be a good idea to include null responses as an option. Against offering them is the idea that you want to force students to commit to an answer rather than do nothing, and also the observation that when provided usually few take the null option, given the anonymity of entering a guess. Furthermore, a respondent could simply not press any button; although that, for the presenter, is ambiguous between a decision rejecting all the alternatives, the equipment giving trouble to some of the audience, or the audience getting bored or disengaged. However if you do include them as standard, it may give you better, quicker feedback about problems. In fact there are at least three usually applicable distinct null options to use:

Assertion-reason questions

I particularly commend asking MCQs that, instead of asking which fact is true, ask which reason for a given fact is the right one.

An extension of this are: Assertion-reason questions.

Covertly related questions: Using 3 questions to make a strong test of understanding one concept

Mark Russell suggests using 3 (say) alternative questions all testing the same key concept. With MCQs with 4 response options, 25% of students will get a question right by accident if they answer at random: not a strong test. He suggests having 3 alternative questions testing exactly the same concept, and only students who get all 3 of these correct should be regarded as having learned the concept. The questions are tacitly linked (by being about the same concept), but not listed adjacently and not using similar structure. He found that students who did not have a sound understanding of the concept did not even recognise that the 3 questions were linked: the disguise does not need to be elaborate (contrary to expert / staff perceptions, who naturally see the 3 questions as "about the same thing" exactly because they grasp the concept).

Russell, Mark (2008) "Using an electronic voting system to enhance learning and teaching" Engineering Education vol.3 no.2 pp.58-65 doi:10.11120/ened.2008.03020058

Some references on MCQ design

  • CAAC (Computer Assisted Assessment Centre) website advice on MCQ design

  • Johnstone, A. H. (1991) "Why is science difficult to learn? Things are seldom what they seem" Journal of Computer Assisted Learning vol.7 no.2 pp.75-83 doi:10.1111/j.1365-2729.1991.tb00230.x
  • See also Roy Tasker's group: http://visualizingchemistry.com/research

  • McBeath, R. J. (ed.) (1992) Instructing and Evaluating Higher Education: A Guidebook for Planning Learning Outcomes (New Jersey: ETP)

  • Russell, Mark (2008) "Using an electronic voting system to enhance learning and teaching" Engineering Education vol.3 no.2 pp.58-65 doi:10.11120/ened.2008.03020058

    Web site logical path: [www.psy.gla.ac.uk] [~steve] [EVSmain] [this page]
    [Top of this page]