Last changed 13 Nov 2009 ............... Length about 1000 words (9,000 bytes).
This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/ilig/qdesign.html.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [EVSmain] [this page]
Question design: [one question] [purposes] [contingent sessions] [feedback] [whole sessions]

Question formats

(written by Steve Draper,   as part of the Interactive Lectures website)

There is a whole art to designing MCQs (multiple choice questions). Much of the literature on this is for assessment. In this context however we don't much care (as that literature does) about fairness, or discriminatory power, but instead will concentrate on what will maximise learning.

Here I just discuss possible formats for a question, without varying the purpose or difficulty. I was in part inspired by Michele Dickson of Strathclyde University. The useful tactic implied by her practice is to vary the way questions are asked about each topic. Applied to statistics this might be:

The idea is to require students to access knowledge of a topic from several different starting points. Here I exercised three kinds of link, and each kind in both directions. Exercising these different types and directions of link is not only important in itself (because understanding requires understanding all of these) but keeps the type of mental demand on the students fresh, even if you are in fact sticking on one topic.

Types of relationship to exercise / test

In the abstract there are three different classes of relationship to test:

The first is that of linking ideas or concepts to particular examples or instances of them e.g. is a whale a fish or a mammal? Another form of this is linking (engineering or maths) problems with the principle or rule that is likely to be used to solve it. However both concepts and instances are represented in more than one way, and practice at these alternative representations and their equivalences is usually an essential aspect of learning a subject. Thus concepts usually have both a technical name, and a definition or description, and testing this relationship is important. Similarly instances usually have more than one standard method of description and, although these are specific to each subject, learners need to master them all, and questions testing these equivalences are important. In teaching French language, both the spelling and the pronounciation of a word needs to be learned. In statistics, an example data set should be represented by a graph, a table of values, as well as a description such as "bell shaped curve with long tails". In chemistry, the name "copper sulfate" should be linked to "CuSO4" and a photograph of blue crystals, and questions should test these links. (See Johnstone, A.H. (1991) "Why is science difficult to learn? Things are seldom what they seem" Journal of computer assisted learning vol.7 no.2 pp.75-83 for an argument related to this based in teaching Chemistry.)

These relationships are all bidirectional, so questions can (and should) be asked in both directions e.g. both "which of these is a mammal" and "to which of these categories do dolphins belong?". Thus a subject with three standard representations for instances plus concept names and concept definitions will have five representations, and so 20 types of question (pick one of five for the question, and one of the remaining four for the response categories). Additional variations come from allowing more than one item as an answer, or asking the question in the negative e.g. "which of these is not a mammal?: mouse, platypus, porpoise?".

The problem of technical vocabulary is a general one, and suggests that the concept name-definition link should be treated especially carefully. If you ask questions that are problems (real-world cases) and ask which concept applies but use only the technical names of the concepts, then students must understand perfectly both concept and the vocabulary; and if they get it wrong you don't know which aspect they got wrong. Asking concept-case questions using not technical vocabulary but paraphrased descriptions of the concepts can separate these; and separate questions to test name-definition (i.e. concept vocabulary).

Further Response Options

The handsets do not directly allow the audience to specify more than one answer per question. However you can offer at least some combinations yourself e.g.
"Is a Black Widow:
  1. A spider
  2. An insect
  3. An arachnid
  4. (1) and (2)
  5. (2) and (3)
  6. (1) and (3)
  7. (1) and (2) and (3)
  8. None of the above

It may or may not be a good idea to include null responses as an option. Against offering them is the idea that you want to force students to commit to an answer rather than do nothing, and also the observation that when provided usually few take the null option, given the anonymity of entering a guess. Furthermore, a respondent could simply not press any button; although that, for the presenter, is ambiguous between a decision rejecting all the alternatives, the equipment giving trouble to some of the audience, or the audience getting bored or disengaged. However if you do include them as standard, it may give you better, quicker feedback about problems. In fact there are at least three usually applicable distinct null options to use:

Assertion-reason questions

I particularly commend asking MCQs that, instead of asking which fact is true, ask which reason for a given fact is the right one.

An extension of this are:

  • Assertion-reason questions

    Some references on MCQ design

  • McBeath, R. J. (ed.) (1992) Instructing and Evaluating Higher Education: A Guidebook for Planning Learning Outcomes (New Jersey: ETP)

  • CAAC (Computer Assisted Assessment Centre) website advice on MCQ design

    Web site logical path: [www.psy.gla.ac.uk] [~steve] [EVSmain] [this page]
    [Top of this page]