Web site logical path: [www.psy.gla.ac.uk] [~steve] [EVSmain] [this page]
Question design: [one question] [purposes] [contingent sessions] [feedback] [whole sessions]

Question formats

There is a whole art to designing MCQs (multiple choice questions). Much of the literature on this is for assessment. In this context however we don't much care (as that literature does) about fairness, or discriminatory power, but instead will concentrate on what will maximise learning.

Here I just discuss possible formats for a question, without varying the purpose or difficulty. I was in part inspired by Michele Dickson of Strathclyde University. The useful tactic implied by her practice is to vary the way questions are asked about each topic. Applied to statistics this might be:

• Name a concept and ask which of several alternative definitions is correct.
• Give a definition, and ask which of several concepts it defines.
• Show a plot of data or other specific case, and ask which of several (conceptual) statements about it is correct.
• Show a statement, and ask which of several plots of data or specific cases match the statement.
• Describe or show a specific real world case rather than data or measurements of it i.e. a description of a real situation being modelled, as opposed to the graphical or tabular representation of particular data; and ask which of several (conceptual) statements about it is correct.
• Show a statement, and ask which of several real world cases match the statement.

The idea is to require students to access knowledge of a topic from several different starting points. Here I exercised three kinds of link, and each kind in both directions. Exercising these different types and directions of link is not only important in itself (because understanding requires understanding all of these) but keeps the type of mental demand on the students fresh, even if you are in fact sticking on one topic.

Types of relationship to exercise / test

• Between concept and instance: category<->specific case
• Between different concept representations: name<->definition or description
• Between different instance representations: representation1 of a case<->representation2 of a case
  

The first is that of linking ideas or concepts to particular examples or instances of them e.g. is a whale a fish or a mammal? Another form of this is linking (engineering or maths) problems with the principle or rule that is likely to be used to solve it. However both concepts and instances are represented in more than one way, and practice at these alternative representations and their equivalences is usually an essential aspect of learning a subject. Thus concepts usually have both a technical name, and a definition or description, and testing this relationship is important. Similarly instances usually have more than one standard method of description and, although these are specific to each subject, learners need to master them all, and questions testing these equivalences are important. In teaching French language, both the spelling and the pronounciation of a word needs to be learned. In statistics, an example data set should be represented by a graph, a table of values, as well as a description such as "bell shaped curve with long tails". In chemistry, the name "copper sulfate" should be linked to "CuSO4" and a photograph of blue crystals, and questions should test these links. (See Johnstone, A.H. (1991) "Why is science difficult to learn? Things are seldom what they seem" Journal of computer assisted learning vol.7 no.2 pp.75-83 for an argument related to this based in teaching Chemistry.)

These relationships are all bidirectional, so questions can (and should) be asked in both directions e.g. both "which of these is a mammal" and "to which of these categories do dolphins belong?". Thus a subject with three standard representations for instances plus concept names and concept definitions will have five representations, and so 20 types of question (pick one of five for the question, and one of the remaining four for the response categories). Additional variations come from allowing more than one item as an answer, or asking the question in the negative e.g. "which of these is not a mammal?: mouse, platypus, porpoise?".

The problem of technical vocabulary is a general one, and suggests that the concept name-definition link should be treated especially carefully. If you ask questions that are problems (real-world cases) and ask which concept applies but use only the technical names of the concepts, then students must understand perfectly both concept and the vocabulary; and if they get it wrong you don't know which aspect they got wrong. Asking concept-case questions using not technical vocabulary but paraphrased descriptions of the concepts can separate these; and separate questions to test name-definition (i.e. concept vocabulary).

Further Response Options

1. A spider
2. An insect
3. An arachnid
4. (1) and (2)
5. (2) and (3)
6. (1) and (3)
7. (1) and (2) and (3)
8. None of the above
   

It may or may not be a good idea to include null responses as an option. Against offering them is the idea that you want to force students to commit to an answer rather than do nothing, and also the observation that when provided usually few take the null option, given the anonymity of entering a guess. Furthermore, a respondent could simply not press any button; although that, for the presenter, is ambiguous between a decision rejecting all the alternatives, the equipment giving trouble to some of the audience, or the audience getting bored or disengaged. However if you do include them as standard, it may give you better, quicker feedback about problems. In fact there are at least three usually applicable distinct null options to use:

• "None of the above" i.e. (I think there's) [a problem with the answer set]
• "I think there's a mistake in the question" [a problem with the question]
• "I have no idea" ("I don't know") [a problem with the respondent]

Assertion-reason questions

An extension of this are: