Last changed 3 Feb 2013 ............... Length about 900 words (6000 bytes).
(Document started on 3 Dec 2012.) This is a WWW document maintained by Steve Draper, installed at You may copy it. How to refer to it.

Web site logical path: [] [~steve] [talks] [this page]

Assessment by pairwise ranking

Title: Assessment by pairwise ranking: Prospects for a completely different method of marking (presentation)


Presenter Steve Draper,   School of Psychology,   University of Glasgow.

Slides: PDF
Handout: PDF file
Related material:


Assessment by pairwise ranking (APR), also referred to by various other terms, has recently emerged from the school sector as a radical departure in assessment, only recently made feasible by technology.  This talk introduces it, and discusses whether and how it could be beneficial and practical to introduce it in Higher Education.

In APR, instead of reading each student script once and deciding its mark, markers see pairs of scripts and decide which should rank above the other on a single complex criterion.  Software assembles these pairwise judgements into a quantitative interval scale (based on Thurstone's "law of comparative judgement").  Finally, grade boundaries are superimposed on the rank order.  Controlled studies using professional markers employed by school exam boards have shown that marking in this way gives much higher (NOT lower) reliability, and that for large numbers of scripts, the total time taken is less.  The statistics can directly identify when sufficient consensus has been reached, which scripts generate most disagreement (send them for a second and third opinion), and which markers agree least with other markers.  Originally designed for higher reliability (fairness) of marks and reduced costs, it can also produce feedback comments.  The talk will also discuss how it applies to different disciplines.

The most interesting underlying issue is that APR is in complete contrast to the approach which is currently, by default, the dominant one of breaking a judgement down into separate judgements against multiple explicit criteria, which at least has the virtue of supporting usefully explicit and diagnostic feedback to learners.  Instead, APR uses a single complex criterion for marking.  However in many ways, real academic values have a large implicit aspect;  and furthermore, are holistic instead of being always and simply reductionist.  APR is particularly appropriate for portfolios of work, and for work where different students may choose to submit in different media (e.g. printed, web pages, audio tapes).

In order to book online and obtain further information about the conference, please visit 6th Annual University of Glasgow Learning and Teaching Conference.

Web site logical path: [] [~steve] [talks] [this page]
[Top of this page]