Last changed 23 Mar 2016 ............... Length about 1,000 words (8,000 bytes).
(Document started on 20 Mar 2016.) This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/apr/ex1.html. You may copy it. How to refer to it.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [apr] [this page] [popup1] [popup instructions] [APRintro] [ex1] [software notes]

Ranking exercise for CERE

By Steve Draper, Department of Psychology, University of Glasgow.

This page is about a post-course optional exercise likely to be of interest to students on the PosPsy course. (CERE course web page CERE course moodle page)

The aim of this exercise is to produce a ranking of some student-written wiki from past years of this course, which could assist students' selections of topics for revision. It does this by using students' collective judgements of the helpfulness of the wikis. The ranking data should also give us a measure of the degree of consensus, or spread of opinions, as well. I expect to make this available on the course Moodle page.

Instead of a conventional voting system, the exercise uses a novel method I call APR (Assessment by Pairwise Ranking) where each participant gets a series of pairs of wiki pages, and simply says which they think is the better of each pair on the given criterion. Participants, including professional exam markers, are reported to find this easier and "better" than normal marking. So, besides being more likely to acquire your cooperation, this will be an interesting method for you to experience. (A brief introduction to the thinking behind the method is here.)

Expected benefits for participating students

The rankings produced should help you decide what topics you select for revision, and suggest what order to inspect the wikis in. Each wiki's value is something I can only estimate: student rankings are likely to be more informative from your point of view.
In the process of contributing judgements, you will be looking over various wikis, which is probably a useful warmup to revision in itself. (I say this, partly because of student feedback from other Peer feedback exercises.)
You will experience being a "marker" in a strikingly novel method of assessment.

Overview of what I'm asking you to do

Contribute many pairwise judgements.
You do this by following the links to the software, which is being made accessible via the PosPsy Moodle page.
You sit at a computer (with a fairly large screen: it is probably intolerable to use it on a phone's screen).
You probably should do this over a number of separate sessions (perhaps of 10-30 min.s each ?) during the next 5 days.
Optionally you can also attend the group session at a booked time. This might be fun for sharing experiences, commenting on it to me, etc.

When you're ready to try it (but you might first want to look over what's below on this page), then go to the course moodle page, and follow the instructions in the box "Optional follow-up exercise".

When, where, how

When: the ranking has started. I hope the process will be sufficiently complete by the end of Friday 25 March, if not sooner.

It is generally best to spread your contribution to the ranking over several times. (This is because the software saves overall work, but uses early ranking judgements to estimate which discriminations are essentially clear; and which are going to be close to each other and therefore require more "votes" i.e. comparisons to clarify. Furthermore, it may be most informative to you to do some very early on, when you get many pairs of very different quality to each other; but we will also need plenty of similar quality pairs later on to firm up exact orderings.)

Where: It can be done online, and/or in a booked room.
It needs a large screen i.e. laptop or desktop, not a phone screen, in order to make the wiki PDFs readable.

Where -- room booking: I have booked a room to allow for a group session where you can discuss both the method and the rankings with each other, and ask me about this whole process. This is Wednesday 23 March, 2pm-4pm in BO 603 (psych labs). You will be able to login to the computers in the labs; or bring a laptop and connect via WiFi.

How: When you're ready to try it (but you might first want to glance over what's below on this page), then go to the course moodle page, and follow the instructions in the box "Optional follow-up exercise".
Use a machine with a large screen (phones are too small for this software to give you a good look). Don't use the browser Internet Explorer (too slow for this): university computers also have Chrome as standard, if you look for it.

Software: credit

The particular software underpinning this exercise is being written at Glasgow University by Niall Barr. It is very new to us, and there will be rough edges to say the least. But it has already been used here e.g. for judging abstracts submitted to a conference.

Finale / finally

When you're ready to try it, then go to the course moodle page, and follow the instructions in the box "Optional follow-up exercise".

Parting slogan:

"Vote early, vote often"