Last changed 4 Sept 2019 ............... Length about 900 words (9,000 bytes).
(Document started on 17 Aug. 2019) This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/talks/apr4.html. You may copy it. How to refer to it.

Web site logical path: [www.psy.gla.ac.uk] [~steve] [talks] [this page]

Scaling Assessment with Adaptive Comparative Judgement

Title: Scaling Assessment with Adaptive Comparative Judgement
Date/time: Wednesday 4 Sept. 2019. Session: 12:15-1:15, (our own slot: 12:15-12:45 [A-099]).
Live stream of our presentation: Open the session page and at 12:15pm, press play on the embedded YouTube player. (General instructions)
Occasion: Alt-C conference
Place: McEwan Hall, University of Edinburgh
How to get there: map

Authors / Presenters
The authors
Passport photo Passport photo Passport photo Passport photo
Sarah Honeychurch Niall Barr Jeremy Singer Steve Draper
Business school IT Services Computing Science School of Psychology
University of Glasgow

Slides: PDF
Related material: A wider set of collected notes and references on the subject

Abstract

As student class sizes increase without a corresponding increase in teaching staff, the challenge to provide timely, formative student feedback increases. It might seem that moving to e-assessment is the answer. However, this does not necessarily remove the need for staff intervention. We have addressed this, and other related issues, at our University by piloting a novel method of peer review using Adaptive Comparative Judgement (ACJ) software. ACJ is a process where students compare pairs of submissions, making a simple binary judgement on which piece of work is better and providing formative feedback (Pollitt 2012). As judgements are made, the software begins to rank the submissions, and the end result is a sorted series of submissions ranging from best to worst. ACJ thus refers to the algorithm used by the software, and a more descriptive phrase for the process that students undertake might be "assessment by pairwise ranking" (APR).

We begin by briefly explaining the principles behind ACJ/APR before showing how this approach is being used here concurrently in a Futurelearn MOOC (n=1000) and an honours course in Computing Science (n=80), and providing examples of student reactions to this approach.

Computing Science students often write code before they have learnt to read other people's code (Glass, 2003). Our use of ACJ attempts to redress this imbalance at the early stages of encountering a new programming language. It is important that students have completed the task themselves before undertaking this peer review activity so that they have a deep understanding of the problem before they start to make judgements on the work of their peers (Nicol, 2018). This activity therefore only allows students to read other people's code after they have themselves completed the coding task.

The problem is deliberately set up so that a range of code solutions is possible. As learners see other learners' code, they rapidly gain insight into the alternative solution space. One student commented, "I'm impressed by how different the solutions are." Another said, "Isn't it always fascinating to see how different solutions might exist for the same problem?" (FutureLearn comments)

While the example we give is discipline specific, ACJ can be used for any task where there are a variety of "correct" solutions, and we would like to extend our use of ACJ/APR to other subjects. A major part of this session will be a group discussion of the applicability of ACJ to the audience's subject areas and suggestions for how this process might be refined.

References

Dale, V.H.M & Singer, J. (2019) "Learner experiences of a blended course incorporating a MOOC on Haskell functional programming" Research in Learning Technology vol.27. doi: 10.25304/rlt.v27.2248

Glass, R.L. (2003) "About Education" in: Facts and Fallacies of Software Engineering Addison-Wesley, Boston, MA.

Nicol, David (2018) "Unlocking generative feedback through peer reviewing" ch.5 p.47-59 in V.Grion & A.Serbati (2018) Assessment of learning or assessment for learning? Towards a culture of sustainable assessment in higher education Pensa Multimedia, pp.47-59.

Pollitt,A. (2012) "The method of Adaptive Comparative Judgement" Assessment in Education: Principles, Policy & Practice Vol.19 no.3 pp.281-300 doi:10.1080/0969594X.2012.665354
Correction to an equation (published vol.19, no.3, p.387 doi: 10.1080/0969594X.2012.694697)

In order to obtain further information about the conference, please visit Alt-C conference

Web site logical path: [www.psy.gla.ac.uk] [~steve] [talks] [this page]
[Top of this page]