Last changed 12 July 1998 ............... Length about 900 words (6,000 bytes).
This is a WWW document by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/courses/cblatom.html. You may copy it. How to refer to it.

ATOM on evaluating CBL

[Notes for learners: main page of the ATOM]

Authored by Steve Draper.

This page describes an "ATOM": a chunk of learning about evaluating CBL (computer based learning). The centrepiece is a practical exercise, with associated reading and some support and video conferences with a remote expert: the author. This version is designed for delivery in Feb-March 1998 to students at Heriot-Watt.

Plan of learner activities

The learning activities for this topic have the following plan:

Basic reading

Various papers can be found on my web page and elsewhere.

First video conference with the remote expert / author

The aims of this first video conference will be:

The agenda and details are on a separate page.

Designing the evaluation

The central exercise is for students to design and carry out an evaluation of a piece of learning and teaching, preferably mediated by technology. The target agreed is the learning on this course carried out in week 6. Students should therefore work on preparing the evaluation (this is where most of the work is) BEFORE week 6. During week 6 they will gather data from each other, using the prepared instruments.

How the work will be divided will be finally decided by the local teachers (Patrick and Alison). Possibilities include the class working as one big team; splitting into two, as the topic for week 6 falls into two parts; or agreeing the overall evaluation as a class, but assigning different instruments (e.g. the quiz, questionnaires etc.) to different students.

The clients in this evaluation are the teachers delivering the course (in this case Patrick and Alison), who may improve how they do that in the light of the evaluation. You will therefore be interviewing them to discover what the various learning resources used are, what their learning objectives for week 6 are, what questions they would like an evaluation to answer. (Good evaluations typically combine issues the teacher wants answered with issues the evaluators think may be important.)

An important part of most evaluations that follow the Integrative approach is to administer a test or quiz. This usually has to be both designed and marked by the teachers: only they have the expert knowledge for this. It is usually a good idea if each question in the quiz corresponds to an explicit learning objective, which similarly must be elicited from the teachers.

Executing the evaluation: gathering data

In week 6, the design of the evaluation and the instruments should all be ready: they just need to be used. Bear in mind that you, the students, are the subjects as well as the evaluators here. You will probably not have the time and patience (as subjects) to fill out 14 different questionnaires. When designing the evaluation and dividing up work, make sure you do not overload your subjects (yourselves). If it is decided to run alternative evaluations in parallel, then you will only get a few students to fill in each instrument. You could of course decide as a class to share data from some instruments (e.g. the quiz) while trying different versions of others (e.g. interviews).

Preliminary report

In real evaluations, the report is not usually written in one go. Instead, interpreting the results depends a lot on what the teachers say, besides comments from others. To do this, it is important to prepare the data gathered and show it the teachers for their comments: i.e. to write a preliminary report with all the data but few conclusions as soon as possible.

Discussion of preliminary reports

Based on the preliminary reports, a class discussion with the teachers and perhaps a video conference discussion with the remote expert will help you explore the interpretations you will want to include in your final reports. If you do not have your data in a form (such as graphs) that can be easily presented in the discussion, you will not be able to get the teachers' interpretations. Our experience has been that this is crucial.