26 Jul 1998 ............... Length about 1000 words (7,000 bytes).
This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/overview.html.
You may copy it. How to refer to it.
Learner main page
Course home page (sample)
CBL evaluation: Overview of the ATOM
Preface: introduction to the notion of an ATOM
this link for a brief introduction to the notion of an ATOM, and terms
such as trail, remote expert, delivery, etc.
State the topic and type of ATOM.
This ATOM is a practical introduction to evaluating CBL material. It
was intended as one week's work on a CBL course, but is probably rather more
than that: one of the bigger exercises on a course. It spreads over several
weeks in elapsed time because of the need to plan, execute, and analyse an
evaluation. (The sequence of activities is given on
the learners' main page.)
Timetabling may be constrained to fit around the delivery time of
the CBL chosen to be evaluated.
How the concept fits into the course
Relationships and constraints on this topic in relation to the
This ATOM was designed for use in a module on CBL (computer based
learning). Evaluation, though often neglected, is a crucial topic.
Conceptually, it could be taught at any point: you could begin with it to set
a standard, or end with it. The kind of evaluation presented here is
empirical, classroom based evaluation i.e. it is about observing and measuring
student reactions and learning outcomes of CBL (not about software features, or
Student prerequisite knowledge
None, although it is assumed that the students will have some knowledge
of HCI evaluation or else they may be unready to accept the importance of
testing real users.
Student prerequisite skills
It is assumed, although perhaps not wholly essential, that the students
will have some experience of HCI evaluation methods, some experience of
designing and using questionnaires, and some experience of receiving CBL
This ATOM has been used at M.Sc. level but there is no obvious reason not to
use it at any level, except perhaps for issues of independence and study habits.
These often depend more upon discipline (arts vs. science) than course level.
However planning and executing an evaluation as an autonomous activity,
without simpler warmup activities, probably does require previous experience
that may not be available except at this level.
Timetabling: how the activity fits into the course
Amount of student work
Intended as one week's work for a module (say 8 hours), but probably
Amount of contact hours
We scheduled 2 2-hour class meetings (using video conference), where
this ATOM was a main item of business.
The plan is for students to work in groups.
Any size is conceivable. Because an evaluation of this kind usually involves
several different instruments (questionnaires), it is natural to have several
(more than two) students per group. Having several groups in the class allows
different designs to be contrasted.
Relationship to rest of course
The key constraint is that this exercise is about evaluating some
(other) piece of CBL. When this has been decided on, then the work for this
ATOM is scheduled around it: with data gathering during that delivery, analysis
and writeup after, and planning before. Thus this ATOM can probably not be the
last activity in a course, and the CBL delivery used as a target cannot be the
first thing in a course. More details on the sequence of activities is given
Learner activity plan
Follow this link for details.
The reports of their group's evaluation written by each student should
probably be marked. In the one delivery so far, this was done by the local
deliverers, but after (taking into account) the verbal comments given to the
students by the remote expert.
The ATOM requires that the students have access to either themselves or
another group of students while they are receiving some CBL material.
State how much work is required of the remote expert, and how essential it is
Used for seminar/tutorial (by video conference) twice: to discuss the
reading and/or preliminary design of the evaluation, then to discuss
preliminary findings by the student groups. Only 2-4 hours of expert's time.
Such an expert is only necessary to the extent that expert discussion
of the (always diverse) methods and findings by the various students is
wanted by the deliverers.
None (other than that CBL material).
None for computing.
Video conferencing for the whole class, if the remote expert is used.
Papers are available on the web and in journals.
History and admin.
Authored and maintained by
at the University of Glasgow.
Number of times delivered: 1
Evaluation reports available: 1
Number of Trails (examples of past student work): none.
Structure of the ATOM's web pages
There are 4 kinds of web page associated with this, perhaps with any,
- The teacher overview page [this page], intended for teachers browsing for
ATOMs they might choose to use in their courses.
Teacher details: fuller notes for teachers, with instructions
for them, and rationales for why the exercise is the way it is.
learner details page: the instructions for learners. This is likely to have
the basic explanation and description of the ATOM, and also acts like a
handout for students.
course home page (for learners) for a particular delivery of an ATOM,
special to the institution, course, and circumstances. This should have all
the links needed for those students, and in turn should act as the entrance
lobby, and be pointed to by the course pages, links from the teachers' home
pages etc. A sample is included here, but its content and format may vary
widely depending on the course within which the ATOM is being used.
Learner main page
Course home page (sample)