HCI Lecture 5

This is a WWW page originally created by students (see other credits on this page) but now copied and stored by Steve Draper as part of the CSCLN ATOM.

Think-Aloud Protocols

The details of this lecture are available on pages 12-14 of the handout.

Useful Links (under construction)

Last lecture
Next lecture
Email Angus Beaton , (page maintainer) with your constructive criticism/comments.

Overview

The main points to be addressed in this lecture were:

What is a think-aloud protocol?
In what way(s) can their usefulness be optimised?

What is a think-aloud protocol?

Think aloud protocols consist of observing a user working with an interface while encouraging them to "think-aloud"; to say what they are thinking and wondering at each moment. Think-aloud protocols are of particular value because they focus on the problems a user has; when the user is working without difficulty, direct observation (and hence the think-aloud protocol itself) is of very limited use. This is because the user is unable to communicate as fast as they think and act, unless a specific problem arises which slows them down. It is at these times when this method really shines as it allows the observer to correlate the actions and statements of the participant.

We see the main advantages of this method as:

Rapid, high-quality, qualitative user feedback (e.g. as compared with questionnaires).
Data is available from a broad range of sources, such as:
- Direct observation of what the subject is doing.
- Hearing what the subject wants, or is trying, to do.
If the subject gets into difficulties, the observer has the chance to clarify the situation.
High degree of flexibility; the experiment may easily be steered by the observer.
The presence of two people allows meaningful, direct dialogue.

Furthermore, the protocol may be used in two distinct scenarios.

The observer specifies a definite task to be accomplished by the subject.
- This allows the observer to concentrate on a specific task they are interested in.
'Open-ended'; no task is specified, and the user is free to choose their own task.
- Allows the observer to concentrate on naturally occurring problems

In what way(s) can their usefulness be optimised?

This may be answered by giving an overview of how a think-aloud protocol should be conducted in practice, in order for its results to be realistic and helpful to the observer. The purpose of the experiment, e.g. to examine in detail a specific 'corner' of the interface, or to gain an outline of its general efficacy, should of course be decided upon initially. This will allow the observer to choose one of the scenarios given above, and to tailor the protocol accordingly.

Introduction

As an observer, you need to make sure of the following:

That user realises that the interface, not they, are under scrutiny.
That the user should at all times comment liberally on his/her actions, intentions and thoughts.
That the user is at ease. This involves explaining that you may give only a bare minimum of help to the user, and apologising in advance for this. The user should try to find their own way as much as possible.
Any help given to the user should be carefully thought out, in order for its effects to be recorded as part of the experiment.

Note taking

The main task of the observer is to jot down what happens.
The moment when an observer notes what a subject is doing/saying only occurs once and thus needs to be recorded.
This can be aided by a structured data sheet, can include categories to observe and also prompts [see handout].
Note taking is still relevant even when videoing the experiment as the video representation cannot capture everything and there is no way of clarifying ambiguities when watching a video.

Use of video recording equipment

Useful because session can be recorded for later analysis
With the observers usual position beside the participant they may miss something
BUT, for the reasons specified above, note taking is still vital
For monetary and accuracy reasons an effective alternative to the videoing of a session may be to simply repeat the session with different participants

Use of Prompts

There is a considerable difference between prompting and biasing the user, basically say as much as necessary to keep user happy without helping and making suggestions. Such questions may include:

What are you thinking now?
Why did you do that?

If a subject is completely stuck a decision may have to be made as to whether to help the participant or not. It is up to the judgement of the observer, but if a decision is made to allow some help the following should be noted:

Ask what the user would do if the oberver wasn't there
Take a note of what you said and what happened afterwards
Was the problem solved?

All of the above maximise the effectiveness of the experiment by noting the effect of your input.

Analysis

Usually done informally at the time by careful note taking but a more quantitative content analysis can be done by examining video evidence after the experiment.

Hints

The method is informal and the most effective single way to maximise effectiveness is to create an informal atmosphere.This is where the subject can feel relaxed and as unaware as possible that their actions are being recorded. As mentioned above hints may well contribute to such an informal atmosphere.

Summary