20 July 1999 ............... Length about 6,000 words (46,000 bytes).
This is a WWW document maintained by Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/resq.html.
You may copy it. How to refer to it.
Web site logical path:
MEASURING LEARNING RESOURCE USE
M.I.Brown, G.F.Doughty, S.W.Draper, F.P.Henderson and E.McAteer
Dept. of Psychology, University of Glasgow, Glasgow G12 8QQ, UK.
A version of this paper appeared in Computers and Education
1996 vol.27, pp. 103-113.
This paper describes work by the evaluation group within the TILT project
(Teaching with Independent Learning Technologies).
Enquiries about TILT generally should be sent to the project director:
email@example.com or G.F.Doughty, Robert Clark Centre, 66 Oakfield
Avenue, University of Glasgow, Glasgow G12 8LS, U.K.
Contact time is only part of a student's learning, especially in higher
education, and teachers and lab classes are only two of the resources students
draw upon. Despite this, there is a prevalence of studies which appear to be
evaluating a piece of courseware in isolation. Learning gains from a CAL
package are important, but since acquisition and retention of knowledge is
really what is ultimately important and will depend on other learning
experiences in the course, an additional question in any learning situation
therefore is: what resources is a student using? And, following up on that,
which are most useful, are some better than others, or do they complement each
other in essential ways?
This paper describes the design and application of the Resource Questionnaire,
the instrument we are developing in an attempt to gather information on the
learning resources used by students. The resources asked about may include not
only lectures, tutorials and courseware, but books, handouts, notes, and
discussions with other students.
Some preliminary results are described and the importance of this information
to teaching staff in assessing and increasing the value of the resources to
students by ensuring their effective integration into a course, is discussed.
This paper reports the development of a questionnaire instrument
designed to measure how a piece of CAL (Computer Assisted Learning) operates as
part of a larger set of resources available to students. We have used this as
part of a larger package of instruments in what we term "Integrative
Evaluation" . Here we focus on how a piece of CAL relates to other aspects
of a course. Despite the fact that acquisition and retention of
knowledge is what is ultimately important to a student, not whether the source
was CAL, and that in most cases learners have multiple resources to draw upon,
many studies appear to be evaluating a piece of courseware (i.e. a CAL package)
in isolation. A few authors have identified this: Laurillard  regards the
integration of the CAL package into the course as a whole to be very important,
and Blondel et al.  discussing computer-assisted language learning (CALL),
state " since computers are not used in isolation, CALL should not be evaluated
Learning gains from a package are important. In a controlled situation (e.g. in
scheduled classes) pre- and post-tests may show learning gains. However, in an
open access situation (which is how a lot of CAL material is used), if students
don't use it, or can't use it, then however good the package was shown to be in
controlled trials, in these situations it will have no positive effect on
student learning. Students may not have been able to access a CAL package due
to a shortage of computers or computer lab hours, or they may lack the time or
the motivation to attempt to use it, or they may just not have been told about
it. As Rowntree  emphasised in 1974, "Innovation may appear ineffective but
may in fact not be properly implemented". Thus effective integration of CAL or
any other innovation into a course is essential otherwise no matter how
good it is, it will not be of value to students.
The CAL package is in general just one of many resources that is available to
students and like other resources may or may not be used, and may or may not be
the one that matters for an individual student, just as some may doze through a
lecture but then learn the material from a textbook. Differences between
students are probably as important as those between teaching interventions: no
intervention will ever suit all students confronting it, but a good
intervention will have only a small number who do not feel it is of use to them
. In fact this point applies much more widely than to recent attempts to
introduce CAL in higher education. Firstly, it applies in principle to all
educational interventions and resources: all of them operate not in a vacuum
but as part of an ensemble of resources, where the main aim is that the
ensemble as a whole should be effective. This applies particularly in higher
education, where students have, and as an educational principle are meant to
have, considerable responsibility and control over how, when, and what they
learn. Nevertheless most evaluation tends to assume that one intervention
(typically the lecture) is the primary resource. For instance many standard
course feedback questionnaires  do not have questions addressing the
relative value of resources.
On the other hand in distance learning and resource based learning course
designers are much more conscious of providing multiple, partly redundant,
resources and are more likely to seek information about their relative roles
and effectiveness, although rather little seems to have been published on this.
This may be because it seems to those working on distance learning too obvious
to mention, even though as noted it is not to those working on evaluation in
campus institutions. In this paper we take the view that CAL should be
evaluated more in the way that distance learning would be than in the way that
lecture based courses usually are.
This is partly because CAL cannot replace all the features of a typical
lecture, but still more because in reality no one resource in Higher Education
is important by itself. The issue for all
courses in fact is of how students use and benefit from each of the resources
available to them. For instance, textbooks are important on most courses, but
students do not use them while in the classroom. In other words, a learning
activity thought by teachers as well as students to be central, goes on out of
sight of the teacher who in fact has almost no information about its quantity,
quality, or importance. Lecturers' "consultation" hours, questions in lectures,
borrowing other student's notes, and peer interaction with classmates are other
largely unmonitored resources. Any study of learning in natural conditions, at
least in Higher Education, needs to address this question of the pattern of
A question in any learning situation therefore is: what resources is each
student using? And, following up on that, which are most useful, are some
better than others, do they complement each other in essential ways? This is of
interest for at least the following reasons:
- Discovering if some new resource (e.g. courseware) is doing anything
- Discovering the set of resources and activities being engaged in.
Laurillard's model  focuses on the set of activities supported by resources,
and she argues that no one resource supports them all in practice.
- Students in higher education may well engage in "auto-compensation" as argued
by us elsewhere : i.e. if any one resource such as a lecture is deficient,
this may not appear in lower test results as students may then be driven to
another resource (e.g. a textbook) to remedy it.
We have developed a questionnaire, the Resource Questionnaire, in order to
measure the use and value of the different learning resources available to
students. This instrument has undergone significant changes since its
inception, and it will undergo more, but it promises to address the need for
measuring where any particular resource being evaluated (e.g. courseware) fits
into the learning situation as a whole and whether it is being successfully
integrated into the course. From this we hope to find out the value of CAL (and
other resources) to students, and to increase the value of CAL by determining
ways to ensure its effective integration into courses in the future.
Downloadable versions of the resource questionnaire may be found
An example resource questionnaire
Page 2 of the Resource Questionnaire given to students studying Portuguese.
Usefulness of Resources
In the following table, tick each resource you have used during the
course, tick how useful you consider each was to you in
learning/understanding Portuguese. Please give reasons for your answers. Extra
comments can be written on the front of the form.
The questionnaire was originally developed in order to determine what
resources students were using to learn or understand a topic or course and how
useful they considered them to be, but also now includes additional questions
relating to the value of the resources to the student and their accessibility.
Learning resources are defined as anything that a student uses to learn and
understand a topic. The questionnaire lists the resources that the teacher
managing the course considered were available to the students studying a
particular topic or a whole course. The resources asked about include not only
lectures, tutorials and courseware, but books, handouts, notes, and discussions
with other students. The questionnaire can be used at the end of or during a
course where CAL use was scheduled, and also where CAL was optional on open
access. It can also be used in courses where no CAL package is available in
order to assess the use and value to the students of other resources.
|| Use tick if used||
|| Usefulness of Resource ||
|| Reason |
|| not at all useful
|| not very useful
|| very useful
|| extremely useful||
|classes with natural language speaker
|computer class using De Tudo Um Pouco
|material from the Audio Visual Library
|| videos? books? tapes? |
|Portuguese course from Audio Visual Library
|| which? |
|on-line material (eg data-bases, dictionaries etc.)
|| what? |
|computer open access De Tudo Um Pouco
|weekly class assignment
|sample exam questions
|past exam papers
|own class notes
|own notes from computer session
|discussions with students on course
|discussions with students or friends, not on course
|discussions with class teacher
|discussions with course tutor
|discussions with demonstrator
|on-line message box
|| who with?|
|| please specify|
The questionnaire asks students to report their views on the following:
- Use of resources and perceived usefulness of each resource
- Amount of time available to use each resource
- Which resources increased/decreased their interest in the topic
- Which resources they could have managed / could not have managed without
(i.e. essential & non essential resources)
- Resources they would use again
- Resources they found difficult to access
Not all these issues are necessarily addressed in every version of the
questionnaire. A question on the issue of auto-compensation is now included in
Packages may be being used by teachers with little computing experience
who may not have the ability or the computing support to set up a system to
monitor student use of a package. A simple questionnaire is therefore often
easier to administer in order to determine if the package is being used and if
students consider it to be useful to them. In any case with computer
monitoring, unless students are tracked through their use of a package, it is
not possible to determine if they are actually using that package or just
logged into it whilst engaged in some other pursuit. In any case, such on-line
monitoring will not give information about other resources.
Although the general design of the Resource Questionnaire is re-used across
studies, it must always be customised for each new application. These choices
include: creating a list of the resources available and significant in the
particular course, tailoring prompts in the "Reason" column, and varying the
design to cover any particular issues the teacher is interested in. This is
done by consulting the teacher in charge.
Figure 1 shows a page from a Resource Questionnaire used in order to determine
the use of resources by students learning Portuguese. It allows students to
indicate their use, and the perceived usefulness of each resource. In each case
the students can indicate their response on a 5 point scale ("not at all
useful" to "extremely useful") and can add reasons for their answer. A
completed page of a questionnaire given to students studying Unix is shown in
Figure 2. It is of interest to look at the percentage of those students on a
course who actually used a resource and then at the percentage of these
students who found it "useful", "very useful" or "extremely useful", as even if
few students use a resource, if they react positively to it then it can
obviously be of value, and the reasons for other students not using it should
be examined. Examination of the resources used by individual students and their
reported degree of usefulness should enable staff to build up learning profiles
of successful students which might enable them to support less able students in
developing good study skills.
Boyd and Mitchell  note the importance to the student of knowing the
amount of time available for using a package as this can affect their
perception of appropriate pacing and/or rate
of achievement. It is therefore important to determine whether students
felt that there was enough time available to use the CAL packages. This is
important when time is scheduled for their use and also if it is available on
open access as, if students do not believe that there is enough time to use a
CAL package or other resource, they may well not even attempt to use it. We
hope that by asking students to report on a five point scale ("far too little"
-"far too much" time available) how much time they considered was available to
them to use the resource then any real problems may be identified. The
percentage of students who report that there was "far too little time" / "not
enough time" to use a resource is important as this should identify if there is
an overload at that part of the course. In addition, looking at the perceived
usefulness of a resource and the perceived availability of time to use it could
identify whether a re-allocation of time within a course would result in more
effective use of student and staff time (e.g. replacing some lectures with
" What matters most to the young is the effective access to the new
educational goods and the presence, at his or her side of a competent tutor."
, and Scanlon et al.  similarly
emphasised the need for careful planning and costing to ensure easy
access to CAL packages. We agree that this is very important as problems of
access to CAL packages which are meant to be available for repeated use could
easily discourage students. In our studies we must determine whether students
do experience difficulties in gaining access. If a large percentage of
students are identified as "not knowing about a resource" or "having difficulty
accessing a resource" then there may be a real problem in the presentation of
that resource to the students and /or the availability of that resource to the
students. Answers to this question should also
determine if disabled students are having problems of access, as
these problems would be specified in the comments.
Kenning and Kenning , considering computer assisted language
learning (CALL), suggest that CALL may well find its justification, not in
improved test scores, but in positive reactions from learners and teachers and
that the value of CALL lies in the fostering of motivation and the enhancement
of language learning. The Resource Questionnaire therefore attempts to further
detect positive/negative reactions from students by asking them about learning
resources which increased or decreased their interest in a topic.
If students report that they will use a resource again, it suggests that
they value it and therefore if they had also rated it positively in terms of
usefulness, this could reinforce their answers on usefulness. However it should
be borne in mind that reporting that they will use it again does not mean that
they will, and a further questionnaire or follow-up interviews would be
necessary to confirm any re-use of the resources.
By asking students which resources they could or could not have managed
without in learning and understanding a topic or course, the questionnaire
further attempts to identify the usefulness of the resources available to the
students. This question is only related to the resources that they actually
used, and does not therefore give information on resources that they actively
did not use because they considered that they could manage without them. This
should be monitored in future questionnaires as students tend to ignore
anything not felt to be essential , and if this is found to be a reason for
non-use of CAL it would need to be addressed.
Versions of the Questionnaire have been used in eight studies so far.
From these we shall concentrate on some of the results obtained which
illustrate the value of the questionnaire to academic staff.
- In a course on 16th Century Musicianship Skills, a CAL
package was available on optional open access, and training in the use of the
package was also optional. There was no scheduled use. This situation was
partly because the package had been produced by a developer in the TILT Project
who considered there was a need for such a package to aid students, and not by
the lecturer who taught the course. As on-line logging was unavailable, the
Resource Questionnaire was administered at two points during the six week
module to assess actual use, rather than relying on predicted use which has
been found by us to be unreliable. Delivery of the Resource Questionnaire to
these students was pressured - only five minutes for questionnaire completion
at the start of each of two lectures was available. This prevented the
necessary full introduction of the questionnaire, and few students were able to
complete it in the time allocated.
Some interesting findings emerged despite these problems. For example, to a
CAL-naive audience, wording is essential. Stating "Have you used the package"
was not enough, as many students had opened the package during the training
session and felt this counted as package use, confounding the results. However
conclusions were drawn from the results which guided the subsequent evaluation.
The package did appear under-used, though most students (including those who
had only opened it up during the training session) felt it would be "useful" to
"extremely useful". In subsequent interviews and focus groups prompted by the
Resource Questionnaire results, reasons given for non-use included time
constraints and inadequate study skills. As the version of the Resource
Questionnaire used in this study was one of the earliest, the specific question
about the amount of time available for resource use was unfortunately not
asked. It was concluded that the lack of use was a reflection of a student
problem or difficulty, and the fact that students were not reminded of the
existence of the package and its potential benefits. External constraints, such
as problems in gaining access to the CAL material, were not reported. Positive
presentation of the CAL to the students, and the possibility of training the
students in study skills (for CAL and other resources) and time management, are
under discussion in light of these results. More recently the lecturer in
charge of the course has realised the students' need for the training that the
CAL package offers and has asked for it to be made available to the students.
Since he has initiated the request the chances of it being positively presented
to the students are much greater.
- Second year B.Tech. Ed. students studying electronic circuits could
access CAL packages during three scheduled labs and also on open access. Only
27% of those completing the questionnaire (three out of eleven students) used
the CAL package. Two of these students found the sections of the package they
accessed "useful" or "very useful". The other found them "not very useful". A
further questionnaire later in the term indicated that a total of six students
had used the package. Of the initial three, the two students who rated it
positively in questionnaire 1 said they would refer to it again. Other students
found it difficult to understand. Information from the lecturer may partly
explain the negative reactions from the students. The scheduled sessions were
not compulsory and due to other commitments, illness etc. neither the lecturer,
technician, nor demonstrator were present at the second class. Changes in the
method of presentation of the CAL to the students with more directed help may
- Twenty one students studying Portuguese all used a CAL package during
scheduled classes where help was available, and all rated it positively, but
33% of the class reported that there was "not enough" time available to use
it. 43% of the class also used CAL on open access and of these, 44% reported
that there was "not enough" or "far too little" time available to use it.
9am-9.15pm (Mon.-Thurs.) and 9am-4.45pm (Fri.), outwith scheduled use by other
classes, and usually 9am-5pm in vacations]. Reasons given were mainly
concerned with the computers not being available due to other classes using
the lab. When asked about resources that increased or decreased their interest
in a topic, the CAL package scored highly (43% reported that it increased
their interest). Similarly the classes with a native speaker were rated highly
by all the students, but again 33% of the class considered that there was "not
enough"/"far too little" time available for this resource. This was partly the
result of administrative problems in ensuring the presence of a native
speaker. 23% of the class reported that these classes increased their interest
in the subject and 19% of the class noted that the audio visual material
increased their interest. In this situation the students are using the CAL
package and rating it positively, but staff will address the issue of
increasing the access to it and the time available to use it. In addition, due
to the feedback on the classes with a native speaker, staff are working to
ensure the continuation and reinforce the role of these natural language
- A CAL package was developed to aid Social Science and Accounting and Finance
students study statistics. In one study, twenty eight Sociology students (Year
3) used the package during a timetabled session and of these only two (7%)
reported that they would not use the package again. This question which is
normally included in the Resource Questionnaire was in this case included in a
Post-Task Questionnaire. As further use of the CAL package was being logged,
this allowed us to compare predicted re-use with actual re-use of CAL. Of the
twenty six students who reported that they would re-use it only four (14%) did
so, in the following four months, two for 5 minutes or less, one for 11.5
minutes to complete all the sections, and one for 42.5 minutes apparently for
revision. This indicated that predicted re-use of CAL is not reliable. However,
the two students who did not want to re-use it also did not appear to value it
as a resource, as they reported that they would not recommend it to others, and
had not learnt anything from it.
- Dental students used CAL packages during scheduled classes with help
available. The Resource Questionnaire indicated that 95% of the class of
forty two students used the packages, and of these 78% rated them positively
(35% of those who used them rated them "very" or "extremely useful"). Only 26%
of the class used the video tapes which were available, but of these 82% rated
them positively. 73% of those who used them considered that there was "not
enough" or "far too little" time available for their use. Other resources that
there was not enough /far too little time available for were textbooks,
teaching clinic, discussions with academics. Resources that were reported as
being difficult to access by the class of Dental students were: video tapes,
24%; textbooks, 17%; CAL, 10%. Some students commented "What videos?" Staff
considered the videos a very useful resource and had not been aware of the
infrequent use of them. In this Dental course it was therefore decided that
the students should be made more aware of the presence and means of accessing
the videos, as those who did use them did find them useful.
- Forty-one students who were learning to use Unix also completed a Resource
Questionnaire. All resources were rated positively by the majority of students
using them. 49% of those who attended the scheduled labs and 27% of those who
attended unscheduled labs, considered that there was not enough time available
for using these two resources. This problem relating to computers is further
identified by the fact that 27% of the class reported the computers as a
resource difficult to access. This was reported as being due to too few
computers being available during scheduled classes and the lab being too
crowded at all times. In addition, 86% of those who used Exercises as a
resource rated them positively and of these 38% did not consider that there was
enough time available to spend on them.
The Resource Questionnaire can highlight problems in all courses not just where
CAL is in use (e.g. in the course on Unix). Also additional questions can be
included in the questionnaire by lecturers or evaluators concerning any aspect
of a course. For instance, in the Unix course, lecturers used the questionnaire
to quantify problems of which they were already aware. Students were asked if
they considered there was sufficient time to work on Unix and whether there was
sufficient time between a lecture and the related lab. (There was a one hour
lunch break). 32% reported that there was not sufficient time to work on Unix,
and 32% that there was not sufficient time between a lecture and the related
lab. The Resource Questionnaire itself then identified where students required
more time (i.e. in the scheduled and unscheduled labs and in doing exercises).
Results obtained from the same students learning C programming showed that 79%
of them considered that there was not sufficient time to work on C, and again
the main problems were identified as not enough time in the labs and on the
exercises. We hope to question these students after a year in order to
determine their opinion of the courses in light of their experience since then.
In the meantime changes are being made in the courses with re-timetabling and
the introduction of recommended summer reading.
From these examples and others, it is apparent that when a new resource (in
this case CAL) is introduced into a course, it is useful to know whether it is
being used and considered useful by the students for whom it was intended. In
addition, in any course it is useful for teachers to know which resources
students are using and their opinion of them. In the examples above, the CAL
packages which were used by students in scheduled classes were used and rated
positively by the majority of students. However, where the CAL packages were
optional in open access or scheduled situations only a small percentage of
students used them, although many of them did rate them positively. That this
could also be true for resources other than CAL was illustrated by the
infrequent student use of videos in Dentistry.
Many people consider that the use of independent learning technologies will
bring benefits to education. What these benefits will be, we believe will
depend to a great extent not only on the quality of the packages but also on
how these packages are delivered to the students. This in turn depends not only
on staff integrating them into courses in such a way that students are
motivated to make full use of them, but also on the provision of a sufficient
number of computers and support to allow students easy access to the packages.
If this is not done effectively then, as we have found, packages may not be
used. This is similar to what Jones & O'Shea  found, which was that
students may ignore anything not felt to be essential and that perceived
educational benefits often have little to do with the amount of use of CAL. Use
often is related to the time available to use it, and the problems associated
with doing so.
We consider that effective integration should increase the value of a package
to students.Staff know that some students "don't listen", or "don't try to
access resources till the last minute if at all". How much credence should be
given to the percentage of students reporting "not enough time available",
"difficulty in accessing a resource", etc. is debatable as some students may
use "lack of time" as an excuse rather than a reason for not using a resource.
However if analysis of the results shows that students who used a resource,
even if it was a small percentage of the class, found it useful but considered
that there was too little time available or that it was difficult to access
then there may be a real problem and further methods such as focus groups may
be necessary to identify it. Similarly if students report that they did not
know about a resource or did not know how to access it then there may be a
problem, and therefore the way students are told about the resource and the
method of accessing it should be improved. Access problems may include shortage
of available computers (too few altogether, or lack of access to computer labs
in the evening etc.).
Brudenelle and Carpenter  emphasise that lack of interaction between
faculty and students and faulty orientation to the CAI (computer assisted
instruction) as two of the reasons for students' negative attitudes towards
CAI. Presentation of the package to the students by the lecturer will be
important as it will affect their perception of task and their motivation. We
have stressed this in , where we emphasise that students should be made
fully aware of the resource's value in helping them fulfil the objectives of a
course, and be clear about the contribution the resource will make in helping
them successfully complete the course's assessments. Similarly in Case Studies
from MIT Project Athena  it is stated "In education tools can be used with
positive or negative results. Education is, after all, one of the subtlest of
human activities. And whether the potential benefits that the computer seems to
offer are actually achieved depends largely on the teachers who use it."
Not only must the courseware or other resource be properly integrated into the
course, but also the evaluation itself should be planned and implemented with
care. The Resource Questionnaire can take up to 20 minutes for students to
complete carefully, allowing time for instruction as to its completion, and for
its return to the researcher. Failure to plan the administration time
carefully can result in confused students and even more confusing results. If
time is limited, the number of resources must also be limited as, if too many
are listed, students cannot be expected to concentrate fully on all their
If the questionnaire is administered during the use of one of the resources,
for example a lecture, then students who do not attend lectures will be omitted
and their reasons for this and their methods of compensation not detected.
Finally predictive questioning, in this case asking students if they will use a
resource again, was not a reliable method of estimating re-use as subsequent
logging of use showed. It did, however, give additional information on the
perceived value of the resource to the student.
Considerably more work could usefully be done on this questionnaire and
its use, and currently studies are underway to validate it by comparing results
using the questionnaire with results obtained by logging the open access use of
more of the courseware we are evaluating. It is in principle possible that
students seriously misreport their use of resources (e.g. because they do not
remember such activities accurately) and so more checks on validity are
appropriate. However comparable work on whether people can report
accurately from memory on their use of and need for commands in user-computer
interfaces found accuracies above 80%  so confidence in our instrument's
validity seems reasonable.
The effective integration of CAL into a course does not take place
automatically, but requires effort on the part of staff involved and may
involve new study skills on the part of students. It is apparent that many
people realise the importance of integrating CAL into the curriculum and have
identified where problems may occur, and like Scanlon et al.  we recognise
the need for careful planning before introducing a CAL package into a course
. However, once a new resource has been introduced, it is important to be
able to determine if the integration of it into the course is effective, and we
consider the Resource Questionnaire to be a simple method of achieving this. In
addition it may also:
- Provide information on student access to, and evaluation of, all resources
- Enable staff to build up a profile of a student's learning method. This could
in the future enable them to direct a student on the basis of what successful
students did in previous years (i.e. to teach study skills)
- Be used on commercial and in house packages in scheduled and open access
- Provide evidence for more funding for more or better computers, for longer
computer lab opening hours and/or for more staff (technical and/or academic)
As regards the effective integration of a resource such as CAL into a course,
which is what we are concentrating on at present, the questionnaire should
enable the lecturer to obtain the answers to these important questions about a
- Do the students use it?
- Do they find it useful?
- Have they sufficient time to use it?
- Can they get easy access to it?
- Can they get help if they need it?
If the answers to any of these are negative, then it may be necessary to adjust
the situation in which CAL is used in order to maximise the benefits to the
students as only then can effective learning take place. By examining the
comments made by the students in the questionnaire and if necessary in
follow-up interviews, it should be possible to determine which of the following
needs to be actively increased:
- Student awareness of it and its value
- Student access to it
- Staff commitment to it
- Student commitment to it
Certainly our experience shows that the questionnaire can enable lecturers to
identify problems associated with the use by students of the different learning
resources available to them, and consequently to improve the value of these
resources to students by improving their integration within the course.
Although we have most often used it in evaluations of the use of CAL, we have
also used it in courses without any CAL component. In principle it is
applicable to any course, is of greater interest the more a course revolves
around multiple resources of comparable importance, and should be applicable to
distance learning courses although we have not yet had an opportunity to test
this. Together with our other instruments , it provides a method of
evaluating the integration of resources (including CAL) within a course.
A version of this paper was originally presented at CAL'95, and we thank the
audience for their feedback. We are furthermore grateful to an anonymous
referee for pointing out the relationship of this work to a longer tradition of
evaluating distance learning.
The work was done as part of the TILT (Teaching with Independent Learning
Technologies) project, funded through the TLTP programme (Teaching and Learning
Technology Programme) by the UK university funding councils (DENI, HEFCE,
HEFCW, SHEFC) and by the University of Glasgow. The studies discussed here
could not have been done without the active participation of all the TILT
Research Assistants and many members of the university teaching staff to whom
we are grateful, particularly: Stephen Arnold, Peter Dickman, Mike Harland and
1. Draper, S.W., Brown, M.I., Henderson, F.P., and McAteer, E.,
Integrative evaluation: An emerging role for classroom studies of CAL,
Computers & Education. 26, 17-32 (1996)
2. Laurillard, D., Rethinking university teaching: A framework for the
effective use of educational technology . Routledge: London. (1993).
3. Blondel, M., Brick, N., Horrocks, G. and McBride, N., Theory and
Reality of Evaluation; Evaluation and Text Jumbler. Computer Assisted
Language Learning, 3, pp.55-67. (1990).
4. Rowntree, D., Educational Technology in Curriculum Development .
Harper & Row. (1974).
5. Creanor, L., Durndell, H., Henderson, F.P., Primrose, C., Brown, M.I.,
Draper, S.W. and McAteer, E., The Hypertext Approach to Information Skills:
Development and Evaluation. Technical report no.4, TILT project, Robert
Clark Centre, University of Glasgow (1995).
6. Marsh, H.W., "Student's evaluations of university teaching: research
findings, methodological issues, and directions for future research" Int.
Journal of Educational Research, 11, 3, pp.253-388. (1987).
7. Draper, S.W., Brown, M.I., Edgerton, E,. Henderson, F.P., McAteer, E.,
Smith, E.D., and Watt, H.D., Observing and Measuring the Performance of
Educational Technology. Technical report no.1, TILT project, Robert Clark
Centre, University of Glasgow (1994).
8. Boyd, G. McI., and Mitchell, P.D., How can Intelligent CAL better
adapt to Learners? CAL91 Symposium, Lancaster. Computers & Education
18, pp. 23-28. (1992).
9. OECD Report, New Information Technologies, A Challenge for
Education.. New Information Technologies Centre for Educational Research
and Innovation. (CERI). OECD. Paris. (1986).
10. Scanlon, E., Jones, A., O'Shea, T., Murphy, P., Whitelegg, E. and Vincent,
T., Computer Assisted Learning. Institutional Research Review, 1,
59-79. Open University Press. (1982).
11. Kenning, M-M., & Kenning, M.J., Computers and Language Learning.
Current Theory and Practice. Ellis Horwood Series in Computers and their
12. Jones, A.C. and O'Shea, T., Barriers to the Use of Computer Assisted
Learning. British Journal of Educational Technology, 3,
pp. 207-217. (1982).
13. Brudenell, I., Carpenter, C.S., Adult Learning Styles and Attitudes Toward
Computer Assisted Instruction. Journal of Nursing Education, 29,
14. Shaw, R.(Editor) (25 authors), Using Learning Technologies. Interim
Conclusions from the TILT Project.. Technical report no.3, TILT project,
Robert Clark Centre, University of Glasgow. (1995).
15. Hodges, M.E. & Sasnet, R.M., (Eds.) Multimedia Computing: Case
Studies from MIT Project Athena. Addison-Wesley Publishing Company, Inc.
16. Edgerton, E.A., Draper, S.W., & Barton, S.B. Feature checklists in
HCI: some basic results. Interchi '93 Adjunct proceedings, pp.189-190.
Ashlund, S., Mullet, K., Henderson, A., Hollnagel, E. & White,T. (Eds.).
Web site logical path:
[Top of this page]