Lessons on delivering tutorial teaching and ATOMs based on MANTCHI evaluation studies

A Technical Report
by
Margaret I. Brown  &  Stephen W. Draper
Department of Psychology
University of Glasgow
Glasgow  G12 8QQ   U.K.
email: steve@psy.gla.ac.uk
WWW URL:   http://www.psy.gla.ac.uk/~steve

Introduction

This report presents collected lessons, findings and recommendations, learned during the MANTCHI (1998) project as a result of 20 or so evaluation studies, concerning tutorials and how to deliver web-based tutorial support (ATOMs) to students on HCI courses in 4 Universities.  (An overview of their findings is given in another report:  Draper & Brown, 1998.)

This report is structured into the following sections:
A.& B.	The lessons (tutorials and delivery of ATOMs): if you want to know what we recommend, just read these.
C.	The basis: a short discussion of the kinds of evidence underlying this report, which you should read if you wonder just how much faith to put in them.
D.	The theoretical view: a short discussion of the kind of lessons these are.

A.  The lessons: Tutorials

This section presents our collection of findings and recommendations from the evaluation of HCI tutorials before the introduction of the MANTCHI ATOM structure for tutorial material.

1. Expectations
Access to past exam papers, specimen questions and worked examples gives students an idea of the approach they should take to learning and processing the lecture and other material.  This is very highly valued by many students.

2. Practical experience
Students reported the importance of practical experience of actual interfaces, exercises, examples etc. and considered that they required more of these on their courses along with more practical experience of "new technology". Some students studying several formalisms suggested applying the different formalisms to the same interactive device. 

3. Feedback
Students valued feedback and considered small tutorial groups were ideal for this. Even without the expected feedback, many still valued the practical experience of exercises.

4. Collaboration
Students were on the whole enthusiastic about collaborating with students in other universities. Those who had been involved in the first MANTCHI collaborations identified some of the benefits (seeing /hearing other students' experiences) and disadvantages (having to be available at the same time as the other students) of collaborating.

5. Information about the students involved
Students at different levels/courses may have different "requirements" and may require different kinds of tutorial support.

6. Video conferences
Students who had been involved in a video conference, considered that these should only be held for a specific, well defined purpose. Technical problems can interfere with a conference especially if lecturers had not experienced video conferencing before.

B. Lessons: Delivery of ATOMs 

This section presents our collection of findings and recommendations from the delivery of ATOMs 

1. Computer and other Technical Support
The Network is not 100% reliable. Adequate technology has to be available and working to 
deliver/support an ATOM. Lecturers delivering ATOMs have to have alternative plans in hand in case the Network goes down. This is one reason for providing paper-based resources. If the ATOM requires students to use other equipment (in the case of the two ATOMs: scanners and pdf readers), these have to be accessible as do sufficient numbers of suitable computers.

2. Web-based vs. Paper Resources
Web-based instructions and resources may also need to be given to students on paper.  During student use of some ATOMs, lecturers handed out paper-based instructions and resources.  In some cases this was because the students were unable to access the web-based resources, in other cases it was because the lecturer wished to give the students additional instructions which superseded those on the ATOM web page. 

Students usually download and print the web-based resources which is less efficient than these resources being centrally copied on to paper and handed out.  Students reported that though it can be useful to access information etc. electronically, this is not always possible and anyway they like having a hard copy that they can make notes on.  This also covers the problem of the network not functioning when needed by the students.  It is also likely that the students will not have continuous access to computers while completing their assignments.

3. Information about the students involved
Knowledge of the students' previous experience is useful to lecturers involved in collaborative teaching before lecturing/conducting a remote tutorial.  Local teachers need to brief remote teachers on this.

4. Assessment of ATOM tasks
If the ATOM tasks are not directly assessed, students may not complete the tasks. Where courses contain more than one ATOM and the tasks are not directly assessed, students may be less likely to complete the tasks for the later ATOMs. If the ATOM tasks are not directly assessed, students are more likely to report that the workload is too heavy than if it is directly assessed.

5. Students' Expectations
Expectations should be clear. ATOMs may involve remote experts, web-based instructions, and learning resources as well as some "in house" lectures, handouts and other resources. Students require information about what is available to them and what is expected of them in the way of self-tuition (resource-based learning) etc. 

6. Content of ATOM
It should be clear if local instructions about the assignments (completion, submission etc.) differ from those on the ATOM home page. The ATOM (or the course web page) should contain clear information on:  which resources will be delivered locally (in house), what to use: (e.g. a real physical radio alarm in an exercise on formal descriptions); access passwords; the date solutions should be submitted; exactly how solutions should be submitted; the approximate date on which web-based feedback will become available. 

7. Time
Instructions about the ATOM resources and assignments have to be sent to students in plenty of time.  Students do not all check their e-mail every day.  Students admitted that even if they are given information in plenty of time they may not act on it.  However where web-based resources (or any resources) have to be used before an  assignment is to be attempted, students have to be given clear instructions in plenty of time for them to be able to plan and use the resources. They have to have the information to allow them to manage their time effectively.

8. Remote Expert and Local Deliverer
It should be clear to students whether the "in-house" teachers are "experts" or "facilitators".  Each ATOM has a domain expert.  The lecturer delivering the ATOM to his/her students need not be an expert in the subject. It is useful if the students are made aware that the lecturer may be "facilitating" rather than "teaching" and also that the work will involve "resource-based" learning utilising the ATOM web-based resources and a domain expert.

9. Feedback from Domain Expert
Students should be alerted when the web-based feedback on their solutions is available.  They should also be alerted when feedback on the solutions from other universities is available.  I.e. posting them on the web without e-mailing an announcement is unsatisfactory.

10. ATOMs involving Group Work
Group work involves extra organisation and time which has to be taken into account.  Students recognised the benefits of group work, but found that it took more time than working in pairs or alone.  This appeared to matter more where the task was not directly assessed.  If possible, group work should be mainly within regular timetabled sessions of course to avoid clashes between courses.  Similarly video conferences should also be within regular timetabled sessions.  (The general problem is that of organising group meetings and irregular class meetings, which suddenly require new times to be found in the face of, for many students, conflicting classes and paid employment.)

11. New Types of Resources
 Students may need to be encouraged to access  and use new types of resources
Students varied in their use and reaction to the resources available on an ATOM. Many students did not use the TRAILs and other solutions and feedback. Until they become familiar with such resources they may need to be encouraged to use them. However we do have some evidence of the resources including solutions and feedback (tertiary resources) being re-used by some students while completing their projects/ essays.

12. HyperNews Discussion Forum
In future it may be necessary to manage the discussion in some way as the discussion forum was hardly used. During the use of the first two ATOMs this was really just used as a notice board for submitting solutions and getting feedback from the "Remote Expert". One student who did not attend the ATOM lab/tutorial sessions reported using the solutions and feedback on Statecharts and ERMIA to learn/understand these formalisms. In later ATOMs, solutions were submitted on web pages.

13. Collaboration between Students from different Universities
Rivalry between students at different Universities can result from ATOM use. Although this can be a good thing, we have to be careful to avoid the collaborations from discouraging some students from actively participating. Collaboration is mainly perceived as a benefit by students but on one of the ATOMs involving students at two Universities, comments from students at both universities indicated some rivalry and annoyance at comparisons used in the feedback. 

14. The Integration of ATOMs into Courses
ATOMs are discrete units.  The point has been raised that ATOMs could fragment a course, reducing the possibility of relating that topic to other parts of the course.  This could be a problem especially if several are used.  It is something that should be kept in mind. Integration of the ATOMs may be improved by asking students  to write a report involving the topics studied on the ATOMs used, as this appeared to be successful with some referring back to the solutions and feedback.

C.  The basis: what is the nature of the evidence

The evaluations on which these recommendations are based were carried out in three phases. Those in Phase 1 led to the findings reported in Section A and those in Phases 2 and 3 led to the lessons in Section B.  We won't give the actual evidence for every lesson for reasons of space and your boredom.  Here is a discussion of the kind of evidence, and as an illustration all the evidence underlying one of the findings.  This should be enough for you to understand the degree of strength and weakness of the evidence, and so to estimate the degree of belief merited by these lessons.

These findings did not mainly emerge from comparable measures designed to test learning outcomes, but usually from open-ended measures that yield (among other things) complaints by students:  mainly open-ended questions in questionnaires administered to whole classes, interviews with a subset of nearly every class we studied, and the direct classroom observations we did in a majority of our studies.  Full lists of the lessons, usually with the student comments transcribed in full, were fed back to the course deliverers for use in improving delivery next time.  For one example item, we give details of the evidence on which the finding and recommendations were based.  Should it be important to clarify an issue first identified by open-ended measures, then a more systematic measure can be applied.  For instance, when the difficulties of group-work, and claims about high work load appeared, we then designed some systematic measures of these to investigate them further.  Similarly, should one of the lessons in this report be particularly important to you, then you should include some specific measures of it in your own evaluations.

An example of the evidence
Here is some of the evidence on which the lesson "Web-based vs. Paper Resources" (no.2 in section B) was based.

In one study, all were asked if they had any problems while accessing the web-based resources.  25% reported some problem, examples being  "Password problems plus early setbacks with software.", "On learning space — crashes".  In a second study, all students were asked "Did you experience any difficulty gaining access to any resources / activities during the use of the ... ATOM?"  3 (13.6%) reported problems:  "Remote web page" "server was down from where I had to access on-line." "Lab was too busy during lab sessions".  They were also asked about resources for which there was insufficient time, which yielded comments including these:  "Remote web page was too remote, took a very long time to view",  but another student said "None! Most are web-based and therefore can be accessed at any time, when most convenient".  In a third study, students were asked "What else would have helped at the two tutorials this week?" which elicited an 83% response rate including this long reply:  "Computer equipment that worked!  A lot of time was wasted in tutorials trying to fight with the equipment being used.  It is not a necessity to teach through the use of computers when teaching to a computer course.  In fact the opposite is true because computer students above all recognise the problems that can occur by over complicating a problem by using advanced computing e.g. the newsgroup on a web site (where a simple newsgroup added on to [the] news server would have achieved the same inter-communication and been far more reliable/faster than web browsing) and using the scanner (where simply drawing the chart on the computer would have been much faster and produced much clearer results for everyone to view). This is not a criticism of the ATOMs or the teaching method but more of the implementation which although seeming perfectly reasonable proved only to hinder our progress in learning about this topic!"  In a fourth study, students were asked "Did you print the ATOM information and scenario from the Web?";  10 (45.5%) said yes, 12 (54.5%) said no.  They were asked if they used the paper or web form: 7 (31.8%) said paper, 10 (45.5%) said web, 4  (18.2%) said both.  They were asked to explain why; among the numerous comments were "I like to save paper", "I took the work home", "some documents don't print well", "Web-based was easier to refer to related documents because of links".  In a fifth study, printed versions were provided but students were asked if they had already printed out the web documents: 25% said yes.  When asked which form they used, 45.8% used paper form, 20.8% the web form, 12.5% used both, and 20.8% didn't answer.  In a sixth study, when asked how the ATOM compared to traditionally delivered units, one student said "Personally, I do not like using the net as a learning aid, I spend enough time working on a PC as it is without having to rely on the World Wide Wait to scroll through text on screen.  Call me old fashioned, but I do prefer reading from books/journals/papers - a bit more portable and quicker to access - I wish I'd  recorded how much time I waste during a week logging on, waiting for Win95 to start, waiting for Netscape etc. etc. etc.  If I have an hour free in between lectures it is just impractical to get any work done on a PC."  In a seventh study, when asked to comment on "How useful do you consider the ... ATOM Web-based resources were to you in learning & understanding ...? ", two explanatory comments (for low usefulness ratings) were "Items in pdf format prohibited many people viewing the docs", and "Paper-based notes are easier to manage and access. Paper notes don't crash!"

D.  The theoretical view

The main product of integrative evaluation (the method we used: Draper et al., 1996) is formative recommendations, not mainly of the software or other material being tested, but of the overall teaching delivery of which it is (only) one part.  From the start of our evaluations in MANTCHI, as has been the case in many other projects, many of the points that emerged as problems were not about learning outcomes nor about the design of the learning material itself, but were practical points about the management or administration of the activities (e.g. informing students properly about resources and deadlines, availability of computing and other resources).  It is this set of problems, and recommendations for avoiding them, that are presented in this report.

In some descriptions of the educational process these issues are called delivery or implementation (cf. Reigeluth; 1983).  From our perspective of seeing learning as the outcome of a whole set of activities (not the one-way delivery of material), we categorise these issues as the management of the learning and teaching process:  about co-ordinating and organising those activities, rather than designing their content.  This view is presented as an extension to the Laurillard model in Draper (1997), and seen as at bottom a process of negotiation (tacit or explicit) between teachers and learners.

Many of the lessons in this report may seem obvious to readers, not so much from hindsight but because they are familiar points in the education literature.  They are often rather less familiar to higher education teachers (who seldom read that literature), and who have very many such practical details to deal with in delivering any course (another reason for calling them "management" issues).  This suggests that many gains in learning and teaching quality might be made, not by technical and pedagogical innovation, but by attention to best practice at this management level, backed by integrative evaluation to detect and feed back those points that emerge strongly as issues in each particular case.

References
Draper, S.W.  (1997, 18 April) "Adding (negotiated) management to models of learning and teaching"  Itforum (email list: invited paper)  [also WWW document].  URL:  http://www.psy.gla.ac.uk/~steve/TLP.management.html

Draper,S.W.,  Brown, M.I.,  Henderson,F.P. &  McAteer,E.  (1996)  "Integrative evaluation: an emerging role for classroom studies of CAL"  Computers and Education  vol.26  no.1-3, pp.17-32

Draper, S.W., & Brown, M.I. (1998) "Evaluating remote collaborative tutorial teaching in MANTCHI" 
	[WWW document]  URL  http://www.psy.gla.ac.uk/~steve/mant/mantchiEval.html

Laurillard, D. (1993)  Rethinking university teaching: A framework for the effective use of educational technology  (Routledge: London).

MANTCHI (1998)  MANTCHI project pages  [WWW document]  URL  http://mantchi.use-of-mans.ac.uk/

Reigeluth,C.M. (1983)  "Instructional design: What is it and why is it?"  ch.1 pp.3-36  in C.M.Reigeluth (ed.)  Instructional-design theories and models: An overview of their current status  (Erlbaum: Hillsdale, NJ)