Lessons on delivering tutorial teaching and ATOMs based on MANTCHI evaluation studies

A. The lessons: Tutorials

2. Practical experience
Students reported the importance of practical experience of actual interfaces, exercises, examples etc. and considered that they required more of these on their courses along with more practical experience of "new technology". Some students studying several formalisms suggested applying the different formalisms to the same interactive device.

3. Feedback
Students valued feedback and considered small tutorial groups were ideal for this. Even without the expected feedback, many still valued the practical experience of exercises.

4. Collaboration
Students were on the whole enthusiastic about collaborating with students in other universities. Those who had been involved in the first MANTCHI collaborations identified some of the benefits (seeing /hearing other students' experiences) and disadvantages (having to be available at the same time as the other students) of collaborating.

5. Information about the students involved
Students at different levels/courses may have different "requirements" and may require different kinds of tutorial support.

6. Video conferences
Students who had been involved in a video conference, considered that these should only be held for a specific, well defined purpose. Technical problems can interfere with a conference especially if lecturers had not experienced video conferencing before.

B. Lessons: Delivery of ATOMs

2. Web-based vs. Paper Resources
Web-based instructions and resources may also need to be given to students on paper. During student use of some ATOMs, lecturers handed out paper-based instructions and resources. In some cases this was because the students were unable to access the web-based resources, in other cases it was because the lecturer wished to give the students additional instructions which superseded those on the ATOM web page.

Students usually download and print the web-based resources which is less efficient than these resources being centrally copied on to paper and handed out. Students reported that though it can be useful to access information etc. electronically, this is not always possible and anyway they like having a hard copy that they can make notes on. This also covers the problem of the network not functioning when needed by the students. It is also likely that the students will not have continuous access to computers while completing their assignments.

3. Information about the students involved
Knowledge of the students' previous experience is useful to lecturers involved in collaborative teaching before lecturing/conducting a remote tutorial. Local teachers need to brief remote teachers on this.

4. Assessment of ATOM tasks
If the ATOM tasks are not directly assessed, students may not complete the tasks. Where courses contain more than one ATOM and the tasks are not directly assessed, students may be less likely to complete the tasks for the later ATOMs. If the ATOM tasks are not directly assessed, students are more likely to report that the workload is too heavy than if it is directly assessed.

5. Students' Expectations
Expectations should be clear. ATOMs may involve remote experts, web-based instructions, and learning resources as well as some "in house" lectures, handouts and other resources. Students require information about what is available to them and what is expected of them in the way of self-tuition (resource-based learning) etc.

6. Content of ATOM
It should be clear if local instructions about the assignments (completion, submission etc.) differ from those on the ATOM home page. The ATOM (or the course web page) should contain clear information on: which resources will be delivered locally (in house), what to use: (e.g. a real physical radio alarm in an exercise on formal descriptions); access passwords; the date solutions should be submitted; exactly how solutions should be submitted; the approximate date on which web-based feedback will become available.

7. Time
Instructions about the ATOM resources and assignments have to be sent to students in plenty of time. Students do not all check their e-mail every day. Students admitted that even if they are given information in plenty of time they may not act on it. However where web-based resources (or any resources) have to be used before an assignment is to be attempted, students have to be given clear instructions in plenty of time for them to be able to plan and use the resources. They have to have the information to allow them to manage their time effectively.

8. Remote Expert and Local Deliverer
It should be clear to students whether the "in-house" teachers are "experts" or "facilitators". Each ATOM has a domain expert. The lecturer delivering the ATOM to his/her students need not be an expert in the subject. It is useful if the students are made aware that the lecturer may be "facilitating" rather than "teaching" and also that the work will involve "resource-based" learning utilising the ATOM web-based resources and a domain expert.

9. Feedback from Domain Expert
Students should be alerted when the web-based feedback on their solutions is available. They should also be alerted when feedback on the solutions from other universities is available. I.e. posting them on the web without e-mailing an announcement is unsatisfactory.

10. ATOMs involving Group Work
Group work involves extra organisation and time which has to be taken into account. Students recognised the benefits of group work, but found that it took more time than working in pairs or alone. This appeared to matter more where the task was not directly assessed. If possible, group work should be mainly within regular timetabled sessions of course to avoid clashes between courses. Similarly video conferences should also be within regular timetabled sessions. (The general problem is that of organising group meetings and irregular class meetings, which suddenly require new times to be found in the face of, for many students, conflicting classes and paid employment.)

11. New Types of Resources
Students may need to be encouraged to access and use new types of resources
Students varied in their use and reaction to the resources available on an ATOM. Many students did not use the TRAILs and other solutions and feedback. Until they become familiar with such resources they may need to be encouraged to use them. However we do have some evidence of the resources including solutions and feedback (tertiary resources) being re-used by some students while completing their projects/ essays.

12. HyperNews Discussion Forum
In future it may be necessary to manage the discussion in some way as the discussion forum was hardly used. During the use of the first two ATOMs this was really just used as a notice board for submitting solutions and getting feedback from the "Remote Expert". One student who did not attend the ATOM lab/tutorial sessions reported using the solutions and feedback on Statecharts and ERMIA to learn/understand these formalisms. In later ATOMs, solutions were submitted on web pages.

13. Collaboration between Students from different Universities
Rivalry between students at different Universities can result from ATOM use. Although this can be a good thing, we have to be careful to avoid the collaborations from discouraging some students from actively participating. Collaboration is mainly perceived as a benefit by students but on one of the ATOMs involving students at two Universities, comments from students at both universities indicated some rivalry and annoyance at comparisons used in the feedback.

14. The Integration of ATOMs into Courses
ATOMs are discrete units. The point has been raised that ATOMs could fragment a course, reducing the possibility of relating that topic to other parts of the course. This could be a problem especially if several are used. It is something that should be kept in mind. Integration of the ATOMs may be improved by asking students to write a report involving the topics studied on the ATOMs used, as this appeared to be successful with some referring back to the solutions and feedback.

C. The basis: what is the nature of the evidence

These findings did not mainly emerge from comparable measures designed to test learning outcomes, but usually from open-ended measures that yield (among other things) complaints by students: mainly open-ended questions in questionnaires administered to whole classes, interviews with a subset of nearly every class we studied, and the direct classroom observations we did in a majority of our studies. Full lists of the lessons, usually with the student comments transcribed in full, were fed back to the course deliverers for use in improving delivery next time. For one example item, we give details of the evidence on which the finding and recommendations were based. Should it be important to clarify an issue first identified by open-ended measures, then a more systematic measure can be applied. For instance, when the difficulties of group-work, and claims about high work load appeared, we then designed some systematic measures of these to investigate them further. Similarly, should one of the lessons in this report be particularly important to you, then you should include some specific measures of it in your own evaluations.

An example of the evidence

In one study, all were asked if they had any problems while accessing the web-based resources. 25% reported some problem, examples being "Password problems plus early setbacks with software.", "On learning space -- crashes". In a second study, all students were asked "Did you experience any difficulty gaining access to any resources / activities during the use of the ... ATOM?" 3 (13.6%) reported problems: "Remote web page" "server was down from where I had to access on-line." "Lab was too busy during lab sessions". They were also asked about resources for which there was insufficient time, which yielded comments including these: "Remote web page was too remote, took a very long time to view", but another student said "None! Most are web-based and therefore can be accessed at any time, when most convenient". In a third study, students were asked "What else would have helped at the two tutorials this week?" which elicited an 83% response rate including this long reply: "Computer equipment that worked! A lot of time was wasted in tutorials trying to fight with the equipment being used. It is not a necessity to teach through the use of computers when teaching to a computer course. In fact the opposite is true because computer students above all recognise the problems that can occur by over complicating a problem by using advanced computing e.g. the newsgroup on a web site (where a simple newsgroup added on to [the] news server would have achieved the same inter-communication and been far more reliable/faster than web browsing) and using the scanner (where simply drawing the chart on the computer would have been much faster and produced much clearer results for everyone to view). This is not a criticism of the ATOMs or the teaching method but more of the implementation which although seeming perfectly reasonable proved only to hinder our progress in learning about this topic!" In a fourth study, students were asked "Did you print the ATOM information and scenario from the Web?"; 10 (45.5%) said yes, 12 (54.5%) said no. They were asked if they used the paper or web form: 7 (31.8%) said paper, 10 (45.5%) said web, 4 (18.2%) said both. They were asked to explain why; among the numerous comments were "I like to save paper", "I took the work home", "some documents don't print well", "Web-based was easier to refer to related documents because of links". In a fifth study, printed versions were provided but students were asked if they had already printed out the web documents: 25% said yes. When asked which form they used, 45.8% used paper form, 20.8% the web form, 12.5% used both, and 20.8% didn't answer. In a sixth study, when asked how the ATOM compared to traditionally delivered units, one student said "Personally, I do not like using the net as a learning aid, I spend enough time working on a PC as it is without having to rely on the World Wide Wait to scroll through text on screen. Call me old fashioned, but I do prefer reading from books/journals/papers - a bit more portable and quicker to access - I wish I'd recorded how much time I waste during a week logging on, waiting for Win95 to start, waiting for Netscape etc. etc. etc. If I have an hour free in between lectures it is just impractical to get any work done on a PC." In a seventh study, when asked to comment on "How useful do you consider the ... ATOM Web-based resources were to you in learning & understanding ...? ", two explanatory comments (for low usefulness ratings) were "Items in pdf format prohibited many people viewing the docs", and "Paper-based notes are easier to manage and access. Paper notes don't crash!"

D. The theoretical view

In some descriptions of the educational process these issues are called delivery or implementation (cf. Reigeluth; 1983). From our perspective of seeing learning as the outcome of a whole set of activities (not the one-way delivery of material), we categorise these issues as the management of the learning and teaching process: about co-ordinating and organising those activities, rather than designing their content. This view is presented as an extension to the Laurillard model in Draper (1997), and seen as at bottom a process of negotiation (tacit or explicit) between teachers and learners.

Many of the lessons in this report may seem obvious to readers, not so much from hindsight but because they are familiar points in the education literature. They are often rather less familiar to higher education teachers (who seldom read that literature), and who have very many such practical details to deal with in delivering any course (another reason for calling them "management" issues). This suggests that many gains in learning and teaching quality might be made, not by technical and pedagogical innovation, but by attention to best practice at this management level, backed by integrative evaluation to detect and feed back those points that emerge strongly as issues in each particular case.

References

Draper,S.W., Brown, M.I., Henderson,F.P. & McAteer,E. (1996) "Integrative evaluation: an emerging role for classroom studies of CAL" Computers and Education vol.26 no.1-3, pp.17-32

Draper, S.W., & Brown, M.I. (1998) "Evaluating remote collaborative tutorial teaching in MANTCHI"
[WWW document] URL http://www.psy.gla.ac.uk/~steve/mant/mantchiEval.html

Laurillard, D. (1993) Rethinking university teaching: A framework for the effective use of educational technology (Routledge: London).

MANTCHI (1998) MANTCHI project pages [WWW document] URL http://mantchi.use-of-mans.ac.uk/

Reigeluth,C.M. (1983) "Instructional design: What is it and why is it?" ch.1 pp.3-36 in C.M.Reigeluth (ed.) Instructional-design theories and models: An overview of their current status (Erlbaum: Hillsdale, NJ)

Web site logical path: [ www.psy.gla.ac.uk] [~steve] [mant] [other formats] [this page]