Web site logical path: [www.psy.gla.ac.uk] [~steve] [grumps] [this page]
Stephen W. Draper
It's about how the first study acted as project-gelling. We could send it to a management journal. Julie knows something about this, and should be able to write the abstract lessons and punchlines for such a paper.
I did a very crude first draft and handed it over to Julie, since when nothing hopeful has been heard. My idea was to spend 2-4 hours each on it in turn, passing it swiftly round the whole team.
In one way or another it actively involved everyone: the 3 PIs, our associate in Oz, the 3 RAs, and an RA on a related project. In fact, it could probably be said that it has reinvigorated the Revelation/TLC project, and by cheering up Julie Cargill saved a valuable RA for the university. It has also managed real links with RT in Perth, since he will use our software there in classes, with the connection embodied by sending students over this June-Sept.
It has addressed one of the two application areas named in the grant proposal (education); but equally it hasn't exercised the other (bioinformatics).
It has exercised: collection of real user data; in big volumes; by instrumenting their pre-existing systems; and collecting it over a network. It has exercised some mobile computers for both collection and delivery. It has (only) slightly exercised retrievals from that data, and re-presentation of retrieved results to the researchers (us) and the end-users. It hasn't exercised the notion of dynamically configurable filters at the collection end; and particularly not, changing the filters in the light of the iteratively refined questions researchers ask of the data. Although it has already confronted us with the gap between what is easy/natural to collect, vs. what is meaningful and hence more directly useful (e.g. window events vs. task changes by the user).
We might say, therefore, that this initial study is only half done so far: it needs a phase 2 focussed on data mining/retrieval/analysis, and ideally that should draw on RT's expertise in such analysis which also hasn't yet been exercised in Grumps. And furthermore, that its single biggest omission from the viewpoint of exercising all aspects in the proposal were: a) the bioinformatics application area; b) a proper closed loop from refining retrievals and questions of the data, to modifying the data collection to address them better.
Clearly I should lead on this; hope to offer something coherent soon.
I have quite a few ideas, and quite a lot of lumps of prose; but am still not clear about the overall message. A summary message should probably be in the newsletter articles. On the other hand, Margaret is only now finishing writing up the evaluation data (never mind interpretation).
As a pilot study it does suggest that the LSS can deliver benefits to its users. It is beginning to help students: it saves arm raising and spending a lot of attention in getting a help request seen. Some students feel more able to ask for help. Some tutors already feel it gives them a better grip on distributing their attention over their group. This would, as has been repeatedly remarked, be a much greater benefit at the start of the academic year. All of this is modulated by: getting the tutor and students introduced to the software (it doesn't sell itself instantly with no introduction), the great differences in interaction feel and style in different groups, the fact that some of the gains will be proportional to the length of use and so the amount of past data available to re-present.
Suggested gains, that should be measured in any future study of the LSS, include:
The LSS, as a design project, looks at one of the main teaching activities (level 1: big class; computing labs/tutorial groups: a major investment in teaching resources). It asks how could computing in general, and information management in particular, help. The students are distributed on a network; the tutors are mobile. (A future redesign here would look at having tutors remote, and save them walking around physically. But this might not be able to retain the central human value of interaction that feels personal to the students.) The tutors (and perhaps the students) need JIT ("just in time") information delivery. The role of computers in supporting telephone retail services is supporting the provider (here the tutor) with everything the organisation (here the dept.) knows about interacting with that client (here the student): the history of past interaction, and everything digitally recorded about that client. But an enhancement over the retail model (and so an original feature) would be to provide the student with everything too, including information on interactions and on the tutor. What is interesting, and draws on the CSCW research area, is that even with very small groups or 1:1 interactions, ICT can nevertheless sometimes and in some ways augment the interaction, even when distance is not a factor (as it is in phone retail services, and in some classic CSCW cases). The notion of recording user actions, and re-presenting this record as a resource for further interaction, is natural here, and is close to the heart of Grumps' research too.
Thus the LSS is essentially the result of looking at computing lab teaching, and asking how it could be enhanced by:
There are a number of types of data here:
From this perspective, our first study was a rather small pre-cursor of what is easily possible if we continue on this path. It also means we could view the LSS as (in part) the design of one delivery system in which end users receive and use retrievals of Grumps data: insinuating these into actual user work patterns.
This may be the major paper from the first study; focus will be the study as DIM, making clear that the study covers a large part of the space. Original notion of centering on the contrast of UAR and LSS will probably not be used much by Huw. Key messages listed below; but this paper would stress DIM first, education and specific design only in passing.
UAR: bulk data (several million items), little prior thought about how to get utility from it, instrumented OS and so put up with raw events in OS terms (instrument OS to get data from all applications), slow turnround from collection to retrieval (20 mins. top speed), no payback to endusers (yet), data collection and redistribution over existing fixed network.
LSS: small data, some volatile and not retained? only collected for pre-designed reasons, design new application (instrument application), fast turnround (help request must be "instantly" visible e.g. 30 seconds, actually much faster), designed for direct benefit to endusers (maybe to researchers later), data collection and redistribution over wireless and fixed network to fixed and mobile units. Data redesigned during study to benefit its use (e.g. changing buttons, and changing properties displayed in queue).
Both: can mine data now collected; all data stored subject to filter at source, educational benefit to end user aimed for in long term; privacy issues voiced by users, though with huge variations in attitude.
Major issues to emerge/discover:
Thus, as we can see is implicit in the contrast between UAR and LSS, we have exercised over a large range of the space of DIM activity. Furthermore, this has raised some meaty, and probably novel, issues to report on and to develop further. This is the heading under which I would then proceed to write up these "big" lessons.
We haven't started on this yet.
In a sense, though, we could perhaps view the development of the LSS during the study as an example. Both collection and display (retrieval) changed under the impact of what users said they wanted, and what was required by this application. This is a very slender argument, perhaps specious. Yet it is true that Murray was modifying the software during the study, in its collection, retrieval, and display aspects; and this was based not just on being late in the design, but also on observations of and interactions with the end users, and with our (i.e. researchers') evolving understandings.
I promised; I haven't delivered.
Web site logical path:
[Top of this page]