The aims for MIRA workshops

by Steve Draper

Contents (click to jump to a section)

• What the MIRA grant proposal said
• What people said at the end of the Dagstuhl workshop (overlapping aims and objectives)
• My regrouping of the aims
• How well does our plan for Nancy fulfill these aims?

The point of this document is to remind us of what MIRA is aiming for, and to allow us to reconsider and perhaps modify it at this stage during MIRA's lifetime. I used it in planning the Nancy workshop. Comments are welcome at any time, and I will be asking everyone for their views in the closing plenary session on Friday, when we can review the Nancy workshop and suggest what should be concentrated on at future workshops.

What the MIRA grant proposal said

• Bring the user back into the evaluation process
  
• Understand the changing nature of IR tasks and their evaluation
  
• 'Evaluate' traditional evaluation methodologies
  
• Consider how evaluation can be prescriptive of IR design
  
• Move towards a balanced approach (system versus user)
  
• Understand how interaction affects evaluation
  
• Support the move from static to dynamic evaluation
  
• Understand how new media affect evaluation
  
• Make evaluation methods more practical for smaller groups
  
• Spawn new projects to develop new evaluation frameworks

What people said at the end of the Dagstuhl workshop (overlapping aims and objectives)

1. To develop a design for a multimedia test collection, and launch a collaborative project around this. [Steve Robertson]
   
2. To apply for an EC "research network" [Keith van Rijsbergen]
   
3. Put together collaborative IR projects involving subsets of MIRA participants: could be anything, but certainly including (1) and (2). [Yves Chiaramella]
   
4. Develop tools and approaches for interactive evaluation (e.g. the HCI of the WWW). [Yves Chiaramella ]
   
5. Create a good environment for graduate students: this means having tools and collections available for them to use wherever they are working. [Keith van Rijsbergen]
   
6. Pool information on, and perhaps use of, existing (multimedia) collections. Pool evaluation data, .... (e.g. from Rutgers, City University in London) [David Harper]
   
7. Open MIRA to commercial end users: get them involved.
   
8. "Bring the user back into evaluation"
   8.2 Perhaps this simply means actually doing and presenting evaluations on interactive use.

My regrouping of the aims

• A. Developing methods and tools for evaluating interactive IR. Possibly the most important activity of all. We made a good start with Annelise's talk at Monselice. We need to put more work and more structure into this.
  
• B. User tasks: one of the biggest messages from Dagstuhl was the importance of studying real users, and their overall goals. This is crucial to sensible evaluation that involves users; it reminds us that we may have to design different IR software for different user communities. But it also is relevant to the goal of involving commercial end users. On the one hand, this is important as part of involving users. On the other hand, we must be careful that we don't pick a very special kind of commercial user and so miss out on the majority. The main impact of improved user interfaces is to widen the set of users: professional retrievers are probably now a tiny minority of those using IR software.
  
• C. To develop a design for a MMTC (multimedia test collection). I feel that some of the sessions at Dagstuhl changed my mind on this: that in fact we made some progress. I hope my workshop report captures some of this. However we need to move it forwards: perhaps a strand at the next workshop should address this specifically.
  
• D. Get together collaborative projects. Provided we meet regularly and the workshops allow plenty of time for talking in the evenings etc., no other special action may be necessary.
  
• E. Pool tools and data, providing support for researchers and graduate students beyond their own offices. Someone needs to take a lead on this: to define what is wanted, how it could be organised, what MIRA will do next to progress this. Here are unsolicited suggestions: a) Keith should lead it, because his version of supporting graduate students was more specific than anyone else's and might give focus to it. b) Part of the "how" might be to appoint existing graduate students for a year or six months at a time to coordinate central information e.g. keep a web site up to date. (In fact, using MIRA funds to pay such a student might be a good idea.) This site should tell a complete outsider how to get access e.g. where resources are, who to contact and to get permission from, what hardware their own department must have, etc. It could also, of course, have or point to the best bibliographies for IR.

How well does our plan for Nancy fulfill these aims?

B. Aim [B] concerns real user tasks and hence workplace studies: Annelise's and Raya's presentation addresses this.

C. Friday morning will address multimedia, and so is a step towards aim [C]. It would be desirable to develop discussions for a multimedia test collection, but there does not seem time to do more than prepare for thinking about that. Possibly this could be discussed in one of the groups on Friday, or during Saturday for those staying on.

D,E. We hope informal discussions may promote aims [D] and [E]. We had hoped to have a doctorial consortium to address [E] explicitly, but those plans have been postponed.