Related web sites: [MIRA] [Dublin workshop] [The draft document] [this page]

Kal's comments on draft 3

Preface

Dear Micheline,
Please find below some comments regarding the Mira wg paper on the EuroMMTC. When reading further, remember that my bullets hit myself equally; I could not have written a better paper. At some distance it is easier to criticise. I may have gone a bit astray somewhere because the rest of us (you) are not keeping me on track. Distribute as you find appropriate.

1

This is a good question in order to be able to perform (or learn to do) evaluation. I would also pose another question: What research questions are there to be answered by evaluating multimedia systems? — Why do we do multimedia systems evaluation? The problems that bring us to evaluation influence (at least should!) greatly the kind of evaluation needed. After that we may find that some further research questions remain to be answered before we can perform the evaluation.

2

I do not believe that there is such a framework. I believe that work and life with their information activities are so diverse that no single framework is sufficient. Therefore we should be selective, focun on some domain and tasks, and develop the framework for that. Annelise's work might be good starting point for development.

3

This aim says to me that we want to develop a overall multimedia evaluation methodology. It has been on the table since the Mira Monselice Meeting and we have not made much progress I dare say. This is a very broad goal and particularly difficult when we hane not performed even the first multimedia systems evaluation (well, some have done some but overall we are not really experienced with this). I think that it requires a lot of work to be able to say more on this broad goal than Annelise already has said. She has a framework which more or less suggests methodologies (if not particular methods) for evaluation. We also could learn from HCI / ethnomethodology people (mentioned later in the paper).

Why not be more moderate? Identify some work contexts and tasks where multimedia is frequently used; try to understand the use of multimedia there; try to identify possible problems in using multimedia and/or the systems in such contexts; identify system / media features related to these problems; then develop evaluation methods (and standards) for these features. That would tell us what kind of TCs are needed.

Actually, Section 3 of the paper goes into this direction.

4

I think that we should be more specific about the tasks. 80% of newsmaking is production line work for the next day's media presentations and 95% based on news agencies daily supply. There is very little retrieval. 20% may be features perhaps requiring extensive research in all kinds of collections, not just digital news libraries. — Statistics by Kal's imagination.

Shall we focus on both? Which mix of media (as output of the tasks)?

Focusing on news consumption? Most of that (96%) is consuming the daily bread with little interest in exploring in the possibly available background material: there is no time. The rest is related to information needs and retrieval problems which vary between the bottom of sea and distant galaxies. Even these could be analyzed, but it might be difficult to build a TC for it. [But this fosters the traditional TREC approach ...]

5

6

The expertise listed is better suited for a traditional engineering focus I dare say. Some of the Mira participant institutions actually do have expertise in work domain analysis (AMP) and user studies (Sheffield, the Royal School, Tampere) but many of their experts have not participated in Mira (e.g., Sheffield or Tampere people in user studies). There is expertise in the news MEDIA domain (yes, we computer people have dealt with the documents!) but not much in news making (the tasks).

An interdisciplinary approach is needed. For the work domain analysis part we should try to integrate people currently outside Mira. The effort should not be dominated by plain computer science parties.

7

In order to materialize the following benefits we really need to pay a lot attention to the tasks:

• Work based design incorporating real needs and real problem solving tasks
• Research driven by real life problems taking account of constraints in the workplace
• Expand the output of IR experimental evaluation/design approaches to include work context

We are much better off in the more technical areas.

8

- This is a way of testing and strengthening the foundation of the EuroMMTC project; the MMTC should be invaluable for some research goals.

Related web sites: [MIRA] [Dublin workshop] [The draft document] [this page]
[Top of this page]