This draft is by Miche and Annelise.
Also, Kal has written some comments.
TREC = ?
MM = multimedia
TC = test collection
MMTC = multimedia test collection
IR = Information Retrieval
Following the Grenoble workshop the working group set out to consider
the following:
- What research questions needed to be addressed for evaluating
multimedia systems?
- What test framework could be used to encompass the full range of
evaluative issues related to multimedia and to promote the development
of appropriate evaluation methodologies?
- How would the concept of a test collection fit into that framework
and how would it serve to address the various research questions?
In the preliminary discussions to consider the feasibility of a test
collection approach for multimedia systems evaluation, it was
recognised that:
- an MMTC could not easily be pre-specified in the way that the ideal
test collection had been to form the basis for TREC.
- an appropriate model would need to be developed in the context of
real world work domains in order to take full account of both user
task domains and systems approaches to evaluation.
Hence the primary objective of a Mira proposal for a project would be
to focus on the development of a framework which would draw on
suitable real world work domains to inform on what would be useful for
evaluating multimedia systems and in what way as well as assemble
illustrative resources for the purpose of carrying out evaluations.
The working group then considered what would be the best approach for
developing that framework. There appears to be two possibilities:
- to identify a sample of work domains from which a generalised
framework for addressing evaluation issues could be defined.
- to identify a single large and varied work domain rich in multimedia
sources and uses which would provide a suitably representative
environment to derive a framework which then could be applied and
verified in other work domains.
The following sections present the aim and outline of a possible
research proposal and approach which could be adopted for developing
such a framework based on a single work centered case study.
The overall research aim is to develop a framework that identifies
appropriate methodologies for evaluating how well multimedia digital
library systems, services and resources support different work
requirements.
To meet that research aim it is suggested that a single domain case
study would be the most effective way of focusing and co-ordinating
the research efforts of the different participants and at the same
time provide enough flexibility for individual research groups to
pursue different aspects of multimedia research.
The news media which includes all the processes for generating
disseminating and using news information sources is being proposed as
a possible domain which would meet the following criteria:
- a domain which is at the forefront of information and communications
technologies where IT has already impacted on work tasks and where new
developments are continuing to take place
- a rich multimedia domain covering text, static/moving images, sound
and different combinations of these
- a domain where many different kinds of single and multimedia
documents are produced and repackaged by different parties and users
and are handled in many different ways, i.e. stand alone, composite,
linked, clustered
- a domain which encompasses real time, dynamic as well as archival
information and where original source materials are as important as
the output information products and services
- a domain which is rich in knowledge acquisition, data and
information handling and retrieval tasks and which supports
collaborative work in a variety of ways. Some of these tasks may be
ill defined and unpredictable and others will be changing as the
technology evolves.
- a domain which makes use of the WWW as well as other highly
interactive systems
- a domain involving a variety of different types of users, experts as
well as non-specialists untrained end users.
A news media case study would involve a combination of ethnographic
and ecological studies as well as parallel experimental research
activities drawing on appropriate conceptual models pertinent to
work-centered evaluation, e.g. models of information need and
interaction in IR.
The direct study of the work domain would involve:
- identifying information needs and characteristics of different types
of users, i.e. different types of journalists as well as different
consumers of news information
- carrying out work domain analyses of news generation and retrieval
tasks
- identifying and gaining access to the various multimedia information
sources and resources pertinent to news media generation or
dissemination which could form the basis for a representative sample
of test materials.
The ethnographic studies would serve to identify existing multimedia
systems in the news media domain as well as provide empirical data for
the specification of prototype systems which integrate multimedia
information sources. Existing systems including web based services as
well as prototypes designed to address evaluation issues, would be the
subject of the evaluative studies.
The approach is to explore the design of a test facility by
incorporating the use of prototype demonstrator systems together with
the development of a sample collection(s) of multimedia sources. The
demonstrator systems would be designed and used specificaly to address
evaluation methodological issues. The evaluative experiments would
include a combination of field and laboratory type studies with real
or perhaps simulated users, focus on the different components of
multimedia systems: i.e. search engines, indexing structures,
interface features as well as explore what types of measures could be
applied. The empirical evaluations will lead to the collection of data
which can be tested in relation to research various questions. Hence
the methodological issues regarding the design of appropriate
evaluative experiments would be addressed in a collaborative
co-ordinated and more systematic way.
In addition to designing and carrying out research into evaluation
experiments, another aspect of the research activity would be to
evaluate the evaluation framework itself and test the validity of the
experiments for measuring to what extent multimedia systems meet the
requirements of the chosen work context. This would involve comparing
test results with different evaluative criteria relating to user
characteristics, different task activities for news generation and
retrieval and support of the work domain requirements more generally.
By evaluating the framework it should also be possible to determine
which aspects or components are likely to be applicable to other work
domains and tested in due course.
The strength of this proposal is that it is based on an
interdisciplinary approach that integrates the different interests of
Mira participants. These include expertise on:
- IR evaluation methodologies and experimental design
- Indexing methods for multimedia
- Multimedia database development
- Search engine development
- Interface design
- Multimedia system design
- Navigational and semantic structures design
- Work domain analysis and user studies
- News media domain knowledge
Some of the benefits envisaged in adopting the proposed approach
include:
- Work based design incorporating real needs and real problem solving
tasks
- Research driven by real life problems taking account of constraints
in the workplace
- Provide a basis for choosing appropriate evaluation methods
- Enable the comparability and transferability of evaluation studies
- Expand the output of IR experimental evaluation/design approaches to
include work context
- Identify new areas for the development of evaluation methodologies
- Identify new research issue and areas
Related web sites:
[MIRA]
[Dublin workshop]
[this page]
[Top of this page]