Last changed
4 July 1998 ............... Length about 900 words (6000 bytes).
This is a WWW document maintained by
Steve Draper, installed at http://www.psy.gla.ac.uk/~steve/dublinmore.html.
Web site logical path:
[MIRA]
[MIRA workshops]
[Dublin workshop]
[programme]
[this page]
More details on Dublin workshop
Each MIRA member will probably want to publish a paper to represent the
result of their work in MIRA. But all papers will be refereed and will only
be published in the conference book if they pass this process. As the
deadline for submission is less than 6 months away (January 1999), it is time
to start planning these papers. At the Dublin MIRA workshop, there will be
sessions for each of us to try out an outline of a paper on each other, and
get some useful feedback in time to act on it before the deadline.
Each MIRA member is advised to to prepare an outline for a paper and present
it to a group in Dublin. This is voluntary, but hopefully can be helpful.
The plan is to divide into small groups of about 6, and each present a short
outline and see what comments the rest of the group make about its suitability
and interest.
At the last MIRA workshop, several working groups were set up. Each working
group is expected to present a report at Dublin. Furthermore, we would like
there to be a set of exercises for people to participate in at the final
conference. This will give them a personal experience that is probably much
more convincing that just an academic paper would be. For instance, Fabio's
working group is looking at the amount of disagreement between people in
making relevance judgements. It should be possible to produce an exercise
that demonstrates this, in which anyone can make some relevance judgements and
then compare their own with other people's. At Dublin, we hope each working
group will present an exercise like this for us all to try out: if it works
well, we can use it at the final conference.
At the Dublin MIRA workshop there will be a series of 3 related sessions.
They will explore issues, often discussed in the literature but not brought
into mainstream consideration in IR evaluation, about the complexities of
"relevance" and "information need". The aim of each of these sessions is to
bring out and juxtapose as many different points on the theme as possible, so
that the workshop can consider how they might be put together. Hence many
very short talks, each making a few distinctive points, would be most useful.
The first session is on "information needs" (what are they, how many different
types are there?). The second is on scalar, non-binary relevance judgements
i.e. on arguments for getting away from the binary relevance judgements used
in most test collections and recording each judgement at least as a number or
ranking. The third is on multiple types or dimensions of relevance: on why
even a single number is inadequate.
In my (Steve Draper's) view, the papers by Stefano Mizzaro are the best
summary in these areas so far, and we should use them as a starting point.
They offer a framework that tries to cover all the issues, and have copious
references to the literature that are useful in relating the various different
terms and distinctions that have been made. It may be that the discussion
will show us how the framework has to be extended to cover everything (the
second of these sessions run by Fabio probably does not fit directly), but
it would be helpful if everyone has looked at Stefano's work first.
One of his papers, also published in Interacting with Computers
is
available on the web . (He has another one with an even more comprehensive
literature review.)
For the first session, on information needs, a useful starting point is his
view of that as a dimension (dimension 2, section 2.2 in his paper).
I have summarised his 4 dimensions on a web page,
but for a fuller explanation and references to the earlier literature, you
need his paper.
Stefano is coming to the workshop, and is interested in learning how he needs
to update his framework.
Web site logical path:
[MIRA]
[MIRA workshops]
[Dublin workshop]
[programme]
[this page]