Wiki for WebCLEF 2007 | Main / HomePage

Welcome to the wiki for WebCLEF 2007.

Here's an initial task description (version October 2006).

The task mixes aspect of WiQA 2006 and a vertical search task; in the task we envisage, the "restricted" domain is cultural heritage, and the data would consist of Wikipedia combined with crawl of cultural heritage institutes, possibly extended with the top N pages from a general web search engine.

Key starting points in the CLEF context:

The task should match some real-world information need
Multi- and cross-lingual from the start, not as an afterthought
Relevance judgements, assessments etc. resulting from the task should be re-usable

Our user is an expert collecting a comprehensive dossier on a specific topic. Real-world scenario: a librarian asked to gather a dossier on a topic. A topic can be an notable person, organization, movement, event, etc. A topic definition consists of the name of person/org/etc. and description of the "kind of information" the expert is interested in. This description is provided as a usual narrative and, in addition, an a set of examples: pieces of text and/or web pages that sufficiently indicate the direction of expert's interest. Examples can be "good texts" the topic in question, or "good texts" over other, explicitly specified topic.

E.g., if an expert is interested in Catherine the Great of Russia http://en.wikipedia.org/wiki/Catherine_II_of_Russia as an art patron, rather than in her achievements as an empress, he might provide as examples some texts/web pages describing her or some other person's patronage. An expert collecting information on Michaelangelo's contribution to mechanics might provide as examples a few texts about other persons' contributions to this field.

Informally: I want to know about X everything that looks like Y

Taking as input the title of a topic, the narrative and the "examples", the system should compile a dossier consisting of two types of objects:

SNIPPET: information snippets: pieces of text from a given collection satisfying the expert's information need
PAGE: id's of the documents in the collection, each document being "mostly about the topic", i.e. can be included in the dossier as a whole.

The collection: Wikipedias and a substantial Web crawl. In particular, the crawl will include top N web pages returned by Google for the topic as a query (those will be explicitely indicated to the systems).

System's output: the list of type/docid/textbit where

type is SNIPPET or PAGE
docid is docid
textbit is a piece of text that either constitutes the SNIPPET, or serves as a description of a PAGE sufficient for the assessor (convincing enough) to add the page to the dossier.

The max. sum of the sizes of all textbits for a given topic is specified in the topic (i.e., the expert says "I can only look at that much data"). The sizes of individual textbits is up to systems.

Assessments: something like marking up "good" text spans in the output of a system. What about repetition? Evaluation: yield, i.e., the total side of "good" text spans. Maybe, should be different for SNIPPET and PAGE.

The eventual judgement of systems' output should be performed by topic creators. We might need double assessments, though.