The main theme of the workshop concerns future challenges in Expertise Retrieval. The workshop is not meant to be a “second TREC Enterprise workshop”, focused on a specific expert finding scenario. Instead, our aim is to broaden the topic area and to seek for potential connections with other related fields.
The workshop would be organized around the following major issues:
Evaluation
Much of the expert finding work has been validated using the W3C collection from the 2005 and 2006 editions of the Enterprise Track. However, TREC 2007 featured a new collection (CSIRO), with different characteristics. In addition to TREC datasets, is it timely to research datasets of other sources? For example, the intranet of a university, or the CiteSeer dataset. Given these collections, the original task definitions may need to be revised and/or new tasks should perhaps be considered. Are our tasks and corresponding metrics generalizable to these new collections? To other collections? Do we have a standardized protocol for evaluating expertise retrieval?
Social aspects
Expertise finding is often addressed by uncovering associations between people and topics; commonly, a co-occurrence of the name of a person with topics in the same context is assumed to be evidence of expertise. In addition, there have been attempts to use social networks of people as another source of evidence. These give rise to a number of questions: (i) What other factors could play a role, are there other sources of evidence? (ii) How can methods from social network analysis be applied to ER? Further, with sites such as LinkedIn attracting many millions of users, how can information mined from such social networks be integrated with topically oriented ER methods?
Links with knowledge management
While ER is relatively new in IR, the task of locating experts and maintaining expertise profiles within an organization dates a while back within the Knowledge Management community. How do recent algorithmic advances contribute to the KM field? Are these new techniques and/or needs within the KM field that could direct continued developments in expertise retrieval? What other fields may benefit from the developed technology, e.g., medicine?
Bring in the user
Research in the field of ER has been carried out by abstracting the user away. This abstraction has been incredibly successful in enabling ER to advance rapidly. However, a greater focus on the user (modeling the user’s background and type, tasks, scenarios, personas, result presentation) would possibly enable further advances, and validate the success of research to date. What are the information needs and search goals of users of ER systems? Furthermore, how should they be rendered at the interface?
Expertise Retrieval on the Web
Current expert search is limited to a particular domain or intranet. A much more challenging task would be to perform expert finding on the web. How do we deal with issues of scalability and data quality in that case? What type of evidence (of expertise) would users of ER systems find acceptable there? How should we deal with ambiguity (which will be a big issue on the web)? And with multilinguality?