LTP2007: Word of the day

This is the suggested topic for the course Language Technology Project 2007.

Introduction

Word of the day is an application which tracks topics in text feeds, for example news or blogs. It requires the basic task of counting phrases but includes many challenges which we will address in this project:

• How can we identify and collapse similar terms like Democrat, Democrats and Democratic?
• Can we split terms which refer to different entities? For example, Armstrong can be a cyclist, musician, astronaut, location, company, ...
• What is the best/most attractive way to visualize the term counts? Examples: term categories, using different font size, graphs, combined graphs, heat maps, ...

Task

Develop a word of the day application for a text feed that is supplied by the teachers of the course. You may decide what language to work with (prefably Dutch or English texts). You will also make the main design decisions. You will use available data streams from news and blogs. The system to be developed will be put online and continue to track live text feeds. There are plans to link it to existing online demos of our department, like MoodSignals and the Dutch election news search.

Modules

Students will work in seperate groups on different modules. Groups will consist of two students. Some modules can be developed by an individual.

• Term identification module: Identify interesting terms in texts.
• Term categization module: Build clusters of marked terms in texts. Label the clusters. Examine ways to collapse similar terms and split ambiguous terms.
• Text processing module: Convert texts in different formats to single format. Identify the language of the texts. Mark the terms in the text. Suggest useful snippets for the terms.
• Visualization and web module: Look for methods and software for visualizing time series. Use available software or create your own implementation. Integrate available views in website. Create online search facility. Propose interesting design features.

For this course, relevant background knowledge consists of knowledge of natural language processing plus general programming experience. Some experience with programming web services or visualization software is useful for the related modules.

The final system will be evaluated by comparing it with other available word of the day applications. Does the new application offer the same views as the earlier applications? Does it offer interesting other views? What are the response times for online queries?

Literature, tools and examples

• Unni Cathrine Eiken, Anja Therese Liseth, Hans Friedrich Witschel, Matthias Richter, Chris Biemann, Ord i dag: Mining Norwegian Daily Newswire. FinTal 2006. http://wortschatz.uni-leipzig.de/~fwitschel/papers/ordIDag.pdf
• P. Rayson and R. Garside. Comparing corpora using frequency profiling. In Proceedings of the ACL-2000 Workshop on Comparing Corpora, Hong Kong, pp. 1-6, 2000. http://www.comp.lancs.ac.uk/computing/users/paul/publications/rg_acl2000.pdf