Language models for information retrieval

Abstract

One of the major challenges in the field of information retrieval (IR) is to specify a formal framework that both describes the important processes involved in finding relevant information, and successfully predicts which techniques will provide good effectiveness in terms of accuracy. A recent approach that has shown considerable promise uses generative models of text (language models) to describe the IR processes. We briefly review the major variations of the language model approach and how they have been used to develop a range of retrieval-related language technologies, including cross-lingual IR and distributed search. We also discuss how this approach could be used with structured data extracted from text.