GOING BEYOND EXPLICIT KNOWLEDGE FOR IMPROVED SEMANTIC BASED INFORMATION RETRIEVAL

Abstract

This paper proposes a novel model that incorporates word semantics into information retrieval, enabling a whole new range of semantic based retrieval applications. Concepts can be sought instead of simple words, and the retrieval of words that are semantically related to the input keywords is also possible. A novel encoding scheme applied to words creates the groundwork for this model and enables the application of complex semantic operators. One of the most powerful operators introduced by the model is the semantic wildcard, able to fulfill information requests expressed along general-specific lines. For instance, a search for animal# will match any concept that is of type animal, going beyond the explicit knowledge stated in texts. Additionally, a new lexical locality operator is introduced, making possible the retrieval of paragraphs rather than entire documents. Both semantic indexing and paragraph indexing paradigms proposed in the model have been implemented in a system that was tested on questions run against an index of 130,000 documents. A significant improvement was observed over classic keyword based retrieval systems, in terms of precision, recall and success rate.