Evaluation issues of query result ranking for semantic search

Abstract

Abstract Application of semantic in information retrieval is a dynamically developing area. Nowadays, elements of semantic search are used in popular systems such as Microsoft Azure, Abbyy Intelligent Search, Google Search with BERT. Using sematic search, it is possible to obtain documents that contain exact meaning instead of set of words. But Lucene is still one of the most popular libraries for search purpose and it has its own syntax for fuzzy, wildcard, proximity and other modifiers for queries. To evaluate precision and recall of search the authors have created a list of queries and divided it into groups according to a query type. The article contains results of this investigation for semantic search with metagraph knowledge base in comparison to Lucene with the same morphological analyzer. The quantity of documents for two types of search may be the same but ranking should be different. Ranking of queries is another issue and its evaluation is not a trivial task. In this article the authors applied Levenstein distance but then proposed a new method for comparison of ranking given by different search engines. All results were obtained on Open Corpora text corpus.