An asymptotically optimal data compression algorithm based on an inverted index

Abstract

Summary form only given. An alternate approach to representing a data sequence is to associate with each source letter, the list of locations at which it appears in the data sequence. We present a data compression algorithm based on a generalization of this idea. The algorithm parses the data with respect to a static dictionary of phrases and associates with each phrase in the dictionary a list of locations at which the phrase appears in the parsed data. Each list of locations is then run-length encoded. This collection of run-length encoded lists constitutes the compressed representation of the data. We refer to the collection of lists as an inverted index. While in information retrieval systems, the inverted index is an adjunct to the main database used to speed up searching, we regard it here as a self-contained representation of the database itself. Further, our inverted index does not necessarily list every occurrence of a phrase in the data, only every occurrence in the parsing. This allows us to be asymptotically optimal in terms of compression, though at the cost of a loss in searching efficiency. We discuss this trade-off between compression and searching efficiency. We prove that in terms of compression, this algorithm is asymptotically optimal universally over the class of discrete memoryless sources. We also show that pattern matching can be performed efficiently in the compressed domain. Compressing and storing data in this manner may be useful in applications which require frequent searching of a large but mostly static database.