Distributed Arrays – An Algebra for Generic Distributed Query Processing

Abstract

We propose a simple model for distributed query processing based on the concept of a distributed array. Such an array has fields of some data type whose values can be stored on different machines. It offers operations to manipulate all fields in parallel within the distributed algebra. Distributed arrays rest on a given set of data types and operations called the basic algebra implemented by some piece of software called the basic engine. It can create and delete objects of such types and evaluate expressions (queries) over objects, constants, and operations of the basic algebra. Data types comprise simple types such as integers or strings, but also relations or indexes on disk or in memory. Hence a basic engine provides a complete environment for query processing on a single machine. We assume this environment is also extensible by types and operations. Operations on distributed arrays are implemented by one basic engine called the master which controls a set of basic engines called the workers. It maps operations on distributed arrays to the respective operations on their fieelds executed by workers. The distributed algebra is completely generic: any type or operation added in the extensible basic engine will be immediately available for distributed query processing. To demonstrate the use of the distributed algebra as a language for distributed query processing, we describe a fairly complex algorithm for distributed density-based similarity clustering. The algorithm is a novel contribution by itself. Its complete implementationmis shown in terms of the distributed algebra and the basic algebra. As a basic engine thenSecondo system is used, a rich environment for extensible query processing, providing useful tools such as main memory M-trees, graphs, or a DBScan implementation.