Dynamic Self-assembling Petaflop Scale Clusters

Abstract

High Performance Computing (HPC) has been studied and used in the scientific community for decades. The Message Passing Interface was first introduced in 1992. Similarly, commercial businesses have been relying on High Throughput Computing (HTC) for the past two decades. Mapreduce platforms became popular with the advent of Very Large Databases (VLDBs) and Big Data. We are now seeing the convergence between HPC and HTC to provide faster and cheaper parallel computation. The emergence of MPI as a scalable and viable parallel platform along with the acceptance of Mapreduce to tackle large data sets now opens the door to a host of new applications particularly in biomedical, public health, scientific, and health informatics research. This convergence is making it possible to have every device be a parallel node. In this paper we explore this convergence and a method for creating dynamic self-assembling clusters using commodity hardware and mobile devices.