SAGE: Geo-Distributed Streaming Data Analysis in Clouds

Abstract

The continuous growth of sensor networks, stock exchanges, climate monitoring or scientific applications produces new streaming data at increasing rates. Managing and processing such data, sometimes generated from multiple geographical locations, raises important challenges as it requires real-time processing or data aggregation. Conventional solutions like DBMS, MapReduce or dedicated solutions adopting single-located environments fail to meet the demands required for processing the Geo-distributed streaming data. Public clouds like Azure, with data centers spread around the globe, offer the infrastructure which can handle such a processing. Our approach, proposes a service-oriented cloud architecture for performing the stream analysis, by composing services which are distributed among multiple cloud data centers. Hence, the computation is moved towards the multiple data sources exploiting the geographical data locality. The initial results showed good scalability of the approach, reaching 1000 cores in the Azure cloud, and performance improvements compared to single location processing of a factor of 3.3.