Network-aware end-to-end data throughput optimization

Abstract

The rapidly advancing optical networking technology allows us high-bandwidth connectivity up to 100Gbps these days. However, the end-users and their applications can only observe a fraction of this available bandwidth capacity due to inefficient transport protocols and other end-system bottlenecks such as disk I/O limitations, processor speed, and NIC restrictions. In this paper, we present a novel network-aware end-to-end throughput prediction and optimization framework which provides us with the best parameter combination (i.e. parallel stream, disk, and CPU numbers) to achieve the highest end-to-end throughput between two end-systems (i.e. clusters, data centers, parallel disk systems) possible. Our experiments show that the model and algorithm we have developed enable us to achieve close-to-optimal end-to-end throughput performance with negligible sampling and prediction overhead.