High Throughput Hierarchical Heavy Hitter Detection in Data Streams

Abstract

Detecting heavy activity aggregation in data streams is a critical task for many networking, data base and data-mining applications. The aggregation points often belong to hierarchical domains (e.g. IP domain, XML data tree, etc.). These aggregation points are referred to as hierarchical heavy hitters. The hierarchical domains is usually very large with respect to both the number of aggregation points and the number of levels in the hierarchy. Due to the huge amount of intermediate data to be maintained and the dependency between hierarchy levels, it is challenging to detect hierarchical heavy hitters in very large hierarchical domains at high throughput. In this work, we propose a novel Sketch-counter-hybrid algorithm that decomposes the online hierarchical heavy hitter detection to independent and parallel online heavy hitter detection at each hierarchy level. We then propose a fully pipelined architecture suitable for FPGA implementation to accelerate the algorithm. We use IP network monitoring as an example of application. The post place-and-route results on a state-of-the-art FPGA show high throughput and scalability. Our architecture achieves 123 Gbps throughput assuming minimum-sized IPv4 packets while supporting the entire 32-bit IPv4 hierarchy on a single FPGA device. It sustains 100+ Gbps throughput while supporting various hierarchy sizes, stream sizes and accuracy requirements. Our architecture achieves much higher throughput than other techniques when accelerating Sketches of similar number of rows and columns, and counter sizes.