A cloud/edge computing streaming system for network traffic monitoring and threat detection

Abstract

The unyielding trend of increasing cyber threats has made cyber security paramount in protecting personal and private intellectual property. To provide a highly secured network environment, network threat detection systems must handle real-time big data from varied places in enterprise networks. In this paper, we introduce a streaming-based threat detection system that can rapidly analyse highly intensive network traffic data in real-time, utilising streaming-based clustering algorithms to detect abnormal network activities. The developed system integrates the high-performance data analysis capabilities of Flume, Spark and Hadoop into a cloud-computing environment to provide network monitoring and intrusion detection. Our performance evaluation validates that the developed system can cope with a significant volume of streaming data in a high detection accuracy and good system performance. We further extend our system for edge computing and discuss some key challenges, as well as some potential solutions, aiming to improve the scalability of our system.