Improving false alarm rate in intrusion detection systems using Hadoop

Abstract

Intrusion Detection Systems are a vital part of an organization's security. This paper gives an account of the existing algorithms for Intrusion Detection using Machine Learning, along with certain new ideas for improving the same. The paper mainly talks about employing the Decision Tree mechanism for Intrusion Detection and improve it with the distributed file system, Hadoop. Initially a method that uses a dirty-flags to check the consistency of the Decision Tree, which changes with every wrong classification of the system is employed. The wrong classification is identified by a certain user who informs the system about the same and helps it learn. In the further sections, a new method which does not use a dirty-flag, but rather modifies the Key-Value pair in the results of the reduce() function is tested as an improvement to the previous method. The two methods are compared, with the help of the Hadoop Simulation Tool — YARN. The main aim of the paper is to propose the use of the Distributed File System for Machine Learning along with some improvements to the current Hadoop File System, so that it reduces the total Time Taken, when Machine Learning algorithms are employed along with it.