Intrusion detection using machine learning

Abstract

Intrusion detection has been an issue since the beginning of computer network technology in the 1980's. Researchers from around the globe are proposing new techniques such as firewalls and encryption to tackle this issue. However, the problem is still a top challenging issue in the IT world with Intrusion Detection Systems (IDS) attracting much attention of researchers today. This thesis develops and implements an agent based robust Intrusion Detection System (IDS) using a hybrid approach to machine learning algorithm. Machine learning is a comparatively new approach, which is based on statistical driven computing concepts that animate search engines and can enable intrusion detection. This research includes an extensive literature review on machine learning algorithms, an integrated multiple (heterogeneous) data from various sources and the selection of a subset of data relevant to the analysis task. The proposed hybrid algorithm outperformed existing methods for intrusion detection. Association Rule Mining (ARM) algorithm and rule based classification were used to generate rules for IDS based on meta learning technique. An evaluation of the competency ARM algorithms is conducted to propose a novel IDS model. Throughout the evaluation process, this research verified the statistical influence on dataset characteristics implementing the meta learning process. While none of the algorithms performed significantly better than others, this initial experiment made a contribution to knowledge by exploring statistical dominations in datasets that would lead a user to pick an appropriate ARM algorithm for generating rules without any problem. The research further carried out performance evaluation of rule-based classification, which could be implemented in the IDS model. Following an extensive literature review on several rules based algorithms, this research selected PART (partial decision tree) and C4.5. At this point the thesis proposed a hybrid approach to select the most significant attribute utilizing entropy and three different types of correlation coefficients. The approach enhanced the accuracy of the algorithms significantly and rolled back the processing time. An emphasis on the improvement of classification based algorithms utilizing Fano's Inequality was made. Using this theorem, attributes of a dataset were discriminated in such a way that the processing time for rule generation was remarkably reduced without compromising rules accuracy. One of the observations during this research showed that classification based algorithms would be a better choice for the proposed IDS model. The thesis presented the proposed novel IDS model. The IDS model deployed classification based algorithms on the basis of experiences from previous experiments of this thesis, a novel approach of attribute selection from dataset and the meta learning process to detect intrusions. Experimental justification was conducted in several ways. All the experiments showed the robustness, efficiency and competency of the proposed IDS model. Expectation of this research was to make significant contributions to the Machine Learning community in order to be able to make more accurate and quick decision to detect intrusion and perform accordingly in more efficient way thus contributing significant insights in increasing the security of network environments, which in turn could support entire system user communities.