Peer to peer botnet detection based on node traffic behavior
University of New Brunswick
A botnet, which is created to conduct large-scale illegal activities, has become a serious threat to the Internet. Recently, botnets started to utilize a decentralized structure in their command and control channel, which is a more robust and resilient communication infrastructure. P2P botnets, created based on a variety of P2P protocols, are the most representative decentralized botnets and have caused great loss to Internet users. Although a lot of botnet detection techniques have been developed, the existing P2P botnet detection methods are still limited. In this thesis, we present a novel P2P botnet detection system based on an analysis of network behavior. The proposed detection system consists of three main components: Network Packets Capturing, Node Feature Extraction, and Online Classifier. In this thesis, we explain the proposed algorithms and implementation methods for each component in detail. Moreover, in this thesis we also present two novel combined classifiers that integrate supervised machine learning and unsupervised machine learning techniques. One, called Sequential Combined Classifier aims at further enhancing the detection rate; the other one, called Parallel Combined Classifier aims at detecting unknown P2P botnet traffic. Based on three real-world network traffic trace sets (i.e. Storm trace, Waledac trace, and normal traffic trace), a series of evaluation experiments are conducted and their results are reported in this thesis. Several contributions from the evaluation results include (1) identification of an appropriate time window size that allows to provide a better detection performance when used in system's packets capturing module; (2) optimized configuration for system's online classifier in each time window size; and (3) evaluated the effectiveness of two proposed combined classifiers and verified their ability to improve detection rate or detect unknown botnet traffic. According experimental results, we obtain the detection accuracy of 99.0% and the false positive rate of 0.1%.