Alzahrani, Abdullah2023-03-012023-03-012016Thesis 9913https://unbscholar.lib.unb.ca/handle/1882/14564Along with increasing security measures in Android platforms, the amount of Android malware that use remote exploits has grown significantly. Using mobile botnets, attackers concentrate on reliable attack vectors such as SMS messages. Short Message Service (SMS) has been increasingly targeted by a number of malicious applications ("apps") that have the ability to abuse SMS features in order to send spam, to transfer command and control (C&C) instructions, to distribute malicious applications via URLs embedded in text messages, to send text messages to premium-rate numbers, and to exploit smartphones. Efficient detection and defence techniques that use filtering and blocking methods for SMS botnets is therefore an urgent necessity. Unfortunately, most botnet detection solutions proposed so far are reactive; that is, they require a large amount of data in order to effectively generate signatures and filtering rules to differentiate between normal and malicious SMS messages. By using proactive approaches such as a multi-agent system, agents can monitor certain environments and report abnormal behaviour in order to protect user data. In this thesis, we propose an SMS-based botnet detection framework using intelligent agents that are used to detect malicious SMS messages and monitor smartphone resources which are typically targeted by SMS botnet attacks. The proposed detection framework is based on a multi-layer model which consists of three modules and intelligent agents. The first is an SMS signature-based detection module which can be used to combat SMS botnets, in which we first apply pattern-matching detection approaches for incoming and outgoing SMS text messages, and then use rule-based techniques to label unknown SMS messages as suspicious or normal. The second module, an anomaly-based detection module, employs unsupervised learning techniques, using clustering algorithms to group SMS messages into four class labels and to classify reported text messages to one of those four classes. The module also uses a robust and efficient behavioural profiling analysis to detect whether there are any correlations between classification results and alerts from profiling analysis. Rule-based correlations are used to label SMS messages as either normal or malicious. The third module is a defence module that can be used as a more proactive approach which directly generates signatures and rules in order to protect Android smartphones from abuse by SMS botnets. This module is used to generate signatures of malicious SMS messages, to update phone number blacklists, to analyze malicious applications and to send feedback to Android smartphones so that the user can take action. Finally, a multi-agent system that can be used to observe Android mobile devices and to interact with service provider agents in order to detect malicious applications and SMS botnet activities on Android mobile devices. We have developed an intelligent and proactive framework that scans incoming and outgoing text messages, monitors Android resources and observes user usage that includes user connectivity time. The framework creates a user profile that is used to perform behavioural profiling analysis in order to identity malicious SMS and cut the C&C Channel. The proposed framework has been implemented using JADE agents. We demonstrate the capability of the multi-agent system, signature-based detection, anomaly-based detection module, and defence module in accurately detecting SMS botnets, we conduct different experiments in three phases. In the first phase, we focus on evaluating the efficiency of the SMS signature detection module in Android devices. This module was evaluated using over 12,000 test messages. It was able to detect all 747 malicious SMS messages in the dataset (100% detection rate with no false negatives). It also flagged 351 SMS messages as suspicious. A comprehensive performance analysis of the anomaly-based detection module is conducted in the second phase. The detection performance of the anomaly-based detection module has an average accuracy of 95% and an average of false negative rate is 3.95% on applied datasets. After having studied the performance of each module individually, in the last phase, we analyzed the overall performance of the proposed framework and provided a thorough analysis of JADE agents monitoring mechanism after demonstrating the capability of each module individually. We used approximately 60,000 test messages to evaluate the proposed framework. The signature detection agents reported 165 malicious SMS messages and 3,081 suspicious SMS messages. The anomaly-based detection module labelled 941 SMS messages as malicious.text/xmlxvi, 197 pages : illustrationselectronicen-CAhttp://purl.org/coar/access_right/c_abf2Android (Electronic resource) -- Security measures.Intrusion detection systems (Computer security)Text messaging (Cell phone systems) -- Security measures.Malware (Computer software)Intelligent agents (Computer software)Mobile computing -- Security measures.Mobile communication systems -- Security measures.Computer networks -- Security measures.Computer security.doctoral thesis2020-04-20Ghorbani, Ali(OCoLC)1151314170Computer Science