Browsing by Author "Alkhateeb, Ehab"

Now showing 1 - 1 of 1
  • Thumbnail Image
    Item
    Unmasking stealthy threats: Techniques for identifying and analyzing obfuscated malware
    (University of New Brunswick, 2024-08) Alkhateeb, Ehab; Ghorbani, Ali A.; Lashkari, Arash Habibi
    In the ever-evolving domain of cybersecurity, the challenges of countering obscured malware and crafting effective Anti-Virus (AV) solutions are formidable. This struggle is particularly evident in packed malware, where malicious actors employ encryption and sophisticated techniques to conceal their payloads, thereby circumventing detection by AV scanners and security analysts. Recent studies reveal that addressing both known and unknown packers poses a significant challenge, often due to insufficient datasets and a reliance on raw features. Furthermore, there is a notable absence of a comprehensive approach for unpacking, which involves identifying packers to streamline the overall process and implementing both profile and generic unpacking methods. This thesis introduces an innovative malware packer classifier, meticulously designed to excel in the identification of packer families and the detection of previously unknown packers in real-world scenarios. Our approach relies on sophisticated feature engineering techniques, involving multiple layers of analysis to extract crucial features used as inputs for the classifier. These features encapsulate the intricacies of packed malware, enabling our classifier to reveal their concealed intentions. Furthermore, to enhance packer identification performance, we have diligently curated a dataset comprising precisely packed samples, ensuring the highest level of data quality and relevance to real-world threats. The proposed packer identifier distinguishes itself with high accuracy in discerning a diverse array of known packers with an accuracy of 99.60%, including previously unidentified packers with an accuracy of 91%. It achieves this while maintaining operational efficiency and effectiveness, addressing a critical need in cybersecurity. Additionally, we have introduced a unified unpacking approach leveraging our identification methodology, to minimize unpacking overhead. This involves a strategic decision-making process to determine whether to employ a profile-based or generic unpacking method. Moreover, we have proposed a script-based profile unpacking technique and an Intel PIN tool for generic unpacking. By advancing the state-of-the-art in malware packer identification and unpacking, this research significantly contributes to fortifying defenses against the persistent threat of obfuscated malware, ultimately enhancing the security of digital ecosystems.