An anomaly detection framework for DNS-over-HTTPS (DoH) tunnel using time-series analysis

dc.contributor.advisorLashkari, Arash Habibi
dc.contributor.authorMontazeriShatoori, Mohammadreza
dc.description.abstractDomain Name System (DNS) as a network protocol is vulnerable to several security loopholes. To cover up some of the vulnerabilities in DNS, a new protocol, named DNS over HTTPS (DoH), is created to improve privacy, and protect from various persistent attacks. The DoH protocol encrypts the DNS requests for the DoH client and sends it through a tunnel to prevent eavesdropping and man-in-the-middle attacks. This research work comprehensively studies these security vulnerabilities, proposes a taxonomy of potential DNS attacks, analyzes the security aspects of DoH protocol, and classifies DNS attacks that are applicable on DoH. To achieve these objectives, we simulated DoH tunnels. The simulated environment covers different DoH deployment scenarios includes DoH within an application, DoH proxy on the name server in the local network, and DoH proxy on a local system as suggested in RFC8484. In this research, we captured malicious and benign DoH traffic and analyzed it as a two-layered approach to classify benign and malicious traffic at first layer and characterize DoH traffic at second layer. It is observed that for statistical features, Random Forest (RF) and Decision Tree (DT) give the best classification and characterization results among prominent machine learning and deep learning classifiers at first and second layer, respectively. Moreover, for time-series features, long short-term memory (LSTM) turns out to be the best classifier for DoH traffic classification and characterization at first and second layers, respectively. The experimental results indicate that while DoH can be abused to create covert communication channels, the proposed solution is sufficient to detect these channels in a timely manner.
dc.description.copyright© Mohammadreza MontazeriShatoori, 2021
dc.format.extentxi, 93 pages
dc.publisherUniversity of New Brunswick
dc.subject.disciplineComputer Science
dc.titleAn anomaly detection framework for DNS-over-HTTPS (DoH) tunnel using time-series analysis
dc.typemaster thesis Science of Computer Science of New Brunswick
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
1.54 MB
Adobe Portable Document Format