A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features

Keshkeh, Kinan (2022) A Machine Learning Classification Approach To Detect Tls-Based Malware Using Entropy-Based Flow Set Features. Masters thesis, Perpustakaan Hamzah Sendut.

Download (1MB) | Preview


As internet encryption has grown to safeguard users’ privacy, malware has evolved to leverage encryption protocols such as Transport Layer Security (TLS) to conceal its hazardous connections. The difficulty and impracticality of decrypting TLS network traffic before it reaches the Intrusion Detection System (IDS) has driven numerous research studies to focus on anomaly-based malware detection without decryption employing various features and Machine Learning (ML) algorithms. Nonetheless, several of these studies used flow features with low feature importance value and poor capability to distinguish malicious flows, such as the number of packets sent and received in a flow or its duration. Furthermore, the outliers and frequency-based flow feature transformations (FTT) applied to mitigate the poor flow feature have several flaws. This thesis proposes a TLS-based malware detection (TLSMalDetect) approach based on ML classification to address flow feature utilization limitations in related work. TLSMalDetect includes periodicity-independent entropy-based flow set (EFS) features produced by an FFT technique. The efficiency of EFS features is assessed in two ways: (1) by comparing them to the relevant related work’s features of outliers and flow using four feature importance methods, and (2) by analyzing the classification performance in the scenarios with and without EFS features. This study also investigates TLSMalDetect detection performance using seven ML classification algorithms and identifies the one with the highest accuracy.

Item Type: Thesis (Masters)
Subjects: Q Science > QA Mathematics > QA75.5-76.95 Electronic computers. Computer science
Divisions: Pusat Pengajian Sains Komputer (School of Computer Sciences) > Thesis
Depositing User: Mr Hasmizar Mansor
Date Deposited: 04 Mar 2024 01:20
Last Modified: 04 Mar 2024 01:20
URI: http://eprints.usm.my/id/eprint/60044

Actions (login required)

View Item View Item