Efficient Identification of Malicious Traffic in TLS Networks Using Machine Learning

Abstract

The increasing implementation of Transport Layer Security (TLS) for encrypted communication, although improving data security and privacy, has posed challenges for network security monitoring. Conventional Network Intrusion Detection Systems (NIDS) exhibit diminished efficacy when cybercriminals employ TLS to obscure illicit activities. This research uses Suricata logs processed based on Flow ID and TLS packets labeled using Suricata rules, resulting in a dataset of 30 million records. This dataset is then used to present a machine learning (ML) method for identifying malicious TLS traffic. To train a hybrid Random Forest (RF) and Long Short-Term Memory (LSTM) model, we extract significant characteristics such as JA3 fingerprints, TLS handshake abnormalities, and statistical flow metrics. The RF-LSTM model surpasses individual machine learning models without incurring computational overhead, with a success rate of $99.9 \%$. The proposed method provides a real-time, scalable, and efficient approach to analyzing TLS traffic in cybersecurity applications.

Access to Document

Other files and links

• Link to publication in Scopus

Fingerprint