Capabilities and limitations of attack classification in encrypted traffic by machine learning methods

Abstract

This paper investigates the capabilities and limitations of applying machine learning methods for computer attacks classifying within encrypted network traffic. This research is relevant due to the widespread adoption of encryption, which complicates the operation of traditional traffic analysis methods. The article proposes a comprehensive data model incorporating statistical flow characteristics, packet sequences, and payload byte distribution. A comparative analysis of machine learning algorithms: Random Forest, Extra Trees, Decision Tree, AdaBoost, and KNN was conducted on three public datasets: CIC-IDS2017, TON IoT, and USTC-TFC2016. Random Forest and Extra Trees algorithms demonstrated the best results, Extra Trees showing slightly higher stability. Hyperparameter optimization for the Extra Trees model was performed, proving its effectiveness when applied within a single dataset. A key limitation of the proposed approach is the model's low generalization capability. Cross-dataset testing revealed a significant decrease in classification performance, indicating a dependency of the results on the specific network environment. Applied methods to improve generalization, including feature filtering, neural network implementation, and domain adaptation, did not yield a sufficiently robust or universally applicable solution. The study concludes that machine learning models are applicable for effective detection and classification of computer attacks within a single network environment. However, the portability of trained models between different network environments remains limited and requires further research.