Vibe coding in cybersecurity

Abstract

This article presents the results of a large-scale experiment in which teams of students from the M.V. Lomonosov Moscow State University's Computational Mathematics and Cybernetics (CM) program, as part of a course on Internet of Things security, developed network traffic classification programs using large-scale language models (LLMs). The use of machine learning (deep learning) models is a classic approach for intrusion detection systems. Available Kaggle datasets were offered as input data. Along with the datasets, hand-coded models were also available on the same website, serving as baseline examples for comparison with automated solutions. The task involved developing prompts, testing them against several LLMs, and, separately, testing the English and Russian language versions of the prompts. Overall, all experiments conducted produced working LLM models that performed as well as existing hand-coded models in terms of metrics. However, the obtained results required verification, just as the architectural solutions chosen by the LLM were not always optimal and sometimes simply incorrect, for example, in terms of data leakage. This still leaves room for programmers (ML engineers) to work.