Vibe coding in cybersecurity

Marat Agranovskiy, Maryam Atalaeva, Maria Varfolomeeva, Kirill Vititnev, Victoria Gorbenko, Artem Goroshko, Mariia Dyskina, Aleksandr Kustenkov, Ekaterina Lavrova, Ilya Lystsov, Mikhail Maister, Darya Mironova, David Muradyan, Maria Oprishko, Ekaterina Orlova, Anastasiia Piontkevich, Ekaterina Pismennaya, Vladislav Riabykin, Dmitrii Fedorov, Uma Khasanova, Dmitry Namiot

Abstract


This article presents the results of a large-scale experiment in which teams of students from the M.V. Lomonosov Moscow State University's Computational Mathematics and Cybernetics (CM) program, as part of a course on Internet of Things security, developed network traffic classification programs using large-scale language models (LLMs). The use of machine learning (deep learning) models is a classic approach for intrusion detection systems. Available Kaggle datasets were offered as input data. Along with the datasets, hand-coded models were also available on the same website, serving as baseline examples for comparison with automated solutions. The task involved developing prompts, testing them against several LLMs, and, separately, testing the English and Russian language versions of the prompts. Overall, all experiments conducted produced working LLM models that performed as well as existing hand-coded models in terms of metrics. However, the obtained results required verification, just as the architectural solutions chosen by the LLM were not always optimal and sometimes simply incorrect,