Artificial Intelligence in social engineering cyber threats: risk analysis, experimental assessment of detection tools, and countermeasure recommendations

Abstract

The development of generative artificial intelligence is changing the operating model of social engineering attacks: the level of automation, personalization, and message realism is increasing. The article outlines the ways generative models are applied at each stage of a social engineering attack. It argues that the key change is not in improving the quality of an individual phishing message, but in moving toward scalable adaptive campaigns, where each target receives unique content and subsequent actions are based on their response. An experimental study was conducted on methods for detecting socially-engineered content using a text corpus of messages, a control subset of messages generated by AI tools, as well as audio and video datasets. Two heuristic baseline approaches were compared with four machine learning models; the effects of class imbalance, feature space dimensionality, and decision threshold were investigated. For classic phishing, the first heuristic baseline approach shows a moderate harmonic mean of precision and recall, while gradient boosting and a recurrent neural network with an attention mechanism deliver significantly higher performance, with the best result coming from the recurrent network. For AI-generated messages, the harmonic mean of precision and recall for the heuristic baseline approach drops sharply, indicating degradation of traditional linguistic indicators. When moving from a balanced class ratio to a heavily imbalanced one, the accuracy of gradient boosting drops significantly, recall decreases moderately, and the area under the precision-recall curve shrinks. A two-level cascade detector architecture is proposed, which provides high detection recall with a very low false positive rate and minimal average processing delay. Practical recommendations for multi-level protection and risk management of AI systems have been outlined.