Cracks in the code: Cybercriminals target AI models to scale attacks

In the report, Cisco notes that the rapid adoption of AI-enabled technology has led to an expanded attack surface and additional security hazards, complicating the risk and threat environments.

“The AI ecosystem’s reliance on shared models, datasets, and libraries expands the attack surface into the AI supply chain,” reads the report in part.

“Adversaries targeting an AI system’s building blocks and related components can be particularly concerning due to their potential for widespread impact across multiple downstream applications and systems.”

Among the highlighted threat fronts are direct security risks to AI models, systems, applications, and infrastructure, the emergence of AI-specific attack vectors, as well as the use of AI to automate and professionalise threat actor cyber operations.

The report further notes that while these threats might be on the horizon for 2025 and beyond, attacks that emerged in 2024 mainly featured AI enhancing existing malicious tactics rather than aiding in creating new ones or significantly automating the kill-chain.

But while most AI threats and vulnerabilities are low to medium risk by themselves, the analyst argues that combining them with the increased velocity of AI adoption and the lagging development of accompanying security practices and safeguards will ultimately grow organisational risks.

“Attackers are focused on targeting infrastructure supporting AI systems and applications, particularly on the unique vulnerabilities of AI deployment environments,” notes Cisco.

“Compromises in AI infrastructure could result in cascading effects that can impact multiple systems and customers simultaneously, and attackers can proceed to conduct additional operations targeting model training jobs and model architecture as well as models’ training data and configurations.”

Among the prominently featured AI-specific attack vectors that involve direct compromise of AI infrastructure are direct prompt injection and jailbreaking.

Prompt injection refers to a technique used to manipulate model responses through specific inputs to alter its behaviour and circumvent an AI model’s in-built safety measures and guardrails, usually to re-task a Large Language Model (LLM) application to conduct some other task.

Jailbreaking, on the other hand, is a prompt where an attacker provides inputs that cause the model to disregard its alignment or safety protocols entirely, and is particularly rampant in chatbots.

In indirect injections, attackers focus on providing compromised source data, such as malicious PDFs or web pages, or even non-human-readable text to input malicious instructions designed to manipulate LLM responses.

“Indirect prompt injections are more difficult to detect because the attack does not require direct access to an AI model, meaning they can bypass traditional prompt injection defenses, and the threat can persist in systems over time,” cautions Cisco.

Other AI-powered attacks include the extraction of training data from deployed AI models, which risks revealing sensitive or confidential information that was used to train the model.

Attackers can also tamper with data used by AI models, compromising the integrity of the model’s outputs and potentially leading to incorrect decisions or harmful actions.

The report also cites data poisoning campaigns, which involve threat actors injecting malicious samples into training datasets to introduce weaknesses or backdoors into AI models, as well as model extraction and model inversion, where attackers try to steal or duplicate a machine learning model by repeatedly querying it and using the responses to train their own copy.

Leveraging AI as a tool for attacks

In using AI tools to scale attacks, attackers have been found to deploy the technology for social engineering as well as for automating malicious activities.

By combining these two components, the threat actors have been found to increase their success rates exponentially, as they can produce higher volumes of socially engineered lures that are of higher quality with the assistance of LLMs and generative AI.

As such, the pundits argue, phishing and other social engineering techniques such as vishing (AI-generated voice cloning) and deepfakes are set to continue improving with AI’s assistance, while spam and phishing detection races to catch up.

Cybercriminals have also been found to be attempting to leverage chatbots to assist in malware development and task automation so as to improve their attack success rates.

→ [email protected]