Recent findings from global cybersecurity leader, Barracuda Networds, sees security researchers reveal how attackers are targeting and manipulating companies’ AI assistants and their security features.
Threat actors are now embedding malicious prompts in emails, which an AI assistant, like Microsoft 365’s Copilot, may inadvertently ingest when scanning for context. This could lead to the silent exfiltration of sensitive data or the execution of malicious commands.
Another key risk is RAG (Retrieval-Augmented Generation) poisoning, where manipulated emails corrupt an AI assistant’s memory, causing it to provide false information or make incorrect decisions. Attackers are also manipulating the AI components of security platforms, with the potential to trigger automated help desk tickets or sensitive data autoreplies.
The report also highlights “confused deputy” attacks, where an AI agent with high-level access performs unauthorised tasks on behalf of a lower-privileged user. To combat this, experts say email defences must become “agent-aware”, incorporating LLM-aware filtering and zero trust AI execution to validate all actions and prevent manipulation.
The report was authored and researched by a team of experts and professors. It stated: “Security researchers are now seeing threat actors manipulate companies’ AI tools and tamper with their AI security features to steal and compromise information and weaken a target’s defences. AI assistants and the Large Language Models (LLMs) that support their functionality are vulnerable to abuse.”