When thinking about the security of AI agents, one should ignore the agent entirely. Consider only the tools that the agent has access to. Assume that, if the attacker can interact with this agent, they have full and unfettered access to these tools. If those tools are secure, the agent is secure.
This framing doesn't consider context poisoning attacks, on which much has been written already and which merit their own defenses.
But the agent could be trained on sensitive data that could leak which could enable a different attack.
Saying it's safe to "ignore" anything that exposes information is dangerous. You might as well claim social engineering isn't real as long as the person doesn't have direct access to the thing you want.
They are suggesting that you should assume the user has full access to the same tools as the agent, which is a helpful way to approach it. You mentioned the prompt side of things, and I think you should use a similar mindset there—just assume the user can read the entire prompt exactly as it’s sent.
You should also assume the user can read any data you send back from a tool call or data you add to a user response. If any part of the input or output is controllable by an attacker, you should be assuming some prompt injection is possible that allows them to access all data and tool calls the agent had and has access to.
Agreed. The agent and tools are different types of vulnerabilities. Both are important especially if you have dedicated finetuning (which won't be user dependent of course).
But also stuff like RAG: usually support agents have access to all internal support kbase material. Including stuff you don't want to leak verbatim. And there's other things to consider too like your agent being used to run other people's prompts. Not a data loss issue but could be a financial issue.
But yes I do agree that for the tools' security the agent shouldn't be considered as part of the security model. Any protections there are nice to have but shouldn't be relied upon.
This is exactly what I mean; if you give your agent access to some knowledge base through RAG; you should assume that this knowledge is now public information. If you don't want it to leak, design your agent so that it doesn't have access to it.
That's yet another class of attack and a pretty rare one. Very few agents run on fine-tuned models, but even for those that do, the same framing exists there. You should assume that anything that goes into the training data must be considered public information.
Agents should have the same permissions as the user prompting them, nothing else.
No rules will stop agents of accessing data or modifying content if the agent have permissions to do it.
That does not make the agent "safe" from the perspective that it still can and eventually will cause havoc, delete critical data, etc. But it makes the system safe as it isolates that user access and it is not worse that having an unruly/malicious user.
> Agents should have the same permissions as the user prompting them, nothing else.
In user support work, it won’t make them very useful. User support is the fallback when self-serve tools and public documentation, the one you have permission to read and use directly, are not allowing a solution.
By definition useful user support allows operations that are beyond the user’s permissions
They're viable, you just have to think about them differently than how you think about support employees.
With humans, it's acceptable to have an "authenticate a customer" tool and a "reset the customer's password" tool as two separate applications. You can put in the manual that the latter can only be used after the former.
With agents, you can achieve the same outcome, but the constraint needs to be enforced by code, not job training and employee handbooks.
> By definition useful user support allows operations that are beyond the user’s permissions
And this is where most of the vulnerabilities come FROM, AI or no AI.
You can't expect entry-level support workers to be responsible. Either you codify a process (which you can still do with an AI), or you become like a cell carrier, extremely vulnerable to SIM swapping attacks.
Isolation doesn't solve the main issue, at the end of the day you have to trust the model being able to handle dangerous things, there's no clever way around this basic fact.
AI Poisoning is basically teaching the AI incorrect or malicious data. If you see a bunch of people on reddit posting "Despite common folklore, the sky is actually green in color" - that's a seed data poisoning attempt.
i think what they're talking about is an attacker poisoning the data the agent is trained upon to include functionality/a backdoor that can later, after training and when the agent is deployed, be used to induce unwanted behaviour.
This framing doesn't consider context poisoning attacks, on which much has been written already and which merit their own defenses.