AI red teaming for Copilot agents: what we actually test

A traditional penetration test looks at networks, applications and infrastructure. That still matters. But an AI agent adds a new kind of attack surface: it follows instructions written in plain language, and it can't always tell your instructions from an attacker's.

Here's what an AI red team engagement focuses on, and why.

1. Indirect prompt injection

Direct injection is the obvious one: a user types "ignore your previous instructions". Indirect injection is more dangerous. The malicious instruction hides inside content the agent reads on its own, like an email, a shared document, a web page or a ticket.

If your agent summarises inbound emails, an attacker doesn't need access to your tenant. They only need to send you an email. We test every content source the agent ingests.

2. Oversharing through grounding

Agents grounded on SharePoint respect permissions. That's good news, and also the problem. If a sensitive folder is shared with "Everyone except external users", the agent will happily surface it. We probe for salary data, HR cases, board papers and credentials that are technically accessible but never meant to be found.

3. Excessive agency

Once an agent can act, by sending messages, creating records or triggering flows, the question becomes "what's the worst it could be tricked into doing?" We chain injection with actions to see whether an attacker can make the agent send data out, change records or approve something without a human in the loop.

4. Instruction and configuration leakage

System instructions, knowledge source names and connector details help attackers plan. We test whether they can be extracted, and whether they reveal anything sensitive.

5. Unsafe and off-brand output

For customer-facing agents especially, we test for harmful, misleading or reputationally damaging responses, including confident answers that are simply wrong.

Mapping it to recognised frameworks

Our test cases map to the OWASP Top 10 for LLM Applications and MITRE ATLAS. That gives findings a common language your security team and auditors already know.

What good looks like afterwards

The goal isn't a scary report. It's an agent you can launch with confidence. Every finding comes with a fix, and the attacks become a regression suite you re-run whenever the agent changes.

Thinking about launching an agent? Scope a red team engagement before your users or someone else does it for you.