AI red teaming is the practice of probing AI systems with adversarial inputs and scenarios to surface security, safety, bias, and reliability failures before production. Modern practice combines human creativity with automated agentic probing, scoring outcomes against threat taxonomies including OWASP, MITRE ATLAS, and NIST AI 100-2.
- Adversarial probing of AI systems to find failures before production
- Combines human creativity with automated, agent-driven probing at scale
- Agent red teaming targets tool descriptions, permissions, and reasoning loops
- Manual-only programs cannot keep pace with thousands of agents
Why is AI red teaming important?
Red teaming matters because it is a recognized input to model and agent deployment decisions, sitting alongside third-party audits and continuous monitoring. The frontier labs treat it as standard practice – both Anthropic's Responsible Scaling Policy and OpenAI's GPT-4 System Card document formal red team programs.
It also appears in the frameworks enterprises are measured against. The NIST AI Risk Management Framework names adversarial testing within its Manage and Measure functions, and Article 9 of the EU AI Act imposes risk-management obligations including testing for high-risk systems, though it does not use the term red teaming verbatim.
The practical constraint is scale. Manual-only red teaming runs into a wall at agent populations measured in thousands, because the rate at which new agents and new tool integrations appear outpaces what any human team can sustainably probe.
What is AI red teaming?
AI red teaming is structured adversarial testing of an AI system, conducted to find failures while they are still cheap to fix. A red team approaches the system as an attacker would, attempting to make it behave outside its intended boundaries, then documents what worked.
What separates it from conventional penetration testing is the target. There is often no software vulnerability to exploit. The system is manipulated through its inputs – a crafted prompt, poisoned content, or a multi-turn conversation that shifts context gradually – so the testing surface is language and behavior rather than code paths.
Agent-specific red teaming extends this to the agent layer, and the attack vectors change accordingly. Prompt injection delivered through tool descriptions rather than user input. Tool abuse, where a permitted tool is used for an unintended purpose. Behavioral manipulation sustained across multi-turn sessions. And privilege escalation achieved through the agent's own reasoning loop, where the agent is persuaded to grant itself scope nobody authorized.
Types of AI red teaming
Practice divides by who conducts the probing and by what is being probed.
Manual red teaming relies on human testers, and it remains the best source of novel attack ideas. It does not scale to a large agent estate. Automated red teaming runs predefined attack suites continuously, covering known techniques at volume with no creative reach beyond its library. Agentic red teaming uses agents to probe agents, generating and adapting attacks against a live target, which recovers some of the creativity while keeping the coverage.
By target, model red teaming probes the model itself for jailbreaks and harmful output. Application red teaming probes the surrounding architecture, including retrieval and how output is handled. Agent red teaming probes identity, permissions, tool access, and the reasoning loop.
Most mature programs run continuous automated coverage with periodic human engagements aimed at what automation would not think to try.
AI red teaming & Onyx
Onyx ships Agentic Red Teaming as a read-only capability inside the Secure AI Control Plane. It continuously tests discovered AI agents against agent-specific attack vectors including prompt injection, tool abuse, behavioral manipulation, and privilege escalation through agent reasoning. Read-only is the current product scope.
The design point is that findings feed defense directly. Results flow into AI Guard runtime policies, so a weakness found by probing becomes an enforced control in the same platform rather than a ticket handed to another team. Typical findings illustrate why the agent layer needs its own testing: a publicly reachable agent able to act beyond the requesting user's scope, or an HR agent permitting enumeration of confidential employee records. Both are authorization failures rather than software defects, and neither shows up in a code scan.
