Adversarial machine learning is the discipline of attacking and defending machine learning systems, covering the data they are trained on, the models themselves, and the inferences they produce at runtime. It includes data poisoning, model evasion, model extraction, and membership inference, and NIST publishes the canonical taxonomy.
- The discipline of attacking and defending machine learning systems
- Four canonical classes: poisoning, evasion, extraction, and inference
- NIST AI 100-2 is the standard taxonomy security teams map against
- Prompt injection is an evasion-class attack delivered through application input
Why is adversarial machine learning important?
Adversarial machine learning matters because it is the academic backbone behind most production AI security categories. The attack classes it names are what the commercial tooling claims to address, so it is the vocabulary that lets a buyer test whether a claim is specific.
Each class has a distinct operational consequence. Data poisoning corrupts training pipelines and software supply chains, and it is invisible to any runtime control because the compromise is already in the model. Evasion attacks defeat the classifiers security teams depend on, including the detection models used to catch other attacks. Extraction recovers a proprietary model's behavior through queries. Membership inference determines whether a specific record was in the training set, which is a privacy exposure rather than a security one.
The operative question for buyers follows directly from that spread: which adversarial classes does a vendor detect and intercept at runtime, and which does it merely catalog? Mapping a product to the taxonomy separates coverage from familiarity.
What is adversarial machine learning?
Adversarial machine learning is the study and practice of manipulating machine learning systems, along with the defensive techniques that resist manipulation. It predates the current generation of language models by more than a decade, and the framing it established is what modern AI security inherits.
NIST AI 100-2 publishes the canonical taxonomy, organizing attacks by the stage they target and the attacker's objective. Training-stage attacks alter what a model learns. Inference-stage attacks alter what it does with what it learned. Some attacks target confidentiality of the model or its data rather than its behavior.
Prompt injection sits at the intersection of adversarial ML and application-layer security. Formally it is an evasion-class attack against a model's instruction-following behavior, delivered through application input channels rather than through crafted numerical perturbation. That dual character is why it appears in both the academic taxonomy and the OWASP application checklist, described differently in each.
Types of adversarial machine learning attacks
The NIST taxonomy gives four classes, and they differ in when they occur and what they compromise.
Poisoning attacks corrupt training data or the training process so the resulting model behaves as the attacker intends. Backdoor variants embed a trigger that activates on a specific input. These are supply chain attacks in effect, and a consuming organization usually cannot detect them at all.
Evasion attacks craft inputs that cause a trained model to produce a wrong result at inference. Prompt injection and jailbreaking both belong here in the formal sense.
Extraction attacks reconstruct a model's parameters or behavior through systematic querying, which is a theft-of-intellectual-property concern for anyone hosting a proprietary model.
Inference attacks, including membership inference, determine properties of the training data from model behavior, exposing whether particular records were used.
Agent deployments add a fifth surface in practice: manipulation of tool descriptions and tool outputs, which is evasion applied to the agent's decision loop rather than the model's classification.
Adversarial machine learning & Onyx
Onyx maps adversarial ML threats to the NIST AI 100-2 taxonomy categories – evasion, poisoning, extraction, and inference – so coverage can be discussed against a standard rather than a feature list. That mapping is also what makes the gaps legible, since no platform intercepts all four classes and the honest position is to say which.
Runtime coverage spans the model inference and tool-use surfaces, including MCP-brokered tool calls, which is where evasion-class attacks against agents are actually delivered. Poisoning and extraction are addressed through supply chain and provenance visibility across models, agents, and MCP servers rather than through control of a third party's training process, because that is the honest limit of what a consuming enterprise can enforce.
