ML security is the practice of keeping machine learning systems trustworthy when someone is actively trying to break them. Traditional security still applies: access control, patched servers, protected secrets. But a learned model also has weaknesses that ordinary software does not, because its behaviour comes from data and it has to interpret whatever inputs arrive.
NIST’s taxonomy of adversarial machine learning groups the main attacks by what the attacker targets.
The main attack families
- Evasion: inputs crafted at inference time so the model gets them wrong, such as an image altered in ways a person would not notice. See adversarial attack.
- Poisoning: corrupting the training data or the model itself. Data poisoning degrades or skews the model; a backdoor attack plants a hidden trigger that produces attacker-chosen output.
- Privacy attacks: extracting what the model should keep private. Membership inference tests whether a specific record was in the training set; model extraction reconstructs a copy of the model through repeated queries.
LLM applications
Language model apps add their own risks. OWASP’s Top 10 for LLM applications lists prompt injection first: text in a user message, web page or document that overrides the developer’s instructions. When the model can call tools or read private data, a successful injection becomes a way to act or leak on the attacker’s behalf.
Defending
No single fix covers everything. Common layers include controlling where training data comes from, adversarial training and robustness testing, differential privacy to limit what a model memorizes, rate limits and monitoring on model APIs, and treating model output as untrusted before it triggers actions. Papernot and colleagues argue there can be real tension between model complexity, accuracy and resilience, so security is a design trade-off rather than a bolt-on.