Source: Lessons From Red Teaming 100 Generative AI Products
Microsoft AI Red Team (Jan 2025): eight lessons from 100+ engagements, notably that simple attacks beat gradient methods, red teaming is not benchmarking, and humans remain essential.
Contents
- What: experience report, arXiv:2501.07238
- Authors: Blake Bullwinkel, Amanda Minnich, … Ram Shankar Siva Kumar, Yonatan Zunger, Chang Kawaguchi, Mark Russinovich (Microsoft AI Red Team)
- Date: 2025-01-13
- Original: arXiv:2501.07238
Summary
The team that builds PyRIT distils what it learned from red-teaming more than 100 generative AI products. It also presents an internal threat-model ontology, which the abstract does not detail.
Key claims
- Start from what the system can do and where it is deployed.
- You don’t need to compute gradients to break an AI system.
- AI red teaming is not safety benchmarking.
- Automation helps cover more of the risk landscape.
- The human element is crucial.
- Responsible-AI harms are pervasive but hard to measure.
- LLMs amplify existing security risks and introduce new ones.
- Securing AI systems is never finished.