Source: Lessons From Red Teaming 100 Generative AI Products

Microsoft AI Red Team (Jan 2025): eight lessons from 100+ engagements, notably that simple attacks beat gradient methods, red teaming is not benchmarking, and humans remain essential.

Contents
  • What: experience report, arXiv:2501.07238
  • Authors: Blake Bullwinkel, Amanda Minnich, … Ram Shankar Siva Kumar, Yonatan Zunger, Chang Kawaguchi, Mark Russinovich (Microsoft AI Red Team)
  • Date: 2025-01-13
  • Original: arXiv:2501.07238

Summary

The team that builds PyRIT distils what it learned from red-teaming more than 100 generative AI products. It also presents an internal threat-model ontology, which the abstract does not detail.

Key claims

  1. Start from what the system can do and where it is deployed.
  2. You don’t need to compute gradients to break an AI system.
  3. AI red teaming is not safety benchmarking.
  4. Automation helps cover more of the risk landscape.
  5. The human element is crucial.
  6. Responsible-AI harms are pervasive but hard to measure.
  7. LLMs amplify existing security risks and introduce new ones.
  8. Securing AI systems is never finished.

Feeds

LLM red teaming

Suggest changes →