Source: A Holistic Approach to Undesired Content Detection in the Real World

OpenAI's AAAI-23 paper on building a production moderation classifier: taxonomy, labelling quality, active learning for rare events, robustness.

Contents
  • What: research paper (arXiv:2208.03274), AAAI-23 oral
  • Authors: Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, Lilian Weng (OpenAI)
  • Date: 5 August 2022 (v2 14 February 2023)
  • Original: arxiv.org/abs/2208.03274

Summary

The paper describes the end-to-end recipe behind OpenAI’s moderation classifier. Its main argument is that model architecture matters less than the surrounding process: a precise taxonomy, consistent labelling, and data collection aimed at rare harmful events.

Key claims

  • Categories covered: sexual, hateful, violence, self-harm, harassment.
  • Success depends on taxonomy and labelling-instruction design, data-quality control, an active-learning pipeline to surface rare events, and techniques that improve robustness and reduce overfitting.
  • The approach generalises to other taxonomies and beats off-the-shelf classifiers.

Feeds

Content moderation

Suggest changes →