Source: Magpie (Xu et al., 2024)

Paper that extracts alignment data from an aligned model by feeding it only the chat template's user-turn prefix, so it writes both query and answer.

Contents
  • What: research paper (arXiv:2406.08464)
  • Author: Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, Bill Yuchen Lin
  • Date: 2024-06-12 (v2 2024-10-07)
  • Original: arXiv:2406.08464

Summary

Magpie generates instruction data with no seed prompts. It gives an aligned model (Llama-3-Instruct) only the chat-template prefix that comes before a user message. Because the model is autoregressive, it completes a plausible user query, then answers that query in a second call.

Key claims

  • No seed questions or prompt engineering are needed. The chat template alone elicits queries.
  • 4M instruction–response pairs were generated, and 300K high-quality ones were selected.
  • Llama-3-8B-Base fine-tuned on Magpie data was comparable on some tasks to the official Llama-3-8B-Instruct, which used about 10M examples plus preference tuning.
  • Magpie SFT alone beat earlier public datasets used with SFT plus preference optimisation (such as UltraFeedback with DPO) on AlpacaEval, Arena-Hard and WildBench.

Feeds

Synthetic data generation

Suggest changes →