Source: Spider (Yu et al., EMNLP 2018)

The cross-domain text-to-SQL benchmark of 10,181 questions over 200 multi-table databases, whose split by database forces models to generalize to unseen schemas.

Contents
  • What: benchmark paper (arXiv:1809.08887, EMNLP 2018)
  • Author/publisher: Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, …, Dragomir Radev (Yale)
  • Date: 2018-09-24 (v5 2019-02-02)
  • Original: arXiv:1809.08887

Summary

Spider made text-to-SQL a cross-domain problem. Earlier datasets reused one database and the same queries across train and test; Spider puts different databases in train and test, so a model must map language onto a schema it has never seen. It became the standard benchmark of the pre-LLM and early-LLM era.

Key claims

  • 10,181 questions, 5,693 unique complex SQL queries, 200 multi-table databases, 138 domains, annotated by 11 college students.
  • Train/test split by database, so test schemas and queries are unseen.
  • Evaluated with exact-match accuracy of predicted SQL against gold SQL.
  • Best baseline at release reached only 12.4% exact match in the database-split setting.

Feeds

Text-to-SQL

Suggest changes →