Source: Neural Machine Translation of Rare Words with Subword Units (Sennrich et al., 2016)
The paper that brought byte pair encoding to neural NLP as a way to handle open vocabularies with subword units.
Contents
- What: research paper (arXiv 1508.07909, ACL 2016)
- Author: Rico Sennrich, Barry Haddow, Alexandra Birch (University of Edinburgh)
- Date: 2015-08-31 (v1); revised 2016-06-10 (v5)
- Original: arXiv:1508.07909
Summary
Neural translation models use a fixed vocabulary, yet real text keeps producing unseen words. The paper adapts byte pair encoding, originally a compression algorithm, to learn a vocabulary of subword units by repeatedly merging the most frequent adjacent symbol pair. Rare words are then represented as sequences of known subwords instead of an unknown-word token.
Key claims
- Translation is an open-vocabulary problem; fixed word vocabularies cannot cover it.
- Encoding rare and unknown words as subword sequences lets the model handle them without a back-off dictionary.
- BPE, a compression technique, is adapted to learn the subword vocabulary.
- Subword models beat a back-off dictionary baseline by 1.1 BLEU (En-De) and 1.3 BLEU (En-Ru) on WMT 15.