Variance-Aware Machine Translation Test Sets
2021
We release 70 small and discriminative test sets for machine translation (MT)
evaluation called variance-aware test sets (VAT), covering 35 translation
directions from WMT16 to WMT20 competitions. VAT is automatically created by a
novel variance-aware filtering method that filters the indiscriminative test
instances of the current MT test sets without any human labor. Experimental
results show that VAT outperforms the original WMT test sets in terms of the
correlation with human judgement across mainstream language pairs and test
sets. Further analysis on the properties of VAT reveals the challenging
linguistic features (e.g., translation of low-frequency words and proper nouns)
for competitive MT systems, providing guidance for constructing future MT test
sets. The test sets and the code for preparing variance-aware MT test sets are
freely available at https://github.com/NLP2CT/Variance-Aware-MT-Test-Sets .
- Correction
- Source
- Cite
- Save
- Machine Reading By IdeaReader
18
References
1
Citations
NaN
KQI