On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation

Abstract

While multilingual neural machine translation has achieved great success, it suffers from the off-target issue, where the translation is in the wrong language.This problem is more pronounced on zero-shot translation tasks.In this work, we find that failing in encoding discriminative target language signal will lead to offtarget and a closer lexical distance (i.e., KLdivergence) between two languages' vocabularies is related with a higher off-target rate.We also find that solely isolating the vocab of different languages in the decoder can alleviate the problem.Motivated by the findings, we propose Language Aware Vocabulary Sharing (LAVS), a simple and effective algorithm to construct the multilingual vocabulary, that greatly alleviates the off-target problem of the translation model by increasing the KLdivergence between languages.We conduct experiments on a multilingual machine translation benchmark in 11 languages.Experiments show that the off-target rate for 90 translation tasks is reduced from 29% to 8%, while the overall BLEU score is improved by an average of 1.9 points without extra training cost or sacrificing the supervised directions' performance.We release the code at https://github.com/PKUnlpicler/Off-Target-MNMTfor reproduction.