XDailyDialog: A Multilingual Parallel Dialogue Corpus

Abstract

High-quality corpora are significant to the development of dialogue models.However, most existing corpora for open-domain dialogue modeling are limited to a single language.The absence of multilingual open-domain dialog corpora not only limits the research on multilingual or cross-lingual transfer learning but also hinders the development of robust opendomain dialogue systems that can be deployed in other parts of the world.In this paper, we provide a multilingual parallel open-domain dialog dataset, XDailyDialog, 1 to enable researchers to explore the challenging task of multilingual and cross-lingual open-domain dialogue.XDailyDialog includes 13K dialogues aligned across 4 languages (52K dialogues and 410K utterances in total).We then propose a dialogue generation model, kNN-Chat, which has a novel kNN-search mechanism to support unified response retrieval for monolingual, multilingual, and cross-lingual dialogue.Experiment results show the effectiveness of this framework.