Personalizing Task-oriented Dialog Systems via Zero-shot Generalizable Reward Function

Abstract

Task-oriented dialog systems enable users to accomplish tasks using natural\nlanguage. State-of-the-art systems respond to users in the same way regardless\nof their personalities, although personalizing dialogues can lead to higher\nlevels of adoption and better user experiences. Building personalized dialog\nsystems is an important, yet challenging endeavor and only a handful of works\ntook on the challenge. Most existing works rely on supervised learning\napproaches and require laborious and expensive labeled training data for each\nuser profile. Additionally, collecting and labeling data for each user profile\nis virtually impossible. In this work, we propose a novel framework, P-ToD, to\npersonalize task-oriented dialog systems capable of adapting to a wide range of\nuser profiles in an unsupervised fashion using a zero-shot generalizable reward\nfunction. P-ToD uses a pre-trained GPT-2 as a backbone model and works in three\nphases. Phase one performs task-specific training. Phase two kicks off\nunsupervised personalization by leveraging the proximal policy optimization\nalgorithm that performs policy gradients guided by the zero-shot generalizable\nreward function. Our novel reward function can quantify the quality of the\ngenerated responses even for unseen profiles. The optional final phase\nfine-tunes the personalized model using a few labeled training examples. We\nconduct extensive experimental analysis using the personalized bAbI dialogue\nbenchmark for five tasks and up to 180 diverse user profiles. The experimental\nresults demonstrate that P-ToD, even when it had access to zero labeled\nexamples, outperforms state-of-the-art supervised personalization models and\nachieves competitive performance on BLEU and ROUGE metrics when compared to a\nstrong fully-supervised GPT-2 baseline\n