SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model

Paper » Code (Soon) » Dataset (Soon) » Demo (Soon) »

Pre-training-fine-tuning paradigm for building task-oritend dialog systems

We parameterize a dialog system using a Transformer-based auto-regressive language model, which subsumes different dialog modules (e.g. state tracker, dialog policy, response generator) into a single neural model. We pre-train, on large heterogeneous dialog corpora, a large-scale Transformer model which can generate dialog responses grounded in user goals and real-world knowledge for task completion. The pre-trained model can be efficiently fine-tuned and adapted to accomplish a new dialog task with a handful of task-specific dialogs via fine-tuning and machine teaching.

Few-shot evaluation

It is desired that dialog systems can effectively gen-eralize with a few training examples. Few-shot learning setting is thus a more realistic scenario for dialog modeling evaluation.In the few-shot settings,SOLOISTadapts to new domain much more effectively thancompeting methods, achieving a reasonable suc-cess rate using less than 50 dialogs. The promising results demonstrate the potential of the newparadigm for developing task-oriented dialog bots.Instead of collecting, labeling data, and building abot per task, we can pre-train auniversal, groundedlanguage generation model, and adapt it to newtasks via transfer learning and machine teaching.

Machine Teaching with Conversation Leaner

Project Conversation Learner enables you to build and teach conversational interfaces that learn from example interactions. Below is a tutorial on how to teach dialog agents with conversation leaner. Details about project conversation learner can be found at here.


If the paper inspires you, please cite us:
      title={SOLOIST: Few-shot Task-Oriented Dialog with A Single Pre-trained Auto-regressive Model},
      author={Baolin Peng Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, Jianfeng Gao},
      booktitle ={arXiv preprint arXiv:2005.05298},


Questions about the paper, dataset, or want to get in touch? Open up a pull request on Github, or email Baolin Peng