Awesome Dialogue State Tracking

Dialogue State Tracking (DST) Papers, Codes, Datasets, Resources

✅  Last update : 22.07.17

[Table of Contents]

📖  Introduction to DST

📝  DST Research Papers

   1. MultiWOZ (Multi-domain Wizard-of-Oz)

      1) Ontology based model

      2) Open vocab based model

      3) Hybrid model (Ontology + Open vocab)

      4) Zero,Few-Shot / Meta / Transfer learning

   2. WOZ (Wizard-of-Oz)

   3. SGD (Schema-Guided Dialogue)

   4. Data Limitation

   5. etc

🗂  Datasets

   1. Single Domain

   2. Multi Domain

      English

      Korean

      Chinese

📌  Evaluation Metrics

🏆  Competition

[1] Introduction to DST

Dialogue state tracking (DST) is a core component in task-oriented dialogue systems, such as restaurant reservation or ticket booking. The goal of DST is to extract user goals/intentions expressed during conversation and to encode them as a compact set of dialogue states, i.e., a set of slots and their corresponding values (Wu et al., 2019)

Dialogue State Tracking (DST) can be categorized into several approaches. In this repository, we divided the dst approach as shown.

[2] DST Research Papers

✅  Paper name, Venue | Model name | [Code]

1. MultiWOZ (Multi-domain Wizard-of-Oz)

1) Ontology based model

• SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking , ACL 2019 | SUMBT | [Code]
• HyST: A Hybrid Approach for Flexible and Accurate Dialogue State Tracking , Interspeech 2019 | HyST | None
• Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering , arXiv preprint| DSTQA | [Code]
• Schema-Guided Multi-Domain Dialogue State Tracking with Graph Attention Neural Networks , AAAI 2020 | SST | None
• Slot Self-Attentive Dialogue State Tracking , WWW 2021 | DST-STAR | [Code]
• Knowledge-Aware Graph-Enhanced GPT-2 for Dialogue State Tracking , EMNLP 2021 | None | [Code]
• LUNA: Learning Slot-Turn Alignment for Dialogue State Tracking , NAACL 2022 | LUNA | None

2) Open vocab based model

• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems , ACL 2019 | TRADE | [Code]

• BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformer , Interspeech 2019 | BERT-DST |[Code]

• Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation , IJCNLP 2019 | COMER | [Code]

• CREDIT: Coarse-to-Fine Sequence Generation for Dialogue State Tracking , arXiv preprint 2020 | CREDIT | None

• Non-Autoregressive Dialog State Tracking , ICLR 2020 | NADST | [Code]

• SimpleTOD: A Simple Language Model for Task-Oriented Dialogue , NeurIPS 2020 | SimpleTOD | [Code]

• SAS: Dialogue State Tracking via Slot Attention and Slot Information Sharing , ACL 2020 | SAS | None

• From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap , ACL 2020 | STARC | None

• Efficient Dialogue State Tracking by Selectively Overwriting Memory , ACL 2020 | SOM-DST | [Code]

• End-to-End Neural Pipeline for Goal-Oriented Dialogue Systems using GPT-2 , ACL 2020 | NP-DST | None

• Efficient Context and Schema Fusion Networks for Multi-Domain Dialogue State Tracking , Findings of EMNLP 2020 | CSFN-DST | None

• Multi-Domain Dialogue State Tracking based on State Graph , arXiv preprint | Graph-DST | None

• GCDST: A Graph-based and Copy-augmented Multi-domain Dialogue State Tracking , Findings of EMNLP 2020 | GCDST | None

• Slot Attention with Value Normalization for Multi-Domain Dialogue State Tracking , ACL 2020 | SAVN | [Code]

• Parallel Interactive Networks for Multi-Domain Dialogue State Generation , EMNLP 2020 | PIN | [Code]

• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue , EMNLP 2020 | TOD-BERT | [Code]

• TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking , SIGDAL 2020 | TripPy | [Code]

• A Sequence-to-Sequence Approach to Dialogue State Tracking , ACL 2021 | Seq2Seq-DU | [Code]

• Jointly Optimizing State Operation Prediction and Value Generation for Dialogue State Tracking , arXiv preprint | Transformer-DST | [Code]

• Amendable Generation for Dialogue State Tracking , ACL 2021 | AG-DST | [Code]

• UBAR: Towards Fully End-to-End Task-Oriented Dialog System with GPT-2 , AAAI 2021 | UBAR | [Code]

• Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System , ACL 2022 | PPTOD | [Code]

• DSTEA: Dialogue State Tracking with Entity Adaptive Pre-training , arXiv preprint | DSTEA | None

3) Hybrid model (Ontology + Open vocab)

• Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking , SEM 2020 | DS-DST | None

• Dual Slot Selector via Local Reliability Verification for Dialogue State Tracking , ACL 2021 | DSS-DST | [Code]

4) Zero,Few-Shot / Meta / Transfer learning

• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems , ACL 2019 | TRADE | [Code]
• Fine-Tuning BERT for Schema-Guided Zero-Shot Dialogue State Tracking , AAAI 2020 | SGP-DST | None
• Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Tracking , ACL 2020 | None | [Code]
• From Machine Reading Comprehension to Dialogue State Tracking: Bridging the Gap , ACL 2020 | None | None
• MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems , EMNLP 2020 | MinTL | [Code]
• Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking , NAACL 2021 | None | None
• Zero-shot Generalization in Dialog State Tracking through Generative Question Answering , EACL 2021 | None | None
• Few Shot Dialogue State Tracking using Meta-learning , EACL 2021 | None | None
• Domain Adaptive Meta-learning for Dialogue State Tracking , TASLP | DAMAML | [Code]
• Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialog State Tracking , ACL 2021 | ScCLog | None
• NeuralWOZ: Learning to Collect Task-Oriented Dialogue via Model-Based Simulation , ACL 2021 | NeuralWOZ | [Code]

2. WoZ (Wizard-of-Oz)

• Neural Belief Tracker: Data-Driven Dialogue State Tracking , ACL 2017 | None | None

• Towards Universal Dialogue State Tracking , EMNLP 2018 | StateNet | [Code]

• Toward Scalable Neural Dialogue State Tracking , NeurIPS 2018 | GCE | [Code]

• Global-Locally Self-Attentive Dialogue State Tracker , ACL 2018 | GLAD | [Code]

• Scalable Neural Dialogue State Tracking , ASRU 2019 | G-SAT | [Code]

• A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems , ICASSP 2020 | None | None

• MA-DST: Multi-Attention-Based Scalable Dialog State Tracking , AAAI 2020 | MA-DST | None

• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue , EMNLP 2020 | TOD-BERT | [Code]

• Neural Dialogue State Tracking with Temporally Expressive Networks , Findings of EMNLP 2020 | TEN | [Code]

• A Sequence-to-Sequence Approach to Dialogue State Tracking , ACL 2021 | Seq2Seq-DU | [Code]

3. SGD (Schema-Guided Dialogue)

• A Fast and Robust BERT-based Dialogue State Tracker for Schema-Guided Dialogue Dataset , KDD 2020 | FastSGT | [Code]

• A Sequence-to-Sequence Approach to Dialogue State Tracking , ACL 2021 | Seq2Seq-DU | [Code]

4. Data Limitation

• COCO: CONTROLLABLE COUNTERFACTUALS FOR EVALUATING DIALOGUE STATE TRACKERS , ICLR 2021 | CoCo | [Code]

• Annotation Inconsistency and Entity Bias in MultiWOZ , SIGDIAL 2021 | None | None

• Oh My Mistake!: Toward Realistic Dialogue State Tracking including Turnback Utterances , arXiv preprint | None | None

5. etc.

• Recent Advances and Challenges in Task-oriented Dialog Systems , SCTS | None | None
• Variational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation , EMNLP 2020 | None | [Code]
• Tutorial : Deeper Conversational AI , NeurIPS 2020 | None | None
• Out-of-Task Training for Dialog State Tracking Models , COLING 2020 | Trippy variant |[Code]
• DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue , arXiv preprint | DialoGLUE | [Code]
• A Comparative Study on Schema-Guided Dialogue State Tracking , NAACL 2021 | None | None
• Comprehensive Study: How the Context Information of Different Granularity Affects Dialogue State Tracking? , ACL 2021 | None | [Code]
• Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialog State Tracking , ACL 2021 | SaCLog | None
• Coreference Augmentation for Multi-Domain Task-Oriented Dialogue State Tracking , Interspeech 2021 | CDST | None

[3] Datasets

✅  Paper name, Venue | Dataset name | Language | [Code]

1. Single Domain

• The Dialog State Tracking Challenge , SIGDIAL 2013 | DSTC | en | [Dataset]

• The Second Dialog State Tracking Challenge , SIGDIAL 2014 | DSTC2 | en | [Dataset]

• A Network-based End-to-End Trainable Task-oriented Dialogue System , EACL 2017 | CamRest/CamRest676 | en | [Dataset]

• Neural Belief Tracker: Data-Driven Dialogue State Tracking , ACL 2017 | WOZ 2.0 | en, de, it | [Dataset]

• AirDialogue: An Environment for Goal-Oriented Dialogue Research , EMNLP 2018 | AirDialogue | en | [Dataset]

• HDRS: Hindi Dialogue Restaurant Search Corpus for Dialogue State Tracking in Task-Oriented Environment , TASLP | HDRS | hi | [Dataset]

2. Multi Domain

English

• The Third Dialog State Tracking Challenge , IEEE SLT 2014 | DSTC3 | en | [Dataset]

• Key-Value Retrieval Networks for Task-Oriented Dialogue , SIGDIAL 2017 | KVReT | en | [Dataset]

• Building a Conversational Agent Overnight with Dialogue Self-Play , arXiv preprint | SimD | en | [Dataset]

• Large-Scale Multi-Domain Belief Tracking with Knowledge Sharing , ACL 2018 | MultiWOZ 1.0 | en | [Dataset]

• MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling , EMNLP 2018 | MultiWOZ 2.0 | en | [Dataset]

• MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines , LREC 2020 | MultiWOZ 2.1 | en | [Dataset]

• Schema-Guided Dialogue State Tracking Task at DSTC8 , AAAI 2020 | SGD | en | [Dataset]

• MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines , ACL 2020 | MultiWOZ 2.2 | en | [Dataset]

• MultiWOZ 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation , arXiv preprint | MultiWOZ 2.3 | en | [Dataset]

• MultiWOZ 2.4: A Multi-Domain Task-Oriented Dialogue Dataset with Essential Annotation Corrections to Improve State Tracking Evaluation , arXiv preprint | MultiWOZ 2.4 | en | [Dataset]

• BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling , NeurIPS 2021 Dataset and Benchmark Track | BiToD | en, ch | [Dataset]

Korean

• KLUE: Korean Language Understanding Evaluation , NeurIPS 2021 Dataset and Benchmark Track | KLUE-DST/WoS | kr | [Dataset]

Chinese

• CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset , TACL | CrossWOZ | ch | [Dataset]

• RiSAWOZ: A Large-Scale Multi-Domain Wizard-of-Oz Dataset with Rich Semantic Annotations for Task-Oriented Dialogue Modeling , EMNLP 2020 | RiSAWOZ | ch | [Dataset]

• BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling , NeurIPS 2021 Dataset and Benchmark Track | BiToD | en, ch | [Dataset]

[4] Evaluation Metrics

✅  Paper name, Venue | Metric name

• Global-Locally Self-Attentive Dialogue State Tracker , ACL 2018 | Joint Goal Accuracy

• Schema-Guided Dialogue State Tracking Task at DSTC8 , AAAI 2020 | Average Goal Accuracy

• Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking, ACL 2022 | Relative Slot Accuracy

• Towards Fair Evaluation of Dialogue State Tracking by Flexible Incorporation of Turn-level Performances, ACL 2022 | Flexible Goal Accuracy

[5] Competition (DSTC)

1. Introduction

DSTC is the most famous competition in the field of Dialogue System. First held in 2013, DSTC started as a Dialogue State Tracking Challenge, but since the dialogue-related researches have been actively expanded, it has been relaunched as the Dialogue System Technology Challenges. DSTC covers the various subjects of dialogue issues such as NLP, Vision, and Speech. The 11th challenge is now taking place with a total of 5 tracks (divided two challenge periods) . More information about DSTC can be found at the link.

2. Related Papers

✅  Paper name, Competition | Model name | [Code]

• An Empirical Study of Cross-Lingual Transferability in Generative Dialogue State Tracker , DSTC9 Workshop - AAAI | None | None

• Efficient Dialogue State Tracking by Masked Hierarchical Transformer , DSTC9 Workshop - AAAI | None | None

💌  Contact Us

Yukyung Lee | Korea University | [email protected]

Kyumin Park | Korea Advanced Institute of Science and Technology | [email protected]