DSTC7: End-to-End Conversation Modeling

DSTC7 has ended on January 27, 2019. This github project is still available 'as is', but we unfortunately no longer have time to maintain the code or to provide assistance with this project.

News

• 10/29/2018: Spreadsheet containing indivdual judgments used for human evaluation.
• 10/23/2018 and 10/15/2018: Automatic and human evaluation results posted. The code to reproduce the automatic evaluation and get the same scores can be found here.
• 10/8/2018: Participants submitted system outputs.
• 9/10/2018-10/8/2018: Evaluation phase, instructions here.
• 7/11/2018: An FAQ section has been added to the data extraction page.
• 7/1/2018: Official training data is up.
• 6/18/2018: Trial data is up.
• 6/1/2018: Task description is up.
• 6/1/2018: Registration for DSTC7 is now open.

Registration

Please register [here] Registration has now closed.

Task

This DSTC7 track presents an end-to-end conversational modeling task, in which the goal is to generate conversational responses that go beyond trivial chitchat by injecting informative responses that are grounded in external knowledge. This task is distinct from what is commonly thought of as goal-oriented, task-oriented, or task-completion dialog in that there is no specific or predefined goal (e.g., booking a flight, or reserving a table at a restaurant). Instead, it targets human-like interactions where the underlying goal is often ill-defined or not known in advance, of the kind seen, for example, in work and other productive environments (e.g.,brainstorming meetings) where people share information.

Please check this description for more details about the task, which follows our previous work "A Knowledge-Grounded Neural Conversation Model" and our original task proposal.

Data

We extend the knowledge-grounded setting, with each system input consisting of two parts:

• Conversational data from Reddit.
• Contextually-relevant “facts”, taken from the website that started the (Reddit) conversation.

Please check the data extraction for the input data pipeline. Note: We are providing scripts to extract the data from a Reddit dump, as we are unable to release the data directly ourselves.

Evaluation

As described in the task description (Section 4), We will evaluate response quality using both automatic and human evaluations on two criteria.

• Appropriateness;
• Informativeness.

We will use automatic evaluation metrics such as BLEU and METEOR to have preliminary score for each submission prior to the human evaluation. Participants can also use these metrics for their own evaluations during the development phase. We will allow participants to submit multiple system outputs with one system marked as “primary” for human evaluation. We will provide a BLEU scoring script to help participants decide which system they want to select as primary.

We will use crowdsourcing for human evaluation. For each response, we ask humans if it is an (1) appropriate and (2) informative response, on a scale from 1 to 5. The system with best average Appropriateness and Informativeness will be determined the winner.

Baseline

A standard seq2seq baseline model will be provided soon.

Timeline

Organizers

• Michel Galley
• Chris Brockett
• Sean Xiang Gao
• Bill Dolan
• Jianfeng Gao

Reference

If you submit any system to DSTC7-Task2 or publish any other work making use of the resources provided on this project, we ask you to cite the following task description paper:

Michel Galley, Chris Brockett, Xiang Gao, Bill Dolan, Jianfeng Gao. End-to-End conversation Modeling: DSTC7 Task 2 Description. In DSTC7 workshop (forthcoming).

Contact Information

• For questions specific to Task 2, you can contact us at [email protected]. (No longer maintained.)
• You can get the latest updates and participate in discussions on DSTC mailing list.