1 of 15

Final Project

Dialog State Tracking Challenge

TAs

adl2016ta@gmail.com

​

2 of 15

Final Project

Release: 2016/11/24 09:00

Code Deadline: 2017/1/8 23:59

Report Deadline: 2017/1/12 09:00

​

3 of 15

Outline

  • Introduction
  • Challenge overview
    • main task
    • pilot task
  • Data
  • Evaluation
  • Links
  • FAQ

4 of 15

Introduction

  • Dialog state tracking:
  • updating the dialog state after each interaction between the system and user
  • Dialog state:
  • the history of the conversation up to the current timestep
  • Difficulty:
  • errors in speech recognition and language understanding
  • ambiguous natural language�

​

5 of 15

Introduction

  • Dialog State Tracking Chanllenge 5:
  • cross-language dialog state tracking
  • Goals:
  • built a tracker for the target language using resources using:�1.existing resources in the source language�2.the corresponding machine translated sentences in the target language

​

6 of 15

Challenge Overview

  • Main task
  • Pilot task
  • Spoken language understanding
  • Speech act prediction
  • Spoken language generation

7 of 15

Challenge Overview

  • Main task ：track dialog states for sub-dialog segments
  • fill out a frame of slot-value pairs considering all dialog history prior to the turn

8 of 15

Challenge Overview

Main task ：track dialog states for sub-dialog segments

1.A training set of labelled dialogs in English�2.A small development set in Chinese�3.built systems for state tracking in Chinese dialogs using English training set�4.evaluate on the unlabelled test set in Chinese by comparing the system outputs with reference annotations

9 of 15

Challenge Overview

  • Example human-human dialog in Chinese and dialog state labels for the main task

10 of 15

Challenge Overview

  • Pilot task
  • Spoken language understanding:�Tagging a given utterance with speech acts and semantics slots.
  • Speech act prediction:�Predicting the speech act of the next turn imitating the policy of one speaker.
  • Spoken language generation:�Generating a response utterance for one speaker by using the corresponding speech act and semantic slot information.

11 of 15

Data

12 of 15

Data

  • label.json:
  • the annotations for each segment
  • 3 different type of labels:�frame structure, speech act, semantics
  • log.json:
  • the information for each session between a given tourist and a given guide
  • translation.json:
  • the translation generate by a MT system for each dialog session �

​

13 of 15

Evaluation

The following metrics were used for evaluation: �• Accuracy: �Fraction of sub-dialogs in which the tracker’s output is equivalent to the gold standard frame structure �• Precision: �Fraction of slot-value pairs in the tracker’s outputs correctly filled�• Recall:� Fraction of slot-value pairs in the gold standard labels correctly filled �• F-measure:� The harmonic mean of precision and recall

​

​

​

14 of 15

Links

​

​

​

15 of 15

FAQ