TL
TL;DR
The dialog state tracking challenge seeks to address this by providing a heterogeneous corpus of 15K human-computer dialogs in a standard format, along with a suite of 11 evaluation metrics, and shows that the suite of performance metrics cluster into 4 natural groups.