2D-ATOMS

Official dataset for Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models. Ziqiao Ma, Jacob Sansom, Run Peng, Joyce Chai. EMNLP Findings, 2023.

We introduce 2D-ATOMS dataset, a novel text-based dataset that evaluates a machine's reasoning process under a situated theory-of-mind setting.

Our dataset includes 9 different ToM evaluation tasks for each mental state under ATOMS framework, and 1 reality-checking task to test LLMs’ understanding of the world. It is important to acknowledge that our experiment serves as a proof of concept and does not aim to cover the entire spectrum of machine ToM, as our case studies are far from being exhaustive or systematic. Here we release the zero-shot version of our dataset, which is used in our paper.