Evaluating Theory of Mind in Question Answering

08/28/2018
by   Aida Nematzadeh, et al.
0

We propose a new dataset for evaluating question answering models with respect to their capacity to reason about beliefs. Our tasks are inspired by theory-of-mind experiments that examine whether children are able to reason about the beliefs of others, in particular when those beliefs differ from reality. We evaluate a number of recent neural models with memory augmentation. We find that all fail on our tasks, which require keeping track of inconsistent states of the world; moreover, the models' accuracy decreases notably when random sentences are introduced to the tasks at test.

READ FULL TEXT

Please sign up or login with your details

Forgot password? Click here to reset