A Large-Scale Analysis of Human Question-Asking Strategies in a Referential Visual Search Task

Alberto Testoni, Postdoctoral Researcher at the Institute for Logic, Language and Computation – University of Amsterdam.

Abstract:

In recent years, a multitude of datasets of human–human conversations has been released for the main purpose of training conversational agents based on data-hungry artificial neural networks. Datasets of this sort represent a useful and underexplored source to validate, complement, and enhance cognitive studies on human behavior and language use. We present a method that leverages the recent development of powerful computational models to obtain the fine-grained annotation required to apply metrics and techniques from Cognitive Science to large datasets. Previous work in Cognitive Science has investigated the question-asking strategies of human participants by employing different variants of the so-called 20-question-game setting and proposing several evaluation methods. In my talk, I focus on GuessWhat, a task proposed within the Computer Vision and Natural Language Processing communities that is similar in structure to the 20-question-game setting. Crucially, the GuessWhat dataset contains tens of thousands of dialogues based on real-world images, making it a suitable setting to investigate the question-asking strategies of human players on a large scale and in a natural setting. Our results demonstrate the effectiveness of computational tools to automatically code how the hypothesis space changes throughout the dialogue in complex visual scenes. On the one hand, we confirm findings from previous work on smaller and more controlled settings. On the other hand, our analyses allow us to highlight the presence of “uninformative” questions (in terms of Expected Information Gain) at specific rounds of the dialogue. We hypothesize that these questions fulfill pragmatic constraints that human players exploit to solve visual tasks in complex scenes successfully. Our work illustrates a method that brings together efforts and findings from different disciplines to gain a better understanding of human question-asking strategies on large-scale datasets, while at the same time posing new questions about the development of conversational systems.

—

Join Zoom Meeting
https://univ-amu-fr.zoom.us/j/86852094022?pwd=YVBvNVk0WlYwZ2hUS0VQcVozZnBVQT09

Meeting ID: 868 5209 4022
Passcode: 790678
One tap mobile
+33170950350,,86852094022# France
+33186995831,,86852094022# France

Dial by your location
+33 1 7095 0350 France
+33 1 8699 5831 France
+33 1 7037 2246 France
+33 1 7037 9729 France
+33 1 7095 0103 France
Meeting ID: 868 5209 4022
Find your local number: https://univ-amu-fr.zoom.us/u/kcAYO6DzTC

Join by SIP
86852094022@139.124.199.80
86852094022@139.124.199.200

Join by H.323
139.124.199.80
139.124.199.200
Meeting ID: 868 5209 4022
Passcode: 790678