Mutual Understanding in Situated Interactions with Conversational User Interfaces: Theory, Studies, and Computation

Download from KTH Royal Institute of Technology (DiVA). Published in March 2022.

This dissertation presents advances in HCI through a series of studies focusing on task-oriented interactions between humans and between humans and machines. The notion of mutual understanding is central, also known as grounding in psycholinguistics, in particular how people establish understanding in conversations and what interactional phenomena are present in that process. Addressing the gap in computational models of understanding, interactions in this dissertation are observed through multisensory input and evaluated with statistical and machine-learning models. As it becomes apparent, miscommunication is ordinary in human conversations and therefore embodied computer interfaces interacting with humans are subject to a large number of conversational failures. Investigating how these inter- faces can evaluate human responses to distinguish whether spoken utterances are understood is one of the central contributions of this thesis.

The first papers (Papers A and B) included in this dissertation describe studies on how humans establish understanding incrementally and how they co-produce utterances to resolve misunderstandings in joint-construction tasks. Utilising the same interaction paradigm from such human-human settings, the remaining papers describe collaborative interactions between humans and machines with two central manipulations: embodiment (Papers C, D, E, and F) and conversational failures (Papers D, E, F, and G). The methods used investigate whether embodiment affects grounding behaviours among speakers and what verbal and non-verbal channels are utilised in response and recovery to miscommunication. For application to robotics and conversational user interfaces, failure detection systems are developed predicting in real-time user uncertainty, paving the way for new multimodal computer interfaces that are aware of dialogue breakdown and system failures.

Through the lens of Theory, Studies, and Computation, a comprehensive overview is presented on how mutual understanding has been observed in interactions with humans and between humans and machines. A summary of literature in mutual understanding from psycholinguistics and human-computer interaction perspectives is reported. An overview is also presented on how prior knowledge in mutual understanding has and can be observed through experimentation and empirical studies, along with perspectives of how knowledge acquired through observation is put into practice through the analysis and development of computational models. Derived from literature and empirical observations, the central thesis of this dissertation is that embodiment and mutual understanding are intertwined in task-oriented interactions, both in successful communication but also in situations of miscommunication.

PAPER A: Estimating Uncertainty in Task-Oriented Dialogue. Dimosthenis Kontogiorgos, Andre Pereira, Joakim Gustafson. International Conference on Multimodal Interaction (ICMI). Shuzhou, China (2019).

PAPER B: Measuring Collaboration Load with Pupillary Responses - Implications for the Design of Instructions in Task-Oriented HRI. Dimosthenis Kontogiorgos, Joakim Gustafson. Frontiers in Psychology (Cognitive Science) (2021).

PAPER C: The Effects of Anthropomorphism and Non-verbal Social Behaviour in Virtual Assistants. Dimosthenis Kontogiorgos, Andre Pereira, Olle Andersson, Marco Koivisto, Elena Gonzalez Rabal, Ville Vartiainen, Joakim Gustafson. International Conference on Intelligent Virtual Agents (IVA). Paris, France (2019).

PAPER D: Embodiment Effects in Interactions with Failing Robots. Dimosthenis Kontogiorgos, Sanne van Waveren, Olle Wallberg, Andre Pereira, Iolanda Leite, Joakim Gustafson. International Conference on Human Factors in Computing Systems (CHI). Honolulu, USA (2020).

PAPER E: Grounding Behaviours with Conversational Interfaces: Effects of Embodiment and Failures. Dimosthenis Kontogiorgos, Andre Pereira, Joakim Gustafson. Journal on Multimodal User Interfaces (JMUI) (2021).

PAPER F: Behavioural Responses to Robot Conversational Failures. Dimosthenis Kontogiorgos, Andre Pereira, Boran Sahindal, Sanne van Waveren, Joakim Gustafson. International Conference on Human-Robot Interaction (HRI). Cambridge, UK (2020).

PAPER G: A Systematic Cross-Corpus Analysis of Human Reactions to Robot Conversational Failures. Dimosthenis Kontogiorgos, Minh Tran, Joakim Gustafson, Mohammad Soleymani. International Conference on Multimodal Interaction (ICMI). Montreal, Canada (2021). *Best Paper Nominee.