CONSONENCE
COllaborative multimodal Narrative SOcial ageNts for immersive ExperieNCEs
Modélisation d’agents sociaux intelligents et interactions multimodales pour des expériences immersives.
Context, positioning, and objective(s)
The growing interest of manufacturers in developing new multimodal interaction devices and standards (e.g., MPEG-I Part 31, Haptics Coding), together with the emergence of nextgeneration interactive content, invites us to rethink the formats and uses of future immersive environments (e.g., multimodal streaming, content enhanced with haptic feedback).
The CONSONENCE project aims to study collaboration between users and virtual agents within heterogeneous immersive environments populated by agents endowed with social interaction capabilities. More specifically, it focuses on the task of listening to narrative content (podcasts, streaming media, oral storytelling, or immersive stereoscopic videos) in a virtual space where embodied agents sustain narrative tension and encourage user interaction with the environment. Such a task, when performed in mixed reality (MR), raises significant scientific and technical challenges: users are physically distant, rely on diverse MR devices (CAVE systems, VR/AR headsets, tablets, screens), and consequently lose access to natural communication channels. This heterogeneity hinders collaboration and collective creativity [1], ultimately degrading user experience quality, engagement, and the emergence of spontaneous interactions.
The project is guided by two main hypotheses: (H1) Integrating multimodal interactions (visual, auditory, haptic) within the virtual environment enhances user experience and engagement in MR. (H2) Populating the virtual environment with socially intelligent agents (SIAs) capable of multimodal interaction improves user experience and promotes interaction, reinforcing copresence and narrative engagement.