Contributors to this document: Roxane Bertrand, Thierry Chaminade, Leonardo Lancia, Noël
Nguyen, Magalie Ochs, Cristel Portes, Béatrice Priego-Valverde [feel free to add your own
name if you make additions/amendments etc.].

In this roundtable, we propose to address three main topics:

• Conversational interactions as the primary frame of reference for studies on brain language relationships
• New paradigms for investigating the neurophysiological and cognitive bases of conversational interactions
• Analytical tools for the characterization of between-individual coordination and information transfer in conversational interactions

Conversational interactions as the primary frame of reference for studies on brain/language
relationships

There is a longstanding tradition of research on the relationships between language, cognition
and the human brain. Until recently, however, investigations in this domain were limited to
studying language production or comprehension in talkers individually exposed to highly controlled
linguistic material. Over the last couple of years, major advances have been
simultaneously accomplished in language sciences, cognitive sciences and neurosciences,
which have brought us on the verge of a new research paradigm. Language sciences have
entered a new phase as they move away from individually-administered protocols towards the
characterization of how spoken language is jointly used by two or more talkers as a shared set
of resources for interacting with each other (Clark,1996). The idea that many aspects of
language interaction are rule governed and can be modeled into the grammar is also assumed
by a growing number of linguists (Ginzburg & Poesio, 2016). This has occurred in conjunction
with the advent of increasingly large databases on conversational spoken language, together
with that of powerful large-scale spoken-language processing tools and techniques. Cognitive
sciences and neurosciences have also undergone a paradigm shift, which has made them pass
from a single-brain to a multi-brain frame of reference (Hasson, et al., 2012; Schilbach et al.
2013) as a new challenge has arisen that consists in understanding how the brains of two
people speaking with each other come to being temporarily coupled. These advances now
make it possible to explore language and the brain in the context in which they both primarily
develop, i.e. social interactions.

Previous work by members of the BLRI and others has shown that phonetic and prosodic
patterns systematically convey information about the interaction in which they are produced. For
example, Meunier & Espesser (2011) found that vowel shortening systematically occurs in
conversational speech compared with read speech, to a greater extent for function than for
content words. On the basis of a large-scale speech corpus, Aubanel and Nguyen (2010)
provided evidence that speakers of different regional varieties of French tend to converge
towards each other at the phonetic level in an interactive situation. Portes, Beyssade, Michelas,
Marandin, & Champagne-Lavau (2014) demonstrated that intonational contours are consistently
related to both the speaker’s attribution of attitude to the interlocutor and speaker’s expectations
about the interlocutor’s upcoming move. Guardiola and Bertrand (2013) have revealed that
between-speaker convergence in a story-telling activity, at a variety of linguistic levels that
include the phonetic and prosodic ones, is contingent upon the degree of alignment (activity
level) and affiliation (stance of speakers) shown by the interlocutor with respect to the main
speaker. On the other hand, Priego-Valverde et al. (submitted) found that conversational
partners spend more time displaying a non-synchronic smiling behavior than a synchronic one.
2 New paradigms for investigating the neurophysiological and cognitive bases of conversational interactions.

Despite multiple advances, neural mechanisms that underlie natural social encounter are still
considered as the “dark matter” of social cognitive neuroscience (Schilbach et al., 2012). Novel
paradigmatic approaches, both theoretically and experimentally, are necessary to overcome
intrinsic limitations of the classical scientific approach, ill suited to study naturally uncontrolled
natural interactions.

One such approach utilizes artificial agents such as computer-animated avatars (e.g. Embodied
Conversational Agents - ECA) and humanoid robots. First of all, artificial agents can be made
interactive but they don’t elicit certain natural mechanisms for social interactions, such as the
attribution of mental states, as real people do (Wykowska, Chaminade, & Cheng, 2016). They
can therefore be used as high-order controls to study such cognitive and physiological
mechanisms involved in natural interactions like in conversations. Secondly, since artificial
agents provide full control of their behaviours, they can be used to replay specific verbal and
nonverbal behaviors to investigate hypotheses on social interactions both with humans (e.g.
Ochs et al., 2017, Ravenet, Ochs, & Pelachaud, 2014) or with others artificial agents (e.g.,
effects of synchrony between agents as for the “virtual parrot”, Lancia et al., under development,
or smiling agents, Prepin, Ochs, & Pelachaud, 2013). However, artificial agents Consequently,
another challenge is to investigate more precisely the effect of these mechanisms on the
interaction with virtual entities.

A second approach is the investigation of brain-to-brain coupling (Hasson et al., 2012), in
particular through direct recording of the brain responses of two interlocutors with
hyperscanning - of fMRI, fNRIS, EEG or MEG signals. Both approaches are rather novel and to
our knowledge haven’t yet been used to investigate natural conversational interactions.
3 Mathematical tools for the characterization of between-individual coordination and
information transfer in conversational interactions.

In the last decades, the interest in coordinative phenomena observed in many disciplines and
research domains has fostered the development of methods that permit characterizing the
exchange of information between heterogeneous processes unfolding over time on different
time scales. The emphasis will be put on tools and techniques based on concepts from
information theory, time-series analysis, dynamical systems, graph analysis and Bayesian
inference (to cite a few) that can be used to identify the web of relations linking linguistic,
physiological, and neural phenomena observed during verbal interactions and conversational
tasks. Similar methods are commonly adopted in the analysis of neurophysiological signals (for
example in studies addressing connectivity or in studies on cortical oscillations, e.g. Gross et al.
2013), but their application to the analysis of speech behavior, as reflected in physiological
activity, motion patterns, and acoustic signals, is much less developed. Some studies conducted
by BLRI members successfully adapted state-space methods (e.g., Lancia et al., 2016),
originally proposed for the analysis of (nearly stationary) dynamical systems, to the analysis of
strongly non-stationary signals from speech production. In our on-going work, these methods
are applied to characterize the mutual influences (in both magnitude and directionality) in pairs
of time-varying processes (e.g. the movements of two different speech articulators, or the
amplitude modulations of speech signals from two different speakers). The next steps involve
the characterization of the coordination between several behavioural and neurophysiological
dimensions and between continuous dimensions (as those representing neural, physiological
and acoustic data) and symbolic dimensions (as those representing linguistic data).

References (authors attending the Porquerolles workshop are in bold)
Bertrand, R., & Espesser, R. (2017). Co-narration in French conversation storytelling: A
quantitative insight. Journal of Pragmatics, 111, 33-53.

Bertrand, R., & Priego-Valverde, B. (submitted). Listing practice in French conversation:
From collaborative achievement to interactional convergence. Discours.

Chaminade, T. (2017). An experimental approach to study the physiology of natural social
interactions. Interaction Studies, in press.
Falk, S., Lanzilotti, C., & Schön, D. (2017). Tuning neural phase entrainment to speech.
Journal of Cognitive Science, in press.

Ginzburg, J., & Poesio, M. (2016). Grammar is a system that characterizes talk in interaction.
Frontiers in Psychology, 7, Article 1938.

Giraud, A. L., & Poeppel, D. (2012). Cortical oscillations and speech processing: emerging
computational principles and operations. Nature Neuroscience, 15, 511-517.

Gross, J., Hoogenboom, N., Thut, G., Schyns, P., Panzeri, S., Belin, P., & Garrod, S. (2013).
Speech rhythms and multiplexed oscillatory sensory coding in the human brain. PLoS Biol, 11,
e1001752.

Guardiola M. & Bertrand R. (2013) Interactional convergence in conversational storytelling:
when reported speech is a cue of alignment and/or affiliation. Frontiers in Psychology, 4, Article
705.

Guillot, A., & Daucé, E., eds. (2002). Approches dynamiques de la cognition artificielle
(Lavoisier).

Heyselaar, E., Hagoort, P., & Segaert, K. (2017). In dialogue with an avatar, language behavior
is identical to dialogue with a human partner. Behavior Research Methods, 49, 46-60.
Lancia, L., Voigt, D., & Krasovitskiy, G. (2016). Characterization of laryngealization as irregular
vocal fold vibration and interaction with prosodic prominence. Journal of Phonetics, 54, 80-97.
Meunier, C., & Espesser, R. (2011). Vowel reduction in conversational speech in French: The
role of lexical factors. Journal of Phonetics, 39, 271-278.

Moulin-Frier, C., Diard, J., Schwartz, J. L., & Bessière, P. (2015). COSMO (“Communicating
about Objects using Sensory–Motor Operations”): A Bayesian modeling framework for studying
speech communication and the emergence of phonological systems. Journal of Phonetics, 53,
5-41.

Nguyen, N., & Delvaux, V. (2015). Role of imitation in the emergence of phonological systems.
Journal of Phonetics, 53, 46-54.

Priego-Valverde, B., Bigi, B., Attardo, S., Pickering, L., Gironzetti, E. (submitted). Is smiling
during humor so obvious? A cross-cultural comparison of smiling behavior in humorous
sequences in American English and French interactions. Intercultural Pragmatics.
Ochs, M., Mckeown, G., & Pelachaud, C. (2017). A user-perception based approach to create
smiling embodied conversational agents. ACM Transactions on Interactive Intelligent Systems,
7, 4.

Ravenet, B., Ochs, M., & Pelachaud, C. (2014). Interpersonal Attitude of a Speaking Agent,
Simulated Group Conversations International Conference on Intelligent Virtual Agent (IVA2014),
Boston, USA, August 2014.

Portes, C., Beyssade, C., Michelas, A., Marandin, J. M., & Champagne-Lavau, M. (2014). The
dialogical dimension of intonational meaning: evidence from French. Journal of Pragmatics, 74,
15-29.

Prepin, K., Ochs, M., & Pelachaud, C. (2013). Beyond backchannels: co-construction of dyadic
stancce by reciprocal reinforcement of smiles between virtual agents, CogSci (Annual
Conference of the Cognitive Science Society), Berlin, July 2013.

Schoot, L., Hagoort, P., & Segaert, K. (2016). What can we learn from a two-brain approach to
verbal interaction? Neuroscience & Biobehavioral Reviews, 68, 454-459.

Strijkers, K., & Costa, A. (2016). The cortical dynamics of speaking: present shortcomings and
future avenues. Language, Cognition and Neuroscience, 31, 484-503.

Wykowska, A., Chaminade, T., & Cheng, G. (2016). Embodied artificial agents for
understanding human social cognition. Phil. Trans. R. Soc. B, 371(1693), 20150375.