This is an example of how hand gestures could help to predict upcoming words and how this might result in the smooth conversations that we all have every day. In my PhD at the Donders Institute for Brain, Cognition & Behaviour (Radboud University Nijmegen), I investigate whether this is something that actually happens. What I had not expected was that my passion for hand gestures would eventually lead me to virtual reality and using virtual avatars! Virtual reality has been described as a “potentially game-changing method for the language sciences” by dr. David Peeters, an Associate Professor at the University of Tilburg who has been using virtual reality in his own research for years. This is the story of why I went in that very cool but unexpected direction.
The constant struggle: How to keep my studies controlled yet natural?
One of the main challenges of my PhD has been keeping my studies controlled (so I can learn what influences what) ánd natural (that is, as similar as possible to real-life everyday conversations). “Experimental control is important in psycholinguistic research, because to study the effect of one variable we need to isolate it and make sure only this variable differs between conditions”, dr. David Peeters explains. “But we should also make sure that in our experiments we create a situation that is similar to what we are ultimately interested in: communication in everyday settings.”
Introducing my friend Ava, a virtual avatar
This is why I now use this new, cool and innovative approach which we can use to make a study both controlled and natural: namely virtual reality (VR)! I’m using a virtual avatar, who looks like she’s from the video game The Sims. I call her Ava. With great help from Thijs Rinsma, ICT developer at the Donders Institute, I can make Ava talk and move exactly as I want to.
In my latest study, I use this method to test whether people can use the hand gestures they see to predict which words someone might say next. To make the experiment natural, I let Ava do hand gestures that I saw in conversations between friends, and I let her ask questions like in real conversation. In one condition, Ava produced a gesture, and in the other she did not. Crucially, I made sure that nothing changed between these conditions except for the hand gesture.
Virtual reality as a tool to study language
David Peeters also uses virtual reality to study human language and communication. He has used it to test whether well-established findings from traditional studies replicate in rich virtual environments, such as how bilinguals switch between two languages. In addition, he has used VR to carry out experiments that are difficult to do in the real world. In one such study, led by Johanne Tromp as part of her PhD studies, they let participants be a waiter in a virtual restaurant to study how people process indirect requests. The restaurant guests would for example say “My soup is cold”, indirectly requesting to be brought a new soup. These experiments were performed in the state-of-the-art Virtual Reality lab at the Max Planck Institute for Psycholinguistics.
When asked what he thought of using virtual avatars to study how people process communicative bodily signals such as gestures, Peeters explained “I think it is great. This way you can look at human communication as something that involves many different sources of information, such as the face, mouth, hands and speech. Compared to the many studies that focused only on speech, using VR to also include communicative bodily signals is a great step forward.” Indeed, virtual avatars make it possible to study communication in a more realistic way, without having to give up on naturalness or experimental control!
I’m curious to see how virtual reality will be used as a tool in future studies. For my research, for example, another great step forward would be to have participants hold conversations with these avatars, so participants are not just listening, but also speaking. Then we could find out whether using Ava’s hand gestures to make predictions, like I predicted my friend might offer me a drink, would also result in faster responses. Hopefully Ava and her future friends will help us unravel this and all the other mysteries underlying human language and communication.
Recommended reading
- To read about how gestures typically start before the corresponding speech and thus make prediction possible: Marlijn ter Bekke, Linda Drijvers & Judith Holler (2020). The predictive potential of hand gestures during conversation: An investigation of the timing of gestures in relation to speech. https://doi.org/10.31234/osf.io/b5zq7
- To read more about using VR as a method for the language sciences: David Peeters (2019). Virtual reality: A game-changing method for the language sciences. https://doi.org/10.3758/s13423-019-01571-3
- To read more about how human communication involves the body as well as speech: Judith Holler & Stephen Levinson (2019). Multimodal language processing in human communication. https://doi.org/10.1016/j.tics.2019.05.006
Editor: Cecilia Hustá
Final editing: Sophie Slaats
Dutch translation: Veerle Wilms
German Translation: Franziska Schulz
