Enhancing Immersive Interaction Design for Games using Interactive Machine Learning Workshop

Introduction and Motivation

Immersive technologies are increasingly more present in everyday devices, with an important rise of consumer Virtual Reality systems used for digital games. Games were contended as the ‘killer application’ for the study of computational creativity by Liapis and colleagues (2014), due to their multifaceted, content-intensive and rich gameplay interaction. However, games created for immersive technologies incorporate sensors that make it difficult to process complex movement interactions. Therefore, creators rely on simple controls for their games that constraint their creativity and limit the player experience. Liapis and colleagues (2014) listed interaction design as one of the key facets of computational game creativity (CGC), but existing research has not looked yet at how to enhance the creativity of movement interaction designers in immersive games.

Movement interaction design for novel interfaces using interactive machine learning (IML) was something previously explored in music motion research (Katan, Grierson and Fiebrink, 2015). The interactive use of machine learning technologies for movement interaction offers the capability to capture the nuances of physical movement when working with sensing technologies (Gillies, 2019) with the opportunity to enhance creators creativity. Existing research looking at supporting movement creativity uses a more generative approach, where the computer generates a diverse combination of movements (Carlson, Schiphorst and Pasquier, 2011). However, this approach fails when having a direct impact on the interactive and play ‘facets’ of CGC, which are key to the player experience (Liapis, Yannakakis and Togelius, 2014).

Is for the above-mentioned reasons that the focus of this workshop is to explore what are the challenges and opportunities on using interactive machine learning to support immersive movement interaction design in games. The idea behind the proposal is to organise a hands-on event, not a "mini-conference". Participants will actively collaborate using movement interaction design techniques to design and create a basic implementation of an immersive game interaction. Embodied sketching will be the technique used to elicit movement interaction design and play (Márquez Segura et al., 2016). Participants will participate in an activity of ‘bodystorming’, which is theoretically grounded in somaesthetics (Lee, Lim and Shusterman, 2014). Márquez Segura and colleagues (2016) show how bodily experience and awareness are of chief importance when designing technologically supported play experiences, and how play can be both the medium and the goal. Hence, participants will investigate immersive movement interaction design using their own physical movements, following the principles of ‘bodystorming’ during this phase.

Conversely, interactive machine learning will be the technique used to support their creative and implementation process (Gillies, 2019). Participants will take part in full body movement implementations following the movement interactions ideated in the previous phase. However, in order to allow participants to fully express themselves, we cannot expect them to sit and write code after having ideated with their body. Participants will then use a software toolkit, InteractML, that will allow them to implement their ‘bodystormed’ ideas by providing physical movement examples and computer responses iteratively (Gonzalez Diaz, Perry and Fiebrink, 2019). InteractML was developed to support interactive machine learning and movement interaction design for games, making it the ideal tool to explore this aspect of computational game creativity. The workshop will end with an opportunity to reflect on the approach and discuss its application to games. Participants who would like to continue with this workflow will then be given access to our tools and long-term support, as well as invited to join the existing online InteractML community1. This workshop is being developed as part of a project funded by the UK Engineering and Physical Science Research Council, Intelligent Games and Games Intelligence (IGGI).

Workshop Plan

This will be an active, hands-on workshop in which participants will learn immersive interaction design techniques and apply them to creating more expressive game interactions. We will provide a short introduction to interactive machine learning to the participants and show examples of how it can enhance their game creativity. We will then be leading the event introducing participants to an active body storming session to generate movement game interactions and then encourage to implement them in the software toolkit. We will provide support and constant guidance in how to use the toolkit. Since the activity will involve taking part in full body movement activities, comfortable clothing and footwear are encouraged. However, participants will choose the movements they make and only need to make movements they are comfortable with. We encourage participation from all along the spectrum of body motion, these experiences will provide valuable insights in the tool design.


  • 9:30 am: Registration
  • 10:00 am: Welcome Presentation / Group Introductions
  • 10:30 am: Machine Learning Presentation and Hands-on Demo
  • 11:00 am: coffee break
  • 11:15 am: Movement Sensitization Exercises
  • 11:30 am: Body storming and Embodied Sketching exercise
  • 12:30 am: lunch break
  • 13:30 am: Prototype Building
  • 16:00 am: Project Presentations
  • 16:30 am: Reflection and group discussion: How does Movement Interaction apply to Games
  • 17:30 am: Finish


To be confirmed


We plan on recording the outcomes of presentations and discussions in the event to be used as the basis for writing a position paper, to be submitted to ICCC 2021, on future directions on this specific facet of Computational Game Creativity - Immersive Game Interaction Design.

We will also make our software and materials available to all participants after the workshop and can provide ongoing support (funded by the Intelligent Games and Games Intelligence project) for them to use it in their own work. We therefore aim to benefit participants in the long term and develop strong collaborative relationships.

If you have any additional questions, comments, concerns or just want to say hello, please feel free to reach out to us directly.

Email: hello@interactml.com


Carlos Gonzalez Diaz

Carlos is a PhD Candidate in the Intelligent Games and Games Intelligence (IGGI) Centre for Doctoral Training at the University of York, Goldsmiths and Queen Mary universities. His research focuses on how the use of interactive machine learning in the design of movement interactions of virtual reality games can affect the player experience. During his PhD, Carlos has collaborated with Sony Interactive Entertainment R&D researching on the PSVR game system as well as participated in a Google-funded project to develop an interactive machine learning framework for the Unity3D game engine. He has recently co-organised academic events, being one of the chairs for ACM CHI Play 2019 and IEEE CoG 2019.