Music Performances in Virtual Reality

Prof. Dr. Bernd Fröhlich
M.Sc. Sebastian Mühlhaus

Music performances offer a vast design space that has historically evolved according to interactive multimedia systems. Artists and consumers alike appreciate a live music performance not only for auditory stimuli but for the sum of all aspects that can be found at a venue. Desktop systems, tablets and smartphones can only address this demand to a limited extent. That is why an increasing number of applications target VR technology to deliver compelling immersive experiences for both artists and audiences.

Our vision for a VR live concert experience includes an artistic representation for the performance, channels for communication of artist to crowd, crowd to artist and crowd to crowd as well as a suitable degree of agency for all participants. Tools and systems supporting such a concert experience require suitable integrations into the creative workflows of artists.

In this project, we will take first steps towards the aforementioned vision. Our investigations will begin by reviewing available applications and related work for VR performances and music-based creation of dynamic virtual environments. We will then evaluate the benefits and drawbacks of the various approaches. Finally, we will implement a set of promising techniques towards a system that enables music performances in VR.

Our students Karoline Brehm, Lucky Chandrautama, Simon Meininger, Margarita Osipova, Jonas Roquette and Tony Zöppig started their investigation by exploring related consumer applications. Along their observations and given academic related work, the group proceeded to identify qualitative factors involved with audiovisual media and music presentations as well as social virtual reality.

After identifying problem spaces with respect to sound spatialization, audience participation, signal processing and sound manipulation using expressive gestures, an outlook towards a contribution for this project was initially drafted. Using agile software engineering principles, the group incrementally implemented and refined their concepts and prototypic implementation.

The team was successful in designing a comprehensive toolset for the creation and collaborative exploration of immersive 3D audiovisual installations in social VR. Several tools such as mock-up sketches, diagrams and intermediate presentations guided the discussion, refinement and prioritization of ideas. As a result, a final demo was developed. It was implemented using Unity and runs on the Oculus Quest 2 HMD.

The MUVR application was showcased at Summaery 2022, where a co-located group of users could meet with remote located visitors of an experimental 3D audiovisual space. Conceptual and practical efforts of this project lead great potential for future research and development at our chair. Specifically the improvement of perceived synchrony in musical XR and building a system for live music performances in social VR continue to be points of interest for our future investigations.