4D-SPACE: 4D Scene and Performance Analysis in Collaborative virtual Environments
Collaborative virtual reality systems, such as our immersive group-to-group telepresence system [1], allow multiple users to interact in a shared virtual environment. Collaboration between distributed parties and in particular gestural communication can be facilitated by including realistic user representations (volumetric avatars). Such systems can be leveraged to analyse human actions and interactions. For example, researchers may want to study social interaction in realistic situations, but desire a strict control over the situation that a real-life setting may not afford [2]. An experiment that takes place in virtual reality can provide that control, while maintaining the plausibility of the situation. In creative fields, the possibility to create realistic virtual user representations gives physical performers like actors and dancers the opportunity to evaluate their movements with richer information than that provided by a simple video stream.
To support retrospective analysis of action and interaction, it is essential that user sessions in virtual environments can be recorded and subsequently replayed for exploration, annotation, and coding. In this project, we aim to develop a tool for 4D scene and performance analysis in collaborative environments. The software will be able to capture and replay multi-modal interaction between users in a virtual environment, as well as dynamic performances recorded in our lab space. Continuous information about users' position and orientation should be recorded, as well as audio streams for speech and conversation analysis. When realistic user representations, such as volumetric avatars are required, these should also be encoded in a manner that allows reconstruction at the original quality level.
The main challenges in this project are recording and synchronizing a plethora of different data streams, and storing them in a compact format that preserves the quality of the live reconstruction and allows the performance to be replayed on-demand for analysis and annotation purposes.
You have an affinity for real-time systems and in particular Unity, feel confident in C++ programming and are interested in asynchronous and concurrent programming? Great! You want to learn about standard compression libraries or even want to explore state-of-the-art compression papers to tackle the challenge of real-time compression of large data streams? Perfect! If at least one of the two sentences describes you, we would look forward to welcoming you in our project! |