Project Gaussian Splatting for Mobile Mixed Reality Devices

Prof. Dr. Bernd Fröhlich
M.Sc. Adrian Kreskowski
M.Sc. Gareth Rendle
M.Sc. Anton Lammert
 

Novel-view synthesis techniques based on  Neural Radiance Fields [Mildenhall et al. 2020], Plenoxels [Fridovich-Keil et al. 2022], or, most recently and possibly most-well known, 3D Gaussian Splatting [Kerbl et al. 2023, Liu et al. 2024] enable the visually high-fidelity representation of surfaces that are hard or even almost impossible to reconstruct using classic photogrammetric approaches. Examples of such surfaces include fur, vegetation, transparent or translucent objects and thin structures in general. The novel-view synthesis approaches perform faithful interpolation of existing color information contained in a set of high-quality input images. Novel views can be rendered in real-time, provided one has access to reasonably powerful graphics hardware.

In a previous project, we explored the Gaussian Splatting literature and optimized an existing Unity-based rendering plugin for efficient rendering of Gaussian-based scenes on desktop graphics hardware. We also identified several challenges in rendering these models on mobile devices.

In this project, we aim to build on these insights and optimize Gaussian Splatting algorithms for mobile mixed reality (MR) devices such as the Meta Quest 3 or other mobile devices such as tablets. We will research, implement, and evaluate promising techniques in areas like visibility culling, output-sensitive rendering, data compression, and hybrid representations. Our goal is to fully leverage mobile hardware for real-time rendering at appropriate quality levels.

In addition to the challenge of efficiently rendering on low-power MR hardware, we want to address related research questions with part of the project team, such as how to interact with scene elements consisting of hundreds of thousands of unstructured Gaussian-based primitives or how to convincingly blend Gaussian-Splatting-based scenes with camera streams obtained by mixed-reality devices. 

If you are experienced with or interested in real-time computer graphics and/or topics in the field of mixed reality, we would be excited to welcome you to our project! 

We will provide you with a Quest 3 for the duration of the project and will address the challenges of rendering photorealistic real-world datasets on low-power MR hardware.

As well as willingness to work in a team, and enthusiasm for learning about and developing rendering techniques on cutting edge hardware, you should have the following competencies:

• Solid programming skills in C++ are required
• Successfully completed Computer Graphics course or equivalent qualifications
• Experience with GPGPU programming or algorithm design is welcome.

If you are in doubt as to whether you fulfil the requirements, or if you have any further questions regarding the project, we are happy to have a discussion with you before or after the project fair, or after that via email to adrian.kreskowski[at]uni-weimar.de.

The final assessment of your work will be conducted based on the project contributions of every team member, including:

• Active participation in the project during and in between weekly meetings
• Design, implementation and evaluation of Mixed-Reality-based Optimizations for efficient rendering of Gaussian Splatting within Unity or an own real-time rendering framework on mobile devices
• Intermediate talks
• Intermediate and final project presentations
• Documentation in form of a short paper

If you are experienced or interested in real-time computer graphics and/or topics in the field of mixed reality, we would be excited to welcome you to our project! 

We will provide you with a Quest 3 for the duration of the project and will address the challenges of rendering photorealistic real-world datasets on low-power MR hardware.