Novel-view synthesis techniques based on Neural Radiance Fields [Mildenhall et al. 2020], Plenoxels [Fridovich-Keil et al. 2022], or, most recently and possibly most-well known, 3D Gaussian Splatting [Kerbl et al. 2023, Liu et al. 2024] enable the visually high-fidelity reconstruction of surfaces which are hard or even near-impossible to reconstruct using classic photogrammetric approaches. Examples of such surfaces include fur, vegetation, transparent or translucent objects and thin structures in general. The novel-view synthesis approaches perform faithful interpolation of existing color information contained in a set of high-quality input images. Novel views can be rendered in real-time, provided one has access to powerful graphics hardware.
First research [Lin et al. 2024] has emerged which aims at reducing the rendering workload of weaker mobile devices using foveated rendering techniques. However, to enable the exploration of high-quality datasets in virtual reality applications, it is necessary to design rendering algorithms with e.g. output-sensitivity in mind. In the first part of this project, we will explore existing rendering and acceleration techniques for novel-view synthesis by example of 3D Gaussian Splatting. After a detailed analysis of the rendering algorithms, we will design, implement and evaluate our own acceleration techniques for enabling real-time 3D Gaussian Splatting at high visual fidelity for state-of-the-art virtual reality devices.
In order to optimize performance for real-world datasets in virtual reality applications, we plan to explore an ocean floor dataset in virtual reality using head-mounted displays. The dataset will be captured and provided to us by the MARUM - Center for Marine Environmental Sciences at the beginning of our project.
If you are experienced or interested in real-time computer graphics and virtual reality, we would be excited to welcome you in our project. We will provide you with a Quest 3 for the duration of the project and together we will get our feet wet with our challenging real-world dataset and Efficient Gaussian Splatting for Virtual Reality! |
