Gaussian Splats are a method for rendering 3D scenes photorealistically on as many devices as possible.
The story so far#
What sets Gaussian Splats apart from traditional rendering methods? Most techniques for displaying 3D data rely on 3D objects that are then textured. Over time, more and more effects (so-called shaders) were added to enable more realistic rendering. This approach works excellently when, for example, a video game needs to be displayed on a PC or a game console, but it is not the best method for scanning real-world objects and presenting them in a web browser.
Why isn’t this always practical?#
When real-world objects are scanned and displayed, the expectation is that these objects look in the virtual world exactly like their real-world counterparts. However, a realistic 3D scene requires more than just the polygons and texture of an object. Shaders are needed to modify the surface properties so that small irregularities can be represented (also known as tessellation) or to add reflections. Lighting must also be reconstructed precisely as it appeared in the original scene. Light must behave true to life and shine through and refract on semi-transparent surfaces.
Is there a better way?#
What sounds complex usually is. This is where Gaussian Splats come into play. They get by without shaders, reflections, or virtual light sources. Strictly speaking, there aren’t even textures involved anymore.
How can this work?#
Instead of pixels, so-called “splats” are used. Unlike pixels, these splats can be deformed into oval shapes to better represent structures of the real world. Instead of applying textures to these splats, each splat carries its own color values. This way, a splat changes its color depending on the viewing angle. Through this mathematical trick, shader problems can be bypassed entirely. To round out the experience, each splat is given a Gaussian blur.
A scene usually consists of thousands to millions of splats in three-dimensional space. The more splats are used, the more demanding the rendering and creation of the scene becomes.
Where’s the catch?#
So far, this sounds too good to be true. Gaussian Splats are a very new technique that competes with methods that are decades old, and therefore it has not yet reached the same technical maturity as alternative approaches. Scenes are created, for example, using a machine learning process, which requires enormous computing power. Depending on the complexity of the scene, even modern smartphones reach their limits, and the dreaded stuttering begins.
What exactly is needed for Gaussian Splats will follow in the next part of this series.