Gaussian Splatting

Exploring 3D Gaussian Splatting for Novel View Synthesis

In this post, I’ll be talking about my current experience with Gaussian Splatting which is a popular 3D Scene Reconstruction technique these days. Gaussian Splatting was originally introduced by Lee Westover in the 1990s but only recently methods like 3D Gaussian splatting has been introduced for real-time radiance field rendering and novel view synthesis.

Iteration 1

3D Gaussian Splatting is a 3D Scene Reconstruction technique which recreates a particular scene given multiple views of the scene and provides access to novel views that weren’t necessarily directly in the original set of views.

3D Gaussian is like yellow smiley ball but more malleable and we have thousands/millions of these optimized to represent the scene after a careful optimization process of fitting the Gaussians to the available views. These Gaussians have gradient, color, transparency, and shape parameters associated with each of them. .

I wanted to start with getting the basics right first so I thought it might just be better to use the already available pipeline as compared to writing my own and get a basic pipeline going. In my current state, my focus is establishing that it’s doable and approachable and I understand it just enough to dive deep into the details. Yes, I want to write it in raw C++, entirely from scratch too but there is time for that.

First things first, 3D Gaussian Splatting is broadly a two-step process:

  • Getting sparse reconstruction and camera poses from COLMAP
  • Utilizing the sparse reconstruction and camera poses for the optimization process of the Gaussians

Sparse Reconstruction with COLMAP

COLMAP is a very popular structure from motion tool that is extensively used throughout the computer vision community for calculating camera poses from video frames or image set taken in an environment. Utilizing COLMAP is a 3-4 steps process and the important steps are:

  • Feature Extraction
  • Feature Matching
  • Sparse Reconstruction
  • (Optional) Dense Reconstruction

Given I was using mac I had to build COLMAP from source, after building it and installing it system-wide, I started to use it for sparse reconstruction. This sparse reconstruction is used by Gaussian splatting pipelines. I had to record a video first from which I’ll fetch the appropriate views.

After recording the video, I used ImageMagick command line utilities to extract 3 frames per second to create the image dataset for the COLMAP sparse reconstruction. Given we have COLMAP built with GUI working and data organized in the structure like:

splat_test/
 - images/
 - 1.jpg
 - 2.jpg
 - .
 - .

We open the COLMAP GUI likely with colmap gui for me at least. Then we start a new project by selecting a new project in Files->New Project and then creating a new database.db in splat_test/ and selecting splat_test/images for Images section in form. After that, there are three steps:

  • Click on Processing->Feature Extraction with parameters of camera Shared for all images selected unless they aren’t and run it with everything else as is. Close the dialog after logs show Elapsed time: abc
  • Click on Processing->Feature Matching with default settings and run it. This uses a GPU implementation of the SIFT detector to match features between images which are finally used to create the sparse reconstruction we need. By default, it does exhaustive feature matching between all combinations. Close the dialog after logs show Elapsed time: abc
  • Finally, we will do Reconstruction->Start Reconstruction which will do a sparse reconstruction of the image views and features amongst them.
  • Finally, we need to export the model to splat_test/ with Files->Export Model and it will output 4 files in splat_test/ and we need to organize it in a structure and it all should look like the below in end:
splat_test/
 - images/
 - 1.jpg
 - 2.jpg
 - .
 - . 
 - sparse/
 - 0/
 - cameras.bin
 - points3D.bin
 - images.bin
 - project.ini
 - database.db
 - database.db-shm
 - database.db-wal

We will be using this as the input for the next part of the process which utilizes OpenSplat.

Optimization process with OpenSplat

I used OpenSplat as it provided Metal GPU support and Results from SFM process using COLMAP generate sparse point cloud which is used to initialize the Gaussians and thereafter begins the optimization process of the Gaussians which is focused on refining these Gaussians to represent the scene better.

Now assume you are in opensplat/build/ at end of it all and there is opensplat executable there.

You need to run:

  • ./opensplat PATH_TO_IT/splatting_test/ -o PATH_TO_IT/splatting_test/test.splat -n 2000

This generates your Gaussian splat which is optimized over Gaussians in 3d.

This likely has a lot of artifacts. Clean up with https://playcanvas.com/supersplat/editor and download your splat again. Lasso Select is pretty useful.

Visualization

The final gaussian splats can be visualized at https://ashwanirathee.com/cs/apps/splat/ which is forked from Kevin Kwok’s splat viewer.

Results:

References

  • https://articles.tomasparks.name/publications/Westover1991.pdf
  • https://ashwanirathee.com/cs/apps/splat/
  • https://playcanvas.com/supersplat/editor
  • https://antimatter15.com/splat/(Kevin Kwok’s Splat Viewer)
  • https://playcanvas.com/supersplat/editor