libframe
ISP and 3D Vision
libframe is a hybrid C/C++ library with dual intent: to learn more about 3D vision and the low level image signal processing pipeline. In process, we also end up finding rendering techniques are useful to learn about the how cameras, lenses and sensors work. I don’t intend this project to be a from scratch project albeit only using other libraries where I feel I am not focussing on a certain area.
Date: 5/5/25
Developed basic understanding of the basic image signal processing pipeline and first few steps of the pipeline from the crux’s openISP repository. Within libframe, I took the test.RAW from openISP and got the image loader running.
Given my interest in learning about cameras, I am finally at the point where I realize that I just need to sit down and write the basic structure for pinhole camera and tinker. I used to think that book like “Multiple View Geometry” and “An Invitation to 3-D Vision” were too compact and where would I even start. Thanks to the experience from voxelminer project and CSE160, I was able to write a basic pinhole camera. I had to simulate the camera behavior as it happens in real world where light rays are captured by the camera sensor. This is what I realized today is PathTracing although I thought RayTracing was the only way I could do this. I followed the notes in “Ray Tracing in One Weekend” and was able to get the most basic scene working shown below:
Importantly, RayTracing for each pixel in image plane, we find the ray from that point to camera’s center and then find the intersection of that ray with the scene which can have anything.
Date: 5/10/25
Planning to completely separate out the graphics portion to just voxelminer and keep the ISP/3D vision portion in libframe. The graphics portion is better suited to be in voxelminer despite an argument could be made that I should create a new separate project. I can make a new project for graphics in Vulkan and keep it separate from voxelminer(webgl2) albeit I don’t want to fragment the projects leading to more confusion for myself. Hence planning to add the desktop focussed vulkan project also in voxelminer.
In meantime, I have been working on the ISP portion of the project where I got dead pixel correction working. I had io for the raw image working last time and now we are able to do dead pixel correction on it. Technique I am using is mean based approach where I check if the center pixel is much different from neighbors and if it is, I replace it with the mean of the neighbors. Finally I had save_to_ppm working which acts like my debugging tool albeit <img src =""> doesn’t work with ppm hence showing a png of the dead pixel corrected image from test.RAW input image.
Next step is to implement black level compensation according to the openISP pipeline.