libframe
ISP and 3D Vision
libframe is a hybrid C/C++ library with dual intent: to learn more about 3D vision and the low level image signal processing pipeline. In process, we also end up finding rendering techniques are useful to learn about the how cameras, lenses and sensors work. I don’t intend this project to be a from scratch project albeit only using other libraries where I feel I am not focussing on a certain area.
Date: 5/5/25
Developed basic understanding of the basic image signal processing pipeline and first few steps of the pipeline from the crux’s openISP repository. Within libframe, I took the test.RAW from openISP and got the image loader running.
Given my interest in learning about cameras, I am finally at the point where I realize that I just need to sit down and write the basic structure for pinhole camera and tinker. I used to think that book like “Multiple View Geometry” and “An Invitation to 3-D Vision” were too compact and where would I even start. Thanks to the experience from voxelminer project and CSE160, I was able to write a basic pinhole camera. I had to simulate the camera behavior as it happens in real world where light rays are captured by the camera sensor. This is what I realized today is PathTracing although I thought RayTracing was the only way I could do this. I followed the notes in “Ray Tracing in One Weekend” and was able to get the most basic scene working shown below:
Importantly, RayTracing for each pixel in image plane, we find the ray from that point to camera’s center and then find the intersection of that ray with the scene which can have anything.
Date: 5/10/25
Planning to completely separate out the graphics portion to just voxelminer and keep the ISP/3D vision portion in libframe. The graphics portion is better suited to be in voxelminer despite an argument could be made that I should create a new separate project. I can make a new project for graphics in Vulkan and keep it separate from voxelminer(webgl2) albeit I don’t want to fragment the projects leading to more confusion for myself. Hence planning to add the desktop focussed vulkan project also in voxelminer.
In meantime, I have been working on the ISP portion of the project where I got dead pixel correction working. I had io for the raw image working last time and now we are able to do dead pixel correction on it. Technique I am using is mean based approach where I check if the center pixel is much different from neighbors and if it is, I replace it with the mean of the neighbors. Finally I had save_to_ppm working which acts like my debugging tool albeit <img src =""> doesn’t work with ppm hence showing a png of the dead pixel corrected image from test.RAW input image.
Next step is to implement black level compensation according to the openISP pipeline.
Date: 6/5/25
I recently ordered Raspberry Pi 5 with 12MP IMX477 camera with it. It got delivered the next day morning. I am so excited about this new tool I have access to, I had been thinking of getting the pi and camera for a long while now. General goal with both those items is to complement my projects, especially this ISP project. I have access to a real camera now where I can control aperture and near/far. I was warmly welcomed to the RasPi by a wallpaper of a fisherman:
Then I wanted to take pictures with this beautiful camera I had. I spent good 20 minutes in figuring out how to wire this up with Raspberry Pi due to 16 different combinations(RPI 5 has 2 cam slots) and lack of information. That’s a blog post for another day on camera serial interface.
I got it wired up finally. libcamera-hello opens up the preview for 5 sec, I wanted to see for longer. Finally with libcamera-hello --timeout 0, I got to see infinite preview. This helped to tinker with aperture, near/far of the camera. I noticed:
- Aperture affects the number of light rays entering the camera, the exposure, brightness. So if it’s too big, leads to blurryness..too small then also bluriness or completely darkness.
- Near/Far helped me focus the rays on the image plane. There was a sweet spot in middle of circular movement which gave a crisp image.
libcamera-jpeg -o test.jpgtakes photo with a certain delay. It’s upside down as you should expect from the camera and also this camera is a rolling shutter based camera. Image taken is shown below:
