View Camera Scene PostProcessing Shader: sampling rgb-d image camera(s)

Problem

In our multi-camera setups with known pose and RGB-Depth images, we can accurately sample pixels from an Image Camera into each pixel in View Camera via a fragment shader. The current approach of treating pixels as discrete squares leads to aliasing and perspective distortion issues, especially with nearby cameras and parallax effects.

Solution

Implement a post-process fragment shader in CesiumJS that:

• Projects each pixel from the View Camera into the Image Camera using basic camera matrices.
• Samples the corresponding color from the Image Camera's RGB-Depth texture.
• Handles parallax and perspective distortions by reconstructing the world position and reprojecting it.

Core Concept

1. World Position Reconstruction

Each pixel in the View Camera is treated as a directional ray on a panosphere, defined by azimuth (ϕ) and elevation (θ) and a guaussian ellipse sample (covariance matrix). The depth texture is used to reconstruct the world position of each pixel as follows:

P_world = M_viewProj-1 * P_clip

• P_clip: Clip space position reconstructed from depth.
• M_viewProj-1: Inverse View-Projection matrix of the View Camera.
• P_world: Reconstructed world position.

2. Reprojection to Image Camera

The world position is then projected into the Image Camera's screen space using its known view and projection matrices:

P_image = M_projimage * M_viewimage * P_world

• M_viewimage: View matrix of the Image Camera.
• M_projimage: Projection matrix of the Image Camera.
• P_image: Projected screen space position on the Image Camera.

The projected coordinates are then converted to texture space (UV) for sampling:

UV_image = (P_image.x / P_image.w * 0.5 + 0.5, P_image.y / P_image.w * 0.5 + 0.5)

3. Image Camera Sampling

Using the calculated UV coordinates, the corresponding color is sampled from the Image Camera's RGB-Depth texture:

Color_sampled = Texture2D(ImageColorTexture, UV_image)

Depth verification (optional) can be done to ensure correct correspondence and manage occlusions.

Rabbit Hole: Advanced Considerations

• Lens Distortion and Fisheye Effects: To be addressed in future iterations with undistortion mapping.
• Gaussian Elliptical Sampling: For smoother interpolation and anti-aliasing, leveraging Gaussian weights.

Risks

• Performance Impact: Multi-sampling and depth verification may require optimization for real-time rendering.
• Edge Cases: Handling occlusions and parallax effects across different camera poses may introduce artifacts.

Why Now?

This implementation is crucial for accurate multi-view compositing and rendering in our current projects using CesiumJS, especially for scenarios involving nearby cameras with RGB-Depth information.

No-Gos

• No complex distortion models or Gaussian sampling in this iteration—focus on basic reprojection first.
• No implementation of fisheye-specific sampling strategies yet.

Next Steps

1. Implement as a PostProcessStage in CesiumJS using GLSL for reprojection calculations.
2. Test with various camera poses, fields of view, and depth configurations.
3. Profile performance and optimize sampling density and texture lookups.