Ray geometry in multi-perspective cameras: a case study of xslit imaging and its applications

A pinhole camera collects rays passing through a common 3D point and its image resembles what would be seen by human eyes. In contrast, a multi-perspective camera combines rays collected by different viewpoints. Such capabilities can potentially benefit a broad class of imaging applications, ranging from scene understanding, to high quality imaging, and to 3D reconstructions. In this dissertation, I thoroughly discuss designing, modeling, and constructing general multi-perspective cameras. The unique approach I adopt is a ray geometry analysis that uniformly models arbitrary multi-perspective cameras as manifolds of rays and ray constraints. Using the theoretical underpinning, I explore the ray geometry in a special type of multi-perspective camera, the XSlit camera. An XSlit camera collects rays that simultaneously pass through two oblique (neither parallel nor coplanar) slits in 3D space. I show that same as in the pinhole camera, images of parallel lines in an XSlit image, although curved, will still converge at a vanishing point, i.e., the XSlit Vanishing Point (XVP). What is different though is that images of coplanar 3D lines will generally intersect at a second common point that we call Coplanar Common Point (CCP). CCP is a special feature of XSlits that does not exist in pinholes. I then present a comprehensive theory to analyze XVPs and CCPs of a Manhattan World (MW) scene. I show that the geometry of 3D lines can be directly recovered from their XVPs and CCPs. I further develop a robust algorithm to distinguish the two types of points of complex MW scenes using a single XSlit image. I further show that it is possible to conduct stereo matching using XSlit cameras. Specifically, I investigate a different, rotational XSlit (R-XSlit) model. I show that R-XSlit stereo can be effectively created by fixing the sensor and slit locations but switching the two slits' directions. I first derive the epipolar geometry of R-XSlit in the 4D light field ray space. The derivation leads to a simple but effective scheme for locating corresponding epipolar curves. To conduct stereo matching, I further define a new disparity term in the R-XSlit model and develop a patch-based graph-cut solution. I show through experiments that R-XSlit provides a potentially advantageous imaging system for conducting fixed-location, dynamic baseline stereo. An important component of this dissertation is to construct a real XSlit camera. I assemble an XSlit lens by using a pair of cylindrical lenses coupled with slit-shaped apertures. The XSlit lens can be mounted on commodity cameras where the slit directions can be easily adjusted. The imaging quality relies heavily on the choice of the slit aperture width: narrow aperture produces a sharp image but the light efficiency is low; whereas a wide aperture generates brighter images but with a narrower Depth-of-Field (DoF). I hence present an XSlit light efficiency analysis as well as develop a novel class of XSlit Depth-from-Defocus (DfD) techniques. In particular, I show that compared with (single) spherical thin lens, XSlit lens produces a deeper DoF with the same light throughput. Further, we can independently code each slit aperture to simultaneously improve kernel invertibility and depth discrepancy in DfD. I foresee that the XSlit imaging system may fundamentally change how people think about cameras and images and significantly advance imaging science.