3D sensing provides the full spatial context of the world, which is important for applications such as augmented reality, virtual reality, and autonomous driving. Unfortunately, conventional cameras only capture a 2D projection of a 3D scene, while depth information is lost. In my research, I propose 3D sensors by jointly designing optics and algorithms. The key idea is to optically encode depth on the sensor measurement, and digitally decode depth using computational solvers. This allows us to recover depth accurately and robustly.

In the first part of my thesis, I explore depth estimation using wavefront sensing, which is useful for scientific systems. Depth is encoded in the phase of a wavefront. I build a novel wavefront imaging sensor with high resolution (a.k.a. WISH), using a programmable spatial light modulator (SLM) and a phase retrieval algorithm. WISH offers fine phase estimation with significantly better spatial resolution as compared to currently available wavefront sensors. However, WISH only provides a micron-scale depth range limited by the optical wavelength. To work for macroscopic objects, I propose WISHED, which increases the depth range by more than 1,000x. It is achieved based on the idea of wavelength diversity by combining the estimated phase at two close optical wavelengths. WISHED is capable of measuring transparent, translucent, and opaque 3D objects with smooth and rough surfaces.

In the second part of my thesis, I study depth recovery with 3D point spread function (PSF) engineering, which has wide applications for commercial devices. Depth is encoded into the blurriness of the image. To increase the PSF variation over depth, I propose to insert a phase mask on the lens aperture. Then, a deep learning-based algorithm is used to predict depth from the sensor image. To optimize the entire system, I developed an end-to-end optimization pipeline. The key insight is to incorporate the learning of hardware parameters by building a differentiable physics simulator that maps the scene to a sensor image. This simulator represents the optical layer of the deep neural network, followed by digital layers that represent the computational algorithm. This network is trained by datasets with a task-specific loss and outputs optimal parameters for both hardware and algorithms. Based on this idea, I develop two prototypes: PhaseCam3D - a passive single view depth sensor, and FreeCam3D - a structured light framework for scene depth estimation and localization with freely moving cameras. In summary, this thesis provides two 3D-sensing solutions with the idea of optical/digital co-design. I envision different modalities of 3D imaging to be widely adopted in the near future, enabling improved capabilities in many existing applications while revealing entirely new, hitherto unexplored application areas.