Browsing by Author "Veeraraghavan, Ashok"

Now showing 1 - 20 of 88
  • Thumbnail Image
    Item
    3D sensing by optics and algorithm co-design
    (2021-04-29) Wu, Yicheng; Veeraraghavan, Ashok
    3D sensing provides the full spatial context of the world, which is important for applications such as augmented reality, virtual reality, and autonomous driving. Unfortunately, conventional cameras only capture a 2D projection of a 3D scene, while depth information is lost. In my research, I propose 3D sensors by jointly designing optics and algorithms. The key idea is to optically encode depth on the sensor measurement, and digitally decode depth using computational solvers. This allows us to recover depth accurately and robustly. In the first part of my thesis, I explore depth estimation using wavefront sensing, which is useful for scientific systems. Depth is encoded in the phase of a wavefront. I build a novel wavefront imaging sensor with high resolution (a.k.a. WISH), using a programmable spatial light modulator (SLM) and a phase retrieval algorithm. WISH offers fine phase estimation with significantly better spatial resolution as compared to currently available wavefront sensors. However, WISH only provides a micron-scale depth range limited by the optical wavelength. To work for macroscopic objects, I propose WISHED, which increases the depth range by more than 1,000x. It is achieved based on the idea of wavelength diversity by combining the estimated phase at two close optical wavelengths. WISHED is capable of measuring transparent, translucent, and opaque 3D objects with smooth and rough surfaces. In the second part of my thesis, I study depth recovery with 3D point spread function (PSF) engineering, which has wide applications for commercial devices. Depth is encoded into the blurriness of the image. To increase the PSF variation over depth, I propose to insert a phase mask on the lens aperture. Then, a deep learning-based algorithm is used to predict depth from the sensor image. To optimize the entire system, I developed an end-to-end optimization pipeline. The key insight is to incorporate the learning of hardware parameters by building a differentiable physics simulator that maps the scene to a sensor image. This simulator represents the optical layer of the deep neural network, followed by digital layers that represent the computational algorithm. This network is trained by datasets with a task-specific loss and outputs optimal parameters for both hardware and algorithms. Based on this idea, I develop two prototypes: PhaseCam3D - a passive single view depth sensor, and FreeCam3D - a structured light framework for scene depth estimation and localization with freely moving cameras. In summary, this thesis provides two 3D-sensing solutions with the idea of optical/digital co-design. I envision different modalities of 3D imaging to be widely adopted in the near future, enabling improved capabilities in many existing applications while revealing entirely new, hitherto unexplored application areas.
  • Thumbnail Image
    Item
    A Robust Algorithm for Identification of Motion Artifacts in Photoplethysmography Signals
    (2018-12-03) Maity, Akash Kumar; Sabharwal, Ashutosh; Veeraraghavan, Ashok; Heckel, Reinhard
    Photoplethysmography(PPG) is commonly used as a means of continuous health monitoring. Many clinically relevant parameters like heart rate (HR), blood oxygenaton level (SPO2) are derived from the sensor measurements using PPG. Presence of motion artifacts in the signal decreases the accuracy of estimating the parameters and therefore reduces the reliabilty of these sensor devices. Motion artifacts can be both periodic or aperiodic. Existing state-of-the-art methods for motion detection rely on the semi-periodic structure of PPG to distinguish from aperiodic motion artifacts. Periodic motion artifacts that can be introduced by perioidic movements like hand tapping, jogging, cannot be detected by current methods reliably. In this thesis, we propose a novel technique, PPGMotion, for identifying all motion artifacts in PPG signals. PPGMotion relies on the morphological structure of artifact-free PPG signal, which has a fast systolic phase and a slowly decaying diastolic phase. We note that in the presence of motion artifacts, the recorded PPG signals do not exhibit the characteristic PPG shape. Our approach uses this prior information about the PPG morphology to reliable detect periodic motion artifacts, without the need of any additional hardware components like an accelerometer. To evaluate the proposed method, we adopt both a simulation and real data collection. For simulation-based iii analysis, we use a generative model for motion artifacts to simulate different cases of motion artifacts. For real data, we have compared our approach against recent works on motion identification using 3 datasets, where we record the PPG from a pulse-oximeter attached to a finger with subjects making (1) random finger movements, (2) periodic movements like periodic finger tapping and (3) PPG recordings from Maxim smartwatch with subjects running on a treadmill. Dataset (2) and (3) are expected to introduce periodic motion artifacts in the measured PPG signals. We demonstrate that while our approach is similar in performance to previous methods when random motion artifacts are introduced, the performance is significantly better in the presence of periodic motion artifacts. We show that for simulated dataset, the performance of PPGMotion is significantly better than existing work as the contaminated PPG tends to become periodic, with an increase in sensitivity of atleast 10% over state-of-the-art method. For real data, PPGMotion is successful in identifying the periodic motion artifacts, with mean sensitivity of 95% and accuracy of 95.8%, compared to the state-of-the-art method with mean sensitivity of 66% and accuracy of 89% for dataset (2). For dataset (1), PPGMotion achieves an accuracy of 96.35% with sensitivity of 95.29%, and for dataset (3), PPGMotion achieves an accuracy of 91.89% and sensitivity of 93.03%, compared to the second best method with accuracy 81.23% and sensitivity 74.99%.
  • Thumbnail Image
    Item
    Accelerated MRI with Un-trained Neural Networks
    (2021-03-05) Zalbagi Darestani, Mohammad; Heckel, Reinhard; Veeraraghavan, Ashok
    Convolutional Neural Networks (CNNs) are highly effective for image reconstruction problems. Typically, CNNs are trained on large amounts of training images. Recently, however, un-trained CNNs such as the Deep Image Prior and Deep Decoder have achieved excellent performance for image reconstruction problems such as denoising and inpainting, without using any training data. In this work, we study the performance of un-trained neural networks for the reconstruction problem arising in accelerated Magnetic Resonance Imaging (MRI). For this purpose, we view the performance from two perspectives: reconstruction accuracy, and reconstruction reliability. Reconstruction accuracy: One of the main goals in solving image reconstruction tasks is obtaining a high-quality image. In order to measure the quality of the images reconstructed by an un-trained network, we first propose a highly-optimized un-trained recovery approach based on a variation of the Deep Decoder. Afterward, through extensive experiments, we show that the resulting method significantly outperforms conventional un-trained methods such as total-variation norm minimization, as well as naive applications of un-trained networks. Most importantly, the proposed approach achieves on-par performance with a standard trained baseline, the U-net, on the FastMRI dataset, a dataset for benchmarking deep-learning based reconstruction methods. This demonstrates that there is less benefit in learning a prior for solving the accelerated MRI reconstruction problem. This conclusion is drawn by comparison with a baseline trained neural network, however, state-of-the-art methods still slightly outperform U-net (and hence the proposed approach in this work). Reconstruction reliability: Recent works on accelerated MRI reconstruction suggest that trained neural networks are not reliable for image reconstruction tasks, albeit achieving excellent accuracy. For example, at the inference time, small changes (also referred to as adversarial perturbations) in the input of the network can result in significant reconstruction artifacts. In this regard, we analyze the robustness of trained and un-trained methods. Specifically, we consider three notions of robustness: (i) robustness against small changes in the input, (ii) robustness in recovering small details in the image, and (iii) robustness to distribution shifts. Our main findings from this analysis are the followings: (i) contrary to the current belief, neither of the trained and un-trained methods is robust to small changes in the input, and (ii) in opposition to trained neural networks, un-trained methods are naturally robust to data distribution shifts, and interestingly, an un-trained neural network outperforms a trained one after the distribution shift. This work promotes the use of un-trained neural networks for accelerated MRI reconstruction through the following conclusions. First, in terms of accuracy, un-trained neural networks yield high-quality reconstructions, significantly better than conventional un-trained methods and similar to baseline trained methods. Second, a key advantage of un-trained networks over trained ones is a better generalization to unseen data distributions.
  • Thumbnail Image
    Item
    An Objective System for Quantitative Assessment of Television Viewing Among Children (Family Level Assessment of Screen Use in the Home-Television): System Development Study
    (JMIR, 2022) Vadathya, Anil Kumar; Musaad, Salma; Beltran, Alicia; Perez, Oriana; Meister, Leo; Baranowski, Tom; Hughes, Sheryl O.; Mendoza, Jason A.; Sabharwal, Ashutosh; Veeraraghavan, Ashok; O'Connor, Teresia
    Background: Television viewing among children is associated with developmental and health outcomes, yet measurement techniques for television viewing are prone to errors, biases, or both. Objective: This study aims to develop a system to objectively and passively measure children’s television viewing time. Methods: The Family Level Assessment of Screen Use in the Home-Television (FLASH-TV) system includes three sequential algorithms applied to video data collected in front of a television screen: face detection, face verification, and gaze estimation. A total of 21 families of diverse race and ethnicity were enrolled in 1 of 4 design studies to train the algorithms and provide proof of concept testing for the integrated FLASH-TV system. Video data were collected from each family in a laboratory mimicking a living room or in the child’s home. Staff coded the video data for the target child as the gold standard. The accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were calculated for each algorithm, as compared with the gold standard. Prevalence and biased adjusted κ scores and an intraclass correlation using a generalized linear mixed model compared FLASH-TV’s estimation of television viewing duration to the gold standard. Results: FLASH-TV demonstrated high sensitivity for detecting faces (95.5%-97.9%) and performed well on face verification when the child’s gaze was on the television. Each of the metrics for estimating the child’s gaze on the screen was moderate to good (range: 55.1% negative predictive value to 91.2% specificity). When combining the 3 sequential steps, FLASH-TV estimation of the child’s screen viewing was overall good, with an intraclass correlation for an overall time watching television of 0.725 across conditions. Conclusions: FLASH-TV offers a critical step forward in improving the assessment of children’s television viewing.
  • Thumbnail Image
    Item
    Bioluminescent flashes drive nighttime schooling behavior and synchronized swimming dynamics in flashlight fish
    (Public Library of Science, 2019) Gruber, David F.; Phillips, Brennan T.; O’Brien, Rory; Boominathan, Vivek; Veeraraghavan, Ashok; Vasan, Ganesh; O’Brien, Peter; Pieribone, Vincent A.; Sparks, John S.
    Schooling fishes, like flocking birds and swarming insects, display remarkable behavioral coordination. While over 25% of fish species exhibit schooling behavior, nighttime schooling has rarely been observed or reported. This is due to vision being the primary modality for schooling, which is corroborated by the fact that most fish schools disperse at critically low light levels. Here we report on a large aggregation of the bioluminescent flashlight fish Anomalops katoptron that exhibited nighttime schooling behavior during multiple moon phases, including the new moon. Data were recorded with a suite of low-light imaging devices, including a high-speed, high-resolution scientific complementary metal-oxide-semiconductor (sCMOS) camera. Image analysis revealed nighttime schooling using synchronized bioluminescent flashing displays, and demonstrated that school motion synchrony exhibits correlation with relative swim speed. A computer model of flashlight fish schooling behavior shows that only a small percentage of individuals need to exhibit bioluminescence in order for school cohesion to be maintained. Flashlight fish schooling is unique among fishes, in that bioluminescence enables schooling in conditions of no ambient light. In addition, some members can still partake in the school while not actively exhibiting their bioluminescence. Image analysis of our field data and model demonstrate that if a small percentage of fish become motivated to change direction, the rest of the school follows. The use of bioluminescence by flashlight fish to enable schooling in shallow water adds an additional ecological application to bioluminescence and suggests that schooling behavior in mesopelagic bioluminescent fishes may be also mediated by luminescent displays.
  • Thumbnail Image
    Item
    Camera based photoplethysmogram estimation
    (2018-04-10) Kumar, Mayank; Veeraraghavan, Ashok; Sabharwal, Ashutosh; Rice University; United States Patent and Trademark Office
    A system for estimating a photoplethysmogram waveform of a target includes an image processor configured to obtain images of the target and a waveform analyzer. The waveform analyzer is configured to determine a weight of a portion of the target. The weight is based on a time variation of a light reflectivity of the portion of the target. The time variation of the light reflectivity of the target is based on the images. The waveform analyzer is further configured to estimate a PPG waveform of the target based on the weight of the portion and the time variation of the light reflectivity of the portion.
  • Thumbnail Image
    ItemEmbargo
    Camera-based Tissue Hemodynamics Imaging
    (2023-04-21) Maity, Akash Kumar; Sabharwal, Ashutosh; Veeraraghavan, Ashok
    Blood flow changes within the human tissue have two main characteristics- i) the temporal pulsatile variation caused by the regular heart beat, and ii) the spatial variation that captures the presence of blood vessels beneath the skin surface. Hemodynamics, which describes blood flow throughout the body, is important to monitor for many medical conditions. The use of visible and near-infrared light for deep tissue hemodynamics imaging is emerging as a low-cost and safe alternative to some of the existing state-of-the-art technologies. However, the best camera-based systems suffer from a poor signal-to-noise ratio and low signal contrast regime due to light interaction in the tissue. Hence, accurate estimation using light-based imaging remains an open problem. In this thesis, we develop two camera-based systems to estimate two dimensions of tissue hemodynamics: i) heart rate as a measure of temporal variation, and ii) deep tissue perfusion to map spatial variation across the tissue. In the first part, we present RobustPPG, a camera-based, motion-robust imaging technique for estimating heart rate accurately from human face videos under normal ambient illumination. We explicitly model and generate motion distortions due to the movements of the person's face using inverse rendering. The generated motion distortion is then used to filter the motion-induced measurements. We demonstrate that our approach performs better than the state-of-the-art methods in extracting a clean blood volume signal with over $2$ dB signal quality improvement and $30\%$ improvement in RMSE of estimated heart rate in intense motion scenarios. In the second part of our thesis, we present SpeckleCam, a camera-based system to recover deep tissue blood perfusion in high resolution. We use a line scanning system and a fast algorithm for recovering high-resolution blood flow deep inside the tissue. Our approach replaces the traditional matrix-multiplication form with a convolution-based forward model that enables us to develop an efficient and fast blood flow reconstruction algorithm, with $10\times$ speedup in runtime compared to the prior methods. We show that our proposed approach can recover complex structures $6$ mm deep inside a tissue-like scattering medium in the reflection geometry. We also demonstrate the sensitivity of our approach to real tissue to detect reduced flow in major blood vessels during arm occlusion.
  • Thumbnail Image
    Item
    Camera-based Vital Signs: Towards Driver Monitoring and Face Liveness Verification
    (2018-08-20) Nowara, Ewa Magdalena; Veeraraghavan, Ashok; Heckel, Reinhard
    I show how remote photoplethysmography (rPPG), which are blood flow induced intensity variations in the skin observed with a camera, can improve driver monitoring and face liveness verifcation. A leading cause of car accidents is driver distraction. These accidents could be prevented by monitoring drivers rPPG signals while driving. However, it is challenging to measure rPPG signals in a moving vehicle due to drastic illumination variations and large motion. I built a narrow-band near-infrared set up to reduce outside illumination variations and I developed an algorithm called SparsePPG to exploit spatial low rankness and sparsity in frequency of rPPG signals. Face recognition algorithms can provide highly secure user authentication due to their high accuracy; however, they cannot distinguish between authentic faces and face attacks, such as photographs. I developed an algorithm called PPGSecure which uses rPPG signals from a face video recording and machine learning to detect these face attacks.
  • Thumbnail Image
    Item
    Compressive Implicit Radar for High-accuracy Millimeter-wave Imaging
    (2023-11-20) Farrell, Sean; Veeraraghavan, Ashok; Sabharwal, Ashutosh
    Millimeter wave (mmWave) imaging is becoming an increased area of interest due to the advancement of commercially available low-cost on-chip mmWave devices and the unique features mmWaves offer. Millimeter waves, unlike optical waves, can penetrate through common visibly degraded environments caused by smoke, fog, or dust. This unique advantage makes mmWave imaging ideal for security monitoring, autonomous vehicles, and simultaneous localization and mapping (SLAM) applications. However, the main limitation of current mmWave devices is they suffer from low angular resolution due to small physical apertures and conventional signal processing techniques. Constructing large physical arrays is expensive and leads to computational challenges due to the increased readout bandwidth. Deep learning approaches have been investigated to achieve high resolution mmWave imaging using downsampled apertures. However, these deep learning techniques typically require large amounts of training data and have limited generalizability. Recently, untrained neural networks (NN) have shown outstanding performance on challenging computational imaging inverse problems such as denoising, inpainting, and deblurring without using any training data. In this work, we design and evaluate the performance of untrained neural networks and sub-sampled apertures for sparse multi-input multi-output (MIMO) radar imaging. We approach the design of our proposed sparse radar imaging method from two aspects: aperture sub-sampling scheme and NN architecture. We design a minimum redundant array (MRA) inspired sub-sampled aperture that reduces the radar read-out bandwidth by $75\%$ compared to conventional MIMO multiple radar chip Nyquist sampled aperture designs. This allows our method to work with a single radar chip and demonstrates that there is significant compressed sensing that can be applied in radar aperture design. The second design aspect of our method is the NN architecture, where we analyzed state-of-art untrained designs using convolutional neural networks (CNN) and multi-layer perceptions (MLP). Through extensive experiments, we find that current untrained NN designs work well for natural images but achieve subpar performance on radar images. We propose a CNN decoder architecture inspired by Deep Decoder that is highly optimized for sparse radar imaging. In our experiments we show that our NN architecture significantly outperforms competing untrained methods. A crucial component in evaluating the performance of untrained methods for sparse radar imaging is analyzing the image reconstruction quality, generalizability to scene changes, and inference time. We find that the proposed untrained method is capable of performing joint deblurring and denoising without requiring any training data. The proposed method suppresses aliasing artifacts from the sparse aperture design as well as sinc-like artifacts from the finite aperture size. We demonstrate that the proposed method achieves high reconstruction quality in a variety of outdoor and indoor scenes using real experimental data collected with a 76-81 GHz mmWave radar. One of the main limitations of untrained methods is the slower inference time compared to trained methods. In response, we propose a new initialization scheme that leverages the fact that between radar frames the scene does not change significantly and thus we use the weights of the trained NN from the previous frame for initialization. We demonstrate that, using our new untrained NN initialization scheme we can achieve a $5\times$ speed up in inference time compared to using random initialization. Overall, this work advocates for the use of untrained NN to achieve high resolution sparse radar imaging. We find that our proposed sub-sampled aperture and untrained NN architecture out perform competing untrained baselines and achieves similar imaging performance to conventional approaches using a Nyquist sampled aperture. Additionally, we find that the proposed method is robust to environmental changes and can achieve inference times on the order of seconds with novel NN initialization schemes.
  • Thumbnail Image
    Item
    CS-ToF: High-resolution compressive time-of-flight imaging
    (Optical Society of America, 2017) Li, Fengqiang; Chen, Huaijin; Pediredla, Adithya; Yeh, Chiakai; He, Kuan; Veeraraghavan, Ashok; Cossairt, Oliver
    Three-dimensional imaging using Time-of-flight (ToF) sensors is rapidly gaining widespread adoption in many applications due to their cost effectiveness, simplicity, and compact size. However, the current generation of ToF cameras suffers from low spatial resolution due to physical fabrication limitations. In this paper, we propose CS-ToF, an imaging architecture to achieve high spatial resolution ToF imaging via optical multiplexing and compressive sensing. Our approach is based on the observation that, while depth is non-linearly related to ToF pixel measurements, a phasor representation of captured images results in a linear image formation model. We utilize this property to develop a CS-based technique that is used to recover high resolution 3D images. Based on the proposed architecture, we developed a prototype 1-megapixel compressive ToF camera that achieves as much as 4× improvement in spatial resolution and 3× improvement for natural scenes. We believe that our proposed CS-ToF architecture provides a simple and low-cost solution to improve the spatial resolution of ToF and related sensors.
  • Thumbnail Image
    Item
    Data-Driven Computational Imaging with Applications to Imaging Through and Around Obstacles
    (2019-01-11) Metzler, Chris A; Baraniuk, Richard G.; Veeraraghavan, Ashok
    This thesis takes a data-driven approach to two of the foremost problems in optics; imaging through scattering media and imaging around corners using visible light.
  • Thumbnail Image
    Item
    Deep imaging in scattering media with selective plane illumination microscopy
    (SPIE, 2016) Pediredla, Adithya Kumar; Zhang, Shizheng; Avants, Ben; Ye, Fan; Nagayama, Shin; Chen, Ziying; Kemere, Caleb; Robinson, Jacob T.; Veeraraghavan, Ashok
    In most biological tissues, light scattering due to small differences in refractive index limits the depth of optical imaging systems. Two-photon microscopy (2PM), which significantly reduces the scattering of the excitation light, has emerged as the most common method to image deep within scattering biological tissue. This technique, however, requires high-power pulsed lasers that are both expensive and difficult to integrate into compact portable systems. Using a combination of theoretical and experimental techniques, we show that if the excitation path length can be minimized, selective plane illumination microscopy (SPIM) can image nearly as deep as 2PM without the need for a high-powered pulsed laser. Compared to other single-photon imaging techniques like epifluorescence and confocal microscopy, SPIM can image more than twice as deep in scattering media (∼10 times the mean scattering length). These results suggest that SPIM has the potential to provide deep imaging in scattering media in situations in which 2PM systems would be too large or costly.
  • Thumbnail Image
    Item
    Deep learning extended depth-of-field microscope for fast and slide-free histology
    (PNAS, 2020) Jin, Lingbo; Tang, Yubo; Wu, Yicheng; Coole, Jackson B.; Tan, Melody T.; Zhao, Xuan; Badaoui, Hawraa; Robinson, Jacob T.; Williams, Michelle D.; Gillenwater, Ann M.; Richards-Kortum, Rebecca R.; Veeraraghavan, Ashok
    Microscopic evaluation of resected tissue plays a central role in the surgical management of cancer. Because optical microscopes have a limited depth-of-field (DOF), resected tissue is either frozen or preserved with chemical fixatives, sliced into thin sections placed on microscope slides, stained, and imaged to determine whether surgical margins are free of tumor cells—a costly and time- and labor-intensive procedure. Here, we introduce a deep-learning extended DOF (DeepDOF) microscope to quickly image large areas of freshly resected tissue to provide histologic-quality images of surgical margins without physical sectioning. The DeepDOF microscope consists of a conventional fluorescence microscope with the simple addition of an inexpensive (less than $10) phase mask inserted in the pupil plane to encode the light field and enhance the depth-invariance of the point-spread function. When used with a jointly optimized image-reconstruction algorithm, diffraction-limited optical performance to resolve subcellular features can be maintained while significantly extending the DOF (200 µm). Data from resected oral surgical specimens show that the DeepDOF microscope can consistently visualize nuclear morphology and other important diagnostic features across highly irregular resected tissue surfaces without serial refocusing. With the capability to quickly scan intact samples with subcellular detail, the DeepDOF microscope can improve tissue sampling during intraoperative tumor-margin assessment, while offering an affordable tool to provide histological information from resected tissue specimens in resource-limited settings.
  • Thumbnail Image
    Item
    Deep-3D microscope: 3D volumetric microscopy of thick scattering samples using a wide-field microscope and machine learning
    (Optica Publishing Group, 2022) Li, Bowen; Tan, Shiyu; Dong, Jiuyang; Lian, Xiaocong; Zhang, Yongbing; Ji, Xiangyang; Ji, Xiangyang; Veeraraghavan, Ashok; Veeraraghavan, Ashok
    Confocal microscopy is a standard approach for obtaining volumetric images of a sample with high axial and lateral resolution, especially when dealing with scattering samples. Unfortunately, a confocal microscope is quite expensive compared to traditional microscopes. In addition, the point scanning in confocal microscopy leads to slow imaging speed and photobleaching due to the high dose of laser energy. In this paper, we demonstrate how the advances in machine learning can be exploited to "teach" a traditional wide-field microscope, one that’s available in every lab, into producing 3D volumetric images like a confocal microscope. The key idea is to obtain multiple images with different focus settings using a wide-field microscope and use a 3D generative adversarial network (GAN) based neural network to learn the mapping between the blurry low-contrast image stacks obtained using a wide-field microscope and the sharp, high-contrast image stacks obtained using a confocal microscope. After training the network with widefield-confocal stack pairs, the network can reliably and accurately reconstruct 3D volumetric images that rival confocal images in terms of its lateral resolution, z-sectioning and image contrast. Our experimental results demonstrate generalization ability to handle unseen data, stability in the reconstruction results, high spatial resolution even when imaging thick (∼40 microns) highly-scattering samples. We believe that such learning-based microscopes have the potential to bring confocal imaging quality to every lab that has a wide-field microscope.
  • Thumbnail Image
    ItemEmbargo
    Deep-learning-enabled computational microscopy for rapid cancer detection
    (2024-01-16) Jin, Lingbo; Veeraraghavan, Ashok
    The gold standard for cancer detection in diagnosis and treatment guidance is based on histopathology, the examination of cells under a microscope. However, the tissue in question will need to be removed from the patient, sectioned into very thin slices, and stained before it can be examined by a pathologist. This preparation process is time-consuming and labor-intensive. The first part of this thesis focuses on an ex vivo ultraviolet extended depth-of-field microscope that can rapidly image large areas of freshly resected tissue, providing histologic quality images without physical sectioning. To overcome challenges in directly imaging thick intact tissue, such as subsurface scattering, high tissue surface irregularities, and difficulties in histology interpretation, the proposed microscopy platform unifies UV surface excitation, end-to-end extended depth-of-field, and GAN-based virtual staining into a single, coherent pipeline. This microscope provides an inexpensive and easy-to-use alternative to standard histopathology. The second part of this thesis extends the capabilities of an in vivo high-resolution micro-endoscopy (HRME) by enabling 3D volume imaging. The HRME is a minimally invasive microscope with an imaging fiber that can be used alongside an endoscope and provides cellular-resolution histological images in real-time. However, the micro-endoscopy can only image the superficial layer of the tissue in 2D. By incorporating a custom-designed phase mask with the fiber, the new Mask-HRME system can perform volumetric imaging with variable focusing ability. This micro-endoscopy will enable physicians to look deeper into the tissue without performing more invasive procedures. In both of the projects, computational imaging methods that jointly design the optics and image processing algorithm enables these new microscopes to break the limit of conventional microscopy and capture more diagnostic information than was previously possible.
  • Thumbnail Image
    Item
    DeepDOF-SE: affordable deep-learning microscopy platform for slide-free histology
    (Springer Nature, 2024) Jin, Lingbo; Tang, Yubo; Coole, Jackson B.; Tan, Melody T.; Zhao, Xuan; Badaoui, Hawraa; Robinson, Jacob T.; Williams, Michelle D.; Vigneswaran, Nadarajah; Gillenwater, Ann M.; Richards-Kortum, Rebecca R.; Veeraraghavan, Ashok
    Histopathology plays a critical role in the diagnosis and surgical management of cancer. However, access to histopathology services, especially frozen section pathology during surgery, is limited in resource-constrained settings because preparing slides from resected tissue is time-consuming, labor-intensive, and requires expensive infrastructure. Here, we report a deep-learning-enabled microscope, named DeepDOF-SE, to rapidly scan intact tissue at cellular resolution without the need for physical sectioning. Three key features jointly make DeepDOF-SE practical. First, tissue specimens are stained directly with inexpensive vital fluorescent dyes and optically sectioned with ultra-violet excitation that localizes fluorescent emission to a thin surface layer. Second, a deep-learning algorithm extends the depth-of-field, allowing rapid acquisition of in-focus images from large areas of tissue even when the tissue surface is highly irregular. Finally, a semi-supervised generative adversarial network virtually stains DeepDOF-SE fluorescence images with hematoxylin-and-eosin appearance, facilitating image interpretation by pathologists without significant additional training. We developed the DeepDOF-SE platform using a data-driven approach and validated its performance by imaging surgical resections of suspected oral tumors. Our results show that DeepDOF-SE provides histological information of diagnostic importance, offering a rapid and affordable slide-free histology platform for intraoperative tumor margin assessment and in low-resource settings.
  • Thumbnail Image
    Item
    DeepDOF: Deep learning extended depth-of-field microscope for fast and slide-free histology of surgical specimens
    (2020-10-22) Jin, Lingbo; Veeraraghavan, Ashok
    Histopathology is the gold standard for cancer diagnosis, but histopathology slide preparation is expensive, time- and labor-intensive. Slide-free pathology with fluo- rescence microscopy could offer a faster and less costly alternative. However, rapidly imaging intact tissue with surface irregularities (∼ 200μm) is fundamentally con- strained by the intrinsic trade-off between resolution and depth-of-field (DOF). In this study, we present a novel computational microscope that can image intact spec- imens with cellular resolution without re-focusing. The system is designed to pro- vide real-time deep learning based histopathology of intact specimen using extended depth-of-field (DeepDOF). the DeepDOF microscope consists of a conventional microscope with the addition of a wavefront-encoding phase mask and a neural network that jointly extends the DOF while maintaining subcellular resolution. Leveraging advances in deep learning, we simultaneously designed and optimized the two key components in the DeepDOF network end-to-end. First, the optical layer simulates the phase mask that creates a depth-dependent and invertible point spread function (PSF). These PSF sections can encode surface texture/intensity information regardless of the surface topology. Sec- ond, an artificial intelligence-based digital layer is used to deconvolve and extract high resolution image information from the captured data. In this study, we trained the DeepDOF network and optimized the microscope design with a large image dataset consisting of varied imaging features from human histology to natural scenes. The optimized phase mask was then fabricated using reactive ion etching and inserted into the aperture plane of a 4x, 0.13 NA epi-fluorescence microscope, which was further integrated with an automated x-y sample stage for tissue mapping. We calibrated the depth dependent PSFs of the DeepDOF microscope using 1 μm fluorescent beads. By imaging resolution target, we show that the DeepDOF microscope can con- sistently resolve subcellular features within a 200 μm depth-of-field, thus allowing the visualization of nuclear morphology on highly irregular tissue surfaces without serial focusing. We validated DeepDOF microscope’s performance by imaging freshly resected and proflavine-stained porcine esophageal tissue and human oral tissue. Fur- thermore, we show that DeepDOF images reveal a variety of important diagnostic features confirmed by standard histopathology. In the long term, the DeepDOF mi- croscope can substantially contribute to histopathological assessment of intact biop- sies and surgical specimens, especially for intraoperative evaluation and in resource- constrained settings.
  • Thumbnail Image
    Item
    Denoising-based Approximate Message Passing
    (2014-12-05) Metzler, Chris A; Baraniuk, Richard G; Veeraraghavan, Ashok; Zhang, Yin
    A denoising algorithm seeks to remove perturbations or errors from a signal. The last three decades have seen extensive research devoted to this arena, and as a result, today's denoisers are highly optimized algorithms that effectively remove large amounts of additive white Gaussian noise. A compressive sensing (CS) reconstruction algorithm seeks to recover a structured signal acquired using a small number of randomized measurements. Typical CS reconstruction algorithms can be cast as iteratively estimating a signal from a perturbed observation. This thesis answers a natural question: How can one effectively employ a generic denoiser in a CS reconstruction algorithm? In response, in this thesis, I propose a denoising-based approximate message passing (D-AMP) algorithm that is capable of high-performance reconstruction. I demonstrate that, when used with a high performance denoiser, D-AMP offers state-of-the-art CS recovery performance for natural images while operating tens of times faster than the only competitive method. In addition, I explain the exceptional performance of D-AMP by analyzing some of its theoretical features. A critical insight into this approach is the use of an appropriate Onsager correction term in the D-AMP iterations, which coerces the signal perturbation at each iteration to be distributed approximately like the white Gaussian noise that denoisers are typically designed to remove. In doing so, this feature enables the algorithm to effectively use nearly any denoiser.
  • Thumbnail Image
    Item
    Depth Limit of Imaging through Scattering Media using Selective Plane of Illumination Microscopy (SPIM)
    (2015-12-03) Zhang, Shizheng; Veeraraghavan, Ashok; Robinson, Jacob T; Kemere, Caleb
    In most biological tissues, the maximum optical imaging depth is limited by light scattering. Confocal and multi-photon microscopy have been developed to increase the imaging depth by limiting the amount of scattered light that reaches the detector, however, these techniques acquire images one point at a time resulting in reduced image acquisition speed. Recently, Selective Plane of Illumination Microscopy (SPIM) has emerged as an alternative 3D microscopy technique with faster image acquisition speeds, enabled by capturing entire 2D planes rather than individual points. While the advantages of SPIM for high speed imaging are understood, here we demonstrate that SPIM also increases the imaging depth in scattering media compared to confocal and epifluorescence techniques. We show both analytically and experimentally that SPIM can image 2-3 times deeper than confocal microscopy (~10x the mean scattering length). The primary reason for the deeper imaging capability of SPIM is the fact that off-axis illumination reduces the out-of-focus fluorescence above the imaging plane. We find that for scattering media, multi- photon microscopy can image deeper than SPIM; however, the fact that SPIM does not require a high-power pulsed laser makes this approach a lower cost alternative to multi-photon microscopy for imaging into scattering media beyond the depths of conventional single photon microscopy techniques.
  • Thumbnail Image
    Item
    Designing miniature computational cameras for photography, microscopy, and artificial intelligence
    (2019-07-25) Boominathan, Vivek; Veeraraghavan, Ashok
    The fields of robotics, Internet of things, health-monitoring, neuroscience, and many others have consistently trended toward miniaturization of sensing devices over the past few decades. This miniaturization has enabled new applications in areas such as connected devices, wearables, implantable medical devices, in vivo microscopy and micro-robotics. By integrating visual sensing capabilities in such devices, we can enable improved spatial, temporal and contextual information collection for artificial intelligence and data processing. However, cameras are reaching their limits in size reduction due to restrictions of traditional lensed-optics. If we can combine the design of camera optics and computational algorithms, we can potentially achieve miniaturization beyond traditional optics. In this dissertation, we explore designing unconventional optics coupled with computational algorithms to achieve miniaturization. Recent works have shown the use of flat diffractive optics, placed at a focus distance from the imaging sensor, as a substitute for lensing and the use of computational algorithms to correct and sharpen the images. We take this a step further and place a thin diffractive mask at a very close distance (range of 100s of microns to a millimeter) from the imaging sensor, thereby achieving an even smaller form-factor. Such flat camera geometry calls for new ways of modeling the system, methods to optimize the mask design and computational algorithms to recover high-resolution images. Moreover, retaining the thin geometry, we develop a framework to design optical masks to off-load some of the computational processing to inherently zero-power optical processing. With the developed methods, we demonstrate (1) ultraminiature microscopy, (2) thickness constrained high-resolution imaging, (3) optical Gabor feature extraction, and an example of (4) hybrid optical-electronic computer vision system.