Advanced liver cancer is often treated with radiotherapy, which requires precise liver segmentation. Deep learning models excel at segmentation but struggle on unseen data, a problem exacerbated by the difficulty of amassing large datasets in medical imaging. Clinicians manually correct these errors, but as models improve, the risk of clinicians overlooking mistakes due to automation bias increases. To ensure quality care for all patients, this thesis aims to offer automated, scalable, and interpretable solutions for detecting liver segmentation model failures.

My first approach prioritized performance and scalability. It applied the Mahalanobis distance (MD) to the features of four Swin UNETR and nnU-net liver segmentation models. I proposed reducing the dimensionality of these features with either principal component analysis (PCA) or uniform manifold approximation and projection (UMAP), resulting in improved performance and efficiency. Additionally, I proposed a k-th nearest neighbors distance (KNN) as a non-parametric alternative to the MD for medical imaging. KNN drastically improved scalability and performance on raw and average-pooled bottleneck features.

My second approach emphasized interpretability by introducing generative modeling for the localization of novel information that a model will fail on. It employed a StyleGAN2 network to model a distribution of 3,234 abdominal computed tomography exams (CTs). It then localized metal artifacts and abnormal fluid buildup, two prevalent causes of liver segmentation model failure, in 55 CTs by reconstructing the scans with backpropagation on the StyleGAN’s input space and focusing on the regions with the highest reconstruction errors.

The computational cost, data requirements, and training complexity of generative adversarial networks, along with a lack of reliable evaluation measures, have impeded their application to medical imaging. Accordingly, a significant portion of this thesis is dedicated to evaluating the applications of StyleGAN2 and the Fréchet Inception Distance (FID), a common measure of synthetic image quality, to medical imaging.

The principal contributions of this thesis are integrating PCA and UMAP with MD, utilizing KNN for out-of-distribution detection in medical imaging, leveraging generative modeling to localize novel information at inference, providing a comprehensive application study of StyleGAN2 to medical imaging, and challenging prevailing assumptions about the FID in medical imaging.