Sabharwal, Ashutosh2024-05-212024-052024-04-17May 2024Chen, Wenwan. Objective Speech-Based Sociability Measure for Mental Health Assessment. (2024). PhD diss., Rice University. https://hdl.handle.net/1911/116105https://hdl.handle.net/1911/116105EMBARGO NOTE: This item is embargoed until 2026-05-01Sociability measures play a pivotal role in mental health assessment due to their associations with mood and symptoms of mental disorders. The conventional approach for assessing sociability relies on self-reports, often biased and error-prone. To address these limitations, I propose Ambiance-aware Social Interaction Measure (ASIM), an objective, fine-grained and comprehensive sociability measure. ASIM captures both individual social interactions and social ambiance – the environmental elements impacting social engagements – through the analysis of unconstrained audio data. Specifically, ASIM is structured as an 8-dimensional vector, with each element originating from proxies of social ambiance and social interactions. The number of concurrent speakers is proposed as a proxy for social ambiance, while a dedicated target speaker detection algorithm is devised to capture individual social interactions. To suit diverse usage scenarios, I present both offline processing and on-device processing solutions. In the offline processing scenario, concurrent speaker count benefits from the knowledge of large-scale self-supervised representations through model fine-tuning, and target speaker detection incorporates unsupervised source separation as a pre-processing step. For on-device processing, the concurrent speaker count is compressed to 5% of its original size to facilitate device-side implementation. Additionally, an Android application has been developed to support data collection, processing, and real-time feedback. Furthermore, target speaker detection leverages the enroll-aware attention statistic pooling method to effectively eliminate interfering speakers and enhance the model's robustness. ASIM's performance is evaluated using benchmark datasets and Rice University campus data, followed by an in-depth error analysis to understand its limitations and utilization. ASIM's utility is further substantiated through application to a clinical dataset. Results show a significant correlation between sociability patterns extracted by ASIM and self-reported metrics. Additionally, ASIM proves effective in predicting self-reported mood and sociability, showcasing its potential as an instrumental tool in both mental health research and clinical settings.application/pdfengCopyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder.sociability measurespeechThesis2024-05-21