Sabharwal, Ashutosh2019-05-172020-05-012019-052019-04-16May 2019Cao, Jian. "Objective Sociability Measures from Multi-modal Smartphone data and Unconstrained Day-long Audio Streams." (2019) Diss., Rice University. <a href="https://hdl.handle.net/1911/105989">https://hdl.handle.net/1911/105989</a>.https://hdl.handle.net/1911/105989Sociability is defined as a tendency to affiliate with others and to prefer being with others instead of staying alone. Sociability is well-known to influence many aspects of an individual, e.g., their quality of life, their health and well-being, their workplace performance and even learning outcomes. Despite its importance, existing sociability measures mainly rely on subjective self-reports, hence they only provide a sparse sampling of an individual's social experiences, and put extra burdens on both the researchers and participants. In this thesis, I propose and develop objective measurements to capture social interactions from multi-modal smartphone data and unconstrained day-long audio streams. The goal is to automatically capture two forms of social interactions (i.e., remote social interactions including phone calls and text messages; in-person social interactions in the form of verbal conversations), and to investigate the correlations between objectively assessed social interactions and wellbeing and mental health outcomes. Towards that goal, I develop a smartphone app that integrates multi-modal sensor and usage data for remote social interaction measure. I also propose and develop the SocialSense framework, which automatically captures in-person verbal interactions from unconstrained audio recordings by wearable devices. SocialSense consists of an utterance segmentation frontend, an unsupervised speaker indexing stage using Siamese Convolutional Neural Network (Siamese-CNN), and a SpeakerRank algorithm to track the most frequent speakers. I evaluate the performance of SocialSense on both public datasets and a private Rice Speech Corpus for different ambient backgrounds, voice clip lengths and the number of speakers, and the results indicate that SocialSense performs reasonably well on unconstrained audio data. Using the smartphone app and SocialSense, I conduct three trials with the clinical population to validate that objectively assessed social interactions are correlated with mental health outcomes. Specifically, the SOLVD-Adult and SOLVD-Teen trials focus on investigating the correlations between remote social interactions and depressive symptoms assessed by clinical instruments. The SocioNet study aims at studying the consistency between objectively captured in-person social interactions, and self-reported sociability level and qualitative clinical observations for depression and psychosis patients. The results indicate that the smartphone app and SocialSense provide an objective, continuous and unobstructive approach to capture both remote and in-person social interactions. The sensor-based sociability measures correlate well with both self-reports and clinical instruments. Besides, SocialSense is able to capture transient behavioral markers that are of significant clinical importance, but are hard to detect with previous measures. Hence, the objective sociability measures have great potential for applications in mental health, team science, and other behavioral research.application/pdfengCopyright is held by the author, unless otherwise indicated. Permission to reuse, publish, or reproduce the work beyond the bounds of fair use or other exemptions to copyright law must be obtained from the copyright holder.Objective sociability measuresocial interactionsSocialSense frameworkunconstrained day-long audio streams.Thesis2019-05-17