Human physiological and behavioral data have been continuously collected and studied, empowered by modern wearable devices. Among plenty of research areas, emotion and wellbeing estimation has become a rising topic. Researchers and developers found that there is potential to decode human emotion and wellbeing with physiological and behavioral data. Several outstanding studies have shown promising results and provide incentives for the future development of this area. Nevertheless, there are challenges that remain, such as modality missing, heterogeneity among subjects, and label sparseness. In this thesis, we addressed obstacles in modeling physiological and behavioral data. (1) We proposed a job-role based multi-task and multi-label learning to build models for different groups of populations with correlated labels; (2) We proposed a modality fusion network to adaptively fit parameters and infer emotion prediction even with missing data modalities; (3) We proposed a personalized attention model to learn the heterogeneity in sensor data and labels among individuals; (4) We designed a semi-supervised learning framework to learn representations from the massive unlabeled physiological sequences. We evaluated the proposed methods on data sets collected in the wild as well as using public data sets. The evaluations showed significant improvements in our approaches compared to the baseline models. Further, we conducted the model interpretability analysis to identify the critical part of the input signal that contributes to the proposed deep learning models.