Contrastive Learning is a popular method for training modern deep neural networks. In this thesis, we explore several methods in the supervised learning and semi-supervised learning setting.

Firstly, we propose a technique called Negative Sampling in Semi-Supervised Learning (NS3L). NS3L exploits implicit negative evidence to improve the top-line performance of deep neural networks in semi-supervised learning. NS3L requires almost no additional computation and overhead and is shown to improve existing state-of-the-art methods.

Secondly, we take the view of implicit contrastive learning and propose the data augmentation method StackMix. Following the “Mix” line of work, StackMix takes pairs of samples and concatenates the inputs while averaging the outputs. This way, the neural network needs to learn to differentiate between the two samples within the concatenated sample. Improved performance is demonstrated on a variety of settings.

Lastly, we tackle the computational requirements of FixMatch, a semi-supervised learning method, and propose Fast FixMatch based on curriculum batch size. Curriculum batch size exploits natural training dynamics by starting with a small batch size and ending with a large batch size. Coupled with two other complementary methods that together perform better than a sum of parts, Fast FixMatch demonstrates substantial decreased training computations compared with FixMatch.