GHz-scale bandwidth in the mmWave spectrum (30 GHz to 300 GHz and beyond) realizes data rates of up to 100 Gb/sec with highly directional links, which can satiate the ever increasing demand for high speed wireless connectivity. However, a key challenge is that end nodes need to continually align their beams to maintain directional links, which incurs significant beam search overhead. This overhead can limit the performance of mmWave networks under nodal or environmental mobility. In this thesis, I present the design, implementation and evaluation of two novel mmWave beam adaptation solutions for achieving out of band steering using passive light sensing with off-the-shelf sensors to completely eliminate the need for in-band training. The key idea is to exploit similar propagation characteristics of higher frequency bands (i.e., light and mmWave bands) to track the dominant component of the mmWave wireless channel solely by using measurements in light band. To this end, I first introduce LiSteer, a system that steers mmWave beams at mobile devices by repurposing indicator LEDs on wireless Access Points (APs) to passively acquire direction estimates. I demonstrate that LiSteer maintains beam alignment at the narrowest beamwidth level even in case of device mobility, without incurring any training overhead at mobile devices. I then present SearchLight, where the key idea is to simultaneously track a mobile device's position and orientation using intensity measurements from lighting infrastructure, and to adapt mmWave beams at both mobile devices and the AP, completely eliminating beam training overhead for mmWave links. My extensive evaluation on a custom dual-band hardware platform comprising highly directional horn antennas as well as practical phased antenna arrays with electronic beam steering shows that both LiSteer and SearchLight achieve direction estimates within 2.5 degrees of ground truth on average. Moreover, both systems track client mobility and achieve up to 3x throughput gains compared to an in-band training strategy, and eliminate milli-second-scale in-band training epochs.