The chemical nature of surface adsorbates affects the localized surface plasmon resonance of metal nanoparticles. However, classical electromagnetic simulations are blind to this effect, whereas experiments are typically plagued by ensemble averaging that also includes size and shape variations. In this work, we are able to isolate the contribution of surface adsorbates to the plasmon resonance by carefully selecting adsorbate isomers, using single-particle spectroscopy to obtain homogeneous linewidths, and comparing experimental results to high-level quantum mechanical calculations based on embedded correlated wavefunction theory. Our approach allows us to indisputably show that nanoparticle plasmons are influenced by the chemical nature of the adsorbates 1,7-dicarbadodecaborane(12)-1-thiol (M1) and 1,7-dicarbadodecaborane(12)-9-thiol (M9). These surface adsorbates induce inside the metal electric dipoles that act as additional scattering centers for plasmon dephasing. In contrast, charge transfer from the plasmon to adsorbates—the most widely suggested mechanism to date—does not play a role here.