Prior evidence suggests domain-specific working memory (WM) buffers for maintaining phonological (i.e., speech sound) and semantic (i.e., meaning) information. The phonological WM buffer’s proposed location is in the left supramarginal gyrus (SMG), whereas semantic WM has been related to the left inferior frontal gyrus (IFG), the middle frontal gyrus (MFG), and the angular gyrus (AG). However, less is known about the white matter correlates of phonological and semantic WM. We tested 45 individuals with left hemisphere brain damage on single word processing, phonological WM, and semantic WM tasks and obtained T1 and diffusion weighted neuroimaging. Virtual dissections were performed for each participants’ arcuate fasciculus (AF), inferior fronto-occipital fasciculus (IFOF), inferior longitudinal fasciculus (ILF), middle longitudinal fasciculus (MLF), and uncinate fasciculus (UF), which connect the proposed domain-specific WM buffers with perceptual or processing regions. The results showed that the left ILF, MLF, IFOF, and the direct segment of the AF were related to semantic WM performance. Phonological WM was related to both the left ILF and the MLF. This work informs our understanding of the white matter correlates of WM, especially semantic WM, which has not previously been investigated. In addition, this work helps to adjudicate between theories of verbal WM, providing some evidence for separate pathways supporting phonological and semantic WM.