Naming pictures and matching words to pictures belonging to the same semantic category negatively affects language production and comprehension. By most accounts, semantic interference arises when accessing lexical representations in naming (e.g., Damian, Vigliocco, & Levelt, 2001) and semantic representations in comprehension (e.g., Forde & Humphreys, 1997). Further, damage to the left inferior frontal gyrus (LIFG), a region implicated in cognitive control, results in increasing semantic interference when items repeat across cycles in both language production and comprehension (Jefferies, Baker, Doran, & Lambon Ralph, 2007). This generates the prediction that the LIFG via white matter connections supports resolution of semantic interference arising from different loci (lexical vsﾠsemantic) in the temporal lobe. However, it remains unclear whether the cognitive and neural mechanisms that resolve semantic interference are the same across tasks. Thus, we examined which gray matter structures [using whole brain and region of interest (ROI) approaches] and white matter connections (using deterministic tractography) when damaged impact semantic interference and its increase across cycles when repeatedly producing and understanding words in 15 speakers with varying lexical-semantic deficits from left hemisphere stroke. We found that damage to distinct brain regions, the posterior versusﾠanterior temporal lobe, was associated with semantic interference (collapsed across cycles) in naming and comprehension, respectively. Further, those with LIFG damage compared to those without exhibited marginally larger increases in semantic interference across cycles in naming but not comprehension. Lastly, the inferior fronto-occipital fasciculus, connecting the LIFG with posterior temporal lobe, related to semantic interference in naming, whereas the inferior longitudinal fasciculus (ILF), connecting posterior with anterior temporal regions related to semantic interference in comprehension. These neuroanatomical-behavioral findings have implications for models of the lexical-semantic language network by demonstrating that semantic interference in language production and comprehension involves different representations which differentially recruit a cognitive control mechanism for interference resolution.