Prior behavioral and neuroimaging evidence supports a separation between working memory capacities in the phonological and orthographic domains. Although these data indicate distinct buffers for orthographic and phonological information, prior neural evidence does indicate that nearby left inferior parietal regions support both of these working memory capacities. Given that no study has directly compared their neural substrates based on data from the same individuals, it is possible that there is a common left inferior parietal region shared by both working memory capacities. In fact, those endorsing an embedded processes account of working memory might suggest that parietal involvement reflects a domain-general attentional system that directs attention to long-term memory representations in the two domains, implying that the same neural region supports the two capacities. Thus, in this work, a multivariate lesion-symptom mapping approach was used to assess the neural basis of phonological and orthographic working memory using behavioral and lesion data from the same set of 37 individuals. The results showed a separation of the neural substrates, with regions in the angular gyrus supporting orthographic working memory and with regions primarily in the supramarginal gyrus supporting phonological working memory. The results thus argue against the parietal involvement as supporting a domain-general attentional mechanism and support a domain-specific buffer account of working memory.