The neural basis of phonological working memory (WM) was investigated through an examination of the effects of irrelevant speech distractors and disruptive neural stimulation from transcranial magnetic stimulation (TMS). Embedded processes models argue that the same regions involved in speech perception are used to support phonological WM whereas buffer models assume that a region separate from speech perception regions is used to support WM. Thus, according to the embedded processes approach but not the buffer approach, irrelevant speech and TMS to the speech perception region should disrupt the decoding of phonological WM representations. According to the buffer account, decoding of WM items should be possible in the buffer region despite distraction and should be disrupted with TMS to this region. Experiment 1 used fMRI and representational similarity analyses (RSA) with a delayed recognition memory paradigm using nonword stimuli. Results showed that decoding of memory items in the speech perception regions (superior temporal gyrus, STG) was possible in the absence of distractors. However, the decoding evidence in the left STG was susceptible to interference from distractors presented during the delay period whereas decoding in the proposed buffer region (supramarginal gyrus, SMG) persisted. Experiment 2 examined the causal roles of the speech processing region and the buffer region in phonological WM performance using TMS. TMS to the SMG during the early delay period caused a disruption in recognition performance for the memory nonwords, whereas stimulations at the STG and an occipital control region did not affect WM performance. Taken together, results from the two experiments are consistent with predictions of a buffer model of phonological WM, pointing to a critical role of the left SMG in maintaining phonological representations.