In two dimensions, dephasing by a bath cuts off Anderson localization that would otherwise occur at any energy density for fermions with disorder. For an isolated system with short-range interactions, the system can be its own bath, exhibiting diffusive (non-Markovian) thermal density fluctuations. We recast the dephasing of weak localization due to a diffusive bath as a self-interacting polymer loop. We investigate the critical behavior of the loop in d=4−ε dimensions, and find a nontrivial fixed point corresponding to a temperature T∗∼ε>0 where the dephasing time diverges. Assuming that this fixed point survives to ε=2, we associate it with a possible instability of the ergodic phase. Our approach may open a new line of attack against the problem of the ergodic to many-body-localized phase transition in d>1 spatial dimensions.