Many problems in science and engineering can be formulated as integral equations with elliptic kernels. In particular, in optimal control and design problems, the domain geometry evolves and results in a sequence of discretized linear systems to be constructed and inverted. While the systems can be constructed and inverted independently, the computational cost is relatively high. In the case where the change in the domain geometry for each new problem is only local, i.e. the geometry remains the same except within a small subdomain, we are able to reduce the cost of inverting the new system by reusing the pre-computed fast direct solvers of the original system. The resulting solver only requires inexpensive matrix-vector multiplications and matrix inversion of small size, thus dramatically reducing the cost of inverting the new linear system.