Complex robotic systems like mobile manipulators and humanoids will soon be employed outside the highly-structured domain of factories, entering cluttered kitchens, warehouses, and even space stations (e.g., NASA's Robonaut 2). For a robot to be effective in these domains, it is essential that the robot be able to manipulate and navigate the environment to achieve desired tasks, e.g., setting a table and fetching an object on the other side of the space station. These problems can be formulated as multi-modal planning problems, that is, there are a finite number of interactions the robot can have with the environment (e.g., picking up an object, grasping a handrail) each of which with a continuous infinity of possibilities (e.g., where the object is grasped or placed). In this work, we propose an efficient and general solver for these multi-modal problems by tackling three key issues. First, we address the challenges in efficiently solving the arbitrary manifold-constrained motion planning problems that arise in multi-modal planning with a framework that maintains theoretical guarantees of sampling-based planners. Second, we improve planner efficiency with an experience-based framework that learns from prior motion plans, leveraging the structure of multi-modal problems. Finally, we propose a guiding heuristic for multi-modal planners that generates a "lead" of specific actions to attempt, enabling efficient solutions to complex, long-horizon problems.