The ability to image the optical near-fields of nanoscale structures, map their morphology and concurrently obtain spectroscopic information, all with high spatiotemporal resolution, is a highly sought-after technique in nanophotonics. As a step towards this goal, we demonstrate the mapping of electromagnetic forces between a nanoscale tip and an optically excited sample consisting of plasmonic nanostructures, with an imaging platform based on atomic force microscopy. We present the first detailed joint experimental-theoretical study of this type of photo-induced force microscopy. We show that the enhancement of near-field optical forces in gold disk dimers and nanorods follows the expected plasmonic field enhancements, with strong polarization sensitivity. We then introduce a new way to evaluate optically-induced tip-sample forces by simulating realistic geometries of the tip and sample. We decompose the calculated forces into in-plane and out-of-plane components and compare the calculated and measured force enhancements in the fabricated plasmonic structures. Finally, we show the usefulness of photo-induced force mapping for characterizing the heterogeneity of near-field enhancements in precisely e-beam fabricated nominally alike nanostructures - a capability of widespread interest for precise nanomanufacturing, SERS and photocatalysis applications.