Damped outrigger system is effective for improving energy dissipation for tall buildings. However, conventional damped outrigger (CDO) system with viscous damping has two limitations: (i) its maximum damping ratio cannot be improved when outrigger/column stiffness is inadequate; (ii) different modes achieve their maximum damping ratios at different outrigger damping values, and thus the dampers cannot be optimized to simultaneously reduce vibrations of multiple modes of concern to their minimum. In this paper, a purely frequency-independent negative stiffness damped outrigger (FI-NSDO) system is proposed by combining frequency-independent damper (FID) and negative stiffness device (NSD). The damped outrigger with FID can achieve the maximum damping ratio for all modes as compared to frequency-dependent damper like viscous damper. As the NSD has the features of assisting and enhancing motion and frequency-independence, the utilization of NSD will considerably improve the maximum damping ratios when outrigger/column stiffness is inadequate and maintain the frequency-independent feature of the whole system. Therefore, the FI-NSDO has the capability of simultaneously increasing the damping ratios of all target modes to their maximum values. Analysis in frequency domain and time domain, demonstrate that the proposed FI-NSDO performs better in controlling the multi-mode vibration of seismic responses.