Circular dichroism spectroscopy is essential for structural characterization of proteins and chiral nanomaterials. Chiral structures from plasmonic materials have extraordinary strong circular dichroism effects compared to their molecular counterparts. While being extensively investigated, the comprehensive account of circular dichroism effects consistent with other plasmonic phenomena is still missing. Here we investigated the circular differential scattering of a simple chiral plasmonic system, a twisted side-by-side Au nanorod dimer, using single-particle circular dichroism spectroscopy complimented with electromagnetic simulations. This approach enabled us to quantify the effects of structural symmetry breaking, namely, size-mismatch between the constituent Au nanorods and large twist angles on the resulting circular differential scattering spectrum. Our results demonstrate that, if only scattering is considered as measured by dark-field spectroscopy, a homodimer of Au nanorods with similar sizes produces a circular differential scattering line shape that is different from the bisignate response of the corresponding conventional CD spectrum, which measures extinction, that is, the sum of scattering and absorption. On the other hand, symmetry breaking in a heterodimer with Au nanorods with different sizes yields a bisignate circular differential scattering line shape. In addition, we provide a general method for correcting linear dichroism artifacts arising from slightly elliptically polarized light in a typical dark-field microscope, as is necessary especially when measuring highly anisotropic nanostructures, such as side-by-side nanorods. This work lays the foundation for understanding absorption and scattering contributions to the CD line shape of single chiroplasmonic nanostructures free from ensemble-averaging, especially important for self-assembled chiral nanostructures that usually exist as both enantiomers.