Studies in the collective motility of organisms use a range of analytical approaches to formulate continuous kinetic models of collective dynamics from rules or equations describing agent interactions. However, the derivation of these kinetic models often relies on Boltzmann’s “molecular chaos” hypothesis, which assumes that correlations between individuals are short-lived. While this assumption is often the simplest way to derive tractable models, it is often not valid in practice due to the high levels of cooperation and self-organization present in biological systems. In this work, we illustrated this point by considering a general Boltzmann-type kinetic model for the alignment of self-propelled rods where rod reorientation occurs upon binary collisions. We examine the accuracy of the kinetic model by comparing numerical solutions of the continuous equations to an agent-based model that implements the underlying rules governing microscopic alignment. Even for the simplest case considered, our comparison demonstrates that the kinetic model fails to replicate the discrete dynamics due to the formation of rod clusters that violate statistical independence. Additionally, we show that introducing noise to limit cluster formation helps improve the agreement between the analytical model and agent simulations but does not restore the agreement completely. These results highlight the need to both develop and disseminate improved moment-closure methods for modeling biological and active matter systems.