Frugivores act as seed-dispersal agents in many ecosystems. Thus, understanding the roles and impacts of seed dispersal by frugivores is important to understand the structure and diversity maintenance of plant communities. Frugivore-mediated seed dispersal is behaviorally driven, generating nonrandom patterns of seed dispersion; but, we know relatively little about how this might affect plant populations or communities. I examined how frugivores affected plants from the individual level to the population and community levels. To do this, I used modeling, trait-based and phylogenetic approaches combined with field observations and experiments, focusing on seed dispersal by three frugivorous lemur species in the biodiverse rainforest of Ranomafana National Park, Madagascar. An analysis of traits suggested that 84% of trees in Ranomafana are adapted for animal dispersal, of which more than 70% are dispersed by these three lemur species indicating their role as generalist dispersers. The distribution of fruit and seed size of bird-dispersed species was nested within the wide spectrum of size distribution associated with lemur dispersal. Nonrandom seed dispersal by these three frugivores increased per capita recruitment of the seeds of a long-lived canopy tree, Cryptocarya crassifolia, by four-fold compared with no dispersal, even though it was not an overall advantage compared to random dispersal. The three frugivores not only dispersed seeds away from parent and conspecific adult-trees, but also biased seed dispersal toward certain microhabitats. By using fruiting trees as seed dispersal foci, lemur frugivores structure the spatial associations between dispersed seeds and adult-trees nonrandomly, in terms of fruiting time, dispersal mode and phylogenetic relatedness. However, lemur-dispersed tree species were not more likely to be each other’s neighbors as adults. Interestingly, co-fruiting neighboring trees sharing lemur dispersers were more phylogenetically distant than expected by chance despite a phylogenetic signal in lemur dispersal mode, although there was no phylogenetic signal in the timing of fruiting among lemur-dispersed tree species. Results suggest an important link between frugivore foraging behavior and the spatial, temporal and phylogenetic patterning of seed dispersal. The critical role of frugivores in structuring seed dispersion and seed-adult plant associations has critical implications for plant-plant interactions, biodiversity patterns and community structure.