Through frugivory and seed dispersal, vertebrates influence plant demography and forest regeneration. Variation in local habitat surrounding fruiting plants can influence frugivore foraging decisions, thereby creating intraspecific variation in seed dispersal services. However, we have little knowledge of drivers of local variation in frugivory. Here, we investigate factors that may influence frugivore diversity and fruit removal at the level of individual plants. We focus on a common understory palm within a continuous Chocó forest with mixed land-use histories in Ecuador. The density of pioneer tree species in the genus Cecropia around focal palms was negatively related to fruit removal and the diversity of frugivores visiting palms. This may relate to the fact that the presence and abundance of Cecropia species often indicate the existence and severity of past disturbances. Local Cecropia density was also related to an overall shift in the frugivore community that corresponded with an increase in fruit removal by lower-quality seed dispersers (rodents). We also found that the local density of fruiting conspecifics was positively related to frugivore diversity, but not fruit removal. Our results provide information on drivers of intraspecific inequalities in plant populations across tropical forest landscapes. The reduction in fruit removal and frugivore diversity associated with local Cecropia abundance suggests that seed dispersal services can be sensitive to fine-scale variation in habitat structure. Furthermore, because Cecropia are often indicative of past disturbances, this indicates that even small-scale habitat degradation by humans can have lasting effects by creating localized pockets of forest unfavored by frugivores.