Natural selection can favour cooperation, but it is unclear when cooperative populations should be larger than less cooperative ones. While experiments have shown that cooperation can increase population size, cooperation and population size can become negatively correlated if spatial processes affect both variables in opposite directions. We use a simple mathematical model of spatial common-pool resource production to investigate how space affects the cooperation-population size relationship. We find that only cooperation that is sufficiently beneficial to neighbours increases population size. However, spatial clustering variations can create a negative cooperation-population relationship between populations even when cooperation is highly beneficial, because clustering selects for cooperation but decreases population size. Individual-based simulations with variable individual movement rates produced variation in spatial clustering and the hypothesized negative cooperation-population relationships. These results suggest that variation in spatial clustering can limit the size of evolutionarily stable cooperating populations - an ecological dilemma of cooperation.
