Single-cell experiments have revealed cell-to-cell variability in generation times and growth rates for genetically identical cells. Theoretical models relating the fluctuating generation times of single cells to the population growth rate are usually based on the assumption that the generation times of mother and daughter cells are uncorrelated. This assumption, however, is inconsistent with the exponential growth of cell volume in time observed for many cell types. Here we develop a more general and biologically relevant model in which cells grow exponentially and generation times are correlated in a manner which controls cell size. In addition to the fluctuating generation times, we also allow the single-cell growth rates to fluctuate and account for their correlations across the lineage tree. Surprisingly, we find that the population growth rate only depends on the distribution of single-cell growth rates and their correlations.