A reduction in fertility can lead to an aging population which leads to a variety of problems, see for example the Demographics of Japan.
A related concern is that high birth rates tend to place a greater burden of child rearing and education on populations already struggling with poverty.