Yeah but it does excite emotions and make people freak out which in our Political system is more important than logic. In my mind it's illogical to shrink the baby population in view of the coming increase in an elderly slanted population. We need more work force to support the baby boomers and I think the economic impact of the shrinking work force will be very negative. Allowing the baby population to be artificially reduced is dangerous to economic sustainability because the baby boomer generation will need sufficient amounts of labor to support their continued existence. Some may argue there are benefits to shrinking populations this way, but I think the long term 30 year effect isn't good for the economy.
There are huge benefits to decreasing the population, as there are finite resources, and the fewer people there are, the less people will have to do without. Also, the fewer people there are, the less pollution there will be. If, just as an arbitrary example, there were only 1 million people in the world, they could all drive gas guzzling, pollution creating SUVs with relative impunity, if they wanted. Of course, only 1 million people would create its own problems, but if we keep going at the rate we are (which, by the way, is increasing on a global scale), there will be massive starvation and wars over resources. That is a very ugly picture.
Simply put, even if we maintain the current population level, things will be ugly enough. I think 4 billion was plenty (and probably too much) for the world population, and having more is just making things worse.
As for population distribution, if a country is not producing enough people domestically, they can always allow people to immigrate in to take care of that. Naturally, the bigots don't tend to like foreigners, but that is a different problem from the population levels.