Where are all the citations going?
The impact factor is not the right measure. The impact factor counts the average number of citations a paper receives in the two years after publication. But in most papers, most of the references given are for papers (substantially) older than two years, and consequently they don't count for the impact factor of the journals cited. That means that the average impact factor over all journals is not equal to the average number of citations.

Second, as others have pointed out, the distribution is heavily skewed. That's true for papers in general, but also for papers by the same researcher. To take just my own papers (see this google scholar page), there are a couple with over a thousand citations, but #10 on the list is already below 100 citations and there is quite a long tail of papers with far fewer citations than they have references themselves.