Mechanism to limit the exponential growth of the number of ancestors that are n-generations away from an organism?
Of course a(n) is limited, and much more tightly than by the number of atoms that exist.  The analysis in the question goes off track with "inbreeding is not germane. Here, all that matters is that each organism has 2 parents. Period."

Consider a person whose parents are first cousins. Then:


a(1) = 2
a(2) = 4
a(3) = 6, not 8


While each of the two parents has four grandparents, together the two parents have only six grandparents because they share one pair of grandparents.  This is due to pedigree collapse.

This table shows the actual collapse for some people with ancestry known to nine generations and with significant inbreeding (from de.Wikipedia):