What are the limits of ASIC Mining? My assumption is that ASIC miners optimize for the "hashes per joule" metric, as did GPU miners over CPU miners.

It would seem that after some time, the decision to mine Bitcoins would be based on if you could source energy cheaply enough.

When do we reach that point?

I'm not a hardware guy so I'm curious to understand what the upperbound limits are of ASIC mining from a technical perspective. How much more can we optimize?

Thanks!
I'm not an expert, but as far as I understand it, Moore's Law applies to ASICs just like other chips. So, every two years you get newer ASICs which can hash more, even though the chip design is not all the different from the previous ASICs, the number of transistors that chip has on it is doubled.  So, there is always going to be a cost of keeping up with the latest chips. This is part of what prevents a 51% attack, since there are literally not enough ASICs is the world for one big bank or something, or one scammer, to buy them all up to mine with.  World BitCoin Network on YouTube goes into this in more detail in the 51% attack video.  

Moore's Law isn't going to hit a limit in the near term, since when we get so small we are on the scale of just a couple of atoms, we will just start stacking chips on top of each other.  Itel is already researching liquid cooling between chip stacks.  So, for the forseeable future, Bitcoin mining is not just going to be a function of cheap energy, or optimized chips, but the financial treadmill of buying new ASICs every two years.