Want to switch to tankless, but can I use my existing wiring?
If a heater requires 8 AWG wire,  it's because it requires more electricity than a smaller wire is capable of safely delivering.  If you used the 12 AWG wire, first the 20A breaker would trip, but if that was upgraded (DO NOT) then the wire itself would turn into a heater and burn down the house.  

What I'm saying here is that when too much electricity is run through a wire that is too small, it heats up.  Wire sizes are based on keeping a wire cool enough to not cause a fire when it's used.  Wire sizes are hard requirements, not suggestions.

The only option here is to run new wire or choose a smaller water heater.