Is there a way to multiply current?
Let me get this straight.  You connected something rated for 6 V to something that produces 12 V, and you're actually surprised something went wrong!!?  You are trying to cram 20 pounds of stuff into a 10 pound bag.  It doesn't take a degree in engineering or much sense at all to expect this not to work out.

The 12 V supply can put out up to 800 mA.  Apparently your heater draws more than that when connected to 12 V.  This type of heater is basically just a resistor.  For it to only draw 800 mA at 12 V, it would need to draw only 400 mA at 6 V.  That means it would be rated as (6 V)(400 mA)= 2.4 W or less at 6 V.  If it is rated at higher power than that, then it will draw more current than 400 mA at 6 V or 800 mA at 12 V and your 12 V supply can't handle it.

Apparently your 12 V supply has a shutdown feature so that it goes to 0 V when you try to abuse it, as you did.  It may periodically try to raise the voltage for a short time, discover the current is still too high, and go back into shutdown.  These attempts to recover would be short blips maybe every second or two, so you'd still see 0 V on a meter measuring the output.

Fortunately for you, all that happened was that the supply shut down.  If you had connected the heater to a 12 V source with more current capability, like a car battery for example, the heater would have to give instead of the supply.  Again, ceramic heaters are basically just resistors with fancy marketing, and generally don't come with any protection circuitry.  The 2x voltage would cause 4x the power dissipation, which would very likely be destructive to the heater within a few seconds.

The 4 AA batteries work with the heater because they are running it within spec.  At roughly 1.5 V per cell, they make 6 V, which is what the heater is designed to operate at.  This again should be no surprise.

There are ways to take 12 VDC at some current and make 6 VDC at not quite twice the current.  A circuit for this is called a buck converter, but that is above your level of electronics at this point.  Nothing called a "transformer" will work since those inherently run on AC, not DC.