Is it safe to hook up a device with a higher voltage battery through a regulator like this?
Traditional LINEAR regulators indeed drop power by heating up. Newer-technology SWITCHING regulators only draw power for fractional amounts of time so that they produce very low heat compared to the old-fashioned ones.

Note, however that many (most?) regulators need some better margin to work with. You will need a rather good (better than average) regulator to produce a 5V output from a 6V input voltage.  Most 5V regulators you will find are not capable of operating on inputs as low as 6V. Now you MIGHT find a very "low-dropout voltage" regulator that can work with only 1 volt of differential.

There are also switch-mode regulators which can handle "drop-out" voltage down practically to zero (slightly more than 5V source for 5V output.