How to keep a linear regulator cool without an actual heatsink?
You won't be able to dissipate that much heat with only copper traces to wick away the heat. (A SOT-89 is also a very small package, are you sure that specific part in that specific package is rated for 3W?)

I use D-Pak sized packages with lots of copper on the four layers and arrays of vias to try to give the device a lot of copper for a heat sink.



This works reasonably well for low duty cycle loads but doesn't work well for continuous load applications (there is high thermal resistance to the air). For high dissipation requirements you need fins and air moving over them, and unless you're building circuit boards differently than I know how to, you're going to need a heat sink to get those fins.