208x240 buck boost transformer putting out ~260v
That transformer is awful small

Your transformer is not a buck-boost transformer as you assumed it to be -- instead, it's a standard (isolating dry type) transformer with a 120/240V primary and a 208V secondary.  As a result, its windings are handling full power, which means that since it's a 500VA unit, it can only handle 2A at 240V coming out.  I'd expect it to overheat quite severely trying to handle the 8FLA@240VAC of a 1HP motor.

As to the high output

I would check the input voltage -- it sounds like the input is on the high side, and doubly so because there will usually be a bit of turns ratio compensation on the design secondary side that acts in the opposite direction from what you are describing in a backfed application such as yours.  However, it should drop to a more normal value under a reference load.