How should I connect a three-wire transformer in a panel with no neutral bus?
Some things don't make sense here. Standard low voltage transformers should have a hot, a neutral and a ground wire. They are single phase and should be installed into the side of a standard junction box through a 1/2" knockout. Then you just match the wires to the circuit wires. The low voltage connections are then on the outside of the box.

Low voltage transformers aren't normally attached to the main circuit panel since their wires are usually too short to reach the connections. There is nothing in the Code saying you can't, you might just have to extend the wires.

Even in 1960's, houses in the US were built with split phase three wire services. The main panel would have two hot legs attached to the breaker bus bars a neutral and a ground bar. At the main panel the neutral and ground wires can be mixed on the same bars. If you choose to attach it to the main panel, the neutral and ground wires should be attached to one of those bars.

The National Electrical Code prohibits the use of course-thread self tapping screws for ground connections. The threads have to be fine enough to engage a minimum of 2 threads in the material of the enclosure. See 2017 NEC 250.8(A)(6).

Good Luck!