Bicycle Trailer pulls back wheel sideways
It is possible that you put not enough torque on axle nuts. The axle moves in the fork.
 If there is enough space, add some extra, broad spacers and then screw it back to the frame.

Use this kind of spacer (it's from Sram I-3, might require some drilling)