render compositor only in 2.8
I have since found the cause - User error.

Blender was running fine but as I was scaling from a 4K to HD image it was processing the 4K image, then scaling then processing the HD image, which took 6 times longer. I exported a HD version of the images and used that and render times dropped to .7s/frame, which is what I was expecting.