Cycles render time increases drastically upon adding simple planar backdrop
The render-time difference isn't too surprising, given that without the background most of the camera rays would simply miss all of the geometry and so those tiles to the side would render extremely quickly compared to the rest. Adding the background means that those camera rays hit a surface and Cycles then needs to determine Diffuse rays from the rest of the geometry to shade each point on the surface.
One potential solution would be to use the new(ish) Adaptive Sampling option. This way you can set a noise threshold and those tiles far from the &quot;track&quot; should render quicker than they otherwise would, while still having enough samples to avoid noise.