Denoiser slows rendering too much
Nothing is wrong, the denoising takes time, that's all.

In your blend, most of the area has literally nothing. The renderer notices that and renders those areas extremely fast. The denoiser still has to treat all the pixels, and does not know that there is nothing there. 

As an exemple (CPU only, render size changed to 1080x1080):
100 samples, no denoise:
your file, ('as is') : ~11s  (GPU 6s)
with added background plane : ~33 s (GPU: 24s)

1 sample, default denoise settings:
your file, ('as is') : ~25s (GPU: 9s)
with added backgound : ~26 s (GPU: 10s)
(render of 1 sample w/o denoise takes &lt;1s, with or w/o background)

I'm not going to give all the results, but for this scene, the denoising seems to add ~25s to the processing on CPU (on my system, and with the reduced output size!), independent of the presence of a background plane or the number of samples.