Cycles lighting Principled BSDF differently on CPU and GPU
OK I found the answer, really stupid mistake.

I had direct light clamping on 10 and it did really weird things where light sources would be randomly clamped out.

I could re-render the same frame several times and achieve different results each time.

I just turned it off now.