Intel denoiser is denosing every frame differently
It seems that the Open Image Denoiser currently does not support temporal coherence. That means it only considers one image at a time, not the surrounding frames. However, according to this issue at their repository, this feature is planned for a future release.
Same problem with the Blender denoiser (denoising option in the render layers tab):

For animation denoising can be used, however it still requires high
sample counts for good results. With low sample counts, low frequency
(blurry) noise can be visible in animation frames, even if it not
becomes immediately apparent in still images. (Blender Documentation)


Conclusion: It seems that your best option is to use more samples if possible (and search for tips to reduce render times). Maybe some of the tricks to reduce noise mentioned in the Blender Documentation can help you as well.