Displacement is blocky
These lines are visible also in the original texture. So to get rid of this you would have to use some image editor to compensate it. Here I used Photoshop's Box Blur filter with range 1 px that wasn't enough so I used it twice, but you loose some details.
Original

Box Blur 1 px

Box Blur 1 px applied twice


There are probably better algorithms for reconstruction or pattern removal like some FFT Fast - Fourier Transform (they are like a magic in some cases) ... so here are some examples

https://www.youtube.com/watch?v=D9ziTuJ3OCw
https://www.wirebiters.com/salt-pepper-image-denoising-using-matlab/
https://www.cs.unm.edu/~brayer/vision/fourier.html

Ten years ago I used some FFT script for Photoshop, but it is too far to remember anything, sorry :)