How to fix wobble in outdoor pavers
I wouldn't remove the paver base, which provides quite a bit of stability, but I'd consider a thin layer of sand over it to act as a buffer. If the base has joints or gaps that would allow the sand to drain, you could install a sheet of plastic or heavy landscape fabric first.