Skip those parts of tests that do not pass on AARCH64
Due to performance reasons the remapper behaves differently on AARCH64 and this consequently leads to test failures when this is checked. Disable the parts of tests that fail on AARCH64 due to this expected difference.

Fixes: #62882 