Area preserving distance optimization
This appears to be an instance of the assignment problem, where you are trying to match up the $nm$ cells of $M$ with the $nm$ cells of $M'$.  For each place where $M[i,j]$ may get permuted to, you can compute the cost of that: namely, if it is permuted to $M'[i',j']$, then its cost is $(i'-x P[M[i,j]])^2 + (j' - y P[M[i,j]])^2$.  Then you can use standard algorithms for the assignment problem, such as the Hungarian algorithm.  Whether this is practical or not may depend on how large $nm$ is.  If it's not practical, you could instead try a heuristic, such as local search algorithm based on making swaps.