Compare two methods of random permutations I want to compare in a practical sense two methods of random permutations -- one theoretically perfect, namely that of Fisher and Yates, and another ad hoc, let's
call it X. A way of comparison I could think of is the following:

One starts from the standard configuration of n objects [0, 1, 2, ...., n-1] and
apply each method a fairly large number of times successively to permute them
and each time one computes the Hamming distance of the result from the standard
configuaration. One obtains thus the frequency distribution of the Hamming distances
for each method. If these are fairly comparable to each other, then X could be
 practically employed in place of the theoretically perfect one.

Is this line of thought of mine correct? Does anyone have an idea of a better method
of comparison? Thanks in advance.
Your method of comparison is very easy to defeat. I assume that by "Hamming distance from the regular configuration" you mean the number of fixed points. Make a table of the number of permutations in $S_n$ having $k$ fixed points (this can be computed recursively), and fix a single permutation with $k$ fixed points for every $k$. This way you can sample a permutation in a ways that matches this particular statistic of random permutations, but probably performs rather differently compared to a truly random permutation.