Test the randomness of a random permutations, that was generated without Fisher-Yates?
One thing that comes to mind when evaluating random permutations is what happens if we shuffle an array that contains consecutive integers and just use the first index as a randomly generated number, which can now be tested for randomness using test suites like PractRand.

No, you can't do that, but almost.  A pseudo/truly random sequence has multiple same values, repetitions and +/- runs of digits. Your solely incremental sequence will easily fail the test even when perfectly permuted, probably even fail something as simple as ent.
You test the shuffle algorithm by inference.

Generate a large file from something like /dev/urandom or the windows equivalent.

Sort it into order, either rising or falling.

Then permute that.

Then test for randomness.


The test hypothesis is that since the original sequence was cryptographically random with the correct $\chi^2$ distribution, a proper shuffle will not make it worse. And it will space out the values into random positions. So the standard randomness tests should then pass it.