Alternatives to Mersenne Twister
Forget the Mersenne Twister and xorshift; maybe CryptMT is secure but it has received very little scrutiny since it was rejected from the eSTREAM portfolio a decade ago.  Cryptography is rife with good pseudorandom number generators, like ChaCha: if you pick the seed uniformly at random, the attacker has no hope of distinguishing the output from uniform random, and the world has high confidence in this state of affairs because it has been subjected to heavy scrutiny and there are trillions of euros of economic value protected by ChaCha in HTTPS around the world.

If you have measured performance, and you have profiled, and you have confirmed that a lot of time is spent in ChaCha, and you are confident that this will never be security-relevant—in particular, if you can quantify the statistical tests you need to defeat and argue that smarter statistical tests will never be relevant—then maybe you will have a reason to look elsewhere.  But the first thing you can do with ChaCha is tune down the number of rounds from the default ChaCha20 to the cheaper ChaCha12 or even cheaper ChaCha8.



Update, for question's addendum:

If the adversary can control the seed, then even if you used a random oracle for the PRNG (which is the strongest possible model—not attained by ChaCha but a reasonable model for, say, SHAKE128), the adversary can spend an expected $t$ trials finding a seed with a property that occurs only $1/t$ of the time, like $t = 2^n$ trials to find a seed under which the first $n$ bits of output are all zero.  No amount of cryptography can help you against this.

But if the adversary can't control the seed, the property you want—nonsingular matrices with high probability—is essentially guaranteed with any cryptographic PRNG.  If it weren't, then the putative cryptographic PRNG would be astonishingly bad, because a very simple test—generate a matrix and test whether it's nonsingular—would serve as an attack against the PRNG that breaks it without even knowing how the PRNG is designed!