Cryptsetup | Given Enough Passphrase Entropy, Is It Necessary To Use a High Iteration Count?
If you are certain that the passphrase contains enough entropy, then iteration is not needed. The iteration is only required when the entropy is low. The only way to preclude brute force search through a small search space is to make each guess take a prohibitively long amount of time. This is why the iteration count is recommended to be high: To ensure that nobody can guess more then a small number of passwords per second.

If the passphrase contains 256 bits of entropy, then it does not matter how fast the hash can be computed - the search space is simply too large for anyone to search through, regardless of how fast they can look.