(Whitebox Crypto) Using ChaCha20, is it safe to reduce the nonce length in a single block cipher?
Am I considerably reducing the complexity of finding the original key this way?


No. If key was only used as input of chacha20_block(key, counter=0, nonce) when preparing the lookup_tables, then finding the original key is demonstrably at least as hard as breaking chacha20 (the algorithm) in its intended usage.


  Am I increasing attack chances?


Depends comparing to what, and under what threat model. Most obvious issue: the nonce becomes predictable with sizable probability.


  I'm looking for ways to increase the nonce length without exploding the memory size.


If we stick to producing the same results as chacha20_block(key, counter=0, nonce), modeling that only as a strong PRF of nonce, then demonstrably memory size in bits is at least: number of possible nonces × output width in bits.

On the other hand, it is possible to increase the nonce length. One trivial way is to have a normal-width nonce with only the low-order bits changing.

With some increase in memory, it is possible to use arbitrary and normal-width nounces, and even to hide their value until use: in the black box store F(nounce) where F is a public PRF such that collision among actually used nounces does not occur (either make F wide enough to have little collisions for random nounces, or restrict actually used nounces to avoid collision); and at each use compute F(nounce) and use the output to decide which tabulated value should be output.



Note: I'm not seeing the whole thing as true white box cryptography, because the would-be box is so severely limited in what it can compute. It is more like a One Time Pad with true randomness replaced by strong-pseudo randomness generated by a cryptographically strong PRNG which key is known to only one side. I see nothing this can achieve that plain symmetric crypto with diversified keys does not also achieve, with the later having the advatage of not requiring massive storage.