how can I change the setting for number of upload retries in the IDE?
The code is not in the arduino ide but in the tool chain (.exe) that it runs. 

Normally it is a program called avrdude.exe that is running so you would need to rebuild the avrdude.exe tool which is not practical.

The easiest way, but rather crude, is to pull the usb cable. If the port disappears avrdude should stop immediately.