How to change the functionality of FILE_EXISTS_RENAME?
There's no way to do it without changing the core function file_create_filename. That could obviously have unwanted side-effects for other parts of the system, and make updating difficult.

If you have to do it differently for whatever reason, you should write your own function and use that instead. Once you see how much code you need to copy and maintain from file_save_upload and file_create_filename, though, you might think twice about whether it's worth it.