Is there any way to mathematically optimise Bitcoin transfer fee? Say you want to transfer X amount of Bitcoins between wallets and Y amount of transaction fee will be levied. Now you have option to send extra Bitcoins for the same amount of transaction fee, so how can you calculate what is the maximum amount of Bitcoins that can be transferred for that Y amount of transaction fee?
The network fee doesn't depend on the amount of BTC that you're sending, it depends on the total size of the transaction. So there is no such thing as "maximum amount of BTC that can be transferred". You transfer as many BTC as you want and you determine the fee per byte. The miners will include your transaction on a block depending on that fee.

To calculate an optimal fee I recommend you have to take a look at the mempool taking into account that the transactions with greater fee will be processed first. In general, except that you're on a hurry or the mempool is overloaded, a fee of 5 satoshis/byte would be enough.

For further details take a look into How is transaction fee calculated?