On a practical level, how exactly is the amount of a Bitcoin transaction fee determined?
Bitcoin transaction fees are usually calculated based on the size (in bytes) that the transaction takes up. This is a number known as 'fee rate', usually written as 'satoshis per byte', and the idea is that you are bidding for space within a block, where most miners will select transactions with the highest sat/byte fee to prioritize for confirmation. The average fee rate according to Bitinfocharts is about 97 sat/byte, meaning for each byte the transaction takes up, you add 0.00000097 BTC in fees. You can probably get away with using anywhere from 50-100 sat/byte and have your transaction confirmed within a reasonable amount of time. The value of the transaction in BTC is completely irrelevant.
If you're handling many withdrawals, often, service operators will batch the withdrawals of multiple users together into a single transaction to save on fees, by generating a transaction that has many outputs, one for each user's withdrawal. Alternatively, you could offer deposits and withdrawals through the Lightning Network to minimize fees.