Accurately calculate fees to sweep all coins
It seems there is some discrepancy between your estimatedFee variable and the feerate being used in the fundrawtransaction step. 

I think you could try simplifying your code to not use the estimatedFee during the vendorXBalance calculation, and then when calling fundrawtransaction, use the subtractFeeFromOutputs option to just have the fee deducted equally from all specified outputs. For example:

balance = Bitcoin.rpc.getbalance '*', 1, true

vendorABalance = vendorAIncome * 95%
vendorBBalance = vendorBIncome * 95%
vendorCBalance = vendorCIncome * 95%

systemBalance = balance - vendorABalance - vendorBBalance - vendorCBalance

rtx = Bitcoin.rpc.createrawtransaction [], [
  { 'Vendor A wallet' =&gt; vendorABalance },
  { 'Vendor B wallet' =&gt; vendorBBalance },
  { 'Vendor C wallet' =&gt; vendorCBalance },
  { 'System wallet' =&gt; systemBalance },
]
Bitcoin.rpc.fundrawtransaction rtx, { includeWatching: true, conf_target: 6, subtractFeeFromOutputs: [vout_index,...] }

# On a offline machine
rtx = bitcoin-cli signrawtransactionwithkey rtx ['private keys']

# Back on the online machine
Bitcoin.rpc.sendrawtransaction rtx


Here is some relevant text from bitcoin-cli help fundrawtransaction (v0.17.0.1): 

"subtractFeeFromOutputs" (array, optional) A json array of integers.
     The fee will be equally deducted from the amount of each specified output.
     The outputs are specified by their zero-based index, before any change 
     output is added.
     Those recipients will receive less bitcoins than you enter in their 
     corresponding amount field. 
     If no outputs are specified here, the sender pays the fee.
       [vout_index,...]