Pull-up resistor value I2C; higher than 400 pf capacitance
Typically the pull up resistor for your bus is chosen depending on the bus capacitance. So higher the bus capacitance, you'll have to use a lower pull-up resistor. The criterion to be satisfied is that at the switching frequency, your pad should be able to charge to 90% of the rail voltage (which is a spec in i2c). The resistor pull-ups generally mentioned, correspond to different modes of operation of i2c.(like fast mode, fast mode plus, high speed mode). FM+ mode uses 1MHz signal and hence the resistor pull-up is calculated such that the bus is charged to rail voltage within 0.5us.

Now in your case, if you have a dc Iol requirement of 3mA @ Vol=0.4V then, it just means that you're supposed to be able to sink 3mA of current through the open drain transistor which you have. You may not need to worry about the drop across the pull-up resistor.

Also, I didn't get your Rmax calculation part, is it an equation mentioned in the i2c data sheet?