At what point one has to use an I2C buffer?
You should use a buffer anytime the bus capacitance will be over 400pF. If you are just barely getting to that then you can make the decision to use stronger pull-ups instead of a buffer.

In general, if you're going off-board to a cable, you should have a buffer there. If you ever wanted to have longer cables you'd be glad there was a buffer to support that. For more information on that, see this question.