When to add an op amp buffer to a voltage reference IC?
Could I use the output of the voltage reference IC, in this case
  LM4041, directly instead of making it passing through an op amp
  buffer?


Not if you want low noise and some decent degree of accuracy. For low noise note the RC on the output of the op-amp - it has a low pass frequency of about 16 kHz so, any noise above that coming from the op-amp or voltage reference is progressively attenuated. There's no reason why the 100 nF couldn't be 10 uF (if the op-amp can handle the RC without oscillating) and get a 3 dB noise point of 160 Hz (even better for noise).

So, given that the LM4041 produces circa 20 uV RMS of wide-band noise it seems sensible to use an RC circuit. 

I'm not a big fan of op-amp buffers because they can bring all sorts of errors to the party but the RC is a must in my book because it simultaneously keeps noise down and offers a low impedance to the ADC at frequencies above where the raw output from the 4041 can't deliver low impedance (look at figures 7 and 8 in the TI data sheet).