Low-frequency / DC circuits --- noise + decoupling issues
I'm wondering: do I really need bypass capacitors for this op-amp?


Yes, you need capacitors to filter the voltage at the output of the potential divider because you don't want to transfer power supply noise through to the reference pin of the AD8421. You will probably only need one capacitor from the non-inverting input to ground; i.e. you don't need two as you have indictated.

(Alternative) Yes, you need op-amp power rail decouplers.