Limit the current draw from Op-Amp
This is the sort of circuit you need to be considering: -



Of importance is that the op-amp's input common mode range MUST include the positive ground rail. This makes the choice of op-amp more difficult but there will be devices that fit the bill.