How do I replace this bathtub handle that has a plastic piece?
If you're not worried about possibly damaging the plastic bit (after all you are looking to remove it), I would get a pair of vice grips onto the square section at the back of the plastic piece.

This will allow you to either rotate the plug (to un-screw it, if it's threaded or to break the bond if it's glued in), or apply horizontal force to pull it out without damaging it too much.