Why does it take a long time for water to start draining from my tub faucet?
The stopper you're describing is called the diverter. The symptoms you're describing sound like it's gotten slightly bent or nicked. I would take it apart and try re-setting the pin. Maybe scrub it a bit with a light abrasive to true it up. If that doesn't work, replace it. I assume your diverter is in your tub spout. You can usually remove that by removing the hex-head set-screw and then rotating the entire spout (the padded handle of a hammer stuck in the spout can help you get a grip).