Why does it take over 4 seconds to eliminate audio popping in this circuit?
Here's what the data sheet says about Cb (the 10 uF capacitor in your circuit): -


  Bypass Capacitor Value
  
  Besides minimizing the input capacitor size, careful consideration
  should be paid to the value of the bypass capacitor, CB. Since CB
  determines how fast the LM4808 settles to quiescent operation, its
  value is critical when minimizing turn-on pops. The slower the
  LM4808's outputs ramp to their quiescent DC voltage (nominally 1/2
  VDD), the smaller the turn-on pop. Choosing CB equal to 1.0μF or
  larger, will minimize turn-on pops. As discussed above, choosing Ci no
  larger than necessary for the desired bandwidth helps minimize clicks
  and pops.


So I would definitely experiment with this value and initially try 1 uF. Also, it makes little sense to use both amplifiers (not op-amps BTW) of this package - attach your headphone decoupling capacitors to the first stage. Cascading two stages might make the popping problem worse.

You should also note that having nearly 2000 uF coupling your headphones (32 ohms or thereabouts) has a high pas cut-off of only 2.5 Hz and this is too low for audio. Make it more like 220 uF and you might find the pop is reduced.