Discharge capacitor faster during reset
You could also use an nchannl mosfet like the 2N7002 off of the inverted reset which is fast and could also prevent noise from reaching the reference as the capacitive coupling between the gate an reference is small and high frequency (well above audio range)

Also, you might want to reduce the reference cap to 10uF (the UDA1334BTS needs at least 10uf) which would also reduce the startup time.