Analog Mute Circuit - Simplest Solution
The PCM5102A datasheet suggests that this can be done on the chip itself.



XSMT, pin 17, input, Soft mute control: Soft mute (Low) / soft un-mute (High).



From the datasheet:


  11.2 Recommended Powerdown Sequence
  
  Under certain conditions, the PCM510xA devices can exhibit some pop on power down. Pops are caused by a
  device not having enough time to detect power loss and start the muting process.
  
  The PCM510xA devices have two auto-mute functions to mute the device upon power loss (intentional or
  unintentional).
  
  XSMT = 0
  
  When the XSMT pin is pulled low, the incoming PCM data is attenuated to 0, closely followed by a hard analog
  mute. This process takes 150 sample times (ts
  ) + 0.2 ms.
  Because this mute time is mainly dominated by the sampling frequency, systems sampling at 192 kHz will mute
  much faster than a 48-kHz system.
  
  Clock Error Detect
  
  When clock error is detected on the incoming data clock, the PCM510xA devices switch to an internal oscillator,
  and continue to the drive the output, while attenuating the data from the last known value. Once this process is
  complete, the PCM510xA outputs are hard muted to ground.


I don't think you'll have any noise.