Grounding without Ground
Motor cars are not galvanically grounded and they produce tens of thousands of volts for the spark plugs. They are "safe" by design.


  My enclosure is metal, I need to ground this enclosure.


If your metal box is powered from an SMPS that is designed to the correct standards then what is the issue?

If it's an issue of EMC then sure, you need to think about how to reduce emissions. If it's a safety issue then for sure you MUST choose a suitable SMPS.


  I am going to connect my circuit return ground (5V return ground) to
  the enclosure


It may be a good method but it might also cause other problems - it depends on how the SMPS is designed - quite a few have an EMC capacitor from the DC output wire back to the AC rectified smoothing capacitor - this reduces EMC to the DC side and can offer some protection the other way round.


  How effective the shield will be on enclosure by doing this.


It may be somewhat effective or it may be worse. You need to do some measurements.