Should I be concerned if my charger is earthed (grounded) or not?
All Power Supplies must meet 4 critieria


1) Safety; for Hipot, leakage to earth and other things.
2) EMI compatibility
3) meet product requirements for  AC-DC or AC-RF mobile charger
4) Low Cost
Will they fail from a 6kV lightning transient? 


possibly (especially in Florida) but not 2~3kV according to regional product power standards in an area protected with line lightning arrestors

Are you safe from secondary breakdown and getting zapped
yes if they are compliant and there are no manufacturing defects.
all manufacturers are only required to do 100% Hipot and leakage tests but not EMI tests


What about EMI?


It is well defined in most countries but with the flooded online aftermarket of 3rd party power adapters, it's a gray zone.


The 3rd party or aftermarket chargers like OEM's are self-regulated and are not tested with every target device as OEM's are required to do. So compatibility with load noise and switcher noise may vary if the step load rates happen to interfere under some load condition. So hypothetically it may become a source of EMI if not an OEM supply or may not have the smart communication signals over DC power to identify the match to the target device.

Radio and power line conducted interference requirements are well specified for consumer and commercial equipment.

Susceptibility has long been a corporate standard and then included in CE standards.  The RF standard levels  are shown in a graph below. But there are more tests for Hipot, leakage current to ground wire , ESD damage and sometimes a few more. ( coke spill test, drop test, temp rise test, short circuit test)

Beside converting AC to DC which usually generates a lot fast rise time current pulses, the purpose is to  minimize this "egress" going out and "ingress from power line transients (PLT) and lightning noise.

There are many methods to this AC-DC conversion.  Long ago bricks just pumped high pulse currents just after a bridge rectifier peak voltage that had to be sustained until the next half cycle. These were notorious for causing external mic buzz issues on some 3rd party desk mics for laptops and even towers with ATX 50kHz SMPS converters.  The converters didn't radiate that far but far enough to be picked up by an unbalanced desktop mic plugged into an ungrounded mobile laptop or even a grounded desktop. Usually ok, but some were caused a buzz.

This is due the common mode stray EMI coupling into an unbalanced high impedance mic( or radio or other elctronic device nearby) and the receiving cable converts the CM noise into a differential voltage due to unbalanced impedance on signal vs ground.

The most common solution is a 2 way Pi Filter , sometimes in two stages. The winding balance is critical and the Y caps to earth ground are plastic high quality and rated for lightning transients for breakdown safety.

Now you have notied that many mobiles with low power mobiles e.g. iPhones, iPads do not use 3 pronged plugs.  Instead they higher frequency Ac-DC converters where the CM noise can be absorbed in the lossy ferrite CM choke instead of being shunt from CM choke to earth ground.

Emission levels


An example of some 2 pronged AC-DC noise filters ( PLT protection not shown)



This does not mean ALL 2 pronged mobile chargers are compliant to conducted/radiated ingress/egress but many are such as Apple's.

So don't be worried. Just be aware of the requirements and possible EMI and susceptibility to damage or interference from lightning or power line transients and changes to local standards in your country.

Just beware of of the requirements and 3rd party chargers that have not been certified with your equipment.  It may be OK, so trust  but verify or don't trust. 

Your choice.

edit

Content References: all from my experience as an EE since 1975 doing R&amp;D, Test Eng , CMS and EMI testing.

Image references: https://interferencetechnology.com/practical-reasons-for-shifting-to-the-application-of-dielectric-independent-emi-filters-with-integral-surge-protection-in-product-designs/