Why does the IPC standard allow for smaller clearance on inner layers than external layers?
External layers are plated-up with additional copper after etching, whereas internal layers usually aren't.  This can lead to larger manufacturing tolerances, though one could argue that it's the board house's responsibility to deal with that rather than the IPC standard.

Also, solder mask has deliberate openings, some registration tolerance (which can be significant) and isn't immune to damage.  Its thickness may not be especially well controlled and it does not have guaranteed dielectric performance, whereas the core or prepreg material surrounding the traces on inner layers does. The primary purpose of solder mask is to aid soldering yield - the insulation and mechanically protective qualities are secondary.  Where those qualities are critical it would be better to use a conformal coat.