Discontinuous large values in depth map
It's hard to say for sure without seeing the file.
But I believe you're probably seeing places where the reflections shoot off into infinity.
Let's think about glass, which is one place where you mentioned problems.  A ray either goes through the glass or reflects from the glass.  You can get vastly differing depths depending on which it does.  The depth needs to take into account the reflection for purposes of misting, DoF, etc (think a mirror).  But there's no sense in mixing depths-- you're not going to get something more right by averaging the depth through and the depth reflected, you're just going to get something completely wrong.  Blender has to pick between reflected depth and refracted depth, and where it changes its pick, you have a discontinuity.
What's the solution?  I've asked around and not heard any suggestions.  If you want your depth to a surface, you can render an Eevee depth map, which won't have this behavior.  You could also consider using multiple render layers, perhaps with material AOV outputs that would let you specify flat depth, reflected depth, or refracted/transparent depth.   But yes, these are frustrating solutions in that they can potentially add a lot of render time and complexity.