Why shouldn't Bump, Normal and Displacement maps be gamma corrected? So I'm trying to wrap my head around this from a fairly technical point of view.

When you add a Bump or Normal or Displacement map in your shader they should not be gamma corrected. But diffuse textures should be. But why?

When you save a file in 8bit (or 16bit Integer) with a format like JPEG, PNG or TIFF they get the gamma of 0.4545 (1/2.2) burnt in. So a camera captures the light from the real world (that is linear) and when saving the photo as a JPEG a gamma value of 1/2.2 is added, as a way to compress the information into 8bit, and then when you view the image on a monitor that adds a gamma of 2.2 making the luminance linear again.

So when using a diffuse textures (like a photo) you need to a "remove" that 1/2.2 gamma to make it linear (called linearizing) so now the texture looks darker but these values are correct from a mathematical standpoint so the renderer will make the correct calculations. Side note: the software used a display gamma of 2.2 to make sure the textures looks correct it's just for the internal calculation that they are linearized. But when I create a Bump map in Photoshop and save that as a JPG or PNG then that 1/2.2 gamma is burnt into that image as well, so why doesn't this Bump map need to be linearized as well?

If I load a Bump map in 3ds Max with a input gamma of 2.2 just like a would do with a Diffuse texture, the diffuse looks right but the bump looks wrong, so I have to set the input gamma to 1.0 for the bump in order to make it look right, but as I said: why, both image files have the 1/2.2 gamma burnt in, right?
The fact that a photo is stored in gamma space doesn't have to do with PNG or JPEG, it has to do with the fact that it's a photo.

The camera detects a 50% intensity but is going to save it as a 187 instead of 127, because that is more efficient to keep the color information that matters to the human eye when you have only 256 different possible values. And so does Photoshop if you tell it so.

For normals or displacement maps, you don't have this need so there is no reason to use gamma space for them (using gamma for them would actually be a mistake, since it would affect how precision is distributed). Thus when you decode a normal texture or a displacement texture, you don't need to apply gamma.

If the normal map is encoded in a different space though, as can be the case for a G-Buffer for example, then you have to apply the corresponding correction.