Does an HDR environment texture affect the Shadow pass?
The HDR environment does not affect the shadow pass. At all. The problem originated with the removal of the lighting setup.

In other words, the shadow pass only supports lamps. Mesh lights and world lighting will not affect it. See Why does the shadow pass not account for mesh lights?