Making a HDRI map more subtle creates a drop shadow
It's because you're blurring the Environment Pass. The Environment Pass is not inclusive of the area behind the building. In other words, there's a black hole where the building is that's being blurred... therefore some of the blue is going to be darker around the edge. 





I've been hoping and waiting for the Blender devs to include the capability of making only specific layers transparent.  As far as I know, this doesn't exist yet.  

If you want to blur the environment without having a halo and without having to save a separate render to disk and then composite the image to your scene, you can use the "Glass Sphere" trick.  

1.) Set Transparent under Render Properties >> Film:





2.) Add a Sphere object (low poly) with Smooth Shading enabled and set the material to be Glass with a IOR of 1.0 and a Color of 1.0 White.  Make it large enough to encompass your Camera and then Parent it to the Camera:





3.) Create two render layers... one for your setting and one for the sphere.  Exclude the Sphere's layer from everything else (this ensures that the sphere itself doesn't affect the lighting on any objects in your scene).  Set the Number of Samples on the Sphere Layer to 1 (it should only take a couple of seconds to render):





4.) After that, blur it as you were with the Environment pass:



EDIT: I forgot... the reason this works is because you now have a shader (albeit a perfectly clear shader) covering the camera plane.  Blender must render the shader and all the rays that are affecting it, hence the environment on the other side of the glass must be visible.  For obvious reasons, a Transparent shader will not work for this.