How to adapt diffuse/GI light mapper to directional light maps?
Take a look at Valve's paper on combining Radiosity and Normal Mapping. It had a lot of useful insights.

If I'm not mistaken, it looks to me like you're trying to compute normal mapping using the energy value at the texel that is a result of Radiosity - e.g. it came from all directions through almost-endless bumping around the room across thousands of surfaces. 

But your point light, for the normal mapping, has a very specific position, direction, distance and intensity. Meaning, you're combining apples and goats.

What I believe you should do is:
- compute the normal mapping separately for those surfaces
- blend the GI solution with normal mapping
- experiment with the blending factor till it looks 'good enough' for you