PBR missing rim darkening on non metallic rough surfaces
The problem is in the diffuse term, which can be seen by making the specular portion of IBL not be added into the result.  The diffuse only render will not have the darkening, and of course, the specular can't make the diffuse darker. (Note, fresnel vs no fresnel is not the issue here).

The core problem is that lambertian diffuse is being used, which is view independent. There are other view independent diffuse terms which give results closer to the ground truth.

Two example diffuse terms are Burley (Disney) and Oren-Nayar.

Burley is talked about here: https://google.github.io/filament/Filament.md.html#toc4.5

An approach used in Titanfall is described here:
https://www.gdcvault.com/play/1024478/PBR-Diffuse-Lighting-for-GGX