Reaction Pathways: What to do when IRC geometries are relaxed (optimized) and the energy changes significantly
IRC is a gradient following algorithm that tries to follow the imaginary mode. Check the algorithm (several one implemented, so really depends on which you are talking about) and you will see that unlike in optimisation algorithms, there is a big emphasis on following the real gradient (and not just jump to the endpoint as fast as possible).

It works well around the TS where the gradient is large and has a well define direction. However getting closer to the optimised geometry, the requirements of following the exact gradient and having an effective optimization algorithm are contradicting. In analytical sense it may be not obvious, but when you have numeric gradients, small second derivatives (these are almost always very soft modes), the direction of the gradient may have big errors and you should keep a small stepsize to check it thoroughly. 

In summary, the situation you wrote down is more or less common at some extent in almost all the IRC calculations due to the numerical nature of the problem. Harder imaginary frequencies may help.