Camera animated using Follow Path contraint wanders then Jumps back to path near end of animation
I was able to fix the problem. At first I thought that it was to do with the rotation applied to the camera. So I rigged the camera and used a null(empty) to rotate it. This did not fix it. 

It was actually rather simple. 

I had to increase the resolution of the active spline, 192 worked well for me. 

 

I hope this helps anyone who ever has the same problem I did.

If you know of a different/better way of fixing said issue, by all means let me know.