Images loading from server on every page load
Caching problem was not with the views, it was actually Adaptive Image module which was causing the frustrating image reload issue.

After trying several caching options, I disabled the Adaptive Styles and use normal image styles and the issue was resolved. 

For adaptive images I am now using Adaptive Image Styles and it is working fine. Though this require some modifications in .htaccess file but that is acceptable.