Failed to load CUDA kernel: rendering crashes after few hundreds of iteration on GPU
As @RobertGützkow mentioned, it was because too many files were opened. Blender was unable to open the CUDA kernel after it reached the system's maximum file descriptor limit. ulimit -n gave me 1024. I increased the limit by ulimit -n 70000 and the error disappeared.