Failed to load CUDA kernel: rendering crashes after few hundreds of iteration on GPU I'm running blender 2.92 via python on Ubuntu 18.04.5 bionic with RTX 2080 Ti GPUs.
I'm iteratively generating images using cycles. After I generate around 300 images the error in the following comes up and blender crashes. The RTX 2080 has 11GB and each rendering process only takes around 1GB in nvidia-smi so I don't think it's a GPU memory error.
Fra:1 Mem:123.13M (Peak 145.52M) | Time:00:00.16 | Mem:0.00M, Peak:0.00M | Scene, RenderLayer | Loading render kernels (may take a few minutes the first time)
Failed to load CUDA kernel from '/playpen3/home/jmincho/workspace/blender-2.92.0-linux64/2.92/scripts/addons/cycles/lib/kernel_sm_75.cubin' (File not found)

Refer to the Cycles GPU rendering documentation for possible solutions:
https://docs.blender.org/manual/en/latest/render/cycles/gpu_rendering.html

Failed to load CUDA kernel from '/playpen3/home/jmincho/workspace/blender-2.92.0-linux64/2.92/scripts/addons/cycles/lib/filter_sm_75.cubin' (File not found)
Invalid handle in cuModuleGetFunction( &amp;functions.adaptive_stopping, cuModule, &quot;kernel_cuda_adaptive_stopping&quot;) (device_cuda_impl.cpp:600)
Invalid handle in cuModuleGetFunction( &amp;functions.adaptive_filter_x, cuModule, &quot;kernel_cuda_adaptive_filter_x&quot;) (device_cuda_impl.cpp:602)
Invalid handle in cuModuleGetFunction( &amp;functions.adaptive_filter_y, cuModule, &quot;kernel_cuda_adaptive_filter_y&quot;) (device_cuda_impl.cpp:604)
Invalid handle in cuModuleGetFunction( &amp;functions.adaptive_scale_samples, cuModule, &quot;kernel_cuda_adaptive_scale_samples&quot;) (device_cuda_impl.cpp:606)
Writing: /tmp/base_scene.crash.txt
Unable to save '/tmp/base_scene.crash.txt': Too many open files
scandir: Too many open files

I don't know what actually causes the error. I confirmed scripts/addons/cycles/lib/kernel_sm_75.cubin exists in the mentioned path. As Too many open files is mentioned, I tried deleting files in /tmp/ but it didn't help either. Any tips or suggestions would be helpful. Thanks.
As @RobertGützkow mentioned, it was because too many files were opened. Blender was unable to open the CUDA kernel after it reached the system's maximum file descriptor limit. ulimit -n gave me 1024. I increased the limit by ulimit -n 70000 and the error disappeared.