In simple terms, what is the relationship between the GPU, Nvidia driver, CUDA and cuDNN in the context for using a deep learning framework?
I have always been doing deep learning on Google Colab or on school clusters that have everything set up nicely. Recently I needed to set up a workstation to do deep learning from scratch and I realized I have very limited understanding of the things that I need to install to run a framework like tensorflow or pytorch on GPU.
So can anyone explain in simple term possible, what is purpose of Nvidia driver, CUDA and cuDNN? How do they work together or on top of one another, and why do I need to install all of them for tensorflow/pytorch?

Python code runs on the CPU, not the GPU. This would be rather slow for complex Neural Network layers like LSTM's or CNN's. Hence, TensorFlow and PyTorch know how to let cuDNN compute those layers. cuDNN requires CUDA, and CUDA requires the NVidia driver. All of the last 3 components are provided by NVidia; they've decided to organize their code that way.
