Is it worth to upgrade CUDA and cuDNN while having older GPUs?
There is no right answer to this question, because there are many factors to consider, for instance:

Indirect dependency on specific/minimum CUDA version: with deep learning frameworks like pytorch and tensorflow, a specific version of the framework depends on a specific version of CUDA/cuDNN, so the decision of which (minimum) CUDA version to have is directly dependent on the framework version you need.

Bug fixes: new versions or old versions may introduce unexpected bugs that make upgrading/not upgrading not an option.

Performance regressions: some new CUDA versions may introduced performance regressions for older cards (e.g. this, this).


About support for new features, these are normally tied to new hardware (i.e. tensorcores), so it is not frequent to gain new features just by upgrading. But it is not impossible either ¯\(ツ)/¯
So, the answer is: it depends on the case.