Delete key inserts a space on Python interpreter after upgrading to Catalina
So after some further testing, it looked like it was an issue with only certain virtual environments that I was using that I had reconfigured by following the guide on anaconda:

https://www.anaconda.com/how-to-restore-anaconda-after-macos-catalina-update/

My system python had also been set to the conda one by default by this, which was why I was getting the issue. 

I just ended up creating a new environment and reinstalling all my packages.