macOS Catalina still uses bash as the default shell
Turns out the terminal app has its own settings for which shell to use.
I had to go to terminal preferences and change the option for Shells open with to Default login shell: