How to use Calculator on MacOS Catalina
RPN Mode

The problem most probably stems from the calculator being in the Reverse Polish Notation (RPN) mode. Things should go back to normal if you simply turn the RPN mode off by pressing ⌘+R.