Enable Finder "Calculate all sizes" checkbox by default
On macOS Big Sur, in Finder, when you 'Show View Options' you get the following window where you can see there is a button at the bottom to use the settings as default. After adjusting the settings, press that button and see if it solves your problem.