How to calculate heat dissipation improvement when adding a fan to the heat sink?
You are dissipating at least 45W in your regulator, which is a lot.

Assume you have the TO-3 package (the TO-220 simply won't work at ambient 25C no matter what heat sink you use) with a theoretical operating die temperature of 125C. Computing the allowable thermal resistance to 25C ambient:

\begin{align}
\Theta_{JA} = \frac{125-25}{45} = 2.22 \frac{C}{W}
\end{align}

However, the TO-3 package version has a 1C/W resistance from the die to case, so you only get 1.22C/W for the heat sink. Heat sink manufacturers will usually give you a specified value of what the thermal resistance to ambient is for force and natural convection.

For example, this heat sink specifies 0.9C/W at 250 LFM (linear feet per minute), which is adequate for this application. There is also a plot of what the thermal resistance vs. air flow speed is, and it looks like roughly 125-150 LFM is the minimum flow speed you can have to maintain the device temperature below 125C.

However, this is running the device at it's limit, and is completely impractical. Even with the specified 0.9C/W thermal resistance, you're going to get an 85.5C rise above ambient, meaning your junction temperature is 110.5C. That's stressing the device way too much.

I would take the suggestions in the comments and use a buck regulator instead of a linear one.

As a side-note, I'm not sure why you are complaining about the heat sink cost. The TO-3 package LM338 is ~$50 unit price, while I could find a heat sink with 0.5C/W @250 LFM thermal resistance for ~$20 unit price. A buck regulator solution is significantly cheaper, and if you really need the linear regulator stability, then add a linear regulator after the buck regulator. Now there's only 10W dissipated in the linear regulator so you might be able to get away with either no heat sink and just use a fan, or just have a heat sink with no fan.

To actually calculate these values is beyond the scope of this answer... needless to say, for any complex geometry you will need to use CFD software.