Function of transformer as a filter in isolated DC-DC converters
The datasheet for your part says:

Isolation capacitance  – Input/Output    380 pF typ.
Isolation resistance – Input/Output    &gt;1‘000 M Ohm  (500 VDC)

Thus, as far as common mode noise is concerned, your converter can be modelled as a 380pF capacitor between input and output. The Isolation resistance would not usually be a concern as it is very high, but capacitance can be a problem.
The internal transformer uses high frequency. Since a transformer can never be perfectly symmetrical due to fabrication tolerances etc, some part of the the input chopped waveform is converted into common mode which appears at the output. This is why the manufacturer will also put a capacitor between transformer primary and secondary to provide a short return path for this stray HF common mode current. Without the capacitor, the current would find a way to close the loop anyway, via the rest of your circuit, turning it into an antenna, and that would be undesirable.

For example lets say we have this buck converter where we want to step down 24V to 5V DC. And imagine at the input these is 1MegHz 1Vpkpk common-mode noise or differential mode noise.

There can't be a 1MHz 1Vpp differential mode noise at the input because there is a decoupling capacitor at the input. Differential mode is measured between both input pins.
However, if there is a 1MHz 1Vpp common mode noise at the input this means both input pins' voltage wiggle relative to ground. In this case, the common mode current through the converter will be the same as you would have through a 380pF cap. It is impossible to know the exact amplitude of this current without knowing its complete path, as it will have to make a loop somehow, which may contain various impedances.

How can we estimate the amount of the noise would appear at the 5V DC output?

The differential noise (ripple) will be as specced by the DC-DC datasheet.
The common mode noise will most likely not prevent whatever is powered by this 5V supply from working, especially if it is a digital circuit, but it may convert the circuit into an antenna and radiate.
Since current always travels into a loop, if this is a problem you can reduce the amount of current by increasing the total loop impedance, ie adding a common mode choke at the input or output.