Alpha in the mist channel
In order to set pixels of the mist transparent where the original render is transparent, you can multiply the alpha pass with the mist pass.
The full steps to achieve this result are:

Enable film transparency (Render Properties &gt; Film &gt; Transparent)
Enable the mist pass (View Layer Properties &gt; Passes &gt; Data &gt; Mist)
Switch to the Compositing workspace or open the compositor and enable Use Nodes
Add a Math node (Add &gt; Converter &gt; Math), set the mode to Multiply and use the Alpha of the render layer and the Mist as inputs. Alternatively this should also work with the Mix node set to Multiply and the factor set to 1. The effect is that the alpha is used to mask the mist. When the alpha is zero the result of the math node is zero as well, thus the result is transparent in exactly the same places as the alpha channel. 
Use the result from the previous step for further compositing.


Below is a visualization of the intermediate steps of the compositing node tree.
Image pass

Alpha pass

Mist pass

Mist pass multiplied with alpha pass (masked mist pass)

Image pass mixed with masked mist pass (final result)

Compositing nodes