Entropy loss from collapsing/merging two categories
I ended up coming up with my own solution. I'm not sure if it's correct so I won't mark it as an answer.

Let's call the original distribution P. Suppose we will be collapsing A and B into a new category X. The entropy of P will be: $S_P = - \sum P_i\ln P_i$ where $i$ is A, B, C, ... Notably, $P_A &gt; 0, P_B &gt; 0, P_X = 0$ .

After collapsing, we get a new distribution $Q$. Now $Q_A=Q_B=0$ and $Q_X =Q_A+Q_B$. This will have entropy $S_Q$.

The entropy we lost by collapsing is $S_P-S_Q = P_A \ln P_A + P_B \ln P_B - Q_X \ln Q_X$. All other terms, such as $P_C\ln P_C$ for category $C$ which is unaffected, are present in both distributions and cancel out.