Complement of Mealy machine
You are right: it is easy to miss corner cases, so whatever you choose to define, you will need to prove that you defined the right thing.

To do so, you need to first define what you actually want, i.e., the language of $\overline{M}$.

From what I can tell, you want
$$\mathcal{L}(\overline M) = \{ (w^I_0 ,w^O_0) (w^I_1 ,w^O_1) \ldots \in (\Sigma \times \Omega)^\omega \mid (w^I_0,w^O_0) (w^I_1 ,w^O_1) \ldots \not\in \mathcal{L}(M)\}$$

If you just just invert the set of possible outputs along an edge, then get an automaton $\overline M$ for which you can then try to prove that it has the right language, i.e., that (1) every word accepted by the automaton is in the language that we defined for it, and (2) vice versa.

Because this seems to be connected to your research, I will not give you the answer whether such a proof works (as I don't want you to force to give non-citation credit to the work of others, which you would need to do if someone else gives you a solution), but rather let you try this out on your own.

Should this approach not work, here is an idea that allows you to cover all corner cases:


Translate the Mealy automaton to a DFA (deterministic automaton over finite words) that
accepts all prefix traces of the Mealy automaton's language
Complement the DFA
Conjunct the DFA with with a DFA accepting all words from $(\Sigma \times \Omega)^*$.
The resulting automaton accepts the words that you want your Mealy automaton to have. So define a suitable translation to a non-deterministic Mealy automaton.


The idea here is the complementation part is done by a well-known and known-to-work algorithm, so you avoid parts of your problem. The last step is still non-trivial, but probably easier than before.