Is there a sense in which we can say error returned from a function is the reverse of context?
A duality between exceptions and states was observed and worked out by Jean-Guillaume Dumas, Dominique Duval, Laurent Fousse, and Jean-Claude Reynaud.

TL;DR: Reader and exception monads are obviously dual.

In your setup, I think it works as follows. Let $S$ be the type of environments in which a program might run (you call them contexts, but I'd rather save that word for typing contexts), and let $E$ be a type of errors that a program may raise. The environments here are read-only.

A program which runs in environment of type $S$, takes input of type $A$, and outputs a result of type $B$ has type
$$S \times A \to B.$$
A program which may raise an error of type $E$, takes input of type $A$, and outputs a result of type $B$ has type
$$A \to E + B.$$
We clearly have a duality here. If we imagine that this is all hapenning in some category $\mathcal{C}$, then $X \times A \to B$ in $\mathcal{C}$ is the same thing as $B \to X + A$ in the dual category $\mathcal{C}^\mathrm{op}$.