Subtracting standard electrode potentials
Let's take a look at what you're doing specifically for computing the potential difference. The potentials at the cathode and the anode are all relative to some standard. Importantly, it must be the same standard. Let's refer to this as ground.

$$E_{\mathrm{cathode}} = V_{\mathrm{cathode}} - V_{\mathrm{ground}}$$
$$E_{\mathrm{anode}} = V_{\mathrm{anode}} - V_{\mathrm{ground}}$$

The $V$'s here represent absolute potentials on some arbitrary scale.

What's the change in potential when I move a test charge from the anode to the cathode? Since potential differences are path independent, we can look at this via states: moving the charge from anode to cathode is the same as moving the charge from anode to ground and then from ground to cathode.

The potential difference is expressed as:

$$E = (V_{\mathrm{cathode}} - V_{\mathrm{ground}}) - (V_{\mathrm{anode}} - V_{\mathrm{ground}}) = E_{\mathrm{cathode}} - E_{\mathrm{anode}}$$

The key is the cancellation of the ground potential, so that we have a potential difference.

This is exactly what are doing when working with half-reactions. And it works because both the reduction and oxidation half reactions are referenced to the standard hydrogen electrode.

Let's look at two half-reactions:
$$\ce{A -&gt; A+ + e-}\ ,\ \ E^{⦵} = -E_{A}$$
$$\ce{B+ + e- -&gt; B}\ ,\ \ E^{⦵} = E_{B}$$

As a general rule, we tabulate reduction potentials, so let's treat the $\ce{A}$ reaction with a reduction potential:

$$\ce{A+ + e- -&gt; A}\ ,\ \ E^{⦵} = E_{A}$$

Now, both of these half reactions are relative to the standard hydrogen electrode.

In the reaction
$$\ce{A + B+ -&gt; A+ + B}$$
we can look at the equivalent process of moving an electron from $\ce{A}$ to a proton in a standard hydrogen electrode, and then taking a electron from dihydrogen in the standard hydrogen electrode and moving it to $\ce{B}$.

The potential difference is:
$$E^{⦵}_{\mathrm{cell}} = E_{B} + (-E_{A}) = E_{B}-E_{A}$$

This is exactly the same process because the anode potential relative to ground is the reduction potential of the oxidation half-reaction. The potential difference is computed by subtracting the potential from the other one.

Alternatively, if you already expressed the anode potential as a oxidation potential (inverse of the reduction potential), then the minus sign is implicitly included, and you can just add the two potentials.