Toggle between Half-adder and Half-subtractor
This would work. You could also use a tristate-inverter. However, the XOR gate idea would work better because XOR gates are more common to have. On a side note, I used to think of XOR gates as controllable inverters.