Adder circuit with sign bit (not twos complement)
This is called "sign-magnitide" or "signed-magnitude" representation. A Google search for "sign-magnitude adder" gives lots of useful results. I know this doesn't answer your question directly (I don't have enough reputation to simply comment your post), but you will find the answer there.

As for only having AND and XOR, this is the classic result that any logic gate can be constructed from AND and NOT. Note that XOR(1,a) = NOT(a). Now you have NAND and NXOR. NOR(a,b) = AND(NOT(a),NOT(b)). Now you have OR by negating NOR. All other gates can be constructed by negating inputs of these gates.