来 去 depending on location
Please read this previously asked question : Are both 回了 and 回来了 correct responses to 你回天津了吗 (have you returned to Tianjin)?

Since you are currently in Beijing, the answer should be ' Yes I have came back to Beijing' (我回北京来了 or simply 回来了 or even 回了)

The answer is in your own perspective


~

If you replied: &quot;Yes, I have gone back to Beijing&quot; (我回北京去了) Then you are pretending you are in Shanghai with your friend, because your voice is heard in Shanghai via the telephone.

The answer  would be in your friend's perspective


In conclusion: both 回来了 ( your perspective) and 回去了 (your friend's perspective) are acceptable over the phone.
If you omit 来 or 去 and just say : &quot;回了&quot; then it can be interpreted either way.
It would be much simpler if both you and your friend were in the same location.