Outside of a Universe and its origin
You are making the mistake of thinking that there was a single spot in space that everything expanded from. Rather, the expansion was not of things in space, it was space that expanded. It was an expansion OF space, not IN space. There wasn't "space" around the singularity.