Backing down from a faculty position. Future career problems?
I suggest contacting the department that hired you (the department chair and/or a few colleagues you trust) and explaining the problem. If it is a good university in a developed country, it is almost certain they would have all kinds of resources (both formal and informal) that you are not aware of to help new faculty members with issues related to their relocation, e.g., housing, health insurance and health care, moving arrangements, and other relevant things to ease your transition. They may also be able to rearrange your teaching to allow you not to teach the first semester, so effectively you could arrive on campus a few months after your initial appointment date. This could be done informally or as part of the institution's official policy for parental leave. My institution for example has quite generous policies for such things, so you may be pleasantly surprised at how flexible they can be.

Remember, your new department hired you, so they are probably just as eager to see you joining them as you are to be joining, and would be just as disappointed if you don't come. Will the move be stressful? Of course, any move is stressful, certainly between countries and with a family. But when you weigh the short-term difficulties against the very long term benefits from taking this dream position, I think it is a no-brainer (at least given the information you've provided in the question). Good luck!