Given a picture of a unit square at an arbitrary angle and position, how can I reconstruct the distance and orientation?
This problem is known as camera calibration. The goal is estimating the various parameters of a "camera":


The intrinsic parameters: Things like focal length, distortion of the lens, sensor parameters etc.
Extrinsic parameters: Orientation of position of the camera.


To robustly determine those from an image that contains labelled points you usually need more than one image. A typical way to do this is recording many images of a chessboard pattern. But as soon as you have determined all the intrinsic parameters, you could save those for future applications as they do not have to be estimated over and over again. 

I recommend reading the introduction to camera calibration of openCV: 

https://docs.opencv.org/3.1.0/dc/dbb/tutorial_py_calibration.html

https://docs.opencv.org/3.1.0/d9/db7/tutorial_py_table_of_contents_calib3d.html