Finding the Best Fitting Plane Given a Set of 3D Points
This is exactly the problem that PCA (principal component analysis) solves.  So, use PCA -- that provides an algorithmic solution to this problem.  See this explanation over at Statistics.SE for an explanation of why PCA solves this problem.

[Least squares fitting is almost a solution to this problem, except that it doesn't minimize the orthogonal distance -- it minimizes the distance along the $y$ axis, or any single axis you might pick.]



If for some reason you didn't want to use PCA, you could frame this as a mathematical optimization problem (the objective function is the total orthogonal distance) and then use any standard optimization technique -- e.g., gradient descent -- to find an approximate optimum.