Mathematical model used in free route airspace
En-route simulation by airtopsoft is a good simulation modeling software package that I have utilized in the past to model airspace routing. It also give you some pretty powerful tools to analyze the data afterward.

Another commonly used mathematical tool for modeling airspace and routing is the "Airspace Concept Evaluation System" (ACES). This platform uses four-degrees-of-freedom equations of motion to model aircraft performance and navigation data. Based on the trajectory data, it generates "pseudo-radar" tracks to help predict collision avoidance patterns. I believe this tool is more commonly used to predict routes in the NAS but I believe you could utilize it for free route airspace by playing with the input parameters. 

Hope this gives you a good place to start!