The first traceable use of the phrase point-to-point in connection with a horse race is in Bell's Life on 10 January 1874.
One of the few remaining point-to-point races run under the original conditions is the New Forest Boxing Day point-to-point, which has a given start and finish point, with riders allowed to choose their own course in-between.