Do comparison sorts have to start at the beginning of the input?
You can make the comparisons in any order you want. If there was only one possible sequence of comparisons, there'd be only one possible sorting algorithm!

But note that, to specify an algorithm, you have to specify it in terms of operations that your model of computation supports. For example, althouh you as an intelligent human can say "compare 1 with 3", to specify an algorithm, you'd need to be able to describe how to choose the elements to be compared next. Of course, you could try to say "compare the smallest with the second-smallest" but, then, you'd have to give an algorithm for finding those two elements which would, itself, probably use further comparisons.  And, of course, if you already knew which were the smallest two, you wouldn't need to compare them anyway.