Views Sort Criteria: Contextual Filter Multiple Value Order
I guess some other people had the same idea I had.

The Views Arguments Extras module handles my situation perfectly. I used the version for 7 (duh :) ), but there is a drupal 6 version as well.