Sorting Order by GETSORTED
What you see in your execution plan is Sybase performing a DISTINCT operator.

When you do a SELECT DISTINCT in Sybase it first builds a Worktable (as you can see in step 1 of your execution plan) and then it selects sorted data removing duplicates from that worktable (which is what you see in step 2). This is documented

The sort algorithm doesn't appear to be disclosed so if the Sybase developers, with all their wisdom, introduced an optimization where they use data length to determine if values are duplicates, that could have the effect you describe as: 


  I have reasons to believe that it does the sorting on the size of the
  data in column for different rows, and for rows of same length, it
  does a alphabetical


The moral of the story is: 

If you want your data sorted add an ORDER BY clause

As to your question


  in what order does GETSORTED actually sorts the data?


In the order the algorithm can remove duplicates most efficiently.