How does PostgreSQL compare strings?
That's just how your locale defines sort order. Obviously  upper case letters act as tiebreakers if the string is otherwise identical - then they sort after lower case equivalents. But 'ba' still sorts after 'B' (and 'BA' sorts after 'b').

Compare to results without collation rules:

SELECT   'b' &gt; 'B'
      , 'ba' &gt; 'B'
      , 'ba' &gt; 'C'
      , 'b'  &gt; 'B' COLLATE "C"
      , 'ba' &gt; 'B' COLLATE "C"
      , 'ba' &gt; 'C' COLLATE "C";



 ?column? | ?column? | ?column? | ?column? | ?column? | ?column?
----------+----------+----------+----------+----------+----------
 f        | t        | f        | t        | t        | t



(My current collation setting German_Germany.1252 happens to behave just like your en_ZA.UTF-8.)