Temp Table Clustered Key Not Being Honored: Bug or Expected Functionality?
There is no guarantee of ORDER without ORDER BY.

The execution plan for both has "Ordered = False". 



This means you may get the results in key order but equally may not. 

Specifically see When can allocation order scans be used?


  The only time such a scan will be used is when there’s no possibility
  of the data changing (e.g. when the TABLOCK hint is specified, or when
  the table is in a read-only database) or when its explicitly stated
  that we don’t care (e.g. when the NOLOCK hint is specifed or under
  READ UNCOMMITTED isolation level). As a further twist, there’s a
  trade-off with setup cost of the allocation order scan against the
  number of pages that will b read – an allocation order scan will only
  be used if there’s more than 64 pages to be read.


As the local temp table is not accessible to other connections you get this behaviour without explicitly taking a table lock however the comment about size of table still applies which is why you see the difference in your two cases.

If you need a specific order add an ORDER BY to get scan in key order (with "Ordered = True").