PySpark RDD SortByKey() Not Working Properly
You are sorting strings - as shown by the u prefix on the output (for Unicode, not that it really matters). Therefore it is working as 10 comes before 2 when tested as a string.
Map all your values to integers before working with them and you should be fine.
I want to sort an RDD that I have, which contains a key range of 0-49995 such that (0, value), ... , (49995, value).
I want to sort it in ascending order, and I am using the SortByKey() function, but it seems like it is not working properly because this is the result that I am getting:



test0.sortByKey(True).take(5)



[(u'0', [u'38737', u'18591', u'27383', u'34211', u'337', u'352', u'1532', u'12143', u'12561', u'17880']), (u'1', [u'35621', u'44891', u'14150', u'15356', u'35630', u'13801', u'13889', u'14078', u'25228', u'13805']), (u'10', [u'83', u'18', u'38', u'89', u'3', u'11', u'29', u'41', u'53', u'55']), (u'100', [u'42704', u'122', u'125', u'128', u'131', u'2501', u'11200', u'12049', u'12576', u'18583']), (u'1000', [u'8671', u'955', u'1012', u'1020', u'1378', u'2413', u'7699', u'10276', u'12625', u'12667'])]
It started at key 0, 1, but then skipped to 10 and jumped to 100, then 1000. It should ascending from 0-5. Can someone please tell me what I am doing wrong here?
Thank you!