title: Pickle serialization order mystery
body: <p><strong>Update 6/8/17</strong></p>
  
  <p>Though 3 years passed, my <a href="https://github.com/tschellenbach/Stream-Framework/pull/53" rel="nofollow noreferrer">PR</a> is still pending as a temporary solution by enforcing the output order. Stream-Framework might reconsider its design on using content as key for notifications. GitHub <a href="https://github.com/tschellenbach/Stream-Framework/issues/153" rel="nofollow noreferrer">Issue #153</a> references this.</p>
  
  <p><strong>Question</strong></p>
  
  <p>See following sample:</p>
  
  <pre><code>import pickle
  x = {'order_number': 'X', 'deal_url': 'J'}
  
  pickle.dumps(x)
  pickle.dumps(pickle.loads(pickle.dumps(x)))
  pickle.dumps(pickle.loads(pickle.dumps(pickle.loads(pickle.dumps(x)))))
  </code></pre>
  
  <p>Results:</p>
  
  <pre><code>(dp0\nS'deal_url'\np1\nS'J'\np2\nsS'order_number'\np3\nS'X'\np4\ns.
  (dp0\nS'order_number'\np1\nS'X'\np2\nsS'deal_url'\np3\nS'J'\np4\ns.
  (dp0\nS'deal_url'\np1\nS'J'\np2\nsS'order_number'\np3\nS'X'\np4\ns.
  </code></pre>
  
  <p>Clearly, serialized output changes for every dump. When I remove a character from any of keys, this doesn't happen. I discovered this as <a href="https://github.com/tschellenbach/Stream-Framework" rel="nofollow noreferrer">Stream-Framework</a> use pickled output as key for storage of notifications on its k/v store. I will pull request if we get a better understanding what is going on here. I have found two solutions to prevent it:</p>
  
  <p>A - Convert to dictionary after sorting (yes, somehow provides the intended side effect)</p>
  
  <pre><code>import operator
  sorted_x = dict(sorted(x.iteritems(), key=operator.itemgetter(1)))
  </code></pre>
  
  <p>B - Remove underscores (but not sure if this always works)</p>
  
  <p>So what causes the mystery under dictionary sorting for pickle?</p>
  
  <p>Proof that calling sort over dict provides dump to produce same result:</p>
  
  <pre><code>import operator
  x = dict(sorted(x.iteritems(), key=operator.itemgetter(1)))
  
  pickle.dumps(x)
  "(dp0\nS'order_number'\np1\nS'X'\np2\nsS'deal_url'\np3\nS'J'\np4\ns."
  
  x = pickle.loads(pickle.dumps(x))
  x = dict(sorted(x.iteritems(), key=operator.itemgetter(1)))
  
  pickle.dumps(x)
  "(dp0\nS'order_number'\np1\nS'X'\np2\nsS'deal_url'\np3\nS'J'\np4\ns."
  </code></pre>