title: Random numbers generation in PySpark
body: <p>Lets start with a simple function which always returns a random integer:</p>
  
  <pre><code>import numpy as np
  
  def f(x):
      return np.random.randint(1000)
  </code></pre>
  
  <p>and a RDD filled with zeros and mapped using <code>f</code>:</p>
  
  <pre><code>rdd = sc.parallelize([0] * 10).map(f)
  </code></pre>
  
  <p>Since above RDD is not persisted I expect I'll get a different output every time I collect:</p>
  
  <pre><code>&gt; rdd.collect()
  [255, 512, 512, 512, 255, 512, 255, 512, 512, 255]
  </code></pre>
  
  <p>If we ignore the fact that distribution of values doesn't really look random it is more or less what happens. Problem starts we we when take only a first element:</p>
  
  <pre><code>assert len(set(rdd.first() for _ in xrange(100))) == 1
  </code></pre>
  
  <p>or</p>
  
  <pre><code>assert len(set(tuple(rdd.take(1)) for _ in xrange(100))) == 1
  </code></pre>
  
  <p>It seems to return the same number each time. I've been able to reproduce this behavior on two different machines with Spark 1.2, 1.3 and 1.4. Here I am using <code>np.random.randint</code> but it behaves the same way with <code>random.randint</code>.</p>
  
  <p>This issue, same as non-exactly-random results with <code>collect</code>, seems to be Python specific and I couldn't reproduce it using Scala:</p>
  
  <pre><code>def f(x: Int) = scala.util.Random.nextInt(1000)
  
  val rdd = sc.parallelize(List.fill(10)(0)).map(f)
  (1 to 100).map(x =&gt; rdd.first).toSet.size
  
  rdd.collect()
  </code></pre>
  
  <p>Did I miss something obvious here? </p>
  
  <p><strong>Edit</strong>:</p>
  
  <p>Turns out the source of the problem is Python RNG implementation. To quote <a href="https://docs.python.org/2/library/random.html" rel="noreferrer">official documentation</a>:</p>
  
  <blockquote>
    <p>The functions supplied by this module are actually bound methods of a hidden instance of the random.Random class. You can instantiate your own instances of Random to get generators that don’t share state. </p>
  </blockquote>
  
  <p>I assume NumPy works the same way and  rewriting <code>f</code> using <code>RandomState</code> instance as follows </p>
  
  <pre><code>import os
  import binascii
  
  def f(x, seed=None):
      seed = (
          seed if seed is not None 
          else int(binascii.hexlify(os.urandom(4)), 16))
      rs = np.random.RandomState(seed)
      return rs.randint(1000)
  </code></pre>
  
  <p>makes it slower but solves the problem.</p>
  
  <p>While above explains not random results from collect I still don't understand how it affects <code>first</code> / <code>take(1)</code> between multiple actions.</p>