title: Calculate UDF once
body: <p>I want to have a UUID column in a pyspark dataframe that is calculated only once, so that I can select the column in a different dataframe and have the UUIDs be the same. However, the UDF for the UUID column is recalculated when I select the column.</p>
  
  <p>Here's what I'm trying to do:</p>
  
  <pre><code>&gt;&gt;&gt; uuid_udf = udf(lambda: str(uuid.uuid4()), StringType())
  &gt;&gt;&gt; a = spark.createDataFrame([[1, 2]], ['col1', 'col2'])
  &gt;&gt;&gt; a = a.withColumn('id', uuid_udf())
  &gt;&gt;&gt; a.collect()
  [Row(col1=1, col2=2, id='5ac8f818-e2d8-4c50-bae2-0ced7d72ef4f')]
  &gt;&gt;&gt; b = a.select('id')
  &gt;&gt;&gt; b.collect()
  [Row(id='12ec9913-21e1-47bd-9c59-6ddbe2365247')]  # Wanted this to be the same ID as above
  </code></pre>
  
  <p><strong>Possible workaround: rand()</strong></p>
  
  <p>A possible workaround might be to use pyspark.sql.functions.rand() as my source of randomness. However, there are two problems:</p>
  
  <p>1) I'd like to have letters, not just numbers, in the UUID, so that it doesn't need to be quite as long</p>
  
  <p>2) Though it technically works, it produces ugly UUIDs:</p>
  
  <pre><code>&gt;&gt;&gt; from pyspark.sql.functions import rand, round
  &gt;&gt;&gt; a = a.withColumn('id', round(rand() * 10e16))
  &gt;&gt;&gt; a.collect()
  [Row(col1=1, col2=2, id=7.34745165108606e+16)]
  </code></pre>