title: Pyspark - DataFrame not updated when applying functions in a loop
body: <p>I'm trying to apply different functions to various columns of a DataFrame depending on a condition. When I do this in a loop, <code>fn1</code> is applied successfully on the first iteration. But the <code>df</code> turns <code>None</code> on the second iteration. I guess the problem is the way I'm initializing the <code>df</code> in the scope of a loop.</p>
  <pre><code>df = spark.createDataFrame([(10,4,2,3),(20,1,3,4),(30,7,4,5),(40,2,1,9)], schema=['id','metric_1','metric_2', 'metric_3'])
  
  cols_info = [{'name':'metric_1','apply_func':'True','method':'fn1'},{'name':'metric_2','apply_func':'True','method':'fn2'}, {'name':'metric_3','apply_func':'True','method':'fn3'}]
  
  def fn1(df, col):
      return df.withColumn(col, F.pow(df[col], 2))
  
  def fn2(df, col):
      return df.withColumn(col, F.hash(df[col]))
  
  def fn3(df, col):
      return df.withColumn(col, F.log2(df[col]))
  
  def process_data(df, columns):
      for col in columns:
          if col[&quot;apply_func&quot;] == &quot;True&quot;:
              if column[&quot;method&quot;] == &quot;fn1&quot;:
                  df = fn1(df, col[&quot;name&quot;])
              if column[&quot;method&quot;] == &quot;fn2&quot;:
                  df = fn2(df, col[&quot;name&quot;])
              if column[&quot;method&quot;] == &quot;fn3&quot;:
                  df = fn3(df, col[&quot;name&quot;])
  
      return df
  </code></pre>
  <p>What is the correct way to apply such transformations with Pyspark DataFrame API?</p>