title: Behavior of 'Column' object within function spark
body: <p>I am writting a code to replace characters with the following pathern : [^\w | ] with '' . The point is that when use the DataFrame 'sentenceDF' within my function 'removePunctuation' i get the following error 'column' object is not callable'.</p>
  
  <pre><code>from pyspark.sql.functions import regexp_replace, trim, col, lower
  
      def removePunctuation(column):
          cleanString = column
          cleanString = cleanString.select(regexp_replace(sentenceDF['sentence'],'[^\w | ]','').alias('sentence'))
          cleanString = cleanString.select(regexp_replace(cleanString['sentence'],'_','').alias('sentence'))
          cleanString = cleanString.select(lower(cleanString['sentence']))
  
          return cleanString
  
  
  
      sentenceDF = sqlContext.createDataFrame([('Hi, you!',),
                                               (' No under_score!',),
                                               (' *      Remove punctuation then spaces  * ',)], ['sentence'])
  
      result = sentenceDF.select(removePunctuation(col('sentence')))
      result.show()
  </code></pre>
  
  <p>TraceBack :</p>
  
  <pre><code>    TypeError: 'Column' object is not callable
      --------------------------------------------------------------------------- TypeError Traceback (most recent call last) 
      &lt;ipython-input-50-aa978fac8bae&gt; in &lt;module&gt;() 
           15 (' * Remove punctuation then spaces * ',)], ['sentence']) 
           16 
      ---&gt; 17 result = sentenceDF.select(removePunctuation(col('sentence')))  
  
           18 result.show() 
  
      &lt;ipython-input-50-aa978fac8bae&gt; in removePunctuation(column) 
           4 def removePunctuation(column): 
           5 cleanString = column 
     ----&gt; 6 cleanString = cleanString.select(regexp_replace(sentenceDF['sentence'],'[^\w | ]','').alias('sentence')) 
           7 cleanString = cleanString.select(regexp_replace(cleanString['sentence'],'_','').alias('sentence')) 
           8 cleanString = cleanString.select(lower(cleanString['sentence'])) TypeError: 'Column' object is not callable 
  
      Command took 0.09 seconds -- by andres.velez.e@gmail.com at 10/30/2016, 2:48:17 PM on My Cluster (6 GB)
  </code></pre>