title: Method pow does not exist error Pyspark error
body: <p>This is a bit strange.</p>
  <p>When I create a dataframe and then do some transformations with the function pow, it works. But when I push it to run in a real world scenario, it does not. The datatype of columns in my dummy and the real world scenario is the same</p>
  <p>Error</p>
  <pre><code>Method pow([class java.lang.Double, class java.lang.Double]) does not exist
  </code></pre>
  <p>This works (with made up data)</p>
  <pre><code>from pyspark.sql.types import StructType,StructField, IntegerType, DoubleType
  
  columns = [&quot;CounterpartID&quot;,&quot;Year&quot;,&quot;Month&quot;,&quot;Day&quot;,&quot;churnprobability&quot;, &quot;deadprobability&quot;]
  data = [(1234, 2021,5,12, 0.85,0.6),(1224, 2022,6,12, 0.75,0.6),(1345, 2022,5,13, 0.8,0.2),(234, 2021,7,12, 0.9,0.8), (1654, 2021,7,12, 1.40,20.0), (7548, 2021,7,12, -1.40,20.0),  (6582, 2021,7,12, -1.40,20.0)]
  
  
  
  schema = StructType([ \
      StructField(&quot;CounterpartID&quot;,IntegerType(),False), \
      StructField(&quot;Year&quot;,IntegerType(),False), \
      StructField(&quot;Month&quot;,IntegerType(),False), \
      StructField(&quot;Day&quot;, IntegerType(), False), \
      StructField(&quot;churnprobability&quot;, DoubleType(), False), \
      StructField(&quot;deadprobability&quot;, DoubleType(), False) \
    ])
   
  df = spark.createDataFrame(data=data,schema=schema)
  df.printSchema()
  df.show(truncate=False)
  
  abc=df.withColumn(&quot;client_id&quot;, f.col(&quot;CounterpartID&quot;))\
        .withColumn(&quot;year&quot;, f.col(&quot;Year&quot;))\
        .withColumn(&quot;month&quot;, f.col(&quot;Month&quot;))\
        .withColumn(&quot;day&quot;, f.col(&quot;Day&quot;))\
        .withColumn(&quot;churn_probability_unit&quot;, f.col(&quot;churnprobability&quot;).cast(IntegerType()))\
        .withColumn(&quot;churn_probability_nanos&quot;, ((f.col(&quot;churnprobability&quot;) - f.col(&quot;churnprobability&quot;).cast(IntegerType())) * pow(10,9)).cast(IntegerType()))\
        .withColumn(&quot;dead_probability_unit&quot;, f.col(&quot;deadprobability&quot;).cast(IntegerType()))\
        .withColumn(&quot;dead_probability_nanos&quot;, (f.col(&quot;deadprobability&quot;) %1 * pow(10,9)).cast(IntegerType()))\
        .select(&quot;client_id&quot;, &quot;year&quot;, &quot;month&quot;, &quot;day&quot;, &quot;churn_probability_unit&quot;, &quot;churn_probability_nanos&quot;, &quot;dead_probability_unit&quot;,&quot;dead_probability_nanos&quot;)\
  
  abc.show()
  
  </code></pre>
  <p>However, instead of df, in the real world scenario (Production job), I have a real dataframe ( of course) and all columns in that has the same datatype as my dummy dataframe above:</p>
  <p>e.g. here:</p>
  <p><a href="https://i.stack.imgur.com/moz0L.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/moz0L.png" alt="Production Dataframe" /></a></p>
  <p>However, when I do the same transformation, it complains of not able to find the pow function with double parameters. Here's the stack trace (below). I have also looked up the docs <a href="https://spark.apache.org/docs/3.1.3/api/python/_modules/pyspark/sql/functions.html#pow" rel="nofollow noreferrer">here</a>, for pow, but it does not talk anything about datatypes. Some S.O posts suggests Double should be ok. where is it going wrong? I can of course change it multiply by the actual number , instead of using pow, but I would still like to understand this better. Any questions, and I can help answer.</p>
  <pre><code>----&gt; 7       .withColumn(&quot;churn_probability_nanos&quot;, ((f.col(&quot;churnprobability&quot;) % 1.0) * pow(10,9)).cast(IntegerType()))\
        8       .withColumn(&quot;dead_probability_unit&quot;, f.col(&quot;deadprobability&quot;).cast(IntegerType()))\
        9       .withColumn(&quot;dead_probability_nanos&quot;, (f.col(&quot;deadprobability&quot;) %1 * pow(10,9)).cast(IntegerType()))\
  
  /databricks/spark/python/pyspark/sql/functions.py in pow(col1, col2)
      737     Returns the value of the first argument raised to the power of the second argument.
      738     &quot;&quot;&quot;
  --&gt; 739     return _invoke_binary_math_function(&quot;pow&quot;, col1, col2)
      740 
      741 
  
  /databricks/spark/python/pyspark/sql/functions.py in _invoke_binary_math_function(name, col1, col2)
       73     and wraps the result with :class:`~pyspark.sql.Column`.
       74     &quot;&quot;&quot;
  ---&gt; 75     return _invoke_function(
       76         name,
       77         # For legacy reasons, the arguments here can be implicitly converted into floats,
  
  /databricks/spark/python/pyspark/sql/functions.py in _invoke_function(name, *args)
       57     &quot;&quot;&quot;
       58     jf = _get_get_jvm_function(name, SparkContext._active_spark_context)
  ---&gt; 59     return Column(jf(*args))
       60 
       61 
  
  /databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py in __call__(self, *args)
     1302 
     1303         answer = self.gateway_client.send_command(command)
  -&gt; 1304         return_value = get_return_value(
     1305             answer, self.gateway_client, self.target_id, self.name)
     1306 
  
  /databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
      108     def deco(*a, **kw):
      109         try:
  --&gt; 110             return f(*a, **kw)
      111         except py4j.protocol.Py4JJavaError as e:
      112             converted = convert_exception(e.java_exception)
  
  /databricks/spark/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
      328                     format(target_id, &quot;.&quot;, name), value)
      329             else:
  --&gt; 330                 raise Py4JError(
      331                     &quot;An error occurred while calling {0}{1}{2}. Trace:\n{3}\n&quot;.
      332                     format(target_id, &quot;.&quot;, name, value))
  
  Py4JError: An error occurred while calling z:org.apache.spark.sql.functions.pow. Trace:
  py4j.Py4JException: Method pow([class java.lang.Double, class java.lang.Double]) does not exist
      at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:341)
      at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:362)
      at py4j.Gateway.invoke(Gateway.java:289)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:251)
      at java.lang.Thread.run(Thread.java:748)
  </code></pre>