title: How to log/print message in pyspark pandas_udf?
body: <p>I have tested that both <code>logger</code> and <code>print</code> can't print message in a <code>pandas_udf</code> ,  either in cluster mode or client mode.</p>
  
  <p>Test code:</p>
  
  <pre class="lang-py prettyprint-override"><code>import sys
  import numpy as np
  import pandas as pd
  
  from pyspark.sql import SparkSession
  from pyspark.sql.functions import *
  import logging
  
  logger = logging.getLogger('test')
  
  spark = (SparkSession
  .builder
  .appName('test')
  .getOrCreate())
  
  
  df = spark.createDataFrame(pd.DataFrame({
      'y': np.random.randint(1, 10, (20,)),
      'ds': np.random.randint(1000, 9999, (20,)),
      'store_id' : ['a'] * 10 + ['b'] *7 + ['q']*3,
      'product_id' : ['c'] * 5 + ['d'] *12 + ['e']*3,
      })
  )
  
  
  @pandas_udf('y int, ds int, store_id string, product_id string', PandasUDFType.GROUPED_MAP)
  def train_predict(pdf):
      print('#'*100)
      logger.info('$'*100)
      logger.error('&amp;'*100)
      return pd.DataFrame([], columns=['y', 'ds','store_id','product_id'])
  
  
  df1 = df.groupby(['store_id', 'product_id']).apply(train_predict)
  </code></pre>
  
  <p>Also note: </p>
  
  <pre class="lang-py prettyprint-override"><code>log4jLogger = spark.sparkContext._jvm.org.apache.log4j
  LOGGER = log4jLogger.LogManager.getLogger(__name__)
  LOGGER.info("#"*50)
  </code></pre>
  
  <p>You can't use this in <code>pandas_udf</code>, because this log beyond to spark context object, you can't refer to spark session/context in a udf.</p>
  
  <p>The only way I know is use <code>Excetion</code> as the answer I wrote below. 
  But it is tricky and with drawback.
  I want to know if there is any way to just print message in pandas_udf.</p>