title: Pyspark not logging to file
body: <p>I am launching a pyspark script with the command <code>spark-submit</code> redirecting the standard output to file too with <code>tee</code> to have a log.</p>
  
  <p>The command is the following:</p>
  
  <pre><code>spark-submit test.py  | tee test.xxx
  </code></pre>
  
  <p>The problem is that <strong>only</strong> the <code>print</code> inside user defined function <code>UDF</code> got printed only on the Terminal but not to the file <code>tee test.xxx</code> while all the the other prints will both write to terminal and file.</p>
  
  <p>To simulate this behaviour I created this minimal complete working example:</p>
  
  <pre><code>from pyspark import SparkContext
  import pyspark.sql.functions as F #udf, col, count, sum, when, avg, mean, min
  from pyspark.sql import SQLContext
  from pyspark.sql.types import *
  def cutURL(input):
      cutURL.lineNumber += 1
      if input==None or input=="" or not(isinstance(input, str)):
          print("WARNING: not proper string URL: empty or null. Possible line: " + str(cutURL.lineNumber))
          res = "Unknown"
          if input==None: print("string is none")
          elif not(isinstance(input, str)): print("input not a string")
          elif input=="": print("empty string")
          return res
      res = input
      try:
          if (bool(re.search("/devices(.+?)&amp;maxdist=[0-9]+", input))):
              res = re.search("/devices(.+?)&amp;maxdist=[0-9]+", input).group()
          else:
              res = re.sub(r'.*?(/devices/[^/]*_)[^/_]*(/read)', r'\1\2', input)
      except:
          print("WARning in cutURL:")
          print(" not matching regular expression: is the string")
      return res
  
  
  sc = SparkContext.getOrCreate()
  sc.setLogLevel("WARN")
  sqlContext = SQLContext(sc)
  cutURL.lineNumber = 0
  print("This will be printed to both file and terminal")
  df = sqlContext.createDataFrame([None, "example",  "others"], "string").toDF("url")
  cut_URL_udf =  F.udf(cutURL, StringType())
  df2 = df.select(cut_URL_udf("url").alias("cut_URL"))
  df2.show()
  </code></pre>
  
  <p>In this case the string <code>WARNING: not proper string URL: empty or null. Possible line:</code> got printed only on terminal but not to file.</p>
  
  <p>How can make output generated inside a pyspark UDF redirected to file too?</p>
  
  <p><strong>EDIT</strong>
  To explain better my problem I add the line <code>print("This will be printed to both file and terminal")</code>. This one will be printed to terminal and logged to file while the <code>print</code> inside udf only to terminal.</p>