title: How to set logLevel in a pyspark job
body: <p>I'm trying to set the log level in a pyspark job. I'm not using the spark shell, so I can't just do what it advises and call sc.setLogLevel(newLevel), since I don't have an sc object.</p>
  
  <p>A lot of sources say to just modify the log4j.properties, but I don't know where to find/put that file. I used pip install pyspark in a virtual environment, so I don't have a $SPARK_HOME environment variable that I've set (the sources usually say the log4j.properties is in the $SPARK_HOME).</p>
  
  <p>I hope I can call this programmatically, but I don't know where to call setLogLevel. Right now my setup code is just this:</p>
  
  <pre><code>spark = SparkSession.builder.master("local").appName("test-mf").getOrCreate()
  </code></pre>