title: Reducing the log output from Spark in a Scala program
body: <p>I'm writing a Scala / Spark program following <a href="https://medium.com/large-scale-data-processing/how-to-kick-start-spark-development-on-intellij-idea-in-4-steps-c7c8f5c2fe63" rel="nofollow noreferrer">this example</a>.  My tools are IntelliJ and sbt.  (I'm not using the scala spark shell.)  I'm using scala-logging with logback-classic and I need to reduce the logging from Spark or direct the spark logging to a different .log file.</p>
  
  <p>I've tried calling <code>sc.setLogLevel("WARN")</code> in my code where <code>sc</code> is the <code>SparkContext</code> but it has no effect.</p>
  
  <p>To make matters worse the Spark log output comes from several different packages (org.apache.spark, o.a.h.m.lib, o.a.h.s.a, etc.)  I hope there is a better way than defining an appender for each package.</p>
  
  <p>How do I turn off Spark logging, or better yet redirect logging from Spark to a different file than the logging calls from my code?</p>