title: Debug, Warn & Info messages from non-main class not visible in spark executor logging
body: <p>We've tried a variety of solutions including changing the log4j.properties file, copying the file to the executors via --file and then telling them to use it as an arg passed to spark via --conf and also tried updating the configuration of the EMR cluster itself.</p>
  
  <p>Warn messages from the system are visible in the executor logs.  Warn messages from the main class are visible but no messages from any other classes are coming through the ether and we're not sure what the problem could be.  </p>
  
  <p>The logging level is fine, as shown by the message generated by Spark but the other class messages are not getting through.</p>
  
  <p>The driver log (from the main page in EMR Console) is showing Debug messages from other classes, the executor logs (via the Spark UI Executor tab) are not.</p>
  
  <p>Any help is hugely appreciated, thanks.</p>
  
  <p>It's a streaming app running on spark 1.6.  Below are some of the options we've tried.</p>
  
  <p>Running the step normally: <code>Arguments: spark-submit --deploy-mode client --master yarn --class main jarLoc</code></p>
  
  <p>Extra spark logging config set at cluster configuration level:</p>
  
  <pre><code>{"classification":"spark-log4j", "properties":{"log4j.logger.MainClass$":"DEBUG", 
  "log4j.logger.org.apache.spark":"WARN", "log4j.logger.org.apache.hadoop":"WARN", "log4j.logger.com.amazonaws.services":"WARN", "log4j.logger.com.companyName":"INFO", "log4j.logger.org.spark-project":"WARN"}, "configurations":[]}
  </code></pre>
  
  <p>Current log4j properties file:</p>
  
  <pre><code>log4j.rootLogger=INFO, STDOUT
  log4j.logger.deng=DEBUG
  log4j.appender.STDOUT=org.apache.log4j.ConsoleAppender
  log4j.appender.STDOUT.layout=org.apache.log4j.PatternLayout
  log4j.appender.STDOUT.layout.ConversionPattern=%d{yyyy-MM-dd hh:mm:ss} %t %x %-5p %-10c:%m%n
  
  log4j.logger.MainClass$=DEBUG
  log4j.logger.com.sessioncam=INFO
  log4j.logger.org.apache.spark=WARN
  log4j.logger.com.amazonaws.services=WARN
  log4j.logger.org.spark-project=WARN
  log4j.logger.org.apache.hadoop=WARN
  </code></pre>
  
  <p>Things I've tried:</p>
  
  <pre><code>spark-submit --deploy-mode client --master yarn --class MainClass--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/tmp/files/log4j.properties /tmp/files/jar.jar
  
  
  Arguments: spark-submit --deploy-mode client --master yarn --class MainClass--files /tmp/files/log4j.properties /tmp/files/jar.jar
  </code></pre>