title: Passing dynamic log file name to log4j.properties in python
body: <p>I can find the answer to this in Java, but so far I haven't seen a Python solution so I'm posting this question.</p>
  
  <p>In my log4j.properties, I have:</p>
  
  <pre><code>log4j.rootLogger=WARN,LOGFILE
  log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
  log4j.appender.LOGFILE.File=log/${scriptname}.log
  log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
  log4j.appender.LOGFILE.Append=false
  log4j.appender.LOGFILE.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
  </code></pre>
  
  <p>In script.py in my main, I call a method to launch spark:</p>
  
  <pre><code>spark_submit(yarn_pool, os.path.basename(__file__))
  </code></pre>
  
  <p>Which is defined here:</p>
  
  <pre><code>def spark_submit(yarn_pool, scriptname):
     spark_submit_command = 'spark2-submit'
     ret_code = subprocess.call([
          spark_submit_command,
          '--master', 'yarn',
          '--queue', yarn_pool,
          '--executor-memory', '16g',
          '--driver-java-options', f'-Dlog4j.debug=true -Dlogfile.name={scriptname}'
  </code></pre>
  
  <p>Late in script.py, I attempt the logging:</p>
  
  <pre><code>conf = SparkConf()
  conf.setAppName("My App")
  spark = SparkContext(conf=conf)
  
  log4jLogger = spark._jvm.org.apache.log4j
  LOGGER = log4jLogger.LogManager.getLogger("root.logger")
  LOGGER.warn("Starting App")
  </code></pre>
  
  <p>I'm trying to find a way to pass the filename of my script into the spark_submit method, and then into log4j.properties, but I cannot figure out the syntax to get the code to actually recognize my scriptname parameter. </p>
  
  <p>I've tried ${sys:scriptname} and ${env:scriptname} as well, and those are also unrecognized. There doesn't seem to be clear documentation on how variables through all these files are passed together, and I'd appreciate help in understanding this. </p>