title: How to pass configuration properties to SparkListener?
body: <p>Stack: </p>
  
  <ul>
  <li>Java 8</li>
  <li>Spark 2.1.0</li>
  <li>Kafka 0.10.2</li>
  <li>HDP 2.5.2</li>
  </ul>
  
  <p>I am creating a custom SparkListener that will write to Kafka when a Spark job fails at a specific task. In order to do that, I am using the KafkaProducer class which takes in properties. However, we are trying to put this in production, and the Kafka broker addresses cannot be hard-coded into the custom SparkListener class. Thus the need for the use of external properties.</p>
  
  <p>It seems like in the lifecycle of a SparkListener, the onApplicationStart method gets called first (after the constructor). Additionally, I am familiar with passing custom properties to the Spark driver before a Spark Conf/Context gets instantiated via Spark-Submit.</p>
  
  <p>Is there any way to pass external-information to the SparkListener?</p>
  
  <pre><code>`public class TestingSparkListener {
      public static void main(String[] a) {
          SparkConf sc = new SparkConf().setAppName("Testing Custom Spark Listener");
          sc.set("spark.extraListeners", "my.package.TestSparkListener");
          JavaSparkContext jsc = new JavaSparkContext(sc);
      }
  }`
  
  `public class CustomSparkListener {
      static KafkaProducer&lt;String, String&gt; prod;
      @Override
      public void onApplicationStart(SparkListenerApplicationStart applicationStart) {
          SparkConf conf = SparkContext.getOrCreate().getConf(); //problem here
      }
  }`
  </code></pre>
  
  <p>Edited to add sample code</p>