title: Log4j JDBC with Apache Storm 1.0.1 not writing log statements
body: <p>I am having issues using Log4J2 (2.1) in storm (1.0.1) with logging to jdbc appender.  I have been following the Microsoft HDInsight doc <a href="https://docs.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-develop-java-topology" rel="nofollow noreferrer" title="Apache Storm develop java topology">Apache Storm develop java topology</a>.  Additional factors include using the Microsoft EventHubs (1.0.2) library for EventHubsSpout configuration, which have tended to cause some Maven "Dependency Hell" issues with SLF4j.</p>
  
  <p>When debugging a local cluster, I have storm-core 1.0.1 being included in the jar (which is then marked provided in the pom.xml when the topology is deployed).  I have a connection factory class for the JDBC connection, and then declare the JDBC appender in the log4j2.xml configuration file.  The local cluster tends to output correctly (which we have tried both a console and a JDBC appender successfully).  </p>
  
  <p>When the topology is submitted to the HDInsight cluster, there are no errors thrown in Storm-UI, but there are no log statements being output to the database.  I have tried:</p>
  
  <ul>
  <li>Implicitly using the storm-core log4j2 dependency.</li>
  <li>Explicitly including log4j2 dependencies for core and api in the pom.xml (See code snippet).</li>
  <li>Using all available log types (Debug -> Error).  I read in other online solutions that some log output types will get picked up by Storm's internal logging to Storm-UI, but that hasn't been the case.</li>
  </ul>
  
  <p>Environmentally, we have confirmed that the cluster can talk to the SQL Server.  We have also confirmed that the jdbc driver works on the cluster to talk to the SQL Server.</p>
  
  <p>Any input as to why this logs successfully as a local cluster, but than "Fails" to log when submitted to the HDInsight cluster (considering the setup is nearly identical to the Microsoft doc with the exception of using JDBC instead of console)?</p>
  
  <p>Code snippets below (Reminder, storm-core -> Scope: Provided is commented out for local cluster):</p>
  
  <p>pom.xml:</p>
  
  <pre><code>&lt;dependencies&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.storm&lt;/groupId&gt;
          &lt;artifactId&gt;storm-core&lt;/artifactId&gt;
          &lt;version&gt;1.0.1&lt;/version&gt;
          &lt;!-- keep storm out of the jar-with-dependencies --&gt;
          &lt;!--&lt;scope&gt;provided&lt;/scope&gt;--&gt;
          &lt;exclusions&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
                  &lt;artifactId&gt;log4j-slf4j-impl&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
                  &lt;artifactId&gt;log4j-over-slf4j&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
                  &lt;artifactId&gt;slf4j-api&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;logback-classic&lt;/groupId&gt;
                  &lt;artifactId&gt;ch.qos.logback&lt;/artifactId&gt;
              &lt;/exclusion&gt;
          &lt;/exclusions&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.microsoft&lt;/groupId&gt;
          &lt;artifactId&gt;eventhubs&lt;/artifactId&gt;
          &lt;version&gt;1.0.2&lt;/version&gt;
          &lt;exclusions&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;org.apache.log4j&lt;/groupId&gt;
                  &lt;artifactId&gt;log4j&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;ch.qos.logback&lt;/groupId&gt;
                  &lt;artifactId&gt;logback-core&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;ch.qos.logback&lt;/groupId&gt;
                  &lt;artifactId&gt;logback-classic&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
                  &lt;artifactId&gt;log4j-over-slf4j&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;org.slf4j&lt;/groupId&gt;
                  &lt;artifactId&gt;slf4j-api&lt;/artifactId&gt;
              &lt;/exclusion&gt;
          &lt;/exclusions&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
          &lt;artifactId&gt;log4j-api&lt;/artifactId&gt;
          &lt;version&gt;2.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
          &lt;artifactId&gt;log4j-core&lt;/artifactId&gt;
          &lt;version&gt;2.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.commons&lt;/groupId&gt;
          &lt;artifactId&gt;commons-dbcp2&lt;/artifactId&gt;
          &lt;version&gt;2.1.1&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;com.microsoft.sqlserver&lt;/groupId&gt;
          &lt;artifactId&gt;mssql-jdbc&lt;/artifactId&gt;
          &lt;version&gt;6.2.2.jre8&lt;/version&gt;
      &lt;/dependency&gt;
  &lt;/dependencies&gt;
  &lt;build&gt;
      &lt;resources&gt;
          &lt;resource&gt;
              &lt;directory&gt;src/main/resources&lt;/directory&gt;
              &lt;filtering&gt;false&lt;/filtering&gt;
              &lt;includes&gt;
                  &lt;include&gt;log4j2.xml&lt;/include&gt;
              &lt;/includes&gt;
          &lt;/resource&gt;
      &lt;/resources&gt;
  &lt;/build&gt;
  </code></pre>
  
  <p>ConnectionFactory:</p>
  
  <pre><code>    public class ConnectionFactory {
      private interface Singleton {
          ConnectionFactory INSTANCE = new ConnectionFactory();
      }
  
      private final BasicDataSource dataSource;
      private static Properties dbProperties;
  
      private static String URL;
      private static String DRIVER;
      private static String USERNAME;
      private static String PASSWORD;
  
      private ConnectionFactory() {
          this.dataSource = new BasicDataSource();
      }
  
      public static void SetConnectionProperties(Properties properties) {
          try {
              //This property is used by logger
              System.setProperty("hostName", InetAddress.getLocalHost().getHostName());
  
              //Get environment properties
              dbProperties = properties;
              URL = dbProperties.getProperty("jdbc.url");
              DRIVER = dbProperties.getProperty("jdbc.driverClassName");
              USERNAME = dbProperties.getProperty("jdbc.username");
              PASSWORD = dbProperties.getProperty("jdbc.password");
  
              //Set connection properties
              Singleton.INSTANCE.dataSource.setUrl(URL);
              Singleton.INSTANCE.dataSource.setDriverClassName(DRIVER);
              Singleton.INSTANCE.dataSource.setUsername(USERNAME);
              Singleton.INSTANCE.dataSource.setPassword(PASSWORD);
          } catch (IOException ex){
  
          }
      }
  
      public static Connection getConnection() throws SQLException {
          return Singleton.INSTANCE.dataSource.getConnection();
      }
  }
  </code></pre>
  
  <p>log4j2.xml:</p>
  
  <pre><code>    &lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;configuration monitorInterval="60"&gt;
      &lt;Appenders&gt;
          &lt;Console name="STDOUT" target="SYSTEM_OUT"&gt;
              &lt;PatternLayout pattern="%d{HH:mm:ss} [%t] %-5level %logger{36} - %msg%n"/&gt;
          &lt;/Console&gt;
          &lt;JDBC name="databaseAppender" tableName="NotificationLog"&gt;
              &lt;ConnectionFactory
                      class="ConnectionFactory" method="getConnection" /&gt;
              &lt;Column name="LocalTimestamp" pattern="%d{yyyy-MM-dd HH:mm:ss.SSS}" /&gt;
              &lt;Column name="UtcTimestamp" pattern="%d{yyyy-MM-dd HH:mm:ss.SSS}{GMT}" /&gt;
              &lt;Column name="Level" pattern="%p" /&gt;
              &lt;Column name="Message" pattern="%m" /&gt;
              &lt;Column name="MachineName" pattern="${hostName}" /&gt;
              &lt;Column name="Exception" pattern="%throwable{full}" /&gt;
          &lt;/JDBC&gt;
      &lt;/Appenders&gt;
      &lt;Loggers&gt;
          &lt;Root level="debug"&gt;
              &lt;AppenderRef ref="STDOUT"/&gt;
              &lt;AppenderRef ref="databaseAppender"/&gt;
          &lt;/Root&gt;
      &lt;/Loggers&gt;
  &lt;/configuration&gt;
  </code></pre>
  
  <p>bolt.java</p>
  
  <pre><code>import org.apache.logging.log4j.Logger;
  import org.apache.logging.log4j.LogManager;
  
  public class Bolt extends BaseRichBolt {
      private OutputCollector collector;
  
      private static final Logger LOG = LogManager.getLogger(Bolt.class);
  
      @SuppressWarnings("rawtypes")
      @Override
      public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {
          this.collector = collector;
      }
  
      @Override
      public void execute(Tuple input) {
          LOG.debug("Test debug");
          LOG.info("Test Info");
          LOG.error("Test Error");
  
          collector.ack(input);
      }
      ...
  }
  </code></pre>
  
  <p>UPDATE (2/20/2018): Adding trace logging statements into the Main method of the jar, we were able to log successfully to the database.  This means the issue seems to be isolated to logging from the bolts.  Could this be an issue with serialization?  If it is, why would this work running the jar on local machine, and start having issues when deployed to the Storm Cluster?</p>
  
  <p>UPDATE (2/23/2018): Though we don't have a solution to this problem, we implemented a work around using the JDBC driver direcly with SQL prepared statements (e.g. made our own logging solution).  </p>
  
  <p>I think the answer to my most recent question has to do with the fact that on the Storm Cluster, the serialized components are being fed to/from the Zookeepers and for some reason Log4J isn't playing nicely.  However, with our own "Logging" solution, we are registering the factory for serialization prior to submitting the topology, and it seems to work (See code snippet).</p>
  
  <pre><code>....
  Config conf = new Config();
  conf.registerSerialization(LoggerFactory.class);
  StormSubmitter.submitTopology(topologyName, conf, builder.createTopology());
  </code></pre>
  
  <p>All of this is just conjecture though, and I would really like to better understand the interaction between the topology, bolts, and injected libraries before I would feel comfortable providing an accepted answer to this particular question.</p>