title: Spring Batch Step does not execute
body: <p>I'm trying to fix a problem in Spring Batch that has been plaguing our system recently. We have a job that, for the most part works just fine. It's a multi-step job that downloads and processes data.</p>
  
  <p>The problem is that sometimes the job will bomb out. Maybe the server we're trying to connect to throws an error or we shut down the server in the middle of the job. At this point the next time our quartz scheduler tries to run the job <strong>it doesn't seem to do anything</strong>. Below is an abridged version of this job definition:</p>
  
  <pre><code>&lt;batch:job id="job.download-stuff" restartable="true"&gt;
  &lt;batch:validator ref="downloadValidator"/&gt;
  &lt;batch:step id="job.download-stuff.download"&gt;
      &lt;batch:tasklet ref="salesChannelOrderDownloader" transaction-manager="transactionManager"&gt;
          &lt;batch:transaction-attributes isolation="READ_UNCOMMITTED" propagation="NOT_SUPPORTED"/&gt;
          &lt;batch:listeners&gt;
              &lt;batch:listener ref="downloadListener"/&gt;
              &lt;batch:listener ref="loggingContextStepListener" /&gt;
          &lt;/batch:listeners&gt;
      &lt;/batch:tasklet&gt;
      &lt;batch:next on="CONTINUE" to="job.download-stuff.process-stuff.step" /&gt;
      &lt;batch:end on="*" /&gt;
  &lt;/batch:step&gt;
  &lt;batch:step id="job.download-stuff.process-stuff.step"&gt;
      ...
  &lt;/batch:step&gt;
  &lt;batch:listeners&gt;
      &lt;batch:listener ref="loggingContextJobListener"/&gt;
  &lt;/batch:listeners&gt;
  </code></pre>
  
  <p></p>
  
  <p>Once it gets into this state, the <code>downloadValidator</code> runs, but it never makes it into the first step <code>download-stuff.download</code>. I set a breakpoint in the tasklet and it never makes it inside. </p>
  
  <p><strong>If I clear out all of the spring batch tables, which are stored in our mysql database, and restart the server it will begin working again, but I'd rather understand what prevents it from operating correctly at this point rather than employ scorched earth tactics to get the job running.</strong></p>
  
  <p>I'm a novice at Spring Batch, to put it mildly, so forgive me if I am omitting important details. I've set breakpoints and turned on logging to learn what I can.</p>
  
  <p>What I have observed so far from going through the database is that entries appear to no longer be written to the BATCH_STEP_EXECUTION and BATCH_JOB_EXECUTION tables.</p>
  
  <p>There are no BATCH_JOB_EXECUTION entries for the job that are not in COMPLETED status and no BATCH_STEP_EXECUTION entries that are not in COMPLETED</p>
  
  <p>You'll see that there is a batch:validator defined, I've confirmed that spring batch calls that validator and that it makes it through successfully (set a breakpoint and stepped through). The first step does not get executed.</p>
  
  <p>Neither the loggingContextJobListener nor the loggingContextStepListener seem to fire either. What could be causing this?</p>
  
  <p><strong>UPDATE</strong>
  I took a closer look at the <code>downloadListener</code> added as a batch:listener. Here's the source code of afterStep:</p>
  
  <pre><code>@Override
  @Transactional(propagation = Propagation.REQUIRES_NEW)
  public ExitStatus afterStep(StepExecution stepExecution) {
      long runSeconds = TimeUnit.NANOSECONDS.toSeconds(System.nanoTime() - nanoStart);
  
      // If Success - we're good
      if (stepExecution.getStatus() == BatchStatus.COMPLETED) {
          Long endTs = stepExecution.getExecutionContext().getLong("toTime");
          Date toTime = new Date(endTs);
          handleSuccess(toTime, stepExecution.getWriteCount());
          return null;
      }
  
      // Otherwise - record errors
      List&lt;Throwable&gt; failures = stepExecution.getFailureExceptions();
      handleError(failures);
      return ExitStatus.FAILED;
  }
  </code></pre>
  
  <p>I confirmed that the <code>return ExitStatus.FAILED</code> line executes and that the exception that was thrown is logged in the failureExceptions. It seems like once that happens the BATCH_JOB_EXECUTION entry is in COMPLETED status (and exit_code) and the STEP_EXECUTION is failed.</p>
  
  <p>At this point, the entries in the BATCH_JOB_EXECUTION_PARAMS table remain. I actually tried modifying the values of their KEY_NAME and value columns, but this still didn't allow the job to run. <strong>As long as there are parameters tied to a JOB_EXECUTION_ID, another job belonging to the same BATCH_JOB_INSTANCE cannot run.</strong></p>
  
  <p>Once I remove the entries in <strong>BATCH_JOB_EXECUTION_PARAMS</strong> for that specific JOB_EXECUTION_ID, another <strong>BATCH_JOB_EXECUTION</strong> can run, even though all the BATCH_JOB_EXECUTION entries are in a completed state.</p>
  
  <p>So I guess I have two questions- is that the correct behavior? And if so, what is preventing the BATCH_JOB_EXECUTION_PARAMS from being removed and how do I remove them?</p>