title: How to repeat Job with Partitioner when data is dynamic with Spring Batch?
body: <p>I am trying to develop a batch process using Spring Batch + Spring Boot (Java config), but I have a problem doing so. I have a software that has a database and a Java API, and I read records from there. The batch process should retrieve all the documents which expiration date is less than a certain date, update the date, and save them again in the same database.</p>
  
  <p>My first approach was reading the records 100 by 100; so the ItemReader retrieve 100 records, I process them 1 by 1, and finally I write them again. In the reader, I put this code:</p>
  
  <pre><code>public class DocumentItemReader implements ItemReader&lt;Document&gt; {
  
      public List&lt;Document&gt; documents = new ArrayList&lt;&gt;();
  
      @Override
      public Document read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
  
          if(documents.isEmpty()) {
              getDocuments(); // This method retrieve 100 documents and store them in "documents" list.
              if(documents.isEmpty()) return null;
          }
  
          Document doc = documents.get(0);
          documents.remove(0);
          return doc;
      }
  }
  </code></pre>
  
  <p>So, with this code, the reader reads from the database until no records are found. When the "getDocuments()" method doesn't retrieve any documents, the List is empty and the reader returns null (so the Job finish). Everything worked fine here.</p>
  
  <p>However, the problem appears if I want to use several threads. In this case, I started using the Partitioner approach instead of Multi-threading. The reason of doing that is because I read from the same database, so if I repeat the full step with several threads, all of them will find the same records, and I cannot use pagination (see below).</p>
  
  <p>Another problem is that database records are updated dynamically, so I cannot use pagination. For example, let's suppose I have 200 records, and all of them are going to expire soon, so the process is going to retrieve them. Now imagine I retrieve 10 with one thread, and before anything else, that thread process one and update it in the same database. The next thread cannot retrieve from 11 to 20 records, as the first record is not going to appear in the search (as it has been processed, its date has been updated, and then it doesn't match the query).</p>
  
  <p>It is a little difficult to understand, and some things may sound strange, but in my project:</p>
  
  <ul>
  <li>I am forced to use the same database to read and write.</li>
  <li>I can have millions of documents, so I cannot read all the records at the same time. I need to read them 100 by 100, or 500 by 500.</li>
  <li>I need to use several threads.</li>
  <li>I cannot use pagination, as the query to the databse will retrieve different documents each time it is executed.</li>
  </ul>
  
  <p>So, after hours thinking, I think the unique possible solution is to repeat the job until the query retrives no documents. Is this possible? I want to do something like the step does: Do something until null is returned - repeat the job until the query return zero records.</p>
  
  <p>If this is not a good approach, I will appreciate other possible solutions.</p>
  
  <p>Thank you.</p>