title: Processing huge data with spring batch partitioning
body: <p>I am implementing spring batch job for processing millions of records in a DB table using partition approach as follows -</p>
  
  <ol>
  <li><p>Fetch a unique partitioning codes from table in a partitioner and set the same in execution context.</p></li>
  <li><p>Create a chunk step with reader,processor and writer to process records based on particular partition code.</p></li>
  </ol>
  
  <p>Is this approach is proper or is there any better approach for situation like this? As some partition codes can have more number of records than others,so those with more records might take more time to process than the ones with less records.</p>
  
  <p>Is it possible to create partition/thread to process like thread1 process 1-1000,thread2 process 1001-2000 etc ?</p>
  
  <p>How do I control number of threads getting created as partition codes can be around 100, I would like to create only 20 thread and process in 5 iteration?</p>
  
  <p>What happens if one partition fails, will all processing stop and reverted back?</p>
  
  <p>Following are configurations -</p>
  
  <pre><code> &lt;bean id="MyPartitioner" class="com.MyPartitioner" /&gt;
   &lt;bean id="itemProcessor" class="com.MyProcessor" scope="step" /&gt;
   &lt;bean id="itemReader" class="org.springframework.batch.item.database.JdbcCursorItemReader" scope="step" &gt;
    &lt;property name="dataSource" ref="dataSource"/&gt;
    &lt;property name="sql" value="select * from mytable WHERE code = '#{stepExecutionContext[code]}' "/&gt;
    &lt;property name="rowMapper"&gt;
        &lt;bean class="com.MyRowMapper" scope="step"/&gt;
    &lt;/property&gt;
  &lt;/bean&gt;
  &lt;bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" &gt;
      &lt;property name="corePoolSize" value="20"/&gt;
      &lt;property name="maxPoolSize" value="20"/&gt;
      &lt;property name="allowCoreThreadTimeOut" value="true"/&gt;
  &lt;/bean&gt;
  
  &lt;batch:step id="Step1" xmlns="http://www.springframework.org/schema/batch"&gt;
      &lt;batch:tasklet transaction-manager="transactionManager"&gt;
          &lt;batch:chunk reader="itemReader"  processor="itemProcessor" writer="itemWriter" commit-interval="200"/&gt;
      &lt;/batch:tasklet&gt;
  &lt;/batch:step&gt;
  &lt;batch:job id="myjob"&gt;
      &lt;batch:step id="mystep"&gt;
          &lt;batch:partition step="Step1" partitioner="MyPartitioner"&gt;
              &lt;batch:handler grid-size="20" task-executor="taskExecutor"/&gt;
          &lt;/batch:partition&gt;
      &lt;/batch:step&gt;
  &lt;/batch:job&gt;
  </code></pre>
  
  <p>Partitioner -</p>
  
  <pre><code>public class MyPartitioner implements Partitioner{
  @Override
  public Map&lt;String, ExecutionContext&gt; partition(int gridSize)
  {
  Map&lt;String, ExecutionContext&gt; partitionMap = new HashMap&lt;String, ExecutionContext&gt;();
  List&lt;String&gt; codes = getCodes();
  
  for (String code : codes)
  {
      ExecutionContext context = new ExecutionContext();
      context.put("code", code);
      partitionMap.put(code, context);
  }
  return partitionMap;}}
  </code></pre>
  
  <p>Thanks</p>