title: Splitting a large XML file with Apache Camel using split, stax, jaxb
body: <p>I have a large XML file (could be around million records) on a sftp server. I don't want to load the entire file in-memory. The intention is that my route picks up the file, splits it and use the stax builder to iterate over the elements and map it to a JAXB object and send it to a queue (or spring batch) in order to persist later to the database.</p>
  <p><strong>input.xml (only 2 records as example)</strong></p>
  <pre><code>&lt;data&gt;
      &lt;PRODUCTNUMBER&gt;
          &lt;PRODUCTNUMBER&gt;8D0201075E&lt;/PRODUCTNUMBER&gt;
          &lt;CURRGROSSPRICE&gt;427.90&lt;/CURRGROSSPRICE&gt;
          &lt;NEXTGROSSPRICE&gt;0.00&lt;/NEXTGROSSPRICE&gt;
          &lt;NEXTPRICEDATE&gt;1900-01-01 00:00:00&lt;/NEXTPRICEDATE&gt;
          &lt;PRODUCTNAME_FR&gt;Some description&lt;/PRODUCTNAME_FR&gt;
          &lt;PRODUCTNAME_NL&gt;Some description&lt;/PRODUCTNAME_NL&gt;
      &lt;/PRODUCTNUMBER&gt;
  
      &lt;PRODUCTNUMBER&gt;
          &lt;PRODUCTNUMBER&gt;99630211802&lt;/PRODUCTNUMBER&gt;
          &lt;CURRGROSSPRICE&gt;3.78&lt;/CURRGROSSPRICE&gt;
          &lt;NEXTGROSSPRICE&gt;0.00&lt;/NEXTGROSSPRICE&gt;
          &lt;NEXTPRICEDATE&gt;1900-01-01 00:00:00&lt;/NEXTPRICEDATE&gt;
          &lt;PRODUCTNAME_FR&gt;Some description&lt;/PRODUCTNAME_FR&gt;
      &lt;/PRODUCTNUMBER&gt;
  &lt;/data&gt;
  </code></pre>
  <p><strong>Camel route</strong></p>
  <pre><code>from(&quot;sftp:localhost:22/in&quot;)
     .split(stax(PartRecords.class)).streaming()
     .marshal().json(JsonLibrary.Jackson, true)
     .to(&quot;rabbitmq://rabbitmq:5672/myExchange?queue=partQueue&amp;routingKey=queue.part&quot;)
     .end();
  </code></pre>
  <p><strong>PartRecord.java</strong></p>
  <pre><code>@XmlRootElement(name = &quot;PRODUCTNUMBER&quot;)
  @XmlAccessorType(XmlAccessType.FIELD)
  @Getter
  @Setter
  @ToString
  public class PartRecord implements Serializable {
  
      @XmlElement(name = &quot;PRODUCTNUMBER&quot;)
      private String productNumber;
  
      @XmlElement(name = &quot;CURRGROSSPRICE&quot;)
      private BigDecimal currentPrice;
  
      @XmlElement(name = &quot;PRODUCTNAME_NL&quot;)
      private String partDescriptionNL;
  
      @XmlElement(name = &quot;PRODUCTNAME_FR&quot;)
      private String partDescriptionFR;
  
  }
  </code></pre>
  <p><strong>PartRecords.java</strong></p>
  <pre><code>@XmlRootElement(name = &quot;data&quot;)
  @XmlAccessorType(XmlAccessType.FIELD)
  @ToString
  public class PartRecords implements Serializable {
  
      @XmlElement(name = &quot;PRODUCTNUMBER&quot;)
      private List&lt;PartRecord&gt; partRecords;
  
      public List&lt;PartRecord&gt; getPartRecords() {
          if (partRecords == null) {
              partRecords = new ArrayList&lt;&gt;();
          }
          return partRecords;
      }
  
  }
  </code></pre>
  <p>The route is working fine and a message is put on the queue but instead of having 1 message per record, the entire file in json is put on the queue. I guess this is normal behavior so I need something extra. I don't know if it's good idea to have 1 message per record but I imagine having 1 message containing the entire file is also not performant.</p>
  <p><em><strong>Current behavior output</strong></em></p>
  <pre><code>{
    &quot;partRecords&quot; : [ {
      &quot;productNumber&quot; : &quot;8D0201075E&quot;,
      &quot;currentPrice&quot; : 427.90,
      &quot;partDescriptionNL&quot; : &quot;Some description&quot;,
      &quot;partDescriptionFR&quot; : &quot;Some description&quot;
    }, {
      &quot;productNumber&quot; : &quot;99630211802&quot;,
      &quot;currentPrice&quot; : 3.78,
      &quot;partDescriptionNL&quot; : null,
      &quot;partDescriptionFR&quot; : &quot;Some description&quot;
    }]
  }
  </code></pre>
  <p>What am I doing wrong? I'm using Spring Boot v2.5.4, Apache Camel v3.11.1. Thanks in advance.</p>