title: GC problems when inserting large CSV file with Apache Camel
body: <p>I have a large CSV file containing about 5.2M lines. I want to parse this file and insert the data into a database. I am using apache camel for this. </p>
  
  <p>The route is fairly easy (simplified for this example)</p>
  
  <pre><code>from("file:work/customer/").id("customerDataRoute")
  .split(body().tokenize("\n")).streaming()
  .parallelProcessing()
  .unmarshal(bindyCustomer)
  .split(body())
  .process(new CustomerProcessor())
  .to("sql:INSERT INTO CUSTOMER_DATA(`FIELD1`,`FIELD2`) VALUES(#,#)");
  </code></pre>
  
  <p>bindyCustomer is a BindyCsvDataFormat for the CSV file and 
  CustomerProcessor is a Processor that returns the data of the Bindy Customer object as an array of objects for the SQL insert. The actual object has 39 fields (simplified above).</p>
  
  <p>This works all okay for the first 800.000 to 1.000.000 lines but then it comes to a halt. </p>
  
  <p>I have monitored the camel instance with JVisualVM and the Visual GC plugin and I can see that the old generation fills up and when it reaches maximum the whole system comes to a halt but it does not crash. 
  At this point the old generation is full, the Eden space is nearly full and both Survivor spaces are empty (as it can't move anything to the old generation I guess).</p>
  
  <p>So what is wrong here? This looks like a memory leak in the Camel SQL component to me. 
  The data is mainly stored in ConcurrentHashMap objects.</p>
  
  <p>When I take out the SQL component the old generation hardly fills at all.</p>
  
  <p>I am using Camel 2.15.1 Will try to use 2.17.1 to see whether that fixes the problem.</p>
  
  <p>Update: I have tried Camel 2.17.1 (same problem) and I have tried to insert the do the inserts in Java using java.sql.Statement.executeUPdate. With this option I managed to insert about 2.6 M rows but then it stopped as well. The funny thing is that I don't get a memory error. It just comes to a halt.</p>