title: String.split() temporary objects and Garbage Collect
body: <p>In my project, we have a requirement to read a very large file, where each line has identifiers separated by a special character ( "|"). Unfortunately I can't use parallelism, since it is necessary to make a validation between the last character of a line with the first of the next line, to decide whether it will or not be extracted. Anyway, the requirement is very simple: break the line into tokens, analyze them and store only some of them in memory. The code is very simple, something like below:</p>
  
  <pre><code>final LineIterator iterator = FileUtils.lineIterator(file)
  while(iterator.hasNext()){
     final String[] tokens = iterator.nextLine().split("\\|");
     //process
  }
  </code></pre>
  
  <p>But this little piece of code is very, very inefficient. The method split() generates too many temporary objects that are not been collected (as best explained here: <a href="http://chrononsystems.com/blog/hidden-evils-of-javas-stringsplit-and-stringr" rel="nofollow noreferrer">http://chrononsystems.com/blog/hidden-evils-of-javas-stringsplit-and-stringr</a>.</p>
  
  <p>For comparison purposes: a 5mb file was using around 35 mb memory at the end of file process.</p>
  
  <p>I tested some alternatives like:</p>
  
  <ul>
  <li>Using a pre compiled pattern (<a href="https://stackoverflow.com/questions/5965767/performance-of-stringtokenizer-class-vs-split-method-in-java">Performance of StringTokenizer class vs. split method in Java</a>)</li>
  <li>Use Guava's Splitter (<a href="https://stackoverflow.com/questions/11001330/java-split-string-performances">Java split String performances</a>)</li>
  <li>Optimize String storage (<a href="http://java-performance.info/string-packing-converting-characters-to-bytes/" rel="nofollow noreferrer">http://java-performance.info/string-packing-converting-characters-to-bytes/</a>)</li>
  <li>Use of optimized collections (<a href="http://blog.takipi.com/5-coding-hacks-to-reduce-gc-overhead" rel="nofollow noreferrer">http://blog.takipi.com/5-coding-hacks-to-reduce-gc-overhead</a>)</li>
  </ul>
  
  <p>But none of them appears to be efficient enough. Using JProfiler, I could see that the amount of memory used by temporary objects is too high (35 mb used, but only 15 mb is actually been used by valid objects).</p>
  
  <p>Then I decide make a simple test: after 50,000 lines read, explicit call to System.gc(). And then, at the end of process, the memory usage has decreased from 35 mb to 16mb. I tested many, many times, and always got the same result.</p>
  
  <p>I know invoke that invoke of System.gc () is a bad practice (as indicated in <a href="https://stackoverflow.com/questions/2414105/why-is-it-bad-practice-to-call-system-gc">Why is it bad practice to call System.gc()?</a>). But is there is any other alternative in a cenario where the split() method could be invoked millions of times?</p>
  
  <p>[UPDATE]
  I use a 5 mb file only for test purpose, but the system should process much larger files (500Mb ~ 1Gb) </p>