title: Reducing memory churn when processing large data set
body: <p>Java has a tendency to create a large number objects that needs to be garbage collected when processing large data set.  This happens fairly frequently when streaming a amounts of data from the database, creating reports, etc.  Is there a strategy to reduce the memory churn.</p>
  
  <p>In this example, the object based version spends significant amount of times (2+ seconds) generating objects and performing garbage collection whereas the boolean array version completes in a fraction of a section without any garbages collection whatsoever.</p>
  
  <p>How do I reduce the memory churn (the need for large number of garbage collections) when processing large data sets? </p>
  
  <pre><code>java -verbose:gc -Xmx500M UniqChars
  ...
  ----------------
  [GC 495441K-&gt;444241K(505600K), 0.0019288 secs] x 45 times
  70000007
  ================
  70000007
  
  
  
  import java.util.HashSet;
  import java.util.Set;
  public class UniqChars {
      static String a=null;
      public static void main(String [] args) {
              //Generate data set
              StringBuffer sb=new StringBuffer("sfdisdf");
              for (int i =0; i&lt; 10000000; i++) {
                      sb.append("sfdisdf");
              }
              a=sb.toString();
              sb=null;  //free sb
              System.out.println("----------------");
              compareAsSet();
              System.out.println("================");
              compareAsAry();
      }
  
      public static void compareAsSet() {
              Set&lt;String&gt; uniqSet = new HashSet&lt;String&gt;();
              int n=0;
              for(int i=0; i&lt;a.length(); i++) {
                      String chr = a.substring(i,i);
                      uniqSet.add(chr);
                      n++;
              }
              System.out.println(n);
      }
  
      public static void compareAsAry() {
              boolean uniqSet[] = new boolean[65536];
              int n=0;
              for(int i=0; i&lt;a.length(); i++) {
                      int chr = (int) a.charAt(i);
                      uniqSet[chr]=true;
                      n++;
              }
              System.out.println(n);
      }
  }
  </code></pre>