title: Collection Framework ,Big data and best Practice
body: <p>I have following class </p>
  
  <pre><code>public class BdFileContent {
      String filecontent;
  }
  </code></pre>
  
  <p>E.g file1.txt has following content:</p>
  
  <pre><code>This is test
  </code></pre>
  
  <ul>
  <li>"This" represents single instance of file content object.</li>
  <li>"is" represents another file content object</li>
  <li>"test" represents another file content object</li>
  </ul>
  
  <p>Suppose following is folder structure:</p>
  
  <pre><code>lineage
  |
  +-folder1
  |    |
  |    +-file1.txt
  |    +-file2.txt
  |
  +-folder2
  |    |
  |    +-file3.txt
  |    +-file4.txt
  +-...
  |
  +-...+-fileN.txt
  </code></pre>
  
  <p>.
  .
  .
  .</p>
  
  <p>N </p>
  
  <p><strong>N>1000 files</strong><br>
  N value will be very huge value </p>
  
  <p>BdFileContent class  represents each string in file in directory.</p>
  
  <p>I have to do lots of data manipulation and need to create a work on complex  data structure .I have to perform computation on both in memory and in disk .</p>
  
  <pre><code>ArrayList&lt;ArrayList&lt;ArrayList&lt;BdFileContent&gt;&gt;&gt; filecontentallFolderFileAsSingleStringToken = new ArrayList&lt;&gt;(); 
  </code></pre>
  
  <p>For example Above object represents all file contents of directory. I have to add this object for tree node  in BdTree .</p>
  
  <p>I am writing my own tree and adding 
  filecontentallFolderFileAsSingleStringToken as node .</p>
  
  <p>In What extend  collection framework data structure is appropriate for huge data.</p>
  
  <p>At this point i want to get some insight of how big company uses data structure to manipulate huge set of data generated every day.</p>
  
  <p><strong>Are they using collection framework?</strong></p>
  
  <p><strong>Do they use there own custom data structure ?</strong></p>
  
  <p><strong>Are they using  multi node data structure with each node running on separate JVM?</strong></p>
  
  <p>Till now collection object  runs on single jvm and can not dynamically use another jvm when there is signal for overflow flow in memory  and lack resource for processing </p>
  
  <p><strong>Normally what other developer approach for data structure for big data ?</strong></p>
  
  <p><strong>How other developer are handling it ?</strong></p>
  
  <p>I want to get some hints for real uses cases and experience.</p>