title: Data structure to store huge amount of data?
body: <p>In my application,I have to load volumedata from set of images (MRC images) and keep the pixel data in memory.(images are grayscaled ,so one byte per pixel).</p>
  
  <p>My development environment is QT framework ,MinGW for Windows and GCC for Linux.</p>
  
  <p>At the moment,I use a simple data structure to store volumedata as :</p>
  
  <pre><code>unsigned char *volumeData;
  </code></pre>
  
  <p>and do one huge allocation as follows.</p>
  
  <pre><code>volumeData=new unsigned char[imageXsize * imageYsize * numofImages];
  </code></pre>
  
  <p>Following are the important methods to access image-data in a given plane,such as</p>
  
  <pre><code>unsigned char* getXYPlaneSlice(int z_value);
  unsigned char* getYZPlaneSlice(int x_value);
  unsigned char* getZXPlaneSlice(int y_value);
  </code></pre>
  
  <p>With my simple data-structure it was easy to implement above methods.</p>
  
  <p>But we might need to adopt to volume size as 2000x2000x1000 (~3.7Gb) in the future.And current datastructure will not be able to handle that huge data.</p>
  
  <ol>
  <li><p>How to avoid fragmentation ? Now,even with 1000x1000x200 data, application crash giving bad_alloc.
  Whats the best way to change the datastructure for this ? shall I use something like linked-list which each chunk is of size 100mb.</p></li>
  <li><p>Also,user should be able to perfome some image-processing filters on volume-data and also should be able to reset to original pixel value.
  That means, I should keep two copies of volume-data. 
  With current implemetation its like.</p>
  
  <p>unsigned char *volumeDataOriginal;</p>
  
  <p>unsigned char *volumeDataCurrent;</p></li>
  </ol>
  
  <p>So with 2000x2000x1000 data-range its going to utilize about 8Gb (4Gb for each volume).
  But in Win32 , the address space is 4GB.How to tackle this ? I should go with 64bit application ?</p>
  
  <p>EDIT :
  Here is a snapshot of my application  <img src="https://i.stack.imgur.com/wB3Q8.png" alt="enter image description here"></p>
  
  <p>Basically,I load the volume-data (from set of images,from MRC format..etc) and display them in different plane-viewers (XY,YX,YZ.Image shows XY-plane-viewer).I need to keep above 3 data-access methods to show an image in a particular plane.using slider-bar user can change which image to show in the selected plane)</p>
  
  <p>Thanks in advance.</p>