title: Writing & Appending arrays of float to the only dataset in hdf5 file in C++
body: <p>I am processing number of files, each processing of the file will output several thousand of arrays of float and I will <strong>store the data of all files in one huge dataset in a single hdf5</strong> for further processing. </p>
  
  <p>The thing is currently I am confused about how to append my data into the hdf5 file. (comment in the code above) In 2 for loops above, as you can see, I want to append 1 dimensional array of float into hdf5 at a time, and not as the whole thing. My data is in terabytes, and we can only append the data into the file.</p>
  
  <p>There are several questions:</p>
  
  <ol>
  <li>How to append the data in this case? What kind of function must I use?</li>
  <li>Right now, I have fdim[0] = 928347543, I have tried put infinity flag of HDF5 in, but the runtime execution complains. Is there a way to do this? I don't want to calculate the data that I have each time; is there a way to just simply keep on adding data in, without caring the value of fdim?</li>
  </ol>
  
  <p>Or is this not possible?</p>
  
  <p>EDIT:</p>
  
  <p>I've been following Simon's suggestion, and currently here is my updated code:</p>
  
  <pre><code>hid_t desFi5;
  hid_t fid1;
  hid_t propList;
  hsize_t fdim[2];
  
  desFi5 = H5Fcreate(saveFilePath, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
  
  fdim[0] = 3;
  fdim[1] = 1;//H5S_UNLIMITED;
  
  fid1 = H5Screate_simple(2, fdim, NULL);
  
  cout &lt;&lt; "----------------------------------Space done\n";
  
  propList = H5Pcreate( H5P_DATASET_CREATE);
  
  H5Pset_layout( propList, H5D_CHUNKED );
  
  int ndims = 2;
  hsize_t chunk_dims[2];
  chunk_dims[0] = 3;
  chunk_dims[1] = 1;
  
  H5Pset_chunk( propList, ndims, chunk_dims );
  
  cout &lt;&lt; "----------------------------------Property done\n";
  
  hid_t dataset1 = H5Dcreate( desFi5, "des", H5T_NATIVE_FLOAT, fid1, H5P_DEFAULT, propList, H5P_DEFAULT);
  
  cout &lt;&lt; "----------------------------------Dataset done\n";
  
  bufi = new float*[1];
  bufi[0] = new float[3];
  bufi[0][0] = 0;
  bufi[0][1] = 1;
  bufi[0][2] = 2;
  
  //hyperslab
  hsize_t start[2] = {0,0};
  hsize_t stride[2] = {1,1};
  hsize_t count[2] = {1,1};
  hsize_t block[2] = {1,3};
  
  H5Sselect_hyperslab( fid1, H5S_SELECT_OR, start, stride, count, block);     
  cout &lt;&lt; "----------------------------------hyperslab done\n";   
  
  H5Dwrite(dataset1, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, *bufi);
  
  fdim[0] = 3;
  fdim[1] = H5S_UNLIMITED;    // COMPLAINS HERE
  H5Dset_extent( dataset1, fdim );
  
  cout &lt;&lt; "----------------------------------extent done\n";
  
  //hyperslab2
  hsize_t start2[2] = {1,0};
  hsize_t stride2[2] = {1,1};
  hsize_t count2[2] = {1,1};
  hsize_t block2[2] = {1,3};
  
  H5Sselect_hyperslab( fid1, H5S_SELECT_OR, start2, stride2, count2, block2);     
  cout &lt;&lt; "----------------------------------hyperslab2 done\n";  
  
  H5Dwrite(dataset1, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, *bufi);
  
  cout &lt;&lt; "----------------------------------H5Dwrite done\n";        
  H5Dclose(dataset1);
  cout &lt;&lt; "----------------------------------dataset closed\n";   
  H5Pclose( propList );   
  cout &lt;&lt; "----------------------------------property list closed\n"; 
  H5Sclose(fid1); 
  cout &lt;&lt; "----------------------------------dataspace fid1 closed\n";    
  H5Fclose(desFi5);       
  cout &lt;&lt; "----------------------------------desFi5 closed\n";    
  </code></pre>
  
  <p>My current output is: </p>
  
  <pre><code>bash-3.2$ ./hdf5AppendTest.out
  ----------------------------------Space done
  ----------------------------------Property done
  ----------------------------------Dataset done
  ----------------------------------hyperslab done
  HDF5-DIAG: Error detected in HDF5 (1.8.10) thread 0:
    #000: /home/hdftest/snapshots-bin-hdf5_1_8_10/current/src/H5D.c line 1103 in H5Dset_extent(): unable to set extend dataset
      major: Dataset
      minor: Unable to initialize object
    #001: /home/hdftest/snapshots-bin-hdf5_1_8_10/current/src/H5Dint.c line 2179 in H5D__set_extent(): unable to modify size of data space
      major: Dataset
      minor: Unable to initialize object
    #002: /home/hdftest/snapshots-bin-hdf5_1_8_10/current/src/H5S.c line 1874 in H5S_set_extent(): dimension cannot exceed the existing maximal size (new: 18446744073709551615 max: 1)
      major: Dataspace
      minor: Bad value
  ----------------------------------extent done
  ----------------------------------hyperslab2 done
  ----------------------------------H5Dwrite done
  ----------------------------------dataset closed
  ----------------------------------property list closed
  ----------------------------------dataspace fid1 closed
  ----------------------------------desFi5 closed
  </code></pre>
  
  <p>Currently, I see that setting things in unlimited with H5Dset_extent still causes a problem during runtime. (problematic function is marked with <code>//COMPLAINS HERE</code> in the code above.) I already got a chunk data as specified by Simon, so what's the problem here?</p>
  
  <p>On the other hand, without H5Dset_extent, I can write a test array of [0, 1, 2] just fine, but how can we make the code above the output the test array to the file like this:</p>
  
  <pre><code>[0, 1, 2]
  [0, 1, 2]
  [0, 1, 2]
  [0, 1, 2]
  ...
  ...
  </code></pre>
  
  <p>Recall: this is just a test array, the real data is bigger, and I cannot hold the whole thing in the RAM, so I must put data in part by part one at a time.</p>
  
  <p>EDIT 2:</p>
  
  <p>I've followed more of Simon's suggestion. Here is the critical part:</p>
  
  <pre><code>hsize_t n = 3, p = 1;
  float *bufi_data = new float[n * p];
  float ** bufi = new float*[n];
  for (hsize_t i = 0; i &lt; n; ++i){
      bufi[i] = &amp;bufi_data[i * n];
  }
  
  bufi[0][0] = 0.1;
  bufi[0][1] = 0.2;
  bufi[0][2] = 0.3;
  
  //hyperslab
  hsize_t start[2] = {0,0};
  hsize_t count[2] = {3,1};
  
  H5Sselect_hyperslab( fid1, H5S_SELECT_SET, start, NULL, count, NULL);
  cout &lt;&lt; "----------------------------------hyperslab done\n";   
  
  H5Dwrite(dataset1, H5T_NATIVE_FLOAT, H5S_ALL, fid1, H5P_DEFAULT, *bufi);
  
  bufi[0][0] = 0.4;
  bufi[0][1] = 0.5;
  bufi[0][2] = 0.6;
  
  hsize_t fdimNew[2];
  fdimNew[0] = 3;
  fdimNew[1] = 2;
  H5Dset_extent( dataset1, fdimNew );
  
  cout &lt;&lt; "----------------------------------extent done\n";
  
  //hyperslab2
  hsize_t start2[2] = {0,0}; //PROBLEM
  hsize_t count2[2] = {3,1};
  
  H5Sselect_hyperslab( fid1, H5S_SELECT_SET, start2, NULL, count2, NULL);     
  cout &lt;&lt; "----------------------------------hyperslab2 done\n";  
  
  H5Dwrite(dataset1, H5T_NATIVE_FLOAT, H5S_ALL, fid1, H5P_DEFAULT, *bufi);
  </code></pre>
  
  <p>From the above, I got the following output for hdf5:</p>
  
  <pre><code>0.4 0.5 0.6
    0   0   0
  </code></pre>
  
  <p>After further experiment with <code>start2</code> and <code>count2</code>, I see these variables only affect starting index and incrementing index for <code>bufi</code>. It does not move the position of the writing index of my dataset at all.</p>
  
  <p>Recall: the final result must be:</p>
  
  <pre><code>0.1 0.2 0.3
  0.4 0.5 0.6
  </code></pre>
  
  <p>Also, it must be <code>bufi</code> instead of <code>*bufi</code> for <code>H5Dwrite</code>, Simon, because <code>bufi</code> gives me completely random numbers.</p>
  
  <p>UPDATE 3:</p>
  
  <p>For the selection part suggested by Simon:</p>
  
  <pre><code>hsize_t start[2] = {0, 0};
  hsize_t count[2] = {1, 3};
  
  hsize_t start[2] = {1, 0};
  hsize_t count[2] = {1, 3};
  </code></pre>
  
  <p>These will give out the following error:</p>
  
  <pre><code>HDF5-DIAG: Error detected in HDF5 (1.8.10) thread 0:
    #000: /home/hdftest/snapshots-bin-hdf5_1_8_10/current/src/H5Dio.c line 245 in H5Dwrite(): file selection+offset not within extent
      major: Dataspace
      minor: Out of range
  </code></pre>
  
  <p><code>count[2]</code> should be <code>{3,1}</code>, rather than <code>{1,3}</code>, I suppose? And for <code>start[2]</code>, if I don't set it as <code>{0,0}</code>, it will always yell out the error above.</p>
  
  <p>Are you sure this is correct?</p>