title: Recover data from corrupted file
body: <p>I have a HDF5 file that for some reason got corrupted. I am trying to retrieve the portion of the file that is essentially fine. I can read all datasets from the groups that do not contain a corrupted field just fine. But I cannot read any of the not-corrupted datasets from a group that has also a corrupted dataset.  </p>
  
  <p>The funny thing is however that I <strong>can easily read</strong> those datasets using HDFView. I.e. I can open them, and find all numerical values. Using HDFView I can only not read the corrupted dataset.</p>
  
  <p>My question is how can I exploit this, and retrieve as much data as I can?</p>
  
  <p>When reading with h5py:</p>
  
  <pre><code>Traceback (most recent call last):
    File "repair.py", line 44, in &lt;module&gt;
      print(data['/dt_yield/000000'][...])
    File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
    File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
    File "/usr/local/lib/python3.6/site-packages/h5py/_hl/group.py", line 167, in __getitem__
      oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
    File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
    File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
    File "h5py/h5o.pyx", line 190, in h5py.h5o.open
  KeyError: 'Unable to open object (bad heap free list)'
  </code></pre>
  
  <p>When reading with C++:</p>
  
  <pre><code>HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
    #000: H5D.c line 294 in H5Dopen2(): unable to open dataset
      major: Dataset
      minor: Can't open object
    #001: H5Dint.c line 1362 in H5D__open_name(): not found
      major: Dataset
      minor: Object not found
    #002: H5Gloc.c line 428 in H5G_loc_find(): can't find object
      major: Symbol table
      minor: Object not found
    #003: H5Gtraverse.c line 867 in H5G_traverse(): internal path traversal failed
      major: Symbol table
      minor: Object not found
    #004: H5Gtraverse.c line 594 in H5G_traverse_real(): can't look up component
      major: Symbol table
      minor: Object not found
    #005: H5Gobj.c line 1156 in H5G__obj_lookup(): can't locate object
      major: Symbol table
      minor: Object not found
    #006: H5Gstab.c line 890 in H5G__stab_lookup(): unable to protect symbol table heap
      major: Symbol table
      minor: Protected metadata error
    #007: H5HL.c line 364 in H5HL_protect(): unable to load heap data block
      major: Heap
      minor: Unable to protect metadata
    #008: H5AC.c line 1763 in H5AC_protect(): H5C_protect() failed
      major: Object cache
      minor: Unable to protect metadata
    #009: H5C.c line 2561 in H5C_protect(): can't load entry
      major: Object cache
      minor: Unable to load metadata into cache
    #010: H5C.c line 6877 in H5C_load_entry(): Can't deserialize image
      major: Object cache
      minor: Unable to load metadata into cache
    #011: H5HLcache.c line 763 in H5HL__cache_datablock_deserialize(): can't initialize free list
      major: Heap
      minor: Unable to initialize object
    #012: H5HLcache.c line 241 in H5HL__fl_deserialize(): bad heap free list
      major: Heap
      minor: Out of range
  libc++abi.dylib: terminating with uncaught exception of type H5::FileIException
  </code></pre>
  
  <h1>Recovery script (using h5py)</h1>
  
  <p>This is the implementation I have so far to recover at least everything that h5py can read directly:</p>
  
  <pre><code>import numpy as np
  import h5py, os, time
  
  def getdatasets(key,archive):
  
    if key[-1] != '/': key += '/'
  
    out = []
  
    for name in archive[key]:
  
      path = key + name
  
      if isinstance(archive[path], h5py.Dataset):
        out += [path]
      else:
        try   : out += getdatasets(path,archive)
        except: pass
  
    return out
  
  data  = h5py.File('data.hdf5' ,'r')
  fixed = h5py.File('fixed.hdf5','w')
  
  datasets = getdatasets('/',data)
  
  groups = list(set([i[::-1].split('/',1)[1][::-1] for i in datasets]))
  groups = [i for i in groups if len(i)&gt;0]
  
  idx    = np.argsort(np.array([len(i.split('/')) for i in groups]))
  groups = [groups[i] for i in idx]
  
  for group in groups:
    fixed.create_group(group)
  
  for path in datasets:
  
    # - check path
    if path not in data: continue
  
    # - try reading
    try   : data[path]
    except: continue
  
    # - get group name
    group = path[::-1].split('/',1)[1][::-1]
  
    # - minimum group name
    if len(group) == 0: group = '/'
  
    # - copy data
    data.copy(path, fixed[group])
  </code></pre>