title: Is it possible to specify the pickle protocol when writing pandas to HDF5?
body: <p>Is there a way to tell Pandas to use a specific pickle protocol (e.g. 4) <strong>when writing an HDF5 file</strong>?</p>
  
  <p>Here is the situation (much simplified):</p>
  
  <ul>
  <li><p>Client A is using <code>python=3.8.1</code> (as well as <code>pandas=1.0.0</code> and <code>pytables=3.6.1</code>). A writes some DataFrame using <code>df.to_hdf(file, key)</code>.</p></li>
  <li><p>Client B is using <code>python=3.7.1</code> (and, as it happened, <code>pandas=0.25.1</code> and <code>pytables=3.5.2</code> --but that's irrelevant). B tries to read the data written by A using <code>pd.read_hdf(file, key)</code>, and fails with <code>ValueError: unsupported pickle protocol: 5</code>.</p></li>
  </ul>
  
  <p>Mind you, this doesn't happen with a purely numerical DataFrame (e.g. <code>pd.DataFrame(np.random.normal(size=(10,10)))</code>. So here is a reproducible example:</p>
  
  <pre><code>(base) $ conda activate py38
  (py38) $ python
  Python 3.8.1 (default, Jan  8 2020, 22:29:32)
  [GCC 7.3.0] :: Anaconda, Inc. on linux
  Type "help", "copyright", "credits" or "license" for more information.
  &gt;&gt;&gt; import pandas as pd
  &gt;&gt;&gt; df = pd.DataFrame(['hello', 'world']))
  &gt;&gt;&gt; df.to_hdf('foo', 'x')
  &gt;&gt;&gt; exit()
  (py38) $ conda deactivate
  (base) $ python
  Python 3.7.4 (default, Aug 13 2019, 20:35:49)
  [GCC 7.3.0] :: Anaconda, Inc. on linux
  Type "help", "copyright", "credits" or "license" for more information.
  &gt;&gt;&gt; import pandas as pd
  &gt;&gt;&gt; df = pd.read_hdf('foo', 'x')
  Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
    File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 407, in read_hdf
      return store.select(key, auto_close=auto_close, **kwargs)
    File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 782, in select
      return it.get_result()
    File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 1639, in get_result
      results = self.func(self.start, self.stop, where)
    File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 766, in func
      return s.read(start=_start, stop=_stop, where=_where, columns=columns)
    File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 3206, in read
      "block{idx}_values".format(idx=i), start=_start, stop=_stop
    File "/opt/anaconda3/lib/python3.7/site-packages/pandas/io/pytables.py", line 2737, in read_array
      ret = node[0][start:stop]
    File "/opt/anaconda3/lib/python3.7/site-packages/tables/vlarray.py", line 681, in __getitem__
      return self.read(start, stop, step)[0]
    File "/opt/anaconda3/lib/python3.7/site-packages/tables/vlarray.py", line 825, in read
      outlistarr = [atom.fromarray(arr) for arr in listarr]
    File "/opt/anaconda3/lib/python3.7/site-packages/tables/vlarray.py", line 825, in &lt;listcomp&gt;
      outlistarr = [atom.fromarray(arr) for arr in listarr]
    File "/opt/anaconda3/lib/python3.7/site-packages/tables/atom.py", line 1227, in fromarray
      return six.moves.cPickle.loads(array.tostring())
  ValueError: unsupported pickle protocol: 5
  &gt;&gt;&gt;
  </code></pre>
  
  <p>Note: I tried also reading using <code>pandas=1.0.0</code> (and <code>pytables=3.6.1</code>) in <code>python=3.7.4</code>. That fails too, so I believe it is simply the Python version (3.8 writer vs 3.7 reader) that causes the problem. This makes sense since pickle protocol 5 was introduced as <a href="https://www.python.org/dev/peps/pep-0574/" rel="noreferrer">PEP-574</a> for Python 3.8.</p>