title: How to load checkpoints across different versions of pytorch (1.3.1 and 1.6.x) using ppc64le and x86?
body: <p>As I outlined here I am stuck using old versions of pytorch and torchvision due to hardware e.g. using ppc64le IBM architectures.</p>
  <p>For this reason, I am having issues when sending and receiving checkpoints between different computers, clusters and my personal mac. I wonder if there is any way to load models in a way to avoid this issue? e.g. perhaps saving models in with a old and new format when using 1.6.x. Of course for the 1.3.1 to 1.6.x is impossible but at leat I was hoping something would work.</p>
  <p>Any advice? Of course my ideal solution is that I don't have to worry about it and I can always load and save my checkpoints and everything I usually pickle uniformly across all my hardware.</p>
  <hr />
  <p>The first error I got was a zip jit error:</p>
  <pre><code>RuntimeError: /home/miranda9/data/f.pt is a zip archive (did you mean to use torch.jit.load()?)
  </code></pre>
  <p>so I used that (and other pickle libraries):</p>
  <pre><code># %%
  import torch
  from pathlib import Path
  
  
  def load(path):
      import torch
      import pickle
      import dill
  
      path = str(path)
      try:
          db = torch.load(path)
          f = db['f']
      except Exception as e:
          db = torch.jit.load(path)
          f = db['f']
          #with open():
          # db = pickle.load(open(path, &quot;r+&quot;))
          # db = dill.load(open(path, &quot;r+&quot;))
          #raise ValueError(f'FAILED: {e}')
      return db, f
  
  p = &quot;~/data/f.pt&quot;
  path = Path(p).expanduser()
  
  db, f = load(path)
  
  Din, nb_examples = 1, 5
  x = torch.distributions.Normal(loc=0.0, scale=1.0).sample(sample_shape=(nb_examples, Din))
  
  y = f(x)
  
  print(y)
  print('Success!\a')
  </code></pre>
  <p>but I get complains of different pytorch versions which I am forced to use:</p>
  <pre><code>Traceback (most recent call last):
    File &quot;hal_pg.py&quot;, line 27, in &lt;module&gt;
      db, f = load(path)
    File &quot;hal_pg.py&quot;, line 16, in load
      db = torch.jit.load(path)
    File &quot;/home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/jit/__init__.py&quot;, line 239, in load
      cpp_module = torch._C.import_ir_module(cu, f, map_location, _extra_files)
  RuntimeError: version_number &lt;= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 1. Your PyTorch installation may be too old. (init at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131)
  frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0xbc (0x7fff7b527b9c in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libc10.so)
  frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1d98 (0x7fff1d293c78 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
  frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0x88 (0x7fff1d2950d8 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
  frame #3: torch::jit::import_ir_module(std::shared_ptr&lt;torch::jit::script::CompilationUnit&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, c10::optional&lt;c10::Device&gt;, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt; &gt; &gt;&amp;) + 0x64 (0x7fff1e624664 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
  frame #4: &lt;unknown function&gt; + 0x70e210 (0x7fff7c0ae210 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
  frame #5: &lt;unknown function&gt; + 0x28efc4 (0x7fff7bc2efc4 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
  &lt;omitting python frames&gt;
  frame #26: &lt;unknown function&gt; + 0x25280 (0x7fff84b35280 in /lib64/libc.so.6)
  frame #27: __libc_start_main + 0xc4 (0x7fff84b35474 in /lib64/libc.so.6)
  </code></pre>
  <p>any ideas how to make everything consistent across the clusters? I can't even open the pickle files.</p>
  <hr />
  <p>maybe this is just impossible with the current pytorch version I am forced to use :(</p>
  <pre><code>RuntimeError: version_number &lt;= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 1. Your PyTorch installation may be too old. (init at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131)
  frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0xbc (0x7fff83ba7b9c in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libc10.so)
  frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1d98 (0x7fff25993c78 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
  frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;) + 0x88 (0x7fff259950d8 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
  frame #3: torch::jit::import_ir_module(std::shared_ptr&lt;torch::jit::script::CompilationUnit&gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, c10::optional&lt;c10::Device&gt;, std::unordered_map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::hash&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::equal_to&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt; &gt; &gt;&amp;) + 0x64 (0x7fff26d24664 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
  frame #4: &lt;unknown function&gt; + 0x70e210 (0x7fff8472e210 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
  frame #5: &lt;unknown function&gt; + 0x28efc4 (0x7fff842aefc4 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
  &lt;omitting python frames&gt;
  frame #23: &lt;unknown function&gt; + 0x25280 (0x7fff8d335280 in /lib64/libc.so.6)
  frame #24: __libc_start_main + 0xc4 (0x7fff8d335474 in /lib64/libc.so.6)
  
  </code></pre>
  <p>using code:</p>
  <pre><code>from pathlib import Path
  
  import torch
  
  path = '/home/miranda9/data/dataset/'
  path = Path(path).expanduser() / 'fi_db.pt'
  path = str(path)
  
  # db = torch.load(path)
  # torch.jit.load(path)
  db = torch.jit.load(str(path))
  
  print(db)
  </code></pre>
  <hr />
  <p>related links:</p>
  <ul>
  <li><a href="https://stackoverflow.com/questions/64141188/how-to-load-checkpoints-across-different-versions-of-pytorch-1-3-1-and-1-6-x-u">How to load checkpoints across different versions of pytorch (1.3.1 and 1.6.x) using ppc64le and x86?</a></li>
  <li><a href="https://discuss.pytorch.org/t/how-to-load-checkpoints-across-different-versions-of-pytorch-1-3-1-and-1-6-x-using-ppc64le-and-x86/97829" rel="noreferrer">https://discuss.pytorch.org/t/how-to-load-checkpoints-across-different-versions-of-pytorch-1-3-1-and-1-6-x-using-ppc64le-and-x86/97829</a></li>
  <li>related gitissue: <a href="https://github.com/pytorch/pytorch/issues/43766" rel="noreferrer">https://github.com/pytorch/pytorch/issues/43766</a></li>
  <li>reddit: <a href="https://www.reddit.com/r/pytorch/comments/jvza7v/how_to_load_checkpoints_across_different_versions/" rel="noreferrer">https://www.reddit.com/r/pytorch/comments/jvza7v/how_to_load_checkpoints_across_different_versions/</a></li>
  </ul>