class_docstr: This class represents a SHA1 checksum based storage dir for state_dict
  and tensors.
  
  This means the same content will not be stored multiple times, resulting
  in space savings. (a.k.a. de-duplication)
  
  To make things easier for the callers, this class accept input data
  as files, state_dict or tensors. This class always returns in-memory
  data, not on-disk files. This class doesn't really care or know the actually
  data types.
  
  A key issue is dealing with content deletion. We use a reference counting
  algorithm, which means the caller must have symmetrical add/remove calls
  for each object.
  
  We used to support children-parent dependency graph and ref counting, but
  it is flawed since a grand-child can have the same SHA1 as the grand-parent,
  resulting in a cycle. This means caller must compute which parent is safe
  to delete in a version tracking graph. The lesson here is that content
  addressibility and dependency graphs do not mix well.
  
  We support multicore compression for the data to be store on per-object basis.
  We use pgzip to do parallel compression/decompression on top of it to use all
  the cores.
  
  Args:
      path (Path):
          The path in which a SHA1_Store will be created.
      init (bool, optional):
          - If ``True``, a new SHA1_Store in the path if not already exists.
          - Default: False
      sha1_buf_size (int):
          Buffer size used for checksumming. Default: 100MB.
      tmp_dir (str):
          Dir for temporary files if input is an in-memory object or output data needs
          to be decompressed first.
      pgzip_threads (int, optional):
          Number of threads (cores) used in compression. Default: None to use all cores.
      pgzip_block_size (int):
          Per-thread block size for compression. Default: 10MB.
class_docstr_tok: ["This","class","represents","a","SHA1","checksum","based","storage","dir","for","state_dict","and","tensors",".","This","means","the","same","content","will","not","be","stored","multiple","times",",","resulting","in","space","savings",".","(","a.k.a",".","de-duplication",")","To","make","things","easier","for","the","callers",",","this","class","accept","input","data","as","files",",","state_dict","or","tensors",".","This","class","always","returns","in-memory","data",",","not","on-disk","files",".","This","class","does","n't","really","care","or","know","the","actually","data","types",".","A","key","issue","is","dealing","with","content","deletion",".","We","use","a","reference","counting","algorithm",",","which","means","the","caller","must","have","symmetrical","add\/remove","calls","for","each","object",".","We","used","to","support","children-parent","dependency","graph","and","ref","counting",",","but","it","is","flawed","since","a","grand-child","can","have","the","same","SHA1","as","the","grand-parent",",","resulting","in","a","cycle",".","This","means","caller","must","compute","which","parent","is","safe","to","delete","in","a","version","tracking","graph",".","The","lesson","here","is","that","content","addressibility","and","dependency","graphs","do","not","mix","well",".","We","support","multicore","compression","for","the","data","to","be","store","on","per-object","basis",".","We","use","pgzip","to","do","parallel","compression\/decompression","on","top","of","it","to","use","all","the","cores",".","Args",":","path","(","Path",")",":","The","path","in","which","a","SHA1_Store","will","be","created",".","init","(","bool",",","optional",")",":","-","If","``","True","``",",","a","new","SHA1_Store","in","the","path","if","not","already","exists",".","-","Default",":","False","sha1_buf_size","(","int",")",":","Buffer","size","used","for","checksumming",".","Default",":","100MB",".","tmp_dir","(","str",")",":","Dir","for","temporary","files","if","input","is","an","in-memory","object","or","output","data","needs","to","be","decompressed","first",".","pgzip_threads","(","int",",","optional",")",":","Number","of","threads","(","cores",")","used","in","compression",".","Default",":","None","to","use","all","cores",".","pgzip_block_size","(","int",")",":","Per-thread","block","size","for","compression",".","Default",":","10MB","."]
docstr: Adds a file/object to this store and the sha1 references accordingly.
  
  First, a sha1 hash is calculated. Utilizing the sha1 hash string, the actual file
  in <file_or_obj> is moved within the store and the reference file is updated.
  If the input is an object, it will be store in the self._tmp_dir and then moved.
  
  If compress is True, the stored file is also compressed, which is useful for tensors
  with a lot of zeros.
  
  We use pickle and numpy for saving, loading because it is more deterministic
  in terms of serialized bytes. They do lose info on device and dtype of
  tensors. Will handle those later.
  
  Args:
      file_or_obj (str or tensor or Dict):
          Path to the file to be added to the store or an in-memory object
          that can be handled by pickle. Note, OrderedDict is used when
          you call `state_dict()` on a nn.Module, and it is an instance
          of a Dict too. A model's state_dict can be a simple dict because
          it may contain both model state_dict and other non-tensor info.
      compress (bool, optional):
          Use gzip compression on this object or not.
          Default: True
      name (str, optional):
          Optional name for this object.
          Default: None
docstr_tok: ["Adds","a","file\/object","to","this","store","and","the","sha1","references","accordingly",".","First",",","a","sha1","hash","is","calculated",".","Utilizing","the","sha1","hash","string",",","the","actual","file","in","<","file_or_obj",">","is","moved","within","the","store","and","the","reference","file","is","updated",".","If","the","input","is","an","object",",","it","will","be","store","in","the","self._tmp_dir","and","then","moved",".","If","compress","is","True",",","the","stored","file","is","also","compressed",",","which","is","useful","for","tensors","with","a","lot","of","zeros",".","We","use","pickle","and","numpy","for","saving",",","loading","because","it","is","more","deterministic","in","terms","of","serialized","bytes",".","They","do","lose","info","on","device","and","dtype","of","tensors",".","Will","handle","those","later",".","Args",":","file_or_obj","(","str","or","tensor","or","Dict",")",":","Path","to","the","file","to","be","added","to","the","store","or","an","in-memory","object","that","can","be","handled","by","pickle",".","Note",",","OrderedDict","is","used","when","you","call","`","state_dict","(",")","`","on","a","nn.Module",",","and","it","is","an","instance","of","a","Dict","too",".","A","model","'s","state_dict","can","be","a","simple","dict","because","it","may","contain","both","model","state_dict","and","other","non-tensor","info",".","compress","(","bool",",","optional",")",":","Use","gzip","compression","on","this","object","or","not",".","Default",":","True","name","(","str",",","optional",")",":","Optional","name","for","this","object",".","Default",":","None"]
code: def add(
      self,
      file_or_obj: Union[Path, Tensor, Dict],
      compress: bool = True,
      name: str = None,
  ) -> str:
      """Adds a file/object to this store and the sha1 references accordingly.
  
      First, a sha1 hash is calculated. Utilizing the sha1 hash string, the actual file
      in <file_or_obj> is moved within the store and the reference file is updated.
      If the input is an object, it will be store in the self._tmp_dir and then moved.
  
      If compress is True, the stored file is also compressed, which is useful for tensors
      with a lot of zeros.
  
      We use pickle and numpy for saving, loading because it is more deterministic
      in terms of serialized bytes. They do lose info on device and dtype of
      tensors. Will handle those later.
  
      Args:
          file_or_obj (str or tensor or Dict):
              Path to the file to be added to the store or an in-memory object
              that can be handled by pickle. Note, OrderedDict is used when
              you call `state_dict()` on a nn.Module, and it is an instance
              of a Dict too. A model's state_dict can be a simple dict because
              it may contain both model state_dict and other non-tensor info.
          compress (bool, optional):
              Use gzip compression on this object or not.
              Default: True
          name (str, optional):
              Optional name for this object.
              Default: None
      """
      start = time.time()
      is_pickle_file = None
  
      # Use `isinstance` not `type() == Path` since pathlib returns OS specific
      # Path types, which inherit from the Path class.
      if isinstance(file_or_obj, (Path, str)):
          # Make sure it is a valid file.
          try:
              pickle.load(open(file_or_obj, "rb"))
              is_pickle_file = True
          except Exception as e:
              is_pickle_file = False
              pass
          file_path = Path(file_or_obj)
          remove_tmp = False
  
      if is_pickle_file is False:
          # Continue to support torch.save()'ed files too by loading it
          # in memory and the next if condition will pickle it.
          file_or_obj = torch.load(cast(Union[Path, str], file_or_obj))
  
      if isinstance(file_or_obj, (Tensor, Dict)):
          # Serialize the object into a tmp file.
          file_path = self._get_tmp_file_path()
          pickle.dump(to_np(file_or_obj), open(file_path, "wb"))
          remove_tmp = True
      else:
          assert False, f"incorrect input {type(file_or_obj)}"
  
      # Get SHA1 from the file.
      assert isinstance(file_path, Path), type(file_path)
      sha1_hash = self._get_sha1_hash(file_path)
  
      # Load json for many meta data operations below. Repeatedly loading
      # can be very slow.
      with self._json_ctx:
          # Add reference.
          ref_count = self._add_ref(sha1_hash, True, compress)
  
          if ref_count == 1:  # First time adding.
              # Create the file dir, if needed.
              repo_fdir = self._sha1_to_dir(sha1_hash)
              if not repo_fdir.exists():
                  try:
                      repo_fdir.mkdir(exist_ok=True, parents=True)
                  except FileExistsError as error:
                      sys.stderr.write(
                          f"An exception occured: {repr(error)}\n"
                      )
                      sys.exit(ExitCode.FILE_EXISTS_ERROR)
  
              # Transfer the file to the store.
              repo_fpath = repo_fdir.joinpath(sha1_hash)
              try:
                  if compress:
                      orig_size, comp_size = _copy_compressed(
                          file_path,
                          repo_fpath,
                          self._pgzip_threads,
                          self._pgzip_block_size,
                      )
                  else:
                      shutil.copy2(file_path, repo_fpath)
                      orig_size = comp_size = file_path.stat().st_size
              except BaseException as error:
                  # Something went wrong, perhaps out of space, or race condition due to lack of locking.
                  # TODO (Min): proper handle the error and recover when we learn more here.
                  sys.stderr.write(
                      f"An exception occured: {repr(error)}\n"
                  )
                  ref_count = self._add_ref(sha1_hash, False, compress)
  
          # Update the sizes for this entry.
          entry = _get_json_entry(self._json_dict[sha1_hash])
          assert (
              ref_count == 1
              or entry[ENTRY_OS_KEY] % (ref_count - 1) == 0
          ), f"incorrect size: {entry[ENTRY_OS_KEY]} and {ref_count}"
          o_diff = (
              orig_size
              if ref_count == 1
              else (entry[ENTRY_OS_KEY] // (ref_count - 1))
          )
          d_diff = orig_size if ref_count == 1 else 0
          c_diff = comp_size if ref_count == 1 else 0
          entry[ENTRY_OS_KEY] += o_diff
          entry[ENTRY_DS_KEY] += d_diff
          entry[ENTRY_CS_KEY] += c_diff
  
          # Update whole store's stats.
          self._json_dict[STORE_OS_KEY] += o_diff
          self._json_dict[STORE_DS_KEY] += d_diff
          self._json_dict[STORE_CS_KEY] += c_diff
  
          # Update the name list for this entry.
          if name:
              if name not in entry[ENTRY_NAMES_KEY].keys():
                  entry[ENTRY_NAMES_KEY][name] = 1
              else:
                  entry[ENTRY_NAMES_KEY][name] += 1
  
      # Clean up if needed.
      if remove_tmp:
          file_path.unlink()
  
      duration = time.time() - start
      if duration > 60:
          logging.warning(f"Add() is taking long: {duration}s")
      return sha1_hash
  
imports: import hashlib
  import json
  import logging
  import os
  from pathlib import Path
  import pickle
  import shutil
  import sys
  import tempfile
  import time
  from typing import Any, Dict, Optional, Tuple, Union, cast
  import pgzip
  import torch
  from torch import Tensor
  from fairscale.internal.containers import from_np, to_np
  from .utils import ExitCode
label_desc: Use image node_id 2 for calling the SHA1_Store obj's underlying member method code with example usage: obj.add(file_or_obj, compress, name) and returns: sha1_hash