class_docstr: ShardedTensor is an torch.Tensor subclass to represent Tensors that are sharded
  across multiple devices and multiple processes.
  
  ShardedTensor is initialized in an SPMD like fashion where each rank
  initializes the ShardedTensor. The ShardedTensor object on each rank
  then only stores the local shard for the Tensor and provides global
  metadata for all the shards.
  
  ShardedTensor doesn't provide any Tensor like operations but is a wrapper
  providing the Tensor representing the local shard and the global metadata.
  Using these, users can build their custom distributed._sharded computations
  on top of this primitive. The local shards are all initialized using the
  create_op specified by tensor_init_params.create_op, e.g., torch.ones, or
  torch.empty
  
  Args:
      sharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification
          describing how to shard the Tensor.
      size (int...): a sequence of integers defining the shape of the output
          tensor. Can be a variable number of arguments or a collection like a list or tuple.
  
  Keyword args:
      dtype (:class:`torch.dtype`, optional): the desired data type of returned tensor.
              Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`).
      layout (:class:`torch.layout`, optional): the desired layout of returned Tensor.
          Default: ``torch.strided``.
      requires_grad (bool, optional): If autograd should record operations on the
          returned tensor. Default: ``False``.
      pin_memory (bool, optional): If set, returned tensor would be allocated in
          the pinned memory. Works only for CPU tensors. Default: ``False``.
      memory_format (:class:`torch.memory_format`, optional): the desired memory format of
          returned Tensor. Default: ``torch.contiguous_format``.
      init_rrefs (bool, optional): Whether or not to initialize
          :class:`torch.distributed.rpc.RRef`s pointing to remote shards.
          Need to initialize the RPC Framework if specified as ``True``.
          Default: ``False``.
  
  .. note:: ShardedTensor uses collectives to do various operations, i.e. it
      uses all_gather to do cross rank validations. For NCCL-based process
      groups, internal tensor representations of objects must be moved to the
      GPU device before communication takes place. In this case, the device
      used is given by ``torch.cuda.current_device()`` and it is the user's
      responsibility to ensure that this is set so that each rank has an
      individual GPU, via ``torch.cuda.set_device()``
code: def __hash__(self):
      return id(self)
  
imports: from __future__ import annotations
  from dataclasses import dataclass
  from typing import Callable, Dict, List, Optional, Sequence, Tuple, cast
  import copy
  import warnings
  from functools import reduce
  import weakref
  import threading
  import torch
  import torch.distributed
  from torch.distributed import rpc
  from torch.distributed import distributed_c10d
  from torch.distributed._shard.metadata import ShardMetadata
  import torch.distributed._shard.sharding_spec
  from torch.distributed._shard.sharding_spec.api import _dispatch_custom_op, _has_custom_op
  from torch.distributed._shard.sharding_spec._internals import check_tensor, validate_non_overlapping_shards_metadata
  from torch.distributed._shard._utils import DEPRECATE_MSG
  from .metadata import TensorProperties, ShardedTensorMetadata
  from .shard import Shard
  from .reshard import reshuffle_local_shard, reshard_local_shard
  from .utils import _flatten_tensor_size, _parse_and_validate_remote_device, _validate_output_tensor_for_gather, build_metadata_from_local_shards, build_global_metadata
  from torch.distributed.remote_device import _remote_device
  from torch.utils import _pytree
  import operator
label_desc: Use image node_id 22 for calling the ShardedTensor obj's underlying member method code with example usage: obj.__hash__() and returns: id