class_docstr: Wraps a :class:`PipelineModulesGraph` model to train on using synchronous pipeline
  parallelism. If the model requires lots of memory and doesn't fit on a single GPU,
  pipeline parallelism is a useful technique to employ for training.
  
  The implementation is based on the torchgpipe_ paper.
  
  .. _torchgpipe: https://arxiv.org/abs/2004.09910
  
  PipelineModulesGraph combines pipeline parallelism with checkpointing to reduce peak
  memory required to train while minimizing device under-utilization.
  
  You should place all the modules on the appropriate rpc workers and devices and wrap
  them into an :class:`PipelineModulesGraph` module defining the connection between them.
  
  Args:
      module (:class:`PipelineModulesGraph`):
      model to be parallelized using pipelining. Each module
          in the graph has to have all of its parameters on a single
          device.
      chunks (int):
          number of micro-batches (default: ``1``)
      checkpoint (str):
          when to enable checkpointing, one of ``'always'``,
          ``'except_last'``, or ``'never'`` (default: ``'except_last'``).
          ``'never'`` disables checkpointing completely, ``'except_last'``
          enables checkpointing for all micro-batches except the last one
          and ``'always'`` enables checkpointing for all micro-batches.
class_docstr_tok: ["Wraps","a",":","class",":","`","PipelineModulesGraph","`","model","to","train","on","using","synchronous","pipeline","parallelism",".","If","the","model","requires","lots","of","memory","and","does","n't","fit","on","a","single","GPU",",","pipeline","parallelism","is","a","useful","technique","to","employ","for","training",".","The","implementation","is","based","on","the","torchgpipe_","paper",".","..","_torchgpipe",":","https",":","\/\/arxiv.org\/abs\/2004.09910","PipelineModulesGraph","combines","pipeline","parallelism","with","checkpointing","to","reduce","peak","memory","required","to","train","while","minimizing","device","under-utilization",".","You","should","place","all","the","modules","on","the","appropriate","rpc","workers","and","devices","and","wrap","them","into","an",":","class",":","`","PipelineModulesGraph","`","module","defining","the","connection","between","them",".","Args",":","module","(",":","class",":","`","PipelineModulesGraph","`",")",":","model","to","be","parallelized","using","pipelining",".","Each","module","in","the","graph","has","to","have","all","of","its","parameters","on","a","single","device",".","chunks","(","int",")",":","number","of","micro-batches","(","default",":","``","1","``",")","checkpoint","(","str",")",":","when","to","enable","checkpointing",",","one","of","``","'always","'","``",",","``","'except_last","'","``",",","or","``","'never","'","``","(","default",":","``","'except_last","'","``",")",".","``","'never","'","``","disables","checkpointing","completely",",","``","'except_last","'","``","enables","checkpointing","for","all","micro-batches","except","the","last","one","and","``","'always","'","``","enables","checkpointing","for","all","micro-batches","."]
code: def cpu(self) -> "DistributedPipeline":
      raise MOVING_DENIED
  
file_path: fairscale/fairscale/experimental/nn/distributed_pipeline/pipeline.py
imports: from dataclasses import dataclass
  from typing import Any, Dict, List, Optional, Union
  import torch
  from torch import Tensor, nn
  from torch.distributed import rpc
  from fairscale.internal import torch_version
  from fairscale.nn.pipe import microbatch
  from .data import DataConsumer
  from .graph import Node, PipelineModulesGraph
  from .partition_handler import DistributedPipelineRecord, PartitionHandler
label_desc: Use image node_id 3 for calling the DistributedPipeline obj's underlying member method code with example usage: obj.cpu() without return types