class_docstr: ``pytorch.Trainer`` is an abstraction on top of a vanilla PyTorch training loop that handles
  many training details under-the-hood, and exposes APIs for configuring training-related features
  such as automatic checkpointing, validation, profiling, metrics reporting, etc.
  
  ``Trainer`` must be initialized and called from within a ``pytorch.PyTorchTrialContext``.
class_docstr_tok: ["``","pytorch.Trainer","``","is","an","abstraction","on","top","of","a","vanilla","PyTorch","training","loop","that","handles","many","training","details","under-the-hood",",","and","exposes","APIs","for","configuring","training-related","features","such","as","automatic","checkpointing",",","validation",",","profiling",",","metrics","reporting",",","etc",".","``","Trainer","``","must","be","initialized","and","called","from","within","a","``","pytorch.PyTorchTrialContext","``","."]
docstr: ``fit()`` trains a ``PyTorchTrial`` configured from the ``Trainer`` and handles
  checkpointing and validation steps, and metrics reporting.
  
  Arguments:
      checkpoint_period: The number of steps to train for before checkpointing. This is
          a ``TrainUnit`` type (``Batch`` or ``Epoch``) which can take an ``int`` or
          instance of ``collections.abc.Container`` (list, tuple, etc.). For example,
          ``Batch(100)`` would checkpoint every 100 batches, while ``Batch([5, 30, 45])``
          would checkpoint after every 5th, 30th, and 45th batch.
      validation_period: The number of steps to train for before validating. This is a
          ``TrainUnit`` type (``Batch`` or ``Epoch``) which can take an ``int`` or instance
          of ``collections.abc.Container`` (list, tuple, etc.). For example, ``Batch(100)``
          would validate every 100 batches, while ``Batch([5, 30, 45])`` would validate
          after every 5th, 30th, and 45th batch.
      max_length: The maximum number of steps to train for. This value is required and
          only applicable in local training mode. For on-cluster training, this value will
          be ignored; the searcher’s ``max_length`` must be configured from the experiment
          configuration. This is a ``TrainUnit`` type (``Batch`` or ``Epoch``) which takes an
          ``int``. For example, ``Epoch(1)`` would train for a maximum length of one epoch.
      reporting_period: The number of steps to train for before reporting metrics and
          searcher progress. For local training mode, metrics are printed to stdout. This
          is a ``TrainUnit`` type (``Batch`` or ``Epoch``) which can take an ``int`` or
          instance of ``collections.abc.Container`` (list, tuple, etc.). For example,
          ``Batch(100)`` would report every 100 batches, while ``Batch([5, 30, 45])`` would
          report after every 5th, 30th, and 45th batch.
      checkpoint_policy: Controls how Determined performs checkpoints after validation
          operations, if at all. Should be set to one of the following values:
  
              best (default): A checkpoint will be taken after every validation operation
                  that performs better than all previous validations for this experiment.
                  Validation metrics are compared according to the ``metric`` and
                  ``smaller_is_better`` fields in the searcher configuration. This option
                  is only supported for on-cluster training.
              all: A checkpoint will be taken after every validation, no matter the
                  validation performance.
              none: A checkpoint will never be taken due to a validation. However,
                  even with this policy selected, checkpoints are still expected to be taken
                  after the trial is finished training, due to cluster scheduling decisions,
                  before search method decisions, or due to ``min_checkpoint_period``.
      latest_checkpoint: Configures the checkpoint used to start or continue training.
          This value should be set to ``det.get_cluster_info().latest_checkpoint`` for
          standard continue training functionality.
      step_zero_validation: Configures whether to perform an initial validation before
          training. Defaults to false.
      test_mode: Runs a minimal loop of training for testing and debugging purposes. Will
          train and validate one batch. Defaults to false.
docstr_tok: ["``","fit","(",")","``","trains","a","``","PyTorchTrial","``","configured","from","the","``","Trainer","``","and","handles","checkpointing","and","validation","steps",",","and","metrics","reporting",".","Arguments",":","checkpoint_period",":","The","number","of","steps","to","train","for","before","checkpointing",".","This","is","a","``","TrainUnit","``","type","(","``","Batch","``","or","``","Epoch","``",")","which","can","take","an","``","int","``","or","instance","of","``","collections.abc.Container","``","(","list",",","tuple",",","etc.",")",".","For","example",",","``","Batch","(","100",")","``","would","checkpoint","every","100","batches",",","while","``","Batch","(","[","5",",","30",",","45","]",")","``","would","checkpoint","after","every","5th",",","30th",",","and","45th","batch",".","validation_period",":","The","number","of","steps","to","train","for","before","validating",".","This","is","a","``","TrainUnit","``","type","(","``","Batch","``","or","``","Epoch","``",")","which","can","take","an","``","int","``","or","instance","of","``","collections.abc.Container","``","(","list",",","tuple",",","etc.",")",".","For","example",",","``","Batch","(","100",")","``","would","validate","every","100","batches",",","while","``","Batch","(","[","5",",","30",",","45","]",")","``","would","validate","after","every","5th",",","30th",",","and","45th","batch",".","max_length",":","The","maximum","number","of","steps","to","train","for",".","This","value","is","required","and","only","applicable","in","local","training","mode",".","For","on-cluster","training",",","this","value","will","be","ignored",";","the","searcher","\u2019","s","``","max_length","``","must","be","configured","from","the","experiment","configuration",".","This","is","a","``","TrainUnit","``","type","(","``","Batch","``","or","``","Epoch","``",")","which","takes","an","``","int","``",".","For","example",",","``","Epoch","(","1",")","``","would","train","for","a","maximum","length","of","one","epoch",".","reporting_period",":","The","number","of","steps","to","train","for","before","reporting","metrics","and","searcher","progress",".","For","local","training","mode",",","metrics","are","printed","to","stdout",".","This","is","a","``","TrainUnit","``","type","(","``","Batch","``","or","``","Epoch","``",")","which","can","take","an","``","int","``","or","instance","of","``","collections.abc.Container","``","(","list",",","tuple",",","etc.",")",".","For","example",",","``","Batch","(","100",")","``","would","report","every","100","batches",",","while","``","Batch","(","[","5",",","30",",","45","]",")","``","would","report","after","every","5th",",","30th",",","and","45th","batch",".","checkpoint_policy",":","Controls","how","Determined","performs","checkpoints","after","validation","operations",",","if","at","all",".","Should","be","set","to","one","of","the","following","values",":","best","(","default",")",":","A","checkpoint","will","be","taken","after","every","validation","operation","that","performs","better","than","all","previous","validations","for","this","experiment",".","Validation","metrics","are","compared","according","to","the","``","metric","``","and","``","smaller_is_better","``","fields","in","the","searcher","configuration",".","This","option","is","only","supported","for","on-cluster","training",".","all",":","A","checkpoint","will","be","taken","after","every","validation",",","no","matter","the","validation","performance",".","none",":","A","checkpoint","will","never","be","taken","due","to","a","validation",".","However",",","even","with","this","policy","selected",",","checkpoints","are","still","expected","to","be","taken","after","the","trial","is","finished","training",",","due","to","cluster","scheduling","decisions",",","before","search","method","decisions",",","or","due","to","``","min_checkpoint_period","``",".","latest_checkpoint",":","Configures","the","checkpoint","used","to","start","or","continue","training",".","This","value","should","be","set","to","``","det.get_cluster_info","(",")",".latest_checkpoint","``","for","standard","continue","training","functionality",".","step_zero_validation",":","Configures","whether","to","perform","an","initial","validation","before","training",".","Defaults","to","false",".","test_mode",":","Runs","a","minimal","loop","of","training","for","testing","and","debugging","purposes",".","Will","train","and","validate","one","batch",".","Defaults","to","false","."]
code: def fit(
      self,
      checkpoint_period: Optional[pytorch.TrainUnit] = None,
      validation_period: Optional[pytorch.TrainUnit] = None,
      max_length: Optional[pytorch.TrainUnit] = None,
      reporting_period: pytorch.TrainUnit = pytorch.Batch(
          100
      ),  # noqa: B008
      checkpoint_policy: str = "best",
      latest_checkpoint: Optional[str] = None,
      step_zero_validation: bool = False,
      test_mode: bool = False,
  ) -> None:
      """
      ``fit()`` trains a ``PyTorchTrial`` configured from the ``Trainer`` and handles
      checkpointing and validation steps, and metrics reporting.
  
      Arguments:
          checkpoint_period: The number of steps to train for before checkpointing. This is
              a ``TrainUnit`` type (``Batch`` or ``Epoch``) which can take an ``int`` or
              instance of ``collections.abc.Container`` (list, tuple, etc.). For example,
              ``Batch(100)`` would checkpoint every 100 batches, while ``Batch([5, 30, 45])``
              would checkpoint after every 5th, 30th, and 45th batch.
          validation_period: The number of steps to train for before validating. This is a
              ``TrainUnit`` type (``Batch`` or ``Epoch``) which can take an ``int`` or instance
              of ``collections.abc.Container`` (list, tuple, etc.). For example, ``Batch(100)``
              would validate every 100 batches, while ``Batch([5, 30, 45])`` would validate
              after every 5th, 30th, and 45th batch.
          max_length: The maximum number of steps to train for. This value is required and
              only applicable in local training mode. For on-cluster training, this value will
              be ignored; the searcher’s ``max_length`` must be configured from the experiment
              configuration. This is a ``TrainUnit`` type (``Batch`` or ``Epoch``) which takes an
              ``int``. For example, ``Epoch(1)`` would train for a maximum length of one epoch.
          reporting_period: The number of steps to train for before reporting metrics and
              searcher progress. For local training mode, metrics are printed to stdout. This
              is a ``TrainUnit`` type (``Batch`` or ``Epoch``) which can take an ``int`` or
              instance of ``collections.abc.Container`` (list, tuple, etc.). For example,
              ``Batch(100)`` would report every 100 batches, while ``Batch([5, 30, 45])`` would
              report after every 5th, 30th, and 45th batch.
          checkpoint_policy: Controls how Determined performs checkpoints after validation
              operations, if at all. Should be set to one of the following values:
  
                  best (default): A checkpoint will be taken after every validation operation
                      that performs better than all previous validations for this experiment.
                      Validation metrics are compared according to the ``metric`` and
                      ``smaller_is_better`` fields in the searcher configuration. This option
                      is only supported for on-cluster training.
                  all: A checkpoint will be taken after every validation, no matter the
                      validation performance.
                  none: A checkpoint will never be taken due to a validation. However,
                      even with this policy selected, checkpoints are still expected to be taken
                      after the trial is finished training, due to cluster scheduling decisions,
                      before search method decisions, or due to ``min_checkpoint_period``.
          latest_checkpoint: Configures the checkpoint used to start or continue training.
              This value should be set to ``det.get_cluster_info().latest_checkpoint`` for
              standard continue training functionality.
          step_zero_validation: Configures whether to perform an initial validation before
              training. Defaults to false.
          test_mode: Runs a minimal loop of training for testing and debugging purposes. Will
              train and validate one batch. Defaults to false.
      """
      # Set defaults.
      if checkpoint_period is None:
          checkpoint_period = pytorch.Batch(sys.maxsize)
  
      if validation_period is None:
          validation_period = pytorch.Batch(sys.maxsize)
  
      if self._local_training:
          if checkpoint_policy == "best":
              logger.warning(
                  "checkpoint_policy='best' is not supported in local training mode. "
                  "Falling back to 'all'."
              )
              checkpoint_policy = "all"
          if max_length is None:
              raise ValueError(
                  "max_length must be defined in local training mode."
              )
  
          if not isinstance(max_length.value, int):
              raise TypeError(
                  "max_length must be configured in TrainUnit(int) types."
              )
  
          if self._det_profiler:
              logger.warning(
                  "Determined profiler will be ignored in local training mode."
              )
  
          smaller_is_better = True
          searcher_metric_name = None
          steps_completed = 0
          global_batch_size = None
      else:
          if test_mode:
              raise ValueError(
                  "test_mode is only supported in local training mode."
              )
  
          if max_length is not None:
              logger.warning(
                  "max_length is ignored when training on-cluster. Please configure the "
                  "searcher instead."
              )
  
          assert self._info, "Unable to detect cluster info."
          if (
              latest_checkpoint is None
              and self._info.latest_checkpoint is not None
          ):
              logger.warning(
                  "latest_checkpoint has not been configured. Pause/resume training will not "
                  "be able to continue from latest checkpoint. Did you mean to set "
                  "`fit(latest_checkpoint=info.latest_checkpoint)'?"
              )
  
          smaller_is_better = bool(
              self._info.trial._config["searcher"]["smaller_is_better"]
          )
          searcher_metric_name = self._info.trial._config["searcher"][
              "metric"
          ]
          steps_completed = int(self._info.trial._steps_completed)
          global_batch_size = self._info.trial.hparams.get(
              "global_batch_size", None
          )
          if global_batch_size:
              global_batch_size = int(global_batch_size)
  
      trial_controller = pytorch._PyTorchTrialController(
          trial_inst=self._trial,
          context=self._context,
          checkpoint_period=checkpoint_period,
          validation_period=validation_period,
          smaller_is_better=smaller_is_better,
          steps_completed=steps_completed,
          latest_checkpoint=latest_checkpoint,
          local_training=self._local_training,
          test_mode=test_mode,
          reporting_period=reporting_period,
          searcher_metric_name=searcher_metric_name,
          checkpoint_policy=checkpoint_policy,
          step_zero_validation=step_zero_validation,
          max_length=max_length,
          det_profiler=self._det_profiler,
          global_batch_size=global_batch_size,
      )
  
      trial_controller.run()
  
file_path: determined/harness/determined/pytorch/_trainer.py
imports: import contextlib
  import logging
  import random
  import sys
  from typing import Any, Dict, Iterator, Optional
  import numpy
  import torch
  import torch.distributed
  import determined
  from determined import core, gpu, horovod, profiler, pytorch
  from determined.horovod import hvd
label_desc: Use image node_id 3 for calling the Trainer obj's underlying member method code with example usage: obj.fit(checkpoint_period, validation_period, max_length, reporting_period, checkpoint_policy, latest_checkpoint, step_zero_validation, test_mode) without return types