docstr: Parameters
  ----------
  model : ``Model``, required.
      An AllenNLP model to be optimized. Pytorch Modules can also be optimized if
      their ``forward`` method returns a dictionary with a "loss" key, containing a
      scalar tensor representing the loss function to be optimized.
  optimizer : ``torch.nn.Optimizer``, required.
      An instance of a Pytorch Optimizer, instantiated with the parameters of the
      model to be optimized.
  iterator : ``DataIterator``, required.
      A method for iterating over a ``Dataset``, yielding padded indexed batches.
  train_dataset : ``Dataset``, required.
      A ``Dataset`` to train on. The dataset should have already been indexed.
  validation_dataset : ``Dataset``, optional, (default = None).
      A ``Dataset`` to evaluate on. The dataset should have already been indexed.
  patience : Optional[int] > 0, optional (default=None)
      Number of epochs to be patient before early stopping: the training is stopped
      after ``patience`` epochs with no improvement. If given, it must be ``> 0``.
      If None, early stopping is disabled.
  validation_metric : str, optional (default="loss")
      Validation metric to measure for whether to stop training using patience
      and whether to serialize an ``is_best`` model each epoch. The metric name
      must be prepended with either "+" or "-", which specifies whether the metric
      is an increasing or decreasing function.
  validation_iterator : ``DataIterator``, optional (default=None)
      An iterator to use for the validation set.  If ``None``, then
      use the training `iterator`.
  shuffle: ``bool``, optional (default=True)
      Whether to shuffle the instances in the iterator or not.
  num_epochs : int, optional (default = 20)
      Number of training epochs.
  serialization_dir : str, optional (default=None)
      Path to directory for saving and loading model files. Models will not be saved if
      this parameter is not passed.
  num_serialized_models_to_keep : ``int``, optional (default=20)
      Number of previous model checkpoints to retain.  Default is to keep 20 checkpoints.
      A value of None or -1 means all checkpoints will be kept.
  keep_serialized_model_every_num_seconds : ``int``, optional (default=None)
      If num_serialized_models_to_keep is not None, then occasionally it's useful to
      save models at a given interval in addition to the last num_serialized_models_to_keep.
      To do so, specify keep_serialized_model_every_num_seconds as the number of seconds
      between permanently saved checkpoints.  Note that this option is only used if
      num_serialized_models_to_keep is not None, otherwise all checkpoints are kept.
  model_save_interval : ``float``, optional (default=None)
      If provided, then serialize models every ``model_save_interval``
      seconds within single epochs.  In all cases, models are also saved
      at the end of every epoch if ``serialization_dir`` is provided.
  cuda_device : ``int``, optional (default = -1)
      An integer specifying the CUDA device to use. If -1, the CPU is used.
  grad_norm : ``float``, optional, (default = None).
      If provided, gradient norms will be rescaled to have a maximum of this value.
  grad_clipping : ``float``, optional (default = ``None``).
      If provided, gradients will be clipped `during the backward pass` to have an (absolute)
      maximum of this value.  If you are getting ``NaNs`` in your gradients during training
      that are not solved by using ``grad_norm``, you may need this.
  learning_rate_scheduler : ``PytorchLRScheduler``, optional, (default = None)
      A Pytorch learning rate scheduler. The learning rate will be decayed with respect to
      this schedule at the end of each epoch. If you use
      :class:`torch.optim.lr_scheduler.ReduceLROnPlateau`, this will use the ``validation_metric``
      provided to determine if learning has plateaued.  To support updating the learning
      rate on every batch, this can optionally implement ``step_batch(batch_num_total)`` which
      updates the learning rate given the batch number.
  summary_interval: ``int``, optional, (default = 100)
      Number of batches between logging scalars to tensorboard
  histogram_interval : ``int``, optional, (default = ``None``)
      If not None, then log histograms to tensorboard every ``histogram_interval`` batches.
      When this parameter is specified, the following additional logging is enabled:
          * Histograms of model parameters
          * The ratio of parameter update norm to parameter norm
          * Histogram of layer activations
      We log histograms of the parameters returned by
      ``model.get_parameters_for_histogram_tensorboard_logging``.
      The layer activations are logged for any modules in the ``Model`` that have
      the attribute ``should_log_activations`` set to ``True``.  Logging
      histograms requires a number of GPU-CPU copies during training and is typically
      slow, so we recommend logging histograms relatively infrequently.
      Note: only Modules that return tensors, tuples of tensors or dicts
      with tensors as values currently support activation logging.
docstr_tok: ["Parameters","--","--","--","--","--","model",":","``","Model","``",",","required",".","An","AllenNLP","model","to","be","optimized",".","Pytorch","Modules","can","also","be","optimized","if","their","``","forward","``","method","returns","a","dictionary","with","a","``","loss","''","key",",","containing","a","scalar","tensor","representing","the","loss","function","to","be","optimized",".","optimizer",":","``","torch.nn.Optimizer","``",",","required",".","An","instance","of","a","Pytorch","Optimizer",",","instantiated","with","the","parameters","of","the","model","to","be","optimized",".","iterator",":","``","DataIterator","``",",","required",".","A","method","for","iterating","over","a","``","Dataset","``",",","yielding","padded","indexed","batches",".","train_dataset",":","``","Dataset","``",",","required",".","A","``","Dataset","``","to","train","on",".","The","dataset","should","have","already","been","indexed",".","validation_dataset",":","``","Dataset","``",",","optional",",","(","default","=","None",")",".","A","``","Dataset","``","to","evaluate","on",".","The","dataset","should","have","already","been","indexed",".","patience",":","Optional","[","int","]",">","0",",","optional","(","default=None",")","Number","of","epochs","to","be","patient","before","early","stopping",":","the","training","is","stopped","after","``","patience","``","epochs","with","no","improvement",".","If","given",",","it","must","be","``",">","0","``",".","If","None",",","early","stopping","is","disabled",".","validation_metric",":","str",",","optional","(","default=","''","loss","''",")","Validation","metric","to","measure","for","whether","to","stop","training","using","patience","and","whether","to","serialize","an","``","is_best","``","model","each","epoch",".","The","metric","name","must","be","prepended","with","either","``","+","''","or","``","-","''",",","which","specifies","whether","the","metric","is","an","increasing","or","decreasing","function",".","validation_iterator",":","``","DataIterator","``",",","optional","(","default=None",")","An","iterator","to","use","for","the","validation","set",".","If","``","None","``",",","then","use","the","training","`","iterator","`",".","shuffle",":","``","bool","``",",","optional","(","default=True",")","Whether","to","shuffle","the","instances","in","the","iterator","or","not",".","num_epochs",":","int",",","optional","(","default","=","20",")","Number","of","training","epochs",".","serialization_dir",":","str",",","optional","(","default=None",")","Path","to","directory","for","saving","and","loading","model","files",".","Models","will","not","be","saved","if","this","parameter","is","not","passed",".","num_serialized_models_to_keep",":","``","int","``",",","optional","(","default=20",")","Number","of","previous","model","checkpoints","to","retain",".","Default","is","to","keep","20","checkpoints",".","A","value","of","None","or","-1","means","all","checkpoints","will","be","kept",".","keep_serialized_model_every_num_seconds",":","``","int","``",",","optional","(","default=None",")","If","num_serialized_models_to_keep","is","not","None",",","then","occasionally","it","'s","useful","to","save","models","at","a","given","interval","in","addition","to","the","last","num_serialized_models_to_keep",".","To","do","so",",","specify","keep_serialized_model_every_num_seconds","as","the","number","of","seconds","between","permanently","saved","checkpoints",".","Note","that","this","option","is","only","used","if","num_serialized_models_to_keep","is","not","None",",","otherwise","all","checkpoints","are","kept",".","model_save_interval",":","``","float","``",",","optional","(","default=None",")","If","provided",",","then","serialize","models","every","``","model_save_interval","``","seconds","within","single","epochs",".","In","all","cases",",","models","are","also","saved","at","the","end","of","every","epoch","if","``","serialization_dir","``","is","provided",".","cuda_device",":","``","int","``",",","optional","(","default","=","-1",")","An","integer","specifying","the","CUDA","device","to","use",".","If","-1",",","the","CPU","is","used",".","grad_norm",":","``","float","``",",","optional",",","(","default","=","None",")",".","If","provided",",","gradient","norms","will","be","rescaled","to","have","a","maximum","of","this","value",".","grad_clipping",":","``","float","``",",","optional","(","default","=","``","None","``",")",".","If","provided",",","gradients","will","be","clipped","`","during","the","backward","pass","`","to","have","an","(","absolute",")","maximum","of","this","value",".","If","you","are","getting","``","NaNs","``","in","your","gradients","during","training","that","are","not","solved","by","using","``","grad_norm","``",",","you","may","need","this",".","learning_rate_scheduler",":","``","PytorchLRScheduler","``",",","optional",",","(","default","=","None",")","A","Pytorch","learning","rate","scheduler",".","The","learning","rate","will","be","decayed","with","respect","to","this","schedule","at","the","end","of","each","epoch",".","If","you","use",":","class",":","`","torch.optim.lr_scheduler.ReduceLROnPlateau","`",",","this","will","use","the","``","validation_metric","``","provided","to","determine","if","learning","has","plateaued",".","To","support","updating","the","learning","rate","on","every","batch",",","this","can","optionally","implement","``","step_batch","(","batch_num_total",")","``","which","updates","the","learning","rate","given","the","batch","number",".","summary_interval",":","``","int","``",",","optional",",","(","default","=","100",")","Number","of","batches","between","logging","scalars","to","tensorboard","histogram_interval",":","``","int","``",",","optional",",","(","default","=","``","None","``",")","If","not","None",",","then","log","histograms","to","tensorboard","every","``","histogram_interval","``","batches",".","When","this","parameter","is","specified",",","the","following","additional","logging","is","enabled",":","*","Histograms","of","model","parameters","*","The","ratio","of","parameter","update","norm","to","parameter","norm","*","Histogram","of","layer","activations","We","log","histograms","of","the","parameters","returned","by","``","model.get_parameters_for_histogram_tensorboard_logging","``",".","The","layer","activations","are","logged","for","any","modules","in","the","``","Model","``","that","have","the","attribute","``","should_log_activations","``","set","to","``","True","``",".","Logging","histograms","requires","a","number","of","GPU-CPU","copies","during","training","and","is","typically","slow",",","so","we","recommend","logging","histograms","relatively","infrequently",".","Note",":","only","Modules","that","return","tensors",",","tuples","of","tensors","or","dicts","with","tensors","as","values","currently","support","activation","logging","."]
code: def __init__(
      self,
      model,
      optimizer,
      iterator,
      train_dataset,
      validation_dataset=None,
      patience=None,
      validation_metric="-loss",
      validation_iterator=None,
      shuffle=True,
      num_epochs=20,
      serialization_dir=None,
      num_serialized_models_to_keep=20,
      keep_serialized_model_every_num_seconds=None,
      model_save_interval=None,
      cuda_device=-1,
      grad_norm=None,
      grad_clipping=None,
      learning_rate_scheduler=None,
      summary_interval=100,
      histogram_interval=None,
  ):
      """
      Parameters
      ----------
      model : ``Model``, required.
          An AllenNLP model to be optimized. Pytorch Modules can also be optimized if
          their ``forward`` method returns a dictionary with a "loss" key, containing a
          scalar tensor representing the loss function to be optimized.
      optimizer : ``torch.nn.Optimizer``, required.
          An instance of a Pytorch Optimizer, instantiated with the parameters of the
          model to be optimized.
      iterator : ``DataIterator``, required.
          A method for iterating over a ``Dataset``, yielding padded indexed batches.
      train_dataset : ``Dataset``, required.
          A ``Dataset`` to train on. The dataset should have already been indexed.
      validation_dataset : ``Dataset``, optional, (default = None).
          A ``Dataset`` to evaluate on. The dataset should have already been indexed.
      patience : Optional[int] > 0, optional (default=None)
          Number of epochs to be patient before early stopping: the training is stopped
          after ``patience`` epochs with no improvement. If given, it must be ``> 0``.
          If None, early stopping is disabled.
      validation_metric : str, optional (default="loss")
          Validation metric to measure for whether to stop training using patience
          and whether to serialize an ``is_best`` model each epoch. The metric name
          must be prepended with either "+" or "-", which specifies whether the metric
          is an increasing or decreasing function.
      validation_iterator : ``DataIterator``, optional (default=None)
          An iterator to use for the validation set.  If ``None``, then
          use the training `iterator`.
      shuffle: ``bool``, optional (default=True)
          Whether to shuffle the instances in the iterator or not.
      num_epochs : int, optional (default = 20)
          Number of training epochs.
      serialization_dir : str, optional (default=None)
          Path to directory for saving and loading model files. Models will not be saved if
          this parameter is not passed.
      num_serialized_models_to_keep : ``int``, optional (default=20)
          Number of previous model checkpoints to retain.  Default is to keep 20 checkpoints.
          A value of None or -1 means all checkpoints will be kept.
      keep_serialized_model_every_num_seconds : ``int``, optional (default=None)
          If num_serialized_models_to_keep is not None, then occasionally it's useful to
          save models at a given interval in addition to the last num_serialized_models_to_keep.
          To do so, specify keep_serialized_model_every_num_seconds as the number of seconds
          between permanently saved checkpoints.  Note that this option is only used if
          num_serialized_models_to_keep is not None, otherwise all checkpoints are kept.
      model_save_interval : ``float``, optional (default=None)
          If provided, then serialize models every ``model_save_interval``
          seconds within single epochs.  In all cases, models are also saved
          at the end of every epoch if ``serialization_dir`` is provided.
      cuda_device : ``int``, optional (default = -1)
          An integer specifying the CUDA device to use. If -1, the CPU is used.
      grad_norm : ``float``, optional, (default = None).
          If provided, gradient norms will be rescaled to have a maximum of this value.
      grad_clipping : ``float``, optional (default = ``None``).
          If provided, gradients will be clipped `during the backward pass` to have an (absolute)
          maximum of this value.  If you are getting ``NaNs`` in your gradients during training
          that are not solved by using ``grad_norm``, you may need this.
      learning_rate_scheduler : ``PytorchLRScheduler``, optional, (default = None)
          A Pytorch learning rate scheduler. The learning rate will be decayed with respect to
          this schedule at the end of each epoch. If you use
          :class:`torch.optim.lr_scheduler.ReduceLROnPlateau`, this will use the ``validation_metric``
          provided to determine if learning has plateaued.  To support updating the learning
          rate on every batch, this can optionally implement ``step_batch(batch_num_total)`` which
          updates the learning rate given the batch number.
      summary_interval: ``int``, optional, (default = 100)
          Number of batches between logging scalars to tensorboard
      histogram_interval : ``int``, optional, (default = ``None``)
          If not None, then log histograms to tensorboard every ``histogram_interval`` batches.
          When this parameter is specified, the following additional logging is enabled:
              * Histograms of model parameters
              * The ratio of parameter update norm to parameter norm
              * Histogram of layer activations
          We log histograms of the parameters returned by
          ``model.get_parameters_for_histogram_tensorboard_logging``.
          The layer activations are logged for any modules in the ``Model`` that have
          the attribute ``should_log_activations`` set to ``True``.  Logging
          histograms requires a number of GPU-CPU copies during training and is typically
          slow, so we recommend logging histograms relatively infrequently.
          Note: only Modules that return tensors, tuples of tensors or dicts
          with tensors as values currently support activation logging.
      """
      self._model = model
      self._iterator = iterator
      self._validation_iterator = validation_iterator
      self._shuffle = shuffle
      self._optimizer = optimizer
      self._train_data = train_dataset
      self._validation_data = validation_dataset
  
      if patience is None:  # no early stopping
          if validation_dataset:
              logger.warning(
                  "You provided a validation dataset but patience was set to None, "
                  "meaning that early stopping is disabled"
              )
      elif (not isinstance(patience, int)) or patience <= 0:
          raise ConfigurationError(
              '{} is an invalid value for "patience": it must be a positive integer '
              "or None (if you want to disable early stopping)".format(
                  patience
              )
          )
      self._patience = patience
      self._num_epochs = num_epochs
  
      self._serialization_dir = serialization_dir
      self._num_serialized_models_to_keep = (
          num_serialized_models_to_keep
      )
      self._keep_serialized_model_every_num_seconds = (
          keep_serialized_model_every_num_seconds
      )
      self._serialized_paths = []
      self._last_permanent_saved_checkpoint_time = time.time()
      self._model_save_interval = model_save_interval
  
      self._grad_norm = grad_norm
      self._grad_clipping = grad_clipping
      self._learning_rate_scheduler = learning_rate_scheduler
  
      increase_or_decrease = validation_metric[0]
      if increase_or_decrease not in ["+", "-"]:
          raise ConfigurationError(
              "Validation metrics must specify whether they should increase "
              "or decrease by pre-pending the metric name with a +/-."
          )
      self._validation_metric = validation_metric[1:]
      self._validation_metric_decreases = increase_or_decrease == "-"
  
      if not isinstance(cuda_device, int) and not isinstance(
          cuda_device, list
      ):
          raise ConfigurationError(
              "Expected an int or list for cuda_device, got {}".format(
                  cuda_device
              )
          )
  
      if isinstance(cuda_device, list):
          logger.info(
              "WARNING: Multiple GPU support is experimental not recommended for use. "
              "In some cases it may lead to incorrect results or undefined behavior."
          )
          self._multiple_gpu = True
          self._cuda_devices = cuda_device
          # data_parallel will take care of transfering to cuda devices,
          # so the iterator keeps data on CPU.
          self._iterator_device = -1
      else:
          self._multiple_gpu = False
          self._cuda_devices = [cuda_device]
          self._iterator_device = cuda_device
  
      if self._cuda_devices[0] != -1:
          self._model = self._model.cuda(self._cuda_devices[0])
  
      self._log_interval = 10  # seconds
      self._summary_interval = summary_interval
      self._histogram_interval = histogram_interval
      self._log_histograms_this_batch = False
      # We keep the total batch number as a class variable because it
      # is used inside a closure for the hook which logs activations in
      # ``_enable_activation_logging``.
      self._batch_num_total = 0
  
      self._last_log = 0.0  # time of last logging
  
      if serialization_dir is not None:
          train_log = SummaryWriter(
              os.path.join(serialization_dir, "log", "train")
          )
          validation_log = SummaryWriter(
              os.path.join(serialization_dir, "log", "validation")
          )
          self._tensorboard = TensorboardWriter(
              train_log, validation_log
          )
      else:
          self._tensorboard = TensorboardWriter()
      self._warned_tqdm_ignores_underscores = False
  
imports: from __future__ import with_statement
  from __future__ import division
  from __future__ import absolute_import
  import logging
  import os
  import shutil
  import time
  import re
  import datetime
  import traceback
  import torch
  import torch.optim.lr_scheduler
  from torch.nn.parallel import replicate, parallel_apply
  from torch.nn.parallel.scatter_gather import scatter_kwargs, gather
  from allennlp.common import Params
  from allennlp.common.checks import ConfigurationError
  from allennlp.common.util import peak_memory_mb, gpu_memory_mb
  from allennlp.common.tqdm import Tqdm
  from allennlp.data.instance import Instance
  from allennlp.data.iterators.data_iterator import DataIterator
  from allennlp.models.model import Model
  from allennlp.nn import util
  from allennlp.training.learning_rate_schedulers import LearningRateScheduler
  from allennlp.training.optimizers import Optimizer
label_desc: Use image node_id 1 to create a new Trainer object from inherited base classes: object with example: obj = Trainer(model, optimizer, iterator, train_dataset, validation_dataset, patience, validation_metric, validation_iterator, shuffle, num_epochs, serialization_dir, num_serialized_models_to_keep, keep_serialized_model_every_num_seconds, model_save_interval, cuda_device, grad_norm, grad_clipping, learning_rate_scheduler, summary_interval, histogram_interval)