class_docstr: Base trainers for models that can be built by calling a tower function under a :class:`TowerContext`.
  
  The assumption of tower function is required by some features that replicates the model
  automatically. For example, TowerTrainer can create a predictor for you automatically,
  by calling the tower function.
  
  To use :class:`TowerTrainer`, set `tower_func` and use it to build the graph.
  Note that `tower_func` can only be set once per instance of `TowerTrainer`.
class_docstr_tok: ["Base","trainers","for","models","that","can","be","built","by","calling","a","tower","function","under","a",":","class",":","`","TowerContext","`",".","The","assumption","of","tower","function","is","required","by","some","features","that","replicates","the","model","automatically",".","For","example",",","TowerTrainer","can","create","a","predictor","for","you","automatically",",","by","calling","the","tower","function",".","To","use",":","class",":","`","TowerTrainer","`",",","set","`","tower_func","`","and","use","it","to","build","the","graph",".","Note","that","`","tower_func","`","can","only","be","set","once","per","instance","of","`","TowerTrainer","`","."]
docstr: TowerTensorHandles: used to access the tower handles by either indices or names.
  
  This property is accessbile only after the graph is set up.
  With :meth:`towers`, you can then access many attributes of each tower:
  
  Example:
  
  .. code-block:: python
  
      # Access the conv1/output tensor in the first training tower
      trainer.towers.training()[0].get_tensor('conv1/output')
docstr_tok: ["TowerTensorHandles",":","used","to","access","the","tower","handles","by","either","indices","or","names",".","This","property","is","accessbile","only","after","the","graph","is","set","up",".","With",":","meth",":","`","towers","`",",","you","can","then","access","many","attributes","of","each","tower",":","Example",":","..","code-block",":",":","python","#","Access","the","conv1\/output","tensor","in","the","first","training","tower","trainer.towers.training","(",")","[","0","]",".get_tensor","(","'conv1\/output","'",")"]
code: def towers(self):
      """
      TowerTensorHandles: used to access the tower handles by either indices or names.
  
      This property is accessbile only after the graph is set up.
      With :meth:`towers`, you can then access many attributes of each tower:
  
      Example:
  
      .. code-block:: python
  
          # Access the conv1/output tensor in the first training tower
          trainer.towers.training()[0].get_tensor('conv1/output')
      """
      return self.tower_func.towers
  
imports: from abc import ABCMeta, abstractmethod
  import six
  import tensorflow
  from ..compat import tfv1, is_tfv2
  from ..input_source import PlaceholderInput
  from ..predict.base import OnlinePredictor
  from ..tfutils.gradproc import FilterNoneGrad
  from ..tfutils.tower import PredictTowerContext, TowerFunc, get_current_tower_context
  from ..utils import logger
  from ..utils.argtools import call_only_once, memoized
  from ..utils.develop import HIDE_DOC
  from .base import Trainer
label_desc: Use image node_id 5 for calling the TowerTrainer obj's underlying member method code with example usage: obj.towers() and returns: self