  
      Each side may have up to 3 generators (training, preview and time-lapse). To conserve VRAM
      these need to share access to the same face information for the images they are processing.
  
      As the cache is populated at run-time, thread safe writes are required for the first epoch.
      Following that, the cache is only used for reads, which is thread safe intrinsically.
  
      It would probably be quicker to set locks on each individual face, but for code complexity
      reasons, and the fact that the lock is only taken up during cache population, and it should
      only be being read multiple times on save iterations, we lock the whole cache during writes.
  
 
</s> remove     Each side may have up to 3 generators (training, preview and time-lapse). To conserve VRAM
    these need to share access to the same face information for the images they are processing.
 </s> add  </s> remove     It would probably be quicker to set locks on each individual face, but for code complexity
    reasons, and the fact that the lock is only taken up during cache population, and it should
    only be being read multiple times on save iterations, we lock the whole cache during writes.
 </s> add     This class is called from :mod:`plugins.train.trainer._base` and launches a background
    iterator that compiles augmented data, target data and sample data. </s> remove class _Cache():
    """ A thread safe mechanism for collecting and holding face meta information (masks, "
    "alignments data etc.) for multiple :class:`TrainingDataGenerator`s.
 </s> add logger = logging.getLogger(__name__)
ConfigType = Dict[str, Union[bool, int, float, str]]  # TODO Dataclass
BatchType = Tuple[np.ndarray, List[np.ndarray]] </s> remove     filenames: list
        The filenames of all the images. This can either be the full path or the base name. If the
        full paths are passed in, they are stripped to base name for use as the cache key.
 </s> add     model: :class:`~plugins.train.model.ModelBase`
        The model that this data generator is feeding </s> remove         set_flag: bool
            ``True`` if the flag should be set to indicate that the cache is being reset because of
            a legacy face set/centering mismatch. ``False`` if the cache is being reset because it
            has detected a reset flag from the opposite cache.
 </s> add         detected_face: list
            The list of :class:`~lib.align.DetectedFace` objects corresponding to the batch
        batch: :class:`numpy.ndarray`
            The preallocated array to apply masks to
        side: str
            '"a"' or '"b"' the side that is being processed
class DataGenerator():
    """ Parent class for Training and Preview Data Generators.
  
      Each side may have up to 3 generators (training, preview and time-lapse). To conserve VRAM
      these need to share access to the same face information for the images they are processing.
  
class DataGenerator():
    """ Parent class for Training and Preview Data Generators.
class DataGenerator():
    """ Parent class for Training and Preview Data Generators.
  
      It would probably be quicker to set locks on each individual face, but for code complexity
      reasons, and the fact that the lock is only taken up during cache population, and it should
      only be being read multiple times on save iterations, we lock the whole cache during writes.
  
keep keep keep keep replace replace keep keep keep keep keep
 Data Augmentation update (#1263)

- lib.detected_face
    - Subclass Masks for Landmark based masks
    - Add training mask propery + methods to DetectedFace
  - lib.training_training
    - subclass TrainingDataGenerator for training and preview data
    - Split cache into own module
    - Reduce thread count to 1 to prevent image corruption + data re-use
    - Process on largest model input/output size rather than stored image size
    - Size and crop masks during caching stage
    - Implement ring buffer for data flow
    - Fix preview reload bug
  - augmentation
    - typing
    - switch color aug order
    - better initialization
    - Fix warp + landmark warp to correctly apply at different image scales
    - Slightly improved warp caching
    - Don't store whether image is_preview. Handle all data as training images implicitly
  - plugins.trainer: Typing and fixes to work with trainingdata refactor 
lib/training/generator.py