class_docstr: The Slurm Evaluator is a class designed to continuously monitor VISSL pretrainings
  and launch evaluations as checkpoints become available. The method takes a
  config dictionary consisting of the training checkpoint directory, an array of
  benchmarks, and information on how often to evaluate the trainings.
docstr: Evaluate the checkpoints. At a high level, this is the structure.
  
  1. Load training YAML config file.
  2. Monitor training checkpoints. When checkpoint is ready, launch the evaluation jobs.
  3. Monitor the evaluation jobs. When evaluation jobs are complete, mark the results.
  3. Once all jobs have been recorded, complete.
docstr_tok: ["Evaluate","the","checkpoints",".","At","a","high","level",",","this","is","the","structure",".","1",".","Load","training","YAML","config","file",".","2",".","Monitor","training","checkpoints",".","When","checkpoint","is","ready",",","launch","the","evaluation","jobs",".","3",".","Monitor","the","evaluation","jobs",".","When","evaluation","jobs","are","complete",",","mark","the","results",".","3",".","Once","all","jobs","have","been","recorded",",","complete","."]
code: def evaluate(self):
      """
      Evaluate the checkpoints. At a high level, this is the structure.
  
      1. Load training YAML config file.
      2. Monitor training checkpoints. When checkpoint is ready, launch the evaluation jobs.
      3. Monitor the evaluation jobs. When evaluation jobs are complete, mark the results.
      3. Once all jobs have been recorded, complete.
      """
      start_time = time.time()
  
      # Wait for the training config to be available. This indicates the training has begun.
      while True:
          if time.time() - start_time > _TRAINING_CONFIG_WAIT_SECONDS:
              raise RuntimeError(
                  f"Training config still doesn't exist after:"
                  f"{_TRAINING_CONFIG_WAIT_SECONDS / 60} minutes"
              )
  
          if (
              g_pathmgr.exists(self.training_checkpoint_file)
              and self._max_training_iterations()
          ):
              # Load training yaml config.
              self._load_training_config()
  
              # Set max training iterations
              self.max_training_iterations = (
                  self._max_training_iterations()
              )
  
              # Generate evaluation results
              self.evaluation_results = (
                  self._generate_initial_benchmark_results()
              )
              self._validate_evaluation_setup()
  
              break
  
          time.sleep(_SLEEP_TIME_SECONDS)
  
      # Save initial evaluation benchmarks, for checkpointing reasons.
      self.save_evaluation_benchmarks()
  
      # Checkpoint folder is now available. Continuously monitor the training checkpoints,
      # launch evaluation jobs as needed, monitor their progress, and record their results.
      while True:
          self._evaluate_checkpoints()
  
          self._check_evaluation_jobs()
  
          # Break if no more checkpoints to evaluate
          if self._finished():
              logging.info("Evaluations are finished")
              break
  
          time.sleep(_SLEEP_TIME_SECONDS)
  
imports: import collections
  import json
  import logging
  import os
  import time
  from datetime import datetime
  from pathlib import Path
  from typing import List
  import submitit
  from iopath.common.file_io import g_pathmgr
  from vissl.config.attr_dict import AttrDict
  from vissl.utils.distributed_launcher import launch_distributed_on_slurm
  from vissl.utils.hydra_config import compose_hydra_configuration, convert_to_attrdict
  from vissl.utils.io import load_file, makedir
  from vissl.utils.misc import flatten_dict, retry
label_desc: Use image node_id 2 for calling the BenchmarkSuiteScheduler obj's underlying member method code with example usage: obj.evaluate() without return types