opennmt/runner.py
def _auto_tune_batch_size(
    config,
    min_batch_size,
    max_batch_size,
    min_range,
    sample_iterations=5,
    num_devices=1,
    scaling_factor=0.7,
    mixed_precision=False,
    timeout=15 * 60,
):
    """Find the largest token-based batch size that can be used with this
    configuration.

    This function runs some training iterations and uses out-of-memory errors as
    search conditions. A binary search is used to converge to a suitable batch
    size.

    We prefer to run the iterations in a different process so that it does not
    alter the current context (OOM may not be safe to recover from, see for
    example https://stackoverflow.com/q/53820713/2529808).

    Args:
      config: The training configuration.
      min_batch_size: The smallest batch size to consider.
      max_batch_size: The largest batch size to consider.
      min_range: Continue searching while the difference between
        :obj:`max_batch_size` and :obj:`min_batch_size` is larger than this value.
      sample_iterations: The number of training iterations.
      num_devices: The number of devices to use.
      scaling_factor: Scale the found batch size by this value.
      mixed_precision: If ``True``, run the autotuning with mixed precision.
      timeout: Consider the training attempt as failed after this many seconds.

    Returns:
      The autotuned batch size.
    """
    tf.get_logger().info(
        "Searching the largest batch size between %d and %d with a precision of %d...",
        min_batch_size,
        max_batch_size,
        min_range,
    )

    model_description = os.path.join(config["model_dir"], MODEL_DESCRIPTION_FILENAME)
    absolute_min_batch_size = min_batch_size
    stderr_data = None

    while max_batch_size - min_batch_size > min_range:
        batch_size = (max_batch_size + min_batch_size) // 2

        with tempfile.TemporaryDirectory() as tmpdir:
            run_config = copy.deepcopy(config)
            run_config["model_dir"] = tmpdir
            run_config["train"]["batch_autotune_mode"] = True
            run_config["train"]["batch_size"] = batch_size
            run_config["train"]["save_checkpoints_steps"] = None
            run_config["train"]["average_last_checkpoints"] = 0
            run_config["train"]["max_step"] = sample_iterations

            config_path = os.path.join(tmpdir, "batch_size_autotuner.yml")
            with tf.io.gfile.GFile(config_path, mode="w") as config_file:
                yaml.dump(run_config, config_file)

            env = os.environ.copy()
            env["TF_CPP_MIN_LOG_LEVEL"] = "2"
            args = [
                sys.executable or "python",
                "-m",
                "opennmt.bin.main",
                "--log_level",
                "ERROR",
                "--config",
                config_path,
                "--model",
                model_description,
            ]
            if mixed_precision:
                args.extend(["--mixed_precision"])
            args.extend(
                [
                    "train",
                    "--num_gpus",
                    str(num_devices),
                ]
            )

            tf.get_logger().info("Trying training with batch size %d...", batch_size)
            with open(os.devnull, "w") as devnull:
                process = subprocess.Popen(
                    args,
                    stdout=devnull,
                    stderr=subprocess.PIPE,
                    env=env,
                )
                try:
                    _, stderr_data = process.communicate(timeout=timeout)
                except subprocess.TimeoutExpired:
                    process.kill()
                    tf.get_logger().info("... failed (timeout).")
                    max_batch_size = batch_size - 1
                else:
                    if process.returncode != 0:
                        tf.get_logger().info("... failed.")
                        max_batch_size = batch_size - 1
                    else:
                        tf.get_logger().info(
                            "... succeeded, continue until the search range is smaller than %d.",
                            min_range,
                        )
                        min_batch_size = batch_size

    if min_batch_size == absolute_min_batch_size:
        if stderr_data is not None:
            tf.get_logger().error(
                'Last training attempt exited with an error:\n\n"""\n%s"""\n'
                % stderr_data.decode("utf-8")
            )
        raise RuntimeError(
            "Batch size autotuning failed: all training attempts exited with an error "
            "(see last error above). Either there is not enough memory to train this "
            "model, or unexpected errors occured. Please try to set a fixed batch size "
            "in the training configuration."
        )

    batch_size = max(int(scaling_factor * min_batch_size), absolute_min_batch_size)
    tf.get_logger().info("Batch size auto tuned to %d.", batch_size)
    return batch_size
Find the largest token-based batch size that can be used with this
    configuration.

    This function runs some training iterations and uses out-of-memory errors as
    search conditions. A binary search is used to converge to a suitable batch
    size.

    We prefer to run the iterations in a different process so that it does not
    alter the current context (OOM may not be safe to recover from, see for
    example https://stackoverflow.com/q/53820713/2529808).

    Args:
      config: The training configuration.
      min_batch_size: The smallest batch size to consider.
      max_batch_size: The largest batch size to consider.
      min_range: Continue searching while the difference between
        :obj:`max_batch_size` and :obj:`min_batch_size` is larger than this value.
      sample_iterations: The number of training iterations.
      num_devices: The number of devices to use.
      scaling_factor: Scale the found batch size by this value.
      mixed_precision: If ``True``, run the autotuning with mixed precision.
      timeout: Consider the training attempt as failed after this many seconds.

    Returns:
      The autotuned batch size.
    
Find the largest token-based batch size that can be used with this
configuration.

This function runs some training iterations and uses out-of-memory errors as
search conditions. A binary search is used to converge to a suitable batch
size.


Find the largest token - based batch size that can be used with this configuration . This function runs some training iterations and uses out - of - memory errors as search conditions . A binary search is used to converge to a suitable batch size .
def _auto_tune_batch_size(
    config,
    min_batch_size,
    max_batch_size,
    min_range,
    sample_iterations=5,
    num_devices=1,
    scaling_factor=0.7,
    mixed_precision=False,
    timeout=15 * 60,
):
    tf.get_logger().info(
        "Searching the largest batch size between %d and %d with a precision of %d...",
        min_batch_size,
        max_batch_size,
        min_range,
    )
    model_description = os.path.join(config["model_dir"], MODEL_DESCRIPTION_FILENAME)
    absolute_min_batch_size = min_batch_size
    stderr_data = None
    while max_batch_size - min_batch_size > min_range:
        batch_size = (max_batch_size + min_batch_size) // 2
        with tempfile.TemporaryDirectory() as tmpdir:
            run_config = copy.deepcopy(config)
            run_config["model_dir"] = tmpdir
            run_config["train"]["batch_autotune_mode"] = True
            run_config["train"]["batch_size"] = batch_size
            run_config["train"]["save_checkpoints_steps"] = None
            run_config["train"]["average_last_checkpoints"] = 0
            run_config["train"]["max_step"] = sample_iterations
            config_path = os.path.join(tmpdir, "batch_size_autotuner.yml")
            with tf.io.gfile.GFile(config_path, mode="w") as config_file:
                yaml.dump(run_config, config_file)
            env = os.environ.copy()
            env["TF_CPP_MIN_LOG_LEVEL"] = "2"
            args = [
                sys.executable or "python",
                "-m",
                "opennmt.bin.main",
                "--log_level",
                "ERROR",
                "--config",
                config_path,
                "--model",
                model_description,
            ]
            if mixed_precision:
                args.extend(["--mixed_precision"])
            args.extend(
                [
                    "train",
                    "--num_gpus",
                    str(num_devices),
                ]
            )
            tf.get_logger().info("Trying training with batch size %d...", batch_size)
            with open(os.devnull, "w") as devnull:
                process = subprocess.Popen(
                    args,
                    stdout=devnull,
                    stderr=subprocess.PIPE,
                    env=env,
                )
                try:
                    _, stderr_data = process.communicate(timeout=timeout)
                except subprocess.TimeoutExpired:
                    process.kill()
                    tf.get_logger().info("... failed (timeout).")
                    max_batch_size = batch_size - 1
                else:
                    if process.returncode != 0:
                        tf.get_logger().info("... failed.")
                        max_batch_size = batch_size - 1
                    else:
                        tf.get_logger().info(
                            "... succeeded, continue until the search range is smaller than %d.",
                            min_range,
                        )
                        min_batch_size = batch_size
    if min_batch_size == absolute_min_batch_size:
        if stderr_data is not None:
            tf.get_logger().error(
                'Last training attempt exited with an error:\n\n"""\n%s"""\n'
                % stderr_data.decode("utf-8")
            )
        raise RuntimeError(
            "Batch size autotuning failed: all training attempts exited with an error "
            "(see last error above). Either there is not enough memory to train this "
            "model, or unexpected errors occured. Please try to set a fixed batch size "
            "in the training configuration."
        )
    batch_size = max(int(scaling_factor * min_batch_size), absolute_min_batch_size)
    tf.get_logger().info("Batch size auto tuned to %d.", batch_size)
    return batch_size
def _auto_tune_batch_size ( config , min_batch_size , max_batch_size , min_range , sample_iterations = 5 , num_devices = 1 , scaling_factor = 0.7 , mixed_precision = False , timeout = 15 * 60 , ) : tf . get_logger ( ) . info ( "Searching the largest batch size between %d and %d with a precision of %d..." , min_batch_size , max_batch_size , min_range , ) model_description = os . path . join ( config [ "model_dir" ] , MODEL_DESCRIPTION_FILENAME ) absolute_min_batch_size = min_batch_size stderr_data = None while max_batch_size - min_batch_size > min_range : batch_size = ( max_batch_size + min_batch_size ) // 2 with tempfile . TemporaryDirectory ( ) as tmpdir : run_config = copy . deepcopy ( config ) run_config [ "model_dir" ] = tmpdir run_config [ "train" ] [ "batch_autotune_mode" ] = True run_config [ "train" ] [ "batch_size" ] = batch_size run_config [ "train" ] [ "save_checkpoints_steps" ] = None run_config [ "train" ] [ "average_last_checkpoints" ] = 0 run_config [ "train" ] [ "max_step" ] = sample_iterations config_path = os . path . join ( tmpdir , "batch_size_autotuner.yml" ) with tf . io . gfile . GFile ( config_path , mode = "w" ) as config_file : yaml . dump ( run_config , config_file ) env = os . environ . copy ( ) env [ "TF_CPP_MIN_LOG_LEVEL" ] = "2" args = [ sys . executable or "python" , "-m" , "opennmt.bin.main" , "--log_level" , "ERROR" , "--config" , config_path , "--model" , model_description , ] if mixed_precision : args . extend ( [ "--mixed_precision" ] ) args . extend ( [ "train" , "--num_gpus" , str ( num_devices ) , ] ) tf . get_logger ( ) . info ( "Trying training with batch size %d..." , batch_size ) with open ( os . devnull , "w" ) as devnull : process = subprocess . Popen ( args , stdout = devnull , stderr = subprocess . PIPE , env = env , ) try : _ , stderr_data = process . communicate ( timeout = timeout ) except subprocess . TimeoutExpired : process . kill ( ) tf . get_logger ( ) . info ( "... failed (timeout)." ) max_batch_size = batch_size - 1 else : if process . returncode != 0 : tf . get_logger ( ) . info ( "... failed." ) max_batch_size = batch_size - 1 else : tf . get_logger ( ) . info ( "... succeeded, continue until the search range is smaller than %d." , min_range , ) min_batch_size = batch_size if min_batch_size == absolute_min_batch_size : if stderr_data is not None : tf . get_logger ( ) . error ( 'Last training attempt exited with an error:\n\n"""\n%s"""\n' % stderr_data . decode ( "utf-8" ) ) raise RuntimeError ( "Batch size autotuning failed: all training attempts exited with an error " "(see last error above). Either there is not enough memory to train this " "model, or unexpected errors occured. Please try to set a fixed batch size " "in the training configuration." ) batch_size = max ( int ( scaling_factor * min_batch_size ) , absolute_min_batch_size ) tf . get_logger ( ) . info ( "Batch size auto tuned to %d." , batch_size ) return batch_size
Find the largest token-based batch size that can be used with this
configuration.
Find the largest token - based batch size that can be used with this configuration .
"""Find the largest token-based batch size that can be used with this
    configuration.

    This function runs some training iterations and uses out-of-memory errors as
    search conditions. A binary search is used to converge to a suitable batch
    size.

    We prefer to run the iterations in a different process so that it does not
    alter the current context (OOM may not be safe to recover from, see for
    example https://stackoverflow.com/q/53820713/2529808).

    Args:
      config: The training configuration.
      min_batch_size: The smallest batch size to consider.
      max_batch_size: The largest batch size to consider.
      min_range: Continue searching while the difference between
        :obj:`max_batch_size` and :obj:`min_batch_size` is larger than this value.
      sample_iterations: The number of training iterations.
      num_devices: The number of devices to use.
      scaling_factor: Scale the found batch size by this value.
      mixed_precision: If ``True``, run the autotuning with mixed precision.
      timeout: Consider the training attempt as failed after this many seconds.

    Returns:
      The autotuned batch size.
    """
The autotuned batch size.
The autotuned batch size .
config
The training configuration.
The training configuration .
min_batch_size
The smallest batch size to consider.
The smallest batch size to consider .
max_batch_size
The largest batch size to consider.
The largest batch size to consider .
min_range
Continue searching while the difference between
:obj:`max_batch_size` and :obj:`min_batch_size` is larger than this value.
Continue searching while the difference between : obj : ` max_batch_size ` and : obj : ` min_batch_size ` is larger than this value .
sample_iterations
The number of training iterations.
The number of training iterations .
num_devices
The number of devices to use.
The number of devices to use .
scaling_factor
Scale the found batch size by this value.
Scale the found batch size by this value .
mixed_precision
If ``True``, run the autotuning with mixed precision.
If ` ` True ` ` run the autotuning with mixed precision .
timeout
Consider the training attempt as failed after this many seconds.
Consider the training attempt as failed after this many seconds .