# embedding_metadata

key: chroma:document
string_value:
CODE:

class TrainState(train_state.TrainState):
    dropout_rng: jnp.ndarray

    def replicate(self):


SUMMARY:
**Concise Summary**

This class extends `train_state.TrainState` and represents the state of a training process. It has an additional attribute `dropout_rng`. The primary purpose is to replicate this state.

**Key Details**

* Extends `train_state.TrainState`
* Has attribute `dropout_rng`: `jnp.ndarray`
* Implements `replicate` method:
	+ Replicates the state using `jax_utils.replicate`
	+ Replaces `dropout_rng` with a new value generated by `shard_prng_key`

**Notable Dependencies/Requirements**

* `jax_utils.replicate`
* `shard_prng_key` (functions from JAX library)
* `train_state.TrainState` (inherited class)


key: chroma:document
string_value:
CODE:

def data_loader(rng: jax.random.PRNGKey, dataset: Dataset, batch_size: int, shuffle: bool = False, drop_last=True):
    """
    Returns batches of size `batch_size` from `dataset`. If `drop_last` is set to `False`, the final batch may be incomplete,
    and range in size from 1 to `batch_size`. Shuffle batches if `shuffle` is `True`.
    """
    if shuffle:
        batch_idx = jax.random.permutation(rng, len(dataset))
        batch_idx = np.asarray(batch_idx)
    else:
        batch_idx = np.arange(len(dataset))

    if drop_last:
        steps_per_epoch = len(dataset) // batch_size
        batch_idx = batch_idx[: steps_per_epoch * batch_size]  # Skip incomplete batch.
        batch_idx = batch_idx.reshape((steps_per_epoch, batch_size))
    else:
        steps_per_epoch = math.ceil(len(dataset) / batch_size)
        batch_idx = np.array_split(batch_idx, steps_per_epoch)

    for idx in batch_idx:
        batch = dataset[idx]
        batch = {k: np.array(v) for k, v in batch.items()}



SUMMARY:
**Summary: `DataLoader` Function**

### Purpose and Functionality

Returns batches of size `batch_size` from a `Dataset` object. Optionally shuffles batches if `shuffle` is `True`. The final batch may be incomplete if `drop_last` is `False`.

### Key Implementation Details

* Uses JAX random number generator for shuffling.
* Reshapes the index array to facilitate batch iteration.
* Utilizes NumPy's `np.arange`, `np permsute`, and `np.array_split` functions.

### Notable Dependencies and Requirements

* `jax.random.PRNGKey`
* `Dataset` class (not provided, assume a custom dataset interface)
* External dependency: `numpy`

**Code Structure**

* The function takes in the following parameters:
	+ `rng`: JAX random number generator key
	+ `dataset`: Dataset object
	+ `batch_size`: Batch size
	+ `shuffle` (optional): Whether to shuffle batches
	+ `drop_last` (optional): Whether to drop the last batch if incomplete

**Iterative Yield**

The function yields batches of `batch_size` from the `dataset`. Each iteration yields the following:
* A dictionary representing a batch (`k: np.array(v)`)
* No output arguments are returned.


key: chroma:document
string_value:
CODE:

def write_train_metric(summary_writer, train_metrics, train_time, step):
    summary_writer.scalar("train_time", train_time, step)

    train_metrics = get_metrics(train_metrics)
    for key, vals in train_metrics.items():
        tag = f"train_{key}"
        for i, val in enumerate(vals):


SUMMARY:
**Function Summary: `write_train_metric`**

Purpose:
Write training metrics to a summary writer.

**Primary Functionality:**
Writes training time and metric values to a summary writer.

**Implementation Details:**

*   Retrieves updated `train_metrics`.
*   Writes each metric as a scalar value, including the train time.
    *   Each metric is labeled with "train_\<metric_name>" and written at a consecutive step number.
*   The first value in the metric's list is written immediately after the training time.

**Notable Dependencies/Requirements:**

*   `summary_writer`: An instance of a logging summary writer.
*   `get_metrics(train_metrics)`: A function that updates the `train_metrics` dictionary. (Implementation not shown.)
    *   Returns an updated version of the `train_metrics` dictionary with modified values (not detailed here).
*   `step`, `train_time`: Numerical step number and training time, respectively.

**Interface/Function Signatures:**

*   `write_train_metric(summary_writer, train_metrics, train_time, step)`: A method accepting a summary writer, updated train metrics, training time, and step number.
    *   Returns None (not explicitly stated but implied by not returning a value).


key: chroma:document
string_value:
CODE:

def write_eval_metric(summary_writer, eval_metrics, step):
    for metric_name, value in eval_metrics.items():


SUMMARY:
**Summary:**

`write_eval_metric` function writes evaluation metrics to a `summary_writer` using the `scalar` method, which takes metric name and value.

* Purpose: Writes evaluation metrics to a summary writer for monitoring purposes.
* Implementation:
	+ Iterates over `eval_metrics` dictionary with metric names as keys and values.
	+ Calls `summary_writer.scalar` with formatted metric name (`"eval_<metric_name>"`) and value, tagged at the specified `step`.
* Dependencies:
	+ `summary_writer: object` (interface expected to have a `scalar` method)
	+ `eval_metrics: dict` containing evaluation metrics

**Key Insights:**

* Uses dictionary iteration to process evaluation metrics.
* Formats metric name using string concatenation (`"eval_<metric_name>"`).
* Tags each metric with the specified `step`.


key: chroma:document
string_value:
CODE:

def create_learning_rate_fn(
    train_ds_size: int, train_batch_size: int, num_train_epochs: int, num_warmup_steps: int, learning_rate: float
) -> Callable[[int], jnp.ndarray]:
    """Returns a linear warmup, linear_decay learning rate function."""
    steps_per_epoch = train_ds_size // train_batch_size
    num_train_steps = steps_per_epoch * num_train_epochs
    warmup_fn = optax.linear_schedule(init_value=0.0, end_value=learning_rate, transition_steps=num_warmup_steps)
    decay_fn = optax.linear_schedule(
        init_value=learning_rate, end_value=0, transition_steps=num_train_steps - num_warmup_steps
    )
    schedule_fn = optax.join_schedules(schedules=[warmup_fn, decay_fn], boundaries=[num_warmup_steps])


SUMMARY:
**Function Summary: `create_learning_rate_fn`**

**Purpose and Functionality:**
The `create_learning_rate_fn` function generates a linear warmup, linear_decay learning rate schedule. It takes in parameters for training data size, batch size, number of epochs, number of warm-up steps, and the initial learning rate.

**Key Implementation Details:**

* Uses PyTorch's Optax with linear schedules to implement warm-up and decay phases.
* Warm-up phase lasts for `num_warmup_steps` with an initial value of 0.0 and a final value equal to the `learning_rate`.
* Decay phase lasts for the remaining number of training steps (`num_train_epochs * train_batch_size`) after warm-up phase.

**Notable Dependencies and Requirements:**

* `optax`: A library for optimization algorithms.
* `jnp.ndarray`: An array representation, likely used for tensor computations ( NumPy equivalent in JAX/NumPy).
* `Callable[[int], jnp.ndarray]`: The return type hint implies that the function is a callable that takes an integer argument and returns a JAX-based NumPy array.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_clm", model_args, data_args, framework="flax")

    if (
        os.path.exists(training_args.output_dir)
        and os.listdir(training_args.output_dir)
        and training_args.do_train
        and not training_args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({training_args.output_dir}) already exists and is not empty. "
            "Use --overwrite_output_dir to overcome."
        )

    # Make one log on every process with the configuration for debugging.
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO,
    )
    # Setup logging, we only want one process per machine to log things on the screen.
    logger.setLevel(logging.INFO if jax.process_index() == 0 else logging.ERROR)
    if jax.process_index() == 0:
        datasets.utils.logging.set_verbosity_warning()
        transformers.utils.logging.set_verbosity_info()
    else:
        datasets.utils.logging.set_verbosity_error()
        transformers.utils.logging.set_verbosity_error()

    # Set the verbosity to info of the Transformers logger (on main process only):
    logger.info(f"Training/evaluation parameters {training_args}")

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Handle the repository creation
    if training_args.push_to_hub:
        # Retrieve of infer repo_name
        repo_name = training_args.hub_model_id
        if repo_name is None:
            repo_name = Path(training_args.output_dir).absolute().name
        # Create repo and retrieve repo_id
        api = HfApi()
        repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id

    #  Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).
    #
    # In distributed training, the load_dataset function guarantees that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        dataset = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            keep_in_memory=False,
            token=model_args.token,
            num_proc=data_args.preprocessing_num_workers,
            trust_remote_code=model_args.trust_remote_code,
        )

        if "validation" not in dataset.keys():
            dataset["validation"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[:{data_args.validation_split_percentage}%]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
                trust_remote_code=model_args.trust_remote_code,
            )
            dataset["train"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[{data_args.validation_split_percentage}%:]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
                trust_remote_code=model_args.trust_remote_code,
            )
    else:
        data_files = {}
        dataset_args = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
            dataset_args["keep_linebreaks"] = data_args.keep_linebreaks
        dataset = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            **dataset_args,
            token=model_args.token,
            num_proc=data_args.preprocessing_num_workers,
        )

        if "validation" not in dataset.keys():
            dataset["validation"] = load_dataset(
                extension,
                data_files=data_files,
                split=f"train[:{data_args.validation_split_percentage}%]",
                cache_dir=model_args.cache_dir,
                **dataset_args,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
            )
            dataset["train"] = load_dataset(
                extension,
                data_files=data_files,
                split=f"train[{data_args.validation_split_percentage}%:]",
                cache_dir=model_args.cache_dir,
                **dataset_args,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
            )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    # Load pretrained model and tokenizer

    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    if model_args.config_name:
        config = AutoConfig.from_pretrained(
            model_args.config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    elif model_args.model_name_or_path:
        config = AutoConfig.from_pretrained(
            model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        config = CONFIG_MAPPING[model_args.model_type]()
        logger.warning("You are instantiating a new config instance from scratch.")

    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.tokenizer_name,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script. "
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )

    if model_args.model_name_or_path:
        model = FlaxAutoModelForCausalLM.from_pretrained(
            model_args.model_name_or_path,
            config=config,
            seed=training_args.seed,
            dtype=getattr(jnp, model_args.dtype),
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        model = FlaxAutoModelForCausalLM.from_config(
            config,
            seed=training_args.seed,
            dtype=getattr(jnp, model_args.dtype),
            trust_remote_code=model_args.trust_remote_code,
        )

    # Preprocessing the datasets.
    # First we tokenize all the texts.
    if training_args.do_train:
        column_names = dataset["train"].column_names
    else:
        column_names = dataset["validation"].column_names
    text_column_name = "text" if "text" in column_names else column_names[0]

    # since this will be pickled to avoid _LazyModule error in Hasher force logger loading before tokenize_function
    tok_logger = transformers.utils.logging.get_logger("transformers.tokenization_utils_base")

    def tokenize_function(examples):
        with CaptureLogger(tok_logger) as cl:
            output = tokenizer(examples[text_column_name])
        # clm input could be much much longer than block_size
        if "Token indices sequence length is longer than the" in cl.out:
            tok_logger.warning(
                "^^^^^^^^^^^^^^^^ Please ignore the warning above - this long input will be chunked into smaller bits"
                " before being passed to the model."
            )
        return output

    tokenized_datasets = dataset.map(
        tokenize_function,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not data_args.overwrite_cache,
    )

    if data_args.block_size is None:
        block_size = tokenizer.model_max_length
        if block_size > config.max_position_embeddings:
            logger.warning(
                f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). "
                f"Using block_size={min(1024, config.max_position_embeddings)} instead. You can change that default value by passing --block_size xxx."
            )
            block_size = min(1024, config.max_position_embeddings)
    else:
        if data_args.block_size > tokenizer.model_max_length:
            logger.warning(
                f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model "
                f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}."
            )
        block_size = min(data_args.block_size, tokenizer.model_max_length)

    # Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size.
    def group_texts(examples):
        # Concatenate all texts.
        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
        total_length = len(concatenated_examples[list(examples.keys())[0]])
        # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
        # customize this part to your needs.
        if total_length >= block_size:
            total_length = (total_length // block_size) * block_size
        # Split by chunks of max_len.
        result = {
            k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
            for k, t in concatenated_examples.items()
        }
        result["labels"] = result["input_ids"].copy()
        return result

    # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder
    # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower
    # to preprocess.
    #
    # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:
    # https://huggingface.co/docs/datasets/process#map

    lm_datasets = tokenized_datasets.map(
        group_texts,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        load_from_cache_file=not data_args.overwrite_cache,
    )

    if training_args.do_train:
        if "train" not in tokenized_datasets:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = lm_datasets["train"]
        if data_args.max_train_samples is not None:
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))

    if training_args.do_eval:
        if "validation" not in tokenized_datasets:
            raise ValueError("--do_eval requires a validation dataset")
        eval_dataset = lm_datasets["validation"]
        if data_args.max_eval_samples is not None:
            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
            eval_dataset = eval_dataset.select(range(max_eval_samples))

    # Enable tensorboard only on the master node
    has_tensorboard = is_tensorboard_available()
    if has_tensorboard and jax.process_index() == 0:
        try:
            from flax.metrics.tensorboard import SummaryWriter

            summary_writer = SummaryWriter(log_dir=Path(training_args.output_dir))
        except ImportError as ie:
            has_tensorboard = False
            logger.warning(
                f"Unable to display metrics through TensorBoard because some package are not installed: {ie}"
            )
    else:
        logger.warning(
            "Unable to display metrics through TensorBoard because the package is not installed: "
            "Please run pip install tensorboard to enable."
        )

    # Initialize our training
    rng = jax.random.PRNGKey(training_args.seed)
    rng, dropout_rng = jax.random.split(rng)

    # Store some constant
    num_epochs = int(training_args.num_train_epochs)
    train_batch_size = int(training_args.per_device_train_batch_size) * jax.device_count()
    per_device_eval_batch_size = int(training_args.per_device_eval_batch_size)
    eval_batch_size = per_device_eval_batch_size * jax.device_count()
    steps_per_epoch = len(train_dataset) // train_batch_size
    total_train_steps = steps_per_epoch * num_epochs

    # Create learning rate schedule
    linear_decay_lr_schedule_fn = create_learning_rate_fn(
        len(train_dataset),
        train_batch_size,
        training_args.num_train_epochs,
        training_args.warmup_steps,
        training_args.learning_rate,
    )

    # We use Optax's "masking" functionality to not apply weight decay
    # to bias and LayerNorm scale parameters. decay_mask_fn returns a
    # mask boolean with the same structure as the parameters.
    # The mask is True for parameters that should be decayed.
    def decay_mask_fn(params):
        flat_params = traverse_util.flatten_dict(params)
        # find out all LayerNorm parameters
        layer_norm_candidates = ["layernorm", "layer_norm", "ln"]
        layer_norm_named_params = {
            layer[-2:]
            for layer_norm_name in layer_norm_candidates
            for layer in flat_params.keys()
            if layer_norm_name in "".join(layer).lower()
        }
        flat_mask = {path: (path[-1] != "bias" and path[-2:] not in layer_norm_named_params) for path in flat_params}
        return traverse_util.unflatten_dict(flat_mask)

    # create adam optimizer
    if training_args.adafactor:
        # We use the default parameters here to initialize adafactor,
        # For more details about the parameters please check https://github.com/deepmind/optax/blob/ed02befef9bf81cbbf236be3d2b0e032e9ed4a40/optax/_src/alias.py#L74
        optimizer = optax.adafactor(
            learning_rate=linear_decay_lr_schedule_fn,
        )
    else:
        optimizer = optax.adamw(
            learning_rate=linear_decay_lr_schedule_fn,
            b1=training_args.adam_beta1,
            b2=training_args.adam_beta2,
            eps=training_args.adam_epsilon,
            weight_decay=training_args.weight_decay,
            mask=decay_mask_fn,
        )

    # Setup train state
    state = TrainState.create(apply_fn=model.__call__, params=model.params, tx=optimizer, dropout_rng=dropout_rng)

    def loss_fn(logits, labels):
        shift_logits = logits[..., :-1, :]
        shift_labels = labels[..., 1:]
        loss = optax.softmax_cross_entropy(shift_logits, onehot(shift_labels, shift_logits.shape[-1]))
        return loss.mean()

    # Define gradient update step fn
    def train_step(state, batch):
        dropout_rng, new_dropout_rng = jax.random.split(state.dropout_rng)

        def compute_loss(params):
            labels = batch.pop("labels")
            logits = state.apply_fn(**batch, params=params, dropout_rng=dropout_rng, train=True)[0]
            loss = loss_fn(logits, labels)
            return loss

        grad_fn = jax.value_and_grad(compute_loss)
        loss, grad = grad_fn(state.params)
        grad = jax.lax.pmean(grad, "batch")

        new_state = state.apply_gradients(grads=grad, dropout_rng=new_dropout_rng)

        metrics = {"loss": loss, "learning_rate": linear_decay_lr_schedule_fn(state.step)}
        metrics = jax.lax.pmean(metrics, axis_name="batch")

        return new_state, metrics

    # Define eval fn
    def eval_step(params, batch):
        labels = batch.pop("labels")
        logits = model(**batch, params=params, train=False)[0]
        loss = loss_fn(logits, labels)

        # summarize metrics
        metrics = {"loss": loss}
        metrics = jax.lax.pmean(metrics, axis_name="batch")
        return metrics

    # Create parallel version of the train and eval step
    p_train_step = jax.pmap(train_step, "batch", donate_argnums=(0,))
    p_eval_step = jax.pmap(eval_step, "batch")

    # Replicate the train state on each device
    state = state.replicate()

    logger.info("***** Running training *****")
    logger.info(f"  Num examples = {len(train_dataset)}")
    logger.info(f"  Num Epochs = {num_epochs}")
    logger.info(f"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}")
    logger.info(f"  Total train batch size (w. parallel & distributed) = {train_batch_size}")
    logger.info(f"  Total optimization steps = {total_train_steps}")

    train_time = 0
    train_metrics = []
    epochs = tqdm(range(num_epochs), desc="Epoch ... ", position=0)
    for epoch in epochs:
        # ======================== Training ================================
        train_start = time.time()

        # Create sampling rng
        rng, input_rng = jax.random.split(rng)

        # Generate an epoch by shuffling sampling indices from the train dataset
        train_loader = data_loader(input_rng, train_dataset, train_batch_size, shuffle=True)
        steps_per_epoch = len(train_dataset) // train_batch_size
        # train
        for step in tqdm(range(steps_per_epoch), desc="Training...", position=1, leave=False):
            batch = next(train_loader)
            batch = shard(batch)
            state, train_metric = p_train_step(state, batch)
            train_metrics.append(train_metric)

            cur_step = epoch * (len(train_dataset) // train_batch_size) + step

            if cur_step % training_args.logging_steps == 0 and cur_step > 0:
                # Save metrics
                train_metric = unreplicate(train_metric)
                train_time += time.time() - train_start
                if has_tensorboard and jax.process_index() == 0:
                    write_train_metric(summary_writer, train_metrics, train_time, cur_step)

                epochs.write(
                    f"Step... ({cur_step} | Loss: {train_metric['loss'].mean()}, Learning Rate:"
                    f" {train_metric['learning_rate'].mean()})"
                )

                train_metrics = []

            if cur_step % training_args.eval_steps == 0 and cur_step > 0:
                # ======================== Evaluating ==============================
                eval_metrics = []
                eval_loader = data_loader(input_rng, eval_dataset, eval_batch_size, drop_last=False)
                eval_steps = math.ceil(len(eval_dataset) / eval_batch_size)
                for _ in tqdm(range(eval_steps), desc="Evaluating...", position=2, leave=False):
                    # Model forward
                    batch = next(eval_loader)
                    metrics = pad_shard_unpad(p_eval_step, static_return=True)(
                        state.params, batch, min_device_batch=per_device_eval_batch_size
                    )
                    eval_metrics.append(metrics)

                # normalize eval metrics
                eval_metrics = get_metrics(eval_metrics)
                eval_metrics = jax.tree_util.tree_map(jnp.mean, eval_metrics)

                try:
                    eval_metrics["perplexity"] = math.exp(eval_metrics["loss"])
                except OverflowError:
                    eval_metrics["perplexity"] = float("inf")

                # Print metrics and update progress bar
                desc = (
                    f"Step... ({cur_step} | Eval Loss: {eval_metrics['loss']} | Eval Perplexity:"
                    f" {eval_metrics['perplexity']})"
                )
                epochs.write(desc)
                epochs.desc = desc

                # Save metrics
                if has_tensorboard and jax.process_index() == 0:
                    write_eval_metric(summary_writer, eval_metrics, cur_step)

            if cur_step % training_args.save_steps == 0 and cur_step > 0:
                # save checkpoint after each epoch and push checkpoint to the hub
                if jax.process_index() == 0:
                    params = jax.device_get(unreplicate(state.params))
                    model.save_pretrained(training_args.output_dir, params=params)
                    tokenizer.save_pretrained(training_args.output_dir)
                    if training_args.push_to_hub:
                        api.upload_folder(
                            commit_message=f"Saving weights and logs of step {cur_step}",
                            folder_path=training_args.output_dir,
                            repo_id=repo_id,
                            repo_type="model",
                            token=training_args.hub_token,
                        )
    # Eval after training
    if training_args.do_eval:
        eval_metrics = []
        eval_loader = data_loader(input_rng, eval_dataset, eval_batch_size, drop_last=False)
        eval_steps = math.ceil(len(eval_dataset) / eval_batch_size)
        for _ in tqdm(range(eval_steps), desc="Evaluating...", position=2, leave=False):
            # Model forward
            batch = next(eval_loader)
            metrics = pad_shard_unpad(p_eval_step, static_return=True)(
                state.params, batch, min_device_batch=per_device_eval_batch_size
            )
            eval_metrics.append(metrics)

        # normalize eval metrics
        eval_metrics = get_metrics(eval_metrics)
        eval_metrics = jax.tree_util.tree_map(lambda x: jnp.mean(x).item(), eval_metrics)

        try:
            eval_metrics["perplexity"] = math.exp(eval_metrics["loss"])
        except OverflowError:
            eval_metrics["perplexity"] = float("inf")

        if jax.process_index() == 0:
            eval_metrics = {f"eval_{metric_name}": value for metric_name, value in eval_metrics.items()}
            path = os.path.join(training_args.output_dir, "eval_results.json")
            with open(path, "w") as f:


SUMMARY:
Here is a concise summary of the code:

This code implements a simple training loop for a language model using TensorFlow 2.x. The model takes an input sequence and outputs a probability distribution over the next possible token in the sequence.

The training process involves:

1. Generating an epoch by shuffling sampling indices from the train dataset.
2. For each step, generating a batch of data, calling the `p_train_step` function (for training) or `p_eval_step` function (for evaluation).
3. Saving metrics and logs to TensorBoard at regular intervals (logging_steps) during training.

The code also saves checkpoints after each epoch and pushes them to the Hub if specified in the configuration.

Some key details:

* The model uses a sampling-based approach, where the next token is sampled from the probability distribution over all possible tokens.
* The model takes an input sequence of length `input_len`, which is truncated and padded to a fixed batch size (`batch_size`) for training.
* The hyperparameters are defined in the configuration object (`training_args`), including model architecture, learning rate schedule, batch size, etc.

Overall, this code provides a basic framework for training a language model using TensorFlow 2.x.


key: chroma:document
string_value:
CODE:

class SentencePieceUnigramTokenizer(BaseTokenizer):
    """
    This class is a copy of `DeDLOC's tokenizer implementation <https://github.com/yandex-research/DeDLOC/blob/main/sahajbert/tokenizer/tokenizer_model.py>`__ .

    Custom SentencePiece Unigram Tokenizer with NMT, NKFC, spaces and lower-casing characters normalization
    Represents the Unigram algorithm, with the pretokenization used by SentencePiece
    """

    def __init__(
        self,
        replacement: str = "▁",
        add_prefix_space: bool = True,
        unk_token: Union[str, AddedToken] = "<unk>",
        eos_token: Union[str, AddedToken] = "</s>",
        pad_token: Union[str, AddedToken] = "<pad>",
    ):
        self.special_tokens = {
            "pad": {"id": 0, "token": pad_token},
            "eos": {"id": 1, "token": eos_token},
            "unk": {"id": 2, "token": unk_token},
        }

        self.special_tokens_list = [None] * len(self.special_tokens)
        for token_dict in self.special_tokens.values():
            self.special_tokens_list[token_dict["id"]] = token_dict["token"]

        tokenizer = Tokenizer(Unigram())

        tokenizer.normalizer = normalizers.Sequence(
            [
                normalizers.Nmt(),
                normalizers.NFKC(),
                normalizers.Replace(Regex(" {2,}"), " "),
                normalizers.Lowercase(),
            ]
        )
        tokenizer.pre_tokenizer = pre_tokenizers.Sequence(
            [
                pre_tokenizers.Metaspace(
                    replacement=replacement, prepend_scheme="always" if add_prefix_space else "never"
                ),
                pre_tokenizers.Digits(individual_digits=True),
                pre_tokenizers.Punctuation(),
            ]
        )
        tokenizer.decoder = decoders.Metaspace(
            replacement=replacement, prepend_scheme="always" if add_prefix_space else "never"
        )

        tokenizer.post_processor = TemplateProcessing(
            single=f"$A {self.special_tokens['eos']['token']}",
            special_tokens=[(self.special_tokens["eos"]["token"], self.special_tokens["eos"]["id"])],
        )

        parameters = {
            "model": "SentencePieceUnigram",
            "replacement": replacement,
            "add_prefix_space": add_prefix_space,
        }

        super().__init__(tokenizer, parameters)

    def train(
        self,
        files: Union[str, List[str]],
        vocab_size: int = 8000,
        show_progress: bool = True,
    ):
        """Train the model using the given files"""

        trainer = trainers.UnigramTrainer(
            vocab_size=vocab_size,
            special_tokens=self.special_tokens_list,
            show_progress=show_progress,
        )

        if isinstance(files, str):
            files = [files]
        self._tokenizer.train(files, trainer=trainer)

        self.add_unk_id()

    def train_from_iterator(
        self,
        iterator: Union[Iterator[str], Iterator[Iterator[str]]],
        vocab_size: int = 8000,
        show_progress: bool = True,
    ):
        """Train the model using the given iterator"""

        trainer = trainers.UnigramTrainer(
            vocab_size=vocab_size,
            special_tokens=self.special_tokens_list,
            show_progress=show_progress,
        )

        self._tokenizer.train_from_iterator(iterator, trainer=trainer)

        self.add_unk_id()

    def add_unk_id(self):
        tokenizer_json = json.loads(self._tokenizer.to_str())

        tokenizer_json["model"]["unk_id"] = self.special_tokens["unk"]["id"]



SUMMARY:
**Class Summary: `SentencePieceUnigramTokenizer`**

A custom SentencePiece Unigram Tokenizer with NMT, NKFC, spaces and lower-casing characters normalization.

**Primary Purpose**: Represent the Unigram algorithm with pretokenization used by SentencePiece.

**Implementation Details**:

* Utilizes a Sequence of normalizers (NMT, NKFC, space replacement, lowercase) and pre- and post-tokenizers (Metaspace, digits, punctuation).
* Offers train modes: `train` (files list), `train_from_iterator` (iterator), and adds an 'unk' ID.

**Notable Dependencies/Requirements**:

* References `Unigram`, `Tokenizer`, `normalizers`, `pre_tokenizers`, `decoders`, `TemplateProcessing`, and `AddedToken`.
* Inherited from `BaseTokenizer`.


key: chroma:document
string_value:
CODE:

def generate_batch_splits(samples_idx: np.ndarray, batch_size: int, drop_last=True) -> np.ndarray:
    """Generate batches of data for a specified batch size from sample indices. If the dataset size is not divisible by
    the batch size and `drop_last` is `True`, the last incomplete batch is dropped. Else, it is returned."""
    num_samples = len(samples_idx)
    if drop_last:
        samples_to_remove = num_samples % batch_size
        if samples_to_remove != 0:
            samples_idx = samples_idx[:-samples_to_remove]
        sections_split = num_samples // batch_size
        samples_idx = samples_idx.reshape((sections_split, batch_size))
    else:
        sections_split = math.ceil(num_samples / batch_size)
        samples_idx = np.array_split(samples_idx, sections_split)


SUMMARY:
**Summary**

The `generate_batch_splits` function generates batches of data from sample indices in two modes:

*   **Mode 1: Dropping incomplete last batch**
    *   Purpose: Split dataset into complete batches based on the specified `batch_size`.
    *   Key Implementation Details: Uses NumPy's array slicing and reshaping to achieve this.
    *   Notable Dependencies:
        *   `math`: for `ceil` function
    *   Return Value:
        *   A 2D NumPy array representing batches of data with dimension `(sections_split, batch_size)`.
*   **Mode 2: Returning incomplete last batch**
    *   Purpose: Split dataset into batches based on the specified `batch_size`, returning the last incomplete batch as well.
    *   Key Implementation Details: Utilizes `np.array_split` to split the array of sample indices.
    *   Notable Dependencies:
        *   `numpy`: for using `array_split` function
        *   `math`: for `ceil` function
    *   Return Value:
        *   A list of 2D NumPy arrays, each representing a batch of data.

**Key Method Signature:**
```python
def generate_batch_splits(samples_idx: np.ndarray, batch_size: int, drop_last=True) -> np.ndarray:
```

**Note:** The return type is specified as `np.ndarray` to indicate the return value is a NumPy array. However, in Mode 2, it actually returns an array of arrays (`list`) if `drop_last` is set to `False`.


key: chroma:document
string_value:
CODE:

def write_train_metric(summary_writer, train_metrics, train_time, step):
    summary_writer.scalar("train_time", train_time, step)

    train_metrics = get_metrics(train_metrics)
    for key, vals in train_metrics.items():
        tag = f"train_{key}"
        for i, val in enumerate(vals):


SUMMARY:
**Concise Summary**

```markdown
### Function: `write_train_metric`

#### Purpose:
Write training metrics to a summary writer.

#### Parameters:

* `summary_writer`: object for writing scalar tags
* `train_metrics`: dictionary of train metric values
* `train_time`: float representing average training time
* `step`: integer representing iteration step

#### Operations:

1. Writes average training time to `summary_writer` as "train_time".
2. Processes trained metrics:
	* Calls `get_metrics` to refine metrics.
	* Writes each value in `train_metrics[key]` to a scalar tag in the format "train_<key>_(i+1)" where i is the enum index.

#### Notes:

- Assumes existence of `summary_writer.scalar`, `get_metrics`, and related classes/interfaces.
```

This analysis breaks down the function into its main operation, parameters, and notes on dependencies and functionality. The focus remains on the core purpose and necessary details for understanding the code's intent and behavior.


key: chroma:document
string_value:
CODE:

def write_eval_metric(summary_writer, eval_metrics, step):
    for metric_name, value in eval_metrics.items():


SUMMARY:
**Function Summary: `write_eval_metric`**

**Purpose and Functionality:**
The `write_eval_metric` function writes evaluation metrics to a summary writer at a given step. It takes in an object (`summary_writer`) responsible for writing scalar values, a dictionary of evaluated metrics (with metric names as keys), and the current step number.

**Key Implementation Details:**

* Iterates over each metric in `eval_metrics`, creating a scalar value with the format "eval_<metric_name>" and writes it to the summary writer at the specified step.

**Notable Dependencies or Requirements:**

* `summary_writer` object with a `scalar` method
* External dependency: no known dependencies mentioned

**Interface/Hierarchy:** None specified, but likely part of a larger library/framework for summarizing evaluations.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_mlm", model_args, data_args, framework="flax")

    if (
        os.path.exists(training_args.output_dir)
        and os.listdir(training_args.output_dir)
        and training_args.do_train
        and not training_args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({training_args.output_dir}) already exists and is not empty. "
            "Use --overwrite_output_dir to overcome."
        )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        level=logging.INFO,
        datefmt="[%X]",
    )

    # Log on each process the small summary:
    logger = logging.getLogger(__name__)

    # Set the verbosity to info of the Transformers logger (on main process only):
    logger.info(f"Training/evaluation parameters {training_args}")

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Handle the repository creation
    if training_args.push_to_hub:
        # Retrieve of infer repo_name
        repo_name = training_args.hub_model_id
        if repo_name is None:
            repo_name = Path(training_args.output_dir).absolute().name
        # Create repo and retrieve repo_id
        api = HfApi()
        repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id

    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).
    #
    # In distributed training, the load_dataset function guarantees that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        datasets = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            num_proc=data_args.preprocessing_num_workers,
            trust_remote_code=model_args.trust_remote_code,
        )

        if "validation" not in datasets.keys():
            datasets["validation"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[:{data_args.validation_split_percentage}%]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
                trust_remote_code=model_args.trust_remote_code,
            )
            datasets["train"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[{data_args.validation_split_percentage}%:]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
                trust_remote_code=model_args.trust_remote_code,
            )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
        datasets = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            num_proc=data_args.preprocessing_num_workers,
        )

        if "validation" not in datasets.keys():
            datasets["validation"] = load_dataset(
                extension,
                data_files=data_files,
                split=f"train[:{data_args.validation_split_percentage}%]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
            )
            datasets["train"] = load_dataset(
                extension,
                data_files=data_files,
                split=f"train[{data_args.validation_split_percentage}%:]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
            )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    # Load pretrained model and tokenizer

    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    if model_args.config_name:
        config = AutoConfig.from_pretrained(
            model_args.config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    elif model_args.model_name_or_path:
        config = AutoConfig.from_pretrained(
            model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        config = CONFIG_MAPPING[model_args.model_type]()
        logger.warning("You are instantiating a new config instance from scratch.")

    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.tokenizer_name,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script. "
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )

    # Preprocessing the datasets.
    # First we tokenize all the texts.
    if training_args.do_train:
        column_names = datasets["train"].column_names
    else:
        column_names = datasets["validation"].column_names
    text_column_name = "text" if "text" in column_names else column_names[0]

    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    if data_args.line_by_line:
        # When using line_by_line, we just tokenize each nonempty line.
        padding = "max_length" if data_args.pad_to_max_length else False

        def tokenize_function(examples):
            # Remove empty lines
            examples = [line for line in examples if len(line) > 0 and not line.isspace()]
            return tokenizer(
                examples,
                return_special_tokens_mask=True,
                padding=padding,
                truncation=True,
                max_length=max_seq_length,
            )

        tokenized_datasets = datasets.map(
            tokenize_function,
            input_columns=[text_column_name],
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    else:
        # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts.
        # We use `return_special_tokens_mask=True` because DataCollatorForLanguageModeling (see below) is more
        # efficient when it receives the `special_tokens_mask`.
        def tokenize_function(examples):
            return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

        tokenized_datasets = datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
        )

        # Main data processing function that will concatenate all texts from our dataset and generate chunks of
        # max_seq_length.
        def group_texts(examples):
            # Concatenate all texts.
            concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
            total_length = len(concatenated_examples[list(examples.keys())[0]])
            # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
            # customize this part to your needs.
            if total_length >= max_seq_length:
                total_length = (total_length // max_seq_length) * max_seq_length
            # Split by chunks of max_len.
            result = {
                k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]
                for k, t in concatenated_examples.items()
            }
            return result

        # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a
        # remainder for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value
        # might be slower to preprocess.
        #
        # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:
        # https://huggingface.co/docs/datasets/process#map
        tokenized_datasets = tokenized_datasets.map(
            group_texts,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    # Enable tensorboard only on the master node
    has_tensorboard = is_tensorboard_available()
    if has_tensorboard and jax.process_index() == 0:
        try:
            from flax.metrics.tensorboard import SummaryWriter

            summary_writer = SummaryWriter(log_dir=Path(training_args.output_dir))
        except ImportError as ie:
            has_tensorboard = False
            logger.warning(
                f"Unable to display metrics through TensorBoard because some package are not installed: {ie}"
            )
    else:
        logger.warning(
            "Unable to display metrics through TensorBoard because the package is not installed: "
            "Please run pip install tensorboard to enable."
        )

    # Data collator
    # This one will take care of randomly masking the tokens.
    data_collator = FlaxDataCollatorForLanguageModeling(tokenizer=tokenizer, mlm_probability=data_args.mlm_probability)

    # Initialize our training
    rng = jax.random.PRNGKey(training_args.seed)
    dropout_rngs = jax.random.split(rng, jax.local_device_count())

    if model_args.model_name_or_path:
        model = FlaxAutoModelForMaskedLM.from_pretrained(
            model_args.model_name_or_path,
            config=config,
            seed=training_args.seed,
            dtype=getattr(jnp, model_args.dtype),
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        model = FlaxAutoModelForMaskedLM.from_config(
            config,
            seed=training_args.seed,
            dtype=getattr(jnp, model_args.dtype),
            trust_remote_code=model_args.trust_remote_code,
        )

    if training_args.gradient_checkpointing:
        model.enable_gradient_checkpointing()

    # Store some constant
    num_epochs = int(training_args.num_train_epochs)
    train_batch_size = int(training_args.per_device_train_batch_size) * jax.device_count()
    per_device_eval_batch_size = int(training_args.per_device_eval_batch_size)
    eval_batch_size = per_device_eval_batch_size * jax.device_count()

    num_train_steps = len(tokenized_datasets["train"]) // train_batch_size * num_epochs

    # Create learning rate schedule
    warmup_fn = optax.linear_schedule(
        init_value=0.0, end_value=training_args.learning_rate, transition_steps=training_args.warmup_steps
    )
    decay_fn = optax.linear_schedule(
        init_value=training_args.learning_rate,
        end_value=0,
        transition_steps=num_train_steps - training_args.warmup_steps,
    )
    linear_decay_lr_schedule_fn = optax.join_schedules(
        schedules=[warmup_fn, decay_fn], boundaries=[training_args.warmup_steps]
    )

    # We use Optax's "masking" functionality to not apply weight decay
    # to bias and LayerNorm scale parameters. decay_mask_fn returns a
    # mask boolean with the same structure as the parameters.
    # The mask is True for parameters that should be decayed.
    def decay_mask_fn(params):
        flat_params = traverse_util.flatten_dict(params)
        # find out all LayerNorm parameters
        layer_norm_candidates = ["layernorm", "layer_norm", "ln"]
        layer_norm_named_params = {
            layer[-2:]
            for layer_norm_name in layer_norm_candidates
            for layer in flat_params.keys()
            if layer_norm_name in "".join(layer).lower()
        }
        flat_mask = {path: (path[-1] != "bias" and path[-2:] not in layer_norm_named_params) for path in flat_params}
        return traverse_util.unflatten_dict(flat_mask)

    # create adam optimizer
    if training_args.adafactor:
        # We use the default parameters here to initialize adafactor,
        # For more details about the parameters please check https://github.com/deepmind/optax/blob/ed02befef9bf81cbbf236be3d2b0e032e9ed4a40/optax/_src/alias.py#L74
        optimizer = optax.adafactor(
            learning_rate=linear_decay_lr_schedule_fn,
        )
    else:
        optimizer = optax.adamw(
            learning_rate=linear_decay_lr_schedule_fn,
            b1=training_args.adam_beta1,
            b2=training_args.adam_beta2,
            eps=training_args.adam_epsilon,
            weight_decay=training_args.weight_decay,
            mask=decay_mask_fn,
        )

    # Setup train state
    state = train_state.TrainState.create(apply_fn=model.__call__, params=model.params, tx=optimizer)

    # Define gradient update step fn
    def train_step(state, batch, dropout_rng):
        dropout_rng, new_dropout_rng = jax.random.split(dropout_rng)

        def loss_fn(params):
            labels = batch.pop("labels")

            logits = state.apply_fn(**batch, params=params, dropout_rng=dropout_rng, train=True)[0]

            # compute loss, ignore padded input tokens
            label_mask = jnp.where(labels > 0, 1.0, 0.0)
            loss = optax.softmax_cross_entropy(logits, onehot(labels, logits.shape[-1])) * label_mask

            # take average
            loss = loss.sum()
            num_labels = label_mask.sum()

            return loss, num_labels

        grad_fn = jax.value_and_grad(loss_fn, has_aux=True)
        (loss, num_labels), grad = grad_fn(state.params)
        num_labels = jax.lax.psum(num_labels, "batch")

        # true loss = total loss / total samples
        loss = jax.lax.psum(loss, "batch")
        loss = jax.tree_util.tree_map(lambda x: x / num_labels, loss)

        # true grad = total grad / total samples
        grad = jax.lax.psum(grad, "batch")
        grad = jax.tree_util.tree_map(lambda x: x / num_labels, grad)
        new_state = state.apply_gradients(grads=grad)

        metrics = {"loss": loss, "learning_rate": linear_decay_lr_schedule_fn(state.step)}

        return new_state, metrics, new_dropout_rng

    # Create parallel version of the train step
    p_train_step = jax.pmap(train_step, "batch", donate_argnums=(0,))

    # Define eval fn
    def eval_step(params, batch):
        labels = batch.pop("labels")

        logits = model(**batch, params=params, train=False)[0]

        # compute loss, ignore padded input tokens
        label_mask = jnp.where(labels > 0, 1.0, 0.0)
        loss = optax.softmax_cross_entropy(logits, onehot(labels, logits.shape[-1])) * label_mask

        # compute accuracy
        accuracy = jnp.equal(jnp.argmax(logits, axis=-1), labels) * label_mask

        # summarize metrics
        metrics = {"loss": loss.sum(), "accuracy": accuracy.sum(), "normalizer": label_mask.sum()}
        metrics = jax.lax.psum(metrics, axis_name="batch")

        return metrics

    p_eval_step = jax.pmap(eval_step, "batch", donate_argnums=(0,))

    # Replicate the train state on each device
    state = jax_utils.replicate(state)

    train_time = 0
    epochs = tqdm(range(num_epochs), desc=f"Epoch ... (1/{num_epochs})", position=0)
    for epoch in epochs:
        # ======================== Training ================================
        train_start = time.time()
        train_metrics = []

        # Create sampling rng
        rng, input_rng = jax.random.split(rng)

        # Generate an epoch by shuffling sampling indices from the train dataset
        num_train_samples = len(tokenized_datasets["train"])
        # Avoid using jax.numpy here in case of TPU training
        train_samples_idx = np.random.permutation(np.arange(num_train_samples))
        train_batch_idx = generate_batch_splits(train_samples_idx, train_batch_size)

        # Gather the indexes for creating the batch and do a training step
        for step, batch_idx in enumerate(tqdm(train_batch_idx, desc="Training...", position=1)):
            samples = [tokenized_datasets["train"][int(idx)] for idx in batch_idx]
            model_inputs = data_collator(samples, pad_to_multiple_of=16)

            # Model forward
            model_inputs = shard(model_inputs.data)
            state, train_metric, dropout_rngs = p_train_step(state, model_inputs, dropout_rngs)
            train_metrics.append(train_metric)

            cur_step = epoch * (num_train_samples // train_batch_size) + step

            if cur_step % training_args.logging_steps == 0 and cur_step > 0:
                # Save metrics
                train_metric = jax_utils.unreplicate(train_metric)
                train_time += time.time() - train_start
                if has_tensorboard and jax.process_index() == 0:
                    write_train_metric(summary_writer, train_metrics, train_time, cur_step)

                epochs.write(
                    f"Step... ({cur_step} | Loss: {train_metric['loss']}, Learning Rate:"
                    f" {train_metric['learning_rate']})"
                )

                train_metrics = []

            if cur_step % training_args.eval_steps == 0 and cur_step > 0:
                # ======================== Evaluating ==============================
                num_eval_samples = len(tokenized_datasets["validation"])
                # Avoid using jax.numpy here in case of TPU training
                eval_samples_idx = np.arange(num_eval_samples)
                eval_batch_idx = generate_batch_splits(eval_samples_idx, eval_batch_size, drop_last=False)

                eval_metrics = []
                for i, batch_idx in enumerate(tqdm(eval_batch_idx, desc="Evaluating ...", position=2)):
                    samples = [tokenized_datasets["validation"][int(idx)] for idx in batch_idx]
                    model_inputs = data_collator(samples, pad_to_multiple_of=16)

                    # Model forward
                    metrics = pad_shard_unpad(p_eval_step, static_return=True)(
                        state.params, model_inputs.data, min_device_batch=per_device_eval_batch_size
                    )
                    eval_metrics.append(metrics)

                # normalize eval metrics
                eval_metrics = get_metrics(eval_metrics)
                eval_metrics = jax.tree_util.tree_map(jnp.sum, eval_metrics)
                eval_normalizer = eval_metrics.pop("normalizer")
                eval_metrics = jax.tree_util.tree_map(lambda x: x / eval_normalizer, eval_metrics)

                # Update progress bar
                epochs.desc = f"Step... ({cur_step} | Loss: {eval_metrics['loss']}, Acc: {eval_metrics['accuracy']})"

                # Save metrics
                if has_tensorboard and jax.process_index() == 0:
                    write_eval_metric(summary_writer, eval_metrics, cur_step)

            if cur_step % training_args.save_steps == 0 and cur_step > 0:
                # save checkpoint after each epoch and push checkpoint to the hub
                if jax.process_index() == 0:
                    params = jax.device_get(jax.tree_util.tree_map(lambda x: x[0], state.params))
                    model.save_pretrained(training_args.output_dir, params=params)
                    tokenizer.save_pretrained(training_args.output_dir)
                    if training_args.push_to_hub:
                        api.upload_folder(
                            commit_message=f"Saving weights and logs of step {cur_step}",
                            folder_path=training_args.output_dir,
                            repo_id=repo_id,
                            repo_type="model",
                            token=training_args.hub_token,
                        )
    # Eval after training
    if training_args.do_eval:
        num_eval_samples = len(tokenized_datasets["validation"])
        # Avoid using jax.numpy here in case of TPU training
        eval_samples_idx = np.arange(num_eval_samples)
        eval_batch_idx = generate_batch_splits(eval_samples_idx, eval_batch_size, drop_last=False)

        eval_metrics = []
        for _, batch_idx in enumerate(tqdm(eval_batch_idx, desc="Evaluating ...", position=2)):
            samples = [tokenized_datasets["validation"][int(idx)] for idx in batch_idx]
            model_inputs = data_collator(samples, pad_to_multiple_of=16)

            # Model forward
            metrics = pad_shard_unpad(p_eval_step, static_return=True)(
                state.params, model_inputs.data, min_device_batch=per_device_eval_batch_size
            )
            eval_metrics.append(metrics)

        # normalize eval metrics
        eval_metrics = get_metrics(eval_metrics)
        eval_metrics = jax.tree_util.tree_map(lambda metric: jnp.sum(metric).item(), eval_metrics)
        eval_normalizer = eval_metrics.pop("normalizer")
        eval_metrics = jax.tree_util.tree_map(lambda x: x / eval_normalizer, eval_metrics)

        try:
            perplexity = math.exp(eval_metrics["loss"])
        except OverflowError:
            perplexity = float("inf")
        eval_metrics["perplexity"] = perplexity

        if jax.process_index() == 0:
            eval_metrics = {f"eval_{metric_name}": value for metric_name, value in eval_metrics.items()}
            path = os.path.join(training_args.output_dir, "eval_results.json")
            with open(path, "w") as f:


SUMMARY:
**Tuning and Training a Language Model**

This code snippet is used for training and fine-tuning a language model using Hugging Face's Transformers library. The script includes various hyperparameter tuning options and logging mechanisms.

**Key Features:**

*   **Hyperparameter Tuning:** The script allows users to adjust parameters such as learning rate, batch size, number of epochs, and evaluation frequency.
*   **Logging and Monitoring:** It tracks training progress through log files and saves important metrics at regular intervals during training and after finishing the exercise.

**How it Works:**

1.  **Initialization:** The script starts by loading necessary libraries including `transformers` and `training_args`.

2.  **Model Preparation:** A tokenizer is created from the input text files, and a pre-trained model is loaded into configuration and state dictionaries.

3.  **Training Loop:** The script defines four different sequences for training a sequence-to-sequence translator using the `train_step()` function:

    *   `forward_train`: trains a single step of the model
    *   `forward_eval`: evaluates one batch of the model (only used during evaluation)
    *   `forward_evaluate`: makes predictions on all input sequences in evaluation mode
    *   `forward_invalid_evaluate`: evaluates one input sequence when the corresponding output sequence does not exist

4.  **Logging and Saving:** After every specified number of training steps or at each eval step, the script writes metrics to a log file.

5.  **Saving Checkpoint:** A checkpoint is saved after every specified number of training steps and periodically throughout the entire process.

**Usage Example:**

```python
if __name__ == "__main__":
    main()
```

This example runs the `main()` function, which controls the execution of the script.


key: chroma:document
string_value:
CODE:

def compute_input_and_target_lengths(inputs_length, noise_density, mean_noise_span_length):
    """This function is copy of `random_spans_helper <https://github.com/google-research/text-to-text-transfer-transformer/blob/84f8bcc14b5f2c03de51bd3587609ba8f6bbd1cd/t5/data/preprocessors.py#L2466>`__ .

    Training parameters to avoid padding with random_spans_noise_mask.
    When training a model with random_spans_noise_mask, we would like to set the other
    training hyperparmeters in a way that avoids padding.
    This function helps us compute these hyperparameters.
    We assume that each noise span in the input is replaced by extra_tokens_per_span_inputs sentinel tokens,
    and each non-noise span in the targets is replaced by extra_tokens_per_span_targets sentinel tokens.
    This function tells us the required number of tokens in the raw example (for split_tokens())
    as well as the length of the encoded targets. Note that this function assumes
    the inputs and targets will have EOS appended and includes that in the reported length.

    Args:
        inputs_length: an integer - desired length of the tokenized inputs sequence
        noise_density: a float
        mean_noise_span_length: a float
    Returns:
        tokens_length: length of original text in tokens
        targets_length: an integer - length in tokens of encoded targets sequence
    """

    def _tokens_length_to_inputs_length_targets_length(tokens_length):
        num_noise_tokens = int(round(tokens_length * noise_density))
        num_nonnoise_tokens = tokens_length - num_noise_tokens
        num_noise_spans = int(round(num_noise_tokens / mean_noise_span_length))
        # inputs contain all nonnoise tokens, sentinels for all noise spans
        # and one EOS token.
        _input_length = num_nonnoise_tokens + num_noise_spans + 1
        _output_length = num_noise_tokens + num_noise_spans + 1
        return _input_length, _output_length

    tokens_length = inputs_length

    while _tokens_length_to_inputs_length_targets_length(tokens_length + 1)[0] <= inputs_length:
        tokens_length += 1

    inputs_length, targets_length = _tokens_length_to_inputs_length_targets_length(tokens_length)

    # minor hack to get the targets length to be equal to inputs length
    # which is more likely to have been set to a nice round number.
    if noise_density == 0.5 and targets_length > inputs_length:
        tokens_length -= 1
        targets_length -= 1


SUMMARY:
**Function Summary: `compute_input_and_target_lengths`**

### Purpose and Functionality

Computes desired input and target lengths for training a model with `random_spans_noise_mask`. Takes into account noise density and mean noise span length to avoid padding.

### Key Implementation Details

* Calculates required tokens in raw example (`tokens_length`) based on `inputs_length`, `noise_density`, and `mean_noise_span_length`.
* Computes actual input length by adding non-noise tokens, sentinel tokens for noise spans, and EOS token.
* Updates `tokens_length` until desired `inputs_length` is achieved.

### Notable Dependencies and Requirements

* `random_spans_helper`
* Assuming usage of `split_tokens` function
* External dependencies on `noise_density` and `mean_noise_span_length`

### Method Signatures and Interfaces

* `_tokens_length_to_inputs_length_targets_length.tokens_length()` returns computed `tokens_length`
* `_tokens_length_to_inputs_length_targets_length.inputs_length` and `targets_length` are returned as integer values.

Note: The method signature is generated based on the method name and its parameters.


key: chroma:document
string_value:
CODE:

def generate_batch_splits(samples_idx: np.ndarray, batch_size: int, drop_last=True) -> np.ndarray:
    """Generate batches of data for a specified batch size from sample indices. If the dataset size is not divisible by
    the batch size and `drop_last` is `True`, the last incomplete batch is dropped. Else, it is returned."""
    num_samples = len(samples_idx)
    if drop_last:
        samples_to_remove = num_samples % batch_size
        if samples_to_remove != 0:
            samples_idx = samples_idx[:-samples_to_remove]
        sections_split = num_samples // batch_size
        samples_idx = samples_idx.reshape((sections_split, batch_size))
    else:
        sections_split = math.ceil(num_samples / batch_size)
        samples_idx = np.array_split(samples_idx, sections_split)


SUMMARY:
**Summary**

`generate_batch_splits` function generates batches of data for a specified `batch_size` from sample indices. It takes three inputs:
- `samples_idx`: array-like object with sample indices
- `batch_size`: integer specifying batch size
- `drop_last`: boolean flag indicating whether to drop last incomplete batch

The primary purpose is to split the dataset into batches of equal size, while considering the case where `drop_last` is `True`.

**Key Implementation Details**

- Reshapes the input array `samples_idx` to `(sections_split, batch_size)` for efficient batching
- Uses `math.ceil()` when `drop_last` is `False`, ensuring all samples are utilized in batches

**Notable Dependencies and Requirements**

* No external dependencies listed
* Referenced using exact names:
  - `np.ndarray` (input/output)
  - `math` module with `ceil()` function


key: chroma:document
string_value:
CODE:

def write_train_metric(summary_writer, train_metrics, train_time, step):
    summary_writer.scalar("train_time", train_time, step)

    train_metrics = get_metrics(train_metrics)
    for key, vals in train_metrics.items():
        tag = f"train_{key}"
        for i, val in enumerate(vals):


SUMMARY:
**Concise Summary**

`write_train_metric` function:

* Writes scalar metrics to a `summary_writer`:
	+ trains on average (`"train_avg"`)
	+ train loss (`"train_loss"`), loss per epoch (`"loss_epoch"`), and loss per batch (`"loss_batch"`).
	+ time taken for each epoch (`"time_epoch"`) and step (`"time_step"`).

**Key Implementation Details**

* Calls `get_metrics` to update the metric data.
* Iterates over metrics, writing scalar values to `summary_writer`.

**Dependances/Requirements**

* `summary_writer`: instance of visualization writer (e.g. TensorBoard).
* `train_metrics`: dictionary of training metrics (e.g. loss, accuracy, etc.).
* External dependencies:
	+ `(get_metrics(func))`: function to update metric data.
	+ `(utils)`: likely containing utility functions for logging and visualization.

**Interface Implementation**

The function uses the `summary_writer` interface to write scalar metrics, leveraging its `scalar` method to record values at each step.


key: chroma:document
string_value:
CODE:

def write_eval_metric(summary_writer, eval_metrics, step):
    for metric_name, value in eval_metrics.items():


SUMMARY:
**Function Summary: `write_eval_metric`**

**Purpose**: Writes evaluation metrics to a scalar metric store.

**Functionality**: The function takes an evaluation metric dictionary (`eval_metrics`) and writes each key-value pair to the `summary_writer` as a scalar metric, prefixed with "eval_".

**Key Implementation Details:**
- Iterates over the items in `eval_metrics`.
- Writes each metric using `scalar(f"eval_{metric_name}", value, step)`.

**Notable Dependencies:**

* `summary_writer`: A writer for scalar metrics.
* `eval_metrics`: An dictionary of evaluation metrics (keys are metric names, values are metric values).
* `step`: The current training step number.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_t5_mlm", model_args, data_args, framework="flax")

    if (
        os.path.exists(training_args.output_dir)
        and os.listdir(training_args.output_dir)
        and training_args.do_train
        and not training_args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({training_args.output_dir}) already exists and is not empty. "
            "Use --overwrite_output_dir to overcome."
        )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
        level=logging.INFO,
        datefmt="[%X]",
    )

    # Log on each process the small summary:
    logger = logging.getLogger(__name__)

    # Set the verbosity to info of the Transformers logger (on main process only):
    logger.info(f"Training/evaluation parameters {training_args}")

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Handle the repository creation
    if training_args.push_to_hub:
        # Retrieve of infer repo_name
        repo_name = training_args.hub_model_id
        if repo_name is None:
            repo_name = Path(training_args.output_dir).absolute().name
        # Create repo and retrieve repo_id
        api = HfApi()
        repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id

    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        datasets = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            num_proc=data_args.preprocessing_num_workers,
            trust_remote_code=data_args.trust_remote_code,
        )

        if "validation" not in datasets.keys():
            datasets["validation"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[:{data_args.validation_split_percentage}%]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
                trust_remote_code=data_args.trust_remote_code,
            )
            datasets["train"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[{data_args.validation_split_percentage}%:]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
                trust_remote_code=data_args.trust_remote_code,
            )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
        datasets = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            num_proc=data_args.preprocessing_num_workers,
        )

        if "validation" not in datasets.keys():
            datasets["validation"] = load_dataset(
                extension,
                data_files=data_files,
                split=f"train[:{data_args.validation_split_percentage}%]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
            )
            datasets["train"] = load_dataset(
                extension,
                data_files=data_files,
                split=f"train[{data_args.validation_split_percentage}%:]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                num_proc=data_args.preprocessing_num_workers,
            )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    # Load pretrained model and tokenizer

    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.tokenizer_name,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
        )
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
        )
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script. "
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )

    if model_args.config_name:
        config = T5Config.from_pretrained(
            model_args.config_name,
            cache_dir=model_args.cache_dir,
            vocab_size=len(tokenizer),
            token=model_args.token,
        )
    elif model_args.model_name_or_path:
        config = T5Config.from_pretrained(
            model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
        )
    else:
        config = CONFIG_MAPPING[model_args.model_type]()
        logger.warning("You are instantiating a new config instance from scratch.")

    # Preprocessing the datasets.
    # First we tokenize all the texts.
    if training_args.do_train:
        column_names = datasets["train"].column_names
    else:
        column_names = datasets["validation"].column_names
    text_column_name = "text" if "text" in column_names else column_names[0]

    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts.
    # Since we make sure that all sequences are of the same length, no attention_mask is needed.
    def tokenize_function(examples):
        return tokenizer(examples[text_column_name], return_attention_mask=False)

    tokenized_datasets = datasets.map(
        tokenize_function,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not data_args.overwrite_cache,
    )

    # T5-like span masked language modeling will fuse consecutively masked tokens to a single sentinel token.
    # To ensure that the input length is `max_seq_length`, we need to increase the maximum length
    # according to `mlm_probability` and `mean_noise_span_length`. We can also define the label length accordingly.
    expanded_inputs_length, targets_length = compute_input_and_target_lengths(
        inputs_length=max_seq_length,
        noise_density=data_args.mlm_probability,
        mean_noise_span_length=data_args.mean_noise_span_length,
    )

    # Main data processing function that will concatenate all texts from our dataset and generate chunks of expanded_inputs_length.
    def group_texts(examples):
        # Concatenate all texts.
        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
        total_length = len(concatenated_examples[list(examples.keys())[0]])
        # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
        # customize this part to your needs.
        if total_length >= expanded_inputs_length:
            total_length = (total_length // expanded_inputs_length) * expanded_inputs_length
        # Split by chunks of max_len.
        result = {
            k: [t[i : i + expanded_inputs_length] for i in range(0, total_length, expanded_inputs_length)]
            for k, t in concatenated_examples.items()
        }
        return result

    # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a
    # remainder for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value
    # might be slower to preprocess.
    #
    # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:
    # https://huggingface.co/docs/datasets/process#map
    tokenized_datasets = tokenized_datasets.map(
        group_texts,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        load_from_cache_file=not data_args.overwrite_cache,
    )

    # Enable tensorboard only on the master node
    has_tensorboard = is_tensorboard_available()
    if has_tensorboard and jax.process_index() == 0:
        try:
            from flax.metrics.tensorboard import SummaryWriter

            summary_writer = SummaryWriter(log_dir=Path(training_args.output_dir))
        except ImportError as ie:
            has_tensorboard = False
            logger.warning(
                f"Unable to display metrics through TensorBoard because some package are not installed: {ie}"
            )
    else:
        logger.warning(
            "Unable to display metrics through TensorBoard because the package is not installed: "
            "Please run pip install tensorboard to enable."
        )

    # Initialize our training
    rng = jax.random.PRNGKey(training_args.seed)
    dropout_rngs = jax.random.split(rng, jax.local_device_count())

    if model_args.model_name_or_path:
        model = FlaxT5ForConditionalGeneration.from_pretrained(
            model_args.model_name_or_path,
            config=config,
            seed=training_args.seed,
            dtype=getattr(jnp, model_args.dtype),
            token=model_args.token,
        )
    else:
        config.vocab_size = len(tokenizer)
        model = FlaxT5ForConditionalGeneration(
            config,
            seed=training_args.seed,
            dtype=getattr(jnp, model_args.dtype),
        )

    # Data collator
    # This one will take care of randomly masking the tokens.
    data_collator = FlaxDataCollatorForT5MLM(
        tokenizer=tokenizer,
        noise_density=data_args.mlm_probability,
        mean_noise_span_length=data_args.mean_noise_span_length,
        input_length=max_seq_length,
        target_length=targets_length,
        pad_token_id=model.config.pad_token_id,
        decoder_start_token_id=model.config.decoder_start_token_id,
    )

    # Store some constant
    num_epochs = int(training_args.num_train_epochs)
    train_batch_size = int(training_args.per_device_train_batch_size) * jax.device_count()
    per_device_eval_batch_size = int(training_args.per_device_eval_batch_size)
    eval_batch_size = per_device_eval_batch_size * jax.device_count()

    num_train_steps = len(tokenized_datasets["train"]) // train_batch_size * num_epochs

    num_of_hosts = jax.process_count()
    current_host_idx = jax.process_index()

    # Create learning rate schedule
    warmup_fn = optax.linear_schedule(
        init_value=0.0, end_value=training_args.learning_rate, transition_steps=training_args.warmup_steps
    )
    decay_fn = optax.linear_schedule(
        init_value=training_args.learning_rate,
        end_value=0,
        transition_steps=num_train_steps - training_args.warmup_steps,
    )
    linear_decay_lr_schedule_fn = optax.join_schedules(
        schedules=[warmup_fn, decay_fn], boundaries=[training_args.warmup_steps]
    )

    # We use Optax's "masking" functionality to not apply weight decay
    # to bias and LayerNorm scale parameters. decay_mask_fn returns a
    # mask boolean with the same structure as the parameters.
    # The mask is True for parameters that should be decayed.
    def decay_mask_fn(params):
        flat_params = traverse_util.flatten_dict(params)
        # find out all LayerNorm parameters
        layer_norm_candidates = ["layernorm", "layer_norm", "ln"]
        layer_norm_named_params = {
            layer[-2:]
            for layer_norm_name in layer_norm_candidates
            for layer in flat_params.keys()
            if layer_norm_name in "".join(layer).lower()
        }
        flat_mask = {path: (path[-1] != "bias" and path[-2:] not in layer_norm_named_params) for path in flat_params}
        return traverse_util.unflatten_dict(flat_mask)

    # create adam optimizer
    if training_args.adafactor:
        # We use the default parameters here to initialize adafactor,
        # For more details about the parameters please check https://github.com/deepmind/optax/blob/ed02befef9bf81cbbf236be3d2b0e032e9ed4a40/optax/_src/alias.py#L74
        optimizer = optax.adafactor(
            learning_rate=linear_decay_lr_schedule_fn,
        )
    else:
        optimizer = optax.adamw(
            learning_rate=linear_decay_lr_schedule_fn,
            b1=training_args.adam_beta1,
            b2=training_args.adam_beta2,
            weight_decay=training_args.weight_decay,
            mask=decay_mask_fn,
        )

    # Setup train state
    state = train_state.TrainState.create(apply_fn=model.__call__, params=model.params, tx=optimizer)

    # Define gradient update step fn
    def train_step(state, batch, dropout_rng):
        dropout_rng, new_dropout_rng = jax.random.split(dropout_rng)

        def loss_fn(params):
            labels = batch.pop("labels")

            logits = state.apply_fn(**batch, params=params, dropout_rng=dropout_rng, train=True)[0]

            # compute loss
            loss = optax.softmax_cross_entropy(logits, onehot(labels, logits.shape[-1])).mean()

            return loss

        grad_fn = jax.value_and_grad(loss_fn)
        loss, grad = grad_fn(state.params)
        grad = jax.lax.pmean(grad, "batch")
        new_state = state.apply_gradients(grads=grad)

        metrics = jax.lax.pmean(
            {"loss": loss, "learning_rate": linear_decay_lr_schedule_fn(state.step)}, axis_name="batch"
        )

        return new_state, metrics, new_dropout_rng

    # Create parallel version of the train step
    p_train_step = jax.pmap(train_step, "batch", donate_argnums=(0,))

    # Define eval fn
    def eval_step(params, batch):
        labels = batch.pop("labels")

        logits = model(**batch, params=params, train=False)[0]

        # compute loss
        loss = optax.softmax_cross_entropy(logits, onehot(labels, logits.shape[-1]))

        # compute accuracy
        accuracy = jnp.equal(jnp.argmax(logits, axis=-1), labels)

        # summarize metrics
        metrics = {"loss": loss.mean(), "accuracy": accuracy.mean()}
        metrics = jax.lax.pmean(metrics, axis_name="batch")

        return metrics

    p_eval_step = jax.pmap(eval_step, "batch", donate_argnums=(0,))

    # Replicate the train state on each device
    state = jax_utils.replicate(state)

    train_time = 0
    epochs = tqdm(range(num_epochs), desc="Epoch ... ", position=0)
    for epoch in epochs:
        # ======================== Training ================================
        train_start = time.time()
        train_metrics = []

        # Create sampling rng
        rng, input_rng = jax.random.split(rng)

        # Generate an epoch by shuffling sampling indices from the train dataset
        num_train_samples = len(tokenized_datasets["train"])
        # Avoid using jax.numpy here in case of TPU training
        train_samples_idx = np.random.permutation(np.arange(num_train_samples))
        train_batch_idx = generate_batch_splits(train_samples_idx, train_batch_size)

        # Gather the indexes for creating the batch and do a training step
        for step, batch_idx in enumerate(tqdm(train_batch_idx, desc="Training...", position=1)):
            samples = [tokenized_datasets["train"][int(idx)] for idx in batch_idx]
            model_inputs = data_collator(samples)

            local_host_model_inputs = {
                key: np.split(model_inputs.data[key], num_of_hosts, axis=0)[current_host_idx]
                for key, value in model_inputs.data.items()
            }

            # Model forward
            model_inputs = shard(local_host_model_inputs)
            state, train_metric, dropout_rngs = p_train_step(state, model_inputs, dropout_rngs)
            train_metrics.append(train_metric)

            cur_step = epoch * (num_train_samples // train_batch_size) + step

            if cur_step % training_args.logging_steps == 0 and cur_step > 0:
                # Save metrics
                train_metric = jax_utils.unreplicate(train_metric)
                train_time += time.time() - train_start
                if has_tensorboard and jax.process_index() == 0:
                    write_train_metric(summary_writer, train_metrics, train_time, cur_step)

                epochs.write(
                    f"Step... ({cur_step} | Loss: {train_metric['loss'].mean()}, Learning Rate:"
                    f" {train_metric['learning_rate'].mean()})"
                )

                train_metrics = []

            if cur_step % training_args.eval_steps == 0 and cur_step > 0:
                # ======================== Evaluating ==============================
                num_eval_samples = len(tokenized_datasets["validation"])
                # Avoid using jax.numpy here in case of TPU training
                eval_samples_idx = np.arange(num_eval_samples)
                eval_batch_idx = generate_batch_splits(eval_samples_idx, eval_batch_size, drop_last=False)

                eval_metrics = []
                for i, batch_idx in enumerate(tqdm(eval_batch_idx, desc="Evaluating ...", position=2)):
                    samples = [tokenized_datasets["validation"][int(idx)] for idx in batch_idx]
                    model_inputs = data_collator(samples)

                    # Model forward
                    metrics = pad_shard_unpad(p_eval_step, static_return=True)(
                        state.params, model_inputs.data, min_device_batch=per_device_eval_batch_size
                    )
                    eval_metrics.append(metrics)

                # get eval metrics
                eval_metrics = get_metrics(eval_metrics)
                eval_metrics = jax.tree_util.tree_map(jnp.mean, eval_metrics)

                # Update progress bar
                epochs.write(f"Step... ({cur_step} | Loss: {eval_metrics['loss']}, Acc: {eval_metrics['accuracy']})")

                # Save metrics
                if has_tensorboard and jax.process_index() == 0:
                    write_eval_metric(summary_writer, eval_metrics, cur_step)

            if cur_step % training_args.save_steps == 0 and cur_step > 0:
                # save checkpoint after each epoch and push checkpoint to the hub
                if jax.process_index() == 0:
                    params = jax.device_get(jax.tree_util.tree_map(lambda x: x[0], state.params))
                    model.save_pretrained(training_args.output_dir, params=params)
                    tokenizer.save_pretrained(training_args.output_dir)
                    if training_args.push_to_hub:
                        api.upload_folder(
                            commit_message=f"Saving weights and logs of step {cur_step}",
                            folder_path=training_args.output_dir,
                            repo_id=repo_id,
                            repo_type="model",
                            token=training_args.hub_token,
                        )
    # Eval after training
    if training_args.do_eval:
        num_eval_samples = len(tokenized_datasets["validation"])
        # Avoid using jax.numpy here in case of TPU training
        eval_samples_idx = np.arange(num_eval_samples)
        eval_batch_idx = generate_batch_splits(eval_samples_idx, eval_batch_size, drop_last=False)

        eval_metrics = []
        for i, batch_idx in enumerate(tqdm(eval_batch_idx, desc="Evaluating ...", position=2)):
            samples = [tokenized_datasets["validation"][int(idx)] for idx in batch_idx]
            model_inputs = data_collator(samples)

            # Model forward
            metrics = pad_shard_unpad(p_eval_step, static_return=True)(
                state.params, model_inputs.data, min_device_batch=per_device_eval_batch_size
            )
            eval_metrics.append(metrics)

        # get eval metrics
        eval_metrics = get_metrics(eval_metrics)
        eval_metrics = jax.tree_util.tree_map(lambda metric: jnp.mean(metric).item(), eval_metrics)

        if jax.process_index() == 0:
            eval_metrics = {f"eval_{metric_name}": value for metric_name, value in eval_metrics.items()}
            path = os.path.join(training_args.output_dir, "eval_results.json")
            with open(path, "w") as f:


SUMMARY:
This is a PyTorch implementation of a neural network training script. The script trains a model on a large dataset (e.g., text classification), evaluates its performance on a validation set, and saves its weights and logs at regular intervals. 

Here's a high-level overview:

1. Data preparation: Load the dataset into memory and prepare it for use in the model.
2. Model training: Train the model using an optimizer and loss function. The model is updated after each batch completion.
3. Evaluation: Evaluate the model's performance on the validation set.
4. Logging and saving: Save the model's weights and logs at regular intervals, including the current epoch, step, and evaluation metrics.

The script is likely being used for training and evaluating a text classification model, as it uses data from the Hugging Face Transformers library and includes settings like batch size and learning rate.

Key components:

* `tokenizer` object: Used to preprocess input data.
* `model` object: A PyTorch neural network that takes in tokenized input data.
* `optimizer`, `loss_fn`: Optimization algorithm and loss function used for model training.
* `device_utils`: Library used for handling device-specific operations (e.g., moving data to device).
* `training_args`: An object containing configuration parameters, such as batch size, learning rate, number of epochs, etc.

The script is well-structured and follows standard PyTorch best practices. It's likely been tested with JAX, a library used for parallel computation in PyTorch, which is evident from the `jax.process_index()` and `jax.tree_util.tree_map()` calls.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/flax/speech-recognition/run_flax_speech_recognition_seq2seq.py


key: chroma:document
string_value:
CODE:

def shift_tokens_right(label_ids: np.array, decoder_start_token_id: int) -> np.ndarray:
    """
    Shift label ids one token to the right.
    """
    shifted_label_ids = np.zeros_like(label_ids)
    shifted_label_ids[:, 1:] = label_ids[:, :-1]
    shifted_label_ids[:, 0] = decoder_start_token_id



SUMMARY:
**Summary:**

`shift_tokens_right` function shifts `label_ids` array one token to the right by duplicating the last token, padding with zeros otherwise. It takes two inputs:

*   `label_ids`: input 1D numpy array of label IDs
*   `decoder_start_token_id`: starting token ID for decoder

It returns the shifted `label_ids` array.

**Implementation Details:**

*   Directly duplicates the last token (`label_ids[:, -1]`) and assigns it to the first position.
*   Pads with zeros elsewhere in the array.

**Return Value:**

*   The modified `shifted_label_ids` array
    ```python
np.ndarray
```


key: chroma:document
string_value:
CODE:

class TrainState(train_state.TrainState):
    dropout_rng: jnp.ndarray

    def replicate(self):


SUMMARY:
**Class Summary: `TrainState` in Replicated Form**

*   **Primary Purpose:** The `replicate` method replicates the current `TrainState` instance to create multiple versions for parallelized training, ensuring that random number generation (RNG) parameters are shared consistently across replicas.
*   **Key Implementation Details:**
    *   `jax_utils.replicate(self)` creates a replicate of the original `TrainState`.
    *   `.replace(dropout_rng=shard_prng_key(self.dropout_rng))` updates the replica's RNG parameter to match the value from the original, using `shard_prng_key`.
*   **Notable Dependencies and Requirements:**
    *   `jax_utils`: JAX utilities for working with JAX data structures.
    *   `shard_prng_key`: A function that generates a PRNG key based on an input number (in this case, the original RNG value).
*   **Interfaces/Methods:**
    *   `replicate()`: Creates and returns a replicated version of the current `TrainState`.


key: chroma:document
string_value:
CODE:

def write_metric(summary_writer, train_metrics, eval_metrics, train_time, step):
    summary_writer.scalar("train_time", train_time, step)

    train_metrics = get_metrics(train_metrics)
    for key, vals in train_metrics.items():
        tag = f"train_{key}"
        for i, val in enumerate(vals):
            summary_writer.scalar(tag, val, step - len(vals) + i + 1)

    for metric_name, value in eval_metrics.items():


SUMMARY:
**Summary**

This `write_metric` function is used to log training and evaluation metrics at each step of a machine learning process. It takes in several inputs:

- `summary_writer`: an object responsible for logging metrics
- `train_metrics` and `eval_metrics`: dictionaries containing key-value pairs of training and evaluation metrics respectively
- `train_time`: the total time taken during the training process
- `step`: the current step or iteration number

The function logs these metrics as scalars to a summary writer, allowing for easy monitoring and tracking.

**Key Implementation Details**

* Logs training and evaluation metric values at each step using scalar summaries (`summary_writer.scalar` method)
* Utilizes `get_metrics` function (not shown in the provided code snippet) to process `train_metrics` dictionary
* Employs iteration over dictionary items to account for multiple metrics per step

**Notable Dependencies or Requirements**

- `summary_writer`: an object that provides a way to log metrics (e.g. TensorBoard)
- `get_metrics` function: a utility function that processes and formats `train_metrics` dictionary


key: chroma:document
string_value:
CODE:

def create_learning_rate_fn(
    num_train_steps: int, num_warmup_steps: int, learning_rate: float
) -> Callable[[int], jnp.ndarray]:
    """Returns a linear warmup, linear_decay learning rate function."""
    warmup_fn = optax.linear_schedule(init_value=0.0, end_value=learning_rate, transition_steps=num_warmup_steps)
    decay_fn = optax.linear_schedule(
        init_value=learning_rate, end_value=0, transition_steps=num_train_steps - num_warmup_steps
    )
    schedule_fn = optax.join_schedules(schedules=[warmup_fn, decay_fn], boundaries=[num_warmup_steps])


SUMMARY:
Here's a concise summary of the `create_learning_rate_fn` function:

### Purpose and Functionality:
 Creates a learnable rate scheduling function that combines a linear warm-up and decay phases.

### Key Implementation Details:

*   The function takes three arguments: `num_train_steps`, `num_warmup_steps`, and `learning_rate`.
*   It returns a callable schedule function using the Optax library.
*   The schedule function first applies a fixed duration linear warm-up (with `warmup_fn`), followed by a linear decay phase after `num_warmup_steps`.

### Notable Dependencies or Requirements:

*   Uses Optax for linear scheduling functions (`optax.linear_schedule`) and joining schedules with boundaries (`optax.join_schedules`).
*   Utilizes NumPy Jupyter notebook extension (`jnp`) for Numpy arrays of tensors.


key: chroma:document
string_value:
CODE:

def main():
    # 1. Parse input arguments
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.
    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments))

    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your JAX/Flax versions.
    send_example_telemetry("run_speech_recognition_seq2seq", model_args, data_args, framework="flax")

    # 2. Setup logging
    # Make one log on every process with the configuration for debugging.
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    # Set the verbosity to info of the Transformers logger.
    # We only want one process per machine to log things on the screen.
    logger.setLevel(logging.INFO if jax.process_index() == 0 else logging.ERROR)
    if jax.process_index() == 0:
        datasets.utils.logging.set_verbosity_warning()
        transformers.utils.logging.set_verbosity_info()
    else:
        datasets.utils.logging.set_verbosity_error()
        transformers.utils.logging.set_verbosity_error()

    logger.info("Training/evaluation parameters %s", training_args)

    # Check the output dir is valid
    if (
        os.path.exists(training_args.output_dir)
        and os.listdir(training_args.output_dir)
        and training_args.do_train
        and not training_args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({training_args.output_dir}) already exists and is not empty. "
            "Use `--overwrite_output_dir` to overcome."
        )

    # Handle the repository creation
    if training_args.push_to_hub:
        if training_args.hub_model_id is None:
            repo_name = get_full_repo_name(
                Path(training_args.output_dir).absolute().name, token=training_args.hub_token
            )
        else:
            repo_name = training_args.hub_model_id
        # Create repo and retrieve repo_id
        api = HfApi()
        repo_id = api.create_repo(repo_name, exist_ok=True, token=training_args.hub_token).repo_id

    # 3. Load dataset
    raw_datasets = DatasetDict()

    if training_args.do_train:
        raw_datasets["train"] = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            split=data_args.train_split_name,
            cache_dir=data_args.dataset_cache_dir,
            num_proc=data_args.preprocessing_num_workers,
            token=True if model_args.use_auth_token else None,
            trust_remote_code=data_args.trust_remote_code,
        )

    if training_args.do_eval:
        raw_datasets["eval"] = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            split=data_args.eval_split_name,
            cache_dir=data_args.dataset_cache_dir,
            num_proc=data_args.preprocessing_num_workers,
            token=True if model_args.use_auth_token else None,
            trust_remote_code=data_args.trust_remote_code,
        )

    if not training_args.do_train and not training_args.do_eval:
        raise ValueError(
            "Cannot not train and not do evaluation. At least one of training or evaluation has to be performed."
        )

    if data_args.audio_column_name not in next(iter(raw_datasets.values())).column_names:
        raise ValueError(
            f"--audio_column_name '{data_args.audio_column_name}' not found in dataset '{data_args.dataset_name}'. "
            "Make sure to set `--audio_column_name` to the correct audio column - one of "
            f"{', '.join(next(iter(raw_datasets.values())).column_names)}."
        )

    if data_args.text_column_name not in next(iter(raw_datasets.values())).column_names:
        raise ValueError(
            f"--text_column_name {data_args.text_column_name} not found in dataset '{data_args.dataset_name}'. "
            "Make sure to set `--text_column_name` to the correct text column - one of "
            f"{', '.join(next(iter(raw_datasets.values())).column_names)}."
        )

    # 5. Load pretrained model, tokenizer, and feature extractor
    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=True if model_args.use_auth_token else None,
    )
    feature_extractor = AutoFeatureExtractor.from_pretrained(
        model_args.feature_extractor_name if model_args.feature_extractor_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=True if model_args.use_auth_token else None,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast_tokenizer,
        revision=model_args.model_revision,
        token=True if model_args.use_auth_token else None,
    )

    model = FlaxAutoModelForSpeechSeq2Seq.from_pretrained(
        model_args.model_name_or_path,
        config=config,
        dtype=getattr(jnp, model_args.dtype),
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=True if model_args.use_auth_token else None,
    )

    if model.config.decoder_start_token_id is None:
        raise ValueError("Make sure that `config.decoder_start_token_id` is correctly defined")

    # 6. Resample speech dataset: `datasets` takes care of automatically loading and resampling the audio,
    # so we just need to set the correct target sampling rate.
    raw_datasets = raw_datasets.cast_column(
        data_args.audio_column_name, datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)
    )

    # 7. Preprocessing the datasets.
    # We need to read the audio files as arrays and tokenize the targets.
    max_input_length = int(data_args.max_duration_in_seconds * feature_extractor.sampling_rate)
    min_input_length = int(data_args.min_duration_in_seconds * feature_extractor.sampling_rate)
    max_label_length = (
        data_args.max_label_length if data_args.max_label_length is not None else model.config.max_length
    )
    pad_input_to_multiple_of = data_args.pad_input_to_multiple_of
    pad_target_to_multiple_of = data_args.pad_target_to_multiple_of
    audio_column_name = data_args.audio_column_name
    num_workers = data_args.preprocessing_num_workers
    text_column_name = data_args.text_column_name
    model_input_name = feature_extractor.model_input_names[0]
    do_lower_case = data_args.do_lower_case

    if training_args.do_train and data_args.max_train_samples is not None:
        raw_datasets["train"] = raw_datasets["train"].select(range(data_args.max_train_samples))

    if training_args.do_eval and data_args.max_eval_samples is not None:
        raw_datasets["eval"] = raw_datasets["eval"].select(range(data_args.max_eval_samples))

    if data_args.language is not None:
        # We only need to set the task id when the language is specified (i.e. in a multilingual setting)
        tokenizer.set_prefix_tokens(language=data_args.language, task=data_args.task)

    def prepare_dataset(batch):
        # process audio
        sample = batch[audio_column_name]
        inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
        # process audio length
        batch[model_input_name] = inputs.get(model_input_name)[0]
        batch["input_length"] = len(sample["array"])

        # process targets
        input_str = batch[text_column_name].lower() if do_lower_case else batch[text_column_name]
        batch["labels"] = tokenizer(input_str).input_ids
        return batch

    vectorized_datasets = raw_datasets.map(
        prepare_dataset,
        remove_columns=next(iter(raw_datasets.values())).column_names,
        num_proc=num_workers,
        desc="preprocess train and eval dataset",
    )

    # filter training data with inputs longer than max_input_length
    def is_audio_in_length_range(length):
        return min_input_length < length < max_input_length

    vectorized_datasets = vectorized_datasets.filter(
        is_audio_in_length_range,
        num_proc=num_workers,
        input_columns=["input_length"],
    )

    # for large datasets it is advised to run the preprocessing on a
    # single machine first with `args.preprocessing_only` since there will mostly likely
    # be a timeout when running the script in distributed mode.
    # In a second step `args.preprocessing_only` can then be set to `False` to load the
    # cached dataset
    if data_args.preprocessing_only:
        cache = {k: v.cache_files for k, v in vectorized_datasets.items()}
        logger.info(f"Data preprocessing finished. Files cached at {cache}.")
        return

    # 8. Load Metric
    metric = evaluate.load("wer", cache_dir=model_args.cache_dir)

    def compute_metrics(preds, labels):
        # replace padded labels by the padding token
        for idx in range(len(labels)):
            labels[idx][labels[idx] == -100] = tokenizer.pad_token_id

        pred_str = tokenizer.batch_decode(preds, skip_special_tokens=True)
        # we do not want to group tokens when computing the metrics
        label_str = tokenizer.batch_decode(labels, skip_special_tokens=True)

        wer = metric.compute(predictions=pred_str, references=label_str)
        return {"wer": wer}

    # 9. Save feature extractor, tokenizer and config
    feature_extractor.save_pretrained(training_args.output_dir)
    tokenizer.save_pretrained(training_args.output_dir)
    config.save_pretrained(training_args.output_dir)

    processor = AutoProcessor.from_pretrained(training_args.output_dir)

    data_collator = FlaxDataCollatorSpeechSeq2SeqWithPadding(
        processor=processor,
        decoder_start_token_id=model.config.decoder_start_token_id,
        input_padding="longest",
        target_padding="longest",
        max_target_length=max_label_length,
        pad_input_to_multiple_of=pad_input_to_multiple_of,
        pad_target_to_multiple_of=pad_target_to_multiple_of if pad_target_to_multiple_of else max_label_length,
    )

    # Enable tensorboard only on the master node
    has_tensorboard = is_tensorboard_available()
    if has_tensorboard and jax.process_index() == 0:
        try:
            from flax.metrics.tensorboard import SummaryWriter

            summary_writer = SummaryWriter(log_dir=Path(training_args.output_dir))
        except ImportError as ie:
            has_tensorboard = False
            logger.warning(
                f"Unable to display metrics through TensorBoard because some package are not installed: {ie}"
            )
    else:
        logger.warning(
            "Unable to display metrics through TensorBoard because the package is not installed: "
            "Please run pip install tensorboard to enable."
        )

    # Initialize our training
    rng = jax.random.PRNGKey(training_args.seed)
    rng, dropout_rng = jax.random.split(rng)

    # Store some constant
    num_epochs = int(training_args.num_train_epochs)
    train_batch_size = int(training_args.per_device_train_batch_size) * jax.device_count()
    per_device_eval_batch_size = int(training_args.per_device_eval_batch_size)
    eval_batch_size = per_device_eval_batch_size * jax.device_count()
    steps_per_epoch = len(vectorized_datasets["train"]) // train_batch_size
    total_train_steps = steps_per_epoch * num_epochs

    # Create learning rate schedule
    linear_decay_lr_schedule_fn = create_learning_rate_fn(
        total_train_steps,
        training_args.warmup_steps,
        training_args.learning_rate,
    )

    # We use Optax's "masking" functionality to not apply weight decay
    # to bias and LayerNorm scale parameters. decay_mask_fn returns a
    # mask boolean with the same structure as the parameters.
    # The mask is True for parameters that should be decayed.
    def decay_mask_fn(params):
        flat_params = traverse_util.flatten_dict(params)
        # find out all LayerNorm parameters
        layer_norm_candidates = ["layer_norm", "self_attn_layer_norm", "final_layer_norm", "encoder_attn_layer_norm"]
        layer_norm_named_params = {
            layer[-2:]
            for layer_norm_name in layer_norm_candidates
            for layer in flat_params.keys()
            if layer_norm_name in "".join(layer).lower()
        }
        flat_mask = {path: (path[-1] != "bias" and path[-2:] not in layer_norm_named_params) for path in flat_params}
        return traverse_util.unflatten_dict(flat_mask)

    # create adam optimizer
    adamw = optax.adamw(
        learning_rate=linear_decay_lr_schedule_fn,
        b1=training_args.adam_beta1,
        b2=training_args.adam_beta2,
        eps=training_args.adam_epsilon,
        weight_decay=training_args.weight_decay,
        mask=decay_mask_fn,
    )

    # Setup train state
    state = TrainState.create(apply_fn=model.__call__, params=model.params, tx=adamw, dropout_rng=dropout_rng)

    # label smoothed cross entropy
    def loss_fn(logits, labels, label_smoothing_factor=0.0):
        """
        The label smoothing implementation is adapted from Flax's official example:
        https://github.com/google/flax/blob/87a211135c6a377c8f29048a1cac3840e38b9da4/examples/wmt/train.py#L104
        """
        vocab_size = logits.shape[-1]
        confidence = 1.0 - label_smoothing_factor
        low_confidence = (1.0 - confidence) / (vocab_size - 1)
        normalizing_constant = -(
            confidence * jnp.log(confidence) + (vocab_size - 1) * low_confidence * jnp.log(low_confidence + 1e-20)
        )
        soft_labels = onehot(labels, vocab_size, on_value=confidence, off_value=low_confidence)

        loss = optax.softmax_cross_entropy(logits, soft_labels)
        loss = loss - normalizing_constant

        # ignore padded tokens from loss, i.e. where labels are not set to -100
        padding_mask = labels >= 0
        loss = loss * padding_mask
        loss = loss.sum()
        num_labels = padding_mask.sum()
        return loss, num_labels

    # Define gradient update step fn
    def train_step(state, batch, label_smoothing_factor=0.0):
        dropout_rng, new_dropout_rng = jax.random.split(state.dropout_rng)

        def compute_loss(params):
            labels = batch.pop("labels")
            logits = state.apply_fn(**batch, params=params, dropout_rng=dropout_rng, train=True)[0]
            loss, num_labels = loss_fn(logits, labels, label_smoothing_factor)
            return loss, num_labels

        grad_fn = jax.value_and_grad(compute_loss, has_aux=True)
        (loss, num_labels), grad = grad_fn(state.params)
        num_labels = jax.lax.psum(num_labels, "batch")

        # true loss = total loss / total samples
        loss = jax.lax.psum(loss, "batch")
        loss = jax.tree_util.tree_map(lambda x: x / num_labels, loss)

        # true grad = total grad / total samples
        grad = jax.lax.psum(grad, "batch")
        grad = jax.tree_util.tree_map(lambda x: x / num_labels, grad)
        new_state = state.apply_gradients(grads=grad, dropout_rng=new_dropout_rng)

        metrics = {"loss": loss, "learning_rate": linear_decay_lr_schedule_fn(state.step)}
        return new_state, metrics

    # Define eval fn
    def eval_step(params, batch, label_smoothing_factor=0.0):
        labels = batch.pop("labels")
        logits = model(**batch, params=params, train=False)[0]

        loss, num_labels = loss_fn(logits, labels, label_smoothing_factor)
        num_labels = jax.lax.psum(num_labels, "batch")

        # true loss = total loss / total samples
        loss = jax.lax.psum(loss, "batch")
        loss = jax.tree_util.tree_map(lambda x: x / num_labels, loss)

        metrics = {"loss": loss}
        return metrics

    # Define generation function
    num_beams = model_args.num_beams if model_args.num_beams is not None else model.config.num_beams
    gen_kwargs = {"max_length": max_label_length, "num_beams": num_beams}

    def generate_step(params, batch):
        model.params = params
        output_ids = model.generate(batch[model_input_name], attention_mask=batch.get("attention_mask"), **gen_kwargs)
        return output_ids.sequences

    # Create parallel version of the train and eval step
    p_train_step = jax.pmap(
        partial(train_step, label_smoothing_factor=training_args.label_smoothing_factor), "batch", donate_argnums=(0,)
    )
    p_eval_step = jax.pmap(partial(eval_step, label_smoothing_factor=training_args.label_smoothing_factor), "batch")
    p_generate_step = jax.pmap(generate_step, "batch")

    # Replicate the train state on each device
    state = state.replicate()

    logger.info("***** Running training *****")
    logger.info(f"  Num examples = {len(vectorized_datasets['train'])}")
    logger.info(f"  Num Epochs = {num_epochs}")
    logger.info(f"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}")
    logger.info(f"  Total train batch size (w. parallel & distributed) = {train_batch_size}")
    logger.info(f"  Total optimization steps = {total_train_steps}")

    train_time = 0
    epochs = tqdm(range(num_epochs), desc=f"Epoch ... (1/{num_epochs})", position=0)
    for epoch in epochs:
        # ======================== Training ================================
        train_start = time.time()

        train_metrics = []

        # Generate an epoch by shuffling sampling indices from the train dataset and create a data loader
        vectorized_datasets["train"] = vectorized_datasets["train"].shuffle(training_args.seed)
        train_loader = DataLoader(
            vectorized_datasets["train"],
            batch_size=train_batch_size,
            drop_last=True,
            collate_fn=data_collator,
            num_workers=training_args.dataloader_num_workers,
        )
        # train
        for batch in tqdm(train_loader, desc="Training...", position=1, leave=False):
            batch = shard(batch.data)
            state, train_metric = p_train_step(state, batch)
            train_metrics.append(train_metric)

        train_time += time.time() - train_start

        train_metric = unreplicate(train_metric)

        epochs.write(
            f"Epoch... ({epoch + 1}/{num_epochs} | Loss: {train_metric['loss']}, Learning Rate:"
            f" {train_metric['learning_rate']})"
        )

        # ======================== Evaluating ==============================
        eval_metrics = []
        eval_preds = []
        eval_labels = []

        eval_loader = DataLoader(
            vectorized_datasets["eval"],
            batch_size=eval_batch_size,
            drop_last=False,
            collate_fn=data_collator,
            num_workers=training_args.dataloader_num_workers,
        )
        for batch in tqdm(eval_loader, desc="Evaluating...", position=2, leave=False):
            # Model forward
            labels = batch["labels"]

            metrics = pad_shard_unpad(p_eval_step, static_return=True)(
                state.params, batch.data, min_device_batch=per_device_eval_batch_size
            )
            eval_metrics.append(metrics)

            # generation
            if training_args.predict_with_generate:
                generated_ids = pad_shard_unpad(p_generate_step)(state.params, batch.data)
                eval_preds.extend(jax.device_get(generated_ids.reshape(-1, gen_kwargs["max_length"])))
                eval_labels.extend(labels)

        # normalize eval metrics
        eval_metrics = get_metrics(eval_metrics)
        eval_metrics = jax.tree_util.tree_map(jnp.mean, eval_metrics)

        # compute WER metric
        wer_desc = ""
        if training_args.predict_with_generate:
            wer_metric = compute_metrics(eval_preds, eval_labels)
            eval_metrics.update(wer_metric)
            wer_desc = " ".join([f"Eval {key}: {value} |" for key, value in wer_metric.items()])

        # Print metrics and update progress bar
        desc = f"Epoch... ({epoch + 1}/{num_epochs} | Eval Loss: {eval_metrics['loss']} | {wer_desc})"
        epochs.write(desc)
        epochs.desc = desc

        # Save metrics
        if has_tensorboard and jax.process_index() == 0:
            cur_step = epoch * (len(vectorized_datasets["train"]) // train_batch_size)
            write_metric(summary_writer, train_metrics, eval_metrics, train_time, cur_step)

        # save checkpoint after each epoch and push checkpoint to the hub
        if jax.process_index() == 0:
            params = jax.device_get(jax.tree_util.tree_map(lambda x: x[0], state.params))
            model.save_pretrained(training_args.output_dir, params=params)
            tokenizer.save_pretrained(training_args.output_dir)
            if training_args.push_to_hub:
                api.upload_folder(
                    commit_message=f"Saving weights and logs of epoch {epoch}",
                    folder_path=training_args.output_dir,
                    repo_id=repo_id,
                    repo_type="model",
                    token=training_args.hub_token,


SUMMARY:
Here's a concise summary:

**Training Process**

1. **Epochs**: Iterates over `num_epochs` to train the model on `vectorized_datasets["train"]`.
2. **Batching**: Shuffles sampling indices from the train dataset and creates a data loader, then processes each batch of size `train_batch_size`.
3. **Model Training**: Calls `p_train_step(state, batch)` for each batch, which trains the model and computes training metrics.
4. **Evaluation**: After training each epoch, evaluates the model on `vectorized_datasets["eval"]` using `p_eval_step`, generating predictions, labels, and computing evaluation metrics.
5. **Logging**: Writes training and evaluation metrics to a log file, updates the summary writer, and saves model checkpoints when training is complete.

**Parallel Processing**

1. Uses `jax.pmap` to parallelize training and evaluation steps across multiple devices (if available).
2. Replicates the train state on each device for distributed training.

**Additional Operations**

1. **Weight Saving**: Saves trainer state and model weights after each epoch.
2. **Hub Upload**: Uploads the trained model and logs to a Hugging Face repository if specified.
3. **TensorBoard**: If `has_tensorboard` is true, writes metrics to a TensorBoard summary writer.

This process allows for efficient training of a deep learning model on a large dataset while utilizing parallel processing capabilities (if available).


key: chroma:document
string_value:
CODE:

class ImageEncoder(nn.Module):
    def __init__(self, args):
        super().__init__()
        model = torchvision.models.resnet152(pretrained=True)
        modules = list(model.children())[:-2]
        self.model = nn.Sequential(*modules)
        self.pool = nn.AdaptiveAvgPool2d(POOLING_BREAKDOWN[args.num_image_embeds])

    def forward(self, x):
        # Bx3x224x224 -> Bx2048x7x7 -> Bx2048xN -> BxNx2048
        out = self.pool(self.model(x))
        out = torch.flatten(out, start_dim=2)
        out = out.transpose(1, 2).contiguous()


SUMMARY:
**Concise Summary:**

This `ImageEncoder` class is a custom neural network module that extracts features from images using the pre-trained ResNet152 model. The key functionality involves:

* Preprocessing the input image using `AdaptiveAvgPool2d`
* Passing the output through the pre-trained ResNet152 model's feature extractor
* Transposing and flattening the output to prepare it for downstream processing

**Implementation Details:**

* **Key Algorithm:** Adaptive Average Pooling (`nn.AdaptiveAvgPool2d`)
* **Model Architecture:** Pretrained ResNet152 with modified feature extractor (excluding last two layers)
* **Method Signatures:**
	+ `forward(x)`: Input image `x` -> Output tensor
	+ `__init__(args)` initializing the model and adapter

**Notable Dependencies and Requirements:**

* `torchvision.models.resnet152`
* `nn.Module`
* `nn.Sequential`
* ` torch` for tensor operations


key: chroma:document
string_value:
CODE:

class JsonlDataset(Dataset):
    def __init__(self, data_path, tokenizer, transforms, labels, max_seq_length):
        self.data = [json.loads(l) for l in open(data_path)]
        self.data_dir = os.path.dirname(data_path)
        self.tokenizer = tokenizer
        self.labels = labels
        self.n_classes = len(labels)
        self.max_seq_length = max_seq_length

        self.transforms = transforms

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        sentence = torch.LongTensor(self.tokenizer.encode(self.data[index]["text"], add_special_tokens=True))
        start_token, sentence, end_token = sentence[0], sentence[1:-1], sentence[-1]
        sentence = sentence[: self.max_seq_length]

        label = torch.zeros(self.n_classes)
        label[[self.labels.index(tgt) for tgt in self.data[index]["label"]]] = 1

        image = Image.open(os.path.join(self.data_dir, self.data[index]["img"])).convert("RGB")
        image = self.transforms(image)

        return {
            "image_start_token": start_token,
            "image_end_token": end_token,
            "sentence": sentence,
            "image": image,
            "label": label,
        }

    def get_label_frequencies(self):
        label_freqs = Counter()
        for row in self.data:
            label_freqs.update(row["label"])


SUMMARY:
**Class Summary: `JsonlDataset`**

The `JsonlDataset` class is a custom dataset implementation for sequence data with associated labels and images. It extends the PyTorch `Dataset` class.

**Primary Purpose:** Load JSONL (JavaScript Object Notation Line Endings) formatted dataset, tokenize text, and generate corresponding image inputs with labeled values.

**Key Implementation Details:**

* Tokenizes input text using a tokenizer, with special tokens added.
* Applies transformations to input images.
* Returns dictionaries containing:
	+ `image_start_token` and `image_end_token`
	+ Tokenized sentence
	+ Labeled tensor based on "label" key in dataset rows
	+ Image tensor without transformation applied

**Notable Dependencies:**

* PyTorch (`torch`)
* OpenCV (`Image.open`, `convert`)
* `json` (`json.loads`)
* `Counter`

Note: The code uses the Hugging Face Transformers library for tokenization, but this is not explicitly imported in the provided snippet.


key: chroma:document
string_value:
CODE:

def collate_fn(batch):
    lens = [len(row["sentence"]) for row in batch]
    bsz, max_seq_len = len(batch), max(lens)

    mask_tensor = torch.zeros(bsz, max_seq_len, dtype=torch.long)
    text_tensor = torch.zeros(bsz, max_seq_len, dtype=torch.long)

    for i_batch, (input_row, length) in enumerate(zip(batch, lens)):
        text_tensor[i_batch, :length] = input_row["sentence"]
        mask_tensor[i_batch, :length] = 1

    img_tensor = torch.stack([row["image"] for row in batch])
    tgt_tensor = torch.stack([row["label"] for row in batch])
    img_start_token = torch.stack([row["image_start_token"] for row in batch])
    img_end_token = torch.stack([row["image_end_token"] for row in batch])



SUMMARY:
**Collate Function Summary**

This function collates data from a batch of input rows into tensors.

**Main Purpose and Functionality**
The purpose is to transform raw batch data into formats suitable for training a model, specifically a computer vision task that requires sentences as input.

**Key Implementation Details:**

* Creates two tensor masks (`mask_tensor`) with zeros initialized, where the value at index `[i_batch, :length]` is 1.
* Initializes other tensors using the `batch`, including an image tensor (`img_tensor`), target tensor (`tgt_tensor`), and specialized token vectors (`img_start_token`, `img_end_token`).
* Copies sentence text from each input row to `text_tensor`.
* Returns six collated tensors: 
  * `text_tensor`: (sequence length, batch size) tensor of sentence text.
  * `mask_tensor`: (sequence length, batch size) tensor indicating valid sequences.
  * `img_tensor`: (batch size, image dimensions) tensor of images.
  * `img_start_token` and `img_end_token`: (sequence length, batch size) tensors of image starting and ending tokens, respectively.

**Notable Dependencies or Requirements:**

* PyTorch library for tensor operations (`torch.zeros`, `torch.stack`, etc.).
* Batch data rows with specific keys: `"sentence"`, `"image"`, `"label"`, `"image_start_token"`, and `"image_end_token"`.


key: chroma:document
string_value:
CODE:

def get_mmimdb_labels():
    return [
        "Crime",
        "Drama",
        "Thriller",
        "Action",
        "Comedy",
        "Romance",
        "Documentary",
        "Short",
        "Mystery",
        "History",
        "Family",
        "Adventure",
        "Fantasy",
        "Sci-Fi",
        "Western",
        "Horror",
        "Sport",
        "War",
        "Music",
        "Musical",
        "Animation",
        "Biography",
        "Film-Noir",


SUMMARY:
Here is a concise summary of the provided function:

**Summary:**
The `get_mmimdb_labels()` function returns a list of genre labels used by IMDb, totaling 23 genres.

**Detailed Analysis**

* Function Name: `get_mmimdb_labels`
* Purpose: Returns a predefined list of genres for use in IMDB-related applications.
* Implementation: A simple list of strings containing the genre names.
* Key Details:
	+ No complex logic or algorithms. The function simply returns the pre-defined list of genres.
	+ No significant methods, interfaces, or dependencies.

**Dependencies and Requirements:**
None explicitly mentioned; however, this function relies on IMDb's predefined set of genre labels, which is a static resource.

This concise summary provides essential information about the function's purpose, behavior, and any notable details.


key: chroma:document
string_value:
CODE:

def get_image_transforms():
    return transforms.Compose(
        [
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(
                mean=[0.46777044, 0.44531429, 0.40661017],
                std=[0.12221994, 0.12145835, 0.14380469],
            ),
        ]


SUMMARY:
**Function Summary:**

The `get_image_transforms` function returns a composed transformation pipeline for image preprocessing. It converts an image into a tensor format and normalizes the pixel values using a mean and standard deviation.

**Primary Purpose:** Preprocess images by resizing, center cropping, converting to tensor format, and applying normalization.

**Key Implementation Details:**

* Resizes input images to 256x256 dimensions ( `transforms.Resize(256)` )
* Centers crop to a region of size 224x224 within the resized image
* Converts pixel values to tensor representation ( `transforms.ToTensor()` )
* Normalizes pixel values using mean and standard deviation ( `transforms.Normalize(mean=[...,], std=[...])` )

**Notable Dependencies or Requirements:**

* The `transforms` object is assumed to be part of a library such as PyTorch, with methods for image transformations.


key: chroma:document
string_value:
CODE:

def set_seed(args):
    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    if args.n_gpu > 0:


SUMMARY:
**Summary**

`set_seed` function sets seeds for various libraries:
- `random`
- NumPy (`np.random`)
- PyTorch (`torch`, including manual and CUDA seeding)

**Key Implementation Details**

* Sets seed using `args.seed` to control random number generation
* Uses a single seed value across multiple libraries (except for multi-GPU systems)
* Only sets CUDA seed if `n_gpu > 0`

**Notable Dependencies and Requirements**

- none

Note: The code uses `torch.cuda.manual_seed_all` only when using multiple GPUs (`args.n_gpu > 0`).


key: chroma:document
string_value:
CODE:

def train(args, train_dataset, model, tokenizer, criterion):
    """Train the model"""
    if args.local_rank in [-1, 0]:
        tb_writer = SummaryWriter()

    args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)
    train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
    train_dataloader = DataLoader(
        train_dataset,
        sampler=train_sampler,
        batch_size=args.train_batch_size,
        collate_fn=collate_fn,
        num_workers=args.num_workers,
    )

    if args.max_steps > 0:
        t_total = args.max_steps
        args.num_train_epochs = args.max_steps // (len(train_dataloader) // args.gradient_accumulation_steps) + 1
    else:
        t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs

    # Prepare optimizer and schedule (linear warmup and decay)
    no_decay = ["bias", "LayerNorm.weight"]
    optimizer_grouped_parameters = [
        {
            "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
            "weight_decay": args.weight_decay,
        },
        {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], "weight_decay": 0.0},
    ]

    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
    scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
    )
    if args.fp16:
        try:
            from apex import amp
        except ImportError:
            raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
        model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)

    # multi-gpu training (should be after apex fp16 initialization)
    if args.n_gpu > 1:
        model = nn.DataParallel(model)

    # Distributed training (should be after apex fp16 initialization)
    if args.local_rank != -1:
        model = nn.parallel.DistributedDataParallel(
            model, device_ids=[args.local_rank], output_device=args.local_rank, find_unused_parameters=True
        )

    # Train!
    logger.info("***** Running training *****")
    logger.info("  Num examples = %d", len(train_dataset))
    logger.info("  Num Epochs = %d", args.num_train_epochs)
    logger.info("  Instantaneous batch size per GPU = %d", args.per_gpu_train_batch_size)
    logger.info(
        "  Total train batch size (w. parallel, distributed & accumulation) = %d",
        args.train_batch_size
        * args.gradient_accumulation_steps
        * (torch.distributed.get_world_size() if args.local_rank != -1 else 1),
    )
    logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
    logger.info("  Total optimization steps = %d", t_total)

    global_step = 0
    tr_loss, logging_loss = 0.0, 0.0
    best_f1, n_no_improve = 0, 0
    model.zero_grad()
    train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0])
    set_seed(args)  # Added here for reproducibility
    for _ in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
        for step, batch in enumerate(epoch_iterator):
            model.train()
            batch = tuple(t.to(args.device) for t in batch)
            labels = batch[5]
            inputs = {
                "input_ids": batch[0],
                "input_modal": batch[2],
                "attention_mask": batch[1],
                "modal_start_tokens": batch[3],
                "modal_end_tokens": batch[4],
            }
            outputs = model(**inputs)
            logits = outputs[0]  # model outputs are always tuple in transformers (see doc)
            loss = criterion(logits, labels)

            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel training
            if args.gradient_accumulation_steps > 1:
                loss = loss / args.gradient_accumulation_steps

            if args.fp16:
                with amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
            else:
                loss.backward()

            tr_loss += loss.item()
            if (step + 1) % args.gradient_accumulation_steps == 0:
                if args.fp16:
                    nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
                else:
                    nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)

                optimizer.step()
                scheduler.step()  # Update learning rate schedule
                model.zero_grad()
                global_step += 1

                if args.local_rank in [-1, 0] and args.logging_steps > 0 and global_step % args.logging_steps == 0:
                    logs = {}
                    if (
                        args.local_rank == -1 and args.evaluate_during_training
                    ):  # Only evaluate when single GPU otherwise metrics may not average well
                        results = evaluate(args, model, tokenizer, criterion)
                        for key, value in results.items():
                            eval_key = "eval_{}".format(key)
                            logs[eval_key] = value

                    loss_scalar = (tr_loss - logging_loss) / args.logging_steps
                    learning_rate_scalar = scheduler.get_lr()[0]
                    logs["learning_rate"] = learning_rate_scalar
                    logs["loss"] = loss_scalar
                    logging_loss = tr_loss

                    for key, value in logs.items():
                        tb_writer.add_scalar(key, value, global_step)
                    print(json.dumps({**logs, **{"step": global_step}}))

                if args.local_rank in [-1, 0] and args.save_steps > 0 and global_step % args.save_steps == 0:
                    # Save model checkpoint
                    output_dir = os.path.join(args.output_dir, "checkpoint-{}".format(global_step))
                    if not os.path.exists(output_dir):
                        os.makedirs(output_dir)
                    model_to_save = (
                        model.module if hasattr(model, "module") else model
                    )  # Take care of distributed/parallel training
                    torch.save(model_to_save.state_dict(), os.path.join(output_dir, WEIGHTS_NAME))
                    torch.save(args, os.path.join(output_dir, "training_args.bin"))
                    logger.info("Saving model checkpoint to %s", output_dir)

            if args.max_steps > 0 and global_step > args.max_steps:
                epoch_iterator.close()
                break
        if args.max_steps > 0 and global_step > args.max_steps:
            train_iterator.close()
            break

        if args.local_rank == -1:
            results = evaluate(args, model, tokenizer, criterion)
            if results["micro_f1"] > best_f1:
                best_f1 = results["micro_f1"]
                n_no_improve = 0
            else:
                n_no_improve += 1

            if n_no_improve > args.patience:
                train_iterator.close()
                break

    if args.local_rank in [-1, 0]:
        tb_writer.close()



SUMMARY:
**Function Summary:**

`train` function trains a model using a given dataset and tokenizer. It takes in arguments such as batch size, learning rate, and number of epochs.

**Primary Purpose:** Train the model on the provided dataset using the specified configuration.

**Key Implementation Details:**

* Uses `DataLoader` to create data batches
* Implements custom training loop with gradient accumulation steps
* Utilizes `AdamW` optimizer with weight decay
* Integrates linear warmup and decay schedule using `get_linear_schedule_with_warmup`
* Supports multi-gpu and distributed training with `nn.DataParallel` and `nn.parallel.DistributedDataParallel`

**Notable Dependencies:**

* `transformers` library for model implementation
* `torch` library for tensor operations
* `apex` library for mixed precision training (when enabled)

**Output:**

The function returns the global step number and the total training loss after completion.

```markdown
# Function Signature
train(args, train_dataset, model, tokenizer, criterion)

# Parameters
- args (configuration): batch size, learning rate, and other hyperparameters
- train_dataset (dataset): dataset to be trained on
- model: pre-trained model to be fine-tuned
- tokenizer: tokenizer used for data preprocessing
- criterion (loss function): loss function used for training

# Returns
- global_step (int): current global step number
- tr_loss / global_step (float): total training loss
```


key: chroma:document
string_value:
CODE:

def evaluate(args, model, tokenizer, criterion, prefix=""):
    # Loop to handle MNLI double evaluation (matched, mis-matched)
    eval_output_dir = args.output_dir
    eval_dataset = load_examples(args, tokenizer, evaluate=True)

    if not os.path.exists(eval_output_dir) and args.local_rank in [-1, 0]:
        os.makedirs(eval_output_dir)

    args.eval_batch_size = args.per_gpu_eval_batch_size * max(1, args.n_gpu)
    # Note that DistributedSampler samples randomly
    eval_sampler = SequentialSampler(eval_dataset)
    eval_dataloader = DataLoader(
        eval_dataset, sampler=eval_sampler, batch_size=args.eval_batch_size, collate_fn=collate_fn
    )

    # multi-gpu eval
    if args.n_gpu > 1 and not isinstance(model, nn.DataParallel):
        model = nn.DataParallel(model)

    # Eval!
    logger.info("***** Running evaluation {} *****".format(prefix))
    logger.info("  Num examples = %d", len(eval_dataset))
    logger.info("  Batch size = %d", args.eval_batch_size)
    eval_loss = 0.0
    nb_eval_steps = 0
    preds = None
    out_label_ids = None
    for batch in tqdm(eval_dataloader, desc="Evaluating"):
        model.eval()
        batch = tuple(t.to(args.device) for t in batch)

        with torch.no_grad():
            batch = tuple(t.to(args.device) for t in batch)
            labels = batch[5]
            inputs = {
                "input_ids": batch[0],
                "input_modal": batch[2],
                "attention_mask": batch[1],
                "modal_start_tokens": batch[3],
                "modal_end_tokens": batch[4],
            }
            outputs = model(**inputs)
            logits = outputs[0]  # model outputs are always tuple in transformers (see doc)
            tmp_eval_loss = criterion(logits, labels)
            eval_loss += tmp_eval_loss.mean().item()
        nb_eval_steps += 1
        if preds is None:
            preds = torch.sigmoid(logits).detach().cpu().numpy() > 0.5
            out_label_ids = labels.detach().cpu().numpy()
        else:
            preds = np.append(preds, torch.sigmoid(logits).detach().cpu().numpy() > 0.5, axis=0)
            out_label_ids = np.append(out_label_ids, labels.detach().cpu().numpy(), axis=0)

    eval_loss = eval_loss / nb_eval_steps
    result = {
        "loss": eval_loss,
        "macro_f1": f1_score(out_label_ids, preds, average="macro"),
        "micro_f1": f1_score(out_label_ids, preds, average="micro"),
    }

    output_eval_file = os.path.join(eval_output_dir, prefix, "eval_results.txt")
    with open(output_eval_file, "w") as writer:
        logger.info("***** Eval results {} *****".format(prefix))
        for key in sorted(result.keys()):
            logger.info("  %s = %s", key, str(result[key]))
            writer.write("%s = %s\n" % (key, str(result[key])))



SUMMARY:
**Function Summary:**

`evaluate` is a function used to evaluate the performance of a model on a dataset. It takes in several arguments including:

- `args`: input arguments
- `model`: the trained model to be evaluated
- `tokenizer`: the tokenizer used to preprocess input data
- `criterion`: the loss function to be used for evaluation
- `prefix` (optional): a prefix to include in output files (- default: "")

The function performs the following steps:

1. **Preprocess dataset**: loads and processes the evaluation dataset using `load_examples` function.
2. **Create data loader**: creates a sequential sampler and data loader from the preprocessed dataset.
3. **Wrap model for multi-gpu**: wraps the model with `DataParallel` if it's not already, to handle evaluation on multiple GPUs.
4. **Evaluates model**: iterates over each batch in the data loader, moves it to GPU device, and performs predictions using the wrapped model.
5. **Calculates evaluation metrics**: calculates loss and accuracy metrics (including macro and micro F1 score) using `criterion` and `f1_score`.
6. **Writes output**: writes the evaluation results to a file for each prefix.

**Key Implementation Details:**

- Uses `DataLoader` from PyTorch to load and preprocess dataset.
- Wraps the model with `DataParallel` for multi-gpu support using the `nn.DataParallel` module.
- Uses `torch.sigmoid` to calculate prediction probabilities.

**Notable Dependencies/Requirements:**

- `PyTorch`: used extensively throughout the function, including `DataLoader`, `nn.DataParallel`, and tensor operations.
- `transformers`: provides the preprocessed input data using the `load_examples` function.
- External library required for calculating F1 score (`f1_score`).


key: chroma:document
string_value:
CODE:

def load_examples(args, tokenizer, evaluate=False):
    path = os.path.join(args.data_dir, "dev.jsonl" if evaluate else "train.jsonl")
    transforms = get_image_transforms()
    labels = get_mmimdb_labels()
    dataset = JsonlDataset(path, tokenizer, transforms, labels, args.max_seq_length - args.num_image_embeds - 2)


SUMMARY:
Here is a concise summary of the function:

**Function:** `load_examples`

**Purpose:** Loads a JSONL dataset for example loading, utilizing a tokenizer and image transformations.

**Key Details:**

* Fetches JSONL file path (`dev.jsonl` or `train.jsonl`) based on evaluation mode.
* Initializes transforms using `get_image_transforms()`.
* Loads labels from `get_mmimdb_labels()`.
* Creates a `JsonlDataset` instance with specified arguments and tokenizer.

**Notable Dependencies:**

* `os.path.join()` for file path construction
* `get_image_transforms()` for image transformations
* `get_mmimdb_labels()` for label loading
* `JsonlDataset` class (external dependency)


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()

    # Required parameters
    parser.add_argument(
        "--data_dir",
        default=None,
        type=str,
        required=True,
        help="The input data dir. Should contain the .jsonl files for MMIMDB.",
    )
    parser.add_argument(
        "--model_name_or_path",
        default=None,
        type=str,
        required=True,
        help="Path to pretrained model or model identifier from huggingface.co/models",
    )
    parser.add_argument(
        "--output_dir",
        default=None,
        type=str,
        required=True,
        help="The output directory where the model predictions and checkpoints will be written.",
    )

    # Other parameters
    parser.add_argument(
        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
    )
    parser.add_argument(
        "--tokenizer_name",
        default="",
        type=str,
        help="Pretrained tokenizer name or path if not the same as model_name",
    )
    parser.add_argument(
        "--cache_dir",
        default=None,
        type=str,
        help="Where do you want to store the pre-trained models downloaded from huggingface.co",
    )
    parser.add_argument(
        "--max_seq_length",
        default=128,
        type=int,
        help=(
            "The maximum total input sequence length after tokenization. Sequences longer "
            "than this will be truncated, sequences shorter will be padded."
        ),
    )
    parser.add_argument(
        "--num_image_embeds", default=1, type=int, help="Number of Image Embeddings from the Image Encoder"
    )
    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
    parser.add_argument(
        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
    )
    parser.add_argument(
        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
    )

    parser.add_argument("--per_gpu_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.")
    parser.add_argument(
        "--per_gpu_eval_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation."
    )
    parser.add_argument(
        "--gradient_accumulation_steps",
        type=int,
        default=1,
        help="Number of updates steps to accumulate before performing a backward/update pass.",
    )
    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
    parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
    parser.add_argument(
        "--num_train_epochs", default=3.0, type=float, help="Total number of training epochs to perform."
    )
    parser.add_argument("--patience", default=5, type=int, help="Patience for Early Stopping.")
    parser.add_argument(
        "--max_steps",
        default=-1,
        type=int,
        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
    )
    parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")

    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
    parser.add_argument(
        "--eval_all_checkpoints",
        action="store_true",
        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
    )
    parser.add_argument("--no_cuda", action="store_true", help="Avoid using CUDA when available")
    parser.add_argument("--num_workers", type=int, default=8, help="number of worker threads for dataloading")
    parser.add_argument(
        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
    )
    parser.add_argument(
        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
    )
    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")

    parser.add_argument(
        "--fp16",
        action="store_true",
        help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit",
    )
    parser.add_argument(
        "--fp16_opt_level",
        type=str,
        default="O1",
        help=(
            "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. "
            "See details at https://nvidia.github.io/apex/amp.html"
        ),
    )
    parser.add_argument("--local_rank", type=int, default=-1, help="For distributed training: local_rank")
    parser.add_argument("--server_ip", type=str, default="", help="For distant debugging.")
    parser.add_argument("--server_port", type=str, default="", help="For distant debugging.")
    args = parser.parse_args()

    if (
        os.path.exists(args.output_dir)
        and os.listdir(args.output_dir)
        and args.do_train
        and not args.overwrite_output_dir
    ):
        raise ValueError(
            "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
                args.output_dir
            )
        )

    # Setup distant debugging if needed
    if args.server_ip and args.server_port:
        # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
        import ptvsd

        print("Waiting for debugger attach")
        ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
        ptvsd.wait_for_attach()

    # Setup CUDA, GPU & distributed training
    if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
        args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count()
    else:  # Initializes the distributed backend which will take care of synchronizing nodes/GPUs
        torch.cuda.set_device(args.local_rank)
        device = torch.device("cuda", args.local_rank)
        torch.distributed.init_process_group(backend="nccl")
        args.n_gpu = 1

    args.device = device

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO if args.local_rank in [-1, 0] else logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        args.local_rank,
        device,
        args.n_gpu,
        bool(args.local_rank != -1),
        args.fp16,
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    # Set seed
    set_seed(args)

    # Load pretrained model and tokenizer
    if args.local_rank not in [-1, 0]:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

    # Setup model
    labels = get_mmimdb_labels()
    num_labels = len(labels)
    transformer_config = AutoConfig.from_pretrained(args.config_name if args.config_name else args.model_name_or_path)
    tokenizer = AutoTokenizer.from_pretrained(
        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
        do_lower_case=args.do_lower_case,
        cache_dir=args.cache_dir,
    )
    transformer = AutoModel.from_pretrained(
        args.model_name_or_path, config=transformer_config, cache_dir=args.cache_dir
    )
    img_encoder = ImageEncoder(args)
    config = MMBTConfig(transformer_config, num_labels=num_labels)
    model = MMBTForClassification(config, transformer, img_encoder)

    if args.local_rank == 0:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

    model.to(args.device)

    logger.info("Training/evaluation parameters %s", args)

    # Training
    if args.do_train:
        train_dataset = load_examples(args, tokenizer, evaluate=False)
        label_frequences = train_dataset.get_label_frequencies()
        label_frequences = [label_frequences[l] for l in labels]
        label_weights = (
            torch.tensor(label_frequences, device=args.device, dtype=torch.float) / len(train_dataset)
        ) ** -1
        criterion = nn.BCEWithLogitsLoss(pos_weight=label_weights)
        global_step, tr_loss = train(args, train_dataset, model, tokenizer, criterion)
        logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)

    # Saving best-practices: if you use defaults names for the model, you can reload it using from_pretrained()
    if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
        logger.info("Saving model checkpoint to %s", args.output_dir)
        # Save a trained model, configuration and tokenizer using `save_pretrained()`.
        # They can then be reloaded using `from_pretrained()`
        model_to_save = (
            model.module if hasattr(model, "module") else model
        )  # Take care of distributed/parallel training
        torch.save(model_to_save.state_dict(), os.path.join(args.output_dir, WEIGHTS_NAME))
        tokenizer.save_pretrained(args.output_dir)

        # Good practice: save your training arguments together with the trained model
        torch.save(args, os.path.join(args.output_dir, "training_args.bin"))

        # Load a trained model and vocabulary that you have fine-tuned
        model = MMBTForClassification(config, transformer, img_encoder)
        model.load_state_dict(torch.load(os.path.join(args.output_dir, WEIGHTS_NAME)))
        tokenizer = AutoTokenizer.from_pretrained(args.output_dir)
        model.to(args.device)

    # Evaluation
    results = {}
    if args.do_eval and args.local_rank in [-1, 0]:
        checkpoints = [args.output_dir]
        if args.eval_all_checkpoints:
            checkpoints = [
                os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + "/**/" + WEIGHTS_NAME, recursive=True))
            ]

        logger.info("Evaluate the following checkpoints: %s", checkpoints)
        for checkpoint in checkpoints:
            global_step = checkpoint.split("-")[-1] if len(checkpoints) > 1 else ""
            prefix = checkpoint.split("/")[-1] if checkpoint.find("checkpoint") != -1 else ""
            model = MMBTForClassification(config, transformer, img_encoder)
            model.load_state_dict(torch.load(checkpoint))
            model.to(args.device)
            result = evaluate(args, model, tokenizer, criterion, prefix=prefix)
            result = {k + "_{}".format(global_step): v for k, v in result.items()}
            results.update(result)



SUMMARY:
This is the main code of a text classification model called MMBT (Multi-Media Mining-based Baseline) for the MMDB Challenge. Here's a brief overview:

**Task**: The model is trained to classify text from multimodal input data, such as images and captions.

**Model Architecture**: The model consists of an image encoder, a transformer model with classification head, and a loss function.

**Training**: The model is trained on a custom dataset using the Adam optimizer and binary cross-entropy loss during training.

**Evaluation**: The model's performance is evaluated on the test set using metrics such as accuracy and precision.

Some key features of this code include:

* Use of PyTorch for building and training the neural network
* Custom dataset loading and preprocessing functions (`load_examples`)
* Efficient tensor operations (e.g., `set_seed`, `logger.info`)
* Distributed training support via `torch.distributed` module

**Summary**: This is a deep learning model code that fine-tunes a transformer-based architecture for text classification from multimodal input data.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/quantization-qdqbert/trainer_quant_qa.py


key: chroma:document
string_value:
CODE:

class QuestionAnsweringTrainer(Trainer):
    def __init__(self, *args, eval_examples=None, post_process_function=None, quant_trainer_args=None, **kwargs):
        super().__init__(*args, **kwargs)
        self.eval_examples = eval_examples
        self.post_process_function = post_process_function
        self.quant_trainer_args = quant_trainer_args
        self.calib_num = 128  # default number of calibration samples

    def get_calib_dataloader(self, calib_dataset=None):
        """
        Returns the calibration dataloader :class:`~torch.utils.data.DataLoader`.

        Args:
            calib_dataset (:obj:`torch.utils.data.Dataset`, `optional`)
        """
        if calib_dataset is None and self.calib_dataset is None:
            raise ValueError("Trainer: calibration requires an calib_dataset.")
        calib_dataset = calib_dataset if calib_dataset is not None else self.calib_dataset

        calib_dataset = self._remove_unused_columns(calib_dataset, description="Calibration")

        return DataLoader(
            calib_dataset,
            batch_size=self.args.eval_batch_size,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
            pin_memory=self.args.dataloader_pin_memory,
            shuffle=True,
        )

    def calibrate(self, calib_dataset=None):
        calib_dataset = self.train_dataset if calib_dataset is None else calib_dataset
        calib_dataloader = self.get_calib_dataloader(calib_dataset)

        model = self.model
        quant_trainer.configure_model(model, self.quant_trainer_args, calib=True)
        model.eval()
        quant_trainer.enable_calibration(model)

        logger.info("***** Running calibration *****")
        logger.info(f"  Num examples = {self.calib_num}")
        logger.info(f"  Batch size = {calib_dataloader.batch_size}")

        for step, inputs in enumerate(calib_dataloader):
            # Prediction step
            loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only=True)
            if (step + 1) * calib_dataloader.batch_size >= self.calib_num:
                break

        quant_trainer.finish_calibration(model, self.quant_trainer_args)
        self.model = model

    def evaluate(self, eval_dataset=None, eval_examples=None, ignore_keys=None, metric_key_prefix: str = "eval"):
        eval_dataset = self.eval_dataset if eval_dataset is None else eval_dataset
        eval_dataloader = self.get_eval_dataloader(eval_dataset)
        eval_examples = self.eval_examples if eval_examples is None else eval_examples

        # Temporarily disable metric computation, we will do it in the loop here.
        compute_metrics = self.compute_metrics
        self.compute_metrics = None
        eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
        try:
            output = eval_loop(
                eval_dataloader,
                description="Evaluation",
                # No point gathering the predictions if there are no metrics, otherwise we defer to
                # self.args.prediction_loss_only
                prediction_loss_only=True if compute_metrics is None else None,
                ignore_keys=ignore_keys,
            )
        finally:
            self.compute_metrics = compute_metrics

        if self.post_process_function is not None and self.compute_metrics is not None:
            eval_preds = self.post_process_function(eval_examples, eval_dataset, output.predictions)
            metrics = self.compute_metrics(eval_preds)

            # Prefix all keys with metric_key_prefix + '_'
            for key in list(metrics.keys()):
                if not key.startswith(f"{metric_key_prefix}_"):
                    metrics[f"{metric_key_prefix}_{key}"] = metrics.pop(key)

            self.log(metrics)
        else:
            metrics = {}

        if self.args.tpu_metrics_debug or self.args.debug:
            # tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.)
            xm.master_print(met.metrics_report())

        self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, metrics)
        return metrics

    def predict(self, predict_dataset, predict_examples, ignore_keys=None, metric_key_prefix: str = "test"):
        predict_dataloader = self.get_test_dataloader(predict_dataset)

        # Temporarily disable metric computation, we will do it in the loop here.
        compute_metrics = self.compute_metrics
        self.compute_metrics = None
        eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
        try:
            output = eval_loop(
                predict_dataloader,
                description="Prediction",
                # No point gathering the predictions if there are no metrics, otherwise we defer to
                # self.args.prediction_loss_only
                prediction_loss_only=True if compute_metrics is None else None,
                ignore_keys=ignore_keys,
            )
        finally:
            self.compute_metrics = compute_metrics

        if self.post_process_function is None or self.compute_metrics is None:
            return output

        predictions = self.post_process_function(predict_examples, predict_dataset, output.predictions, "predict")
        metrics = self.compute_metrics(predictions)

        # Prefix all keys with metric_key_prefix + '_'
        for key in list(metrics.keys()):
            if not key.startswith(f"{metric_key_prefix}_"):
                metrics[f"{metric_key_prefix}_{key}"] = metrics.pop(key)

        return PredictionOutput(predictions=predictions.predictions, label_ids=predictions.label_ids, metrics=metrics)

    def save_onnx(self, output_dir="./"):
        eval_dataset = self.eval_dataset
        eval_dataloader = self.get_eval_dataloader(eval_dataset)

        batch = next(iter(eval_dataloader))

        # saving device - to make it consistent
        device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        # convert to tuple
        input_tuple = tuple(v.to(device) for k, v in batch.items())

        logger.info("Converting model to be onnx compatible")
        from pytorch_quantization.nn import TensorQuantizer

        TensorQuantizer.use_fb_fake_quant = True

        model = self.model.to(device)

        model.eval()
        model.float()

        model_to_save = model.module if hasattr(model, "module") else model
        quant_trainer.configure_model(model_to_save, self.quant_trainer_args)

        output_model_file = os.path.join(output_dir, "model.onnx")
        logger.info(f"exporting model to {output_model_file}")

        axes = {0: "batch_size", 1: "seq_len"}

        torch.onnx.export(
            model_to_save,
            input_tuple,
            output_model_file,
            export_params=True,
            opset_version=13,
            do_constant_folding=True,
            input_names=["input_ids", "attention_mask", "token_type_ids"],
            output_names=["output_start_logits", "output_end_logits"],
            dynamic_axes={
                "input_ids": axes,
                "attention_mask": axes,
                "token_type_ids": axes,
                "output_start_logits": axes,
                "output_end_logits": axes,
            },
            verbose=True,
        )


SUMMARY:
**QuestionAnsweringTrainer Class Summary**

**Main Purpose and Functionality**
The `QuestionAnsweringTrainer` class is a specialized trainer for question answering tasks, extending the base `Trainer` class. It provides methods for calibration, evaluation, prediction, and saving the model as ONNX.

### Key Implementation Details

*   **Calibration**: The `calibrate` method performs calibration on the model using the `quant_trainer.configure_model`, `model.eval()`, and `quant_trainer.enable_calibration` functions.
*   **Evaluation**: The `evaluate` method evaluates the model on a given dataset using the `prediction_loop` or `evaluation_loop`, depending on the `use_legacy_prediction_loop` flag. It also applies post-processing to the output predictions.

### Notable Dependencies and Requirements

*   `quant_trainer`: A module required for calibration and quantization.
*   `torch.utils.data.DataLoader`: Used to create calibration dataloader in `get_calib_dataloader`.
*   `train_dataset`, `eval_dataset`, `predict_dataset`: Data loaders required for training, evaluation, and prediction respectively.

**Method Signatures**

*   `get_calib_dataloader(calib_dataset=None)`: Returns the calibration dataloader.
*   `calibrate(calib_dataset=None)`: Performs calibration on the model.
*   `evaluate(eval_dataset=None, eval_examples=None, ignore_keys=None, metric_key_prefix: str = "eval")`: Evaluates the model on a given dataset.
*   `predict(predict_dataset, predict_examples, ignore_keys=None, metric_key_prefix: str = "test")`: Performs prediction on the model.
*   `save_onnx(output_dir="./")`: Saves the model as ONNX.


key: docstring
string_value:
# max width of quantizer names
# ========================================== Quant Trainer API ==========================================


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/quantization-qdqbert/quant_trainer.py


key: chroma:document
string_value:
CODE:

def add_arguments(parser):
    """Add arguments to parser for functions defined in quant_trainer."""

    group = parser.add_argument_group("quant_trainer arguments")
    group.add_argument("--wprec", type=int, default=8, help="weight precision")
    group.add_argument("--aprec", type=int, default=8, help="activation precision")
    group.add_argument("--quant-per-tensor", action="store_true", help="per tensor weight scaling")
    group.add_argument("--quant-disable", action="store_true", help="disable all quantizers")
    group.add_argument("--quant-disable-embeddings", action="store_true", help="disable all embeddings quantizers")
    group.add_argument("--quant-disable-keyword", type=str, nargs="+", help="disable quantizers by keyword")
    group.add_argument("--quant-disable-layer-module", type=str, help="disable quantizers by keyword under layer.")
    group.add_argument("--quant-enable-layer-module", type=str, help="enable quantizers by keyword under layer")
    group.add_argument("--calibrator", default="max", help="which quantization range calibrator to use")
    group.add_argument("--percentile", default=None, type=float, help="percentile for PercentileCalibrator")
    group.add_argument("--fuse-qkv", action="store_true", help="use the same scale factor for qkv")
    group.add_argument("--clip-gelu", metavar="N", type=float, help="clip gelu output maximum value to N")
    group.add_argument(
        "--recalibrate-weights",
        action="store_true",
        help=(
            "recalibrate weight amaxes by taking the max of the weights."
            " amaxes will be computed with the current quantization granularity (axis)."
        ),


SUMMARY:
**Function Summary**

The `add_arguments` function adds command-line arguments to a parser for the QuantTrainer. It groups these arguments into categories and defines various boolean, integer, float, and string parameters.

**Main Purpose and Functionality**
==============================

*   The purpose of this function is to configure and add argument options for functions defined in `quant_trainer`.
*   It uses the `ArgumentParser` class from the `argparse` library to define the arguments.
*   These arguments are categorized into groups, with a focus on various quantization-related settings.

**Key Implementation Details**
-----------------------------

*   **Quantization Options**: `--wprec`, `--aprec`, `--quant-per-tensor`, `--quant-disable`, `--quant-disable-embeddings`, `--quant-disable-keyword`, and `--quant-disable-layer-module` control the behavior of various quantization algorithms.
*   **Calibration Options**: `--calibrator` and `--percentile` determine which calibrators to use and their respective percentile values.

**Notable Dependencies or Requirements**
---------------------------------------

*   The function relies on the `argparse` library for creating and manipulating command-line arguments.

No inherited or implemented interfaces are referenced in this code snippet.


key: chroma:document
string_value:
CODE:

def set_default_quantizers(args):
    """Set default quantizers before creating the model."""

    if args.calibrator == "max":
        calib_method = "max"
    elif args.calibrator == "percentile":
        if args.percentile is None:
            raise ValueError("Specify --percentile when using percentile calibrator")
        calib_method = "histogram"
    elif args.calibrator == "mse":
        calib_method = "histogram"
    else:
        raise ValueError(f"Invalid calibrator {args.calibrator}")

    input_desc = QuantDescriptor(num_bits=args.aprec, calib_method=calib_method)
    weight_desc = QuantDescriptor(num_bits=args.wprec, axis=(None if args.quant_per_tensor else (0,)))
    quant_nn.QuantLinear.set_default_quant_desc_input(input_desc)


SUMMARY:
**Summary**

Sets default quantizers for the model creation process, handling calibration methods based on the specified `calibrator` argument.

**Main Purpose and Functionality**
```python
def set_default_quantizers(args):
    """Set default quantizers before creating the model."""
```
Preserves exact technical terms:
- QuantDescriptor
- num_bits
- calib_method

 Primary purpose: Set calibration method based on `args.calibrator`.

**Key Implementation Details**

* Handles three calibration methods:
	+ Max (`max` calibrator)
	+ Percentile (`percentile` calibrator) with optional `--percentile` specification
	+ MSE (`mse` calibrator)

* Sets QuantDescriptor inputs and weights for QuantLinear module.

**Notable Dependencies or Requirements**

- QuantDescriptor (class)
- num_bits (parameter)
- calib_method (parameter)
- quant_nn.QquantLinear (module)
```
Note: External dependency on `quant_nn.QquantLinear` is present.


key: chroma:document
string_value:
CODE:

def configure_model(model, args, calib=False, eval=False):
    """Function called before the training loop."""

    logger.info("Configuring Model for Quantization")
    logger.info(f"using quantization package {pytorch_quantization.__file__}")

    if not calib:
        if args.quant_disable_embeddings:
            set_quantizer_by_name(model, ["embeddings"], which="weight", _disabled=True)

        if args.quant_disable:
            set_quantizer_by_name(model, [""], _disabled=True)

        if args.quant_disable_keyword:
            set_quantizer_by_name(model, args.quant_disable_keyword, _disabled=True)

        if args.quant_disable_layer_module:
            set_quantizer_by_name(model, [r"layer.\d+." + args.quant_disable_layer_module], _disabled=True)

        if args.quant_enable_layer_module:
            set_quantizer_by_name(model, [r"layer.\d+." + args.quant_enable_layer_module], _disabled=False)

        if args.recalibrate_weights:
            recalibrate_weights(model)

        if args.fuse_qkv:
            fuse_qkv(model, args)

    if args.clip_gelu:
        clip_gelu(model, args.clip_gelu)

    # if args.local_rank in [-1, 0] and not calib:


SUMMARY:
**Configure Model Function Summary**

### Purpose and Functionality

The `configure_model` function, called before the training loop, configures the behavior of a PyTorch model for quantization. It takes into account various parameters to control embedding, layer, and keyword quantization, as well as weight recalibration.

### Key Implementation Details

*   Configures Quantizer by name for:
    *   `embeddings`
    *   Specific layers using regular expressions
*   Enables or disables specific layers with `fuse_qkv` function
*   Recalibrates weights based on `recalibrate_weights` function
*   Clips GELU values based on `clip_gelu` function

### Notable Dependencies and Requirements

*   PyTorch Quantization package (`pytorch_quantization`)
*   `set_quantizer_by_name`, `fuse_qkv`, `recalibrate_weights`, `clip_gelu` functions
*   Logging module for logging information events (e.g., "Configuring Model for Quantization")

### Interface and Inheritance

The function relies on the following interfaces:

*   `Logger`: for logging information events
*   `set_quantizer_by_name`
*   `fuse_qkv`
*   `recalibrate_weights`
*   `clip_gelu`

Note that the specific implementation of these functions is not provided in this code chunk, but they are imported or referenced as described above.


key: chroma:document
string_value:
CODE:

def enable_calibration(model):
    """Enable calibration of all *_input_quantizer modules in model."""

    logger.info("Enabling Calibration")
    for name, module in model.named_modules():
        if name.endswith("_quantizer"):
            if module._calibrator is not None:
                module.disable_quant()
                module.enable_calib()
            else:
                module.disable()


SUMMARY:
**Summary**

```python
def enable_calibration(model):
    """
    Enables calibration of all *_input_quantizer modules in a given model.
    """

    
    # Disable quantization and enable calibration for each module with _quantizer suffix,
    # logging the module details before modification.
    for name, module in model.named_modules():
        if name.endswith("_quantizer"):
            if module._calibrator is not None:
                module.disable_quant()
                module.enable_calib()
            else:
                module.disable()
```

**Implementation Details**

*   The function iterates over the named modules of a given `model`.
*   It checks each module's suffix for `_quantizer` and applies specific behavior:
    *   If the module has a calibrator, it disables quantization and enables calibration.
    *   Otherwise, it simply disables the module.
*   For each module, it logs the module details before modification.

**Dependencies/Foundation**

*   The function relies on the `model` object provided as an argument.
*   It uses the `logger` object for logging purposes.


key: chroma:document
string_value:
CODE:

def finish_calibration(model, args):
    """Disable calibration and load amax for all "*_input_quantizer modules in model."""

    logger.info("Loading calibrated amax")
    for name, module in model.named_modules():
        if name.endswith("_quantizer"):
            if module._calibrator is not None:
                if isinstance(module._calibrator, calib.MaxCalibrator):
                    module.load_calib_amax()
                else:
                    module.load_calib_amax("percentile", percentile=args.percentile)
                module.enable_quant()
                module.disable_calib()
            else:
                module.enable()
    model.cuda()


SUMMARY:
**Summary:**

`finish_calibration` is a function that disables calibration and loads the Amax quantizer for all modules with "_quantizer" in their names within a given `model`. 

- **Primary Purpose:** Load calibrated Amax quantizer and enable quantization.

**Key Implementation Details:**

* Iterates through all named modules of the `model`.
* Checks if a module is of type `_quantizer` and has a `_calibrator`.
	+ If it's a `MaxCalibrator`, loads `amax` data, enables quantization, and disables calibration.
	+ Otherwise, loads `amax` data with custom percentile, enables quantization, and disables calibration.
* Enables CUDA context for the `model`.
* Prints a summary of the model using `print_quant_summary`.

**Notable Dependencies or Requirements:**

* Access to a logger instance (`logger.info`)
* A "percentile" argument (e.g., from `args` dictionary)
* Presence of `print_quant_summary` function
* Amax quantizer loading and calibration logic (`lib.calib.MaxCalibrator`)
* CUDA context support for the model (`torch.cuda`)


key: docstring
string_value:
# ========================================== Helper Function ==========================================


key: chroma:document
string_value:
CODE:

def fuse_qkv(model, args):
    """Adjust quantization ranges to match an implementation where the QKV projections are implemented with a single GEMM.
    Force the weight and output scale factors to match by taking the max of (Q,K,V).
    """

    def fuse3(qq, qk, qv):
        for mod in [qq, qk, qv]:
            if not hasattr(mod, "_amax"):
                print("          WARNING: NO AMAX BUFFER")
                return
        q = qq._amax.detach().item()
        k = qk._amax.detach().item()
        v = qv._amax.detach().item()

        amax = max(q, k, v)
        qq._amax.fill_(amax)
        qk._amax.fill_(amax)
        qv._amax.fill_(amax)
        logger.info(f"          q={q:5.2f} k={k:5.2f} v={v:5.2f} -> {amax:5.2f}")

    for name, mod in model.named_modules():
        if name.endswith(".attention.self"):
            logger.info(f"FUSE_QKV: {name:{name_width}}")
            fuse3(mod.matmul_q_input_quantizer, mod.matmul_k_input_quantizer, mod.matmul_v_input_quantizer)
            if args.quant_per_tensor:


SUMMARY:
**Summary**

Fuses QKV projections by matching quantization ranges with single GEMM implementation. This function takes two inputs:
- `model`: A PyTorch model containing attention modules
- `args`: Model arguments, including quantization per tensor flag (`quant_per_tensor`)


The function performs the following operations:

1. Defines a helper function `fuse3`, which:
	* Finds the maximum value of Q, K, and V buffers.
	* Updates the `_amax` buffer values in each dimension to match this maximum.

2. Loops through attention self-modules in the model, identifying `matmul_q_input_quantizer`, `matmul_k_input_quantizer`, and `matmul_v_input_quantizer`.

3. If `quant_per_tensor` is True, applies the fusion factor from the query, key, or value weight quantizers.

**Key Implementation Details**

- `fuse3` function updates `_amax` buffer values in each dimension using `fill_()` operations.
   - The maximum value is computed across Q, K, and V buffers using `max()` function.
  
- Single GEMM implementation matches Q, K, and V projections by setting the scale factors to a common maximum.

**Notable Dependencies or Requirements**

- PyTorch model with attention modules
- Model arguments (`args`) containing `quant_per_tensor` flag


key: chroma:document
string_value:
CODE:

def clip_gelu(model, maxval):
    """Clip activations generated by GELU to maxval when quantized.
    Implemented by adjusting the amax of the following input_quantizer.
    """

    for name, mod in model.named_modules():
        if name.endswith(".output.dense") and not name.endswith("attention.output.dense"):
            amax_init = mod._input_quantizer._amax.data.detach().item()
            mod._input_quantizer._amax.data.detach().clamp_(max=maxval)
            amax = mod._input_quantizer._amax.data.detach().item()


SUMMARY:
**Summary**

`clip_gelu`: A function that clips activations generated by GELU activation functions in a PyTorch model to maximum values (`maxval`) during quantization, potentially improving model accuracy.

**Key Implementation Details**

* Clips `amax` value of `_input_quantizer` module outputs (i.e., dense layers) within models.
* Log debug message for each clipped output with original and new `amax` values.

**Notable Dependencies/Requirements**

* PyTorch model (`model`) with quantization support
* `InputQuantizer` module, specifically its `.amax` attribute.


key: chroma:document
string_value:
CODE:

def expand_amax(model):
    """Expand per-tensor amax to be per channel, where each channel is assigned the per-tensor amax."""

    for name, mod in model.named_modules():
        if hasattr(mod, "_weight_quantizer") and mod._weight_quantizer.axis is not None:
            k = mod.weight.shape[0]
            amax = mod._weight_quantizer._amax.detach()
            mod._weight_quantizer._amax = torch.ones(k, dtype=amax.dtype, device=amax.device) * amax


SUMMARY:
**Summary:**

Expands per-tensor `amax` to be per-channel in a PyTorch model, where each channel is assigned the per-tensor `amax` using tensor broadcasting.

**Key Implementation Details:**

* Expands `_amax` attribute of `WeightQuantizer` modules by creating new tensors with shape `(k,)`, where `k` is the number of channels.
* Assigns the original amax value to a new tensor in the expanded model for each channel.

**Notable Dependencies and Requirements:**

* PyTorch library (`torch`)
* WeightQuantizer class (part of PyTorch's Quantization module)


key: chroma:document
string_value:
CODE:

def recalibrate_weights(model):
    """Performs max calibration on the weights and updates amax."""

    for name, mod in model.named_modules():
        if hasattr(mod, "_weight_quantizer"):
            if not hasattr(mod.weight_quantizer, "_amax"):
                print("RECALIB: {name:{name_width}} WARNING: NO AMAX BUFFER")
                continue

            # determine which axes to reduce across
            # e.g. a 4D tensor quantized per axis 0 should reduce over (1,2,3)
            axis_set = set() if mod._weight_quantizer.axis is None else set(mod._weight_quantizer.axis)
            reduce_axis = set(range(len(mod.weight.size()))) - axis_set
            amax = pytorch_quantization.utils.reduce_amax(mod.weight, axis=reduce_axis, keepdims=True).detach()
            logger.info(f"RECALIB: {name:{name_width}} {mod._weight_quantizer._amax.flatten()} -> {amax.flatten()}")


SUMMARY:
**Recalibration Function Summary**

This function recalibrates weights in a model by calculating the maximum absolute value (amax) of each weight tensor, reducing across unused axes, and updating the `_amax` buffer.

**Function Keys:**

* `name`: Module name
* `mod`: Model module
* `axis_set`: Axes to reduce across (set or list)
* `reduce_axis`: Unused axes coordinates (int list)
* `amax`: Maximum absolute value tensor

**Implementation Highlights:**

* Iterates over named modules in the model
* Checks for `_weight_quantizer` and its associated `_amax`
* Calculates amax by reducing aweight tensors along unused axes
* Updates `_amax` buffer with maximum absolute values

**Key Method Signature:**
```python
reduce_amax(mod.weight, axis=reduce_axis, keepdims=True)
```

**Notable Dependencies/Requirements:**

* `pytorch_quantization.utils`
* External dependency on PyTorch quantization utilities
                


key: chroma:document
string_value:
CODE:

def print_model_summary(model, name_width=25, line_width=180, ignore=None):
    """Print model quantization configuration."""

    if ignore is None:
        ignore = []
    elif not isinstance(ignore, list):
        ignore = [ignore]

    name_width = 0
    for name, mod in model.named_modules():
        if not hasattr(mod, "weight"):
            continue
        name_width = max(name_width, len(name))

    for name, mod in model.named_modules():
        input_q = getattr(mod, "_input_quantizer", None)
        weight_q = getattr(mod, "_weight_quantizer", None)
        if not hasattr(mod, "weight"):
            continue
        if type(mod) in ignore:
            continue
        if [True for s in ignore if isinstance(s, str) and s in name]:
            continue
        act_str = f"Act:{input_q.extra_repr()}"
        wgt_str = f"Wgt:{weight_q.extra_repr()}"
        s = f"{name:{name_width}} {act_str} {wgt_str}"
        if len(s) <= line_width:
            logger.info(s)
        else:
            logger.info(f"{name:{name_width}} {act_str}")


SUMMARY:
**Function Summary: `print_model_summary`**

Prints model quantization configuration.

*   Purpose: This function prints detailed information about the model's quantization configuration for each module.
*   Input Parameters:
    *   `model`: The PyTorch model instance to analyze.
    *   `name_width`: A maximum width for printing module names (default=25).
    *   `line_width`: Maximum length of a single line in the output (default=180).
    *   `ignore`: An optional list or string(s) specifying modules to exclude from the summary (default=None).

Key Implementation Details:

*   Iterates over model modules and filters based on whether they have a weight attribute.
*   Extracts quantization information for inputs and weights using extra_repr() methods.
*   Prints module names with their respective quantization details in two lines if width exceeds line limit.

Notable Dependencies/Requirements:

*   PyTorch (model instance)


key: chroma:document
string_value:
CODE:

def print_quant_summary(model):
    """Print summary of all quantizer modules in the model."""

    count = 0
    for name, mod in model.named_modules():
        if isinstance(mod, pytorch_quantization.nn.TensorQuantizer):
            print(f"{name:80} {mod}")
            count += 1


SUMMARY:
**Function Summary: `print_quant_summary`**

### Purpose and Functionality

Prints a summary of all tensor quantizer modules (TensorQuantizer) in the given PyTorch model.

**Key Implementation Details**

* Iterates through each named module in the model using `.named_modules()`.
* Checks if each module is an instance of `pytorch_quantization.nn.TensorQuantizer` and prints its details.
* Increments a counter to track the number of tensor quantizers found.

### Notable Dependencies/Requirements

* PyTorch (`model`)
* `pytorch_quantization(nn)`
* External dependency: `pytorch-quantization` library (not explicitly imported)


key: chroma:document
string_value:
CODE:

def set_quantizer(name, mod, quantizer, k, v):
    """Set attributes for mod.quantizer."""

    quantizer_mod = getattr(mod, quantizer, None)
    if quantizer_mod is not None:
        assert hasattr(quantizer_mod, k)
        setattr(quantizer_mod, k, v)
    else:


SUMMARY:
**Function Summary: `set_quantizer`**

Sets attributes on the quantizer module of an object `mod`.

* **Purpose:** Updates or sets an attribute on the existing quantizer module of `mod`.
* **Key Implementation Details:**
	+ Retrieves the quantizer module using `getattr` and ensures it's not None.
	+ Asserts that the quantizer module has the specified key (`k`) and updates its value with `v` if it exists.
* **Notable Dependencies/Requirements:**
	+ `mod.quantizer`: The object's quantizer module.
	+ `logger`: A logging instance (not shown in this code snippet).

This summary captures the essential details of the function, omitting unnecessary information and focusing on the core functionality.


key: chroma:document
string_value:
CODE:

def set_quantizers(name, mod, which="both", **kwargs):
    """Set quantizer attributes for mod."""

    s = f"Warning: changing {which} quantizers of {name:{qname_width}}"
    for k, v in kwargs.items():
        s += f" {k}={v}"
        if which in ["input", "both"]:
            set_quantizer(name, mod, "_input_quantizer", k, v)
        if which in ["weight", "both"]:
            set_quantizer(name, mod, "_weight_quantizer", k, v)


SUMMARY:
**Summary:**

`set_quantizers` function: sets quantizer attributes for model (`mod`) of `name`. Provides flexibility to select specific quantization layers (input/output) and accepts additional keyword arguments.

**Key Implementation Details:**

* Uses `logging` module with informative message (`s`).
* Iterates through keyword arguments (`kwargs`) to set quantizers for "input" and/or "weight" layers.
* Calls `set_quantizer` function within the loop, passing relevant parameters.

**Notable Dependencies or Requirements:**

* `logger` object (imported but not defined in this code snippet)
* `qname_width` variable (value not specified here, assumed to be defined elsewhere)

Note: The purpose of this summary is to provide a high-level understanding of the function's behavior and key components.


key: chroma:document
string_value:
CODE:

def set_quantizer_by_name(model, names, **kwargs):
    """Set quantizer attributes for layers where name contains a substring in names."""

    for name, mod in model.named_modules():
        if hasattr(mod, "_input_quantizer") or hasattr(mod, "_weight_quantizer"):
            for n in names:
                if re.search(n, name):
                    set_quantizers(name, mod, **kwargs)
        elif name.endswith("_quantizer"):
            for n in names:
                if re.search(n, name):
                    s = f"Warning: changing {name:{name_width}}"
                    for k, v in kwargs.items():
                        s += f" {k}={v}"
                        setattr(mod, k, v)


SUMMARY:
**Function Summary:**

`set_quantizer_by_name(model, names, **kwargs)`

Sets quantizer attributes for layers in the input model where the layer name contains a substring from `names`. Notifies warnings and updates attributions if applicable.

**Key Details:**

- Iterates through model layers using `named_modules()`.
- Checks for "_input_quantizer" or "_weight_quantizer" attributes.
- Updates quantizers based on name pattern matching in `names`.

**Implementation:**

This function modifies existing Quantization configurations and appends the updated attribute(s) to log a warning message.

**Notable Dependencies/Requirements:**

*   Related classes/functions:
    -   `model.named_modules()`: A module that allows iterating over model layers.
-   External dependencies:
    -   `re`: Python's standard library for regular expressions.

**Summary Conclusion:**
The function iterates through the model layers, identifying those with applicable quantizers and updating or adding configurations based on provided names.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/quantization-qdqbert/utils_qa.py


key: chroma:document
string_value:
CODE:

def postprocess_qa_predictions(
    examples,
    features,
    predictions: Tuple[np.ndarray, np.ndarray],
    version_2_with_negative: bool = False,
    n_best_size: int = 20,
    max_answer_length: int = 30,
    null_score_diff_threshold: float = 0.0,
    output_dir: Optional[str] = None,
    prefix: Optional[str] = None,
    log_level: Optional[int] = logging.WARNING,
):
    """
    Post-processes the predictions of a question-answering model to convert them to answers that are substrings of the
    original contexts. This is the base postprocessing functions for models that only return start and end logits.

    Args:
        examples: The non-preprocessed dataset (see the main script for more information).
        features: The processed dataset (see the main script for more information).
        predictions (:obj:`Tuple[np.ndarray, np.ndarray]`):
            The predictions of the model: two arrays containing the start logits and the end logits respectively. Its
            first dimension must match the number of elements of :obj:`features`.
        version_2_with_negative (:obj:`bool`, `optional`, defaults to :obj:`False`):
            Whether or not the underlying dataset contains examples with no answers.
        n_best_size (:obj:`int`, `optional`, defaults to 20):
            The total number of n-best predictions to generate when looking for an answer.
        max_answer_length (:obj:`int`, `optional`, defaults to 30):
            The maximum length of an answer that can be generated. This is needed because the start and end predictions
            are not conditioned on one another.
        null_score_diff_threshold (:obj:`float`, `optional`, defaults to 0):
            The threshold used to select the null answer: if the best answer has a score that is less than the score of
            the null answer minus this threshold, the null answer is selected for this example (note that the score of
            the null answer for an example giving several features is the minimum of the scores for the null answer on
            each feature: all features must be aligned on the fact they `want` to predict a null answer).

            Only useful when :obj:`version_2_with_negative` is :obj:`True`.
        output_dir (:obj:`str`, `optional`):
            If provided, the dictionaries of predictions, n_best predictions (with their scores and logits) and, if
            :obj:`version_2_with_negative=True`, the dictionary of the scores differences between best and null
            answers, are saved in `output_dir`.
        prefix (:obj:`str`, `optional`):
            If provided, the dictionaries mentioned above are saved with `prefix` added to their names.
        log_level (:obj:`int`, `optional`, defaults to ``logging.WARNING``):
            ``logging`` log level (e.g., ``logging.WARNING``)
    """
    if len(predictions) != 2:
        raise ValueError("`predictions` should be a tuple with two elements (start_logits, end_logits).")
    all_start_logits, all_end_logits = predictions

    if len(predictions[0]) != len(features):
        raise ValueError(f"Got {len(predictions[0])} predictions and {len(features)} features.")

    # Build a map example to its corresponding features.
    example_id_to_index = {k: i for i, k in enumerate(examples["id"])}
    features_per_example = collections.defaultdict(list)
    for i, feature in enumerate(features):
        features_per_example[example_id_to_index[feature["example_id"]]].append(i)

    # The dictionaries we have to fill.
    all_predictions = collections.OrderedDict()
    all_nbest_json = collections.OrderedDict()
    if version_2_with_negative:
        scores_diff_json = collections.OrderedDict()

    # Logging.
    logger.setLevel(log_level)
    logger.info(f"Post-processing {len(examples)} example predictions split into {len(features)} features.")

    # Let's loop over all the examples!
    for example_index, example in enumerate(tqdm(examples)):
        # Those are the indices of the features associated to the current example.
        feature_indices = features_per_example[example_index]

        min_null_prediction = None
        prelim_predictions = []

        # Looping through all the features associated to the current example.
        for feature_index in feature_indices:
            # We grab the predictions of the model for this feature.
            start_logits = all_start_logits[feature_index]
            end_logits = all_end_logits[feature_index]
            # This is what will allow us to map some the positions in our logits to span of texts in the original
            # context.
            offset_mapping = features[feature_index]["offset_mapping"]
            # Optional `token_is_max_context`, if provided we will remove answers that do not have the maximum context
            # available in the current feature.
            token_is_max_context = features[feature_index].get("token_is_max_context", None)

            # Update minimum null prediction.
            feature_null_score = start_logits[0] + end_logits[0]
            if min_null_prediction is None or min_null_prediction["score"] > feature_null_score:
                min_null_prediction = {
                    "offsets": (0, 0),
                    "score": feature_null_score,
                    "start_logit": start_logits[0],
                    "end_logit": end_logits[0],
                }

            # Go through all possibilities for the `n_best_size` greater start and end logits.
            start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()
            end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()
            for start_index in start_indexes:
                for end_index in end_indexes:
                    # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond
                    # to part of the input_ids that are not in the context.
                    if (
                        start_index >= len(offset_mapping)
                        or end_index >= len(offset_mapping)
                        or offset_mapping[start_index] is None
                        or len(offset_mapping[start_index]) < 2
                        or offset_mapping[end_index] is None
                        or len(offset_mapping[end_index]) < 2
                    ):
                        continue
                    # Don't consider answers with a length that is either < 0 or > max_answer_length.
                    if end_index < start_index or end_index - start_index + 1 > max_answer_length:
                        continue
                    # Don't consider answer that don't have the maximum context available (if such information is
                    # provided).
                    if token_is_max_context is not None and not token_is_max_context.get(str(start_index), False):
                        continue

                    prelim_predictions.append(
                        {
                            "offsets": (offset_mapping[start_index][0], offset_mapping[end_index][1]),
                            "score": start_logits[start_index] + end_logits[end_index],
                            "start_logit": start_logits[start_index],
                            "end_logit": end_logits[end_index],
                        }
                    )
        if version_2_with_negative:
            # Add the minimum null prediction
            prelim_predictions.append(min_null_prediction)
            null_score = min_null_prediction["score"]

        # Only keep the best `n_best_size` predictions.
        predictions = sorted(prelim_predictions, key=lambda x: x["score"], reverse=True)[:n_best_size]

        # Add back the minimum null prediction if it was removed because of its low score.
        if version_2_with_negative and not any(p["offsets"] == (0, 0) for p in predictions):
            predictions.append(min_null_prediction)

        # Use the offsets to gather the answer text in the original context.
        context = example["context"]
        for pred in predictions:
            offsets = pred.pop("offsets")
            pred["text"] = context[offsets[0] : offsets[1]]

        # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid
        # failure.
        if len(predictions) == 0 or (len(predictions) == 1 and predictions[0]["text"] == ""):
            predictions.insert(0, {"text": "empty", "start_logit": 0.0, "end_logit": 0.0, "score": 0.0})

        # Compute the softmax of all scores (we do it with numpy to stay independent from torch/tf in this file, using
        # the LogSumExp trick).
        scores = np.array([pred.pop("score") for pred in predictions])
        exp_scores = np.exp(scores - np.max(scores))
        probs = exp_scores / exp_scores.sum()

        # Include the probabilities in our predictions.
        for prob, pred in zip(probs, predictions):
            pred["probability"] = prob

        # Pick the best prediction. If the null answer is not possible, this is easy.
        if not version_2_with_negative:
            all_predictions[example["id"]] = predictions[0]["text"]
        else:
            # Otherwise we first need to find the best non-empty prediction.
            i = 0
            while predictions[i]["text"] == "":
                i += 1
            best_non_null_pred = predictions[i]

            # Then we compare to the null prediction using the threshold.
            score_diff = null_score - best_non_null_pred["start_logit"] - best_non_null_pred["end_logit"]
            scores_diff_json[example["id"]] = float(score_diff)  # To be JSON-serializable.
            if score_diff > null_score_diff_threshold:
                all_predictions[example["id"]] = ""
            else:
                all_predictions[example["id"]] = best_non_null_pred["text"]

        # Make `predictions` JSON-serializable by casting np.float back to float.
        all_nbest_json[example["id"]] = [
            {k: (float(v) if isinstance(v, (np.float16, np.float32, np.float64)) else v) for k, v in pred.items()}
            for pred in predictions
        ]

    # If we have an output_dir, let's save all those dicts.
    if output_dir is not None:
        if not os.path.isdir(output_dir):
            raise EnvironmentError(f"{output_dir} is not a directory.")

        prediction_file = os.path.join(
            output_dir, "predictions.json" if prefix is None else f"{prefix}_predictions.json"
        )
        nbest_file = os.path.join(
            output_dir, "nbest_predictions.json" if prefix is None else f"{prefix}_nbest_predictions.json"
        )
        if version_2_with_negative:
            null_odds_file = os.path.join(
                output_dir, "null_odds.json" if prefix is None else f"{prefix}_null_odds.json"
            )

        logger.info(f"Saving predictions to {prediction_file}.")
        with open(prediction_file, "w") as writer:
            writer.write(json.dumps(all_predictions, indent=4) + "\n")
        logger.info(f"Saving nbest_preds to {nbest_file}.")
        with open(nbest_file, "w") as writer:
            writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
        if version_2_with_negative:
            logger.info(f"Saving null_odds to {null_odds_file}.")
            with open(null_odds_file, "w") as writer:
                writer.write(json.dumps(scores_diff_json, indent=4) + "\n")



SUMMARY:
This is a function that takes in input data from a passage and outputs a prediction of the answer. The function processes the input data into individual predictions by identifying key phrases or spans with their corresponding scores, which are then ranked to determine the most likely answer.

The output format consists of two primary dictionaries:

* `all_predictions`: A dictionary where each key corresponds to an original passage ID and its value is a single string representing the predicted answer.
* `all_nbest_json`: Similarly a dictionary but containing JSON representations of individual prediction results.

This function is part of a larger Natural Language Processing (NLP) pipeline, designed for text classification applications where identifying most likely answers from a passage involves key phrase extraction and ranking. 

Key features:

*   Utilizes both LogSumExp trick for efficient scoring calculation using NumPy vectors, which maintains independence with T.F/Torch.
* Incorporates probability computation using softmax function to provide continuous scores over discrete predictions resulting in probabilities matching NBECE requirements.
* Provides the ability to generate "all nbest predictions" where a set of top-scoring solutions can be picked based on individual user preferences.


key: chroma:document
string_value:
CODE:

def postprocess_qa_predictions_with_beam_search(
    examples,
    features,
    predictions: Tuple[np.ndarray, np.ndarray],
    version_2_with_negative: bool = False,
    n_best_size: int = 20,
    max_answer_length: int = 30,
    start_n_top: int = 5,
    end_n_top: int = 5,
    output_dir: Optional[str] = None,
    prefix: Optional[str] = None,
    log_level: Optional[int] = logging.WARNING,
):
    """
    Post-processes the predictions of a question-answering model with beam search to convert them to answers that are substrings of the
    original contexts. This is the postprocessing functions for models that return start and end logits, indices, as well as
    cls token predictions.

    Args:
        examples: The non-preprocessed dataset (see the main script for more information).
        features: The processed dataset (see the main script for more information).
        predictions (:obj:`Tuple[np.ndarray, np.ndarray]`):
            The predictions of the model: two arrays containing the start logits and the end logits respectively. Its
            first dimension must match the number of elements of :obj:`features`.
        version_2_with_negative (:obj:`bool`, `optional`, defaults to :obj:`False`):
            Whether or not the underlying dataset contains examples with no answers.
        n_best_size (:obj:`int`, `optional`, defaults to 20):
            The total number of n-best predictions to generate when looking for an answer.
        max_answer_length (:obj:`int`, `optional`, defaults to 30):
            The maximum length of an answer that can be generated. This is needed because the start and end predictions
            are not conditioned on one another.
        start_n_top (:obj:`int`, `optional`, defaults to 5):
            The number of top start logits too keep when searching for the :obj:`n_best_size` predictions.
        end_n_top (:obj:`int`, `optional`, defaults to 5):
            The number of top end logits too keep when searching for the :obj:`n_best_size` predictions.
        output_dir (:obj:`str`, `optional`):
            If provided, the dictionaries of predictions, n_best predictions (with their scores and logits) and, if
            :obj:`version_2_with_negative=True`, the dictionary of the scores differences between best and null
            answers, are saved in `output_dir`.
        prefix (:obj:`str`, `optional`):
            If provided, the dictionaries mentioned above are saved with `prefix` added to their names.
        log_level (:obj:`int`, `optional`, defaults to ``logging.WARNING``):
            ``logging`` log level (e.g., ``logging.WARNING``)
    """
    if len(predictions) != 5:
        raise ValueError("`predictions` should be a tuple with five elements.")
    start_top_log_probs, start_top_index, end_top_log_probs, end_top_index, cls_logits = predictions

    if len(predictions[0]) != len(features):
        raise ValueError(f"Got {len(predictions[0])} predictions and {len(features)} features.")

    # Build a map example to its corresponding features.
    example_id_to_index = {k: i for i, k in enumerate(examples["id"])}
    features_per_example = collections.defaultdict(list)
    for i, feature in enumerate(features):
        features_per_example[example_id_to_index[feature["example_id"]]].append(i)

    # The dictionaries we have to fill.
    all_predictions = collections.OrderedDict()
    all_nbest_json = collections.OrderedDict()
    scores_diff_json = collections.OrderedDict() if version_2_with_negative else None

    # Logging.
    logger.setLevel(log_level)
    logger.info(f"Post-processing {len(examples)} example predictions split into {len(features)} features.")

    # Let's loop over all the examples!
    for example_index, example in enumerate(tqdm(examples)):
        # Those are the indices of the features associated to the current example.
        feature_indices = features_per_example[example_index]

        min_null_score = None
        prelim_predictions = []

        # Looping through all the features associated to the current example.
        for feature_index in feature_indices:
            # We grab the predictions of the model for this feature.
            start_log_prob = start_top_log_probs[feature_index]
            start_indexes = start_top_index[feature_index]
            end_log_prob = end_top_log_probs[feature_index]
            end_indexes = end_top_index[feature_index]
            feature_null_score = cls_logits[feature_index]
            # This is what will allow us to map some the positions in our logits to span of texts in the original
            # context.
            offset_mapping = features[feature_index]["offset_mapping"]
            # Optional `token_is_max_context`, if provided we will remove answers that do not have the maximum context
            # available in the current feature.
            token_is_max_context = features[feature_index].get("token_is_max_context", None)

            # Update minimum null prediction
            if min_null_score is None or feature_null_score < min_null_score:
                min_null_score = feature_null_score

            # Go through all possibilities for the `n_start_top`/`n_end_top` greater start and end logits.
            for i in range(start_n_top):
                for j in range(end_n_top):
                    start_index = int(start_indexes[i])
                    j_index = i * end_n_top + j
                    end_index = int(end_indexes[j_index])
                    # Don't consider out-of-scope answers (last part of the test should be unnecessary because of the
                    # p_mask but let's not take any risk)
                    if (
                        start_index >= len(offset_mapping)
                        or end_index >= len(offset_mapping)
                        or offset_mapping[start_index] is None
                        or offset_mapping[end_index] is None
                    ):
                        continue
                    # Don't consider answers with a length negative or > max_answer_length.
                    if end_index < start_index or end_index - start_index + 1 > max_answer_length:
                        continue
                    # Don't consider answer that don't have the maximum context available (if such information is
                    # provided).
                    if token_is_max_context is not None and not token_is_max_context.get(str(start_index), False):
                        continue
                    prelim_predictions.append(
                        {
                            "offsets": (offset_mapping[start_index][0], offset_mapping[end_index][1]),
                            "score": start_log_prob[i] + end_log_prob[j_index],
                            "start_log_prob": start_log_prob[i],
                            "end_log_prob": end_log_prob[j_index],
                        }
                    )

        # Only keep the best `n_best_size` predictions.
        predictions = sorted(prelim_predictions, key=lambda x: x["score"], reverse=True)[:n_best_size]

        # Use the offsets to gather the answer text in the original context.
        context = example["context"]
        for pred in predictions:
            offsets = pred.pop("offsets")
            pred["text"] = context[offsets[0] : offsets[1]]

        # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid
        # failure.
        if len(predictions) == 0:
            predictions.insert(0, {"text": "", "start_logit": -1e-6, "end_logit": -1e-6, "score": -2e-6})

        # Compute the softmax of all scores (we do it with numpy to stay independent from torch/tf in this file, using
        # the LogSumExp trick).
        scores = np.array([pred.pop("score") for pred in predictions])
        exp_scores = np.exp(scores - np.max(scores))
        probs = exp_scores / exp_scores.sum()

        # Include the probabilities in our predictions.
        for prob, pred in zip(probs, predictions):
            pred["probability"] = prob

        # Pick the best prediction and set the probability for the null answer.
        all_predictions[example["id"]] = predictions[0]["text"]
        if version_2_with_negative:
            scores_diff_json[example["id"]] = float(min_null_score)

        # Make `predictions` JSON-serializable by casting np.float back to float.
        all_nbest_json[example["id"]] = [
            {k: (float(v) if isinstance(v, (np.float16, np.float32, np.float64)) else v) for k, v in pred.items()}
            for pred in predictions
        ]

    # If we have an output_dir, let's save all those dicts.
    if output_dir is not None:
        if not os.path.isdir(output_dir):
            raise EnvironmentError(f"{output_dir} is not a directory.")

        prediction_file = os.path.join(
            output_dir, "predictions.json" if prefix is None else f"{prefix}_predictions.json"
        )
        nbest_file = os.path.join(
            output_dir, "nbest_predictions.json" if prefix is None else f"{prefix}_nbest_predictions.json"
        )
        if version_2_with_negative:
            null_odds_file = os.path.join(
                output_dir, "null_odds.json" if prefix is None else f"{prefix}_null_odds.json"
            )

        logger.info(f"Saving predictions to {prediction_file}.")
        with open(prediction_file, "w") as writer:
            writer.write(json.dumps(all_predictions, indent=4) + "\n")
        logger.info(f"Saving nbest_preds to {nbest_file}.")
        with open(nbest_file, "w") as writer:
            writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
        if version_2_with_negative:
            logger.info(f"Saving null_odds to {null_odds_file}.")
            with open(null_odds_file, "w") as writer:
                writer.write(json.dumps(scores_diff_json, indent=4) + "\n")



SUMMARY:
This is a Python function that generates predictions for text classification tasks. The function takes in several inputs and outputs a dictionary of prediction results, as well as two optional dictionaries for null odds.

Summary:

* The function processes a list of examples, each containing a context string and an id.
* For each example, it generates predictions by iterating over scores from the input data, selecting the best-scoring predictions (up to `n_best_size`), and computing their corresponding text offsets in the original context.
* It then uses these offsets to extract the answer texts from the contexts and creates a dictionary of prediction results.
* If `version_2_with_negative` is True, it also computes and returns the null odds for each example.

Key features:

* The function supports handling edge cases where there are no scoring predictions (`n_best_size == 0`).
* It uses NumPy arrays to efficiently compute and store intermediate scores and scores before computation.
* It iterates over scores from input data multiple times, resulting in a relatively high computational complexity.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/quantization-qdqbert/run_quant_qa.py


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    # quant_trainer arguments
    quant_trainer.add_arguments(parser)

    # if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
    #     # If we pass only one argument to the script and it's the path to a json file,
    #     # let's parse it to get our arguments.
    #     model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    # else:

    model_args, data_args, training_args, quant_trainer_args = parser.parse_args_into_dataclasses()

    # setup QAT training args for scheduler (default to use cosine annealing learning rate schedule)
    training_args.lr_scheduler_type = SchedulerType.COSINE

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )

    log_level = training_args.get_process_log_level()
    logger.setLevel(log_level)
    datasets.utils.logging.set_verbosity(log_level)
    transformers.utils.logging.set_verbosity(log_level)
    transformers.utils.logging.enable_default_handler()
    transformers.utils.logging.enable_explicit_format()

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
        + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    logger.info(f"Training/evaluation parameters {training_args}")

    # Detecting last checkpoint.
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        raw_datasets = load_dataset(
            data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir
        )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]

        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if data_args.test_file is not None:
            data_files["test"] = data_args.test_file
            extension = data_args.test_file.split(".")[-1]
        raw_datasets = load_dataset(extension, data_files=data_files, field="data", cache_dir=model_args.cache_dir)
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    # set default quantization parameters before building model
    quant_trainer.set_default_quantizers(quant_trainer_args)

    # Load pretrained model and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    config = QDQBertConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=True if model_args.use_auth_token else None,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=True,
        revision=model_args.model_revision,
        token=True if model_args.use_auth_token else None,
    )
    model = QDQBertForQuestionAnswering.from_pretrained(
        model_args.model_name_or_path,
        from_tf=bool(".ckpt" in model_args.model_name_or_path),
        config=config,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=True if model_args.use_auth_token else None,
    )

    # Tokenizer check: this script requires a fast tokenizer.
    if not isinstance(tokenizer, PreTrainedTokenizerFast):
        raise ValueError(
            "This example script only works for models that have a fast tokenizer. Checkout the big table of models at"
            " https://huggingface.co/transformers/index.html#supported-frameworks to find the model types that meet"
            " this requirement"
        )

    # Preprocessing the datasets.
    # Preprocessing is slightly different for training and evaluation.
    if training_args.do_train or model_args.do_calib:
        column_names = raw_datasets["train"].column_names
    elif training_args.do_eval or model_args.save_onnx:
        column_names = raw_datasets["validation"].column_names
    else:
        column_names = raw_datasets["test"].column_names
    question_column_name = "question" if "question" in column_names else column_names[0]
    context_column_name = "context" if "context" in column_names else column_names[1]
    answer_column_name = "answers" if "answers" in column_names else column_names[2]

    # Padding side determines if we do (question|context) or (context|question).
    pad_on_right = tokenizer.padding_side == "right"

    if data_args.max_seq_length > tokenizer.model_max_length:
        logger.warning(
            f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the "
            f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
        )
    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    # Training preprocessing
    def prepare_train_features(examples):
        # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results
        # in one example possible giving several features when a context is long, each of those features having a
        # context that overlaps a bit the context of the previous feature.
        tokenized_examples = tokenizer(
            examples[question_column_name if pad_on_right else context_column_name],
            examples[context_column_name if pad_on_right else question_column_name],
            truncation="only_second" if pad_on_right else "only_first",
            max_length=max_seq_length,
            stride=data_args.doc_stride,
            return_overflowing_tokens=True,
            return_offsets_mapping=True,
            padding="max_length" if data_args.pad_to_max_length else False,
        )

        # Since one example might give us several features if it has a long context, we need a map from a feature to
        # its corresponding example. This key gives us just that.
        sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")
        # The offset mappings will give us a map from token to character position in the original context. This will
        # help us compute the start_positions and end_positions.
        offset_mapping = tokenized_examples.pop("offset_mapping")

        # Let's label those examples!
        tokenized_examples["start_positions"] = []
        tokenized_examples["end_positions"] = []

        for i, offsets in enumerate(offset_mapping):
            # We will label impossible answers with the index of the CLS token.
            input_ids = tokenized_examples["input_ids"][i]
            cls_index = input_ids.index(tokenizer.cls_token_id)

            # Grab the sequence corresponding to that example (to know what is the context and what is the question).
            sequence_ids = tokenized_examples.sequence_ids(i)

            # One example can give several spans, this is the index of the example containing this span of text.
            sample_index = sample_mapping[i]
            answers = examples[answer_column_name][sample_index]
            # If no answers are given, set the cls_index as answer.
            if len(answers["answer_start"]) == 0:
                tokenized_examples["start_positions"].append(cls_index)
                tokenized_examples["end_positions"].append(cls_index)
            else:
                # Start/end character index of the answer in the text.
                start_char = answers["answer_start"][0]
                end_char = start_char + len(answers["text"][0])

                # Start token index of the current span in the text.
                token_start_index = 0
                while sequence_ids[token_start_index] != (1 if pad_on_right else 0):
                    token_start_index += 1

                # End token index of the current span in the text.
                token_end_index = len(input_ids) - 1
                while sequence_ids[token_end_index] != (1 if pad_on_right else 0):
                    token_end_index -= 1

                # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).
                if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):
                    tokenized_examples["start_positions"].append(cls_index)
                    tokenized_examples["end_positions"].append(cls_index)
                else:
                    # Otherwise move the token_start_index and token_end_index to the two ends of the answer.
                    # Note: we could go after the last offset if the answer is the last word (edge case).
                    while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:
                        token_start_index += 1
                    tokenized_examples["start_positions"].append(token_start_index - 1)
                    while offsets[token_end_index][1] >= end_char:
                        token_end_index -= 1
                    tokenized_examples["end_positions"].append(token_end_index + 1)

        return tokenized_examples

    if training_args.do_train or model_args.do_calib:
        if "train" not in raw_datasets:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = raw_datasets["train"]
        if data_args.max_train_samples is not None:
            # We will select sample from whole data if argument is specified
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))
        # Create train feature from dataset
        with training_args.main_process_first(desc="train dataset map pre-processing"):
            train_dataset = train_dataset.map(
                prepare_train_features,
                batched=True,
                num_proc=data_args.preprocessing_num_workers,
                remove_columns=column_names,
                load_from_cache_file=not data_args.overwrite_cache,
                desc="Running tokenizer on train dataset",
            )
        if data_args.max_train_samples is not None:
            # Number of samples might increase during Feature Creation, We select only specified max samples
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))

    # Validation preprocessing
    def prepare_validation_features(examples):
        # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results
        # in one example possible giving several features when a context is long, each of those features having a
        # context that overlaps a bit the context of the previous feature.
        tokenized_examples = tokenizer(
            examples[question_column_name if pad_on_right else context_column_name],
            examples[context_column_name if pad_on_right else question_column_name],
            truncation="only_second" if pad_on_right else "only_first",
            max_length=max_seq_length,
            stride=data_args.doc_stride,
            return_overflowing_tokens=True,
            return_offsets_mapping=True,
            padding="max_length" if data_args.pad_to_max_length else False,
        )

        # Since one example might give us several features if it has a long context, we need a map from a feature to
        # its corresponding example. This key gives us just that.
        sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

        # For evaluation, we will need to convert our predictions to substrings of the context, so we keep the
        # corresponding example_id and we will store the offset mappings.
        tokenized_examples["example_id"] = []

        for i in range(len(tokenized_examples["input_ids"])):
            # Grab the sequence corresponding to that example (to know what is the context and what is the question).
            sequence_ids = tokenized_examples.sequence_ids(i)
            context_index = 1 if pad_on_right else 0

            # One example can give several spans, this is the index of the example containing this span of text.
            sample_index = sample_mapping[i]
            tokenized_examples["example_id"].append(examples["id"][sample_index])

            # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token
            # position is part of the context or not.
            tokenized_examples["offset_mapping"][i] = [
                (o if sequence_ids[k] == context_index else None)
                for k, o in enumerate(tokenized_examples["offset_mapping"][i])
            ]

        return tokenized_examples

    if training_args.do_eval or model_args.save_onnx:
        if "validation" not in raw_datasets:
            raise ValueError("--do_eval requires a validation dataset")
        eval_examples = raw_datasets["validation"]
        if data_args.max_eval_samples is not None:
            # We will select sample from whole data
            max_eval_samples = min(len(eval_examples), data_args.max_eval_samples)
            eval_examples = eval_examples.select(range(max_eval_samples))
        # Validation Feature Creation
        with training_args.main_process_first(desc="validation dataset map pre-processing"):
            eval_dataset = eval_examples.map(
                prepare_validation_features,
                batched=True,
                num_proc=data_args.preprocessing_num_workers,
                remove_columns=column_names,
                load_from_cache_file=not data_args.overwrite_cache,
                desc="Running tokenizer on validation dataset",
            )
        if data_args.max_eval_samples is not None:
            # During Feature creation dataset samples might increase, we will select required samples again
            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
            eval_dataset = eval_dataset.select(range(max_eval_samples))

    if training_args.do_predict:
        if "test" not in raw_datasets:
            raise ValueError("--do_predict requires a test dataset")
        predict_examples = raw_datasets["test"]
        if data_args.max_predict_samples is not None:
            # We will select sample from whole data
            predict_examples = predict_examples.select(range(data_args.max_predict_samples))
        # Predict Feature Creation
        with training_args.main_process_first(desc="prediction dataset map pre-processing"):
            predict_dataset = predict_examples.map(
                prepare_validation_features,
                batched=True,
                num_proc=data_args.preprocessing_num_workers,
                remove_columns=column_names,
                load_from_cache_file=not data_args.overwrite_cache,
                desc="Running tokenizer on prediction dataset",
            )
        if data_args.max_predict_samples is not None:
            # During Feature creation dataset samples might increase, we will select required samples again
            max_predict_samples = min(len(predict_dataset), data_args.max_predict_samples)
            predict_dataset = predict_dataset.select(range(max_predict_samples))

    # Data collator
    # We have already padded to max length if the corresponding flag is True, otherwise we need to pad in the data
    # collator.
    data_collator = (
        default_data_collator
        if data_args.pad_to_max_length
        else DataCollatorWithPadding(tokenizer, pad_to_multiple_of=8 if training_args.fp16 else None)
    )

    # Post-processing:
    def post_processing_function(examples, features, predictions, stage="eval"):
        # Post-processing: we match the start logits and end logits to answers in the original context.
        predictions = postprocess_qa_predictions(
            examples=examples,
            features=features,
            predictions=predictions,
            version_2_with_negative=data_args.version_2_with_negative,
            n_best_size=data_args.n_best_size,
            max_answer_length=data_args.max_answer_length,
            null_score_diff_threshold=data_args.null_score_diff_threshold,
            output_dir=training_args.output_dir,
            log_level=log_level,
            prefix=stage,
        )
        # Format the result to the format the metric expects.
        if data_args.version_2_with_negative:
            formatted_predictions = [
                {"id": k, "prediction_text": v, "no_answer_probability": 0.0} for k, v in predictions.items()
            ]
        else:
            formatted_predictions = [{"id": k, "prediction_text": v} for k, v in predictions.items()]

        references = [{"id": ex["id"], "answers": ex[answer_column_name]} for ex in examples]
        return EvalPrediction(predictions=formatted_predictions, label_ids=references)

    metric = load_metric("squad_v2" if data_args.version_2_with_negative else "squad")

    def compute_metrics(p: EvalPrediction):
        return metric.compute(predictions=p.predictions, references=p.label_ids)

    # Initialize our Trainer
    trainer = QuestionAnsweringTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset if training_args.do_train or model_args.do_calib else None,
        eval_dataset=eval_dataset if training_args.do_eval or model_args.save_onnx else None,
        eval_examples=eval_examples if training_args.do_eval or model_args.save_onnx else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
        post_process_function=post_processing_function,
        compute_metrics=compute_metrics,
        quant_trainer_args=quant_trainer_args,
    )

    # Calibration
    if model_args.do_calib:
        logger.info("*** Calibrate ***")
        results = trainer.calibrate()
        trainer.save_model()

    # Training
    if training_args.do_train:
        checkpoint = None
        if training_args.resume_from_checkpoint is not None:
            checkpoint = training_args.resume_from_checkpoint
        elif last_checkpoint is not None:
            checkpoint = last_checkpoint

        quant_trainer.configure_model(trainer.model, quant_trainer_args)

        train_result = trainer.train(resume_from_checkpoint=checkpoint)
        trainer.save_model()  # Saves the tokenizer too for easy upload

        metrics = train_result.metrics
        max_train_samples = (
            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)
        )
        metrics["train_samples"] = min(max_train_samples, len(train_dataset))

        trainer.log_metrics("train", metrics)
        trainer.save_metrics("train", metrics)
        trainer.save_state()

    # Evaluation
    if training_args.do_eval:
        logger.info("*** Evaluate ***")
        quant_trainer.configure_model(trainer.model, quant_trainer_args, eval=True)
        metrics = trainer.evaluate()

        max_eval_samples = data_args.max_eval_samples if data_args.max_eval_samples is not None else len(eval_dataset)
        metrics["eval_samples"] = min(max_eval_samples, len(eval_dataset))

        trainer.log_metrics("eval", metrics)
        trainer.save_metrics("eval", metrics)

    # Prediction
    if training_args.do_predict:
        logger.info("*** Predict ***")
        results = trainer.predict(predict_dataset, predict_examples)
        metrics = results.metrics

        max_predict_samples = (
            data_args.max_predict_samples if data_args.max_predict_samples is not None else len(predict_dataset)
        )
        metrics["predict_samples"] = min(max_predict_samples, len(predict_dataset))

        trainer.log_metrics("predict", metrics)
        trainer.save_metrics("predict", metrics)

    if training_args.push_to_hub:
        kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "question-answering"}
        if data_args.dataset_name is not None:
            kwargs["dataset_tags"] = data_args.dataset_name
            if data_args.dataset_config_name is not None:
                kwargs["dataset_args"] = data_args.dataset_config_name
                kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
            else:
                kwargs["dataset"] = data_args.dataset_name

        trainer.push_to_hub(**kwargs)

    if model_args.save_onnx:
        logger.info("Exporting model to onnx")


SUMMARY:
Here is a concise summary of the provided code:

The code implements a Question Answering Trainer for the Hugging Face library using the Squad V2 dataset. It trains, evaluates, and predicts on a given dataset with optional calibration and model saving. The trainer uses data collators and post-processing functions to optimize performance. The code also includes options for resume from checkpoint, eval, prediction, and saving metrics.

The code consists of several sections:

1. Importing libraries, datasets, and models.
2. Preprocessing the data (tokenizerization).
3. Training a Question Answering model with calibration.
4. Evaluating the model on a test dataset.
5. Making predictions on new instances.
6. Exporting the trained model to ONNX format.

The code is designed to be modular, allowing for easy extension and customization of the training process.


key: chroma:document
string_value:
CODE:

def _mp_fn(index):
    # For xla_spawn (TPUs)


SUMMARY:
**Function Summary**

`_mp_fn` appears to be an execution context handler for `xla_spawn` (TensorFlow's Platform-Independent compilation and runtime) on TPUs. It calls the main function, likely as part of a multi-processing or parallel processing framework.

**Primary Purpose**: Executes the main function within an XLA-Spawn context.

**Implementation Details**:
- Calls `main()`.
 
**Dependencies and Requirements**:
- TensorFlow (for xla_spawn).
- Unknown main() function signature.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/quantization-qdqbert/evaluate-hf-trt-qa.py


key: chroma:document
string_value:
CODE:
# run inference with TRT
def model_infer(inputs, context, d_inputs, h_output0, h_output1, d_output0, d_output1, stream):
    input_ids = np.asarray(inputs["input_ids"], dtype=np.int32)
    attention_mask = np.asarray(inputs["attention_mask"], dtype=np.int32)
    token_type_ids = np.asarray(inputs["token_type_ids"], dtype=np.int32)

    # Copy inputs
    cuda.memcpy_htod_async(d_inputs[0], input_ids.ravel(), stream)
    cuda.memcpy_htod_async(d_inputs[1], attention_mask.ravel(), stream)
    cuda.memcpy_htod_async(d_inputs[2], token_type_ids.ravel(), stream)
    # start time
    start_time = time.time()
    # Run inference
    context.execute_async(
        bindings=[int(d_inp) for d_inp in d_inputs] + [int(d_output0), int(d_output1)], stream_handle=stream.handle
    )
    # Transfer predictions back from GPU
    cuda.memcpy_dtoh_async(h_output0, d_output0, stream)
    cuda.memcpy_dtoh_async(h_output1, d_output1, stream)
    # Synchronize the stream and take time
    stream.synchronize()
    # end time
    end_time = time.time()
    infer_time = end_time - start_time
    outputs = (h_output0, h_output1)
    # print(outputs)


SUMMARY:
**Summary**

`model_infer` Function Analysis
==================================

### Purpose and Functionality

The `model_infer` function runs inference on a model using the Tensor Rocket (TRT) framework. It takes in input data, context, and various output parameters.

**Key Implementation Details**

*   The function copies input data to the GPU using CUDA.
*   It executes the asynchronous inference request using the `context.execute_async` method, passing in bindings for input inputs and output parameters.
*   The predictions are transferred back from the GPU using CUDA's `memcpy_dtoh_async` methods.

**Notable Dependencies and Requirements**

*   Required:
    *   `inputs`: dictionary containing `"input_ids"`, `"attention_mask"`, and `"token_type_ids"`
    *   `d_inputs`: list of length 3, representing device memory locations for input data
    *   `h_output0` and `h_output1`: device memory locations for output predictions
*   External Dependencies:
    *   CUDA (NVIDIA GPU acceleration engine)
    *   OpenTensorRuntime (TensorRT Integration)

### Method Signatures and Interfaces

*   `context.execute_async`
*   `cuda.memcpy_htod_async`
*   `cuda.memcpy_dtoh_async`

Note: The function returns a tuple containing the output predictions (`(h_output0, h_output1)`).


key: chroma:document
string_value:
CODE:
# Validation preprocessing
def prepare_validation_features(examples):
    # Some of the questions have lots of whitespace on the left, which is not useful and will make the
    # truncation of the context fail (the tokenized question will take a lots of space). So we remove that
    # left whitespace
    examples[question_column_name] = [q.lstrip() for q in examples[question_column_name]]

    # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results
    # in one example possible giving several features when a context is long, each of those features having a
    # context that overlaps a bit the context of the previous feature.
    tokenized_examples = tokenizer(
        examples[question_column_name if pad_on_right else context_column_name],
        examples[context_column_name if pad_on_right else question_column_name],
        truncation="only_second" if pad_on_right else "only_first",
        max_length=max_seq_length,
        stride=args.doc_stride,
        return_overflowing_tokens=True,
        return_offsets_mapping=True,
        padding="max_length",
    )

    # Since one example might give us several features if it has a long context, we need a map from a feature to
    # its corresponding example. This key gives us just that.
    sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

    # For evaluation, we will need to convert our predictions to substrings of the context, so we keep the
    # corresponding example_id and we will store the offset mappings.
    tokenized_examples["example_id"] = []

    for i in range(len(tokenized_examples["input_ids"])):
        # Grab the sequence corresponding to that example (to know what is the context and what is the question).
        sequence_ids = tokenized_examples.sequence_ids(i)
        context_index = 1 if pad_on_right else 0

        # One example can give several spans, this is the index of the example containing this span of text.
        sample_index = sample_mapping[i]
        tokenized_examples["example_id"].append(examples["id"][sample_index])

        # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token
        # position is part of the context or not.
        tokenized_examples["offset_mapping"][i] = [
            (o if sequence_ids[k] == context_index else None)
            for k, o in enumerate(tokenized_examples["offset_mapping"][i])
        ]



SUMMARY:
**Concise Summary**

### Main Purpose and Functionality

Prepares validation features by preprocessing text data. The function:

1. Removes left whitespace from questions
2. Tokenizes input IDs and attention masks with truncation and padding, preserving context overlap
3. Maintains a mapping between feature indices and example IDs
4. Prepends example IDs and offset mappings to the tokenized output

### Key Implementation Details

* Uses `tokenizer` function to perform sequence encodings
* Specifies truncation strategy, max length, stride, and padding scheme for tokenization
* Uses `return_overflowing_tokens=True` and `return_offsets_mapping=True` to preserve overlapping tokens and context offsets

### Notable Dependencies or Requirements

* `tokenizer` function (Bert tokenizer or similar)
* `args.doc_stride`

**RAG System Representation**
```
ValidationPrep_preprocess
  MainPurposeAndFunctionality
    RemoveLeftWhitespaceFromQuestions
    TokenizeInputIdsAndAttentionMasks
  KeyImplementationDetails
    UseTruncationForTokenization
    SpecifyPaddingScheme
  NotableDependenciesOrRequirements
    BertTokenizer
    args.doc_stride
```


key: chroma:document
string_value:
CODE:
# Post-processing:
def post_processing_function(examples, features, predictions, stage="eval"):
    # Post-processing: we match the start logits and end logits to answers in the original context.
    predictions = postprocess_qa_predictions(
        examples=examples,
        features=features,
        predictions=predictions,
        version_2_with_negative=args.version_2_with_negative,
        n_best_size=args.n_best_size,
        max_answer_length=args.max_answer_length,
        null_score_diff_threshold=args.null_score_diff_threshold,
        output_dir=args.output_dir,
        prefix=stage,
    )
    # Format the result to the format the metric expects.
    if args.version_2_with_negative:
        formatted_predictions = [
            {"id": k, "prediction_text": v, "no_answer_probability": 0.0} for k, v in predictions.items()
        ]
    else:
        formatted_predictions = [{"id": k, "prediction_text": v} for k, v in predictions.items()]

    references = [{"id": ex["id"], "answers": ex[answer_column_name]} for ex in examples]


SUMMARY:
**Function Summary**

`post_processing_function`
=====================================

**Purpose**
-----------

Post-processesQA prediction results to format them for evaluation metrics.

**Main Functionality**

1. Calls `postprocess_qa_predictions` function to refine prediction ratings by matching start logits and end logits with context answers.
2. Formats predictions into a specified output format based on `version_2_with_negative` argument.
3. Constructs references for evaluation metric calculation using original example data.

**Key Implementation Details**

* Uses `args.version_2_with_negative`, `args.n_best_size`, `args.max_answer_length`, and `args.null_score_diff_threshold` as input parameters.
* Formats predictions using dictionary comprehensions to match expected output format.
* References: accesses `examples` and extracts relevant columns for constructing references.

**Dependencies/Requirements**

* `postprocess_qa_predictions`
* `args`: object with arguments (`version_2_with_negative`, `n_best_size`, `max_answer_length`, `null_score_diff_threshold`, `output_dir`, `prefix`)


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/self-training-text-classification/finetuning.py


key: chroma:document
string_value:
CODE:

class Split(ExplicitEnum):
    TRAIN = "train"
    EVAL = "eval"
    TEST = "test"


SUMMARY:
**Summary:**

This `Split` class defines an enum with four values (`TRAIN`, `EVAL`, `TEST`, and `INFER`) used to specify data split configurations. It inherits from `ExplicitEnum`. No external dependencies are referenced.

**Key Details:**

- **Class Functionality**: Specifies data split configurations.
- **Method Signatures:** None directly exposed, but value enumerations facilitate their use.
- **Notable Dependencies:** None, as it's an Enum-based class with only its enumeration values specified.

Note: An explicit enum declaration is likely to be part of a larger context or specific framework requirements.


key: chroma:document
string_value:
CODE:

def train(args, accelerator, model, tokenizer, train_dataloader, optimizer, lr_scheduler, eval_dataloader=None):
    """Train a model on the given training data."""

    total_batch_size = args.per_device_train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps

    logger.info("***** Running training *****")
    logger.info("  Num examples = %d", args.num_examples[Split.TRAIN.value])
    logger.info("  Instantaneous batch size per device = %d", args.per_device_train_batch_size)
    logger.info("  Total train batch size (w. parallel, distributed & accumulation) = %d", total_batch_size)
    logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
    logger.info("  Total optimization steps = %d", args.max_steps)

    # Only show the progress bar once on each machine.
    progress_bar = tqdm(range(args.max_steps), disable=not accelerator.is_local_main_process)

    checkpoints = None
    eval_results = None
    best_checkpoint = None
    best_eval_result = None
    early_stopping_patience_counter = 0
    should_training_stop = False
    epoch = 0
    completed_steps = 0
    train_loss = 0.0
    model.zero_grad()

    for _ in range(args.num_train_epochs):
        epoch += 1
        model.train()
        for step, batch in enumerate(train_dataloader):
            outputs = model(**batch)
            loss = outputs.loss
            loss = loss / args.gradient_accumulation_steps
            accelerator.backward(loss)
            train_loss += loss.item()

            if step % args.gradient_accumulation_steps == 0 or step == len(train_dataloader) - 1:
                optimizer.step()
                lr_scheduler.step()
                optimizer.zero_grad()
                progress_bar.update(1)
                completed_steps += 1

                # Evaluate during training
                if (
                    eval_dataloader is not None
                    and args.eval_strategy == IntervalStrategy.STEPS.value
                    and args.eval_steps > 0
                    and completed_steps % args.eval_steps == 0
                ):
                    accelerator.wait_for_everyone()
                    new_checkpoint = f"checkpoint-{IntervalStrategy.STEPS.value}-{completed_steps}"
                    new_eval_result = evaluate(args, accelerator, eval_dataloader, "eval", model, new_checkpoint)[
                        args.eval_metric
                    ]
                    logger.info(
                        "Evaluation result at step %d: %s = %f", completed_steps, args.eval_metric, new_eval_result
                    )
                    if checkpoints is None:
                        checkpoints = np.array([new_checkpoint])
                        eval_results = np.array([new_eval_result])
                        best_checkpoint = new_checkpoint
                        best_eval_result = new_eval_result
                    else:
                        if new_eval_result - best_eval_result > args.early_stopping_threshold:
                            best_checkpoint = new_checkpoint
                            best_eval_result = new_eval_result
                            early_stopping_patience_counter = 0
                        else:
                            if new_eval_result == best_eval_result:
                                best_checkpoint = new_checkpoint
                                best_eval_result = new_eval_result
                            early_stopping_patience_counter += 1

                        if early_stopping_patience_counter >= args.early_stopping_patience:
                            should_training_stop = True

                        checkpoints = np.append(checkpoints, [new_checkpoint], axis=0)
                        eval_results = np.append(eval_results, [new_eval_result], axis=0)
                        sorted_ids = np.argsort(eval_results)
                        eval_results = eval_results[sorted_ids]
                        checkpoints = checkpoints[sorted_ids]

                    if len(checkpoints) > args.keep_checkpoint_max:
                        # Delete the current worst checkpoint
                        checkpoint_to_remove, *checkpoints = checkpoints
                        eval_results = eval_results[1:]
                        if checkpoint_to_remove != new_checkpoint:
                            if accelerator.is_main_process:
                                shutil.rmtree(os.path.join(args.output_dir, checkpoint_to_remove), ignore_errors=True)
                            accelerator.wait_for_everyone()

                    if new_checkpoint in checkpoints:
                        # Save model checkpoint
                        checkpoint_output_dir = os.path.join(args.output_dir, new_checkpoint)
                        if accelerator.is_main_process:
                            if not os.path.exists(checkpoint_output_dir):
                                os.makedirs(checkpoint_output_dir)
                        accelerator.wait_for_everyone()
                        unwrapped_model = accelerator.unwrap_model(model)
                        unwrapped_model.save_pretrained(checkpoint_output_dir, save_function=accelerator.save)
                        if accelerator.is_main_process:
                            tokenizer.save_pretrained(checkpoint_output_dir)
                            logger.info("Saving model checkpoint to %s", checkpoint_output_dir)

            if completed_steps >= args.max_steps:
                break

            if should_training_stop:
                break

        # Evaluate during training
        if eval_dataloader is not None and args.eval_strategy == IntervalStrategy.EPOCH.value:
            accelerator.wait_for_everyone()
            new_checkpoint = f"checkpoint-{IntervalStrategy.EPOCH.value}-{epoch}"
            new_eval_result = evaluate(args, accelerator, eval_dataloader, "eval", model, new_checkpoint)[
                args.eval_metric
            ]
            logger.info("Evaluation result at epoch %d: %s = %f", epoch, args.eval_metric, new_eval_result)

            if checkpoints is None:
                checkpoints = np.array([new_checkpoint])
                eval_results = np.array([new_eval_result])
                best_checkpoint = new_checkpoint
                best_eval_result = new_eval_result
            else:
                if new_eval_result - best_eval_result > args.early_stopping_threshold:
                    best_checkpoint = new_checkpoint
                    best_eval_result = new_eval_result
                    early_stopping_patience_counter = 0
                else:
                    if new_eval_result == best_eval_result:
                        best_checkpoint = new_checkpoint
                        best_eval_result = new_eval_result
                    early_stopping_patience_counter += 1

                if early_stopping_patience_counter >= args.early_stopping_patience:
                    should_training_stop = True

                checkpoints = np.append(checkpoints, [new_checkpoint], axis=0)
                eval_results = np.append(eval_results, [new_eval_result], axis=0)
                sorted_ids = np.argsort(eval_results)
                eval_results = eval_results[sorted_ids]
                checkpoints = checkpoints[sorted_ids]

            if len(checkpoints) > args.keep_checkpoint_max:
                # Delete the current worst checkpoint
                checkpoint_to_remove, *checkpoints = checkpoints
                eval_results = eval_results[1:]
                if checkpoint_to_remove != new_checkpoint:
                    if accelerator.is_main_process:
                        shutil.rmtree(os.path.join(args.output_dir, checkpoint_to_remove), ignore_errors=True)
                    accelerator.wait_for_everyone()

            if new_checkpoint in checkpoints:
                # Save model checkpoint
                checkpoint_output_dir = os.path.join(args.output_dir, new_checkpoint)
                if accelerator.is_main_process:
                    if not os.path.exists(checkpoint_output_dir):
                        os.makedirs(checkpoint_output_dir)
                accelerator.wait_for_everyone()
                unwrapped_model = accelerator.unwrap_model(model)
                unwrapped_model.save_pretrained(checkpoint_output_dir, save_function=accelerator.save)
                if accelerator.is_main_process:
                    tokenizer.save_pretrained(checkpoint_output_dir)
                    logger.info("Saving model checkpoint to %s", checkpoint_output_dir)

        if completed_steps >= args.max_steps:
            break

        if should_training_stop:
            break

    if best_checkpoint is not None:
        # Save the best checkpoint
        logger.info("Best checkpoint: %s", best_checkpoint)
        logger.info("Best evaluation result: %s = %f", args.eval_metric, best_eval_result)
        best_checkpoint_output_dir = os.path.join(args.output_dir, best_checkpoint)
        if accelerator.is_main_process:
            shutil.move(best_checkpoint_output_dir, os.path.join(args.output_dir, "best-checkpoint"))
            shutil.rmtree(best_checkpoint_output_dir, ignore_errors=True)
        accelerator.wait_for_everyone()

    else:
        # Assume that the last checkpoint is the best checkpoint and save it
        checkpoint_output_dir = os.path.join(args.output_dir, "best-checkpoint")
        if not os.path.exists(checkpoint_output_dir):
            os.makedirs(checkpoint_output_dir)

        accelerator.wait_for_everyone()
        unwrapped_model = accelerator.unwrap_model(model)
        unwrapped_model.save_pretrained(checkpoint_output_dir, save_function=accelerator.save)
        if accelerator.is_main_process:
            tokenizer.save_pretrained(checkpoint_output_dir)
            logger.info("Saving model checkpoint to %s", checkpoint_output_dir)


SUMMARY:
**Function Summary:**

`train`

*Trains a model on given training data.*

### Purpose and Functionality:

* Trains a model using the provided dataset (`train_dataloader`) for `args.num_train_epochs` epochs.
* Accumulates gradients over `args.gradient_accumulation_steps` steps.
* Evaluates during training using `eval_dataloader` at intervals specified by `args.eval_strategy`.
* Saves checkpoints and tracks best checkpoint based on evaluation results.

### Key Implementation Details:

* Model is wrapped with an accelerator (`accelerator.unwrap_model(model)`) for distributed training.
* Gradient accumulation is performed using a loop that checks `completed_steps >= args.max_steps` to exit early.
* Best checkpoint is determined by evaluating the model on the validation dataset and selecting the checkpoint with the highest evaluation metric.

### External Dependencies:

* Model architecture
* Dataloader for training and evaluation data
* Accelerator for distributed training

Note: The provided summary only captures the main functionality of the code. For a detailed understanding, please refer to the original codebase.


key: chroma:document
string_value:
CODE:

def evaluate(args, accelerator, dataloader, eval_set, model, checkpoint, has_labels=True, write_to_file=True):
    """Evaluate a model checkpoint on the given evaluation data."""

    num_examples = args.num_examples[eval_set]
    eval_metric = None
    completed_steps = 0
    eval_loss = 0.0
    all_predictions = None
    all_references = None
    all_probabilities = None

    if has_labels:
        # Get the metric function
        eval_metric = load_metric(args.eval_metric)

    eval_results = {}
    model.eval()
    for _, batch in enumerate(dataloader):
        with torch.no_grad():
            outputs = model(**batch)

        eval_loss += outputs.loss.item()
        logits = outputs.logits
        predictions = logits.argmax(dim=-1) if not args.is_regression else logits.squeeze()
        predictions = accelerator.gather(predictions)

        if all_predictions is None:
            all_predictions = predictions.detach().cpu().numpy()
        else:
            all_predictions = np.append(all_predictions, predictions.detach().cpu().numpy(), axis=0)

        if not args.is_regression:
            probabilities = logits.softmax(dim=-1).max(dim=-1).values
            probabilities = accelerator.gather(probabilities)
            if all_probabilities is None:
                all_probabilities = probabilities.detach().cpu().numpy()
            else:
                all_probabilities = np.append(all_probabilities, probabilities.detach().cpu().numpy(), axis=0)

        if has_labels:
            references = batch["labels"]
            references = accelerator.gather(references)
            if all_references is None:
                all_references = references.detach().cpu().numpy()
            else:
                all_references = np.append(all_references, references.detach().cpu().numpy(), axis=0)

            eval_metric.add_batch(
                predictions=predictions,
                references=references,
            )
        completed_steps += 1

    if has_labels:
        eval_results.update(eval_metric.compute())
        eval_results["completed_steps"] = completed_steps
        eval_results["avg_eval_loss"] = eval_loss / completed_steps

        if write_to_file:
            accelerator.wait_for_everyone()
            if accelerator.is_main_process:
                results_file = os.path.join(args.output_dir, f"{eval_set}_results_{checkpoint}.json")
                with open(results_file, "w") as f:
                    json.dump(eval_results, f, indent=4, sort_keys=True)

    if write_to_file:
        accelerator.wait_for_everyone()
        if accelerator.is_main_process:
            output_file = os.path.join(args.output_dir, f"{eval_set}_output_{checkpoint}.csv")
            if not args.is_regression:
                assert len(all_predictions) == len(all_probabilities)
                df = pd.DataFrame(list(zip(all_predictions, all_probabilities)), columns=["prediction", "probability"])
            else:
                df = pd.DataFrame(all_predictions, columns=["prediction"])
            df = df.head(num_examples)
            df.to_csv(output_file, header=True, index=False)


SUMMARY:
**Function Summary: `evaluate`**

Evaluates a model checkpoint on the given evaluation data.

**Purpose**
The main purpose of this function is to evaluate a pre-trained model checkpoint on a specified evaluation set. It computes the model's performance metrics, such as accuracy or loss, and saves the results to files if required.

**Key Implementation Details:**

* Loads a metric function based on the `args.eval_metric` for evaluating model performance.
* Evaluates the model on each batch of data in the `dataloader` using the `model`.
* Computes evaluation metrics (e.g., accuracy, loss) by gathering predictions, references, and probabilities from the accelerator.
* Stores the evaluation results in a dictionary.

**Notable Dependencies:**

* `pytorch`: The PyTorch framework is used for building and evaluating the model.
* `load_metric`: Function to load metric functions, such as accuracy or loss functions.
* `accelerator`: Uses the `accelerator` library (e.g., `loky`, `torch.distributed`) for parallelizing computations.

**Assumptions:**

* The model has been trained and saved in a checkpoint format suitable for evaluation.
* The `dataloader` contains the training data for the evaluation set.


key: chroma:document
string_value:
CODE:

def load_from_pretrained(args, pretrained_model_name_or_path):
    """Load the pretrained model and tokenizer."""

    # In distributed training, the .from_pretrained methods guarantee that only
    # one local process can concurrently perform this procedure.

    config = AutoConfig.from_pretrained(
        pretrained_model_name_or_path,
        num_labels=args.num_labels if hasattr(args, "num_labels") else None,
        finetuning_task=args.task_name.lower(),
        cache_dir=args.cache_dir,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        pretrained_model_name_or_path, use_fast=args.use_fast_tokenizer, cache_dir=args.cache_dir
    )
    model = AutoModelForSequenceClassification.from_pretrained(
        pretrained_model_name_or_path,
        from_tf=bool(".ckpt" in args.model_name_or_path),
        config=config,
        ignore_mismatched_sizes=True,
        cache_dir=args.cache_dir,
    )


SUMMARY:
**Function Summary: Load Pretrained Model and Tokenizer**

Loads a pre-trained Hugging Face model, tokenizer, and configuration.
**Primary Purpose:** Loads pre-trained model and its associated tokenization model from a given model name/path.

**Key Implementation Details:**

* Uses `AutoConfig`, `AutoTokenizer`, and `AutoModelForSequenceClassification` to load the respective models.
	+ Model architecture is determined by the input `pretrained_model_name_or_path`.
* Configuring model for sequence classification, including:
  * Using fine-tuning task (`finetuning_task`) specified in `args`.
  * Ignoring size mismatches between configuration and model (`ignore_mismatched_sizes=True`).
* Returns three components:
  + Model configuration (`config`)
  + Tokenizer instance (`tokenizer`)
  + Pre-trained model instance (`model`)

**Notable Dependencies:**

* Hugging Face libraries (`transformers`, `auto_model_for_sequence_classification`, etc.)
* External dependency on `args` object (input arguments)

**Interfaces and Inheritance:** None notable.


key: chroma:document
string_value:
CODE:

def finetune(accelerator, model_name_or_path, train_file, output_dir, **kwargs):
    """Fine-tuning a pre-trained model on a downstream task.

    Args:
      accelerator: An instance of an accelerator for distributed training (on
        multi-GPU, TPU) or mixed precision training.
      model_name_or_path: Path to pretrained model or model identifier from
        huggingface.co/models.
      train_file: A csv or a json file containing the training data.
      output_dir: The output directory where the model predictions and checkpoints
        will be written.
      **kwargs: Dictionary of key/value pairs with which to update the
        configuration object after loading. The values in kwargs of any keys which
        are configuration attributes will be used to override the loaded values.
    """
    # Make one log on every process with the configuration for debugging.
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO,
    )
    logger.info(accelerator.state)

    # Setup logging, we only want one process per machine to log things on the
    # screen. accelerator.is_local_main_process is only True for one process per
    # machine.
    logger.setLevel(logging.INFO if accelerator.is_local_main_process else logging.ERROR)

    model_args = FTModelArguments(model_name_or_path=model_name_or_path)
    data_args = FTDataArguments(train_file=train_file)
    training_args = FTTrainingArguments(output_dir=output_dir)
    args = argparse.Namespace()

    for arg_class in (model_args, data_args, training_args):
        for key, value in vars(arg_class).items():
            setattr(args, key, value)

    for key, value in kwargs.items():
        if hasattr(args, key):
            setattr(args, key, value)

    # Sanity checks
    data_files = {}
    args.data_file_extension = None

    # You need to provide the training data as we always run training
    args.do_train = True
    assert args.train_file is not None
    data_files[Split.TRAIN.value] = args.train_file

    if args.do_eval or args.eval_strategy != IntervalStrategy.NO.value:
        assert args.eval_file is not None
        data_files[Split.EVAL.value] = args.eval_file

    if args.do_eval and args.test_file is not None:
        data_files[Split.TEST.value] = args.test_file

    if args.do_predict:
        assert args.infer_file is not None
        data_files[Split.INFER.value] = args.infer_file

    for key in data_files:
        extension = data_files[key].split(".")[-1]
        assert extension in ["csv", "json"], f"`{key}_file` should be a csv or a json file."
        if args.data_file_extension is None:
            args.data_file_extension = extension
        else:
            assert extension == args.data_file_extension, f"`{key}_file` should be a {args.data_file_extension} file`."

    assert (
        args.eval_metric in datasets.list_metrics()
    ), f"{args.eval_metric} not in the list of supported metrics {datasets.list_metrics()}."

    # Handle the output directory creation
    if accelerator.is_main_process:
        if args.output_dir is not None:
            os.makedirs(args.output_dir, exist_ok=True)
    accelerator.wait_for_everyone()

    # If passed along, set the training seed now.
    if args.seed is not None:
        set_seed(args.seed)

    # You need to provide your CSV/JSON data files.
    #
    # For CSV/JSON files, this script will use as labels the column called 'label'
    # and as pair of sentences the sentences in columns called 'sentence1' and
    # 'sentence2' if these columns exist or the first two columns not named
    # 'label' if at least two columns are provided.
    #
    # If the CSVs/JSONs contain only one non-label column, the script does single
    # sentence classification on this single column.
    #
    # In distributed training, the load_dataset function guarantees that only one
    # local process can download the dataset.

    # Loading the dataset from local csv or json files.
    raw_datasets = load_dataset(args.data_file_extension, data_files=data_files)

    # Labels
    is_regression = raw_datasets[Split.TRAIN.value].features["label"].dtype in ["float32", "float64"]
    args.is_regression = is_regression

    if args.is_regression:
        label_list = None
        num_labels = 1
    else:
        label_list = args.label_list
        assert label_list is not None
        label_list.sort()  # Let's sort it for determinism
        num_labels = len(label_list)
    args.num_labels = num_labels

    # Load pre-trained model
    config, tokenizer, model = load_from_pretrained(args, args.model_name_or_path)

    # Preprocessing the datasets
    non_label_column_names = [name for name in raw_datasets[Split.TRAIN.value].column_names if name != "label"]
    if "sentence1" in non_label_column_names and "sentence2" in non_label_column_names:
        sentence1_key, sentence2_key = "sentence1", "sentence2"
    else:
        if len(non_label_column_names) >= 2:
            sentence1_key, sentence2_key = non_label_column_names[:2]
        else:
            sentence1_key, sentence2_key = non_label_column_names[0], None

    label_to_id = {v: i for i, v in enumerate(label_list)}
    config.label2id = label_to_id
    config.id2label = {id: label for label, id in config.label2id.items()}
    padding = "max_length" if args.pad_to_max_length else False

    def preprocess_function(examples):
        # Tokenize the texts
        texts = (
            (examples[sentence1_key],) if sentence2_key is None else (examples[sentence1_key], examples[sentence2_key])
        )
        result = tokenizer(*texts, padding=padding, max_length=args.max_length, truncation=True)

        if "label" in examples:
            if label_to_id is not None:
                # Map labels to IDs (not necessary for GLUE tasks)
                result["labels"] = [label_to_id[l] for l in examples["label"]]
            else:
                # In all cases, rename the column to labels because the model will
                # expect that.
                result["labels"] = examples["label"]
        return result

    with accelerator.main_process_first():
        processed_datasets = raw_datasets.map(
            preprocess_function,
            batched=True,
            remove_columns=raw_datasets[Split.TRAIN.value].column_names,
            desc="Running tokenizer on dataset",
        )

    num_examples = {}
    splits = [s.value for s in Split]
    for split in splits:
        if split in processed_datasets:
            num_examples[split] = len(processed_datasets[split])
    args.num_examples = num_examples

    train_dataset = processed_datasets[Split.TRAIN.value]
    eval_dataset = processed_datasets[Split.EVAL.value] if Split.EVAL.value in processed_datasets else None
    test_dataset = processed_datasets[Split.TEST.value] if Split.TEST.value in processed_datasets else None
    infer_dataset = processed_datasets[Split.INFER.value] if Split.INFER.value in processed_datasets else None

    # Log a few random samples from the training set:
    for index in random.sample(range(len(train_dataset)), 3):
        logger.info("Sample %d of the training set: %s.", index, train_dataset[index])

    # DataLoaders creation:
    if args.pad_to_max_length:
        # If padding was already done ot max length, we use the default data
        # collator that will just convert everything to tensors.
        data_collator = default_data_collator
    else:
        # Otherwise, `DataCollatorWithPadding` will apply dynamic padding for us (by
        # padding to the maximum length of the samples passed). When using mixed
        # precision, we add `pad_to_multiple_of=8` to pad all tensors to multiple of
        # 8s, which will enable the use of Tensor Cores on NVIDIA hardware with
        # compute capability >= 7.5 (Volta).
        # For fp8, we pad to multiple of 16.
        if accelerator.mixed_precision == "fp8":
            pad_to_multiple_of = 16
        elif accelerator.mixed_precision != "no":
            pad_to_multiple_of = 8
        else:
            pad_to_multiple_of = None
        data_collator = DataCollatorWithPadding(tokenizer, pad_to_multiple_of=pad_to_multiple_of)

    train_dataloader = DataLoader(
        train_dataset,
        batch_size=args.per_device_train_batch_size,
        shuffle=True,
        collate_fn=data_collator,
    )
    eval_dataloader, test_dataloader, infer_dataloader = None, None, None

    if eval_dataset is not None:
        eval_dataloader = DataLoader(
            eval_dataset, batch_size=args.per_device_eval_batch_size, collate_fn=data_collator
        )

    if test_dataset is not None:
        test_dataloader = DataLoader(
            test_dataset, batch_size=args.per_device_eval_batch_size, collate_fn=data_collator
        )

    if infer_dataset is not None:
        infer_dataloader = DataLoader(
            infer_dataset, batch_size=args.per_device_eval_batch_size, collate_fn=data_collator
        )

    # Optimizer
    # Split weights in two groups, one with weight decay and the other not.
    no_decay = ["bias", "LayerNorm.weight"]
    optimizer_grouped_parameters = [
        {
            "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
            "weight_decay": args.weight_decay,
        },
        {
            "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
            "weight_decay": 0.0,
        },
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate)

    # Prepare everything with our `accelerator`.
    model, optimizer, train_dataloader, eval_dataloader, test_dataloader, infer_dataloader = accelerator.prepare(
        model, optimizer, train_dataloader, eval_dataloader, test_dataloader, infer_dataloader
    )

    # Note -> the training dataloader needs to be prepared before we grab its
    # length below (cause its length will be shorter in multiprocess)

    # Scheduler and math around the number of training steps.
    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)
    if args.max_steps == -1:
        args.max_steps = args.num_train_epochs * num_update_steps_per_epoch
    else:
        args.num_train_epochs = math.ceil(args.max_steps / num_update_steps_per_epoch)

    lr_scheduler = get_scheduler(
        name=args.lr_scheduler_type,
        optimizer=optimizer,
        num_warmup_steps=args.warmup_steps,
        num_training_steps=args.max_steps,
    )

    # Train
    completed_steps, avg_train_loss = train(
        args, accelerator, model, tokenizer, train_dataloader, optimizer, lr_scheduler, eval_dataloader
    )
    accelerator.wait_for_everyone()
    logger.info("Training job completed: completed_steps = %d, avg_train_loss = %f", completed_steps, avg_train_loss)

    args.model_name_or_path = os.path.join(args.output_dir, "best-checkpoint")
    logger.info("Loading the best checkpoint: %s", args.model_name_or_path)
    config, tokenizer, model = load_from_pretrained(args, args.model_name_or_path)
    model = accelerator.prepare(model)

    if args.do_eval:
        # Evaluate
        if eval_dataloader is not None:
            logger.info("***** Running evaluation on the eval data using the best checkpoint *****")
            eval_results = evaluate(args, accelerator, eval_dataloader, Split.EVAL.value, model, "best-checkpoint")
            avg_eval_loss = eval_results["avg_eval_loss"]
            eval_metric = eval_results[args.eval_metric]
            logger.info("Evaluation job completed: avg_eval_loss = %f", avg_eval_loss)
            logger.info("Evaluation result for the best checkpoint: %s = %f", args.eval_metric, eval_metric)

        if test_dataloader is not None:
            logger.info("***** Running evaluation on the test data using the best checkpoint *****")
            eval_results = evaluate(args, accelerator, test_dataloader, Split.TEST.value, model, "best-checkpoint")
            avg_eval_loss = eval_results["avg_eval_loss"]
            eval_metric = eval_results[args.eval_metric]
            logger.info("Test job completed: avg_test_loss = %f", avg_eval_loss)
            logger.info("Test result for the best checkpoint: %s = %f", args.eval_metric, eval_metric)

    if args.do_predict:
        # Predict
        if infer_dataloader is not None:
            logger.info("***** Running inference using the best checkpoint *****")
            evaluate(
                args, accelerator, infer_dataloader, Split.INFER.value, model, "best-checkpoint", has_labels=False
            )
            logger.info("Inference job completed.")

    # Release all references to the internal objects stored and call the garbage
    # collector. You should call this method between two trainings with different
    # models/optimizers.


SUMMARY:
This code is written in Python and appears to be a trainer for a machine learning model using the Hugging Face Transformers library and PyTorch. Here's a concise summary of what the code does:

1. **Initialization**: The script initializes various hyperparameters, such as batch sizes, number of epochs, optimizer parameters, etc.
2. **Model Preparation**: The model is loaded, tokenizer is prepared, and data loaders are created for training, evaluation, testing, and inference.
3. **Optimizer and Scheduler**: An AdamW optimizer with a learning rate scheduler is set up to control the learning rate during training.
4. **Training Loop**: A training loop is implemented that iterates over the number of steps defined in the `args`. At each step:
	* The input data is loaded and fed into the model.
	* The loss is computed using the `model` and the optimizer updates the parameters using "step" gradients from AdamW.
	* The loss for evaluation or testing tasks (if applicable) is recorded, and a logging statement is printed with the current training step and average training loss.
5. **Evaluation**: After training, if `args.do_eval` is True, the trained model performs an additional evaluation on the validation and test sets to compute metrics such as loss, accuracy, etc.
6. **Prediction**: If `args.do_predict` is True, the trained model performs predictions for inference data using the same best model weights stored after training.

The code also ensures proper memory management by freeing any internal objects stored during the execution of the script.

To write this code from scratch:

1. Install the necessary libraries: Hugging Face Transformers, PyTorch.
2. Import the required libraries and configure your environment for training a transformer-based model (if needed).
3. Initialize hyperparameters like batch sizes, number of epochs, optimizer parameters, etc.
4. Set up the model by loading it or creating one yourself if you have custom modules.
5. Prepare the tokenizer using the Hugging Face Transformers library.
6. Split your dataset into training and evaluation sets (if applicable) and create data loaders for both.
7. Implement the optimizer and scheduler as done in this script.
8. Set up a training loop that iterates over multiple epochs, updating the model parameters, logging losses, etc.
9. Evaluate and predict on test or custom sets if required.

Please note that the Hugging Face library provides many utility functions to simplify the above code and offers more convenient ways of handling these tasks using pre-built tools such as `Trainer`, which this script appears to be building upon.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/self-training-text-classification/selftraining.py


key: chroma:document
string_value:
CODE:

def create_pseudo_labeled_data(args, infer_input, infer_output, eval_result, id2label, next_data_dir):
    """Create pseudeo labeled data for the next self-training iteration."""

    dataset = datasets.concatenate_datasets([infer_input, infer_output], axis=1)

    if args.do_filter_by_confidence:
        dataset = dataset.filter(lambda example: example["probability"] > args.confidence_threshold)

    if args.do_filter_by_val_performance:
        assert eval_result >= 0.0 and eval_result <= 1.0
        num_selected_rows = int(eval_result * len(dataset))
        print(num_selected_rows)
        dataset = dataset.sort("probability", reverse=True)
        dataset = dataset.select(range(num_selected_rows))

    dataset = dataset.remove_columns(["label", "probability"])
    dataset = dataset.rename_column("prediction", "label")
    dataset = dataset.map(lambda example: {"label": id2label[example["label"]]})
    dataset = dataset.shuffle(seed=args.seed)

    pseudo_labeled_data_file = os.path.join(next_data_dir, f"train_pseudo.{args.data_file_extension}")
    if args.data_file_extension == "csv":
        dataset.to_csv(pseudo_labeled_data_file, index=False)
    else:


SUMMARY:
**Summary**

`create_pseudo_labeled_data` function creates pseudo-labeled data for self-training by concatenating `infer_input` and `infer_output` datasets. It applies filters by confidence threshold and validation performance, then processes the resulting dataset.

**Key Details:**

* Uses `datasets.concatenate_datasets` to merge input and output datasets.
* Applies filters:
	+ Confidence threshold (`args.do_filter_by_confidence`)
	+ Validation performance (`args.do_filter_by_val_performance`)
* Processes dataset:
	+ Removes columns "label" and "probability"
	+ Renames column "prediction" to "label"
	+ Maps labels using `id2label` dictionary
	+ Shuffles dataset with specified seed
* Saves processed dataset to CSV/SJSON file.

**Dependencies:**

* `datasets`
* `os`
* `id2label` dictionary (assumed external dependency)

**Function Signature:** 
```python
def create_pseudo_labeled_data(args, infer_input, infer_output, eval_result, id2label, next_data_dir)
```

**Note:** This summary focuses on the main purpose, key implementation details, and notable dependencies. It excludes generic comments and non-essential information.


key: chroma:document
string_value:
CODE:

def selftrain(model_name_or_path, train_file, infer_file, output_dir, **kwargs):
    """Self-training a pre-trained model on a downstream task.

    Args:
      model_name_or_path: Path to pretrained model or model identifier from
        huggingface.co/models.
      train_file: A csv or a json file containing the training data.
      infer_file: A csv or a json file containing the data to predict on.
      output_dir: The output directory where the model predictions and checkpoints
        will be written.
      **kwargs: Dictionary of key/value pairs with which to update the
        configuration object after loading. The values in kwargs of any keys which
        are configuration attributes will be used to override the loaded values.
    """
    # Initialize the accelerator. We will let the accelerator handle device
    # placement for us.
    accelerator = Accelerator()
    # Make one log on every process with the configuration for debugging.
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO,
    )
    logger.info(accelerator.state)

    # Setup logging, we only want one process per machine to log things on the
    # screen. accelerator.is_local_main_process is only True for one process per
    # machine.
    logger.setLevel(logging.INFO if accelerator.is_local_main_process else logging.ERROR)

    if accelerator.is_local_main_process:
        datasets.utils.logging.set_verbosity_warning()
        transformers.utils.logging.set_verbosity_info()
    else:
        datasets.utils.logging.set_verbosity_error()
        transformers.utils.logging.set_verbosity_error()

    model_args = STModelArguments(model_name_or_path=model_name_or_path)
    data_args = STDataArguments(train_file=train_file, infer_file=infer_file)
    training_args = STTrainingArguments(output_dir=output_dir)
    args = argparse.Namespace()

    for arg_class in (model_args, data_args, training_args):
        for key, value in vars(arg_class).items():
            setattr(args, key, value)

    for key, value in kwargs.items():
        if hasattr(args, key):
            setattr(args, key, value)

    # Sanity checks
    data_files = {}
    args.data_file_extension = None

    # You need to provide the training data and the data to predict on
    assert args.train_file is not None
    assert args.infer_file is not None
    data_files["train"] = args.train_file
    data_files["infer"] = args.infer_file

    if args.eval_strategy != IntervalStrategy.NO.value:
        assert args.eval_file is not None
        data_files["eval"] = args.eval_file

    for key in data_files:
        extension = data_files[key].split(".")[-1]
        assert extension in ["csv", "json"], f"`{key}_file` should be a csv or a json file."
        if args.data_file_extension is None:
            args.data_file_extension = extension
        else:
            assert extension == args.data_file_extension, f"`{key}_file` should be a {args.data_file_extension} file`."

    assert (
        args.eval_metric in datasets.list_metrics()
    ), f"{args.eval_metric} not in the list of supported metrics {datasets.list_metrics()}."

    # If passed along, set the training seed now.
    if args.seed is not None:
        set_seed(args.seed)

    logger.info("Creating the initial data directory for self-training...")
    data_dir_format = f"{args.output_dir}/self-train_iter-{{}}".format
    initial_data_dir = data_dir_format(0)

    if accelerator.is_main_process:
        if args.output_dir is not None:
            os.makedirs(args.output_dir, exist_ok=True)
            os.makedirs(initial_data_dir, exist_ok=True)
    accelerator.wait_for_everyone()

    best_iteration = None
    best_eval_result = None
    early_stopping_patience_counter = 0
    should_training_stop = False
    # Show the progress bar
    progress_bar = tqdm(range(args.max_selftrain_iterations), disable=not accelerator.is_local_main_process)

    # Self-train
    for iteration in range(0, int(args.max_selftrain_iterations)):
        current_data_dir = data_dir_format(iteration)
        assert os.path.exists(current_data_dir)

        # Stage 1: initial fine-tuning for iteration = 0 or pseudo-training for
        # iteration > 0
        current_output_dir = os.path.join(current_data_dir, "stage-1")
        arguments_dict = {
            "accelerator": accelerator,
            "model_name_or_path": args.model_name_or_path,
            "cache_dir": args.cache_dir,
            "do_train": True,
            "train_file": data_files["train"] if iteration == 0 else data_files["train_pseudo"],
            "do_eval": True if args.eval_file is not None else False,
            "eval_file": data_files["eval"],
            "do_predict": True,
            "infer_file": data_files["infer"],
            "task_name": args.task_name,
            "label_list": args.label_list,
            "output_dir": current_output_dir,
            "eval_metric": args.eval_metric,
            "eval_strategy": args.eval_strategy,
            "early_stopping_patience": args.early_stopping_patience,
            "early_stopping_threshold": args.early_stopping_threshold,
            "seed": args.seed,
        }
        # Add additional training arguments
        for key, value in kwargs.items():
            if key not in arguments_dict and not hasattr(training_args, key):
                arguments_dict.update({key: value})

        model_bin_file_path = os.path.join(current_output_dir, "best-checkpoint", MODEL_BIN_FILE)
        if os.path.exists(model_bin_file_path):
            logger.info(
                "Found existing model checkpoint at %s. Skipping self-training: iteration: %d, stage: 1.",
                model_bin_file_path,
                iteration,
            )
        else:
            logger.info("***** Running self-training: iteration: %d, stage: 1 *****", iteration)
            finetune(**arguments_dict)
            accelerator.wait_for_everyone()
            assert os.path.exists(model_bin_file_path)
            logger.info("Self-training job completed: iteration: %d, stage: 1.", iteration)

        if iteration > 0 and args.finetune_on_labeled_data:
            # Stage 2 (optional): fine-tuning on the original labeled data
            model_path = os.path.join(current_output_dir, "best-checkpoint")
            current_output_dir = os.path.join(current_data_dir, "stage-2")
            # Update arguments_dict
            arguments_dict["model_name_or_path"] = model_path
            arguments_dict["train_file"] = data_files["train"]
            arguments_dict["output_dir"] = current_output_dir

            model_bin_file_path = os.path.join(current_output_dir, "best-checkpoint", MODEL_BIN_FILE)
            if os.path.exists(model_bin_file_path):
                logger.info(
                    "Found existing model checkpoint at %s. Skipping self-training: iteration: %d, stage: 2.",
                    model_bin_file_path,
                    iteration,
                )
            else:
                logger.info("***** Running self-training: iteration: %d, stage: 2 *****", iteration)
                finetune(**arguments_dict)
                accelerator.wait_for_everyone()
                assert os.path.exists(model_bin_file_path)
                logger.info("Self-training job completed: iteration: %d, stage: 2.", iteration)

        new_iteration = iteration
        next_data_dir = data_dir_format(iteration + 1)

        config = AutoConfig.from_pretrained(os.path.join(current_output_dir, "best-checkpoint"))
        id2label = config.id2label
        eval_results_file = os.path.join(current_output_dir, "eval_results_best-checkpoint.json")
        test_results_file = os.path.join(current_output_dir, "test_results_best-checkpoint.json")
        assert os.path.exists(eval_results_file)

        with open(eval_results_file, "r") as f:
            eval_result = float(json.load(f)[args.eval_metric])
        infer_output_file = os.path.join(current_output_dir, "infer_output_best-checkpoint.csv")
        assert os.path.exists(infer_output_file)
        # Loading the dataset from local csv or json files.
        infer_input = load_dataset(args.data_file_extension, data_files={"data": data_files["infer"]})["data"]
        infer_output = load_dataset("csv", data_files={"data": infer_output_file})["data"]

        if accelerator.is_main_process:
            os.makedirs(next_data_dir, exist_ok=True)
            shutil.copy(eval_results_file, os.path.join(output_dir, f"eval_results_iter-{iteration}.json"))
            if os.path.exists(test_results_file):
                shutil.copy(eval_results_file, os.path.join(output_dir, f"test_results_iter-{iteration}.json"))
            create_pseudo_labeled_data(args, infer_input, infer_output, eval_result, id2label, next_data_dir)
        accelerator.wait_for_everyone()

        data_files["train_pseudo"] = os.path.join(next_data_dir, f"train_pseudo.{args.data_file_extension}")

        if args.eval_strategy != IntervalStrategy.NO.value:
            new_eval_result = eval_result

            if best_iteration is None:
                best_iteration = new_iteration
                best_eval_result = new_eval_result
            else:
                if new_eval_result - best_eval_result > args.early_stopping_threshold:
                    best_iteration = new_iteration
                    best_eval_result = new_eval_result
                    early_stopping_patience_counter = 0
                else:
                    if new_eval_result == best_eval_result:
                        best_iteration = new_iteration
                        best_eval_result = new_eval_result
                    early_stopping_patience_counter += 1

                if early_stopping_patience_counter >= args.early_stopping_patience:
                    should_training_stop = True

        progress_bar.update(1)

        if should_training_stop:
            break

    if best_iteration is not None:
        # Save the best iteration
        logger.info("Best iteration: %d", best_iteration)
        logger.info("Best evaluation result: %s = %f", args.eval_metric, best_eval_result)
        accelerator.wait_for_everyone()
        if accelerator.is_main_process:
            shutil.copy(
                os.path.join(output_dir, f"eval_results_iter-{iteration}.json"),
                os.path.join(output_dir, "eval_results_best-iteration.json"),
            )
    else:
        # Assume that the last iteration is the best
        logger.info("Best iteration: %d", args.max_selftrain_iterations - 1)
        logger.info("Best evaluation result: %s = %f", args.eval_metric, eval_result)
        accelerator.wait_for_everyone()
        if accelerator.is_main_process:
            shutil.copy(
                os.path.join(output_dir, f"eval_results_iter-{args.max_selftrain_iterations - 1}.json"),
                os.path.join(output_dir, "eval_results_best-iteration.json"),


SUMMARY:
The provided code is a script for self-supervised learning using the Hugging Face library. The script performs several iterations of finetuning a pre-trained model on three different tasks:

1. Supervised task: Finetuning the model on labeled data to predict certain features or metrics.
2. Unsupervised task (pseudolabeling): Creating pseudo-labeled data by evaluating the model's performance on some test data set and assuming that the labels are correct for a while (until the model significantly improves its performance).
3. Test/evaluation: Evaluating the model's performance on new unseen data to provide a baseline of the model's performance.

Here is a concise summary:

* Iterate between three main stages:
  - Supervised task: Fine-tune the model.
  - Unsupervised task (pseudolabeling): Create a pseudo-labeled dataset and update the original dataset with this pseudo-labeling data. Use newly updated data to fine-tune the model further.
    * This process is repeated until convergence of performance (i.e., the best evaluation result).
  - Testing/evaluation: Save and use the best-performing model by evaluating its performance on some test data (which may be very similar to original but was used for testing during the early stages) and compare with a baseline iteration.
* Provide outputs like `eval_results_best-iteration.json` containing evaluation metrics of best performing instance.

Note that this script uses various Hugging Face library functions to load and preprocess data, fine-tune models, and create pseudo-labeled datasets.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/vqgan-clip/img_processing.py


key: chroma:document
string_value:
CODE:

def preprocess(img, target_image_size=256):
    s = min(img.size)

    if s < target_image_size:
        raise ValueError(f"min dim for image {s} < {target_image_size}")

    r = target_image_size / s
    s = (round(r * img.size[1]), round(r * img.size[0]))
    img = TF.resize(img, s, interpolation=PIL.Image.LANCZOS)
    img = TF.center_crop(img, output_size=2 * [target_image_size])
    img = torch.unsqueeze(T.ToTensor()(img), 0)


SUMMARY:
Here is the concise code sum:
```
preprocess function:

Purpose: Resizes input image to target size while maintaining aspect ratio.

Key implementation details:

- Scales smaller images up to target_size using Lanczos interpolation
- Centers cropped images on target_size square canvas
- Converts image to PyTorch tensor and adds batch dimension

Important notes:

- Raises ValueError if original image is too small for target size (target_image_size)
- Dependencies:
  - TF (TensorFlow library, unspecified functions)
    - resize
    - center_crop
  - PIL (Python Imaging Library, unspecified functions)
    - Image.LANCZOS
    - ToTensor
```


key: chroma:document
string_value:
CODE:

def preprocess_vqgan(x):
    x = 2.0 * x - 1.0


SUMMARY:
**Function Summary**

The `preprocess_vqgan` function takes an input `x` and applies a standardization transformation to it.

**Description**
Normalizes input values to the range [-1.0, 1.0] for potential use in VQ-VAE (Variational Quantized Autoencoder) models.

**Key Implementation Details**

*   Input variable: `x`
*   Transformation applied: `(2.0 * x) - 1.0`

**Notable Dependencies or Requirements**
The function uses a simple linear transformation but may require understanding of image or data pre-processing techniques in the context of VQ-VAE models.

This summary provides essential information about the purpose and functionality of the `preprocess_vqgan` function, highlighting key implementation details while omitting unnecessary or generic descriptions.


key: chroma:document
string_value:
CODE:

def custom_to_pil(x, process=True, mode="RGB"):
    x = x.detach().cpu()
    if process:
        x = post_process_tensor(x)
    x = x.numpy()
    if process:
        x = (255 * x).astype(np.uint8)
    x = Image.fromarray(x)
    if not x.mode == mode:
        x = x.convert(mode)


SUMMARY:
**Function Summary:**

`custom_to_pil` converts a tensor to a PIL image. It:

*   Moves the tensor to CPU and detaches it from the gradient computation
*   Applies post-processing on the tensor (`x`) if specified
*   Converts the numpy array representation of `x` to an unsigned 8-bit integer type
*   Creates a PIL image from the processed numpy array

Key implementation details:

*   Method signatures: `post_process_tensor`, `detach`, `cpu`, `numpy`, `astype`
*   Design choice: Applying post-processing only if specified by `process=True`

Notable dependencies and requirements:

*   `Image` module (built-in, no explicit import)
*   PIL library for image processing
*   NumPy library for numpy array representation
*   TensorFlow/PyTorch libraries for tensor operations (`attach`, `cpu`, `detach`)


key: chroma:document
string_value:
CODE:

def post_process_tensor(x):
    x = torch.clamp(x, -1.0, 1.0)
    x = (x + 1.0) / 2.0
    x = x.permute(1, 2, 0)


SUMMARY:
**Concise Summary:**

```python
summarized_function_name: post_process_tensor

**Purpose and Functionality:** This function normalizes floating point tensor values to the range [-1, 1] using a sigmoid mapping and permutes its dimensions for further processing.

**Key Implementation Details:**

* Uses `torch.clamp` with bounds (-1.0, 1.0) for normalization.
* Applies a sigmoid mapping (x + 1.0) / 2.0 to squeeze the range into [-1, 1].
* Permutations the dimensions of the tensor using `.permute(1, 2, 0)`.

**Notable Dependencies and Requirements:**

* torch library for tensor operations.
```

In the RAG system, this summary could be stored with a reference ID to the original code snippet, allowing developers to quickly locate the implementation details.


key: chroma:document
string_value:
CODE:

def loop_post_process(x):
    x = post_process_tensor(x.squeeze())


SUMMARY:
**Summary:**

This is a Python function named `loop_post_process` which takes an input tensor `x`. 

**Main Purpose and Functionality:**
The primary purpose of this function is to process the input tensor `x` after it has been squeezed.

**Key Implementation Details:**

- Squeezes the input tensor `x` using the `squeeze()` method.
- Applies post-processing to the tensor using the `post_process_tensor()` function (not shown in this snippet).
- Permutates the resulting tensor by rearranging its dimensions from `(C, H, W)` to `(C, B, H, W)` using the `.permute()` method with indices (2, 0, 1).
- Adds a batch dimension at the beginning of the tensor using the `.unsqueeze(0)` method.

**Notable Dependencies or Requirements:**

- The `post_process_tensor()` function is not shown in this snippet but is assumed to be defined elsewhere.
- The `squeeze()`, `.permute()`, and `.unsqueeze()` methods are part of PyTorch's tensor API.


key: chroma:document
string_value:
CODE:

def load_config(config_path, display=False):
    config = OmegaConf.load(config_path)
    if display:
        print(yaml.dump(OmegaConf.to_container(config)))


SUMMARY:
**Summary:**

Loads configuration from file at specified `config_path` and returns it as an OmegaConf object. If `display` is True, prints the loaded configuration as YAML.

**Key Implementation Details:**

* Uses `OmegaConf.load(config_path)` to load configuration
* Optional `display=True` parameter to print parsed config using `yaml.dump` and `OmegaConf.to_container`

**Notable Dependencies:**

* OmegaConf library (`import omegaconf`)
* YAML parsing library (`import yaml`)

**Return Value:**
Returns an OmegaConf object representing the loaded configuration.


key: chroma:document
string_value:
CODE:

def load_vqgan(device, conf_path=None, ckpt_path=None):
    if conf_path is None:
        conf_path = "./model_checkpoints/vqgan_only.yaml"
    config = load_config(conf_path, display=False)
    model = VQModel(**config.model.params)
    if ckpt_path is None:
        ckpt_path = "./model_checkpoints/vqgan_only.pt"
    sd = torch.load(ckpt_path, map_location=device)
    if ".ckpt" in ckpt_path:
        sd = sd["state_dict"]
    model.load_state_dict(sd, strict=True)
    model.to(device)
    del sd


SUMMARY:
**Function Summary: `load_vqgan`**

* **Purpose:** Loads a pre-trained VQGAN model from a checkpoint file.
* 
* The function takes three parameters:
  - `device`: The GPU device to use for model loading
  - `conf_path`: Path to configuration file (defaults to "./model_checkpoints/vqgan_only.yaml")
  - `ckpt_path`: Path to checkpoint file (defaults to "./model_checkpoints/vqgan_only.pt")
* 
* The function:
  - Loads the configuration and sets up the VQModel instance
  - Loads the pre-trained model state dictionary from the checkpoint file
  - Moves the loaded model to the specified device
  - Returns the loaded VQModel instance

**Key Implementation Details:**

* Uses `torch.load` with `map_location=device` for efficient loading.
* Checks if the checkpoint path ends in ".ckpt" and loads only the state dictionary if necessary.

**Notable Dependencies/Requirements:**

* `config`: Loaded from a configuration file using the `load_config` function (not shown in this code snippet)
* `VQModel`: PyTorch model class imported from an external library or module.
* `torch`: Required for loading and manipulating models.


key: chroma:document
string_value:
CODE:

def reconstruct_with_vqgan(x, model):
    z, _, [_, _, indices] = model.encode(x)
    print(f"VQGAN --- {model.__class__.__name__}: latent shape: {z.shape[2:]}")
    xrec = model.decode(z)


SUMMARY:
**Summary**

The `reconstruct_with_vqgan` function reconstructs input data `x` using the VQGAN (Variational Quantized Generative Adversarial Network) model.

### Main Purpose and Functionality
- Reconsitute input data `x`
- Passes input data through VQGAN encode process
- Prints latent shape after encoding

### Key Implementation Details

- `model.encode(x)`: Calls the `encode` method on the VQGAN model to perform encoding.
- `model.decode(z)`: Calls the `decode` method on the VQGAN model to reconstruct input data `x`.

### Notable Dependencies or Requirements
- `model.__class__.__name__`: References the `VQGANModel` class.
- No external dependencies listed.

### Summary in RAG format:

* Purpose: Reconstruct input data `x`
* Functionality: Calls VQGAN encode and decode methods
* Notes:
	+ `model.encode(x)`
	+ `model.decode(z)`


key: chroma:document
string_value:
CODE:

def get_obj_from_str(string, reload=False):
    module, cls = string.rsplit(".", 1)
    if reload:
        module_imp = importlib.import_module(module)
        importlib.reload(module_imp)


SUMMARY:
**Function Summary: `get_obj_from_str`**

 Creates an object instance from a string representation of the form `<module>.<class>`. 

### Key Implementation Details:

* Uses `rsplit` to separate module and class names
* Checks whether to reload the imported module on demand (`reload` parameter)
* Uses `getattr` with custom module import to instantiate the desired class

### Notable Dependencies:

* `importlib`: for importing modules dynamically
* `getattr`: for accessing attributes (class instances) within imported modules


key: chroma:document
string_value:
CODE:

def instantiate_from_config(config):
    if "target" not in config:
        raise KeyError("Expected key `target` to instantiate.")


SUMMARY:
**instantiate_from_config Function Summary**

* **Primary Purpose**: Instantiates an object from a configuration dictionary, using the specified class and overriding optional parameters.
* **Implementation Details**:
	+ Verifies that the `target` key is present in the config.
	+ Calls `get_obj_from_str` with the target class name to instantiate it with the provided parameters (`params`).
* **Notable Dependencies**:
	+ `get_obj_from_str`: a function that returns an instance of a class given its string representation (exact function not shown in this code snippet).


key: chroma:document
string_value:
CODE:

def load_model_from_config(config, sd, gpu=True, eval_mode=True):
    model = instantiate_from_config(config)
    if sd is not None:
        model.load_state_dict(sd)
    if gpu:
        model.cuda()
    if eval_mode:
        model.eval()


SUMMARY:
**Summary**

`load_model_from_config` function: loads a neural network model from configuration and state dictionary, optionally moving to GPU and enabling evaluation mode.

**Key Details**

* Functionality: Instantiates a model from `config`, loads state dictionary if provided, moves to specified GPU (if applicable), and sets evaluation mode (if applicable).
* Important Parameters:
	+ `config`: instance of class or configuration data
	+ `sd` (optional): state dictionary loaded from file
	+ `gpu` (bool=False, defaults to True): move model to specified GPU when calling
	+ `eval_mode` (bool=True, defaults to True): set model as evaluation mode when calling

**External Dependencies**

* `instantiate_from_config`: function to instantiate model from configuration data
* `load_state_dict`: method for loading state dictionary from file
* `cuda`: context manager or method for moving model to specified GPU


key: chroma:document
string_value:
CODE:

def load_model(config, ckpt, gpu, eval_mode):
    # load the specified checkpoint
    if ckpt:
        pl_sd = torch.load(ckpt, map_location="cpu")
        global_step = pl_sd["global_step"]
        print(f"loaded model from global step {global_step}.")
    else:
        pl_sd = {"state_dict": None}
        global_step = None
    model = load_model_from_config(config.model, pl_sd["state_dict"], gpu=gpu, eval_mode=eval_mode)["model"]


SUMMARY:
**Summary:**

### Purpose and Functionality

The `load_model` function loads a PyTorch model from a checkpoint using the Hugging Face ` Transformers` library. It takes four arguments:

*   `config`: configuration object
*   `ckpt`: checkpoint file path (optional)
*   `gpu`: GPU index for model computation
*   `eval_mode`: evaluation mode

The function returns the loaded model and its global step.

### Key Implementation Details

*   Loads PyTorch state dictionary using `torch.load`
*   Uses Hugging Face's ` Transformers` library to load the model from checkpoint
*   Returns the loaded model ("model" key in config) with accompanying global step

### Notable Dependencies or Requirements

*   PyTorch
*   Hugging Face's `Transformers` library


key: chroma:document
string_value:
CODE:

class ProcessorGradientFlow:
    """
    This wraps the huggingface CLIP processor to allow backprop through the image processing step.
    The original processor forces conversion to PIL images, which is faster for image processing but breaks gradient flow.
    We call the original processor to get the text embeddings, but use our own image processing to keep images as torch tensors.
    """

    def __init__(self, device: str = "cpu", clip_model: str = "openai/clip-vit-large-patch14") -> None:
        self.device = device
        self.tokenizer = CLIPTokenizerFast.from_pretrained(clip_model)
        self.image_mean = [0.48145466, 0.4578275, 0.40821073]
        self.image_std = [0.26862954, 0.26130258, 0.27577711]
        self.normalize = torchvision.transforms.Normalize(self.image_mean, self.image_std)
        self.resize = torchvision.transforms.Resize(224)
        self.center_crop = torchvision.transforms.CenterCrop(224)

    def preprocess_img(self, images):
        images = self.resize(images)
        images = self.center_crop(images)
        images = self.normalize(images)
        return images

    def __call__(self, text=None, images=None, **kwargs):
        encoding = self.tokenizer(text=text, **kwargs)
        encoding["pixel_values"] = self.preprocess_img(images)
        encoding = {key: value.to(self.device) for (key, value) in encoding.items()}


SUMMARY:
**Summary:**

`ProcessorGradientFlow` class is a custom processor wrapper for HuggingFace's CLIP model. Its primary purpose is to enable backprop through the image processing step while maintaining gradient flow.

**Key Functionality:**

* Wraps the original CLIP processor with custom image preprocessing
* Utilizes FastTokenizers and torchvision transformations (resize, center crops, and normalization)
* Maintains PyTorch tensor data type for images

**Implementation Details:**

* Custom initialization (`__init__`) sets device, tokenizer, mean, and standard deviation for image normalization
* `preprocess_img` function applies resizing, cropping, and normalizing images
* `__call__` method processes input text and images while maintaining gradient flow

**Notable Dependencies:**

* HuggingFace's FastTokenizer (`CLiptokenizerFast`)
* PyTorch (`torchvision.transforms`, `torch.tensor`)
* CLIP model (`openai/clip-vit-large-patch14`)


key: chroma:document
string_value:
CODE:

class VQGAN_CLIP(nn.Module):
    def __init__(
        self,
        iterations=10,
        lr=0.01,
        vqgan=None,
        vqgan_config=None,
        vqgan_checkpoint=None,
        clip=None,
        clip_preprocessor=None,
        device=None,
        log=False,
        save_vector=True,
        return_val="image",
        quantize=True,
        save_intermediate=False,
        show_intermediate=False,
        make_grid=False,
    ) -> None:
        """
        Instantiate a VQGAN_CLIP model. If you want to use a custom VQGAN model, pass it as vqgan.
        """
        super().__init__()
        self.latent = None
        self.device = device if device else get_device()
        if vqgan:
            self.vqgan = vqgan
        else:
            self.vqgan = load_vqgan(self.device, conf_path=vqgan_config, ckpt_path=vqgan_checkpoint)
        self.vqgan.eval()
        if clip:
            self.clip = clip
        else:
            self.clip = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
        self.clip.to(self.device)
        self.clip_preprocessor = ProcessorGradientFlow(device=self.device)

        self.iterations = iterations
        self.lr = lr
        self.log = log
        self.make_grid = make_grid
        self.return_val = return_val
        self.quantize = quantize
        self.latent_dim = self.vqgan.decoder.z_shape

    def make_animation(self, input_path=None, output_path=None, total_duration=5, extend_frames=True):
        """
        Make an animation from the intermediate images saved during generation.
        By default, uses the images from the most recent generation created by the generate function.
        If you want to use images from a different generation, pass the path to the folder containing the images as input_path.
        """
        images = []
        if output_path is None:
            output_path = "./animation.gif"
        if input_path is None:
            input_path = self.save_path
        paths = sorted(glob(input_path + "/*"))
        if not len(paths):
            raise ValueError(
                "No images found in save path, aborting (did you pass save_intermediate=True to the generate"
                " function?)"
            )
        if len(paths) == 1:
            print("Only one image found in save path, (did you pass save_intermediate=True to the generate function?)")
        frame_duration = total_duration / len(paths)
        durations = [frame_duration] * len(paths)
        if extend_frames:
            durations[0] = 1.5
            durations[-1] = 3
        for file_name in paths:
            if file_name.endswith(".png"):
                images.append(imageio.imread(file_name))
        imageio.mimsave(output_path, images, duration=durations)
        print(f"gif saved to {output_path}")

    def _get_latent(self, path=None, img=None):
        if not (path or img):
            raise ValueError("Input either path or tensor")
        if img is not None:
            raise NotImplementedError
        x = preprocess(Image.open(path), target_image_size=256).to(self.device)
        x_processed = preprocess_vqgan(x)
        z, *_ = self.vqgan.encode(x_processed)
        return z

    def _add_vector(self, transform_vector):
        """Add a vector transform to the base latent and returns the resulting image."""
        base_latent = self.latent.detach().requires_grad_()
        trans_latent = base_latent + transform_vector
        if self.quantize:
            z_q, *_ = self.vqgan.quantize(trans_latent)
        else:
            z_q = trans_latent
        return self.vqgan.decode(z_q)

    def _get_clip_similarity(self, prompts, image, weights=None):
        clip_inputs = self.clip_preprocessor(text=prompts, images=image, return_tensors="pt", padding=True)
        clip_outputs = self.clip(**clip_inputs)
        similarity_logits = clip_outputs.logits_per_image
        if weights is not None:
            similarity_logits = similarity_logits * weights
        return similarity_logits.sum()

    def _get_clip_loss(self, pos_prompts, neg_prompts, image):
        pos_logits = self._get_clip_similarity(pos_prompts["prompts"], image, weights=(1 / pos_prompts["weights"]))
        if neg_prompts:
            neg_logits = self._get_clip_similarity(neg_prompts["prompts"], image, weights=neg_prompts["weights"])
        else:
            neg_logits = torch.tensor([1], device=self.device)
        loss = -torch.log(pos_logits) + torch.log(neg_logits)
        return loss

    def _optimize_CLIP(self, original_img, pos_prompts, neg_prompts):
        vector = torch.randn_like(self.latent, requires_grad=True, device=self.device)
        optim = torch.optim.Adam([vector], lr=self.lr)

        for i in range(self.iterations):
            optim.zero_grad()
            transformed_img = self._add_vector(vector)
            processed_img = loop_post_process(transformed_img)
            clip_loss = self._get_CLIP_loss(pos_prompts, neg_prompts, processed_img)
            print("CLIP loss", clip_loss)
            if self.log:
                wandb.log({"CLIP Loss": clip_loss})
            clip_loss.backward(retain_graph=True)
            optim.step()
            if self.return_val == "image":
                yield custom_to_pil(transformed_img[0])
            else:
                yield vector

    def _init_logging(self, positive_prompts, negative_prompts, image_path):
        wandb.init(reinit=True, project="face-editor")
        wandb.config.update({"Positive Prompts": positive_prompts})
        wandb.config.update({"Negative Prompts": negative_prompts})
        wandb.config.update({"lr": self.lr, "iterations": self.iterations})
        if image_path:
            image = Image.open(image_path)
            image = image.resize((256, 256))
            wandb.log("Original Image", wandb.Image(image))

    def process_prompts(self, prompts):
        if not prompts:
            return []
        processed_prompts = []
        weights = []
        if isinstance(prompts, str):
            prompts = [prompt.strip() for prompt in prompts.split("|")]
        for prompt in prompts:
            if isinstance(prompt, (tuple, list)):
                processed_prompt = prompt[0]
                weight = float(prompt[1])
            elif ":" in prompt:
                processed_prompt, weight = prompt.split(":")
                weight = float(weight)
            else:
                processed_prompt = prompt
                weight = 1.0
            processed_prompts.append(processed_prompt)
            weights.append(weight)
        return {
            "prompts": processed_prompts,
            "weights": torch.tensor(weights, device=self.device),
        }

    def generate(
        self,
        pos_prompts,
        neg_prompts=None,
        image_path=None,
        show_intermediate=True,
        save_intermediate=False,
        show_final=True,
        save_final=True,
        save_path=None,
    ):
        """Generate an image from the given prompts.
        If image_path is provided, the image is used as a starting point for the optimization.
        If image_path is not provided, a random latent vector is used as a starting point.
        You must provide at least one positive prompt, and optionally provide negative prompts.
        Prompts must be formatted in one of the following ways:
        - A single prompt as a string, e.g "A smiling woman"
        - A set of prompts separated by pipes: "A smiling woman | a woman with brown hair"
        - A set of prompts and their weights separated by colons: "A smiling woman:1 | a woman with brown hair: 3" (default weight is 1)
        - A list of prompts, e.g ["A smiling woman", "a woman with brown hair"]
        - A list of prompts and weights, e.g [("A smiling woman", 1), ("a woman with brown hair", 3)]
        """
        if image_path:
            self.latent = self._get_latent(image_path)
        else:
            self.latent = torch.randn(self.latent_dim, device=self.device)
        if self.log:
            self._init_logging(pos_prompts, neg_prompts, image_path)

        assert pos_prompts, "You must provide at least one positive prompt."
        pos_prompts = self.process_prompts(pos_prompts)
        neg_prompts = self.process_prompts(neg_prompts)
        if save_final and save_path is None:
            save_path = os.path.join("./outputs/", "_".join(pos_prompts["prompts"]))
        if not os.path.exists(save_path):
            os.makedirs(save_path)
        else:
            save_path = save_path + "_" + get_timestamp()
            os.makedirs(save_path)
        self.save_path = save_path

        original_img = self.vqgan.decode(self.latent)[0]
        if show_intermediate:
            print("Original Image")
            show_pil(custom_to_pil(original_img))

        original_img = loop_post_process(original_img)
        for iter, transformed_img in enumerate(self._optimize_CLIP(original_img, pos_prompts, neg_prompts)):
            if show_intermediate:
                show_pil(transformed_img)
            if save_intermediate:
                transformed_img.save(os.path.join(self.save_path, f"iter_{iter:03d}.png"))
            if self.log:
                wandb.log({"Image": wandb.Image(transformed_img)})
        if show_final:
            show_pil(transformed_img)
        if save_final:


SUMMARY:
The code provided appears to be part of a Python class-based implementation of a neural face editor. The class is designed to optimize the output of a VAE (Variational Autoencoder) model using CLIP (Contrastive Learning of Images To Beat the Best Human Performers At A Low Resource) loss, which encourages the generated image to match a target sample while minimizing the distance between it and other negative samples.

Here's a concise summary:

**Key Features:**

1. The class takes in VAE-generated latent vectors and optimizes them using CLIP loss.
2. It supports multiple positive and negative prompts, with respective weights.
3. Optimization yields intermediate images at each iteration, which can be saved or displayed for supervision.
4. The final image can also be saved alongside the optimized intermediate outputs.

**Key Functions:**

1. `_init_logging`: Initializes W&B (WandB) logging for project setup and configuration.
2. `process_prompts`: Processes prompts into the required format for W&B logging.
3. `generate`: The main function that generates images from provided prompts, optimizes latents using CLIP loss, and saves intermediate and final outputs.

**Architecture:**

The architecture revolves around VAEs and CLIP losses, with the optimization loop iterating through multiple positive and negative samples to find an optimal image representation.

This class appears to be a valuable tool for face editors, allowing users to fine-tune their models using human-like guidance from prompt inputs.


key: chroma:document
string_value:
CODE:

def freeze_module(module):
    for param in module.parameters():


SUMMARY:
**Function Summary: Freeze Module**

**Purpose:** Temporarily sets `requires_grad` to `False` for all parameters of a given `module`.

**Key Implementation Detail:** The function iterates over each parameter in the module and directly modifies its `requires_grad` attribute.

**Notable Dependencies/Requirements:** None, as this is a self-contained function.


key: chroma:document
string_value:
CODE:

def get_device():
    device = "cuda" if torch.cuda.is_available() else "cpu"
    if torch.backends.mps.is_available() and torch.backends.mps.is_built():
        device = "mps"
    if device == "mps":
        print(
            "WARNING: MPS currently doesn't seem to work, and messes up backpropagation without any visible torch"
            " errors. I recommend using CUDA on a colab notebook or CPU instead if you're facing inexplicable issues"
            " with generations."
        )


SUMMARY:
**Function Summary: `get_device()`**

**Main Purpose and Functionality**
The `get_device()` function determines the preferred device for computations. It checks availability of CUDA, MPS (Mobile GPU), and returns the most suitable device.

**Key Implementation Details**

* Uses Tensorflow's `torch` module to check device availability.
* Returns "cuda" if available, otherwise falls back to other options:
	+ If both CUDA and MPS are available but not built, uses "mps".
	+ If only "mps" is available (but built), prints a warning message and recommends using CUDA or CPU.

**Notable Dependencies and Requirements**

* `torch`: Tensorflow backend for PyTorch.
* `torch.backends.mps`: Mobile GPU support.


key: chroma:document
string_value:
CODE:

def show_pil(img):
    fig = plt.imshow(img)
    fig.axes.get_xaxis().set_visible(False)
    fig.axes.get_yaxis().set_visible(False)


SUMMARY:
**Summary**

This `show_pil` function displays an image using pandas' internal PIL (Python Imaging Library) functionality. It:

*   Displays the image with plotly's `imshow` function
*   Hides the x and y axes of the plot
*   Shows the resulting plot
*   **Main Purpose:** Displaying an image
*   **Key Implementation Detail:** Using `plt.imshow` to display the image, followed by hiding and showing the axes using `fig.axes.get_xaxis()` and `fig.axes.get_yaxis()`

**Analysis Notes:**

*   External dependencies:
    *   pandas
    *   numpy (not shown in this code snippet)
    *   plotly
    *   matplotlib.pyplot

**Key Algorithm/Pattern:** Utilizing the `imshow` function to display images.


key: chroma:document
string_value:
CODE:

def get_timestamp():
    current_time = datetime.now()
    timestamp = current_time.strftime("%H:%M:%S")


SUMMARY:
**Function Summary**

The `get_timestamp` function retrieves and returns the current time in HH:MM:SS format using the `datetime` module.

### Key Details
- Uses `datetime.now()` to get the current time
- Formats the time as `%H:%M:%S` using `strftime`
- Returns the timestamp string

### Dependencies
- `datetime` class


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/seq2seq-distillation/finetune.py


key: chroma:document
string_value:
CODE:

class SummarizationModule(BaseTransformer):
    mode = "summarization"
    loss_names = ["loss"]
    metric_names = ROUGE_KEYS
    default_val_metric = "rouge2"

    def __init__(self, hparams, **kwargs):
        if hparams.sortish_sampler and hparams.gpus > 1:
            hparams.replace_sampler_ddp = False
        elif hparams.max_tokens_per_batch is not None:
            if hparams.gpus > 1:
                raise NotImplementedError("Dynamic Batch size does not work for multi-gpu training")
            if hparams.sortish_sampler:
                raise ValueError("--sortish_sampler and --max_tokens_per_batch may not be used simultaneously")

        super().__init__(hparams, num_labels=None, mode=self.mode, **kwargs)
        use_task_specific_params(self.model, "summarization")
        save_git_info(self.hparams.output_dir)
        self.metrics_save_path = Path(self.output_dir) / "metrics.json"
        self.hparams_save_path = Path(self.output_dir) / "hparams.pkl"
        pickle_save(self.hparams, self.hparams_save_path)
        self.step_count = 0
        self.metrics = defaultdict(list)
        self.model_type = self.config.model_type
        self.vocab_size = self.config.tgt_vocab_size if self.model_type == "fsmt" else self.config.vocab_size

        self.dataset_kwargs: dict = {
            "data_dir": self.hparams.data_dir,
            "max_source_length": self.hparams.max_source_length,
            "prefix": self.model.config.prefix or "",
        }
        n_observations_per_split = {
            "train": self.hparams.n_train,
            "val": self.hparams.n_val,
            "test": self.hparams.n_test,
        }
        self.n_obs = {k: v if v >= 0 else None for k, v in n_observations_per_split.items()}

        self.target_lens = {
            "train": self.hparams.max_target_length,
            "val": self.hparams.val_max_target_length,
            "test": self.hparams.test_max_target_length,
        }
        assert self.target_lens["train"] <= self.target_lens["val"], f"target_lens: {self.target_lens}"
        assert self.target_lens["train"] <= self.target_lens["test"], f"target_lens: {self.target_lens}"
        if self.hparams.freeze_embeds:
            freeze_embeds(self.model)
        if self.hparams.freeze_encoder:
            freeze_params(self.model.get_encoder())
            assert_all_frozen(self.model.get_encoder())

        self.hparams.git_sha = get_git_info()["repo_sha"]
        self.num_workers = hparams.num_workers
        self.decoder_start_token_id = None  # default to config
        if self.model.config.decoder_start_token_id is None and isinstance(self.tokenizer, MBartTokenizer):
            self.decoder_start_token_id = self.tokenizer.lang_code_to_id[hparams.tgt_lang]
            self.model.config.decoder_start_token_id = self.decoder_start_token_id
        self.dataset_class = (
            Seq2SeqDataset if hasattr(self.tokenizer, "prepare_seq2seq_batch") else LegacySeq2SeqDataset
        )
        self.already_saved_batch = False
        self.eval_beams = self.model.config.num_beams if self.hparams.eval_beams is None else self.hparams.eval_beams
        if self.hparams.eval_max_gen_length is not None:
            self.eval_max_length = self.hparams.eval_max_gen_length
        else:
            self.eval_max_length = self.model.config.max_length
        self.val_metric = self.default_val_metric if self.hparams.val_metric is None else self.hparams.val_metric

    def save_readable_batch(self, batch: Dict[str, torch.Tensor]) -> Dict[str, List[str]]:
        """A debugging utility"""
        readable_batch = {
            k: self.tokenizer.batch_decode(v.tolist()) if "mask" not in k else v.shape for k, v in batch.items()
        }
        save_json(readable_batch, Path(self.output_dir) / "text_batch.json")
        save_json({k: v.tolist() for k, v in batch.items()}, Path(self.output_dir) / "tok_batch.json")

        self.already_saved_batch = True
        return readable_batch

    def forward(self, input_ids, **kwargs):
        return self.model(input_ids, **kwargs)

    def ids_to_clean_text(self, generated_ids: List[int]):
        gen_text = self.tokenizer.batch_decode(
            generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
        )
        return lmap(str.strip, gen_text)

    def _step(self, batch: dict) -> Tuple:
        pad_token_id = self.tokenizer.pad_token_id
        src_ids, src_mask = batch["input_ids"], batch["attention_mask"]
        tgt_ids = batch["labels"]
        if isinstance(self.model, T5ForConditionalGeneration):
            decoder_input_ids = self.model._shift_right(tgt_ids)
        else:
            decoder_input_ids = shift_tokens_right(tgt_ids, pad_token_id)
        if not self.already_saved_batch:  # This would be slightly better if it only happened on rank zero
            batch["decoder_input_ids"] = decoder_input_ids
            self.save_readable_batch(batch)

        outputs = self(src_ids, attention_mask=src_mask, decoder_input_ids=decoder_input_ids, use_cache=False)
        lm_logits = outputs["logits"]
        if self.hparams.label_smoothing == 0:
            # Same behavior as modeling_bart.py, besides ignoring pad_token_id
            ce_loss_fct = nn.CrossEntropyLoss(ignore_index=pad_token_id)

            assert lm_logits.shape[-1] == self.vocab_size
            loss = ce_loss_fct(lm_logits.view(-1, lm_logits.shape[-1]), tgt_ids.view(-1))
        else:
            lprobs = nn.functional.log_softmax(lm_logits, dim=-1)
            loss, nll_loss = label_smoothed_nll_loss(
                lprobs, tgt_ids, self.hparams.label_smoothing, ignore_index=pad_token_id
            )
        return (loss,)

    @property
    def pad(self) -> int:
        return self.tokenizer.pad_token_id

    def training_step(self, batch, batch_idx) -> Dict:
        loss_tensors = self._step(batch)

        logs = dict(zip(self.loss_names, loss_tensors))
        # tokens per batch
        logs["tpb"] = batch["input_ids"].ne(self.pad).sum() + batch["labels"].ne(self.pad).sum()
        logs["bs"] = batch["input_ids"].shape[0]
        logs["src_pad_tok"] = batch["input_ids"].eq(self.pad).sum()
        logs["src_pad_frac"] = batch["input_ids"].eq(self.pad).float().mean()
        # TODO(SS): make a wandb summary metric for this
        return {"loss": loss_tensors[0], "log": logs}

    def validation_step(self, batch, batch_idx) -> Dict:
        return self._generative_step(batch)

    def validation_epoch_end(self, outputs, prefix="val") -> Dict:
        self.step_count += 1
        losses = {k: torch.stack([x[k] for x in outputs]).mean() for k in self.loss_names}
        loss = losses["loss"]
        generative_metrics = {
            k: np.array([x[k] for x in outputs]).mean() for k in self.metric_names + ["gen_time", "gen_len"]
        }
        metric_val = (
            generative_metrics[self.val_metric] if self.val_metric in generative_metrics else losses[self.val_metric]
        )
        metric_tensor: torch.FloatTensor = torch.tensor(metric_val).type_as(loss)
        generative_metrics.update({k: v.item() for k, v in losses.items()})
        losses.update(generative_metrics)
        all_metrics = {f"{prefix}_avg_{k}": x for k, x in losses.items()}
        all_metrics["step_count"] = self.step_count
        self.metrics[prefix].append(all_metrics)  # callback writes this to self.metrics_save_path
        preds = flatten_list([x["preds"] for x in outputs])
        return {
            "log": all_metrics,
            "preds": preds,
            f"{prefix}_loss": loss,
            f"{prefix}_{self.val_metric}": metric_tensor,
        }

    def calc_generative_metrics(self, preds, target) -> Dict:
        return calculate_rouge(preds, target)

    def _generative_step(self, batch: dict) -> dict:
        t0 = time.time()

        # parser.add_argument('--eval_max_gen_length', type=int, default=None, help='never generate more than n tokens')
        generated_ids = self.model.generate(
            batch["input_ids"],
            attention_mask=batch["attention_mask"],
            use_cache=True,
            decoder_start_token_id=self.decoder_start_token_id,
            num_beams=self.eval_beams,
            max_length=self.eval_max_length,
        )
        gen_time = (time.time() - t0) / batch["input_ids"].shape[0]
        preds: List[str] = self.ids_to_clean_text(generated_ids)
        target: List[str] = self.ids_to_clean_text(batch["labels"])
        loss_tensors = self._step(batch)
        base_metrics = dict(zip(self.loss_names, loss_tensors))
        rouge: Dict = self.calc_generative_metrics(preds, target)
        summ_len = np.mean(lmap(len, generated_ids))
        base_metrics.update(gen_time=gen_time, gen_len=summ_len, preds=preds, target=target, **rouge)
        return base_metrics

    def test_step(self, batch, batch_idx):
        return self._generative_step(batch)

    def test_epoch_end(self, outputs):
        return self.validation_epoch_end(outputs, prefix="test")

    def get_dataset(self, type_path) -> Seq2SeqDataset:
        n_obs = self.n_obs[type_path]
        max_target_length = self.target_lens[type_path]
        dataset = self.dataset_class(
            self.tokenizer,
            type_path=type_path,
            n_obs=n_obs,
            max_target_length=max_target_length,
            **self.dataset_kwargs,
        )
        return dataset

    def get_dataloader(self, type_path: str, batch_size: int, shuffle: bool = False) -> DataLoader:
        dataset = self.get_dataset(type_path)

        if self.hparams.sortish_sampler and type_path != "test" and type_path != "val":
            sampler = dataset.make_sortish_sampler(batch_size, distributed=self.hparams.gpus > 1)
            return DataLoader(
                dataset,
                batch_size=batch_size,
                collate_fn=dataset.collate_fn,
                shuffle=False,
                num_workers=self.num_workers,
                sampler=sampler,
            )

        elif self.hparams.max_tokens_per_batch is not None and type_path != "test" and type_path != "val":
            batch_sampler = dataset.make_dynamic_sampler(
                self.hparams.max_tokens_per_batch, distributed=self.hparams.gpus > 1
            )
            return DataLoader(
                dataset,
                batch_sampler=batch_sampler,
                collate_fn=dataset.collate_fn,
                # shuffle=False,
                num_workers=self.num_workers,
                # batch_size=None,
            )
        else:
            return DataLoader(
                dataset,
                batch_size=batch_size,
                collate_fn=dataset.collate_fn,
                shuffle=shuffle,
                num_workers=self.num_workers,
                sampler=None,
            )

    def train_dataloader(self) -> DataLoader:
        dataloader = self.get_dataloader("train", batch_size=self.hparams.train_batch_size, shuffle=True)
        return dataloader

    def val_dataloader(self) -> DataLoader:
        return self.get_dataloader("val", batch_size=self.hparams.eval_batch_size)

    def test_dataloader(self) -> DataLoader:
        return self.get_dataloader("test", batch_size=self.hparams.eval_batch_size)

    @staticmethod
    def add_model_specific_args(parser, root_dir):
        BaseTransformer.add_model_specific_args(parser, root_dir)
        add_generic_args(parser, root_dir)
        parser.add_argument(
            "--max_source_length",
            default=1024,
            type=int,
            help=(
                "The maximum total input sequence length after tokenization. Sequences longer "
                "than this will be truncated, sequences shorter will be padded."
            ),
        )
        parser.add_argument(
            "--max_target_length",
            default=56,
            type=int,
            help=(
                "The maximum total input sequence length after tokenization. Sequences longer "
                "than this will be truncated, sequences shorter will be padded."
            ),
        )
        parser.add_argument(
            "--val_max_target_length",
            default=142,  # these defaults are optimized for CNNDM. For xsum, see README.md.
            type=int,
            help=(
                "The maximum total input sequence length after tokenization. Sequences longer "
                "than this will be truncated, sequences shorter will be padded."
            ),
        )
        parser.add_argument(
            "--test_max_target_length",
            default=142,
            type=int,
            help=(
                "The maximum total input sequence length after tokenization. Sequences longer "
                "than this will be truncated, sequences shorter will be padded."
            ),
        )
        parser.add_argument("--freeze_encoder", action="store_true")
        parser.add_argument("--freeze_embeds", action="store_true")
        parser.add_argument("--sortish_sampler", action="store_true", default=False)
        parser.add_argument("--overwrite_output_dir", action="store_true", default=False)
        parser.add_argument("--max_tokens_per_batch", type=int, default=None)
        parser.add_argument("--logger_name", type=str, choices=["default", "wandb", "wandb_shared"], default="default")
        parser.add_argument("--n_train", type=int, default=-1, required=False, help="# examples. -1 means use all.")
        parser.add_argument("--n_val", type=int, default=500, required=False, help="# examples. -1 means use all.")
        parser.add_argument("--n_test", type=int, default=-1, required=False, help="# examples. -1 means use all.")
        parser.add_argument(
            "--task", type=str, default="summarization", required=False, help="# examples. -1 means use all."
        )
        parser.add_argument("--label_smoothing", type=float, default=0.0, required=False)
        parser.add_argument("--src_lang", type=str, default="", required=False)
        parser.add_argument("--tgt_lang", type=str, default="", required=False)
        parser.add_argument("--eval_beams", type=int, default=None, required=False)
        parser.add_argument(
            "--val_metric", type=str, default=None, required=False, choices=["bleu", "rouge2", "loss", None]
        )
        parser.add_argument("--eval_max_gen_length", type=int, default=None, help="never generate more than n tokens")
        parser.add_argument("--save_top_k", type=int, default=1, required=False, help="How many checkpoints to save")
        parser.add_argument(
            "--early_stopping_patience",
            type=int,
            default=-1,
            required=False,
            help=(
                "-1 means never early stop. early_stopping_patience is measured in validation checks, not epochs. So"
                " val_check_interval will effect it."
            ),
        )


SUMMARY:
This code provides a concise overview of the Hugging Face Transformers library's transformer architecture for text generation tasks such as summarization and question answering.

Here are the key points:

*   The transformer model, named `BaseTransformer`, is defined with various arguments that can be passed to the constructor. These include:
    *   `src_lang` and `tgt_lang`: source and target language codes.
    *   `task`: task type (e.g., summarization).
    *   `label_smoothing`: smoothing factor for label smoothing regularization.
    *   `overwrite_output_dir`: whether to overwrite the output directory with default values.
    *   `n_train`, `n_val`, `n_test`: number of examples for training, validation, and testing.
    *   `train_batch_size` and `eval_batch_size`: training and evaluation batch sizes.

*   The model is added to a parser using the `add_model_specific_args` function.

Overall, this code sets up the Transformers library with various options for text generation tasks such as summarization and question answering.


key: chroma:document
string_value:
CODE:

class TranslationModule(SummarizationModule):
    mode = "translation"
    loss_names = ["loss"]
    metric_names = ["bleu"]
    default_val_metric = "bleu"

    def __init__(self, hparams, **kwargs):
        super().__init__(hparams, **kwargs)
        self.dataset_kwargs["src_lang"] = hparams.src_lang
        self.dataset_kwargs["tgt_lang"] = hparams.tgt_lang

    def calc_generative_metrics(self, preds, target) -> dict:


SUMMARY:
**TranslationModule Class Summary**

* **Purpose:** Translation module for summarization pipeline, responsible for translating source text to target language.
* **Key Functionality:**
	+ Initializes with hypothesis parameters and dataset settings.
	+ Calculates BLEU score using `calculate_bleu` function from target text.
* **Notable Implementation Details:**
	+ Uses `super().__init__` to inherit from parent class (`SummarizationModule`) with updated dataset parameters.
	+ Applies custom metric calculation for validation using `bleu` score.

**Dependences and Interfaces:**

* Inherits from `SummarizationModule`
* Utilizes external function `calculate_bleu` from unknown module/namespace.


key: chroma:document
string_value:
CODE:

def main(args, model=None) -> SummarizationModule:
    Path(args.output_dir).mkdir(exist_ok=True)
    check_output_dir(args, expected_items=3)

    if model is None:
        if "summarization" in args.task:
            model: SummarizationModule = SummarizationModule(args)
        else:
            model: SummarizationModule = TranslationModule(args)
    dataset = Path(args.data_dir).name
    if (
        args.logger_name == "default"
        or args.fast_dev_run
        or str(args.output_dir).startswith("/tmp")
        or str(args.output_dir).startswith("/var")
    ):
        logger = True  # don't pollute wandb logs unnecessarily
    elif args.logger_name == "wandb":
        from pytorch_lightning.loggers import WandbLogger

        project = os.environ.get("WANDB_PROJECT", dataset)
        logger = WandbLogger(name=model.output_dir.name, project=project)

    elif args.logger_name == "wandb_shared":
        from pytorch_lightning.loggers import WandbLogger

        logger = WandbLogger(name=model.output_dir.name, project=f"hf_{dataset}")

    if args.early_stopping_patience >= 0:
        es_callback = get_early_stopping_callback(model.val_metric, args.early_stopping_patience)
    else:
        es_callback = False

    lower_is_better = args.val_metric == "loss"
    trainer: pl.Trainer = generic_train(
        model,
        args,
        logging_callback=Seq2SeqLoggingCallback(),
        checkpoint_callback=get_checkpoint_callback(
            args.output_dir, model.val_metric, args.save_top_k, lower_is_better
        ),
        early_stopping_callback=es_callback,
        logger=logger,
    )
    pickle_save(model.hparams, model.output_dir / "hparams.pkl")
    if not args.do_predict:
        return model

    model.hparams.test_checkpoint = ""
    checkpoints = sorted(glob.glob(os.path.join(args.output_dir, "*.ckpt"), recursive=True))
    if checkpoints:
        model.hparams.test_checkpoint = checkpoints[-1]
        trainer.resume_from_checkpoint = checkpoints[-1]
    trainer.logger.log_hyperparams(model.hparams)

    # test() without a model tests using the best checkpoint automatically
    trainer.test()


SUMMARY:
**Main Function Summary**

```python
def main(args, model=None) -> SummarizationModule:
    """
    The main entry point of the codebase.

    Args:
        args: Command line arguments containing task, data directory, output directory, etc.
        model: Can be any of SummarizationModule or TranslationModule.

    Returns:
        model (SummarizationModule): After training a model.
    """

    # Create output directory and check for valid items
    Path(args.output_dir).mkdir(exist_ok=True)
    check_output_dir(args, expected_items=3)

    # Initialize logger based on command line argument
    if args.logger_name == "default" or args.fast_dev_run:
        logger = True  # don't pollute wandb logs unnecessarily
    elif args.logger_name == "wandb":
        from pytorch_lightning.loggers import WandbLogger
        project = os.environ.get("WANDB_PROJECT", dataset)
        logger = WandbLogger(name=model.output_dir.name, project=project)
    elif args.logger_name == "wandb_shared":
        from pytorch_lightning.loggers import WandbLogger
        logger = WandbLogger(name=model.output_dir.name, project=f"hf_{dataset}")

    # Set up early stopping if patience is greater than 0
    es_callback = get_early_stopping_callback(model.val_metric, args.early_stopping_patience)

    # Initialize trainer for training the model
    lower_is_better = args.val_metric == "loss"
    trainer: pl.Trainer = generic_train(
        model,
        args,
        logging_callback=Seq2SeqLoggingCallback(),
        checkpoint_callback=get_checkpoint_callback(
            args.output_dir, model.val_metric, args.save_top_k, lower_is_better
        ),
        early_stopping_callback=es_callback,
        logger=logger,
    )

    # Pickle and save hparams to output directory
    pickle_save(model.hparams, model.output_dir / "hparams.pkl")

    # Train or test the model if do_predict is set to False
    model.hparams.test_checkpoint = ""
    checkpoints = sorted(glob.glob(os.path.join(args.output_dir, "*.ckpt"), recursive=True))
    if checkpoints:
        model.hparams.test_checkpoint = checkpoints[-1]
        trainer.resume_from_checkpoint = checkpoints[-1]

    # Log hyperparameters and test the model
    trainer(logger=logger)
    trainer.residual_forward pass: None  # Remove this to avoid error
```

**Key Implementation Details**

* The function initializes a logger based on the command line argument `args.logger_name`.
* It sets up early stopping using the `get_early_stopping_callback` function if patience is greater than 0.
* The trainer for training the model is initialized with various callbacks and logging parameters.
* The function saves hyperparameters to a pickle file named "hparams.pkl" in the output directory.

**Notable Dependencies or Requirements**

* `pytorch-lightning`
* `wandb` (version 2.3.0) for logging purposes
* ` Seq2SeqLoggingCallback`, `get_early_stopping_callback`, `get_checkpoint_callback`
* `hf_{dataset}` as project name in WandbLogger when logger_name = "wandb_shared"


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/seq2seq-distillation/lightning_base.py


key: chroma:document
string_value:
CODE:

class BaseTransformer(pl.LightningModule):
    def __init__(
        self,
        hparams: argparse.Namespace,
        num_labels=None,
        mode="base",
        config=None,
        tokenizer=None,
        model=None,
        **config_kwargs,
    ):
        """Initialize a model, tokenizer and config."""
        super().__init__()
        # TODO: move to self.save_hyperparameters()
        # self.save_hyperparameters()
        # can also expand arguments into trainer signature for easier reading

        self.save_hyperparameters(hparams)
        self.step_count = 0
        self.output_dir = Path(self.hparams.output_dir)
        cache_dir = self.hparams.cache_dir if self.hparams.cache_dir else None
        if config is None:
            self.config = AutoConfig.from_pretrained(
                self.hparams.config_name if self.hparams.config_name else self.hparams.model_name_or_path,
                **({"num_labels": num_labels} if num_labels is not None else {}),
                cache_dir=cache_dir,
                **config_kwargs,
            )
        else:
            self.config: PretrainedConfig = config

        extra_model_params = ("encoder_layerdrop", "decoder_layerdrop", "dropout", "attention_dropout")
        for p in extra_model_params:
            if getattr(self.hparams, p, None):
                assert hasattr(self.config, p), f"model config doesn't have a `{p}` attribute"
                setattr(self.config, p, getattr(self.hparams, p))

        if tokenizer is None:
            self.tokenizer = AutoTokenizer.from_pretrained(
                self.hparams.tokenizer_name if self.hparams.tokenizer_name else self.hparams.model_name_or_path,
                cache_dir=cache_dir,
            )
        else:
            self.tokenizer: PreTrainedTokenizer = tokenizer
        self.model_type = MODEL_MODES[mode]
        if model is None:
            self.model = self.model_type.from_pretrained(
                self.hparams.model_name_or_path,
                from_tf=bool(".ckpt" in self.hparams.model_name_or_path),
                config=self.config,
                cache_dir=cache_dir,
            )
        else:
            self.model = model

    def load_hf_checkpoint(self, *args, **kwargs):
        self.model = self.model_type.from_pretrained(*args, **kwargs)

    def get_lr_scheduler(self):
        get_schedule_func = arg_to_scheduler[self.hparams.lr_scheduler]
        scheduler = get_schedule_func(
            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=self.total_steps()
        )
        scheduler = {"scheduler": scheduler, "interval": "step", "frequency": 1}
        return scheduler

    def configure_optimizers(self):
        """Prepare optimizer and schedule (linear warmup and decay)"""
        model = self.model
        no_decay = ["bias", "LayerNorm.weight"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
                "weight_decay": self.hparams.weight_decay,
            },
            {
                "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
                "weight_decay": 0.0,
            },
        ]
        if self.hparams.adafactor:
            optimizer = Adafactor(
                optimizer_grouped_parameters, lr=self.hparams.learning_rate, scale_parameter=False, relative_step=False
            )

        else:
            optimizer = AdamW(
                optimizer_grouped_parameters, lr=self.hparams.learning_rate, eps=self.hparams.adam_epsilon
            )
        self.opt = optimizer

        scheduler = self.get_lr_scheduler()

        return [optimizer], [scheduler]

    def test_step(self, batch, batch_nb):
        return self.validation_step(batch, batch_nb)

    def test_epoch_end(self, outputs):
        return self.validation_end(outputs)

    def total_steps(self) -> int:
        """The number of total training steps that will be run. Used for lr scheduler purposes."""
        num_devices = max(1, self.hparams.gpus)  # TODO: consider num_tpu_cores
        effective_batch_size = self.hparams.train_batch_size * self.hparams.accumulate_grad_batches * num_devices
        return (self.dataset_size / effective_batch_size) * self.hparams.max_epochs

    def setup(self, mode):
        if mode == "test":
            self.dataset_size = len(self.test_dataloader().dataset)
        else:
            self.train_loader = self.get_dataloader("train", self.hparams.train_batch_size, shuffle=True)
            self.dataset_size = len(self.train_dataloader().dataset)

    def get_dataloader(self, type_path: str, batch_size: int, shuffle: bool = False):
        raise NotImplementedError("You must implement this for your task")

    def train_dataloader(self):
        return self.train_loader

    def val_dataloader(self):
        return self.get_dataloader("dev", self.hparams.eval_batch_size, shuffle=False)

    def test_dataloader(self):
        return self.get_dataloader("test", self.hparams.eval_batch_size, shuffle=False)

    def _feature_file(self, mode):
        return os.path.join(
            self.hparams.data_dir,
            "cached_{}_{}_{}".format(
                mode,
                list(filter(None, self.hparams.model_name_or_path.split("/"))).pop(),
                str(self.hparams.max_seq_length),
            ),
        )

    @pl.utilities.rank_zero_only
    def on_save_checkpoint(self, checkpoint: Dict[str, Any]) -> None:
        save_path = self.output_dir.joinpath("best_tfmr")
        self.model.config.save_step = self.step_count
        self.model.save_pretrained(save_path)
        self.tokenizer.save_pretrained(save_path)

    @staticmethod
    def add_model_specific_args(parser, root_dir):
        parser.add_argument(
            "--model_name_or_path",
            default=None,
            type=str,
            required=True,
            help="Path to pretrained model or model identifier from huggingface.co/models",
        )
        parser.add_argument(
            "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
        )
        parser.add_argument(
            "--tokenizer_name",
            default=None,
            type=str,
            help="Pretrained tokenizer name or path if not the same as model_name",
        )
        parser.add_argument(
            "--cache_dir",
            default="",
            type=str,
            help="Where do you want to store the pre-trained models downloaded from huggingface.co",
        )
        parser.add_argument(
            "--encoder_layerdrop",
            type=float,
            help="Encoder layer dropout probability (Optional). Goes into model.config",
        )
        parser.add_argument(
            "--decoder_layerdrop",
            type=float,
            help="Decoder layer dropout probability (Optional). Goes into model.config",
        )
        parser.add_argument(
            "--dropout",
            type=float,
            help="Dropout probability (Optional). Goes into model.config",
        )
        parser.add_argument(
            "--attention_dropout",
            type=float,
            help="Attention dropout probability (Optional). Goes into model.config",
        )
        parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
        parser.add_argument(
            "--lr_scheduler",
            default="linear",
            choices=arg_to_scheduler_choices,
            metavar=arg_to_scheduler_metavar,
            type=str,
            help="Learning rate scheduler",
        )
        parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
        parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
        parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
        parser.add_argument("--num_workers", default=4, type=int, help="kwarg passed to DataLoader")
        parser.add_argument("--num_train_epochs", dest="max_epochs", default=3, type=int)
        parser.add_argument("--train_batch_size", default=32, type=int)
        parser.add_argument("--eval_batch_size", default=32, type=int)


SUMMARY:
**Transformer Base Class Summary**

### Purpose and Functionality
This is the base class for transformer models in PyTorch Lightning. It initializes a model, tokenizer, and config, and provides functionality for training, validation, and testing.

### Key Implementation Details

*   Initializes `config` using AutoConfig from Hugging Face's transformers library.
*   Creates `optimizer` using either Adafactor or AdamW based on the specified hyperparameters.
*   Defines a custom data loader (`get_dataloader`) that needs to be implemented by subclasses.

### Notable Dependencies and Requirements

*   `transformers`: Hugging Face Transformers library
*   `pytorch`: PyTorch library and its Lightning wrapper (PyTorch Lightning)

### Key Hyperparameters

*   `learning_rate`: Initial learning rate for Adam optimization.
*   `lr_scheduler`: Learning rate scheduler, e.g., linear, step, etc.
*   `weight_decay` : Weight decay value for optimization.
*   `adam_epsilon`: Epsilon value for Adam optimizer.
*   `warmup_steps`: Warm-up steps in training phase.


key: chroma:document
string_value:
CODE:

class LoggingCallback(pl.Callback):
    def on_batch_end(self, trainer, pl_module):
        lr_scheduler = trainer.lr_schedulers[0]["scheduler"]
        lrs = {f"lr_group_{i}": lr for i, lr in enumerate(lr_scheduler.get_lr())}
        pl_module.logger.log_metrics(lrs)

    def on_validation_end(self, trainer: pl.Trainer, pl_module: pl.LightningModule):
        rank_zero_info("***** Validation results *****")
        metrics = trainer.callback_metrics
        # Log results
        for key in sorted(metrics):
            if key not in ["log", "progress_bar"]:
                rank_zero_info("{} = {}\n".format(key, str(metrics[key])))

    def on_test_end(self, trainer: pl.Trainer, pl_module: pl.LightningModule):
        rank_zero_info("***** Test results *****")
        metrics = trainer.callback_metrics
        # Log and save results to file
        output_test_results_file = os.path.join(pl_module.hparams.output_dir, "test_results.txt")
        with open(output_test_results_file, "w") as writer:
            for key in sorted(metrics):
                if key not in ["log", "progress_bar"]:
                    rank_zero_info("{} = {}\n".format(key, str(metrics[key])))


SUMMARY:
**Class Summary: LoggingCallback**

A PyTorch Lightning module that extends the `pl.Callback` class to provide logging functionality after each training, validation, and test batch or iteration.

**Main Purpose and Functionality**

This class logs various metrics after each training, validation, and test event. It also saves key results to a text file.

**Key Implementation Details**

* Extracts learning rate values from the `lr_scheduler` and logs them using `pl_module.logger`.
* Logs validation and test metrics using ` trainer.callback_metrics`, excluding specific keys like "log" and "progress_bar".
* Saves test results to a file with the specified output directory.

**Notable Dependencies or Requirements**

* PyTorch Lightning (`pl.Callback`, `pl.Trainer`, `pl.LightningModule`, `rank_zero_info`)
* `pl_module.logger` (uses logging functionality)
* External dependency:
	+ `os.path.join` for joining output directory and file path

**Primary Method Signatures:**

* `on_batch_end(trainer, pl_module)`: Update learning rates and log metrics.
* `on_validation_end(trainer, pl_module)`: Log validation results.
* `on_test_end(trainer, pl_module)`: Log test results and save to file.


key: chroma:document
string_value:
CODE:

def add_generic_args(parser, root_dir) -> None:
    #  To allow all pl args uncomment the following line
    #  parser = pl.Trainer.add_argparse_args(parser)
    parser.add_argument(
        "--output_dir",
        default=None,
        type=str,
        required=True,
        help="The output directory where the model predictions and checkpoints will be written.",
    )
    parser.add_argument(
        "--fp16",
        action="store_true",
        help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit",
    )

    parser.add_argument(
        "--fp16_opt_level",
        type=str,
        default="O2",
        help=(
            "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. "
            "See details at https://nvidia.github.io/apex/amp.html"
        ),
    )
    parser.add_argument("--n_tpu_cores", dest="tpu_cores", type=int)
    parser.add_argument("--max_grad_norm", dest="gradient_clip_val", default=1.0, type=float, help="Max gradient norm")
    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
    parser.add_argument("--do_predict", action="store_true", help="Whether to run predictions on the test set.")
    parser.add_argument(
        "--gradient_accumulation_steps",
        dest="accumulate_grad_batches",
        type=int,
        default=1,
        help="Number of updates steps to accumulate before performing a backward/update pass.",
    )
    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
    parser.add_argument(
        "--data_dir",
        default=None,
        type=str,
        required=True,
        help="The input data dir. Should contain the training files for the CoNLL-2003 NER task.",


SUMMARY:
**Summary:**

`add_generic_args(parser, root_dir)`: Configures a parser for a PyTorch Lightning project with various command-line arguments.

**Main Purpose and Functionality:**

- Adds several key command-line arguments:
  - `--output_dir` (str): Output directory for model predictions and checkpoints.
  - `--fp16` (bool): Enables or disables 16-bit precision mode.
  - `--fp16_opt_level` (str): Apex AMP optimization level.
  - Other arguments control training, prediction, gradient accumulation, and miscellaneous settings.

**Key Implementation Details:**

- Used argparse library for argument parsing.
- Key method signatures include:
  - `add_argument()`: adds specific command-line arguments to the parser.
  - `dest` attribute is used to map argument names with Python variable names.

**Notable Dependencies or Requirements:**

- PyTorch Lightning library (`pl.Trainer` class).
- `argparse` for command-line argument parsing.


key: chroma:document
string_value:
CODE:

def generic_train(
    model: BaseTransformer,
    args: argparse.Namespace,
    early_stopping_callback=None,
    logger=True,  # can pass WandbLogger() here
    extra_callbacks=[],
    checkpoint_callback=None,
    logging_callback=None,
    **extra_train_kwargs,
):
    pl.seed_everything(args.seed)

    # init model
    odir = Path(model.hparams.output_dir)
    odir.mkdir(exist_ok=True)

    # add custom checkpoints
    if checkpoint_callback is None:
        checkpoint_callback = pl.callbacks.ModelCheckpoint(
            filepath=args.output_dir, prefix="checkpoint", monitor="val_loss", mode="min", save_top_k=1
        )
    if early_stopping_callback:
        extra_callbacks.append(early_stopping_callback)
    if logging_callback is None:
        logging_callback = LoggingCallback()

    train_params = {}

    # TODO: remove with PyTorch 1.6 since pl uses native amp
    if args.fp16:
        train_params["precision"] = 16
        train_params["amp_level"] = args.fp16_opt_level

    if args.gpus > 1:
        train_params["distributed_backend"] = "ddp"

    train_params["accumulate_grad_batches"] = args.accumulate_grad_batches
    train_params["accelerator"] = extra_train_kwargs.get("accelerator", None)
    train_params["profiler"] = extra_train_kwargs.get("profiler", None)

    trainer = pl.Trainer.from_argparse_args(
        args,
        weights_summary=None,
        callbacks=[logging_callback] + extra_callbacks,
        logger=logger,
        checkpoint_callback=checkpoint_callback,
        **train_params,
    )

    if args.do_train:
        trainer.fit(model)



SUMMARY:
Here is a concise summary of the code:

**Function:** `generic_train`

**Purpose:**
Trains an instance of the `BaseTransformer` model using the specified hyperparameters and callbacks.

**Key Implementation Details:**

* Initializes model directory and sets up checkpoint callback
* Configures PyTorch optimizer, scheduler, and loss function via `train_params`
	+ Supports FP16 mixed precision training with optional optimization level
	+ Distributed training backend using (`ddp` on multiple GPUs)
	+ Accumulates gradients over batches and enables profiler for acceleration

**Callbacks:**

* Customizable with `early_stopping_callback`, `logging_callback`, and `checkpoint_callback`
* Logging callback automatically enabled if not specified

**Notable Dependencies:**
- Hugging Face's Transformers library (`BaseTransformer` model)
- PyTorch Lightning (PL) library for training and evaluation
- WandbLogger library for logging with Weights & Biases (optional)

**External Requirements:**
- `PyTorch`
- `Hugging Face's Transformers`
- `PyTorch Lightning`
- `Weights & Biases` (optional)


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/seq2seq-distillation/convert_pl_checkpoint_to_hf.py


key: chroma:document
string_value:
CODE:

def remove_prefix(text: str, prefix: str):
    if text.startswith(prefix):
        return text[len(prefix) :]


SUMMARY:
**Summary**

The `remove_prefix` function takes in two string parameters, `text` and `prefix`, and checks if the former starts with the latter. If it does, it returns a new string containing all characters after the prefix. Otherwise, it returns the original text.

### Key Implementation Details

- Checks for prefix match using the `startswith()` method.
- Returns a slice of `text` starting from the index after the prefix length (`len(prefix)`) if matched.

### Notable Dependencies and Requirements

- None noted.

### Example Output

- If `text` is `"hello world"` and `prefix` is `" hello "`, returns `"world"`.
- Otherwise, returns the original `text`.


key: chroma:document
string_value:
CODE:

def sanitize(sd):


SUMMARY:
**Functional Summary**

The `sanitize` function takes input dictionary `sd` and returns a sanitized version of the data by removing the prefix "model." from all keys.

* Primary purpose: Data preprocessing
* Key implementation detail: Uses dictionary comprehension with `remove_prefix` function to modify key names.
* Method signature: `def sanitize(sd): return { ... }`
* Notable dependencies:
	+ `remove_prefix` function (not shown in this code snippet)


key: chroma:document
string_value:
CODE:

def average_state_dicts(state_dicts: List[Dict[str, torch.Tensor]]):
    new_sd = {}
    for k in state_dicts[0].keys():
        tensors = [sd[k] for sd in state_dicts]
        new_t = sum(tensors) / len(tensors)
        assert isinstance(new_t, torch.Tensor)
        new_sd[k] = new_t


SUMMARY:
**Summary**

This function calculates the averaged values of tensors across multiple state dictionaries (`state_dicts`), preserving their structure and labels.

**Functionality:**

*   Calculate the average tensor value for each key in `state_dicts[0]`
*   Compute the sum of all tensors with that key and divide by the total number of tensors
*   Store the averaged tensor values in a new dictionary (`new_sd`)

**Key Implementation Details:**

*   Uses list comprehension to extract tensors from each state dictionary
*   Applies `assert isinstance(new_t, torch.Tensor)` to ensure averaged value is a tensor

**Notable Dependencies/Requirements:**

*   Uses Python's built-in data structures (lists, dictionaries)
*   Utilizes the PyTorch library for tensor operations (`torch.Tensor`)
*   Assumes that all state dictionaries have the same structure and labels as the first dictionary in `state_dicts`


key: chroma:document
string_value:
CODE:

def convert_pl_to_hf(pl_ckpt_path: str, hf_src_model_dir: str, save_path: str) -> None:
    """Cleanup a pytorch-lightning .ckpt file or experiment dir and save a huggingface model with that state dict.
    Silently allows extra pl keys (like teacher.) Puts all ckpt models into CPU RAM at once!

    Args:
        pl_ckpt_path (:obj:`str`): Path to a .ckpt file saved by pytorch_lightning or dir containing ckpt files.
            If a directory is passed, all .ckpt files inside it will be averaged!
        hf_src_model_dir (:obj:`str`): Path to a directory containing a correctly shaped checkpoint
        save_path (:obj:`str`): Directory to save the new model

    """
    hf_model = AutoModelForSeq2SeqLM.from_pretrained(hf_src_model_dir)
    if os.path.isfile(pl_ckpt_path):
        ckpt_files = [pl_ckpt_path]
    else:
        assert os.path.isdir(pl_ckpt_path)
        ckpt_files = list(Path(pl_ckpt_path).glob("*.ckpt"))
        assert ckpt_files, f"could not find any ckpt files inside the {pl_ckpt_path} directory"

    if len(ckpt_files) > 1:
        logger.info(f"averaging the weights of {ckpt_files}")

    state_dicts = [sanitize(torch.load(x, map_location="cpu")["state_dict"]) for x in ckpt_files]
    state_dict = average_state_dicts(state_dicts)

    missing, unexpected = hf_model.load_state_dict(state_dict, strict=False)
    assert not missing, f"missing keys: {missing}"
    hf_model.save_pretrained(save_path)
    try:
        tok = AutoTokenizer.from_pretrained(hf_src_model_dir)
        tok.save_pretrained(save_path)
    except Exception:
        pass


SUMMARY:
**Function Summary:**

`convert_pl_to_hf`

Converts a PyTorch Lightning checkpoint(.ckpt file or dir) to a Hugging Face model and saves the result.

**Purpose & Functionality:**

1. Loads a source Hugging Face model and creates an instance from `hf_src_model_dir`.
2. If `pl_ckpt_path` is a file, loads and sanitizes its state dict; if it's a dir, loads and averages all `.ckpt` files inside.
3. Saves the averaged state dict to the new model.
4. Optionally saves tokenizer weights.

**Key Implementation Details:**

1. Uses `torch.load()` to read .ckpt files on CPU RAM.
2. Averages state dicts using an unnamed custom function (`average_state_dicts()`) with input `state_dicts`.
3. Loads Hugging Face model weights using `hf_model.load_state_dict()` with `strict=False` for missing keys.

**Dependencies:**

1. PyTorch Lightning
2. Hugging Face Transformers

**Notable Interfaces and Classes:**

1. `AutoModelForSeq2SeqLM`
2. `AutoTokenizer`


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/seq2seq-distillation/sentence_splitter.py


key: chroma:document
string_value:
CODE:

def add_newline_to_end_of_each_sentence(x: str) -> str:
    """This was added to get rougeLsum scores matching published rougeL scores for BART and PEGASUS."""
    re.sub("<n>", "", x)  # remove pegasus newline char
    assert NLTK_AVAILABLE, "nltk must be installed to separate newlines between sentences. (pip install nltk)"


SUMMARY:
**Function Summary:**

`add_newline_to_end_of_each_sentence`

**Purpose:** Replaces obsolete newline characters with standard newlines and tokenizes input text into sentences.

**Implementation Details:**

* Removes Pegasus-specific newline character `<n>`
* Tokenizes input text using NLTK's `sent_tokenize`
	+ Returns joined strings with spaces between sentences, defaulting to the original string in case of an error

**Dependencies:**

*NLTK library with `nltk` module
* pip-installed nltk package for availability of language processing tools.

**Notable Interface Methods:** 

- `re.sub()`: Subpattern replacement
- `NLTK_AVAILABLE`
- `np.join()` : Used as `join()` on string


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/seq2seq-distillation/make_student.py


key: chroma:document
string_value:
CODE:

def copy_layers(src_layers: nn.ModuleList, dest_layers: nn.ModuleList, layers_to_copy: List[int]) -> None:
    layers_to_copy = nn.ModuleList([src_layers[i] for i in layers_to_copy])
    assert len(dest_layers) == len(layers_to_copy), f"{len(dest_layers)} != {len(layers_to_copy)}"


SUMMARY:
**Function Summary**

```python
copy_layers(
    src_layers: nn.ModuleList,
    dest_layers: nn.ModuleList,
    layers_to_copy: List[int]
) -> None:
    
    """
    Copies specified modules from the source network to the destination network.
    
    Args:
        src_layers (nn.ModuleList): Modules to copy from in the source network.
        dest_layers (nn.ModuleList): Modules to copy to in the destination network.
        layers_to_copy (List[int]): Indices of source modules to copy.
    
    Notes:
        - Only copies specified modules, not the entire network.
    """
```

**Implementation Details**

* Copies `layers_to_copy` modules from `src_layers` to `dest_layers`.
* Verifies that the number of modules in `dest_layers` matches the number of layers being copied.

**Notable Dependencies and Requirements**

* `nn.ModuleList`: PyTorch's module list container.
* `nn.Module`: PyTorch neural network module.


key: chroma:document
string_value:
CODE:

def pick_layers_to_copy(n_student, n_teacher):
    try:
        val = LAYERS_TO_COPY[n_teacher][n_student]
        return val
    except KeyError:
        if n_student != n_teacher:
            warnings.warn(
                f"no hardcoded layers to copy for teacher {n_teacher} -> student {n_student}, defaulting to first"
                f" {n_student}"
            )


SUMMARY:
**Summary:**

This function, `pick_layers_to_copy`, determines the layers to be copied from students to teachers in an educational system. It retrieves the copy layers for a specific teacher-student pair from the `LAYERS_TO_COPY` dictionary. If no data is found, it issues a warning and returns default layers for that student.

**Implementation Details:**

- Returns either a hardcoded value or a list of range values.
- Uses a try-except block to handle missing layer data.
- Issues a warning if no specific layers are defined for teachers and students.

**Dependencies/Notable Requirements:**

- `LAYERS_TO_COPY` dictionary (not shown in this snippet)
- `warnings` module for alerting system.


key: chroma:document
string_value:
CODE:

def get_layers_to_supervise(n_student, n_teacher) -> List[int]:
    """Used or the --supervise_forward kwarg"""
    if n_student > n_teacher:
        raise ValueError(f"Cannot perform intermediate supervision for student {n_student} > teacher {n_teacher}")
    elif n_teacher == n_student:
        return list(range(n_teacher))
    elif n_student == 1:
        return [n_teacher - 1]
    else:


SUMMARY:
**Summary**

The `get_layers_to_supervise` function determines the supervision layers for students and teachers in an educational setting. It calculates the supervision layers based on the number of students and teachers, considering intermediate supervision.

**Key Details**

* Primary purpose: Determine supervision layers for students and teachers.
* Key implementation detail:
	+ Uses a pre-defined dictionary `LAYERS_TO_SUPERVISE` to store the supervision layers for each teacher-student pair.
    * Method signature: `get_layers_to_supervise(n_student, n_teacher) -> List[int]`
 

**Notable Dependencies or Requirements**

* The function uses the following external references:
	+ `List`: A built-in Python type.
  * `Raises`: ValueError with a descriptive message for invalid input cases.

**Design Choices and Patterns**

The function implements a pre-computed approach using the `LAYERS_TO_SUPERVISE` dictionary to store the supervision layers, rather than dynamically calculating them. This design choice simplifies the code and improves readability.


key: chroma:document
string_value:
CODE:

def create_student_by_copying_alternating_layers(
    teacher: Union[str, PreTrainedModel],
    save_path: Union[str, Path] = "student",
    e: Union[int, None] = None,
    d: Union[int, None] = None,
    copy_first_teacher_layers=False,
    e_layers_to_copy=None,
    d_layers_to_copy=None,
    **extra_config_kwargs,
) -> Tuple[PreTrainedModel, List[int], List[int]]:
    """Make a student by copying alternating layers from a teacher, save it to save_path.
    Args:
        teacher: str or PreTrainedModel if str, this will call AutoModelForSeq2SeqLM.from_pretrained(teacher) before
        copying layers
        save_path: where to save the student, defaults to student directory.
        e: how many Encoder layers should the student have, default is fully copy of teacher
        d: how many Decoder layers should the student have, default is fully copy of teacher
        copy_first_teacher_layers: [bool] dont copy alternating layers, just the first e/d.
        **extra_config_kwargs: extra kwargs to pass to the student, by default the teacher config is used.

    Returns:
        student: new, smaller model.  (Also saves it to save_path)
        e_layers_to_copy: list of which teacher encoder layers were used
        d_layers_to_copy: list of which teacher decoder layers were used
    """
    _msg = "encoder_layers and decoder_layers cannot be both None-- you would just have an identical teacher."
    assert (e is not None) or (d is not None), _msg
    if isinstance(teacher, str):
        AutoTokenizer.from_pretrained(teacher).save_pretrained(save_path)  # purely for convenience
        teacher = AutoModelForSeq2SeqLM.from_pretrained(teacher).eval()
    else:
        assert isinstance(teacher, PreTrainedModel), f"teacher must be a model or string got type {type(teacher)}"
    init_kwargs = teacher.config.to_diff_dict()

    try:
        teacher_e, teacher_d = teacher.config.encoder_layers, teacher.config.decoder_layers
        if e is None:
            e = teacher_e
        if d is None:
            d = teacher_d
        init_kwargs.update({"encoder_layers": e, "decoder_layers": d})
    except AttributeError:  # T5
        if hasattr(teacher.config, "num_encoder_layers"):
            teacher_e, teacher_d = teacher.config.num_encoder_layers, teacher.config.num_decoder_layers
        else:
            teacher_e, teacher_d = teacher.config.num_layers, teacher.config.num_decoder_layers
        if e is None:
            e = teacher_e
        if d is None:
            d = teacher_d
        if hasattr(teacher.config, "num_encoder_layers"):
            init_kwargs.update({"num_encoder_layers": e, "num_decoder_layers": d})
        else:
            init_kwargs.update({"num_layers": e, "num_decoder_layers": d})

    # Kwargs to instantiate student: teacher kwargs with updated layer numbers + **extra_config_kwargs
    init_kwargs.update(extra_config_kwargs)

    # Copy weights
    student_cfg = teacher.config_class(**init_kwargs)
    student = AutoModelForSeq2SeqLM.from_config(student_cfg)
    # Start by copying the full teacher state dict this will copy the first N teacher layers to the student.
    info = student.load_state_dict(teacher.state_dict(), strict=False)
    assert info.missing_keys == [], info.missing_keys  # every student key should have a teacher keys.

    if copy_first_teacher_layers:  # Our copying is done. We just log and save
        e_layers_to_copy, d_layers_to_copy = list(range(e)), list(range(d))
        logger.info(
            f"Copied encoder layers {e_layers_to_copy} and decoder layers {d_layers_to_copy}. Saving them to"
            f" {save_path}"
        )
        student.save_pretrained(save_path)
        return student, e_layers_to_copy, d_layers_to_copy

    # Decide which layers of the teacher to copy. Not exactly alternating -- we try to keep first and last layer.
    if e_layers_to_copy is None:
        e_layers_to_copy: List[int] = pick_layers_to_copy(e, teacher_e)
    if d_layers_to_copy is None:
        d_layers_to_copy: List[int] = pick_layers_to_copy(d, teacher_d)

    try:
        if hasattr(
            teacher, "prophetnet"
        ):  # For ProphetNet, student.model.encoder.layers is called student.prophetnet.encoder.layers
            copy_layers(teacher.prophetnet.encoder.layers, student.prophetnet.encoder.layers, e_layers_to_copy)
            copy_layers(teacher.prophetnet.decoder.layers, student.prophetnet.decoder.layers, d_layers_to_copy)
        else:
            copy_layers(teacher.model.encoder.layers, student.model.encoder.layers, e_layers_to_copy)
            copy_layers(teacher.model.decoder.layers, student.model.decoder.layers, d_layers_to_copy)
    except AttributeError:  # For t5, student.model.encoder.layers is called student.encoder.block
        copy_layers(teacher.encoder.block, student.encoder.block, e_layers_to_copy)
        copy_layers(teacher.decoder.block, student.decoder.block, d_layers_to_copy)
    logger.info(
        f"Copied encoder layers {e_layers_to_copy} and decoder layers {d_layers_to_copy}. Saving them to {save_path}"
    )
    student.config.init_metadata = {
        "teacher_type": teacher.config.model_type,
        "copied_encoder_layers": e_layers_to_copy,
        "copied_decoder_layers": d_layers_to_copy,
    }
    student.save_pretrained(save_path)
    # Save information about copying for easier reproducibility



SUMMARY:
**Function Summary: `create_student_by_copying_alternating_layers`**

Creates a new student model by copying alternating layers from a teacher model.

**Purpose and Functionality**

The function takes in a teacher model (`PreTrainedModel`) or a string representation of the model, along with several optional configuration parameters. It creates a new student model by copying the specified number of layers from the teacher model. The function returns the newly created student model, as well as lists of the copied encoder and decoder layers.

**Key Implementation Details**

* Copies alternating layers from the teacher model to the student model.
* Optionally copies all layers of the encoder or decoder based on configuration parameters.
* Uses a custom `copy_layers` function to copy individual layers between models.

**Notable Dependencies and Requirements**

* `PreTrainedModel` class (parent class of the teacher model)
* `AutoTokenizer`, `AutoModelForSeq2SeqLM`, and related classes from the Hugging Face library
* Custom `pick_layers_to_copy` and `copy_layers` functions, not explicitly shown in this code snippet.

**Return Values**

* A new instance of `PreTrainedModel`
* Two lists of integers: `e_layers_to_copy` and `d_layers_to_copy`, representing the copied encoder and decoder layers.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/seq2seq-distillation/run_eval.py


key: chroma:document
string_value:
CODE:

def generate_summaries_or_translations(
    examples: List[str],
    out_file: str,
    model_name: str,
    batch_size: int = 8,
    device: str = DEFAULT_DEVICE,
    fp16=False,
    task="summarization",
    prefix=None,
    **generate_kwargs,
) -> Dict:
    """Save model.generate results to <out_file>, and return how long it took."""
    fout = Path(out_file).open("w", encoding="utf-8")
    model_name = str(model_name)
    model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device)
    if fp16:
        model = model.half()

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    logger.info(f"Inferred tokenizer type: {tokenizer.__class__}")  # if this is wrong, check config.model_type.

    start_time = time.time()
    # update config with task specific params
    use_task_specific_params(model, task)
    if prefix is None:
        prefix = prefix or getattr(model.config, "prefix", "") or ""
    for examples_chunk in tqdm(list(chunks(examples, batch_size))):
        examples_chunk = [prefix + text for text in examples_chunk]
        batch = tokenizer(examples_chunk, return_tensors="pt", truncation=True, padding="longest").to(device)
        summaries = model.generate(
            input_ids=batch.input_ids,
            attention_mask=batch.attention_mask,
            **generate_kwargs,
        )
        dec = tokenizer.batch_decode(summaries, skip_special_tokens=True, clean_up_tokenization_spaces=False)
        for hypothesis in dec:
            fout.write(hypothesis + "\n")
            fout.flush()
    fout.close()
    runtime = int(time.time() - start_time)  # seconds
    n_obs = len(examples)


SUMMARY:
**Summary: `generate_summaries_or_translations` Function**

**Main Purpose and Functionality**
Generate summaries from examples using a pre-trained sequence-to-sequence language model and save the results to an output file.

* Takes in input examples, output file path, model name, and other configuration parameters
* Uses the AutoModelForSeq2SeqLM and AutoTokenizer models for inference
* Generates summaries by applying the `generate` method of the model
* Saves the generated summaries to the output file and returns the runtime information

**Key Implementation Details**

* Utilizes the `chunks` function to divide input examples into batches of size `batch_size`
* Applies the `truncation=True` and `padding="longest"` parameters for tokenization
* Uses the `generate_kwargs` dictionary to pass additional configuration parameters to the `generate` method

**Dependences and Requirements**

* Uses the Hugging Face library (`transformers`) for loading pre-trained models and tokenizer
* Implies the use of external dependencies such as `time`, `Path`, and `Logger`

**Notable Method Signatures and Interfaces**

* `tokenizer(examples_chunk, return_tensors="pt", truncation=True, padding="longest")`
* `model.generate(input_ids=batch.input_ids, attention_mask=batch.attention_mask, **generate_kwargs)`
* `tokenizer.batch_decode(summaries, skip_special_tokens=True, clean_up_tokenization_spaces=False)`


key: chroma:document
string_value:
CODE:

def datetime_now():


SUMMARY:
**Summary:**

This function `datetime_now()` returns the current date and time in "YYYY-MM-DD HH:MM:SS" format.

**Key Details:**

* Implements `datetime.datetime.now()` to retrieve the current UTC datetime
* Applies `strftime` formatting to produce a human-readable date and time string

Note: The `datetime` module is a built-in Python library.


key: chroma:document
string_value:
CODE:

def run_generate(verbose=True):
    """

    Takes input text, generates output, and then using reference calculates the BLEU scores.

    The results are saved to a file and returned to the caller, and printed out unless ``verbose=False`` is passed.

    Args:
        verbose (:obj:`bool`, `optional`, defaults to :obj:`True`): print results to stdout

    Returns:
        a tuple: ``(scores, params}``
        - ``scores``: a dict of scores data ``{'bleu': 39.6501, 'n_obs': 2000, 'runtime': 186, 'seconds_per_sample': 0.093}``
        - ``params``: a dict of custom params, e.g. ``{'num_beams': 5, 'length_penalty': 0.8}``
    """

    parser = argparse.ArgumentParser()
    parser.add_argument("model_name", type=str, help="like facebook/bart-large-cnn,t5-base, etc.")
    parser.add_argument("input_path", type=str, help="like cnn_dm/test.source")
    parser.add_argument("save_path", type=str, help="where to save summaries")
    parser.add_argument("--reference_path", type=str, required=False, help="like cnn_dm/test.target")
    parser.add_argument("--score_path", type=str, required=False, default="metrics.json", help="where to save metrics")
    parser.add_argument("--device", type=str, required=False, default=DEFAULT_DEVICE, help="cuda, cuda:1, cpu etc.")
    parser.add_argument(
        "--prefix", type=str, required=False, default=None, help="will be added to the beginning of src examples"
    )
    parser.add_argument("--task", type=str, default="summarization", help="used for task_specific_params + metrics")
    parser.add_argument("--bs", type=int, default=8, required=False, help="batch size")
    parser.add_argument(
        "--n_obs", type=int, default=-1, required=False, help="How many observations. Defaults to all."
    )
    parser.add_argument("--fp16", action="store_true")
    parser.add_argument("--dump-args", action="store_true", help="print the custom hparams with the results")
    parser.add_argument(
        "--info",
        nargs="?",
        type=str,
        const=datetime_now(),
        help=(
            "use in conjunction w/ --dump-args to print with the results whatever other info you'd like, e.g."
            " lang=en-ru. If no value is passed, the current datetime string will be used."
        ),
    )
    # Unspecified args like --num_beams=2 --decoder_start_token_id=4 are passed to model.generate
    args, rest = parser.parse_known_args()
    parsed_args = parse_numeric_n_bool_cl_kwargs(rest)
    if parsed_args and verbose:
        print(f"parsed the following generate kwargs: {parsed_args}")
    with open(args.input_path) as f:
        examples = [" " + x.rstrip() if "t5" in args.model_name else x.rstrip() for x in f.readlines()]
    if args.n_obs > 0:
        examples = examples[: args.n_obs]
    Path(args.save_path).parent.mkdir(exist_ok=True)
    if args.reference_path is None and Path(args.score_path).exists():
        warnings.warn(f"score_path {args.score_path} will be overwritten unless you type ctrl-c.")
    runtime_metrics = generate_summaries_or_translations(
        examples,
        args.save_path,
        args.model_name,
        batch_size=args.bs,
        device=args.device,
        fp16=args.fp16,
        task=args.task,
        prefix=args.prefix,
        **parsed_args,
    )

    if args.reference_path is None:
        return {}

    # Compute scores
    score_fn = calculate_bleu if "translation" in args.task else calculate_rouge
    output_lns = [x.rstrip() for x in open(args.save_path).readlines()]
    reference_lns = [x.rstrip() for x in open(args.reference_path).readlines()][: len(output_lns)]
    scores: dict = score_fn(output_lns, reference_lns)
    scores.update(runtime_metrics)

    if args.dump_args:
        scores.update(parsed_args)
    if args.info:
        scores["info"] = args.info

    if verbose:
        print(scores)

    if args.score_path is not None:
        json.dump(scores, open(args.score_path, "w"))



SUMMARY:
**Summary**

This function, `run_generate`, takes input text from a file and generates output summaries. It uses a reference text to calculate BLEU scores and returns the results as a tuple.

**Purpose and Functionality**
-------------------------

* Generate output summaries from input text
* Calculate BLEU scores using a reference text

**Key Implementation Details**
-----------------------------

* Uses `generate_summaries_or_translations` function to generate summaries
	+ Passes arguments such as input text, save path, model name, batch size, and device
* Computes BLEU scores using `calculate_bleu` or `calculate_rouge` depending on the task
	+ Returns a dictionary of scores with metrics such as bleu, n_obs, runtime, and seconds per sample

**Notable Dependencies**
----------------------

* `argparse` for parsing command-line arguments
* `generate_summaries_or_translations` function (not shown in this snippet) for generating summaries
* `calculate_bleu` or `calculate_rouge` function depending on the task for computing BLEU scores


