# embedding_metadata

key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/language-modeling-tpu/prepare_tfrecord_shards.py


key: chroma:document
string_value:
CODE:

def parse_args():
    parser = argparse.ArgumentParser(
        description="Prepare TFRecord shards from pre-tokenized samples of the wikitext dataset."
    )
    parser.add_argument(
        "--dataset_name",
        type=str,
        default="wikitext",
        help="Name of the training. Explore datasets at: hf.co/datasets.",
    )
    parser.add_argument(
        "--dataset_config", type=str, default="wikitext-103-raw-v1", help="Configuration name of the dataset."
    )
    parser.add_argument(
        "--trust_remote_code",
        action="store_true",
        help=(
            "Whether to trust the execution of code from datasets/models defined on the Hub."
            " This option should only be set to `True` for repositories you trust and in which you have read the"
            " code, as it will execute code present on the Hub on your local machine."
        ),
    )
    parser.add_argument(
        "--tokenizer_name_or_path",
        type=str,
        default="sayakpaul/unigram-tokenizer-wikitext",
        help="Tokenizer identifier. Can be a local filepath or a Hub identifier.",
    )
    parser.add_argument(
        "--shard_size",
        type=int,
        default=1000,
        help="Number of entries to go in a single shard.",
    )
    parser.add_argument("--split", type=str, default="train", choices=["train", "test", "validation"])
    parser.add_argument(
        "--limit",
        default=None,
        type=int,
        help="Limit the number of shards (used for debugging).",
    )
    parser.add_argument(
        "--max_length",
        type=int,
        default=512,
        help="Maximum sequence length. For training on TPUs, it helps to have a maximum"
        " sequence length that is a multiple of 8.",
    )
    parser.add_argument(
        "--output_dir",
        default="tf-tpu",
        type=str,
        help="Output directory where the TFRecord shards will be saved. If the"
        " path is appended with `gs://` ('gs://tf-tpu', for example) then the TFRecord"
        " shards will be directly saved to a Google Cloud Storage bucket.",
    )

    args = parser.parse_args()


SUMMARY:
**Function Summary:**

The `parse_args` function is used to parse command-line arguments for preparing TFRecord shards from pre-tokenized samples of the wikitext dataset. It utilizes the `argparse` library to define and validate parameters.

**Key Implementation Details:**

*   Parses various configuration options, including `dataset_name`, `tokenizer_name_or_path`, and `output_dir`.
*   Defines several argument types (e.g., `str`, `int`) with default values.
*   Includes optional arguments for customizing the training process (e.g., `trust_remote_code`).
*   Validates user input through various constraints (e.g., `choices=["train", "test", "validation"]`).

**Notable Dependencies or Requirements:**

*   The function relies on the `argparse` library to define and parse command-line arguments.
*   It uses several external classes and functions, including:
    *   `argparse.ArgumentParser`
    *   `argparse.Argument`
    *   `argparse.Action`
    *   `argparse.parse_args`

**Inherited or Implemented Interfaces:**

*   None.

**Primary Purpose:**

The function is designed to parse command-line arguments and validate user input, providing a structured configuration for preparing TFRecord shards.


key: chroma:document
string_value:
CODE:

def tokenize_function(tokenizer):
    def fn(examples):
        return tokenizer(examples["text"])



SUMMARY:
**Concise Summary:**

`tokenize_function` takes a `tokenizer` object as input and returns a function `fn`. The returned `fn` takes a dictionary `examples` containing a `"text"` key, and applies the `tokenizer` to it. This suggests that the primary purpose of this function is to wrap a text tokenizer with a custom interface.

**Key Implementation Details:**

* The returned function `fn` delegates text processing to the input `tokenizer`.
* It returns the tokenized result from the `tokenizer` interface (`return tokenizer(examples["text"])`).

**Notable Dependencies or Requirements:**

* A valid `tokenizer` object is required for the `tokenize_function`.


key: chroma:document
string_value:
CODE:

def get_serialized_examples(tokenized_data):
    records = []
    for i in range(len(tokenized_data["input_ids"])):
        features = {
            "input_ids": tf.train.Feature(int64_list=tf.train.Int64List(value=tokenized_data["input_ids"][i])),
            "attention_mask": tf.train.Feature(
                int64_list=tf.train.Int64List(value=tokenized_data["attention_mask"][i])
            ),
        }
        features = tf.train.Features(feature=features)
        example = tf.train.Example(features=features)
        record_bytes = example.SerializeToString()
        records.append(record_bytes)


SUMMARY:
**Function Summary: `get_serialized_examples`**

**Purpose:** Serializes input IDs and attention masks into TensorFlow examples for downstream tokenization tasks.

**Functionality:**

* Iterates through a list of tokens (`input_ids`) and their corresponding attention masks.
* Creates TF features with serialized input IDs and attention masks.
* Serializes each example using `tf.train.Example` and stores the result in a list of bytes (`record_bytes`).
* Returns a list of serialized examples.

**Key Implementation Details:**

* Uses TensorFlow's `int64_list` to serialize integer values efficiently.
* Implements `tf.train.Features` to create a nested feature structure.

**Notable Dependencies/Requirements:**

* TensorFlow library (specifically, its feature and example serialization APIs)
* Tokenized data with `input_ids` and `attention_mask` keys.


key: chroma:document
string_value:
CODE:

def main(args):
    dataset = datasets.load_dataset(
        args.dataset_name, args.dataset_config, split=args.split, trust_remote_code=args.trust_remote_code
    )

    if args.limit is not None:
        max_samples = min(len(dataset), args.limit)
        dataset = dataset.select(range(max_samples))
        print(f"Limiting the dataset to {args.limit} entries.")

    tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_name_or_path)

    # Handle output directory creation.
    # For serializing into a Google Cloud Storage Bucket, one needs to first
    # create a bucket.
    if "gs" not in args.output_dir:
        if not os.path.exists(args.output_dir):
            os.makedirs(args.output_dir)
        split_dir = os.path.join(args.output_dir, args.split)
        if not os.path.exists(split_dir):
            os.makedirs(split_dir)
    else:
        split_dir = os.path.join(args.output_dir, args.split)

    # Tokenize the whole dataset at once.
    tokenize_fn = tokenize_function(tokenizer)
    dataset_tokenized = dataset.map(tokenize_fn, batched=True, num_proc=4, remove_columns=["text"])

    # We need to concatenate all our texts together, and then split the result
    # into chunks of a fixed size, which we will call block_size. To do this, we
    # will use the map method again, with the option batched=True. When we use batched=True,
    # the function we pass to map() will be passed multiple inputs at once, allowing us
    # to group them into more or fewer examples than we had in the input.
    # This allows us to create our new fixed-length samples. The advantage of this
    # method is that we don't lose a whole lot of content from the dataset compared to the
    # case where we simply tokenize with a pre-defined max_length.

    def group_texts(examples):
        # Concatenate all texts.
        concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
        total_length = len(concatenated_examples[list(examples.keys())[0]])
        # We drop the small remainder, though you could add padding instead if the model supports it
        # In this, as in all things, we advise you to follow your heart 🫀
        total_length = (total_length // args.max_length) * args.max_length
        # Split by chunks of max_len.
        result = {
            k: [t[i : i + args.max_length] for i in range(0, total_length, args.max_length)]
            for k, t in concatenated_examples.items()
        }
        return result

    grouped_dataset = dataset_tokenized.map(group_texts, batched=True, batch_size=1000, num_proc=4)

    shard_count = 0
    total_records = 0
    for shard in range(0, len(grouped_dataset), args.shard_size):
        dataset_snapshot = grouped_dataset[shard : shard + args.shard_size]
        records_containing = len(dataset_snapshot["input_ids"])
        filename = os.path.join(split_dir, f"dataset-{shard_count}-{records_containing}.tfrecord")
        serialized_examples = get_serialized_examples(dataset_snapshot)

        with tf.io.TFRecordWriter(filename) as out_file:
            for i in range(len(serialized_examples)):
                example = serialized_examples[i]
                out_file.write(example)
            print("Wrote file {} containing {} records".format(filename, records_containing))

        shard_count += 1
        total_records += records_containing

    with open(f"split-{args.split}-records-count.txt", "w") as f:


SUMMARY:
**Function Summary**

This function, `main`, appears to be part of a data processing pipeline for text datasets. It:

1. Loads a dataset from Hugging Face's Datasets API based on user input (dataset_name, split, and trust_remote_code). If a limit is specified, it limits the number of samples in the dataset.
2. Tokenizes the entire dataset using an AutoTokenizer instance and maps it to a new format.
3. Concatenates all texts into a single sequence and splits it into chunks of a fixed size (max_length).
4. Creates TFRecord files from these chunked sequences, one shard per file.

**Key Implementation Details**

* Tokenization is performed using `AutoTokenizer.from_pretrained` and `map` method with `batched=True`.
* Text concatenation and chunking is done in `group_texts` function.
* Chunked sequences are written to TFRecord files using `tf.io.TFRecordWriter`.

**Notable Dependencies and Interfaces**

* `datasets`: Hugging Face's Datasets API (load_dataset, select)
* `AutoTokenizer`: Hugging Face's AutoTokenizer instance (from_pretrained)
* `get_serialized_examples`: presumably a custom function to serialize examples into TFRecords.
* `tf.io.TFRecordWriter`: TensorFlow's TFRecord writer interface.
* `os.path.exists`, `os.makedirs`, and ` os.path.join` for directory management.

**Primary Functionality**

The primary purpose of this code is to process a text dataset by tokenizing, concatenating, and chunking the text into TFRecord files. The input dataset can be limited and processed in parallel using multiple CPU processes.


key: chroma:document
string_value:
CODE:

def parse_args():
    parser = argparse.ArgumentParser(description="Train a masked language model on TPU.")
    parser.add_argument(
        "--pretrained_model_config",
        type=str,
        default="FacebookAI/roberta-base",
        help="The model config to use. Note that we don't copy the model's weights, only the config!",
    )
    parser.add_argument(
        "--tokenizer",
        type=str,
        default="unigram-tokenizer-wikitext",
        help="The name of the tokenizer to load. We use the pretrained tokenizer to initialize the model's vocab size.",
    )

    parser.add_argument(
        "--per_replica_batch_size",
        type=int,
        default=8,
        help="Batch size per TPU core.",
    )

    parser.add_argument(
        "--no_tpu",
        action="store_true",
        help="If set, run on CPU and don't try to initialize a TPU. Useful for debugging on non-TPU instances.",
    )

    parser.add_argument(
        "--tpu_name",
        type=str,
        help="Name of TPU resource to initialize. Should be blank on Colab, and 'local' on TPU VMs.",
        default="local",
    )

    parser.add_argument(
        "--tpu_zone",
        type=str,
        help="Google cloud zone that TPU resource is located in. Only used for non-Colab TPU nodes.",
    )

    parser.add_argument(
        "--gcp_project", type=str, help="Google cloud project name. Only used for non-Colab TPU nodes."
    )

    parser.add_argument(
        "--bfloat16",
        action="store_true",
        help="Use mixed-precision bfloat16 for training. This is the recommended lower-precision format for TPU.",
    )

    parser.add_argument(
        "--train_dataset",
        type=str,
        help="Path to training dataset to load. If the path begins with `gs://`"
        " then the dataset will be loaded from a Google Cloud Storage bucket.",
    )

    parser.add_argument(
        "--shuffle_buffer_size",
        type=int,
        default=2**18,  # Default corresponds to a 1GB buffer for seq_len 512
        help="Size of the shuffle buffer (in samples)",
    )

    parser.add_argument(
        "--eval_dataset",
        type=str,
        help="Path to evaluation dataset to load. If the path begins with `gs://`"
        " then the dataset will be loaded from a Google Cloud Storage bucket.",
    )

    parser.add_argument(
        "--num_epochs",
        type=int,
        default=1,
        help="Number of epochs to train for.",
    )

    parser.add_argument(
        "--learning_rate",
        type=float,
        default=1e-4,
        help="Learning rate to use for training.",
    )

    parser.add_argument(
        "--weight_decay_rate",
        type=float,
        default=1e-3,
        help="Weight decay rate to use for training.",
    )

    parser.add_argument(
        "--max_length",
        type=int,
        default=512,
        help="Maximum length of tokenized sequences. Should match the setting used in prepare_tfrecord_shards.py",
    )

    parser.add_argument(
        "--mlm_probability",
        type=float,
        default=0.15,
        help="Fraction of tokens to mask during training.",
    )

    parser.add_argument("--output_dir", type=str, required=True, help="Path to save model checkpoints to.")
    parser.add_argument("--hub_model_id", type=str, help="Model ID to upload to on the Hugging Face Hub.")

    args = parser.parse_args()


SUMMARY:
**Function Summary:**

`parse_args` is a function that parses command-line arguments for training a masked language model using TensorFlow and TPU. It uses `argparse` to define and validate input parameters.

**Key Implementations:**

* Configures the `pretrained_model_config`, tokenizer, batch size per TPU core (`per_replica_batch_size`), and hyperparameters (e.g., learning rate, weight decay rate) using command-line arguments.
* Allows flexible usage on both CPU and TPU environments with options for debugging on non-TPU instances (`--no_tpu`) or initializing a TPU resource (`tpu_name`, `tpu_zone`, `gcp_project`).

**Notable Dependencies/Requirements:**

* TensorFlow
* `argparse`
* `tpu` (optional)
* Google Cloud Storage (for dataset loading)

**Interfaces/Exports:**

None, returns the parsed command-line arguments (`args`) as output.


key: chroma:document
string_value:
CODE:

def initialize_tpu(args):
    try:
        if args.tpu_name:
            tpu = tf.distribute.cluster_resolver.TPUClusterResolver(
                args.tpu_name, zone=args.tpu_zone, project=args.gcp_project
            )
        else:
            tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
    except ValueError:
        raise RuntimeError(
            "Couldn't connect to TPU! Most likely you need to specify --tpu_name, --tpu_zone, or "
            "--gcp_project. When running on a TPU VM, use --tpu_name local."
        )

    tf.config.experimental_connect_to_cluster(tpu)
    tf.tpu.experimental.initialize_tpu_system(tpu)



SUMMARY:
**Summary:**

`initialize_tpu(args)` Function
=====================================

**Purpose and Functionality:**
Establishes a connection to a Google Cloud TPU cluster.

**Key Implementation Details:**

* Creates a `TPUClusterResolver` instance with provided TPU name, zone, and project.
* Triggers the connection to a TPU cluster using `tf.config.experimental_connect_to_cluster`.
* Initializes the TPU system using `tf.tpu.experimental.initialize_tpu_system`.

**Notable Dependencies or Requirements:**

* `tf`: TensorFlow library for distributed computation on TPU clusters.
* `gcp_project`, `tpu_name`, and `tpu_zone` are assumed to be provided through command-line arguments.

**Return Value:**
Returns the initialized `TPUClusterResolver` instance.


key: chroma:document
string_value:
CODE:

def count_samples(file_list):
    num_samples = 0
    for file in file_list:
        filename = file.split("/")[-1]
        sample_count = re.search(r"-\d+-(\d+)\.tfrecord", filename).group(1)
        sample_count = int(sample_count)
        num_samples += sample_count



SUMMARY:
**Summary:**

The `count_samples` function takes a list of file paths as input and returns the total number of samples extracted from TFRecord files. It achieves this by:

* Splitting each file path to extract the sample count from the filename using a regular expression
* Parsing the extracted string to obtain an integer sample count
* Summing up the individual sample counts

**Key Implementation Details:**

* Uses `re.search` and `.group(1)` to extract the sample count from the format "-<count>-<id>.tfrecord"
* Performs type conversion on the sample count using `int()`

**Notable Dependencies:**

* Uses regular expression (`re`) module for string parsing
* Assumes TFRecord file format is in the specified pattern.


key: chroma:document
string_value:
CODE:

def prepare_dataset(records, decode_fn, mask_fn, batch_size, shuffle, shuffle_buffer_size=None):
    num_samples = count_samples(records)
    dataset = tf.data.Dataset.from_tensor_slices(records)
    if shuffle:
        dataset = dataset.shuffle(len(dataset))
    dataset = tf.data.TFRecordDataset(dataset, num_parallel_reads=AUTO)
    # TF can't infer the total sample count because it doesn't read all the records yet, so we assert it here
    dataset = dataset.apply(tf.data.experimental.assert_cardinality(num_samples))
    dataset = dataset.map(decode_fn, num_parallel_calls=AUTO)
    if shuffle:
        assert shuffle_buffer_size is not None
        dataset = dataset.shuffle(args.shuffle_buffer_size)
    dataset = dataset.batch(batch_size, drop_remainder=True)
    dataset = dataset.map(mask_fn, num_parallel_calls=AUTO)
    dataset = dataset.prefetch(AUTO)


SUMMARY:
**Function Summary:** `prepare_dataset`

**Purpose:**
Creates and prepares a TensorFlow dataset for training from the given records.

**Key Purpose:** Dataset creation and preprocessing with optional shuffling, batching, and masking.

### Implementation Details:

- **Dataset Creation:** Utilizes `tf.data.Dataset.from_tensor_slices(records)`.
- **Shuffling (if applicable):** Adds shuffle buffer size assertion if enabled.
- **TFRecordDataset (optional):** Set `num_parallel_reads=AUTO` for efficient data loading.
- **Decoding and Masking:**
  - Applies `decode_fn` mapping on all elements (`dataset.map(decode_fn, num_parallel_calls=AUTO)`).
  - Applies `mask_fn` mapping on batched dataset (`dataset.batch(batch_size, drop_remainder=True).map(mask_fn, num_parallel_calls=AUTO)`).

### Dependencies and Requirements:

- `tf.data`
- `count_samples` function (not shown in this snippet)
- `decode_fn` and `mask_fn` functions (not shown in this snippet)


key: chroma:document
string_value:
CODE:

def main(args):
    if not args.no_tpu:
        tpu = initialize_tpu(args)
        strategy = tf.distribute.TPUStrategy(tpu)
    else:
        strategy = tf.distribute.OneDeviceStrategy(device="/gpu:0")

    if args.bfloat16:
        keras.mixed_precision.set_global_policy("mixed_bfloat16")

    tokenizer = AutoTokenizer.from_pretrained(args.tokenizer)
    config = AutoConfig.from_pretrained(args.pretrained_model_config)
    config.vocab_size = tokenizer.vocab_size

    training_records = tf.io.gfile.glob(os.path.join(args.train_dataset, "*.tfrecord"))
    if not training_records:
        raise ValueError(f"No .tfrecord files found in {args.train_dataset}.")
    eval_records = tf.io.gfile.glob(os.path.join(args.eval_dataset, "*.tfrecord"))
    if not eval_records:
        raise ValueError(f"No .tfrecord files found in {args.eval_dataset}.")

    num_train_samples = count_samples(training_records)

    steps_per_epoch = num_train_samples // (args.per_replica_batch_size * strategy.num_replicas_in_sync)
    total_train_steps = steps_per_epoch * args.num_epochs

    with strategy.scope():
        model = TFAutoModelForMaskedLM.from_config(config)
        model(model.dummy_inputs)  # Pass some dummy inputs through the model to ensure all the weights are built
        optimizer, schedule = create_optimizer(
            num_train_steps=total_train_steps,
            num_warmup_steps=total_train_steps // 20,
            init_lr=args.learning_rate,
            weight_decay_rate=args.weight_decay_rate,
        )

        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, metrics=["accuracy"])

    def decode_fn(example):
        features = {
            "input_ids": tf.io.FixedLenFeature(dtype=tf.int64, shape=(args.max_length,)),
            "attention_mask": tf.io.FixedLenFeature(dtype=tf.int64, shape=(args.max_length,)),
        }
        return tf.io.parse_single_example(example, features)

    # Many of the data collators in Transformers are TF-compilable when return_tensors == "tf", so we can
    # use their methods in our data pipeline.
    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm_probability=args.mlm_probability, mlm=True, return_tensors="tf"
    )

    def mask_with_collator(batch):
        # TF really needs an isin() function
        special_tokens_mask = (
            ~tf.cast(batch["attention_mask"], tf.bool)
            | (batch["input_ids"] == tokenizer.cls_token_id)
            | (batch["input_ids"] == tokenizer.sep_token_id)
        )
        batch["input_ids"], batch["labels"] = data_collator.tf_mask_tokens(
            batch["input_ids"],
            vocab_size=len(tokenizer),
            mask_token_id=tokenizer.mask_token_id,
            special_tokens_mask=special_tokens_mask,
        )
        return batch

    batch_size = args.per_replica_batch_size * strategy.num_replicas_in_sync

    train_dataset = prepare_dataset(
        training_records,
        decode_fn=decode_fn,
        mask_fn=mask_with_collator,
        batch_size=batch_size,
        shuffle=True,
        shuffle_buffer_size=args.shuffle_buffer_size,
    )

    eval_dataset = prepare_dataset(
        eval_records,
        decode_fn=decode_fn,
        mask_fn=mask_with_collator,
        batch_size=batch_size,
        shuffle=False,
    )

    callbacks = []
    if args.hub_model_id:
        callbacks.append(
            PushToHubCallback(output_dir=args.output_dir, hub_model_id=args.hub_model_id, tokenizer=tokenizer)
        )

    model.fit(
        train_dataset,
        validation_data=eval_dataset,
        epochs=args.num_epochs,
        callbacks=callbacks,
    )



SUMMARY:
**Function Summary: Training Hugging Face's AutoModelForMaskedLM**

### Main Functionality

This function is the entry point for training the `AutoModelForMaskedLM` model. It takes in several command-line arguments and uses them to configure the model, tokenizer, data loading, and training parameters.

### Implementation Details

* The function initializes a `TPUStrategy` or `OneDeviceStrategy` based on whether `--no_tpu` is provided.
* It sets up the mixed precision policy for bfloat16 if specified.
* It loads the pre-trained tokenizer and config from Hugging Face's model hub.
* It defines a data decoding function (`decode_fn`) to parse TFRecord files.
* It creates a data collator using `DataCollatorForLanguageModeling` with custom masking logic.
* It trains the model on the training dataset, evaluating on the evaluation dataset after each epoch.

### Notable Dependencies and Requirements

* TensorFlow 2.x
* Hugging Face's `AutoTokenizer`, `AutoConfig`
* TFRecord files containing training and evaluation datasets
* External dependencies:
	+ `tf.distributestrategy` for parallel training
	+ `tf.io.gfile` for file I/O
	+ `os` for path manipulation

### Key Method Signatures and Interfaces

* `initilm.fit`: trains the model on the training dataset
* `model(tf.Tensor):` passes dummy inputs through the model to build weights

This summary provides a concise overview of the main functionality, key implementation details, and notable dependencies and requirements of this code chunk.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/language-modeling-tpu/train_unigram.py


key: chroma:document
string_value:
CODE:

def parse_args():
    parser = argparse.ArgumentParser(description="Train a unigram tokenizer on the wikitext dataset.")
    parser.add_argument(
        "--dataset_name",
        type=str,
        default="wikitext",
        help="Name of the training. Explore datasets at: hf.co/datasets.",
    )
    parser.add_argument(
        "--dataset_config", type=str, default="wikitext-103-raw-v1", help="Configuration name of the dataset."
    )
    parser.add_argument(
        "--trust_remote_code",
        action="store_true",
        help=(
            "Whether to trust the execution of code from datasets/models defined on the Hub."
            " This option should only be set to `True` for repositories you trust and in which you have read the"
            " code, as it will execute code present on the Hub on your local machine."
        ),
    )
    parser.add_argument(
        "--batch_size",
        type=int,
        default=1000,
        help="Batch size during training.",
    )
    parser.add_argument(
        "--vocab_size",
        type=int,
        default=10048,
        help="Size of the desired vocabulary.",
    )
    parser.add_argument(
        "--limit",
        default=None,
        type=int,
        help="Limit the number of shards (used for debugging).",
    )
    parser.add_argument(
        "--export_to_hub",
        action="store_true",
    )

    args = parser.parse_args()


SUMMARY:
**Summary**

The `parse_args` function creates an argparse parser to define command-line arguments for training a unigram tokenizer on the wikitext dataset. The main functionality is to:

* Parse input arguments using `argparse.ArgumentParser`
* Define 6 parameters:
	+ `--dataset_name`: dataset name (defaults to "wikitext")
	+ `--dataset_config`: configuration name of the dataset (defaults to "wikitext-103-raw-v1")
	+ `--trust_remote_code`: trust the execution of code from datasets/models on the Hub (bool)
	+ `--batch_size`: batch size during training (defaults to 1000)
	+ `--vocab_size`: size of the desired vocabulary (defaults to 10048)
	+ `--limit`: limit the number of shards for debugging (optional, int)
	+ `--export_to_hub`: export the trained model to the Hub (bool)

**Key Details**

* The parser is created with a description and default values for the parameters.
* The `store_true` action is used for boolean flags (`--trust_remote_code` and `--export_to_hub`).
* The `default=None` value allows an optional integer argument for the `--limit` parameter.

**Dependencies/Requirements**

* The `argparse` library is required.


key: chroma:document
string_value:
CODE:

def main(args):
    dataset = datasets.load_dataset(
        args.dataset_name, args.dataset_config, split="train", trust_remote_code=args.trust_remote_code
    )

    if args.limit is not None:
        max_train_samples = min(len(dataset), args.limit)
        dataset = dataset.select(range(max_train_samples))
        logger.info(f"Limiting the dataset to {args.limit} entries.")

    def batch_iterator():
        for i in range(0, len(dataset), args.batch_size):
            yield dataset[i : i + args.batch_size]["text"]

    # Prepare the tokenizer.
    tokenizer = Tokenizer(Unigram())
    tokenizer.normalizer = normalizers.Sequence([normalizers.Replace("``", '"'), normalizers.Replace("''", '"')])
    tokenizer.pre_tokenizer = pre_tokenizers.Metaspace()

    # Prepare the trainer.
    trainer = UnigramTrainer(
        unk_token="<unk>",
        special_tokens=["[CLS]", "[SEP]", "<unk>", "<pad>", "[MASK]"],
        vocab_size=args.vocab_size,
    )

    logger.info("Training the tokenizer.")
    tokenizer.train_from_iterator(batch_iterator(), trainer=trainer)
    logger.info("Tokenizer training complete!")

    cls_token_id = tokenizer.token_to_id("[CLS]")
    sep_token_id = tokenizer.token_to_id("[SEP]")
    tokenizer.post_processor = processors.TemplateProcessing(
        single="[CLS]:0 $A:0 [SEP]:0",
        pair="[CLS]:0 $A:0 [SEP]:0 $B:1 [SEP]:1",
        special_tokens=[
            ("[CLS]", cls_token_id),
            ("[SEP]", sep_token_id),
        ],
    )
    tokenizer.decoder = decoders.Metaspace()

    if args.export_to_hub:
        logger.info("Exporting the trained tokenizer to Hub.")
        new_tokenizer = AlbertTokenizerFast(tokenizer_object=tokenizer)


SUMMARY:
**Concise Summary**

### Main Purpose

This function trains and modifies a BERT-based model, specifically the Unigram tokenizer, for natural language processing tasks. The primary purpose is to fine-tune the model on a pre-defined dataset with user-specified parameters.

### Key Implementation Details

* Trains the UnigramTokenizer using `train_from_iterator` method, feeding in batches generated by the `batch_iterator`.
* Modifies the tokenizer's configuration, including token normalization and preprocessing.
* Defines custom processing templates for class prediction and pair-based classification.
* Exports the trained model to a Hub repository if specified.

### Notable Dependencies

* `datasets` library (for loading dataset)
* `Vectorizer()` from `transformers` library
* UnigramTrainer and Tokenizer classes
* normalizers, pre_tokenizers, Metaspace, and TemplateProcessing modules from transformers library


key: chroma:document
string_value:
CODE:

def crop_to_square(image):
    height, width = tf.shape(image)[0], tf.shape(image)[1]
    if height > width:
        image = tf.image.crop_to_bounding_box(image, (height - width) // 2, 0, width, width)
    elif width > height:
        image = tf.image.crop_to_bounding_box(image, 0, (width - height) // 2, height, height)


SUMMARY:
**Summary:**

This code defines a `crop_to_square` function that takes an image as input and adjusts its bounding box to become a square. The primary purpose is to ensure the aspect ratio of the image remains unchanged.

**Key Implementation Details:**

The cropping algorithm works dynamically based on the relative size of the image height and width, adjusting the cropping rectangle accordingly.

**Notable Dependencies/Requirements:**

* TensorFlow (TF) library for computing `tf.shape(image)` and applying image transformations (`tf.image.crop_to_bounding_box`).
 
**Method Signature:**

`def crop_to_square(image: [image]) -> image`

Note: The type hint for the function is missing, assuming it accepts a single image tensor as input.


key: chroma:document
string_value:
CODE:

def load_as_tf_dataset(dataset, image_column, image_size, mean, std, batch_size, shuffle):
    dataset = dataset.with_format("tensorflow")[:]  # Load the dataset as tensor slices, but not the images yet!
    tf_dataset = tf.data.Dataset.from_tensor_slices(dataset)

    def load_image(sample):
        image_path = sample[image_column]
        image = tf.io.read_file(image_path)
        image = tf.image.decode_image(image, channels=3, expand_animations=False)
        image = crop_to_square(image)
        image = tf.image.resize(image, [image_size, image_size], method="bicubic", antialias=True)
        image = image / 255.0
        image = (image - mean) / std
        image = tf.transpose(image, perm=[2, 0, 1])  # Convert to channels-first
        sample["pixel_values"] = image
        del sample[image_column]
        return sample

    if shuffle:
        tf_dataset = tf_dataset.shuffle(len(tf_dataset))
    tf_dataset = tf_dataset.map(load_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)
    tf_dataset = tf_dataset.batch(batch_size, drop_remainder=shuffle)
    tf_dataset = tf_dataset.prefetch(tf.data.experimental.AUTOTUNE)



SUMMARY:
**Function Summary:**

`load_as_tf_dataset`

Loads a dataset as TensorFlow Datasets and preprocesses images for use with TensorFlow.

**Primary Purpose:** Load data from TFRecords format into a TensorFlow Dataset, applying image preprocessing steps.

**Key Implementation Details:**

- Loads dataset in "tensorflow" format using `with_format("tensorflow")[:]`.
- Maps input samples to preprocessed images with:
  - Image reading and decoding
  - Resizing using bicubic interpolation
  - Normalization (mean, standard deviation)
  - Data type conversion

**Notable Dependencies:**

* TensorFlow (`tf`, `tf.data`, `tf.io`)

**External Dependencies:**
- `crop_to_square` (not shown in this code snippet)


key: chroma:document
string_value:
CODE:

def main():
    # 1. Parse input arguments
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    if model_args.model_name_or_path is not None:
        if model_args.vision_model_name_or_path is not None or model_args.text_model_name_or_path is not None:
            raise ValueError(
                "If using model_name_or_path, you cannot specify separate image/text model paths as well!"
            )

    if model_args.vision_model_name_or_path is not None or model_args.text_model_name_or_path is not None:
        if model_args.model_name_or_path is not None:
            raise ValueError(
                "If using separate image/text model paths, you cannot specify model_name_or_path as well!"
            )
        if not (model_args.vision_model_name_or_path is not None and model_args.text_model_name_or_path is not None):
            raise ValueError(
                "If using separate image/text model paths, you must specify both vision_model_name_or_path "
                "and text_model_name_or_path!"
            )

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/TensorFlow versions.
    send_example_telemetry("run_clip", model_args, data_args, framework="tensorflow")

    # 2. Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )

    # The default of training_args.log_level is passive, so we set log level at info here to have that default.
    transformers.utils.logging.set_verbosity_info()

    log_level = training_args.get_process_log_level()
    logger.setLevel(log_level)
    transformers.utils.logging.set_verbosity(log_level)
    transformers.utils.logging.enable_default_handler()
    transformers.utils.logging.enable_explicit_format()

    # Log on each process the small summary:
    logger.info(f"Training/evaluation parameters {training_args}")

    # 3. Detecting last checkpoint and eventually continue from last checkpoint
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    # 4. Load dataset
    # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files this script will use the first column for the full image path and the second column for the
    # captions (unless you specify column names for this with the `image_column` and `caption_column` arguments).
    #
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        dataset = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            keep_in_memory=False,
            data_dir=data_args.data_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if data_args.test_file is not None:
            data_files["test"] = data_args.test_file
            extension = data_args.test_file.split(".")[-1]
        dataset = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
        )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    # 5. Load pretrained model, tokenizer, and image processor
    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.tokenizer_name,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    elif model_args.text_model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.text_model_name_or_path,
            cache_dir=model_args.cache_dir,
            use_fast=model_args.use_fast_tokenizer,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script. "
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )

    if model_args.model_name_or_path:
        # Load image_processor, in this script we only use this to get the mean and std for normalization.
        image_processor = AutoImageProcessor.from_pretrained(
            model_args.image_processor_name or model_args.model_name_or_path,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
        with training_args.strategy.scope():
            model = TFAutoModel.from_pretrained(
                model_args.model_name_or_path,
                cache_dir=model_args.cache_dir,
                revision=model_args.model_revision,
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
    else:
        # Load image_processor, in this script we only use this to get the mean and std for normalization.
        image_processor = AutoImageProcessor.from_pretrained(
            model_args.image_processor_name or model_args.vision_model_name_or_path,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
        with training_args.strategy.scope():
            model = TFVisionTextDualEncoderModel.from_vision_text_pretrained(
                vision_model_name_or_path=model_args.vision_model_name_or_path,
                text_model_name_or_path=model_args.text_model_name_or_path,
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
    config = model.config

    if model_args.freeze_vision_model:
        model.vision_model.trainable = False

    if model_args.freeze_text_model:
        model.text_model.trainable = False

    # Preprocessing the datasets.
    # We need to tokenize inputs and targets.
    if training_args.do_train:
        column_names = dataset["train"].column_names
    elif training_args.do_eval:
        column_names = dataset["validation"].column_names
    elif training_args.do_predict:
        column_names = dataset["test"].column_names
    else:
        logger.info("There is nothing to do. Please pass `do_train`, `do_eval` and/or `do_predict`.")
        return

    # 6. Get the column names for input/target.
    dataset_columns = dataset_name_mapping.get(data_args.dataset_name, None)
    if data_args.image_column is None:
        image_column = dataset_columns[0] if dataset_columns is not None else column_names[0]
    else:
        image_column = data_args.image_column
        if image_column not in column_names:
            raise ValueError(
                f"--image_column' value '{data_args.image_column}' needs to be one of: {', '.join(column_names)}"
            )
    if data_args.caption_column is None:
        caption_column = dataset_columns[1] if dataset_columns is not None else column_names[1]
    else:
        caption_column = data_args.caption_column
        if caption_column not in column_names:
            raise ValueError(
                f"--caption_column' value '{data_args.caption_column}' needs to be one of: {', '.join(column_names)}"
            )

    # # 7. Preprocessing the datasets.

    # We need to tokenize input captions and transform the images.
    def tokenize_captions(examples):
        captions = list(examples[caption_column])
        text_inputs = tokenizer(captions, max_length=data_args.max_seq_length, padding="max_length", truncation=True)
        examples["input_ids"] = text_inputs.input_ids
        examples["attention_mask"] = text_inputs.attention_mask
        return examples

    def filter_corrupt_images(examples):
        """remove problematic images"""
        valid_images = []
        for image_file in examples[image_column]:
            try:
                Image.open(image_file)
                valid_images.append(True)
            except Exception:
                valid_images.append(False)
        return valid_images

    if training_args.do_train:
        if "train" not in dataset:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = dataset["train"]
        if data_args.max_train_samples is not None:
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))

        train_dataset = train_dataset.filter(
            filter_corrupt_images, batched=True, num_proc=data_args.preprocessing_num_workers
        )
        train_dataset = train_dataset.map(
            function=tokenize_captions,
            batched=True,
            remove_columns=[col for col in column_names if col != image_column],
            num_proc=data_args.preprocessing_num_workers,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on train dataset",
        )

        tf_train_dataset = load_as_tf_dataset(
            dataset=train_dataset,
            batch_size=training_args.per_device_train_batch_size,
            image_column=image_column,
            image_size=config.vision_config.image_size,
            mean=image_processor.image_mean,
            std=image_processor.image_std,
            shuffle=True,
        )

    if training_args.do_eval:
        if "validation" not in dataset:
            raise ValueError("--do_eval requires a train validation")
        eval_dataset = dataset["validation"]
        if data_args.max_eval_samples is not None:
            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
            eval_dataset = eval_dataset.select(range(max_eval_samples))

        eval_dataset = eval_dataset.filter(
            filter_corrupt_images, batched=True, num_proc=data_args.preprocessing_num_workers
        )
        eval_dataset = eval_dataset.map(
            function=tokenize_captions,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=[col for col in column_names if col != image_column],
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on validation dataset",
        )

        tf_eval_dataset = load_as_tf_dataset(
            dataset=eval_dataset,
            batch_size=training_args.per_device_eval_batch_size,
            image_column=image_column,
            image_size=config.vision_config.image_size,
            mean=image_processor.image_mean,
            std=image_processor.image_std,
            shuffle=False,
        )

    # 8. Preparing push_to_hub and model card
    push_to_hub_model_id = training_args.push_to_hub_model_id
    if model_args.model_name_or_path is not None:
        model_name = model_args.model_name_or_path.split("/")[-1]
    else:
        vision_name = model_args.vision_model_name_or_path.split("/")[-1]
        text_name = model_args.text_model_name_or_path.split("/")[-1]
        model_name = f"{vision_name}-{text_name}"
    if not push_to_hub_model_id:
        if data_args.dataset_name is not None:
            push_to_hub_model_id = f"{model_name}-finetuned-{data_args.dataset_name}"
        else:
            push_to_hub_model_id = f"{model_name}-finetuned-contrastive-image-text-modeling"

    model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "contrastive-image-text-modeling"}
    if data_args.dataset_name is not None:
        model_card_kwargs["dataset_tags"] = data_args.dataset_name
        if data_args.dataset_config_name is not None:
            model_card_kwargs["dataset_args"] = data_args.dataset_config_name
            model_card_kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
        else:
            model_card_kwargs["dataset"] = data_args.dataset_name

    if training_args.push_to_hub:
        callbacks = [
            PushToHubCallback(
                output_dir=training_args.output_dir,
                hub_model_id=push_to_hub_model_id,
                hub_token=training_args.push_to_hub_token,
                tokenizer=tokenizer,
                **model_card_kwargs,
            )
        ]
    else:
        callbacks = []

    # # 9. Training
    if training_args.do_train:
        num_train_steps = int(len(tf_train_dataset) * int(training_args.num_train_epochs))
        if training_args.warmup_steps > 0:
            num_warmup_steps = training_args.warmup_steps
        elif training_args.warmup_ratio > 0:
            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
        else:
            num_warmup_steps = 0
        optimizer, lr_schedule = create_optimizer(
            init_lr=training_args.learning_rate,
            num_train_steps=num_train_steps,
            num_warmup_steps=num_warmup_steps,
            adam_beta1=training_args.adam_beta1,
            adam_beta2=training_args.adam_beta2,
            adam_epsilon=training_args.adam_epsilon,
            weight_decay_rate=training_args.weight_decay,
            adam_global_clipnorm=training_args.max_grad_norm,
        )
        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, jit_compile=training_args.xla)

        if not training_args.do_eval:
            tf_eval_dataset = None
        model.fit(
            tf_train_dataset,
            validation_data=tf_eval_dataset,
            epochs=int(training_args.num_train_epochs),
            callbacks=callbacks,
        )

    # # 10. Evaluation

    if training_args.do_eval and not training_args.do_train:


SUMMARY:
This is a Python script for training a Contrastive-Image-Text modeling (CITM) model using the Hugging Face Transformers library. The script performs the following tasks:

1. Loads and configures the dataset, including downloading and preprocessing images.
2. Defines the model architecture, including the CITM model and its corresponding optimizer.
3. Compiles the model with a custom loss function.
4. Trains the model using Hugging Face's Trainer class, which handles loading and validation datasets, training, and evaluation.
5. Evaluates the trained model on a test dataset.

The script also includes several optional features:

* Pushing the trained model to a GitHub repository, where it can be accessed by other users.
* Creating a model card with metadata about the CITM model, including its author, tasks, and dataset schema.
* Tracking metrics during training using Hugging Face's Trainer callbacks.


key: chroma:document
string_value:
CODE:

def main():
    # region Argument Parsing
    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_ner", model_args, data_args, framework="tensorflow")
    # endregion

    # region Setup logging
    # we only want one process per machine to log things on the screen.
    # accelerator.is_local_main_process is only True for one process per machine.
    logger.setLevel(logging.INFO)
    datasets.utils.logging.set_verbosity_warning()
    transformers.utils.logging.set_verbosity_info()

    # If passed along, set the training seed now.
    if training_args.seed is not None:
        set_seed(training_args.seed)
    # endregion

    # region Loading datasets
    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets for token classification task available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'tokens' or the first column if no column called
    # 'tokens' is found. You can easily tweak this behavior (see below).
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        raw_datasets = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        raw_datasets = load_dataset(
            extension,
            data_files=data_files,
            token=model_args.token,
        )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    if raw_datasets["train"] is not None:
        column_names = raw_datasets["train"].column_names
        features = raw_datasets["train"].features
    else:
        column_names = raw_datasets["validation"].column_names
        features = raw_datasets["validation"].features

    if data_args.text_column_name is not None:
        text_column_name = data_args.text_column_name
    elif "tokens" in column_names:
        text_column_name = "tokens"
    else:
        text_column_name = column_names[0]

    if data_args.label_column_name is not None:
        label_column_name = data_args.label_column_name
    elif f"{data_args.task_name}_tags" in column_names:
        label_column_name = f"{data_args.task_name}_tags"
    else:
        label_column_name = column_names[1]

    # In the event the labels are not a `Sequence[ClassLabel]`, we will need to go through the dataset to get the
    # unique labels.
    def get_label_list(labels):
        unique_labels = set()
        for label in labels:
            unique_labels = unique_labels | set(label)
        label_list = list(unique_labels)
        label_list.sort()
        return label_list

    if isinstance(features[label_column_name].feature, ClassLabel):
        label_list = features[label_column_name].feature.names
        # No need to convert the labels since they are already ints.
        label_to_id = {i: i for i in range(len(label_list))}
    else:
        label_list = get_label_list(raw_datasets["train"][label_column_name])
        label_to_id = {l: i for i, l in enumerate(label_list)}
    num_labels = len(label_list)
    # endregion

    # region Load config and tokenizer
    #
    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    if model_args.config_name:
        config = AutoConfig.from_pretrained(
            model_args.config_name,
            num_labels=num_labels,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    elif model_args.model_name_or_path:
        config = AutoConfig.from_pretrained(
            model_args.model_name_or_path,
            num_labels=num_labels,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        config = CONFIG_MAPPING[model_args.model_type]()
        logger.warning("You are instantiating a new config instance from scratch.")

    tokenizer_name_or_path = model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path
    if not tokenizer_name_or_path:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script. "
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )

    if config.model_type in {"gpt2", "roberta"}:
        tokenizer = AutoTokenizer.from_pretrained(
            tokenizer_name_or_path,
            use_fast=True,
            add_prefix_space=True,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        tokenizer = AutoTokenizer.from_pretrained(
            tokenizer_name_or_path,
            use_fast=True,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    # endregion

    # region Preprocessing the raw datasets
    # First we tokenize all the texts.
    padding = "max_length" if data_args.pad_to_max_length else False

    # Tokenize all texts and align the labels with them.

    def tokenize_and_align_labels(examples):
        tokenized_inputs = tokenizer(
            examples[text_column_name],
            max_length=data_args.max_length,
            padding=padding,
            truncation=True,
            # We use this argument because the texts in our dataset are lists of words (with a label for each word).
            is_split_into_words=True,
        )

        labels = []
        for i, label in enumerate(examples[label_column_name]):
            word_ids = tokenized_inputs.word_ids(batch_index=i)
            previous_word_idx = None
            label_ids = []
            for word_idx in word_ids:
                # Special tokens have a word id that is None. We set the label to -100 so they are automatically
                # ignored in the loss function.
                if word_idx is None:
                    label_ids.append(-100)
                # We set the label for the first token of each word.
                elif word_idx != previous_word_idx:
                    label_ids.append(label_to_id[label[word_idx]])
                # For the other tokens in a word, we set the label to either the current label or -100, depending on
                # the label_all_tokens flag.
                else:
                    label_ids.append(label_to_id[label[word_idx]] if data_args.label_all_tokens else -100)
                previous_word_idx = word_idx

            labels.append(label_ids)
        tokenized_inputs["labels"] = labels
        return tokenized_inputs

    processed_raw_datasets = raw_datasets.map(
        tokenize_and_align_labels,
        batched=True,
        remove_columns=raw_datasets["train"].column_names,
        desc="Running tokenizer on dataset",
    )

    train_dataset = processed_raw_datasets["train"]
    eval_dataset = processed_raw_datasets["validation"]

    if data_args.max_train_samples is not None:
        max_train_samples = min(len(train_dataset), data_args.max_train_samples)
        train_dataset = train_dataset.select(range(max_train_samples))

    if data_args.max_eval_samples is not None:
        max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
        eval_dataset = eval_dataset.select(range(max_eval_samples))

    # Log a few random samples from the training set:
    for index in random.sample(range(len(train_dataset)), 3):
        logger.info(f"Sample {index} of the training set: {train_dataset[index]}.")
    # endregion

    with training_args.strategy.scope():
        # region Initialize model
        if model_args.model_name_or_path:
            model = TFAutoModelForTokenClassification.from_pretrained(
                model_args.model_name_or_path,
                config=config,
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
        else:
            logger.info("Training new model from scratch")
            model = TFAutoModelForTokenClassification.from_config(
                config, token=model_args.token, trust_remote_code=model_args.trust_remote_code
            )

        # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch
        # on a small vocab and want a smaller embedding size, remove this test.
        embeddings = model.get_input_embeddings()

        # Matt: This is a temporary workaround as we transition our models to exclusively using Keras embeddings.
        #       As soon as the transition is complete, all embeddings should be keras.Embeddings layers, and
        #       the weights will always be in embeddings.embeddings.
        if hasattr(embeddings, "embeddings"):
            embedding_size = embeddings.embeddings.shape[0]
        else:
            embedding_size = embeddings.weight.shape[0]
        if len(tokenizer) > embedding_size:
            model.resize_token_embeddings(len(tokenizer))
        # endregion

        # region Create TF datasets

        # We need the DataCollatorForTokenClassification here, as we need to correctly pad labels as
        # well as inputs.
        collate_fn = DataCollatorForTokenClassification(tokenizer=tokenizer, return_tensors="np")
        num_replicas = training_args.strategy.num_replicas_in_sync
        total_train_batch_size = training_args.per_device_train_batch_size * num_replicas

        dataset_options = tf.data.Options()
        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF

        # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
        # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
        # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
        # yourself if you use this method, whereas they are automatically inferred from the model input names when
        # using model.prepare_tf_dataset()
        # For more info see the docs:
        # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
        # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset

        tf_train_dataset = model.prepare_tf_dataset(
            train_dataset,
            collate_fn=collate_fn,
            batch_size=total_train_batch_size,
            shuffle=True,
        ).with_options(dataset_options)
        total_eval_batch_size = training_args.per_device_eval_batch_size * num_replicas
        tf_eval_dataset = model.prepare_tf_dataset(
            eval_dataset,
            collate_fn=collate_fn,
            batch_size=total_eval_batch_size,
            shuffle=False,
        ).with_options(dataset_options)

        # endregion

        # region Optimizer, loss and compilation
        num_train_steps = int(len(tf_train_dataset) * training_args.num_train_epochs)
        if training_args.warmup_steps > 0:
            num_warmup_steps = training_args.warmup_steps
        elif training_args.warmup_ratio > 0:
            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
        else:
            num_warmup_steps = 0

        optimizer, lr_schedule = create_optimizer(
            init_lr=training_args.learning_rate,
            num_train_steps=num_train_steps,
            num_warmup_steps=num_warmup_steps,
            adam_beta1=training_args.adam_beta1,
            adam_beta2=training_args.adam_beta2,
            adam_epsilon=training_args.adam_epsilon,
            weight_decay_rate=training_args.weight_decay,
            adam_global_clipnorm=training_args.max_grad_norm,
        )
        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, jit_compile=training_args.xla)
        # endregion

        # Metrics
        metric = evaluate.load("seqeval", cache_dir=model_args.cache_dir)

        def get_labels(y_pred, y_true):
            # Transform predictions and references tensos to numpy arrays

            # Remove ignored index (special tokens)
            true_predictions = [
                [label_list[p] for (p, l) in zip(pred, gold_label) if l != -100]
                for pred, gold_label in zip(y_pred, y_true)
            ]
            true_labels = [
                [label_list[l] for (p, l) in zip(pred, gold_label) if l != -100]
                for pred, gold_label in zip(y_pred, y_true)
            ]
            return true_predictions, true_labels

        def compute_metrics():
            results = metric.compute()
            if data_args.return_entity_level_metrics:
                # Unpack nested dictionaries
                final_results = {}
                for key, value in results.items():
                    if isinstance(value, dict):
                        for n, v in value.items():
                            final_results[f"{key}_{n}"] = v
                    else:
                        final_results[key] = value
                return final_results
            else:
                return {
                    "precision": results["overall_precision"],
                    "recall": results["overall_recall"],
                    "f1": results["overall_f1"],
                    "accuracy": results["overall_accuracy"],
                }

        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            if data_args.dataset_name is not None:
                push_to_hub_model_id = f"{model_name}-finetuned-{data_args.dataset_name}"
            else:
                push_to_hub_model_id = f"{model_name}-finetuned-token-classification"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "token-classification"}
        if data_args.dataset_name is not None:
            model_card_kwargs["dataset_tags"] = data_args.dataset_name
            if data_args.dataset_config_name is not None:
                model_card_kwargs["dataset_args"] = data_args.dataset_config_name
                model_card_kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
            else:
                model_card_kwargs["dataset"] = data_args.dataset_name

        if training_args.push_to_hub:
            callbacks = [
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            ]
        else:
            callbacks = []
        # endregion

        # region Training
        logger.info("***** Running training *****")
        logger.info(f"  Num examples = {len(train_dataset)}")
        logger.info(f"  Num Epochs = {training_args.num_train_epochs}")
        logger.info(f"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}")
        logger.info(f"  Total train batch size = {total_train_batch_size}")
        # Only show the progress bar once on each machine.

        model.fit(
            tf_train_dataset,
            validation_data=tf_eval_dataset,
            epochs=int(training_args.num_train_epochs),
            callbacks=callbacks,
        )
        # endregion

        # region Predictions
        # If you have variable batch sizes (i.e. not using pad_to_max_length), then
        # this bit might fail on TF < 2.8 because TF can't concatenate outputs of varying seq
        # length from predict().

        try:
            predictions = model.predict(tf_eval_dataset, batch_size=training_args.per_device_eval_batch_size)["logits"]
        except tf.python.framework.errors_impl.InvalidArgumentError:
            raise ValueError(
                "Concatenating predictions failed! If your version of TensorFlow is 2.8.0 or older "
                "then you will need to use --pad_to_max_length to generate predictions, as older "
                "versions of TensorFlow cannot concatenate variable-length predictions as RaggedTensor."
            )
        if isinstance(predictions, tf.RaggedTensor):
            predictions = predictions.to_tensor(default_value=-100)
        predictions = tf.math.argmax(predictions, axis=-1).numpy()
        if "label" in eval_dataset:
            labels = eval_dataset.with_format("tf")["label"]
        else:
            labels = eval_dataset.with_format("tf")["labels"]
        if isinstance(labels, tf.RaggedTensor):
            labels = labels.to_tensor(default_value=-100)
        labels = labels.numpy()
        attention_mask = eval_dataset.with_format("tf")["attention_mask"]
        if isinstance(attention_mask, tf.RaggedTensor):
            attention_mask = attention_mask.to_tensor(default_value=-100)
        attention_mask = attention_mask.numpy()
        labels[attention_mask == 0] = -100
        preds, refs = get_labels(predictions, labels)
        metric.add_batch(
            predictions=preds,
            references=refs,
        )
        eval_metric = compute_metrics()
        logger.info("Evaluation metrics:")
        for key, val in eval_metric.items():
            logger.info(f"{key}: {val:.4f}")

        if training_args.output_dir is not None:
            output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
            with open(output_eval_file, "w") as writer:
                writer.write(json.dumps(eval_metric))
        # endregion

    if training_args.output_dir is not None and not training_args.push_to_hub:
        # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
This is a Python script for fine-tuning a pre-trained language model for token classification tasks. Here's a concise summary:

**Script Overview**

The script fine-tunes a pre-trained language model on a specific dataset, performs evaluation and prediction, and saves the model.

**Key Features**

1. **Fine-tuning**: Fine-tunes a pre-trained model on a specified dataset.
2. **Evaluation**: Evaluates the model's performance using metrics such as precision, recall, F1 score, and accuracy.
3. **Prediction**: Uses the model to make predictions on a test set.
4. **Saving**: Saves the model to a local directory for later use.

**Notes**

* The script requires pre-trained language models and dataset files.
* It assumes that the dataset is in TFRecord format.
* The output directory for saving the model and evaluation results can be specified by the user.
* If the output directory is not specified, the model will still be saved locally.


key: chroma:document
string_value:
CODE:

def postprocess_qa_predictions(
    examples,
    features,
    predictions: Tuple[np.ndarray, np.ndarray],
    version_2_with_negative: bool = False,
    n_best_size: int = 20,
    max_answer_length: int = 30,
    null_score_diff_threshold: float = 0.0,
    output_dir: Optional[str] = None,
    prefix: Optional[str] = None,
    log_level: Optional[int] = logging.WARNING,
):
    """
    Post-processes the predictions of a question-answering model to convert them to answers that are substrings of the
    original contexts. This is the base postprocessing functions for models that only return start and end logits.

    Args:
        examples: The non-preprocessed dataset (see the main script for more information).
        features: The processed dataset (see the main script for more information).
        predictions (:obj:`Tuple[np.ndarray, np.ndarray]`):
            The predictions of the model: two arrays containing the start logits and the end logits respectively. Its
            first dimension must match the number of elements of :obj:`features`.
        version_2_with_negative (:obj:`bool`, `optional`, defaults to :obj:`False`):
            Whether or not the underlying dataset contains examples with no answers.
        n_best_size (:obj:`int`, `optional`, defaults to 20):
            The total number of n-best predictions to generate when looking for an answer.
        max_answer_length (:obj:`int`, `optional`, defaults to 30):
            The maximum length of an answer that can be generated. This is needed because the start and end predictions
            are not conditioned on one another.
        null_score_diff_threshold (:obj:`float`, `optional`, defaults to 0):
            The threshold used to select the null answer: if the best answer has a score that is less than the score of
            the null answer minus this threshold, the null answer is selected for this example (note that the score of
            the null answer for an example giving several features is the minimum of the scores for the null answer on
            each feature: all features must be aligned on the fact they `want` to predict a null answer).

            Only useful when :obj:`version_2_with_negative` is :obj:`True`.
        output_dir (:obj:`str`, `optional`):
            If provided, the dictionaries of predictions, n_best predictions (with their scores and logits) and, if
            :obj:`version_2_with_negative=True`, the dictionary of the scores differences between best and null
            answers, are saved in `output_dir`.
        prefix (:obj:`str`, `optional`):
            If provided, the dictionaries mentioned above are saved with `prefix` added to their names.
        log_level (:obj:`int`, `optional`, defaults to ``logging.WARNING``):
            ``logging`` log level (e.g., ``logging.WARNING``)
    """
    if len(predictions) != 2:
        raise ValueError("`predictions` should be a tuple with two elements (start_logits, end_logits).")
    all_start_logits, all_end_logits = predictions

    if len(predictions[0]) != len(features):
        raise ValueError(f"Got {len(predictions[0])} predictions and {len(features)} features.")

    # Build a map example to its corresponding features.
    example_id_to_index = {k: i for i, k in enumerate(examples["id"])}
    features_per_example = collections.defaultdict(list)
    for i, feature in enumerate(features):
        features_per_example[example_id_to_index[feature["example_id"]]].append(i)

    # The dictionaries we have to fill.
    all_predictions = collections.OrderedDict()
    all_nbest_json = collections.OrderedDict()
    if version_2_with_negative:
        scores_diff_json = collections.OrderedDict()

    # Logging.
    logger.setLevel(log_level)
    logger.info(f"Post-processing {len(examples)} example predictions split into {len(features)} features.")

    # Let's loop over all the examples!
    for example_index, example in enumerate(tqdm(examples)):
        # Those are the indices of the features associated to the current example.
        feature_indices = features_per_example[example_index]

        min_null_prediction = None
        prelim_predictions = []

        # Looping through all the features associated to the current example.
        for feature_index in feature_indices:
            # We grab the predictions of the model for this feature.
            start_logits = all_start_logits[feature_index]
            end_logits = all_end_logits[feature_index]
            # This is what will allow us to map some the positions in our logits to span of texts in the original
            # context.
            offset_mapping = features[feature_index]["offset_mapping"]
            # Optional `token_is_max_context`, if provided we will remove answers that do not have the maximum context
            # available in the current feature.
            token_is_max_context = features[feature_index].get("token_is_max_context", None)

            # Update minimum null prediction.
            feature_null_score = start_logits[0] + end_logits[0]
            if min_null_prediction is None or min_null_prediction["score"] > feature_null_score:
                min_null_prediction = {
                    "offsets": (0, 0),
                    "score": feature_null_score,
                    "start_logit": start_logits[0],
                    "end_logit": end_logits[0],
                }

            # Go through all possibilities for the `n_best_size` greater start and end logits.
            start_indexes = np.argsort(start_logits)[-1 : -n_best_size - 1 : -1].tolist()
            end_indexes = np.argsort(end_logits)[-1 : -n_best_size - 1 : -1].tolist()
            for start_index in start_indexes:
                for end_index in end_indexes:
                    # Don't consider out-of-scope answers, either because the indices are out of bounds or correspond
                    # to part of the input_ids that are not in the context.
                    if (
                        start_index >= len(offset_mapping)
                        or end_index >= len(offset_mapping)
                        or offset_mapping[start_index] is None
                        or len(offset_mapping[start_index]) < 2
                        or offset_mapping[end_index] is None
                        or len(offset_mapping[end_index]) < 2
                    ):
                        continue
                    # Don't consider answers with a length that is either < 0 or > max_answer_length.
                    if end_index < start_index or end_index - start_index + 1 > max_answer_length:
                        continue
                    # Don't consider answer that don't have the maximum context available (if such information is
                    # provided).
                    if token_is_max_context is not None and not token_is_max_context.get(str(start_index), False):
                        continue

                    prelim_predictions.append(
                        {
                            "offsets": (offset_mapping[start_index][0], offset_mapping[end_index][1]),
                            "score": start_logits[start_index] + end_logits[end_index],
                            "start_logit": start_logits[start_index],
                            "end_logit": end_logits[end_index],
                        }
                    )
        if version_2_with_negative and min_null_prediction is not None:
            # Add the minimum null prediction
            prelim_predictions.append(min_null_prediction)
            null_score = min_null_prediction["score"]

        # Only keep the best `n_best_size` predictions.
        predictions = sorted(prelim_predictions, key=lambda x: x["score"], reverse=True)[:n_best_size]

        # Add back the minimum null prediction if it was removed because of its low score.
        if (
            version_2_with_negative
            and min_null_prediction is not None
            and not any(p["offsets"] == (0, 0) for p in predictions)
        ):
            predictions.append(min_null_prediction)

        # Use the offsets to gather the answer text in the original context.
        context = example["context"]
        for pred in predictions:
            offsets = pred.pop("offsets")
            pred["text"] = context[offsets[0] : offsets[1]]

        # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid
        # failure.
        if len(predictions) == 0 or (len(predictions) == 1 and predictions[0]["text"] == ""):
            predictions.insert(0, {"text": "empty", "start_logit": 0.0, "end_logit": 0.0, "score": 0.0})

        # Compute the softmax of all scores (we do it with numpy to stay independent from torch/tf in this file, using
        # the LogSumExp trick).
        scores = np.array([pred.pop("score") for pred in predictions])
        exp_scores = np.exp(scores - np.max(scores))
        probs = exp_scores / exp_scores.sum()

        # Include the probabilities in our predictions.
        for prob, pred in zip(probs, predictions):
            pred["probability"] = prob

        # Pick the best prediction. If the null answer is not possible, this is easy.
        if not version_2_with_negative:
            all_predictions[example["id"]] = predictions[0]["text"]
        else:
            # Otherwise we first need to find the best non-empty prediction.
            i = 0
            while predictions[i]["text"] == "":
                i += 1
            best_non_null_pred = predictions[i]

            # Then we compare to the null prediction using the threshold.
            score_diff = null_score - best_non_null_pred["start_logit"] - best_non_null_pred["end_logit"]
            scores_diff_json[example["id"]] = float(score_diff)  # To be JSON-serializable.
            if score_diff > null_score_diff_threshold:
                all_predictions[example["id"]] = ""
            else:
                all_predictions[example["id"]] = best_non_null_pred["text"]

        # Make `predictions` JSON-serializable by casting np.float back to float.
        all_nbest_json[example["id"]] = [
            {k: (float(v) if isinstance(v, (np.float16, np.float32, np.float64)) else v) for k, v in pred.items()}
            for pred in predictions
        ]

    # If we have an output_dir, let's save all those dicts.
    if output_dir is not None:
        if not os.path.isdir(output_dir):
            raise EnvironmentError(f"{output_dir} is not a directory.")

        prediction_file = os.path.join(
            output_dir, "predictions.json" if prefix is None else f"{prefix}_predictions.json"
        )
        nbest_file = os.path.join(
            output_dir, "nbest_predictions.json" if prefix is None else f"{prefix}_nbest_predictions.json"
        )
        if version_2_with_negative:
            null_odds_file = os.path.join(
                output_dir, "null_odds.json" if prefix is None else f"{prefix}_null_odds.json"
            )

        logger.info(f"Saving predictions to {prediction_file}.")
        with open(prediction_file, "w") as writer:
            writer.write(json.dumps(all_predictions, indent=4) + "\n")
        logger.info(f"Saving nbest_preds to {nbest_file}.")
        with open(nbest_file, "w") as writer:
            writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
        if version_2_with_negative:
            logger.info(f"Saving null_odds to {null_odds_file}.")
            with open(null_odds_file, "w") as writer:
                writer.write(json.dumps(scores_diff_json, indent=4) + "\n")



SUMMARY:
This code is a part of a text classification model, specifically designed to generate answers based on context. Here's a concise summary:

**Functionality**

* The function takes an input dictionary with context and ID as keys, and returns two dictionaries: `all_predictions` and `all_nbest_json`.
* It identifies the best prediction by comparing the logit scores of the null answer and non-null predictions.
* The function uses NumPy to compute softmax scores and includes probabilities in the predictions.

**Input/Output**

* Input dictionary with context, ID, and threshold values (not shown in the code snippet).
* Output dictionaries: `all_predictions` contains best prediction text, and `all_nbest_json` contains all non-null predictions (including probability scores).

**Implementation Details**

* The function uses NumPy to handle floating-point numbers and perform calculations.
* It uses JSON serialization to convert output data into a human-readable format.

Overall, this code is responsible for determining the most likely answer based on input context and returns two sets of results: one without non-null predictions (null prediction) and another with all predicted answers.


key: chroma:document
string_value:
CODE:

def postprocess_qa_predictions_with_beam_search(
    examples,
    features,
    predictions: Tuple[np.ndarray, np.ndarray],
    version_2_with_negative: bool = False,
    n_best_size: int = 20,
    max_answer_length: int = 30,
    start_n_top: int = 5,
    end_n_top: int = 5,
    output_dir: Optional[str] = None,
    prefix: Optional[str] = None,
    log_level: Optional[int] = logging.WARNING,
):
    """
    Post-processes the predictions of a question-answering model with beam search to convert them to answers that are substrings of the
    original contexts. This is the postprocessing functions for models that return start and end logits, indices, as well as
    cls token predictions.

    Args:
        examples: The non-preprocessed dataset (see the main script for more information).
        features: The processed dataset (see the main script for more information).
        predictions (:obj:`Tuple[np.ndarray, np.ndarray]`):
            The predictions of the model: two arrays containing the start logits and the end logits respectively. Its
            first dimension must match the number of elements of :obj:`features`.
        version_2_with_negative (:obj:`bool`, `optional`, defaults to :obj:`False`):
            Whether or not the underlying dataset contains examples with no answers.
        n_best_size (:obj:`int`, `optional`, defaults to 20):
            The total number of n-best predictions to generate when looking for an answer.
        max_answer_length (:obj:`int`, `optional`, defaults to 30):
            The maximum length of an answer that can be generated. This is needed because the start and end predictions
            are not conditioned on one another.
        start_n_top (:obj:`int`, `optional`, defaults to 5):
            The number of top start logits too keep when searching for the :obj:`n_best_size` predictions.
        end_n_top (:obj:`int`, `optional`, defaults to 5):
            The number of top end logits too keep when searching for the :obj:`n_best_size` predictions.
        output_dir (:obj:`str`, `optional`):
            If provided, the dictionaries of predictions, n_best predictions (with their scores and logits) and, if
            :obj:`version_2_with_negative=True`, the dictionary of the scores differences between best and null
            answers, are saved in `output_dir`.
        prefix (:obj:`str`, `optional`):
            If provided, the dictionaries mentioned above are saved with `prefix` added to their names.
        log_level (:obj:`int`, `optional`, defaults to ``logging.WARNING``):
            ``logging`` log level (e.g., ``logging.WARNING``)
    """
    if len(predictions) != 5:
        raise ValueError("`predictions` should be a tuple with five elements.")
    start_top_log_probs, start_top_index, end_top_log_probs, end_top_index, cls_logits = predictions

    if len(predictions[0]) != len(features):
        raise ValueError(f"Got {len(predictions[0])} predictions and {len(features)} features.")

    # Build a map example to its corresponding features.
    example_id_to_index = {k: i for i, k in enumerate(examples["id"])}
    features_per_example = collections.defaultdict(list)
    for i, feature in enumerate(features):
        features_per_example[example_id_to_index[feature["example_id"]]].append(i)

    # The dictionaries we have to fill.
    all_predictions = collections.OrderedDict()
    all_nbest_json = collections.OrderedDict()
    scores_diff_json = collections.OrderedDict() if version_2_with_negative else None

    # Logging.
    logger.setLevel(log_level)
    logger.info(f"Post-processing {len(examples)} example predictions split into {len(features)} features.")

    # Let's loop over all the examples!
    for example_index, example in enumerate(tqdm(examples)):
        # Those are the indices of the features associated to the current example.
        feature_indices = features_per_example[example_index]

        min_null_score = None
        prelim_predictions = []

        # Looping through all the features associated to the current example.
        for feature_index in feature_indices:
            # We grab the predictions of the model for this feature.
            start_log_prob = start_top_log_probs[feature_index]
            start_indexes = start_top_index[feature_index]
            end_log_prob = end_top_log_probs[feature_index]
            end_indexes = end_top_index[feature_index]
            feature_null_score = cls_logits[feature_index]
            # This is what will allow us to map some the positions in our logits to span of texts in the original
            # context.
            offset_mapping = features[feature_index]["offset_mapping"]
            # Optional `token_is_max_context`, if provided we will remove answers that do not have the maximum context
            # available in the current feature.
            token_is_max_context = features[feature_index].get("token_is_max_context", None)

            # Update minimum null prediction
            if min_null_score is None or feature_null_score < min_null_score:
                min_null_score = feature_null_score

            # Go through all possibilities for the `n_start_top`/`n_end_top` greater start and end logits.
            for i in range(start_n_top):
                for j in range(end_n_top):
                    start_index = int(start_indexes[i])
                    j_index = i * end_n_top + j
                    end_index = int(end_indexes[j_index])
                    # Don't consider out-of-scope answers (last part of the test should be unnecessary because of the
                    # p_mask but let's not take any risk)
                    if (
                        start_index >= len(offset_mapping)
                        or end_index >= len(offset_mapping)
                        or offset_mapping[start_index] is None
                        or len(offset_mapping[start_index]) < 2
                        or offset_mapping[end_index] is None
                        or len(offset_mapping[end_index]) < 2
                    ):
                        continue

                    # Don't consider answers with a length negative or > max_answer_length.
                    if end_index < start_index or end_index - start_index + 1 > max_answer_length:
                        continue
                    # Don't consider answer that don't have the maximum context available (if such information is
                    # provided).
                    if token_is_max_context is not None and not token_is_max_context.get(str(start_index), False):
                        continue
                    prelim_predictions.append(
                        {
                            "offsets": (offset_mapping[start_index][0], offset_mapping[end_index][1]),
                            "score": start_log_prob[i] + end_log_prob[j_index],
                            "start_log_prob": start_log_prob[i],
                            "end_log_prob": end_log_prob[j_index],
                        }
                    )

        # Only keep the best `n_best_size` predictions.
        predictions = sorted(prelim_predictions, key=lambda x: x["score"], reverse=True)[:n_best_size]

        # Use the offsets to gather the answer text in the original context.
        context = example["context"]
        for pred in predictions:
            offsets = pred.pop("offsets")
            pred["text"] = context[offsets[0] : offsets[1]]

        # In the very rare edge case we have not a single non-null prediction, we create a fake prediction to avoid
        # failure.
        if len(predictions) == 0:
            # Without predictions min_null_score is going to be None and None will cause an exception later
            min_null_score = -2e-6
            predictions.insert(0, {"text": "", "start_logit": -1e-6, "end_logit": -1e-6, "score": min_null_score})

        # Compute the softmax of all scores (we do it with numpy to stay independent from torch/tf in this file, using
        # the LogSumExp trick).
        scores = np.array([pred.pop("score") for pred in predictions])
        exp_scores = np.exp(scores - np.max(scores))
        probs = exp_scores / exp_scores.sum()

        # Include the probabilities in our predictions.
        for prob, pred in zip(probs, predictions):
            pred["probability"] = prob

        # Pick the best prediction and set the probability for the null answer.
        all_predictions[example["id"]] = predictions[0]["text"]
        if version_2_with_negative:
            scores_diff_json[example["id"]] = float(min_null_score)

        # Make `predictions` JSON-serializable by casting np.float back to float.
        all_nbest_json[example["id"]] = [
            {k: (float(v) if isinstance(v, (np.float16, np.float32, np.float64)) else v) for k, v in pred.items()}
            for pred in predictions
        ]

    # If we have an output_dir, let's save all those dicts.
    if output_dir is not None:
        if not os.path.isdir(output_dir):
            raise EnvironmentError(f"{output_dir} is not a directory.")

        prediction_file = os.path.join(
            output_dir, "predictions.json" if prefix is None else f"{prefix}_predictions.json"
        )
        nbest_file = os.path.join(
            output_dir, "nbest_predictions.json" if prefix is None else f"{prefix}_nbest_predictions.json"
        )
        if version_2_with_negative:
            null_odds_file = os.path.join(
                output_dir, "null_odds.json" if prefix is None else f"{prefix}_null_odds.json"
            )

        logger.info(f"Saving predictions to {prediction_file}.")
        with open(prediction_file, "w") as writer:
            writer.write(json.dumps(all_predictions, indent=4) + "\n")
        logger.info(f"Saving nbest_preds to {nbest_file}.")
        with open(nbest_file, "w") as writer:
            writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
        if version_2_with_negative:
            logger.info(f"Saving null_odds to {null_odds_file}.")
            with open(null_odds_file, "w") as writer:
                writer.write(json.dumps(scores_diff_json, indent=4) + "\n")



SUMMARY:
Here is a concise summary of the provided function:

**Function Name**: `compute_prediction`

**Description**: This function computes and returns predictions for a given text input. It uses a scoring system to rank possible answers based on their likelihood. It outputs two dictionaries: one containing the best predicted answer (`all_predictions`), and another with null odds scores if the model supports it.

**Input Parameters**:

* `context`: The input text context.
* `p_mask`: A mask indicating which parts of the input are relevant (not used in this implementation).
* `output_dir`: An optional directory to save output files to.
* `n_best_size`: The number of predictions to return.
* `token_is_max_context`: A function to determine if a token has the maximum context available.

**Return Values**:

* `all_predictions`: A dictionary mapping input IDs to their best predicted answers.
* `scores_diff_json` (optional): A dictionary with null odds scores for each input ID, if applicable.


key: chroma:document
string_value:
CODE:
# region Helper classes
class SavePretrainedCallback(keras.callbacks.Callback):
    # Hugging Face models have a save_pretrained() method that saves both the weights and the necessary
    # metadata to allow them to be loaded as a pretrained model in future. This is a simple Keras callback
    # that saves the model with this method after each epoch.
    def __init__(self, output_dir, **kwargs):
        super().__init__()
        self.output_dir = output_dir

    def on_epoch_end(self, epoch, logs=None):


SUMMARY:
**SavePretrainedCallback Class Summary**

* **Purpose**: Saves a Keras model using Hugging Face's `save_pretrained()` method after each epoch, preserving weights and necessary metadata.

* **Key Implementation Details**:
	+ Saves the model using `self.model.save_pretrained(self.output_dir)`
	+ Utilizes Hugging Face's `save_pretrained()` method to save both weights and metadata

* **Notable Dependencies/Requirements**:
	+ None explicitly listed
	+ Assumes the presence of `Docstring` (potentially a helper class or function)

* **Code Structure**:
	+继承自 Keras Callback (`super().__init__()`)
	+有一个输出目录 (`self.output_dir`) 和其他可选参数 (`**kwargs`)
	+具有一个epoch结束的回调事件处理函数（`on_epoch_end()`)


key: chroma:document
string_value:
CODE:

def main():
    # region Argument parsing
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_qa", model_args, data_args, framework="tensorflow")

    output_dir = Path(training_args.output_dir)
    output_dir.mkdir(parents=True, exist_ok=True)
    # endregion

    # region Checkpoints
    checkpoint = None
    if len(os.listdir(training_args.output_dir)) > 0 and not training_args.overwrite_output_dir:
        if (output_dir / CONFIG_NAME).is_file() and (output_dir / TF2_WEIGHTS_NAME).is_file():
            checkpoint = output_dir
            logger.info(
                f"Checkpoint detected, resuming training from checkpoint in {training_args.output_dir}. To avoid this"
                " behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )
        else:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to continue regardless."
            )
    # endregion

    # region Logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if training_args.should_log else logging.WARN)

    # Set the verbosity to info of the Transformers logger (on main process only):
    if training_args.should_log:
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    logger.info(f"Training/evaluation parameters {training_args}")
    # endregion

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # region Load Data
    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        datasets = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]

        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if data_args.test_file is not None:
            data_files["test"] = data_args.test_file
            extension = data_args.test_file.split(".")[-1]
        datasets = load_dataset(
            extension,
            data_files=data_files,
            field="data",
            cache_dir=model_args.cache_dir,
            token=model_args.token,
        )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.
    # endregion

    # region Load pretrained model and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=True,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    # endregion

    # region Tokenizer check: this script requires a fast tokenizer.
    if not isinstance(tokenizer, PreTrainedTokenizerFast):
        raise ValueError(
            "This example script only works for models that have a fast tokenizer. Checkout the big table of models at"
            " https://huggingface.co/transformers/index.html#supported-frameworks to find the model types that meet"
            " this requirement"
        )
    # endregion

    # region Preprocessing the datasets
    # Preprocessing is slightly different for training and evaluation.
    if training_args.do_train:
        column_names = datasets["train"].column_names
    elif training_args.do_eval:
        column_names = datasets["validation"].column_names
    else:
        column_names = datasets["test"].column_names
    question_column_name = "question" if "question" in column_names else column_names[0]
    context_column_name = "context" if "context" in column_names else column_names[1]
    answer_column_name = "answers" if "answers" in column_names else column_names[2]

    # Padding side determines if we do (question|context) or (context|question).
    pad_on_right = tokenizer.padding_side == "right"

    if data_args.max_seq_length > tokenizer.model_max_length:
        logger.warning(
            f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the "
            f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
        )
    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    if data_args.pad_to_max_length or isinstance(training_args.strategy, tf.distribute.TPUStrategy):
        logger.info("Padding all batches to max length because argument was set or we're on TPU.")
        padding = "max_length"
    else:
        padding = False

    # Training preprocessing
    def prepare_train_features(examples):
        # Some of the questions have lots of whitespace on the left, which is not useful and will make the
        # truncation of the context fail (the tokenized question will take a lots of space). So we remove that
        # left whitespace
        examples[question_column_name] = [q.lstrip() for q in examples[question_column_name]]

        # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results
        # in one example possible giving several features when a context is long, each of those features having a
        # context that overlaps a bit the context of the previous feature.
        tokenized_examples = tokenizer(
            examples[question_column_name if pad_on_right else context_column_name],
            examples[context_column_name if pad_on_right else question_column_name],
            truncation="only_second" if pad_on_right else "only_first",
            max_length=max_seq_length,
            stride=data_args.doc_stride,
            return_overflowing_tokens=True,
            return_offsets_mapping=True,
            padding=padding,
        )

        # Since one example might give us several features if it has a long context, we need a map from a feature to
        # its corresponding example. This key gives us just that.
        sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")
        # The offset mappings will give us a map from token to character position in the original context. This will
        # help us compute the start_positions and end_positions.
        offset_mapping = tokenized_examples.pop("offset_mapping")

        # Let's label those examples!
        tokenized_examples["start_positions"] = []
        tokenized_examples["end_positions"] = []

        for i, offsets in enumerate(offset_mapping):
            # We will label impossible answers with the index of the CLS token.
            input_ids = tokenized_examples["input_ids"][i]
            cls_index = input_ids.index(tokenizer.cls_token_id)

            # Grab the sequence corresponding to that example (to know what is the context and what is the question).
            sequence_ids = tokenized_examples.sequence_ids(i)

            # One example can give several spans, this is the index of the example containing this span of text.
            sample_index = sample_mapping[i]
            answers = examples[answer_column_name][sample_index]
            # If no answers are given, set the cls_index as answer.
            if len(answers["answer_start"]) == 0:
                tokenized_examples["start_positions"].append(cls_index)
                tokenized_examples["end_positions"].append(cls_index)
            else:
                # Start/end character index of the answer in the text.
                start_char = answers["answer_start"][0]
                end_char = start_char + len(answers["text"][0])

                # Start token index of the current span in the text.
                token_start_index = 0
                while sequence_ids[token_start_index] != (1 if pad_on_right else 0):
                    token_start_index += 1

                # End token index of the current span in the text.
                token_end_index = len(input_ids) - 1
                while sequence_ids[token_end_index] != (1 if pad_on_right else 0):
                    token_end_index -= 1

                # Detect if the answer is out of the span (in which case this feature is labeled with the CLS index).
                if not (offsets[token_start_index][0] <= start_char and offsets[token_end_index][1] >= end_char):
                    tokenized_examples["start_positions"].append(cls_index)
                    tokenized_examples["end_positions"].append(cls_index)
                else:
                    # Otherwise move the token_start_index and token_end_index to the two ends of the answer.
                    # Note: we could go after the last offset if the answer is the last word (edge case).
                    while token_start_index < len(offsets) and offsets[token_start_index][0] <= start_char:
                        token_start_index += 1
                    tokenized_examples["start_positions"].append(token_start_index - 1)
                    while offsets[token_end_index][1] >= end_char:
                        token_end_index -= 1
                    tokenized_examples["end_positions"].append(token_end_index + 1)

        return tokenized_examples

    processed_datasets = {}
    if training_args.do_train:
        if "train" not in datasets:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = datasets["train"]
        if data_args.max_train_samples is not None:
            # We will select sample from whole data if argument is specified
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))
        # Create train feature from dataset
        train_dataset = train_dataset.map(
            prepare_train_features,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
        )
        if data_args.max_train_samples is not None:
            # Number of samples might increase during Feature Creation, We select only specified max samples
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))
        processed_datasets["train"] = train_dataset

    # Validation preprocessing
    def prepare_validation_features(examples):
        # Some of the questions have lots of whitespace on the left, which is not useful and will make the
        # truncation of the context fail (the tokenized question will take a lots of space). So we remove that
        # left whitespace
        examples[question_column_name] = [q.lstrip() for q in examples[question_column_name]]

        # Tokenize our examples with truncation and maybe padding, but keep the overflows using a stride. This results
        # in one example possible giving several features when a context is long, each of those features having a
        # context that overlaps a bit the context of the previous feature.
        tokenized_examples = tokenizer(
            examples[question_column_name if pad_on_right else context_column_name],
            examples[context_column_name if pad_on_right else question_column_name],
            truncation="only_second" if pad_on_right else "only_first",
            max_length=max_seq_length,
            stride=data_args.doc_stride,
            return_overflowing_tokens=True,
            return_offsets_mapping=True,
            padding=padding,
        )

        # Since one example might give us several features if it has a long context, we need a map from a feature to
        # its corresponding example. This key gives us just that.
        sample_mapping = tokenized_examples.pop("overflow_to_sample_mapping")

        # For evaluation, we will need to convert our predictions to substrings of the context, so we keep the
        # corresponding example_id and we will store the offset mappings.
        tokenized_examples["example_id"] = []

        for i in range(len(tokenized_examples["input_ids"])):
            # Grab the sequence corresponding to that example (to know what is the context and what is the question).
            sequence_ids = tokenized_examples.sequence_ids(i)
            context_index = 1 if pad_on_right else 0

            # One example can give several spans, this is the index of the example containing this span of text.
            sample_index = sample_mapping[i]
            tokenized_examples["example_id"].append(examples["id"][sample_index])

            # Set to None the offset_mapping that are not part of the context so it's easy to determine if a token
            # position is part of the context or not.
            tokenized_examples["offset_mapping"][i] = [
                (o if sequence_ids[k] == context_index else None)
                for k, o in enumerate(tokenized_examples["offset_mapping"][i])
            ]

        return tokenized_examples

    if training_args.do_eval:
        if "validation" not in datasets:
            raise ValueError("--do_eval requires a validation dataset")
        eval_examples = datasets["validation"]
        if data_args.max_eval_samples is not None:
            # We will select sample from whole data
            max_eval_samples = min(len(eval_examples), data_args.max_eval_samples)
            eval_examples = eval_examples.select(range(max_eval_samples))
        # Validation Feature Creation
        eval_dataset = eval_examples.map(
            prepare_validation_features,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
        )
        if data_args.max_eval_samples is not None:
            # During Feature creation dataset samples might increase, we will select required samples again
            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
            eval_dataset = eval_dataset.select(range(max_eval_samples))
        processed_datasets["validation"] = eval_dataset

    if training_args.do_predict:
        if "test" not in datasets:
            raise ValueError("--do_predict requires a test dataset")
        predict_examples = datasets["test"]
        if data_args.max_predict_samples is not None:
            # We will select sample from whole data
            predict_examples = predict_examples.select(range(data_args.max_predict_samples))
        # Predict Feature Creation
        predict_dataset = predict_examples.map(
            prepare_validation_features,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
        )
        if data_args.max_predict_samples is not None:
            # During Feature creation dataset samples might increase, we will select required samples again
            max_predict_samples = min(len(predict_dataset), data_args.max_predict_samples)
            predict_dataset = predict_dataset.select(range(max_predict_samples))
        processed_datasets["test"] = predict_dataset
    # endregion

    # region Metrics and Post-processing:
    def post_processing_function(examples, features, predictions, stage="eval"):
        # Post-processing: we match the start logits and end logits to answers in the original context.
        predictions = postprocess_qa_predictions(
            examples=examples,
            features=features,
            predictions=predictions,
            version_2_with_negative=data_args.version_2_with_negative,
            n_best_size=data_args.n_best_size,
            max_answer_length=data_args.max_answer_length,
            null_score_diff_threshold=data_args.null_score_diff_threshold,
            output_dir=training_args.output_dir,
            prefix=stage,
        )
        # Format the result to the format the metric expects.
        if data_args.version_2_with_negative:
            formatted_predictions = [
                {"id": k, "prediction_text": v, "no_answer_probability": 0.0} for k, v in predictions.items()
            ]
        else:
            formatted_predictions = [{"id": k, "prediction_text": v} for k, v in predictions.items()]

        references = [{"id": ex["id"], "answers": ex[answer_column_name]} for ex in examples]
        return EvalPrediction(predictions=formatted_predictions, label_ids=references)

    metric = evaluate.load(
        "squad_v2" if data_args.version_2_with_negative else "squad", cache_dir=model_args.cache_dir
    )

    def compute_metrics(p: EvalPrediction):
        return metric.compute(predictions=p.predictions, references=p.label_ids)

    # endregion

    with training_args.strategy.scope():
        dataset_options = tf.data.Options()
        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF
        num_replicas = training_args.strategy.num_replicas_in_sync

        # region Load model and prepare datasets
        if checkpoint is None:
            model_path = model_args.model_name_or_path
        else:
            model_path = checkpoint
        model = TFAutoModelForQuestionAnswering.from_pretrained(
            model_path,
            config=config,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
        if training_args.do_train:
            training_dataset = model.prepare_tf_dataset(
                processed_datasets["train"],
                shuffle=True,
                batch_size=training_args.per_device_train_batch_size * num_replicas,
                tokenizer=tokenizer,
            )

            training_dataset = training_dataset.with_options(dataset_options)

            num_train_steps = len(training_dataset) * training_args.num_train_epochs
            if training_args.warmup_steps > 0:
                num_warmup_steps = training_args.warmup_steps
            elif training_args.warmup_ratio > 0:
                num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
            else:
                num_warmup_steps = 0

            optimizer, schedule = create_optimizer(
                init_lr=training_args.learning_rate,
                num_train_steps=len(training_dataset) * training_args.num_train_epochs,
                num_warmup_steps=num_warmup_steps,
                adam_beta1=training_args.adam_beta1,
                adam_beta2=training_args.adam_beta2,
                adam_epsilon=training_args.adam_epsilon,
                weight_decay_rate=training_args.weight_decay,
                adam_global_clipnorm=training_args.max_grad_norm,
            )

            # Transformers models compute the right loss for their task by default when labels are passed, and will
            # use this for training unless you specify your own loss function in compile().
            model.compile(optimizer=optimizer, jit_compile=training_args.xla, metrics=["accuracy"])

        else:
            # Optimizer doesn't matter as it won't be used anyway
            model.compile(optimizer="sgd", jit_compile=training_args.xla, metrics=["accuracy"])
            training_dataset = None

        if training_args.do_eval:
            eval_dataset = model.prepare_tf_dataset(
                processed_datasets["validation"],
                shuffle=False,
                batch_size=training_args.per_device_train_batch_size * num_replicas,
                tokenizer=tokenizer,
            )
            eval_dataset = eval_dataset.with_options(dataset_options)
        else:
            eval_dataset = None

        if training_args.do_predict:
            predict_dataset = model.prepare_tf_dataset(
                processed_datasets["test"],
                shuffle=False,
                batch_size=training_args.per_device_eval_batch_size * num_replicas,
                tokenizer=tokenizer,
            )
            predict_dataset = predict_dataset.with_options(dataset_options)
        else:
            predict_dataset = None

        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            if data_args.dataset_name is not None:
                push_to_hub_model_id = f"{model_name}-finetuned-{data_args.dataset_name}"
            else:
                push_to_hub_model_id = f"{model_name}-finetuned-question-answering"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "question-answering"}
        if data_args.dataset_name is not None:
            model_card_kwargs["dataset_tags"] = data_args.dataset_name
            if data_args.dataset_config_name is not None:
                model_card_kwargs["dataset_args"] = data_args.dataset_config_name
                model_card_kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
            else:
                model_card_kwargs["dataset"] = data_args.dataset_name

        if training_args.push_to_hub:
            callbacks = [
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            ]
        else:
            callbacks = []
        # endregion

        # region Training and Evaluation

        if training_args.do_train:
            # Note that the validation and test datasets have been processed in a different way to the
            # training datasets in this example, and so they don't have the same label structure.
            # As such, we don't pass them directly to Keras, but instead get model predictions to evaluate
            # after training.
            model.fit(training_dataset, epochs=int(training_args.num_train_epochs), callbacks=callbacks)

        if training_args.do_eval:
            logger.info("*** Evaluation ***")

            # In this example, we compute advanced metrics at the end of training, but
            # if you'd like to compute metrics every epoch that are too complex to be written as
            # standard Keras metrics, you can use our KerasMetricCallback. See
            # https://huggingface.co/docs/transformers/main/en/main_classes/keras_callbacks

            eval_predictions = model.predict(eval_dataset)
            if isinstance(eval_predictions.start_logits, tf.RaggedTensor):
                # If predictions are RaggedTensor, we densify them. Since they are logits, padding with 0 is a bad idea!
                # The reason is that a logit of 0 can often end up as quite a high probability value, sometimes even
                # the highest probability in a sample. Instead, we use a large negative value, which ensures that the
                # padding positions are correctly masked.
                eval_start_logits = eval_predictions.start_logits.to_tensor(default_value=-1000).numpy()
                eval_end_logits = eval_predictions.end_logits.to_tensor(default_value=-1000).numpy()
            else:
                eval_start_logits = eval_predictions.start_logits
                eval_end_logits = eval_predictions.end_logits

            post_processed_eval = post_processing_function(
                datasets["validation"],
                processed_datasets["validation"],
                (eval_start_logits, eval_end_logits),
            )
            metrics = compute_metrics(post_processed_eval)
            logging.info("Evaluation metrics:")
            for metric, value in metrics.items():
                logging.info(f"{metric}: {value:.3f}")
            if training_args.output_dir is not None:
                output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
                with open(output_eval_file, "w") as writer:
                    writer.write(json.dumps(metrics))
        # endregion

        # region Prediction
        if training_args.do_predict:
            logger.info("*** Predict ***")

            test_predictions = model.predict(predict_dataset)
            if isinstance(test_predictions.start_logits, tf.RaggedTensor):
                # If predictions are RaggedTensor, we densify them. Since they are logits, padding with 0 is a bad idea!
                # The reason is that a logit of 0 can often end up as quite a high probability value, sometimes even
                # the highest probability in a sample. Instead, we use a large negative value, which ensures that the
                # padding positions are correctly masked.
                test_start_logits = test_predictions.start_logits.to_tensor(default_value=-1000).numpy()
                test_end_logits = test_predictions.end_logits.to_tensor(default_value=-1000).numpy()
            else:
                test_start_logits = test_predictions.start_logits
                test_end_logits = test_predictions.end_logits
            post_processed_test = post_processing_function(
                datasets["test"],
                processed_datasets["test"],
                (test_start_logits, test_end_logits),
            )
            metrics = compute_metrics(post_processed_test)

            logging.info("Test metrics:")
            for metric, value in metrics.items():
                logging.info(f"{metric}: {value:.3f}")
        # endregion

    if training_args.output_dir is not None and not training_args.push_to_hub:
        # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
This code is a template for a command-line interface (CLI) in PyTorch for training and fine-tuning a pre-trained Hugging Face Transformers model. 

The CLI provides options to train, evaluate, and predict with the model after it has been loaded from a checkpoint file.

Key Features:

1. Training: The CLI allows users to specify the number of epochs, batch size, learning rate, decay factor, and other hyperparameters for training.
2. Evaluation: Users can specify which datasets to use for evaluation during training (validation and/or test). Advanced metrics are also computed after evaluation is complete.
3. Prediction: If desired, users can predict answers from a specific dataset using the trained model.

Improvements That Could Be Suggested:

1. Type Hints: The function parameters could have type hints to make the code more readable and self-documenting.
2. Hyperparameter Tuning: A simple grid search or Random Search for hyperparameters would improve the performance of the model by finding optimal settings.
3. Early Stopping: Implementing early stopping during training to prevent overfitting.
4. Model Monitoring: Inbuilt mechanisms like monitoring loss, accuracy, and predictions on validation sets to track model's performance effectively.

Overall, this code serves as a solid foundation for creating custom scripts using PyTorch Transformer models.


key: chroma:document
string_value:
CODE:

def main():
    # region Argument parsing
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_summarization", model_args, data_args, framework="tensorflow")
    # endregion

    # region Logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO)
    datasets.utils.logging.set_verbosity(logging.INFO)
    transformers.utils.logging.set_verbosity(logging.INFO)

    # Log on each process the small summary:
    logger.info(f"Training/evaluation parameters {training_args}")
    # endregion

    # region T5 special-casing
    if data_args.source_prefix is None and model_args.model_name_or_path in [
        "google-t5/t5-small",
        "google-t5/t5-base",
        "google-t5/t5-large",
        "google-t5/t5-3b",
        "google-t5/t5-11b",
    ]:
        logger.warning(
            "You're running a t5 model but didn't provide a source prefix, which is the expected, e.g. with "
            "`--source_prefix 'summarize: ' `"
        )
    # endregion

    # region Detecting last checkpoint
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None and training_args.resume_from_checkpoint is None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )
    # endregion

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # region Load datasets
    # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files this script will use the first column for the full texts and the second column for the
    # summaries (unless you specify column names for this with the `text_column` and `summary_column` arguments).
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        raw_datasets = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if data_args.test_file is not None:
            data_files["test"] = data_args.test_file
            extension = data_args.test_file.split(".")[-1]
        raw_datasets = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
        )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.
    # endregion

    # region Load model config and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.

    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast_tokenizer,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )

    prefix = data_args.source_prefix if data_args.source_prefix is not None else ""
    # endregion

    # region Dataset preprocessing
    # We need to tokenize inputs and targets.
    if training_args.do_train:
        column_names = raw_datasets["train"].column_names
    elif training_args.do_eval:
        column_names = raw_datasets["validation"].column_names
    else:
        logger.info("There is nothing to do. Please pass `do_train`, and/or `do_eval`.")
        return

    # Get the column names for input/target.
    dataset_columns = summarization_name_mapping.get(data_args.dataset_name, None)
    if data_args.text_column is None:
        text_column = dataset_columns[0] if dataset_columns is not None else column_names[0]
    else:
        text_column = data_args.text_column
        if text_column not in column_names:
            raise ValueError(
                f"--text_column' value '{data_args.text_column}' needs to be one of: {', '.join(column_names)}"
            )
    if data_args.summary_column is None:
        summary_column = dataset_columns[1] if dataset_columns is not None else column_names[1]
    else:
        summary_column = data_args.summary_column
        if summary_column not in column_names:
            raise ValueError(
                f"--summary_column' value '{data_args.summary_column}' needs to be one of: {', '.join(column_names)}"
            )

    # Temporarily set max_target_length for training.
    max_target_length = data_args.max_target_length
    padding = "max_length" if data_args.pad_to_max_length else False

    def preprocess_function(examples):
        inputs = examples[text_column]
        targets = examples[summary_column]
        inputs = [prefix + inp for inp in inputs]
        model_inputs = tokenizer(inputs, max_length=data_args.max_source_length, padding=padding, truncation=True)

        # Tokenize targets with the `text_target` keyword argument
        labels = tokenizer(text_target=targets, max_length=max_target_length, padding=padding, truncation=True)

        # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
        # padding in the loss.
        if padding == "max_length" and data_args.ignore_pad_token_for_loss:
            labels["input_ids"] = [
                [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
            ]

        model_inputs["labels"] = labels["input_ids"]
        return model_inputs

    if training_args.do_train:
        if "train" not in raw_datasets:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = raw_datasets["train"]
        if data_args.max_train_samples is not None:
            max_train_samples = min(len(train_dataset), data_args.max_train_samples)
            train_dataset = train_dataset.select(range(max_train_samples))
        train_dataset = train_dataset.map(
            preprocess_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on train dataset",
        )
    else:
        train_dataset = None

    if training_args.do_eval:
        max_target_length = data_args.val_max_target_length
        if "validation" not in raw_datasets:
            raise ValueError("--do_eval requires a validation dataset")
        eval_dataset = raw_datasets["validation"]
        if data_args.max_eval_samples is not None:
            max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
            eval_dataset = eval_dataset.select(range(max_eval_samples))
        eval_dataset = eval_dataset.map(
            preprocess_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on validation dataset",
        )
    else:
        eval_dataset = None
    # endregion

    # region Text preprocessing
    def postprocess_text(preds, labels):
        preds = [pred.strip() for pred in preds]
        labels = [label.strip() for label in labels]

        # rougeLSum expects newline after each sentence
        preds = ["\n".join(nltk.sent_tokenize(pred)) for pred in preds]
        labels = ["\n".join(nltk.sent_tokenize(label)) for label in labels]

        return preds, labels

    # endregion

    with training_args.strategy.scope():
        # region Prepare model
        model = TFAutoModelForSeq2SeqLM.from_pretrained(
            model_args.model_name_or_path,
            config=config,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )

        # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch
        # on a small vocab and want a smaller embedding size, remove this test.
        embeddings = model.get_input_embeddings()

        # Matt: This is a temporary workaround as we transition our models to exclusively using Keras embeddings.
        #       As soon as the transition is complete, all embeddings should be keras.Embeddings layers, and
        #       the weights will always be in embeddings.embeddings.
        if hasattr(embeddings, "embeddings"):
            embedding_size = embeddings.embeddings.shape[0]
        else:
            embedding_size = embeddings.weight.shape[0]
        if len(tokenizer) > embedding_size:
            model.resize_token_embeddings(len(tokenizer))
        # endregion

        # region Prepare TF Dataset objects
        if model.config.decoder_start_token_id is None:
            raise ValueError("Make sure that `config.decoder_start_token_id` is correctly defined")

        label_pad_token_id = -100 if data_args.ignore_pad_token_for_loss else tokenizer.pad_token_id
        data_collator = DataCollatorForSeq2Seq(
            tokenizer,
            model=model,
            label_pad_token_id=label_pad_token_id,
            pad_to_multiple_of=128,  # Reduce the number of unique shapes for XLA, especially for generation
            return_tensors="np",
        )

        dataset_options = tf.data.Options()
        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF

        num_replicas = training_args.strategy.num_replicas_in_sync
        total_train_batch_size = training_args.per_device_train_batch_size * num_replicas
        total_eval_batch_size = training_args.per_device_eval_batch_size * num_replicas

        # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
        # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
        # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
        # yourself if you use this method, whereas they are automatically inferred from the model input names when
        # using model.prepare_tf_dataset()
        # For more info see the docs:
        # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
        # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset

        tf_train_dataset = model.prepare_tf_dataset(
            train_dataset,
            collate_fn=data_collator,
            batch_size=total_train_batch_size,
            shuffle=True,
        ).with_options(dataset_options)
        tf_eval_dataset = model.prepare_tf_dataset(
            eval_dataset,
            collate_fn=data_collator,
            batch_size=total_eval_batch_size,
            shuffle=False,
        ).with_options(dataset_options)
        # endregion

        # region Optimizer, loss and LR scheduling
        num_train_steps = int(len(tf_train_dataset) * training_args.num_train_epochs)
        if training_args.warmup_steps > 0:
            num_warmup_steps = training_args.warmup_steps
        elif training_args.warmup_ratio > 0:
            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
        else:
            num_warmup_steps = 0
        if training_args.do_train:
            optimizer, lr_schedule = create_optimizer(
                init_lr=training_args.learning_rate,
                num_train_steps=num_train_steps,
                num_warmup_steps=num_warmup_steps,
                adam_beta1=training_args.adam_beta1,
                adam_beta2=training_args.adam_beta2,
                adam_epsilon=training_args.adam_epsilon,
                weight_decay_rate=training_args.weight_decay,
                adam_global_clipnorm=training_args.max_grad_norm,
            )
        else:
            optimizer = "sgd"  # Just write anything because we won't be using it

        # endregion

        # region Metric and KerasMetricCallback
        if training_args.do_eval:
            metric = evaluate.load("rouge", cache_dir=model_args.cache_dir)

            if data_args.val_max_target_length is None:
                data_args.val_max_target_length = data_args.max_target_length

            gen_kwargs = {
                "max_length": data_args.val_max_target_length if data_args is not None else config.max_length,
                "num_beams": data_args.num_beams,
                "no_repeat_ngram_size": 0,  # Not supported under XLA right now, and some models set it by default
            }

            def compute_metrics(preds):
                predictions, labels = preds
                if isinstance(predictions, tuple):
                    predictions = predictions[0]
                decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
                labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
                decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
                decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)
                metrics = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
                # Only print the mid f-measures, but there are a lot of other statistics in there too!
                metrics = {key: round(val.mid.fmeasure * 100, 4) for key, val in metrics.items()}
                return metrics

            # The KerasMetricCallback allows metrics that are too complex to write as standard Keras metrics
            # to be computed each epoch. Any Python code can be included in the metric_fn. This is especially
            # useful for metrics like BLEU and ROUGE that perform string comparisons on decoded model outputs.
            # For more information, see the docs at
            # https://huggingface.co/docs/transformers/main_classes/keras_callbacks#transformers.KerasMetricCallback

            metric_callback = KerasMetricCallback(
                metric_fn=compute_metrics,
                eval_dataset=tf_eval_dataset,
                predict_with_generate=True,
                use_xla_generation=True,
                generate_kwargs=gen_kwargs,
            )
            callbacks = [metric_callback]
        else:
            callbacks = []
        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            if data_args.dataset_name is not None:
                push_to_hub_model_id = f"{model_name}-finetuned-{data_args.dataset_name}"
            else:
                push_to_hub_model_id = f"{model_name}-finetuned-summarization"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "summarization"}
        if data_args.dataset_name is not None:
            model_card_kwargs["dataset_tags"] = data_args.dataset_name
            if data_args.dataset_config_name is not None:
                model_card_kwargs["dataset_args"] = data_args.dataset_config_name
                model_card_kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
            else:
                model_card_kwargs["dataset"] = data_args.dataset_name

        if training_args.push_to_hub:
            # Because this training can be quite long, we save once per epoch.
            callbacks.append(
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            )
        # endregion

        # region Training
        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, jit_compile=training_args.xla)
        eval_metrics = None
        if training_args.do_train:
            logger.info("***** Running training *****")
            logger.info(f"  Num examples = {len(train_dataset)}")
            logger.info(f"  Num Epochs = {training_args.num_train_epochs}")
            logger.info(f"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}")
            logger.info(f"  Total train batch size = {total_train_batch_size}")
            logger.info(f"  Total optimization steps = {num_train_steps}")

            if training_args.xla and not data_args.pad_to_max_length:
                logger.warning(
                    "XLA training may be slow at first when --pad_to_max_length is not set "
                    "until all possible shapes have been compiled."
                )
            history = model.fit(tf_train_dataset, epochs=int(training_args.num_train_epochs), callbacks=callbacks)
            eval_metrics = {key: val[-1] for key, val in history.history.items()}
        # endregion

        # region Validation

        if training_args.do_eval and not training_args.do_train:
            # Do a standalone evaluation run
            logger.info("Evaluation...")

            # Compiling generation with XLA yields enormous speedups, see https://huggingface.co/blog/tf-xla-generate
            @tf.function(jit_compile=True)
            def generate(**kwargs):
                return model.generate(**kwargs)

            for batch, labels in tf_eval_dataset:
                batch.update(gen_kwargs)
                generated_tokens = generate(**batch)
                if isinstance(generated_tokens, tuple):
                    generated_tokens = generated_tokens[0]
                decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
                labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
                decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
                decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

                metric.add_batch(predictions=decoded_preds, references=decoded_labels)

            eval_metrics = metric.compute(use_stemmer=True)

            result = {key: round(val.mid.fmeasure * 100, 4) for key, val in eval_metrics.items()}
            logger.info(result)
        # endregion

        if training_args.output_dir is not None and eval_metrics is not None:
            output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
            with open(output_eval_file, "w") as writer:
                writer.write(json.dumps(eval_metrics))

        if training_args.output_dir is not None and not training_args.push_to_hub:
            # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
Here is a concise summary of the code:

The code sets up and trains a Hugging Face Transformers model for summarization. It takes in various training parameters, including dataset name, configuration, push to Hub, and output directory. It defines several callbacks for tracking progress, saving models, and logging results.

Key components include:

1. Model setup: importing necessary libraries, loading pre-trained model weights, and initializing model configuration.
2. Data preparation: loading dataset, tokenizing input/outputs, and building dataloaders.
3. Callbacks: setting up various callbacks for tracking progress (PushToHubCallback), saving models (model.save_pretrained()), logging results (KerasMetricCallback).
4. Training loop: training the model using the Transformers `fit` method, passing in training dataset and epochs.
5. Evaluation: running a standalone evaluation script after training, calculating evaluation metrics using the Hugging Face metric API.

Overall, this code sets up a complete pipeline for training a summarization model with various customizations and logging capabilities.


key: chroma:document
string_value:
CODE:

def main():
    # region Argument Parsing
    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_mlm", model_args, data_args, framework="tensorflow")

    # Sanity checks
    if data_args.dataset_name is None and data_args.train_file is None and data_args.validation_file is None:
        raise ValueError("Need either a dataset name or a training/validation file.")
    else:
        if data_args.train_file is not None:
            extension = data_args.train_file.split(".")[-1]
            assert extension in ["csv", "json", "txt"], "`train_file` should be a csv, json or txt file."
        if data_args.validation_file is not None:
            extension = data_args.validation_file.split(".")[-1]
            assert extension in ["csv", "json", "txt"], "`validation_file` should be a csv, json or txt file."

    if training_args.output_dir is not None:
        training_args.output_dir = Path(training_args.output_dir)
        os.makedirs(training_args.output_dir, exist_ok=True)

    if isinstance(training_args.strategy, tf.distribute.TPUStrategy) and not data_args.pad_to_max_length:
        logger.warning("We are training on TPU - forcing pad_to_max_length")
        data_args.pad_to_max_length = True
    # endregion

    # region Checkpoints
    # Detecting last checkpoint.
    checkpoint = None
    if len(os.listdir(training_args.output_dir)) > 0 and not training_args.overwrite_output_dir:
        config_path = training_args.output_dir / CONFIG_NAME
        weights_path = training_args.output_dir / TF2_WEIGHTS_NAME
        if config_path.is_file() and weights_path.is_file():
            checkpoint = training_args.output_dir
            logger.warning(
                f"Checkpoint detected, resuming training from checkpoint in {training_args.output_dir}. To avoid this"
                " behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )
        else:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to continue regardless."
            )

    # endregion

    # region Setup logging
    # accelerator.is_local_main_process is only True for one process per machine.
    logger.setLevel(logging.INFO)
    datasets.utils.logging.set_verbosity_warning()
    transformers.utils.logging.set_verbosity_info()
    # endregion

    # If passed along, set the training seed now.
    if training_args.seed is not None:
        set_seed(training_args.seed)

    # region Load datasets
    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        raw_datasets = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
        if "validation" not in raw_datasets.keys():
            raw_datasets["validation"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[:{data_args.validation_split_percentage}%]",
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
            raw_datasets["train"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[{data_args.validation_split_percentage}%:]",
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
        raw_datasets = load_dataset(
            extension,
            data_files=data_files,
            token=model_args.token,
        )

    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.
    # endregion

    # region Load pretrained model and tokenizer
    #
    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    if checkpoint is not None:
        config = AutoConfig.from_pretrained(
            checkpoint, token=model_args.token, trust_remote_code=model_args.trust_remote_code
        )
    elif model_args.config_name:
        config = AutoConfig.from_pretrained(
            model_args.config_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code
        )
    elif model_args.model_name_or_path:
        config = AutoConfig.from_pretrained(
            model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code
        )
    else:
        config = CONFIG_MAPPING[model_args.model_type]()
        logger.warning("You are instantiating a new config instance from scratch.")

    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.tokenizer_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code
        )
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code
        )
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script. "
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )
    # endregion

    # region Dataset preprocessing
    # First we tokenize all the texts.
    column_names = raw_datasets["train"].column_names
    text_column_name = "text" if "text" in column_names else column_names[0]

    if data_args.max_seq_length is None:
        max_seq_length = tokenizer.model_max_length
        if max_seq_length > 1024:
            logger.warning(
                f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). "
                "Picking 1024 instead. You can reduce that default value by passing --max_seq_length xxx."
            )
            max_seq_length = 1024
    else:
        if data_args.max_seq_length > tokenizer.model_max_length:
            logger.warning(
                f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the "
                f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
            )
        max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    if data_args.line_by_line:
        # When using line_by_line, we just tokenize each nonempty line.
        padding = "max_length" if data_args.pad_to_max_length else False

        def tokenize_function(examples):
            # Remove empty lines
            examples[text_column_name] = [
                line for line in examples[text_column_name] if len(line) > 0 and not line.isspace()
            ]
            return tokenizer(
                examples[text_column_name],
                padding=padding,
                truncation=True,
                max_length=max_seq_length,
                # We use this option because DataCollatorForLanguageModeling (see below) is more efficient when it
                # receives the `special_tokens_mask`.
                return_special_tokens_mask=True,
            )

        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=[text_column_name],
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on dataset line_by_line",
        )
    else:
        # Otherwise, we tokenize every text, then concatenate them together before splitting them in smaller parts.
        # We use `return_special_tokens_mask=True` because DataCollatorForLanguageModeling (see below) is more
        # efficient when it receives the `special_tokens_mask`.
        def tokenize_function(examples):
            return tokenizer(examples[text_column_name], return_special_tokens_mask=True)

        tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not data_args.overwrite_cache,
            desc="Running tokenizer on every text in dataset",
        )

        # Main data processing function that will concatenate all texts from our dataset and generate chunks of
        # max_seq_length.
        def group_texts(examples):
            # Concatenate all texts.
            concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
            total_length = len(concatenated_examples[list(examples.keys())[0]])
            # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
            # customize this part to your needs.
            if total_length >= max_seq_length:
                total_length = (total_length // max_seq_length) * max_seq_length
            # Split by chunks of max_len.
            result = {
                k: [t[i : i + max_seq_length] for i in range(0, total_length, max_seq_length)]
                for k, t in concatenated_examples.items()
            }
            return result

        # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a
        # remainder for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value
        # might be slower to preprocess.
        #
        # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:
        # https://huggingface.co/docs/datasets/process#map

        tokenized_datasets = tokenized_datasets.map(
            group_texts,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            load_from_cache_file=not data_args.overwrite_cache,
            desc=f"Grouping texts in chunks of {max_seq_length}",
        )

    train_dataset = tokenized_datasets["train"]

    if data_args.validation_file is not None:
        eval_dataset = tokenized_datasets["validation"]
    else:
        logger.info(
            f"Validation file not found: using {data_args.validation_split_percentage}% of the dataset as validation"
            " as provided in data_args"
        )
        train_indices, val_indices = train_test_split(
            list(range(len(train_dataset))), test_size=data_args.validation_split_percentage / 100
        )

        eval_dataset = train_dataset.select(val_indices)
        train_dataset = train_dataset.select(train_indices)

    if data_args.max_train_samples is not None:
        max_train_samples = min(len(train_dataset), data_args.max_train_samples)
        train_dataset = train_dataset.select(range(max_train_samples))
    if data_args.max_eval_samples is not None:
        max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
        eval_dataset = eval_dataset.select(range(max_eval_samples))

    # Log a few random samples from the training set:
    for index in random.sample(range(len(train_dataset)), min(3, len(train_dataset))):
        logger.info(f"Sample {index} of the training set: {train_dataset[index]}.")
    # endregion

    with training_args.strategy.scope():
        # region Prepare model
        if checkpoint is not None:
            model = TFAutoModelForMaskedLM.from_pretrained(
                checkpoint, config=config, token=model_args.token, trust_remote_code=model_args.trust_remote_code
            )
        elif model_args.model_name_or_path:
            model = TFAutoModelForMaskedLM.from_pretrained(
                model_args.model_name_or_path,
                config=config,
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
        else:
            logger.info("Training new model from scratch")
            model = TFAutoModelForMaskedLM.from_config(
                config, token=model_args.token, trust_remote_code=model_args.trust_remote_code
            )

        # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch
        # on a small vocab and want a smaller embedding size, remove this test.
        embeddings = model.get_input_embeddings()

        # Matt: This is a temporary workaround as we transition our models to exclusively using Keras embeddings.
        #       As soon as the transition is complete, all embeddings should be keras.Embeddings layers, and
        #       the weights will always be in embeddings.embeddings.
        if hasattr(embeddings, "embeddings"):
            embedding_size = embeddings.embeddings.shape[0]
        else:
            embedding_size = embeddings.weight.shape[0]
        if len(tokenizer) > embedding_size:
            model.resize_token_embeddings(len(tokenizer))
        # endregion

        # region TF Dataset preparation
        num_replicas = training_args.strategy.num_replicas_in_sync
        data_collator = DataCollatorForLanguageModeling(
            tokenizer=tokenizer, mlm_probability=data_args.mlm_probability, return_tensors="np"
        )
        options = tf.data.Options()
        options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF

        # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
        # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
        # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
        # yourself if you use this method, whereas they are automatically inferred from the model input names when
        # using model.prepare_tf_dataset()
        # For more info see the docs:
        # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
        # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset

        tf_train_dataset = model.prepare_tf_dataset(
            train_dataset,
            shuffle=True,
            batch_size=num_replicas * training_args.per_device_train_batch_size,
            collate_fn=data_collator,
        ).with_options(options)

        tf_eval_dataset = model.prepare_tf_dataset(
            eval_dataset,
            # labels are passed as input, as we will use the model's internal loss
            shuffle=False,
            batch_size=num_replicas * training_args.per_device_eval_batch_size,
            collate_fn=data_collator,
            drop_remainder=True,
        ).with_options(options)
        # endregion

        # region Optimizer and loss
        num_train_steps = len(tf_train_dataset) * int(training_args.num_train_epochs)
        if training_args.warmup_steps > 0:
            num_warmup_steps = training_args.warmup_steps
        elif training_args.warmup_ratio > 0:
            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
        else:
            num_warmup_steps = 0

        # Bias and layernorm weights are automatically excluded from the decay
        optimizer, lr_schedule = create_optimizer(
            init_lr=training_args.learning_rate,
            num_train_steps=num_train_steps,
            num_warmup_steps=num_warmup_steps,
            adam_beta1=training_args.adam_beta1,
            adam_beta2=training_args.adam_beta2,
            adam_epsilon=training_args.adam_epsilon,
            weight_decay_rate=training_args.weight_decay,
            adam_global_clipnorm=training_args.max_grad_norm,
        )

        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, jit_compile=training_args.xla)
        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            if data_args.dataset_name is not None:
                push_to_hub_model_id = f"{model_name}-finetuned-{data_args.dataset_name}"
            else:
                push_to_hub_model_id = f"{model_name}-finetuned-mlm"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "fill-mask"}
        if data_args.dataset_name is not None:
            model_card_kwargs["dataset_tags"] = data_args.dataset_name
            if data_args.dataset_config_name is not None:
                model_card_kwargs["dataset_args"] = data_args.dataset_config_name
                model_card_kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
            else:
                model_card_kwargs["dataset"] = data_args.dataset_name

        if training_args.push_to_hub:
            callbacks = [
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            ]
        else:
            callbacks = []
        # endregion

        # region Training and validation
        logger.info("***** Running training *****")
        logger.info(f"  Num examples = {len(train_dataset)}")
        logger.info(f"  Num Epochs = {training_args.num_train_epochs}")
        logger.info(f"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}")
        logger.info(f"  Total train batch size = {training_args.per_device_train_batch_size * num_replicas}")

        # For long training runs, you may wish to use the PushToHub() callback here to save intermediate checkpoints
        # to the Hugging Face Hub rather than just pushing the finished model.
        # See https://huggingface.co/docs/transformers/main_classes/keras_callbacks#transformers.PushToHubCallback

        history = model.fit(
            tf_train_dataset,
            validation_data=tf_eval_dataset,
            epochs=int(training_args.num_train_epochs),
            callbacks=callbacks,
        )
        train_loss = history.history["loss"][-1]
        try:
            train_perplexity = math.exp(train_loss)
        except OverflowError:
            train_perplexity = math.inf
        logger.info(f"  Final train loss: {train_loss:.3f}")
        logger.info(f"  Final train perplexity: {train_perplexity:.3f}")

    validation_loss = history.history["val_loss"][-1]
    try:
        validation_perplexity = math.exp(validation_loss)
    except OverflowError:
        validation_perplexity = math.inf
    logger.info(f"  Final validation loss: {validation_loss:.3f}")
    logger.info(f"  Final validation perplexity: {validation_perplexity:.3f}")

    if training_args.output_dir is not None:
        output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
        results_dict = {}
        results_dict["train_loss"] = train_loss
        results_dict["train_perplexity"] = train_perplexity
        results_dict["eval_loss"] = validation_loss
        results_dict["eval_perplexity"] = validation_perplexity
        with open(output_eval_file, "w") as writer:
            writer.write(json.dumps(results_dict))
        # endregion

    if training_args.output_dir is not None and not training_args.push_to_hub:
        # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
This code is for training a masked language modeling (MLM) model using the Hugging Face Transformers library. It takes in a large dataset and fine-tunes the model on it to predict the missing words in sentences.

Here's a simplified breakdown of what the code does:

1. **Data Preparation**: The code loads a dataset and splits it into training and validation sets.
2. **Model Initialization**: A new instance of the MLM model is created with the specified configuration.
3. **Optimizer and Loss Function**: An optimizer (AdamW) and loss function are defined for the model, which will update the model's parameters to minimize the loss.
4. **Training Loop**: The training loop iterates through the training data, updating the model's parameters using the optimizer. Every few epochs, the model also evaluates its performance on the validation set.
5. **Model Evaluation**: After training is complete, the final accuracy, perplexity, and other metrics are calculated for both the training and validation sets.
6. **Model Saving**: If desired, a local copy of the trained model is saved to disk.

This code can be used as a base for fine-tuning pre-trained language models on custom datasets. It provides a solid framework for experimenting with different model architectures and hyperparameters.


key: chroma:document
string_value:
CODE:

def main():
    # region Argument Parsing
    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/PyTorch versions.
    send_example_telemetry("run_clm", model_args, data_args, framework="tensorflow")

    # Sanity checks
    if data_args.dataset_name is None and data_args.train_file is None and data_args.validation_file is None:
        raise ValueError("Need either a dataset name or a training/validation file.")
    else:
        if data_args.train_file is not None:
            extension = data_args.train_file.split(".")[-1]
            assert extension in ["csv", "json", "txt"], "`train_file` should be a csv, json or txt file."
        if data_args.validation_file is not None:
            extension = data_args.validation_file.split(".")[-1]
            assert extension in ["csv", "json", "txt"], "`validation_file` should be a csv, json or txt file."

    if training_args.output_dir is not None:
        training_args.output_dir = Path(training_args.output_dir)
        os.makedirs(training_args.output_dir, exist_ok=True)
    # endregion

    # region Checkpoints
    # Detecting last checkpoint.
    checkpoint = None
    if len(os.listdir(training_args.output_dir)) > 0 and not training_args.overwrite_output_dir:
        config_path = training_args.output_dir / CONFIG_NAME
        weights_path = training_args.output_dir / TF2_WEIGHTS_NAME
        if config_path.is_file() and weights_path.is_file():
            checkpoint = training_args.output_dir
            logger.info(
                f"Checkpoint detected, resuming training from checkpoint in {training_args.output_dir}. To avoid this"
                " behavior, change the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )
        else:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to continue regardless."
            )

    # endregion

    # region Setup logging
    # accelerator.is_local_main_process is only True for one process per machine.
    logger.setLevel(logging.INFO)
    datasets.utils.logging.set_verbosity_warning()
    transformers.utils.logging.set_verbosity_info()
    # endregion

    # If passed along, set the training seed now.
    if training_args.seed is not None:
        set_seed(training_args.seed)

    # region Load datasets
    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        raw_datasets = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
        if "validation" not in raw_datasets.keys():
            raw_datasets["validation"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[:{data_args.validation_split_percentage}%]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
            raw_datasets["train"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[{data_args.validation_split_percentage}%:]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
    else:
        data_files = {}
        dataset_args = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
        extension = (
            data_args.train_file.split(".")[-1]
            if data_args.train_file is not None
            else data_args.validation_file.split(".")[-1]
        )
        if extension == "txt":
            extension = "text"
            dataset_args["keep_linebreaks"] = data_args.keep_linebreaks
        raw_datasets = load_dataset(
            extension,
            data_files=data_files,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            **dataset_args,
        )
        # If no validation data is there, validation_split_percentage will be used to divide the dataset.
        if "validation" not in raw_datasets.keys():
            raw_datasets["validation"] = load_dataset(
                extension,
                data_files=data_files,
                split=f"train[:{data_args.validation_split_percentage}%]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                **dataset_args,
            )
            raw_datasets["train"] = load_dataset(
                extension,
                data_files=data_files,
                split=f"train[{data_args.validation_split_percentage}%:]",
                cache_dir=model_args.cache_dir,
                token=model_args.token,
                **dataset_args,
            )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.
    # endregion

    # region Load pretrained model and tokenizer
    #
    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    if model_args.config_name:
        config = AutoConfig.from_pretrained(
            model_args.config_name,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    elif model_args.model_name_or_path:
        config = AutoConfig.from_pretrained(
            model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code
        )
    else:
        config = CONFIG_MAPPING[model_args.model_type]()
        logger.warning("You are instantiating a new config instance from scratch.")

    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.tokenizer_name, token=model_args.token, trust_remote_code=model_args.trust_remote_code
        )
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(
            model_args.model_name_or_path, token=model_args.token, trust_remote_code=model_args.trust_remote_code
        )
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script. "
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )
    # endregion

    # region Dataset preprocessing
    # First we tokenize all the texts.
    column_names = raw_datasets["train"].column_names
    text_column_name = "text" if "text" in column_names else column_names[0]

    def tokenize_function(examples):
        return tokenizer(examples[text_column_name])

    tokenized_datasets = raw_datasets.map(
        tokenize_function,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        remove_columns=column_names,
        load_from_cache_file=not data_args.overwrite_cache,
        desc="Running tokenizer on dataset",
    )

    if data_args.block_size is None:
        block_size = tokenizer.model_max_length
        if block_size > config.max_position_embeddings:
            logger.warning(
                f"The tokenizer picked seems to have a very large `model_max_length` ({tokenizer.model_max_length}). "
                f"Using block_size={min(1024, config.max_position_embeddings)} instead. You can change that default value by passing --block_size xxx."
            )
            block_size = min(1024, config.max_position_embeddings)
    else:
        if data_args.block_size > tokenizer.model_max_length:
            logger.warning(
                f"The block_size passed ({data_args.block_size}) is larger than the maximum length for the model "
                f"({tokenizer.model_max_length}). Using block_size={tokenizer.model_max_length}."
            )
        block_size = min(data_args.block_size, tokenizer.model_max_length)

    # Main data processing function that will concatenate all texts from our dataset and generate chunks of block_size.
    def group_texts(examples):
        # Concatenate all texts.
        concatenated_examples = {k: list(chain(*examples[k])) for k in examples.keys()}
        total_length = len(concatenated_examples[list(examples.keys())[0]])
        # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
        # customize this part to your needs.
        if total_length >= block_size:
            total_length = (total_length // block_size) * block_size
        # Split by chunks of max_len.
        result = {
            k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
            for k, t in concatenated_examples.items()
        }
        result["labels"] = result["input_ids"].copy()
        return result

    # Note that with `batched=True`, this map processes 1,000 texts together, so group_texts throws away a remainder
    # for each of those groups of 1,000 texts. You can adjust that batch_size here but a higher value might be slower
    # to preprocess.
    #
    # To speed up this part, we use multiprocessing. See the documentation of the map method for more information:
    # https://huggingface.co/docs/datasets/process#map

    lm_datasets = tokenized_datasets.map(
        group_texts,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        load_from_cache_file=not data_args.overwrite_cache,
        desc=f"Grouping texts in chunks of {block_size}",
    )

    train_dataset = lm_datasets["train"]
    if data_args.validation_file is not None:
        eval_dataset = lm_datasets["validation"]
    else:
        logger.info(
            f"Validation file not found: using {data_args.validation_split_percentage}% of the dataset as validation"
            " as provided in data_args"
        )
        train_indices, val_indices = train_test_split(
            list(range(len(train_dataset))), test_size=data_args.validation_split_percentage / 100
        )

        eval_dataset = train_dataset.select(val_indices)
        train_dataset = train_dataset.select(train_indices)

    if data_args.max_train_samples is not None:
        max_train_samples = min(len(train_dataset), data_args.max_train_samples)
        train_dataset = train_dataset.select(range(max_train_samples))
    if data_args.max_eval_samples is not None:
        max_eval_samples = min(len(eval_dataset), data_args.max_eval_samples)
        eval_dataset = eval_dataset.select(range(max_eval_samples))

    # Log a few random samples from the training set:
    for index in random.sample(range(len(train_dataset)), min(3, len(train_dataset))):
        logger.info(f"Sample {index} of the training set: {train_dataset[index]}.")
    # endregion

    with training_args.strategy.scope():
        # region Prepare model
        if checkpoint is not None:
            model = TFAutoModelForCausalLM.from_pretrained(
                checkpoint, config=config, token=model_args.token, trust_remote_code=model_args.trust_remote_code
            )
        elif model_args.model_name_or_path:
            model = TFAutoModelForCausalLM.from_pretrained(
                model_args.model_name_or_path,
                config=config,
                token=model_args.token,
                trust_remote_code=model_args.trust_remote_code,
            )
        else:
            logger.info("Training new model from scratch")
            model = TFAutoModelForCausalLM.from_config(
                config, token=model_args.token, trust_remote_code=model_args.trust_remote_code
            )

        # We resize the embeddings only when necessary to avoid index errors. If you are creating a model from scratch
        # on a small vocab and want a smaller embedding size, remove this test.
        embeddings = model.get_input_embeddings()

        # Matt: This is a temporary workaround as we transition our models to exclusively using Keras embeddings.
        #       As soon as the transition is complete, all embeddings should be keras.Embeddings layers, and
        #       the weights will always be in embeddings.embeddings.
        if hasattr(embeddings, "embeddings"):
            embedding_size = embeddings.embeddings.shape[0]
        else:
            embedding_size = embeddings.weight.shape[0]
        if len(tokenizer) > embedding_size:
            model.resize_token_embeddings(len(tokenizer))
        # endregion

        # region TF Dataset preparation
        num_replicas = training_args.strategy.num_replicas_in_sync
        options = tf.data.Options()
        options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF

        # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
        # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
        # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
        # yourself if you use this method, whereas they are automatically inferred from the model input names when
        # using model.prepare_tf_dataset()
        # For more info see the docs:
        # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
        # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset

        tf_train_dataset = model.prepare_tf_dataset(
            train_dataset,
            shuffle=True,
            batch_size=num_replicas * training_args.per_device_train_batch_size,
        ).with_options(options)

        tf_eval_dataset = model.prepare_tf_dataset(
            eval_dataset,
            shuffle=False,
            batch_size=num_replicas * training_args.per_device_eval_batch_size,
            drop_remainder=True,
        ).with_options(options)
        # endregion

        # region Optimizer and loss
        num_train_steps = len(tf_train_dataset) * int(training_args.num_train_epochs)
        if training_args.warmup_steps > 0:
            num_warmup_steps = training_args.warmup_steps
        elif training_args.warmup_ratio > 0:
            num_warmup_steps = int(num_train_steps * training_args.warmup_ratio)
        else:
            num_warmup_steps = 0

        # Bias and layernorm weights are automatically excluded from the decay
        optimizer, lr_schedule = create_optimizer(
            init_lr=training_args.learning_rate,
            num_train_steps=num_train_steps,
            num_warmup_steps=num_warmup_steps,
            adam_beta1=training_args.adam_beta1,
            adam_beta2=training_args.adam_beta2,
            adam_epsilon=training_args.adam_epsilon,
            weight_decay_rate=training_args.weight_decay,
            adam_global_clipnorm=training_args.max_grad_norm,
        )

        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, jit_compile=training_args.xla)
        # endregion

        # region Preparing push_to_hub and model card
        push_to_hub_model_id = training_args.push_to_hub_model_id
        model_name = model_args.model_name_or_path.split("/")[-1]
        if not push_to_hub_model_id:
            if data_args.dataset_name is not None:
                push_to_hub_model_id = f"{model_name}-finetuned-{data_args.dataset_name}"
            else:
                push_to_hub_model_id = f"{model_name}-finetuned-clm"

        model_card_kwargs = {"finetuned_from": model_args.model_name_or_path, "tasks": "text-generation"}
        if data_args.dataset_name is not None:
            model_card_kwargs["dataset_tags"] = data_args.dataset_name
            if data_args.dataset_config_name is not None:
                model_card_kwargs["dataset_args"] = data_args.dataset_config_name
                model_card_kwargs["dataset"] = f"{data_args.dataset_name} {data_args.dataset_config_name}"
            else:
                model_card_kwargs["dataset"] = data_args.dataset_name

        if training_args.push_to_hub:
            callbacks = [
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=tokenizer,
                    **model_card_kwargs,
                )
            ]
        else:
            callbacks = []
        # endregion

        # region Training and validation
        logger.info("***** Running training *****")
        logger.info(f"  Num examples = {len(train_dataset)}")
        logger.info(f"  Num Epochs = {training_args.num_train_epochs}")
        logger.info(f"  Instantaneous batch size per device = {training_args.per_device_train_batch_size}")
        logger.info(f"  Total train batch size = {training_args.per_device_train_batch_size * num_replicas}")

        # For long training runs, you may wish to use the PushToHub() callback here to save intermediate checkpoints
        # to the Hugging Face Hub rather than just pushing the finished model.
        # See https://huggingface.co/docs/transformers/main_classes/keras_callbacks#transformers.PushToHubCallback

        history = model.fit(
            tf_train_dataset,
            validation_data=tf_eval_dataset,
            epochs=int(training_args.num_train_epochs),
            callbacks=callbacks,
        )
        train_loss = history.history["loss"][-1]
        try:
            train_perplexity = math.exp(train_loss)
        except OverflowError:
            train_perplexity = math.inf
        logger.info(f"  Final train loss: {train_loss:.3f}")
        logger.info(f"  Final train perplexity: {train_perplexity:.3f}")
        validation_loss = history.history["val_loss"][-1]
        try:
            validation_perplexity = math.exp(validation_loss)
        except OverflowError:
            validation_perplexity = math.inf
        logger.info(f"  Final validation loss: {validation_loss:.3f}")
        logger.info(f"  Final validation perplexity: {validation_perplexity:.3f}")

        if training_args.output_dir is not None:
            output_eval_file = os.path.join(training_args.output_dir, "all_results.json")
            results_dict = {}
            results_dict["train_loss"] = train_loss
            results_dict["train_perplexity"] = train_perplexity
            results_dict["eval_loss"] = validation_loss
            results_dict["eval_perplexity"] = validation_perplexity
            with open(output_eval_file, "w") as writer:
                writer.write(json.dumps(results_dict))
        # endregion

    if training_args.output_dir is not None and not training_args.push_to_hub:
        # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
Here's a concise summary of the code:

**Training a Text Generation Model**

This is a script that trains a text generation model using the Hugging Face Transformers library. The model takes in a set of input data and generates text based on the task provided.

**Key Steps:**

1. **Data Preparation**: Load the dataset into the model.
2. **Model Initialization**: Initialize the model with the given parameters (e.g., model name, learning rate).
3. **Loss Function and Optimizer**: Define the loss function and optimizer for training.
4. **Model Compilation**: Compile the model with the chosen optimizer and loss function.
5. **Training**: Train the model on the dataset using the defined hyperparameters.
6. **Validation**: Validate the model's performance on a separate validation set.

**Key Hyperparameters:**

* Learning rate
* Number of epochs
* Batch size
* Regularization parameters (e.g., weight decay, Adam beta-values)

**Additional Features:**

* Pushing the final trained model to the Hugging Face Hub for sharing.
* Saving the evaluation results as a JSON file.

Overall, this script provides a flexible framework for training and fine-tuning text generation models using Transformers.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/tensorflow/image-classification/run_image_classification.py


key: chroma:document
string_value:
CODE:

def pil_loader(path: str):
    with open(path, "rb") as f:
        im = Image.open(f)


SUMMARY:
**pil_loader Function Summary**

* **Purpose:** Loads and converts an image to RGB format using the Python Imaging Library (PIL).

**Function Details:**

* Reads an image from the specified `path` file in binary mode (`"rb"`).
* Uses PIL's `Image.open()` function to load the image.
* Converts the loaded image to the "RGB" color space using the `convert()` method.

**External Dependencies:**
- Python Imaging Library (PIL)

No additional interfaces or classes are implemented or inherited by this function.


key: chroma:document
string_value:
CODE:

def center_crop(image, size):
    size = (size, size) if isinstance(size, int) else size
    orig_height, orig_width, _ = image.shape
    crop_height, crop_width = size
    top = (orig_height - orig_width) // 2
    left = (orig_width - crop_width) // 2
    image = tf.image.crop_to_bounding_box(image, top, left, crop_height, crop_width)


SUMMARY:
**Summary:**

This function `(center_crop)` takes an `image` and optional `size` parameters. It centers a cropped region to the specified size within the original image.

**Implementation Details:**

- **Crop Region Calculation**: The top-left coordinates are calculated based on the difference between the image dimensions and the crop size.
- **Image Cropping**: The `tf.image.crop_to_bounding_box` function is used to extract the centered cropped region from the original image.

**Notable Dependencies/Requirements:**

- `tf` (TensorFlow) for image processing functionality
- `(size, size)` or exact integer size as input, defaulting to square if an integer is provided


key: docstring
string_value:
# Numpy and TensorFlow compatible version of PyTorch RandomResizedCrop. Code adapted from:
# https://pytorch.org/vision/main/_modules/torchvision/transforms/transforms.html#RandomResizedCrop


key: chroma:document
string_value:
CODE:
# https://pytorch.org/vision/main/_modules/torchvision/transforms/transforms.html#RandomResizedCrop
def random_crop(image, scale=(0.08, 1.0), ratio=(3.0 / 4.0, 4.0 / 3.0)):
    height, width, _ = image.shape
    area = height * width
    log_ratio = np.log(ratio)
    for _ in range(10):
        target_area = np.random.uniform(*scale) * area
        aspect_ratio = np.exp(np.random.uniform(*log_ratio))
        w = int(round(np.sqrt(target_area * aspect_ratio)))
        h = int(round(np.sqrt(target_area / aspect_ratio)))
        if 0 < w <= width and 0 < h <= height:
            i = np.random.randint(0, height - h + 1)
            j = np.random.randint(0, width - w + 1)
            return image[i : i + h, j : j + w, :]

    # Fallback to central crop
    in_ratio = float(width) / float(height)
    w = width if in_ratio < min(ratio) else int(round(height * max(ratio)))
    h = height if in_ratio > max(ratio) else int(round(width / min(ratio)))
    i = (height - h) // 2
    j = (width - w) // 2


SUMMARY:
**Concise Summary**

```python
def random_crop(image, 
               scale=(0.08, 1.0), 
               ratio=(3.0 / 4.0, 4.0 / 3.0)):
    """
    Returns a randomly resized crop of the input image.
    
    Args:
        image: Input image tensor.
        scale: Tuple of (min_scale, max_scale) for area scaling.
        ratio: Tuple of (width_ratio, height_ratio) for aspect ratio.
    
    Returns:
        A cropped image tensor.
    """
```

**Implementation Details**

* The function generates a random crop by iterating 10 times through different scales and ratios to find a valid crop.
* On the 11th iteration, if no valid crop is found, it falls back to a central crop using a median aspect ratio.

**Dependencies**

* `numpy` for numerical computations
* No explicit dependencies on TensorFlow (TF) since this code snippet only defines the `random_crop` function.

**Key Insights**

* The function uses both area scaling and aspect ratio scaling for random cropping.
* It employs a fallback to central crop in case no valid crop is found after 10 iterations.


key: chroma:document
string_value:
CODE:

def random_resized_crop(image, size, scale=(0.08, 1.0), ratio=(3.0 / 4.0, 4.0 / 3.0)):
    size = (size, size) if isinstance(size, int) else size
    image = random_crop(image, scale, ratio)
    image = tf.image.resize(image, size)


SUMMARY:
**Summary:**

This function (`random_resized_crop`) generates a randomized cropped image with a specified `size`, while maintaining its aspect ratio.

**Key Details:**

* Resizes an input `image` to a random size within the specified `scale` and `ratio`.
* Utilizes TensorFlow's `tf.image.resize` to resize the image.

**Implementation Parameters:**

* `size`: The target size of the output image (int or tuple).
* `scale`: A scaling factor for randomly resizing the image (tuple).
* `ratio`: The aspect ratio constraint for the resized image (tuple).
* `random_crop`: Utilized function from TensorFlow for cropping (not shown in this code snippet).

**Notable Dependencies:**

* TensorFlow (`tf`): For image processing and randomization utilities.
* `random_crop`: A custom or external function implemented by the library.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.
    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TFTrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Sending telemetry. Tracking the example usage helps us better allocate resources to maintain them. The
    # information sent is the one passed as arguments along with your Python/TensorFlow versions.
    send_example_telemetry("run_image_classification", model_args, data_args, framework="tensorflow")

    # Checkpoints. Find the checkpoint the use when loading the model.
    checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        checkpoint = get_last_checkpoint(training_args.output_dir)
        if checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif checkpoint is not None and training_args.resume_from_checkpoint is None:
            logger.info(
                f"Checkpoint detected, resuming training at {checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    log_level = training_args.get_process_log_level()
    logger.setLevel(log_level)

    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    logger.info(f"Training/evaluation parameters {training_args}")

    # region Dataset and labels
    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Initialize our dataset and prepare it for the 'image-classification' task.
    if data_args.dataset_name is not None:
        dataset = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            cache_dir=model_args.cache_dir,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
        )
    else:
        data_files = {}
        if data_args.train_dir is not None:
            data_files["train"] = os.path.join(data_args.train_dir, "**")
        if data_args.validation_dir is not None:
            data_files["validation"] = os.path.join(data_args.validation_dir, "**")
        dataset = load_dataset(
            "imagefolder",
            data_files=data_files,
            cache_dir=model_args.cache_dir,
        )
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    # Prepare label mappings.
    # We'll include these in the model's config to get human readable labels in the Inference API.
    labels = dataset["train"].features["labels"].names
    label2id, id2label = {}, {}
    for i, label in enumerate(labels):
        label2id[label] = str(i)
        id2label[str(i)] = label

    # Load model image processor and configuration
    config = AutoConfig.from_pretrained(
        model_args.config_name or model_args.model_name_or_path,
        num_labels=len(labels),
        label2id=label2id,
        id2label=id2label,
        finetuning_task="image-classification",
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )
    image_processor = AutoImageProcessor.from_pretrained(
        model_args.image_processor_name or model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        token=model_args.token,
        trust_remote_code=model_args.trust_remote_code,
    )

    # If we don't have a validation split, split off a percentage of train as validation.
    data_args.train_val_split = None if "validation" in dataset.keys() else data_args.train_val_split
    if isinstance(data_args.train_val_split, float) and data_args.train_val_split > 0.0:
        split = dataset["train"].train_test_split(data_args.train_val_split)
        dataset["train"] = split["train"]
        dataset["validation"] = split["test"]

    # Define our data preprocessing function. It takes an image file path as input and returns
    # Write a note describing the resizing behaviour.
    if "shortest_edge" in image_processor.size:
        # We instead set the target size as (shortest_edge, shortest_edge) to here to ensure all images are batchable.
        image_size = (image_processor.size["shortest_edge"], image_processor.size["shortest_edge"])
    else:
        image_size = (image_processor.size["height"], image_processor.size["width"])

    def _train_transforms(image):
        img_size = image_size
        image = keras.utils.img_to_array(image)
        image = random_resized_crop(image, size=img_size)
        image = tf.image.random_flip_left_right(image)
        image /= 255.0
        image = (image - image_processor.image_mean) / image_processor.image_std
        image = tf.transpose(image, perm=[2, 0, 1])
        return image

    def _val_transforms(image):
        image = keras.utils.img_to_array(image)
        image = tf.image.resize(image, size=image_size)
        # image = np.array(image) # FIXME - use tf.image function
        image = center_crop(image, size=image_size)
        image /= 255.0
        image = (image - image_processor.image_mean) / image_processor.image_std
        image = tf.transpose(image, perm=[2, 0, 1])
        return image

    def train_transforms(example_batch):
        """Apply _train_transforms across a batch."""
        example_batch["pixel_values"] = [
            _train_transforms(pil_img.convert("RGB")) for pil_img in example_batch["image"]
        ]
        return example_batch

    def val_transforms(example_batch):
        """Apply _val_transforms across a batch."""
        example_batch["pixel_values"] = [_val_transforms(pil_img.convert("RGB")) for pil_img in example_batch["image"]]
        return example_batch

    train_dataset = None
    if training_args.do_train:
        if "train" not in dataset:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = dataset["train"]
        if data_args.max_train_samples is not None:
            train_dataset = train_dataset.shuffle(seed=training_args.seed).select(range(data_args.max_train_samples))
        train_dataset = train_dataset.map(
            train_transforms,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    eval_dataset = None
    if training_args.do_eval:
        if "validation" not in dataset:
            raise ValueError("--do_eval requires a validation dataset")
        eval_dataset = dataset["validation"]
        if data_args.max_eval_samples is not None:
            eval_dataset = eval_dataset.select(range(data_args.max_eval_samples))
        # Set the validation transforms
        eval_dataset = eval_dataset.map(
            val_transforms,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    predict_dataset = None
    if training_args.do_predict:
        if "test" not in dataset:
            raise ValueError("--do_predict requires a test dataset")
        predict_dataset = dataset["test"]
        if data_args.max_predict_samples is not None:
            predict_dataset = predict_dataset.select(range(data_args.max_predict_samples))
        # Set the test transforms
        predict_dataset = predict_dataset.map(
            val_transforms,
            batched=True,
            num_proc=data_args.preprocessing_num_workers,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    collate_fn = DefaultDataCollator(return_tensors="np")

    # Load the accuracy metric from the datasets package
    metric = evaluate.load("accuracy", cache_dir=model_args.cache_dir)

    # Define our compute_metrics function. It takes an `EvalPrediction` object (a namedtuple with a
    # predictions and label_ids field) and has to return a dictionary string to float.
    def compute_metrics(p):
        """Computes accuracy on a batch of predictions"""
        logits, label_ids = p
        predictions = np.argmax(logits, axis=-1)
        metrics = metric.compute(predictions=predictions, references=label_ids)
        return metrics

    with training_args.strategy.scope():
        if checkpoint is None:
            model_path = model_args.model_name_or_path
        else:
            model_path = checkpoint

        model = TFAutoModelForImageClassification.from_pretrained(
            model_path,
            config=config,
            from_pt=bool(".bin" in model_path),
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=model_args.token,
            trust_remote_code=model_args.trust_remote_code,
            ignore_mismatched_sizes=model_args.ignore_mismatched_sizes,
        )
        num_replicas = training_args.strategy.num_replicas_in_sync
        total_train_batch_size = training_args.per_device_train_batch_size * num_replicas
        total_eval_batch_size = training_args.per_device_eval_batch_size * num_replicas

        dataset_options = tf.data.Options()
        dataset_options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.OFF

        if training_args.do_train:
            num_train_steps = int(len(train_dataset) * training_args.num_train_epochs)
            if training_args.warmup_steps > 0:
                num_warmpup_steps = int(training_args.warmup_steps)
            elif training_args.warmup_ratio > 0:
                num_warmpup_steps = int(training_args.warmup_ratio * num_train_steps)
            else:
                num_warmpup_steps = 0

            optimizer, _ = create_optimizer(
                init_lr=training_args.learning_rate,
                num_train_steps=num_train_steps,
                num_warmup_steps=num_warmpup_steps,
                adam_beta1=training_args.adam_beta1,
                adam_beta2=training_args.adam_beta2,
                adam_epsilon=training_args.adam_epsilon,
                weight_decay_rate=training_args.weight_decay,
                adam_global_clipnorm=training_args.max_grad_norm,
            )
            # model.prepare_tf_dataset() wraps a Hugging Face dataset in a tf.data.Dataset which is ready to use in
            # training. This is the recommended way to use a Hugging Face dataset when training with Keras. You can also
            # use the lower-level dataset.to_tf_dataset() method, but you will have to specify things like column names
            # yourself if you use this method, whereas they are automatically inferred from the model input names when
            # using model.prepare_tf_dataset()
            # For more info see the docs:
            # https://huggingface.co/docs/transformers/main/en/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset
            # https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.to_tf_dataset
            train_dataset = model.prepare_tf_dataset(
                train_dataset,
                shuffle=True,
                batch_size=total_train_batch_size,
                collate_fn=collate_fn,
            ).with_options(dataset_options)
        else:
            optimizer = "sgd"  # Just write anything because we won't be using it

        if training_args.do_eval:
            eval_dataset = model.prepare_tf_dataset(
                eval_dataset,
                shuffle=False,
                batch_size=total_eval_batch_size,
                collate_fn=collate_fn,
            ).with_options(dataset_options)

        if training_args.do_predict:
            predict_dataset = model.prepare_tf_dataset(
                predict_dataset,
                shuffle=False,
                batch_size=total_eval_batch_size,
                collate_fn=collate_fn,
            ).with_options(dataset_options)

        # Transformers models compute the right loss for their task by default when labels are passed, and will
        # use this for training unless you specify your own loss function in compile().
        model.compile(optimizer=optimizer, jit_compile=training_args.xla, metrics=["accuracy"])

        push_to_hub_model_id = training_args.push_to_hub_model_id
        if not push_to_hub_model_id:
            model_name = model_args.model_name_or_path.split("/")[-1]
            push_to_hub_model_id = f"{model_name}-finetuned-image-classification"

        model_card_kwargs = {
            "finetuned_from": model_args.model_name_or_path,
            "tasks": "image-classification",
            "dataset": data_args.dataset_name,
            "tags": ["image-classification", "tensorflow", "vision"],
        }

        callbacks = []
        if eval_dataset is not None:
            callbacks.append(KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=eval_dataset))
        if training_args.push_to_hub:
            callbacks.append(
                PushToHubCallback(
                    output_dir=training_args.output_dir,
                    hub_model_id=push_to_hub_model_id,
                    hub_token=training_args.push_to_hub_token,
                    tokenizer=image_processor,
                    **model_card_kwargs,
                )
            )

        if training_args.do_train:
            model.fit(
                train_dataset,
                validation_data=eval_dataset,
                epochs=int(training_args.num_train_epochs),
                callbacks=callbacks,
            )

        if training_args.do_eval:
            n_eval_batches = len(eval_dataset)
            eval_predictions = model.predict(eval_dataset, steps=n_eval_batches)
            eval_labels = dataset["validation"]["labels"][: n_eval_batches * total_eval_batch_size]
            eval_metrics = compute_metrics((eval_predictions.logits, eval_labels))
            logging.info("Eval metrics:")
            for metric_name, value in eval_metrics.items():
                logging.info(f"{metric_name}: {value:.3f}")

        if training_args.output_dir is not None:
            os.makedirs(training_args.output_dir, exist_ok=True)
            with open(os.path.join(training_args.output_dir, "all_results.json"), "w") as f:
                f.write(json.dumps(eval_metrics))

        if training_args.do_predict:
            n_predict_batches = len(predict_dataset)
            test_predictions = model.predict(predict_dataset, steps=n_predict_batches)
            test_labels = dataset["validation"]["labels"][: n_predict_batches * total_eval_batch_size]
            test_metrics = compute_metrics((test_predictions.logits, test_labels))
            logging.info("Test metrics:")
            for metric_name, value in test_metrics.items():
                logging.info(f"{metric_name}: {value:.3f}")

        if training_args.output_dir is not None and not training_args.push_to_hub:
            # If we're not pushing to hub, at least save a local copy when we're done


SUMMARY:
This is a Python script that trains a Hugging Face Transformers model on an image classification task. Here's a concise summary:

1. The script sets up the training environment, including the model, optimizer, and hyperparameters.
2. It prepares the datasets for training and evaluation using `model.prepare_tf_dataset()`.
3. The model is compiled with the specified loss function, optimizer, and metrics (accuracy).
4. The model is trained on the training dataset, evaluated on the validation dataset, and predicts on the test dataset.
5. After training, the model's performance is logged, and results are saved to a JSON file if desired.

Key features:

* Uses Hugging Face Transformers for image classification
* Supports optional evaluation, prediction, and saving modes
* Allows for hyperparameter tuning and output directory customization

Note that this script assumes you have a basic understanding of PyTorch and Keras, as well as the Hugging Face Transformers library.


key: chroma:document
string_value:
CODE:

def main():
    parser = HfArgumentParser(TensorFlowBenchmarkArguments)
    benchmark_args = parser.parse_args_into_dataclasses()[0]
    benchmark = TensorFlowBenchmark(args=benchmark_args)
    try:
        benchmark_args = parser.parse_args_into_dataclasses()[0]
    except ValueError as e:
        arg_error_msg = "Arg --no_{0} is no longer used, please use --no-{0} instead."
        begin_error_msg = " ".join(str(e).split(" ")[:-1])
        full_error_msg = ""
        depreciated_args = eval(str(e).split(" ")[-1])
        wrong_args = []
        for arg in depreciated_args:
            # arg[2:] removes '--'
            if arg[2:] in TensorFlowBenchmark.deprecated_args:
                # arg[5:] removes '--no_'
                full_error_msg += arg_error_msg.format(arg[5:])
            else:
                wrong_args.append(arg)
        if len(wrong_args) > 0:
            full_error_msg = full_error_msg + begin_error_msg + str(wrong_args)
        raise ValueError(full_error_msg)


SUMMARY:
**Function Summary**

*   **Purpose**: The main function `main()` parses command-line arguments, creates a TensorFlowBenchmark instance, and runs the benchmark.
*   **Functionality**: It utilizes an `HfArgumentParser` to parse arguments into dataclasses, handles deprecated argument warnings, and executes the benchmark using the `TensorFlowBenchmark` class.

**Key Implementation Details**

*   Uses an `HfArgumentParser` with `TensorFlowBenchmarkArguments` to parse command-line arguments.
*   Handles deprecated arguments raised by Python's `eval()` function on a string of deprecated options.
*   Includes error handling for invalid argument values, raising a `ValueError` with a formatted error message.

**Notable Dependencies and Requirements**

*   `HfArgumentParser`: A class from the Hugging Face Transformers library used to parse command-line arguments into dataclasses.
*   `TensorFlowBenchmarkArguments`: A custom class or interface extended from `arguments.BaseArguments`.
*   `TensorFlowBenchmark`: The benchmarking class executed after parsing arguments.
*   `python's eval() function`: Used for parsing deprecated argument names.

**Notes**: This code assumes the availability of specific classes and interfaces (`HfArgumentParser`, `TensorFlowBenchmarkArguments`, `TensorflowBenchmark`) in the project, which may require external libraries or custom definitions.


key: chroma:document
string_value:
CODE:

def list_field(default=None, metadata=None):


SUMMARY:
**Summary**

Concats `field` function with factory method creating an instance of the same type as `default`, injecting `metadata`. This function acts as a shortcut to create and configure custom fields.

**Key Implementation Details:**

- Utilizes `field` function provided by `django-felix` library.
- Employs `factory_method` to dynamically create an instance based on the `default` value.
- Passes `metadata` directly to the parent `field()` method for configuration.

**Notable Dependencies/Requirements:**

- Requires `django-felix` library, referencing `Field`.


key: chroma:document
string_value:
CODE:

def can_convert_to_int(string):
    try:
        int(string)
        return True
    except ValueError:


SUMMARY:
**Summary:**

The `can_convert_to_int` function takes a string as input and attempts to convert it to an integer using the built-in `int()` function. If successful, it returns `True`. Otherwise, it catches the `ValueError` exception and returns `False`.

* Main purpose: Determine if a given string can be converted to an integer.
* Key implementation detail: Conversion with try-except block for handling potential `ValueError`.
* Notable dependency/interface: Built-in Python data type `int()`.

**RAG Code Summary:** 

`can_convert_to_int(str string) : bool`

`def can_convert_to_int(string):`
  `try: int(string)`
`except ValueError: return False`
`return True`


key: chroma:document
string_value:
CODE:

def can_convert_to_float(string):
    try:
        float(string)
        return True
    except ValueError:


SUMMARY:
**Summary:**

```markdown
**Function:** `can_convert_to_float`
**Purpose:** Checks if a given string can be converted to a Float.

* Attempts to convert the input string to a float using `float()`.
* Returns True (representing successful conversion) or False on failure.

## Key Implementation Details

* Uses `try-except` block to handle `ValueError` exceptions raised during floating-point conversion.

## Notable Dependencies/Requirements**

* None. This function does not import any external modules or classes.
```

**RAG (Required Action Guide) Tags:**

* Primary Purpose: Convert input string to float
* Key Implementation Details: Floating-point conversion attempt and `try-except` block handling
* Notable Dependencies: None


key: chroma:document
string_value:
CODE:

class Plot:
    def __init__(self, args):
        self.args = args
        self.result_dict = defaultdict(lambda: {"bsz": [], "seq_len": [], "result": {}})

        with open(self.args.csv_file, newline="") as csv_file:
            reader = csv.DictReader(csv_file)
            for row in reader:
                model_name = row["model"]
                self.result_dict[model_name]["bsz"].append(int(row["batch_size"]))
                self.result_dict[model_name]["seq_len"].append(int(row["sequence_length"]))
                if can_convert_to_int(row["result"]):
                    # value is not None
                    self.result_dict[model_name]["result"][(int(row["batch_size"]), int(row["sequence_length"]))] = (
                        int(row["result"])
                    )
                elif can_convert_to_float(row["result"]):
                    # value is not None
                    self.result_dict[model_name]["result"][(int(row["batch_size"]), int(row["sequence_length"]))] = (
                        float(row["result"])
                    )

    def plot(self):
        fig, ax = plt.subplots()
        title_str = "Time usage" if self.args.is_time else "Memory usage"
        title_str = title_str + " for training" if self.args.is_train else title_str + " for inference"

        if not self.args.no_log_scale:
            # set logarithm scales
            ax.set_xscale("log")
            ax.set_yscale("log")

        for axis in [ax.xaxis, ax.yaxis]:
            axis.set_major_formatter(ScalarFormatter())

        for model_name_idx, model_name in enumerate(self.result_dict.keys()):
            batch_sizes = sorted(set(self.result_dict[model_name]["bsz"]))
            sequence_lengths = sorted(set(self.result_dict[model_name]["seq_len"]))
            results = self.result_dict[model_name]["result"]

            (x_axis_array, inner_loop_array) = (
                (batch_sizes, sequence_lengths) if self.args.plot_along_batch else (sequence_lengths, batch_sizes)
            )

            label_model_name = (
                model_name if self.args.short_model_names is None else self.args.short_model_names[model_name_idx]
            )

            for inner_loop_value in inner_loop_array:
                if self.args.plot_along_batch:
                    y_axis_array = np.asarray(
                        [results[(x, inner_loop_value)] for x in x_axis_array if (x, inner_loop_value) in results],
                        dtype=int,
                    )
                else:
                    y_axis_array = np.asarray(
                        [results[(inner_loop_value, x)] for x in x_axis_array if (inner_loop_value, x) in results],
                        dtype=np.float32,
                    )

                (x_axis_label, inner_loop_label) = (
                    ("batch_size", "len") if self.args.plot_along_batch else ("in #tokens", "bsz")
                )

                x_axis_array = np.asarray(x_axis_array, int)[: len(y_axis_array)]
                plt.scatter(
                    x_axis_array, y_axis_array, label=f"{label_model_name} - {inner_loop_label}: {inner_loop_value}"
                )
                plt.plot(x_axis_array, y_axis_array, "--")

            title_str += f" {label_model_name} vs."

        title_str = title_str[:-4]
        y_axis_label = "Time in s" if self.args.is_time else "Memory in MB"

        # plot
        plt.title(title_str)
        plt.xlabel(x_axis_label)
        plt.ylabel(y_axis_label)
        plt.legend()

        if self.args.figure_png_file is not None:
            plt.savefig(self.args.figure_png_file)
        else:


SUMMARY:
**Summary:**

The `Plot` class is designed to visualize training and inference time(memory) usage patterns of different models. It takes a CSV file containing model information as input.

**Main Purpose and Functionality:**

1. Load the specified CSV file, which contains model names and their corresponding training/inference statistics (e.g., batch size, sequence length, result values).
2. Plot the result values for each model in a scatter plot, with x-axis representing one of the statistics (batch size or sequence length) and y-axis representing the other statistic and its result value.

**Key Implementation Details:**

1. The `plot` method creates two subplots if plotting along batch vs sequence length, and otherwise plots results as functions of one statistic.
2. The scatter plot is generated using the seaborn `scatter` function, with line plots connecting each point for consistency.
3. The plot includes labels, titles, and legends to explain the data points.

**Notable Dependencies and Requirements:**

1. The class uses the `defaultdict` from the `collections` module to efficiently group results by model name.
2. It relies on the pandas library through CSV file reading (`csv.DictReader`).
3. The plot generation utilizes the matplotlib library.
4. Optional dependencies include seaborn for the scatter plot function and the plt.show() function for displaying plots, as well as PIL for saving PNG files.

**Interfaces Implemented:**

1. The `Plot` class adheres to the standard plotting APIs in matplotlib (e.g., `fig`, `ax`, `plt.scatter`, etc.) and supports additional options via the matplotlib rcParams (e.g., `'no_log_scale'`)


key: chroma:document
string_value:
CODE:

def main():
    parser = HfArgumentParser(PlotArguments)
    plot_args = parser.parse_args_into_dataclasses()[0]
    plot = Plot(args=plot_args)


SUMMARY:
**Summary:**

`main()` function initializes and executes a plot generation pipeline. It uses an `HfArgumentParser` to parse command-line arguments, creates a `Plot` object with the parsed data, and calls its `plot()` method.

**Key Details:**

*   Main entry point for the application
*   Uses `HfArgumentParser` to parse command-line arguments of type `PlotArguments`
*   Creates and initializes a `Plot` object (`plot`) with a reference to the parsed argument data.
*   Calls the `plot()` method on the `Plot` object.

**Notable Dependencies/Requirements:**

*   `HfArgumentParser`: A class for parsing command-line arguments
*   `PlotArguments` : The type of arguments to be parsed by `HfArgumentParser`
*   `Plot`: A class used for generating plots, with a method `plot()` that implements the actual plotting functionality.


key: chroma:document
string_value:
CODE:

def find_backend(line: str) -> Optional[str]:
    """
    Find one (or multiple) backend in a code line of the init.

    Args:
        line (`str`): A code line in an init file.

    Returns:
        Optional[`str`]: If one (or several) backend is found, returns it. In the case of multiple backends (the line
        contains `if is_xxx_available() and `is_yyy_available()`) returns all backends joined on `_and_` (so
        `xxx_and_yyy` for instance).
    """
    if _re_test_backend.search(line) is None:
        return None
    backends = [b[0] for b in _re_backend.findall(line)]
    backends.sort()


SUMMARY:
**Summary**

This function, `find_backend`, extracts and aggregates backend names (`b[0]`) from a given code line. It uses regular expressions to search for matching patterns and then returns the found backends in sorted order, optionally joined with `_and_`.

**Key Implementation Details**

* Uses `_re_test_backend` and `_re_backend` regular expression objects to search and find backend names.
* Returns `None` if no match is found.

**Dependencies**

* `_re_test_backend`, `_re_backend`
* `Optional[str]`

**Return Value**

* A string containing one or more backend names, separated by `_and_`. 

**Primary Functionality**
Find backend names from a given code line in an init file.


key: chroma:document
string_value:
CODE:

def read_init() -> Dict[str, List[str]]:
    """
    Read the init and extract backend-specific objects.

    Returns:
        Dict[str, List[str]]: A dictionary mapping backend name to the list of object names requiring that backend.
    """
    with open(os.path.join(PATH_TO_TRANSFORMERS, "__init__.py"), "r", encoding="utf-8", newline="\n") as f:
        lines = f.readlines()

    # Get to the point we do the actual imports for type checking
    line_index = 0
    while not lines[line_index].startswith("if TYPE_CHECKING"):
        line_index += 1

    backend_specific_objects = {}
    # Go through the end of the file
    while line_index < len(lines):
        # If the line is an if is_backend_available, we grab all objects associated.
        backend = find_backend(lines[line_index])
        if backend is not None:
            while not lines[line_index].startswith("    else:"):
                line_index += 1
            line_index += 1

            objects = []
            # Until we unindent, add backend objects to the list
            while len(lines[line_index]) <= 1 or lines[line_index].startswith(" " * 8):
                line = lines[line_index]
                single_line_import_search = _re_single_line_import.search(line)
                if single_line_import_search is not None:
                    # Single-line imports
                    objects.extend(single_line_import_search.groups()[0].split(", "))
                elif line.startswith(" " * 12):
                    # Multiple-line imports (with 3 indent level)
                    objects.append(line[12:-2])
                line_index += 1

            backend_specific_objects[backend] = objects
        else:
            line_index += 1



SUMMARY:
**Function Summary: `read_init`**

* **Purpose and Functionality**: Extracts backend-specific objects from the `__init__.py` file of a Hugging Face transformer model, mapping each backend to its associated object names.
* **Key Implementation Details**:
	+ Reads and parses the entire `__init__.py` file.
	+ Searches for `if is_backend_available` statements to identify backend-specific imports.
	+ Parses multiple lines using regular expressions to extract object names from import statements.
* **Dependencies and Requirements**:
	+ Hugging Face transformer models (specifically, `PATH_TO_TRANSFORMERS`)
	+ `_re_single_line_import` regex pattern (internal function)
	+ `find_backend` function (imported but not explicitly shown in this code snippet)
* **Interfaces/Inherited:** This function inherits from no specific interface.

**Notes:**

* The function returns a dictionary with the format `{backend_name: [object_names]}`.
* The `_re_single_line_import` regex pattern is used to extract object names from single-line import statements.


key: chroma:document
string_value:
CODE:

def create_dummy_object(name: str, backend_name: str) -> str:
    """
    Create the code for a dummy object.

    Args:
        name (`str`): The name of the object.
        backend_name (`str`): The name of the backend required for that object.

    Returns:
        `str`: The code of the dummy object.
    """
    if name.isupper():
        return DUMMY_CONSTANT.format(name)
    elif name.islower():
        return DUMMY_FUNCTION.format(name, backend_name)
    else:


SUMMARY:
**Function Summary**

`create_dummy_object`: Returns code for a dummy object of specified type based on input `name`. The type is determined by the casing of `name`.

* **Purpose**: Generate code for a dummy object with given name and backend requirement.
* **Implementation Details**:
	+ Uses format strings (`DUMMY_CONSTANT`, `DUMMY_FUNCTION`, `DUMMY_CLASS`) to generate code.
	+ Includes `backend_name` in generated code when `name` is not all upper case or lower case.
* **Notable Dependencies**: Uses string formatting (`format`) and likely inherited from a parent class/inheritance hierarchy containing `DUMMY_CONSTANT`, `DUMMY_FUNCTION`, and `DUMMY_CLASS`. 

**Key Interfaces/Methods:**

- `DUMMY_CONSTANT`
- `DUMMY FUNCTION`
- `DUMMY CLASS`

**External Dependencies:** None


key: chroma:document
string_value:
CODE:

def create_dummy_files(backend_specific_objects: Optional[Dict[str, List[str]]] = None) -> Dict[str, str]:
    """
    Create the content of the dummy files.

    Args:
        backend_specific_objects (`Dict[str, List[str]]`, *optional*):
            The mapping backend name to list of backend-specific objects. If not passed, will be obtained by calling
            `read_init()`.

    Returns:
        `Dict[str, str]`: A dictionary mapping backend name to code of the corresponding backend file.
    """
    if backend_specific_objects is None:
        backend_specific_objects = read_init()

    dummy_files = {}

    for backend, objects in backend_specific_objects.items():
        backend_name = "[" + ", ".join(f'"{b}"' for b in backend.split("_and_")) + "]"
        dummy_file = "# This file is autogenerated by the command `make fix-copies`, do not edit.\n"
        dummy_file += "from ..utils import DummyObject, requires_backends\n\n"
        dummy_file += "\n".join([create_dummy_object(o, backend_name) for o in objects])
        dummy_files[backend] = dummy_file



SUMMARY:
**Function Summary: create_dummy_files**

**Purpose**: Creates the content of dummy files based on backend-specific objects.

**Functionality**

* Takes an optional dictionary `backend_specific_objects` mapping backend names to lists of backend-specific objects.
* If none provided, reads them from `read_init()`.
* Creates a dictionary `dummy_files` with backend names as keys and their corresponding code as values.
* For each backend:
	+ Constructs the code by concatenating object imports and calls to `create_dummy_object` method.
	+ Prefers formatted output for list of backend objects.

**Key Implementation Details**

* Uses `read_init()` to retrieve backend-specific objects when not provided.
* Employed string formatting to nicely display list of backend objects.

**Notable Dependencies**

* `read_init()`
* `create_dummy_object()
* `DummyObject` class from `..utils` module
* External dependencies: requires `make` command and directory structure


key: chroma:document
string_value:
CODE:

def check_dummies(overwrite: bool = False):
    """
    Check if the dummy files are up to date and maybe `overwrite` with the right content.

    Args:
        overwrite (`bool`, *optional*, default to `False`):
            Whether or not to overwrite the content of the dummy files. Will raise an error if they are not up to date
            when `overwrite=False`.
    """
    dummy_files = create_dummy_files()
    # For special correspondence backend name to shortcut as used in utils/dummy_xxx_objects.py
    short_names = {"torch": "pt"}

    # Locate actual dummy modules and read their content.
    path = os.path.join(PATH_TO_TRANSFORMERS, "utils")
    dummy_file_paths = {
        backend: os.path.join(path, f"dummy_{short_names.get(backend, backend)}_objects.py")
        for backend in dummy_files.keys()
    }

    actual_dummies = {}
    for backend, file_path in dummy_file_paths.items():
        if os.path.isfile(file_path):
            with open(file_path, "r", encoding="utf-8", newline="\n") as f:
                actual_dummies[backend] = f.read()
        else:
            actual_dummies[backend] = ""

    # Compare actual with what they should be.
    for backend in dummy_files.keys():
        if dummy_files[backend] != actual_dummies[backend]:
            if overwrite:
                print(
                    f"Updating transformers.utils.dummy_{short_names.get(backend, backend)}_objects.py as the main "
                    "__init__ has new objects."
                )
                with open(dummy_file_paths[backend], "w", encoding="utf-8", newline="\n") as f:
                    f.write(dummy_files[backend])
            else:
                raise ValueError(
                    "The main __init__ has objects that are not present in "
                    f"transformers.utils.dummy_{short_names.get(backend, backend)}_objects.py. Run `make fix-copies` "
                    "to fix this."


SUMMARY:
**Function Summary: check_dummies**

**Purpose:** Verify the integrity of dummy files and optionally update their content.

**Functionality:**

1. Checks if dummy file paths exist and reads their contents.
2. Compares actual contents with expected ones. If updated, updates the file.
3. Raises `ValueError` if dummy files are outdated (non-present) when `overwrite=False`.

**Key Implementation Details:**

* Uses `create_dummy_files()` to generate dictionary of dummy files.
* Uses a mapping (`short_names`) for shortcutting backend names.

**Notable Dependencies and Requirements:**

* `utils/dummy_xxx_objects.py` file
* `os` module (for interacting with the file system)
* `pathlib` (not explicitly mentioned, but implied by `PATH_TO_TRANSFORMERS`)
* `make` command (to run `fix-copies` to update outdated dummy files)


key: chroma:document
string_value:
CODE:

def get_module_source_from_name(module_name: str) -> str:
    # Extract the source code from the module name
    spec = importlib.util.find_spec(module_name)
    if spec is None or spec.origin is None:
        raise ValueError(f"Cannot open file associated with {module_name} module.")

    with open(spec.origin, "r", encoding="utf-8") as file:
        source_code = file.read()


SUMMARY:
**Summary**

`get_module_source_from_name`: Extracts(source code) from a given Python module name (`module_name`). 

### Key Details

* Uses `importlib.util.find_spec` to locate the module spec.
* Returns the read contents of the module's associated file.

### Implementation Notes

* Checks for existence and validity of the module spec before proceeding.
* Raises a `ValueError` if the file associated with the module is not found or cannot be opened.
* reads the source code from the file using UTF-8 encoding.

### Dependencies and Interfaces

* `importlib.util`: Module to locate Python modules.
* `python files/utf-8 encoding (`encoding="utf-8"`): Specifies character encoding for reading the file.


key: chroma:document
string_value:
CODE:

def preserve_case_replace(text, patterns: dict, default_name: str):
    # Create a regex pattern to match all variations
    regex_pattern = "|".join(re.escape(key) for key in patterns.keys())
    compiled_regex = re.compile(f"(?<![a-z0-9])({regex_pattern})(.|$)", re.IGNORECASE | re.DOTALL)

    def replace(match):
        matched_pattern = match.group(1)
        next_char = match.group(2)
        new_pattern = patterns.get(matched_pattern, default_name)

        # In this case, the cased old model did not respect CamelCase and was all UPPERCASE, so we need to rely on next char
        # The heuristic is: if next char is not a letter, then it is not part of a model name and result should be `new_name`.upper()
        if len(patterns) == 2 and matched_pattern.isupper():
            if not next_char.isalpha():
                # `new_name.upper()` is just the other entry for `matched_pattern.lower()`, uppercased
                new_pattern = patterns[matched_pattern.lower()].upper()

        return new_pattern + next_char



SUMMARY:
**Function Summary: preserve_case_replace**

Preserves sentence case in a given `text` by replacing matched patterns with corresponding names from the `patterns` dictionary. The function uses a regex pattern to match all variations of model names.

**Key Functionality:**

*   Preserves original text while replacing matched patterns.
*   Replaces patterns based on the order and key-value pairs in `patterns`.
*   Uses case-insensitive matching for non-alphanumeric characters.

**Important Implementation Details:**

*   Regex pattern `(?<![a-z0-9])({regex_pattern})(.|$)` matches all variations of model names while preserving sentence case.
*   The `replace` function uses the matched pattern, next character, and corresponding name from `patterns` to replace the original text.

**Notable Dependencies:**

*   `re` module for regex operations.
*   Python's built-in string methods (e.g., `isalpha()`, `upper()`).

**External Dependencies:**

*   None specified in the provided code chunk. However, it can be assumed that any library or framework being used by the target project may also require this function to function appropriately.


key: chroma:document
string_value:
CODE:

def get_cased_name(lowercase_name: str) -> str:
    """From a model name in lowercase in the format `my_model`, return the cased name in the format `MyModel`."""
    if lowercase_name in CONFIG_MAPPING_NAMES:
        return CONFIG_MAPPING_NAMES[lowercase_name].replace("Config", "")
    else:


SUMMARY:
**Summary:**

This function, `get_cased_name`, takes a lowercase model name as input and returns the cased version. If the input matches a predefined mapping, it replaces "Config" with an empty string; otherwise, it applies title case to the split and joined components.

**Key Implementation Details:**

* Replaces "Config" for matched mappings
* Applies title case using `str.title()` for unmatched inputs

**Dependencies/Interfaces:**

* `CONFIG_MAPPING_NAMES`: a dictionary-like data structure (not defined in this code snippet)

**Notable Method Signatures:**

* `get_cased_name(lowercase_name: str) -> str`


key: chroma:document
string_value:
CODE:

def get_lowercase_name(cased_name: str) -> str:
    """From a model name in Camelcase in the format `MyModel`, return the lowercase name in the format `my_model`."""
    inverse_mapping = {value: key for key, value in CONFIG_MAPPING_NAMES.items()}
    if cased_name + "Config" in inverse_mapping:
        return inverse_mapping[cased_name + "Config"]
    else:


SUMMARY:
**Function Summary:**

`get_lowercase_name`

* **Purpose:** Convert a Camelcase name to lowercase and format according to the ` CONFIG_MAPPING_NAMES`.
* **Functionality:** 
	+ Checks if a Config variant exists in the inverse mapping for the given Camelcase name.
	+ If not, returns the lowered Camelcase name by splitting into words and joining with underscores.

**Key Implementation Details:**

* Uses regular expressions (`re.findall`) to extract words from the Camelcase input string.
* Applies `str.lower()` to each word before rejoining them with underscores.

**Notable Dependencies or Requirements:**

* `CONFIG_MAPPING_NAMES` (injected as a constant)
* `re` (regular expression module)


key: chroma:document
string_value:
CODE:

class ReplaceNameTransformer(m.MatcherDecoratableTransformer):
    """A transformer that replaces `old_name` with `new_name` in comments, string and any references.
    It should take into account name like `MyNewModel`, or `my_new_model`. Without using the AUTO_MAPPING.
    Supported renaming patterns:
        - llama -> my_new_model     and     my_new_model    -> llama
        - Llama -> MyNewModel       and     MyNewModel      -> Llama
        - LLAMA -> MY_NEW_MODEL     and     MY_NEW_MODEL    -> LLAMA
        - LLaMa -> MyNewModel       abd     MyNewModel      -> Llama
    """

    def __init__(self, old_name: str, new_name: str, original_new_model_name: str = "", only_doc: bool = False):
        super().__init__()
        self.old_name = old_name
        self.new_name = new_name
        self.cased_new_name = get_cased_name(self.new_name)
        self.cased_old_name = get_cased_name(self.old_name)
        self.patterns = {
            old_name: new_name,
            old_name.upper(): new_name.upper(),
            # For some old models, `self.cased_old_name` == `old_name.upper()` in which case this overwrite previous entry
            self.cased_old_name: self.cased_new_name,
        }
        # In case new_name is a prefix alias, and not the original new model name
        self.original_new_model_name = original_new_model_name
        self.only_doc = only_doc

    def _replace_name(self, original_node, updated_node):
        if re.findall(r"# Copied from", updated_node.value):
            return cst.RemoveFromParent()
        update = preserve_case_replace(updated_node.value, self.patterns, self.cased_new_name)
        return updated_node.with_changes(value=update)

    @m.leave(m.SimpleString() | m.Comment())
    def replace_name(self, original_node, updated_node):
        return self._replace_name(original_node, updated_node)

    def leave_Name(self, original_node, updated_node):
        if not self.only_doc:
            return self._replace_name(original_node, updated_node)
        return updated_node

    def leave_ImportFrom(self, original_node, updated_node):
        """The imports from other file types (configuration, processing etc) should use original model name."""
        if self.original_new_model_name != self.new_name and m.matches(updated_node.module, m.Name()):
            patterns = "|".join(ALL_FILE_TYPES)
            regex = rf"({patterns})_{self.new_name}"
            new_source = re.sub(
                regex, lambda m: f"{m.group(1)}_{self.original_new_model_name}", updated_node.module.value
            )
            updated_node = updated_node.with_changes(module=updated_node.module.with_changes(value=new_source))


SUMMARY:
**Summary**

The `ReplaceNameTransformer` class is designed to replace specific names in comments, strings, and references. It handles various naming patterns, considering case sensitivity and prefixes/qualifiers.

**Key Functionality:**

1.  Replaces `old_name` with `new_name` in:
    *   Comments
    *   Strings
    *   References

2.  Supports different renaming patterns:

    *   `llama -> my_new_model`
    *   `Llama -> MyNewModel`
    *   `LLAMA -> MY_NEW_MODEL`
    *   `(prefix)_LLaMa_(suffix)`

**Implementation Details:**

*   `_replace_name` method applies the replacement rules to an updated node.

*   The transformer considers the case of the original name and uses a `patterns` dictionary for replacements.
*   It preserves the original meaning by allowing certain suffixes (like `.py`) to remain unchanged.

**Dependencies/Notable Requirements:**

*   Uses Python's `re` module for regular expressions
*   Leverages the `cst` library for parsing nodes

Here's a concise summary of key classes and functions:

*   `class ReplaceNameTransformer(m.MatcherDecoratableTransformer):`

    *   Initializes with `old_name`, `new_name`, optional `original_new_model_name`, and `only_doc`.
    *   Provides `replace_name`, `leave_Name` and `leave_ImportFrom` methods to apply the transformer transformations

The full implementation showcases how this transformer can be used to standardize naming conventions within a larger system.


key: chroma:document
string_value:
CODE:

def SUPER_CALL_NODE(func_name):


SUMMARY:
**Summary:**

The `SUPER_CALL_NODE` function generates an abstract syntax tree (AST) node for the `super()` call in Python.

**Key Implementation Details:**

* The `m.Call` node represents the call operation with its attributes.
* The `attr` attribute within the `Call` node specifies the `func_name` parameter, which is passed from the caller function.
* A nested `m.Call` node represents the built-in `super()` function and has a specific structure, calling another inner `m.Name` node named `"super"`.
* Both nodes rely on another higher-level node `m.Attribute`, indicating method calls.

**Notable Dependencies/Requirements:**

* The function uses the `ast` module (implied but not shown) to create an abstract syntax tree.
* It relies on a meta-programming system, possibly a Python parser generator like Parsec or PyParsing.


key: chroma:document
string_value:
CODE:

def is_call_to_super(node, func_name):
    return m.matches(
        node, m.SimpleStatementLine(body=[m.Return(SUPER_CALL_NODE(func_name)) | m.Expr(SUPER_CALL_NODE(func_name))])


SUMMARY:
**Summary:**

`is_call_to_super` function checks if the given `node` represents a call to the superclass (__Super__) from a method named `func_name`. 

**Key Implementation Details:**
- Uses regular expression (`m.matches`) to match a specific pattern in the node.
- Expects a match with either a return statement calling `SUPER_CALL_NODE(func_name)` or an expression also calling `SUPER_CALL_NODE(func_name)`.

**Notable Dependencies/Requirements:**

- `SuperCallNode` is assumed to be a previously defined class or function.
- External dependency on regular expression engine (`m`) using `m.matches`. 

Note: Without additional context, the exact nature of `SUPER_CALL_NODE(func_name)`, `m`, and related classes/functions cannot be precisely determined.


key: chroma:document
string_value:
CODE:

def get_full_attribute_name(node: cst.Attribute | cst.Name) -> str | None:
    """Get the full name of an Attribute or Name node (e.g. `"nn.Module"` for an Attribute representing it). If the
    successive value of an Attribute are not Name nodes, return `None`."""
    if m.matches(node, m.Name()):
        return node.value
    elif m.matches(node, m.Attribute()):
        if not m.matches(node.attr, m.Name()):
            return None
        name = node.attr.value
        new_node = node.value
        while m.matches(new_node, m.Attribute()):
            if not m.matches(new_node.attr, m.Name()):
                return None
            name = new_node.attr.value + "." + name
            new_node = new_node.value
        if not m.matches(new_node, m.Name()):
            return None
        return new_node.value + "." + name


SUMMARY:
**Summary**

`get_full_attribute_name` function extracts the full attribute name from an AST node (either `Attribute` or `Name`). It traverses attributes and checks if successive values are `Name` nodes. If successful, it returns the fully qualified name.

**Key Details**

* Function takes an `AST` node (`cst.Attribute | cst.Name`) as input
* Returns a string representing the full attribute name or `None` if incomplete
* Uses a regular expression (m) to match node types

**Implementation Choices**

* Uses recursive descent approach for traversing attributes
* Employed a try-and-contract approach to handle cases with incomplete successive values


key: docstring
string_value:
# Transformer class to replace ClassB.call_to_method and ClassB().call_to_method with super().call_to_method


key: chroma:document
string_value:
CODE:
# Transformer class to replace ClassB.call_to_method and ClassB().call_to_method with super().call_to_method
class ReplaceMethodCallTransformer(cst.CSTTransformer):
    def __init__(self, all_bases: Set[str]):
        self.all_bases = all_bases

    def leave_Attribute(self, original_node: cst.Attribute, updated_node: cst.Attribute) -> cst.CSTNode:
        # Handle ClassB.call_to_method or module.classB.call_to_method
        if (
            m.matches(original_node.value, m.Name() | m.Attribute())
            and get_full_attribute_name(original_node.value) in self.all_bases
            and m.matches(original_node.attr, m.Name())
        ):
            # Replace with super().call_to_method
            return updated_node.with_changes(
                value=cst.Call(cst.Name("super")),
            )
        # Handle ClassB().call_to_method or module.ClassB().call_to_method
        elif (
            m.matches(original_node.value, m.Call())
            and m.matches(original_node.value.func, m.Name() | m.Attribute())
            and get_full_attribute_name(original_node.value.func) in self.all_bases
            and m.matches(original_node.attr, m.Name())
        ):
            # Replace with super().call_to_method
            return updated_node.with_changes(value=cst.Call(cst.Name("super")))
        return updated_node

    def leave_Call(self, original_node: cst.Call, updated_node: cst.Call) -> cst.CSTNode:
        # Check if the function being called is of the form ClassB().func_a or ClassB.func_a
        if m.matches(original_node.func, m.Attribute()) and (
            # Match ClassB().func_a(...) or module
            (
                m.matches(original_node.func.value, m.Call())
                and m.matches(original_node.func.value.func, m.Name() | m.Attribute())
                and get_full_attribute_name(original_node.func.value.func) in self.all_bases
            )
            or
            # Match ClassB.func_a(...)
            (
                m.matches(original_node.func.value, m.Name() | m.Attribute())
                and get_full_attribute_name(original_node.func.value) in self.all_bases
            )
        ):
            # Check if the first argument is 'self', and remove it
            if len(original_node.args) > 0 and m.matches(original_node.args[0].value, m.Name("self")):
                # Create the new argument list without 'self'
                new_args = updated_node.args[1:]
            else:
                new_args = updated_node.args

            return updated_node.with_changes(args=new_args)


SUMMARY:
**Class Summary**

`ReplaceMethodCallTransformer` is a CST transformer that replaces `ClassB.call_to_method` and `ClassB().call_to_method` with `super().call_to_method`.

**Primary Purpose**

The main purpose of this class is to transform Python code by replacing method calls in the specified formats with the equivalent Super class call.

**Key Implementation Details**

1. **leave_Attribute**: This function handles case where attribute name is present as a string (`ClassB.call_to_method`). It checks if the attribute name is in `all_bases` and replaces it with `super().call_to_method`.
2. **leave_Call**: This function handles cases where the method call contains an instance of `(module)` or only the class method call (e.g. `ClassB.func_a`). It checks for the presence of 'self' as an argument, removes it if present and updates the arguments of the function call.

**Notable Dependencies/Requirements**

* `cst` module for working with Python abstract syntax trees (ASTs)
* `Set[str]` for storing class names
* `get_full_attribute_name` and `m.matches` functions appear to be custom utility functions not explicitly included in the provided code.


key: chroma:document
string_value:
CODE:

def get_docstring_indent(docstring):
    # Match the first line after the opening triple quotes
    match = re.search(r'(?:"""|\'\'\'|```)\n(\s+)', docstring)
    if match:
        # Return the indentation spaces captured
        return len(match.group(1))


SUMMARY:
**Function Summary**

```
get_docstring_indent function
=============================

Returns the number of indentation spaces preceding the first line after triple quotes in a docstring.

### Purpose

Extracts and returns the indentation level from the matched opening docstring triplet.

### Implementation Details

- Uses regular expression to match the opening triplet and capture the following whitespace content.
- Returns the length of the captured whitespace as the indentation distance.

### Dependency

None
```


key: chroma:document
string_value:
CODE:

def merge_docstrings(original_docstring, updated_docstring):
    # indent_level = get_docstring_indent(updated_docstring)
    original_level = get_docstring_indent(original_docstring)
    if not re.findall(r"\n\s*Args:\n", updated_docstring):
        # Split the docstring at the example section, assuming `"""` is used to define the docstring
        parts = original_docstring.split("```")
        if "```" in updated_docstring and len(parts) > 1:
            updated_docstring = updated_docstring.lstrip('r"')
            new_parts = updated_docstring.split("```")
            if len(new_parts) != 3:
                raise ValueError("There should only be one example, and it should have opening and closing '```'")
            parts[1] = new_parts[1]
            updated_docstring = "".join(
                [
                    parts[0].rstrip(" \n") + new_parts[0],
                    f"\n{original_level*' '}```",
                    parts[1],
                    "```",
                    parts[2],
                ]
            )
        elif updated_docstring not in original_docstring:
            # add tabulation if we are at the lowest level.
            if re.search(r"\n\s*.*\(.*\)\:\n\s*\w", updated_docstring):
                updated_docstring = updated_docstring.replace("\n    ", "\n        ")
            updated_docstring = original_docstring.rstrip('"') + "\n" + updated_docstring.lstrip('r"\n')


SUMMARY:
**Summary**

The `merge_docstrings` function merges two docstrings, aiming to blend the style of both into a single cohesive doc string. Specifically:

### Main Purpose and Functionality

* Merges two doc strings (`original_docstring` and `updated_docstring`) into one.
* Preserves indentation structure from `original_docstring`.
* Updates `Args:` section with new content from `updated_docstring`.

### Key Implementation Details

* Uses regular expression to detect and replace `Args:` format in `updated_docstring`.
* Corrects inconsistent indentation by tabulation.
* Handles different cases of doc string formatting, including single-step and multi-step examples.

### Notable Dependencies or Requirements

* Utilizes string manipulation functions (e.g., `rstrip`, `lstrip`, `split`) and regular expressions (e.g., `re.findall`, `re.search`).
* Assumes the use of triple quoting (`"""`) for defining docstrings.
* Expected input: Two doc strings, with an optional third example section using triple quotes.


key: chroma:document
string_value:
CODE:

class SuperTransformer(cst.CSTTransformer):
    METADATA_DEPENDENCIES = (ParentNodeProvider,)

    def __init__(self, python_module: cst.Module, original_methods, updated_methods, all_bases=None):
        self.python_module = python_module
        self.original_methods = original_methods
        self.updated_methods = updated_methods
        self.all_assign_target = {}
        self.deleted_targets = {}  # child node can delete some arguments
        self.all_bases = all_bases or []
        self.transformer = ReplaceMethodCallTransformer(set(self.all_bases))

    def update_body(self, existing_body, new_statements):
        """
        Helper method to update the body by removing duplicates before adding new statements.
        `existing_body` is the body of the original method, the parent class
        `new_statements` are the additional statements
        """
        deduplicated_new_body = []
        existing_nodes = set()
        for node in new_statements:
            if m.matches(node, m.SimpleStatementLine(body=[m.Assign()])):
                target = self.python_module.code_for_node(node.body[0].targets[0].target)
                self.all_assign_target[target] = node
            if m.matches(node, m.SimpleStatementLine(body=[m.Del()])):
                target = self.python_module.code_for_node(node.body[0].target)
                self.deleted_targets[target] = node

        for stmt in existing_body:
            if m.matches(stmt, m.SimpleStatementLine(body=[m.Assign()])):
                target = self.python_module.code_for_node(stmt.body[0].targets[0].target)
                if target in self.deleted_targets:
                    continue
                if target in self.all_assign_target:
                    stmt = self.all_assign_target[target]
            # Skip the docstring (will be added later on, at the beginning)
            elif m.matches(stmt, DOCSTRING_NODE):
                continue
            comment_less_code = re.sub(r"#.*", "", self.python_module.code_for_node(stmt)).strip()
            comment_less_code = re.sub(r"\ *\n", "\n", comment_less_code).strip()
            deduplicated_new_body.append(stmt)
            existing_nodes.add(comment_less_code)

        for node in new_statements:
            code = self.python_module.code_for_node(node)
            comment_less_code = re.sub(r"#.*", "", code).strip()
            comment_less_code = re.sub(r"\ *\n", "\n", comment_less_code).strip()
            if node not in deduplicated_new_body and comment_less_code not in existing_nodes:
                if not m.matches(node, m.SimpleStatementLine(body=[m.Del()])):
                    deduplicated_new_body.append(node)
                    existing_nodes.add(comment_less_code)

        deduplicated_new_body = self._fix_post_init_location(deduplicated_new_body)

        return deduplicated_new_body

    def _fix_post_init_location(self, new_body: list[cst.CSTNode]):
        """Fix the location of the `post_init()` in the new body, if we added statements after the call to
        `super()` (it needs to be the very last statement called)"""
        # Fix the post_init() that has to be last
        for i, node in enumerate(new_body):
            code = self.python_module.code_for_node(node)
            comment_less_code = re.sub(r"#.*", "", code).strip()
            comment_less_code = re.sub(r"\ *\n", "\n", comment_less_code).strip()
            if "self.post_init(" in comment_less_code and i < len(new_body) - 1:
                # Remove it and add it again at the end
                new_body.pop(i)
                new_body.append(node)
                break
        return new_body

    def _fix_init_location(self, new_body):
        """Fix the location of the `super().__init__()` in the new body, if we had new statements before it."""
        start_index = 0
        for i, node in enumerate(new_body):
            if m.matches(node, DOCSTRING_NODE) and i == start_index:
                start_index += 1
                continue
            code = self.python_module.code_for_node(node)
            comment_less_code = re.sub(r"#.*", "", code).strip()
            comment_less_code = re.sub(r"\ *\n", "\n", comment_less_code).strip()
            if "super().__init__" in comment_less_code and i > start_index:
                # Remove it and add it again at the top after the docstrings
                node = new_body.pop(i)
                new_body = new_body[:start_index] + [node] + new_body[start_index:]
                break
        return new_body

    def replace_super_calls(self, node: cst.IndentedBlock, func_name: str) -> cst.CSTNode:
        """Updates the body of the input `node`'s `func_name` function by replacing calls
        to super().func_name() with the source code of the parent class' `func_name`.
        It keeps everything that is defined before `super().func_name()`.
        """
        self.has_docstring = False
        parent_has_docstring = False
        if func_name in self.original_methods:
            parent_has_docstring = m.matches(self.original_methods[func_name].body.body[0], DOCSTRING_NODE)
        new_body = []
        has_super_call = False

        for i, expr in enumerate(node.body):
            if is_call_to_super(expr, func_name):
                has_super_call = True
                new_body.extend(self.update_body(self.original_methods[func_name].body.body, node.body[i + 1 :]))
                new_body = self._fix_init_location(new_body)
            else:
                expr = expr.visit(self.transformer)
            if m.matches(expr, DOCSTRING_NODE):
                self.has_docstring = True
                if parent_has_docstring:  # actually here we ought to de-duplicate?
                    original_docstring = self.original_methods[func_name].body.body[0].body[0].value.value
                    updated_docstring = expr.body[0].value.value
                    merged_doc = merge_docstrings(original_docstring, updated_docstring)
                    new_node = [expr.with_changes(body=[cst.Expr(value=cst.SimpleString(value=merged_doc))])]
                else:
                    new_node = [expr]
                new_body.extend(new_node)
            elif not m.matches(expr, m.SimpleStatementLine(body=[m.Del()])) and not has_super_call:
                new_body.append(expr)
        if not self.has_docstring and parent_has_docstring:
            new_body = [self.original_methods[func_name].body.body[0]] + new_body
        return node.with_changes(body=new_body)

    def leave_FunctionDef(self, original_node: cst.Call, updated_node: cst.Call) -> cst.CSTNode:
        if updated_node.name.value in self.updated_methods:
            name = updated_node.name.value
            new_body = self.replace_super_calls(updated_node.body, name)
            return updated_node.with_changes(body=new_body, params=updated_node.params)
        return updated_node

    def leave_Return(self, original_node: cst.Return, updated_node: cst.Return) -> cst.CSTNode:
        """ "When a return statement is reached, it is replaced with the unrolled super code"""
        if m.matches(updated_node.value, m.Call(func=m.Attribute(attr=m.Name("super")))):
            func_def = self.get_metadata(ParentNodeProvider, original_node)
            if m.matched(func_def, m.FunctionDef()) and func_def.name.value in self.original_methods:
                updated_return_value = updated_node.value.with_changes(
                    args=[
                        cst.Arg(
                            value=cst.Call(func=cst.Name("super"), args=[cst.Arg(value=cst.Name(func_def.name.value))])
                        )
                    ]
                )
                return updated_node.with_changes(value=updated_return_value)


SUMMARY:
**Summary:**

This class `SuperTransformer` is designed to transform Python code that references `super()` calls. It achieves this by:

*   **Replacing `super()` calls**: Replaces calls to `super()` with the source code of the parent class's method implementation.
*   **De-duplicating docstrings**: Merges overlapping docstrings from different sources in a unified way.

**Key Implementation Details:**

1.  **Main Functionality**: `replace_super_calls` function
    *   Updates the body of a function node by replacing calls to `super()` with the source code of the parent class's method implementation.
2.  **Docstring Handling**
    *   Uses separate functions `_fix_post_init_location` and `_fix_init_location` to handle docstrings in different locations (at the end, or at the top).
3.  **Visitor Pattern**: Utilizes a visitor pattern (`visit`) to transform AST nodes.

**Notable Dependencies:**

1.  `ParentNodeProvider`
2.  `m` (a module for parsing and manipulating code)
3.  `DOCSTRING_NODE` (a node representing docstrings in the AST)


key: chroma:document
string_value:
CODE:

def find_all_dependencies(
    dependency_mapping: Dict[str, set],
    start_entity: str | None = None,
    initial_dependencies: set | None = None,
    initial_checked_dependencies: set | None = None,
    return_parent: bool = False,
) -> list | set:
    """Return all the dependencies of the given `start_entity` or `initial_dependencies`. This is basically some kind of
    BFS traversal algorithm. It can either start from `start_entity`, or `initial_dependencies`.

    Args:
        dependency_mapping (`Dict[str, set]`):
            A mapping from entities (usually function/assignment names), to immediate dependencies. That is, for function names,
            a mapping {"foo": {"bar", "test"}} would indicate that functions `bar` and `test` are immediately called
            in `foo`'s definition.
        start_entity (str | None, *optional*):
            A key of `dependency_mapping`, indicating from which entity to start the search.
        initial_dependencies (set | None, *optional*):
            If `start_entity` is not provided, this can be used as an alternative. In this case, the search will continue
            from all the entities in `initial_dependencies`, if they are in `dependency_mapping`.
        initial_checked_dependencies (set | None, *optional*):
            If provided, entities already present in `initial_checked_dependencies` will not be part of the returned dependencies.
        return_parent (bool, *optional*):
            If `True`, will return a list consisting of tuples (dependency, parent) instead of a simple set of dependencies. Note
            that the order of the items in the list reflects the traversal order. Thus, no parent can ever appear before childs.
    Returns:
        A set of all the dependencies, or a list of tuples `(dependency, parent)` if `return_parent=True`.

    Example:
    Given the following structure in the `modular_xxx.py` file:
    ```
    def foo1():
        pass

    def foo2():
        pass

    def bar():
        foo1()

    def foobar():
        bar()
        foo2()

    class MyLayer(SomeOtherModelLayer):
        def forward(...):
            foobar()
    ```
    and the `dependency_mapping` created when visiting the `modular_xxx.py` file, we get:
    ```
    dependency_mapping = {'bar': {'foo1'}, 'foobar': {'bar', 'foo2'}}
    find_all_dependencies(dependency_mapping, start_entity='foobar', return_parent=True)
    >>> [('bar', 'foobar'), ('foo2', 'foobar'), ('foo1', 'bar')]
    ```
    That is, all the functions needed (and potentially their immediate parent) so that the function to be added
    in MyLayer (`foobar`) can work correctly.
    """
    if initial_dependencies is None and start_entity is not None:
        initial_dependencies = dependency_mapping[start_entity]
    if initial_checked_dependencies is None:
        initial_checked_dependencies = set()

    dependency_queue = deque(initial_dependencies)
    all_dependencies = set()
    all_dependencies_with_parent = []
    checked_dependencies = set(initial_checked_dependencies)
    parents = {initial_dep: start_entity for initial_dep in initial_dependencies}
    while len(dependency_queue) > 0:
        # Pick element to visit
        current = dependency_queue.popleft()
        if current not in checked_dependencies:
            # Add the dependencies
            all_dependencies.add(current)
            all_dependencies_with_parent += [(current, parents[current])]
            if current in dependency_mapping.keys():
                # Update dependency queue
                dependency_queue.extend(dependency_mapping[current])
                parents.update({dep: current for dep in dependency_mapping[current]})
            # add visited node to the list
            checked_dependencies.add(current)

    if not return_parent:
        return all_dependencies
    # no child can ever appear before its parent thanks to the queue (needed to add them at the correct location in the body later)


SUMMARY:
Here is a concise summary of the code:

**Function Purpose and Functionality**

This function performs a breadth-first traversal (BFS) on a dependency graph, where each entity represents a function or assignment name. It finds all dependencies for a given `start_entity` or a set of initial dependencies.

**Main Implementation Details**

* The function uses a queue to efficiently traverse the graph.
* It keeps track of visited entities and their dependencies using sets.
* If `return_parent=True`, it returns a list of tuples (dependency, parent) instead of just a set of dependencies.

**Key Method Signatures and Interfaces**

* `find_all_dependencies(Dict[str, set], str | None = None, ..., return_parent: bool = False -> list | set)`
* `deque()`
* `set()`

**Notable Dependencies or Requirements**

* `dependency_mapping`: a dictionary mapping entities to their dependencies.
* `start_entity`, `initial_dependencies`, and `return_parent` are optional parameters used to customize the traversal.

**Example Use Case**

Given a `modular_xxx.py` file with function definitions, this function returns all dependencies needed for a given entity or initial set of dependencies.


key: chroma:document
string_value:
CODE:

class ClassDependencyMapper(CSTVisitor):
    """A visitor which is designed to analyze a single class node to get all its dependencies that are shared with the set of
    `global_names`.
    """

    def __init__(
        self, class_name: str, global_names: set[str], objects_imported_from_modeling: set[str] | None = None
    ):
        super().__init__()
        self.class_name = class_name
        self.dependencies = set()
        self.global_names = global_names
        self.objects_imported_from_modeling = (
            set() if objects_imported_from_modeling is None else objects_imported_from_modeling
        )

    def visit_Name(self, node):
        if (
            node.value != self.class_name
            and node.value in self.global_names
            and node.value not in self.objects_imported_from_modeling
        ):


SUMMARY:
**ClassDependencyMapper Summary**

The `ClassDependencyMapper` class is a custom visitor for analyzing class nodes to identify shared dependencies with `global_names`. It primarily maps shared node values to set of dependencies.

**Key Implementation Details:**

* Visits each name node in the class definition.
* Checks if name value matches global names (excluding imported from modeling) and adds it to the dependencies set.

**Notable Dependencies or Requirements:**

* `global_names`: a set of global names used for dependency analysis.
* `objects_imported_from_modeling`: an optional set of objects imported from modeling, defaulting to an empty set if not provided.
* `_CSTVisitor`: the base visitor class inherited by this class.

Note that external dependencies are not explicitly mentioned in this code snippet.


key: chroma:document
string_value:
CODE:

def dependencies_for_class_node(node: cst.ClassDef, global_names: set[str]) -> set:
    """Create immediate dependencies for a class node based on the `global_names`."""
    temp_module = cst.Module(body=[node])
    visitor = ClassDependencyMapper(node.name.value, global_names)
    temp_module.visit(visitor)


SUMMARY:
**Summary:**

Creates immediate dependencies for a class node based on `global_names`.

**Main Purpose and Functionality:** 
The function `dependencies_for_class_node` takes a `ClassDef` node and a set of global names, and returns the dependencies required by that class.

**Key Implementation Details:**

- Creates a temporary module with the input class node
- Instantiates a `ClassDependencyMapper` visitor with the class name and global names
- Visits the temporary module, allowing the visitor to inspect dependencies

**Notable Dependencies or Requirements:**

- `cst` (Abstract Syntax Tree) module for parsing
- `ClassDependencyMapper` visitor for dependency resolution
- `global_names` set of imported names for dependency filtering


key: chroma:document
string_value:
CODE:

def augmented_dependencies_for_class_node(
    node: cst.ClassDef, mapper: "ModuleMapper", objects_imported_from_modeling: set[str] | None = None
) -> set:
    """Create augmented dependencies for a class node based on a `mapper`.
    Augmented dependencies means immediate dependencies + recursive function and assignments dependencies.
    """
    temp_module = cst.Module(body=[node])
    visitor = ClassDependencyMapper(node.name.value, set(mapper.global_nodes.keys()), objects_imported_from_modeling)
    temp_module.visit(visitor)


SUMMARY:
**Summary:**

`augmented_dependencies_for_class_node` function:
 Creates augmented dependencies for a class node by traversing its members and recursively visiting dependent modules.

**Key Highlights:**

* Takes in `cst.ClassDef`, `ModuleMapper`, and `objects_imported_from_modeling` parameters.
* Visits the class node using a custom `ClassDependencyMapper` visitor, which collects dependencies.
* Returns augmented dependencies via the ` augment_dependencies` method of the `ModuleMapper`.

**Implementation Details:**

* Creates a temporary module with the class node as its body.
* Uses the `ClassDependencyMapper` to recursively traverse dependent modules.
* Augments the collected dependencies using the `augment_dependencies` method.


key: chroma:document
string_value:
CODE:

class ModuleMapper(CSTVisitor, ABC):
    """An abstract visitor class which analyses a module, creating a mapping of dependencies for classes, functions and assignments.
    Class dependencies are computed with `compute_class_dependencies()`, while function and assignment dependencies are stored in
    `self.object_recursive_dependency_mapping` (can be computed by `_compute_recursive_object_dependencies()`).
    It defines common visiting patterns (i.e. common visit_xxx/leave_xxx functions) between the modular file and the
    modeling files that will be visited.
    """

    METADATA_DEPENDENCIES = (ParentNodeProvider, PositionProvider)

    def __init__(self, python_module: cst.Module):
        # fmt: off
        self.python_module: cst.Module = python_module             # original cst.Module being visited
        self.classes: Dict[str, cst.ClassDef] = {}                 # mapping from class names to Nodes (it will be ordered by default!!)
        self.imports = []                                          # stores all import statements
        self.functions: Dict[str, cst.FunctionDef] = {}            # mapping of global scope function names to Nodes
        self.object_dependency_mapping = defaultdict(set)          # immediate function/assignment dependency mapping (i.e. dependencies immediately in the function/assignment definition)
        self.assignments: Dict[str, cst.SimpleStatementLine] = {}  # mapping of global assignments names to Nodes
        self.current_function = None                               # this keeps track of the current module-scope function
        self.current_assignment = None                             # this keeps track of the current module-scope assignment
        # this keeps track of objects imported from modeling files (`from .configuration import Config`) -> `Config` should not be a dependency
        self.objects_imported_from_modeling = set()
        # regex pattern joining every possible file type
        self.match_patterns = "|".join(ALL_FILE_TYPES)
        # fmt: on

    def visit_ImportFrom(self, node):
        """This keeps track of objects imported from neighbor modeling files (e.g. in `modeling_xxx.py, we have
        `from .configuration_xxx import Config`, then `Config` should be recorded as it is not a dependency that needs
        to be added (because it will be part of the imports)"""
        import_module = self.python_module.code_for_node(node.module)
        import_statement = "." * len(node.relative) + import_module
        if re.search(rf"^\.({self.match_patterns})_.*", import_statement):
            for imported_object in node.names:
                # If an alias is present, we record it and not the original name
                if imported_object.evaluated_alias is not None:
                    self.objects_imported_from_modeling.add(imported_object.evaluated_alias)
                else:
                    self.objects_imported_from_modeling.add(imported_object.evaluated_name)

    def visit_SimpleStatementLine(self, node):
        """
        Global Assigns like `GEMMA_INPUT_DOCSTRING = 'THIS IS THE INPUT'` and all import statements
        are extracted and saved in their corresponding dict. They are then used when updating dependency mappings.
        """
        parent_node = self.get_metadata(cst.metadata.ParentNodeProvider, node)
        simple_top_level_assign_structure = m.SimpleStatementLine(
            body=[m.Assign(targets=[m.AssignTarget(target=m.Name())])]
        )
        if m.matches(parent_node, m.Module()):
            if m.matches(node, simple_top_level_assign_structure):
                left_hand_side = node.body[0].targets[0].target.value
                self.current_assignment = left_hand_side
                self.assignments[left_hand_side] = node
            elif m.matches(node, m.SimpleStatementLine(body=[m.Import() | m.ImportFrom()])):
                self.imports.append(node)

    def leave_SimpleStatementLine(self, node):
        # No need to check for the parent here -> everytime we exit one, it should be None anyway independently of where the
        # SimpleStatement is located
        self.current_assignment = None

    def visit_FunctionDef(self, node):
        parent_node = self.get_metadata(cst.metadata.ParentNodeProvider, node)
        if m.matches(parent_node, m.Module()):
            self.current_function = node.name.value
            self.functions[node.name.value] = node

    def leave_FunctionDef(self, node):
        parent_node = self.get_metadata(cst.metadata.ParentNodeProvider, node)
        if m.matches(parent_node, m.Module()):
            self.current_function = None

    def visit_If(self, node):
        for stmt in node.body.body:
            if m.matches(stmt, m.SimpleStatementLine(body=[m.ImportFrom() | m.Import()])):
                self.imports.append(node)

    def visit_ClassDef(self, node: ClassDef) -> None:
        """Record class nodes to create their dependencies at the end."""
        self.classes[node.name.value] = node

    def visit_Name(self, node: cst.Call):
        """This is used to create a mapping from module-scope functions and assignments to objects used inside them."""
        if self.current_function is not None:
            self.object_dependency_mapping[self.current_function].add(node.value)
        if self.current_assignment is not None:
            self.object_dependency_mapping[self.current_assignment].add(node.value)

    def leave_Module(self, node):
        """When leaving the module, we store the position of each global scoped node to allow sorting the dependencies
        based on their position in the code later. We use the PositionProvider metadata wrapper for this.
        We also make sure to update `self.object_dependency_mapping` so that it contains only names recorded in
        `self.global_nodes`.
        """
        # assign all nodes
        self.global_nodes = {**self.assignments, **self.classes, **self.functions}
        # now sort the class dependency_mapping based on the position of the nodes
        self.start_lines = {}
        for id, node in self.global_nodes.items():
            self.start_lines[id] = self.get_metadata(cst.metadata.PositionProvider, node).start.line

    def _restrict_dependencies_to_known_entities(self):
        """Since we added every Name as part of `self.object_dependency_mapping`, we need to remove those that
        are not part of the recorded objects in `self.global_nodes` (i.e. built-in variables, imports, etc).
        This should be called only after all merging operations have been finalized!!"""
        global_objects = set(self.global_nodes.keys())
        for object_name, dependencies in self.object_dependency_mapping.items():
            self.object_dependency_mapping[object_name] = {dep for dep in dependencies if dep in global_objects}

    def _compute_recursive_object_dependencies(self) -> dict[str, set]:
        """Based on immediate dependency mapping, create the recursive dependency mapping. For example, given the
        following file:
        ```
        def foo():
            pass

        def bar():
            foo()

        def test():
            bar()
        ```
        this visitor can only record immediate dependencies, i.e. it will record the following
        `self.object_dependency_mapping = {"test": {"bar"}, "bar": {"foo}}`. This function is used to create
        the recursive mapping, i.e. `recursive_dependencies = {"test": {"bar", "foo"}, "bar": {"foo}}`.
        """
        recursive_dependencies = {}
        for object_name in self.object_dependency_mapping.keys():
            all_dependencies = find_all_dependencies(self.object_dependency_mapping, start_entity=object_name)
            recursive_dependencies[object_name] = all_dependencies
        return recursive_dependencies

    def augment_dependencies(self, dependencies: set[str]) -> set[str]:
        """For a set of `dependencies`, augment them by adding all potential dependencies of the **functions** and
        **assignments** present in the `dependencies`.
        """
        new_dependencies = dependencies.copy()
        # Go through the set of dependencies
        for dep in tuple(dependencies):
            if dep in self.object_recursive_dependency_mapping.keys():
                new_dependencies.update(self.object_recursive_dependency_mapping[dep])
        return new_dependencies

    def compute_class_dependencies(self):
        """For each visited class, find its dependencies based on visiting the current file + potential merged dependencies."""
        self.class_dependency_mapping = {}
        for class_name, class_node in self.classes.items():
            dependencies = dependencies_for_class_node(class_node, set(self.global_nodes.keys()))
            # Correctly augment class dependencies with all needed objects
            self.class_dependency_mapping[class_name] = self.augment_dependencies(dependencies)

    @abstractmethod
    def compute_relative_order(self, missing_dependencies: set) -> dict[str, int]:


SUMMARY:
**ModuleMapper Summary**

The `ModuleMapper` class is an abstract visitor that analyzes a Python module, creating a mapping of dependencies for classes, functions, and assignments. It computes dependencies using `compute_class_dependencies()` and `_compute_recursive_object_dependencies()`, storing them in `self.object_recursive_dependency_mapping`.

**Main Purpose and Functionality:**

* Analyzes a Python module to create a dependency mapping
* Computes class dependencies using `compute_class_dependencies()`
* Stores object dependencies using `_compute_recursive_object_dependencies()`
* Augments dependencies using `augment_dependencies()` and `dependencies_for_class_node()`

**Class Relationship Mapping:**

* Records class nodes in `self.classes` for future dependencies
* Maps functions and assignments to objects used inside them using `visit_Name()` and `visit_ClassDef()`
* Updates `self.object_recursive_dependency_mapping` with recursive dependencies

**Dependencies Computation:**

* Computes absolute dependencies using `_compute_recursive_object_dependencies()`
* Augments dependencies with potential dependencies of functions and assignments

**Abstract Methods:**

* `compute_relative_order(missing_dependencies: set)`: returns a dictionary mapping class names to their relative order for missing dependencies.

Note: The code uses various internal classes, methods, and metadata wrappers that are not included in this summary.


key: chroma:document
string_value:
CODE:

class ModelFileMapper(ModuleMapper):
    """A mapper designed to parse modeling files (like `modeling_llama.py`). When encountering such a file
    in the `modular_xxx.py` file, we need to correctly visit it and merge the dependencies of the modular and current file.
    For this reason, this class should only be instantiated from the class method `visit_and_merge_dependencies`, which takes
    care of correctly merging dependencies, then finalizes all dependency graph computations.
    Note that we only merge functions and assignments here, as classes will be treated later on as they may be modified.
    For example, if you redefine `apply_rotary_pos_emb()` in the modular, the new node should be used in the dependencies
    of the modeling files as well.
    """

    def __init__(self, python_module: cst.Module):
        super().__init__(python_module)

    def compute_relative_order(self, missing_dependencies: set[str]) -> dict[str, int]:
        """Compute in which relative order the `missing_dependencies` should appear when the nodes are added to the final file that
        will be created based on the modular.
        """
        relative_order = {}
        idx = 0
        classes = sorted(
            [dep for dep in tuple(missing_dependencies) if dep in self.classes], key=lambda x: self.start_lines[x]
        )
        # This is because for merged dependencies, we only have relative order in the other visited file, so we need
        # to track dependency order relative to a given class
        if len(classes) > 0 and not hasattr(self, "class_dependency_mapping"):
            raise ValueError("Cannot correctly find the relative order of the dependencies.")

        remaining_dependencies = missing_dependencies.copy()

        # Start by tracking relative order class by class
        for class_name in classes:
            class_dependencies = tuple(self.class_dependency_mapping[class_name] & remaining_dependencies)
            original_dependencies = []
            merged_dependencies = []
            # We need to differentiate between nodes that were already present (we can get relative order globally) and
            # nodes that were merged (we can get relative order only relative to the class the dependencies relate to)
            for class_dep in class_dependencies:
                if class_dep in self.start_lines:
                    original_dependencies.append(class_dep)
                else:
                    merged_dependencies.append(class_dep)
            # Sort both list according to the order in their respective file
            original_dependencies = sorted(original_dependencies, key=lambda x: self.start_lines[x])
            merged_dependencies = sorted(merged_dependencies, key=lambda x: self.modular_file_start_lines[x])

            # Add all original node first, then merged ones
            for dep in original_dependencies + merged_dependencies:
                remaining_dependencies.remove(dep)
                relative_order[dep] = idx
                idx += 1
            # Add the class itself (it can sometimes already be present if the order of classes in the source file
            # does not make sense, i.e. a class is used somewhere before being defined like in `rt_detr`...)
            if class_name in remaining_dependencies:
                remaining_dependencies.remove(class_name)
                relative_order[class_name] = idx
                idx += 1

        # Now add what still remains
        remaining_dependencies = tuple(remaining_dependencies)
        original_dependencies = []
        merged_dependencies = []
        for dep in remaining_dependencies:
            if dep in self.modular_file_start_lines:
                merged_dependencies.append(dep)
            else:
                original_dependencies.append(dep)
        # Sort both list according to the order in their respective file
        original_dependencies = sorted(original_dependencies, key=lambda x: self.start_lines[x])
        merged_dependencies = sorted(merged_dependencies, key=lambda x: self.modular_file_start_lines[x])

        # Add all original node first, then merged ones
        for dep in original_dependencies + merged_dependencies:
            relative_order[dep] = idx
            idx += 1

        return relative_order

    def _merge_functions(self, functions: dict[str, cst.CSTNode], object_mapping: dict[str, set]):
        """Update the global nodes and function dependency mapping with those from the modular file.

        Merging rule: if any function with the same name was redefined in the modular, use it and its dependencies
        instead of the original ones (this may mean to add new functions as well, if any redefined function uses a new one).
        """
        # Add/overwrite all needed function nodes and dependencies
        self.functions.update(functions)
        self.object_dependency_mapping.update(
            {obj: dep for obj, dep in object_mapping.items() if obj in functions.keys()}
        )
        # Add them to global nodes
        self.global_nodes.update(self.functions)

    def _merge_assignments(self, assignments: dict[str, cst.CSTNode], object_mapping: dict[str, set]):
        """Update the global nodes with the assignment from the modular file.

        Merging rule: if any assignment with the same name was redefined in the modular, we use it and its dependencies ONLY if it is
        in `ASSIGNMENTS_TO_KEEP`. Otherwise, we use the original value and dependencies. This rule was chosen to avoid having to rewrite the
        big docstrings.
        """
        for assignment, node in assignments.items():
            if assignment in ASSIGNMENTS_TO_KEEP or assignment not in self.assignments:
                self.assignments[assignment] = node
                if assignment in object_mapping:
                    self.object_dependency_mapping[assignment] = object_mapping[assignment]
        # Add them to global nodes
        self.global_nodes.update(self.assignments)

    def _merge_classes(self, classes: dict[str, cst.CSTNode]):
        """Update the global nodes with the new classes from the modular (i.e. classes which do not exist in current file, and
        are not imported). We do NOT update any dependency mapping here. This is because we only need the names of newly defined
        classes in the modular to be discoverable when computing dependencies for new nodes later on. For this reason, we
        do not add the new classes to `self.classes`, but only to `global_nodes`.
        """
        # Add/overwrite all needed function nodes and dependencies
        self.global_nodes.update(
            {
                name: node
                for name, node in classes.items()
                if name not in self.classes and name not in self.objects_imported_from_modeling
            }
        )

    def merge_modular_dependencies(self, classes, functions, assignments, object_mapping, start_lines):
        """Merge classes, functions and assignments from the modular definitions into the current module file,
        then record the relative order of all nodes.
        Note: This function takes care of updating `global_nodes` and `object_recursive_dependency_mapping` as well after the
        merge with other files dependencies.
        """
        self._merge_functions(functions, object_mapping)
        self._merge_assignments(assignments, object_mapping)
        self._merge_classes(classes)
        self.modular_file_start_lines = start_lines

        # Restrict the dependency mappings to the known entities to avoid Python's built-ins and imports
        self._restrict_dependencies_to_known_entities()
        # Create the global mapping of recursive dependencies for functions and assignments
        self.object_recursive_dependency_mapping = self._compute_recursive_object_dependencies()

    @classmethod
    def visit_and_merge_dependencies(
        cls, module: cst.Module, classes, functions, assignments, object_mapping, start_lines
    ) -> "ModelFileMapper":
        wrapper = MetadataWrapper(module)
        mapper = cls(module)
        wrapper.visit(mapper)
        # Merge dependencies
        mapper.merge_modular_dependencies(classes, functions, assignments, object_mapping, start_lines)
        # Create the class dependencies graph
        mapper.compute_class_dependencies()


SUMMARY:
**ModelFileMapper Class Summary**

The `ModelFileMapper` class is designed to parse modeling files and merge their dependencies with those of the modular file.

### Purpose
To correctly visit and merge dependencies between modules, ensuring that functions, assignments, and classes are properly updated and ordered relative to each other.

### Key Implementation Details

*   **`merge_modular_dependencies` method**: Merges classes, functions, and assignments from the modular definitions into the current module file.
*   **`compute_relative_order` method**: Computes the relative order of all nodes based on their dependencies and ordering conventions in the source code.
*   **Methods for merging specific node types (functions, assignments, classes)**
*   **Updating global nodes and dependency mappings** after merging with modular file dependencies

### Notable Dependencies or Requirements

*   `cst` module for parsing Python code
*   `MetadataWrapper` class for accessing metadata of the parsed module
*   External dependencies are restricted to known entities using `_restrict_dependencies_to_known_entities` method.

### Class Inheritance and Interfaces

None, as it is a custom implementation.

**Example Usage**

```python
import cst

# Sample module definition
module_def = """class MyClass:
    def __init__(self):
        pass"""

# Create an instance of the ModelFileMapper class
mapper = ModelFileMapper(module)  # Assuming 'module' is a cst module object

# Call the visit_and_merge_dependencies method to process the module
mapper.visit_and_merge_dependencies(...
```

This code snippet provides a basic overview of how to use the `ModelFileMapper` class.


key: chroma:document
string_value:
CODE:

def common_partial_suffix(str1: str, str2: str) -> str:
    """Return the biggest common suffix between 2 strings. If one string is a full suffix of the other string,
    we do not consider it a common suffix and return `""`"""
    common_suffix = ""
    for i in range(1, min(len(str1), len(str2)) + 1):
        if str1[-i] == str2[-i]:
            common_suffix = str1[-i] + common_suffix
        else:
            break
    # We do not allow full string suffix
    if common_suffix == str1 or common_suffix == str2:
        common_suffix = ""


SUMMARY:
**Summary:**

### Function Name:
`common_partial_suffix`

### Primary Purpose:
Returns the largest common partial suffix between two input strings.

### Key Implementation Details:

* Iterates from the shortest string's end to find matching characters.
* Builds the `common_suffix` by prepending matching characters from `str1`.
* If one string is a full suffix of the other, returns an empty string.

### Notable Dependencies/Requirements:

* No external dependencies mentioned.
* The function uses built-in Python string concatenation and iteration.


key: chroma:document
string_value:
CODE:

def replace_class_node(
    mapper: ModelFileMapper, class_node: cst.ClassDef, renamed_super_class: str, original_super_class: str
):
    """
    Replace a class node which inherits from another modeling class. This function works in the following way:
    - start from the base class node of the inherited class (a cst.Node)
    - replace all methods of the base node with the methods defined in the child class
    - append all new methods defined in the child class
    - replace all calls to super() with the unravelled code

                    |    ```python                          |               |    ```python
                    |    class GemmaModel(LlamaModel):      |               |       class GemmaModel(nn.Module):
                    |        def __init__(self):            |               |           def __init__(self):
    Going from:     |            super().__init__()         |       to:     |               super().__init__(config)
                    |            self.dropout = 0.2         |               |               self.dropout = 0.2
                    |     ```                               |               |               self.padding_idx = config.pad_token_id
                                                                            |               self.vocab_size = config.vocab_size
                                                                            |               self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
                                                                            |               self.layers = nn.ModuleList(
                                                                            |                   [LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
                                                                            |               )
                                                                            |               self.norm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
                                                                            |               self.gradient_checkpointing = False
                                                                            |               # Initialize weights and apply final processing
                                                                            |               self.post_init()
                                                                            |     ```
    """
    all_bases = [get_full_attribute_name(k.value) for k in class_node.bases]
    if any(base is None for base in all_bases):
        raise ValueError(f"Could not parse the name of the bases for {class_node.name.value}")

    original_node = mapper.classes[renamed_super_class]
    # Always use the new name of the class (in case we use e.g. `ColPaliForRetrieval` inheriting from `PaliGemmaForConditionalGeneration`)
    new_name = class_node.name

    # If the new class name is different from the renamed super class name, we need to update the docstrings/comments accordingly
    if new_name.value != renamed_super_class:
        common_suffix = common_partial_suffix(new_name.value, renamed_super_class)
        # Note that this works even without common prefix, in which case it does not replace anything
        old, new = renamed_super_class.replace(common_suffix, ""), new_name.value.replace(common_suffix, "")
        temp_module = cst.Module(body=[original_node])
        original_node = temp_module.visit(
            ReplaceNameTransformer(get_lowercase_name(old), get_lowercase_name(new), only_doc=True)
        ).body[0]

    # If we explicitly passed a new base with common suffix to an old base, it is for switching the prefix
    # e.g. if the "natural" parent class is `PreTrainedModel` but we wanted to rename it to `PreTrainedVisionModel`
    additional_bases = [base for base in all_bases if base != original_super_class]
    new_bases = []
    for original_base in original_node.bases:
        new_base = original_base
        # we only potentially switch base for Name-based bases, not Attribute
        if m.matches(original_base.value, m.Name()):
            original_base_name = original_base.value.value
            for additional_base_name in additional_bases:
                suffix = common_partial_suffix(original_base_name, additional_base_name)
                if len(suffix) > 0 and suffix[0].isupper():
                    new_name_node = original_base.value.with_changes(value=additional_base_name)
                    new_base = original_base.with_changes(value=new_name_node)
                    break
        new_bases.append(new_base)

    original_methods = {
        f.name.value if hasattr(f, "name") else mapper.python_module.code_for_node(f): f
        for f in original_node.body.body
    }
    updated_methods = {
        f.name.value if hasattr(f, "name") else mapper.python_module.code_for_node(f): f for f in class_node.body.body
    }
    end_meth = []

    assign_targets = {}
    docstring_node = []
    # Iterate directly from node.body as there can be property/setters with same names which are overwritten when we use a dict
    for func in original_node.body.body:
        name = func.name.value if hasattr(func, "name") else mapper.python_module.code_for_node(func)
        if m.matches(func, m.FunctionDef()) and name in updated_methods and updated_methods[name] is not None:
            new_params = updated_methods[name].params
            # Replace the method in the replacement class, preserving decorators
            kwarg_name = getattr(updated_methods[name].params, "star_kwarg", None)
            if kwarg_name and kwarg_name.name.value == "super_kwargs":
                parent_params = {k.name.value: k for k in func.params.params}
                parent_params.update({k.name.value: k for k in new_params.params[1:]})
                new_params = new_params.with_changes(
                    params=list(parent_params.values()), star_kwarg=func.params.star_kwarg
                )
            # Keep decorators in `modular_xxx.py` if any, else original decorators
            new_decorators = (
                updated_methods[name].decorators if len(updated_methods[name].decorators) > 0 else func.decorators
            )
            if not re.match(
                r"\ndef .*\(.*\):\n    raise.*Error\(.*",
                mapper.python_module.code_for_node(updated_methods[name]),
            ):
                func = func.with_changes(body=updated_methods[name].body, params=new_params, decorators=new_decorators)
            else:
                continue

        if m.matches(func, m.SimpleStatementLine(body=[m.Assign()])):
            target = mapper.python_module.code_for_node(func.body[0].targets[0])
            assign_targets[target] = func
        elif m.matches(func, m.SimpleStatementLine(body=[m.AnnAssign()])):
            target = mapper.python_module.code_for_node(func.body[0].target)
            assign_targets[target] = func
        elif m.matches(func, DOCSTRING_NODE):
            docstring_node = [func]
        else:
            end_meth.append(func)

    # Port new methods that are defined only in modular-file and append at the end
    for func in class_node.body.body:
        name = func.name.value if hasattr(func, "name") else mapper.python_module.code_for_node(func)
        if m.matches(func, DOCSTRING_NODE):  # This processes the docstring of the class!
            # Extract the original docstring
            updated_docstring = func.body[0].value.value
            if len(docstring_node) == 0:  # If the original docstring is empty, just create one from the updated.
                docstring_node = [
                    cst.SimpleStatementLine(body=[cst.Expr(value=cst.SimpleString(value=updated_docstring))])
                ]
            else:
                original_docstring = docstring_node[0].body[0].value.value
                merged_doc = merge_docstrings(original_docstring, updated_docstring)
                # Update the docstring in the original function
                docstring_node = [
                    docstring_node[0].with_changes(body=[cst.Expr(value=cst.SimpleString(value=merged_doc))])
                ]
        if name not in original_methods and func is not None and isinstance(func, cst.FunctionDef):
            end_meth.append(func)
        if m.matches(func, m.SimpleStatementLine(body=[m.Assign()])):
            # TODO we only use single assign might cause issues
            target = mapper.python_module.code_for_node(func.body[0].targets[0])
            assign_targets[target] = func
        if m.matches(func, m.SimpleStatementLine(body=[m.AnnAssign()])):
            target = mapper.python_module.code_for_node(func.body[0].target)
            assign_targets[target] = func
    end_meth = docstring_node + list(assign_targets.values()) + end_meth

    # Replace the calls to `super()` with the unrolled code
    result_node = original_node.with_changes(body=cst.IndentedBlock(body=end_meth))
    temp_module = cst.Module(body=[result_node])
    new_module = MetadataWrapper(temp_module)
    new_replacement_class = new_module.visit(
        SuperTransformer(temp_module, original_methods, updated_methods, all_bases)
    )
    new_replacement_body = new_replacement_class.body[0].body  # get the indented block

    # Use decorators redefined in `modular_xxx.py` if any
    new_decorators = class_node.decorators if len(class_node.decorators) > 0 else original_node.decorators

    return original_node.with_changes(
        body=new_replacement_body, decorators=new_decorators, bases=new_bases, name=new_name


SUMMARY:
This function replaces a Python class node in an abstract syntax tree (AST) with a new implementation. The replacement process involves:

1. Updating the base classes and names of the class.
2. Replacing methods defined in the base classes with those defined in the child class.
3. Appending new methods not present in the parent class but present in the child class.
4. Replacing calls to `super()` with the updated code.

The function takes as input a model file mapper, a node representing the class to be replaced, and the names of the superclasses and the child's superclass. It returns a new node representing the modified class.


key: chroma:document
string_value:
CODE:

def find_file_type(class_name: str) -> str:
    """Based on a class name, find the file type corresponding to the class.
    If the class name is `LlamaConfig` it will return `configuration`.
    The list of suffixes is in `TYPE_TO_FILE_TYPE`. If there are no match, we match by default to `modeling`
    """
    match_pattern = "|".join(TYPE_TO_FILE_TYPE.keys())
    match = re.search(rf"({match_pattern})$", class_name)
    if match:
        file_type = TYPE_TO_FILE_TYPE[match.group(1)]
    else:
        file_type = "modeling"


SUMMARY:
**Summary:**

### Function Purpose and Functionality

`find_file_type` function takes a `class_name` string as input and returns the corresponding file type. It uses a lookup table `TYPE_TO_FILE_TYPE` to match class names against predefined file types.

### Key Implementation Details

* Uses regular expressions to search for matching file types
* Falls back to "modeling" if no match is found in `TYPE_TO_FILE_TYPE`

### Notable Dependencies and Requirements

* `re` (regular expression) module
* `TYPE_TO_FILE_TYPE` dictionary (not shown in this snippet)

### Method Signatures and Interfaces

* `find_file_type(class_name: str) -> str`: returns a file type string based on the input class name


key: chroma:document
string_value:
CODE:

def append_new_import_node(
    node: cst.CSTNode, unused_imports: set[str], added_names: set, imports_to_keep: list[cst.CSTNode]
):
    """Insert the new `node` to the list of `imports_to_keep` in-place, if it is not part of the `unused_imports` or `added_names`.
    Also modifies `added_names` in-place accordingly."""
    import_node = node.body[0]
    names_to_keep = []
    for name in import_node.names:
        name_value = name.evaluated_name
        if name_value not in unused_imports and name_value not in added_names:
            names_to_keep.append(name.with_changes(comma=cst.MaybeSentinel.DEFAULT))
            added_names.add(name_value)
    if len(names_to_keep) > 0:
        new_node = node.with_changes(body=[import_node.with_changes(names=names_to_keep)])


SUMMARY:
**Summary of `append_new_import_node` function**

Inserts a new import node (`node`) to the `imports_to_keep` list if it's not part of unused or added imports. Modifies `added_names` accordingly.

**Key Implementation Details:**

* Inserts `import_node` into `imports_to_keep` with filtered names.
	+ `names_to_keep = [name_value]` where `name_value != unused_imports` and `name_value != added_names`.
* Applies modifications to `node` using:
	+ `with_changes(body=[import_node.with_changes(names=names_to_keep)])`
	+ `imports_to_keep.append(new_node)`.

**Notable Dependencies/Requirements:**

* `cst.CSTNode`: Input node type.
* `added_names`: Set of names to keep track of.
* `unused_imports`: Set of unused imports.

**External Dependency:**

* `cst.MaybeSentinel.DEFAULT`.


key: chroma:document
string_value:
CODE:

def get_needed_imports(body: dict[str, dict], all_imports: list[cst.CSTNode]) -> list[cst.CSTNode]:
    """Get all the imports needed in the `body`, from the list of `all_imports`.
    `body` is a dict with the following structure `{str: {"insert_idx": int, "node": cst.CSTNode}}`.
    Note: we need to use `isinstance` on scope assignements, m.matches apparently does not work here yet!
    """
    new_body = [k[1]["node"] for k in sorted(body.items(), key=lambda x: x[1]["insert_idx"])]
    wrapper = MetadataWrapper(cst.Module(body=all_imports + new_body))
    scopes = set(wrapper.resolve(ScopeProvider).values())
    unused_imports = set()
    import_ref_count = defaultdict(lambda: 0)
    for scope in scopes:
        for assignment in scope.assignments:
            node = assignment.node
            if isinstance(assignment, cst.metadata.Assignment) and isinstance(node, (cst.Import, cst.ImportFrom)):
                ref_count = len(assignment.references)
                name = assignment.name
                import_ref_count[name] = max(ref_count, import_ref_count[name])
    # Similar imports may be redefined, and only used between their 1st and 2nd definition so if we already have
    # a ref count > 0 at any point, the imports is actually used
    unused_imports = {name for name, count in import_ref_count.items() if count <= 0 or name in body.keys()}

    imports_to_keep = []
    # We need to keep track of which names were already imported, because some import may be duplicated from multiple sources
    # or be both protected and unprotected due to inconsistency between models
    added_names = set()
    existing_protected_statements = set()  # str repr of the import nodes - does not work with the nodes directly
    for node in all_imports:
        if m.matches(node, m.If()):  # handle safe imports
            new_statements = []
            for stmt_node in node.body.body:
                append_new_import_node(stmt_node, unused_imports, added_names, new_statements)
            new_statements = [stmt for stmt in new_statements if str(stmt) not in existing_protected_statements]
            if len(new_statements) > 0:
                new_node = node.with_changes(body=node.body.with_changes(body=new_statements))
                imports_to_keep.append(new_node)
                existing_protected_statements.update({str(stmt) for stmt in new_statements})
        else:
            append_new_import_node(node, unused_imports, added_names, imports_to_keep)

    protected_import_nodes = [node for node in imports_to_keep if m.matches(node, m.If())]
    usual_import_nodes = [node for node in imports_to_keep if not m.matches(node, m.If())]

    # Protected imports always appear at the end of all imports


SUMMARY:
**Function Summary: `get_needed_imports`**

*   Purpose:
    *   Generates needed imports for a given code segment (`body`) from a list of provided imports (`all_imports`)
*   Functionality:

    1.  **Sorting and Preparation**: Sorts the `body` dictionary based on insert indices, creates a metadata wrapper around the combined `all_imports + new_body`, and resolves scopes to identify assignments.
    2.  **Import Ref Counting**: Iterates through scope assignments to count reference counts for each import name.
    3.  **Unused Imports Detection**: Identifies unused imports based on ref count threshold and existing references in the code segment (`body`).
*   Output:
    *   A list of needed imports, separated into `usual` (non-protected) and `protected` nodes.

**Key Implementation Details:**

*   Uses a dictionary-based approach to sort and prepare the input data.
*   Employs a metadata wrapper to analyze import statements.
*   Utilizes scope resolution to identify assignments and determine ref counts.
*   Applies thresholds for identifying unused imports.

**Notable Dependencies/Requirements:**

*   `cst.CSTNode` (used for parsing and analyzing code structure)
*   `MetadataWrapper` (custom class used for metadata-driven analysis)
*   `ScopeProvider` (interface for scope resolution)
*   Custom modules (`m.matches`, `append_new_import_node`) not explicitly referenced in the provided function definition.


key: chroma:document
string_value:
CODE:

def split_all_assignment(node: cst.CSTNode) -> dict[str, cst.CSTNode]:
    """Split the `__all__` assignment found in the modular between each corresponding files."""
    all_all_per_file = {}
    assign_node = node.body[0]
    if isinstance(assign_node.value, cst.List):
        # Extract the elements from the list
        all_all_to_add = defaultdict(list)
        for element in assign_node.value.elements:
            if isinstance(element.value, cst.SimpleString):
                # Remove quotes and add the string to the elements list
                class_name = element.value.value
                file = find_file_type(element.value.evaluated_value)
                all_all_to_add[file] += [class_name]
        for file, new_alls in all_all_to_add.items():
            new_node = assign_node.with_changes(
                value=cst.List(elements=[cst.Element(value=cst.SimpleString(value=k)) for k in new_alls])
            )
            all_all_per_file[file] = node.with_changes(body=[new_node])


SUMMARY:
**Function Summary: `split_all_assignment`**

**Purpose:** Splits the `__all__` assignment in a modular into separate assignments for corresponding files.

**Main Functionality:**
Extracts class names from the list of values assigned to `__all__`, determines the file type, and regroups them by file using a dictionary. This function splits the initial assignment into multiple assignments, one per file.

**Key Implementation Details:**

* Iterates through a list of values (`node.body[0].value.elements`) and extracts class names from strings.
* Determines the file type for each class name using `find_file_type(element.value.evaluated_value)`.
* Regroups class names by file using a dictionary (`all_all_to_add`).

**Method Signatures and Interfaces:**

* `assign_node.with_changes`: Replaces the value of an assignment node with a new list.
* `node.with_changes`: Wraps the entire node with new body.

**Notable Dependencies and Requirements:**

* `cst.CSTNode`
* ` defaultdict(list)`
* `find_file_type` (not provided)


key: chroma:document
string_value:
CODE:

class ModularFileMapper(ModuleMapper):
    """This is a Mapper to visit a modular file (like `modular_llama.py`). It visits the whole file, recording dependency,
    then visits all imported modeling files (like `modeling_llama.py`), and manages their mutual dependencies.
    Calling the method `create_modules()` after visit will create all modules based on this modular file.
    """

    def __init__(self, python_module, new_name):
        super().__init__(python_module)
        # fmt: off
        self.model_name = new_name  # name of the model being defined. Should be in the format of `llama` or `layout_xlm` or `phi3`

        self.model_specific_imported_objects: Dict[str, str] = {}  # e.g. {"LlamaModel": "transformers.models.llama.modeling_llama"}
        self.model_specific_modules: Dict[str, cst.Module] = {}  # e.g. {"transformers.models.llama.modeling_llama": cst.Module}

        self.all_all_to_add = {}
        # fmt: on

    def visit_ImportFrom(self, node: cst.ImportFrom) -> None:
        """When visiting imports from modeling files (i.e. `transformers.models.xxx`) we get the code, parse it,
        and save it in `self.model_specific_modules` to later visit. The imported objects are saved in `self.model_specific_imported_objects`.
        """
        import_module = self.python_module.code_for_node(node.module)
        import_statement = "." * len(node.relative) + import_module
        if any(import_to_skip in import_statement for import_to_skip in IMPORTS_TO_SKIP_IN_MODULAR):
            return
        if m.matches(node.module, m.Attribute()):
            for imported_ in node.names:
                _import = re.search(
                    rf"(?:transformers\.models\.)|(?:\.\.)\w+\.({self.match_patterns})_.*", import_statement
                )
                if _import:
                    source = _import.group(1)
                    if source == "modeling" and "Config" in self.python_module.code_for_node(imported_):
                        raise ValueError(
                            f"You are importing {self.python_module.code_for_node(imported_)} from the modeling file. Import from the `configuration_xxxx.py` file instead"
                        )
                    if import_module not in self.model_specific_modules:
                        if "models" not in import_module:
                            import_module = "models." + import_module
                        if "transformers" not in import_module:
                            import_module = "transformers." + import_module
                        source_code = get_module_source_from_name(import_module)
                        tree = cst.parse_module(source_code)
                        self.model_specific_modules[import_module] = tree
                    imported_object = self.python_module.code_for_node(imported_.name)
                    self.model_specific_imported_objects[imported_object] = import_module
        if m.matches(node.module, m.Name()):
            if "transformers" == import_module:
                raise ValueError(
                    f"You are importing from {import_module} directly using global imports. Import from the correct local path"
                )

    def visit_SimpleStatementLine(self, node):
        """If we visit an import statement not previously visited, record it. If we visit a module-scope assignment,
        simply record it or, if it is `__all__`, split it between files where we should dispatch it.
        """
        parent_node = self.get_metadata(cst.metadata.ParentNodeProvider, node)
        simple_top_level_assign_structure = m.SimpleStatementLine(
            body=[m.Assign(targets=[m.AssignTarget(target=m.Name())])]
        )
        if m.matches(parent_node, m.Module()):
            if m.matches(node, m.SimpleStatementLine(body=[m.Import()])):
                self.imports.append(node)
            elif m.matches(node, m.SimpleStatementLine(body=[m.ImportFrom()])):
                import_module = self.python_module.code_for_node(node.body[0].module)
                import_statement = "." * len(node.body[0].relative) + import_module
                if not (
                    re.search(rf"(?:transformers\.models\.)|(?:\.\.)\w+\.({self.match_patterns})_.*", import_statement)
                    and not any(import_to_skip in import_statement for import_to_skip in IMPORTS_TO_SKIP_IN_MODULAR)
                ):
                    self.imports.append(node)
            elif m.matches(node, simple_top_level_assign_structure):
                assigned_variable = node.body[0].targets[0].target.value
                # __all__ is treated differently and not added to general assignments
                if assigned_variable == "__all__":
                    self.all_all_to_add = split_all_assignment(node)
                else:
                    self.current_assignment = assigned_variable
                    self.assignments[assigned_variable] = node

    def leave_Module(self, node):
        """When we leave the modular file, we do the following in order:
        1. for each modeling file found in the imports, rename it with the new model name, visit it, and update
        its dependency graph with the new function and assignment definitions found in the modular
        2. update the modular dependency graph with the imported functions and assignments (found when visiting the matching files)
        3. compute the nested (recursive) function and assignment dependencies
        """
        # Takes care of finalizing our visit
        super().leave_Module(node)

        # 1. for each modeling file found in the imports, rename it with the new model name, visit it, and update dependencies
        self.visited_modules = {}
        self.renamers = {}
        name_prefixes = self.infer_new_model_name()
        for file, module in self.model_specific_modules.items():
            file_model_name = file.split(".")[-2]
            new_name = name_prefixes[file]
            renamer = ReplaceNameTransformer(file_model_name, new_name, self.model_name)
            renamed_module = module.visit(renamer)
            self.visited_modules[file] = ModelFileMapper.visit_and_merge_dependencies(
                renamed_module,
                self.classes,
                self.functions,
                self.assignments,
                self.object_dependency_mapping,
                self.start_lines,
            )
            # We record it so that we can rename classes later the exact same way
            self.renamers[file] = renamer

        # 2. in turn, we need to add the imported functions/assignments to the dependencies of the modular mapper, using the
        # definitions found in the visited files
        self.merge_model_specific_imports(self.visited_modules)

        # 3. compute the nested (recursive) function and assignment dependencies
        self.object_recursive_dependency_mapping = self._compute_recursive_object_dependencies()

        # We need to keep track of which objects were imported directly into which modeling file to not add them wrongly later
        # Note that we may visit several of the same file types, thus we save them per file type, not file
        self.imported_objects_per_file = defaultdict(set)
        for file, mapper in self.visited_modules.items():
            file_type = re.search(rf"^transformers\.models\.\w+\.({self.match_patterns})_.*", file).group(1)
            self.imported_objects_per_file[file_type].update(mapper.objects_imported_from_modeling)

    def merge_model_specific_imports(self, visited_modules):
        """Merge the functions and assignments imported from the modeling files to the modular nodes and dependency graph,
        based on the visited files."""
        self.start_lines_file_mapping = {}
        self.added_objects_file_mapping = {}
        for object_name, file in self.model_specific_imported_objects.items():
            visited_module = visited_modules[file]
            self.start_lines_file_mapping[file] = visited_module.start_lines
            # Add functions and their dependencies
            if object_name in visited_module.functions and object_name not in self.functions:
                self.functions[object_name] = visited_module.functions[object_name]
                self.added_objects_file_mapping[object_name] = file
                dependencies = visited_module.object_dependency_mapping.get(object_name, None)
                if dependencies is not None:
                    self.object_dependency_mapping[object_name] = dependencies
                    for dep in dependencies:
                        if dep not in self.global_nodes:
                            self.added_objects_file_mapping[dep] = file
                            self.functions[dep] = visited_module.global_nodes[dep]

            # Add assignments and their dependencies
            elif object_name in visited_module.assignments and object_name not in self.assignments:
                self.assignments[object_name] = visited_module.assignments[object_name]
                self.added_objects_file_mapping[object_name] = file
                dependencies = visited_module.object_dependency_mapping.get(object_name, None)
                if dependencies is not None:
                    self.object_dependency_mapping[object_name] = dependencies
                    for dep in dependencies:
                        if dep not in self.global_nodes:
                            self.added_objects_file_mapping[dep] = file
                            self.assignments[dep] = visited_module.global_nodes[dep]

        # Do not forget to re-assign all nodes after the merge
        self.global_nodes = {**self.assignments, **self.classes, **self.functions}
        # And restric dependencies to those nodes only
        self._restrict_dependencies_to_known_entities()

    def compute_relative_order(self, missing_dependencies: set) -> dict[str, int]:
        """Compute in which relative order the `missing_dependencies` should appear when the nodes are added to the final file that
        will be created based on the modular.
        """
        relative_order = {}
        idx = 0

        original_dependencies = []
        other_files_dependencies = defaultdict(list)
        for dep in tuple(missing_dependencies):
            if dep in self.added_objects_file_mapping:
                file = self.added_objects_file_mapping[dep]
                other_files_dependencies[file].append(dep)
            else:
                original_dependencies.append(dep)
        # Sort all lists according to the order in their respective file
        all_dependencies = []
        for file, dependencies in other_files_dependencies.items():
            sorted_dependencies = sorted(dependencies, key=lambda x: self.start_lines_file_mapping[file][x])
            all_dependencies += sorted_dependencies
        all_dependencies += sorted(original_dependencies, key=lambda x: self.start_lines[x])

        # Add all original node first, then merged ones (one file at a time)
        for dep in all_dependencies:
            relative_order[dep] = idx
            idx += 1

        return relative_order

    def infer_new_model_name(self) -> dict:
        """Infer whether we are using a model name prefix different from the usual model name as defined from the filename.
        This is useful e.g. when we define a new multi-modal model, and only the text part inherits from `LlamaModel`,
        so we have something like:
        ```python
        class NewModelNameTextDecoderLayer(LlamaDecoderLayer):
            pass
        ```
        with the `Text` prefix added to the model name.
        However, in case of multiple prefix used, we raise a warning and use the most frequent prefix, to avoid parsing
        the same file multiple times and inconsistencies in the objects added from dependencies.
        If the new prefix collides with a prefix of another class in the file where we are importing from, then we also
        raise a warning, and use the default prefix (model name) to avoid collisions in dependencies.
        """
        prefix_model_name_mapping = defaultdict(Counter)
        cased_default_name = get_cased_name(self.model_name)
        # Iterate over all new classes to get modeling super classes
        for class_name, class_node in self.classes.items():
            modeling_bases = [
                k.value.value for k in class_node.bases if k.value.value in self.model_specific_imported_objects
            ]
            if len(modeling_bases) > 1:
                raise ValueError(
                    f"{class_name} was defined with more than 1 model-specific super class. This is unsupported. We found {*modeling_bases,}."
                )
            if len(modeling_bases) == 1:
                filename = self.model_specific_imported_objects[modeling_bases[0]]
                cased_model_name = cased_default_name  # the default name prefix
                suffix = common_partial_suffix(class_name, modeling_bases[0])
                if len(suffix) > 0 and suffix[0].isupper():
                    cased_model_name = class_name.replace(suffix, "")
                prefix_model_name_mapping[filename].update([cased_model_name])

        # Check if we found multiple prefixes for some modeling files
        final_name_mapping = {}
        for file, prefixes_counter in prefix_model_name_mapping.items():
            if len(prefixes_counter) > 1:
                _, total = prefixes_counter.most_common(1)[0]
                most_used_entities = [name for name, count in prefixes_counter.most_common() if count == total]
                # if the default name is in the pool of equally used prefixes, use it, otherwise last encountered
                final_name = cased_default_name if cased_default_name in most_used_entities else most_used_entities[-1]
            else:
                final_name = list(prefixes_counter)[0]
            # Check if the prefix can be used without collisions in the names
            old_cased_model_name = get_cased_name(file.split(".")[-2])
            old_model_name_prefix = final_name.replace(cased_default_name, old_cased_model_name)
            # Raise adequate warning depending on the situation
            has_prefix_collision = f"\nclass {old_model_name_prefix}" in get_module_source_from_name(file)
            if final_name != cased_default_name and has_prefix_collision:
                if len(prefixes_counter) > 1:
                    logger.warning(
                        f"We detected multiple prefix names when inheriting from {file}: {*set(prefixes_counter),}. However, the "
                        f"most used one, '{final_name}', is already present in the source file and will likely cause consistency "
                        f"issues. For this reason we fallback to the default prefix '{cased_default_name}' when grabbing args "
                        "and dependencies. Make sure to subclass the intermediate classes with the prefix you want (if different "
                        f"from '{cased_default_name}') or use a single prefix in all the modular (best)."
                    )
                else:
                    logger.warning(
                        f"We detected the use of the new default prefix {final_name} when inheriting from {file}. However, it is "
                        "already present in the source file and will likely cause consistency issues. For this reason we fallback "
                        f"to the default prefix '{cased_default_name}' when grabbing args and dependencies. Make sure to subclass "
                        f"the intermediate classes with the prefix you want (if different from '{cased_default_name}')"
                    )
                final_name = cased_default_name
            elif len(prefixes_counter) > 1:
                logger.warning(
                    f"We detected multiple prefix names when inheriting from {file}: {*set(prefixes_counter),}. We will only "
                    f"use the most used '{final_name}' prefix when grabbing args and dependencies. Make sure to subclass the "
                    f"intermediate classes with the prefix you want (if different from '{final_name}') or use a single prefix "
                    "in all the modular (best)."
                )
            final_name_mapping[file] = get_lowercase_name(final_name)

        # Check we are not missing imported files
        for file in self.model_specific_modules.keys():
            if file not in final_name_mapping.keys():
                final_name_mapping[file] = self.model_name



SUMMARY:
This is a Python code that appears to be a part of the `transformers` library, specifically handling model merging and renaming. Here's a concise summary:

The code defines several methods for merging models, renaming classes, and handling naming conflicts. It checks if multiple prefixes are used in a modeling file and attempts to use the most frequent or default prefix instead. If a collision is detected, it raises a warning and may use a fallback value.

Key features include:

* Merging models by checking common prefixes
* Renaming classes to avoid naming conflicts
* Fallback behavior for missing imported files

The code requires several external modules, including `tokenizers`, `utils`, and `text_utils`. Some custom functions, such as `get_cased_name` and `common_partial_suffix`, are also used.


key: chroma:document
string_value:
CODE:

def check_dependencies_and_create_import_node(
    file_type: str, new_dependencies: set[str], mapper: ModuleMapper, new_name: str
) -> tuple[set[str], dict[str, cst.CSTNode]]:
    """Check that all class nodes in the `new_dependencies` belong to the correct `file_type`. If this is not the case,
    we need to remove it from the dependencies, and create a new import to it instead.
    This scenario may appear in the following case:
    If a new class in the `modular_xxx.py` file does not belong to `type_xxx.py`, but is used somewhere in `other_type_xxx.py`
    (e.g. as a type hint), but none of the visited files had a similar class, then it would be imported in `type_xxx.py` as
    part of the standard dependency graph (because we never encountered an import towards this new class in any file).
    For example imagine the following `modular.py`:
    ```
    from ..llama.modeling_llama import LlamaModel

    class NewNameTextConfig(PretrainedConfig):
        ...

    class NewNameConfig(PretrainedConfig):
        ...

    class NewNameModel(LlamaModel):
        config = NewNameConfig()
        text_config = NewNameTextConfig()
        ...
    ```
    then without the help of this function, `NewNameTextConfig` would be imported in the `modeling_newname.py` as well as
    `configuration_newname.py`, because `modeling_llama.py` tells us to not import `NewNameConfig`, but has no
    knowledge of `NewNameTextConfig`.
    """
    class_dependencies = {dep for dep in new_dependencies if m.matches(mapper.global_nodes[dep], m.ClassDef())}
    corrected_dependencies = new_dependencies.copy()
    new_imports = {}
    for class_name in class_dependencies:
        class_file_type = find_file_type(class_name)
        # In this case, we need to remove it from the dependencies and create a new import instead
        if class_file_type != file_type:
            corrected_dependencies.remove(class_name)
            import_statement = f"from .{class_file_type}_{new_name} import {class_name}"
            new_imports[class_name] = cst.parse_statement(import_statement)



SUMMARY:
**Function Summary: `check_dependencies_and_create_import_node`**

* **Purpose**: Ensure that class nodes in `new_dependencies` belong to the correct file type. If not, remove it and create a new import statement.
* **Key Implementation Details**:
	+ Filter class dependencies based on their file types using the `matches` method of a `ModuleMapper`.
	+ Create a dictionary `corrected_dependencies` with remaining class nodes that belong to the correct file type.
	+ Generate `import_statement` for each misclassified class node and store it in the `new_imports` dictionary.

**Method Signatures and Interfaces**:

* `ModuleMapper.matches()`: Matches a module with a specific node type (in this case, `ClassDef()`).
* `find_file_type(class_name)`: Returns the file type of a given class name.
* `cst.parse_statement(import_statement)`: Parses an import statement into a CST (Abstract Syntax Tree) Node.

**Notable Dependencies and Requirements**:

* `ModuleMapper`: A module mapper object that provides the necessary methods.
* `find_file_type(class_name)` function: Not shown in this code snippet but presumably defined elsewhere in the project.
* External dependencies:
	+ `cst` (Collection of Statements, Functions, Variables, etc.)


key: chroma:document
string_value:
CODE:

def get_class_node_and_dependencies(
    modular_mapper: ModularFileMapper, class_name: str, node: cst.CSTNode, files: dict[str, dict]
) -> tuple[dict, str, dict]:
    """Return a single class node (and all its dependency nodes), to be added to the `files`. It creates the new
    class node based on the inherited classes if needed. Also returns any new imports of a new class defined in
    the modular that we nay need.
    """
    # An exception was already raised if this has len > 1
    model_specific_bases = [
        k.value.value for k in node.bases if k.value.value in modular_mapper.model_specific_imported_objects
    ]
    super_class = model_specific_bases[0] if len(model_specific_bases) == 1 else None

    file_type = find_file_type(class_name)
    file_to_update = files[file_type]
    model_name = modular_mapper.model_name

    # This is used to avoid adding objects to the dependencies graph if they will be imported already
    imported_objects = modular_mapper.imported_objects_per_file[file_type]

    # We need to replace the class node with the transformers (modeling file) super class node
    if super_class is not None:
        super_file_name = modular_mapper.model_specific_imported_objects[super_class]

        # Get the mapper corresponding to the inherited class
        mapper = modular_mapper.visited_modules[super_file_name]
        # Rename the super class according to the exact same rule we used when renaming the whole module
        renamer = modular_mapper.renamers[super_file_name]
        renamed_super_class = preserve_case_replace(super_class, renamer.patterns, renamer.cased_new_name)

        # Create the new class node
        updated_node = replace_class_node(mapper, node, renamed_super_class, super_class)

        # Grab all immediate dependencies of the new node
        new_node_dependencies = augmented_dependencies_for_class_node(updated_node, mapper, imported_objects)

        # At this point, if any class dependency is found, but belongs to another file, it means that we need to remove
        # it from the dependencies, and add a new import of it instead
        new_node_dependencies, new_imports = check_dependencies_and_create_import_node(
            file_type, new_node_dependencies, mapper, model_name
        )

        # The node was modified -> look for all recursive dependencies of the new node
        all_dependencies_to_add = find_all_dependencies(
            dependency_mapping=mapper.class_dependency_mapping,
            initial_dependencies=new_node_dependencies,
            initial_checked_dependencies=set(file_to_update.keys()),
        )

        relative_dependency_order = mapper.compute_relative_order(all_dependencies_to_add)
        nodes_to_add = {
            dep: (relative_dependency_order[dep], mapper.global_nodes[dep]) for dep in all_dependencies_to_add
        }

    # No transformers (modeling file) super class, just check functions and assignments dependencies
    else:
        updated_node = node
        # The node was NOT modified -> no need to look recursively for other class dependencies. Indeed, even if they are not
        # already defined (which would mean a weird order of the code in the modular...), they will be in the future
        all_dependencies_to_add = augmented_dependencies_for_class_node(updated_node, modular_mapper, imported_objects)

        # At this point, if any class dependency is found, but belongs to another file, it means that we need to remove
        # it from the dependencies, and add a new import of it instead
        all_dependencies_to_add, new_imports = check_dependencies_and_create_import_node(
            file_type, all_dependencies_to_add, modular_mapper, model_name
        )

        relative_dependency_order = modular_mapper.compute_relative_order(all_dependencies_to_add)
        nodes_to_add = {
            dep: (relative_dependency_order[dep], modular_mapper.global_nodes[dep])
            for dep in all_dependencies_to_add
            if dep not in file_to_update.keys()
        }

    # Add the class node itself to the nodes to add
    class_idx = max(relative_dependency_order.values()) + 1 if len(relative_dependency_order) > 0 else 0
    nodes_to_add[class_name] = (class_idx, updated_node)



SUMMARY:
**Summary: get_class_node_and_dependencies Function**

The `get_class_node_and_dependencies` function generates a single class node and its dependencies for a given modular file. It:

1. **Creates the Class Node**: Builds the class node based on inherited classes, if needed, using the provided modular mappings.
2. **Generates New Imports**: Finds new imports of classes defined in the modular that may be required.
3. **Analyzes Dependencies**: Checks dependencies and creates import nodes if necessary, recursively finding all dependencies.
4. **Orders Dependencies**: Computes relative order for the generated dependencies.

**Key Function Details**

* Returns a tuple containing the class node with its updated dependencies, file type, and new imports.
* Uses modular mappings to rename super classes according to exact rules.

**Notable Dependencies and Requirements**

* `modular_mapper` object
	+ `model_specific_imported_objects`
	+ `imported_objects_per_file`
	+ `visited_modules`
	+ `renamers`
* External dependencies:
	+ `find_file_type`
	+ `augmented_dependencies_for_class_node`
	+ `check_dependencies_and_create_import_node`

**Interfaces and Dependencies**

* Uses interfaces from `modular_mapper` object, including `mapper`, `renamer`, and `visited_modules`.
* Returns nodes using a dictionary-like structure with relative dependency order.


key: chroma:document
string_value:
CODE:

def create_modules(modular_mapper: ModularFileMapper) -> dict[str, cst.Module]:
    """Create all the new modules based on visiting the modular file. It replaces all classes as necesary."""
    files = defaultdict(dict)
    current_file_indices = defaultdict(lambda: 0)

    # For each class defined in modular, potentially replace the node and add it with its dependencies
    for class_name, node in modular_mapper.classes.items():
        nodes_to_add, file_type, new_imports = get_class_node_and_dependencies(modular_mapper, class_name, node, files)

        # Add the new potential new imports that we may need to the `modular_mapper` variable
        modular_mapper.imported_objects_per_file[file_type].update(new_imports.keys())
        modular_mapper.imports.extend(list(new_imports.values()))

        # Sort the nodes according to their relative order
        nodes_to_add = sorted(nodes_to_add.items(), key=lambda x: x[1][0])
        # Write all nodes to file
        for dependency, (_, node) in nodes_to_add:
            # This is used to keep certain variables at the beginning of the file
            try:
                # The -1000 is arbitrary -> just keep it bigger than the list
                idx = -1000 + VARIABLES_AT_THE_BEGINNING.index(dependency)
            except ValueError:
                idx = current_file_indices[file_type]
                current_file_indices[file_type] += 1
            files[file_type][dependency] = {"insert_idx": idx, "node": node}

    # Add the __all__ statement to files at the end
    for file_type, node in modular_mapper.all_all_to_add.items():
        idx = current_file_indices[file_type]
        files[file_type]["__all__"] = {"insert_idx": idx, "node": node}

    # Aggregate all the imports statements (we look for duplicates with the code_for_node, not the nodes themselves because
    # they are wrapped in SimpleStatementLine or If which could have different newlines, blanks etc)
    all_imports = modular_mapper.imports.copy()
    all_imports_code = {modular_mapper.python_module.code_for_node(node).strip() for node in all_imports}
    for file, mapper in modular_mapper.visited_modules.items():
        new_imports = [
            node for node in mapper.imports if mapper.python_module.code_for_node(node).strip() not in all_imports_code
        ]
        new_imports_code = {mapper.python_module.code_for_node(node).strip() for node in new_imports}
        all_imports.extend(new_imports)
        all_imports_code.update(new_imports_code)

    # Find the correct imports, and write the new modules
    for file, body in files.items():
        new_body = [k[1]["node"] for k in sorted(body.items(), key=lambda x: x[1]["insert_idx"])]
        needed_imports = get_needed_imports(body, all_imports)
        full_module = needed_imports + new_body
        new_module = cst.Module(body=full_module, header=modular_mapper.python_module.header)
        files[file] = new_module



SUMMARY:
**Summary: Create Modules Function**

*   **Purpose:** Creates new modules based on visiting modular files by replacing classes and adding dependencies.

**Main Purpose and Functionality:**

The `create_modules` function takes a `ModularFileMapper` object as input and returns a dictionary of new modules. It achieves the following key functionality:

1.  Replaces classes in each class definition with updated nodes and added dependencies.
2.  Adds potential new imports to the `modular_mapper.imported_objects_per_file` map.
3.  Sorts nodes according to their relative order for file writing.
4.  Writes all nodes to files, keeping certain variables at the beginning of the file.
5.  Aggregates and finds correct imports to write full modules.

**Key Implementation Details:**

*   Iterates through each class definition in `modular_mapper.classes`.
*   Uses `get_class_node_and_dependencies` function to get updated nodes and dependencies for each class.
*   Updates `imported_objects_per_file` map with new imports.
*   Sorts nodes for file writing using a custom sorting key.
*   Creates a new module by combining needed imports and body.

**Notable Dependencies or Requirements:**

*   `ModularFileMapper` object
*   `get_class_node_and_dependencies` function
*   `VARIABLES_AT_THE_BEGINNING` list
*   `cst.Module` class

**Additional Information:**

This code appears to be part of a larger system designed to work with modular files and their dependencies. The function is closely tied to the `ModularFileMapper` object, which seems to provide necessary data for creating new modules.

The code uses various external libraries or classes (e.g., `cst.Module`, `VARIABLES_AT_THE_BEGINNING`) that are not defined in this snippet.


key: chroma:document
string_value:
CODE:

def convert_modular_file(modular_file):
    pattern = re.search(r"modular_(.*)(?=\.py$)", modular_file)
    output = {}
    if pattern is not None:
        model_name = pattern.groups()[0]
        # Parse the Python file
        with open(modular_file, "r", encoding="utf-8") as file:
            code = file.read()
        module = cst.parse_module(code)
        wrapper = MetadataWrapper(module)
        cst_transformers = ModularFileMapper(module, model_name)
        wrapper.visit(cst_transformers)
        for file, module in create_modules(cst_transformers).items():
            if module != {}:
                # Get relative path starting from src/transformers/
                relative_path = re.search(
                    r"(src/transformers/.*|examples/.*)", os.path.abspath(modular_file).replace("\\", "/")
                ).group(1)

                header = AUTO_GENERATED_MESSAGE.format(
                    relative_path=relative_path, short_name=os.path.basename(relative_path)
                )
                ruffed_code = run_ruff(header + module.code, True)
                formatted_code = run_ruff(ruffed_code, False)
                output[file] = [formatted_code, ruffed_code]
        return output
    else:
        print(f"modular pattern not found in {modular_file}, exiting")


SUMMARY:
**Function Summary:**

`convert_modular_file(modular_file)`

Converts a modular Python file to conform to formatting standards.

**Key Implementation Details:**

- Parses the input modular file using regular expressions and CST (Common SyntaxTree) parser.
- Applies transformations using `MetadataWrapper` and `ModularFileMapper`.
- Formats code using Ruff library on two rounds of review (initial and final).

**Notable Dependencies and Requirements:**

- `re`, `os`, `cst`, and `Ruff` libraries for parsing and formatting purposes.
- Custom classes (`MetadataWrapper`, `ModularFileMapper`) are used to implement transformation logic.


key: chroma:document
string_value:
CODE:

def save_modeling_file(modular_file, converted_file):
    for file_type in converted_file.keys():
        file_name_prefix = file_type.split("*")[0]
        file_name_suffix = file_type.split("*")[-1] if "*" in file_type else ""
        new_file_name = modular_file.replace("modular_", f"{file_name_prefix}_").replace(
            ".py", f"{file_name_suffix}.py"
        )
        non_comment_lines = len(
            [line for line in converted_file[file_type][0].strip().split("\n") if not line.strip().startswith("#")]
        )
        if len(converted_file[file_type][0].strip()) > 0 and non_comment_lines > 0:
            with open(new_file_name, "w", encoding="utf-8") as f:
                f.write(converted_file[file_type][0])
        else:
            non_comment_lines = len(
                [line for line in converted_file[file_type][0].strip().split("\n") if not line.strip().startswith("#")]
            )
            if len(converted_file[file_type][1].strip()) > 0 and non_comment_lines > 0:
                logger.warning("The modeling code contains errors, it's written without formatting")
                with open(new_file_name, "w", encoding="utf-8") as f:


SUMMARY:
**Summary:**

Save Modeling File Functionality
==============================

**Purpose and Functionality**
-----------------------------

This function `save_modeling_file` takes two inputs, `modular_file` and `converted_file`, and saves the contents of `converted_file` to new files with modified names based on their original file types.

**Key Implementation Details:**

*   Iterates over each file type in `converted_file` keys.
*   Creates a new file name by replacing "modular_" prefix and adding `.py` extension suffix based on file type.
*   Checks if the converted file lines contain any non-comment content and writes it to the new file.

**Notable Dependencies and Requirements:**

*   Uses `logger` for warnings during file writing.
*   External dependency on Python's built-in `logging` module (referenced as "logger").
*   Does not inherit or implement any specific interfaces.


key: chroma:document
string_value:
CODE:

def test_custom_files_are_present(transformers_path):
    # Test all the extensions added in the setup
    for file in FILES_TO_FIND:
        if not (transformers_path / file).exists():
            return False


SUMMARY:
**Summary:**

`test_custom_files_are_present`: Verifies that custom files exist at a specified `transformers_path`.

**Key Implementation Details:**

* Iterates through `FILES_TO_FIND` and checks existence of each file using `exists()`.
* Returns `True` if all files are present, `False` otherwise.

**Notable Dependencies/Requirements:**

* `FILES_TO_FIND`: a list of filenames to check (assumed to be defined elsewhere).
* External dependencies not mentioned.
* `transformers_path` is assumed to be an existing directory path.


key: chroma:document
string_value:
CODE:

def get_all_model_names():
    model_names = set()
    # Each auto modeling files contains multiple mappings. Let's get them in a dynamic way.
    for module_name in ["modeling_auto", "modeling_tf_auto", "modeling_flax_auto"]:
        module = getattr(transformers.models.auto, module_name, None)
        if module is None:
            continue
        # all mappings in a single auto modeling file
        mapping_names = [
            x
            for x in dir(module)
            if x.endswith("_MAPPING_NAMES")
            and (x.startswith("MODEL_") or x.startswith("TF_MODEL_") or x.startswith("FLAX_MODEL_"))
        ]
        for name in mapping_names:
            mapping = getattr(module, name)
            if mapping is not None:
                for v in mapping.values():
                    if isinstance(v, (list, tuple)):
                        model_names.update(v)
                    elif isinstance(v, str):
                        model_names.add(v)



SUMMARY:
**Function Summary: get_all_model_names()**

The `get_all_model_names()` function extracts and returns a set of unique model names from auto-modeling files in the `transformers.models.auto` namespace.

**Key Implementation Details:**

* Iterates through three auto-modeling files (`modeling_auto`, `modeling_tf_auto`, `modeling_flax_auto`) to gather model mappings.
* Uses string matching (`str.endswith()`) and object existence checks (`isinstance()` and `is None`) to identify relevant model names.

**Notable Dependencies/Requirements:**

* `transformers.models.auto` namespace
* No external dependencies listed.


key: chroma:document
string_value:
CODE:

def get_tiny_model_names_from_repo():
    # All model names defined in auto mappings
    model_names = set(get_all_model_names())

    with open("tests/utils/tiny_model_summary.json") as fp:
        tiny_model_info = json.load(fp)
    tiny_models_names = set()
    for model_base_name in tiny_model_info:
        tiny_models_names.update(tiny_model_info[model_base_name]["model_classes"])

    # Remove a tiny model name if one of its framework implementation hasn't yet a tiny version on the Hub.
    not_on_hub = model_names.difference(tiny_models_names)
    for model_name in copy.copy(tiny_models_names):
        if not model_name.startswith("TF") and f"TF{model_name}" in not_on_hub:
            tiny_models_names.remove(model_name)
        elif model_name.startswith("TF") and model_name[2:] in not_on_hub:
            tiny_models_names.remove(model_name)



SUMMARY:
**Concise Summary: `get_tiny_model_names_from_repo`**

**Purpose:** Retrieves the names of tiny models defined in auto-mappings and filtering out those not on the Hub.

**Key Implementation Details:**

* Uses `get_all_model_names()` to get all model names, then filters using a set operation (`difference` and `update`) between model names and their implementations on the Hub (`tiny_models_names`).
* Checks for the presence of "TF" prefix in a specific format (e.g., "TFx[model_name]"), removing models that don't have a matching tiny version.
* Uses JSON data from "tests/utils/tiny_model_summary.json" to infer available model classes.

**Dependencies/Requirements:**

* `get_all_model_names()`
* `json` module
* `copy` module
* External dependency on "tests/utils/tiny_model_summary.json" file.


key: chroma:document
string_value:
CODE:

def get_tiny_model_summary_from_hub(output_path):
    special_models = COMPOSITE_MODELS.values()

    # All tiny model base names on Hub
    model_names = get_all_model_names()
    models = hf_api.list_models(
        filter=ModelFilter(
            author="hf-internal-testing",
        )
    )
    _models = set()
    for x in models:
        model = x.id
        org, model = model.split("/")
        if not model.startswith("tiny-random-"):
            continue
        model = model.replace("tiny-random-", "")
        if not model[0].isupper():
            continue
        if model not in model_names and model not in special_models:
            continue
        _models.add(model)

    models = sorted(_models)
    # All tiny model names on Hub
    summary = {}
    for model in models:
        repo_id = f"hf-internal-testing/tiny-random-{model}"
        model = model.split("-")[0]
        try:
            repo_info = hf_api.repo_info(repo_id)
            content = {
                "tokenizer_classes": set(),
                "processor_classes": set(),
                "model_classes": set(),
                "sha": repo_info.sha,
            }
        except Exception:
            continue
        try:
            time.sleep(1)
            tokenizer_fast = AutoTokenizer.from_pretrained(repo_id)
            content["tokenizer_classes"].add(tokenizer_fast.__class__.__name__)
        except Exception:
            pass
        try:
            time.sleep(1)
            tokenizer_slow = AutoTokenizer.from_pretrained(repo_id, use_fast=False)
            content["tokenizer_classes"].add(tokenizer_slow.__class__.__name__)
        except Exception:
            pass
        try:
            time.sleep(1)
            img_p = AutoImageProcessor.from_pretrained(repo_id)
            content["processor_classes"].add(img_p.__class__.__name__)
        except Exception:
            pass
        try:
            time.sleep(1)
            feat_p = AutoFeatureExtractor.from_pretrained(repo_id)
            if not isinstance(feat_p, BaseImageProcessor):
                content["processor_classes"].add(feat_p.__class__.__name__)
        except Exception:
            pass
        try:
            time.sleep(1)
            model_class = getattr(transformers, model)
            m = model_class.from_pretrained(repo_id)
            content["model_classes"].add(m.__class__.__name__)
        except Exception:
            pass
        try:
            time.sleep(1)
            model_class = getattr(transformers, f"TF{model}")
            m = model_class.from_pretrained(repo_id)
            content["model_classes"].add(m.__class__.__name__)
        except Exception:
            pass

        content["tokenizer_classes"] = sorted(content["tokenizer_classes"])
        content["processor_classes"] = sorted(content["processor_classes"])
        content["model_classes"] = sorted(content["model_classes"])

        summary[model] = content
        with open(os.path.join(output_path, "hub_tiny_model_summary.json"), "w") as fp:


SUMMARY:
**Function Summary: `get_tiny_model_summary_from_hub`**

**Purpose:** Fetches summaries for tiny models from the Hugging Face model hub.

**Main Purpose**: The function retrieves a list of tiny models from the Hugging Face model hub, constructs their metadata (e.g., tokenizer, processor, and model classes), and stores it in a JSON file.

**Key Implementation Details:**

*   Filters models by author and namespace.
*   Iterates through each model, retrieving its metadata using the `hf_api` interface.
*   Handles cases where model metadata is not available, waiting 1 second between attempts.
*   Extracts tokenizer classes from slow and fast versions of the AutoTokenizer.
*   Extracts processor classes from image processors (AutoImageProcessor).
*   Extracts model classes from transformers module using string-based lookup.

**Notable Dependencies:**

*   `hf_api`: Hugging Face API interface for interacting with the model hub.
*   `transformers`: Hugging Face transformer library for loading pre-trained models.
*   ` AutoTokenizer`, `AutoImageProcessor`, and `AutoFeatureExtractor` classes from transformers module.

**External Requirements:**

*   Requires access to Hugging Face API subscription or trial account.


key: chroma:document
string_value:
CODE:

def check_missing_backends():
    """
    Checks if all backends are installed (otherwise the check of this script is incomplete). Will error in the CI if
    that's not the case but only throw a warning for users running this.
    """
    missing_backends = []
    if not is_torch_available():
        missing_backends.append("PyTorch")
    if not is_tf_available():
        missing_backends.append("TensorFlow")
    if not is_flax_available():
        missing_backends.append("Flax")
    if len(missing_backends) > 0:
        missing = ", ".join(missing_backends)
        if os.getenv("TRANSFORMERS_IS_CI", "").upper() in ENV_VARS_TRUE_VALUES:
            raise Exception(
                "Full repo consistency checks require all backends to be installed (with `pip install -e '.[dev]'` in the "
                f"Transformers repo, the following are missing: {missing}."
            )
        else:
            warnings.warn(
                "Full repo consistency checks require all backends to be installed (with `pip install -e '.[dev]'` in the "
                f"Transformers repo, the following are missing: {missing}. While it's probably fine as long as you "
                "didn't make any change in one of those backends modeling files, you should probably execute the "
                "command above to be on the safe side."


SUMMARY:
**Summary**

The `check_missing_backends` function checks if all required backends (PyTorch, TensorFlow, and Flax) are installed. It:

* Compares availability of each backend using `is_`:tch_available`, `is_tf_available`, and `is_flax_available`
* If any backend is missing, it appends the name to `missing_backends` list
* Raises an exception in CI if all backends are not installed or a warning for users otherwise
* Provides instructions on how to install missing backends using pip

**Key Implementation Details**

* Checks availability of three backends using separate functions (`is_torch_available`, `is_tf_available`, and `is_flax_available`)
* Uses an exception to raise error in CI if all backends are not installed
* Uses a warning to notify users about the missing backends
* Provides a pip installation command to install missing backends

**Notable Dependencies/Requirements**

* `pytorch`: PyTorch backend
* `tensorflow`: TensorFlow backend
* `flax`: Flax backend
* `transformers`: Transformers repository
* `os`: Operating system dependency for environment variable access


key: chroma:document
string_value:
CODE:

def check_model_list():
    """
    Checks the model listed as subfolders of `models` match the models available in `transformers.models`.
    """
    # Get the models from the directory structure of `src/transformers/models/`
    models_dir = os.path.join(PATH_TO_TRANSFORMERS, "models")
    _models = []
    for model in os.listdir(models_dir):
        if model == "deprecated":
            continue
        model_dir = os.path.join(models_dir, model)
        if os.path.isdir(model_dir) and "__init__.py" in os.listdir(model_dir):
            _models.append(model)

    # Get the models in the submodule `transformers.models`
    models = [model for model in dir(transformers.models) if not model.startswith("__")]

    missing_models = sorted(set(_models).difference(models))
    if missing_models:
        raise Exception(
            f"The following models should be included in {models_dir}/__init__.py: {','.join(missing_models)}."


SUMMARY:
**Function Summary:** `check_model_list`

**Primary Purpose:** Validates that the models listed as subfolders of `src/transformers/models` match the available models in `transformers.models`.

**Key Implementation Details:**

*   Loads all non-deprecated subfolder models from `src/transformers/models/` into `_models`.
*   Compares the list of models in `_models` against the models listed in `transformers.models`.
*   Raising exception when there are missing models that should be included.

**Notable Dependencies:**

*   `os.path`: for path manipulation (e.g., joining paths).
*   `PATH_TO_TRANSFORMERS`: a defined constant referencing to the transformers directory.


key: docstring
string_value:
# If some modeling modules should be ignored for all checks, they should be added in the nested list
# _ignore_modules of this function.


key: chroma:document
string_value:
CODE:
# _ignore_modules of this function.
def get_model_modules() -> List[str]:
    """Get all the model modules inside the transformers library (except deprecated models)."""
    _ignore_modules = [
        "modeling_auto",
        "modeling_encoder_decoder",
        "modeling_marian",
        "modeling_retribert",
        "modeling_flax_auto",
        "modeling_flax_encoder_decoder",
        "modeling_speech_encoder_decoder",
        "modeling_flax_speech_encoder_decoder",
        "modeling_flax_vision_encoder_decoder",
        "modeling_timm_backbone",
        "modeling_tf_auto",
        "modeling_tf_encoder_decoder",
        "modeling_tf_vision_encoder_decoder",
        "modeling_vision_encoder_decoder",
    ]
    modules = []
    for model in dir(transformers.models):
        # There are some magic dunder attributes in the dir, we ignore them
        if model == "deprecated" or model.startswith("__"):
            continue

        model_module = getattr(transformers.models, model)
        for submodule in dir(model_module):
            if submodule.startswith("modeling") and submodule not in _ignore_modules:
                modeling_module = getattr(model_module, submodule)
                modules.append(modeling_module)


SUMMARY:
**Summary**

This function, `get_model_modules`, retrieves all model modules from the `transformers` library, excluding deprecated and specific "modeling" submodules. The primary purpose is to gather a list of models for use in a larger application.

**Key Implementation Details:**

*   ignores ignored module names (`_ignore_modules`) and magic dunder attributes in the directory listing.
*   uses `getattr()` to dynamically access individual model modules and their "modeling" submodules.
*   excludes deprecated model names detected as "__deprecated__".

**Notable Dependencies/Requirements:**

*   The `transformers` library, specifically its `models` module.

**Method Signatures Interfaces:**

*   `get_model_modules() -> List[str]`: Returns a list of model modules in the transformed format, excluding obsolete ones.


key: chroma:document
string_value:
CODE:

def get_models(module: types.ModuleType, include_pretrained: bool = False) -> List[Tuple[str, type]]:
    """
    Get the objects in a module that are models.

    Args:
        module (`types.ModuleType`):
            The module from which we are extracting models.
        include_pretrained (`bool`, *optional*, defaults to `False`):
            Whether or not to include the `PreTrainedModel` subclass (like `BertPreTrainedModel`) or not.

    Returns:
        List[Tuple[str, type]]: List of models as tuples (class name, actual class).
    """
    models = []
    model_classes = (transformers.PreTrainedModel, transformers.TFPreTrainedModel, transformers.FlaxPreTrainedModel)
    for attr_name in dir(module):
        if not include_pretrained and ("Pretrained" in attr_name or "PreTrained" in attr_name):
            continue
        attr = getattr(module, attr_name)
        if isinstance(attr, type) and issubclass(attr, model_classes) and attr.__module__ == module.__name__:
            models.append((attr_name, attr))


SUMMARY:
Here's a concise and focused summary of the given function:

### Function Summary: `get_models`

**Purpose:** Extract objects from a module that are models.

**Primary Purpose:** Return a list of model classes as tuples (class name, actual class) based on their inheritance relation to predefined model classes.

**Key Implementation Details:**

* Iterates over attributes in the given module.
* Filters out attributes whose names contain keywords related to `PreTrainedModel` subclass (excluding them by default).
* Checks if each attribute is a type and inherits from defined model classes using multiple inheritance.
* Appends validated attributes to the result list.

**Notable Dependencies/Requirements:**

* `transformers`: Module containing definition of predefined model classes.
* `types`: Module for accessing built-in types like `ModuleType`.

### Return Value:

A list of tuples (class name, actual class) representing extracted model objects.


key: chroma:document
string_value:
CODE:

def is_building_block(model: str) -> bool:
    """
    Returns `True` if a model is a building block part of a bigger model.
    """
    if model.endswith("Wrapper"):
        return True
    if model.endswith("Encoder"):
        return True
    if model.endswith("Decoder"):
        return True
    if model.endswith("Prenet"):


SUMMARY:
Here is a concise summary of the function:

**Function Summary**

`is_building_block(model: str) -> bool`

Checks if a machine learning model is a building block part of a larger model. It returns `True` for models ending with "Wrapper", "Encoder", "Decoder", or "Prenet".

**Key Implementation Details**

* Uses string suffix matching (`str.endswith()`)
* Only checks for specific model prefixes (no regular expression)

**Dependencies and Interfaces Noted**

* No dependencies are mentioned in the function code.
* Inherited or implemented interfaces not applicable.

This summary is concise, accurate, and focused on providing essential information about the function's purpose and implementation details.


key: chroma:document
string_value:
CODE:

def is_a_private_model(model: str) -> bool:
    """Returns `True` if the model should not be in the main init."""
    if model in PRIVATE_MODELS:
        return True


SUMMARY:
**Summary**

Determines whether a given model should exclude itself from the main initialization.

**Functionality**

- Checks if the model is in the `PRIVATE Modes` list, returning `True` if it is.
- Falls back to calling `is_building_block` with the model as an argument to determine its behavior.

**Key Implementation Details**

- Method signature: `def is_a_private_model(model: str) -> bool`
- Calls `PRIVATE_MODELS` set to determine private models
- Calls `is_building_block` method on a given model

**Notable Dependencies or Requirements**

- Uses the `PRIVATE.Models` set, which is not defined in this snippet. 
  - References related class 'PrivateModels'
  
Note: No external dependencies are listed as this function only references internal sets and methods.


key: chroma:document
string_value:
CODE:

def check_models_are_in_init():
    """Checks all models defined in the library are in the main init."""
    models_not_in_init = []
    dir_transformers = dir(transformers)
    for module in get_model_modules():
        models_not_in_init += [
            model[0] for model in get_models(module, include_pretrained=True) if model[0] not in dir_transformers
        ]

    # Remove private models
    models_not_in_init = [model for model in models_not_in_init if not is_a_private_model(model)]
    if len(models_not_in_init) > 0:


SUMMARY:
**Function Summary**

`check_models_are_in_init()`
=====================

Checks if defined model libraries are deployed in the main application initialization.

**Primary Purpose**

Verifies that all models imported from external modules are included in the main `init.py` file of the library.

**Key Implementation Details**

* Iterates over model modules using `get_model_modules()` and collects models not present in `transformers` directory
* Filters out private models using `is_a_private_model()`
* Raises an exception if any models are missing from main init

**Notable Dependencies and Requirements**

* `transformers`: used to define the expected models directory
* `get_model_modules()`, `get_models()`, and `is_a_private_model()`: functions not shown in this code snippet, but assumed to be defined elsewhere in the library.


key: docstring
string_value:
# If some test_modeling files should be ignored when checking models are all tested, they should be added in the
# nested list _ignore_files of this function.


key: chroma:document
string_value:
CODE:
# nested list _ignore_files of this function.
def get_model_test_files() -> List[str]:
    """
    Get the model test files.

    Returns:
        `List[str]`: The list of test files. The returned files will NOT contain the `tests` (i.e. `PATH_TO_TESTS`
        defined in this script). They will be considered as paths relative to `tests`. A caller has to use
        `os.path.join(PATH_TO_TESTS, ...)` to access the files.
    """

    _ignore_files = [
        "test_modeling_common",
        "test_modeling_encoder_decoder",
        "test_modeling_flax_encoder_decoder",
        "test_modeling_flax_speech_encoder_decoder",
        "test_modeling_marian",
        "test_modeling_tf_common",
        "test_modeling_tf_encoder_decoder",
    ]
    test_files = []
    model_test_root = os.path.join(PATH_TO_TESTS, "models")
    model_test_dirs = []
    for x in os.listdir(model_test_root):
        x = os.path.join(model_test_root, x)
        if os.path.isdir(x):
            model_test_dirs.append(x)

    for target_dir in [PATH_TO_TESTS] + model_test_dirs:
        for file_or_dir in os.listdir(target_dir):
            path = os.path.join(target_dir, file_or_dir)
            if os.path.isfile(path):
                filename = os.path.split(path)[-1]
                if "test_modeling" in filename and os.path.splitext(filename)[0] not in _ignore_files:
                    file = os.path.join(*path.split(os.sep)[1:])
                    test_files.append(file)



SUMMARY:
**Function Summary:**

`get_model_test_files()` 

 Retrieves and returns a list of test files ("test_modeling" files) within the `models` directory under the test root. Excludes ignored files from specified `_ignore_files`.

**Key Details:**

* Ignored files are specified in `_ignore_files`.
* Files with "test_modeling" in their filename, excluding those in `_ignore_files`, are considered.
* Files are joined using `os.path.join()` to construct absolute paths.

**Notable Dependencies:**

* `os`: for file system interactions
* `PATH_TO_TESTS`: script-defined constant for test root path


key: docstring
string_value:
# This is a bit hacky but I didn't find a way to import the test_file as a module and read inside the tester class
# for the all_model_classes variable.


key: chroma:document
string_value:
CODE:
# for the all_model_classes variable.
def find_tested_models(test_file: str) -> List[str]:
    """
    Parse the content of test_file to detect what's in `all_model_classes`. This detects the models that inherit from
    the common test class.

    Args:
        test_file (`str`): The path to the test file to check

    Returns:
        `List[str]`: The list of models tested in that file.
    """
    with open(os.path.join(PATH_TO_TESTS, test_file), "r", encoding="utf-8", newline="\n") as f:
        content = f.read()
    all_models = re.findall(r"all_model_classes\s+=\s+\(\s*\(([^\)]*)\)", content)
    # Check with one less parenthesis as well
    all_models += re.findall(r"all_model_classes\s+=\s+\(([^\)]*)\)", content)
    if len(all_models) > 0:
        model_tested = []
        for entry in all_models:
            for line in entry.split(","):
                name = line.strip()
                if len(name) > 0:
                    model_tested.append(name)


SUMMARY:
**Function Summary**

`find_tested_models(test_file: str) -> List[str]`

Detects models tested in a given test file by parsing its content, specifically extracting `all_model_classes`.

**Key Functionality**

* Reads the content of a test file
* Uses regular expressions to extract `all_model_classes` patterns
* Parses pattern entries to extract model names

**Implementation Details**

* Method signatures: `re.findall` with two pattern strings for `all_model_classes`
* Algorithm: Combines multiple pattern findings into a single list using `.append`

**Notable Dependencies/Requirements**

* `os.path.join` and `open` for file operations
* `re` module for regular expression matching

**No notable inherited or implemented interfaces.**


key: chroma:document
string_value:
CODE:

def should_be_tested(model_name: str) -> bool:
    """
    Whether or not a model should be tested.
    """
    if model_name in IGNORE_NON_TESTED:
        return False


SUMMARY:
**Function Summary**

### `should_be_tested`

Determines whether a model with the given name should be tested.

**Purpose**: Checks if a model's existence in an IGNORE_NON_TESTED list or if it's a building block model to exclude testing.

**Key Implementation Details**:

* Returns `False` for models in `IGNORE NON_TESTED`.
* Excludes models with this method signature from being tested.
* References `is_building_block(model_name)` using the exact name of the function.
* Inherently uses an external list (`IGNORE_NON_TESTED`) without defining it within this code snippet.

**Notable Dependencies/Requirements**: 

* `IGNORE_NON_TESTED` external list or constant.


key: chroma:document
string_value:
CODE:

def check_models_are_tested(module: types.ModuleType, test_file: str) -> List[str]:
    """Check models defined in a module are all tested in a given file.

    Args:
        module (`types.ModuleType`): The module in which we get the models.
        test_file (`str`): The path to the file where the module is tested.

    Returns:
        `List[str]`: The list of error messages corresponding to models not tested.
    """
    # XxxPreTrainedModel are not tested
    defined_models = get_models(module)
    tested_models = find_tested_models(test_file)
    if tested_models is None:
        if test_file.replace(os.path.sep, "/") in TEST_FILES_WITH_NO_COMMON_TESTS:
            return
        return [
            f"{test_file} should define `all_model_classes` to apply common tests to the models it tests. "
            + "If this intentional, add the test filename to `TEST_FILES_WITH_NO_COMMON_TESTS` in the file "
            + "`utils/check_repo.py`."
        ]
    failures = []
    for model_name, _ in defined_models:
        if model_name not in tested_models and should_be_tested(model_name):
            failures.append(
                f"{model_name} is defined in {module.__name__} but is not tested in "
                + f"{os.path.join(PATH_TO_TESTS, test_file)}. Add it to the all_model_classes in that file."
                + "If common tests should not applied to that model, add its name to `IGNORE_NON_TESTED`"
                + "in the file `utils/check_repo.py`."
            )


SUMMARY:
**Summary:**

This function checks if all models defined in a module (`module`) are tested in a given test file (`test_file`). It returns a list of error messages for models that are not tested.

**Key Details:**

* Defines and tests models from `defined_models` and `tested_models` respectively
* Returns error messages for models that are not tested but should be (`model_name`)
* Notifies user to add model to `all_model_classes` in test file or ignore with `IGNORE_NON_TESTED`
 
**Function Signature:**
```python
def check_models_are_tested(module: types.ModuleType, test_file: str) -> List[str]:
```

**Notable Dependencies/Requirements:**

* `get_models`: module for getting defined models
* `find_tested_models`: function for finding tested models in a file
* `TEST_FILES_WITH_NO_COMMON_TESTS`: dictionary of files with no common tests
* `utils/check_repo.py`: configuration file for repository checks
* `types.ModuleType`: type hint for a Python module


key: chroma:document
string_value:
CODE:

def check_all_models_are_tested():
    """Check all models are properly tested."""
    modules = get_model_modules()
    test_files = get_model_test_files()
    failures = []
    for module in modules:
        # Matches a module to its test file.
        test_file = [file for file in test_files if f"test_{module.__name__.split('.')[-1]}.py" in file]
        if len(test_file) == 0:
            failures.append(f"{module.__name__} does not have its corresponding test file {test_file}.")
        elif len(test_file) > 1:
            failures.append(f"{module.__name__} has several test files: {test_file}.")
        else:
            test_file = test_file[0]
            new_failures = check_models_are_tested(module, test_file)
            if new_failures is not None:
                failures += new_failures
    if len(failures) > 0:


SUMMARY:
**Summary**

*   Function: `check_all_models_are_tested()`
*   **Purpose:** Verifies if all models have corresponding test files and reports any mismatches.
*   Key implementation:
    *   Retrieves module metadata (`get_model_modules()`).
    *   Iterates over each module, matching it to its test file based on the module name prefix.
    *   Calls a separate function (`check_models_are_tested()`) for modules with test files and combines any returned errors.
*   **Notable dependencies:**
    *   `get_model_modules()` (module metadata retrieval)
    *   `get_model_test_files()` (test file retrieval)

**External interfaces/deependencies:**

    - Check Model Functions
   


key: chroma:document
string_value:
CODE:

def get_all_auto_configured_models() -> List[str]:
    """Return the list of all models in at least one auto class."""
    result = set()  # To avoid duplicates we concatenate all model classes in a set.
    if is_torch_available():
        for attr_name in dir(transformers.models.auto.modeling_auto):
            if attr_name.startswith("MODEL_") and attr_name.endswith("MAPPING_NAMES"):
                result = result | set(get_values(getattr(transformers.models.auto.modeling_auto, attr_name)))
    if is_tf_available():
        for attr_name in dir(transformers.models.auto.modeling_tf_auto):
            if attr_name.startswith("TF_MODEL_") and attr_name.endswith("MAPPING_NAMES"):
                result = result | set(get_values(getattr(transformers.models.auto.modeling_tf_auto, attr_name)))
    if is_flax_available():
        for attr_name in dir(transformers.models.auto.modeling_flax_auto):
            if attr_name.startswith("FLAX_MODEL_") and attr_name.endswith("MAPPING_NAMES"):
                result = result | set(get_values(getattr(transformers.models.auto.modeling_flax_auto, attr_name)))


SUMMARY:
**Function Summary: `get_all_auto_configured_models`**

**Purpose:** Extracts and returns a list of model names from auto-configured models in at least one available framework (PyTorch, TensorFlow, or Flax).

**Implementation Details:**

* Iterates over specific attribute names (`MODEL_`, `TF_MODEL_`, and `FLAX_MODEL_`) and their corresponding mapping names within each auto-configured model.
* Uses set operations to combine unique model names from different frameworks.

**Notable Dependencies:**

* `transformers.models.auto.modeling_auto`
* `transformers.models.auto.modeling_tf_auto`
* `transformers.models.auto.modeling_flax_auto`
* `is_torch_available()`, `is_tf_available()`, and `is_flax_available()` (functions to check availability of respective frameworks)

**Return Value:** A list of strings representing model names from auto-configured models in at least one available framework.


key: chroma:document
string_value:
CODE:

def ignore_unautoclassed(model_name: str) -> bool:
    """Rules to determine if a model should be in an auto class."""
    # Special white list
    if model_name in IGNORE_NON_AUTO_CONFIGURED:
        return True
    # Encoder and Decoder should be ignored
    if "Encoder" in model_name or "Decoder" in model_name:
        return True


SUMMARY:
**Summary:**
This function determines whether a model with the given `model_name` should be in an auto-class. It checks against a special whitelist and ensures that specific models like "Encoder" and "Decoder" are ignored.

**Key Details:**

* **Functionality:** `ignore_unautoclassed(model_name) -> bool`
* **Purpose:** Determine if a model is auto-configured
* **Design Choice:** Whitelisting approach with specific exceptions for "Encoder" and "Decoder" models.
* **Dependencies/Requirements:**
	+ `IGNORE_NON_AUTO_CONFIGURED`: A predefined list of accepted non-auto-configured model names.

Note: The original code snippet included the `IGNORE_NON_AUTO_CONFIGURED` variable, which is likely defined elsewhere in the project. This summary assumes its existence for the purpose of analysis.


key: chroma:document
string_value:
CODE:

def check_models_are_auto_configured(module: types.ModuleType, all_auto_models: List[str]) -> List[str]:
    """
    Check models defined in module are each in an auto class.

    Args:
        module (`types.ModuleType`):
            The module in which we get the models.
        all_auto_models (`List[str]`):
            The list of all models in an auto class (as obtained with `get_all_auto_configured_models()`).

    Returns:
        `List[str]`: The list of error messages corresponding to models not tested.
    """
    defined_models = get_models(module)
    failures = []
    for model_name, _ in defined_models:
        if model_name not in all_auto_models and not ignore_unautoclassed(model_name):
            failures.append(
                f"{model_name} is defined in {module.__name__} but is not present in any of the auto mapping. "
                "If that is intended behavior, add its name to `IGNORE_NON_AUTO_CONFIGURED` in the file "
                "`utils/check_repo.py`."
            )


SUMMARY:
**Summary:**

### Function Purpose:

`check_models_are_auto_configured` checks if models defined in a module are part of an auto class.

### Key Implementation Details:

*   Calculates the list of defined models from the input `module`.
*   Iterates over each model, verifying if it exists in the `all_auto_models` list and ignoring unautoclassed models.
*   Appends error messages to the `failures` list for non-existent models.

### Notable Dependencies/Requirements:

*   `get_all_auto_configured_models()`: Returns a list of all models in an auto class.
*   `ignore_unautoclassed(model_name)`: Function indicating whether a model is intended to be ignored.
*   `IGNORE_NON_AUTO_CONFIGURED` constant: File location for modifying ignore behavior.


key: chroma:document
string_value:
CODE:

def check_all_models_are_auto_configured():
    """Check all models are each in an auto class."""
    # This is where we need to check we have all backends or the check is incomplete.
    check_missing_backends()
    modules = get_model_modules()
    all_auto_models = get_all_auto_configured_models()
    failures = []
    for module in modules:
        new_failures = check_models_are_auto_configured(module, all_auto_models)
        if new_failures is not None:
            failures += new_failures
    if len(failures) > 0:


SUMMARY:
**Summary:**

`check_all_models_are_auto_configured()` function checks whether all models are configured as auto-configurable. It verifies that all required backends exist and iterates through modules to validate individual model configurations, accumulating failures in a list.

**Key Details:**

* Verifies existence of all required backends
* Iterates through modules to validate model configurations
* Checks for `None` return value from `check_models_are_auto_configured()`
* Accumulates failure messages and raises an exception

**Notable Dependencies/Requirements:**

* `get_model_modules()` function
* `get_all_auto_configured_models()` function
* `check_missing_backends()` function


key: chroma:document
string_value:
CODE:

def check_all_auto_object_names_being_defined():
    """Check all names defined in auto (name) mappings exist in the library."""
    # This is where we need to check we have all backends or the check is incomplete.
    check_missing_backends()

    failures = []
    mappings_to_check = {
        "TOKENIZER_MAPPING_NAMES": TOKENIZER_MAPPING_NAMES,
        "IMAGE_PROCESSOR_MAPPING_NAMES": IMAGE_PROCESSOR_MAPPING_NAMES,
        "FEATURE_EXTRACTOR_MAPPING_NAMES": FEATURE_EXTRACTOR_MAPPING_NAMES,
        "PROCESSOR_MAPPING_NAMES": PROCESSOR_MAPPING_NAMES,
    }

    # Each auto modeling files contains multiple mappings. Let's get them in a dynamic way.
    for module_name in ["modeling_auto", "modeling_tf_auto", "modeling_flax_auto"]:
        module = getattr(transformers.models.auto, module_name, None)
        if module is None:
            continue
        # all mappings in a single auto modeling file
        mapping_names = [x for x in dir(module) if x.endswith("_MAPPING_NAMES")]
        mappings_to_check.update({name: getattr(module, name) for name in mapping_names})

    for name, mapping in mappings_to_check.items():
        for _, class_names in mapping.items():
            if not isinstance(class_names, tuple):
                class_names = (class_names,)
                for class_name in class_names:
                    if class_name is None:
                        continue
                    # dummy object is accepted
                    if not hasattr(transformers, class_name):
                        # If the class name is in a model name mapping, let's not check if there is a definition in any modeling
                        # module, if it's a private model defined in this file.
                        if name.endswith("MODEL_MAPPING_NAMES") and is_a_private_model(class_name):
                            continue
                        if name.endswith("MODEL_FOR_IMAGE_MAPPING_NAMES") and is_a_private_model(class_name):
                            continue
                        failures.append(
                            f"`{class_name}` appears in the mapping `{name}` but it is not defined in the library."
                        )
    if len(failures) > 0:


SUMMARY:
**Summary:**

The `check_all_auto_object_names_being_defined()` function checks if all names defined in auto mapping mappings exist in the library. It iterates through various auto modeling files, maps, and class definitions to identify missing objects.

**Key Functionality:**

* Checks for missing classes derived from `TOKENIZER_MAPPING_NAMES`, `IMAGE_PROCESSOR_MAPPING_NAMES`, etc.
* Iterates through auto modeling files (`modeling_auto`, `modeling_tf_auto`, `modeling_flax_auto`) and extracts mapping names and their corresponding class definitions
* Reports missing classes in the following format: "<class_name> appears in <mapping-name> but it is not defined in the library."

**Notable Dependencies/Requirements:**

* `transformers.models.auto` module
* External dependencies:
	+ `transformers`
	+ `torch`
	+ `jittor`
	(+ other transformer-based libraries)
* Inherited Interfaces:
	+ `TOKENIZER_MAPPING_NAMES`, `IMAGE_PROCESSOR_MAPPING_NAMES`, etc.

**Implementation Details:**

* Uses dynamic imports and attribute access to extract mapping names and class definitions.
* Includes private model checks for certain mapping types.

**Primary Purpose:**
The primary purpose of this function is to identify missing classes derived from auto mapping mappings, raising an exception if any are found.


key: chroma:document
string_value:
CODE:

def check_all_auto_mapping_names_in_config_mapping_names():
    """Check all keys defined in auto mappings (mappings of names) appear in `CONFIG_MAPPING_NAMES`."""
    # This is where we need to check we have all backends or the check is incomplete.
    check_missing_backends()

    failures = []
    # `TOKENIZER_PROCESSOR_MAPPING_NAMES` and `AutoTokenizer` is special, and don't need to follow the rule.
    mappings_to_check = {
        "IMAGE_PROCESSOR_MAPPING_NAMES": IMAGE_PROCESSOR_MAPPING_NAMES,
        "FEATURE_EXTRACTOR_MAPPING_NAMES": FEATURE_EXTRACTOR_MAPPING_NAMES,
        "PROCESSOR_MAPPING_NAMES": PROCESSOR_MAPPING_NAMES,
    }

    # Each auto modeling files contains multiple mappings. Let's get them in a dynamic way.
    for module_name in ["modeling_auto", "modeling_tf_auto", "modeling_flax_auto"]:
        module = getattr(transformers.models.auto, module_name, None)
        if module is None:
            continue
        # all mappings in a single auto modeling file
        mapping_names = [x for x in dir(module) if x.endswith("_MAPPING_NAMES")]
        mappings_to_check.update({name: getattr(module, name) for name in mapping_names})

    for name, mapping in mappings_to_check.items():
        for model_type in mapping:
            if model_type not in CONFIG_MAPPING_NAMES:
                failures.append(
                    f"`{model_type}` appears in the mapping `{name}` but it is not defined in the keys of "
                    "`CONFIG_MAPPING_NAMES`."
                )
    if len(failures) > 0:


SUMMARY:
**CONCISE SUMMARY**

### Function Purpose:

DefectsCheckAutoMappingNames checks if all keys defined in auto mapping names appear in CONFIG_MAPPING_NAMES.

### Key Implementation Details:

- Iterates through special mapping names and their corresponding values from `TOKENIZER_PROCESSOR_MAPPING_NAMES` and `AutoTokenizer`.
- Compares model types in auto modeling files with those in CONFIG_MAPPING_NAMES.
- Raises an exception when inconsistencies are found.

### Notable Dependencies/Requirements:

- `CONFIG_MAPPING_NAMES`
- `IMAGE_PROCESSOR_MAPPING_NAMES`
- `FEATURE_EXTRACTOR_MAPPING_NAMES`
- `PROCESSOR_MAPPING_NAMES`
- `transformers.models.auto` (dynamic import)


key: chroma:document
string_value:
CODE:

def check_all_auto_mappings_importable():
    """Check all auto mappings can be imported."""
    # This is where we need to check we have all backends or the check is incomplete.
    check_missing_backends()

    failures = []
    mappings_to_check = {}
    # Each auto modeling files contains multiple mappings. Let's get them in a dynamic way.
    for module_name in ["modeling_auto", "modeling_tf_auto", "modeling_flax_auto"]:
        module = getattr(transformers.models.auto, module_name, None)
        if module is None:
            continue
        # all mappings in a single auto modeling file
        mapping_names = [x for x in dir(module) if x.endswith("_MAPPING_NAMES")]
        mappings_to_check.update({name: getattr(module, name) for name in mapping_names})

    for name in mappings_to_check:
        name = name.replace("_MAPPING_NAMES", "_MAPPING")
        if not hasattr(transformers, name):
            failures.append(f"`{name}`")
    if len(failures) > 0:


SUMMARY:
**Summary**

`check_all_auto_mappings_importable` function checks whether all auto-mapping functions can be imported.

**Primary Purpose**: Verify if all auto-mapping files are properly installed and importable.

**Key Implementation Details**:

* Iterates through `modeling_auto`, `modeling_tf_auto`, and `modeling_flax_auto` modules to collect their mapping names.
* Checks for the existence of each mapped function in the `transformers` namespace using `hasattr`.

**Notable Detections**:

* Failure: `MAPPING_NAME` (e.g. `_MAPPING_NAMES`) not found in `transformers`.
* Raised exception if any failure is detected, listing all failed mappings.

**Dependencies and Requirements**:
- `transformers.models.auto`
- External dependencies: Not explicitly listed, but assumed to be part of the `transformers` package.
- Inherited/Implemeted interfaces: No notable references.


key: chroma:document
string_value:
CODE:

def check_objects_being_equally_in_main_init():
    """
    Check if a (TensorFlow or Flax) object is in the main __init__ iif its counterpart in PyTorch is.
    """
    attrs = dir(transformers)

    failures = []
    for attr in attrs:
        obj = getattr(transformers, attr)
        if not hasattr(obj, "__module__") or "models.deprecated" in obj.__module__:
            continue

        module_path = obj.__module__
        module_name = module_path.split(".")[-1]
        module_dir = ".".join(module_path.split(".")[:-1])
        if (
            module_name.startswith("modeling_")
            and not module_name.startswith("modeling_tf_")
            and not module_name.startswith("modeling_flax_")
        ):
            parent_module = sys.modules[module_dir]

            frameworks = []
            if is_tf_available():
                frameworks.append("TF")
            if is_flax_available():
                frameworks.append("Flax")

            for framework in frameworks:
                other_module_path = module_path.replace("modeling_", f"modeling_{framework.lower()}_")
                if os.path.isfile("src/" + other_module_path.replace(".", "/") + ".py"):
                    other_module_name = module_name.replace("modeling_", f"modeling_{framework.lower()}_")
                    other_module = getattr(parent_module, other_module_name)
                    if hasattr(other_module, f"{framework}{attr}"):
                        if not hasattr(transformers, f"{framework}{attr}"):
                            if f"{framework}{attr}" not in OBJECT_TO_SKIP_IN_MAIN_INIT_CHECK:
                                failures.append(f"{framework}{attr}")
                    if hasattr(other_module, f"{framework}_{attr}"):
                        if not hasattr(transformers, f"{framework}_{attr}"):
                            if f"{framework}_{attr}" not in OBJECT_TO_SKIP_IN_MAIN_INIT_CHECK:
                                failures.append(f"{framework}_{attr}")
    if len(failures) > 0:


SUMMARY:
**Summary:**

This function, `check_objects_being_equally_in_main_init`, checks if the objects present in TensorFlow/Flax counterpart are also present in PyTorch's main module. It iterates through all attributes of the `transformers` object and verifies the existence of corresponding attributes with prefix "modeling_". If found, it raises an exception indicating failures.

**Key Details:**

- **Implementation:** This function uses attribute iteration to check for equivalent objects between TensorFlow/Flax and PyTorch.
- **Important Method Signatures:**
  - `is_tf_available()`: checks if the TF framework is available.
  - `is_flax_available()`: checks if the Flax framework is available.
  
**Notable Dependencies and Requirements:**

- **Related Classes/Fuctions:** 
  - `transformers` (the object to be checked)
  - `sys`, `os`
- **External Dependencies:**
  - TensorFlow
  - Flax
  - PyTorch
- **Implemented Interfaces:**
    - No specific interfaces are mentioned, but the function appears to follow a generic structure for comparing attributes between frameworks.
-


