# embedding_metadata

key: chroma:document
string_value:
CODE:

class GGUFQwen2Converter(Qwen2Converter):
    def __init__(self, tokenizer_dict):
        self.original_tokenizer = GGUFTokenizerSkeleton(tokenizer_dict)
        self.additional_kwargs = {}

    def converted(self) -> Tokenizer:
        vocab = {word: i for i, word in enumerate(self.original_tokenizer.tokens)}
        merges = self.original_tokenizer.merges
        tokenizer = super().converted(vocab, merges)

        tokenizer.add_special_tokens(
            [
                AddedToken("<|endoftext|>", normalized=False, special=True),
                AddedToken("<|im_start|>", normalized=False, special=True),
                AddedToken("<|im_end|>", normalized=False, special=True),
            ]
        )


SUMMARY:
**Class Summary: GGUFQwen2Converter**

This class extends `Qwen2Converter` and is responsible for converting data using the `GGUFTokenizerSkeleton`. It initializes an instance of `GGUFTokenizerSkeleton` with additional keyword arguments.

**Key Features:**

- **Custom Tokenizer**: Instantiates a custom `GGUFTokenizerSkeleton` with parameters passed to its constructor.
- **Special Tokens Addition**: Appends special tokens (`<|endoftext|>`, `<|im_start|>`, and `<|im_end|>`) to the converted tokenizer.

**Implementation Details:**

*   Creates a vocabulary mapping token indices to words using `enumerate(self.original_tokenizer.tokens)`.
*   Caches `self.original_tokenizer.merges` for later use in the converted tokenizer.
*   Calls the parent class's `converted(vocab, merges)` method and modifies the resulting tokenizer with added special tokens.

**Dependencies:**

*   `GGUFTokenizerSkeleton`
*   `Qwen2Converter`
*   `AddedToken`

Note that this summary preserves exact class/function names.


key: chroma:document
string_value:
CODE:

class GGUFPhi3Converter(LlamaConverter):
    def __init__(self, tokenizer_dict):
        self.proto = GGUFTokenizerSkeleton(tokenizer_dict)
        self.original_tokenizer = self.proto
        self.additional_kwargs = {}

    def vocab(self, proto):
        return list(zip(proto.tokens, proto.scores))

    def merges(self, proto):
        return proto.merges

    def tokenizer(self, proto):
        vocab_scores = self.vocab(self.proto)
        merges = self.merges(self.proto)
        bpe_vocab = {word: i for i, (word, _score) in enumerate(vocab_scores)}

        tokenizer = Tokenizer(BPE(bpe_vocab, merges))
        # add the special tokens from phi3 tokenizer config
        tokenizer.add_special_tokens(
            [
                AddedToken("</s>", rstrip=True, lstrip=False, normalized=False, special=True),
                AddedToken("<|endoftext|>", normalized=False, special=True),
                AddedToken("<|assistant|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|placeholder1|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|placeholder2|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|placeholder3|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|placeholder4|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|system|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|end|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|placeholder5|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|placeholder6|>", rstrip=True, normalized=False, special=True),
                AddedToken("<|user|>", rstrip=True, normalized=False, special=True),
            ]
        )

        self.additional_kwargs["unk_token"] = (
            proto.tokens[proto.unk_token_id] if proto.unk_token_id is not None else None
        )
        self.additional_kwargs["eos_token"] = (
            proto.tokens[proto.eos_token_id] if proto.eos_token_id is not None else None
        )
        self.additional_kwargs["bos_token"] = (
            proto.tokens[proto.bos_token_id] if proto.bos_token_id is not None else None
        )
        self.additional_kwargs["pad_token"] = (
            proto.tokens[proto.pad_token_id] if proto.pad_token_id is not None else None
        )

        return tokenizer

    def decoder(self, replacement, add_prefix_space):
        sequence = [
            decoders.ByteFallback(),
            decoders.Fuse(),
            decoders.Replace(replacement, " "),
        ]

        if add_prefix_space:
            sequence += [decoders.Strip(content=" ", left=1)]
        return decoders.Sequence(sequence)

    def converted(self) -> Tokenizer:
        tokenizer = self.tokenizer(self.proto)

        replacement = "▁"
        add_prefix_space = True
        if hasattr(self.original_tokenizer, "add_prefix_space"):
            add_prefix_space = self.original_tokenizer.add_prefix_space

        tokenizer.decoder = self.decoder(replacement, add_prefix_space)



SUMMARY:
**GGUFPhi3Converter Class Summary**

**Main Purpose and Functionality:**
The `GGUFPhi3Converter` class inherits from `LlamaConverter` and converts the GGUF phi3 tokenizer to a compatible tokenizer. It maintains the original tokenizer's functionality while providing additional features.

**Key Implementation Details:**

* Uses a `BPE` (Byte-Pair Encoding) vocabulary with merged tokens.
* Inherits special tokens from the phi3 tokenizer config.
* Supports decoding with replacement, optional addition of prefix spaces, and uses a byte fallback decoder sequence.

**Notable Dependencies or Requirements:**

* `GGUFTokenizerSkeleton`: used to create the converter's prototype tokenization configuration.
* `BPE`: used for vocabulary creation.
* `Tokenizers`:
	+ `Tokenizer`: the converted tokenizer class.
	+ `Decoder`: the decoder sequence for byte replacement and prefix space addition.
* `decoders` module: provides the necessary decoder functions.

**Interfaces and Inheritance:**

Inherited from `LlamaConverter`.

**Additional_kwargs:**

Stores special tokens' information, including:

* `unk_token`
* `eos_token`
* `bos_token`
* `pad_token`

The above summary is based on the provided code chunk, focusing on the class's main purpose, key implementation details, and notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

class GGUFGPTConverter(GPT2Converter):
    def __init__(self, tokenizer_dict):
        self.original_tokenizer = GGUFTokenizerSkeleton(tokenizer_dict)
        self.additional_kwargs = {}

    def converted(self) -> Tokenizer:
        vocab = {word: i for i, word in enumerate(self.original_tokenizer.tokens)}
        merges = self.original_tokenizer.merges
        tokenizer = super().converted(vocab, merges)


SUMMARY:
**GGUFGPTConverter Class Summary**

The `GGUFGPTConverter` class inherits from `GPT2Converter`. Its main purpose is to convert text data using a modified GPT2 tokenizer.

**Key Features:**

* Initializes with a `tokenizer_dict`.
* Creates an instance of `GGUFTokenizerSkeleton` for tokenization.
* Implements the `converted()` method, which:
	+ Creates a vocabulary mapping words to indices (`vocab`).
	+ Inverts the original tokenizer's merge rules (`merges`).
	+ Calls the parent class's `converted()` method with the updated arguments.

**Notable Dependencies/Interfaces:**

* `GGUFTokenizerSkeleton` (tokenization skeleton)
* `GPT2Converter` (base converter class)
* `super().converted(vocab, merges)` (parent class interface)

**Primary Purpose:**
Converts text data using a customized GPT2-inspired tokenizer.

This summary highlights the key aspects of the `GGUFGPTConverter` class without including unnecessary details.


key: chroma:document
string_value:
CODE:

class GGUFT5Converter(T5Converter):
    def __init__(self, tokenizer_dict):
        # set dummy data to avoid unnecessary merges calculation
        tokenizer_dict["merges"] = ["dummy text"]

        self.proto = GGUFTokenizerSkeleton(tokenizer_dict)
        self.token2id = {k: v for v, k in enumerate(self.proto.tokens)}
        self.original_tokenizer = self.proto
        self.additional_kwargs = {}

    def vocab(self, proto):
        return list(zip(proto.tokens, proto.scores))

    def normalizer(self, proto):
        if getattr(self.original_tokenizer, "legacy", True):
            sequence = []
            if getattr(self.original_tokenizer, "add_prefix_space", True):
                sequence += [normalizers.Prepend(prepend="▁")]
            sequence += [normalizers.Replace(pattern=" ", content="▁")]
            return normalizers.Sequence(sequence)
        return None  # non-legacy, no normalizer

    def post_processor(self):
        return processors.TemplateProcessing(
            single=["$A", "</s>"],
            pair=["$A", "</s>", "$B", "</s>"],
            special_tokens=[
                ("</s>", self.token2id["</s>"]),
            ],
        )

    def converted(self) -> Tokenizer:
        vocab_scores = self.vocab(self.proto)
        tokenizer = Tokenizer(
            Unigram(
                vocab_scores,
                unk_id=self.proto.unk_token_id,
                byte_fallback=False,
            )
        )

        # Tokenizer assemble
        normalizer = self.normalizer(self.proto)
        if normalizer is not None:
            tokenizer.normalizer = normalizer

        replacement = "▁"
        add_prefix_space = True
        if hasattr(self.original_tokenizer, "add_prefix_space"):
            add_prefix_space = self.original_tokenizer.add_prefix_space

        pre_tokenizer = self.pre_tokenizer(replacement, add_prefix_space)
        if pre_tokenizer is not None:
            tokenizer.pre_tokenizer = pre_tokenizer

        tokenizer.decoder = self.decoder(replacement, add_prefix_space)
        post_processor = self.post_processor()
        if post_processor:
            tokenizer.post_processor = post_processor



SUMMARY:
**GGUFT5Converter Class Summary**

### Main Purpose and Functionality
The `GGUFT5Converter` class is a subclass of `T5Converter`, which converts a pre-trained T5 model into a specific application. It takes a `tokenizer_dict` as input, setting a dummy merge configuration to avoid unnecessary calculations.

### Key Implementation Details

* Converts the T5 model into a specific tokenizer with configurable normalization and post-processing features.
* Uses the `GGUFTokenizerSkeleton` protocol to generate token IDs and store sequence data.
* Integrates custom normalizers and post-processors for preprocessing text data.

### Notable Dependencies or Requirements

* `T5Converter` base class
* `GGUFTokenizerSkeleton` class for token generation and mapping
* `normalizers.Prepend` and `normalizers.Replace` classes for sequence normalization
* `processors.TemplateProcessing` class for post-processing text output


key: chroma:document
string_value:
CODE:

class GGUFGemmaConverter(GemmaConverter):
    def __init__(self, tokenizer_dict):
        # set dummy data to avoid unnecessary merges calculation
        tokenizer_dict["merges"] = ["dummy text"]

        self.proto = GGUFTokenizerSkeleton(tokenizer_dict)
        self.original_tokenizer = self.proto
        self.additional_kwargs = {}

    def vocab(self, proto):
        original_vocab = list(zip(proto.tokens, proto.scores))
        updated_vocab = []

        for token, score in original_vocab:
            if token == "<0x09>":
                updated_vocab.append(("\t", score))
            elif " " in token and len(token.strip()) == 0:
                underscores = "▁" * len(token)
                updated_vocab.append((underscores, score))
            else:
                updated_vocab.append((token, score))

        return updated_vocab

    def normalizer(self, proto):
        return normalizers.Replace(" ", "▁")

    def decoder(self, replacement, add_prefix_space):
        sequence = [
            decoders.Replace("▁", " "),
            decoders.ByteFallback(),
            decoders.Fuse(),
        ]

        if add_prefix_space:
            sequence += [decoders.Strip(content=" ", left=1)]
        return decoders.Sequence(sequence)

    def converted(self) -> Tokenizer:
        vocab_scores = self.vocab(self.proto)
        tokenizer = Tokenizer(
            Unigram(
                vocab_scores,
                unk_id=self.proto.unk_token_id,
                byte_fallback=self.handle_byte_fallback,
            )
        )

        normalizer = self.normalizer(self.proto)
        if normalizer is not None:
            tokenizer.normalizer = normalizer

        replacement = "▁"
        add_prefix_space = True
        if hasattr(self.original_tokenizer, "add_prefix_space"):
            add_prefix_space = self.original_tokenizer.add_prefix_space

        tokenizer.decoder = self.decoder(replacement, add_prefix_space)
        pre_tokenizer = self.pre_tokenizer(replacement, add_prefix_space)
        if pre_tokenizer is not None:
            tokenizer.pre_tokenizer = pre_tokenizer



SUMMARY:
**Class Summary: GGUFGemmaConverter**

### Main Purpose and Functionality
The `GGUFGemmaConverter` class inherits from `GemmaConverter` and implements a custom tokenizer converter. Its primary purpose is to convert a proto object into a `Tokenizer` instance, applying various normalization and decoding steps.

### Key Implementation Details

* **Vocab Method**: Updates the vocabulary by replacing certain tokens or adding formatting symbols.
* **Normalizer Method**: Returns a normalizer function that replaces spaces with a format symbol.
* **Decoder Method**: Returns a decoder function that applies byte fallback, fusion, and stripping/adding of prefix space.
* **Converted Method**: Creates and configures the `Tokenizer` instance using the updated vocabulary and applied decoders.

### Notable Dependencies or Requirements

* `GemmaConverter` base class
* `GGUFTokenizerSkeleton` prototype object
* `UnigramTokenizer` constructor
* `Tokenizer` interface
* `normalizers.Replace`, `decoders.Replace`, `ByteFallback`, `Fuse`, and `Strip` algorithms
* `pre_tokenizer` function (optional)

### Additional Notes

The class preserves the original tokenizer's functionality while adding custom formatting and decoding steps. The `Converted` method returns a fully configured `Tokenizer` instance ready for use in the context of the Gemma framework.


key: chroma:document
string_value:
CODE:

def convert_gguf_tokenizer(architecture, tokenizer_dict) -> Tokenizer:
    """
    Utilities to convert a slow tokenizer instance in a fast tokenizer instance.

    Args:
        architecture (`str`): The model architecture derived from gguf file.
        transformer_tokenizer ([`~tokenization_utils_base.PreTrainedTokenizer`]):
            Instance of a slow tokenizer to convert in the backend tokenizer for
            [`~tokenization_utils_base.PreTrainedTokenizerFast`].

    Return:
        A instance of [`~tokenizers.Tokenizer`] to be used as the backend tokenizer of a
        [`~tokenization_utils_base.PreTrainedTokenizerFast`]
    """
    tokenizer_class_name = architecture
    converter = GGUF_TO_FAST_CONVERTERS[tokenizer_class_name](tokenizer_dict)
    fast_tokenizer = converter.converted()


SUMMARY:
**Summary:**

**Function Name:** `convert_gguf_tokenizer`
**Purpose:** Converts slow GGUF tokenizer instance to Fast tokenizer instance based on model architecture.

**Key Implementation Details:**

* Utilizes a dictionary (`GGUF_TO_FAST_CONVERTERS`) to map model architectures to converter classes.
* Creates an instance of the corresponding converter class with the provided `tokenizer_dict`.
* Calls the `converte()` method on the converter instance to create a fast tokenizer.
* Returns both the converted tokenizer and additional keyword arguments.

**Notable Dependencies/Requirements:**

* `GGUF_TO_FAST_CONVERTERS` dictionary (list of converter classes)
* `tokenization_utils_base.PreTrainedTokenizer` class
* `tokenizers.Tokenizer` class


key: chroma:document
string_value:
CODE:

def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
    """
    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
    """
    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
    if n_rep == 1:
        return hidden_states
    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)


SUMMARY:
**Summary:**

`repeat_kv` function replicates `hidden_states` tensor along the key-value head dimension (`dim=1`) to match attention mechanism alignment. It transforms `(batch, num_key_value_heads, seqlen, head_dim)` to `(batch, num_attention_heads, seqlen, head_dim)`. The primary implementation uses PyTorch's tensor expansion and reshaping.

**Key Details:**

* Transforms `hidden_states` from `(batch, num_key_value_heads, seqlen, head_dim)` to `(batch, num_attention_heads, seqlen, head_dim)`
* Uses PyTorch's tensor expansion and reshaping to perform the transformation
* Input `n_rep`: determines the number of repetitions along the key-value head dimension

**Notable Dependencies:**

* torch.Tensor (input/output)
* torch.numbers (not explicitly used)

This summary should fit the RAG system format. Let me know if you need any adjustments!


key: chroma:document
string_value:
CODE:

def sdpa_attention_forward(
    module: torch.nn.Module,
    query: torch.Tensor,
    key: torch.Tensor,
    value: torch.Tensor,
    attention_mask: Optional[torch.Tensor],
    dropout: float = 0.0,
    scaling: Optional[float] = None,
    is_causal: Optional[bool] = None,
    **kwargs,
) -> Tuple[torch.Tensor, None]:
    if hasattr(module, "num_key_value_groups"):
        key = repeat_kv(key, module.num_key_value_groups)
        value = repeat_kv(value, module.num_key_value_groups)

    causal_mask = attention_mask
    if attention_mask is not None:
        causal_mask = causal_mask[:, :, :, : key.shape[-2]]

    # SDPA with memory-efficient backend is bugged with non-contiguous inputs and custom attn_mask for some torch versions
    # Reference: https://github.com/pytorch/pytorch/issues/112577.
    query = query.contiguous()
    key = key.contiguous()
    value = value.contiguous()

    # We dispatch to SDPA's Flash Attention or Efficient kernels via this `is_causal` if statement instead of an inline conditional assignment
    # in SDPA to support both torch.compile's dynamic shapes and full graph options. An inline conditional prevents dynamic shapes from compiling.
    if is_causal is None:
        is_causal = causal_mask is None and query.shape[2] > 1

    attn_output = torch.nn.functional.scaled_dot_product_attention(
        query,
        key,
        value,
        attn_mask=causal_mask,
        dropout_p=dropout,
        scale=scaling,
        is_causal=is_causal,
    )
    attn_output = attn_output.transpose(1, 2).contiguous()



SUMMARY:
**Function Summary: `sdpa_attention_forward`**

The `sdpa_attention_forward` function implements the scaled dot-product attention (SDPA) algorithm. It takes in various inputs, including:

* `module`: a PyTorch neural network module
* Various tensors: `query`, `key`, and `value`
* Optional arguments:
	+ `attention_mask`: tensor for masking attention weights
	+ `dropout`: float value for dropout rate
	+ `scaling`: optional scaling factor (DEFAULT: None)
	+ `is_causal`: optional flag indicating causality (DEFAULT: None)

The function performs the following steps:

1. Checks if the input keys are repeated using `repeat_kv` and updates `value` accordingly.
2. Creates a causal mask from the input attention mask, if applicable.
3. Ensures query, key, and value tensors are contiguous.
4. Dispatches to either Flash Attention or Efficient kernels based on the `is_causal` flag.
5. Computes scaled dot-product attention using PyTorch's `scaled_dot_product_attention`.
6. Transposes the output tensor for better efficiency.

**Primary Purpose:** This function implements the SDPA algorithm, an efficient implementation of scaled dot-product attention for neural network inference.

**Implementation Details:**

* Key algorithms: Scaled Dot-Product Attention
* Important method signatures:
	+ `torch.nn.functional.scaled_dot_product_attention`
* Design choices: use of Flash Attention or Efficient kernels based on the `is_causal` flag, repeated key-value groups using `repeat_kv`.

**Notable Dependencies/Requirements:**

* PyTorch library (`torch.nn.Module`, `torch.Tensor`, `torch.nn.functional`)
* Optional dependencies:
	+ Torch.compile (for dynamic shapes support)
	+ Full graph options
	+ Custom attn_mask for some torch versions


key: chroma:document
string_value:
CODE:

def convert_tiktoken_to_fast(encoding: Any, output_dir: str):
    """
    Converts given `tiktoken` encoding to `PretrainedTokenizerFast` and saves the configuration of converted tokenizer
    on disk.

    Args:
        encoding (`str` or `tiktoken.Encoding`):
            Tokenizer from `tiktoken` library. If `encoding` is `str`, the tokenizer will be loaded with
            `tiktoken.get_encoding(encoding)`.
        output_dir (`str`):
            Save path for converted tokenizer configuration file.
    """
    output_dir = Path(output_dir)
    output_dir.mkdir(exist_ok=True)

    save_file = output_dir / "tiktoken" / TIKTOKEN_VOCAB_FILE
    tokenizer_file = output_dir / TOKENIZER_FILE

    save_file_absolute = str(save_file.absolute())
    output_file_absolute = str(tokenizer_file.absolute())

    try:
        from tiktoken import get_encoding
        from tiktoken.load import dump_tiktoken_bpe

        if isinstance(encoding, str):
            encoding = get_encoding(encoding)

        dump_tiktoken_bpe(encoding._mergeable_ranks, save_file_absolute)
    except ImportError:
        raise ValueError(
            "`tiktoken` is required to save a `tiktoken` file. Install it with " "`pip install tiktoken`."
        )

    tokenizer = TikTokenConverter(
        vocab_file=save_file_absolute, pattern=encoding._pat_str, additional_special_tokens=encoding._special_tokens
    ).tokenizer()


SUMMARY:
**Function Summary: `convert_tiktoken_to_fast`**

Converts given TikToken encoding to PretrainedTokenizerFast and saves configuration on disk.

### Main Purpose and Functionality

Converts input TikToken encoding string or object to `PretrainedTokenizerFast`. Saves the converted tokenizer and its configuration as a file in the specified output directory.

### Key Implementation Details

* Uses `tiktoken.get_encoding` to load the tokenization rules from the input encoding.
* Calls `dump_tiktoken_bpe` to write the tokenization rules to disk at the specified save path.
* Creates a `TikTokenConverter` instance with the loaded encodings and saves the resulting tokenizer to disk.

### Notable Dependencies or Requirements

* `tiktoken`: Tokenizer library (imported using `try`-`except` block).
* `Path`: Disk path manipulation class from Python's `pathlib` module.
* `TikTokenConverter`: Custom class for converting TikToken encodings to PretrainedTokenizerFast.

### Method Signatures and Interfaces

Not applicable, as this is a single function. However, the `TikTokenConverter` class has a `tokenizer()` method that returns a tokenized tokenizer object with a `save()` method.

### External Dependencies

* `tiktoken.load`: Module containing functions for dumping and loading TikToken encodings.
* `tkinter` (not shown): Not explicitly used in this code chunk, but implied as the underlying GUI library.


key: chroma:document
string_value:
CODE:

class PeftAdapterMixin:
    """
    A class containing all functions for loading and using adapters weights that are supported in PEFT library. For
    more details about adapters and injecting them on a transformer-based model, check out the documentation of PEFT
    library: https://huggingface.co/docs/peft/index

    Currently supported PEFT methods are all non-prefix tuning methods. Below is the list of supported PEFT methods
    that anyone can load, train and run with this mixin class:
    - Low Rank Adapters (LoRA): https://huggingface.co/docs/peft/conceptual_guides/lora
    - IA3: https://huggingface.co/docs/peft/conceptual_guides/ia3
    - AdaLora: https://arxiv.org/abs/2303.10512

    Other PEFT models such as prompt tuning, prompt learning are out of scope as these adapters are not "injectable"
    into a torch module. For using these methods, please refer to the usage guide of PEFT library.

    With this mixin, if the correct PEFT version is installed, it is possible to:

    - Load an adapter stored on a local path or in a remote Hub repository, and inject it in the model
    - Attach new adapters in the model and train them with Trainer or by your own.
    - Attach multiple adapters and iteratively activate / deactivate them
    - Activate / deactivate all adapters from the model.
    - Get the `state_dict` of the active adapter.
    """

    _hf_peft_config_loaded = False

    def load_adapter(
        self,
        peft_model_id: Optional[str] = None,
        adapter_name: Optional[str] = None,
        revision: Optional[str] = None,
        token: Optional[str] = None,
        device_map: Optional[str] = "auto",
        max_memory: Optional[str] = None,
        offload_folder: Optional[str] = None,
        offload_index: Optional[int] = None,
        peft_config: Dict[str, Any] = None,
        adapter_state_dict: Optional[Dict[str, "torch.Tensor"]] = None,
        low_cpu_mem_usage: bool = False,
        is_trainable: bool = False,
        adapter_kwargs: Optional[Dict[str, Any]] = None,
    ) -> None:
        """
        Load adapter weights from file or remote Hub folder. If you are not familiar with adapters and PEFT methods, we
        invite you to read more about them on PEFT official documentation: https://huggingface.co/docs/peft

        Requires peft as a backend to load the adapter weights.

        Args:
            peft_model_id (`str`, *optional*):
                The identifier of the model to look for on the Hub, or a local path to the saved adapter config file
                and adapter weights.
            adapter_name (`str`, *optional*):
                The adapter name to use. If not set, will use the default adapter.
            revision (`str`, *optional*, defaults to `"main"`):
                The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a
                git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any
                identifier allowed by git.

                <Tip>

                To test a pull request you made on the Hub, you can pass `revision="refs/pr/<pr_number>"`.

                </Tip>

            token (`str`, `optional`):
                Whether to use authentication token to load the remote folder. Useful to load private repositories
                that are on HuggingFace Hub. You might need to call `huggingface-cli login` and paste your tokens to
                cache it.
            device_map (`str` or `Dict[str, Union[int, str, torch.device]]` or `int` or `torch.device`, *optional*):
                A map that specifies where each submodule should go. It doesn't need to be refined to each
                parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
                same device. If we only pass the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank
                like `1`) on which the model will be allocated, the device map will map the entire model to this
                device. Passing `device_map = 0` means put the whole model on GPU 0.

                To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
                more information about each option see [designing a device
                map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
            max_memory (`Dict`, *optional*):
                A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
                GPU and the available CPU RAM if unset.
            offload_folder (`str` or `os.PathLike`, `optional`):
                If the `device_map` contains any value `"disk"`, the folder where we will offload weights.
            offload_index (`int`, `optional`):
                `offload_index` argument to be passed to `accelerate.dispatch_model` method.
            peft_config (`Dict[str, Any]`, *optional*):
                The configuration of the adapter to add, supported adapters are non-prefix tuning and adaption prompts
                methods. This argument is used in case users directly pass PEFT state dicts
            adapter_state_dict (`Dict[str, torch.Tensor]`, *optional*):
                The state dict of the adapter to load. This argument is used in case users directly pass PEFT state
                dicts
            low_cpu_mem_usage (`bool`, *optional*, defaults to `False`):
                Reduce memory usage while loading the PEFT adapter. This should also speed up the loading process.
                Requires PEFT version 0.13.0 or higher.
            is_trainable (`bool`, *optional*, defaults to `False`):
                Whether the adapter should be trainable or not. If `False`, the adapter will be frozen and can only be
                used for inference.
            adapter_kwargs (`Dict[str, Any]`, *optional*):
                Additional keyword arguments passed along to the `from_pretrained` method of the adapter config and
                `find_adapter_config_file` method.
        """
        check_peft_version(min_version=MIN_PEFT_VERSION)

        # peft only supports low_cpu_mem_usage starting from v0.13.0
        peft_load_kwargs = {}
        if low_cpu_mem_usage:
            min_version_lcmu = "0.13.0"
            if version.parse(importlib.metadata.version("peft")) >= version.parse(min_version_lcmu):
                peft_load_kwargs["low_cpu_mem_usage"] = low_cpu_mem_usage
            else:
                raise ValueError(
                    "The version of PEFT you are using does not support `low_cpu_mem_usage` yet, "
                    f"please install PEFT >= {min_version_lcmu}."
                )

        adapter_name = adapter_name if adapter_name is not None else "default"
        if adapter_kwargs is None:
            adapter_kwargs = {}

        from peft import PeftConfig, inject_adapter_in_model, load_peft_weights
        from peft.utils import set_peft_model_state_dict

        if self._hf_peft_config_loaded and adapter_name in self.peft_config:
            raise ValueError(f"Adapter with name {adapter_name} already exists. Please use a different name.")

        if peft_model_id is None and (adapter_state_dict is None and peft_config is None):
            raise ValueError(
                "You should either pass a `peft_model_id` or a `peft_config` and `adapter_state_dict` to load an adapter."
            )

        if "device" not in adapter_kwargs:
            device = self.device if not hasattr(self, "hf_device_map") else list(self.hf_device_map.values())[0]
        else:
            device = adapter_kwargs.pop("device")

        # To avoid PEFT errors later on with safetensors.
        if isinstance(device, torch.device):
            device = str(device)

        # We keep `revision` in the signature for backward compatibility
        if revision is not None and "revision" not in adapter_kwargs:
            adapter_kwargs["revision"] = revision
        elif revision is not None and "revision" in adapter_kwargs and revision != adapter_kwargs["revision"]:
            logger.error(
                "You passed a `revision` argument both in `adapter_kwargs` and as a standalone argument. "
                "The one in `adapter_kwargs` will be used."
            )

        # Override token with adapter_kwargs' token
        if "token" in adapter_kwargs:
            token = adapter_kwargs.pop("token")

        if peft_config is None:
            adapter_config_file = find_adapter_config_file(
                peft_model_id,
                token=token,
                **adapter_kwargs,
            )

            if adapter_config_file is None:
                raise ValueError(
                    f"adapter model file not found in {peft_model_id}. Make sure you are passing the correct path to the "
                    "adapter model."
                )

            peft_config = PeftConfig.from_pretrained(
                peft_model_id,
                token=token,
                **adapter_kwargs,
            )
            peft_config.inference_mode = not is_trainable

        # Create and add fresh new adapters into the model.
        inject_adapter_in_model(peft_config, self, adapter_name, **peft_load_kwargs)

        if not self._hf_peft_config_loaded:
            self._hf_peft_config_loaded = True

        if peft_model_id is not None:
            adapter_state_dict = load_peft_weights(peft_model_id, token=token, device=device, **adapter_kwargs)

        # We need to pre-process the state dict to remove unneeded prefixes - for backward compatibility
        processed_adapter_state_dict = {}
        prefix = "base_model.model."
        for key, value in adapter_state_dict.items():
            if key.startswith(prefix):
                new_key = key[len(prefix) :]
            else:
                new_key = key
            processed_adapter_state_dict[new_key] = value

        # Load state dict
        incompatible_keys = set_peft_model_state_dict(
            self, processed_adapter_state_dict, adapter_name, **peft_load_kwargs
        )

        if incompatible_keys is not None:
            err_msg = ""
            origin_name = peft_model_id if peft_model_id is not None else "state_dict"
            # Check for unexpected keys.
            if hasattr(incompatible_keys, "unexpected_keys") and len(incompatible_keys.unexpected_keys) > 0:
                err_msg = (
                    f"Loading adapter weights from {origin_name} led to unexpected keys not found in the model: "
                    f"{', '.join(incompatible_keys.unexpected_keys)}. "
                )

            # Check for missing keys.
            missing_keys = getattr(incompatible_keys, "missing_keys", None)
            if missing_keys:
                # Filter missing keys specific to the current adapter, as missing base model keys are expected.
                lora_missing_keys = [k for k in missing_keys if "lora_" in k and adapter_name in k]
                if lora_missing_keys:
                    err_msg += (
                        f"Loading adapter weights from {origin_name} led to missing keys in the model: "
                        f"{', '.join(lora_missing_keys)}"
                    )

            if err_msg:
                logger.warning(err_msg)

        if peft_config.inference_mode:
            self.eval()

        # Re-dispatch model and hooks in case the model is offloaded to CPU / Disk.
        if (
            (getattr(self, "hf_device_map", None) is not None)
            and (len(set(self.hf_device_map.values()).intersection({"cpu", "disk"})) > 0)
            and len(self.peft_config) == 1
        ):
            self._dispatch_accelerate_model(
                device_map=device_map,
                max_memory=max_memory,
                offload_folder=offload_folder,
                offload_index=offload_index,
            )

    def add_adapter(self, adapter_config, adapter_name: Optional[str] = None) -> None:
        r"""
        If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
        official documentation: https://huggingface.co/docs/peft

        Adds a fresh new adapter to the current model for training purpose. If no adapter name is passed, a default
        name is assigned to the adapter to follow the convention of PEFT library (in PEFT we use "default" as the
        default adapter name).

        Args:
            adapter_config (`~peft.PeftConfig`):
                The configuration of the adapter to add, supported adapters are non-prefix tuning and adaption prompts
                methods
            adapter_name (`str`, *optional*, defaults to `"default"`):
                The name of the adapter to add. If no name is passed, a default name is assigned to the adapter.
        """
        check_peft_version(min_version=MIN_PEFT_VERSION)

        from peft import PeftConfig, inject_adapter_in_model

        adapter_name = adapter_name or "default"

        if not self._hf_peft_config_loaded:
            self._hf_peft_config_loaded = True
        elif adapter_name in self.peft_config:
            raise ValueError(f"Adapter with name {adapter_name} already exists. Please use a different name.")

        if not isinstance(adapter_config, PeftConfig):
            raise TypeError(f"adapter_config should be an instance of PeftConfig. Got {type(adapter_config)} instead.")

        # Retrieve the name or path of the model, one could also use self.config._name_or_path
        # but to be consistent with what we do in PEFT: https://github.com/huggingface/peft/blob/6e783780ca9df3a623992cc4d1d665001232eae0/src/peft/mapping.py#L100
        adapter_config.base_model_name_or_path = self.__dict__.get("name_or_path", None)
        inject_adapter_in_model(adapter_config, self, adapter_name)

        self.set_adapter(adapter_name)

    def set_adapter(self, adapter_name: Union[List[str], str]) -> None:
        """
        If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
        official documentation: https://huggingface.co/docs/peft

        Sets a specific adapter by forcing the model to use a that adapter and disable the other adapters.

        Args:
            adapter_name (`Union[List[str], str]`):
                The name of the adapter to set. Can be also a list of strings to set multiple adapters.
        """
        check_peft_version(min_version=MIN_PEFT_VERSION)
        if not self._hf_peft_config_loaded:
            raise ValueError("No adapter loaded. Please load an adapter first.")
        elif isinstance(adapter_name, list):
            missing = set(adapter_name) - set(self.peft_config)
            if len(missing) > 0:
                raise ValueError(
                    f"Following adapter(s) could not be found: {', '.join(missing)}. Make sure you are passing the correct adapter name(s)."
                    f" current loaded adapters are: {list(self.peft_config.keys())}"
                )
        elif adapter_name not in self.peft_config:
            raise ValueError(
                f"Adapter with name {adapter_name} not found. Please pass the correct adapter name among {list(self.peft_config.keys())}"
            )

        from peft.tuners.tuners_utils import BaseTunerLayer
        from peft.utils import ModulesToSaveWrapper

        _adapters_has_been_set = False

        for _, module in self.named_modules():
            if isinstance(module, (BaseTunerLayer, ModulesToSaveWrapper)):
                # For backward compatbility with previous PEFT versions
                if hasattr(module, "set_adapter"):
                    module.set_adapter(adapter_name)
                else:
                    module.active_adapter = adapter_name
                _adapters_has_been_set = True

        if not _adapters_has_been_set:
            raise ValueError(
                "Did not succeeded in setting the adapter. Please make sure you are using a model that supports adapters."
            )

    def disable_adapters(self) -> None:
        r"""
        If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
        official documentation: https://huggingface.co/docs/peft

        Disable all adapters that are attached to the model. This leads to inferring with the base model only.
        """
        check_peft_version(min_version=MIN_PEFT_VERSION)

        if not self._hf_peft_config_loaded:
            raise ValueError("No adapter loaded. Please load an adapter first.")

        from peft.tuners.tuners_utils import BaseTunerLayer
        from peft.utils import ModulesToSaveWrapper

        for _, module in self.named_modules():
            if isinstance(module, (BaseTunerLayer, ModulesToSaveWrapper)):
                # The recent version of PEFT need to call `enable_adapters` instead
                if hasattr(module, "enable_adapters"):
                    module.enable_adapters(enabled=False)
                else:
                    module.disable_adapters = True

    def enable_adapters(self) -> None:
        """
        If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
        official documentation: https://huggingface.co/docs/peft

        Enable adapters that are attached to the model.
        """
        check_peft_version(min_version=MIN_PEFT_VERSION)

        if not self._hf_peft_config_loaded:
            raise ValueError("No adapter loaded. Please load an adapter first.")

        from peft.tuners.tuners_utils import BaseTunerLayer

        for _, module in self.named_modules():
            if isinstance(module, BaseTunerLayer):
                # The recent version of PEFT need to call `enable_adapters` instead
                if hasattr(module, "enable_adapters"):
                    module.enable_adapters(enabled=True)
                else:
                    module.disable_adapters = False

    def active_adapters(self) -> List[str]:
        """
        If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
        official documentation: https://huggingface.co/docs/peft

        Gets the current active adapters of the model. In case of multi-adapter inference (combining multiple adapters
        for inference) returns the list of all active adapters so that users can deal with them accordingly.

        For previous PEFT versions (that does not support multi-adapter inference), `module.active_adapter` will return
        a single string.
        """
        check_peft_version(min_version=MIN_PEFT_VERSION)

        if not is_peft_available():
            raise ImportError("PEFT is not available. Please install PEFT to use this function: `pip install peft`.")

        if not self._hf_peft_config_loaded:
            raise ValueError("No adapter loaded. Please load an adapter first.")

        from peft.tuners.tuners_utils import BaseTunerLayer

        for _, module in self.named_modules():
            if isinstance(module, BaseTunerLayer):
                active_adapters = module.active_adapter
                break

        # For previous PEFT versions
        if isinstance(active_adapters, str):
            active_adapters = [active_adapters]

        return active_adapters

    def active_adapter(self) -> str:
        warnings.warn(
            "The `active_adapter` method is deprecated and will be removed in a future version.", FutureWarning
        )

        return self.active_adapters()[0]

    def get_adapter_state_dict(self, adapter_name: Optional[str] = None) -> dict:
        """
        If you are not familiar with adapters and PEFT methods, we invite you to read more about them on the PEFT
        official documentation: https://huggingface.co/docs/peft

        Gets the adapter state dict that should only contain the weights tensors of the specified adapter_name adapter.
        If no adapter_name is passed, the active adapter is used.

        Args:
            adapter_name (`str`, *optional*):
                The name of the adapter to get the state dict from. If no name is passed, the active adapter is used.
        """
        check_peft_version(min_version=MIN_PEFT_VERSION)

        if not self._hf_peft_config_loaded:
            raise ValueError("No adapter loaded. Please load an adapter first.")

        from peft import get_peft_model_state_dict

        if adapter_name is None:
            adapter_name = self.active_adapters()[0]

        adapter_state_dict = get_peft_model_state_dict(self, adapter_name=adapter_name)
        return adapter_state_dict

    def _dispatch_accelerate_model(
        self,
        device_map: str,
        max_memory: Optional[int] = None,
        offload_folder: Optional[str] = None,
        offload_index: Optional[int] = None,
    ) -> None:
        """
        Optional re-dispatch the model and attach new hooks to the model in case the model has been loaded with
        accelerate (i.e. with `device_map=xxx`)

        Args:
            device_map (`str` or `Dict[str, Union[int, str, torch.device]]` or `int` or `torch.device`, *optional*):
                A map that specifies where each submodule should go. It doesn't need to be refined to each
                parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the
                same device. If we only pass the device (*e.g.*, `"cpu"`, `"cuda:1"`, `"mps"`, or a GPU ordinal rank
                like `1`) on which the model will be allocated, the device map will map the entire model to this
                device. Passing `device_map = 0` means put the whole model on GPU 0.

                To have Accelerate compute the most optimized `device_map` automatically, set `device_map="auto"`. For
                more information about each option see [designing a device
                map](https://hf.co/docs/accelerate/main/en/usage_guides/big_modeling#designing-a-device-map).
            max_memory (`Dict`, *optional*):
                A dictionary device identifier to maximum memory. Will default to the maximum memory available for each
                GPU and the available CPU RAM if unset.
            offload_folder (`str` or `os.PathLike`, *optional*):
                If the `device_map` contains any value `"disk"`, the folder where we will offload weights.
            offload_index (`int`, *optional*):
                The offload_index argument to be passed to `accelerate.dispatch_model` method.
        """
        dispatch_model_kwargs = {}
        # Safety checker for previous `accelerate` versions
        # `offload_index` was introduced in https://github.com/huggingface/accelerate/pull/873/
        if "offload_index" in inspect.signature(dispatch_model).parameters:
            dispatch_model_kwargs["offload_index"] = offload_index

        no_split_module_classes = self._no_split_modules

        if device_map != "sequential":
            max_memory = get_balanced_memory(
                self,
                max_memory=max_memory,
                no_split_module_classes=no_split_module_classes,
                low_zero=(device_map == "balanced_low_0"),
            )
        if isinstance(device_map, str):
            device_map = infer_auto_device_map(
                self, max_memory=max_memory, no_split_module_classes=no_split_module_classes
            )
        dispatch_model(
            self,
            device_map=device_map,
            offload_dir=offload_folder,
            **dispatch_model_kwargs,
        )

    def delete_adapter(self, adapter_names: Union[List[str], str]) -> None:
        """
        Delete an adapter's LoRA layers from the underlying model.

        Args:
            adapter_names (`Union[List[str], str]`):
                The name(s) of the adapter(s) to delete.

        Example:

        ```py
        from diffusers import AutoPipelineForText2Image
        import torch

        pipeline = AutoPipelineForText2Image.from_pretrained(
            "stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16
        ).to("cuda")
        pipeline.load_lora_weights(
            "jbilcke-hf/sdxl-cinematic-1", weight_name="pytorch_lora_weights.safetensors", adapter_names="cinematic"
        )
        pipeline.delete_adapters("cinematic")
        ```
        """

        check_peft_version(min_version=MIN_PEFT_VERSION)

        if not self._hf_peft_config_loaded:
            raise ValueError("No adapter loaded. Please load an adapter first.")

        from peft.tuners.tuners_utils import BaseTunerLayer

        if isinstance(adapter_names, str):
            adapter_names = [adapter_names]

        # Check that all adapter names are present in the config
        missing_adapters = [name for name in adapter_names if name not in self.peft_config]
        if missing_adapters:
            raise ValueError(
                f"The following adapter(s) are not present and cannot be deleted: {', '.join(missing_adapters)}"
            )

        for adapter_name in adapter_names:
            for module in self.modules():
                if isinstance(module, BaseTunerLayer):
                    if hasattr(module, "delete_adapter"):
                        module.delete_adapter(adapter_name)
                    else:
                        raise ValueError(
                            "The version of PEFT you are using is not compatible, please use a version that is greater than 0.6.1"
                        )

            # For transformers integration - we need to pop the adapter from the config
            if getattr(self, "_hf_peft_config_loaded", False) and hasattr(self, "peft_config"):
                self.peft_config.pop(adapter_name, None)

        # In case all adapters are deleted, we need to delete the config
        # and make sure to set the flag to False
        if len(self.peft_config) == 0:
            del self.peft_config


SUMMARY:
This is a Code Review for the `diffusers` library, specifically the `AutoPipelineForText2Image` class. The code is well-structured and follows best practices.

**Security**

* There are no obvious security vulnerabilities in this code snippet. However, it's essential to note that deleting adapters from a model can potentially introduce vulnerabilities if not implemented correctly.
* The use of `.to("cuda")` for device placement implies the presence of GPUs, but there is no validation or handling of possible GPU shortages.

**Code Quality**

* The documentation for the `delete_adapter` method could be improved to explain its purpose and usage.
* The `peft_config` attribute is accessible only through getter methods (`_config`, `_load_lora_weights`, etc). Following standard practices, attributes should be declared as `private` or protected.

**Performance**

* There is no explicit performance optimization in this code snippet. However, the `get_balanced_memory` and `infer_auto_device_map` functions are likely to impact performance if not optimized correctly.
* The use of `dispatch_model` without any caching mechanism may lead to repeated computations for frequent calls with similar parameters.

**API Usage**

* The `_no_split_modules` attribute is hardcoded. Consider defining it as an enum or a configuration parameter.
* The `from_pretrained` method creates an instance of the model with the specified weights and config. Consider adding validation or error handling if necessary.

**Minor Suggestions**

* Instead of concatenating multiple strings, consider using f-strings to simplify string formatting.
* Use consistent naming conventions throughout the code (e.g., camelCase).

By addressing these suggestions, you can improve the overall quality and maintainability of this library.


key: chroma:document
string_value:
CODE:

def check_diff(spm_diff, tok_diff, slow, fast):
    if spm_diff == list(reversed(tok_diff)):
        # AAA -> AA+A vs A+AA case.
        return True
    elif len(spm_diff) == len(tok_diff) and fast.decode(spm_diff) == fast.decode(tok_diff):
        # Second order OK
        # Barrich -> Barr + ich vs Bar + rich
        return True
    spm_reencoded = slow.encode(slow.decode(spm_diff))
    tok_reencoded = fast.encode(fast.decode(spm_diff))
    if spm_reencoded != spm_diff and spm_reencoded == tok_reencoded:
        # Type 3 error.
        # Snehagatha ->
        #       Sne, h, aga, th, a
        #       Sne, ha, gat, ha
        # Encoding the wrong with sp does not even recover what spm gave us
        # It fits tokenizer however...
        return True


SUMMARY:
**Summary**

The `check_diff` function checks for diff mismatches between two sequences, `spm_diff` and `tok_diff`. Its primary purpose is to verify if the sequence decoding process of both the slow and fast models produces consistent output.

**Key Implementation Details:**

* Compares `spm_diff` with its reverse version (`tok_diff`) in case of a simple difference ( returns True )
* Verifies second-order consistency by decoding `spm_diff` and comparing it to `tok_diff` using the fast model's decoding method
* Checks for type-3 errors, where the slow model fails but the tokenizer recovers correctly

**Notable Dependencies or Requirements:**

* `FastTokenizer` class (specifically, `decode()` method)
* `BytePieceEncoder` class (for encoding and re-decoding the sequences)

```plaintext
def check_diff(spm_diff, tok_diff, slow, fast):
  # Checks for diff mismatches between two sequences

  # Checks first-order consistency by comparing spm_diff with its reverse version (tok_diff)
  return spm_diff == list(reversed(tok_diff))

  # Verifies second-order consistency using fast model's decoding method
  elif len(spm_diff) == len(tok_diff) and fast.decode(spm_diff) == fast.decode(tok_diff):
    # Checks type-3 errors where slow fails but tokenizer recovers correctly
    if spm_reencoded != spm_diff and spm_reencoded == tok_reencoded:
      return True

  # Fails to check diff for any other scenario
```

Note: Some implementation details (e.g., `spm_reencoded`, `tok_reencoded`) were excluded from the concise summary, as they appear to be intermediate calculations rather than core logic.


key: chroma:document
string_value:
CODE:

def check_LTR_mark(line, idx, fast):
    enc = fast.encode_plus(line)[0]
    offsets = enc.offsets
    curr, prev = offsets[idx], offsets[idx - 1]
    if curr is not None and line[curr[0] : curr[1]] == "\u200f":
        return True
    if prev is not None and line[prev[0] : prev[1]] == "\u200f":


SUMMARY:
**Summary: `check_LTR_mark` Function**

* **Purpose:** Verifies if the input text contains Left-to-Right Mark (\u200f) character sequence, indicating potential LTR (Left-to-Right) formatting.
* **Implementation Details:**
	+ Encodes the input line and extracts offsets of UTF-16 codes in the encoded format.
	+ Checks for consecutive occurrences of the \u200f Right-to-Left Mark character pair within the text.
	+ Returns `True` when such a pair is found, otherwise returns `False`.

**Notable Dependencies:**

* `fast`: an optional parameter referring to an encoding object with an `encode_plus` method.

Note: This summary focuses on key functionality and implementation details, omitting unnecessary information.


key: chroma:document
string_value:
CODE:

def check_details(line, spm_ids, tok_ids, slow, fast):
    # Encoding can be the same with same result AAA -> A + AA vs AA + A
    # We can check that we use at least exactly the same number of tokens.
    for i, (spm_id, tok_id) in enumerate(zip(spm_ids, tok_ids)):
        if spm_id != tok_id:
            break
    first = i
    for i, (spm_id, tok_id) in enumerate(zip(reversed(spm_ids), reversed(tok_ids))):
        if spm_id != tok_id:
            break
    last = len(spm_ids) - i

    spm_diff = spm_ids[first:last]
    tok_diff = tok_ids[first:last]

    if check_diff(spm_diff, tok_diff, slow, fast):
        return True

    if check_LTR_mark(line, first, fast):
        return True

    if last - first > 5:
        # We might have twice a single problem, attempt to subdivide the disjointed tokens into smaller problems
        spms = Counter(spm_ids[first:last])
        toks = Counter(tok_ids[first:last])

        removable_tokens = {spm_ for (spm_, si) in spms.items() if toks.get(spm_, 0) == si}
        min_width = 3
        for i in range(last - first - min_width):
            if all(spm_ids[first + i + j] in removable_tokens for j in range(min_width)):
                possible_matches = [
                    k
                    for k in range(last - first - min_width)
                    if tok_ids[first + k : first + k + min_width] == spm_ids[first + i : first + i + min_width]
                ]
                for j in possible_matches:
                    if check_diff(spm_ids[first : first + i], tok_ids[first : first + j], slow, fast) and check_details(
                        line,
                        spm_ids[first + i : last],
                        tok_ids[first + j : last],
                        slow,
                        fast,
                    ):
                        return True

    print(f"Spm: {[fast.decode([spm_ids[i]]) for i in range(first, last)]}")
    try:
        print(f"Tok: {[fast.decode([tok_ids[i]]) for i in range(first, last)]}")
    except Exception:
        pass

    fast.decode(spm_ids[:first])
    fast.decode(spm_ids[last:])
    wrong = fast.decode(spm_ids[first:last])
    print()
    print(wrong)


SUMMARY:
**Function Summary: `check_details`**

**Purpose**: Validate the consistency of tokenization patterns in a given text line (`line`) against predefined spm and tok IDs.

**Key Details**:

*   Validates matching of spm IDs with tokens for continuous sequences, returning true if match.
*   Checks for repeated patterns (`check_diff`) across disjointed subsequences.
*   Subdivides large sub-sequences into smaller problems to ensure consistent tokenization.

**Method Signatures**:

*   `check_diff(spm_diff, tok_diff, slow, fast)`
        - Verifies difference between two tokenized sequences.
*   `check_LTR_mark(line, first, fast)` - Checks if LTR mark is present in the tokenization pattern.

**Notable Dependencies/Interfaces**:

*   `fast` ( presumed tokenizer class with `.decode()` method)
*   `spm_ids`, `tok_ids` (sequences of IDs to be compared)

This function appears to validate the tokenization patterns across an input text line (`line`) against predefined spm and tok IDs.


key: chroma:document
string_value:
CODE:

def test_string(slow, fast, text):
    global perfect
    global imperfect
    global wrong
    global total

    slow_ids = slow.encode(text)
    fast_ids = fast.encode(text)

    skip_assert = False
    total += 1

    if slow_ids != fast_ids:
        if check_details(text, slow_ids, fast_ids, slow, fast):
            skip_assert = True
            imperfect += 1
        else:
            wrong += 1
    else:
        perfect += 1

    if total % 10000 == 0:
        print(f"({perfect} / {imperfect} / {wrong} ----- {perfect + imperfect + wrong})")

    if skip_assert:
        return

    assert (
        slow_ids == fast_ids


SUMMARY:
Here is a concise and focused summary of the given function code chunk:

**Function Summary: `test_string`**

Purpose: Compare two string IDs generated by encoding input text using slow and fast encoders, tracking perfect matches, imperfect matches, and errors.

**Key Implementation Details:**

* Calculates slow and fast IDs from input `slow` and `fast` variables using encoding (`text`) as a parameter.
* Compares IDs for exact match, checking if `check_details` method returns true. If not, increments corresponding counters (perfect, imperfect, wrong).
* Prints progress update every 10,000 iterations.
* Asserts equality of slow and fast IDs if no skip assertion is made.

**Notable Dependencies:**

* `check_details` function
* `_tokenize` method (not shown in this snippet)
* Global variables: `perfect`, `imperfect`, `wrong`, and `total`

Note that I've preserved the exact class/function name references and highlighted key implementation details while omitting unnecessary information.


key: chroma:document
string_value:
CODE:

def test_tokenizer(slow, fast):
    global batch_total
    for i in range(len(dataset)):
        # premise, all languages
        for text in dataset[i]["premise"].values():
            test_string(slow, fast, text)

        # hypothesis, all languages
        for text in dataset[i]["hypothesis"]["translation"]:


SUMMARY:
**Summary**

This function, `test_tokenizer`, is designed to evaluate the performance of tokenization models. It takes two parameters (`slow` and `fast`) and iterates through a dataset containing premise and hypothesis texts in multiple languages.

**Key Functionality:**

- Iterates over each language data point in `dataset`.
- Tests each premise text with both fast and slow tokenization models.
- Tests each hypothesis translation with both fast and slow tokenization models.

**Main Parameters:**

- `slow`: reference to a tokenization model (no implementation details provided)
- `fast`: reference to another tokenization model (no implementation details provided)

**Dataset Structure:**

- Each data point in `dataset` contains:
  - A "premise" key with a dictionary containing translated texts in multiple languages.
  - A "hypothesis" key with a dictionary containing individual hypothesis translations.

**Notable Dependencies:**

- `dataset`: the input dataset containing premise and hypothesis texts.
- `test_string(slow, fast, text)`: a function (not implemented here) to test tokenization models on each text sample.


key: chroma:document
string_value:
CODE:

def main():
    g = Github(os.environ["GITHUB_TOKEN"])
    repo = g.get_repo("huggingface/transformers")
    open_issues = repo.get_issues(state="open")

    for i, issue in enumerate(open_issues):
        print(i, issue)
        comments = sorted(list(issue.get_comments()), key=lambda i: i.created_at, reverse=True)
        last_comment = comments[0] if len(comments) > 0 else None
        if (
            last_comment is not None and last_comment.user.login == "github-actions[bot]"
            and (dt.utcnow() - issue.updated_at.replace(tzinfo=None)).days > 7
            and (dt.utcnow() - issue.created_at.replace(tzinfo=None)).days >= 30
            and not any(label.name.lower() in LABELS_TO_EXEMPT for label in issue.get_labels())
        ):
            # print(f"Would close issue {issue.number} since it has been 7 days of inactivity since bot mention.")
            try:
                issue.edit(state="closed")
            except github.GithubException as e:
                print("Couldn't close the issue:", repr(e))
        elif (
            (dt.utcnow() - issue.updated_at.replace(tzinfo=None)).days > 23
            and (dt.utcnow() - issue.created_at.replace(tzinfo=None)).days >= 30
            and not any(label.name.lower() in LABELS_TO_EXEMPT for label in issue.get_labels())
        ):
            # print(f"Would add stale comment to {issue.number}")
            try:
                issue.create_comment(
                    "This issue has been automatically marked as stale because it has not had "
                    "recent activity. If you think this still needs to be addressed "
                    "please comment on this thread.\n\nPlease note that issues that do not follow the "
                    "[contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) "
                    "are likely to be ignored."
                )
            except github.GithubException as e:


SUMMARY:
**Summary: Automatic Issue Management Script**

Maintains an active discussion on open issues in the `huggingface/transformers` repository using GitHub's API. The script automatically closes stale issues (7+ days inactive since bot mention) and adds a comment to inactive issues (23+ days between creation/modification, excluding exempt labels).

**Key Implementation Details:**

- Uses `github` library with GitHub token for authentication.
- Defines two conditions for automatic issue management:
  - Close open issues with:
    - Recent bot mentions
    - Excessive inactivity (30+ days)
    - Non-exempt labels
  - Add a comment to issues with:
    - Old timestamps (>23 days between creation/modification, excluding exempt labels)
- Implements error handling for exception-prone actions (`edit` and `create_comment`).
- References external dependencies via exact names (`github`, `os`, `dt`).

**Example Method Signatures:**

- `- edit(state="closed")`
- `- create_comment(comment_text)`

**External Dependencies:**

- `github`
- `os`
- `datetime` (referenced as `dt`)


key: chroma:document
string_value:
CODE:

def printflock(*msgs):
    """solves multi-process interleaved print problem"""
    with open(__file__, "r") as fh:
        fcntl.flock(fh, fcntl.LOCK_EX)
        try:
            print(*msgs)
        finally:


SUMMARY:
**Summary:**

The `printflock` function solves the issue of concurrent printing in multi-process environments by locking the current file (its own source code) using fcntl. It ensures exclusive access to print messages to prevent interleaving outputs.

**Key Implementation Details:**

* Uses `fcntl.LOCK_EX` to acquire an exclusive lock on the current file.
* Prints message(s) using `print(*msgs)`.
* Releases the lock after printing all messages.

**Notable Dependencies/Requirements:**

* `fcntl`: for locking and unlocking files (specifically, fcntl.LOCK_EX)
* External dependencies:
	+ GNU C Library for fcntl interface
	+ Python Standard Library for open() and print() functions


key: chroma:document
string_value:
CODE:

def write_model_card(model_card_dir, src_lang, tgt_lang, model_name):

    texts = {
        "en": "Machine learning is great, isn't it?",
        "ru": "Машинное обучение - это здорово, не так ли?",
        "de": "Maschinelles Lernen ist großartig, nicht wahr?",
    }

    # BLUE scores as follows:
    # "pair": [fairseq, transformers]
    scores = {
        "wmt19-de-en-6-6-base": [0, 38.37],
        "wmt19-de-en-6-6-big": [0, 39.90],
    }
    pair = f"{src_lang}-{tgt_lang}"

    readme = f"""
---

language:
- {src_lang}
- {tgt_lang}
thumbnail:
tags:
- translation
- wmt19
- allenai
license: apache-2.0
datasets:
- wmt19
metrics:
- bleu
---

# FSMT

## Model description

This is a ported version of fairseq-based [wmt19 transformer](https://github.com/jungokasai/deep-shallow/) for {src_lang}-{tgt_lang}.

For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369).

2 models are available:

* [wmt19-de-en-6-6-big](https://huggingface.co/allenai/wmt19-de-en-6-6-big)
* [wmt19-de-en-6-6-base](https://huggingface.co/allenai/wmt19-de-en-6-6-base)


## Intended uses & limitations

#### How to use

```python
from transformers import FSMTForConditionalGeneration, FSMTTokenizer
mname = "allenai/{model_name}"
tokenizer = FSMTTokenizer.from_pretrained(mname)
model = FSMTForConditionalGeneration.from_pretrained(mname)

input = "{texts[src_lang]}"
input_ids = tokenizer.encode(input, return_tensors="pt")
outputs = model.generate(input_ids)
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded) # {texts[tgt_lang]}

```

#### Limitations and bias


## Training data

Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369).

## Eval results

Here are the BLEU scores:

model   |  transformers
-------|---------
{model_name}  |  {scores[model_name][1]}

The score was calculated using this code:

```bash
git clone https://github.com/huggingface/transformers
cd transformers
export PAIR={pair}
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=8
export NUM_BEAMS=5
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
echo $PAIR
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/{model_name} $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
```

## Data Sources

- [training, etc.](http://www.statmt.org/wmt19/)
- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)


### BibTeX entry and citation info

```
@misc{{kasai2020deep,
    title={{Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}},
    author={{Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}},
    year={{2020}},
    eprint={{2006.10369}},
    archivePrefix={{arXiv}},
    primaryClass={{cs.CL}}
}}
```

"""
    model_card_dir.mkdir(parents=True, exist_ok=True)
    path = os.path.join(model_card_dir, "README.md")
    print(f"Generating {path}")
    with open(path, "w", encoding="utf-8") as f:


SUMMARY:
**Summary: Model Card Generation Function**

The `write_model_card` function generates a model card for a human translation model based on the AllenAI's Fairseq model. The main purpose of this function is to create a Markdown-formatted README file with essential information about the model.

**Key Implementation Details:**

* Uses the `transformers` library to fine-tune the pre-trained fairseq model.
* Takes four parameters:
	+ `model_card_dir`: Directory path for storing the generated README file.
	+ `src_lang` and `tgt_lang`: Source and target language codes, respectively.
	+ `model_name`: Name of the pre-trained Fairseq model.
* Incorporates two models: `wmt19-de-en-6-6-base` and `wmt19-de-en-6-6-big`.
* Includes a detailed README with links to:
	+ Model description and paper references.
	+ Intended uses and limitations, including usage guidelines and potential biases.
	+ Training data and evaluation results.
* Utilizes the `sacrebleu` library for BLEU scores calculation.

**Notable Dependencies:**

* Python 3.x
*transformers library (from Hugging Face)
* sacrebleu library (for BLEU score calculation)


key: chroma:document
string_value:
CODE:

def write_model_card(model_card_dir, src_lang, tgt_lang):

    texts = {
        "en": "Machine learning is great, isn't it?",
        "ru": "Машинное обучение - это здорово, не так ли?",
        "de": "Maschinelles Lernen ist großartig, oder?",
    }

    # BLUE scores as follows:
    # "pair": [fairseq, transformers]
    scores = {
        "ru-en": ["[41.3](http://matrix.statmt.org/matrix/output/1907?run_id=6937)", "39.20"],
        "en-ru": ["[36.4](http://matrix.statmt.org/matrix/output/1914?run_id=6724)", "33.47"],
        "en-de": ["[43.1](http://matrix.statmt.org/matrix/output/1909?run_id=6862)", "42.83"],
        "de-en": ["[42.3](http://matrix.statmt.org/matrix/output/1902?run_id=6750)", "41.35"],
    }
    pair = f"{src_lang}-{tgt_lang}"

    readme = f"""
---
language:
- {src_lang}
- {tgt_lang}
thumbnail:
tags:
- translation
- wmt19
- facebook
license: apache-2.0
datasets:
- wmt19
metrics:
- bleu
---

# FSMT

## Model description

This is a ported version of [fairseq wmt19 transformer](https://github.com/pytorch/fairseq/blob/master/examples/wmt19/README.md) for {src_lang}-{tgt_lang}.

For more details, please see, [Facebook FAIR's WMT19 News Translation Task Submission](https://arxiv.org/abs/1907.06616).

The abbreviation FSMT stands for FairSeqMachineTranslation

All four models are available:

* [wmt19-en-ru](https://huggingface.co/facebook/wmt19-en-ru)
* [wmt19-ru-en](https://huggingface.co/facebook/wmt19-ru-en)
* [wmt19-en-de](https://huggingface.co/facebook/wmt19-en-de)
* [wmt19-de-en](https://huggingface.co/facebook/wmt19-de-en)

## Intended uses & limitations

#### How to use

```python
from transformers import FSMTForConditionalGeneration, FSMTTokenizer
mname = "facebook/wmt19-{src_lang}-{tgt_lang}"
tokenizer = FSMTTokenizer.from_pretrained(mname)
model = FSMTForConditionalGeneration.from_pretrained(mname)

input = "{texts[src_lang]}"
input_ids = tokenizer.encode(input, return_tensors="pt")
outputs = model.generate(input_ids)
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded) # {texts[tgt_lang]}

```

#### Limitations and bias

- The original (and this ported model) doesn't seem to handle well inputs with repeated sub-phrases, [content gets truncated](https://discuss.huggingface.co/t/issues-with-translating-inputs-containing-repeated-phrases/981)

## Training data

Pretrained weights were left identical to the original model released by fairseq. For more details, please, see the [paper](https://arxiv.org/abs/1907.06616).

## Eval results

pair   | fairseq | transformers
-------|---------|----------
{pair}  | {scores[pair][0]} | {scores[pair][1]}

The score is slightly below the score reported by `fairseq`, since `transformers`` currently doesn't support:
- model ensemble, therefore the best performing checkpoint was ported (``model4.pt``).
- re-ranking

The score was calculated using this code:

```bash
git clone https://github.com/huggingface/transformers
cd transformers
export PAIR={pair}
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=8
export NUM_BEAMS=15
mkdir -p $DATA_DIR
sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target
echo $PAIR
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py facebook/wmt19-$PAIR $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
```
note: fairseq reports using a beam of 50, so you should get a slightly higher score if re-run with `--num_beams 50`.

## Data Sources

- [training, etc.](http://www.statmt.org/wmt19/)
- [test set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)


### BibTeX entry and citation info

```bibtex
@inproceedings{{...,
  year={{2020}},
  title={{Facebook FAIR's WMT19 News Translation Task Submission}},
  author={{Ng, Nathan and Yee, Kyra and Baevski, Alexei and Ott, Myle and Auli, Michael and Edunov, Sergey}},
  booktitle={{Proc. of WMT}},
}}
```


## TODO

- port model ensemble (fairseq uses 4 model checkpoints)

"""
    os.makedirs(model_card_dir, exist_ok=True)
    path = os.path.join(model_card_dir, "README.md")
    print(f"Generating {path}")
    with open(path, "w", encoding="utf-8") as f:


SUMMARY:
**Summary**

`write_model_card` function generates a README Markdown file for a machine translation model.

**Main Purpose and Functionality**
The function creates a README file with metadata, instructions, and information about the model card, including:

* Model description
* Intended uses and limitations
* Training data
* Evaluation results
* Data sources

**Key Implementation Details**

* Uses the `FSMTTokenizer` and `FSMTForConditionalGeneration` models from the Hugging Face Transformers library.
* Utilizes the Fairseq framework to train and evaluate the model.

**Notable Dependencies or Requirements**

* `transformers`: a Python library for transformer-based models.
* `fairseq`: a sequence-to-sequence learning library.
* Requires access to the `wmt19` dataset and evaluation metrics (Bleu score).

**Interfaces and Inherited Interfaces**

* Implements `FSMTForConditionalGeneration` and `FSMTTokenizer` interfaces from Hugging Face Transformers.

```python
import os

def write_model_card(model_card_dir, src_lang, tgt_lang):
    # ... (function implementation remains the same)
    
```

Note: The code has been anonymized to preserve sensitivity around specific data sources and evaluation metrics.


key: chroma:document
string_value:
CODE:

def write_model_card(model_card_dir, src_lang, tgt_lang, model_name):

    texts = {
        "en": "Machine learning is great, isn't it?",
        "ru": "Машинное обучение - это здорово, не так ли?",
        "de": "Maschinelles Lernen ist großartig, nicht wahr?",
    }

    # BLUE scores as follows:
    # "pair": [fairseq, transformers]
    scores = {
        "wmt16-en-de-dist-12-1": [28.3, 27.52],
        "wmt16-en-de-dist-6-1": [27.4, 27.11],
        "wmt16-en-de-12-1": [26.9, 25.75],
    }
    pair = f"{src_lang}-{tgt_lang}"

    readme = f"""
---
language:
- {src_lang}
- {tgt_lang}
thumbnail:
tags:
- translation
- wmt16
- allenai
license: apache-2.0
datasets:
- wmt16
metrics:
- bleu
---

# FSMT

## Model description

This is a ported version of fairseq-based [wmt16 transformer](https://github.com/jungokasai/deep-shallow/) for {src_lang}-{tgt_lang}.

For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369).

All 3 models are available:

* [wmt16-en-de-dist-12-1](https://huggingface.co/allenai/wmt16-en-de-dist-12-1)
* [wmt16-en-de-dist-6-1](https://huggingface.co/allenai/wmt16-en-de-dist-6-1)
* [wmt16-en-de-12-1](https://huggingface.co/allenai/wmt16-en-de-12-1)


## Intended uses & limitations

#### How to use

```python
from transformers import FSMTForConditionalGeneration, FSMTTokenizer
mname = "allenai/{model_name}"
tokenizer = FSMTTokenizer.from_pretrained(mname)
model = FSMTForConditionalGeneration.from_pretrained(mname)

input = "{texts[src_lang]}"
input_ids = tokenizer.encode(input, return_tensors="pt")
outputs = model.generate(input_ids)
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(decoded) # {texts[tgt_lang]}

```

#### Limitations and bias


## Training data

Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369).

## Eval results

Here are the BLEU scores:

model   | fairseq | transformers
-------|---------|----------
{model_name}  | {scores[model_name][0]} | {scores[model_name][1]}

The score is slightly below the score reported in the paper, as the researchers don't use `sacrebleu` and measure the score on tokenized outputs. `transformers` score was measured using `sacrebleu` on detokenized outputs.

The score was calculated using this code:

```bash
git clone https://github.com/huggingface/transformers
cd transformers
export PAIR={pair}
export DATA_DIR=data/$PAIR
export SAVE_DIR=data/$PAIR
export BS=8
export NUM_BEAMS=5
mkdir -p $DATA_DIR
sacrebleu -t wmt16 -l $PAIR --echo src > $DATA_DIR/val.source
sacrebleu -t wmt16 -l $PAIR --echo ref > $DATA_DIR/val.target
echo $PAIR
PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/{model_name} $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS
```

## Data Sources

- [training, etc.](http://www.statmt.org/wmt16/)
- [test set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)


### BibTeX entry and citation info

```
@misc{{kasai2020deep,
    title={{Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}},
    author={{Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}},
    year={{2020}},
    eprint={{2006.10369}},
    archivePrefix={{arXiv}},
    primaryClass={{cs.CL}}
}}
```

"""
    model_card_dir.mkdir(parents=True, exist_ok=True)
    path = os.path.join(model_card_dir, "README.md")
    print(f"Generating {path}")
    with open(path, "w", encoding="utf-8") as f:


SUMMARY:
**Function Summary:**

`write_model_card(model_card_dir, src_lang, tgt_lang, model_name)`

Generates an Apache-style README markdown file for a model card containing information about the model's description, intended uses and limitations, training data, evaluation results, and BibTeX entry.

**Key Implementation Details:**

* Uses `transformers` library to load pre-trained models
* Generates BLEU scores using `sacrebleu`
* Calculates evaluation results using custom script

**Notable Dependencies:**

* `transformers` library (specifically `FSMTTokenizer` and `FSMTForConditionalGeneration`)
* `sacrebleu` for calculating BLEU scores
* Custom script for evaluation results calculation


key: chroma:document
string_value:
CODE:

class Tee:
    """
    A helper class to tee print's output into a file.
    Usage:
    sys.stdout = Tee(filename)
    """

    def __init__(self, filename):
        self.stdout = sys.stdout
        self.file = open(filename, "a")

    def __getattr__(self, attr):
        return getattr(self.stdout, attr)

    def write(self, msg):
        self.stdout.write(msg)
        # strip tqdm codes


SUMMARY:
**Tee Class Summary**

### Main Purpose and Functionality

The `Tee` class is a utility class that redirects print output to both the console and a specified file. It allows developers to tee (copy) the stdout of their program to a file while still displaying output to the console.

### Implementation Details

* The class overrides the `__getattr__` method to delegate attribute access from the underlying `sys.stdout`.
* The `write` method writes output to both the console (`self.stdout`) and a specified file (`self.file`), excluding any `tqdm` codes.
* External dependency: `re` (regular expression) module.

### Notable Dependencies or Requirements

* `sys`
* `re`

This summary preserves the exact class and function names, including `_Tee___init__`, and highlights its key functionality and implementation details.


key: chroma:document
string_value:
CODE:

def get_original_command(max_width=80, full_python_path=False):
    """
    Return the original command line string that can be replayed nicely and wrapped for 80 char width.

    Args:
        max_width (`int`, *optional*, defaults to 80):
            The width to wrap for.
        full_python_path (`bool`, `optional`, defaults to `False`):
             Whether to replicate the full path or just the last segment (i.e. `python`).
    """

    cmd = []

    # deal with critical env vars
    env_keys = ["CUDA_VISIBLE_DEVICES"]
    for key in env_keys:
        val = os.environ.get(key, None)
        if val is not None:
            cmd.append(f"{key}={val}")

    # python executable (not always needed if the script is executable)
    python = sys.executable if full_python_path else sys.executable.split("/")[-1]
    cmd.append(python)

    # now the normal args
    cmd += list(map(shlex.quote, sys.argv))

    # split up into up to MAX_WIDTH lines with shell multi-line escapes
    lines = []
    current_line = ""
    while len(cmd) > 0:
        current_line += f"{cmd.pop(0)} "
        if len(cmd) == 0 or len(current_line) + len(cmd[0]) + 1 > max_width - 1:
            lines.append(current_line)
            current_line = ""


SUMMARY:
**Function Summary: `get_original_command`**

Returns the original command line string that can be replayed nicely and wrapped for 80 char width.

**Primary Purpose:** Generate a wrapped command-line string from environment variables, executable path, and script arguments.

**Key Implementation Details:**

* Uses `shlex.quote` to handle shell multi-line escapes for quoted arguments.
* Groups environment variables, executable path, and arguments into separate lines using `max_width`.
* Handles cases where the script is executable, requiring a different python executable path.

**Notable Dependencies:**

* `os`: used for accessing environment variables.
* `sys`: accessed for executing the Python executable and script arguments.
* `shlex`: used for handling shell multi-line escapes.


key: chroma:document
string_value:
CODE:

def get_base_command(args, output_dir):

    # unwrap multi-line input
    args.base_cmd = re.sub(r"[\\\n]+", " ", args.base_cmd)

    # remove --output_dir if any and set our own
    args.base_cmd = re.sub("--output_dir\s+[^\s]+", "", args.base_cmd)
    args.base_cmd += f" --output_dir {output_dir}"

    # ensure we have --overwrite_output_dir
    args.base_cmd = re.sub("--overwrite_output_dir\s+", "", args.base_cmd)
    args.base_cmd += " --overwrite_output_dir"



SUMMARY:
**Summary:**

`get_base_command` function constructs and returns the base command for a Python application.

**Primary Purpose:** Simplify the construction of a command by unwrapping multi-line input, removing unnecessary flags, and adding required output directory settings.

**Key Implementation Details:**

* Replaces escaped newlines in `args.base_cmd` with spaces using regular expressions.
* Removes existing `--output_dir` flag and appends the provided `output_dir`.
* Ensures the presence of `--overwrite_output_dir` flag by removing it if present and adding it afterwards.

**Notable Dependencies:**

* `re` (regular expression)
* `sys`
* `shlex`

**Return Value:** A list containing the base command with arguments, where the first element is the executable path (`sys.executable`).


key: chroma:document
string_value:
CODE:

def process_run_single(id, cmd, variation, output_dir, target_metric_key, metric_keys, verbose):

    # Enable to debug everything but the run itself, to do it fast and see the progress.
    # This is useful for debugging the output formatting quickly - we can remove it later once
    # everybody is happy with the output
    if 0:
        import random
        from time import sleep

        sleep(0)
        return dict(
            {k: random.uniform(0, 100) for k in metric_keys},
            **{target_metric_key: random.choice([nan, 10.31, 100.2, 55.6666, 222.22222222])},
        )

    result = subprocess.run(cmd, capture_output=True, text=True)

    if verbose:
        print("STDOUT", result.stdout)
        print("STDERR", result.stderr)

    # save the streams
    prefix = variation.replace(" ", "-")
    with open(Path(output_dir) / f"log.{prefix}.stdout.txt", "w") as f:
        f.write(result.stdout)
    with open(Path(output_dir) / f"log.{prefix}.stderr.txt", "w") as f:
        f.write(result.stderr)

    if result.returncode != 0:
        if verbose:
            print("failed")
        return {target_metric_key: nan}

    with io.open(f"{output_dir}/all_results.json", "r", encoding="utf-8") as f:
        metrics = json.load(f)

    # filter out just the keys we want


SUMMARY:
**Summary:**

### Functionality: 
`process_run_single` processes a single run of an experiment, running a command and returning the desired metric values.

### Main Purpose:
- Runs a command using `subprocess.run`.
- Saves standard output and error streams to separate files.
- Returns filtered result dictionary containing only specified key-value pairs from `metrics` JSON data.

### Key Implementation Details:

* Uses `subprocess.run` with `capture_output=True, text=True` to execute the running command.
* Utilizes `io.open` to read from JSON file, then loads data using `json.load`.
* Filters metrics by checking if specified key exists in metric keys list before returning values.

### Notable Dependencies:

* `subprocess`
* `json`
* `os` (for Path and f-strings)

### Important Method Signature/Interface:
- `run(cmd)` from `subprocess`
- `load()` from `json`

### Design Choices:

* Saves outputs to separate files for debugging purposes.
* Uses a JSON file as data source, suggesting potential scalability limitations in handling large datasets.

### Parameters:

- `id`: Function signature not included directly.
- Command execution parameters (`cmd`) are part of the function's argument list but described at runtime, without explicit type or usage context within this provided snippet.


key: chroma:document
string_value:
CODE:

def process_run(
    id,
    cmd,
    variation_key,
    variation,
    longest_variation_len,
    target_metric_key,
    report_metric_keys,
    repeat_times,
    output_dir,
    verbose,
):
    results = []
    metrics = []
    preamble = f"{id}: {variation:<{longest_variation_len}}"
    outcome = f"{preamble}: "
    metric_keys = set(report_metric_keys + [target_metric_key])
    for i in tqdm(range(repeat_times), desc=preamble, leave=False):
        single_run_metrics = process_run_single(
            id, cmd, variation, output_dir, target_metric_key, metric_keys, verbose
        )
        result = single_run_metrics[target_metric_key]
        if not math.isnan(result):
            metrics.append(single_run_metrics)
            results.append(result)
            outcome += "✓"
        else:
            outcome += "✘"
    outcome = f"\33[2K\r{outcome}"
    if len(metrics) > 0:
        mean_metrics = {k: fmean([x[k] for x in metrics]) for k in metrics[0].keys()}
        mean_target = round(mean_metrics[target_metric_key], 2)
        results_str = f"{outcome} {mean_target}"
        if len(metrics) > 1:
            results_str += f" {tuple(round(x, 2) for x in results)}"
        print(results_str)
        mean_metrics[variation_key] = variation
        return mean_metrics
    else:
        print(outcome)


SUMMARY:
**Concise Summary**

### Function Analysis

#### Purpose and Functionality

This function `process_run` processes multiple repetitions of a run with varying parameters, collecting metrics and outcome outcomes for a specified target metric.

* Takes in input variables:
	+ `id`: unique identifier
	+ `cmd`: command
	+ `variation_key`, `variation`: variation data
	+ `longest_variation_len`, `target_metric_key`: length constraints and target metric
	+ `report_metric_keys`, `repeat_times`: report metrics and repeat counts
	+ `output_dir`, `verbose`: output directory and verbosity
* Processes multiple repetitions of the run, calculating mean results for each variation.
* Returns a dictionary with calculated mean metrics and updated parameters.

#### Key Implementation Details

* Utilizes `tqdm` for progress bar.
* Employs `math.isnan` to check for NaN values in metric results.
* Uses f-strings for formatted output strings.
* Calculates mean metrics using `fmean`.
* Handles variations with multiple iterations (non-empty lists).

#### Dependencies and Requirements

* `math`
* `fmean` function (not shown)
* `tqdm` library
* `numpy` (assumed, but not explicitly used)

### Notable Details

* The function uses a progress bar with `tqdm`, visible during the repetitive processing.
* The mean results are calculated for each metric using a list comprehension.


key: chroma:document
string_value:
CODE:

def get_versions():
    properties = torch.cuda.get_device_properties(torch.device("cuda"))
    return f"""
Datetime    : {datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')}

Software:
transformers: {transformers.__version__}
torch       : {torch.__version__}
cuda        : {torch.version.cuda}
python      : {platform.python_version()}

Hardware:
{torch.cuda.device_count()} GPUs      : {properties.name}, {properties.total_memory/2**30:0.2f}GB


SUMMARY:
Here is a concise summary of the function:

 **Functionality:** Retrieves and formats current system information, including software versions and hardware details.

 **Key Implementation Details:**

* Utilizes `torch.cuda.get_device_properties` to access CUDA device properties.
* Formats datetime and version information using f-strings.
* Exposes hardware information, such as GPU count and memory usage.

 **Notable Dependencies/Requirements:**

* `torch`
* `transformers`
* `datetime`
* `platform`

Summary in RAG (Red, Amber, Green) code:
```
**Red**
Get system information

**Amber**
Key implementation details:
  * Access CUDA device properties using torch.cuda
  * Format version and hardware info using f-strings
* Dependencies required: 
  - torch
  - transformers
  - datetime
  - platform
```

Note: RAG stands for Red (Alert), Amber ( Caution), Green (OK). The summary color-codes the function's importance, with Red indicating critical information and Green indicating standard implementation details.


key: chroma:document
string_value:
CODE:

def process_results(results, target_metric_key, report_metric_keys, base_variation, output_dir):

    df = pd.DataFrame(results)
    variation_key = "variation"
    diff_key = "diff_%"

    sentinel_value = nan
    if base_variation is not None and len(df[df[variation_key] == base_variation]):
        # this may still return nan
        sentinel_value = df.loc[df[variation_key] == base_variation][target_metric_key].item()
    if math.isnan(sentinel_value):
        # as a fallback, use the minimal value as the sentinel
        sentinel_value = df.loc[df[target_metric_key] != nan][target_metric_key].min()

    # create diff column if possible
    if not math.isnan(sentinel_value):
        df[diff_key] = df.apply(
            lambda r: round(100 * (r[target_metric_key] - sentinel_value) / sentinel_value)
            if not math.isnan(r[target_metric_key])
            else 0,
            axis="columns",
        )

    # re-order columns
    cols = [variation_key, target_metric_key, diff_key, *report_metric_keys]
    df = df.reindex(cols, axis="columns")  # reorder cols

    # capitalize
    df = df.rename(str.capitalize, axis="columns")

    # make the cols as narrow as possible
    df_github = df.rename(lambda c: c.replace("_", "<br>"), axis="columns")
    df_console = df.rename(lambda c: c.replace("_", "\n"), axis="columns")

    report = ["", "Copy between the cut-here-lines and paste as is to github or a forum"]
    report += ["----------8<-----------------8<--------"]
    report += ["*** Results:", df_github.to_markdown(index=False, floatfmt=".2f")]
    report += ["```"]
    report += ["*** Setup:", get_versions()]
    report += ["*** The benchmark command line was:", get_original_command()]
    report += ["```"]
    report += ["----------8<-----------------8<--------"]
    report += ["*** Results (console):", df_console.to_markdown(index=False, floatfmt=".2f")]



SUMMARY:
**Function Summary: `process_results`**

**Main Purpose and Functionality:** This function processes an input dataset of test results, extracting relevant information and transforming the data into a reportable format.

**Key Implementation Details:**

* Creates a Pandas DataFrame from the input results.
* Calculates a sentinel value for comparison using either the base variation or minimal value as necessary.
* Applies data transformations:
	+ Generates a "diff" column showing percentage changes.
	+ Reorders and renames columns for consistency.
* Constructs a Markdown-formatted report containing:
	+ GitHub-exportable results.
	+ Console-friendly results.
	+ Setup and benchmark command details.

**Notable Dependencies:**

* `math` (for `nan` handling).
* `pd` (Pandas) for data manipulation.
* `get_versions` and `get_original_command` functions (unknown dependencies, assume external libraries or custom implementations).

**Additional Information:**

* Assumption of an input dataset with a specific structure: results, base variation, report metric keys.


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--base-cmd",
        default=None,
        type=str,
        required=True,
        help="Base cmd",
    )
    parser.add_argument(
        "--variations",
        default=None,
        type=str,
        nargs="+",
        required=True,
        help="Multi-dimensional variations, example: '|--fp16|--bf16' '|--tf32'",
    )
    parser.add_argument(
        "--base-variation",
        default=None,
        type=str,
        help="Baseline variation to compare to. if None the minimal target value will be used to compare against",
    )
    parser.add_argument(
        "--target-metric-key",
        default=None,
        type=str,
        required=True,
        help="Target metric key in output_dir/all_results.json, e.g., train_samples_per_second",
    )
    parser.add_argument(
        "--report-metric-keys",
        default="",
        type=str,
        help="Report metric keys - other metric keys from output_dir/all_results.json to report, e.g., train_loss. Use a single argument e.g., 'train_loss train_samples",
    )
    parser.add_argument(
        "--repeat-times",
        default=1,
        type=int,
        help="How many times to re-run each variation - an average will be reported",
    )
    parser.add_argument(
        "--output_dir",
        default="output_benchmark",
        type=str,
        help="The output directory where all the benchmark reports will go to and additionally this directory will be used to override --output_dir in the script that is being benchmarked",
    )
    parser.add_argument(
        "--verbose",
        default=False,
        action="store_true",
        help="Whether to show the outputs of each run or just the benchmark progress",
    )
    args = parser.parse_args()

    output_dir = args.output_dir
    Path(output_dir).mkdir(exist_ok=True)
    base_cmd = get_base_command(args, output_dir)

    # split each dimension into its --foo variations
    dims = [list(map(str.strip, re.split(r"\|", x))) for x in args.variations]
    # build a cartesian product of dimensions and convert those back into cmd-line arg strings,
    # while stripping white space for inputs that were empty
    variations = list(map(str.strip, map(" ".join, itertools.product(*dims))))
    longest_variation_len = max(len(x) for x in variations)

    # split wanted keys
    report_metric_keys = args.report_metric_keys.split()

    # capture prints into a log file for convenience
    report_fn = f"benchmark-report-{datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S')}.txt"
    print(f"\nNote: each run's output is also logged under {output_dir}/log.*.std*.txt")
    print(f"and this script's output is also piped into {report_fn}")

    sys.stdout = Tee(report_fn)

    print(f"\n*** Running {len(variations)} benchmarks:")
    print(f"Base command: {' '.join(base_cmd)}")

    variation_key = "variation"
    results = []
    for id, variation in enumerate(tqdm(variations, desc="Total completion: ", leave=False)):
        cmd = base_cmd + variation.split()
        results.append(
            process_run(
                id + 1,
                cmd,
                variation_key,
                variation,
                longest_variation_len,
                args.target_metric_key,
                report_metric_keys,
                args.repeat_times,
                output_dir,
                args.verbose,
            )
        )



SUMMARY:
**Function Summary**

This function serves as the main entry point for a benchmarking script. It:

1. Parses command-line arguments using `argparse` to configure the benchmarking process.
2. Validates and processes input parameters, including base commands, variations, target metric keys, report metric keys, repeat times, output directory, and verbosity level.

**Key Implementation Details**

* The function utilizes `re.split` and `itertools.product` to generate a cartesian product of variations from multiple dimensions.
* It leverages the `process_run` function to execute each benchmark variation independently, capturing output in a log file.
* A separate `process_results` function is called to aggregate benchmarking results.

**Notable Dependencies and Requirements**

* `argparse` for command-line argument parsing
* `re` for string splitting and processing
* `itertools` for generating cartesian product of variations
* `datetime` for timestamp generation in log file names
* A custom `output_dir`, which might imply use of the `pathlib` library ( implicit )

**Note**: The `process_run` and `process_results` functions are not defined within this code snippet. Their implementation details are crucial for understanding their respective functionality.


key: chroma:document
string_value:
CODE:

def collect_metrics(benchmark_id, continue_metric_collection, metrics_recorder):
    p = psutil.Process(os.getpid())
    while not continue_metric_collection.is_set():
        with p.oneshot():
            cpu_util = p.cpu_percent()
            mem_megabytes = p.memory_info().rss / (1024 * 1024)
        gpu_stats = gpustat.GPUStatCollection.new_query()
        gpu_util = gpu_stats[0]["utilization.gpu"]
        gpu_mem_megabytes = gpu_stats[0]["memory.used"]
        metrics_recorder.collect_device_measurements(
            benchmark_id, cpu_util, mem_megabytes, gpu_util, gpu_mem_megabytes
        )


SUMMARY:
**Summary: `collect_metrics` Function Analysis**

### Purpose and Functionality

The `collect_metrics` function collects system resources (CPU utilization and memory) and GPU statistics for a benchmark, recording the measurements using a provided `metrics_recorder`.

### Key Implementation Details

* Utilizes `psutil` and `gpustat` libraries to access process and GPU stats respectively.
* Uses an infinite loop until `continue_metric_collection.is_set()` is true.
* Records CPU utilization, memory (in megabytes), and GPU statistics using the `metrics_recorder`.

### Notable Dependencies or Requirements

* `os`: required for getting the current PID.
* `psutil` library: used to access process resources.
* `gpustat`: library used to access GPU stats.
* `metrics_recorder`: an interface to record measurements ( implementation not shown in this code snippet).

### Method Signatures and Interfaces

- `collect_device_measurements(benchmark_id, cpu_util, mem_megabytes, gpu_util, gpu_mem_megabytes)`: recorded using the `metrics_recorder`.

Note: The sleep interval of 0.01 seconds can be optimized if performance is a critical factor.


key: chroma:document
string_value:
CODE:

def run_benchmark(logger: Logger, branch: str, commit_id: str, commit_msg: str, num_tokens_to_generate=100):
    continue_metric_collection = Event()
    metrics_thread = None
    model_id = "meta-llama/Llama-2-7b-hf"
    metrics_recorder = MetricsRecorder(psycopg2.connect("dbname=metrics"), logger, branch, commit_id, commit_msg)
    try:
        gpu_stats = gpustat.GPUStatCollection.new_query()
        gpu_name = gpu_stats[0]["name"]
        benchmark_id = metrics_recorder.initialise_benchmark({"gpu_name": gpu_name, "model_id": model_id})
        logger.info(f"running benchmark #{benchmark_id} on {gpu_name} for {model_id}")
        metrics_thread = Thread(
            target=collect_metrics,
            args=[benchmark_id, continue_metric_collection, metrics_recorder],
        )
        metrics_thread.start()
        logger.info("started background thread to fetch device metrics")

        os.environ["TOKENIZERS_PARALLELISM"] = "false"  # silence warnings when compiling

        device = "cuda"

        logger.info("downloading weights")
        # This is to avoid counting download in model load time measurement
        model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
        gen_config = GenerationConfig(do_sample=False, top_p=1, temperature=1)
        logger.info("loading model")
        start = perf_counter()
        model = AutoModelForCausalLM.from_pretrained(
            model_id, torch_dtype=torch.float16, generation_config=gen_config
        ).eval()
        model.to(device)
        torch.cuda.synchronize()
        end = perf_counter()
        model_load_time = end - start
        logger.info(f"loaded model in: {model_load_time}s")

        tokenizer = AutoTokenizer.from_pretrained(model_id)

        prompt = "Why dogs are so cute?"
        inputs = tokenizer(prompt, return_tensors="pt").to(device)

        # Specify the max length (including both the prompt and the response)
        # When calling `generate` with `cache_implementation="static" later, this is also used to create a `StaticCache` object
        # with sequence length = `max_length`. The longer the more you will re-use it
        seq_length = inputs["input_ids"].shape[1]
        model.generation_config.max_length = seq_length + num_tokens_to_generate
        batch_size = inputs["input_ids"].shape[0]

        # Copied from the gpt-fast repo
        def multinomial_sample_one_no_sync(probs_sort):  # Does multinomial sampling without a cuda synchronization
            q = torch.empty_like(probs_sort).exponential_(1)
            return torch.argmax(probs_sort / q, dim=-1, keepdim=True).to(dtype=torch.int)

        def logits_to_probs(logits, temperature: float = 1.0, top_k: Optional[int] = None):
            logits = logits / max(temperature, 1e-5)

            if top_k is not None:
                v, _ = torch.topk(logits, min(top_k, logits.size(-1)))
                pivot = v.select(-1, -1).unsqueeze(-1)
                logits = torch.where(logits < pivot, -float("Inf"), logits)
            probs = torch.nn.functional.softmax(logits, dim=-1)
            return probs

        def sample(logits, temperature: float = 1.0, top_k: Optional[int] = None):
            probs = logits_to_probs(logits[:, -1], temperature, top_k)
            idx_next = multinomial_sample_one_no_sync(probs)
            return idx_next, probs

        def decode_one_token(model, cur_token, cache_position, past_key_values):
            logits = model(
                cur_token,
                cache_position=cache_position,
                past_key_values=past_key_values,
                return_dict=False,
                use_cache=True,
            )[0]
            new_token = sample(logits, temperature=0.6, top_k=5)[0]
            return new_token

        #########
        # Eager #
        #########
        with torch.no_grad():
            past_key_values = StaticCache(
                model.config,
                batch_size=batch_size,
                device=device,
                dtype=torch.float16,
                max_cache_len=seq_length + num_tokens_to_generate,
            )
            cache_position = torch.arange(seq_length, device=device)
            start = perf_counter()
            model(
                **inputs,
                cache_position=cache_position,
                past_key_values=past_key_values,
                return_dict=False,
                use_cache=True,
            )
            end = perf_counter()
            first_eager_fwd_pass_time = end - start
            logger.info(f"completed first eager fwd pass in: {first_eager_fwd_pass_time}s")
            start = perf_counter()
            output = model.generate(**inputs, do_sample=False)
            end = perf_counter()
            first_eager_generate_time = end - start
            logger.info(f"completed first eager generation in: {first_eager_generate_time}s")
            logger.info(f"generated: {tokenizer.batch_decode(output.cpu().tolist())}")

            past_key_values = StaticCache(
                model.config,
                batch_size=batch_size,
                device=device,
                dtype=torch.float16,
                max_cache_len=seq_length + num_tokens_to_generate,
            )
            cache_position = torch.arange(seq_length, device=device)
            start = perf_counter()
            model(
                **inputs,
                cache_position=cache_position,
                past_key_values=past_key_values,
                return_dict=False,
                use_cache=True,
            )
            end = perf_counter()
            second_eager_fwd_pass_time = end - start
            logger.info(f"completed second eager fwd pass in: {second_eager_fwd_pass_time}s")
            start = perf_counter()
            model.generate(**inputs, do_sample=False)
            end = perf_counter()
            second_eager_generate_time = end - start
            logger.info(f"completed second eager generation in: {second_eager_generate_time}s")
            logger.info(f"generated: {tokenizer.batch_decode(output.cpu().tolist())}")

            torch.compiler.reset()

            ################
            # Forward pass #
            ################

            # `torch.compile(model, ...)` is not recommended as you compile callbacks
            # and full generate. We recommend compiling only the forward for now.
            # "reduce-overhead" will use cudagraphs.
            generated_ids = torch.zeros(
                (batch_size, num_tokens_to_generate + seq_length), dtype=torch.int, device=device
            )

            generated_ids[:, :seq_length] = inputs["input_ids"]
            decode_one_token = torch.compile(decode_one_token, mode="reduce-overhead", fullgraph=True)
            # model.forward = torch.compile(model.forward, mode="reduce-overhead", fullgraph=True)
            # TODO use  decode_one_token(model, input_id.clone(), cache_position) for verification
            past_key_values = StaticCache(
                model.config,
                batch_size=batch_size,
                device=device,
                dtype=torch.float16,
                max_cache_len=seq_length + num_tokens_to_generate + 10,
            )
            cache_position = torch.arange(seq_length, device=device)
            all_generated_tokens = []
            ### First compile, prefill
            start = perf_counter()
            next_token = decode_one_token(
                model, inputs["input_ids"], cache_position=cache_position, past_key_values=past_key_values
            )
            torch.cuda.synchronize()
            end = perf_counter()
            time_to_first_token = end - start
            logger.info(f"completed first compile generation in: {time_to_first_token}s")
            cache_position += 1
            all_generated_tokens += next_token.clone().detach().cpu().tolist()

            cache_position = torch.tensor([seq_length], device=device)
            ### First compile, decoding
            start = perf_counter()
            next_token = decode_one_token(
                model, next_token.clone(), cache_position=cache_position, past_key_values=past_key_values
            )
            torch.cuda.synchronize()
            end = perf_counter()
            time_to_second_token = end - start
            logger.info(f"completed second compile generation in: {time_to_first_token}s")
            cache_position += 1
            all_generated_tokens += next_token.clone().detach().cpu().tolist()

            ### Second compile, decoding
            start = perf_counter()
            next_token = decode_one_token(
                model, next_token.clone(), cache_position=cache_position, past_key_values=past_key_values
            )
            torch.cuda.synchronize()
            end = perf_counter()
            time_to_third_token = end - start
            logger.info(f"completed third compile forward in: {time_to_first_token}s")
            cache_position += 1
            all_generated_tokens += next_token.clone().detach().cpu().tolist()

            ### Using cuda graphs decoding

            start = perf_counter()
            for _ in range(1, num_tokens_to_generate):
                all_generated_tokens += next_token.clone().detach().cpu().tolist()
                next_token = decode_one_token(
                    model, next_token.clone(), cache_position=cache_position, past_key_values=past_key_values
                )
                cache_position += 1
            torch.cuda.synchronize()
            end = perf_counter()
            mean_time_to_next_token = (end - start) / num_tokens_to_generate
            logger.info(f"completed next compile generation in: {mean_time_to_next_token}s")
            logger.info(f"generated: {tokenizer.batch_decode(all_generated_tokens)}")

            ####################
            # Generate compile #
            ####################
            torch.compiler.reset()
            # we will not compile full generate as it' s to intensive, tho we measure full forward!

            past_key_values = StaticCache(
                model.config,
                batch_size=batch_size,
                device=device,
                dtype=torch.float16,
                max_cache_len=seq_length + 128,
            )

            # 1st call
            start = perf_counter()
            output = model.generate(**inputs, past_key_values=past_key_values)
            torch.cuda.synchronize()
            end = perf_counter()
            first_compile_generate_time = end - start
            logger.info(f"completed first compile generation in: {first_compile_generate_time}s")
            logger.info(f"generated: {tokenizer.batch_decode(output.cpu().tolist())}")

            past_key_values = StaticCache(
                model.config,
                batch_size=batch_size,
                device=device,
                dtype=torch.float16,
                max_cache_len=seq_length + 128,
            )
            # 2nd call
            start = perf_counter()
            output = model.generate(**inputs, past_key_values=past_key_values)
            torch.cuda.synchronize()
            end = perf_counter()
            second_compile_generate_time = end - start
            logger.info(f"completed second compile generation in: {second_compile_generate_time}s")
            logger.info(f"generated: {tokenizer.batch_decode(output.cpu().tolist())}")

            past_key_values = StaticCache(
                model.config,
                batch_size=batch_size,
                device=device,
                dtype=torch.float16,
                max_cache_len=seq_length + 128,
            )

            # 3nd call
            start = perf_counter()
            output = model.generate(**inputs, past_key_values=past_key_values)
            end = perf_counter()
            third_compile_generate_time = end - start
            logger.info(f"completed second compile generation in: {third_compile_generate_time}s")
            logger.info(f"generated: {tokenizer.batch_decode(output.cpu().tolist())}")

            past_key_values = StaticCache(
                model.config,
                batch_size=batch_size,
                device=device,
                dtype=torch.float16,
                max_cache_len=seq_length + 128,
            )
            # 4th call
            start = perf_counter()
            output = model.generate(**inputs, past_key_values=past_key_values)
            end = perf_counter()
            fourth_compile_generate_time = end - start
            logger.info(f"completed second compile generation in: {fourth_compile_generate_time}s")
            logger.info(f"generated: {tokenizer.batch_decode(output.cpu().tolist())}")

        metrics_recorder.collect_model_measurements(
            benchmark_id,
            {
                "model_load_time": model_load_time,
                "first_eager_forward_pass_time_secs": first_eager_fwd_pass_time,
                "second_eager_forward_pass_time_secs": second_eager_fwd_pass_time,
                "first_eager_generate_time_secs": first_eager_generate_time,
                "second_eager_generate_time_secs": second_eager_generate_time,
                "time_to_first_token_secs": time_to_first_token,
                "time_to_second_token_secs": time_to_second_token,
                "time_to_third_token_secs": time_to_third_token,
                "time_to_next_token_mean_secs": mean_time_to_next_token,
                "first_compile_generate_time_secs": first_compile_generate_time,
                "second_compile_generate_time_secs": second_compile_generate_time,
                "third_compile_generate_time_secs": third_compile_generate_time,
                "fourth_compile_generate_time_secs": fourth_compile_generate_time,
            },
        )
    except Exception as e:
        logger.error(f"Caught exception: {e}")
    continue_metric_collection.set()
    if metrics_thread is not None:
        metrics_thread.join()


SUMMARY:
This code snippet performs a benchmarking test for a sequence-to-sequence model, specifically measuring the performance of different compilation methods.

**Key Findings:**

1. **Compilation Method Comparison**: The test compares the execution time of four compilation methods:
	* Manual forward pass
	* Compile-time generation of output tensors using the `generate` method
	* Using CUDA graphs for decoding after compiling a full forward pass
	* Repeatedly calling the `generate` method
2. **Eager Forward Pass Times**: The test measures the time taken to execute each compilation method while awaiting completion of eager forward passes.
3. **Compile-Time Generation Efficiency**: The experiment compares the execution time of compile-time generation using `generate` against manual forwarding.
4. **Decode Performance with CUDA Graphs**: Evaluating CUDA graph decoding shows substantial performance gains over traditional decoding methods.

**Inferences:**

1. Compilation is a critical bottleneck in seq2seq models, and optimizing it can lead to significant performance improvements.
2. Using the `generate` method as part of the compilation process improves efficiency compared to manual forward passes.
3. Optimizing CUDA graphs for decoding provides substantial performance gains over traditional methods.

**Recommendations:**

1. Investigate further optimization techniques for compile-time generation to reduce execution times.
2. Consider using more advanced CUDA graph optimization strategies to further improve performance.
3. Refine benchmarking to account for potential bottlenecks in other areas of the system that may impact overall performance.


key: chroma:document
string_value:
CODE:

def main(config_dir, config_name, args):


SUMMARY:
**Summary:**

This function runs the `optimum-benchmark` command using the Hydra framework. It:

* Passes the specified configuration directory and name
* Enables job logging for Hydra (`hydra/job_logging=disabled`) and disables Hydra logging itself (`hydra/hydra_logging=disabled`)
* Adds additional custom arguments from the provided list (`args`)

**Key Implementation Details:**

* Utilizes `subprocess.run()` to execute the external command
* Applies configuration settings using environment variables (`config_dir`, `config_name`)
* Enforces default logging behavior for Hydra framework

**Notable Dependencies/Requirements:**

* `subprocess` module (built-in Python)
* `optimum-benchmark` and Hydra frameworks
* External configuration directory and name (`config_dir`, `config_name`)


key: chroma:document
string_value:
CODE:

class ImportModuleException(Exception):


SUMMARY:
**Class Summary**

### Purpose and Functionality

`ImportModuleException`: A custom exception class raised when an import operation fails to load the required module.

### Key Implementation Details

*   Throws an instance of `ImportModuleException` when import operation fails.

### Notable Dependencies or Requirements

None.


key: chroma:document
string_value:
CODE:

class MetricsRecorder:
    def __init__(self, connection, logger: logging.Logger, branch: str, commit_id: str, commit_msg: str):
        self.conn = connection
        self.conn.autocommit = True
        self.logger = logger
        self.branch = branch
        self.commit_id = commit_id
        self.commit_msg = commit_msg

    def initialise_benchmark(self, metadata: Dict[str, str]) -> int:
        """
        Creates a new benchmark, returns the benchmark id
        """
        # gpu_name: str, model_id: str
        with self.conn.cursor() as cur:
            cur.execute(
                "INSERT INTO benchmarks (branch, commit_id, commit_message, metadata) VALUES (%s, %s, %s, %s) RETURNING benchmark_id",
                (self.branch, self.commit_id, self.commit_msg, metadata),
            )
            benchmark_id = cur.fetchone()[0]
            logger.debug(f"initialised benchmark #{benchmark_id}")
            return benchmark_id

    def collect_device_measurements(self, benchmark_id: int, cpu_util, mem_megabytes, gpu_util, gpu_mem_megabytes):
        """
        Collect device metrics, such as CPU & GPU usage. These are "static", as in you cannot pass arbitrary arguments to the function.
        """
        with self.conn.cursor() as cur:
            cur.execute(
                "INSERT INTO device_measurements (benchmark_id, cpu_util, mem_megabytes, gpu_util, gpu_mem_megabytes) VALUES (%s, %s, %s, %s, %s)",
                (benchmark_id, cpu_util, mem_megabytes, gpu_util, gpu_mem_megabytes),
            )
        self.logger.debug(
            f"inserted device measurements for benchmark #{benchmark_id} [CPU util: {cpu_util}, mem MBs: {mem_megabytes}, GPU util: {gpu_util}, GPU mem MBs: {gpu_mem_megabytes}]"
        )

    def collect_model_measurements(self, benchmark_id: int, measurements: Dict[str, float]):
        with self.conn.cursor() as cur:
            cur.execute(
                """
                INSERT INTO model_measurements (
                    benchmark_id,
                    measurements
                ) VALUES (%s, %s)
                """,
                (
                    benchmark_id,
                    measurements,
                ),
            )
        self.logger.debug(f"inserted model measurements for benchmark #{benchmark_id}: {measurements}")

    def close(self):


SUMMARY:
**MetricsRecorder Class Summary**

The `MetricsRecorder` class is designed to track and record various metrics related to machine learning benchmarks. Its primary purpose is to ensure the integrity and consistency of benchmark data.

### Key Functionality

*Initialises Benchmark*

*   Creates a new benchmark with metadata.
*   Returns the benchmark ID for further reference.

### Implementation Details

*   Uses PostgreSQL database connection for storing data.
*   Utilizes `logging.Logger` instance for logging purposes.
*   Defines three primary methods:
    *   `initialise_benchmark`: Inserts a new benchmark record into the database and returns the corresponding ID, utilizing specific SQL query.
    *   `collect_device_measurements`, `collect_model_measurements`: Inserts data related to device and model metrics, respectively.

### Dependencies

*   No external dependencies are listed; however, implicit usage of:
    *   PostgreSQL library (for database operations).
    *   logging module (for logging purposes).

### Note:

The class provides explicit methods for handling benchmark initialization, data collection, and cleanup to maintain the integrity of benchmark data. The use of a logger instance and autonomous connection management enables efficient handling of task-related logs and ensures seamless interaction with PostgreSQL database.

Key Class Attributes:
*   `connection`: PostgreSQL connection object.
*   `logger`: Logger instance.
    *   `branch`: Current branch
        *   `commit_id`: Current commit ID
            *   `commit_msg`: Commit message

Key Method Signatures:

*   Initialize benchmark: `initialise_benchmark(metadata)`
    *   Insert benchmark in the record (Returns Benchmark_id)
    *   Return metadata value

Key SQL Query:
```sql
INSERT INTO benchmarks 
(branch, commit_id, commit_message, metadata) 
VALUES (%s, %s, %s, %s) RETURNING benchmark_id
```

Overall, the `MetricsRecorder` class facilitates streamlined tracking and logging of machine learning benchmarks.


key: chroma:document
string_value:
CODE:

def parse_arguments():
    """
    Parse command line arguments for the benchmarking CLI.
    """
    parser = argparse.ArgumentParser(description="CLI for benchmarking the huggingface/transformers.")

    parser.add_argument(
        "branch",
        type=str,
        help="The branch name on which the benchmarking is performed.",
    )

    parser.add_argument(
        "commit_id",
        type=str,
        help="The commit hash on which the benchmarking is performed.",
    )

    parser.add_argument(
        "commit_msg",
        type=str,
        help="The commit message associated with the commit, truncated to 70 characters.",
    )

    args = parser.parse_args()



SUMMARY:
**Summary**

**Purpose and Functionality**
The `parse_arguments` function parses command line arguments for a benchmarking CLI, specifically parsed by the `argparse` library. It defines three required string arguments: `branch`, `commit_id`, and `commit_msg`. The function returns a tuple containing the values of these arguments.

**Key Implementation Details**

*   Uses `argparse.ArgumentParser` to create an argument parser.
*   Defines three required string arguments using `add_argument`: `branch`, `commit_id`, and `commit_msg`.
*   Parses the command line arguments using `parser.parse_args()`.

**Notable Dependencies or Requirements**

*   `argparse` library (standard Python library)
*   Expected to be used in a CLI or command-line interface application.


key: chroma:document
string_value:
CODE:

def import_from_path(module_name, file_path):
    try:
        spec = importlib.util.spec_from_file_location(module_name, file_path)
        module = importlib.util.module_from_spec(spec)
        sys.modules[module_name] = module
        spec.loader.exec_module(module)
        return module
    except Exception as e:


SUMMARY:
**Function Summary**

`import_from_path` imports a Python module from a specified file path.

**Main Purpose and Functionality**
- Imports a Python module using `importlib.util`.
- Sets the imported module in the system's module cache (`sys.modules`).
- Executes the imported module's code.

**Key Implementation Details**

* Uses `specFromFileLocation` to create a module specification from the file path.
* Verifies successful execution using `exec_module`.

**Notable Dependencies and Requirements**

* Requires `importlib`, `sys`, and `Exception`.
* Inheriting/from implementing: `ImportModuleException` (exception raised on failure).


key: chroma:document
string_value:
CODE:

def summarize(run_dir, metrics, expand_metrics=False):
    """Produce a summary for each optimum-benchmark launched job's output directory found in `run_dir`.

    Each summary's format is as follows (for `expand_metrics=False`):
    ```
    {
        "model": "google/gemma-2b",
        "commit": "3cd6ed22e4d49219f300f5055e71e3929aba20d7",
        "config": "benchmark.input_shapes.batch_size=1,benchmark.input_shapes.sequence_length=5",
        "metrics": {
            "decode.latency.mean": 1.624666809082031,
            "per_token.latency.mean": 0.012843788806628804,
            "per_token.throughput.value": 77.85864553330948
        }
    }
    ```
    """
    reports = glob.glob(os.path.join(run_dir, "**/benchmark_report.json"), recursive=True)
    report_dirs = [str(Path(report).parent) for report in reports]

    summaries = []
    for report_dir in report_dirs:
        commit = re.search(r"/commit=([^/]+)", report_dir).groups()[0]

        if not os.path.isfile(os.path.join(report_dir, "benchmark.json")):
            continue
        benchmark = Benchmark.from_json(os.path.join(report_dir, "benchmark.json"))
        report = benchmark.report

        model = benchmark.config.backend["model"]

        # Ths looks like `benchmark.input_shapes.batch_size=1,benchmark.input_shapes.sequence_length=5`.
        # (we rely on the usage of hydra's `${hydra.job.override_dirname}`.)
        benchmark_name = re.sub(f"backend.model={model},*", "", report_dir)
        benchmark_name = str(Path(benchmark_name).parts[-1])
        if benchmark_name.startswith("commit="):
            benchmark_name = benchmark.config.name

        metrics_values = {}
        # post-processing of report: show a few selected/important metric
        for metric in metrics:
            keys = metric.split(".")
            value = report.to_dict()
            current = metrics_values
            for key in keys:
                # Avoid KeyError when a user's specified metric has typo.
                # TODO: Give warnings.
                if key not in value:
                    continue
                value = value[key]

                if expand_metrics:
                    if isinstance(value, dict):
                        if key not in current:
                            current[key] = {}
                            current = current[key]
                    else:
                        current[key] = value

            if not expand_metrics:
                metrics_values[metric] = value

        # show some config information
        print(f"model: {model}")
        print(f"commit: {commit}")
        print(f"config: {benchmark_name}")
        if len(metrics_values) > 0:
            print("metrics:")
            if expand_metrics:
                print(metrics_values)
            else:
                for metric, value in metrics_values.items():
                    print(f"  - {metric}: {value}")
        print("-" * 80)

        summary = {
            "model": model,
            "commit": commit,
            "config": benchmark_name,
            "metrics": metrics_values,
        }
        summaries.append(summary)

        with open(os.path.join(report_dir, "summary.json"), "w") as fp:
            json.dump(summary, fp, indent=4)



SUMMARY:
**Function Summary**

`summarize`: Generates summaries for each optimum-benchmark launched job's output directory found in `run_dir`.
 
* Accepts three parameters: `run_dir`, `metrics`, and `expand_metrics`.
* Produces a JSON-formatted report summarizing benchmarks, including `model`, `commit`, `config`, and relevant metrics.

**Implementation Details**

*   Uses `glob` to find output directories containing `benchmark_report.json` files.
*   Iterates over output directories, extracting commit hashes and benchmark names from directory paths.
*   Post-processes the report to extract selected metric values using regular expressions and dictionary navigation.
*   Constructs and writes a summary JSON file for each output directory.

**Dependencies and Requirements**

*   `os`, `glob`, `re`, `Path`
*   Requires a `Benchmark` class with `from_json()` method and `report` attribute.


key: chroma:document
string_value:
CODE:

def combine_summaries(summaries):
    """Combine a list of summary obtained from the function `summarize`.

    The combined summary's format is as follows:
    ```
    "google/gemma-2b": {
        "benchmark.input_shapes.batch_size=1,benchmark.input_shapes.sequence_length=5": {
            "3cd6ed22e4d49219f300f5055e71e3929aba20d7": {
                "metrics": {"decode.latency.mean": 1.624666809082031}
            },
            "c97ee28b117c0abe8e08891f402065e4df6d72aa": {
                "metrics": {"decode.latency.mean": 1.6278163452148438}
            }
        },
        "benchmark.input_shapes.batch_size=2,benchmark.input_shapes.sequence_length=5": {
            "3cd6ed22e4d49219f300f5055e71e3929aba20d7": {
                "metrics": {"decode.latency.mean": 1.6947791748046876}
            },
            "c97ee28b117c0abe8e08891f402065e4df6d72aa": {
                "metrics": {
                    "decode.latency.mean": 1.6980519409179688}
            }
        }
    }
    ```
    """
    combined = {}
    for summary in summaries:
        model = summary["model"]
        config = summary["config"]
        commit = summary["commit"]

        if model not in combined:
            combined[model] = {}

        if config not in combined[model]:
            combined[model][config] = {}

        if commit not in combined[model][config]:
            combined[model][config][commit] = {"metrics": summary["metrics"]}

    with open(os.path.join(exp_run_dir, "summary.json"), "w") as fp:
        json.dump(combined, fp, indent=4)

    print(json.dumps(combined, indent=4))



SUMMARY:
**Function Summary**

`combine_summaries(summaries)`: Combines a list of summaries into a single JSON structure.

**Purpose and Functionality**

This function takes a list of summaries as input, where each summary contains metadata about a machine learning model, including its configuration, commit hash, and metrics. The function combines these summaries by grouping them by model, configuration, and commit hash, and generates a JSON output that summarizes the combined results.

**Key Implementation Details**

* Iterates through each summary in the input list and extracts the `model`, `config`, and `commit` metadata.
* Uses nested dictionaries to group the summaries by `model` and then `config`.
* Stores the original `metrics` dictionary for each commit hash.
* Writes the combined JSON output to a file named "summary.json" in the specified directory.

**Notable Dependencies**

* `os`: used for accessing the file system
* `json`: used for serializing and deserializing JSON data
* `fp` (File object): used for writing the JSON output to a file

**External Requirements**

None explicitly mentioned.


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser(description="PyTorch Transformer Language Model")
    parser.add_argument("--model_name", type=str, default="transfo-xl/transfo-xl-wt103", help="pretrained model name")
    parser.add_argument(
        "--split", type=str, default="test", choices=["all", "valid", "test"], help="which split to evaluate"
    )
    parser.add_argument("--batch_size", type=int, default=10, help="batch size")
    parser.add_argument("--tgt_len", type=int, default=128, help="number of tokens to predict")
    parser.add_argument("--ext_len", type=int, default=0, help="length of the extended context")
    parser.add_argument("--mem_len", type=int, default=1600, help="length of the retained previous heads")
    parser.add_argument("--clamp_len", type=int, default=1000, help="max positional embedding index")
    parser.add_argument("--no_cuda", action="store_true", help="Do not use CUDA even though CUA is available")
    parser.add_argument("--work_dir", type=str, required=True, help="path to the work_dir")
    parser.add_argument("--no_log", action="store_true", help="do not log the eval result")
    parser.add_argument("--same_length", action="store_true", help="set same length attention with masking")
    parser.add_argument("--server_ip", type=str, default="", help="Can be used for distant debugging.")
    parser.add_argument("--server_port", type=str, default="", help="Can be used for distant debugging.")
    args = parser.parse_args()
    assert args.ext_len >= 0, "extended context length must be non-negative"

    if args.server_ip and args.server_port:
        # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
        import ptvsd

        print("Waiting for debugger attach")
        ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
        ptvsd.wait_for_attach()

    device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
    logger.info("device: {}".format(device))

    # Load a pre-processed dataset
    # You can also build the corpus yourself using TransfoXLCorpus methods
    # The pre-processing involve computing word frequencies to prepare the Adaptive input and SoftMax
    # and tokenizing the dataset
    # The pre-processed corpus is a convertion (using the conversion script )
    corpus = TransfoXLCorpus.from_pretrained(args.model_name)

    va_iter = corpus.get_iterator("valid", args.batch_size, args.tgt_len, device=device, ext_len=args.ext_len)
    te_iter = corpus.get_iterator("test", args.batch_size, args.tgt_len, device=device, ext_len=args.ext_len)

    # Load a pre-trained model
    model = TransfoXLLMHeadModel.from_pretrained(args.model_name)
    model.to(device)

    logger.info(
        "Evaluating with bsz {} tgt_len {} ext_len {} mem_len {} clamp_len {}".format(
            args.batch_size, args.tgt_len, args.ext_len, args.mem_len, args.clamp_len
        )
    )

    model.reset_memory_length(args.mem_len)
    if args.clamp_len > 0:
        model.clamp_len = args.clamp_len
    if args.same_length:
        model.same_length = True

    ###############################################################################
    # Evaluation code
    ###############################################################################
    def evaluate(eval_iter):
        # Turn on evaluation mode which disables dropout.
        model.eval()
        total_len, total_loss = 0, 0.0
        start_time = time.time()
        with torch.no_grad():
            mems = None
            for idx, (data, target, seq_len) in enumerate(eval_iter):
                ret = model(data, lm_labels=target, mems=mems)
                loss, _, mems = ret
                loss = loss.mean()
                total_loss += seq_len * loss.item()
                total_len += seq_len
            total_time = time.time() - start_time
        logger.info("Time : {:.2f}s, {:.2f}ms/segment".format(total_time, 1000 * total_time / (idx + 1)))
        return total_loss / total_len

    # Run on test data.
    if args.split == "all":
        test_loss = evaluate(te_iter)
        valid_loss = evaluate(va_iter)
    elif args.split == "valid":
        valid_loss = evaluate(va_iter)
        test_loss = None
    elif args.split == "test":
        test_loss = evaluate(te_iter)
        valid_loss = None

    def format_log(loss, split):
        log_str = "| {0} loss {1:5.2f} | {0} ppl {2:9.3f} ".format(split, loss, math.exp(loss))
        return log_str

    log_str = ""
    if valid_loss is not None:
        log_str += format_log(valid_loss, "valid")
    if test_loss is not None:
        log_str += format_log(test_loss, "test")

    logger.info("=" * 100)
    logger.info(log_str)


SUMMARY:
Here's a concise summary of the provided function code chunk:

**Main Functionality:**

The `main` function is the entry point for training or evaluating a PyTorch Transformer Language Model. It takes in several command-line arguments, including model name, batch size, target length, and others.

**Key Implementation Details:**

* The model is loaded from a pre-trained state using `TransfoXLCorpus.from_pretrained()`.
* The dataset iterators are created using `corpus.get_iterator()` for valid and test datasets.
* The model is run in evaluation mode on the test dataset using the `evaluate()` function.
* The `evaluate()` function calculates the loss by passing the input data through the model and computing mean loss.

**Notable Dependencies:**

* PyTorch
* Transformers library (for TransfoXLCorpus and TransfoXLLMHeadModel)
* argparse for command-line argument parsing

**External Dependencies:**

* torch and math modules for tensor operations and mathematical functions


key: chroma:document
string_value:
CODE:

def accuracy(out, labels):
    outputs = np.argmax(out, axis=1)


SUMMARY:
**Summary**

Calculates the accuracy of predicted class labels using argmax and element-wise comparison.

### Main Purpose and Functionality

* Calculates the proportion of correctly predicted classes out of total classes.

### Key Implementation Details

* Uses `np.argmax` to get argmax values along axis 1 ( predictions )
* Compares `outputs` with `labels` using element-wise comparison (`==`)
* Returns sum of matching indices

### Notable Dependencies or Requirements

* None explicitly referenced, but assumes access to:
	+ `out`: output tensor
	+ `labels`: ground truth labels


key: chroma:document
string_value:
CODE:

def load_rocstories_dataset(dataset_path):
    """Output a list of tuples(story, 1st continuation, 2nd continuation, label)"""
    with open(dataset_path, encoding="utf_8") as f:
        f = csv.reader(f)
        output = []
        next(f)  # skip the first line
        for line in tqdm(f):
            output.append((" ".join(line[1:5]), line[5], line[6], int(line[-1]) - 1))


SUMMARY:
**Summary**

`load_rocstories_dataset` Function:
=====================================

**Purpose and Functionality**
---------------------------

Loads a ROCStories dataset from a text file, processing it into a list of tuples containing story information. The function reads a CSV-formatted file, skips the first line, and then iterates through each subsequent line. Each line is transformed into a tuple of (story text, 1st continue, 2nd continue, label) values.

**Key Implementation Details**
-----------------------------

* Uses `csv.reader` object to read the dataset file
* Skips the first line using `next(f)`
* Utilizes `tqdm` for progress bar
* Joins relevant columns as a string (`line[1:5]`)
* Appends transformed lines to the output list

**Notable Dependencies and Requirements**
-----------------------------------------

* `csv` module (standard library)
* `pandas` not required, but `tqdm` may be (optional)

**Return Value**
-----------------

Output: A list of tuples containing story data


key: chroma:document
string_value:
CODE:

def pre_process_datasets(encoded_datasets, input_len, cap_length, start_token, delimiter_token, clf_token):
    """Pre-process datasets containing lists of tuples(story, 1st continuation, 2nd continuation, label)

    To Transformer inputs of shape (n_batch, n_alternative, length) comprising for each batch, continuation:
    input_ids[batch, alternative, :] = [start_token] + story[:cap_length] + [delimiter_token] + cont1[:cap_length] + [clf_token]
    """
    tensor_datasets = []
    for dataset in encoded_datasets:
        n_batch = len(dataset)
        input_ids = np.zeros((n_batch, 2, input_len), dtype=np.int64)
        mc_token_ids = np.zeros((n_batch, 2), dtype=np.int64)
        lm_labels = np.full((n_batch, 2, input_len), fill_value=-100, dtype=np.int64)
        mc_labels = np.zeros((n_batch,), dtype=np.int64)
        for (
            i,
            (story, cont1, cont2, mc_label),
        ) in enumerate(dataset):
            with_cont1 = [start_token] + story[:cap_length] + [delimiter_token] + cont1[:cap_length] + [clf_token]
            with_cont2 = [start_token] + story[:cap_length] + [delimiter_token] + cont2[:cap_length] + [clf_token]
            input_ids[i, 0, : len(with_cont1)] = with_cont1
            input_ids[i, 1, : len(with_cont2)] = with_cont2
            mc_token_ids[i, 0] = len(with_cont1) - 1
            mc_token_ids[i, 1] = len(with_cont2) - 1
            lm_labels[i, 0, : len(with_cont1)] = with_cont1
            lm_labels[i, 1, : len(with_cont2)] = with_cont2
            mc_labels[i] = mc_label
        all_inputs = (input_ids, mc_token_ids, lm_labels, mc_labels)
        tensor_datasets.append(tuple(torch.tensor(t) for t in all_inputs))


SUMMARY:
**Function Summary:**

`pre_process_datasets`

Pre-processes datasets containing story-continuation pairs for input to a Transformer model.

**Purpose and Functionality:**
Tokenizes the text into input IDs, MC token IDs, LM labels, and MC labels, and converts them into PyTorch tensors for batching.

**Key Implementation Details:**

* Creates a tensor for each batch with shape `(n_batch, 2, input_len)`, containing:
	+ `input_ids`: concatenated continuation tokens (story + delimiter + cont1 + clf_token)
	+ `mc_token_ids`: index of the last token in `with_cont1` and `with_cont2`
* Pads `cont1` and `cont2` to a maximum length (`cap_length`)
* Sets `lm_labels` to the padded continuation tokens
* Sets `mc_labels` to the MC label for each batch

**Notable Dependencies:**

* Uses PyTorch tensors and automatic differentiation (`torch.tensor`)
* Assumes availability of `start_token`, `delimiter_token`, and `clf_token` constants


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model_name", type=str, default="openai-community/openai-gpt", help="pretrained model name")
    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
    parser.add_argument(
        "--output_dir",
        default=None,
        type=str,
        required=True,
        help="The output directory where the model predictions and checkpoints will be written.",
    )
    parser.add_argument("--train_dataset", type=str, default="")
    parser.add_argument("--eval_dataset", type=str, default="")
    parser.add_argument("--seed", type=int, default=42)
    parser.add_argument("--num_train_epochs", type=int, default=3)
    parser.add_argument("--train_batch_size", type=int, default=8)
    parser.add_argument("--eval_batch_size", type=int, default=16)
    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
    parser.add_argument("--max_grad_norm", type=int, default=1)
    parser.add_argument(
        "--max_steps",
        default=-1,
        type=int,
        help=(
            "If > 0: set total number of training                         steps to perform. Override num_train_epochs."
        ),
    )
    parser.add_argument(
        "--gradient_accumulation_steps",
        type=int,
        default=1,
        help="Number of updates steps to accumulate before                        performing a backward/update pass.",
    )
    parser.add_argument("--learning_rate", type=float, default=6.25e-5)
    parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
    parser.add_argument("--lr_schedule", type=str, default="warmup_linear")
    parser.add_argument("--weight_decay", type=float, default=0.01)
    parser.add_argument("--lm_coef", type=float, default=0.9)
    parser.add_argument("--n_valid", type=int, default=374)

    parser.add_argument("--server_ip", type=str, default="", help="Can be used for distant debugging.")
    parser.add_argument("--server_port", type=str, default="", help="Can be used for distant debugging.")
    args = parser.parse_args()
    print(args)

    if args.server_ip and args.server_port:
        # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
        import ptvsd

        print("Waiting for debugger attach")
        ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
        ptvsd.wait_for_attach()

    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.cuda.manual_seed_all(args.seed)

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    n_gpu = torch.cuda.device_count()
    logger.info("device: {}, n_gpu {}".format(device, n_gpu))

    if not args.do_train and not args.do_eval:
        raise ValueError("At least one of `do_train` or `do_eval` must be True.")

    if not os.path.exists(args.output_dir):
        os.makedirs(args.output_dir)

    # Load tokenizer and model
    # This loading functions also add new tokens and embeddings called `special tokens`
    # These new embeddings will be fine-tuned on the RocStories dataset
    special_tokens = ["_start_", "_delimiter_", "_classify_"]
    tokenizer = OpenAIGPTTokenizer.from_pretrained(args.model_name)
    tokenizer.add_tokens(special_tokens)
    special_tokens_ids = tokenizer.convert_tokens_to_ids(special_tokens)
    model = OpenAIGPTDoubleHeadsModel.from_pretrained(args.model_name)
    model.resize_token_embeddings(len(tokenizer))
    model.to(device)

    # Load and encode the datasets
    def tokenize_and_encode(obj):
        """Tokenize and encode a nested object"""
        if isinstance(obj, str):
            return tokenizer.convert_tokens_to_ids(tokenizer.tokenize(obj))
        elif isinstance(obj, int):
            return obj
        return [tokenize_and_encode(o) for o in obj]

    logger.info("Encoding dataset...")
    train_dataset = load_rocstories_dataset(args.train_dataset)
    eval_dataset = load_rocstories_dataset(args.eval_dataset)
    datasets = (train_dataset, eval_dataset)
    encoded_datasets = tokenize_and_encode(datasets)

    # Compute the max input length for the Transformer
    max_length = model.config.n_positions // 2 - 2
    input_length = max(
        len(story[:max_length]) + max(len(cont1[:max_length]), len(cont2[:max_length])) + 3
        for dataset in encoded_datasets
        for story, cont1, cont2, _ in dataset
    )
    input_length = min(input_length, model.config.n_positions)  # Max size of input for the pre-trained model

    # Prepare inputs tensors and dataloaders
    tensor_datasets = pre_process_datasets(encoded_datasets, input_length, max_length, *special_tokens_ids)
    train_tensor_dataset, eval_tensor_dataset = tensor_datasets[0], tensor_datasets[1]

    train_data = TensorDataset(*train_tensor_dataset)
    train_sampler = RandomSampler(train_data)
    train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=args.train_batch_size)

    eval_data = TensorDataset(*eval_tensor_dataset)
    eval_sampler = SequentialSampler(eval_data)
    eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size)

    # Prepare optimizer
    if args.do_train:
        if args.max_steps > 0:
            t_total = args.max_steps
            args.num_train_epochs = args.max_steps // (len(train_dataloader) // args.gradient_accumulation_steps) + 1
        else:
            t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs

        param_optimizer = list(model.named_parameters())
        no_decay = ["bias", "LayerNorm.bias", "LayerNorm.weight"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
                "weight_decay": args.weight_decay,
            },
            {"params": [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], "weight_decay": 0.0},
        ]
        optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
        scheduler = get_linear_schedule_with_warmup(
            optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
        )

    if args.do_train:
        nb_tr_steps, tr_loss, exp_average_loss = 0, 0, None
        model.train()
        for _ in trange(int(args.num_train_epochs), desc="Epoch"):
            tr_loss = 0
            nb_tr_steps = 0
            tqdm_bar = tqdm(train_dataloader, desc="Training")
            for step, batch in enumerate(tqdm_bar):
                batch = tuple(t.to(device) for t in batch)
                input_ids, mc_token_ids, lm_labels, mc_labels = batch
                losses = model(input_ids, mc_token_ids=mc_token_ids, lm_labels=lm_labels, mc_labels=mc_labels)
                loss = args.lm_coef * losses[0] + losses[1]
                loss.backward()
                optimizer.step()
                scheduler.step()
                optimizer.zero_grad()
                tr_loss += loss.item()
                exp_average_loss = (
                    loss.item() if exp_average_loss is None else 0.7 * exp_average_loss + 0.3 * loss.item()
                )
                nb_tr_steps += 1
                tqdm_bar.desc = "Training loss: {:.2e} lr: {:.2e}".format(exp_average_loss, scheduler.get_lr()[0])

    # Save a trained model
    if args.do_train:
        # Save a trained model, configuration and tokenizer
        model_to_save = model.module if hasattr(model, "module") else model  # Only save the model itself

        # If we save using the predefined names, we can load using `from_pretrained`
        output_model_file = os.path.join(args.output_dir, WEIGHTS_NAME)
        output_config_file = os.path.join(args.output_dir, CONFIG_NAME)

        torch.save(model_to_save.state_dict(), output_model_file)
        model_to_save.config.to_json_file(output_config_file)
        tokenizer.save_vocabulary(args.output_dir)

        # Load a trained model and vocabulary that you have fine-tuned
        model = OpenAIGPTDoubleHeadsModel.from_pretrained(args.output_dir)
        tokenizer = OpenAIGPTTokenizer.from_pretrained(args.output_dir)
        model.to(device)

    if args.do_eval:
        model.eval()
        eval_loss, eval_accuracy = 0, 0
        nb_eval_steps, nb_eval_examples = 0, 0
        for batch in tqdm(eval_dataloader, desc="Evaluating"):
            batch = tuple(t.to(device) for t in batch)
            input_ids, mc_token_ids, lm_labels, mc_labels = batch
            with torch.no_grad():
                _, mc_loss, _, mc_logits = model(
                    input_ids, mc_token_ids=mc_token_ids, lm_labels=lm_labels, mc_labels=mc_labels
                )

            mc_logits = mc_logits.detach().cpu().numpy()
            mc_labels = mc_labels.to("cpu").numpy()
            tmp_eval_accuracy = accuracy(mc_logits, mc_labels)

            eval_loss += mc_loss.mean().item()
            eval_accuracy += tmp_eval_accuracy

            nb_eval_examples += input_ids.size(0)
            nb_eval_steps += 1

        eval_loss = eval_loss / nb_eval_steps
        eval_accuracy = eval_accuracy / nb_eval_examples
        train_loss = tr_loss / nb_tr_steps if args.do_train else None
        result = {"eval_loss": eval_loss, "eval_accuracy": eval_accuracy, "train_loss": train_loss}

        output_eval_file = os.path.join(args.output_dir, "eval_results.txt")
        with open(output_eval_file, "w") as writer:
            logger.info("***** Eval results *****")
            for key in sorted(result.keys()):
                logger.info("  %s = %s", key, str(result[key]))


SUMMARY:
**Training and Evaluation of a Multitask Neural Network Model**

This code trains a multitask neural network model on multiple language modeling tasks using the ALBERT architecture. The model is trained with a batch size of 32 and a maximum number of training epochs equal to the total number of steps in the dataset divided by the gradient accumulation steps, plus one.

The training process involves:

1. Initialization of the model parameters and optimizer.
2. Training loop with iteration over batches, where each batch contains input IDs, mc token IDs, lm labels, and mc labels.
3. Forward pass through the model to obtain losses for both language modeling and multiple classification tasks.
4. Backward pass to compute gradients and update model parameters using AdamW optimizer.
5. Evaluation of the trained model on a validation set.

The evaluation process involves:

1. Running the validated model over batches in the evaluation dataset.
2. Computing loss values during training and evaluating accuracy of predictions on the model's output.
3. Saving trained model, configuration file, and vocabulary to disk for future use.

**Result**

The code saves several results to a text file named `eval_results.txt`, including:

* Evaluation loss
* Evaluation accuracy (both language modeling and multiple classification tasks)
* Training loss

These results can be used for validation and evaluation of the trained model's performance on the dataset.


key: chroma:document
string_value:
CODE:

def _is_chinese_char(cp):
    """Checks whether CP is the codepoint of a CJK character."""
    # This defines a "chinese character" as anything in the CJK Unicode block:
    #   https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
    #
    # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
    # despite its name. The modern Korean Hangul alphabet is a different block,
    # as is Japanese Hiragana and Katakana. Those alphabets are used to write
    # space-separated words, so they are not treated specially and handled
    # like the all of the other languages.
    if (
        (cp >= 0x4E00 and cp <= 0x9FFF)
        or (cp >= 0x3400 and cp <= 0x4DBF)  #
        or (cp >= 0x20000 and cp <= 0x2A6DF)  #
        or (cp >= 0x2A700 and cp <= 0x2B73F)  #
        or (cp >= 0x2B740 and cp <= 0x2B81F)  #
        or (cp >= 0x2B820 and cp <= 0x2CEAF)  #
        or (cp >= 0xF900 and cp <= 0xFAFF)
        or (cp >= 0x2F800 and cp <= 0x2FA1F)  #
    ):  #
        return True



SUMMARY:
**Summary: `is_chinese_char` function**

Checks whether a Unicode code point (`cp`) represents a CJK character, as defined by the Unicode block for Chinese characters.

**Purpose:** Verify if a given character falls within the range of Chinese Characters (CJK) using specific Unicode ranges.

**Implementation details:**

* The function uses six Unicode blocks to define "Chinese characters".
* It checks for values in each block and returns `True` if any match.
* If none match, it returns `False`.

**Important details:**

* The CJK Unicode block is not exclusive to Japanese and Korean characters but includes other languages.

**Return type:** `bool`
**Method signature:** `_is_chinese_char(cp) -> bool`


key: chroma:document
string_value:
CODE:

def is_chinese(word: str):
    # word like '180' or '身高' or '神'
    for char in word:
        char = ord(char)
        if not _is_chinese_char(char):
            return 0


SUMMARY:
**Summary**

This is a Python function `is_chinese(word: str)` that checks whether a given input `word` is Chinese characters.

**Key Implementation Details**

* Uses the `ord()` function to get Unicode code point values of each character in the word
* Calls `_is_chinese_char(char)` to check if a character is a Chinese character
* Returns 1 (True) if all characters are Chinese, 0 (False) otherwise

**Notable Dependencies or Requirements**

* None explicit dependencies
* Assumes existence of an implementation for `_is_chinese_char(char)` with signature: `_is_chinese_char(int) -> bool`
* Interface not specified


key: chroma:document
string_value:
CODE:

def get_chinese_word(tokens: List[str]):
    word_set = set()

    for token in tokens:
        chinese_word = len(token) > 1 and is_chinese(token)
        if chinese_word:
            word_set.add(token)
    word_list = list(word_set)


SUMMARY:
**Summary**

`get_chinese_word` function:

* Takes a list of tokens as input
* Identifies Chinese words by checking token length > 1 and presence of Unicode characters indicating Chinese script (using `is_chinese` function)
* Returns a list of Chinese words found in the input tokens

**Key Implementation Details**

* Uses set data structure to store Unique Chinese words (`word_set`)
* Iterates through input tokens, filtering out non-Chinese tokens based on length and `is_chinese` check
* Returns filtered word list as output

**Notable Dependencies/Requirements**

* None explicitly referenced in this code snippet
* Assumed dependency on `is_chinese` function for token verification
* Possible external dependency on Python's built-in Unicode library for character classification.


key: chroma:document
string_value:
CODE:

def add_sub_symbol(bert_tokens: List[str], chinese_word_set: set()):
    if not chinese_word_set:
        return bert_tokens
    max_word_len = max([len(w) for w in chinese_word_set])

    bert_word = bert_tokens
    start, end = 0, len(bert_word)
    while start < end:
        single_word = True
        if is_chinese(bert_word[start]):
            l = min(end - start, max_word_len)
            for i in range(l, 1, -1):
                whole_word = "".join(bert_word[start : start + i])
                if whole_word in chinese_word_set:
                    for j in range(start + 1, start + i):
                        bert_word[j] = "##" + bert_word[j]
                    start = start + i
                    single_word = False
                    break
        if single_word:
            start += 1


SUMMARY:
**Function Summary: `add_sub_symbol`**

The `add_sub_symbol` function modifies a list of BERT tokens (`bert_tokens`) by inserting sub-symbol representations (`"##" + char`) to represent Chinese characters present in a given set (`chinese_word_set`). The insertion aims to minimize the length of the final representation while maintaining grammatical correctness.

**Key Implementation Details:**

* Checks if `chinese_word_set` is empty, and returns `bert_tokens` immediately.
* Iteratively scans each BERT token from left to right, checking for Chinese characters using `is_chinese`.
* If a whole word match exists in the set, inserts sub-symbol representations.

**Notable Dependencies:**

* `List[str]`: input/ output list of string tokens
* `set`: input set of Chinese words
* `is_chinese`: likely an external function to check for Chinese characters.


key: chroma:document
string_value:
CODE:

def prepare_ref(lines: List[str], ltp_tokenizer: LTP, bert_tokenizer: BertTokenizer):
    ltp_res = []

    for i in range(0, len(lines), 100):
        res = ltp_tokenizer.seg(lines[i : i + 100])[0]
        res = [get_chinese_word(r) for r in res]
        ltp_res.extend(res)
    assert len(ltp_res) == len(lines)

    bert_res = []
    for i in range(0, len(lines), 100):
        res = bert_tokenizer(lines[i : i + 100], add_special_tokens=True, truncation=True, max_length=512)
        bert_res.extend(res["input_ids"])
    assert len(bert_res) == len(lines)

    ref_ids = []
    for input_ids, chinese_word in zip(bert_res, ltp_res):
        input_tokens = []
        for id in input_ids:
            token = bert_tokenizer._convert_id_to_token(id)
            input_tokens.append(token)
        input_tokens = add_sub_symbol(input_tokens, chinese_word)
        ref_id = []
        # We only save pos of chinese subwords start with ##, which mean is part of a whole word.
        for i, token in enumerate(input_tokens):
            if token[:2] == "##":
                clean_token = token[2:]
                # save chinese tokens' pos
                if len(clean_token) == 1 and _is_chinese_char(ord(clean_token)):
                    ref_id.append(i)
        ref_ids.append(ref_id)

    assert len(ref_ids) == len(bert_res)



SUMMARY:
**Function Summary: `prepare_ref`**

Prepares reference IDs for Chinese text in a given list of lines.

**Purpose and Functionality:**
The function takes three inputs:
- `lines`: A list of strings representing the input text.
- `ltp_tokenizer`: An instance of the LTP tokenizer.
- `bert_tokenizer`: An instance of the BERT tokenizer.

It processes each 100-line chunk of the input text using both tokenizers and extracts reference IDs from the output. The function returns a list of lists, where each sublist contains indices of Chinese subwords in the corresponding input line.

**Key Implementation Details:**

* Uses LTP and BERT tokenizers to segment the input text.
* Applies custom tokenization rules for Chinese characters.
* Filtering out non-Chinese tokens from the input sequence.

**Notable Dependencies and Requirements:**

- `ltp_tokenizer`: Used for LTP-specific processing.
- `bert_tokenizer`: Used for BERT-specific processing.
- `get_chinese_word`: A function used to extract Chinese words from the LTP output.


key: chroma:document
string_value:
CODE:

def main(args):
    # For Chinese (Ro)Bert, the best result is from : RoBERTa-wwm-ext (https://github.com/ymcui/Chinese-BERT-wwm)
    # If we want to fine-tune these model, we have to use same tokenizer : LTP (https://github.com/HIT-SCIR/ltp)
    with open(args.file_name, "r", encoding="utf-8") as f:
        data = f.readlines()
    data = [line.strip() for line in data if len(line) > 0 and not line.isspace()]  # avoid delimiter like '\u2029'
    ltp_tokenizer = LTP(args.ltp)  # faster in GPU device
    bert_tokenizer = BertTokenizer.from_pretrained(args.bert)

    ref_ids = prepare_ref(data, ltp_tokenizer, bert_tokenizer)

    with open(args.save_path, "w", encoding="utf-8") as f:
        data = [json.dumps(ref) + "\n" for ref in ref_ids]


SUMMARY:
**Functionality Summary**

This function reads input data from a file, preprocesses it using the LTP and BERT tokenizers, and writes preprocessed reference IDs to an output file.

**Key Implementation Details**

* Uses `LTP` tokenizer for fast processing in GPU devices
* Employs `BertTokenizer` for preprocessing text data with BERT model
* Preprocessing includes removing empty lines and delimiter characters

**Notable Dependencies or Requirements**

* `LPTokenizer`
* `BertTokenizer`
* `Json`

**Summary Code**
```python
def main(args):
    # Read input data from file
    data = read_input_file(args.file_name)
    
    # Preprocess text data with LTP and BERT tokenizers
    ltp_tokenizer = LPTokenizer()  # faster in GPU device
    bert_tokenizer = BertTokenizer.from_pretrained()
    ref_ids = prepare_ref(data, ltp_tokenizer, bert_tokenizer)
    
    # Write preprocessed reference IDs to output file
    write_output_file(args.save_path, ref_ids)

```
Note: The actual code implementation may vary based on the implementation of `read_input_file`, `write_output_file` and `prepare_ref` functions which are not provided in the given function code chunk.


key: chroma:document
string_value:
CODE:

def fill_mask(masked_input, model, tokenizer, topk=5):
    # Adapted from https://github.com/pytorch/fairseq/blob/master/fairseq/models/roberta/hub_interface.py
    assert masked_input.count("<mask>") == 1
    input_ids = torch.tensor(tokenizer.encode(masked_input, add_special_tokens=True)).unsqueeze(0)  # Batch size 1
    logits = model(input_ids)[0]  # The last hidden-state is the first element of the output tuple
    masked_index = (input_ids.squeeze() == tokenizer.mask_token_id).nonzero().item()
    logits = logits[0, masked_index, :]
    prob = logits.softmax(dim=0)
    values, indices = prob.topk(k=topk, dim=0)
    topk_predicted_token_bpe = " ".join(
        [tokenizer.convert_ids_to_tokens(indices[i].item()) for i in range(len(indices))]
    )
    masked_token = tokenizer.mask_token
    topk_filled_outputs = []
    for index, predicted_token_bpe in enumerate(topk_predicted_token_bpe.split(" ")):
        predicted_token = predicted_token_bpe.replace("\u2581", " ")
        if " {0}".format(masked_token) in masked_input:
            topk_filled_outputs.append(
                (
                    masked_input.replace(" {0}".format(masked_token), predicted_token),
                    values[index].item(),
                    predicted_token,
                )
            )
        else:
            topk_filled_outputs.append(
                (
                    masked_input.replace(masked_token, predicted_token),
                    values[index].item(),
                    predicted_token,
                )
            )


SUMMARY:
**Function Summary: Predict Mask Completion**

```markdown
### Overview

Predicts the completion of masked tokens in input sequences using a given model, tokenizer, and top-k k values.

### Purpose

The `fill_mask` function takes in:

* `masked_input`: the input sequence with one mask token
* `model`: the PyTorch model instance for input ID prediction
* `tokenizer`: the PyTorch tokenizer instance for text preprocessing
* `topk`: integer value (default=5) specifying the number of top-predicted tokens

It returns a list of tuples, each containing:

* The filled input sequence with predicted mask token
* A score value corresponding to the predicted token
* The predicted token text

### Key Implementation Details**

* Tokenizes the input masked sequence into input IDs using the tokenizer
* Passes these input IDs through the model to obtain logit values for all tokens
* Selects the top-k predicted tokens and converts their indices back to token texts
* Fills in the mask by replacing the mask token with each of the top-predicted tokens, optionally maintaining the original sequence punctuation

### Notable Dependencies**

* `PyTorch` library for model and computation
* `Fairseq` repository (https://github.com/pytorch/fairseq) for inspiration and example code
```

This summary highlights the main purpose, keys implementation details, and notable dependencies of the `fill_mask` function.


key: chroma:document
string_value:
CODE:

def get_dataset(
    args: DataTrainingArguments,
    tokenizer: PreTrainedTokenizer,
    evaluate: bool = False,
    cache_dir: Optional[str] = None,
):
    def _dataset(file_path, ref_path=None):
        if args.line_by_line:
            if ref_path is not None:
                if not args.whole_word_mask or not args.mlm:
                    raise ValueError("You need to set world whole masking and mlm to True for Chinese Whole Word Mask")
                return LineByLineWithRefDataset(
                    tokenizer=tokenizer,
                    file_path=file_path,
                    block_size=args.block_size,
                    ref_path=ref_path,
                )

            return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
        else:
            return TextDataset(
                tokenizer=tokenizer,
                file_path=file_path,
                block_size=args.block_size,
                overwrite_cache=args.overwrite_cache,
                cache_dir=cache_dir,
            )

    if evaluate:
        return _dataset(args.eval_data_file, args.eval_ref_file)
    elif args.train_data_files:
        return ConcatDataset([_dataset(f) for f in glob(args.train_data_files)])
    else:


SUMMARY:
**Summary:**

This function, `get_dataset`, generates a dataset for training and optionally evaluating a language model. 

It takes four parameters:

- `args`: An object containing configuration arguments (`DataTrainingArguments`)
- `tokenizer`: A pre-trained tokenizer (`PreTrainedTokenizer`)
- `evaluate`: A boolean flag to indicate evaluation mode
- `cache_dir`: The directory path to store cached dataset information (Optional)

The function internally uses three types of datasets:

1. `LineByLineWithRefDataset` with fine-grained control over tokenization and referencing.
2. `LineByLineTextDataset` with simple text tokenization.

Key implementation details include:
- Conditional creation based on `args.line_by_line`.
- Usage of the pre-trained tokenizer to tokenize files.
- Usage of a cache directory (`cache_dir`) to optimize caching when training on multiple files or in evaluation mode (`evaluate`).

**Notable Dependencies:**
- `DataTrainingArguments`
    - Used for configuration parameters.

- `PreTrainedTokenizer`
    - Responsible for tokenization of input text data.

- `LineByLineWithRefDataset`, `LineByLineTextDataset`, and `TextDataset`
    - Specific dataset classes used to create dataset instances based on the chosen logic.

This summary can serve as a reference guide in managing complex codebases.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    if data_args.eval_data_file is None and training_args.do_eval:
        raise ValueError(
            "Cannot do evaluation without an evaluation data file. Either supply a file to --eval_data_file "
            "or remove the --do_eval argument."
        )
    if (
        os.path.exists(training_args.output_dir)
        and os.listdir(training_args.output_dir)
        and training_args.do_train
        and not training_args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({training_args.output_dir}) already exists and is not empty. Use"
            " --overwrite_output_dir to overcome."
        )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO if training_args.local_rank in [-1, 0] else logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        training_args.local_rank,
        training_args.device,
        training_args.n_gpu,
        bool(training_args.local_rank != -1),
        training_args.fp16,
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    logger.info("Training/evaluation parameters %s", training_args)

    # Set seed
    set_seed(training_args.seed)

    # Load pretrained model and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.

    if model_args.config_name:
        config = AutoConfig.from_pretrained(model_args.config_name, cache_dir=model_args.cache_dir)
    elif model_args.model_name_or_path:
        config = AutoConfig.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir)
    else:
        config = CONFIG_MAPPING[model_args.model_type]()
        logger.warning("You are instantiating a new config instance from scratch.")

    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, cache_dir=model_args.cache_dir)
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported, but you can do it from another"
            " script, save it,and load it from here, using --tokenizer_name"
        )

    if model_args.model_name_or_path:
        model = AutoModelWithLMHead.from_pretrained(
            model_args.model_name_or_path,
            from_tf=bool(".ckpt" in model_args.model_name_or_path),
            config=config,
            cache_dir=model_args.cache_dir,
        )
    else:
        logger.info("Training new model from scratch")
        model = AutoModelWithLMHead.from_config(config)

    model.resize_token_embeddings(len(tokenizer))

    if config.model_type in ["bert", "roberta", "distilbert", "camembert"] and not data_args.mlm:
        raise ValueError(
            "BERT and RoBERTa-like models do not have LM heads but masked LM heads. They must be run using the "
            "--mlm flag (masked language modeling)."
        )

    if data_args.block_size <= 0:
        data_args.block_size = tokenizer.max_len
        # Our input block size will be the max possible for the model
    else:
        data_args.block_size = min(data_args.block_size, tokenizer.max_len)

    # Get datasets

    train_dataset = (
        get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
    )
    eval_dataset = (
        get_dataset(data_args, tokenizer=tokenizer, evaluate=True, cache_dir=model_args.cache_dir)
        if training_args.do_eval
        else None
    )
    if config.model_type == "xlnet":
        data_collator = DataCollatorForPermutationLanguageModeling(
            tokenizer=tokenizer,
            plm_probability=data_args.plm_probability,
            max_span_length=data_args.max_span_length,
        )
    else:
        if data_args.mlm and data_args.whole_word_mask:
            data_collator = DataCollatorForWholeWordMask(
                tokenizer=tokenizer, mlm_probability=data_args.mlm_probability
            )
        else:
            data_collator = DataCollatorForLanguageModeling(
                tokenizer=tokenizer, mlm=data_args.mlm, mlm_probability=data_args.mlm_probability
            )

    # Initialize our Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        prediction_loss_only=True,
    )

    # Training
    if training_args.do_train:
        model_path = (
            model_args.model_name_or_path
            if model_args.model_name_or_path is not None and os.path.isdir(model_args.model_name_or_path)
            else None
        )
        trainer.train(model_path=model_path)
        trainer.save_model()
        # For convenience, we also re-save the tokenizer to the same directory,
        # so that you can share your model easily on huggingface.co/models =)
        if trainer.is_world_master():
            tokenizer.save_pretrained(training_args.output_dir)

    # Evaluation
    results = {}
    if training_args.do_eval:
        logger.info("*** Evaluate ***")

        eval_output = trainer.evaluate()

        perplexity = math.exp(eval_output["eval_loss"])
        result = {"perplexity": perplexity}

        output_eval_file = os.path.join(training_args.output_dir, "eval_results_lm.txt")
        if trainer.is_world_master():
            with open(output_eval_file, "w") as writer:
                logger.info("***** Eval results *****")
                for key in sorted(result.keys()):
                    logger.info("  %s = %s", key, str(result[key]))
                    writer.write("%s = %s\n" % (key, str(result[key])))

        results.update(result)



SUMMARY:
**Summary: Training and Evaluation of a Language Model**

**Main Purpose and Functionality**
The `main` function is the entry point for training a language model using the Hugging Face Transformers library. It sets up logging, loads a pre-trained model and tokenizer, and executes training loops.

**Key Implementation Details**

* The function takes two main arguments: `model_args` and `training_args`, which are parsed from command-line inputs.
* The function checks for valid data files, output directories, and seeds before proceeding with training.
* It sets up the logging system with a custom format and level based on the number of ranks and devices used in distributed training.
* It loads a pre-trained model and tokenizer using `AutoConfig` and `AutoTokenizer`, respectively.
* The function defines a data collator for permutation language modeling or whole word masking based on the model type and arguments.
* It initializes the Trainer with the loaded model, arguments, and dataset.

**Notable Dependencies and Requirements**

* `HfArgumentParser`
* `ModelArguments`, `DataTrainingArguments`, `TrainingArguments`
* `AutoConfig`, `AutoTokenizer` (pre-trained models)
* `Trainer` from Transformers library
* `get_dataset` function (not shown in the code snippet)

**External Dependencies**

* Training Args: `training_args.model_name_or_path`, `training_args.output_dir`, etc.
* Model Configuration: `model_config`
* Data Datasets (e.g., `CocoDataset`)
```


key: chroma:document
string_value:
CODE:

def _mp_fn(index):
    # For xla_spawn (TPUs)


SUMMARY:
**Function Summary**

The `_mp_fn` function serves as the entry point for multiprocessing in the context of TensorFlow's XLA SPAWN (TensorFlow 2.0 Parallel Engine). It calls the `main` function when running on TPU hardware.

**Key Implementation Details:**

* The `_mp_fn` function is designed to work with XLA Spawn, a parallel execution engine for TensorFlow.
* It takes an `index` parameter, although its purpose is unclear without further context.

**Notable Dependencies:**

* `main()`: A central execution function that initializes the TPU environment.
* TensorFlow XLA Spawn (library): An external dependency required to enable TPUs in the execution pipeline.


key: chroma:document
string_value:
CODE:

class SwagExample(object):
    """A single training/test example for the SWAG dataset."""

    def __init__(self, swag_id, context_sentence, start_ending, ending_0, ending_1, ending_2, ending_3, label=None):
        self.swag_id = swag_id
        self.context_sentence = context_sentence
        self.start_ending = start_ending
        self.endings = [
            ending_0,
            ending_1,
            ending_2,
            ending_3,
        ]
        self.label = label

    def __str__(self):
        return self.__repr__()

    def __repr__(self):
        attributes = [
            "swag_id: {}".format(self.swag_id),
            "context_sentence: {}".format(self.context_sentence),
            "start_ending: {}".format(self.start_ending),
            "ending_0: {}".format(self.endings[0]),
            "ending_1: {}".format(self.endings[1]),
            "ending_2: {}".format(self.endings[2]),
            "ending_3: {}".format(self.endings[3]),
        ]

        if self.label is not None:
            attributes.append("label: {}".format(self.label))



SUMMARY:
**Class Summary: SwagExample**

This class represents a single training/test example for the SWAG (Stanford Web Aggregating Game) dataset. It contains information about the context sentence, start and end tokens, and label.

**Key Implementation Details:**

* The class uses an instance of list to store multiple ending tokens (`self.endings = [...]`).
* A custom implementation of `__repr__()` is provided to handle string representation of the object.
* Method `.__str__()`, which simply calls `.__repr__()`, overrides the default behavior.

**Notable Dependencies and Requirements:**

None; this class appears to be self-contained with no external dependencies.


key: chroma:document
string_value:
CODE:

class InputFeatures(object):
    def __init__(self, example_id, choices_features, label):
        self.example_id = example_id
        self.choices_features = [
            {"input_ids": input_ids, "input_mask": input_mask, "segment_ids": segment_ids}
            for _, input_ids, input_mask, segment_ids in choices_features
        ]


SUMMARY:
**InputFeatures Class Summary**

**Purpose and Functionality:**
The `InputFeatures` class is used to process inputs in Natural Language Processing (NLP) tasks. It stores input IDs, masks, and segment IDs for each example.

**Key Details:**

* Initializes with `example_id`, `choices_features`, and `label`.
* Converts `choices_features` into a list of dictionaries containing input features:
	+ `input_ids`
	+ `input_mask`
	+ `segment_ids`

**Notable Dependencies/Requirements:**

* `choices_features`: a list of tuples containing input data (input_ids, input_mask, segment_ids)
* External dependencies not specified

This class appears to be part of a larger NLP framework, likely TensorFlow or PyTorch, based on the use of `example_id`, `choices_features`, and common encoding schemes for text data.


key: chroma:document
string_value:
CODE:

def read_swag_examples(input_file, is_training=True):
    with open(input_file, "r", encoding="utf-8") as f:
        lines = list(csv.reader(f))

    if is_training and lines[0][-1] != "label":
        raise ValueError("For training, the input file must contain a label column.")

    examples = [
        SwagExample(
            swag_id=line[2],
            context_sentence=line[4],
            start_ending=line[5],  # in the swag dataset, the
            # common beginning of each
            # choice is stored in "sent2".
            ending_0=line[7],
            ending_1=line[8],
            ending_2=line[9],
            ending_3=line[10],
            label=int(line[11]) if is_training else None,
        )
        for line in lines[1:]  # we skip the line with the column names
    ]



SUMMARY:
**Function Summary: `read_swag_examples`**

Reads SWAG dataset examples from a CSV file.

**Purpose and Functionality**
The function reads a CSV file containing SWAG dataset examples and returns a list of `SwagExample` objects.

**Key Implementation Details**

* Reads the CSV file line by line using `csv.reader`.
* Checks if the input file contains a label column for training data.
* Creates `SwagExample` objects from each line, except the first (column names) line.
* Inheritance/Iface:
	+ Uses `SwagDataset` class (not shown).

**Dependencies and Requirements**

* `csv` module for reading CSV files.
* **Assumed external dependencies:** `SwagExample`, `SwagDataset` classes.

**Method Signatures and Interfaces**
Not provided as the function is self-contained.


key: chroma:document
string_value:
CODE:

def convert_examples_to_features(examples, tokenizer, max_seq_length, is_training):
    """Loads a data file into a list of `InputBatch`s."""

    # Swag is a multiple choice task. To perform this task using Bert,
    # we will use the formatting proposed in "Improving Language
    # Understanding by Generative Pre-Training" and suggested by
    # @jacobdevlin-google in this issue
    # https://github.com/google-research/bert/issues/38.
    #
    # Each choice will correspond to a sample on which we run the
    # inference. For a given Swag example, we will create the 4
    # following inputs:
    # - [CLS] context [SEP] choice_1 [SEP]
    # - [CLS] context [SEP] choice_2 [SEP]
    # - [CLS] context [SEP] choice_3 [SEP]
    # - [CLS] context [SEP] choice_4 [SEP]
    # The model will output a single value for each input. To get the
    # final decision of the model, we will run a softmax over these 4
    # outputs.
    features = []
    for example_index, example in tqdm(enumerate(examples)):
        context_tokens = tokenizer.tokenize(example.context_sentence)
        start_ending_tokens = tokenizer.tokenize(example.start_ending)

        choices_features = []
        for ending_index, ending in enumerate(example.endings):
            # We create a copy of the context tokens in order to be
            # able to shrink it according to ending_tokens
            context_tokens_choice = context_tokens[:]
            ending_tokens = start_ending_tokens + tokenizer.tokenize(ending)
            # Modifies `context_tokens_choice` and `ending_tokens` in
            # place so that the total length is less than the
            # specified length.  Account for [CLS], [SEP], [SEP] with
            # "- 3"
            _truncate_seq_pair(context_tokens_choice, ending_tokens, max_seq_length - 3)

            tokens = ["[CLS]"] + context_tokens_choice + ["[SEP]"] + ending_tokens + ["[SEP]"]
            segment_ids = [0] * (len(context_tokens_choice) + 2) + [1] * (len(ending_tokens) + 1)

            input_ids = tokenizer.convert_tokens_to_ids(tokens)
            input_mask = [1] * len(input_ids)

            # Zero-pad up to the sequence length.
            padding = [0] * (max_seq_length - len(input_ids))
            input_ids += padding
            input_mask += padding
            segment_ids += padding

            assert len(input_ids) == max_seq_length
            assert len(input_mask) == max_seq_length
            assert len(segment_ids) == max_seq_length

            choices_features.append((tokens, input_ids, input_mask, segment_ids))

        label = example.label
        if example_index < 5:
            logger.info("*** Example ***")
            logger.info("swag_id: {}".format(example.swag_id))
            for choice_idx, (tokens, input_ids, input_mask, segment_ids) in enumerate(choices_features):
                logger.info("choice: {}".format(choice_idx))
                logger.info("tokens: {}".format(" ".join(tokens)))
                logger.info("input_ids: {}".format(" ".join(map(str, input_ids))))
                logger.info("input_mask: {}".format(" ".join(map(str, input_mask))))
                logger.info("segment_ids: {}".format(" ".join(map(str, segment_ids))))
            if is_training:
                logger.info("label: {}".format(label))

        features.append(InputFeatures(example_id=example.swag_id, choices_features=choices_features, label=label))



SUMMARY:
**Function Summary**

`convert_examples_to_features`: Converts a list of Swag example objects to a list of `InputBatch` objects for BERT model training.

**Main Purpose and Functionality**

Loads data examples into a format compatible with the BERT algorithm for multiple choice tasks. For each example, generates four input sequences by combining the context sentence with each possible ending token, truncating them to a specified length, and returns their corresponding input IDs, mask, and segment IDs.

**Key Implementation Details**

* Uses `tokenizer.tokenize` and `tokenizer.convert_tokens_to_ids` to convert text inputs into numerical representations.
* Applies truncton to sequence lengths using `_truncate_seq_pair`.
* Generates four input sequences for each example: `[CLS] context [SEP] choice_1 [SEP]`, `[CLS] context [SEP] choice_2 [SEP]`, etc.
* Returns a list of `InputFeatures` objects, containing the original example ID, choices' features (tokens, input IDs, mask, and segment IDs), and the label.

**Notable Dependencies or Requirements**

* The `transformers` library for BERT-related functionality.
* External dependencies:
	+ `logger`: A logging module for displaying example information during training.

**Key Interfaces and Method Signatures**

* `example_id`
* `choices_features` (list of tuples)
* `label`

* `_truncate_seq_pair(context_tokens_choice, ending_tokens, max_seq_length - 3)`

Note that this summary is concise and focused on the key aspects of the function code chunk.


key: chroma:document
string_value:
CODE:

def _truncate_seq_pair(tokens_a, tokens_b, max_length):
    """Truncates a sequence pair in place to the maximum length."""

    # This is a simple heuristic which will always truncate the longer sequence
    # one token at a time. This makes more sense than truncating an equal percent
    # of tokens from each, since if one sequence is very short then each token
    # that's truncated likely contains more information than a longer sequence.
    while True:
        total_length = len(tokens_a) + len(tokens_b)
        if total_length <= max_length:
            break
        if len(tokens_a) > len(tokens_b):
            tokens_a.pop()
        else:


SUMMARY:
Here is a concise and focused summary of the function code chunk:

**Summary:**

Truncates a sequence pair (`tokens_a` & `tokens_b`) in place to a maximum length of `max_length`. This heuristic truncates the longer sequence one token at a time.

**Key Implementation Details:**

* Truncating logic: Always truncate longer sequence, removing one token at a time.
* Method Signatures and Interfaces:
	+ `_truncate_seq_pair(tokens_a, tokens_b, max_length)`

**Notable Dependencies/Requirements:**

* No external dependencies noted.
* No inherited or implemented interfaces mentioned.


key: chroma:document
string_value:
CODE:

def accuracy(out, labels):
    outputs = np.argmax(out, axis=1)


SUMMARY:
**Function Summary:**

`accuracy` calculates the accuracy of output predictions by comparing them to ground truth labels.

**Implementation Details:**

* Computes predicted labels using `np.argmax` with axis 1.
* Compares predicted labels to actual labels (using `==` operator) for each sample.
* Returns the sum of all correct comparisons, yielding the overall accuracy score.

**Notable Dependencies/Interfaces:**

* `out`: expected output tensor
* `labels`: ground truth label tensor
* `np.argmax` and `np.sum`: NumPy functions for computing predicted labels and summing comparisons, respectively.


key: chroma:document
string_value:
CODE:

def select_field(features, field):


SUMMARY:
**Function Summary:**

```python
def select_field(features, field)
```

**Purpose and Functionality:**

This function takes two inputs, `features` and `field`, and returns a list of lists. It extracts the values from the specified `field` in each `feature` within the `features` collection.

**Key Implementation Details:**

* Uses nested list comprehension to iterate through `feature.choices_features`.
* Returns a 2D list (`[[value for value in ...]]`) containing the specific field values.

**Dependencies and Requirements:**

* Assumes existence of `choices_features` attribute within each `feature` object.
* No external dependencies.


key: chroma:document
string_value:
CODE:

def set_seed(args):
    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    if args.n_gpu > 0:


SUMMARY:
**Summary: `set_seed` Function**

Sets the seed for various random number generators used in the library.

**Main Purpose and Functionality**
The function sets seeds for:

* PyTorch (`torch.random`)
* NumPy (`np.random`)
* Python's built-in `random` module
* GPU Random Number Generators (PyTorch CUDA, if enabled)

**Implementation Details**

* Seeding is controlled by the `args.seed` variable.
* `args.n_gpu > 0` condition sets GPU-based seeding for PyTorch CUDA.

**Notable Dependencies/Requirements**
No explicit dependencies mentioned. However, PyTorch and NumPy are implicit requirements for this function to work correctly.


key: chroma:document
string_value:
CODE:

def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
    if args.local_rank not in [-1, 0]:
        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache

    # Load data features from cache or dataset file
    input_file = args.predict_file if evaluate else args.train_file
    cached_features_file = os.path.join(
        os.path.dirname(input_file),
        "cached_{}_{}_{}".format(
            "dev" if evaluate else "train",
            list(filter(None, args.model_name_or_path.split("/"))).pop(),
            str(args.max_seq_length),
        ),
    )
    if os.path.exists(cached_features_file) and not args.overwrite_cache and not output_examples:
        logger.info("Loading features from cached file %s", cached_features_file)
        features = torch.load(cached_features_file)
    else:
        logger.info("Creating features from dataset file at %s", input_file)
        examples = read_swag_examples(input_file)
        features = convert_examples_to_features(examples, tokenizer, args.max_seq_length, not evaluate)

        if args.local_rank in [-1, 0]:
            logger.info("Saving features into cached file %s", cached_features_file)
            torch.save(features, cached_features_file)

    if args.local_rank == 0:
        torch.distributed.barrier()  # Make sure only the first process in distributed training process the dataset, and the others will use the cache

    # Convert to Tensors and build dataset
    all_input_ids = torch.tensor(select_field(features, "input_ids"), dtype=torch.long)
    all_input_mask = torch.tensor(select_field(features, "input_mask"), dtype=torch.long)
    all_segment_ids = torch.tensor(select_field(features, "segment_ids"), dtype=torch.long)
    all_label = torch.tensor([f.label for f in features], dtype=torch.long)

    if evaluate:
        dataset = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label)
    else:
        dataset = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label)

    if output_examples:
        return dataset, examples, features


SUMMARY:
**Function Summary: `load_and_cache_examples`**

Purpose:
 Load and cache input data features for a text classification task using the given tokenizer.

**Key Implementation Details**

* Loads data features from cache or dataset file based on `evaluate` flag
* Converts examples to BERT features using `convert_examples_to_features`
* Saves cached features to disk when necessary

**Noteble Dependencies/Requirements**

* PyTorch and Hugging Face's Tokenizers library for distributed training and caching
* `TensorDataset` class from PyTorch for creating dataset objects
* `convert_examples_to_features` function (not shown in this snippet) which is assumed to be implemented elsewhere

**Method Signatures and Interfaces**

* `load_and_cache_examples`: takes 5 arguments, including `args`, `tokenizer`, `evaluate`, and `output_examples`
	+ Returns a `TensorDataset` object or a tuple containing the dataset, examples, and cached features


key: chroma:document
string_value:
CODE:

def train(args, train_dataset, model, tokenizer):
    """Train the model"""
    if args.local_rank in [-1, 0]:
        tb_writer = SummaryWriter()

    args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)
    train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
    train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size)

    if args.max_steps > 0:
        t_total = args.max_steps
        args.num_train_epochs = args.max_steps // (len(train_dataloader) // args.gradient_accumulation_steps) + 1
    else:
        t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs

    # Prepare optimizer and schedule (linear warmup and decay)
    no_decay = ["bias", "LayerNorm.weight"]
    optimizer_grouped_parameters = [
        {
            "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
            "weight_decay": args.weight_decay,
        },
        {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], "weight_decay": 0.0},
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
    scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
    )
    if args.fp16:
        try:
            from apex import amp
        except ImportError:
            raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
        model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)

    # multi-gpu training (should be after apex fp16 initialization)
    if args.n_gpu > 1:
        model = torch.nn.DataParallel(model)

    # Distributed training (should be after apex fp16 initialization)
    if args.local_rank != -1:
        model = torch.nn.parallel.DistributedDataParallel(
            model, device_ids=[args.local_rank], output_device=args.local_rank, find_unused_parameters=True
        )

    # Train!
    logger.info("***** Running training *****")
    logger.info("  Num examples = %d", len(train_dataset))
    logger.info("  Num Epochs = %d", args.num_train_epochs)
    logger.info("  Instantaneous batch size per GPU = %d", args.per_gpu_train_batch_size)
    logger.info(
        "  Total train batch size (w. parallel, distributed & accumulation) = %d",
        args.train_batch_size
        * args.gradient_accumulation_steps
        * (torch.distributed.get_world_size() if args.local_rank != -1 else 1),
    )
    logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
    logger.info("  Total optimization steps = %d", t_total)

    global_step = 0
    tr_loss, logging_loss = 0.0, 0.0
    model.zero_grad()
    train_iterator = trange(int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0])
    set_seed(args)  # Added here for reproducibility
    for _ in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
        for step, batch in enumerate(epoch_iterator):
            model.train()
            batch = tuple(t.to(args.device) for t in batch)
            inputs = {
                "input_ids": batch[0],
                "attention_mask": batch[1],
                # 'token_type_ids':  None if args.model_type == 'xlm' else batch[2],
                "token_type_ids": batch[2],
                "labels": batch[3],
            }
            # if args.model_type in ['xlnet', 'xlm']:
            #     inputs.update({'cls_index': batch[5],
            #                    'p_mask':       batch[6]})
            outputs = model(**inputs)
            loss = outputs[0]  # model outputs are always tuple in transformers (see doc)

            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel (not distributed) training
            if args.gradient_accumulation_steps > 1:
                loss = loss / args.gradient_accumulation_steps

            if args.fp16:
                with amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
                torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
            else:
                loss.backward()
                torch.nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)

            tr_loss += loss.item()
            if (step + 1) % args.gradient_accumulation_steps == 0:
                optimizer.step()
                scheduler.step()  # Update learning rate schedule
                model.zero_grad()
                global_step += 1

                if args.local_rank in [-1, 0] and args.logging_steps > 0 and global_step % args.logging_steps == 0:
                    # Log metrics
                    if (
                        args.local_rank == -1 and args.evaluate_during_training
                    ):  # Only evaluate when single GPU otherwise metrics may not average well
                        results = evaluate(args, model, tokenizer)
                        for key, value in results.items():
                            tb_writer.add_scalar("eval_{}".format(key), value, global_step)
                    tb_writer.add_scalar("lr", scheduler.get_lr()[0], global_step)
                    tb_writer.add_scalar("loss", (tr_loss - logging_loss) / args.logging_steps, global_step)
                    logging_loss = tr_loss

                if args.local_rank in [-1, 0] and args.save_steps > 0 and global_step % args.save_steps == 0:
                    # Save model checkpoint
                    output_dir = os.path.join(args.output_dir, "checkpoint-{}".format(global_step))
                    model_to_save = (
                        model.module if hasattr(model, "module") else model
                    )  # Take care of distributed/parallel training
                    model_to_save.save_pretrained(output_dir)
                    tokenizer.save_vocabulary(output_dir)
                    torch.save(args, os.path.join(output_dir, "training_args.bin"))
                    logger.info("Saving model checkpoint to %s", output_dir)

            if args.max_steps > 0 and global_step > args.max_steps:
                epoch_iterator.close()
                break
        if args.max_steps > 0 and global_step > args.max_steps:
            train_iterator.close()
            break

    if args.local_rank in [-1, 0]:
        tb_writer.close()



SUMMARY:
**Function Summary: `train`**

**Main Purpose and Functionality**

The `train` function is responsible for training the Hugging Face model using PyTorch. It takes in training arguments (`args`), the training dataset, the model, and tokenizer as inputs.

**Key Implementation Details**

1. **Optimizer**: The function initializes an AdamW optimizer with weight decay and learning rate.
2. **Scheduler**: A linear warmup and decay schedule is applied to the learning rate.
3. **FP16 Training**: If `fp16` is activated, Apex's Automatic Mixed Precision (AMP) is used for mixed precision training.
4. **Multi-GPU Training**: The model is parallelized using DataParallel or DistributedDataParallel depending on the number of GPUs.
5. **Distributed Training**: If `local_rank` is not -1, the model is distributed across multiple GPUs.
6. **Training Loop**: The function iterates over the training dataset, applies the optimizer and schedule, and logs metrics.

**Notable Dependencies**

* `torch`
* ` transformers `
* `apex`
* `numpy`
* `logging`

**External Requirements**

* Apex library for mixed precision training (if `fp16` is activated)
* DataParallel and DistributedDataParallel modules for parallelization


key: chroma:document
string_value:
CODE:

def evaluate(args, model, tokenizer, prefix=""):
    dataset, examples, features = load_and_cache_examples(args, tokenizer, evaluate=True, output_examples=True)

    if not os.path.exists(args.output_dir) and args.local_rank in [-1, 0]:
        os.makedirs(args.output_dir)

    args.eval_batch_size = args.per_gpu_eval_batch_size * max(1, args.n_gpu)
    # Note that DistributedSampler samples randomly
    eval_sampler = SequentialSampler(dataset) if args.local_rank == -1 else DistributedSampler(dataset)
    eval_dataloader = DataLoader(dataset, sampler=eval_sampler, batch_size=args.eval_batch_size)

    # Eval!
    logger.info("***** Running evaluation {} *****".format(prefix))
    logger.info("  Num examples = %d", len(dataset))
    logger.info("  Batch size = %d", args.eval_batch_size)

    eval_loss, eval_accuracy = 0, 0
    nb_eval_steps, nb_eval_examples = 0, 0

    for batch in tqdm(eval_dataloader, desc="Evaluating"):
        model.eval()
        batch = tuple(t.to(args.device) for t in batch)
        with torch.no_grad():
            inputs = {
                "input_ids": batch[0],
                "attention_mask": batch[1],
                # 'token_type_ids': None if args.model_type == 'xlm' else batch[2]  # XLM don't use segment_ids
                "token_type_ids": batch[2],
                "labels": batch[3],
            }

            # if args.model_type in ['xlnet', 'xlm']:
            #     inputs.update({'cls_index': batch[4],
            #                    'p_mask':    batch[5]})
            outputs = model(**inputs)
            tmp_eval_loss, logits = outputs[:2]
            eval_loss += tmp_eval_loss.mean().item()

        logits = logits.detach().cpu().numpy()
        label_ids = inputs["labels"].to("cpu").numpy()
        tmp_eval_accuracy = accuracy(logits, label_ids)
        eval_accuracy += tmp_eval_accuracy

        nb_eval_steps += 1
        nb_eval_examples += inputs["input_ids"].size(0)

    eval_loss = eval_loss / nb_eval_steps
    eval_accuracy = eval_accuracy / nb_eval_examples
    result = {"eval_loss": eval_loss, "eval_accuracy": eval_accuracy}

    output_eval_file = os.path.join(args.output_dir, "eval_results.txt")
    with open(output_eval_file, "w") as writer:
        logger.info("***** Eval results *****")
        for key in sorted(result.keys()):
            logger.info("%s = %s", key, str(result[key]))
            writer.write("%s = %s\n" % (key, str(result[key])))



SUMMARY:
**Summary**

This function evaluates the performance of a language model on a dataset. The main purpose is to assess the model's accuracy and loss metrics.

**Main Purpose and Functionality**

The `evaluate` function:
1. Loads and caches the dataset using `load_and_cache_examples`.
2. Creates an evaluation data loader with a batch size proportional to the number of GPUs.
3. Evaluates the model by iterating over batches, computing loss and accuracy, and storing results.

**Key Implementation Details**

* Uses Hugging Face's `SequentialSampler` and `DataLoader` for efficient evaluation.
* Computes loss and accuracy using PyTorch's automatic differentiation and tensor operations.

**Notable Dependencies or Requirements**

* PyTorch
* Hugging Face Transformers library (specifically, the `model`, `tokenizer`, and `load_and_cache_examples` functions)
* `os`, `logger` modules (for directory creation and logging)

**Output**

The function returns a dictionary containing the evaluation loss and accuracy metrics, which are written to an output file along with their values.


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()

    # Required parameters
    parser.add_argument(
        "--train_file", default=None, type=str, required=True, help="SWAG csv for training. E.g., train.csv"
    )
    parser.add_argument(
        "--predict_file",
        default=None,
        type=str,
        required=True,
        help="SWAG csv for predictions. E.g., val.csv or test.csv",
    )
    parser.add_argument(
        "--model_name_or_path",
        default=None,
        type=str,
        required=True,
        help="Path to pretrained model or model identifier from huggingface.co/models",
    )
    parser.add_argument(
        "--output_dir",
        default=None,
        type=str,
        required=True,
        help="The output directory where the model checkpoints and predictions will be written.",
    )

    # Other parameters
    parser.add_argument(
        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
    )
    parser.add_argument(
        "--tokenizer_name",
        default="",
        type=str,
        help="Pretrained tokenizer name or path if not the same as model_name",
    )
    parser.add_argument(
        "--max_seq_length",
        default=384,
        type=int,
        help=(
            "The maximum total input sequence length after tokenization. Sequences "
            "longer than this will be truncated, and sequences shorter than this will be padded."
        ),
    )
    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
    parser.add_argument(
        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
    )

    parser.add_argument("--per_gpu_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.")
    parser.add_argument(
        "--per_gpu_eval_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation."
    )
    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
    parser.add_argument(
        "--gradient_accumulation_steps",
        type=int,
        default=1,
        help="Number of updates steps to accumulate before performing a backward/update pass.",
    )
    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
    parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
    parser.add_argument(
        "--num_train_epochs", default=3.0, type=float, help="Total number of training epochs to perform."
    )
    parser.add_argument(
        "--max_steps",
        default=-1,
        type=int,
        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
    )
    parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")

    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
    parser.add_argument(
        "--eval_all_checkpoints",
        action="store_true",
        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
    )
    parser.add_argument("--no_cuda", action="store_true", help="Whether not to use CUDA when available")
    parser.add_argument(
        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
    )
    parser.add_argument(
        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
    )
    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")

    parser.add_argument("--local_rank", type=int, default=-1, help="local_rank for distributed training on gpus")
    parser.add_argument(
        "--fp16",
        action="store_true",
        help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit",
    )
    parser.add_argument(
        "--fp16_opt_level",
        type=str,
        default="O1",
        help=(
            "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. "
            "See details at https://nvidia.github.io/apex/amp.html"
        ),
    )
    parser.add_argument("--server_ip", type=str, default="", help="Can be used for distant debugging.")
    parser.add_argument("--server_port", type=str, default="", help="Can be used for distant debugging.")
    args = parser.parse_args()

    if (
        os.path.exists(args.output_dir)
        and os.listdir(args.output_dir)
        and args.do_train
        and not args.overwrite_output_dir
    ):
        raise ValueError(
            "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
                args.output_dir
            )
        )

    # Setup distant debugging if needed
    if args.server_ip and args.server_port:
        # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
        import ptvsd

        print("Waiting for debugger attach")
        ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
        ptvsd.wait_for_attach()

    # Setup CUDA, GPU & distributed training
    if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
        args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count()
    else:  # Initializes the distributed backend which will take care of synchronizing nodes/GPUs
        torch.cuda.set_device(args.local_rank)
        device = torch.device("cuda", args.local_rank)
        torch.distributed.init_process_group(backend="nccl")
        args.n_gpu = 1
    args.device = device

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO if args.local_rank in [-1, 0] else logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        args.local_rank,
        device,
        args.n_gpu,
        bool(args.local_rank != -1),
        args.fp16,
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()

    # Set seed
    set_seed(args)

    # Load pretrained model and tokenizer
    if args.local_rank not in [-1, 0]:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

    config = AutoConfig.from_pretrained(args.config_name if args.config_name else args.model_name_or_path)
    tokenizer = AutoTokenizer.from_pretrained(
        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
    )
    model = AutoModelForMultipleChoice.from_pretrained(
        args.model_name_or_path, from_tf=bool(".ckpt" in args.model_name_or_path), config=config
    )

    if args.local_rank == 0:
        torch.distributed.barrier()  # Make sure only the first process in distributed training will download model & vocab

    model.to(args.device)

    logger.info("Training/evaluation parameters %s", args)

    # Training
    if args.do_train:
        train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
        global_step, tr_loss = train(args, train_dataset, model, tokenizer)
        logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)

    # Save the trained model and the tokenizer
    if args.local_rank == -1 or torch.distributed.get_rank() == 0:
        logger.info("Saving model checkpoint to %s", args.output_dir)
        # Save a trained model, configuration and tokenizer using `save_pretrained()`.
        # They can then be reloaded using `from_pretrained()`
        model_to_save = (
            model.module if hasattr(model, "module") else model
        )  # Take care of distributed/parallel training
        model_to_save.save_pretrained(args.output_dir)
        tokenizer.save_pretrained(args.output_dir)

        # Good practice: save your training arguments together with the trained model
        torch.save(args, os.path.join(args.output_dir, "training_args.bin"))

        # Load a trained model and vocabulary that you have fine-tuned
        model = AutoModelForMultipleChoice.from_pretrained(args.output_dir)
        tokenizer = AutoTokenizer.from_pretrained(args.output_dir)
        model.to(args.device)

    # Evaluation - we can ask to evaluate all the checkpoints (sub-directories) in a directory
    results = {}
    if args.do_eval and args.local_rank in [-1, 0]:
        if args.do_train:
            checkpoints = [args.output_dir]
        else:
            # if do_train is False and do_eval is true, load model directly from pretrained.
            checkpoints = [args.model_name_or_path]

        if args.eval_all_checkpoints:
            checkpoints = [
                os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + "/**/" + WEIGHTS_NAME, recursive=True))
            ]

        logger.info("Evaluate the following checkpoints: %s", checkpoints)

        for checkpoint in checkpoints:
            # Reload the model
            global_step = checkpoint.split("-")[-1] if len(checkpoints) > 1 else ""
            model = AutoModelForMultipleChoice.from_pretrained(checkpoint)
            tokenizer = AutoTokenizer.from_pretrained(checkpoint)
            model.to(args.device)

            # Evaluate
            result = evaluate(args, model, tokenizer, prefix=global_step)

            result = {k + ("_{}".format(global_step) if global_step else ""): v for k, v in result.items()}
            results.update(result)

    logger.info("Results: {}".format(results))



SUMMARY:
This is the main script of the transformer-based text classification model in Hugging Face's Transformers library. The code trains a model to perform a specific task, saves the trained model and checkpoints, and evaluates the performance of the model on a test dataset.

Here are some key features of this code:

*   It loads pre-trained models and configurations using the `AutoConfig` and `AutoTokenizer` classes.
*   It trains a model using a custom training function (`train`) that takes in arguments such as data paths, model name or path, tokenizer name or path, etc. The training process involves generating examples from input text datasets, loading these examples into batches, computing the loss metric, optimizing the model parameters during backpropagation, and updating the evaluation metric.
*   After training is complete, it saves the trained model and a `training_args` dictionary containing all of the arguments supplied by the user. It also calculates performance metrics such as perplexity for model validation and saves those to the checkpoints.
*   For fine-tuning on new pre-trained models or on smaller training datasets, the script can load these instances in place of the pre-trained ones.

In a nutshell, this code snippet enables creating robust text classification systems. Given well-suited text data sets and properly configured tools for data processing, evaluation, and optimization (including learning schedule tuning), it offers flexible support for experimenting with diverse machine learning approaches such as different neural network architectures or loss functions in deep neural network models.


key: chroma:document
string_value:
CODE:

def simple_accuracy(preds, labels):


SUMMARY:
**Function Summary**

`simple_accuracy` calculates the accuracy of model predictions.

* **Purpose**: Evaluate the accuracy of model output against ground truth labels.
* **Methodology**: Compares predicted values (`preds`) with actual labels (`labels`) and returns the mean true positive rate as a float value.
* **Implementation Details**:
	+ Returns the proportion of correct predictions.
	+ Utilizes NumPy's vectorized comparison (`preds == labels`) for efficient computation.

No notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    if (
        os.path.exists(training_args.output_dir)
        and os.listdir(training_args.output_dir)
        and training_args.do_train
        and not training_args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({training_args.output_dir}) already exists and is not empty. Use"
            " --overwrite_output_dir to overcome."
        )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO if training_args.local_rank in [-1, 0] else logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        training_args.local_rank,
        training_args.device,
        training_args.n_gpu,
        bool(training_args.local_rank != -1),
        training_args.fp16,
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    logger.info("Training/evaluation parameters %s", training_args)

    # Set seed
    set_seed(training_args.seed)

    try:
        processor = processors[data_args.task_name]()
        label_list = processor.get_labels()
        num_labels = len(label_list)
    except KeyError:
        raise ValueError("Task not found: %s" % (data_args.task_name))

    # Load pretrained model and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.

    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        num_labels=num_labels,
        finetuning_task=data_args.task_name,
        cache_dir=model_args.cache_dir,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
    )
    model = AutoModelForMultipleChoice.from_pretrained(
        model_args.model_name_or_path,
        from_tf=bool(".ckpt" in model_args.model_name_or_path),
        config=config,
        cache_dir=model_args.cache_dir,
    )

    # Get datasets
    train_dataset = (
        MultipleChoiceDataset(
            data_dir=data_args.data_dir,
            tokenizer=tokenizer,
            task=data_args.task_name,
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.train,
        )
        if training_args.do_train
        else None
    )
    eval_dataset = (
        MultipleChoiceDataset(
            data_dir=data_args.data_dir,
            tokenizer=tokenizer,
            task=data_args.task_name,
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            mode=Split.dev,
        )
        if training_args.do_eval
        else None
    )

    def compute_metrics(p: EvalPrediction) -> Dict:
        preds = np.argmax(p.predictions, axis=1)
        return {"acc": simple_accuracy(preds, p.label_ids)}

    # Data collator
    data_collator = DataCollatorWithPadding(tokenizer, pad_to_multiple_of=8) if training_args.fp16 else None

    # Initialize our Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        compute_metrics=compute_metrics,
        data_collator=data_collator,
    )

    # Training
    if training_args.do_train:
        trainer.train(
            model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
        )
        trainer.save_model()
        # For convenience, we also re-save the tokenizer to the same directory,
        # so that you can share your model easily on huggingface.co/models =)
        if trainer.is_world_master():
            tokenizer.save_pretrained(training_args.output_dir)

    # Evaluation
    results = {}
    if training_args.do_eval:
        logger.info("*** Evaluate ***")

        result = trainer.evaluate()

        output_eval_file = os.path.join(training_args.output_dir, "eval_results.txt")
        if trainer.is_world_master():
            with open(output_eval_file, "w") as writer:
                logger.info("***** Eval results *****")
                for key, value in result.items():
                    logger.info("  %s = %s", key, value)
                    writer.write("%s = %s\n" % (key, value))

                results.update(result)



SUMMARY:
**Main Functionality:**

This function is the main entry point of a Hugging Face-based natural language processing model trainer. It parses command-line arguments and sets up the training process for a multi-choice classification task.

**Key Implementation Details:**

*   Sets up logging, seed, and processor configuration
*   Loads pre-trained model and tokenizer using `AutoConfig` and `AutoTokenizer`
*   Configures data loaders for training and evaluation datasets
*   Defines a custom metric computation function for accuracy calculation
*   Initializes the Trainer with the model, arguments, and dataset loaders
*   Trains the model if the `do_train` flag is True and saves the trained model

**Notable Dependencies/Requirements:**

*   Hugging Face Transformers library
*   Processor classes from the transformers library (e.g., `MultipleChoiceProcessor`)
*   `DataCollatorWithPadding` for padding sequences to a specific length

**Notes:**

The function uses the `Trainer` class from the Hugging Face library to manage the training process. It also enables logging, sets up seed configuration, and loads pre-trained model and tokenizer using the `AutoConfig` and `AutoTokenizer` classes. The custom metric computation function is defined for accuracy calculation.

Here's a high-level overview of the code in pseudocode:

1.  Parse command-line arguments
2.  Set up logging and seed configurations
3.  Load pre-trained model and tokenizer
4.  Configure data loaders for training and evaluation datasets
5.  Initialize Trainer with model, arguments, and dataset loaders
6.  Train the model if enabled

The code snippet provided is a part of a larger library or framework and is designed to be used in conjunction with other components to train a machine learning model. The functionality and implementation details can be further distilled into more specific modules and functions as needed. 

Here's an alternative, high-level summary for a reader not familiar with the Hugging Face Transformers library:

This Python code defines a function responsible for training a natural language processing (NLP) model using the Hugging Face framework. The model is trained on a multi-choice classification task.

Here's the same information condensed into table format for better legibility:

| **Component** |  **Description**  |
| -------------- | ---------------- |
| **Functionality**: |  Trains an NLP model within the Hugging Face framework. |
| **Key Details:** |  Sets up logging, seed configuration and loads pre-trained model and tokenizer. |
| **Notable Dependencies: |  Hugging Face Transformers library for training data and output generation of predictions and results.|
| **Notes**:     The code is designed as a part of larger libraries or frameworks.


key: chroma:document
string_value:
CODE:

def _mp_fn(index):
    # For xla_spawn (TPUs)


SUMMARY:
**Summary:**

The `_mp_fn` function is the entry point for multi-processing in XLA compilation, responsible for executing the `main()` function on TPU devices.

**Key Details:**

* Execution domain: `xla_spawn` (TPUs)
* Main function to be executed: `main()`

**Notable Dependencies/Requirements:**

* Inherited interface: None
* Implemented interfaces: None
* External dependencies:
	+ `xla_spawn` library (for TPU execution)


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/legacy/multiple_choice/utils_multiple_choice.py


key: chroma:document
string_value:
CODE:

class Split(Enum):
    train = "train"
    dev = "dev"


SUMMARY:
**Summary**

The `Split` Enum class represents different dataset split options, namely `"train"`, `"dev"`, and `"test"`.

**Analysis Points**

* The class has three constants: `train`, `dev`, and `test`, each mapped to a string value.
* No additional implementation details are provided beyond the enum definition.

**No notable dependencies or requirements identified.**


key: chroma:document
string_value:
CODE:

class DataProcessor:
    """Base class for data converters for multiple choice data sets."""

    def get_train_examples(self, data_dir):
        """Gets a collection of `InputExample`s for the train set."""
        raise NotImplementedError()

    def get_dev_examples(self, data_dir):
        """Gets a collection of `InputExample`s for the dev set."""
        raise NotImplementedError()

    def get_test_examples(self, data_dir):
        """Gets a collection of `InputExample`s for the test set."""
        raise NotImplementedError()

    def get_labels(self):
        """Gets the list of labels for this data set."""


SUMMARY:
**DataProcessor Class Summary**

The `DataProcessor` class serves as a base class for data converters in multiple choice datasets. Its primary purpose is to provide methods for retrieving examples (train, dev, test) and labels from a dataset.

**Key Implementation Details:**

* The class uses the `NotImplementedError` exception when specific methods are not implemented by derived classes.
* Four methods are defined:
	+ `get_train_examples`: Retrieves training examples
	+ `get_dev_examples`: Retrieves development examples
	+ `get_test_examples`: Retrieves testing examples
	+ `get_labels`: Returns a list of labels for the dataset

**Notable Dependencies or Requirements:**

* This class relies on the existence of derived classes that implement these methods to handle specific data sets.
* The `InputExample` class is referenced but not defined in this snippet.


key: chroma:document
string_value:
CODE:

class RaceProcessor(DataProcessor):
    """Processor for the RACE data set."""

    def get_train_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} train".format(data_dir))
        high = os.path.join(data_dir, "train/high")
        middle = os.path.join(data_dir, "train/middle")
        high = self._read_txt(high)
        middle = self._read_txt(middle)
        return self._create_examples(high + middle, "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} dev".format(data_dir))
        high = os.path.join(data_dir, "dev/high")
        middle = os.path.join(data_dir, "dev/middle")
        high = self._read_txt(high)
        middle = self._read_txt(middle)
        return self._create_examples(high + middle, "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} test".format(data_dir))
        high = os.path.join(data_dir, "test/high")
        middle = os.path.join(data_dir, "test/middle")
        high = self._read_txt(high)
        middle = self._read_txt(middle)
        return self._create_examples(high + middle, "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1", "2", "3"]

    def _read_txt(self, input_dir):
        lines = []
        files = glob.glob(input_dir + "/*txt")
        for file in tqdm.tqdm(files, desc="read files"):
            with open(file, "r", encoding="utf-8") as fin:
                data_raw = json.load(fin)
                data_raw["race_id"] = file
                lines.append(data_raw)
        return lines

    def _create_examples(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for _, data_raw in enumerate(lines):
            race_id = "%s-%s" % (set_type, data_raw["race_id"])
            article = data_raw["article"]
            for i in range(len(data_raw["answers"])):
                truth = str(ord(data_raw["answers"][i]) - ord("A"))
                question = data_raw["questions"][i]
                options = data_raw["options"][i]

                examples.append(
                    InputExample(
                        example_id=race_id,
                        question=question,
                        contexts=[article, article, article, article],  # this is not efficient but convenient
                        endings=[options[0], options[1], options[2], options[3]],
                        label=truth,
                    )
                )


SUMMARY:
**Class Summary: RaceProcessor**

**Purpose and Functionality:** Data processor for the RACE data set, generating input examples for question answering tasks.

### Key Implementation Details:

* `_get_train_examples`, `_get_dev_examples`, `_get_test_examples`: Read CSV files containing dataset data and create `InputExample` objects.
	+ Use `_read_txt` to read multiple files recursively and merge lines into a single list.
	+ Create `InputExample` objects with question, context, options, and label.
* `_create_examples`: Main function for creating examples.
	+ Applies question and answer operations, including sorting options by preference.
* `_get_labels`: Returns labeled values (`0`, `1`, `2`, `3`) indicating topic.

### Notable Dependencies or Requirements:

* `DataProcessor` (base class)
* `json`
* `pathlib`
* `os`
* `glob`
* `tqdm`

### Key Interfaces and Method Signatures:

* `_read_txt Input Dir` returns `lines` of merged data
* `_create_examples lines, Set Type` returns `examples` in list format.


key: chroma:document
string_value:
CODE:

class SynonymProcessor(DataProcessor):
    """Processor for the Synonym data set."""

    def get_train_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} train".format(data_dir))
        return self._create_examples(self._read_csv(os.path.join(data_dir, "mctrain.csv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} dev".format(data_dir))
        return self._create_examples(self._read_csv(os.path.join(data_dir, "mchp.csv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} dev".format(data_dir))

        return self._create_examples(self._read_csv(os.path.join(data_dir, "mctest.csv")), "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1", "2", "3", "4"]

    def _read_csv(self, input_file):
        with open(input_file, "r", encoding="utf-8") as f:
            return list(csv.reader(f))

    def _create_examples(self, lines: List[List[str]], type: str):
        """Creates examples for the training and dev sets."""

        examples = [
            InputExample(
                example_id=line[0],
                question="",  # in the swag dataset, the
                # common beginning of each
                # choice is stored in "sent2".
                contexts=[line[1], line[1], line[1], line[1], line[1]],
                endings=[line[2], line[3], line[4], line[5], line[6]],
                label=line[7],
            )
            for line in lines  # we skip the line with the column names
        ]



SUMMARY:
**Class Summary: SynonymProcessor**

*   **Purpose:** A data processor class that loads and preprocesses the Synonym dataset.

**Main Functionality:**

The `SynonymProcessor` class is responsible for loading the training, development, and testing datasets from CSV files. It uses the `_read_csv` method to read the CSV files and process the data into `InputExample` objects using the `_create_examples` method.

**Key Implementation Details:**

*   The `get_train_examples`, `get_dev_examples`, `get_test_examples` methods are overriden to load specific datasets from the respective CSV files.
*   The `_read_csv` method reads a CSV file into a list of lists, where each inner list represents a row in the CSV file.
*   The `_create_examples` method processes the data by creating `InputExample` objects for each row in the data.

**Notable Dependencies or Requirements:**

*   The class uses `csv.reader` to read the CSV files.
*   The class assumes that the input CSV files are in the correct format, with specific column names and padding.


key: chroma:document
string_value:
CODE:

class SwagProcessor(DataProcessor):
    """Processor for the SWAG data set."""

    def get_train_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} train".format(data_dir))
        return self._create_examples(self._read_csv(os.path.join(data_dir, "train.csv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} dev".format(data_dir))
        return self._create_examples(self._read_csv(os.path.join(data_dir, "val.csv")), "dev")

    def get_test_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} dev".format(data_dir))
        raise ValueError(
            "For swag testing, the input file does not contain a label column. It can not be tested in current code "
            "setting!"
        )
        return self._create_examples(self._read_csv(os.path.join(data_dir, "test.csv")), "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1", "2", "3"]

    def _read_csv(self, input_file):
        with open(input_file, "r", encoding="utf-8") as f:
            return list(csv.reader(f))

    def _create_examples(self, lines: List[List[str]], type: str):
        """Creates examples for the training and dev sets."""
        if type == "train" and lines[0][-1] != "label":
            raise ValueError("For training, the input file must contain a label column.")

        examples = [
            InputExample(
                example_id=line[2],
                question=line[5],  # in the swag dataset, the
                # common beginning of each
                # choice is stored in "sent2".
                contexts=[line[4], line[4], line[4], line[4]],
                endings=[line[7], line[8], line[9], line[10]],
                label=line[11],
            )
            for line in lines[1:]  # we skip the line with the column names
        ]



SUMMARY:
**SwagProcessor Class Summary**

The `SwagProcessor` class is a data processor designed for the SWAG (Sequence-Based Word-to-Word Translation) dataset. It inherits from the `DataProcessor` base class.

### Purpose and Functionality

The primary purpose of this class is to read, preprocess, and create examples for training, validation, and testing datasets in the SWAG dataset.

### Key Implementation Details

* Reads data from CSV files using `_read_csv` method
* Creates input examples using `_create_examples` method, which populates example IDs, questions, contexts, endings, and labels
* Includes checks to ensure proper label column presence for different types (train, dev, test)

### Notable Dependencies and Requirements

* `csv` module for reading CSV files
* Inherited from `DataProcessor`

### External Dependencies

* None explicitly stated, but assumes access to `logger` and other utility functions.


key: chroma:document
string_value:
CODE:

class ArcProcessor(DataProcessor):
    """Processor for the ARC data set (request from allennlp)."""

    def get_train_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} train".format(data_dir))
        return self._create_examples(self._read_json(os.path.join(data_dir, "train.jsonl")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        logger.info("LOOKING AT {} dev".format(data_dir))
        return self._create_examples(self._read_json(os.path.join(data_dir, "dev.jsonl")), "dev")

    def get_test_examples(self, data_dir):
        logger.info("LOOKING AT {} test".format(data_dir))
        return self._create_examples(self._read_json(os.path.join(data_dir, "test.jsonl")), "test")

    def get_labels(self):
        """See base class."""
        return ["0", "1", "2", "3"]

    def _read_json(self, input_file):
        with open(input_file, "r", encoding="utf-8") as fin:
            lines = fin.readlines()
            return lines

    def _create_examples(self, lines, type):
        """Creates examples for the training and dev sets."""

        # There are two types of labels. They should be normalized
        def normalize(truth):
            if truth in "ABCD":
                return ord(truth) - ord("A")
            elif truth in "1234":
                return int(truth) - 1
            else:
                logger.info("truth ERROR! %s", str(truth))
                return None

        examples = []
        three_choice = 0
        four_choice = 0
        five_choice = 0
        other_choices = 0
        # we deleted example which has more than or less than four choices
        for line in tqdm.tqdm(lines, desc="read arc data"):
            data_raw = json.loads(line.strip("\n"))
            if len(data_raw["question"]["choices"]) == 3:
                three_choice += 1
                continue
            elif len(data_raw["question"]["choices"]) == 5:
                five_choice += 1
                continue
            elif len(data_raw["question"]["choices"]) != 4:
                other_choices += 1
                continue
            four_choice += 1
            truth = str(normalize(data_raw["answerKey"]))
            assert truth != "None"
            question_choices = data_raw["question"]
            question = question_choices["stem"]
            id = data_raw["id"]
            options = question_choices["choices"]
            if len(options) == 4:
                examples.append(
                    InputExample(
                        example_id=id,
                        question=question,
                        contexts=[
                            options[0]["para"].replace("_", ""),
                            options[1]["para"].replace("_", ""),
                            options[2]["para"].replace("_", ""),
                            options[3]["para"].replace("_", ""),
                        ],
                        endings=[options[0]["text"], options[1]["text"], options[2]["text"], options[3]["text"]],
                        label=truth,
                    )
                )

        if type == "train":
            assert len(examples) > 1
            assert examples[0].label is not None
        logger.info("len examples: %s}", str(len(examples)))
        logger.info("Three choices: %s", str(three_choice))
        logger.info("Five choices: %s", str(five_choice))
        logger.info("Other choices: %s", str(other_choices))
        logger.info("four choices: %s", str(four_choice))



SUMMARY:
**Class Summary: ArcProcessor**

The `ArcProcessor` class processes data from the ARC dataset, a request from Allennlp. It inherits from `DataProcessor`.

**Primary Purpose:** To read and normalize data from JSON files and create example inputs for the training and development sets.

**Key Implementation Details:**

* `_read_json`: reads files in UTF-8 encoding and returns lines
* `_create_examples`: creates examples by reading JSON, normalizing labels, and extracting question choices
	+ Uses `normalize` function to normalize labels
	+ Extracts specific information from each line (question, choices)
	+ Creates `InputExample` objects with contexts, endings, and label
* `_read_json` is used in the following methods:
	+ `get_train_examples`
	+ `get_dev_examples`
	+ `get_test_examples`

**Notable Dependencies/Requirements:**

* `tqdm`
* `json`
* `InputExample` class (not shown in the provided code)

The class uses the `tqdm` library for progress bars and `json` to parse JSON files. It assumes that the `InputExample` class is defined elsewhere in the project.

**Interfaces Implemented:**

* None explicitly mentioned, but the class inherits from `DataProcessor`

Note that this summary focuses on the main functionality of the class and its key implementation details.


key: chroma:document
string_value:
CODE:

def convert_examples_to_features(
    examples: List[InputExample],
    label_list: List[str],
    max_length: int,
    tokenizer: PreTrainedTokenizer,
) -> List[InputFeatures]:
    """
    Loads a data file into a list of `InputFeatures`
    """

    label_map = {label: i for i, label in enumerate(label_list)}

    features = []
    for ex_index, example in tqdm.tqdm(enumerate(examples), desc="convert examples to features"):
        if ex_index % 10000 == 0:
            logger.info("Writing example %d of %d" % (ex_index, len(examples)))
        choices_inputs = []
        for ending_idx, (context, ending) in enumerate(zip(example.contexts, example.endings)):
            text_a = context
            if example.question.find("_") != -1:
                # this is for cloze question
                text_b = example.question.replace("_", ending)
            else:
                text_b = example.question + " " + ending

            inputs = tokenizer(
                text_a,
                text_b,
                add_special_tokens=True,
                max_length=max_length,
                padding="max_length",
                truncation=True,
                return_overflowing_tokens=True,
            )
            if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                logger.info(
                    "Attention! you are cropping tokens (swag task is ok). "
                    "If you are training ARC and RACE and you are poping question + options, "
                    "you need to try to use a bigger max seq length!"
                )

            choices_inputs.append(inputs)

        label = label_map[example.label]

        input_ids = [x["input_ids"] for x in choices_inputs]
        attention_mask = (
            [x["attention_mask"] for x in choices_inputs] if "attention_mask" in choices_inputs[0] else None
        )
        token_type_ids = (
            [x["token_type_ids"] for x in choices_inputs] if "token_type_ids" in choices_inputs[0] else None
        )

        features.append(
            InputFeatures(
                example_id=example.example_id,
                input_ids=input_ids,
                attention_mask=attention_mask,
                token_type_ids=token_type_ids,
                label=label,
            )
        )

    for f in features[:2]:
        logger.info("*** Example ***")
        logger.info("feature: %s" % f)



SUMMARY:
**Function Summary**

`convert_examples_to_features`

Loads data into a list of `InputFeatures` and preprocesses examples for model training.

**Main Purpose and Functionality**

* Takes in a list of `InputExample`, label list, maximum length, and pre-trained tokenizer as inputs
* Converts examples to features by tokenizing question-text pairs and creating input IDs, attention masks, and token type IDs
* Appends prepared features to a list for further processing

**Key Implementation Details**

* Uses a custom label mapping (`label_map`) to associate labels with indices
* Tokenizes questions and text pairs using the `tokenizer` with specified configuration options (max length, padding, truncation)
* Handles cases where tokens are truncated by logging a warning message

**Notable Dependencies or Requirements**

* `List[InputExample]`: input data structure for examples
* `PreTrainedTokenizer`: pre-trained tokenizer instance for tokenization
* External dependencies:
	+ `tqdm` for progress bar visualization (logging statements)
	+ Logging library for logging purposes


key: chroma:document
string_value:
CODE:

def parse_args():
    """
    Helper function parsing the command line options
    @retval ArgumentParser
    """
    parser = ArgumentParser(
        description=(
            "PyTorch TPU distributed training launch helper utility that will spawn up multiple distributed processes"
        )
    )

    # Optional arguments for the launch helper
    parser.add_argument("--num_cores", type=int, default=1, help="Number of TPU cores to use (1 or 8).")

    # positional
    parser.add_argument(
        "training_script",
        type=str,
        help=(
            "The full path to the single TPU training "
            "program/script to be launched in parallel, "
            "followed by all the arguments for the "
            "training script"
        ),
    )

    # rest from the training program
    parser.add_argument("training_script_args", nargs=REMAINDER)



SUMMARY:
**Summary:**

*   **Functionality:** `parse_args` is a helper function that parses command line options using the `ArgumentParser` class from the `argparse` module.
*   **Key Implementation Details:**
    *   Optionally specifies 1-8 TPU cores to use for distributed training (`--num_cores`) and validates input type as integers.
    *   Specifies a positional argument `training_script`, requiring a full path to a single TPU training program/script, followed by all arguments for the script.
*   **Notable Dependencies or Requirements:**
    *   Uses `ArgumentParser` from `argparse` module.

        **RAG System Representation:**


        |-  parse_args (method)
            |- description:
                |-  launch helper utility for PyTorch TPU distributed training
            |   |- num_cores:
                    |-  (1 or 8, integer type default)
            |- Args:
                |- --num_cores (optional):
                |- `training_script` (positional required path to training script with all arguments)
                |- `training_script_args` (remaining positional arguments)


key: chroma:document
string_value:
CODE:

def main():
    args = parse_args()

    # Import training_script as a module.
    script_fpath = Path(args.training_script)
    sys.path.append(str(script_fpath.parent.resolve()))
    mod_name = script_fpath.stem
    mod = importlib.import_module(mod_name)

    # Patch sys.argv
    sys.argv = [args.training_script] + args.training_script_args + ["--tpu_num_cores", str(args.num_cores)]



SUMMARY:
**Summary**

This `main` function serves as the entry point for a training script execution.

**Primary Purpose**: Executes a trained model using multiple TPU cores by importing and patching the corresponding Python module.

**Key Implementation Details**:

* Imports a training script module (`script_fpath`) and patches its arguments with additional TPU core count and other required parameters.
* Modifies system path to include the parent directory of the imported script for correct module resolution.
* Spawns the underlying multi-threading function `mod._mp_fn` from the patched script module, using up to `args.num_cores` worker processes.

**Notable Dependencies/Requirements**:

* Python 3.x (specifically, imports Python modules with `importlib`)
* PyTorch XLA (via `xmp.spawn`, `tpu_num_cores` argument)
* Argparse library for command-line arguments handling (`parse_args` function)


key: chroma:document
string_value:
CODE:

def _dump_articles(path: Path, articles: list):
    content = "\n".join(articles)


SUMMARY:
**Summary**

`_dump_articles` function writes a concatenated string of articles to a file at the specified path. It takes two inputs:

*   `path`: The file path where the content will be written (`Path() -> Path`)
*   `articles`: A list of article strings to be written to the file (`list[str]`)

The implementation uses the `Path` class and its methods to open, write, and close the file.

**Implementation Details**

*   Uses `join()` to concatenate article strings into a single string
*   Writes the concatenated content to a file at the specified path in write mode ("w") with the `writelines()` method

Note: This summary focuses on the main purpose, key implementation details, and notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

def make_test_data_dir(tmp_dir):
    for split in ["train", "val", "test"]:
        _dump_articles(os.path.join(tmp_dir, f"{split}.source"), ARTICLES)
        _dump_articles(os.path.join(tmp_dir, f"{split}.target"), SUMMARIES)


SUMMARY:
**Summary:**

`make_test_data_dir` function generates test dataset directories for training and evaluation splits by dumping articles and summaries into separate folders within a provided temporary directory.

**Key Details:**

* Takes a temporary directory (`tmp_dir`) as input
* Creates three subdirectories: `train.source`, `val.source`, and `test.source` for article dumps of respective categories
	+ Calls `_dump_articles` with `ARTICLES` on `train.source`
	+ Calls `_dump_articles` with `SUMMARIES` on `val.source`
	+ For `test.source`, no action is taken ( implicit )
* Also creates corresponding target directories (`...target`) for each subdirectory, but only applies `_dump_articles` to `SUMMARIES`, leaving the test split empty
* Returns the input `tmp_dir`

**Implementation Notes:**

* No external dependencies explicitly listed
* Uses predefined constants `ARTICLES` and `SUMMARIES`
* Design choice to split files into source/target folders for article summaries


key: chroma:document
string_value:
CODE:

class TestAll(TestCasePlus):
    @parameterized.expand(
        [
            MBART_TINY,
            MARIAN_TINY,
            T5_TINY,
            BART_TINY,
            PEGASUS_XSUM,
        ],
    )
    @slow
    def test_seq2seq_dataset_truncation(self, tok_name):
        tokenizer = AutoTokenizer.from_pretrained(tok_name)
        tmp_dir = make_test_data_dir(tmp_dir=self.get_auto_remove_tmp_dir())
        max_len_source = max(len(tokenizer.encode(a)) for a in ARTICLES)
        max_len_target = max(len(tokenizer.encode(a)) for a in SUMMARIES)
        max_src_len = 4
        max_tgt_len = 8
        assert max_len_target > max_src_len  # Will be truncated
        assert max_len_source > max_src_len  # Will be truncated
        src_lang, tgt_lang = "ro_RO", "de_DE"  # ignored for all but mbart, but never causes error.
        train_dataset = Seq2SeqDataset(
            tokenizer,
            data_dir=tmp_dir,
            type_path="train",
            max_source_length=max_src_len,
            max_target_length=max_tgt_len,  # ignored
            src_lang=src_lang,
            tgt_lang=tgt_lang,
        )
        dataloader = DataLoader(train_dataset, batch_size=2, collate_fn=train_dataset.collate_fn)
        for batch in dataloader:
            assert isinstance(batch, dict)
            assert batch["attention_mask"].shape == batch["input_ids"].shape
            # show that articles were trimmed.
            assert batch["input_ids"].shape[1] == max_src_len
            # show that targets are the same len
            assert batch["labels"].shape[1] == max_tgt_len
            if tok_name != MBART_TINY:
                continue
            # check language codes in correct place
            batch["decoder_input_ids"] = shift_tokens_right(batch["labels"], tokenizer.pad_token_id)
            assert batch["decoder_input_ids"][0, 0].item() == tokenizer.lang_code_to_id[tgt_lang]
            assert batch["decoder_input_ids"][0, -1].item() == tokenizer.eos_token_id
            assert batch["input_ids"][0, -2].item() == tokenizer.eos_token_id
            assert batch["input_ids"][0, -1].item() == tokenizer.lang_code_to_id[src_lang]

            break  # No need to test every batch

    @parameterized.expand([BART_TINY, BERT_BASE_CASED])
    def test_legacy_dataset_truncation(self, tok):
        tokenizer = AutoTokenizer.from_pretrained(tok)
        tmp_dir = make_test_data_dir(tmp_dir=self.get_auto_remove_tmp_dir())
        max_len_source = max(len(tokenizer.encode(a)) for a in ARTICLES)
        max_len_target = max(len(tokenizer.encode(a)) for a in SUMMARIES)
        trunc_target = 4
        train_dataset = LegacySeq2SeqDataset(
            tokenizer,
            data_dir=tmp_dir,
            type_path="train",
            max_source_length=20,
            max_target_length=trunc_target,
        )
        dataloader = DataLoader(train_dataset, batch_size=2, collate_fn=train_dataset.collate_fn)
        for batch in dataloader:
            assert batch["attention_mask"].shape == batch["input_ids"].shape
            # show that articles were trimmed.
            assert batch["input_ids"].shape[1] == max_len_source
            assert 20 >= batch["input_ids"].shape[1]  # trimmed significantly
            # show that targets were truncated
            assert batch["labels"].shape[1] == trunc_target  # Truncated
            assert max_len_target > trunc_target  # Truncated
            break  # No need to test every batch

    def test_pack_dataset(self):
        tokenizer = AutoTokenizer.from_pretrained("facebook/mbart-large-cc25")

        tmp_dir = Path(make_test_data_dir(tmp_dir=self.get_auto_remove_tmp_dir()))
        orig_examples = tmp_dir.joinpath("train.source").open().readlines()
        save_dir = Path(make_test_data_dir(tmp_dir=self.get_auto_remove_tmp_dir()))
        pack_data_dir(tokenizer, tmp_dir, 128, save_dir)
        orig_paths = {x.name for x in tmp_dir.iterdir()}
        new_paths = {x.name for x in save_dir.iterdir()}
        packed_examples = save_dir.joinpath("train.source").open().readlines()
        # orig: [' Sam ate lunch today.\n', 'Sams lunch ingredients.']
        # desired_packed: [' Sam ate lunch today.\n Sams lunch ingredients.']
        assert len(packed_examples) < len(orig_examples)
        assert len(packed_examples) == 1
        assert len(packed_examples[0]) == sum(len(x) for x in orig_examples)
        assert orig_paths == new_paths

    @pytest.mark.skipif(not FAIRSEQ_AVAILABLE, reason="This test requires fairseq")
    def test_dynamic_batch_size(self):
        if not FAIRSEQ_AVAILABLE:
            return
        ds, max_tokens, tokenizer = self._get_dataset(max_len=64)
        required_batch_size_multiple = 64
        batch_sampler = ds.make_dynamic_sampler(max_tokens, required_batch_size_multiple=required_batch_size_multiple)
        batch_sizes = [len(x) for x in batch_sampler]
        assert len(set(batch_sizes)) > 1  # it's not dynamic batch size if every batch is the same length
        assert sum(batch_sizes) == len(ds)  # no dropped or added examples
        data_loader = DataLoader(ds, batch_sampler=batch_sampler, collate_fn=ds.collate_fn, num_workers=2)
        failures = []
        num_src_per_batch = []
        for batch in data_loader:
            src_shape = batch["input_ids"].shape
            bs = src_shape[0]
            assert bs % required_batch_size_multiple == 0 or bs < required_batch_size_multiple
            num_src_tokens = np.product(batch["input_ids"].shape)
            num_src_per_batch.append(num_src_tokens)
            if num_src_tokens > (max_tokens * 1.1):
                failures.append(num_src_tokens)
        assert num_src_per_batch[0] == max(num_src_per_batch)
        if failures:
            raise AssertionError(f"too many tokens in {len(failures)} batches")

    def test_sortish_sampler_reduces_padding(self):
        ds, _, tokenizer = self._get_dataset(max_len=512)
        bs = 2
        sortish_sampler = ds.make_sortish_sampler(bs, shuffle=False)

        naive_dl = DataLoader(ds, batch_size=bs, collate_fn=ds.collate_fn, num_workers=2)
        sortish_dl = DataLoader(ds, batch_size=bs, collate_fn=ds.collate_fn, num_workers=2, sampler=sortish_sampler)

        pad = tokenizer.pad_token_id

        def count_pad_tokens(data_loader, k="input_ids"):
            return [batch[k].eq(pad).sum().item() for batch in data_loader]

        assert sum(count_pad_tokens(sortish_dl, k="labels")) < sum(count_pad_tokens(naive_dl, k="labels"))
        assert sum(count_pad_tokens(sortish_dl)) < sum(count_pad_tokens(naive_dl))
        assert len(sortish_dl) == len(naive_dl)

    def _get_dataset(self, n_obs=1000, max_len=128):
        if os.getenv("USE_REAL_DATA", False):
            data_dir = "examples/seq2seq/wmt_en_ro"
            max_tokens = max_len * 2 * 64
            if not Path(data_dir).joinpath("train.len").exists():
                save_len_file(MARIAN_TINY, data_dir)
        else:
            data_dir = "examples/seq2seq/test_data/wmt_en_ro"
            max_tokens = max_len * 4
            save_len_file(MARIAN_TINY, data_dir)

        tokenizer = AutoTokenizer.from_pretrained(MARIAN_TINY)
        ds = Seq2SeqDataset(
            tokenizer,
            data_dir=data_dir,
            type_path="train",
            max_source_length=max_len,
            max_target_length=max_len,
            n_obs=n_obs,
        )
        return ds, max_tokens, tokenizer

    def test_distributed_sortish_sampler_splits_indices_between_procs(self):
        ds, max_tokens, tokenizer = self._get_dataset()
        ids1 = set(DistributedSortishSampler(ds, 256, num_replicas=2, rank=0, add_extra_examples=False))
        ids2 = set(DistributedSortishSampler(ds, 256, num_replicas=2, rank=1, add_extra_examples=False))
        assert ids1.intersection(ids2) == set()

    @parameterized.expand(
        [
            MBART_TINY,
            MARIAN_TINY,
            T5_TINY,
            BART_TINY,
            PEGASUS_XSUM,
        ],
    )
    def test_dataset_kwargs(self, tok_name):
        tokenizer = AutoTokenizer.from_pretrained(tok_name, use_fast=False)
        if tok_name == MBART_TINY:
            train_dataset = Seq2SeqDataset(
                tokenizer,
                data_dir=make_test_data_dir(tmp_dir=self.get_auto_remove_tmp_dir()),
                type_path="train",
                max_source_length=4,
                max_target_length=8,
                src_lang="EN",
                tgt_lang="FR",
            )
            kwargs = train_dataset.dataset_kwargs
            assert "src_lang" in kwargs and "tgt_lang" in kwargs
        else:
            train_dataset = Seq2SeqDataset(
                tokenizer,
                data_dir=make_test_data_dir(tmp_dir=self.get_auto_remove_tmp_dir()),
                type_path="train",
                max_source_length=4,
                max_target_length=8,
            )
            kwargs = train_dataset.dataset_kwargs
            assert "add_prefix_space" not in kwargs if tok_name != BART_TINY else "add_prefix_space" in kwargs


SUMMARY:
This test suite appears to be written for a Python library used for sequence-to-sequence tasks, specifically for training machine translation models. The tests cover various aspects of the library's functionality, including:

1. **Dataset creation and loading**: Tests that the dataset can be created with different configurations and loaded correctly.
2. **Batching and shuffling**: Verifies that batches are generated correctly and that shuffling is done properly.
3. **Distributed training**: Ensures that distributed training works as expected, including splitting indices between processes.
4. **Sorting**: Tests the behavior of sorting indices during training to reduce padding.

The tests cover different library versions (e.g., `MarianTiny`, `BartTiny`) and confirm that the library's functionality is consistent across versions.

Some key insights:

* The library uses a custom dataset class (`Seq2SeqDataset`) to create datasets for machine translation tasks.
* The library supports batched learning, using techniques like sorting indices to reduce padding during training.
* Distributed training is supported, allowing the experiment to scale horizontally.

To write similar tests, follow these steps:

1. **Identify library-specific functionality**: Understand how the library creates datasets, handles batching and shuffling, and implements distributed training.
2. **Choose test scenarios**: Select specific test scenarios that cover various aspects of the library's behavior, such as loading datasets, batching and shuffling, or distributed training.
3. **Write test code**: Use a testing framework (e.g., Pytest) to write concise tests that verify expected library behavior.
4. **Test for consistency across versions**: Ensure that tests produce consistent results across different library versions.

Note: Without more information about the specific library being tested, this summary provides general advice on writing effective tests for similar libraries.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/legacy/seq2seq/old_test_seq2seq_examples_multi_gpu.py


key: chroma:document
string_value:
CODE:

class TestSummarizationDistillerMultiGPU(TestCasePlus):
    @classmethod
    def setUpClass(cls):
        return cls

    @slow
    @require_torch_gpu
    def test_distributed_eval(self):
        output_dir = self.get_auto_remove_tmp_dir()
        args = f"""
            --model_name Helsinki-NLP/opus-mt-en-ro
            --save_dir {output_dir}
            --data_dir {self.test_file_dir_str}/test_data/wmt_en_ro
            --num_beams 2
            --task translation
        """.split()

        # we want this test to run even if there is only one GPU, but if there are more we use them all
        n_gpu = get_gpu_count()
        distributed_args = f"""
            -m torch.distributed.launch
            --nproc_per_node={n_gpu}
            {self.test_file_dir}/run_distributed_eval.py
        """.split()
        cmd = [sys.executable] + distributed_args + args
        execute_subprocess_async(cmd, env=self.get_env())

        metrics_save_path = os.path.join(output_dir, "test_bleu.json")
        metrics = load_json(metrics_save_path)
        # print(metrics)


SUMMARY:
**Class Summary: `TestSummarizationDistillerMultiGPU`**

* **Purpose:** Evaluates the distributed translation model on a multigPU setup.

    Main Functionality:
    - Initializes test environment with auto-remove temporary directory and specified data path.
    - Runs distributed evaluation of language model in multigPU mode.
    - Verifies BLEU score greater than or equal to 25.


**Implementation Details:**

* Uses TorchDistributedLaunch for multiple GPU support.
* Iterates over available GPUs and uses all of them for distributed training.
* Calls `run_distributed_eval.py` script with specified command-line arguments.


**Notable Dependencies:**

* Torch
* PyTorch Distributed
* `get_gpu_count` function
* `sys.executable`
* `load_json` function
* `os.path` module.


key: chroma:document
string_value:
CODE:

def add_newline_to_end_of_each_sentence(x: str) -> str:
    """This was added to get rougeLsum scores matching published rougeL scores for BART and PEGASUS."""
    re.sub("<n>", "", x)  # remove pegasus newline char
    assert NLTK_AVAILABLE, "nltk must be installed to separate newlines between sentences. (pip install nltk)"


SUMMARY:
**Summary:**

The `add_newline_to_end_of_each_sentence` function takes a string input `x` and:
1. Removes the Pegasus newline character (`<n>`).
2. Ensures NLTK is available for sentence tokenization (fails if not installed) using an assertion.
3. Returns a new string with each sentence on a separate line, separated by newline characters.
 
**Implementation Details:**

- Uses regular expression substitution to remove `<n>`.
- Utilizes the `sent_tokenize` function from the NLTK library to split sentences.

**Notable Dependencies/Requirements:**

- `NLTK` ( Natural Language Toolkit ) library
- `re` module for regular expression substitution (part of Python's standard library)


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/legacy/seq2seq/save_randomly_initialized_model.py


key: chroma:document
string_value:
CODE:

def save_randomly_initialized_version(config_name: str, save_dir: str, **config_kwargs):
    """Save a randomly initialized version of a model using a pretrained config.
    Args:
        config_name: which config to use
        save_dir: where to save the resulting model and tokenizer
        config_kwargs: Passed to AutoConfig

    Usage::
        save_randomly_initialized_version("facebook/bart-large-cnn", "distilbart_random_cnn_6_3", encoder_layers=6, decoder_layers=3, num_beams=3)
    """
    cfg = AutoConfig.from_pretrained(config_name, **config_kwargs)
    model = AutoModelForSeq2SeqLM.from_config(cfg)
    model.save_pretrained(save_dir)
    AutoTokenizer.from_pretrained(config_name).save_pretrained(save_dir)


SUMMARY:
**Function Summary:**

`save_randomly_initialized_version`

* Saves a randomly initialized version of a sequence-to-sequence model using a predefined configuration.
* Parameters:
	+ `config_name`: Predefined config to use (e.g. "facebook/bart-large-cnn")
	+ `save_dir`: Directory to save the model and tokenizer
	+ `**config_kwargs`: Keyword arguments passed to `AutoConfig`
* Returns: The initialized model

**Implementation Details:**

* Uses `AutoConfig` and `AutoModelForSeq2SeqLM` from Hugging Face Transformers library.
* Saves `model` using `save_pretrained` method with the provided `save_dir`.
* Saves pre-trained tokenizer using `save_pretrained` method with the same `save_dir`.

**Dependencies:**

* Hugging Face Transformers library (`transformers`)
* Required config names (e.g. "facebook/bart-large-cnn")
* No external dependencies listed


key: chroma:document
string_value:
CODE:

def test_disaggregated_scores_are_determinstic():
    no_aggregation = calculate_rouge(PRED, TGT, bootstrap_aggregation=False, rouge_keys=["rouge2", "rougeL"])
    assert isinstance(no_aggregation, defaultdict)
    no_aggregation_just_r2 = calculate_rouge(PRED, TGT, bootstrap_aggregation=False, rouge_keys=["rouge2"])
    assert (
        pd.DataFrame(no_aggregation["rouge2"]).fmeasure.mean()
        == pd.DataFrame(no_aggregation_just_r2["rouge2"]).fmeasure.mean()


SUMMARY:
**Function Summary**

### Test: Disaggregated Scores are Deterministic

This test verifies that the aggregated scores (specifically, ROUGE-2 and ROUGE-L) from both the full aggregation calculation (`calculate_rouge(PRED, TGT)`) and a subset of the calculations (`no_aggregation = calculate_rouge(PRED, TGT, roug_keys=["rouge2", "roupeL"])`) are deterministic.

**Key Insights:**

* The `calculate_rouge` function returns a dictionary-like object represented as a `defaultdict`.
* The test checks if running the full calculation and then the partial calculation produces identical results for both ROUGE-2 and ROUGE-L scores.
* The agreement score is checked using `.fmeasure.mean()`, which calculates the f-score (average of precision and recall) for each score set.

**Key Functions/Classes:**

* `calculate_rouge`
* `PRED` and `TGT`
* `defaultdict`


key: chroma:document
string_value:
CODE:

def test_newline_cnn_improvement():
    k = "rougeLsum"
    score = calculate_rouge(PRED, TGT, newline_sep=True, rouge_keys=[k])[k]
    score_no_sep = calculate_rouge(PRED, TGT, newline_sep=False, rouge_keys=[k])[k]


SUMMARY:
**Summary: Evaluating Newline-aware CNN Improvement**

* **Purpose:** Compare the effect of using line breaks on the Rouge-L score for similarity between two text outputs (`PRED` and `TGT`) generated by models trained with and without newline separation.
* **Implementation Details:**
	+ Uses `calculate_rouge()` function with differing values for `newline_sep`.
	+ Compares scores obtained from both configurations, leveraging the Rouge-L metric (`k="rougeLsum"`).
* **Key Method Signature:** `calculate_rouge(PRED, TGT, newline_sep=True/False, rouge_keys=["rougeLsum"]);`
* **Notable Dependencies:**
	+ `calculate_rouge()` function and its parameters.
	+ `PRED` and `TGT` variables (not defined in this snippet).


key: chroma:document
string_value:
CODE:

def test_newline_irrelevant_for_other_metrics():
    k = ["rouge1", "rouge2", "rougeL"]
    score_sep = calculate_rouge(PRED, TGT, newline_sep=True, rouge_keys=k)
    score_no_sep = calculate_rouge(PRED, TGT, newline_sep=False, rouge_keys=k)


SUMMARY:
**Summary: Cross-Validation of Rouge Metrics**

This function tests the invariance of ROUGE scores to newlines.

**Key Details:**

* Calculates ROUGE scores for `PRED` and `TGT` using `calculate_rouge` with two variants:
	+ `newline_sep=True`
	+ `newline_sep=False`
* Verifies that both variants yield identical scores
* Uses the `rouge1`, `rouge2`, and `rougeL` scores calculated from a predefined list of ROUGE keys (`k=["rouge1", "rouge2", "rougeL"]`)
* Asserts equality between the two score calculations

**Dependencies:**

* `calculate_rouge` function ( imported but not explicitly referenced)
* The `PRED`, `TGT`, and `newline_sep=True/False` variables are assumed to be defined in the surrounding context.


key: chroma:document
string_value:
CODE:

def test_single_sent_scores_dont_depend_on_newline_sep():
    pred = [
        "Her older sister, Margot Frank, died in 1945, a month earlier than previously thought.",
        'Marseille prosecutor says "so far no videos were used in the crash investigation" despite media reports .',
    ]
    tgt = [
        "Margot Frank, died in 1945, a month earlier than previously thought.",
        'Prosecutor: "No videos were used in the crash investigation" German papers say they saw a cell phone video of'
        " the final seconds on board Flight 9525.",
    ]


SUMMARY:
**Function Summary:**

`test_single_sent_scores_dont_depend_on_newline_sep`

* Purpose: Verifies that ROUGE (Recall-Oriented Understudy for Gisting Evaluation) scores are independent of newline separator settings.
* Test Case:
	+ Input: Pair of predicted (`pred`) and target (`tgt`) sentence pairs.
	+ Expected Outcome: Same ROUGE scores obtained with different newline separators (`newline_sep=True` and `newline_sep=False`).
* Key Implementation Details:
	+ Utilizes `calculate_rouge` function from `...` (not shown in this snippet) for scoring computation.

**Notable Dependencies/Requirements:**

* `calculate_rouge` function, assumed to be defined elsewhere.


key: chroma:document
string_value:
CODE:

def test_pegasus_newline():
    pred = [
        """" "a person who has such a video needs to immediately give it to the investigators," prosecutor says .<n> "it is a very disturbing scene," editor-in-chief of bild online tells "erin burnett: outfront" """
    ]
    tgt = [
        """ Marseille prosecutor says "so far no videos were used in the crash investigation" despite media reports . Journalists at Bild and Paris Match are "very confident" the video clip is real, an editor says . Andreas Lubitz had informed his Lufthansa training school of an episode of severe depression, airline says ."""
    ]

    prev_score = calculate_rouge(pred, tgt, rouge_keys=["rougeLsum"], newline_sep=False)["rougeLsum"]
    new_score = calculate_rouge(pred, tgt, rouge_keys=["rougeLsum"])["rougeLsum"]


SUMMARY:
**Concise Summary**

This Python function `test_pegasus_newline` exercises the `calculate_rouge` function with the following parameters:

- **Pred** (Source): A list of text samples referencing a video investigation.
- **Tgt** (Target): A list of desired output, including a Marseille prosecutor's statement about not using videos in crash investigations.
- **Rouge Keys**: Calculating the ROUGE-L sum score with `rougeLsum` metric.

Key Implementation Details:

* Compares two scoring metrics (`prev_score` and `new_score`) before-and-after applying the `newline_sep=False` flag to examine the effect of newline separation on Rouge-L score calculation.
* Asserts that the new score is greater than the previous score, indicating an improvement when ignoring newline separation.

Notable Dependencies:

- `calculate_rouge` function ( external dependency)
- `rougeLsum` interface (from Rouge Lib)

Preserved Class/Function/Variable Names: None

Primary Purpose:
To test and validate the performance of the `calculate_rougue` function in ROUGE-L score calculations for text matching tasks, specifically when ignoring newline separation.


key: chroma:document
string_value:
CODE:

def test_rouge_cli():
    data_dir = Path("examples/seq2seq/test_data/wmt_en_ro")
    metrics = calculate_rouge_path(data_dir.joinpath("test.source"), data_dir.joinpath("test.target"))
    assert isinstance(metrics, dict)
    metrics_default_dict = calculate_rouge_path(
        data_dir.joinpath("test.source"), data_dir.joinpath("test.target"), bootstrap_aggregation=False
    )


SUMMARY:
**Code Summary**

This function, `test_rouge_cli`, tests the `calculate_rouge_path` function with two scenarios:
1. Standard calculation of ROUGE metrics between "test.source" and "test.target".
2. Calculation with default settings (including bootstrap aggregation) on top of an empty dictionary.

Key implementation details:

* Calculates ROUGE metrics on "text.file" pairs using `calculate_rouge_path`.
* Verifies returned result type (`dict` for standard calculation, `defaultdict` for default settings).

Notable dependencies and requirements:
- `Path` class from the `pathlib` library.
- The `calculate_rouge_path` function, which is not shown in this snippet.

Primary purpose: To verify the correctness of ROUGE metric calculations using a testing framework.


key: chroma:document
string_value:
CODE:

def generate_summaries_or_translations(
    examples: List[str],
    out_file: str,
    model_name: str,
    batch_size: int = 8,
    device: str = DEFAULT_DEVICE,
    fp16=False,
    task="summarization",
    prefix=None,
    **generate_kwargs,
) -> Dict:
    """Save model.generate results to <out_file>, and return how long it took."""
    fout = Path(out_file).open("w", encoding="utf-8")
    model_name = str(model_name)
    model = AutoModelForSeq2SeqLM.from_pretrained(model_name).to(device)
    if fp16:
        model = model.half()

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    logger.info(f"Inferred tokenizer type: {tokenizer.__class__}")  # if this is wrong, check config.model_type.

    start_time = time.time()
    # update config with task specific params
    use_task_specific_params(model, task)
    if prefix is None:
        prefix = prefix or getattr(model.config, "prefix", "") or ""
    for examples_chunk in tqdm(list(chunks(examples, batch_size))):
        examples_chunk = [prefix + text for text in examples_chunk]
        batch = tokenizer(examples_chunk, return_tensors="pt", truncation=True, padding="longest").to(device)
        summaries = model.generate(
            input_ids=batch.input_ids,
            attention_mask=batch.attention_mask,
            **generate_kwargs,
        )
        dec = tokenizer.batch_decode(summaries, skip_special_tokens=True, clean_up_tokenization_spaces=False)
        for hypothesis in dec:
            fout.write(hypothesis + "\n")
            fout.flush()
    fout.close()
    runtime = int(time.time() - start_time)  # seconds
    n_obs = len(examples)


SUMMARY:
**Function Summary: `generate_summaries_or_translations`**

**Purpose:** Generate summaries or translations using a pre-trained model and save the results to a file.


**Implementation Details:**

*   Uses `AutoModelForSeq2SeqLM` and `AutoTokenizer` from Hugging FaceTransformers.
*   Takes in:
    *   `examples`: input text data
    *   `out_file`: output file path
    *   `model_name`: model name from Hugging Face Hub
    *   `batch_size`: batch size for generating summaries (default: 8)
    *   `device`: device to run the model on (default: default device)
    *   `fp16`: use half-precision training (default: False)
    *   `task`: task type ("summarization" or "translation")
    *   `prefix`: optional prefix for output
*   Updates the model's configuration with task-specific parameters.
*   Uses `tqdm` for progress bars and `chunks` to handle large input data.
*   Saves generated summaries to the output file, flushed after each write.

**Notable Dependencies and Interfaces:**

*   Hugging FaceTransformers (`AutoModelForSeq2SeqLM`, `AutoTokenizer`)
*   PyTorch (` torch` )
*   `tqdm`
*   `chunks`

**Return Value:**
A dictionary containing the number of observations (`n_obs`), runtime, and seconds per sample.


key: chroma:document
string_value:
CODE:

def datetime_now():


SUMMARY:
**Function Summary**

`datetime_now`: Returns the current date and time in ISO 8601 format.

**Key Implementation Details**

* Uses `datetime.datetime.now()` to retrieve the current UTC datetime object
* Formats the object as a string using `strftime("%Y-%m-%d %H:%M:%S")`

**Notable Dependencies**

* `datetime` module (built-in Python library)
* `strftime` method

**Interface**

None, returns a plain text string.


key: chroma:document
string_value:
CODE:

def run_generate(verbose=True):
    """

    Takes input text, generates output, and then using reference calculates the BLEU scores.

    The results are saved to a file and returned to the caller, and printed out unless ``verbose=False`` is passed.

    Args:
        verbose (:obj:`bool`, `optional`, defaults to :obj:`True`): print results to stdout

    Returns:
        a tuple: ``(scores, params}``
        - ``scores``: a dict of scores data ``{'bleu': 39.6501, 'n_obs': 2000, 'runtime': 186, 'seconds_per_sample': 0.093}``
        - ``params``: a dict of custom params, e.g. ``{'num_beams': 5, 'length_penalty': 0.8}``
    """

    parser = argparse.ArgumentParser()
    parser.add_argument("model_name", type=str, help="like facebook/bart-large-cnn,google-t5/t5-base, etc.")
    parser.add_argument("input_path", type=str, help="like cnn_dm/test.source")
    parser.add_argument("save_path", type=str, help="where to save summaries")
    parser.add_argument("--reference_path", type=str, required=False, help="like cnn_dm/test.target")
    parser.add_argument("--score_path", type=str, required=False, default="metrics.json", help="where to save metrics")
    parser.add_argument("--device", type=str, required=False, default=DEFAULT_DEVICE, help="cuda, cuda:1, cpu etc.")
    parser.add_argument(
        "--prefix", type=str, required=False, default=None, help="will be added to the beginning of src examples"
    )
    parser.add_argument("--task", type=str, default="summarization", help="used for task_specific_params + metrics")
    parser.add_argument("--bs", type=int, default=8, required=False, help="batch size")
    parser.add_argument(
        "--n_obs", type=int, default=-1, required=False, help="How many observations. Defaults to all."
    )
    parser.add_argument("--fp16", action="store_true")
    parser.add_argument("--dump-args", action="store_true", help="print the custom hparams with the results")
    parser.add_argument(
        "--info",
        nargs="?",
        type=str,
        const=datetime_now(),
        help=(
            "use in conjunction w/ --dump-args to print with the results whatever other info you'd like, e.g."
            " lang=en-ru. If no value is passed, the current datetime string will be used."
        ),
    )
    # Unspecified args like --num_beams=2 --decoder_start_token_id=4 are passed to model.generate
    args, rest = parser.parse_known_args()
    parsed_args = parse_numeric_n_bool_cl_kwargs(rest)
    if parsed_args and verbose:
        print(f"parsed the following generate kwargs: {parsed_args}")
    examples = [" " + x.rstrip() if "t5" in args.model_name else x.rstrip() for x in open(args.input_path).readlines()]
    if args.n_obs > 0:
        examples = examples[: args.n_obs]
    Path(args.save_path).parent.mkdir(exist_ok=True)

    if args.reference_path is None and Path(args.score_path).exists():
        warnings.warn(f"score_path {args.score_path} will be overwritten unless you type ctrl-c.")

    if args.device == "cpu" and args.fp16:
        # this mix leads to RuntimeError: "threshold_cpu" not implemented for 'Half'
        raise ValueError("Can't mix --fp16 and --device cpu")

    runtime_metrics = generate_summaries_or_translations(
        examples,
        args.save_path,
        args.model_name,
        batch_size=args.bs,
        device=args.device,
        fp16=args.fp16,
        task=args.task,
        prefix=args.prefix,
        **parsed_args,
    )

    if args.reference_path is None:
        return {}

    # Compute scores
    score_fn = calculate_bleu if "translation" in args.task else calculate_rouge
    output_lns = [x.rstrip() for x in open(args.save_path).readlines()]
    reference_lns = [x.rstrip() for x in open(args.reference_path).readlines()][: len(output_lns)]
    scores: dict = score_fn(output_lns, reference_lns)
    scores.update(runtime_metrics)

    if args.dump_args:
        scores.update(parsed_args)
    if args.info:
        scores["info"] = args.info

    if verbose:
        print(scores)

    if args.score_path is not None:
        json.dump(scores, open(args.score_path, "w"))



SUMMARY:
**Function Summary: `run_generate`**

The `run_generate` function generates summaries for text input using a specified model and calculates BLEU scores. It takes input text, generates output, and then uses reference data to compute BLEU scores.

**Key Implementation Details:**

*   Uses the `generate_summaries_or_translations` function from another module.
*   Calculates BLEU scores using either `calculate_bleu` or `calculate_rouge` depending on the task type.
*   Saves results to a file and returns them as a dictionary containing score data and custom parameters.

**Notable Dependencies and Requirements:**

*   Requires the `argparse`, `datetime`, and `json` libraries.
*   Assumes the existence of other modules and functions used in `generate_summaries_or_translations`.
*   Uses device-specific settings for GPU or CPU acceleration.


key: chroma:document
string_value:
CODE:

def minify(src_dir: str, dest_dir: str, n: int):
    """Write first n lines of each file f in src_dir to dest_dir/f"""
    src_dir = Path(src_dir)
    dest_dir = Path(dest_dir)
    dest_dir.mkdir(exist_ok=True)
    for path in src_dir.iterdir():
        new = [x.rstrip() for x in list(path.open().readlines())][:n]
        dest_path = dest_dir.joinpath(path.name)
        print(dest_path)


SUMMARY:
**Function Summary: `minify`**

* **Purpose:** Copies the first `n` lines of each file in `src_dir` to corresponding files in `dest_dir`.
* **Key Implementation Details:**
	+ Writes first `n` lines to destination file using `readlines()` and `\n`.join().
	+ Uses `path.open()` with `w` mode to write to destination file.
* **Notable Dependencies:**
	+ `Path` from `pathlib` module for path manipulation.
	+ No external dependencies. 

**Code Analysis Notes:**

* The function takes three parameters: `src_dir`, `dest_dir`, and `n`.
* It uses `iterdir()` to traverse the directory tree in `src_dir`, and writes to files in `dest_dir` using `joinpath()`.
* The `exist_ok=True` argument is used when creating the destination directory, which means that the function will not raise an error if the directory already exists.


key: chroma:document
string_value:
CODE:

def parse_search_arg(search):
    groups = search.split()
    entries = dict((g.split("=") for g in groups))
    entry_names = list(entries.keys())
    sets = [[f"--{k} {v}" for v in vs.split(":")] for k, vs in entries.items()]
    matrix = [list(x) for x in itertools.product(*sets)]


SUMMARY:
**Summary**

The `parse_search_arg` function takes a `search` string as input and parses it into a search query. It:

1. Splits the input string into groups separated by spaces (`groups = search.split()`).
2. Creates a dictionary mapping group pairs (e.g., `"keyword=value"`) to values.
3. Generates a list of key names from the dictionary's keys.
4. Iterates over each key-value pair and creates sets of attribute-value pairs (`sets = [...]`).
5. Uses `itertools.product` to generate all possible permutations of these sets, forming a matrix structure.
6. Returns the generated matrix and a list of entry names.

**Implementation Details**

* Algorithm: [Cartesian Product](https://en.wikipedia.org/wiki/Cartesian_product)
* Design Choice: Using tuples (e.g., `f"--{k} {v}"`) to represent attribute-value pairs

**Dependencies**

* `itertools` module for generating Cartesian product
* No external dependencies explicitly listed, but assumes standard Python library imports.

**Interfaces/Inherited Interfaces**

* None specified, but inherits from the general case of iterating over dictionaries and lists.


key: chroma:document
string_value:
CODE:

def run_search():
    """
     Run parametric search over the desired hparam space with help of ``run_eval.py``.

     All the arguments except ``--search`` are passed to ``run_eval.py`` as is. The values inside of "--search" are parsed, reformatted and fed to ``run_eval.py`` as additional args.

    The format for the ``--search`` value is a simple string with hparams and colon separated values to try, e.g.:
    ```
     --search "num_beams=5:10 length_penalty=0.8:1.0:1.2 early_stopping=true:false"
    ```
    which will generate ``12`` ``(2*3*2)`` searches for a product of each hparam. For example the example that was just used will invoke ``run_eval.py`` repeatedly with:

    ```
     --num_beams 5 --length_penalty 0.8 --early_stopping true
     --num_beams 5 --length_penalty 0.8 --early_stopping false
     [...]
     --num_beams 10 --length_penalty 1.2 --early_stopping false
    ```

    On completion, this function prints a markdown table of the results sorted by the best BLEU score and the winning arguments.


    """
    prog = sys.argv[0]

    parser = argparse.ArgumentParser(
        usage=(
            "\n\nImportant: this script accepts all arguments `run_eval.py` accepts and then a few extra, therefore"
            " refer to `run_eval.py -h` for the complete list."
        )
    )
    parser.add_argument(
        "--search",
        type=str,
        required=False,
        help='param space to search, e.g. "num_beams=5:10 length_penalty=0.8:1.0:1.2"',
    )
    parser.add_argument(
        "--bs", type=int, default=8, required=False, help="initial batch size (may get reduced if it's too big)"
    )
    parser.add_argument("--task", type=str, help="used for task_specific_params + metrics")
    parser.add_argument(
        "--info",
        nargs="?",
        type=str,
        const=datetime_now(),
        help=(
            "add custom notes to be printed before the results table. If no value is passed, the current datetime"
            " string will be used."
        ),
    )
    args, args_main = parser.parse_known_args()
    # we share some of the args
    args_main.extend(["--task", args.task])
    args_normal = [prog] + args_main

    # to support variations like translation_en_to_de"
    task = "translation" if "translation" in args.task else "summarization"

    matrix, col_names = parse_search_arg(args.search)
    col_names[0:0] = task_score_names[task]  # score cols first
    col_widths = {col: len(str(col)) for col in col_names}
    results = []
    for r in matrix:
        hparams = dict((x.replace("--", "").split() for x in r))
        args_exp = " ".join(r).split()
        args_exp.extend(["--bs", str(args.bs)])  # in case we need to reduce its size due to CUDA OOM
        sys.argv = args_normal + args_exp

        # XXX: need to trap CUDA OOM and lower args.bs if that happens and retry

        scores = run_generate(verbose=False)
        # make sure scores are first in the table
        result = OrderedDict()
        for score in task_score_names[task]:
            result[score] = scores[score]
        result.update(hparams)
        results.append(result)

        # find widest entries
        for k, v in result.items():
            l = len(str(v))
            if l > col_widths[k]:
                col_widths[k] = l

    results_sorted = sorted(results, key=operator.itemgetter(*task_score_names[task]), reverse=True)
    print(" | ".join([f"{col:{col_widths[col]}}" for col in col_names]))
    print(" | ".join([f"{'-'*col_widths[col]}" for col in col_names]))
    for row in results_sorted:
        print(" | ".join([f"{row[col]:{col_widths[col]}}" for col in col_names]))

    best = results_sorted[0]
    for score in task_score_names[task]:
        del best[score]
    best_args = [f"--{k} {v}" for k, v in best.items()]
    dyn_args = ["--bs", str(args.bs)]
    if args.info:
        print(f"\nInfo: {args.info}")
    print("\nBest score args:")
    print(" ".join(args_main + best_args + dyn_args))



SUMMARY:
**Function Summary**

`run_search`

Runs parametric search over the desired hparam space using `run_eval.py`. Accepts additional arguments, parses and reformats them for use with `run_eval.py`.

**Purpose**
Parametrized search tool to find optimal hyperparameters.

**Key Implementation Details**

* Parses and reformats `--search` argument string to generate multiple sets of command-line arguments.
* Calls `run_generate` function from `run_eval.py` with shared system arguments to evaluate each set of hparams.
* Sorts results by best BLEU score, prints a markdown table of the results.

**Important Method Signatures and Interfaces**

* `parse_search_arg`
* `run_generate`

**Notable Dependencies or Requirements**

* `run_eval.py` (imported)
* `argparse` (used for parsing arguments)
* `datetime_now` (used for current timestamp)


key: chroma:document
string_value:
CODE:

class Seq2SeqTrainer(Trainer):
    def __init__(self, config=None, data_args=None, *args, **kwargs):
        super().__init__(*args, **kwargs)

        if config is None:
            assert isinstance(self.model, PreTrainedModel), (
                "If no `config` is passed the model to be trained has to be of type `PreTrainedModel`, but is"
                f" {self.model.__class__}"
            )
            self.config = self.model.config
        else:
            self.config = config

        self.data_args = data_args
        self.vocab_size = self.config.tgt_vocab_size if isinstance(self.config, FSMTConfig) else self.config.vocab_size

        if self.args.label_smoothing != 0 or (self.data_args is not None and self.data_args.ignore_pad_token_for_loss):
            assert self.config.pad_token_id is not None, (
                "Make sure that `config.pad_token_id` is correctly defined when ignoring `pad_token` for loss"
                " calculation or doing label smoothing."
            )

        if self.config.pad_token_id is None and self.config.eos_token_id is not None:
            logger.warning(
                f"The `config.pad_token_id` is `None`. Using `config.eos_token_id` = {self.config.eos_token_id} for"
                " padding.."
            )

        if self.args.label_smoothing == 0:
            self.loss_fn = torch.nn.CrossEntropyLoss(ignore_index=self.config.pad_token_id)
        else:
            # dynamically import label_smoothed_nll_loss
            from utils import label_smoothed_nll_loss

            self.loss_fn = label_smoothed_nll_loss

    def create_optimizer_and_scheduler(self, num_training_steps: int):
        """
        Setup the optimizer and the learning rate scheduler.

        We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the
        Trainer's init through :obj:`optimizers`, or subclass and override this method in a subclass.
        """
        if self.optimizer is None:
            no_decay = ["bias", "LayerNorm.weight"]
            optimizer_grouped_parameters = [
                {
                    "params": [p for n, p in self.model.named_parameters() if not any(nd in n for nd in no_decay)],
                    "weight_decay": self.args.weight_decay,
                },
                {
                    "params": [p for n, p in self.model.named_parameters() if any(nd in n for nd in no_decay)],
                    "weight_decay": 0.0,
                },
            ]
            optimizer_cls = Adafactor if self.args.adafactor else AdamW
            if self.args.adafactor:
                optimizer_cls = Adafactor
                optimizer_kwargs = {"scale_parameter": False, "relative_step": False}
            else:
                optimizer_cls = AdamW
                optimizer_kwargs = {
                    "betas": (self.args.adam_beta1, self.args.adam_beta2),
                    "eps": self.args.adam_epsilon,
                }
            optimizer_kwargs["lr"] = self.args.learning_rate
            self.optimizer = optimizer_cls(optimizer_grouped_parameters, **optimizer_kwargs)

        if self.lr_scheduler is None:
            self.lr_scheduler = self._get_lr_scheduler(num_training_steps)
        else:  # ignoring --lr_scheduler
            logger.warning("scheduler is passed to `Seq2SeqTrainer`, `--lr_scheduler` arg is ignored.")

    def _get_lr_scheduler(self, num_training_steps):
        schedule_func = arg_to_scheduler[self.args.lr_scheduler]
        if self.args.lr_scheduler == "constant":
            scheduler = schedule_func(self.optimizer)
        elif self.args.lr_scheduler == "constant_w_warmup":
            scheduler = schedule_func(self.optimizer, num_warmup_steps=self.args.warmup_steps)
        else:
            scheduler = schedule_func(
                self.optimizer, num_warmup_steps=self.args.warmup_steps, num_training_steps=num_training_steps
            )
        return scheduler

    def _get_train_sampler(self) -> Optional[torch.utils.data.Sampler]:
        if isinstance(self.train_dataset, torch.utils.data.IterableDataset):
            return None
        elif is_torch_xla_available():
            return get_tpu_sampler(self.train_dataset)
        else:
            if self.args.sortish_sampler:
                self.train_dataset.make_sortish_sampler(
                    self.args.per_device_train_batch_size,
                    distributed=(self.args.parallel_mode == ParallelMode.DISTRIBUTED),
                )

            return (
                RandomSampler(self.train_dataset)
                if self.args.local_rank == -1
                else DistributedSampler(self.train_dataset)
            )

    def _compute_loss(self, model, inputs, labels):
        if self.args.label_smoothing == 0:
            if self.data_args is not None and self.data_args.ignore_pad_token_for_loss:
                # force training to ignore pad token
                logits = model(**inputs, use_cache=False)[0]
                loss = self.loss_fn(logits.view(-1, logits.shape[-1]), labels.view(-1))
            else:
                # compute usual loss via models
                loss, logits = model(**inputs, labels=labels, use_cache=False)[:2]
        else:
            # compute label smoothed loss
            logits = model(**inputs, use_cache=False)[0]
            lprobs = torch.nn.functional.log_softmax(logits, dim=-1)
            loss, _ = self.loss_fn(lprobs, labels, self.args.label_smoothing, ignore_index=self.config.pad_token_id)
        return loss, logits

    def compute_loss(self, model, inputs):
        labels = inputs.pop("labels")
        loss, _ = self._compute_loss(model, inputs, labels)
        return loss

    def prediction_step(
        self,
        model: nn.Module,
        inputs: Dict[str, Union[torch.Tensor, Any]],
        prediction_loss_only: bool,
        ignore_keys: Optional[List[str]] = None,
    ) -> Tuple[Optional[float], Optional[torch.Tensor], Optional[torch.Tensor]]:
        """
        Perform an evaluation step on :obj:`model` using obj:`inputs`.

        Subclass and override to inject custom behavior.

        Args:
            model (:obj:`nn.Module`):
                The model to evaluate.
            inputs (:obj:`Dict[str, Union[torch.Tensor, Any]]`):
                The inputs and targets of the model.

                The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
                argument :obj:`labels`. Check your model's documentation for all accepted arguments.
            prediction_loss_only (:obj:`bool`):
                Whether or not to return the loss only.

        Return:
            Tuple[Optional[float], Optional[torch.Tensor], Optional[torch.Tensor]]:
            A tuple with the loss, logits and labels (each being optional).
        """
        inputs = self._prepare_inputs(inputs)

        gen_kwargs = {
            "max_length": self.data_args.val_max_target_length
            if self.data_args is not None
            else self.config.max_length,
            "num_beams": self.data_args.eval_beams if self.data_args is not None else self.config.num_beams,
        }

        if self.args.predict_with_generate and not self.args.prediction_loss_only:
            generated_tokens = self.model.generate(
                inputs["input_ids"],
                attention_mask=inputs["attention_mask"],
                **gen_kwargs,
            )
            # in case the batch is shorter than max length, the output should be padded
            if generated_tokens.shape[-1] < gen_kwargs["max_length"]:
                generated_tokens = self._pad_tensors_to_max_len(generated_tokens, gen_kwargs["max_length"])

        labels = inputs.pop("labels")
        with torch.no_grad():
            # compute loss on predict data
            loss, logits = self._compute_loss(model, inputs, labels)

        loss = loss.mean().detach()
        if self.args.prediction_loss_only:
            return (loss, None, None)

        logits = generated_tokens if self.args.predict_with_generate else logits

        if labels.shape[-1] < gen_kwargs["max_length"]:
            labels = self._pad_tensors_to_max_len(labels, gen_kwargs["max_length"])

        return (loss, logits, labels)

    def _pad_tensors_to_max_len(self, tensor, max_length):
        # If PAD token is not defined at least EOS token has to be defined
        pad_token_id = self.config.pad_token_id if self.config.pad_token_id is not None else self.config.eos_token_id

        if pad_token_id is None:
            raise ValueError(
                "Make sure that either `config.pad_token_id` or `config.eos_token_id` is defined if tensor has to be"
                f" padded to `max_length`={max_length}"
            )

        padded_tensor = pad_token_id * torch.ones(
            (tensor.shape[0], max_length), dtype=tensor.dtype, device=tensor.device
        )
        padded_tensor[:, : tensor.shape[-1]] = tensor


SUMMARY:
This is a class `Seq2SeqTrainer` that inherits from `Trainer`. It's designed to train sequences-to-sequences models, typically used in natural language processing tasks like machine translation. The class has several methods for setting up the training process, computing losses and prediction steps.

Key features:

* Initialization: sets up the model, config, data arguments, and optimizers/schedulers.
* `create_optimizer_and_scheduler`: sets up the optimizer and scheduler based on the training configuration.
* `_compute_loss`: computes the loss during training or evaluation. If label smoothing is enabled, it uses a smoothed loss function.
* `prediction_step`: performs an evaluation step using the model, inputs, and optional labels.
* Data preparation: prepends data with a pad token to ensure compatibility.

The class aims to provide a flexible framework for training sequences-to-sequences models, allowing users to customize various parameters like optimizer, scheduler, label smoothing, and prediction settings.


key: chroma:document
string_value:
CODE:

def label_smoothed_nll_loss(lprobs, target, epsilon, ignore_index=-100):
    """From fairseq"""
    if target.dim() == lprobs.dim() - 1:
        target = target.unsqueeze(-1)
    nll_loss = -lprobs.gather(dim=-1, index=target)
    smooth_loss = -lprobs.sum(dim=-1, keepdim=True)
    if ignore_index is not None:
        pad_mask = target.eq(ignore_index)
        nll_loss.masked_fill_(pad_mask, 0.0)
        smooth_loss.masked_fill_(pad_mask, 0.0)
    else:
        nll_loss = nll_loss.squeeze(-1)
        smooth_loss = smooth_loss.squeeze(-1)

    nll_loss = nll_loss.sum()  # mean()? Scared to break other math.
    smooth_loss = smooth_loss.sum()
    eps_i = epsilon / lprobs.size(-1)
    loss = (1.0 - epsilon) * nll_loss + eps_i * smooth_loss


SUMMARY:
**Concise Summary:**

The `label_smoothed_nll_loss` function calculates the weighted Nash loss between smoothed negative log-probability (NLP) losses and raw NLP losses. It applies epsilon-smoothing to regularize the model's output.

**Key Functionality:**

* Calculates raw NLL loss (`nll_loss`) by gathering probabilities along the target indices.
* Calculates smoothed NLL loss (`smooth_loss`) as a sum of all probabilities.
* Averages the two losses and applies epsilon-smoothing to smooth the loss.

**Implementation Details:**

* Employs PyTorch's `gather` and `sum` operations for efficient computation.
* Utilizes integer indexing for masking, allowing for dynamic handling of ignore indexes (`ignore_index = -100`).

**Notable Dependencies/Requirements:**

* PyTorch library and tensor operations (e.g., `lprobs`, `target`).
* No external dependencies listed.


key: chroma:document
string_value:
CODE:

def lmap(f: Callable, x: Iterable) -> List:
    """list(map(f, x))"""


SUMMARY:
**Function Summary**

The `lmap` function is an alias for the built-in Python function `map()`. It takes a callable `f` and an iterable `x`, applies `f` to each element of `x`, and returns the result as a list.

**Implementation Details**

* Applies function `f` to each element of iterable `x`
* Returns the result as a list
* Equivalent to built-in `list(map(f, x))`

**Notable Dependencies/Requirements**

* Callable type hint: `f: Callable`
* Iterable type hint: `x: Iterable`
* List data structure (imported implicitly by Python)


key: chroma:document
string_value:
CODE:

def calculate_bleu(output_lns, refs_lns, **kwargs) -> dict:
    """Uses sacrebleu's corpus_bleu implementation."""


SUMMARY:
**Function Summary**

`calculate_bleu` calculates the BLEU score between two sets of output lines (`output_lns`) and reference lines (`refs_lns`). It utilizes the `sacrebleu` library to compute the score.

**Key Implementation Details**

* Takes in `output_lns`, `refs_lns`, and additional keyword arguments (`kwargs`) as input.
* Calls `corpus_bleu` from `sacrebleu` with both sets of lines, along with `kwargs`.
* Returns a dictionary containing the BLEU score rounded to 4 decimal places.

**Notable Dependencies**

* `sacrebleu`: The library used for calculating BLEU scores.
* `score`: The calculated BLEU score.


key: chroma:document
string_value:
CODE:

def build_compute_metrics_fn(task_name: str, tokenizer: PreTrainedTokenizer) -> Callable[[EvalPrediction], Dict]:
    def non_pad_len(tokens: np.ndarray) -> int:
        return np.count_nonzero(tokens != tokenizer.pad_token_id)

    def decode_pred(pred: EvalPrediction) -> Tuple[List[str], List[str]]:
        pred_ids = pred.predictions
        label_ids = pred.label_ids
        pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
        label_ids[label_ids == -100] = tokenizer.pad_token_id
        label_str = tokenizer.batch_decode(label_ids, skip_special_tokens=True)
        pred_str = lmap(str.strip, pred_str)
        label_str = lmap(str.strip, label_str)
        return pred_str, label_str

    def summarization_metrics(pred: EvalPrediction) -> Dict:
        pred_str, label_str = decode_pred(pred)
        rouge: Dict = calculate_rouge(pred_str, label_str)
        summ_len = np.round(np.mean(lmap(non_pad_len, pred.predictions)), 1)
        rouge.update({"gen_len": summ_len})
        return rouge

    def translation_metrics(pred: EvalPrediction) -> Dict:
        pred_str, label_str = decode_pred(pred)
        bleu: Dict = calculate_bleu(pred_str, label_str)
        gen_len = np.round(np.mean(lmap(non_pad_len, pred.predictions)), 1)
        bleu.update({"gen_len": gen_len})
        return bleu

    compute_metrics_fn = summarization_metrics if "summarization" in task_name else translation_metrics


SUMMARY:
**Function Summary: `build_compute_metrics_fn`**

**Purpose and Functionality:**
The `build_compute_metrics_fn` function constructs a comprehension function (`compute_metrics_fn`) for calculating metrics on specific types of text generation tasks (summarization or translation). It takes a task name and a pre-trained tokenizer as input.

**Key Implementation Details:**

* The function uses several supporting functions:
	+ `non_pad_len`: calculates the length of non-padded tokens in a sequence.
	+ `decode_pred`: decodes prediction IDs to strings, skipping special tokens and padding characters.
	+ `summarization_metrics` and `translation_metrics`: calculate ROUGE (text similarity) and BLEU (quality metric) scores for summarization and translation tasks, respectively.

**Notable Dependencies and Requirements:**

* `PreTrainedTokenizer`: a pre-trained tokenizer for text processing.
* `EvalPrediction`: a data type representing evaluation predictions.
* External dependencies:
	+ `calculate_rouge`: calculates ROUGE (text similarity) score.
	+ `calculate_bleu`: calculates BLEU (quality metric) score.

**Return Value:**

The constructed comprehension function (`compute_metrics_fn`) is returned, which takes an `EvalPrediction` as input and returns a dictionary of metrics.


key: chroma:document
string_value:
CODE:

def trim_batch(
    input_ids,
    pad_token_id,
    attention_mask=None,
):
    """Remove columns that are populated exclusively by pad_token_id"""
    keep_column_mask = input_ids.ne(pad_token_id).any(dim=0)
    if attention_mask is None:
        return input_ids[:, keep_column_mask]
    else:


SUMMARY:
**Summary**

The `trim_batch` function trims columns from an input tensor (`input_ids`) that are exclusively populated with the padding token ID (`pad_token_id`). It returns two tensors: one for `input_ids` and optionally, a corresponding `attention_mask`. The primary purpose is to exclude sparse padding from batch processing.

**Key Details**

* **Primary Functionality**: Trimming columns exclusively filled with pad_token_id
* **Implementation Choice**: Using `.ne()` and `.any()` on input dimensions 0

**Notable Dependencies/Requirements**

* `input_ids` (tensor)
* `pad_token_id` (int)
* `attention_mask` (optional tensor)


key: chroma:document
string_value:
CODE:

class AbstractSeq2SeqDataset(Dataset):
    def __init__(
        self,
        tokenizer,
        data_dir,
        max_source_length,
        max_target_length,
        type_path="train",
        n_obs=None,
        prefix="",
        **dataset_kwargs,
    ):
        super().__init__()
        self.src_file = Path(data_dir).joinpath(type_path + ".source")
        self.tgt_file = Path(data_dir).joinpath(type_path + ".target")
        self.len_file = Path(data_dir).joinpath(type_path + ".len")
        if os.path.exists(self.len_file):
            self.src_lens = pickle_load(self.len_file)
            self.used_char_len = False
        else:
            self.src_lens = self.get_char_lens(self.src_file)
            self.used_char_len = True
        self.max_source_length = max_source_length
        self.max_target_length = max_target_length
        assert min(self.src_lens) > 0, f"found empty line in {self.src_file}"
        self.tokenizer = tokenizer
        self.prefix = prefix if prefix is not None else ""

        if n_obs is not None:
            self.src_lens = self.src_lens[:n_obs]
        self.pad_token_id = self.tokenizer.pad_token_id
        self.dataset_kwargs = dataset_kwargs
        dataset_kwargs.update({"add_prefix_space": True} if isinstance(self.tokenizer, BartTokenizer) else {})

    def __len__(self):
        return len(self.src_lens)

    @staticmethod
    def get_char_lens(data_file):
        return [len(x) for x in Path(data_file).open().readlines()]

    @cached_property
    def tgt_lens(self):
        """Length in characters of target documents"""
        return self.get_char_lens(self.tgt_file)

    def make_sortish_sampler(self, batch_size, distributed=False, shuffle=True, **kwargs):
        if distributed:
            return DistributedSortishSampler(self, batch_size, shuffle=shuffle, **kwargs)
        else:
            return SortishSampler(self.src_lens, batch_size, shuffle=shuffle)

    def make_dynamic_sampler(self, max_tokens_per_batch=1024, **kwargs):
        assert FAIRSEQ_AVAILABLE, "Dynamic batch size requires `pip install fairseq`"
        assert not self.used_char_len, "You must call  python make_len_file.py before calling make_dynamic_sampler"
        sorted_indices = list(self.make_sortish_sampler(1024, shuffle=False))

        def num_tokens_in_example(i):
            return min(self.src_lens[i], self.max_target_length)

        # call fairseq cython function
        batch_sampler: List[List[int]] = batch_by_size(
            sorted_indices,
            num_tokens_fn=num_tokens_in_example,
            max_tokens=max_tokens_per_batch,
            required_batch_size_multiple=64,
        )
        shuffled_batches = [batch_sampler[i] for i in np.random.permutation(range(len(batch_sampler)))]
        # move the largest batch to the front to OOM quickly (uses an approximation for padding)
        approximate_toks_per_batch = [max(self.src_lens[i] for i in batch) * len(batch) for batch in shuffled_batches]
        largest_batch_idx = np.argmax(approximate_toks_per_batch)
        shuffled_batches[0], shuffled_batches[largest_batch_idx] = (
            shuffled_batches[largest_batch_idx],
            shuffled_batches[0],
        )
        return shuffled_batches

    def __getitem__(self, item):
        raise NotImplementedError("You must implement this")

    def collate_fn(self, batch):


SUMMARY:
**Class Summary: AbstractSeq2SeqDataset**

The `AbstractSeq2SeqDataset` class is a base class for datasets used in sequence-to-sequence (seq2seq) training, designed for transformer-based models. It inherits from PyTorch's `Dataset` class.

**Main Purpose:**
To provide a structured interface for storing and retrieving seq2seq data, handling character length information and sampling strategies.

**Key Implementation Details:**

1. **Initialization**: The class initializes with essential parameters such as tokenizer, data directory, maximum sequence lengths, type path, and optional dataset keyword arguments.
2. **Character Lengths**: It loads or calculates the character lengths of source sequences from a file, determining if prefix characters should be used.
3. **Data Sampling**: It provides methods for creating sortish and dynamic batch samplers based on source sequence lengths.

**Notable Dependencies/Requirements:**

1. PyTorch (`Dataset` class)
2. Fairseq (for dynamic batch sizing in `make_dynamic_sampler`)
3. Pickle library (for loading saved character lengths)

**Additional Notes:**

* The `__getitem__` and `collate_fn` methods are intended to be overridden by child classes.
* The `make_sortish_sampler`, `make_dynamic_sampler` functions leverage various batching strategies for efficient training data retrieval.


key: chroma:document
string_value:
CODE:

class LegacySeq2SeqDataset(AbstractSeq2SeqDataset):
    def __getitem__(self, index) -> Dict[str, torch.Tensor]:
        """Call tokenizer on src and tgt_lines"""
        index = index + 1  # linecache starts at 1
        source_line = self.prefix + linecache.getline(str(self.src_file), index).rstrip("\n")
        tgt_line = linecache.getline(str(self.tgt_file), index).rstrip("\n")
        assert source_line, f"empty source line for index {index}"
        assert tgt_line, f"empty tgt line for index {index}"
        source_inputs = self.encode_line(self.tokenizer, source_line, self.max_source_length)
        target_inputs = self.encode_line(self.tokenizer, tgt_line, self.max_target_length)

        source_ids = source_inputs["input_ids"].squeeze()
        target_ids = target_inputs["input_ids"].squeeze()
        src_mask = source_inputs["attention_mask"].squeeze()
        return {
            "input_ids": source_ids,
            "attention_mask": src_mask,
            "labels": target_ids,
        }

    def encode_line(self, tokenizer, line, max_length, pad_to_max_length=True, return_tensors="pt"):
        """Only used by LegacyDataset"""
        return tokenizer(
            [line],
            max_length=max_length,
            padding="max_length" if pad_to_max_length else None,
            truncation=True,
            return_tensors=return_tensors,
            **self.dataset_kwargs,
        )

    def collate_fn(self, batch) -> Dict[str, torch.Tensor]:
        input_ids = torch.stack([x["input_ids"] for x in batch])
        masks = torch.stack([x["attention_mask"] for x in batch])
        target_ids = torch.stack([x["labels"] for x in batch])
        pad_token_id = self.pad_token_id
        y = trim_batch(target_ids, pad_token_id)
        source_ids, source_mask = trim_batch(input_ids, pad_token_id, attention_mask=masks)
        batch = {
            "input_ids": source_ids,
            "attention_mask": source_mask,
            "labels": y,
        }


SUMMARY:
**Summary**

The `LegacySeq2SeqDataset` class is an implementation of the `AbstractSeq2SeqDataset` interface, designed to load and preprocess sequence-to-sequence data.

**Main Purpose and Functionality**

It provides a way to retrieve and encode source and target lines from files using a tokenizer.

**Key Implementation Details**

* The `__getitem__` method:
	+ Loads source and target lines from files using `linecache`.
	+ Encodes source and target lines into input IDs, attention masks, and labels.
* The `encode_line` method:
	+ Takes in a line, maximum length, and padding settings.
	+ Calls the tokenizer to encode the line into input IDs, attention masks, and labels.
* The `collate_fn` method:
	+ Combines batched data into a single dictionary.
	+ Trims padded sequences at the end.

**Notable Dependencies or Requirements**

* `tokenizer`: The dataset uses a tokenizer from an external library (implied by the `return_tensors="pt"` argument in `encode_line`).
* `linecache`: A built-in Python module for loading lines from files.
* `torch`: The PyTorch library is required for tensor handling and stack operations.

**Interfaces**

* `AbstractSeq2SeqDataset`
* `tokenizer` (external)


key: chroma:document
string_value:
CODE:

class Seq2SeqDataset(AbstractSeq2SeqDataset):
    """A dataset that calls prepare_seq2seq_batch."""

    def __getitem__(self, index) -> Dict[str, str]:
        index = index + 1  # linecache starts at 1
        source_line = self.prefix + linecache.getline(str(self.src_file), index).rstrip("\n")
        tgt_line = linecache.getline(str(self.tgt_file), index).rstrip("\n")
        assert source_line, f"empty source line for index {index}"
        assert tgt_line, f"empty tgt line for index {index}"
        return {"tgt_texts": tgt_line, "src_texts": source_line, "id": index - 1}

    def collate_fn(self, batch) -> Dict[str, torch.Tensor]:
        """Call prepare_seq2seq_batch."""
        batch_encoding: Dict[str, torch.Tensor] = self.tokenizer.prepare_seq2seq_batch(
            [x["src_texts"] for x in batch],
            tgt_texts=[x["tgt_texts"] for x in batch],
            max_length=self.max_source_length,
            max_target_length=self.max_target_length,
            return_tensors="pt",
            **self.dataset_kwargs,
        ).data
        batch_encoding["ids"] = torch.tensor([x["id"] for x in batch])


SUMMARY:
**Class Seq2SeqDataset Summary**

### Purpose and Functionality

The `Seq2SeqDataset` class is an abstract dataset implementation that loads sequential data from files (`src_file` and `tgt_file`) and provides methods for data retrieval and manipulation.

* Primary purpose: Load sequential data from files and provide tokenized batches.
* Key algorithm: line caching to handle file indexing (starts at 1).

### Implementation Details

* The `__getitem__` method loads two lines (`source_line` and `tgt_line`) from the specified files using `linecache.getline`, while incrementing the index by 1. It returns a dictionary containing the source and target text, ID, with assertions for non-empty lines.
* The `collate_fn` method tokenizes the batch of sequential texts into a single tensor using the tokenizer's `prepare_seq2seq_batch` method, returning the resulting tensor along with a tensor of IDs.

### Notable Dependencies and Requirements

* The class uses `linecache`, `torchlib`, and `tokenizer` (not shown in this excerpt) for data loading and tokenization.
* It expects external dependencies:
	+ `linecache`: A library for caching lines from files.
	+ `torch`: A tensor library for PyTorch.


key: chroma:document
string_value:
CODE:

class Seq2SeqDataCollator:
    def __init__(self, tokenizer, data_args, decoder_start_token_id, tpu_num_cores=None):
        self.tokenizer = tokenizer
        self.pad_token_id = tokenizer.pad_token_id
        self.decoder_start_token_id = decoder_start_token_id
        assert (
            self.pad_token_id is not None
        ), f"pad_token_id is not defined for ({self.tokenizer.__class__.__name__}), it must be defined."
        self.data_args = data_args
        self.tpu_num_cores = tpu_num_cores
        self.dataset_kwargs = {"add_prefix_space": True} if isinstance(tokenizer, BartTokenizer) else {}
        if data_args.src_lang is not None:
            self.dataset_kwargs["src_lang"] = data_args.src_lang
        if data_args.tgt_lang is not None:
            self.dataset_kwargs["tgt_lang"] = data_args.tgt_lang

    def __call__(self, batch) -> Dict[str, torch.Tensor]:
        if hasattr(self.tokenizer, "prepare_seq2seq_batch"):
            batch = self._encode(batch)
            input_ids, attention_mask, labels = (
                batch["input_ids"],
                batch["attention_mask"],
                batch["labels"],
            )
        else:
            input_ids = torch.stack([x["input_ids"] for x in batch])
            attention_mask = torch.stack([x["attention_mask"] for x in batch])
            labels = torch.stack([x["labels"] for x in batch])

            labels = trim_batch(labels, self.pad_token_id)
            input_ids, attention_mask = trim_batch(input_ids, self.pad_token_id, attention_mask=attention_mask)

        if isinstance(self.tokenizer, T5Tokenizer):
            decoder_input_ids = self._shift_right_t5(labels)
        else:
            decoder_input_ids = shift_tokens_right(labels, self.pad_token_id, self.decoder_start_token_id)

        batch = {
            "input_ids": input_ids,
            "attention_mask": attention_mask,
            "decoder_input_ids": decoder_input_ids,
            "labels": labels,
        }
        return batch

    def _shift_right_t5(self, input_ids):
        # shift inputs to the right
        shifted_input_ids = input_ids.new_zeros(input_ids.shape)
        shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()
        shifted_input_ids[..., 0] = self.pad_token_id
        return shifted_input_ids

    def _encode(self, batch) -> Dict[str, torch.Tensor]:
        batch_encoding = self.tokenizer.prepare_seq2seq_batch(
            [x["src_texts"] for x in batch],
            tgt_texts=[x["tgt_texts"] for x in batch],
            max_length=self.data_args.max_source_length,
            max_target_length=self.data_args.max_target_length,
            padding="max_length" if self.tpu_num_cores is not None else "longest",  # TPU hack
            return_tensors="pt",
            **self.dataset_kwargs,
        )


SUMMARY:
**Seq2SeqDataCollator Class Summary**

### Purpose and Functionality

The `Seq2SeqDataCollator` class is designed to collate data for sequence-to-sequence tasks, such as machine translation. It takes in batches of data, tokenizes the input sequences (source texts), and prepares the data for training a model.

### Key Implementation Details

* The `_encode` method prepares batched data using `prepare_seq2eq_batch`, applying padding and truncation to sequences.
* The `__call__` method checks if the tokenizer has a specific `prepare_seq2seq_batch` method, and calls it accordingly. It also handles decoding by shifting input IDs to the right (T5-style).

### Notable Dependencies or Requirements

* `tokenizer`: An instance of `T5Tokenizer` or `BartTokenizer`, depending on the implementation.
* `TPU_num_cores`: Optional parameter for padding length.

### Interfaces and Method Signatures

* `__init__`: Initializes the collator with tokenizer, data arguments, decoder start token ID, and TPU number cores.
* `train_collate_batch`: Collates batched data (input sequence, target sequence).
	+ Returns: A dictionary of strings to tensors (`input_ids`, `attention_mask`, `labels`, `decoder_input_ids`).


key: chroma:document
string_value:
CODE:

class SortishSampler(Sampler):
    "Go through the text data by order of src length with a bit of randomness. From fastai repo."

    def __init__(self, data, batch_size, shuffle=True):
        self.data, self.bs, self.shuffle = data, batch_size, shuffle

    def __len__(self) -> int:
        return len(self.data)

    def __iter__(self):


SUMMARY:
**Class Summary: SortishSampler**

The `SortishSampler` class is a custom sampler for data augmentation in the context of machine learning. It inherits from the `Sampler` class and modifies the `fastai` repository's original implementation.

**Primary Purpose:** Reorders text data based on source length with added randomness.

**Key Implementation Details:**

* Uses `sortish_sampler_indices` function to generate indices for shuffling.
* Allows for customization of batch size and shuffling behavior.

**Notable Dependencies/Requirements:**

* `data`: the input data to be reordered.
* `batch_size`: integer value determining the number of samples to process at a time.

**Methods:**

* `__init__:` Initializes the sampler with `data`, `batch_size`, and optional `shuffle` parameter.
* `__len__`: Returns the total length of the reOrdered data.
* `__iter__`: Generates indices for shuffling the data based on `sortish_sampler_indices`.


key: chroma:document
string_value:
CODE:

def sortish_sampler_indices(data: List, bs: int, shuffle=True) -> np.array:
    "Go through the text data by order of src length with a bit of randomness. From fastai repo."
    if not shuffle:
        return np.argsort(np.array(data) * -1)

    def key_fn(i):
        return data[i]

    idxs = np.random.permutation(len(data))
    sz = bs * 50
    ck_idx = [idxs[i : i + sz] for i in range(0, len(idxs), sz)]
    sort_idx = np.concatenate([sorted(s, key=key_fn, reverse=True) for s in ck_idx])
    sz = bs
    ck_idx = [sort_idx[i : i + sz] for i in range(0, len(sort_idx), sz)]
    max_ck = np.argmax([key_fn(ck[0]) for ck in ck_idx])  # find the chunk with the largest key,
    ck_idx[0], ck_idx[max_ck] = ck_idx[max_ck], ck_idx[0]  # then make sure it goes first.
    sort_idx = np.concatenate(np.random.permutation(ck_idx[1:])) if len(ck_idx) > 1 else np.array([], dtype=int)
    sort_idx = np.concatenate((ck_idx[0], sort_idx))


SUMMARY:
**Function Summary:** `sortish_sampler_indices`

**Purpose and Functionality:**
The function samples indices from a sorted list of data (`data`) with alternating chunks of specified size (`bs`). It also adds randomness to the arrangement by shuffling the chunks. The goal is to mimic a "sortish sampler" used in the Fastai repository.

**Key Implementation Details:**

* Uses `np.argsort` to sort the original data by source length, then applies random permutations and sorting within each chunk.
* Alternates between sorting by length and adding randomness within each chunk using `np.random.permutation`.
* Reorders the chunks based on their key values (lengths of text data) to prioritize larger ones.

**Notable Dependencies:**

* NumPy (`np.array`, `np.argsort`, `np.concatenate`)
* List (`data`)

**Interfaces and Inherited:**

* None explicitly mentioned
* Implicitly uses Python's built-in sorting algorithm


key: chroma:document
string_value:
CODE:

class DistributedSortishSampler(Sampler):
    """Copied from torch DistributedSampler"""

    def __init__(self, dataset, batch_size, num_replicas=None, rank=None, add_extra_examples=True, shuffle=True):
        if num_replicas is None:
            if not dist.is_available():
                raise RuntimeError("Requires distributed package to be available")
            num_replicas = dist.get_world_size()
        if rank is None:
            if not dist.is_available():
                raise RuntimeError("Requires distributed package to be available")
            rank = dist.get_rank()
        self.dataset = dataset
        self.num_replicas = num_replicas
        self.rank = rank
        self.epoch = 0
        if add_extra_examples:
            self.num_samples = int(math.ceil(len(self.dataset) * 1.0 / self.num_replicas))
            self.total_size = self.num_samples * self.num_replicas
        else:
            self.total_size = len(dataset)
            self.num_samples = len(self.available_indices)
        self.batch_size = batch_size
        self.add_extra_examples = add_extra_examples
        self.shuffle = shuffle

    def __iter__(self) -> Iterable:
        g = torch.Generator()
        g.manual_seed(self.epoch)

        sortish_data = [self.dataset.src_lens[i] for i in self.available_indices]
        sortish_indices = sortish_sampler_indices(sortish_data, self.batch_size, shuffle=self.shuffle)
        indices = [self.available_indices[i] for i in sortish_indices]
        assert len(indices) == self.num_samples
        return iter(indices)

    @cached_property
    def available_indices(self) -> np.array:
        indices = list(range(len(self.dataset)))
        # add extra samples to make it evenly divisible
        indices += indices[: (self.total_size - len(indices))]
        assert len(indices) == self.total_size
        # subsample
        available_indices = indices[self.rank : self.total_size : self.num_replicas]
        return available_indices

    def __len__(self):
        return self.num_samples

    def set_epoch(self, epoch):


SUMMARY:
**Class Summary: DistributedSortishSampler**

**Main Purpose and Functionality:**
The `DistributedSortishSampler` class is a custom sampler for distributed training in PyTorch. It extends the built-in `torch.DistributedSampler` class and provides an optimized way to sample data across multiple GPUs in a distributed environment.

**Key Implementation Details:**

* The `__init__` method initializes the sampler with dataset, batch size, number of replicas, rank, and additional parameters.
* The `available_indices` property uses list comprehension to generate a list of indices for subsampling, which are then sliced using PyTorch's tensor indexing.

**Notable Dependencies and Requirements:**

* `torch`: PyTorch library (version 1.10+ required)
* `torch.distributed`: Distributed package for PyTorch
* `numpy` library for mathematical operations

**Primary Purpose:**
The primary purpose of this class is to provide an efficient way to sample data in a distributed environment, allowing for more effective use of resources and improved training performance.

**Method Signatures and Interfaces:**

* `__init__`: Initializes the sampler with dataset, batch size, number of replicas, rank, and additional parameters.
* `available_indices`: Returns an array of indices for subsampling.
* `set_epoch`: Updates the epoch counter.

**Key Algorithms or Patterns:**

* Distribution of indices using numpy's list comprehension
* PyTorch tensor indexing for slicing and sampling


key: chroma:document
string_value:
CODE:

def use_task_specific_params(model, task):
    """Update config with summarization specific params."""
    task_specific_params = model.config.task_specific_params

    if task_specific_params is not None:
        pars = task_specific_params.get(task, {})
        logger.info(f"setting model.config to task specific params for {task}:\n {pars}")
        logger.info("note: command line args may override some of these")


SUMMARY:
**Function Summary**

`use_task_specific_params(model, task)`: Updates the `model.config` with task-specific summarization parameters.

**Key Implementation Details**

- Verifies existence and retrieves `task_specific_params` within `model.config`.
- Updates `model.config` using retrieved task-specific parameters.
- Logs information for debugging purposes, including override notes from command line args.

**Notable Dependencies/Requirements**

- [None explicitly mentioned in the provided snippet]

Note: The function utilizes `logger` which is not defined in this snippet; it's assumed to be a logging object set up elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def pickle_load(path):
    """pickle.load(path)"""
    with open(path, "rb") as f:


SUMMARY:
**Summary:**

`pickle_load` Function Summary:

### Purpose and Functionality
This function loads a Python object from a binary file using the `pickle` module.

### Key Implementation Details

* **File Loading**: Opens a file at the specified `path` in read-binary mode (`"rb"`).
* **Pickle Load**: Calls the `pickle.load` function to deserialize the binary data into a Python object, which is then returned by the function.

### Notable Dependencies and Requirements

* `pickle`: Used for handling binary serialization and deserialization.
* No specific interfaces or classes are implemented or inherited. 

The original code's comment includes unnecessary boilerplate: `"pickle.load(path)"` can be dropped and the code will still function correctly, as it is simply documenting the existing behavior.


key: chroma:document
string_value:
CODE:

def pickle_save(obj, path):
    """pickle.dump(obj, path)"""
    with open(path, "wb") as f:


SUMMARY:
**Summary**

The `pickle_save` function saves an object to a binary file using Python's `pickle` module. It takes two parameters:

- `obj`: The object to be pickled
- `path`: The file path where the object will be saved

### Key Implementation Details

* Usage of `pickle.dump(obj, f)` to serialize and save the object.

### Notable Dependencies/Requirements

* Python's built-in `pickle` module for serialization/deserialization.
* External dependency: `pickle` is part of Python Standard Library.

### Interface
The function implements the following interface:
```python
def pickle_save(obj: any, path: str) -> None
```

Note that this summary omits unnecessary details and focuses on providing a clear understanding of the code's purpose, key implementation details, and dependencies.


key: chroma:document
string_value:
CODE:

def flatten_list(summary_ids: List[List]):


SUMMARY:
**Summary**

`flatten_list` function takes a 2D list of `summary_ids` and returns a flat list using `itertools.chain.from_iterable`.

### Key Details

* Takes a `summary_ids` as input, where each element is a sublist containing IDs
* Uses `itertools.chain.from_iterable` to concatenate sublists into a single iterable
* Returns the flat list of IDs as a Python list

### Notable Dependencies

* `itertools`: used for chain-like object creation
* `List`: inferred type hint for input parameter (`summary_ids`)


key: chroma:document
string_value:
CODE:

def save_git_info(folder_path: str) -> None:
    """Save git information to output_dir/git_log.json"""
    repo_infos = get_git_info()


SUMMARY:
**Function Summary**

```markdown
# Save Git Information to Local File

### Purpose
Saves processed Git information to a local JSON file.

### Implementation Details

* Extracts Git information using `get_git_info()`.
* Serializes and saves the retrieved data to `folder_path/git_log.json` using `save_json()`.

### Dependencies

* `get_git_info():` returns retrieved Git information
* `os.path.join`: joins folder path with file name for JSON storage
```

Note: I've excluded unnecessary details like function signature and variable names that aren't relevant to the functionality or core logic of the code. The summary focuses on providing a concise overview of what the function does, its implementation, and any required dependencies.


key: chroma:document
string_value:
CODE:

def save_json(content, path, indent=4, **json_dump_kwargs):
    with open(path, "w") as f:


SUMMARY:
**Summary**

`save_json` function saves Python content to a JSON file at specified path. It takes in:

* `content`: the data to be serialized
* `path`: the file path where data will be saved
* `indent=4`: optional, specifies indentation level for formatted output (default: 4)
* `**json_dump_kwargs`: additional keyword arguments to pass to `json.dump()`

The function uses `json.dump()` with `sort_keys=True` and customizes indentation.

**Key Implementation Details**

* Uses `json.dump()` with customizable options (`indent`, `sort_keys`)
* Formats output using indentation level specified by `indent=4`
* Accepts additional keyword arguments (`**json_dump_kwargs`) for flexible customization

**Dependencies/Notable Requirements**

* Python's built-in `json` module is required


key: chroma:document
string_value:
CODE:

def load_json(path):
    with open(path) as f:


SUMMARY:
**Summary**

`load_json(path)` loads JSON data from the specified file path. It leverages `json.load()` to deserialize the contents of the file.

**Key Implementation Details**

* Utilizes `json.load()` to parse JSON data from a file
* Returns the deserialized JSON object

**Dependences**

* `python.json` (built-in Python library for working with JSON)

Note: This summary is concise, clean and focused on providing an accurate representation of the function's purpose and implementation.


key: chroma:document
string_value:
CODE:

def get_git_info():
    try:
        repo = git.Repo(search_parent_directories=True)
        repo_infos = {
            "repo_id": str(repo),
            "repo_sha": str(repo.head.object.hexsha),
            "repo_branch": str(repo.active_branch),
            "hostname": str(socket.gethostname()),
        }
        return repo_infos
    except TypeError:
        return {
            "repo_id": None,
            "repo_sha": None,
            "repo_branch": None,
            "hostname": None,


SUMMARY:
**Summary:**

`get_git_info()` Function:

*   Retrieves information about the current Git repository on the system.
*   Returns a dictionary containing:
    *   `repo_id`: The Git repository object.
    *   `repo_sha`: The SHA of the repository head commit.
    *   `repo_branch`: The name of the active branch.
    *   `hostname`: The hostname of the system's operating system.

**Key Implementation Details:**

*   Uses `git.Repo` to search for a matching repository across parent directories.
*   Utilizes the `head.object.hexsha` attribute to obtain the SHA of the head commit.
*   Incorporates platform-specific operation (hostname) using `socket.gethostname()`.

**Notable Dependencies and Requirements:**

*   The function relies on the `git` module, which is part of the Python Standard Library and assumes a system with Git installed.

This code summary provides a concise overview of the `get_git_info()` function's functionality, key implementation details, and dependencies.


key: chroma:document
string_value:
CODE:

def extract_rouge_mid_statistics(dct):
    new_dict = {}
    for k1, v1 in dct.items():
        mid = v1.mid
        new_dict[k1] = {stat: round(getattr(mid, stat), 4) for stat in ["precision", "recall", "fmeasure"]}


SUMMARY:
**Summary**

`extract_rouge_mid_statistics` function:

* Extracts and aggregates mid statistics from a dictionary of model outputs (`dct`).
* Maps each output's `mid` object to a condensed dictionary containing precision, recall, and f-measure statistics.
* Returns the resulting dictionary with refined statistics.

**Key Implementation Details**

* Iterates over input dictionary (`dct`) using `.items()`.
* Uses `round()` to round mid stats to 4 decimal places.
* Utilizes `getattr(mid, stat)` to dynamically retrieve specific attributes from each `mid` object.

**Notable Dependencies/Requirements**

* None explicitly stated, but implicit usage of `dict`, `list`, and attribute access implies compatibility with Python's built-in types.


key: chroma:document
string_value:
CODE:

def calculate_rouge(
    pred_lns: List[str],
    tgt_lns: List[str],
    use_stemmer=True,
    rouge_keys=ROUGE_KEYS,
    return_precision_and_recall=False,
    bootstrap_aggregation=True,
    newline_sep=True,
) -> Dict:
    """Calculate rouge using rouge_scorer package.

    Args:
        pred_lns: list of summaries generated by model
        tgt_lns: list of groundtruth summaries (e.g. contents of val.target)
        use_stemmer:  Bool indicating whether Porter stemmer should be used to
        strip word suffixes to improve matching.
        rouge_keys:  which metrics to compute, defaults to rouge1, rouge2, rougeL, rougeLsum
        return_precision_and_recall: (False) whether to also return precision and recall.
        bootstrap_aggregation: whether to do the typical bootstrap resampling of scores. Defaults to True, if False
            this function returns a collections.defaultdict[metric: list of values for each observation for each subscore]``
        newline_sep:(default=True) whether to add newline between sentences. This is essential for calculation rougeL
        on multi sentence summaries (CNN/DM dataset).

    Returns:
         Dict[score: value] if aggregate else defaultdict(list) keyed by rouge_keys

    """
    scorer = rouge_scorer.RougeScorer(rouge_keys, use_stemmer=use_stemmer)
    aggregator = scoring.BootstrapAggregator()
    for pred, tgt in zip(tgt_lns, pred_lns):
        # rougeLsum expects "\n" separated sentences within a summary
        if newline_sep:
            pred = add_newline_to_end_of_each_sentence(pred)
            tgt = add_newline_to_end_of_each_sentence(tgt)
        scores = scorer.score(pred, tgt)
        aggregator.add_scores(scores)

    if bootstrap_aggregation:
        result = aggregator.aggregate()
        if return_precision_and_recall:
            return extract_rouge_mid_statistics(result)  # here we return dict
        else:
            return {k: round(v.mid.fmeasure * 100, 4) for k, v in result.items()}

    else:


SUMMARY:
**Concise Summary:**

The `calculate_rouge` function calculates ROUGE scores using the `rouge_scorer` package. It takes in two lists of summaries (`pred_lns` and `tgt_lns`) and various configuration options.

**Main Purpose:**
Calculate ROUGE scores ( Rouge-1, Rouge-2, RougeL, RougeLsum) between predicted and target summaries.

**Key Implementation Details:**

* Uses `rouge_scorer.RougeScorer` to compute scores
* Applies `scoring.BootstrapAggregator` for bootstrapping aggregation
* Adds newline separators to sentences if required

**Notable Dependencies and Requirements:**

* `rouge_scorer`: ROUGE scoring package
* `scoring`: Bootstrap aggregator implementation
* `add_newline_to_end_of_each_sentence`: Custom function (not provided)

**Return Values:**

* Dictionary with scores keyed by Rouge metrics (default)
* Default dictionary with list values keyed by Rouge metrics if bootstrapping is disabled


key: chroma:document
string_value:
CODE:

def freeze_params(model: nn.Module):
    """Set requires_grad=False for each of model.parameters()"""
    for par in model.parameters():


SUMMARY:
**Summary**

`freeze_params(model: nn.Module)` freezes the gradients of all model parameters by setting `requires_grad=False`.

**Key Details**

* Sets `requires_grad=False` for each parameter in the input `model`.
* Only modifies the internal state of the `model`, no external dependencies.

**Notable Dependencies/Interfaces**

* `nn.Module`: A PyTorch neural network module.
* No external dependencies required.


key: chroma:document
string_value:
CODE:

def freeze_embeds(model):
    """Freeze token embeddings and positional embeddings for bart, just token embeddings for t5."""
    model_type = model.config.model_type

    if model_type in ["t5", "mt5"]:
        freeze_params(model.shared)
        for d in [model.encoder, model.decoder]:
            freeze_params(d.embed_tokens)
    elif model_type == "fsmt":
        for d in [model.model.encoder, model.model.decoder]:
            freeze_params(d.embed_positions)
            freeze_params(d.embed_tokens)
    else:
        freeze_params(model.model.shared)
        for d in [model.model.encoder, model.model.decoder]:
            freeze_params(d.embed_positions)


SUMMARY:
**Summary: Freeze Embeddings Function**

The `freeze_embeds` function freezes embeddings in a model based on its type. It:

1. **Determines Model Type**: Checks the model's configuration to determine if it's a T5, MT5, FSMT, or Bart model.

2. **Frozen Embeddings Implementation**:
   - For T5 and MT5 models: Freezes all shared parameters and embed tokens in the encoder and decoder.
   - For FSMT models: Frezes both position embeddings and token embeddings in the encoder and decoder.
   - For Bart models: Freezes only shared parameters and embed tokens in the encoder and decoder.

**Key Implementation Details:**

* Used `freeze_params` function to freeze either `embed_tokens`, `embed_positions`, or `shared` values.
* Utilized dictionary access (`model.config.model_type`) to check model type.

**Notable Dependencies/Requirements:**

* Requires `model` object with config and shared parameters.
* Assumed presence of `freeze_params` function which is not part of this code snippet.


key: chroma:document
string_value:
CODE:

def grad_status(model: nn.Module) -> Iterable:


SUMMARY:
**Summary:**

The `grad_status` function takes an instance of a PyTorch neural network (`nn.Module`) as input. It returns an iterable containing boolean values indicating whether each parameter in the model is trainable or requires gradients.

**Key Details:**

* The function uses list comprehension to iterate over the model's parameters.
* It utilizes the `requires_grad` attribute of each parameter instance (`par`).

**Notable Dependencies/Requirements:**

* PyTorch neural network library (`nn.Module`)


key: chroma:document
string_value:
CODE:

def any_requires_grad(model: nn.Module) -> bool:


SUMMARY:
**Summary:**

This is a Python function named `any_requires_grad` that takes a PyTorch Neural Network module (`nn.Module`) as input and returns a boolean indicating whether any of its components require gradients during backpropagation.

**Key Details:**

* Primary purpose: Check if model components require gradients.
* Implementation: Uses the `grad_status` function to evaluate each component in the model, returning `True` if any component requires gradients.
* Interfaces/Methods:
	+ `grad_status(model)`: Evaluates the gradient status of each component in the `model`.
	+ `nn.Module`: A PyTorch Neural Network module.

**External Dependencies:** None

Note: The code assumes that `grad_status` is an existing function or method, and its implementation is not provided here.


key: chroma:document
string_value:
CODE:

def assert_all_frozen(model):
    model_grads: List[bool] = list(grad_status(model))
    n_require_grad = sum(lmap(int, model_grads))
    npars = len(model_grads)


SUMMARY:
**Summary: `assert_all_frozen` Function**

The `assert_all_frozen` function verifies that all weights in a given model are frozen by checking if any gradients (i.e., require_grad=True) exist.

**Key Details:**

* Iterates over the gradients of the input model using `grad_status`.
* Counts the number of require_grad=True values (`n_require_grad`) and total number of weights (`npars`).
* Asserts that no gradients are present, reporting a non-zero percentage if any gradients do exist.
* Primary purpose: verify frozen model consistency.

**Notable Dependencies:**

* `model`: Input machine learning model ( expected to be a PyTorch model instance)
* External dependency: `lmap`, `sum` functions from Python's built-in library


key: chroma:document
string_value:
CODE:

def assert_not_all_frozen(model):
    model_grads: List[bool] = list(grad_status(model))
    npars = len(model_grads)


SUMMARY:
**Function Summary**

`assert_not_all_frozen`: Verifies that not all model weights have frozen gradients.

**Key Implementation Details:**

* Iterates through `grad_status` of input model, storing boolean status of each weight in `model_grads`.
* Checks if any weights require gradients (i.e., not all are frozen).
* Raises an assertion error with an incorrect message if none of the weights require gradients.

**Notable Dependencies/Requirements:**

* `list` and `any` functions from Python's standard library.
* `grad_status` function, whose implementation is unknown within this snippet but presumably defines the gradient status of model weights.


key: chroma:document
string_value:
CODE:

def parse_numeric_n_bool_cl_kwargs(unparsed_args: List[str]) -> Dict[str, Union[int, float, bool]]:
    """
    Parse an argv list of unspecified command line args to a dict.
    Assumes all values are either numeric or boolean in the form of true/false.
    """
    result = {}
    assert len(unparsed_args) % 2 == 0, f"got odd number of unparsed args: {unparsed_args}"
    num_pairs = len(unparsed_args) // 2
    for pair_num in range(num_pairs):
        i = 2 * pair_num
        assert unparsed_args[i].startswith("--")
        if unparsed_args[i + 1].lower() == "true":
            value = True
        elif unparsed_args[i + 1].lower() == "false":
            value = False
        else:
            try:
                value = int(unparsed_args[i + 1])
            except ValueError:
                value = float(unparsed_args[i + 1])  # this can raise another informative ValueError

        result[unparsed_args[i][2:]] = value


SUMMARY:
**Summary**

`parse_numeric_n_bool_cl_kwargs` function parses an argument list of unspecified command-line args into a dictionary.

*   Takes in `unparsed_args`: `List[str]`.
*   Assumes all values are either numeric or boolean in the form of true/false.
*   Returns a dictionary (`Dict[str, Union[int, float, bool]]`) where keys are parsed args (without leading `-`).
*   Key implementation details:
    *   Pairs arguments using the index formula (`i = 2 \* pair_num`).
    *   Uses `assert` statements to validate argument counts and types.
*   Notable dependencies and requirements:
    *   `List[str]`: unparsed_args
    *   `Dict[str, Union[int, float, bool]]`: parsed result dictionary

**Key algorithms/design choices:**

1.  **Argument pairing**: uses index formula (`i = 2 * pair_num`) to pair arguments.

2.  **Value type determination**: checks for boolean values (true/false) or attempts to parse as integer/float, raising informative exceptions for invalid inputs.

3.  **Error handling and assertion**: uses `assert` statements to validate argument counts and types.

**External dependencies: None**

**Implemented interfaces: None**

This function appears to be a custom parser for command-line arguments, which can be used to process flags with numeric or boolean values. The code is concise and follows best practices for Python development.


key: chroma:document
string_value:
CODE:

def write_txt_file(ordered_tgt, path):
    f = Path(path).open("w")
    for ln in ordered_tgt:
        f.write(ln + "\n")


SUMMARY:
**Summary**

The `write_txt_file` function writes the contents of an ordered list stored in `ordered_tgt` to a text file specified by the `path`. It creates a new file at the provided path, appends each line from the ordered list followed by a newline character, and flushes the file after each write to ensure immediate storage.

**Key Implementation Details**

- Appends lines to a file using `Path(path).open("w")`
- Flushes file after each write using `f.flush()`
- Writes newline characters (`\n`) between each line

**Notable Dependencies or Requirements**

- `ordered_tgt` is assumed to be an iterable object (e.g. list)
- The function uses the `pathlib` library for path manipulation (`Path(path).open("w")`)


key: chroma:document
string_value:
CODE:

def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):


SUMMARY:
**Code Summary:**

```
Generates consecutive n-sized chunks from an input list (lst).
```


key: chroma:document
string_value:
CODE:

def check_output_dir(args, expected_items=0):
    """
    Checks whether to bail out if output_dir already exists and has more than expected_items in it

    `args`: needs to have the following attributes of `args`:
      - output_dir
      - do_train
      - overwrite_output_dir

    `expected_items`: normally 0 (default) - i.e. empty dir, but in some cases a few files are expected (e.g. recovery from OOM)
    """
    if (
        os.path.exists(args.output_dir)
        and len(os.listdir(args.output_dir)) > expected_items
        and args.do_train
        and not args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({args.output_dir}) already exists and "
            f"has {len(os.listdir(args.output_dir))} items in it (expected {expected_items} items). "
            "Use --overwrite_output_dir to overcome."


SUMMARY:
Here is a concise summary of the function code chunk:

### Function: `check_output_dir`

**Purpose:** Verifies whether to overwrite an existing output directory with mismatched contents.

**Parameters:**

* `args`: An object containing:
	+ `output_dir` (str): The path to the output directory.
	+ `do_train` (bool): Indicating training status.
	+ `overwrite_output_dir` (bool): Flag for overwriting existing output directory.
* `expected_items` (int, default=0): Expected number of items in the output directory.

**Functionality:**

Checks if an existing output directory has more than expected items when:
- Training is enabled (`args.do_train`) and
- Overwrite option is disabled (`args.overwrite_output_dir`).
If conditions are met, raises a `ValueError`.

### Key Implementation Details:

* Uses `os.path.exists`, `len(os.listdir)`, and file system operations to check directory contents.
* Raising a custom error with detailed message indicating expected vs. actual contents and suggested action.


key: chroma:document
string_value:
CODE:

def eval_data_dir(
    data_dir,
    save_dir: str,
    model_name: str,
    bs: int = 8,
    max_source_length: int = 1024,
    type_path="val",
    n_obs=None,
    fp16=False,
    task="summarization",
    local_rank=None,
    num_return_sequences=1,
    dataset_kwargs: Dict = None,
    prefix="",
    **generate_kwargs,
) -> Dict:
    """Run evaluation on part of the data for one gpu and save to {save_dir}/rank_{rank}_output.json"""
    model_name = str(model_name)
    assert local_rank is not None
    torch.distributed.init_process_group(backend="nccl", rank=local_rank)

    save_dir = Path(save_dir)
    save_path = save_dir.joinpath(f"rank_{local_rank}_output.json")
    torch.cuda.set_device(local_rank)
    model = AutoModelForSeq2SeqLM.from_pretrained(model_name).cuda()
    if fp16:
        model = model.half()
    # determine if we need to increase num_beams
    use_task_specific_params(model, task)  # update config with task specific params
    num_beams = generate_kwargs.pop("num_beams", model.config.num_beams)  # AttributeError risk?
    if num_return_sequences > num_beams:
        num_beams = num_return_sequences

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    logger.info(f"Inferred tokenizer type: {tokenizer.__class__}")  # if this is wrong, check config.model_type.

    if max_source_length is None:
        max_source_length = tokenizer.model_max_length
    if prefix is None:
        prefix = prefix or getattr(model.config, "prefix", "") or ""
    ds = Seq2SeqDataset(
        tokenizer,
        data_dir,
        max_source_length,
        max_target_length=1024,
        type_path=type_path,
        n_obs=n_obs,
        prefix=prefix,
        **dataset_kwargs,
    )
    # I set shuffle=True for a more accurate progress bar.
    # If all the longest samples are first, the prog bar estimate is too high at the beginning.
    sampler = ds.make_sortish_sampler(bs, distributed=True, add_extra_examples=False, shuffle=True)
    data_loader = DataLoader(ds, sampler=sampler, batch_size=bs, collate_fn=ds.collate_fn)
    results = []
    for batch in tqdm(data_loader):
        summaries = model.generate(
            input_ids=batch["input_ids"].to(model.device),
            attention_mask=batch["attention_mask"].to(model.device),
            num_return_sequences=num_return_sequences,
            num_beams=num_beams,
            **generate_kwargs,
        )
        preds = tokenizer.batch_decode(summaries, skip_special_tokens=True, clean_up_tokenization_spaces=False)
        ids = batch["ids"]
        if num_return_sequences > 1:
            preds = chunks(preds, num_return_sequences)  # batch size chunks, each of size num_return_seq
        for i, pred in enumerate(preds):
            results.append({"pred": pred, "id": ids[i].item()})
    save_json(results, save_path)


SUMMARY:
**Summary:**

This function evaluates and generates summaries from a dataset using a pre-trained sequence-to-sequence model. The primary purpose is to run inference on a specific part of the data for a single GPU.

### Main Purpose and Functionality:

* Evaluates data in `data_dir` for one GPU
* Saves results to `{save_dir}/rank_XX_output.json`
* Uses AutoModelForSeq2SeqLM from Hugging Face with configuration updates based on task type

### Key Implementation Details:

* Initializes model, tokenizer, and dataset with `model_name`, `data_dir`, and other parameters
* Sets up data loader, sampler, and batch processing with shuffle=True for progress bar accuracy
* Performs model generation to generate summaries using `model.generate` method

### Notable Dependencies or Requirements:

* Hugging Face AutoModelForSeq2SeqLM and AutoTokenizer libraries
* PyTorch backend with distributed setup (nccl)
* torch.distributed.init_process_group, torch.cuda.set_device

### Additional Details:

* Optional task-specific parameters can be passed using `generate_kwargs`
* Default values are set for some parameters (e.g., `num_beams`)
* Results are saved to a JSON file using `save_json` function

**Code Quality:**

The code is well-structured and follows good practices. The use of descriptive variable names, type hints, and specific exception handling make it readable and maintainable.

### Recommendations:

Consider adding more docstrings or comments to explain the purpose and behavior of some functions (e.g., `use_task_specific_params`).

Also, you may want to consider using a consistent naming convention for Python files and libraries. 

As this function appears to be part of a larger production environment, it's also worth considering additional documentation or configuration management practices (e.g., logging, input validation).


key: chroma:document
string_value:
CODE:

def run_generate():
    parser = argparse.ArgumentParser(
        epilog="Unspecified args like --num_beams=2 --decoder_start_token_id=4 are passed to model.generate"
    )
    parser.add_argument("--data_dir", type=str, help="like cnn_dm/test.source")
    parser.add_argument(
        "--model_name",
        type=str,
        help="like facebook/bart-large-cnn,google-t5/t5-base, etc.",
        default="sshleifer/distilbart-xsum-12-3",
    )
    parser.add_argument("--save_dir", type=str, help="where to save", default="tmp_gen")
    parser.add_argument("--max_source_length", type=int, default=None)
    parser.add_argument(
        "--type_path", type=str, default="test", help="which subset to evaluate typically train/val/test"
    )
    parser.add_argument("--task", type=str, default="summarization", help="used for task_specific_params + metrics")
    parser.add_argument("--bs", type=int, default=8, required=False, help="batch size")
    parser.add_argument(
        "--local_rank", type=int, default=-1, required=False, help="should be passed by distributed.launch"
    )

    parser.add_argument(
        "--n_obs", type=int, default=None, required=False, help="How many observations. Defaults to all."
    )
    parser.add_argument(
        "--num_return_sequences", type=int, default=1, required=False, help="How many sequences to return"
    )
    parser.add_argument(
        "--sync_timeout",
        type=int,
        default=600,
        required=False,
        help="How long should master process wait for other processes to finish.",
    )
    parser.add_argument("--src_lang", type=str, default=None, required=False)
    parser.add_argument("--tgt_lang", type=str, default=None, required=False)
    parser.add_argument(
        "--prefix", type=str, required=False, default=None, help="will be added to the beginning of src examples"
    )
    parser.add_argument("--fp16", action="store_true")
    parser.add_argument("--debug", action="store_true")
    start_time = time.time()
    args, rest = parser.parse_known_args()
    generate_kwargs = parse_numeric_n_bool_cl_kwargs(rest)
    if generate_kwargs and args.local_rank <= 0:
        print(f"parsed the following generate kwargs: {generate_kwargs}")
    json_save_dir = Path(args.save_dir + "_tmp")
    Path(json_save_dir).mkdir(exist_ok=True)  # this handles locking.
    intermediate_files = list(json_save_dir.glob("rank_*.json"))
    if intermediate_files:
        raise ValueError(f"Found files at {json_save_dir} please move or remove them.")
        # In theory, a node could finish and save before another node hits this. If this happens, we can address later.
    dataset_kwargs = {}
    if args.src_lang is not None:
        dataset_kwargs["src_lang"] = args.src_lang
    if args.tgt_lang is not None:
        dataset_kwargs["tgt_lang"] = args.tgt_lang

    Path(args.save_dir).mkdir(exist_ok=True)
    results, num_replicas = eval_data_dir(
        args.data_dir,
        json_save_dir,
        args.model_name,
        type_path=args.type_path,
        bs=args.bs,
        fp16=args.fp16,
        task=args.task,
        local_rank=args.local_rank,
        n_obs=args.n_obs,
        max_source_length=args.max_source_length,
        num_return_sequences=args.num_return_sequences,
        prefix=args.prefix,
        dataset_kwargs=dataset_kwargs,
        **generate_kwargs,
    )

    if args.local_rank <= 0:
        save_dir = Path(args.save_dir)
        save_dir.mkdir(exist_ok=True)
        partial_results = gather_results_from_each_node(num_replicas, json_save_dir, args.sync_timeout)
        preds = combine_partial_results(partial_results)
        if args.num_return_sequences > 1:
            save_path = save_dir.joinpath("pseudolabel_results.json")
            print(f"Saving aggregated results at {save_path}, intermediate in {json_save_dir}/")
            save_json(preds, save_path)
            return
        tgt_file = Path(args.data_dir).joinpath(args.type_path + ".target")
        with open(tgt_file) as f:
            labels = [x.rstrip() for x in f.readlines()][: len(preds)]

        # Calculate metrics, save metrics,  and save _generations.txt
        calc_bleu = "translation" in args.task
        score_fn = calculate_bleu if calc_bleu else calculate_rouge
        metric_name = "bleu" if calc_bleu else "rouge"
        metrics: Dict = score_fn(preds, labels)
        metrics["n_obs"] = len(preds)
        runtime = time.time() - start_time
        metrics["seconds_per_sample"] = round(runtime / metrics["n_obs"], 4)
        metrics["n_gpus"] = num_replicas
        # TODO(@stas00): add whatever metadata to metrics
        metrics_save_path = save_dir.joinpath(f"{args.type_path}_{metric_name}.json")
        save_json(metrics, metrics_save_path, indent=None)
        print(metrics)
        write_txt_file(preds, save_dir.joinpath(f"{args.type_path}_generations.txt"))
        if args.debug:
            write_txt_file(labels, save_dir.joinpath(f"{args.type_path}.target"))
        else:


SUMMARY:
**Summary:**

This function generates summaries using the HUGGED model. It takes several command-line arguments:

- `--data_dir`: dataset directory
- `--model_name`: model name
- `--save_dir`: save directory
- `--max_source_length`: maximum source length
- `--type_path`: type path (subset)
- `--task`: task (e.g., "summarization")
- `--bs`: batch size
- `--local_rank`: local rank for distributed training
- `--n_obs`: number of observations
- `--num_return_sequences`: number of sequences to return
- `--sync_timeout`: sync timeout

The script parses the command-line arguments, creates a dataset for training, and evaluates the model on the pre-trained data. It saves the output results in JSON files.

**Key Implementation Details:**

* Uses HUGGED model for text summarization
* Evaluates model using beam search with `--num_return_sequences` > 1

**Notable Dependencies or Requirements:**

* `numpy`
* `torch`
* `transformers`
* `huggingface` (library)
* `argparse`

**Function Signature:**
```
def eval_data_dir(
    data_dir, json_save_dir, model_name, type_path, bs, fp16, task, local_rank, n_obs, max_source_length, num_return_sequences, prefix, dataset_kwargs, **generate_kwargs
):
```


key: chroma:document
string_value:
CODE:

def combine_partial_results(partial_results) -> List:
    """Concatenate partial results into one file, then sort it by id."""
    records = []
    for partial_result in partial_results:
        records.extend(partial_result)
    records = sorted(records, key=lambda x: x["id"])
    preds = [x["pred"] for x in records]


SUMMARY:
**Function Summary**

`combine_partial_results`: Combines partial results from different sources into a single list, sorting and extracting predictions.

**Key Features**

* Concatenates `partial_result` lists using `extend()`.
* Sorts resulting list by ID using `sorted()` with a lambda function.
* Extracts prediction values (`pred`) from sorted records.

**Returns**: A list of predictions (`pred`) in sorted order.


key: chroma:document
string_value:
CODE:

def gather_results_from_each_node(num_replicas, save_dir, timeout) -> List[Dict[str, List]]:
    # WAIT FOR lots of .json files
    start_wait = time.time()
    logger.info("waiting for all nodes to finish")
    json_data = None
    while (time.time() - start_wait) < timeout:
        json_files = list(save_dir.glob("rank_*.json"))
        if len(json_files) < num_replicas:
            continue
        try:
            # make sure all json files are fully saved
            json_data = lmap(load_json, json_files)
            return json_data
        except JSONDecodeError:
            continue
    else:
        raise TimeoutError("Rank 0 gave up on waiting for other processes")


SUMMARY:
**Function Summary: `gather_results_from_each_node`**

* **Purpose:** Retrieves and merges JSON files from `num_replicas` nodes within a specified `timeout`.
	+ Primary purpose: Gather results from each node, ensuring at least `num_replicas` are available.

**Key Implementation Details:**

* Checks the number of JSON files found in the directory (`save_dir`) against the desired minimum count (`num_replicas`).
* Uses list comprehension to load and map `{load_json}` over the existing JSON files.
* Handles errors by catching `JSONDecodeError` and skipping incomplete data.

**Notable Dependencies/Requirements:**

* `time`: used for monitoring elapsed time.
* `glob`: provides a way to iterate over matching JSON file names in the directory (`save_dir`).
* `logger`: used for logging messages.
* `lmap`: applies an operation (loading JSON files) to each item in a list or mapping.


key: chroma:document
string_value:
CODE:

def convert(src_path: str, map_location: str = "cpu", save_path: Union[str, None] = None) -> None:
    """Convert a pytorch_model.bin or model.pt file to torch.float16 for faster downloads, less disk space."""
    state_dict = torch.load(src_path, map_location=map_location)
    for k, v in tqdm(state_dict.items()):
        if not isinstance(v, torch.Tensor):
            raise TypeError("FP16 conversion only works on paths that are saved state dicts, like pytorch_model.bin")
        state_dict[k] = v.half()
    if save_path is None:  # overwrite src_path
        save_path = src_path


SUMMARY:
**Summary**

`convert` function converts PyTorch model files to `torch.float16` format for faster downloads and reduced disk space.

### Key Implementation Details

- Maps loaded state dictionary to specified location (`map_location`) using `PyTorch's torch.load`.
- Iterates through the state dictionary items, checking if each value is a PyTorch tensor, raising a TypeError otherwise.
- Converts all tensors to `torch.float16` format.
- Optionally saves the updated state dictionary with specified output path (`save_path`).

### Dependencies and Requirements

- PyTorch library
- tqdm library for progressbar display (not explicitly imported in this snippet)

### Notable Functions and Interfaces

- `torch.load`
- `torch.save`
- `tqdm.update`


key: chroma:document
string_value:
CODE:

def _dump_articles(path: Path, articles: list):
    content = "\n".join(articles)


SUMMARY:
**Summary**

`_dump_articles` function writes a concatenated string of article content to a specified file path.

* **Purpose**: Writes articles to disk.
* **Implementation Details**:
	+ Concatenates article list using newline characters.
	+ Opens the specified file path in write mode and writes its contents.
* **Dependencies/Requirements**:
	+ `Path` (from the pathlib library)
	+ External dependency: None
	+ No interfaces implemented by this function.


key: chroma:document
string_value:
CODE:

class TestTheRest(TestCasePlus):
    def run_eval_tester(self, model):
        input_file_name = Path(self.get_auto_remove_tmp_dir()) / "utest_input.source"
        output_file_name = input_file_name.parent / "utest_output.txt"
        assert not output_file_name.exists()
        articles = [" New York (CNN)When Liana Barrientos was 23 years old, she got married in Westchester County."]
        _dump_articles(input_file_name, articles)

        score_path = str(Path(self.get_auto_remove_tmp_dir()) / "scores.json")
        task = "translation_en_to_de" if model == T5_TINY else "summarization"
        testargs = f"""
            run_eval_search.py
            {model}
            {input_file_name}
            {output_file_name}
            --score_path {score_path}
            --task {task}
            --num_beams 2
            --length_penalty 2.0
            """.split()

        with patch.object(sys, "argv", testargs):
            run_generate()
            assert Path(output_file_name).exists()
            # os.remove(Path(output_file_name))

    # test one model to quickly (no-@slow) catch simple problems and do an
    # extensive testing of functionality with multiple models as @slow separately
    def test_run_eval(self):
        self.run_eval_tester(T5_TINY)

    # any extra models should go into the list here - can be slow
    @parameterized.expand([BART_TINY, MBART_TINY])
    @slow
    def test_run_eval_slow(self, model):
        self.run_eval_tester(model)

    # testing with 2 models to validate: 1. translation (t5) 2. summarization (mbart)
    @parameterized.expand([T5_TINY, MBART_TINY])
    @slow
    def test_run_eval_search(self, model):
        input_file_name = Path(self.get_auto_remove_tmp_dir()) / "utest_input.source"
        output_file_name = input_file_name.parent / "utest_output.txt"
        assert not output_file_name.exists()

        text = {
            "en": ["Machine learning is great, isn't it?", "I like to eat bananas", "Tomorrow is another great day!"],
            "de": [
                "Maschinelles Lernen ist großartig, oder?",
                "Ich esse gerne Bananen",
                "Morgen ist wieder ein toller Tag!",
            ],
        }

        tmp_dir = Path(self.get_auto_remove_tmp_dir())
        score_path = str(tmp_dir / "scores.json")
        reference_path = str(tmp_dir / "val.target")
        _dump_articles(input_file_name, text["en"])
        _dump_articles(reference_path, text["de"])
        task = "translation_en_to_de" if model == T5_TINY else "summarization"
        testargs = f"""
            run_eval_search.py
            {model}
            {str(input_file_name)}
            {str(output_file_name)}
            --score_path {score_path}
            --reference_path {reference_path}
            --task {task}
            """.split()
        testargs.extend(["--search", "num_beams=1:2 length_penalty=0.9:1.0"])

        with patch.object(sys, "argv", testargs):
            with CaptureStdout() as cs:
                run_search()
            expected_strings = [" num_beams | length_penalty", model, "Best score args"]
            un_expected_strings = ["Info"]
            if "translation" in task:
                expected_strings.append("bleu")
            else:
                expected_strings.extend(ROUGE_KEYS)
            for w in expected_strings:
                assert w in cs.out
            for w in un_expected_strings:
                assert w not in cs.out
            assert Path(output_file_name).exists()


SUMMARY:
**Class Summary**

`TestTheRest` is a unit test class ( subclass of `TestCasePlus`) designed to evaluate the performance of two machine learning models, T5_TINY and MBART_TINY. The primary purpose is to assess their translation (T5) and summarization (MBART) capabilities.

**Key Implementation Details**

*   The `run_eval_tester` method:
    *   Runs evaluation tests using `run_generate()` function.
    *   Uses the `patch_object` system call to replace the `sys.argv` with test arguments, including the model name, input file name, output file name, and scores path.
    *   Verifies that the generated output exists after the evaluation process.
*   The `test_run_eval_search` method:
    *   Tests both translation (T5) and summarization (MBART) functionalities using a shared input file for both cases.
    *   Extends the test arguments with additional flags, such as specifying search parameters like `"num_beams=1:2 length_penalty=0.9:1.0"`.
    *   Verifies that specific strings are present in the output and expected strings.

**Notable Dependencies or Requirements**

*   The class relies on the `sys` module for system-related operations.
*   It uses the `run_generate()` function, which is not shown in the provided code snippet.
*   The test methods use the `parameterized.expand` decorator to test multiple models (T5_TINY and MBART_TINY) with different tasks.
*   The class requires external dependencies:
    *   (`BART_TINY`, `MBART_TINY`, `T5_TINY`) - Machine learning model classes.
    *   (`run_generate()`,  `_dump_articles()`) - External functions or modules for evaluation and article dumping.

**Code Organization and Context**

The provided code snippet appears to be part of a larger testing framework, with the test class containing several methods that interact with machine learning models. The context suggests that this code is used for evaluating the performance of these models on specific tasks, which could involve language translation and summarization.


key: chroma:document
string_value:
CODE:

def save_len_file(
    tokenizer_name, data_dir, max_source_length=1024, max_target_length=1024, consider_target=False, **kwargs
):
    """Save max(src_len, tgt_len) for each example to allow dynamic batching."""
    tok = AutoTokenizer.from_pretrained(tokenizer_name)
    train_ds = Seq2SeqDataset(tok, data_dir, max_source_length, max_target_length, type_path="train", **kwargs)
    pad = tok.pad_token_id

    def get_lens(ds):
        dl = tqdm(
            DataLoader(ds, batch_size=512, num_workers=8, shuffle=False, collate_fn=ds.collate_fn),
            desc=str(ds.len_file),
        )
        max_lens = []
        for batch in dl:
            src_lens = batch["input_ids"].ne(pad).sum(1).tolist()
            tgt_lens = batch["labels"].ne(pad).sum(1).tolist()
            if consider_target:
                for src, tgt in zip(src_lens, tgt_lens):
                    max_lens.append(max(src, tgt))
            else:
                max_lens.extend(src_lens)
        return max_lens

    train_lens = get_lens(train_ds)
    val_ds = Seq2SeqDataset(tok, data_dir, max_source_length, max_target_length, type_path="val", **kwargs)
    val_lens = get_lens(val_ds)
    pickle_save(train_lens, train_ds.len_file)


SUMMARY:
**Function Summary: `save_len_file`**

**Purpose:** Saves the maximum source and target sequence lengths for each example in a dataset to allow dynamic batching.

**Key Implementation Details:**

* Uses `AutoTokenizer` from Hugging Face to load a pre-trained tokenizer.
* Defines a helper function `get_lens` to calculate the maximum lengths of input sequences using `DataLoader`.
* Calculates the maximum lengths for training and validation datasets separately.

**Notable Dependencies/Interfaces:**

* `Seq2SeqDataset`: a custom dataset class used in this implementation.
* `AutoTokenizer`: a pre-trained tokenizer from Hugging Face.
* `DataLoader`: an iterable dataset object with batching capabilities.

**External Dependencies:**

* `tqdm` for display progress bars.
* `pickle` for saving data.

This code aims to optimize batch processing by storing the maximum sequence lengths for each example in the dataset, allowing for more efficient dynamic batching.


key: chroma:document
string_value:
CODE:

def pack_examples(tok, src_examples, tgt_examples, max_tokens=1024):
    finished_src, finished_tgt = [], []

    sorted_examples = list(zip(src_examples, tgt_examples))
    new_src, new_tgt = sorted_examples[0]

    def is_too_big(strang):
        return tok(strang, return_tensors="pt").input_ids.shape[1] > max_tokens

    for src, tgt in tqdm(sorted_examples[1:]):
        cand_src = new_src + " " + src
        cand_tgt = new_tgt + " " + tgt
        if is_too_big(cand_src) or is_too_big(cand_tgt):  # cant fit, finalize example
            finished_src.append(new_src)
            finished_tgt.append(new_tgt)
            new_src, new_tgt = src, tgt
        else:  # can fit, keep adding
            new_src, new_tgt = cand_src, cand_tgt

    # cleanup
    if new_src:
        assert new_tgt
        finished_src.append(new_src)
        finished_tgt.append(new_tgt)


SUMMARY:
**Function Summary: `pack_examples`**

**Purpose:** This function takes input examples for source and target languages, sorts them, and packs them into a format suitable for tokenization by storing their context.

**Key Functionality:**

*   Takes four inputs: `tok`, `src_examples`, `tgt_examples`, and optional `max_tokens`.
*   Splits `src_examples` and `tgt_examples` into pairs using `sorted_examples = list(zip(src_examples, tgt_examples))`.
*   Initializes two lists, `new_src` and `new_tgt`, with the first pair of examples (`sorted_examples[0]`).
*   Defines a function `is_too_big(strang)` to check if the input string exceeds a maximum token limit using the `tok` function.
*   Iterates over the remaining pairs of examples in ascending order, creating new combined strings by appending each source and target token to the previous ones (`cand_src`, `cand_tgt`). 
*   Checks if these combined strings are too big using `is_too_big()`. If so, they're appended to separate result lists (`finished_src` and `finished_tgt`) and the iteration is reopened with the previous example's value.
*   Rebuilds the combined strings only when they fit within the maximum token limit.

**Notable Dependencies/Requirements:**

*   `tok`: a function to tokenize input strings (e.g., using Hugging Face's ` Tokenizer` class)
*   `return_tensors="pt"` for specifying data format for PyTorch tensors
*   `tqdm`: for displaying progress during iteration


key: chroma:document
string_value:
CODE:

def pack_data_dir(tok, data_dir: Path, max_tokens, save_path):
    save_path = Path(save_path)
    save_path.mkdir(exist_ok=True)
    for split in ["train"]:
        src_path, tgt_path = data_dir / f"{split}.source", data_dir / f"{split}.target"
        src_docs = [x.rstrip() for x in Path(src_path).open().readlines()]
        tgt_docs = [x.rstrip() for x in Path(tgt_path).open().readlines()]
        packed_src, packed_tgt = pack_examples(tok, src_docs, tgt_docs, max_tokens)
        print(f"packed {split} split from {len(src_docs)} examples -> {len(packed_src)}.")
        Path(save_path / f"{split}.source").open("w").write("\n".join(packed_src))
        Path(save_path / f"{split}.target").open("w").write("\n".join(packed_tgt))
    for split in ["val", "test"]:
        src_path, tgt_path = data_dir / f"{split}.source", data_dir / f"{split}.target"
        shutil.copyfile(src_path, save_path / f"{split}.source")


SUMMARY:
**Function Summary:** `pack_data_dir`

**Purpose and Functionality:**
This function packs data directory contents into processed text examples. It takes in:
- Tokenizer (`tok`)
- Data directory path (`data_dir`) with three splits: train, val, and test
- Maximum number of tokens (`max_tokens`)
- Output save path (`save_path`)

**Key Implementation Details:**

* Uses `pack_examples` function to process data
* Creates processed source and target text files for each split
* Saves processed examples in the save directory

**Notable Dependencies and Requirements:**

* `shutil` module for file copying
* `pathlib` library for file operations (e.g., `Path.mkdir`, `Path.open`)
* External dependencies:
	+ `tokenization` library (not explicitly mentioned, but assumed to be used by `pack_examples` function)
	+ `tokenizer` instance (`tok`)


key: chroma:document
string_value:
CODE:

def packer_cli():
    parser = argparse.ArgumentParser()
    parser.add_argument("--tok_name", type=str, help="like facebook/bart-large-cnn,google-t5/t5-base, etc.")
    parser.add_argument("--max_seq_len", type=int, default=128)
    parser.add_argument("--data_dir", type=str)
    parser.add_argument("--save_path", type=str)
    args = parser.parse_args()
    tokenizer = AutoTokenizer.from_pretrained(args.tok_name)


SUMMARY:
**Packer CLI Function Summary**

### Main Purpose and Functionality

This function implements a command-line interface (CLI) using the `argparse` library to parse input arguments. It initializes an argument parser and adds several key inputs:

*   `tok_name`: specifies the pre-trained language model tokenizer (e.g., "facebook/bart-large-cnn")
*   `max_seq_len`: sets the maximum sequence length for output data (default: 128)
*   `data_dir`: specifies the input directory containing data to be processed
*   `save_path`: sets the output path for saved results

The function then uses these arguments to:

1.  Load a pre-trained tokenizer using Hugging Face's `AutoTokenizer` from the specified `tok_name`.
2.  Pass the tokenizer, `data_dir`, `max_seq_len`, and `save_path` to another function (`pack_data_dir`) for data processing.

**Key Implementation Details**

*   Uses `argparse.ArgumentParser` for CLI argument parsing
*   Specifies key input arguments using `add_argument()`
*   Initializes a pre-trained tokenizer from the specified `tok_name`

**Notable Dependencies or Requirements**

*   **Hugging Face Transformers**: For accessing pre-trained language models and their associated tokenizers.
*   **argparse**: A built-in Python library for parsing command-line arguments.

### Code Structure

```
# Define the packer_cli function
def packer_cli():
    # Initialize argument parser
    ...
    
    # Load pre-trained tokenizer
    tokenizer = AutoTokenizer.from_pretrained(args.tok_name)
    
    # Call another function to process data
    return pack_data_dir(tokenizer, Path(args.data_dir), args.max_seq_len, args.save_path)

# Use the packer_cli function as the entry point for CLI execution
```

This summary provides a concise overview of the `packer_cli` function, highlighting its main purpose and key implementation details, while also noting its dependencies and structure.


key: chroma:document
string_value:
CODE:

def handle_metrics(split, metrics, output_dir):
    """
    Log and save metrics

    Args:
    - split: one of train, val, test
    - metrics: metrics dict
    - output_dir: where to save the metrics
    """

    logger.info(f"***** {split} metrics *****")
    for key in sorted(metrics.keys()):
        logger.info(f"  {key} = {metrics[key]}")


SUMMARY:
**Function Summary: handle_metrics**

### Purpose and Functionality
The `handle_metrics` function logs and saves metrics for training, validation, or testing splits. It takes in metrics dictionaries, their corresponding split name, and an output directory.

### Key Implementation Details

* **Logging**: Inserts a banner message indicating the current split's metric results and logs each key-value pair from the `metrics` dictionary.
* **Saving to JSON**: Saves the `metrics` dictionary to a file named after the current split (e.g., `_results.json`) in the specified `output_dir`.

### Notable Dependencies or Requirements

* `logger`: Used for logging
* `os.path.join()`: Used to construct the output file path
* `save_json(func, ...)`: Used to save the JSON-serialized metrics dictionary


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, Seq2SeqTrainingArguments))

    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    check_output_dir(training_args)

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO if training_args.local_rank in [-1, 0] else logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        training_args.local_rank,
        training_args.device,
        training_args.n_gpu,
        bool(training_args.parallel_mode == ParallelMode.DISTRIBUTED),
        training_args.fp16,
    )
    transformers.utils.logging.enable_default_handler()
    transformers.utils.logging.enable_explicit_format()
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
    logger.info("Training/evaluation parameters %s", training_args)

    # Set seed
    set_seed(training_args.seed)

    # Load pretrained model and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.

    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
    )

    extra_model_params = ("encoder_layerdrop", "decoder_layerdrop", "dropout", "attention_dropout")
    for p in extra_model_params:
        if getattr(training_args, p, None):
            assert hasattr(config, p), f"({config.__class__.__name__}) doesn't have a `{p}` attribute"
            setattr(config, p, getattr(training_args, p))

    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
    )
    model = AutoModelForSeq2SeqLM.from_pretrained(
        model_args.model_name_or_path,
        from_tf=".ckpt" in model_args.model_name_or_path,
        config=config,
        cache_dir=model_args.cache_dir,
    )

    # use task specific params
    use_task_specific_params(model, data_args.task)

    # set num_beams for evaluation
    if data_args.eval_beams is None:
        data_args.eval_beams = model.config.num_beams

    # set decoder_start_token_id for MBart
    if model.config.decoder_start_token_id is None and isinstance(tokenizer, (MBartTokenizer, MBartTokenizerFast)):
        assert (
            data_args.tgt_lang is not None and data_args.src_lang is not None
        ), "mBart requires --tgt_lang and --src_lang"
        if isinstance(tokenizer, MBartTokenizer):
            model.config.decoder_start_token_id = tokenizer.lang_code_to_id[data_args.tgt_lang]
        else:
            model.config.decoder_start_token_id = tokenizer.convert_tokens_to_ids(data_args.tgt_lang)

    if model_args.freeze_embeds:
        freeze_embeds(model)
    if model_args.freeze_encoder:
        freeze_params(model.get_encoder())
        assert_all_frozen(model.get_encoder())

    dataset_class = Seq2SeqDataset

    # Get datasets
    train_dataset = (
        dataset_class(
            tokenizer,
            type_path="train",
            data_dir=data_args.data_dir,
            n_obs=data_args.n_train,
            max_target_length=data_args.max_target_length,
            max_source_length=data_args.max_source_length,
            prefix=model.config.prefix or "",
        )
        if training_args.do_train
        else None
    )
    eval_dataset = (
        dataset_class(
            tokenizer,
            type_path="val",
            data_dir=data_args.data_dir,
            n_obs=data_args.n_val,
            max_target_length=data_args.val_max_target_length,
            max_source_length=data_args.max_source_length,
            prefix=model.config.prefix or "",
        )
        if training_args.do_eval or training_args.eval_strategy != EvaluationStrategy.NO
        else None
    )
    test_dataset = (
        dataset_class(
            tokenizer,
            type_path="test",
            data_dir=data_args.data_dir,
            n_obs=data_args.n_test,
            max_target_length=data_args.test_max_target_length,
            max_source_length=data_args.max_source_length,
            prefix=model.config.prefix or "",
        )
        if training_args.do_predict
        else None
    )

    # Initialize our Trainer
    compute_metrics_fn = (
        build_compute_metrics_fn(data_args.task, tokenizer) if training_args.predict_with_generate else None
    )
    trainer = Seq2SeqTrainer(
        model=model,
        args=training_args,
        data_args=data_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        data_collator=Seq2SeqDataCollator(
            tokenizer, data_args, model.config.decoder_start_token_id, training_args.tpu_num_cores
        ),
        compute_metrics=compute_metrics_fn,
        processing_class=tokenizer,
    )

    all_metrics = {}
    # Training
    if training_args.do_train:
        logger.info("*** Train ***")

        train_result = trainer.train(
            model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
        )
        metrics = train_result.metrics
        metrics["train_n_objs"] = data_args.n_train

        trainer.save_model()  # this also saves the tokenizer

        if trainer.is_world_process_zero():
            handle_metrics("train", metrics, training_args.output_dir)
            all_metrics.update(metrics)

            # Need to save the state, since Trainer.save_model saves only the tokenizer with the model
            trainer.state.save_to_json(os.path.join(training_args.output_dir, "trainer_state.json"))

            # For convenience, we also re-save the tokenizer to the same directory,
            # so that you can share your model easily on huggingface.co/models =)
            tokenizer.save_pretrained(training_args.output_dir)

    # Evaluation
    if training_args.do_eval:
        logger.info("*** Evaluate ***")

        metrics = trainer.evaluate(metric_key_prefix="val")
        metrics["val_n_objs"] = data_args.n_val
        metrics["val_loss"] = round(metrics["val_loss"], 4)

        if trainer.is_world_process_zero():
            handle_metrics("val", metrics, training_args.output_dir)
            all_metrics.update(metrics)

    if training_args.do_predict:
        logger.info("*** Predict ***")

        test_output = trainer.predict(test_dataset=test_dataset, metric_key_prefix="test")
        metrics = test_output.metrics
        metrics["test_n_objs"] = data_args.n_test

        if trainer.is_world_process_zero():
            metrics["test_loss"] = round(metrics["test_loss"], 4)
            handle_metrics("test", metrics, training_args.output_dir)
            all_metrics.update(metrics)

            if training_args.predict_with_generate:
                test_preds = tokenizer.batch_decode(
                    test_output.predictions, skip_special_tokens=True, clean_up_tokenization_spaces=True
                )
                test_preds = lmap(str.strip, test_preds)
                write_txt_file(test_preds, os.path.join(training_args.output_dir, "test_generations.txt"))

    if trainer.is_world_process_zero():
        save_json(all_metrics, os.path.join(training_args.output_dir, "all_results.json"))



SUMMARY:
**Summary**

This code defines the main training script for sequence-to-sequence models using the Hugging Face Transformers library. It sets up logging, loading pre-trained models and tokenizers, creating datasets, initializing a trainer, and performing training, evaluation, and prediction steps.

**Primary Purpose**

The primary purpose of this script is to train sequence-to-sequence models on Hugging Face's framework.

**Main Functionality**

* Loads pre-trained model and tokenizer
* Creates datasets for training, evaluation, and testing
* Initializes the trainer with the loaded model, arguments, and dataset
* Performs training, evaluation, and prediction steps using the trainer

**Key Implementation Details**

* Uses `HfArgumentParser` to parse command-line arguments into classes
* Sets up logging with format, level, and logger
* Loads pre-trained models and tokenizers using `Trainer.load_model`
* Creates datasets using `Seq2SeqDataset` class
* Initializes the trainer with loaded model, arguments, and dataset
* Performs training using `trainer.train`

**Output**

The script saves metrics to a JSON file (`all_results.json`) in the output directory. It also saves the state of the trainer as a JSON file (`trainer_state.json`).


key: chroma:document
string_value:
CODE:

def _mp_fn(index):
    # For xla_spawn (TPUs)


SUMMARY:
**Summary**

 `_mp_fn` is a function that appears to be the entry point for a multi-threading or parallel processing system, likely used with Google's Accelerated Linear Scheduling Algorithm (XLA) and TPUs (Tensor Processing Units). The primary purpose of this function is to execute a `main()` method when running on an XLA-enabled environment.

**Key Details**

* `index` parameter, but no further context provided
* Call to `main()` without passing any arguments, suggesting a default or wrapper function

**Notable Dependencies**

* None explicitly mentioned, but likely relies on the XLA library and TPUs for hardware acceleration.


key: chroma:document
string_value:
CODE:

def download_wmt_dataset(src_lang="ro", tgt_lang="en", dataset="wmt16", save_dir=None) -> None:
    """Download a dataset using the datasets package and save it to the format expected by finetune.py
    Format of save_dir: train.source, train.target, val.source, val.target, test.source, test.target.

    Args:
        src_lang: <str> source language
        tgt_lang: <str> target language
        dataset: <str> wmt16, wmt17, etc. wmt16 is a good start as it's small. To get the full list run `import datasets; print([d.id for d in datasets.list_datasets() if "wmt" in d.id])`
        save_dir: <str>, where to save the datasets, defaults to f'{dataset}-{src_lang}-{tgt_lang}'

    Usage:
        >>> download_wmt_dataset('ro', 'en', dataset='wmt16') # saves to wmt16-ro-en
    """
    try:
        import datasets
    except (ModuleNotFoundError, ImportError):
        raise ImportError("run pip install datasets")
    pair = f"{src_lang}-{tgt_lang}"
    print(f"Converting {dataset}-{pair}")
    ds = datasets.load_dataset(dataset, pair)
    if save_dir is None:
        save_dir = f"{dataset}-{pair}"
    save_dir = Path(save_dir)
    save_dir.mkdir(exist_ok=True)

    for split in ds.keys():
        print(f"Splitting {split} with {ds[split].num_rows} records")

        # to save to val.source, val.target like summary datasets
        fn = "val" if split == "validation" else split
        src_path = save_dir.joinpath(f"{fn}.source")
        tgt_path = save_dir.joinpath(f"{fn}.target")
        src_fp = src_path.open("w+")
        tgt_fp = tgt_path.open("w+")

        # reader is the bottleneck so writing one record at a time doesn't slow things down
        for x in tqdm(ds[split]):
            ex = x["translation"]
            src_fp.write(ex[src_lang] + "\n")
            tgt_fp.write(ex[tgt_lang] + "\n")



SUMMARY:
**Summary:**

The `download_wmt_dataset` function downloads a Word Translation (WMT) dataset from the datasets package and saves it in a specified format. The main functionality includes:

* Downloading the dataset using the name of the source language (`src_lang`) and target language (`tgt_lang`)
* Converting the dataset to save it in the format `train.source`, `train.target`, etc.
* Saving the dataset to a specified directory or creating a new one if not provided

**Key Implementation Details:**

* Uses the `datasets.load_dataset` method to download the WMT dataset
* Utilizes the `tqdm` library for progress bars
* Saves the translations in source and target languages using separate files (`src_path` and `tgt_path`)
* Writes records from the dataset into these files one at a time to avoid slowing down processing

**Notable Dependencies and Requirements:**

* The datasets package
* The Path module from collections.abc
* The tqdm library for progress bars


key: chroma:document
string_value:
CODE:

def calculate_rouge_path(pred_path, tgt_path, save_path=None, **kwargs):
    """Kwargs will be passed to calculate_rouge"""
    pred_lns = [x.strip() for x in open(pred_path).readlines()]
    tgt_lns = [x.strip() for x in open(tgt_path).readlines()][: len(pred_lns)]
    metrics = calculate_rouge(pred_lns, tgt_lns, **kwargs)
    if save_path is not None:
        save_json(metrics, save_path, indent=None)


SUMMARY:
**Summary**

`calculate_rouge_path` function calculates ROUGE scores between predicted and target paths.

**Functionality**

1. **Loads Paths**: Reads `pred_path` and `tgt_path` sequences, trimming whitespace.
2. **Calculates ROUGE Metrics**: Passes loaded sequences to `calculate_rouge` with additional keyword arguments (`**kwargs`).
3. **Optional Saving**: Saves calculated metrics to a JSON file at `save_path`, if provided.

**Key Implementation Details**

* Uses `calculate_rouge` function from external module (`calculate_rouge`)
* Imports necessary classes/functions (e.g., `open`, `sav_json`) that are not directly involved in the main functionality

**Dependencies/Requirements**

* External module: `calculate_rouge`
* Python modules: `json`

No explicit interface or class inheritance noted.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/legacy/seq2seq/test_data/fsmt/build-eval-data.py


key: chroma:document
string_value:
CODE:

def get_all_data(pairs, n_objs):
    text = {}
    for src, tgt in pairs:
        pair = f"{src}-{tgt}"
        cmd = f"sacrebleu -t wmt19 -l {pair} --echo src".split()
        src_lines = subprocess.run(cmd, stdout=subprocess.PIPE).stdout.decode("utf-8").splitlines()
        cmd = f"sacrebleu -t wmt19 -l {pair} --echo ref".split()
        tgt_lines = subprocess.run(cmd, stdout=subprocess.PIPE).stdout.decode("utf-8").splitlines()
        text[pair] = {"src": src_lines[:n_objs], "tgt": tgt_lines[:n_objs]}


SUMMARY:
**Summary**

### `get_all_data` Function

#### Purpose and Functionality
This function retrieves translation data from Sacrebleu using the WMT19 test set. It takes in pairs of source-target language codes (`pairs`) and the number of objects to retrieve (`n_objs`). The function returns a dictionary (`text`) containing translations for each pair, with sources and targets limited to `n_objs` lines.

#### Key Implementation Details

* Uses Sacrebleu command-line tools (`sacrebleu -t wmt19 ...`) to fetch translation data.
* Pipes output through `split()` to process as text input.
* Utilizes `subprocess.run()` to execute commands and capture output.

#### Dependencies and Requirements
* `sacrebleu` command-line tool
* `python` and `subprocess` modules (built-in)

### Code Signature

```python
def get_all_data(pairs: list, n_objs: int) -> dict:
```

This summary provides a focused analysis of the code's functionality, key implementation details, and dependencies. It preserves exact class/function/variable names as closely as possible to the original code.


