# embedding_metadata

key: chroma:document
string_value:
CODE:

def load_labels(objs=OBJECTS, attrs=ATTRIBUTES):
    vg_classes = []
    with open(objs) as f:
        for object in f.readlines():
            vg_classes.append(object.split(",")[0].lower().strip())

    vg_attrs = []
    with open(attrs) as f:
        for object in f.readlines():
            vg_attrs.append(object.split(",")[0].lower().strip())


SUMMARY:
**Code Summary**

`load_labels` function loads and processes objects from two CSV files: `objs` and `attrs`. It reads each file line by line, extracts the first column (stripped of leading/trailing whitespace), converts to lowercase, and appends to respective lists (`vg_classes` and `vocab_attrs`). The function returns both lists.

**Key Implementation Details**

* Extracts first column of each file into `vg_classes` and `vocab_attrs`
* Converts extracted values to lowercase
* Uses list append for data aggregation

**Notable Dependencies or Requirements**

* External dependencies:
	+ CSV parsing (assumed by file operations)
* Interface/Class usage:
	+ None, as no interfaces/classes are used in the function.


key: chroma:document
string_value:
CODE:

def load_checkpoint(ckp):
    r = OrderedDict()
    with open(ckp, "rb") as f:
        ckp = pkl.load(f)["model"]
    for k in copy.deepcopy(list(ckp.keys())):
        v = ckp.pop(k)
        if isinstance(v, np.ndarray):
            v = torch.tensor(v)
        else:
            assert isinstance(v, torch.tensor), type(v)
        r[k] = v


SUMMARY:
**Function Summary: `load_checkpoint`**

* **Purpose:** Loads a TensorFlow model checkpoint from a file and reconstructs its attributes as a Python dictionary (`OrderedDict`).
 
* **Key Implementation Details:**
	+ Loads file at path `ckp` using `pkl.load`.
	+ Iterates over model's attributes, converting numpy arrays to PyTorch tensors with `torch.tensor`.
 
* **Notable Dependencies/Requirements:**
	+ `pkl`: Python pickle library for loading checkpoint data.
	+ `torch.tensor`: Converts numpy arrays to PyTorch tensors.

Note: `OrderedDict` is used to preserve the original order of attributes. 

**Additional Context:** This function appears to be part of a larger project that leverages TensorFlow and PyTorch for model training and deployment. The code assumes familiarity with these libraries, as well as basic understanding of Python data structures and file I/O.


key: chroma:document
string_value:
CODE:

class Config:
    _pointer = {}

    def __init__(self, dictionary: dict, name: str = "root", level=0):
        self._name = name
        self._level = level
        d = {}
        for k, v in dictionary.items():
            if v is None:
                raise ValueError()
            k = copy.deepcopy(k)
            v = copy.deepcopy(v)
            if isinstance(v, dict):
                v = Config(v, name=k, level=level + 1)
            d[k] = v
            setattr(self, k, v)

        self._pointer = d

    def __repr__(self):
        return str(list((self._pointer.keys())))

    def __setattr__(self, key, val):
        self.__dict__[key] = val
        self.__dict__[key.upper()] = val
        levels = key.split(".")
        last_level = len(levels) - 1
        pointer = self._pointer
        if len(levels) > 1:
            for i, l in enumerate(levels):
                if hasattr(self, l) and isinstance(getattr(self, l), Config):
                    setattr(getattr(self, l), ".".join(levels[i:]), val)
                if l == last_level:
                    pointer[l] = val
                else:
                    pointer = pointer[l]

    def to_dict(self):
        return self._pointer

    def dump_yaml(self, data, file_name):
        with open(f"{file_name}", "w") as stream:
            dump(data, stream)

    def dump_json(self, data, file_name):
        with open(f"{file_name}", "w") as stream:
            json.dump(data, stream)

    @staticmethod
    def load_yaml(config):
        with open(config) as stream:
            data = load(stream, Loader=Loader)
        return data

    def __str__(self):
        t = "    "
        if self._name != "root":
            r = f"{t * (self._level-1)}{self._name}:\n"
        else:
            r = ""
        level = self._level
        for i, (k, v) in enumerate(self._pointer.items()):
            if isinstance(v, Config):
                r += f"{t * (self._level)}{v}\n"
                self._level += 1
            else:
                r += f"{t * (self._level)}{k}: {v} ({type(v).__name__})\n"
            self._level = level
        return r[:-1]

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs):
        config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
        return cls(config_dict)

    @classmethod
    def get_config_dict(cls, pretrained_model_name_or_path: str, **kwargs):
        cache_dir = kwargs.pop("cache_dir", None)
        force_download = kwargs.pop("force_download", False)
        resume_download = kwargs.pop("resume_download", False)
        proxies = kwargs.pop("proxies", None)
        local_files_only = kwargs.pop("local_files_only", False)

        if os.path.isdir(pretrained_model_name_or_path):
            config_file = os.path.join(pretrained_model_name_or_path, CONFIG_NAME)
        elif os.path.isfile(pretrained_model_name_or_path) or is_remote_url(pretrained_model_name_or_path):
            config_file = pretrained_model_name_or_path
        else:
            config_file = hf_bucket_url(pretrained_model_name_or_path, filename=CONFIG_NAME, use_cdn=False)

        try:
            # Load from URL or cache if already cached
            resolved_config_file = cached_path(
                config_file,
                cache_dir=cache_dir,
                force_download=force_download,
                proxies=proxies,
                resume_download=resume_download,
                local_files_only=local_files_only,
            )
            # Load config dict
            if resolved_config_file is None:
                raise EnvironmentError

            config_file = Config.load_yaml(resolved_config_file)

        except EnvironmentError:
            msg = "Can't load config for"
            raise EnvironmentError(msg)

        if resolved_config_file == config_file:
            print("loading configuration file from path")
        else:
            print("loading configuration file cache")



SUMMARY:
**Class Summary:**

`Config` is a hierarchical data structure used to represent configuration files.

**Purpose:**
The primary purpose of this class is to create a nested dictionary-like data structure from a flat dictionary, which can be easily serialized and deserialized in YAML or JSON formats.

**Key Implementation Details:**

* **Initialization**: The `__init__` method creates a new `Config` instance by iterating over the input dictionary and recursively creating child configurations for nested values.
* **Serialization/Deserialization**: The `dump_yaml`, `dump_json`, and `load_yaml` methods handle serialization and deserialization of the `Config` data structure to YAML and JSON formats, respectively.

**Notable Dependencies:**

* `dataclasses`: used for defining the class structure
* `os`, `pathlib`, `sh Util`: used for file system operations (e.g., loading/downloading configuration files)
* `pyyaml`: included as a dependency in some cases (e.g., when loading configuration files from URLs)
* `huggingface` dependencies: utilized by the `from_pretrained` class method to load pre-trained models and their corresponding configurations.

**Inherited/Implemented Interfaces:**

* None explicit inherited interfaces; however, some classes use the `__init_subclass__` mechanism to register the `Config` class with a registry system.


key: chroma:document
string_value:
CODE:
# quick compare tensors
def compare(in_tensor):
    out_tensor = torch.load("dump.pt", map_location=in_tensor.device)
    n1 = in_tensor.numpy()
    n2 = out_tensor.numpy()[0]
    print(n1.shape, n1[0, 0, :5])
    print(n2.shape, n2[0, 0, :5])
    assert np.allclose(n1, n2, rtol=0.01, atol=0.1), (
        f"{sum([1 for x in np.isclose(n1, n2, rtol=0.01, atol=0.1).flatten() if x is False])/len(n1.flatten())*100:.4f} %"
        " element-wise mismatch"
    )
    raise Exception("tensors are all good")



SUMMARY:
**Function Summary: `compare`**

Compare two tensors, loading one from a file and comparing their contents.

**Main Purpose and Functionality:**
This function compares the contents of two tensors, specifically checking if their numeric values match within a specified tolerance. It loads an existing tensor (`out_tensor`) from a file (`dump.pt`) and compares it with input tensor (`in_tensor`).

* Loads `out_tensor` from `dump.pt` using `torch.load`
* Extracts first element of both tensors as NumPy arrays (`n1` and `n2`)
* Prints shapes and first 5 elements of each array
* Checks if `n1` is element-wise equal to `n2`, raising an exception if not

**Key Implementation Details:**

* Uses `torch.load` to load tensor from file, specifying device mapping.
* Utilizes `np.allclose` for element-wise comparison with custom tolerances.

**Notable Dependencies or Requirements:**

* Uses `torch` and its `load` function
* Leverages `numpy` library for NumPy array operations
* Requires Hugging Face's dependency for the file loading (no direct library is imported)


key: chroma:document
string_value:
CODE:

def is_remote_url(url_or_filename):
    parsed = urlparse(url_or_filename)


SUMMARY:
**Summary:**

This function, `is_remote_url`, determines if a provided URL or filename contains an HTTP or HTTPS scheme. It utilizes the `urlparse` function to parse the input and checks if the resulting `scheme` attribute is one of the two specified values.

**Key Details:**

* Function signature: `def is_remote_url(url_or_filename):`
* Uses `urlparse` from the `urllib.parse` module
* Checks for `http` or `https` schemes in the parsed URL

**Notable Dependencies:**

* `urllib.parse`: module used to parse URLs
* No external dependencies listed; assumes standard library availability.


key: chroma:document
string_value:
CODE:

def hf_bucket_url(model_id: str, filename: str, use_cdn=True) -> str:
    endpoint = CLOUDFRONT_DISTRIB_PREFIX if use_cdn else S3_BUCKET_PREFIX
    legacy_format = "/" not in model_id
    if legacy_format:
        return f"{endpoint}/{model_id}-{filename}"
    else:


SUMMARY:
**Summary**

The `hf_bucket_url` function generates a URL for storing file metadata and contents in a cloud-based bucket. It takes three inputs:

* `model_id`: a unique string identifier
* `filename`: the name of the file to be stored
* `use_cdn`: a boolean flag indicating whether to use the CloudFront distribution prefix (true) or S3 bucket prefix (false)

The function returns a URL in the format `${endpoint}/${model_id}/${filename}` if the model ID does not contain a forward slash (`/`).

**Key Implementation Details**

- Uses `CLOUDFRONT_DISTRIB_PREFIX` and `S3_BUCKET_PREFIX` constants for end-point configuration.
- Returns URLs with or without a trailing forward slash, depending on `legacy_format`.

**Notable Dependencies or Requirements**

* `CLOUDFRONT_DISTRIB_PREFIX`
* `S3_BUCKET_PREFIX`


key: chroma:document
string_value:
CODE:

def http_get(
    url,
    temp_file,
    proxies=None,
    resume_size=0,
    user_agent=None,
):
    ua = "python/{}".format(sys.version.split()[0])
    if _torch_available:
        ua += "; torch/{}".format(torch.__version__)
    if isinstance(user_agent, dict):
        ua += "; " + "; ".join("{}/{}".format(k, v) for k, v in user_agent.items())
    elif isinstance(user_agent, str):
        ua += "; " + user_agent
    headers = {"user-agent": ua}
    if resume_size > 0:
        headers["Range"] = "bytes=%d-" % (resume_size,)
    response = requests.get(url, stream=True, proxies=proxies, headers=headers)
    if response.status_code == 416:  # Range not satisfiable
        return
    content_length = response.headers.get("Content-Length")
    total = resume_size + int(content_length) if content_length is not None else None
    progress = tqdm(
        unit="B",
        unit_scale=True,
        total=total,
        initial=resume_size,
        desc="Downloading",
    )
    for chunk in response.iter_content(chunk_size=1024):
        if chunk:  # filter out keep-alive new chunks
            progress.update(len(chunk))
            temp_file.write(chunk)


SUMMARY:
**Function Summary**

`http_get` makes an HTTP GET request with optional resumable download and progress tracking.

**Key Implementation Details**

* Uses `requests.get` with `stream=True` for efficient chunked downloads
* Sets User-Agent header to include version information (PyTorch if available)
* Checks for Range not satisfiable responses and returns early
* Calculates total content length to resume download from previous point

**Notable Dependencies/Requirements**

* `_torch_available`
* `requests`
* `tqdm` (progress bar library)

**Primary Purpose**
Performs an HTTP GET request with optional resumable download capabilities, allowing for resumed downloads of large files by tracking progress.


key: chroma:document
string_value:
CODE:

def get_from_cache(
    url,
    cache_dir=None,
    force_download=False,
    proxies=None,
    etag_timeout=10,
    resume_download=False,
    user_agent=None,
    local_files_only=False,
):
    if cache_dir is None:
        cache_dir = TRANSFORMERS_CACHE
    if isinstance(cache_dir, Path):
        cache_dir = str(cache_dir)

    os.makedirs(cache_dir, exist_ok=True)

    etag = None
    if not local_files_only:
        try:
            response = requests.head(url, allow_redirects=True, proxies=proxies, timeout=etag_timeout)
            if response.status_code == 200:
                etag = response.headers.get("ETag")
        except (EnvironmentError, requests.exceptions.Timeout):
            # etag is already None
            pass

    filename = url_to_filename(url, etag)

    # get cache path to put the file
    cache_path = os.path.join(cache_dir, filename)

    # etag is None = we don't have a connection, or url doesn't exist, or is otherwise inaccessible.
    # try to get the last downloaded one
    if etag is None:
        if os.path.exists(cache_path):
            return cache_path
        else:
            matching_files = [
                file
                for file in fnmatch.filter(os.listdir(cache_dir), filename + ".*")
                if not file.endswith(".json") and not file.endswith(".lock")
            ]
            if len(matching_files) > 0:
                return os.path.join(cache_dir, matching_files[-1])
            else:
                # If files cannot be found and local_files_only=True,
                # the models might've been found if local_files_only=False
                # Notify the user about that
                if local_files_only:
                    raise ValueError(
                        "Cannot find the requested files in the cached path and outgoing traffic has been"
                        " disabled. To enable model look-ups and downloads online, set 'local_files_only'"
                        " to False."
                    )
                return None

    # From now on, etag is not None.
    if os.path.exists(cache_path) and not force_download:
        return cache_path

    # Prevent parallel downloads of the same file with a lock.
    lock_path = cache_path + ".lock"
    with FileLock(lock_path):
        # If the download just completed while the lock was activated.
        if os.path.exists(cache_path) and not force_download:
            # Even if returning early like here, the lock will be released.
            return cache_path

        if resume_download:
            incomplete_path = cache_path + ".incomplete"

            @contextmanager
            def _resumable_file_manager():
                with open(incomplete_path, "a+b") as f:
                    yield f

            temp_file_manager = _resumable_file_manager
            if os.path.exists(incomplete_path):
                resume_size = os.stat(incomplete_path).st_size
            else:
                resume_size = 0
        else:
            temp_file_manager = partial(tempfile.NamedTemporaryFile, dir=cache_dir, delete=False)
            resume_size = 0

        # Download to temporary file, then copy to cache dir once finished.
        # Otherwise you get corrupt cache entries if the download gets interrupted.
        with temp_file_manager() as temp_file:
            print(
                "%s not found in cache or force_download set to True, downloading to %s",
                url,
                temp_file.name,
            )

            http_get(
                url,
                temp_file,
                proxies=proxies,
                resume_size=resume_size,
                user_agent=user_agent,
            )

        os.replace(temp_file.name, cache_path)

        meta = {"url": url, "etag": etag}
        meta_path = cache_path + ".json"
        with open(meta_path, "w") as meta_file:
            json.dump(meta, meta_file)



SUMMARY:
**Summary:**

The `get_from_cache` function retrieves data from a URL while utilizing a local cache. Its primary purpose is to minimize network downloads by checking the local cache for incomplete files and updating existing ones.

**Key Implementation Details:**

*   Checks for ETAG headers in HTTP responses to determine if the file already exists in the cache or needs to be downloaded.
*   Uses `requests.head` to quickly check if a file exists without requiring credentials.
*   Implements a lock mechanism to prevent parallel downloads of the same file.
*   Supports resuming interrupted downloads using temporary files and a custom file manager.

**Notable Dependencies and Requirements:**

*   `requests` for making HTTP requests
*   `os` for creating directories and files
*   `Path` (from `pathlib`) for handling path manipulation
*   `FileLock` for synchronizing access to shared resources
*   `json` for serializing metadata

**Design Choices:**

*   Utilizes a caching mechanism to store downloaded files locally.
*   Supports resuming interrupted downloads using a temporary file.
*   Implements a lock to prevent parallel downloads of the same file.


key: chroma:document
string_value:
CODE:

def url_to_filename(url, etag=None):
    url_bytes = url.encode("utf-8")
    url_hash = insecure_hashlib.sha256(url_bytes)
    filename = url_hash.hexdigest()

    if etag:
        etag_bytes = etag.encode("utf-8")
        etag_hash = insecure_hashlib.sha256(etag_bytes)
        filename += "." + etag_hash.hexdigest()

    if url.endswith(".h5"):
        filename += ".h5"



SUMMARY:
**Function Summary: `url_to_filename`**

**Purpose and Functionality:**
Converts a URL to a unique file name based on its contents, optionally including an ETag value.

**Implementation Details:**
- Uses SHA-256 hashing to create a unique hash from the URL bytes.
- If an ETag is provided, appends it to the filename by creating another SHA-256 hash.
- If the resulting URL ends with ".h5", prefixes the filename with ".h5".

**Method Signatures and Interfaces:**
- `insecure_hashlib.sha256(url_bytes)`

**Notable Dependencies or Requirements:**
- `insecure_hashlib` (not a standard library, likely imported from an external package)
- The `utf-8` encoding

**Summary of Class/Function Names:**
`url_to_filename`: 
The function takes two parameters:
- `url`
- `etag`

Note that the function does not directly utilize any built-in Python classes or interfaces. It relies on the `insecure_hashlib` package, which should be replaced with a secure hashing library in a production environment.


key: chroma:document
string_value:
CODE:

def cached_path(
    url_or_filename,
    cache_dir=None,
    force_download=False,
    proxies=None,
    resume_download=False,
    user_agent=None,
    extract_compressed_file=False,
    force_extract=False,
    local_files_only=False,
):
    if cache_dir is None:
        cache_dir = TRANSFORMERS_CACHE
    if isinstance(url_or_filename, Path):
        url_or_filename = str(url_or_filename)
    if isinstance(cache_dir, Path):
        cache_dir = str(cache_dir)

    if is_remote_url(url_or_filename):
        # URL, so get it from the cache (downloading if necessary)
        output_path = get_from_cache(
            url_or_filename,
            cache_dir=cache_dir,
            force_download=force_download,
            proxies=proxies,
            resume_download=resume_download,
            user_agent=user_agent,
            local_files_only=local_files_only,
        )
    elif os.path.exists(url_or_filename):
        # File, and it exists.
        output_path = url_or_filename
    elif urlparse(url_or_filename).scheme == "":
        # File, but it doesn't exist.
        raise EnvironmentError("file {} not found".format(url_or_filename))
    else:
        # Something unknown
        raise ValueError("unable to parse {} as a URL or as a local path".format(url_or_filename))

    if extract_compressed_file:
        if not is_zipfile(output_path) and not tarfile.is_tarfile(output_path):
            return output_path

        # Path where we extract compressed archives
        # We avoid '.' in dir name and add "-extracted" at the end: "./model.zip" => "./model-zip-extracted/"
        output_dir, output_file = os.path.split(output_path)
        output_extract_dir_name = output_file.replace(".", "-") + "-extracted"
        output_path_extracted = os.path.join(output_dir, output_extract_dir_name)

        if os.path.isdir(output_path_extracted) and os.listdir(output_path_extracted) and not force_extract:
            return output_path_extracted

        # Prevent parallel extractions
        lock_path = output_path + ".lock"
        with FileLock(lock_path):
            shutil.rmtree(output_path_extracted, ignore_errors=True)
            os.makedirs(output_path_extracted)
            if is_zipfile(output_path):
                with ZipFile(output_path, "r") as zip_file:
                    zip_file.extractall(output_path_extracted)
                    zip_file.close()
            elif tarfile.is_tarfile(output_path):
                tar_file = tarfile.open(output_path)
                tar_file.extractall(output_path_extracted)
                tar_file.close()
            else:
                raise EnvironmentError("Archive format of {} could not be identified".format(output_path))

        return output_path_extracted



SUMMARY:
**Function Summary: `cached_path`**

The `cached_path` function retrieves or downloads a file from a cache, based on the provided URL or filename.

### Primary Purpose

1. **Cache Retrieval**: If the input is a URL, it retrieves the file from the cache and returns its path.
2. **File Retrieval**: If the input is a local file, it checks if it exists and returns its path.
3. **Extract Compressed File**: If `extract_compressed_file` is enabled, it extracts compressed archives (zip or tar files) to a new directory.

### Key Implementation Details

* It handles remote URLs by retrieving data from an internal cache.
* It checks local file existence before attempting to retrieve it from the network.
* When extracting compressed files, it uses `ZipFile` and `tarfile` modules to handle archive formats.
* A lock is used to prevent parallel extractions of the same archive.

### Notable Dependencies

* `FileLock`: Used for synchronization during archive extraction.
* `shutil`, `os`, `zipfile`, and `tarfile`: Modules for handling file operations and archives.


key: chroma:document
string_value:
CODE:

def get_data(query, delim=","):
    assert isinstance(query, str)
    if os.path.isfile(query):
        with open(query) as f:
            data = eval(f.read())
    else:
        req = requests.get(query)
        try:
            data = requests.json()
        except Exception:
            data = req.content.decode()
            assert data is not None, "could not connect"
            try:
                data = eval(data)
            except Exception:
                data = data.split("\n")
        req.close()


SUMMARY:
**Function Summary: `get_data`**

Retrieves data from either a local file or an external API, and attempts to parse it as JSON.

**Main Purpose and Functionality:**
The `get_data` function retrieves data from either a specified local file (if it exists) or sends a GET request to the provided URL. It then attempts to parse the response as JSON, evaluating any string results using the `eval()` function. If parsing fails, it falls back to decoding the content as text and splitting it into lines.

**Key Implementation Details:**

* Uses `requests` to make HTTP GET requests
* Attempts to parse response as JSON using `requests.json()`
* Falls back to decoding response content as text using `.content.decode()` if parsing fails

**Notable Dependencies or Requirements:**

* `os.path.isfile(query)` checks for local file existence
* `requests` library for API interactions
* `json` module (presumably imported from `requests`)

Example Use Case:
```python
data = get_data("https://example.com/api/data", delim=",")
print(data)
```
Note: The use of `eval()` poses security risks, as it can evaluate arbitrary code. This function should be used with caution and ideally sanitized inputs.


key: chroma:document
string_value:
CODE:

def get_image_from_url(url):
    response = requests.get(url)
    img = np.array(Image.open(BytesIO(response.content)))


SUMMARY:
**Summary:**

This function retrieves an image from a URL and returns it as a numpy array. 

**Main Purpose and Functionality:**
The primary purpose is to fetch and convert the received web-based image data into a format suitable for computer vision processing.

**Key Implementation Details:**

* Uses `requests` library to send a GET request to the specified URL.
* Utilizes `Image.open()` from Pillow library to open and load the image from the response content.
* Converts the loaded image into a numpy array using `np.array()`.
* Returns the resulting numpy array.

**Notable Dependencies or Requirements:**

* `requests` library (for HTTP requests)
* `Pillow` library (`Image` and `BytesIO` modules) 
* `numpy` library (for array conversion)

Note that the function does not perform any error handling or validation, suggesting it should be extended to handle potential exceptions.


key: chroma:document
string_value:
CODE:
# to load legacy frcnn checkpoint from detectron
def load_frcnn_pkl_from_url(url):
    fn = url.split("/")[-1]
    if fn not in os.listdir(os.getcwd()):
        wget.download(url)
    with open(fn, "rb") as stream:
        weights = pkl.load(stream)
    model = weights.pop("model")
    new = {}
    for k, v in model.items():
        new[k] = torch.from_numpy(v)
        if "running_var" in k:
            zero = torch.tensor([0])
            k2 = k.replace("running_var", "num_batches_tracked")
            new[k2] = zero


SUMMARY:
**Function Summary**
=====================

`load_frcnn_pkl_from_url(url)`: Loads pre-trained Faster R-CNN (FRCNN) model weights from a URL and stores them as PyTorch tensors.

### Key Implementation Details

* Downloads the file if it doesn't exist locally using `wget`.
* Unpacks the PyTorch `pkl` file, extracting the loaded "model" tensor.
* Converts the model's fields to PyTorch tensors, except for those containing "running_var", which are replaced with zeros.

### Notable Dependencies and Requirements

* `os`: used for interacting with the file system.
* `wget`: used for downloading files from URLs if they don't exist locally.
* `torch`: a deep learning library, specifically for PyTorch models.

### Key Interfaces/Classes

* `pkl.load(stream)`: loads a PyTorch Pickled object from a binary stream.
* `torch.from_numpy(v)`: converts a NumPy array to a PyTorch tensor.


key: chroma:document
string_value:
CODE:

def get_demo_path():


SUMMARY:
**Summary**

This function calculates and returns the absolute path to a demo IPython notebook by traversing up the directory tree from the current working location (`PATH`) and appending "demo.ipynb".

**Key Details**

* `get_demo_path` function
* Uses `os.path.abspath` and `PATH` environment variable
* Returns an absolute file path string with "demo.ipynb" appended

**Notable Dependencies/Requirements**

* `PATH` environment variable (set in current working directory)
* `os` module (for accessing operating system-dependent functions)


key: chroma:document
string_value:
CODE:

def img_tensorize(im, input_format="RGB"):
    assert isinstance(im, str)
    if os.path.isfile(im):
        img = cv2.imread(im)
    else:
        img = get_image_from_url(im)
        assert img is not None, f"could not connect to: {im}"
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    if input_format == "RGB":
        img = img[:, :, ::-1]


SUMMARY:
**Function Summary**

`img_tensorize`

Converts an image to RGB format and prepares it for tensorization.

**Purpose**
Preprocesses an input image (`im`) by converting its color space from BGR to RGB, reversing the channel order if necessary, and returning the resulting tensorized image.

**Key Implementations**

* Reads or retrieves image data using `cv2.imread()` or `get_image_from_url()`
* Converts BGR to RGB using `cv2.cvtColor()`
* Reverses color order if input format is "RGB" (`img[:, :, ::-1]`)

**Dependencies and Requirements**

* `os.path.isfile()` for file existence check
* `cv2.imread()` and `get_image_from_url()` for image loading or retrieval
* `cv2.cvtColor()` for color space conversion


key: chroma:document
string_value:
CODE:

def chunk(images, batch=1):


SUMMARY:
**Summary:**

The `chunk` function splits a list of images into batches. It takes an iterable of images and an optional argument `batch` with a default value of 1.

**Key Details:**

* Returns an iterator that yields batches of a specified size (`i`, where `i + batch` is the next element in the chunk)
* Primary purpose: efficient image processing by dividing data into manageable chunks

**Notable Dependencies/Requirements:**

* None explicitly mentioned, but assumes Python iteration and indexing capabilities.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/visual_bert/visualizing_image.py


key: chroma:document
string_value:
CODE:

class SingleImageViz:
    def __init__(
        self,
        img,
        scale=1.2,
        edgecolor="g",
        alpha=0.5,
        linestyle="-",
        saveas="test_out.jpg",
        rgb=True,
        pynb=False,
        id2obj=None,
        id2attr=None,
        pad=0.7,
    ):
        """
        img: an RGB image of shape (H, W, 3).
        """
        if isinstance(img, torch.Tensor):
            img = img.numpy().astype("np.uint8")
        if isinstance(img, str):
            img = img_tensorize(img)
        assert isinstance(img, np.ndarray)

        width, height = img.shape[1], img.shape[0]
        fig = mplfigure.Figure(frameon=False)
        dpi = fig.get_dpi()
        width_in = (width * scale + 1e-2) / dpi
        height_in = (height * scale + 1e-2) / dpi
        fig.set_size_inches(width_in, height_in)
        ax = fig.add_axes([0.0, 0.0, 1.0, 1.0])
        ax.axis("off")
        ax.set_xlim(0.0, width)
        ax.set_ylim(height)

        self.saveas = saveas
        self.rgb = rgb
        self.pynb = pynb
        self.img = img
        self.edgecolor = edgecolor
        self.alpha = 0.5
        self.linestyle = linestyle
        self.font_size = int(np.sqrt(min(height, width)) * scale // 3)
        self.width = width
        self.height = height
        self.scale = scale
        self.fig = fig
        self.ax = ax
        self.pad = pad
        self.id2obj = id2obj
        self.id2attr = id2attr
        self.canvas = FigureCanvasAgg(fig)

    def add_box(self, box, color=None):
        if color is None:
            color = self.edgecolor
        (x0, y0, x1, y1) = box
        width = x1 - x0
        height = y1 - y0
        self.ax.add_patch(
            mpl.patches.Rectangle(
                (x0, y0),
                width,
                height,
                fill=False,
                edgecolor=color,
                linewidth=self.font_size // 3,
                alpha=self.alpha,
                linestyle=self.linestyle,
            )
        )

    def draw_boxes(self, boxes, obj_ids=None, obj_scores=None, attr_ids=None, attr_scores=None):
        if len(boxes.shape) > 2:
            boxes = boxes[0]
        if len(obj_ids.shape) > 1:
            obj_ids = obj_ids[0]
        if len(obj_scores.shape) > 1:
            obj_scores = obj_scores[0]
        if len(attr_ids.shape) > 1:
            attr_ids = attr_ids[0]
        if len(attr_scores.shape) > 1:
            attr_scores = attr_scores[0]
        if isinstance(boxes, torch.Tensor):
            boxes = boxes.numpy()
        if isinstance(boxes, list):
            boxes = np.array(boxes)
        assert isinstance(boxes, np.ndarray)
        areas = np.prod(boxes[:, 2:] - boxes[:, :2], axis=1)
        sorted_idxs = np.argsort(-areas).tolist()
        boxes = boxes[sorted_idxs] if boxes is not None else None
        obj_ids = obj_ids[sorted_idxs] if obj_ids is not None else None
        obj_scores = obj_scores[sorted_idxs] if obj_scores is not None else None
        attr_ids = attr_ids[sorted_idxs] if attr_ids is not None else None
        attr_scores = attr_scores[sorted_idxs] if attr_scores is not None else None

        assigned_colors = [self._random_color(maximum=1) for _ in range(len(boxes))]
        assigned_colors = [assigned_colors[idx] for idx in sorted_idxs]
        if obj_ids is not None:
            labels = self._create_text_labels_attr(obj_ids, obj_scores, attr_ids, attr_scores)
            for i in range(len(boxes)):
                color = assigned_colors[i]
                self.add_box(boxes[i], color)
                self.draw_labels(labels[i], boxes[i], color)

    def draw_labels(self, label, box, color):
        x0, y0, x1, y1 = box
        text_pos = (x0, y0)
        instance_area = (y1 - y0) * (x1 - x0)
        small = _SMALL_OBJ * self.scale
        if instance_area < small or y1 - y0 < 40 * self.scale:
            if y1 >= self.height - 5:
                text_pos = (x1, y0)
            else:
                text_pos = (x0, y1)

        height_ratio = (y1 - y0) / np.sqrt(self.height * self.width)
        lighter_color = self._change_color_brightness(color, brightness_factor=0.7)
        font_size = np.clip((height_ratio - 0.02) / 0.08 + 1, 1.2, 2)
        font_size *= 0.75 * self.font_size

        self.draw_text(
            text=label,
            position=text_pos,
            color=lighter_color,
        )

    def draw_text(
        self,
        text,
        position,
        color="g",
        ha="left",
    ):
        rotation = 0
        font_size = self.font_size
        color = np.maximum(list(mplc.to_rgb(color)), 0.2)
        color[np.argmax(color)] = max(0.8, np.max(color))
        bbox = {
            "facecolor": "black",
            "alpha": self.alpha,
            "pad": self.pad,
            "edgecolor": "none",
        }
        x, y = position
        self.ax.text(
            x,
            y,
            text,
            size=font_size * self.scale,
            family="sans-serif",
            bbox=bbox,
            verticalalignment="top",
            horizontalalignment=ha,
            color=color,
            zorder=10,
            rotation=rotation,
        )

    def save(self, saveas=None):
        if saveas is None:
            saveas = self.saveas
        if saveas.lower().endswith(".jpg") or saveas.lower().endswith(".png"):
            cv2.imwrite(
                saveas,
                self._get_buffer()[:, :, ::-1],
            )
        else:
            self.fig.savefig(saveas)

    def _create_text_labels_attr(self, classes, scores, attr_classes, attr_scores):
        labels = [self.id2obj[i] for i in classes]
        attr_labels = [self.id2attr[i] for i in attr_classes]
        labels = [
            f"{label} {score:.2f} {attr} {attr_score:.2f}"
            for label, score, attr, attr_score in zip(labels, scores, attr_labels, attr_scores)
        ]
        return labels

    def _create_text_labels(self, classes, scores):
        labels = [self.id2obj[i] for i in classes]
        if scores is not None:
            if labels is None:
                labels = ["{:.0f}%".format(s * 100) for s in scores]
            else:
                labels = ["{} {:.0f}%".format(li, s * 100) for li, s in zip(labels, scores)]
        return labels

    def _random_color(self, maximum=255):
        idx = np.random.randint(0, len(_COLORS))
        ret = _COLORS[idx] * maximum
        if not self.rgb:
            ret = ret[::-1]
        return ret

    def _get_buffer(self):
        if not self.pynb:
            s, (width, height) = self.canvas.print_to_buffer()
            if (width, height) != (self.width, self.height):
                img = cv2.resize(self.img, (width, height))
            else:
                img = self.img
        else:
            buf = io.BytesIO()  # works for cairo backend
            self.canvas.print_rgba(buf)
            width, height = self.width, self.height
            s = buf.getvalue()
            img = self.img

        buffer = np.frombuffer(s, dtype="uint8")
        img_rgba = buffer.reshape(height, width, 4)
        rgb, alpha = np.split(img_rgba, [3], axis=2)

        try:
            import numexpr as ne  # fuse them with numexpr

            visualized_image = ne.evaluate("img * (1 - alpha / 255.0) + rgb * (alpha / 255.0)")
        except ImportError:
            alpha = alpha.astype("float32") / 255.0
            visualized_image = img * (1 - alpha) + rgb * alpha

        return visualized_image.astype("uint8")

    def _change_color_brightness(self, color, brightness_factor):
        assert brightness_factor >= -1.0 and brightness_factor <= 1.0
        color = mplc.to_rgb(color)
        polygon_color = colorsys.rgb_to_hls(*mplc.to_rgb(color))
        modified_lightness = polygon_color[1] + (brightness_factor * polygon_color[1])
        modified_lightness = 0.0 if modified_lightness < 0.0 else modified_lightness
        modified_lightness = 1.0 if modified_lightness > 1.0 else modified_lightness
        modified_color = colorsys.hls_to_rgb(polygon_color[0], modified_lightness, polygon_color[2])


SUMMARY:
This code is part of the PyTorch visualization module, designed for visualizing deep learning models. It provides various methods and modules to create images showing the internal activations or outputs of neural networks. 

The `Visualizer` class is the main function that encapsulates most of the functionality, including calculating image buffers from activations maps, drawing annotations, and saving images to files in multiple formats.

Here's a summary:

*   **Image Buffer Generation**: The `_get_buffer` method calculates an image buffer from data either directly provided by the `Visualizer` instance or via matplotlib backend.
*   **Text Annotation**: A set of methods (`draw_text`, `draw_labels`) are used to create labels on the generated images for class names, scores, or text, allowing for visualization and explanation of deep learning models' internal state.
*   **Color Changes**: The code allows for customizing colors through functions like `_random_color` and `_change_color_brightness`.
*   **Image Saving**: The `save`, `saveas` methods are used to save a frame as an image file.

By using this module, you can effectively explore the inner workings of deep learning models and gain better understanding of how they process information.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/visual_bert/processing_image.py


key: chroma:document
string_value:
CODE:

class ResizeShortestEdge:
    def __init__(self, short_edge_length, max_size=sys.maxsize):
        """
        Args:
            short_edge_length (list[min, max])
            max_size (int): maximum allowed longest edge length.
        """
        self.interp_method = "bilinear"
        self.max_size = max_size
        self.short_edge_length = short_edge_length

    def __call__(self, imgs):
        img_augs = []
        for img in imgs:
            h, w = img.shape[:2]
            # later: provide list and randomly choose index for resize
            size = np.random.randint(self.short_edge_length[0], self.short_edge_length[1] + 1)
            if size == 0:
                return img
            scale = size * 1.0 / min(h, w)
            if h < w:
                newh, neww = size, scale * w
            else:
                newh, neww = scale * h, size
            if max(newh, neww) > self.max_size:
                scale = self.max_size * 1.0 / max(newh, neww)
                newh = newh * scale
                neww = neww * scale
            neww = int(neww + 0.5)
            newh = int(newh + 0.5)

            if img.dtype == np.uint8:
                pil_image = Image.fromarray(img)
                pil_image = pil_image.resize((neww, newh), PILImageResampling.BILINEAR)
                img = np.asarray(pil_image)
            else:
                img = img.permute(2, 0, 1).unsqueeze(0)  # 3, 0, 1)  # hw(c) -> nchw
                img = nn.functional.interpolate(
                    img, (newh, neww), mode=self.interp_method, align_corners=False
                ).squeeze(0)
            img_augs.append(img)



SUMMARY:
**Summary:**

The `ResizeShortestEdge` class enables image resizing while preserving aspect ratio and enforcing bounds. Its primary purpose is to:

1. **Resize images**: Using either bilinear interpolation or pixel shuffling for downsampling/upsampling, depending on the target data type (`np.uint8` or float32).
2. **Adjust aspect ratio**: Scaling short edge while maintaining aspect ratio.
3. **Enforce maximum size**: Cap longest side at `max_size` (defaulting to system maxsize).

**Key Implementation Details:**

* Resizes images by choosing a random width between the shortest and longest possible values, determined by `short_edge_length`.
* Applies interpolation method (`bilinear`) or pixel shuffling based on image data type.
* Adjusts aspect ratio while scaling short edge.

**Notable Dependencies/Requirements:**

* NumPy (`np`)
* OpenCV (uses `Image` and `PILImageResampling`)
* PyTorch (`nn.functional`, `torchvision` package)


key: chroma:document
string_value:
CODE:

class Preprocess:
    def __init__(self, cfg):
        self.aug = ResizeShortestEdge([cfg.INPUT.MIN_SIZE_TEST, cfg.INPUT.MIN_SIZE_TEST], cfg.INPUT.MAX_SIZE_TEST)
        self.input_format = cfg.INPUT.FORMAT
        self.size_divisibility = cfg.SIZE_DIVISIBILITY
        self.pad_value = cfg.PAD_VALUE
        self.max_image_size = cfg.INPUT.MAX_SIZE_TEST
        self.device = cfg.MODEL.DEVICE
        self.pixel_std = torch.tensor(cfg.MODEL.PIXEL_STD).to(self.device).view(len(cfg.MODEL.PIXEL_STD), 1, 1)
        self.pixel_mean = torch.tensor(cfg.MODEL.PIXEL_MEAN).to(self.device).view(len(cfg.MODEL.PIXEL_STD), 1, 1)
        self.normalizer = lambda x: (x - self.pixel_mean) / self.pixel_std

    def pad(self, images):
        max_size = tuple(max(s) for s in zip(*[img.shape for img in images]))
        image_sizes = [im.shape[-2:] for im in images]
        images = [
            nn.functional.pad(
                im,
                [0, max_size[-1] - size[1], 0, max_size[-2] - size[0]],
                value=self.pad_value,
            )
            for size, im in zip(image_sizes, images)
        ]

        return torch.stack(images), torch.tensor(image_sizes)

    def __call__(self, images, single_image=False):
        with torch.no_grad():
            if not isinstance(images, list):
                images = [images]
            if single_image:
                assert len(images) == 1
            for i in range(len(images)):
                if isinstance(images[i], torch.Tensor):
                    images.insert(i, images.pop(i).to(self.device).float())
                elif not isinstance(images[i], torch.Tensor):
                    images.insert(
                        i,
                        torch.as_tensor(img_tensorize(images.pop(i), input_format=self.input_format))
                        .to(self.device)
                        .float(),
                    )
            # resize smallest edge
            raw_sizes = torch.tensor([im.shape[:2] for im in images])
            images = self.aug(images)
            # transpose images and convert to torch tensors
            # images = [torch.as_tensor(i.astype("float32")).permute(2, 0, 1).to(self.device) for i in images]
            # now normalize before pad to avoid useless arithmetic
            images = [self.normalizer(x) for x in images]
            # now pad them to do the following operations
            images, sizes = self.pad(images)
            # Normalize

            if self.size_divisibility > 0:
                raise NotImplementedError()
            # pad
            scales_yx = torch.true_divide(raw_sizes, sizes)
            if single_image:
                return images[0], sizes[0], scales_yx[0]
            else:


SUMMARY:
**Class Summary: `Preprocess`**

### Purpose and Functionality

The `Preprocess` class is designed to preprocess input images for a machine learning model. It applies various transformations, including:

1. Resizing to the shortest edge
2. Padding to a uniform size
3. Normalization (subtracting mean and dividing by standard deviation)

The primary purpose of this class is to ensure that all input images are in a consistent format before being fed into the model.

### Key Implementation Details

* `__init__`: Initializes the preprocessor with configuration settings (`cfg`).
	+ Sets up the data augmentation module using `ResizeShortestEdge`.
	+ Stores the mean and standard deviation of pixels as tensors.
* `pad`: Pads images to a uniform size by adding zeros.
	+ Uses the maximum row and column values to determine padding sizes.
* `__call__`: Processes a list of images through the preprocessing pipeline.
	+ Resizes images using data augmentation.
	+ Normalizes pixel values.
	+ Returns padded images, their original sizes, and scaling factors.

### Notable Dependencies and Requirements

* PyTorch (`torch` and its related functions)
* Configuration object `cfg` (assuming it contains input settings like `.INPUT.FORMAT`, `MODEL.DEVICE`, etc.)

### Interfaces Implemented

* None explicitly mentioned in the code. However, it appears to implement a custom data preprocessing pipeline tailored for the specific model requirements.

### External Dependencies

Assuming the presence of external dependencies not explicitly listed in this code:

* Model configuration file (`cfg`)
* Data augmentation library (not shown here)


key: chroma:document
string_value:
CODE:

def _scale_box(boxes, scale_yx):
    boxes[:, 0::2] *= scale_yx[:, 1]
    boxes[:, 1::2] *= scale_yx[:, 0]


SUMMARY:
**Summary**

The `_scale_box` function scales the x and y coordinates of rectangles in a 2D grid (`boxes`) by corresponding scaling factors (`scale_yx`). Specifically, it performs column-wise indexing on `boxes` to extract individual rectangles and then multiplies their dimensions with scaled x/y coordinates.

**Key Implementation Details:**

* Scales x coordinates using `scale_yx[:, 1]`
* Scales y coordinates using `scale_yx[:, 0]`
* Performs column-wise indexing: `[:, 0::2]` for x coordinates and `[:, 1::2]` for y coordinates

**Notable Dependencies/Requirements:**

* `boxes`: a 2D NumPy array of rectangle dimensions (width and height)
* `scale_yx`: a 2D NumPy array of scaling factors (x and y)


key: chroma:document
string_value:
CODE:

def _clip_box(tensor, box_size: Tuple[int, int]):
    assert torch.isfinite(tensor).all(), "Box tensor contains infinite or NaN!"
    h, w = box_size
    tensor[:, 0].clamp_(min=0, max=w)
    tensor[:, 1].clamp_(min=0, max=h)
    tensor[:, 2].clamp_(min=0, max=w)


SUMMARY:
**Summary:**

The `_clip_box` function clips the values of a 4-channel tensor `tensor` to fit within a rectangular box defined by `box_size`. The box's dimensions are stored in `tensor[:, 0]` (height) and `tensor[:, 1]` (width). Values exceeding these limits are clamped to zero, while those below reach their respective maximum values.

**Key Implementation Details:**

* Clamping of `tensor[:, 2]` and `tensor[:, 3]` is inconsistent with the specified box dimensions. It should be either set to `max=w` or `min=0` instead.
* No error checking for `box_size`, which could lead to an infinite or NaN tensor.

**Assumptions and Dependencies:**

* The input tensor has shape `(num_channels, num_boxes, 4)`.
* The `Tuple[int, int]` return type hint is specific to PyTorch, implying that `torch` is a dependency.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_bnb.py


key: chroma:document
string_value:
CODE:

def list_field(default=None, metadata=None):


SUMMARY:
**Function Summary**

The `list_field` function creates and returns a Django `Field` instance with custom defaults. It takes two arguments:
- `default`: the default value for the field (optional; defaults to None)
- `metadata`: additional metadata for the field (optional)

Key Implementation:

* Creates a new Django `Field` instance
* Sets the default value using a lambda function that returns `default`
* Assigns the provided `metadata` to the field

Notable Dependency:
 None, as it only relies on built-in Python and Django libraries.


key: chroma:document
string_value:
CODE:

def create_vocabulary_from_data(
    datasets: DatasetDict,
    word_delimiter_token: Optional[str] = None,
    unk_token: Optional[str] = None,
    pad_token: Optional[str] = None,
):
    # Given training and test labels create vocabulary
    def extract_all_chars(batch):
        all_text = " ".join(batch["target_text"])
        vocab = list(set(all_text))
        return {"vocab": [vocab], "all_text": [all_text]}

    vocabs = datasets.map(
        extract_all_chars,
        batched=True,
        batch_size=-1,
        keep_in_memory=True,
        remove_columns=datasets["train"].column_names,
    )

    # take union of all unique characters in each dataset
    vocab_set = functools.reduce(
        lambda vocab_1, vocab_2: set(vocab_1["vocab"][0]) | set(vocab_2["vocab"][0]), vocabs.values()
    )

    vocab_dict = {v: k for k, v in enumerate(sorted(vocab_set))}

    # replace white space with delimiter token
    if word_delimiter_token is not None:
        vocab_dict[word_delimiter_token] = vocab_dict[" "]
        del vocab_dict[" "]

    # add unk and pad token
    if unk_token is not None:
        vocab_dict[unk_token] = len(vocab_dict)

    if pad_token is not None:
        vocab_dict[pad_token] = len(vocab_dict)



SUMMARY:
**Code Summary**

### Functionality

Creates a vocabulary from given `datasets` by extracting unique characters and adding custom tokens.

### Main Purpose

`create_vocabulary_from_data`:

* Extracts all unique characters from both training and test labels
* Maps the extraction to datasets using `DatasetDict`
* Calculates character union and sorts it
* Replaces whitespace with a delimiter token (if specified)
* Adds unk and pad tokens (if specified)

### Key Implementation Details

* Uses `functools.reduce` to combine unique characters from each dataset into a set
* Manages custom tokens (`word_delimiter_token`, `unk_token`, `pad_token`) using an ordered dictionary

### Notable Dependencies or Requirements

* `DatasetDict`
* `functools.reduce`

### Method Signatures and Interfaces

* `extract_all_chars(batch)`: Extracts unique characters from batch data.
* `map()` on `datasets`: Applies the extraction function to datasets.

### Technical Terms Usage

* `DatasetDict`
* `batched=True` and `batch_size=-1` in `map()`

Note: The code follows best practices, using precise technical terms and accurately preserving class/function/variable names.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Detecting last checkpoint.
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, "
        f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
    logger.info("Training/evaluation parameters %s", training_args)

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # 1. First, let's load the dataset
    raw_datasets = DatasetDict()

    if training_args.do_train:
        raw_datasets["train"] = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            split=data_args.train_split_name,
            token=data_args.use_auth_token,
        )

        if data_args.audio_column_name not in raw_datasets["train"].column_names:
            raise ValueError(
                f"--audio_column_name '{data_args.audio_column_name}' not found in dataset '{data_args.dataset_name}'."
                " Make sure to set `--audio_column_name` to the correct audio column - one of"
                f" {', '.join(raw_datasets['train'].column_names)}."
            )

        if data_args.text_column_name not in raw_datasets["train"].column_names:
            raise ValueError(
                f"--text_column_name {data_args.text_column_name} not found in dataset '{data_args.dataset_name}'. "
                "Make sure to set `--text_column_name` to the correct text column - one of "
                f"{', '.join(raw_datasets['train'].column_names)}."
            )

        if data_args.max_train_samples is not None:
            raw_datasets["train"] = raw_datasets["train"].select(range(data_args.max_train_samples))

    if training_args.do_eval:
        raw_datasets["eval"] = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            split=data_args.eval_split_name,
            token=data_args.use_auth_token,
        )

        if data_args.max_eval_samples is not None:
            raw_datasets["eval"] = raw_datasets["eval"].select(range(data_args.max_eval_samples))

    # 2. We remove some special characters from the datasets
    # that make training complicated and do not help in transcribing the speech
    # E.g. characters, such as `,` and `.` do not really have an acoustic characteristic
    # that could be easily picked up by the model
    chars_to_ignore_regex = (
        f'[{"".join(data_args.chars_to_ignore)}]' if data_args.chars_to_ignore is not None else None
    )
    text_column_name = data_args.text_column_name

    def remove_special_characters(batch):
        if chars_to_ignore_regex is not None:
            batch["target_text"] = re.sub(chars_to_ignore_regex, "", batch[text_column_name]).lower() + " "
        else:
            batch["target_text"] = batch[text_column_name].lower() + " "
        return batch

    with training_args.main_process_first(desc="dataset map special characters removal"):
        raw_datasets = raw_datasets.map(
            remove_special_characters,
            remove_columns=[text_column_name],
            desc="remove special characters from datasets",
        )

    # save special tokens for tokenizer
    word_delimiter_token = data_args.word_delimiter_token
    unk_token = data_args.unk_token
    pad_token = data_args.pad_token

    # 3. Next, let's load the config as we might need it to create
    # the tokenizer
    # load config
    config = AutoConfig.from_pretrained(
        model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token
    )

    # 4. Next, if no tokenizer file is defined,
    # we create the vocabulary of the model by extracting all unique characters from
    # the training and evaluation datasets
    # We need to make sure that only first rank saves vocabulary
    # make sure all processes wait until vocab is created
    tokenizer_name_or_path = model_args.tokenizer_name_or_path
    tokenizer_kwargs = {}
    if tokenizer_name_or_path is None:
        # save vocab in training output dir
        tokenizer_name_or_path = training_args.output_dir

        vocab_file = os.path.join(tokenizer_name_or_path, "vocab.json")

        with training_args.main_process_first():
            if training_args.overwrite_output_dir and os.path.isfile(vocab_file):
                os.remove(vocab_file)

        with training_args.main_process_first(desc="dataset map vocabulary creation"):
            if not os.path.isfile(vocab_file):
                os.makedirs(tokenizer_name_or_path, exist_ok=True)
                vocab_dict = create_vocabulary_from_data(
                    raw_datasets,
                    word_delimiter_token=word_delimiter_token,
                    unk_token=unk_token,
                    pad_token=pad_token,
                )

                # save vocab dict to be loaded into tokenizer
                with open(vocab_file, "w") as file:
                    json.dump(vocab_dict, file)

        # if tokenizer has just been created
        # it is defined by `tokenizer_class` if present in config else by `model_type`
        tokenizer_kwargs = {
            "config": config if config.tokenizer_class is not None else None,
            "tokenizer_type": config.model_type if config.tokenizer_class is None else None,
            "unk_token": unk_token,
            "pad_token": pad_token,
            "word_delimiter_token": word_delimiter_token,
        }

    # 5. Now we can instantiate the feature extractor, tokenizer and model
    # Note for distributed training, the .from_pretrained methods guarantee that only
    # one local process can concurrently download model & vocab.

    # load feature_extractor and tokenizer
    tokenizer = AutoTokenizer.from_pretrained(
        tokenizer_name_or_path,
        token=data_args.use_auth_token,
        **tokenizer_kwargs,
    )
    feature_extractor = AutoFeatureExtractor.from_pretrained(
        model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token
    )

    # adapt config
    config.update(
        {
            "feat_proj_dropout": model_args.feat_proj_dropout,
            "attention_dropout": model_args.attention_dropout,
            "hidden_dropout": model_args.hidden_dropout,
            "final_dropout": model_args.final_dropout,
            "mask_time_prob": model_args.mask_time_prob,
            "mask_time_length": model_args.mask_time_length,
            "mask_feature_prob": model_args.mask_feature_prob,
            "mask_feature_length": model_args.mask_feature_length,
            "gradient_checkpointing": training_args.gradient_checkpointing,
            "layerdrop": model_args.layerdrop,
            "ctc_loss_reduction": model_args.ctc_loss_reduction,
            "pad_token_id": tokenizer.pad_token_id,
            "vocab_size": len(tokenizer),
            "activation_dropout": model_args.activation_dropout,
        }
    )

    # create model
    model = AutoModelForCTC.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        config=config,
        token=data_args.use_auth_token,
    )

    # freeze encoder
    if model_args.freeze_feature_encoder:
        model.freeze_feature_encoder()

    # 6. Now we preprocess the datasets including loading the audio, resampling and normalization
    # Thankfully, `datasets` takes care of automatically loading and resampling the audio,
    # so that we just need to set the correct target sampling rate and normalize the input
    # via the `feature_extractor`

    # make sure that dataset decodes audio with correct sampling rate
    dataset_sampling_rate = next(iter(raw_datasets.values())).features[data_args.audio_column_name].sampling_rate
    if dataset_sampling_rate != feature_extractor.sampling_rate:
        raw_datasets = raw_datasets.cast_column(
            data_args.audio_column_name, datasets.features.Audio(sampling_rate=feature_extractor.sampling_rate)
        )

    # derive max & min input length for sample rate & max duration
    max_input_length = data_args.max_duration_in_seconds * feature_extractor.sampling_rate
    min_input_length = data_args.min_duration_in_seconds * feature_extractor.sampling_rate
    audio_column_name = data_args.audio_column_name
    num_workers = data_args.preprocessing_num_workers

    # `phoneme_language` is only relevant if the model is fine-tuned on phoneme classification
    phoneme_language = data_args.phoneme_language

    # Preprocessing the datasets.
    # We need to read the audio files as arrays and tokenize the targets.
    def prepare_dataset(batch):
        # load audio
        sample = batch[audio_column_name]

        inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
        batch["input_values"] = inputs.input_values[0]
        batch["input_length"] = len(batch["input_values"])

        # encode targets
        additional_kwargs = {}
        if phoneme_language is not None:
            additional_kwargs["phonemizer_lang"] = phoneme_language

        batch["labels"] = tokenizer(batch["target_text"], **additional_kwargs).input_ids
        return batch

    with training_args.main_process_first(desc="dataset map preprocessing"):
        vectorized_datasets = raw_datasets.map(
            prepare_dataset,
            remove_columns=next(iter(raw_datasets.values())).column_names,
            num_proc=num_workers,
            desc="preprocess datasets",
        )

        def is_audio_in_length_range(length):
            return length > min_input_length and length < max_input_length

        # filter data that is shorter than min_input_length
        vectorized_datasets = vectorized_datasets.filter(
            is_audio_in_length_range,
            num_proc=num_workers,
            input_columns=["input_length"],
        )

    # 7. Next, we can prepare the training.
    # Let's use word error rate (WER) as our evaluation metric,
    # instantiate a data collator and the trainer

    # Define evaluation metrics during training, *i.e.* word error rate, character error rate
    eval_metrics = {metric: load_metric(metric) for metric in data_args.eval_metrics}

    # for large datasets it is advised to run the preprocessing on a
    # single machine first with ``args.preprocessing_only`` since there will mostly likely
    # be a timeout when running the script in distributed mode.
    # In a second step ``args.preprocessing_only`` can then be set to `False` to load the
    # cached dataset
    if data_args.preprocessing_only:
        logger.info(f"Data preprocessing finished. Files cached at {vectorized_datasets.cache_files}")
        return

    def compute_metrics(pred):
        pred_logits = pred.predictions
        pred_ids = np.argmax(pred_logits, axis=-1)

        pred.label_ids[pred.label_ids == -100] = tokenizer.pad_token_id

        pred_str = tokenizer.batch_decode(pred_ids)
        # we do not want to group tokens when computing the metrics
        label_str = tokenizer.batch_decode(pred.label_ids, group_tokens=False)

        metrics = {k: v.compute(predictions=pred_str, references=label_str) for k, v in eval_metrics.items()}

        return metrics

    # Now save everything to be able to create a single processor later
    if is_main_process(training_args.local_rank):
        # save feature extractor, tokenizer and config
        feature_extractor.save_pretrained(training_args.output_dir)
        tokenizer.save_pretrained(training_args.output_dir)
        config.save_pretrained(training_args.output_dir)

    try:
        processor = AutoProcessor.from_pretrained(training_args.output_dir)
    except (OSError, KeyError):
        warnings.warn(
            "Loading a processor from a feature extractor config that does not"
            " include a `processor_class` attribute is deprecated and will be removed in v5. Please add the following "
            " attribute to your `preprocessor_config.json` file to suppress this warning: "
            " `'processor_class': 'Wav2Vec2Processor'`",
            FutureWarning,
        )
        processor = Wav2Vec2Processor.from_pretrained(training_args.output_dir)

    # Instantiate custom data collator
    data_collator = DataCollatorCTCWithPadding(processor=processor)

    decay_parameters = get_parameter_names(model, [torch.nn.LayerNorm])
    decay_parameters = [name for name in decay_parameters if "bias" not in name]
    optimizer_grouped_parameters = [
        {
            "params": [p for n, p in model.named_parameters() if n in decay_parameters],
            "weight_decay": training_args.weight_decay,
        },
        {
            "params": [p for n, p in model.named_parameters() if n not in decay_parameters],
            "weight_decay": 0.0,
        },
    ]
    optimizer = bnb.optim.Adam8bit(
        params=optimizer_grouped_parameters,
        lr=training_args.learning_rate,
        betas=(training_args.adam_beta1, training_args.adam_beta2),
        eps=training_args.adam_epsilon,
    )

    optimizers = (optimizer, None)

    # Initialize Trainer
    trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=vectorized_datasets["train"] if training_args.do_train else None,
        eval_dataset=vectorized_datasets["eval"] if training_args.do_eval else None,
        tokenizer=feature_extractor,
        optimizers=optimizers,
    )

    # 8. Finally, we can start training

    # Training
    if training_args.do_train:
        # use last checkpoint if exist
        if last_checkpoint is not None:
            checkpoint = last_checkpoint
        elif os.path.isdir(model_args.model_name_or_path):
            checkpoint = model_args.model_name_or_path
        else:
            checkpoint = None

        train_result = trainer.train(resume_from_checkpoint=checkpoint)
        trainer.save_model()

        metrics = train_result.metrics
        max_train_samples = (
            data_args.max_train_samples
            if data_args.max_train_samples is not None
            else len(vectorized_datasets["train"])
        )
        metrics["train_samples"] = min(max_train_samples, len(vectorized_datasets["train"]))

        trainer.log_metrics("train", metrics)
        trainer.save_metrics("train", metrics)
        trainer.save_state()

    # Evaluation
    results = {}
    if training_args.do_eval:
        logger.info("*** Evaluate ***")
        metrics = trainer.evaluate()
        max_eval_samples = (
            data_args.max_eval_samples if data_args.max_eval_samples is not None else len(vectorized_datasets["eval"])
        )
        metrics["eval_samples"] = min(max_eval_samples, len(vectorized_datasets["eval"]))

        trainer.log_metrics("eval", metrics)
        trainer.save_metrics("eval", metrics)

    # Write model card and (optionally) push to hub
    config_name = data_args.dataset_config_name if data_args.dataset_config_name is not None else "na"
    kwargs = {
        "finetuned_from": model_args.model_name_or_path,
        "tasks": "automatic-speech-recognition",
        "tags": ["automatic-speech-recognition", data_args.dataset_name],
        "dataset_args": (
            f"Config: {config_name}, Training split: {data_args.train_split_name}, Eval split:"
            f" {data_args.eval_split_name}"
        ),
        "dataset": f"{data_args.dataset_name.upper()} - {config_name.upper()}",
    }
    if "common_voice" in data_args.dataset_name:
        kwargs["language"] = config_name

    if training_args.push_to_hub:
        trainer.push_to_hub(**kwargs)
    else:
        trainer.create_model_card(**kwargs)



SUMMARY:
This is the code for building, training, and evaluating an Automatic Speech Recognition (ASR) model using the Hugging Face Transformers library. Here's a concise summary:

**Overview**

* Define a configuration with ASR-related settings, such as the dataset, model architecture, and hyperparameters.
* Load the pre-trained ASR model and create a custom data collator for CTC (Connec-Typed Character) optimization.
* Instantiate an Adam8bit optimizer and define hyperparameters (e.g., learning rate, weight decay).
* Create a [Trainer] instance with the model, custom data collator, arguments, computation metric, and input dataset.

**Training**

* If training is enabled (`do_train=True`), train the model using the `train()` method on the training dataset.
* Save the model and metrics after training.
* Evaluate the model on the evaluation dataset and log the results.
* Optionally, push the trained model to a Hub repository.

**Evaluation**

* Evaluate the model on the test dataset using the `evaluate()` method.
* Log the metrics (e.g., accuracy, F1 score) with evaluation samples.
* Optionally, create a model card with metadata.

Note that this code assumes a specific structure for the configuration files and directory layout.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/robust-speech-event/run_speech_recognition_ctc_streaming.py


key: chroma:document
string_value:
CODE:

def list_field(default=None, metadata=None):


SUMMARY:
**Summary**

This is a generator function `list_field` that creates and returns a Django model field with optional default value (`default`) and metadata. The primary purpose is to simplify the creation of fields with default values.

**Key Details:**

* Uses `field(default_factory=lambda: default)` to create a default value.
* Sets `metadata=metadata` to provide additional metadata with the created field.
* Does not explicitly specify the return type, as it is designed to be used within the Django framework.

This function does not have notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Detecting last checkpoint.
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}, "
        f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
    logger.info("Training/evaluation parameters %s", training_args)

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # 1. First, let's load the dataset
    raw_datasets = IterableDatasetDict()
    raw_column_names = {}

    def load_streaming_dataset(split, sampling_rate, **kwargs):
        if "+" in split:
            dataset_splits = [load_dataset(split=split_name, **kwargs) for split_name in split.split("+")]
            # `features` and `cast_column` won't be available after interleaving, so we'll use them here
            features = dataset_splits[0].features
            # make sure that the dataset decodes audio with a correct sampling rate
            dataset_splits = [
                dataset.cast_column(data_args.audio_column_name, datasets.features.Audio(sampling_rate=sampling_rate))
                for dataset in dataset_splits
            ]

            interleaved_dataset = interleave_datasets(dataset_splits)
            return interleaved_dataset, features
        else:
            dataset = load_dataset(split=split, **kwargs)
            features = dataset.features
            # make sure that the dataset decodes audio with a correct sampling rate
            dataset = dataset.cast_column(
                data_args.audio_column_name, datasets.features.Audio(sampling_rate=sampling_rate)
            )
            return dataset, features

    # `datasets` takes care of automatically loading and resampling the audio,
    # so we just need to set the correct target sampling rate and normalize the input
    # via the `feature_extractor`
    feature_extractor = AutoFeatureExtractor.from_pretrained(
        model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token
    )

    if training_args.do_train:
        raw_datasets["train"], train_features = load_streaming_dataset(
            path=data_args.dataset_name,
            name=data_args.dataset_config_name,
            split=data_args.train_split_name,
            token=data_args.use_auth_token,
            streaming=True,
            sampling_rate=feature_extractor.sampling_rate,
        )
        raw_column_names["train"] = list(train_features.keys())

        if data_args.audio_column_name not in raw_column_names["train"]:
            raise ValueError(
                f"--audio_column_name '{data_args.audio_column_name}' not found in dataset '{data_args.dataset_name}'."
                " Make sure to set `--audio_column_name` to the correct audio column - one of"
                f" {', '.join(raw_column_names['train'])}."
            )

        if data_args.text_column_name not in raw_column_names["train"]:
            raise ValueError(
                f"--text_column_name {data_args.text_column_name} not found in dataset '{data_args.dataset_name}'. "
                "Make sure to set `--text_column_name` to the correct text column - one of "
                f"{', '.join(raw_column_names['train'])}."
            )

        if data_args.max_train_samples is not None:
            raw_datasets["train"] = raw_datasets["train"].take(range(data_args.max_train_samples))

    if training_args.do_eval:
        raw_datasets["eval"], eval_features = load_streaming_dataset(
            path=data_args.dataset_name,
            name=data_args.dataset_config_name,
            split=data_args.eval_split_name,
            token=data_args.use_auth_token,
            streaming=True,
            sampling_rate=feature_extractor.sampling_rate,
        )
        raw_column_names["eval"] = list(eval_features.keys())

        if data_args.max_eval_samples is not None:
            raw_datasets["eval"] = raw_datasets["eval"].take(range(data_args.max_eval_samples))

    # 2. We remove some special characters from the datasets
    # that make training complicated and do not help in transcribing the speech
    # E.g. characters, such as `,` and `.` do not really have an acoustic characteristic
    # that could be easily picked up by the model
    chars_to_ignore_regex = (
        f'[{"".join(data_args.chars_to_ignore)}]' if data_args.chars_to_ignore is not None else None
    )
    text_column_name = data_args.text_column_name

    def remove_special_characters(batch):
        if chars_to_ignore_regex is not None:
            batch["target_text"] = re.sub(chars_to_ignore_regex, "", batch[text_column_name]).lower() + " "
        else:
            batch["target_text"] = batch[text_column_name].lower() + " "
        return batch

    with training_args.main_process_first(desc="dataset map special characters removal"):
        for split, dataset in raw_datasets.items():
            raw_datasets[split] = dataset.map(
                remove_special_characters,
            ).remove_columns([text_column_name])

    # 3. Next, let's load the config as we might need it to create
    # the tokenizer
    config = AutoConfig.from_pretrained(
        model_args.model_name_or_path, cache_dir=model_args.cache_dir, token=data_args.use_auth_token
    )

    # 4. Now we can instantiate the tokenizer and model
    # Note for distributed training, the .from_pretrained methods guarantee that only
    # one local process can concurrently download model & vocab.

    tokenizer_name_or_path = model_args.tokenizer_name_or_path
    if tokenizer_name_or_path is None:
        raise ValueError(
            "Tokenizer has to be created before training in streaming mode. Please specify --tokenizer_name_or_path"
        )
    # load feature_extractor and tokenizer
    tokenizer = AutoTokenizer.from_pretrained(
        tokenizer_name_or_path,
        config=config,
        token=data_args.use_auth_token,
    )

    # adapt config
    config.update(
        {
            "feat_proj_dropout": model_args.feat_proj_dropout,
            "attention_dropout": model_args.attention_dropout,
            "hidden_dropout": model_args.hidden_dropout,
            "final_dropout": model_args.final_dropout,
            "mask_time_prob": model_args.mask_time_prob,
            "mask_time_length": model_args.mask_time_length,
            "mask_feature_prob": model_args.mask_feature_prob,
            "mask_feature_length": model_args.mask_feature_length,
            "gradient_checkpointing": training_args.gradient_checkpointing,
            "layerdrop": model_args.layerdrop,
            "ctc_loss_reduction": model_args.ctc_loss_reduction,
            "pad_token_id": tokenizer.pad_token_id,
            "vocab_size": len(tokenizer),
            "activation_dropout": model_args.activation_dropout,
        }
    )

    # create model
    model = AutoModelForCTC.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        config=config,
        token=data_args.use_auth_token,
    )

    # freeze encoder
    if model_args.freeze_feature_encoder:
        model.freeze_feature_encoder()

    # 5. Now we preprocess the datasets including loading the audio, resampling and normalization
    audio_column_name = data_args.audio_column_name

    # `phoneme_language` is only relevant if the model is fine-tuned on phoneme classification
    phoneme_language = data_args.phoneme_language

    # Preprocessing the datasets.
    # We need to read the audio files as arrays and tokenize the targets.
    def prepare_dataset(batch):
        # load audio
        sample = batch[audio_column_name]

        inputs = feature_extractor(sample["array"], sampling_rate=sample["sampling_rate"])
        batch["input_values"] = inputs.input_values[0]
        batch["input_length"] = len(batch["input_values"])

        # encode targets
        additional_kwargs = {}
        if phoneme_language is not None:
            additional_kwargs["phonemizer_lang"] = phoneme_language

        batch["labels"] = tokenizer(batch["target_text"], **additional_kwargs).input_ids
        return batch

    vectorized_datasets = IterableDatasetDict()
    with training_args.main_process_first(desc="dataset map preprocessing"):
        for split, dataset in raw_datasets.items():
            vectorized_datasets[split] = (
                dataset.map(prepare_dataset)
                .remove_columns(raw_column_names[split] + ["target_text"])
                .with_format("torch")
            )
            if split == "train":
                vectorized_datasets[split] = vectorized_datasets[split].shuffle(
                    buffer_size=data_args.shuffle_buffer_size,
                    seed=training_args.seed,
                )

    # 6. Next, we can prepare the training.
    # Let's use word error rate (WER) as our evaluation metric,
    # instantiate a data collator and the trainer

    # Define evaluation metrics during training, *i.e.* word error rate, character error rate
    eval_metrics = {metric: load_metric(metric) for metric in data_args.eval_metrics}

    def compute_metrics(pred):
        pred_logits = pred.predictions
        pred_ids = np.argmax(pred_logits, axis=-1)

        pred.label_ids[pred.label_ids == -100] = tokenizer.pad_token_id

        pred_str = tokenizer.batch_decode(pred_ids)
        # we do not want to group tokens when computing the metrics
        label_str = tokenizer.batch_decode(pred.label_ids, group_tokens=False)

        metrics = {k: v.compute(predictions=pred_str, references=label_str) for k, v in eval_metrics.items()}

        return metrics

    # Now save everything to be able to create a single processor later
    if is_main_process(training_args.local_rank):
        # save feature extractor, tokenizer and config
        feature_extractor.save_pretrained(training_args.output_dir)
        tokenizer.save_pretrained(training_args.output_dir)
        config.save_pretrained(training_args.output_dir)

    try:
        processor = AutoProcessor.from_pretrained(training_args.output_dir)
    except (OSError, KeyError):
        warnings.warn(
            "Loading a processor from a feature extractor config that does not"
            " include a `processor_class` attribute is deprecated and will be removed in v5. Please add the following "
            " attribute to your `preprocessor_config.json` file to suppress this warning: "
            " `'processor_class': 'Wav2Vec2Processor'`",
            FutureWarning,
        )
        processor = Wav2Vec2Processor.from_pretrained(training_args.output_dir)

    # Instantiate custom data collator
    max_input_length = data_args.max_duration_in_seconds * feature_extractor.sampling_rate
    data_collator = DataCollatorCTCWithPadding(processor=processor, max_length=max_input_length)

    # trainer callback to reinitialize and reshuffle the streamable datasets at the beginning of each epoch
    class ShuffleCallback(TrainerCallback):
        def on_epoch_begin(self, args, state, control, train_dataloader, **kwargs):
            if isinstance(train_dataloader.dataset, IterableDatasetShard):
                pass  # set_epoch() is handled by the Trainer
            elif isinstance(train_dataloader.dataset, IterableDataset):
                train_dataloader.dataset.set_epoch(train_dataloader.dataset._epoch + 1)

    # Initialize Trainer
    trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=vectorized_datasets["train"] if training_args.do_train else None,
        eval_dataset=vectorized_datasets["eval"] if training_args.do_eval else None,
        tokenizer=processor,
        callbacks=[ShuffleCallback()],
    )

    # 7. Finally, we can start training

    # Training
    if training_args.do_train:
        # use last checkpoint if exist
        if last_checkpoint is not None:
            checkpoint = last_checkpoint
        elif os.path.isdir(model_args.model_name_or_path):
            checkpoint = model_args.model_name_or_path
        else:
            checkpoint = None

        train_result = trainer.train(resume_from_checkpoint=checkpoint)
        trainer.save_model()

        metrics = train_result.metrics
        if data_args.max_train_samples:
            metrics["train_samples"] = data_args.max_train_samples

        trainer.log_metrics("train", metrics)
        trainer.save_metrics("train", metrics)
        trainer.save_state()

    # Evaluation
    results = {}
    if training_args.do_eval:
        logger.info("*** Evaluate ***")
        metrics = trainer.evaluate()
        if data_args.max_eval_samples:
            metrics["eval_samples"] = data_args.max_eval_samples

        trainer.log_metrics("eval", metrics)
        trainer.save_metrics("eval", metrics)

    # Write model card and (optionally) push to hub
    config_name = data_args.dataset_config_name if data_args.dataset_config_name is not None else "na"
    kwargs = {
        "finetuned_from": model_args.model_name_or_path,
        "tasks": "automatic-speech-recognition",
        "tags": ["automatic-speech-recognition", data_args.dataset_name],
        "dataset_args": (
            f"Config: {config_name}, Training split: {data_args.train_split_name}, Eval split:"
            f" {data_args.eval_split_name}"
        ),
        "dataset": f"{data_args.dataset_name.upper()} - {config_name.upper()}",
    }
    if "common_voice" in data_args.dataset_name:
        kwargs["language"] = config_name

    if training_args.push_to_hub:
        trainer.push_to_hub(**kwargs)
    else:
        trainer.create_model_card(**kwargs)



SUMMARY:
This is an implementation of the Hugging Face Transformers library in Python, specifically for training a speech recognition model using Wav2Vec2. Here's a concise summary:

**Overview**

The code defines a script that trains a Wav2Vec2 model on a set of audio data (e.g., speech recordings) to recognize specific words or phrases.

**Key Components**

1. **Data Preparation**: The script loads and preprocesses the audio data, splitting it into training, evaluation, and validation datasets.
2. **Model Initialization**: A Wav2Vec2 model is initialized with a pre-trained tokenizer and feature extractor.
3. **Training Loop**: The script defines a custom trainer with a data collator, which batches audio samples together to create input sequences for the model. The trainer uses word error rate (WER) as the evaluation metric.
4. **Hyperparameter Tuning**: The script allows users to specify hyperparameters such as maximum training epochs, batch size, and learning rate.

**Results**

The script logs metrics such as WER and character error rate during training, which can be used for model evaluation and optimization. After training, the script saves the final model checkpoint and creates a model card describing the experiment configuration.

Overall, this code provides a flexible framework for training speech recognition models using Wav2Vec2, with options for customizing hyperparameters and evaluating performance metrics.


key: chroma:document
string_value:
CODE:

def log_results(result: Dataset, args: Dict[str, str]):
    """DO NOT CHANGE. This function computes and logs the result metrics."""

    log_outputs = args.log_outputs
    dataset_id = "_".join(args.dataset.split("/") + [args.config, args.split])

    # load metric
    wer = load_metric("wer")
    cer = load_metric("cer")

    # compute metrics
    wer_result = wer.compute(references=result["target"], predictions=result["prediction"])
    cer_result = cer.compute(references=result["target"], predictions=result["prediction"])

    # print & log results
    result_str = f"WER: {wer_result}\nCER: {cer_result}"
    print(result_str)

    with open(f"{dataset_id}_eval_results.txt", "w") as f:
        f.write(result_str)

    # log all results in text file. Possibly interesting for analysis
    if log_outputs is not None:
        pred_file = f"log_{dataset_id}_predictions.txt"
        target_file = f"log_{dataset_id}_targets.txt"

        with open(pred_file, "w") as p, open(target_file, "w") as t:
            # mapping function to write output
            def write_to_file(batch, i):
                p.write(f"{i}" + "\n")
                p.write(batch["prediction"] + "\n")
                t.write(f"{i}" + "\n")
                t.write(batch["target"] + "\n")



SUMMARY:
**Code Summary: `log_results` Function**

### Purpose and Functionality

This function computes and logs result metrics for speech recognition tasks. It takes a `Dataset` object and additional arguments as input, loads required metrics (`WER`, `CER`) from external functions, and calculates the results using these metrics. The function prints and logs the computed metrics to files.

### Key Implementation Details

*   Computes WER and CER metrics using `wer.compute()` and `cer.compute()`.
*   Prints and logs results to a text file named after the dataset ID.
*   Optionally writes batch predictions and targets to separate files for further analysis.

### Notable Dependencies or Requirements

*   External "load_metric" function that loads required WER and CER metrics.
*   A Dataset object with attributes `target` and `prediction`.
*   Additional external dependencies or requirements not specified in this snippet.

### Key Method Signatures and Interfaces

*   `compute(references, predictions)` method of the loaded metric functions (`wer` and `cer`).


key: chroma:document
string_value:
CODE:

def normalize_text(text: str) -> str:
    """DO ADAPT FOR YOUR USE CASE. this function normalizes the target text."""

    chars_to_ignore_regex = '[,?.!\-\;\:"“%‘”�—’…–]'  # noqa: W605 IMPORTANT: this should correspond to the chars that were ignored during training

    text = re.sub(chars_to_ignore_regex, "", text.lower())

    # In addition, we can normalize the target text, e.g. removing new lines characters etc...
    # note that order is important here!
    token_sequences_to_ignore = ["\n\n", "\n", "   ", "  "]

    for t in token_sequences_to_ignore:
        text = " ".join(text.split(t))



SUMMARY:
**Function Summary: `normalize_text`**

**Purpose:** Normalize a given text by converting it to lowercase, removing specified characters, new lines, and extra spaces.

**Key Implementation Details:**

* Uses a regular expression (`chars_to_ignore_regex`) to remove specified characters from the input text.
* Applies additional normalization steps:
	+ Removes newline characters (`\n`).
	+ Removes sequences of three or more spaces (`"\n\n"`, `"   "`, and `"  "`).

**Notable Dependencies:**

* `re` module for regular expression operations.

**Return Type:** A normalized string.


key: chroma:document
string_value:
CODE:

def main(args):
    # load dataset
    dataset = load_dataset(args.dataset, args.config, split=args.split, token=True)

    # for testing: only process the first two examples as a test
    # dataset = dataset.select(range(10))

    # load processor
    feature_extractor = AutoFeatureExtractor.from_pretrained(args.model_id)
    sampling_rate = feature_extractor.sampling_rate

    # resample audio
    dataset = dataset.cast_column("audio", Audio(sampling_rate=sampling_rate))

    # load eval pipeline
    if args.device is None:
        args.device = 0 if torch.cuda.is_available() else -1
    asr = pipeline("automatic-speech-recognition", model=args.model_id, device=args.device)

    # map function to decode audio
    def map_to_pred(batch):
        prediction = asr(
            batch["audio"]["array"], chunk_length_s=args.chunk_length_s, stride_length_s=args.stride_length_s
        )

        batch["prediction"] = prediction["text"]
        batch["target"] = normalize_text(batch["sentence"])
        return batch

    # run inference on all examples
    result = dataset.map(map_to_pred, remove_columns=dataset.column_names)

    # compute and log_results
    # do not change function below


SUMMARY:
**Function Summary: Text-to-Speech Generation Inference**

**Main Purpose and Functionality**

This function generates speech recognition results for audio data using a pre-trained ASR model. It takes a dataset with labeled text, resamples the audio data to a specified sampling rate, and runs inference on all examples using the ASR pipeline.

**Key Implementation Details**

* Resamples audio data to a specific sampling rate using `AutoFeatureExtractor`.
* Defines a custom mapping function `map_to_pred` that:
	+ Runs ASR on each batch of audio data with chunking.
	+ Extracts predictions, normalizes text, and assigns labels.
* Applies the mapping function to all dataset examples using `dataset.map`.

**Notable Dependencies and Requirements**

* `load_dataset`, `AutoFeatureExtractor`, `pipeline`, `Audio`, `torch`
* ASR model is loaded from a pre-trained checkpoint (`args.model_id`) on device specified by `args.device`.
*$`log_results` function is called with the resulting dataset and input arguments.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/onnx/summarization/run_onnx_exporter.py


key: chroma:document
string_value:
CODE:

def parse_args():
    parser = argparse.ArgumentParser(description="Export Bart model + Beam Search to ONNX graph.")
    parser.add_argument(
        "--validation_file", type=str, default=None, help="A csv or a json file containing the validation data."
    )
    parser.add_argument(
        "--max_length",
        type=int,
        default=5,
        help="The maximum total input sequence length after tokenization.",
    )
    parser.add_argument(
        "--num_beams",
        type=int,
        default=None,
        help=(
            "Number of beams to use for evaluation. This argument will be "
            "passed to ``model.generate``, which is used during ``evaluate`` and ``predict``."
        ),
    )
    parser.add_argument(
        "--model_name_or_path",
        type=str,
        help="Path to pretrained model or model identifier from huggingface.co/models.",
        required=True,
    )
    parser.add_argument(
        "--config_name",
        type=str,
        default=None,
        help="Pretrained config name or path if not the same as model_name",
    )
    parser.add_argument(
        "--device",
        type=str,
        default="cpu",
        help="Device where the model will be run",
    )
    parser.add_argument("--output_file_path", type=str, default=None, help="Where to store the final ONNX file.")

    args = parser.parse_args()



SUMMARY:
**Function Summary: `parse_args`**

**Purpose:** Parse command-line arguments for exporting a pre-trained BART model with Beam Search to an ONOX graph.

**Functionality:**

* Parses command-line arguments using `argparse`
* Defined arguments:
	+ `validation_file`: validation data file (csv or json)
	+ `max_length`: maximum input sequence length
	+ `num_beams`: number of beams for evaluation (optional)
	+ `model_name_or_path`: path to pre-trained model or model identifier
	+ `config_name`: pretrained config name or path (optional)
	+ `device`: device where the model will be run (cpu or other GPU/TPU option)
	+ `output_file_path`: file path for final ONNX output

**Key Implementation Details:**

* Uses `argparse` to define command-line arguments with type hints and default values
* Returns a parsed `Namespace` object containing the argument values

**Notable Dependencies or Requirements:**

* `argparse` library for parsing command-line arguments
* `huggingface.co/models` library for accessing pre-trained models


key: chroma:document
string_value:
CODE:

def load_model_tokenizer(model_name, device="cpu"):
    huggingface_model = model_dict[model_name].from_pretrained(model_name).to(device)
    tokenizer = tokenizer_dict[model_name].from_pretrained(model_name)

    if model_name in ["facebook/bart-base"]:
        huggingface_model.config.no_repeat_ngram_size = 0
        huggingface_model.config.forced_bos_token_id = None
        huggingface_model.config.min_length = 0



SUMMARY:
**Summary: `load_model_tokenizer` Function**

The `load_model_tokenizer` function loads and configures a Hugging Face model tokenizer for a specified `model_name`. It takes into account special configurations required for a specific model variant.

*   **Purpose**: Load and initialize the Hugging Face model and tokenizer.
*   **Key Implementation Details**:
    *   Loads the model from `model_dict` using `from_pretrained`.
    *   Configures the model according to the specific `model_name` variant:
        *   For `facebook/bart-base`, sets `no_repeat_ngram_size` to 0, `forced_bos_token_id` to None, and `min_length` to 0.
*   **Notable Dependencies**:
    *   Hugging Face libraries (`from_pretrained`)
    *   `model_dict`
    *   `tokenizer_dict`
    
This summary provides a concise overview of the function's main purpose, key implementation details, and notable dependencies. It enables quick understanding of the code's functionality and requirements.


key: chroma:document
string_value:
CODE:

def export_and_validate_model(model, tokenizer, onnx_file_path, num_beams, max_length):
    model.eval()

    ort_sess = None
    bart_script_model = torch.jit.script(BARTBeamSearchGenerator(model))

    with torch.no_grad():
        ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
        inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors="pt").to(model.device)

        summary_ids = model.generate(
            inputs["input_ids"],
            attention_mask=inputs["attention_mask"],
            num_beams=num_beams,
            max_length=max_length,
            early_stopping=True,
            decoder_start_token_id=model.config.decoder_start_token_id,
        )

        torch.onnx.export(
            bart_script_model,
            (
                inputs["input_ids"],
                inputs["attention_mask"],
                num_beams,
                max_length,
                model.config.decoder_start_token_id,
            ),
            onnx_file_path,
            opset_version=14,
            input_names=["input_ids", "attention_mask", "num_beams", "max_length", "decoder_start_token_id"],
            output_names=["output_ids"],
            dynamic_axes={
                "input_ids": {0: "batch", 1: "seq"},
                "output_ids": {0: "batch", 1: "seq_out"},
            },
            example_outputs=summary_ids,
        )

        logger.info("Model exported to {}".format(onnx_file_path))

        new_onnx_file_path = remove_dup_initializers(os.path.abspath(onnx_file_path))

        logger.info("Deduplicated and optimized model written to {}".format(new_onnx_file_path))

        ort_sess = onnxruntime.InferenceSession(new_onnx_file_path)
        ort_out = ort_sess.run(
            None,
            {
                "input_ids": inputs["input_ids"].cpu().numpy(),
                "attention_mask": inputs["attention_mask"].cpu().numpy(),
                "num_beams": np.array(num_beams),
                "max_length": np.array(max_length),
                "decoder_start_token_id": np.array(model.config.decoder_start_token_id),
            },
        )

        np.testing.assert_allclose(summary_ids.cpu().numpy(), ort_out[0], rtol=1e-3, atol=1e-3)

        logger.info("Model outputs from torch and ONNX Runtime are similar.")


SUMMARY:
**Function Summary**

`export_and_validate_model`

Purpose: Exports and validates a BART-based summarization model by generating a summary, exporting it to an ONNX file, and then reloading the ONNX file for validation.

### Main Functionality

*   Evaluates the model
*   Generates a summary using `model.generate()`
*   Exporting the generated summary to an ONNX file using `torch.onnx.export()`
*   Reloading the exported ONNX file using `onnxruntime.InferenceSession()` for validation

### Key Implementation Details

*   Uses `torch.jit.script()` to create a torch script model (`bart_script_model`)
*   Utilizes `model.generate()` with input parameters: `attention_mask`, `num_beams`, `max_length`

### Notable Dependencies and Requirements

*   `tokenizer` object (not shown in this snippet)
*   `BARTBeamSearchGenerator` PyTorch module
*   `onnxruntime` Python library for ONNX file loading and running.
*   `logger` object (not shown in this snippet)

### Key Method Signatures

*   `model.generate()`: takes inputs: `attention_mask`, `num_beams`, `max_length`
*   `torch.jit.script()`: creates a torch script model from the `BARTBeamSearchGenerator` model
*   `onnxruntime.InferenceSession()`: loads and runs the ONNX file on the runtime session.


key: chroma:document
string_value:
CODE:

def main():
    args = parse_args()
    max_length = 5
    num_beams = 4

    # Make one log on every process with the configuration for debugging.
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO,
    )

    logger.setLevel(logging.INFO)
    transformers.utils.logging.set_verbosity_error()

    device = torch.device(args.device)

    model, tokenizer = load_model_tokenizer(args.model_name_or_path, device)

    if model.config.decoder_start_token_id is None:
        raise ValueError("Make sure that `config.decoder_start_token_id` is correctly defined")

    model.to(device)

    if args.max_length:
        max_length = args.max_length

    if args.num_beams:
        num_beams = args.num_beams

    if args.output_file_path:
        output_name = args.output_file_path
    else:
        output_name = "BART.onnx"

    logger.info("Exporting model to ONNX")


SUMMARY:
**Function Summary: Model Export Script**

### Purpose and Functionality
This function exports a pre-trained language model (from Hugging Face's BART variant) to an ONNX format file using the `transformers` library. The script takes user input arguments for model selection, export parameters, and output file path.

### Key Implementation Details

* Utilizes the `parse_args()` function to gather user input arguments.
* Configures logging settings for debugging purposes.
* Sets up the device (GPU or CPU) based on the user-provided argument.
* Loads the selected model and tokenizer using `load_model_tokenizer()`.
* Exports the model to an ONNX file, validating its integrity during the process.

### Dependencies
- `transformers` library (Hugging Face)
  - `parse_args()` function (custom implementation)
  - `load_model_tokenizer()`)
- `torch` library for device management


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/onnx/summarization/bart_onnx/generation_onnx.py


key: chroma:document
string_value:
CODE:

def _convert_past_list_to_tuple(past_key_values):
    """
    In Bart model, the type of past_key_values is tuple(tuple(torch.FloatTensor)) which is not
    TorchScript-compatible. To support this, we have to convert it during the export process.
    This function will convert past values from a list to tuple(tuple(torch.FloatTensor)) for
    the inner decoder.

    According to the definition of past_key_values, each inner tuple(torch.FloatTensor) has 4 tensors,
    so we convert every 4 elements in the list as a tuple(torch.FloatTensor).
    """
    count_of_each_inner_tuple = 4
    results = ()
    temp_result = ()
    count_n = len(past_key_values) // count_of_each_inner_tuple
    for idx in range(count_n):
        real_idx = idx * count_of_each_inner_tuple
        temp_result = tuple(past_key_values[real_idx : real_idx + count_of_each_inner_tuple])
        results += ((temp_result),)



SUMMARY:
**Function Summary: `_convert_past_list_to_tuple`**

**Purpose:** Converts past values from a list to a tuple of tuples, each containing 4 torch.FloatTensor tensors.

**Implementation Details:**

* Converts every 4 elements in the input list `past_key_values` to a tuple of tensor values.
* Uses tuple unpacking and iteration to generate the result.

**Key Method Signature:** 
```python
def _convert_past_list_to_tuple(self, past_key_values):
    ...
```
No external dependencies are specified. The function utilizes PyTorch (`torch.FloatTensor`) within its implementation. 

Note: This summary is generated based on the provided code snippet and might require additional context or information to be fully comprehensive.


key: chroma:document
string_value:
CODE:

class EncoderForONNX(torch.nn.Module):
    def __init__(self, encoder):
        super().__init__()
        self.encoder = encoder

    def forward(self, input_ids, attention_mask):
        return self.encoder(
            input_ids=input_ids,
            attention_mask=attention_mask,
            return_dict=False,


SUMMARY:
**Summary:**

`EncoderForONNX` is an ONNX-compatible encoding module based on PyTorch's `Module` class. It inherits from PyTorch's `nn.Module`.

**Key Details:**

* **Primary Functionality:** Takes input IDs and attention masks as input, passes them through the underlying `encoder`, and returns encoded output.
* **Implementation Details:** Utilizes the `return_dict=False` method in `forward()` to bypass dictionary-level result handling.

**Notable Dependencies:**

* `PyTorch's nn.Module`
* `ONNX`

**Model Structure:**

* Has a single submodule (`encoder`) which is a configurable, inheritable module.


key: chroma:document
string_value:
CODE:

class DecoderForONNX(torch.nn.Module):
    def __init__(self, decoder):
        super().__init__()
        self.decoder = decoder

    def forward(self, input_ids, encoder_state, attention_mask, past=None):
        all_results = None
        if past is not None:
            all_results = _convert_past_list_to_tuple(past)
            input_ids = input_ids[:, -1:]

        last_hidden_state, past_key_values = self.decoder(
            input_ids=input_ids,
            encoder_hidden_states=encoder_state,
            encoder_attention_mask=attention_mask,
            past_key_values=all_results,
            return_dict=False,
        )

        past_values = []
        for past in past_key_values:
            past_values = past_values + list(past)


SUMMARY:
**DecoderForONNX Class Summary**

The `DecoderForONNX` class, a PyTorch-based neural network model, is designed to perform forward pass processing on input sequences.

**Main Purpose and Functionality:**
This decoder model takes in four inputs: `input_ids`, `encoder_state`, `attention_mask`, and optional `past` sequence values. It uses the `_convert_past_list_to_tuple` helper function to convert past sequences into a tuple format. The model then passes these inputs through another decoder (`decoder`) instance, which computes last hidden state and past key values.

**Key Implementation Details:**

* Inputs:
	+ `input_ids`: input tensor (batch x seq_len)
	+ `encoder_state`: encoder hidden states (batch x seq_len x hid_dim)
	+ `attention_mask`: attention mask scalar or tensor
	+ `past`: optional past sequence values (batch x seq_len x hid_dim)
* Decoder method signature:
	+ Input arguments: `input_ids`, `encoder_hidden_states`, `encoder_attention_mask`, `past_key_values`, `return_dict=False`
* Model output:
	+ Last hidden state: (batch x seq_len x hid_dim)
	+ Past values list: (batch x seq_len x hid_dim)

**Notable Dependencies and Requirements:**

* Custom `_convert_past_list_to_tuple` function (not shown in code snippet)
* PyTorch library (`torch.nn.Module`)


key: chroma:document
string_value:
CODE:

def _create_traced_encoder(encoder, input_ids, attention_mask):
    encoder_c = copy.deepcopy(encoder)
    encoder_for_onnx = EncoderForONNX(encoder_c)



SUMMARY:
**Summary**

 Creates a traced version of an ONNX-optimized model. It:

*   Deepcopies the input encoder to create a temporary copy (`encoder_c`).
*   Wraps this copy with `EncoderForONNX` using copy.deepcopy.
*   Traces the resulting object with PyTorch's torch.jit.trace, creating a dynamic computational graph.

**Key Details**

*   Method Signature: `_create_traced_encoder(encoder: torch.nn.Module, input_ids: torch LongTensor, attention_mask: torch LongTensor) -> torch.jit.ScriptModule`
*   Uses `EncoderForONNX` wrapper to adapt the encoder for ONNX export.

**Dependencies**

*   `torch`: The PyTorch library.
*   `EncoderForONNX`: A custom ONNX-specific encoder wrapper.


key: chroma:document
string_value:
CODE:

def _create_traced_decoder(decoder, input_ids, encoder_state, attention_mask, past=None):
    decoder_c = copy.deepcopy(decoder)
    decoder_for_onnx = DecoderForONNX(decoder_c)
    past_values = list(itertools.chain.from_iterable(past or ()))

    # Do this twice so we got 2 different decoders for further work.
    if past_values:
        return torch.jit.trace(decoder_for_onnx, (input_ids, encoder_state, attention_mask, past_values))
    else:


SUMMARY:
**Function Summary: `_create_traced_decoder`**

Creates a traced version of the provided decoder for ONNX export.

**Purpose and Functionality**

The primary purpose of this function is to create a traced version of the decoder using PyTorch's `torch.jit.trace`. The traced decoder can be used for further processing, including ONNX export. It takes in various inputs, including the decoder itself, input IDs, encoder state, attention mask, and past values.

**Key Implementation Details**

*   Traces the `decoder_for_onnx` using `torch.jit.trace`.
*   Returns two versions of the traced decoder based on whether `past_values` is present or not.
*   Uses `copy.deepcopy` to create a deep copy of the original decoder (`decoder_c`).

**Notable Dependencies and Requirements**

*   Uses PyTorch's dynamic computation graph functionality via `torch.jit.trace`.
*   `DecoderForONNX` class for ONNX-specific modifications to the decoder.
*   Utilizes `itertools.chain.from_iterable` for handling multiple past values.

Note: This summary focuses on the core aspects of the function, aiming to provide a concise overview without unnecessary details.


key: chroma:document
string_value:
CODE:

class BartConfigTS(BartConfig, torch.nn.Module):
    """
    BartConfigTS is a TorchScript-compatible transformers.models.bart.configuration_bart.BartConfig.
    TorchScript only supports sub-classes of torch.nn.Module.
    """

    def __init__(self, config):
        BartConfig.__init__(self, config)


SUMMARY:
**Summary:**

```python
BartConfigTS Class:
--------------------

A TorchScript-compatible subclass of `torch.nn.Module` that inherits from `BartConfig`.
Provides an initial configuration for the BART model, making it compatible with TorchScript.

   - **Purpose:** Compatible transformer configuration for TorchScript.
   - **Key implementation details:**
     - Extends `BartConfig` and adds compatibility with `torch.nn.Module`.
   - **Notable dependencies:** None.
```

**Class Hierarchy:**

- `BartConfigTS`
  - `BartConfig`
    - (base class)
- `torch.nn.Module`


key: chroma:document
string_value:
CODE:

class MinLengthLogitsProcessorTS(torch.nn.Module):
    r"""
    :class:`transformers.LogitsProcessor` enforcing a min-length by setting EOS probability to 0.

    Args:
        min_length (:obj:`int`):
            The minimum length below which the score of :obj:`eos_token_id` is set to :obj:`-float("Inf")`.
        eos_token_id (:obj:`int`):
            The id of the `end-of-sequence` token.
    """

    def __init__(self, min_length: int, eos_token_id: int):
        super().__init__()

        if not isinstance(min_length, int) or min_length < 0:
            raise ValueError(f"`min_length` has to be a positive integer, but is {min_length}")

        if not isinstance(eos_token_id, int) or eos_token_id < 0:
            raise ValueError(f"`eos_token_id` has to be a positive integer, but is {eos_token_id}")

        self.min_length = min_length
        self.eos_token_id = eos_token_id

    def forward(self, input_ids, scores) -> torch.Tensor:
        cur_len = input_ids.shape[-1]
        if cur_len < self.min_length:
            scores[:, self.eos_token_id] = -float("inf")


SUMMARY:
**Class Summary: MinLengthLogitsProcessorTS**

A PyTorch module implementing an `MinLengthLogitsProcessor` class. This processor enforces a minimum length constraint by setting the EOS probability to 0.

**Primary Purpose:** Set EOS token score to -inf below a specified min_length threshold for input sequences.

**Key Implementation Details:**

*   Sets the EOS token ID and minimum length thresholds.
*   Modifies scores during the forward pass if the current sequence length is less than `min_length`.

**Notable Dependencies/Requirements:**

*   `transformers.LogitsProcessor`
*   PyTorch library (`torch.tensor`, `super()`)
*   Integer validation for min_length and eos_token_id
*   Floating-point constant for EOS probability value (-inf)


key: chroma:document
string_value:
CODE:

class BARTGenerator(torch.nn.Module, GenerationMixin):
    def __init__(self, model):
        super().__init__()
        self.config = BartConfigTS(model.config)
        self.config.force_bos_token_to_be_generated = False
        self._trace_modules(model)
        self.logits_processor = MinLengthLogitsProcessorTS(self.config.min_length, self.config.eos_token_id)
        self.final_logits_weight = model.model.shared.weight
        self.final_logits_bias = model.final_logits_bias
        self.decoder_layers = model.config.decoder_layers

    def _trace_modules(self, model):
        input_ids = torch.tensor(
            [
                [
                    19,
                    669,
                    18,
                    420,
                    8,
                    664,
                    57,
                    42,
                    8,
                    664,
                    21,
                    3028,
                    195,
                    4445,
                    331,
                    1293,
                    34,
                    21,
                    10,
                    6174,
                    1100,
                    6,
                    69,
                    104,
                    42,
                    32,
                    2621,
                    1638,
                    144,
                    4,
                    6174,
                    558,
                    108,
                    4419,
                    1091,
                    28,
                    4,
                    1668,
                    9,
                    1509,
                    1621,
                    279,
                    35,
                    867,
                    2734,
                    85,
                    11,
                    2216,
                    2734,
                    85,
                    203,
                    2244,
                    7,
                    6,
                    15,
                    8102,
                    7,
                    57,
                    8629,
                    5,
                    model.config.eos_token_id,
                ]
            ],
            device=model.device,
            dtype=torch.long,
        )
        attention_mask = torch.tensor(
            [[True] * input_ids.shape[-1]],
            device=model.device,
            dtype=torch.bool,
        )
        self.encoder = _create_traced_encoder(model.get_encoder(), input_ids, attention_mask)
        encoder_outputs = model.get_encoder()(input_ids, attention_mask=attention_mask, return_dict=True)
        decoder = model.model.decoder
        decoder_outputs = decoder(input_ids, attention_mask, encoder_outputs["last_hidden_state"], None, None, None)
        self.decoder_no_past = _create_traced_decoder(
            model.model.decoder, input_ids, encoder_outputs["last_hidden_state"], attention_mask
        )
        self.decoder_with_past = _create_traced_decoder(
            model.model.decoder, input_ids, encoder_outputs["last_hidden_state"], attention_mask, decoder_outputs[1]
        )

    def _encoder_forward(self, input_ids, attention_mask):
        return self.encoder(input_ids, attention_mask)[0]

    @staticmethod
    def _init_sequence_length_for_generation(
        input_ids: torch.LongTensor, max_length: int
    ) -> Tuple[torch.Tensor, torch.Tensor, int]:
        unfinished_sequences = torch.zeros(input_ids.shape[0], dtype=torch.long, device=input_ids.device) + 1
        sequence_lengths = torch.zeros(input_ids.shape[0], dtype=torch.long, device=input_ids.device) + max_length

        cur_len = input_ids.shape[-1]
        return sequence_lengths, unfinished_sequences, cur_len

    def _decoder_forward(self, input_ids, encoder_output, attention_mask, past: List[torch.Tensor]):
        # Update here to use different decoder for different values of past.
        if past is None or len(past) == 0:
            decoder_output, past = self.decoder_no_past(
                input_ids=input_ids, encoder_state=encoder_output, attention_mask=attention_mask
            )
        else:
            decoder_output, past = self.decoder_with_past(
                input_ids=input_ids, encoder_state=encoder_output, attention_mask=attention_mask, past=past
            )

        lm_logits = F.linear(decoder_output, self.final_logits_weight, bias=self.final_logits_bias)

        return lm_logits, past

    def greedy_search(
        self, input_ids, encoder_output, attention_mask, max_length, pad_token_id: int, eos_token_id: int
    ):
        # init sequence length tensors
        sequence_lengths, unfinished_sequences, cur_len = self._init_sequence_length_for_generation(
            input_ids, max_length
        )

        past: List[torch.Tensor] = []
        while cur_len < max_length:
            logits, past = self._decoder_forward(input_ids, encoder_output, attention_mask, past)
            next_token_logits = logits[:, -1, :]

            # pre-process distribution
            scores = self.logits_processor(input_ids, next_token_logits)

            # argmax
            next_tokens = torch.argmax(scores, dim=-1)

            # add code that transfomers next_tokens to tokens_to_add
            if eos_token_id is not None:
                assert pad_token_id is not None, "If eos_token_id is defined, make sure that pad_token_id is defined."
                next_tokens = next_tokens * unfinished_sequences + (pad_token_id) * (1 - unfinished_sequences)

            # add token and increase length by one
            input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)

            # update sequence length
            if eos_token_id is not None:
                sequence_lengths, unfinished_sequences = self._update_seq_length_for_generation(
                    sequence_lengths, unfinished_sequences, cur_len, next_tokens == eos_token_id
                )

            # stop when there is a </s> in each sentence, or if we exceed the maximul length
            if unfinished_sequences.max() == 0:
                break

            # increase cur_len
            cur_len = cur_len + 1

        return input_ids

    def _prepare_decoder_input_ids_for_generation(
        self,
        input_ids: torch.LongTensor,
        decoder_start_token_id,
        bos_token_id: Optional[int] = None,
    ) -> torch.LongTensor:
        decoder_input_ids = (
            torch.ones((input_ids.shape[0], 1), dtype=input_ids.dtype, device=input_ids.device)
            * decoder_start_token_id
        )
        return decoder_input_ids

    def forward(self, input_ids, attention_mask, max_length, decoder_start_token_id):
        pad_token_id = self.config.pad_token_id
        bos_token_id = self.config.bos_token_id
        eos_token_id = self.config.eos_token_id

        # special case if pad_token_id is not defined
        if pad_token_id is None and eos_token_id is not None:
            # Setting `pad_token_id` to `eos_token_id`:{eos_token_id} for open-end generation.
            pad_token_id = eos_token_id

        encoder_output = self._encoder_forward(input_ids, attention_mask)

        input_ids = self._prepare_decoder_input_ids_for_generation(
            input_ids,
            decoder_start_token_id=decoder_start_token_id,
            bos_token_id=bos_token_id,
        )

        return self.greedy_search(
            input_ids,
            encoder_output,
            attention_mask,
            max_length=max_length,
            pad_token_id=pad_token_id,
            eos_token_id=eos_token_id,


SUMMARY:
**BARTGenerator Class Summary**

### Main Purpose and Functionality
The `BARTGenerator` class is designed to generate text based on the BART (Bidirectional Abstractive Token Transformer) model architecture. It extends PyTorch's `Module` and `GenerationMixin` classes.

Its primary functionality is to take in input IDs, attention mask, max length, and decoder start token ID as inputs, and returns a generated sequence of tokens.

### Key Implementation Details

*   The class initializes the BARTConfigTS configuration, logits processor, and final logits weight/bias.
*   It utilizes two tracing modules (`self.encoder` and `self.decoder_no_past`/`self.decoder_with_past`) to create a traced version of the model for efficient inference.
*   The `_encoder_forward` method processes input IDs through the encoder, while the `_decoder_forward` method generates output tokens based on the decoder's past state.
*   The `greedy_search` method iterates over the generation process, updating sequence lengths and token predictions until termination conditions are met.

### Notable Dependencies and Requirements

*   PyTorch (`torch`)
*   BERT (`transformers`) library for encoder and decoder implementations
*   Custom configurations (`BartConfigTS`) and custom logging and processor functions (`MinLengthLogitsProcessorTS`, `_create_traced_encoder`/ `_create_traced_decoder`)
*   `GenerationMixin` class (base class for sequence generation)

### Interfaces and Dependencies

*   Extends PyTorch's `Module` class.
*   Utilizes BERT's transformer implementation for encoder and decoder.
*   Implements custom processing functions (`_update_seq_length_for_generation`, `_prepare_decoder_input_ids_for_generation`).


key: chroma:document
string_value:
CODE:
# TorchScript compatible BeamSearchScorer
class BeamSearchScorerTS(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.max_length: int = 200
        self.num_beams: int = 3
        self.batch_size: int = 1
        self.length_penalty: float = 1.0
        self.do_early_stopping: bool = True
        self.num_beam_hyps_to_keep: int = 1
        self.num_beam_groups: int = 1
        self.group_size: int = self.num_beams // self.num_beam_groups
        self._done = torch.zeros(self.batch_size, dtype=torch.bool)
        self._beam_hyps_count = torch.zeros(self.batch_size, dtype=torch.long)
        self._beam_hyps_worst_scores = torch.zeros(self.batch_size) + 1e9
        self._beam_hyps_max_length: int = self.max_length - 1
        self._beam_hyps: List[torch.Tensor] = [torch.zeros(2)]  # placeholder for TorchScript compatibility
        self._beam_scores: List[torch.Tensor] = [torch.zeros(2)]  # placeholder for TorchScript compatibility

    def is_done(self) -> torch.Tensor:
        return self._done.all()

    def init(
        self,
        batch_size: int,
        max_length: int,
        num_beams: int,
        device: torch.device,
        length_penalty: float = 1.0,
        do_early_stopping: bool = False,
        num_beam_hyps_to_keep: int = 1,
        num_beam_groups: int = 1,
    ):
        self.max_length = max_length
        self.num_beams = num_beams
        self.batch_size = batch_size
        self.length_penalty = length_penalty
        self.do_early_stopping = do_early_stopping
        self.num_beam_hyps_to_keep = num_beam_hyps_to_keep
        self.num_beam_groups = num_beam_groups
        self.group_size = self.num_beams // self.num_beam_groups

        # NOTE: TorchScript does not support List of Modules
        #       Rewritten BeamHypotheses with tensors and list of tensors.
        self._done = torch.zeros(batch_size, dtype=torch.bool, device=device)
        self._beam_hyps_count = torch.zeros(batch_size, dtype=torch.long, device=device)
        self._beam_hyps_worst_scores = torch.zeros(batch_size, device=device) + 1e9
        self._beam_hyps = []
        self._beam_scores = []

        self._beam_hyps_max_length = max_length - 1  # ignoring bos_token

        if not isinstance(num_beams, int) or num_beams <= 1:
            raise ValueError(
                f"`num_beams` has to be an integer strictly greater than 1, but is {num_beams}. For `num_beams` == 1,"
                " one should make use of `greedy_search` instead."
            )

        if not isinstance(num_beam_groups, int) or (num_beam_groups > num_beams) or (num_beams % num_beam_groups != 0):
            raise ValueError(
                "`num_beam_groups` has to be an integer smaller or equal than `num_beams` and `num_beams` has to be"
                f" divisible by `num_beam_groups`, but is {num_beam_groups} with `num_beams` being {num_beams}."
            )

    def hypo_len(self, hypo_idx: int):
        """
        Number of hypotheses in the list.
        """
        return self._beam_hyps_count[hypo_idx]

    def hypo_add(self, hyp: torch.Tensor, sum_logprobs: float, hypo_idx: int):
        """
        Add a new hypothesis to the list.
        """
        score = sum_logprobs / (hyp.shape[-1] ** self.length_penalty)
        hyps_count = self.hypo_len(hypo_idx)
        if hyps_count < self.num_beams or score > self._beam_hyps_worst_scores[hypo_idx]:
            # NOTE: work around difference of torch.sum(empty_tensor) == 0, while error in onnx.
            # Bug: https://msdata.visualstudio.com/Vienna/_workitems/edit/1486599
            beam_idx = (
                torch.sum(self._beam_hyps_count[:hypo_idx]) if hypo_idx != 0 else torch.tensor(0, dtype=torch.long)
            )
            self._beam_scores.insert(beam_idx, torch.tensor([score]))
            self._beam_hyps.insert(beam_idx, hyp)
            if hyps_count + 1 > self.num_beams:
                sorted_next_scores, sorted_indices = torch.topk(
                    torch.cat(self._beam_scores)[beam_idx : beam_idx + hyps_count + 1], hyps_count + 1, largest=False
                )
                del self._beam_hyps[int((sorted_indices[0] + beam_idx))]
                del self._beam_scores[int((sorted_indices[0] + beam_idx))]
                self._beam_hyps_worst_scores[hypo_idx] = sorted_next_scores[1]
            else:
                self._beam_hyps_worst_scores[hypo_idx] = min(score, self._beam_hyps_worst_scores[hypo_idx])
                self._beam_hyps_count[hypo_idx] = hyps_count + 1

    def hypo_is_done(self, hypo_idx: int, best_sum_logprobs: float, cur_len: int) -> bool:
        """
        If there are enough hypotheses and that none of the hypotheses being generated can become better than the worst
        one in the heap, then we are done with this sentence.
        """
        if self.hypo_len(hypo_idx) < self.num_beams:
            return False
        elif self.do_early_stopping:
            return True
        else:
            cur_score = best_sum_logprobs / cur_len**self.length_penalty
            ret = self._beam_hyps_worst_scores[hypo_idx].item() >= cur_score
            return ret

    def process(
        self,
        input_ids: torch.Tensor,
        next_scores: torch.Tensor,
        next_tokens: torch.Tensor,
        next_indices: torch.Tensor,
        pad_token_id: Optional[int] = None,
        eos_token_id: Optional[int] = None,
    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        cur_len = input_ids.shape[-1]
        batch_size = len(self._beam_hyps_count)
        assert batch_size == (input_ids.shape[0] // self.group_size)

        device = input_ids.device
        next_beam_scores = torch.zeros((batch_size, self.group_size), dtype=next_scores.dtype, device=device)
        next_beam_tokens = torch.zeros((batch_size, self.group_size), dtype=next_tokens.dtype, device=device)
        next_beam_indices = torch.zeros((batch_size, self.group_size), dtype=next_indices.dtype, device=device)

        for batch_idx in range(batch_size):
            if self._done[batch_idx]:
                assert (
                    self.hypo_len(batch_idx) >= self.num_beams
                ), "Batch can only be done if at least {} beams have been generated".format(self.num_beams)
                assert (
                    eos_token_id is not None and pad_token_id is not None
                ), "generated beams >= num_beams -> eos_token_id and pad_token have to be defined"
                # pad the batch
                next_beam_scores[batch_idx, :] = 0
                next_beam_tokens[batch_idx, :] = pad_token_id
                next_beam_indices[batch_idx, :] = 0
                continue

            # next tokens for this sentence
            beam_idx = 0
            for beam_token_rank, (next_token, next_score, next_index) in enumerate(
                zip(next_tokens[batch_idx], next_scores[batch_idx], next_indices[batch_idx])
            ):
                batch_beam_idx = batch_idx * self.group_size + next_index
                # add to generated hypotheses if end of sentence
                if (eos_token_id is not None) and (next_token == eos_token_id):
                    # if beam_token does not belong to top num_beams tokens, it should not be added
                    is_beam_token_worse_than_top_num_beams = beam_token_rank >= self.group_size
                    if is_beam_token_worse_than_top_num_beams:
                        continue
                    self.hypo_add(
                        input_ids[batch_beam_idx].clone(),
                        next_score.item(),
                        batch_idx,
                    )
                else:
                    # add next predicted token since it is not eos_token
                    next_beam_scores[batch_idx, beam_idx] = next_score
                    next_beam_tokens[batch_idx, beam_idx] = next_token
                    next_beam_indices[batch_idx, beam_idx] = batch_beam_idx
                    beam_idx += 1

                # once the beam for next step is full, don't add more tokens to it.
                if beam_idx == self.group_size:
                    break

            if beam_idx < self.group_size:
                raise ValueError(
                    f"At most {self.group_size} tokens in {next_tokens[batch_idx]} can be equal to `eos_token_id:"
                    f" {eos_token_id}`. Make sure {next_tokens[batch_idx]} are corrected."
                )

            # Check if we are done so that we can save a pad step if all(done)
            self._done[batch_idx] = self._done[batch_idx] or self.hypo_is_done(
                batch_idx,
                next_scores[batch_idx].max().item(),
                cur_len,
            )

        return next_beam_scores.view(-1), next_beam_tokens.view(-1), next_beam_indices.view(-1)

    def finalize(
        self,
        input_ids: torch.Tensor,
        final_beam_scores: torch.Tensor,
        final_beam_tokens: torch.Tensor,
        final_beam_indices: torch.Tensor,
        pad_token_id: int,
        eos_token_id: int,
    ) -> Tuple[torch.Tensor, torch.Tensor]:
        batch_size = len(self._beam_hyps_count)

        # finalize all open beam hypotheses and add to generated hypotheses
        for batch_idx in range(batch_size):
            if self._done[batch_idx]:
                continue

            # all open beam hypotheses are added to the beam hypothesis
            # beam hypothesis class automatically keeps the best beams
            for beam_id in range(self.num_beams):
                batch_beam_idx = batch_idx * self.num_beams + beam_id
                final_score = final_beam_scores[batch_beam_idx].item()
                final_tokens = input_ids[batch_beam_idx]
                self.hypo_add(final_tokens, final_score, batch_idx)

        # select the best hypotheses
        # NOTE: torch.Tensor.new_zeros() is not scriptable
        sent_lengths = torch.zeros(batch_size * self.num_beam_hyps_to_keep, dtype=torch.long)
        best = []
        best_scores = torch.zeros(
            batch_size * self.num_beam_hyps_to_keep, device=input_ids.device, dtype=torch.float32
        )
        # retrieve best hypotheses
        for i in range(batch_size):
            # NOTE: lambda is not scriptable
            batch_hypo_start = torch.sum(self._beam_hyps_count[:i]) if i > 0 else torch.tensor(0, dtype=torch.long)
            batch_hypo_end = torch.sum(self._beam_hyps_count[: i + 1])
            beam_scores = torch.cat(self._beam_scores)[batch_hypo_start:batch_hypo_end]
            sorted_next_scores, sorted_indices = torch.topk(beam_scores, len(beam_scores), largest=True)
            for j in range(self.num_beam_hyps_to_keep):
                best_score = beam_scores[sorted_indices[j]]
                best_hyp = self._beam_hyps[batch_hypo_start + sorted_indices[j]]
                sent_lengths[self.num_beam_hyps_to_keep * i + j] = len(best_hyp)
                # append to lists
                best.append(best_hyp)
                best_scores[i * self.num_beam_hyps_to_keep + j] = best_score

        # prepare for adding eos
        sent_max_len = min(sent_lengths.max() + 1, self.max_length)
        decoded = torch.zeros(batch_size * self.num_beam_hyps_to_keep, sent_max_len, dtype=torch.long)
        # shorter batches are padded if needed
        if sent_lengths.min() != sent_lengths.max():
            assert pad_token_id is not None, "`pad_token_id` has to be defined"
            decoded.fill_(pad_token_id)

        # fill with hypotheses and eos_token_id if the latter fits in
        for i, hypo in enumerate(best):
            decoded[i, : sent_lengths[i]] = hypo
            if sent_lengths[i] < self.max_length:
                decoded[i, sent_lengths[i]] = eos_token_id



SUMMARY:
Here is a succinct and thorough summary of the `BeamSearchScorer` code:

**Overview**

The `BeamSearchScorer` is a class that implements beam search, a technique used in natural language processing (NLP) to generate a sequence of predictions. It takes in several inputs, including the input tensor, scores for each token in the vocabulary, and indices for the top-k tokens.

**Methodology**

The `BeamSearchScorer` uses the following steps to perform beam search:

1. Initialize an empty list to store the current hypothesis.
2. For each token in the input sequence:
   1. Check if the hypothesized token is among the top-k tokens based on their scores.
   2. If it is, add the token and its score to the end of the current hypothesis.
3. Repeat step 2 for all tokens until a stopping criterion is met (e.g., max length reached).
4. Once the stopping criterion is met, the last hypothesized sequence in the list is considered the best one.
5. The `finalize` method is called on each hypothesis to convert it into a complete sentence with padding andEOS token if needed.

**Key Functions**

* `process_output`: This function processes the output tensor for beam search. It first sorts the output tensor based on scores, then selects tokens using these sorted indices.
	+ If the hypothesized token is among the top-k tokens, it adds the token and its score to the end of the current hypothesis.
	+ Otherwise, it skips to the next iteration in case `k` changes between iterations.
* `finalize`: This function finalizes each hypothesis after computing its scores. 
   - First it fills all shorter hypotheses with pad_token_id.
   - Then for every other hypothesis, if there is enough space in max-length, fill that section of the hypothesis with token EOS_id
     using pad_token_id as default

**Class Structure**

The `BeamSearchScorer` class consists of several methods:

* `__init__`: Initializes variables such as input tensor, beam size, and other settings.
* `process_output`: This method performs beam search on a given output tensor.
* `finalize`: Finalizes each hypothesized sentence into a complete one.
* The constructor (`__init__`) initializes the object with some defaults for configuration parameters 

**TorchScript Compatibility**

The code is designed to be compatible with TorchScript, which allows it to be used in the context of model serialization and inference.

Here is an edited version of your code snippet, including all the methods described in this summary:

```python
import torch

class BeamSearchScorer:
    def __init__(self, k=5, vocab_size=None):
        """
        Initialize the search options.
        :param k: Max tokens allowed to keep at each step 
        :param vocab_size: number to use for padding and length calculation. This should be provided in the training setup
        """
        self.hypotheses = []
        self.max_lengths = []
        self.k = k
        self.vocab_size = None

    def process_output(self, output):
        """
        Fill hypotheses from prediction tensor.
        :param input: shape (batch_size, sequence_length)
        """
        if not hasattr(output, 'max'):
            raise NotImplementedError

        batch_size, _ = output.shape
        # Initialize an empty list to save hypothesied answers of that epoch
        self.hypotheses.append(torch.zeros(batch_size, self.k))
        self.max_lengths.extend([self.k] * batch_size)

        sorted_indices = torch.topk(output, k=self.k, dim=1 ).indices

        for i, idxes in enumerate(sorted_indices):
            if len(self.max_lengths[i]) < 1:
                continue
            for idx1 in idxes:
                hypo_idx = i * self.k + idx1 
                new_score = output[0,hypo_idx]
                # Append the hypothesized token and its score to the end of the current hypothesis.
                if new_score > self.hypotheses[i][-self.max_lengths[i]]: #if this is better than last value
                    self.hypotheses[i][-self.max_lengths[i]: ] = output[0,hypo_idx][:-1]
```

However, the `process_output` does not take into account the eos token or the max_length so these have to be added manually later in code using `finalize`: 

Please note that you may need to adjust it according to your model. Also you could use a more advanced version of this code which can store the input_ids and use the pre-outputted ones only once.

If there is more space, we can go further, including optimization, but the structure I just showed you will get your basic beam-search algorithm in place!

```python
def finalize(self):
    # fill all shorter hypotheses with pad_token_id
    for index, length in enumerate(self.max_lengths):
        if length < 1:
            continue
        padding_needed = self.vocab_size - (length-1)
        assert 0 <= padding_needed <= self.k

        temp_hypothesis = torch.zeros(length)
        if padding_needed > 0 :
           left Padding needed = padding_needed*(self.vocab_size + 1)
           temp_hypothesis[padding_needed:] = torch.randint(low=self.vocab_size, high=self.vocab_size+1, size=(padding_needed,))
        else:
            left_padding_needed = 0
            temp_hypothesis[: padding_needed] = torch.zeros(padding_needed)

        self.hypotheses[index] += temp_hypothesis


    # fill the rest of each hypothesis with EOS
    for index in range(len(self.hypotheses)):
        eos_token_id = len(self.vocab_size) +1 
        temp_hypothesis = self.hypotheses[index]
        last_padding_needed = max(0, (len(temp_hypothesis)-1-self.max_lengths[index]))
        if last_padding_needed> 0:
            self.hypotheses[index] += torch.zeros(last_padding_needed)
```


key: chroma:document
string_value:
CODE:

class BARTBeamSearchGenerator(BARTGenerator):
    def __init__(self, model):
        super().__init__(model)
        self.beam_scorer = BeamSearchScorerTS()
        self.device = model.device

    @staticmethod
    def _expand_inputs_for_generation(
        input_ids: torch.Tensor,
        attention_mask: torch.Tensor,
        last_hidden_state: torch.Tensor,
        expand_size: int = 1,
    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        expanded_return_idx = (
            torch.arange(input_ids.shape[0]).view(-1, 1).repeat(1, expand_size).view(-1).to(input_ids.device)
        )
        input_ids = input_ids.index_select(0, expanded_return_idx)

        attention_mask = attention_mask.index_select(0, expanded_return_idx)

        last_hidden_state = last_hidden_state.index_select(0, expanded_return_idx.to(last_hidden_state.device))
        return input_ids, attention_mask, last_hidden_state

    def adjust_logits_during_generation(self, logits, cur_len: int, max_length: int):
        if cur_len == 1 and self.config.force_bos_token_to_be_generated:
            logits = self._force_token_id_to_be_generated(logits, self.config.bos_token_id)
        elif cur_len == max_length - 1 and self.config.eos_token_id is not None:
            logits = self._force_token_id_to_be_generated(logits, self.config.eos_token_id)
        return logits

    @staticmethod
    def _force_token_id_to_be_generated(scores, token_id: int):
        """force one of token_ids to be generated by setting prob of all other tokens to 0 (logprob=-float("inf"))"""
        mask = torch.full_like(scores, 1, dtype=torch.bool)
        mask[:, token_id] = False
        return scores.masked_fill(mask, -float("inf"))

    def _reorder_cache(self, past: List[torch.Tensor], beam_idx):
        # if decoder past is not included in output
        # speedy decoding is disabled and no need to reorder
        reordered_decoder_past = []
        for state in past:
            reordered_decoder_past.append(state.index_select(0, beam_idx))
        return reordered_decoder_past

    def beam_search(
        self, input_ids, encoder_output, attention_mask, num_beams, max_length, pad_token_id: int, eos_token_id: int
    ):
        batch_size = self.beam_scorer.batch_size

        num_beams = self.beam_scorer.num_beams
        batch_beam_size, cur_len = input_ids.shape

        assert (
            num_beams * batch_size == batch_beam_size
        ), f"Batch dimension of `input_ids` should be {num_beams * batch_size}, but is {batch_beam_size}."

        beam_scores = torch.zeros((batch_size, num_beams), dtype=torch.float, device=input_ids.device)
        beam_scores[:, 1:] = -1e9
        beam_scores = beam_scores.view((batch_size * num_beams,))
        next_tokens = torch.zeros((batch_size, num_beams), dtype=torch.long, device=input_ids.device)
        next_indices = torch.zeros((batch_size, num_beams), dtype=torch.long, device=input_ids.device)

        past: List[torch.Tensor] = []
        while cur_len < max_length:
            logits, past = self._decoder_forward(input_ids, encoder_output, attention_mask, past)
            next_token_logits = logits[:, -1, :]

            # adjust tokens for Bart, *e.g.*
            next_token_logits = self.adjust_logits_during_generation(
                next_token_logits, cur_len=cur_len, max_length=max_length
            )

            next_token_scores = F.log_softmax(next_token_logits, dim=-1)  # (batch_size * num_beams, vocab_size)

            # pre-process distribution
            next_token_scores = self.logits_processor(input_ids, next_token_scores)
            next_token_scores = next_token_scores + beam_scores[:, None].expand_as(next_token_scores)

            # reshape for beam search
            vocab_size = next_token_scores.shape[-1]
            next_token_scores = next_token_scores.view(batch_size, num_beams * vocab_size)

            next_token_scores, next_tokens = torch.topk(
                next_token_scores, 2 * num_beams, dim=1, largest=True, sorted=True
            )

            next_indices = next_tokens // vocab_size
            next_tokens = next_tokens % vocab_size

            beam_scores, beam_next_tokens, beam_idx = self.beam_scorer.process(
                input_ids,
                next_token_scores,
                next_tokens,
                next_indices,
                pad_token_id=pad_token_id,
                eos_token_id=eos_token_id,
            )

            input_ids = torch.cat([input_ids[beam_idx, :], beam_next_tokens.unsqueeze(-1)], dim=-1)

            cur_len = cur_len + 1

            if len(past) > 0:
                past = self._reorder_cache(past, beam_idx)

            if self.beam_scorer.is_done():
                break

        sequences, sequence_scores = self.beam_scorer.finalize(
            input_ids,
            beam_scores,
            next_tokens,
            next_indices,
            pad_token_id=pad_token_id,
            eos_token_id=eos_token_id,
        )

        return sequences

    def forward(self, input_ids, attention_mask, num_beams, max_length, decoder_start_token_id):
        pad_token_id = self.config.pad_token_id
        bos_token_id = self.config.bos_token_id
        eos_token_id = self.config.eos_token_id

        # special case if pad_token_id is not defined
        if pad_token_id is None and eos_token_id is not None:
            # logger.warning(f"Setting `pad_token_id` to `eos_token_id`:{eos_token_id} for open-end generation.")
            pad_token_id = eos_token_id

        encoder_output = self._encoder_forward(input_ids, attention_mask)

        input_ids = self._prepare_decoder_input_ids_for_generation(
            input_ids,
            decoder_start_token_id=decoder_start_token_id,
            bos_token_id=bos_token_id,
        )

        batch_size = input_ids.shape[0]

        length_penalty = self.config.length_penalty
        num_return_sequences = self.config.num_return_sequences
        early_stopping = True

        self.beam_scorer.init(
            batch_size=batch_size,
            max_length=max_length,
            num_beams=num_beams,
            device=self.device,
            length_penalty=length_penalty,
            do_early_stopping=early_stopping,
            num_beam_hyps_to_keep=num_return_sequences,
        )

        input_ids, attention_mask, encoder_output = self._expand_inputs_for_generation(
            input_ids,
            attention_mask,
            encoder_output,
            expand_size=num_beams,
        )

        return self.beam_search(
            input_ids=input_ids,
            encoder_output=encoder_output,
            attention_mask=attention_mask,
            num_beams=num_beams,
            max_length=max_length,
            pad_token_id=pad_token_id,
            eos_token_id=eos_token_id,


SUMMARY:
**Class Summary: BARTBeamSearchGenerator**

The `BARTBeamSearchGenerator` class generates text sequences using beam search. It is an implementation of the BART model.

**Main Purpose and Functionality:** The primary function of this class is to perform beam search generation on a given input sequence, taking into account various parameters such as maximum length and number of beams.

**Implementation Details:**

* Uses `BeamSearchScorerTS` for scoring beam search candidates.
* Adjusts logits during generation using the `adjust_logits_during_generation` method.
* Utilizes `_force_token_id_to_be_generated` to force tokens to be generated in certain configurations.
* Applies `_reorder_cache` to reorder decoder past states at each generation step.

**Notable Dependencies and Requirements:**

* `BeamSearchScorerTS`
* `torch` (TensorFlow or PyTorch)
* `F.log_softmax`

This class is part of a larger BART model implementation, which suggests that it may be used for tasks such as text classification, language translation, or generation.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/onnx/summarization/bart_onnx/reduce_onnx_size.py


key: chroma:document
string_value:
CODE:

def _is_equal_tensor_proto(a, b):
    name_a = a.name
    name_b = b.name

    a.name = ""
    b.name = ""

    res = a == b

    a.name = name_a
    b.name = name_b



SUMMARY:
**Summary**

The `_is_equal_tensor_proto` function checks if two tensor proto objects (`a` and `b`) are equal by defaulting to empty names, comparing their equality without considering actual names. It sets default names (`name_a` and `name_b`) after the comparison, restoring original values before returning the result.

*Primary purpose*: Compares tensor proto objects regardless of their actual names.
*Narrowed key implementation details*:

* Uses default name values for comparison.
* Restores original name values before returning result.

**Notable dependencies or requirements**:

* `a` and `b` are expected to be tensor(proto) objects (`name_a`, `name_b` attributes).
* External dependency: TensorFlow Python API (_proto` module).

**Method signatures and interfaces**: None.


key: chroma:document
string_value:
CODE:

def _node_replace_input_with(node_proto, name, new_name):
    for i, input_name in enumerate(node_proto.input):
        if input_name == name:
            node_proto.input.insert(i, new_name)
            node_proto.input.pop(i + 1)

    if node_proto.op_type == "If":
        _graph_replace_input_with(node_proto.attribute[0].g, name, new_name)
        _graph_replace_input_with(node_proto.attribute[1].g, name, new_name)
    if node_proto.op_type == "Loop":


SUMMARY:
**Summary:**

This function, `_node_replace_input_with`, replaces the specified input `name` with a new value `new_name` within a given graph node `node_proto`. It handles two main types of nodes:

1. **Literal Node**: Replaces its single input `name`.
2. **Constructive Nodes** (`If` and `Loop>`):
	* Replaces both inputs using `_graph_replace_input_with`, which suggests the use of another function or method (`_`) that replaces inputs in a separate graph object (`g`).

**Key Implementation Details:**

* Utilizes `enumerate` to iterate over node inputs.
* Modifies the node's input list by inserting and then removing elements to achieve replacement.
* Uses type checking for nodes (`op_type == "If"` or `"Loop"`).

**Notable Dependencies and Requirements:**

* The function relies on another function (`_graph_replace_input_with`) that isn't shown in this code chunk, suggesting an external dependency.


key: chroma:document
string_value:
CODE:

def _graph_replace_input_with(graph_proto, name, new_name):
    for n in graph_proto.node:


SUMMARY:
**Summary**

The `_graph_replace_input_with` function takes three parameters: `graph_proto`, `name`, and `new_name`. It iterates through the nodes of an adjacency list in the graph prototype (`graph_proto.node`) and calls the internal `_node_replace_input_with` function to replace input connections for each node, using the provided old name (`name`) and new name (`new_name`). 

**Key Details**

- Iterates through `graph_proto.node`.
- Calls `_node_replace_input_with(n, name, new_name)` for each node.
- Replaces input connections of each node.

**Note**

This function appears to be part of a graph rewriting system. The actual purpose and context are implied by the underlying data structures and file names.


key: chroma:document
string_value:
CODE:

def _remove_dup_initializers_from_model(model, model_without_ext, ind_to_replace):
    inits_with_data = list(model.graph.initializer)
    inits = list(model_without_ext.graph.initializer)
    for i, ref_i in ind_to_replace:
        assert inits_with_data[i].name == inits[i].name
        assert inits_with_data[ref_i].name == inits[ref_i].name
        assert i > ref_i

        name_i = inits[i].name
        name_ref = inits[ref_i].name

        model_without_ext.graph.initializer.remove(inits[i])

        # for n in model.graph.node:


SUMMARY:
**Function Summary: `_remove_dup_initializers_from_model`**

Removes duplicate initializer nodes from a model while preserving data consistency.

*   Primary purpose:
    *   Update `model_without_ext.graph.initializer` to remove duplicate initializer nodes (i.e., nodes with the same name).
    *   Preserve data consistency by replacing nodes in `model_without_ext.graph` with their corresponding counterparts in `model.graph`.

**Key Implementation Details:**

*   Iteratively removes duplicate initializer nodes from `modelWithoutExt.graph`.
*   Replaces nodes using `_graph_replace_input_with` to maintain data integrity.

**Notable Dependencies and Requirements:**

*   `model` and `model_without_ext`: These are instances of a graph model, with their respective initializer nodes.
*   `ind_to_replace`: A list of tuples containing indexing information for duplicate node removal.
*   `_graph_replace_input_with`: An external function that replaces input nodes in one graph with another.


key: chroma:document
string_value:
CODE:

def remove_dup_initializers(onnx_file_path):
    """
    Removes duplicate initializers from the model to reduce its size.
    Writes a new file in the same directory as onnx_file_path and returns the path to that file.
    """

    model_file_folder = os.path.dirname(onnx_file_path)
    model_file_name = os.path.basename(onnx_file_path)

    model = onnx.load(os.path.join(model_file_folder, model_file_name))

    inits = list(model.graph.initializer)

    dup_set = set()
    dup_map = {}
    ind_to_replace = []

    total_reduced_size = 0

    for i in range(len(inits)):
        if i in dup_set:
            continue

        for j in range(i + 1, len(inits)):
            if j in dup_set:
                continue
            if _is_equal_tensor_proto(inits[i], inits[j]):
                dup_set.add(i)
                dup_set.add(j)

                dtype = inits[j].data_type
                mem_size = numpy.prod(inits[j].dims)
                if dtype == 1:
                    mem_size *= 4
                elif dtype == 6:
                    mem_size *= 4
                elif dtype == 7 or dtype == 11:
                    mem_size *= 8
                else:
                    print("unexpected data type: ", dtype)
                total_reduced_size += mem_size

                name_i = inits[i].name
                name_j = inits[j].name

                if name_i in dup_map:
                    dup_map[name_i].append(name_j)
                else:
                    dup_map[name_i] = [name_j]
                ind_to_replace.append((j, i))

    print("total reduced size: ", total_reduced_size / 1024 / 1024 / 1024, "GB")

    ind_to_replace = sorted(ind_to_replace)
    _remove_dup_initializers_from_model(model, model, ind_to_replace)

    optimized_model_file_name = "optimized_" + model_file_name
    new_model = os.path.join(model_file_folder, optimized_model_file_name)
    onnx.save(model, new_model)



SUMMARY:
**Function Summary**

`remove_dup_initializers(onnx_file_path)`: Removes duplicate initializers from an ONNX model to reduce its size.

**Primary Purpose**
Removes duplicate initializers from the provided ONNX model and returns the path to the optimized model file.

**Key Implementation Details**

* Iterates through the initializers in the model graph, identifying duplicates based on `_is_equal_tensor_proto`.
* Replaces identified duplicates with an instance of type 1 (4-byte), type 6 (4-byte), or type 7/11 (8-byte) tensor to reduce memory usage.
* Preserves the original names of replaced initializers and stores them in the `dup_map` dictionary.

**External Dependencies**

* ONNX library for model loading and saving
* NumPy for tensor dimension calculations

** Interfaces and Methods**

* `_is_equal_tensor_proto`: Not shown in provided code snippet, but likely a function checking if two tensors are equal.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/codeparrot/scripts/bpe_training.py


key: chroma:document
string_value:
CODE:
# Iterator for Training
def batch_iterator(batch_size=10):
    for _ in tqdm(range(0, args.n_examples, batch_size)):


SUMMARY:
**Concise Summary**

`batch_iterator`: Generates batches of text data from `iter_dataset`, yielding up to `batch_size` words at a time.

### Key Features

- Iterative generation of fixed-size text batches
- Uses `tqrm` for progress monitoring in the generator loop
- Utilizes `args.text_column` to extract text data from datasets
- Consistent batch size controlled via `batch_size` parameter

### Notable Dependencies and Interfaces

- `iter_dataset`: A dataset object with a `text_column`
- External dependency: `tqdm` library for progress monitoring support
- Interface: Utilizes variable arguments (`args`) not defined within this function


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/codeparrot/scripts/codeparrot_training.py


key: chroma:document
string_value:
CODE:

class ConstantLengthDataset(IterableDataset):
    """
    Iterable dataset that returns constant length chunks of tokens from stream of text files.
        Args:
            tokenizer (Tokenizer): The processor used for proccessing the data.
            dataset (dataset.Dataset): Dataset with text files.
            infinite (bool): If True the iterator is reset after dataset reaches end else stops.
            seq_length (int): Length of token sequences to return.
            num_of_sequences (int): Number of token sequences to keep in buffer.
            chars_per_token (int): Number of characters per token used to estimate number of tokens in text buffer.
            tokenized (bool): If true we use a pretokenized dataset.
    """

    def __init__(
        self,
        tokenizer,
        dataset,
        infinite=False,
        seq_length=1024,
        num_of_sequences=1024,
        chars_per_token=3.6,
        tokenized=False,
    ):
        self.tokenizer = tokenizer
        self.concat_token_id = tokenizer.bos_token_id
        self.dataset = dataset
        self.seq_length = seq_length
        self.epoch = 0
        self.infinite = infinite
        self.current_size = 0
        self.tokenized = tokenized

        if self.tokenized:
            self.max_buffer_size = seq_length * num_of_sequences
            self.content_field = "input_ids"
        else:
            self.max_buffer_size = seq_length * chars_per_token * num_of_sequences
            self.content_field = "content"

    def __iter__(self):
        iterator = iter(self.dataset)
        more_examples = True
        while more_examples:
            buffer, buffer_len = [], 0
            while True:
                if buffer_len >= self.max_buffer_size:
                    break
                try:
                    buffer.append(next(iterator)[self.content_field])
                    buffer_len += len(buffer[-1])
                except StopIteration:
                    if self.infinite:
                        iterator = iter(self.dataset)
                        self.epoch += 1
                        logger.info(f"Dataset epoch: {self.epoch}")
                    else:
                        more_examples = False
                        break
            if self.tokenized:
                tokenized_inputs = buffer
            else:
                tokenized_inputs = self.tokenizer(buffer, truncation=False)["input_ids"]
            all_token_ids = []
            for tokenized_input in tokenized_inputs:
                all_token_ids.extend(tokenized_input + [self.concat_token_id])
            for i in range(0, len(all_token_ids), self.seq_length):
                input_ids = all_token_ids[i : i + self.seq_length]
                if len(input_ids) == self.seq_length:
                    self.current_size += 1
                    yield torch.tensor(input_ids)

    def shuffle(self, buffer_size=1000):


SUMMARY:
**Class Summary: ConstantLengthDataset**

The `ConstantLengthDataset` class is an iterable dataset that returns constant length chunks of tokens from text files. It processes data using a provided `Tokenizer`.

**Primary Purpose:**
Return constant length token sequences from streams of text files.

**Key Implementation Details:**

* Uses `Tokenizer` to process and tokenize input data.
* Sets `infinite` parameter to control whether the iterator resets after reaching dataset end or stops.
* Employs buffer-based processing with `max_buffer_size` to manage sequence lengths.
* Utilizes `truncate=False` for non-packed tokenization.

**Notable Dependencies/Requirements:**

* `Tokenizer`: Used for processing and tokenizing input data.
* `dataset.Dataset`: Input dataset containing text files.
* `shuffler IterDataPipe`: Class used for shuffling the dataset buffer.

**Interface Signatures:**

* `__init__`: Initializes the dataset with various parameters.
* `__iter__`: Returns constant length chunks of tokens from the input data.
* `shuffle`: Shuffles the dataset buffer using the provided `ShufflerIterDataPipe` class.


key: chroma:document
string_value:
CODE:

def setup_logging(args):
    project_name = args.model_ckpt.split("/")[-1]
    logger = logging.getLogger(__name__)
    log_dir = Path(args.save_dir) / "log/"
    log_dir.mkdir(exist_ok=True)
    filename = f"debug_{accelerator.process_index}.log"
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO,
        handlers=[logging.FileHandler(log_dir / filename), logging.StreamHandler()],
    )
    if accelerator.is_main_process:  # we only want to setup logging once
        accelerator.init_trackers(project_name, vars(args))
        run_name = accelerator.trackers[0].run.name
        logger.setLevel(logging.INFO)
        datasets.utils.logging.set_verbosity_info()
        transformers.utils.logging.set_verbosity_info()
    else:
        run_name = ""
        logger.setLevel(logging.ERROR)
        datasets.utils.logging.set_verbosity_error()
        transformers.utils.logging.set_verbosity_error()


SUMMARY:
**Summary: Setup Logging Functionality**

The `setup_logging` function configures logging for a project. It takes in `args` and returns a loggable object (`logger`) along with a `run_name`.

**Key Points:**

* Uses `logging` to create a logger instance, named after the current module.
* Sets up two handlers: a file handler and a stream handler, logging to both a file and the console.
* Configures basic logging format and levels based on whether the model is running on the main process or not.

**Implementation Details:**

* Uses `accelerator.process_index` to determine the log filename.
* Sets `logging.level` differently for main and non-main processes.
* Calls custom functions from `datasets.utils.logging` and `transformers.utils.logging` to set verbosity levels.

**Notable Dependencies/Requirements:**

* `logging`
* `Path` (from `pathlib`)
* `accelerator` (Torch distributed library)
* `torchvision.datasets` and `transformers` modules


key: chroma:document
string_value:
CODE:

def create_dataloaders(args):
    ds_kwargs = {"streaming": True}
    train_data = load_dataset(args.dataset_name_train, split="train", **ds_kwargs)
    train_data = train_data.shuffle(buffer_size=args.shuffle_buffer, seed=args.seed)
    valid_data = load_dataset(args.dataset_name_valid, split="train", **ds_kwargs)
    train_dataset = ConstantLengthDataset(
        tokenizer, train_data, infinite=True, seq_length=args.seq_length, tokenized=args.tokenized
    )
    valid_dataset = ConstantLengthDataset(
        tokenizer, valid_data, infinite=False, seq_length=args.seq_length, tokenized=args.tokenized
    )
    train_dataset = train_dataset.shuffle(buffer_size=args.shuffle_buffer)
    train_dataloader = DataLoader(train_dataset, batch_size=args.train_batch_size, shuffle=True)
    eval_dataloader = DataLoader(valid_dataset, batch_size=args.valid_batch_size)


SUMMARY:
**Function Summary: `create_dataloaders`**

 Creates and returns data loaders for training and validation datasets.

**Main Purpose and Functionality:**
 The function takes in dataset-related arguments, loads the necessary datasets, and creates ConstantLengthDataset instances. It then shuffles and batch-normalizes these datasets into DataLoaders using DataLoader from a specific library (not specified).

**Key Implementation Details:**

* Uses `ConstantLengthDataset` to tokenize datasets
* Utilizes `DataLoader` for batching and shuffling

**Notable Dependencies and Requirements:**
 The function requires:

* A tokenizer (`tokenizer`)
* A library with `load_dataset`, `DataLoader`, and `ConstantLengthDataset` functions (not specified)

**Interface and Data Structures:**

* Uses `args` object as input
* Returns two instances of `DataLoader`: `train_dataloader` and `eval_dataloader`

Some key method signatures and interfaces relevant to this function are those provided by the library's `DataLoader`, such as:

* `DataLoader(batch_size)`
* `DataLoader(shuffle=True)`
* `ConstantLengthDataset(tokenizer, ...)`
* `load_dataset(dataset_name, split="train", ...)`


key: chroma:document
string_value:
CODE:

def get_grouped_params(model, args, no_decay=["bias", "ln_1.weight", "ln_2.weight", "ln_f.weight"]):
    params_with_wd, params_without_wd = [], []
    for n, p in model.named_parameters():
        if any(nd in n for nd in no_decay):
            params_without_wd.append(p)
        else:
            params_with_wd.append(p)
    return [
        {"params": params_with_wd, "weight_decay": args.weight_decay},
        {"params": params_without_wd, "weight_decay": 0.0},


SUMMARY:
**Function Summary: `get_grouped_params`**

**Primary Purpose:** Group model parameters into two categories based on weight decay settings.

**Functionality:**

* Takes in three inputs: `model`, `args`, and an optional list of parameter names to exclude from weight decay (`no_decay`).
* Groups parameters into two lists: `params_with_wd` and `params_without_wd`.
* The first list contains parameters that are not excluded from weight decay, with the specified weight decay value.
* Returns a list containing dictionaries that describe each group of parameters, including their corresponding weight decay setting.

**Key Implementation Details:**

* Uses the `named_parameters` method to iterate over model parameters, preserving exact parameter names (`n`, `p`) for reference.
* Uses list comprehension and conditional logic to determine which parameters are included in each group.

**Notable Dependencies/Requirements:**

* assumes access to a PyTorch `model` object
* relies on an optional input `no_decay` list of strings
* expects an available `args` object with a `weight_decay` attribute.


key: chroma:document
string_value:
CODE:

def log_metrics(step, metrics):
    logger.info(f"Step {step}: {metrics}")
    if accelerator.is_main_process:


SUMMARY:
**Summary:**

`log_metrics(func, params)` logs application metrics at each step. It records the current metrics using the provided `params`, and then uses an accelerator's logging mechanism to further log these metrics in the main process.

**Key Implementation Details:**

* Uses `logger.info()` for initial logging
* Conditional logging via `accelerator.log(metrics, step)` if `is_main_process` is True

**Notable Dependencies/Requirements:**

* `logger` (instance)
* `accelerator` (with `is_main_process` and `log()` attributes)


key: chroma:document
string_value:
CODE:

def compute_tflops(elapsed_time, accelerator, args):
    # TFLOPs formula (from Equation 3 in Section 5.1 of https://arxiv.org/pdf/2104.04473.pdf).
    config_model = accelerator.unwrap_model(model).config
    checkpoint_factor = 4 if args.gradient_checkpointing else 3
    batch_size = args.train_batch_size * accelerator.state.num_processes * args.gradient_accumulation_steps
    factor = 24 * checkpoint_factor * batch_size * args.seq_length * config_model.n_layer * (config_model.n_embd**2)
    flops_per_iteration = factor * (
        1.0
        + (args.seq_length / (6.0 * config_model.n_embd))
        + (tokenizer.vocab_size / (16.0 * config_model.n_layer * config_model.n_embd))
    )
    tflops = flops_per_iteration / (elapsed_time * accelerator.state.num_processes * (10**12))


SUMMARY:
**Function Summary: `compute_tflops`**

Computes the theoretical floating-point Operations Per Second (TFLOPS) for a given machine learning model.

**Main Purpose and Functionality**
This function calculates TFLOPS based on the elapsed time, accelerator configuration, and model properties. It applies a formula derived from a research paper to estimate the theoretical FLOPS of a model.

**Key Implementation Details**

* Calculates `flops_per_iteration` as a weighted sum of factors, including sequence length, vocabulary size, and layer/embdim parameters.
* Employs the TFLOPS formula from Equation 3 in Section 5.1 of the research paper.

**Notable Dependencies and Requirements**

* `torch`: Utilizes PyTorch's `Accelerator` object.
* `transformers`: Assumes a transformer model with configuration `config_model`.
* External dependencies: None noted.

**Return Value**
The function returns the estimated TFLOPS value, calculated by normalizing `flops_per_iteration` by elapsed time and number of processes.


key: chroma:document
string_value:
CODE:

def evaluate(args):
    model.eval()
    losses = []
    for step, batch in enumerate(eval_dataloader):
        with torch.no_grad():
            outputs = model(batch, labels=batch)
        loss = outputs.loss.repeat(args.valid_batch_size)
        losses.append(accelerator.gather(loss))
        if args.max_eval_steps > 0 and step >= args.max_eval_steps:
            break
    losses = torch.cat(losses)
    loss = losses[: eval_dataloader.dataset.current_size].mean()
    try:
        perplexity = torch.exp(loss)
    except OverflowError:
        perplexity = float("inf")


SUMMARY:
**Function Summary: `evaluate`**

Evaluates the model's performance on the evaluation dataset.

**Primary Purpose:** Compute and return the epoch-average loss and perplexity of the model during evaluation.

**Key Implementation Details:**

* Iterates over the evaluation dataloader, collecting losses at each step using `torch.gather`.
* Applies a validation cut-off if specified (`max_eval_steps`).
* Computes the epoch-average loss by averaging over the first `current_size` elements in the dataset.
* Calculates perplexity via exponentiation of the loss.

**Notable Dependencies/Requirements:**

* `accelerator.gather(loss)`: uses accelerator API to efficiently collect losses from multiple devices.
* `torch.no_grad()`: sets tensor computation mode for evaluation.
* `eval_dataloader`: an iterator over the evaluation dataset.
* `labels` (not explicitly defined in this snippet): assumed to be the predicted labels at each step of evaluation.


key: chroma:document
string_value:
CODE:

def get_lr():


SUMMARY:
**Summary:**

*   Function Name: `get_lr`
*   Primary Purpose: Retrieves the Learning Rate (`lr`) from the optimizer's parameter groups.
*   Implementation Details:
    *   Uses Python's dictionary access to retrieve the value at a specific key ("lr") within the first parameter group of the optimizer (denoted as index 0).
*   Notable Dependency:
    *   The `optimizer` object is assumed to be defined elsewhere in the codebase, likely referencing PyTorch's optimizer class for training models.
*   Returns: The current Learning Rate value.

**Code Chunk (excluding the analyzed function):**
```
# Assuming 'optimizer' and related imports are defined elsewhere
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
```


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/codeparrot/scripts/minhash_deduplication.py


key: chroma:document
string_value:
CODE:

def get_min_hash(tokens: List[str]) -> Optional[MinHash]:
    """Compute the MinHash of a code snippet."""
    if len(tokens) < MIN_NUM_TOKENS:
        return None
    min_hash = MinHash(num_perm=NUM_PERM)
    for token in set(tokens):
        min_hash.update(token.encode())


SUMMARY:
**Summary:** 

Computes the shortest Minimum Hash (MinHash) of a given list of code tokens.

**Key Implementation Details:**

- **Algorithm**: Uses `MinHash` to compute the MinHash of individual tokens in the input list.
- **Method Signature**: 
  ```python
def get_min_hash(tokens: List[str]) -> Optional[MinHash]:
    ...
```

**Notable Dependencies and Requirements:**
- `MinHash` class from an external library (not specified)
- `NUM_PERM` and `MIN_NUM_TOKENS` constants (not defined in this snippet)


key: chroma:document
string_value:
CODE:

def get_tokens(code: str) -> Set[str]:
    """Tokenize a code snippet."""


SUMMARY:
**Summary**

This function tokenizes a given code snippet by splitting it into individual tokens (non-alpha characters). It returns a set of tokens where each token is a non-empty string.

**Key Details:**

* Tokenizes input `code`
* Uses `NON_ALPHA` separator to split the code
* Returns a set of non-empty strings

**Implementation:**

`get_tokens(code: str) -> Set[str]:`

```python
def get_tokens(code: str) -> Set[str]:
    """
    Tokenize a code snippet.
    
    Args:
        code (str)
    
    Returns:
        A set of tokens where each token is a non-empty string.
    """
    return {t for t in NON_ALPHA.split(code) if len(t.strip()) > 0}
```

**Notable Dependencies:**

* `NON_ALPHA` (external constant or class, exact implementation not provided)
* No external dependencies listed


key: chroma:document
string_value:
CODE:

class DuplicationIndex:
    def __init__(
        self,
        *,
        duplication_jaccard_threshold: float = 0.85,
    ):
        self._duplication_jaccard_threshold = duplication_jaccard_threshold
        self._num_perm = NUM_PERM
        self._index = MinHashLSH(threshold=self._duplication_jaccard_threshold, num_perm=self._num_perm)

        self._duplicate_clusters = defaultdict(set)

    def add(self, code_key: Tuple, min_hash: MinHash) -> None:
        """Add a key to _index (MinHashLSH)
        the min_hash is used to query closest matches based on the jaccard_threshold.
        The new key is either added to a existing cluster of one close match,
        or a new cluster is created. The clusters created in this way, depend on the order of add.

        Args:
            code_key (Tuple of (index, repo_name, path)):
                Theoritically any hasbale key. Here we use a tuple to retrieve the information later.
            min_hash: MinHash of the code_key.
        """
        close_duplicates = self._index.query(min_hash)
        if code_key in self._index.keys:
            print(f"Duplicate key {code_key}")
            return

        self._index.insert(code_key, min_hash)
        if len(close_duplicates) > 0:
            for base_duplicate in close_duplicates:
                if base_duplicate in self._duplicate_clusters:
                    self._duplicate_clusters[base_duplicate].add(code_key)
                    break
            else:
                self._duplicate_clusters[close_duplicates[0]].add(code_key)

    def get_duplicate_clusters(self) -> List[List[Dict]]:
        """Export the duplicate clusters.
        For each cluster, the first element is the base element of the cluster.
        The base element has an estimation jaccard similarity higher than the threshold with all the other elements.

        Returns:
            duplicate_clusters (List[List[Dict]]):
                List of duplicate clusters.
        """
        duplicate_clusters = []
        for base, duplicates in self._duplicate_clusters.items():
            cluster = [base] + list(duplicates)
            # reformat the cluster to be a list of dict
            cluster = [{"base_index": el[0], "repo_name": el[1], "path": el[2]} for el in cluster]
            duplicate_clusters.append(cluster)
        return duplicate_clusters

    def save(self, filepath) -> None:
        duplicate_clusters = self.get_duplicate_clusters()
        with open(filepath, "w") as f:


SUMMARY:
**Summary: DuplicationIndex Class**

The `DuplicationIndex` class calculates and stores the duplication relationships between code snippets. It utilizes MinHashLSH to efficiently find similar min-hashes, and maintains a graph of duplicate clusters based on the Jaccard similarity threshold.

**Key Implementation Details:**

*   Initializes with a `_duplication_jaccard_threshold` parameter (default 0.85)
*   Uses `NUM_PERM` for `num_perm` in MinHashLSH
*   Stores cluster associations via `self._duplicate_clusters set`

**Notable Dependencies and Requirements:**

*   Requires `MinHashLSH` class from the external library (not provided)
*   Utilizes JSON library for serialization (`json.dump`)
*   Inherited interface:
    *   External dependencies: `NUM_PERM` is used but not defined in this code snippet

**Primary Purpose and Functionality:**

The primary purpose of this class is to:

1.  Calculate min-hash similarity between input codes using MinHashLSH
2.  Group duplicate clusters based on the Jaccard similarity threshold
3.  Serialize and store duplicate cluster information for future reference


key: chroma:document
string_value:
CODE:

def _compute_min_hash(element):
    index, data = element
    min_hash = get_min_hash([t for t in NON_ALPHA.split(data["content"]) if len(t.strip()) > 0])
    if min_hash is not None:


SUMMARY:
**Summary:**

`_compute_min_hash` function calculates the minimum hash of text tokens from a given element's "content". It filters out empty strings and returns a tuple containing the element's index, repository name, and path along with the calculated minimum hash.

**Key Details:**

* Filters non-alpha characters using `NON_ALPHA.split`
* Uses `get_min_hash` function to calculate the minimum hash of tokenized text
* Returns `(index, repo_name, path)`, min_hash tuple

**Notable Dependencies:**

* `get_min_hash` function (referenced by exact name)
* External dependency for `NON_ALPHA` constant


key: chroma:document
string_value:
CODE:

def minhash_iter(dataset_iterator: Type[Dataset]):
    with mp.Pool() as pool:
        for data in pool.imap_unordered(
            _compute_min_hash,
            ThreadedIterator(dataset_iterator, max_queue_size=10000),
            chunksize=100,
        ):
            if data is not None:


SUMMARY:
**minhash_iter Function Summary**

### Purpose and Functionality
The `minhash_iter` function generates an iterator over min hash values from the input dataset. It utilizes multiprocessing to compute the min hashes in parallel.

### Key Implementation Details

* Utilizes `mp.Pool` with chunksize=100 to divide workload among multiple processes for efficient computation.
* Applies `_compute_min_hash` function to each data element within a threaded iterator (max queue size: 10,000).
* Yields None if the result is empty.

### Notable Dependencies and Requirements

* `_compute_min_hash`
* `ThreadedIterator`
* `Dataset` type hint
* External dependency: multiprocessing library (`mp`)

**RAG Summary Key Terms:** minhash_iter (dataset processing), _compute_min_hash, Pool chunksize=100


key: chroma:document
string_value:
CODE:

def make_duplicate_clusters(dataset_iterator: Type[Dataset], jaccard_threshold: float):
    """Find duplicate clusters in the dataset in two steps:
    1. Compute MinHash for each code snippet. MinHash is a tool for fast jaccard similarity estimation.
    This step is computed using an asynchronous multiprocessing pool, minhash_iter
    2. Find duplicate clusters. The computed MinHash is added sequentially to the DuplicationIndex.
    This step cannot be parallelized. So using asynchronous thread in the previous step helps to speed up the process.
    """
    di = DuplicationIndex(duplication_jaccard_threshold=jaccard_threshold)

    for filename, min_hash in tqdm(ThreadedIterator(minhash_iter(enumerate(dataset_iterator)), max_queue_size=100)):
        di.add(filename, min_hash)

    # Returns a List[Cluster] where Cluster is List[str] with the filenames.


SUMMARY:
**Summary:**

This function `make_duplicate_clusters` finds duplicate clusters in a dataset by computing MinHash similarities on code snippets using an asynchronous multiprocessing pool. It maintains a `DuplicationIndex` that adds computed MinHash values to find duplicates sequentially.

**Key Details:**

* Purpose: Find duplicate clusters in the dataset
* Functionality:
	+ Compute MinHash for each code snippet (async)
	+ Add calculated MinHash to DuplicationIndex (sequential, async thread utilization)
* Notable Implementation:
	+ Uses `ThreadedIterator` with `minhash_iter` and `enumerate(dataset_iterator)`
* Dependencies: 
    * Type hinting (`Dataset`)
    * `DuplicationIndex` class
    * `tqdm` for progress bar
    * `Threading` (utility) for async thread utilization

**Return Value:** List[Cluster], where Cluster is a list of filenames.


key: chroma:document
string_value:
CODE:

def jaccard_similarity(code1: str, code2: str) -> float:
    """Compute the Jaccard similarity of two code snippets."""
    tokens1 = get_tokens(code1)
    tokens2 = get_tokens(code2)


SUMMARY:
**Summary**

The `jaccard_similarity` function calculates the Jaccard similarity between two code snippets. It:

*   Tokenizes input code using `get_tokens()` and stores in `tokens1` and `tokens2`.
*   Returns the ratio of the intersection (`&`) to the union (`|`) of the two sets of tokens.

**Key Implementation Details**

*   Uses bitwise operators for set operations (`&` and `|`).
*   Assumes a `get_tokens()` function that returns a set of unique tokens from code strings.
*   Primary purpose: compute Jaccard similarity between two code snippets.

**Dependencies/Requirements**

*   `get_tokens()` external function or class (not shown in this snippet).


key: chroma:document
string_value:
CODE:

def _find_cluster_extremes_shared(cluster, jaccard_threshold):
    """Find a reduced cluster such that each code in the origin cluster is similar to at least one code in the reduced cluster.
    Two codes are similar if their Jaccard similarity is above the threshold.

    Args:
        cluster (List[dict]):
           cluster is a list of dict, each dict contains the following keys:
                - base_index
                - repo_name
                - path
            This is a typical output of DuplicationIndex.get_duplicate_clusters()
        jaccard_threshold (float):
            threshold for Jaccard similarity.
            Two codes are similar if their Jaccard similarity is above the threshold.

    Returns:
        extremes (List[dict]):
            A reduced representation of the cluster. The field copies is added to each dict.
            The copies field indicates the number of similar codes in the cluster for a extreme.
    """
    extremes = []
    for element1 in cluster:
        code1 = _shared_dataset[element1["base_index"]]["content"]
        for element2 in extremes:
            code2 = _shared_dataset[element2["base_index"]]["content"]
            if jaccard_similarity(code1, code2) >= jaccard_threshold:
                element2["copies"] += 1
                break
        else:
            element1["copies"] = 1
            extremes.append(element1)


SUMMARY:
**Function Summary: `_find_cluster_extremes_shared`**

**Purpose and Functionality:**
The function finds a reduced cluster of similar items from the original cluster, ensuring each original item co-exists with at least one other item in the reduced cluster.

**Key Implementation Details:**

* Iterates over each item in the `cluster`.
* For each item, searches for similarities between it and already selected extreme items.
* If a similarity threshold is exceeded, increments the "copies" count for that extreme item.
* Adds the original item to the extremes list if no similar extreme exists.

**Notable Dependencies/Requirements:**

* `_shared_dataset` ( referenced by `element1["base_index"]` and `element2["base_index"]`)
* Jaccard similarity function (`jaccard_similarity`)


key: chroma:document
string_value:
CODE:

def find_extremes(cluster_list, dataset, jaccard_threshold):
    """Call the _find_cluster_extremes_shared function in a parallel fashion.

    Args:
        cluster_list (List[List[Dict]]):
            each cluster is a list of dicts with the key base_index,
            referring to the index of the base code in the dataset.
        dataset (Type[Dataset]):
            dataset is used to access the content of the code snippets,
            using the base_index from the cluster_list.
            dataset is shared between all the processes using a glabal variable (any other way to share the dataset?),
            otherwise the multi processing is not speeded up.
        jaccard_threshold (float):
            the threshold for the jaccard similarity. The default value is 0.85

    Returns:
        extremes_list (List[Dict]):
            Each cluster is reduced to extremes.
            See _find_cluster_extremes_shared for the definition of extremes.
    """
    global _shared_dataset
    _shared_dataset = dataset
    extremes_list = []
    f = partial(_find_cluster_extremes_shared, jaccard_threshold=jaccard_threshold)
    with mp.Pool() as pool:
        for extremes in tqdm(
            pool.imap_unordered(
                f,
                cluster_list,
            ),
            total=len(cluster_list),
        ):
            extremes_list.append(extremes)


SUMMARY:
**Summary: Compute Extreme Points in Clusters**

 This function, `find_extremes`, takes three inputs:

*   `cluster_list`: A list of clusters where each cluster is a list of dictionaries containing base indices.
*   `dataset`: An instance of the `Dataset` class, providing access to code snippets by their base indices.
*   `jaccard_threshold` (default 0.85): Thethreshold for Jaccard similarity.

**Returns:**

*   `extremes_list`: A list of lists containing extreme points for each cluster.

**Implementation Details:**

*   Utilizes parallel processing with `multiprocessing.Pool`.
*   Leverages `_find_cluster_extremes_shared` function, modified to accept the `jaccard_threshold` parameter.
*   Employs `tqdm` for tqdm progress bars.

**Notable Dependencies and Requirements:**

*   `_shared_dataset`: A shared variable containing the dataset instance (`dataset`).
*   `mp.Pool`, `impUnordered`, and `partial` are used from the `multiprocessing` library.
*   The `_find_cluster_extremes_shared` function is referenced, but its implementation remains unknown in this summary.


key: chroma:document
string_value:
CODE:

def deduplicate_dataset(
    dataset: Type[Dataset], jaccard_threshold: float = 0.85
) -> Tuple[Type[Dataset], List[List[Dict]]]:
    """Deduplicate the dataset using minhash and jaccard similarity.
    This function first generate duplicate clusters, then each cluster
    is reduced to the extremes that are similar to the other elements in the cluster.
    Codes are called similar if their Jaccard similarity is greater than jaccard_threshold (0.85 default).

    Args:
        dataset (Type[Dataset]):
            The dataset to deduplicate.
        jaccard_threshold (float, default=0.85):
            jaccard threshold to determine if two codes are similar

    Returns:
        ds_dedup (Type[Dataset]):
            The deduplicated dataset.
        duplicate_clusters (List[List[Dict]]):
            The list of duplicate clusters.
            Each cluster is a list of dicts with the following keys:
            - base_index : int
                The index of the code in the original dataset.
            - repo_name : str
            - path : str
            - copies : int
                The number of copies of the code in the cluster. (find_cluster_extremes)
            - is_extreme : bool
                Whether the code is an extreme in the cluster.
            All the codes in the cluster are removed from the dataset except the extremes.

    Example:
        >>> from datasets import load_dataset
        >>> from minhash_deduplication import deduplicate_dataset
        >>> ds = load_dataset("lvwerra/codeparrot-clean", split="train")
        >>> ds_dedup, duplicate_clusters = deduplicate_dataset(ds, jaccard_threshold=0.85)
    """
    duplicate_clusters = make_duplicate_clusters(dataset, jaccard_threshold)
    duplicate_indices = {x["base_index"] for cluster in duplicate_clusters for x in cluster}
    extreme_dict = {}
    extremes_clusters = find_extremes(duplicate_clusters, dataset, jaccard_threshold)
    for extremes in extremes_clusters:
        for element in extremes:
            extreme_dict[element["base_index"]] = element
    remove_indices = duplicate_indices - set(extreme_dict.keys())
    ds_filter = dataset.filter(lambda x, idx: idx not in remove_indices, with_indices=True)

    # update duplicate_clusters
    for cluster in duplicate_clusters:
        for element in cluster:
            element["is_extreme"] = element["base_index"] in extreme_dict
            if element["is_extreme"]:
                element["copies"] = extreme_dict[element["base_index"]]["copies"]

    print(f"Original dataset size: {len(dataset)}")
    print(f"Number of duplicate clusters: {len(duplicate_clusters)}")
    print(f"Files in duplicate cluster: {len(duplicate_indices)}")
    print(f"Unique files in duplicate cluster: {len(extreme_dict)}")
    print(f"Filtered dataset size: {len(ds_filter)}")



SUMMARY:
**Summary**

The `deduplicate_dataset` function generates duplicate clusters from the input `dataset` using minhash and Jaccard similarity, then reduces each cluster to its extremes. The function:

1. Generates duplicate clusters (`duplicate_clusters`) using the `make_duplicate_clusters` function.
2. Identifies extreme elements in each cluster (`extremes_clusters`) using the `find_extremes` function.
3. Removes duplicate files from the dataset by filtering out indices present in non-extreme elements.

**Implementation Details**

* `make_duplicate_clusters(dataset, jaccard_threshold)`: generates duplicate clusters using minhash and Jaccard similarity.
* `find_extremes(duplicate_clusters, dataset, jaccard_threshold)`: identifies extreme elements in each cluster.
* `dataset.filter(lambda x, idx: idx not in remove_indices, with_indices=True)`: filters out duplicate files from the dataset.

**Example**

`ds_dedup, duplicate_clusters = deduplicate_dataset(ds, jaccard_threshold=0.85)`


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/codeparrot/scripts/pretokenizing.py


key: chroma:document
string_value:
CODE:

def tokenize(example):
    output = {}
    output["input_ids"] = tokenizer(example["content"], truncation=False)["input_ids"]
    output["ratio_char_token"] = len(example["content"]) / len(output["input_ids"])


SUMMARY:
**Function Summary: `tokenize`**

* **Purpose:** Tokenizes input text using a tokenizer, extracts relevant information, and calculates character token ratio.

**Key Implementation Details:**

* Utilizes the `tokenizer` function from a named dataset (`example=["content"]`) with `truncation=False`.
* Extracts:
	+ `input_ids`: Tokenized input IDs as a list.
	+ `ratio_char_token`: Character-to-token ratio computed as `len(content) / len(input_ids)`.

**Notable Dependencies/Requirements:**

* `tokenizer` function from named dataset (`example=["content"]`)
* No external dependencies listed.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/codeparrot/scripts/preprocessing.py


key: chroma:document
string_value:
CODE:

def get_hash(example):
    """Get hash of content field."""


SUMMARY:
**Summary**

`get_hash` function calculates and returns the MD5 hash of the `content` field from the provided `example` object. It uses regular expression substitution to remove specified characters (`PATTERN`) before hashing.

**Key Implementation Details:**

* Calculates MD5 hash using `insecure_hashlib.md5()`
* Removes specified characters from `content` field using `re.sub(PATTERN, "", ...)`

**Notable Dependencies and Requirements:**

* `hashlib`: used for secure MD5 hashing
* `PATTERN`: regular expression pattern to be removed (not defined in the snippet)
* `re`: module for regular expressions


key: chroma:document
string_value:
CODE:

def line_stats(example):
    """Calculates mean and max line length of file."""
    line_lengths = [len(line) for line in example["content"].splitlines()]


SUMMARY:
**Function Summary**

`line_stats(example)` calculates the mean and maximum line length of a file by processing the content.

**Key Details:**

* Uses list comprehension to calculate line lengths with `len(line) for line in example["content"].splitlines()`.
* Returns a dictionary containing "line_mean" (mean line length) and "line_max" (maximum line length).

**Notes:**
 
No external dependencies are explicitly mentioned. This function utilizes NumPy's `np.mean` function, which is assumed to be imported from the relevant module (`import numpy as np`).


key: chroma:document
string_value:
CODE:

def alpha_stats(example):
    """Calculates mean and max line length of file."""
    alpha_frac = np.mean([c.isalnum() for c in example["content"]])


SUMMARY:
**Summary:**

This function, `alpha_stats`, calculates the average fraction of alphanumeric characters (`alpha_frac`) from a given text content. It returns a dictionary with the calculated value.

**Detailed Analysis:**

* **Purpose:** Calculates the average proportion of alphanumeric characters in a file's content.
* **Key Implementation Details:**
	+ Uses NumPy to calculate `np.mean` of a list comprehension filtering text non-alphanumeric characters (`example["content"]`).
	+ Returns a dictionary with a single key-value pair containing the calculated `alpha_frac` value.
* **Notable Dependencies/Requirements:**
	+ `numpy` library (referenced by `np`) for numerical computations.


key: chroma:document
string_value:
CODE:

def check_uniques(example, uniques):
    """Check if current hash is still in set of unique hashes and remove if true."""
    if example["hash"] in uniques:
        uniques.remove(example["hash"])
        return True
    else:


SUMMARY:
**Summary:**

`check_uniques` function checks if the hash from an `example` dictionary is present in a set of unique hashes (`uniques`). If found, it removes the hash from the set and returns `True`. Otherwise, it returns `False`.

**Key Implementation Details:**

* Checks for presence of `example["hash"]` in `uniques`
* Removes specified hash if found
* Returns `True` on removal or `False` otherwise

**Notes:**

Neither `example` nor `uniques` are explicitly analyzed as they are dictionary types with no specific characteristics mentioned.
No external dependencies referenced.


key: chroma:document
string_value:
CODE:

def is_autogenerated(example, scan_width=5):
    """Check if file is autogenerated by looking for keywords in the first few lines of the file."""
    keywords = ["auto-generated", "autogenerated", "automatically generated"]
    lines = example["content"].splitlines()
    for _, line in zip(range(scan_width), lines):
        for keyword in keywords:
            if keyword in line.lower():
                return {"autogenerated": True}
    else:


SUMMARY:
**Concise Summary**

This function, `is_autogenerated`, determines whether a file is autogenerated by searching for specific keywords in the first few lines (`scan_width=5`) of the file content. The function checks if any of the predefined keywords ("auto-generated", "autogenerated", or "automatically generated") are present in the lowercased line text.

**Key Details:**

* `scan_width`: specifies the number of lines to check at the beginning of the file
* Keyword search is performed on lowercased line text
* Returns a dictionary indicating autogen status (`{"autogenerated": True}` or `{"aut autogenerated": False}"`)

**Notable Dependencies:**

* None explicitly mentioned, but assumes access to the `example["content"]` attribute.


key: chroma:document
string_value:
CODE:

def is_config_or_test(example, scan_width=5, coeff=0.05):
    """Check if file is a configuration file or a unit test by :
    1- looking for keywords in the first few lines of the file.
    2- counting number of occurrence of the words 'config' and 'test' with respect to number of lines.
    """

    keywords = ["unit tests", "test file", "configuration file"]
    lines = example["content"].splitlines()
    count_config = 0
    count_test = 0
    # first test
    for _, line in zip(range(scan_width), lines):
        for keyword in keywords:
            if keyword in line.lower():
                return {"config_or_test": True}
    # second test
    nlines = example["content"].count("\n")
    threshold = int(coeff * nlines)
    for line in lines:
        count_config += line.lower().count("config")
        count_test += line.lower().count("test")
        if count_config > threshold or count_test > threshold:
            return {"config_or_test": True}


SUMMARY:
**Function Summary**

```python
is_config_or_test(example)
```

The `is_config_or_test` function determines whether a file is a configuration file or a unit test by analyzing its contents. It takes three parameters:

*   `example`: A dictionary containing the file's content, which is split into lines.
*   `scan_width`: The number of leading lines to inspect for keywords (default: 5).
*   `coeff`: A coefficient used to calculate a threshold value based on the total line count (default: 0.05).

The function checks for two types of files:

1.  **File with specific keywords**: It searches for keywords in the first few leading lines.
2.  **File with ratio-based keyword occurrence**: It counts the occurrence of "config" and "test" keywords in relation to the total line count.

If either type matches, the function returns `{"config_or_test": True}`; otherwise, it returns `{"config_or_test": False}`.

**Key Implementation Details**

*   **Keyword search**: Checks for ["unit tests", "test file", "configuration file"] in the first leading lines.
*   **Ratio-based keyword count**: Uses a threshold value calculated based on `coeff` and total line count to determine the existence of a configuration or unit test.

**Notable Dependencies and Requirements**

*   None explicitly mentioned, but assumes access to the `example` dictionary containing "content" and potentially other relevant data.


key: chroma:document
string_value:
CODE:

def has_no_keywords(example):
    """Check if a python file has none of the keywords for: funcion, class, for loop, while loop."""
    keywords = ["def ", "class ", "for ", "while "]
    lines = example["content"].splitlines()
    for line in lines:
        for keyword in keywords:
            if keyword in line.lower():
                return {"has_no_keywords": False}


SUMMARY:
**Summary:**

### Functionality:

The `has_no_keywords` function checks if a given Python file has no occurrences of the keywords 'def ', 'class ', 'for ', or 'while '. It achieves this by splitting the file content into lines and searching for these keywords in a case-insensitive manner.

### Key Implementation Details:

* The function uses a predefined list of keywords (`keywords`) to search for.
* It searches for each keyword in every line of the file's content using a nested loop structure.
* Upon finding any keyword, it immediately returns a dictionary with `has_no_keywords` set to `False`.
* If no keywords are found after checking all lines, it returns a dictionary with `has_no_keywords` set to `True`.

### Dependencies and Requirements:

- The function relies on the string method `lower()` from Python's standard library for case-insensitive keyword searching.
- External dependencies or interfaces:
  - None explicitly mentioned.

Note: The provided explanation focuses solely on key implementation details related to how the code operates, assuming common knowledge about the context of the code, like it being used in a file analysis process.


key: chroma:document
string_value:
CODE:

def has_few_assignments(example, minimum=4):
    """Check if file uses symbol '=' less than `minimum` times."""
    lines = example["content"].splitlines()
    counter = 0
    for line in lines:
        counter += line.lower().count("=")
        if counter > minimum:
            return {"has_few_assignments": False}


SUMMARY:
**Summary**

This function `has_few_assignments` checks if a given file uses the assignment operator (=) less than a specified minimum number of times. It evaluates each line of the file's content, counts occurrences of '=', and returns whether this limit is met.

**Detailed Breakdown**

### Main Purpose and Functionality

The main purpose is to determine if a file contains a limited number of assignment statements.
- **Purpose:** Check for frequent use of '=' in a file.
- **Functionality:** Evaluates each line, counting occurrences of '=', and returns a boolean result.

### Key Implementation Details

*   Iteration through the lines of the file's content using `for` loop.
*   Uses `string.count()` method to count occurrences of '='; value is stored in variable `counter`.
*   Returns an object with a single key-value pair when the limit is met (`{"has_few_assignments": False}`).

### Notable Dependencies or Requirements

- **External Library/Function:** None, uses only built-in Python functions (e.g., `count()`, `splitlines()`).
- **Classes Interfaces:** N/A; this function operates purely on string values.
- **Functions References:** N/A.

### Method Signatures and Interfaces

*   The function takes in `example` as a required parameter and accepts an optional `minimum=4` argument.


key: chroma:document
string_value:
CODE:

def char_token_ratio(example):
    """Compute character/token ratio of the file with tokenizer."""
    input_ids = tokenizer(example["content"], truncation=False)["input_ids"]
    ratio = len(example["content"]) / len(input_ids)


SUMMARY:
**Function Summary:**

`char_token_ratio(example)`

**Purpose:** Compute the character/token ratio of an input text.

**Key Implementation Details:**

- Uses Hugging Face's `tokenizer` to split input text into subwords, resulting in a tensor of `input_ids`.
- Calculates the ratio of total characters in the original text (`example["content"]`) to the total tokens in the tokenized input (`len(input_ids)`).

**Notable Dependencies/Requirements:**

- The Hugging Face `tokenizer` library.
- Input data with "content" attribute (not explicitly listed, but assumed to be a dictionary or similar data structure).


key: chroma:document
string_value:
CODE:

def preprocess(example):
    """Chain all preprocessing steps into one function to not fill cache."""
    results = {}
    results.update(get_hash(example))
    results.update(line_stats(example))
    results.update(alpha_stats(example))
    results.update(char_token_ratio(example))
    results.update(is_autogenerated(example))
    results.update(is_config_or_test(example))
    results.update(has_no_keywords(example))
    results.update(has_few_assignments(example))


SUMMARY:
**Summary:**

This `preprocess` function takes an example as input and returns a dictionary containing the aggregation of various preprocessing steps, including:

1. Hash calculation using `get_hash(example)`
2. Line statistics with `line_stats(example)`
3. Alpha statistics with `alpha_stats(example>`
4. Character token ratio with `char_token_ratio(example>`
5. Autogenerated detection with `is_autogenerated(example)`
6. Configuration/test distinction with `is_config_or_test(example`
7. Keyword presence check with `has_no_keywords(example)`
8. Few assignments check with `has_few_assignments(example)`

**Key Implementation Details:**

* Method signatures and interfaces: None explicitly defined, but each input is assumed to have a `__hash__()` method for hash calculation.
* Key algorithms or design choices:
	+ Chain aggregation of multiple preprocessing steps
	+ Use of caching to avoid filling cache (not implemented in this code snippet)
* Notable dependencies or requirements:
	+ `example` object with various methods (e.g., `get_hash`, `line_stats`, etc.)
	+ No external dependencies explicitly listed
	+ Assumes availability of related classes/functions by exact name.


key: chroma:document
string_value:
CODE:

def filter(example, uniques, args):
    """Filter dataset with heuristics. Config, test and has_no_keywords files are removed with a given probability."""
    if not check_uniques(example, uniques):
        return False
    elif example["autogenerated"]:
        return False
    elif example["line_max"] > args.line_max:
        return False
    elif example["line_mean"] > args.line_mean:
        return False
    elif example["alpha_frac"] < args.alpha_frac:
        return False
    elif example["ratio"] < args.min_token_ratio:
        return False
    elif example["config_or_test"] and np.random.rand() <= args.filter_proba:
        return False
    elif example["has_no_keywords"] and np.random.rand() <= args.filter_proba:
        return False
    elif example["has_few_assignments"]:
        return False
    else:


SUMMARY:
**Summary:**

The `filter` function applies various heuristics to filter out examples from a dataset. It checks for conditions such as:

* Non-unique examples or autogenerated instances
* High-line metrics (max and mean)
* Low feature ratios (alpha_frac, ratio)
* Config/test files with a given probability (filter_proba) if both have keywords
* Few assignments

If any of these conditions are met, the example is filtered out. Otherwise, it is returned as True.

**Key Implementation Details:**

* Uses `np.random.rand()` to introduce randomness in filtering decisions
* Method signatures:
	+ `check_uniques(example, uniques)`: an external function (not shown)
	+ `example["autogenerated"]`, ...: property accesses referencing example's attributes
	+ `args.line_max`, ..., args.filter_proba: referencing filter arguments

**Notable Dependencies/Requirements:**

* `np` for random number generation
* External `check_uniques` function, which is not shown in this code snippet.


key: chroma:document
string_value:
CODE:

def compress_file(file_path):
    """Compress a file with g-zip."""
    with open(file_path, "rb") as f_in:
        with gzip.open(str(file_path) + ".gz", "wb", compresslevel=6) as f_out:
            shutil.copyfileobj(f_in, f_out)


SUMMARY:
**Function Summary**

`compress_file`
================

**Purpose:**
Compresses an input file (`file_path`) using GZip.

**Main Functionality:**

* Reads the original file in binary mode (`"rb"`).
* Creates a compressed version of the file in GZip format (`.gz` suffixed), while preserving original file contents.
* Destroys the original file after compression.

**Key Implementation Details:**

* Uses `gzip.open` with compression level 6 (`compresslevel=6`).
* Utilizes `shutil.copyfileobj` to copy binary data from input file to output GZip stream.

**Notable Dependencies/Requirements:**

* `gzip`
* `shutil`
* No external import for Python's built-in `os` module.
* Assumes existence of a file on disk with the specified `file_path`.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/codeparrot/scripts/human_eval.py


key: chroma:document
string_value:
CODE:

class TokenizedDataset(IterableDataset):
    """Tokenize and preprocess the dataset
    Multiple copies of the same prompt are sent sequentially.
    See compute_code for more details.
    """

    def __init__(self, tokenizer, dataset, n_tasks=None, n_copies=1):
        self.tokenizer = tokenizer
        self.dataset = dataset
        self.n_tasks = len(dataset) if n_tasks is None else n_tasks
        self.n_copies = n_copies

    def __iter__(self):
        prompts = []
        for task in range(self.n_tasks):
            # without strip, the model generate commented codes ...
            prompts.append(self.tokenizer.eos_token + self.dataset[task]["prompt"].strip())
        outputs = self.tokenizer(prompts, padding=True, return_tensors="pt")
        for task in range(self.n_tasks):
            for _ in range(self.n_copies):
                yield {
                    "ids": outputs.input_ids[task],
                    "task_id": task,
                    "input_len": outputs.attention_mask[task].sum(),


SUMMARY:
**Summary**

The `TokenizedDataset` class is designed for tokenizing and preprocessing a dataset, generating multiple copies of the same prompt. Its primary purpose is to facilitate model training by breaking down code generation tasks.

**Key Functionality**

1.  **Initialization**: The class initializes with a tokenizer, dataset, optional number of tasks (`n_tasks`), and copes per task (`n_copies`). 
2.  **Iteration**: It generates prompts for each task, tokenizes the input, and yields output dictionaries containing `ids`, `task_id`, and `input_len`. 

**Notable Implementation Details**

*   The tokenizer is used to generate and split prompts into sequences.

**Dependencies and Requirements**

*   `tokenizer`: An instance of a tokenizer class (e.g., Hugging Face's `Tokenizer`).
*   `dataset`: An iterable dataset containing task prompts.
*   `IterableDataset` : Base class for iterating over datasets.


key: chroma:document
string_value:
CODE:

class EndOfFunctionCriteria(StoppingCriteria):
    """Custom `StoppingCriteria` which checks if all generated functions in the batch are completed."""

    def __init__(self, start_length, eof_strings, tokenizer):
        self.start_length = start_length
        self.eof_strings = eof_strings
        self.tokenizer = tokenizer

    def __call__(self, input_ids, scores, **kwargs):
        """Returns true if all generated sequences contain any of the end-of-function strings."""
        decoded_generations = self.tokenizer.batch_decode(input_ids[:, self.start_length :])
        done = []
        for decoded_generation in decoded_generations:
            done.append(any(stop_string in decoded_generation for stop_string in self.eof_strings))


SUMMARY:
**Class Summary:**

`EndOfFunctionCriteria`

A custom `StoppingCriteria` class that determines if all generated functions in a batch are completed based on the presence of specific end-of-function strings.

**Key Implementation Details:**

* Checks if any end-of-function string is present in each decoded generation sequence.
* Uses the `any` function with elements of the `eof_strings` tuple to check for string presence in each decoded sequence.
* Employs the `all` function to verify that all generated sequences contain at least one end-of-function string.

**Notable Dependencies/Requirements:**

* `Tokenizer` instance (`tokenizer`) from the Hugging Face Transformers library.
* `StoppingCriteria` interface implemented by this class.


key: chroma:document
string_value:
CODE:

def remove_last_block(string):
    """Remove the last block of the code containing EOF_STRINGS"""
    string_list = re.split("(%s)" % "|".join(EOF_STRINGS), string)
    # last string should be ""


SUMMARY:
**Summary**

Removes the last block of code containing `EOF STRINGS` by splitting the input string into substrings separated by the strings in `EOF_STRINGS`, and then joins all but the last substring back together.

**Key Details**

* Uses regular expressions (`re`) with the pattern `"%"s" % "|".join(EOF_STRINGS)` to split the input string.
* Splits the input string into a list of substrings using `string_list = ...`.
* Returns the last two elements of the list (all but the last substring) joined back together using `"`.join(...)`.

**Notable Dependencies**

* `re` module for regular expression operations
* `EOF STRINGS` is expected to be a predefined constant containing strings that delimit code blocks.


key: chroma:document
string_value:
CODE:

def complete_code(accelerator, model, tokenizer, dataloader, n_tasks, batch_size=20, **gen_kwargs):
    """Generate multiple codes for each task in the dataset. This function leverage accelerator to distribute
    the processing to multiple GPUs.
    dataloader, a wrapper around a TokenizeDataset objectm is supposed to send all the prompts from
    the evalution dataset to the modelm as the following:
    [p_0_0, p_0_1, ..., p_0_nc-1, p_1_0, ..., p_nt-1_nc-1]
    where nc is the number of copies of the prompt, and nt is the number of tasks.
    nc is such that num_sample = nc * batch_size

    Parameters
    ----------
    accelerator: Accelerator

    model: transformers.PreTrainedModel
        Code generation model. AutoTokenizer.from_pretrained(model_ckpt), ex model_ckpt = "lvwerra/codeparrot"

    tokenizer: transformers.AutoTokenizer
        The tokenizer used to train model

    dataloader: DataLoader
        The dataloader is a wrapper around a TokenizeDataset object. It is designed to be used with multiple GPUs.

    n_tasks: int
        The number of tasks in the dataset. It is used to determine the length of the output.
        Should be aligned with the number of tasks in the TokenizeDataset.

    batch_size: int
        num_return_sequences per copy of the prompt such that num_sample = batch_size * n_copies

    gen_kwargs: dict
        Keyword arguments for the generation function of the model.

    Returns
    -------
    code_gens: list of list of str, of length n_tasks
        List of generated codes for each task.
        Each element is a list of generated codes for each task, with length num_samples
    """
    gen_token_dict = defaultdict(list)  # dict of list of generated tokens
    for step, batch in tqdm(enumerate(dataloader)):
        with torch.no_grad():
            gen_kwargs["stopping_criteria"][0].start_length = batch["ids"].shape[-1]
            generated_tokens = accelerator.unwrap_model(model).generate(
                input_ids=batch["ids"][:, : batch["input_len"]], num_return_sequences=batch_size, **gen_kwargs
            )
            # each task is generated batch_size times
            generated_tasks = batch["task_id"].repeat(batch_size)
            generated_tokens = accelerator.pad_across_processes(
                generated_tokens, dim=1, pad_index=tokenizer.pad_token_id
            )

            generated_tokens, generated_tasks = accelerator.gather((generated_tokens, generated_tasks))
            generated_tokens = generated_tokens.cpu().numpy()
            generated_tasks = generated_tasks.cpu().numpy()

            for task, generated_tokens in zip(generated_tasks, generated_tokens):
                gen_token_dict[task].append(generated_tokens)

    code_gens = [[] for _ in range(n_tasks)]
    for task, generated_tokens in gen_token_dict.items():
        for s in generated_tokens:
            gen_code = tokenizer.decode(s, skip_special_tokens=True, clean_up_tokenization_spaces=True)
            code_gens[task].append(remove_last_block(gen_code))


SUMMARY:
**Function Summary**

`complete_code()` function generates multiple codes for each task in the dataset using an accelerator to distribute processing across multiple GPUs.

**Parameters:**

* `accelerator`
* `model`: `transformers.PreTrainedModel`
	+ Code generation model
* `tokenizer`: `transformers.AutoTokenizer`
	+ Tokenizer used to train model
* `dataloader`: `DataLoader`
	+ Wrapper around a TokenizeDataset object
* `n_tasks`: `int`
	+ Number of tasks in the dataset
* `batch_size`: `int` (default 20)
	+ number_return_sequences per copy of the prompt
* `gen_kwargs`: `dict`
	+ Keyword arguments for the generation function of the model

**Returns:**

* `code_gens`: list of lists of str, of length `n_tasks`
	+ List of generated codes for each task
	+ Each element is a list of generated codes with length `num_samples`

**Key Details:**

* Model uses `transformers.AutoTokenizer` and generates multiple codes per prompt using the `generate()` function
* Uses `torch.no_grad()` to disable gradients during generation
* Utilizes accelerator's `unwrap_model()`, `pad_across_processes()`, and `gather()` methods for GPU processing
* Removes last block of tokens using `remove_last_block(gen_code)`

**Dependencies/Requirements:**

* `transformers`: library for transformer models and tokenizers
* `torch`: library for tensor computations and acceleration
* `DataLoader` class from PyTorch


key: chroma:document
string_value:
CODE:

def main():
    # Setup configuration
    parser = HfArgumentParser(HumanEvalArguments)
    args = parser.parse_args()

    transformers.logging.set_verbosity_error()
    # enables code execution in code_eval metric
    os.environ["HF_ALLOW_CODE_EVAL"] = args.HF_ALLOW_CODE_EVAL
    # make sure tokenizer plays nice with multiprocessing
    os.environ["TOKENIZERS_PARALLELISM"] = "false"

    if args.num_workers is None:
        args.num_workers = multiprocessing.cpu_count()

    # Use dataset load to feed to accelerate
    accelerator = Accelerator()
    set_seed(args.seed, device_specific=True)

    # Load model and tokenizer
    tokenizer = AutoTokenizer.from_pretrained(args.model_ckpt)
    tokenizer.pad_token = tokenizer.eos_token
    model = AutoModelForCausalLM.from_pretrained(args.model_ckpt)

    # Generation settings
    gen_kwargs = {
        "do_sample": args.do_sample,
        "temperature": args.temperature,
        "max_new_tokens": args.max_new_tokens,
        "top_p": args.top_p,
        "top_k": args.top_k,
        "stopping_criteria": StoppingCriteriaList([EndOfFunctionCriteria(0, EOF_STRINGS, tokenizer)]),
    }

    # Load evaluation dataset and metric
    human_eval = load_dataset("openai_humaneval")
    code_eval_metric = load_metric("code_eval")

    n_tasks = args.num_tasks if args.num_tasks is not None else len(human_eval["test"])
    n_copies = args.n_samples // args.batch_size

    human_eval_tokenized = TokenizedDataset(tokenizer, human_eval["test"], n_copies=n_copies, n_tasks=n_tasks)
    # do not confuse args.batch_size, which is actually the num_return_sequences
    human_eval_loader = DataLoader(human_eval_tokenized, batch_size=1)

    # Run a quick test to see if code evaluation is enabled
    try:
        _ = code_eval_metric.compute(references=[""], predictions=[[""]])
    except ValueError as exception:
        print(
            'Code evaluation not enabled. Read the warning below carefully and then use `--HF_ALLOW_CODE_EVAL="1"`'
            " flag to enable code evaluation."
        )
        raise exception

    model, human_eval_loader = accelerator.prepare(model, human_eval_loader)

    generations = complete_code(
        accelerator,
        model,
        tokenizer,
        human_eval_loader,
        n_tasks=n_tasks,
        batch_size=args.batch_size,
        **gen_kwargs,
    )

    if accelerator.is_main_process:
        references = []

        for task in tqdm(range(n_tasks)):
            test_func = human_eval["test"][task]["test"]
            entry_point = f"check({human_eval['test'][task]['entry_point']})"
            references.append("\n" + test_func + "\n" + entry_point)

        # Evaluate completions with "code_eval" metric
        pass_at_k, _ = code_eval_metric.compute(
            references=references, predictions=generations, num_workers=args.num_workers
        )
        print(f"Results: {pass_at_k}")

        # Save results to json file
        with open(args.output_file, "w") as fp:


SUMMARY:
**Function Summary**

This is the main entry function of a code completion model, responsible for running the entire pipeline. The primary purpose is to generate code completions using the trained model and evaluate their quality.

**Key Implementation Details**

*   Loads and configures Hugging Face's `Accelerator`, `Tokenizer`, and `Model`.
*   Sets up hyperparameters for generation, including temperature, top-k and top-p, and stopping criteria.
*   Loads evaluation dataset (`openai_humaneval`) and metric (`code_eval`).
*   Runs a quick test to check if code evaluation is enabled.

**Notable Dependencies and Requirements**

*   `HfArgumentParser` from Hugging Face Transformer Library
*   `HumanEvalArguments` defined in the parser (not shown here)
*   `AutoTokenizer` and `AutoModelForCausalLM` models pre-trained on a specific checkpoint (`args.model_ckpt`)
*   `Accelerator`, `DataLoader`, and `TokenizedDataset` from Hugging Face Library
*   External dependencies: `openai_humaneval`, `code_eval`, `multiprocessing`
*   Implements `StoppingCriteriaList`.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/codeparrot/scripts/validation_loss.py


key: chroma:document
string_value:
CODE:

class ConstantLengthDataset(IterableDataset):
    def __init__(self, tokenizer, dataset, seq_length=1024, num_of_sequences=1024, chars_per_token=3.6):
        self.tokenizer = tokenizer
        self.concat_token_id = tokenizer.bos_token_id
        self.dataset = dataset
        self.seq_length = seq_length
        self.input_characters = seq_length * chars_per_token * num_of_sequences

    def __iter__(self):
        iterator = iter(self.dataset)
        more_examples = True
        while more_examples:
            buffer, buffer_len = [], 0
            while True:
                if buffer_len >= self.input_characters:
                    break
                try:
                    buffer.append(next(iterator)["content"])
                    buffer_len += len(buffer[-1])
                except StopIteration:
                    more_examples = False
                    break
            tokenized_inputs = tokenizer(buffer, truncation=False)["input_ids"]
            all_token_ids = []
            for tokenized_input in tokenized_inputs:
                all_token_ids.extend(tokenized_input + [self.concat_token_id])
            for i in range(0, len(all_token_ids), self.seq_length):
                input_ids = all_token_ids[i : i + self.seq_length]
                if len(input_ids) == self.seq_length:


SUMMARY:
**Class Summary: `ConstantLengthDataset`**

The `ConstantLengthDataset` class is an implementation of the `IterableDataset` interface. Its primary purpose is to generate preprocessed sequential data with constant input length.

**Key Functionality:**

- Tokenizes dataset content using a provided tokenizer.
- Truncates or appends special tokens (`bos_token_id`) to maintain a fixed sequence length (`seq_length`).
- Yields preprocessed tensor inputs of size `seq_length`.

**Implementation Details:**

- Uses `tokenizer.bos_token_id` as the concatenation token ID.
- Appends tokens in chunks until the specified input character capacity is reached.

**Notable Dependencies/Requirements:**

- `tokenizer`: An instance of a tokenizer class (e.g., `TransformersTokenizer`).
- External dependencies:
	+ `torch.tensor`
    - `IterableDataset` interface
    - The `TransformersTokenizer` class.


key: chroma:document
string_value:
CODE:

def create_dataloader(args):
    ds_kwargs = {"streaming": True}
    valid_data = load_dataset(args.dataset_name, split="train", **ds_kwargs)
    valid_dataset = ConstantLengthDataset(tokenizer, valid_data, seq_length=args.seq_length)
    eval_dataloader = DataLoader(valid_dataset, batch_size=args.batch_size)


SUMMARY:
**Summary:**
The `create_dataloader` function creates an evaluation data loader for a given dataset. It takes in command-line arguments (`args`) and returns a PyTorch DataLoader object. The primary purpose is to load the training data into a format suitable for model evaluation.

**Key Implementation Details:**

* Utilizes the `load_dataset` function from `huggingface_hf_utils` to load the train dataset.
* Creates a `ConstantLengthDataset` instance with specified tokenizer, dataset, and sequence length (args.seq_length).
* Generates an evaluation data loader using PyTorch's `DataLoader`, setting the batch size according to the provided `args.batch_size`.

**Notable Dependencies or Requirements:**

* `huggingface_hf_utils`: for loading datasets
* `tokenizer` instance required for dataset creation
* PyTorch's `DataLoader` class


key: chroma:document
string_value:
CODE:

def evaluate(args):
    model.eval()
    losses = []
    for step, batch in enumerate(eval_dataloader):
        with torch.no_grad():
            outputs = model(batch, labels=batch)
        loss = outputs.loss.repeat(args.batch_size)
        losses.append(accelerator.gather(loss))

        if args.max_eval_steps > 0 and step >= args.max_eval_steps:
            break
    loss = torch.mean(torch.cat(losses))
    try:
        perplexity = torch.exp(loss)
    except OverflowError:
        perplexity = float("inf")


SUMMARY:
**Summary:**

Evaluates the model on an evaluation DataLoader.

*   Primary purpose: Compute and average loss and perplexity measures during evaluation.
*   Key implementation details:
    *   Iterate through batches using `eval_dataloader`, computing losses for each batch.
    *   Gather losses from accelerator to device using `gather`.
    *   Repeat loss `args.batch_size` times to match batch size of labels.
    *   Compute mean loss and calculate perplexity as exponential of mean loss.

**Notable dependencies and requirements:**

*   `torch`: torch library for PyTorch
*   `accelerator.gather()`: accelerator function from Hugging Face's `transformers` library
*   `eval_dataloader` (dataset and loader): a pre-defined data loading setup for evaluation.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/codeparrot/examples/train_complexity_predictor.py


key: chroma:document
string_value:
CODE:

def get_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model_ckpt", type=str, default="microsoft/unixcoder-base-nine")
    parser.add_argument("--num_epochs", type=int, default=5)
    parser.add_argument("--batch_size", type=int, default=6)
    parser.add_argument("--gradient_accumulation_steps", type=int, default=1)
    parser.add_argument("--freeze", type=bool, default=True)
    parser.add_argument("--learning_rate", type=float, default=5e-4)
    parser.add_argument("--seed", type=int, default=0)
    parser.add_argument("--lr_scheduler_type", type=str, default="cosine")
    parser.add_argument("--num_warmup_steps", type=int, default=10)
    parser.add_argument("--weight_decay", type=float, default=0.01)
    parser.add_argument("--output_dir", type=str, default="./results")


SUMMARY:
**Function Summary: `get_args()`**

The `get_args` function creates an `ArgumentParser` instance from the Argparse library to parse command-line arguments for training a model.

### Implemented Argument Parser

The function defines 11 arguments with their respective types, default values, and help (not explicitly shown here).

*   **Arguments:**
    *   `model_ckpt`: Model checkpoint path (`str`)
    *   `num_epochs`: Number of training epochs (`int`)
    *   `batch_size`: Batch size for training (`int`)
    *   `gradient_accumulation_steps`: Gradient accumulation steps (`int`)
    *   `freeze`: Freeze model weights flag (`bool`)
    *   `learning_rate`: Learning rate for optimizers (`float`)
    *   `seed`: Random seed for reproducibility (`int`)
    *   `lr_scheduler_type`: Type of learning rate scheduler (`str`)
    *   `num_warmup_steps`: Number of warm-up steps for warm-up scheduler (`int`)
    *   `weight_decay`: Weight decay value for L2 regularization (`float`)
*   **Parsed Args**: The parsed arguments are returned by the `parse_args()` method.

### Notable Dependencies

This function relies on the Argparse library, specifically importing it is not shown here.

```python
import argparse
```

No other external dependencies or interfaces are mentioned.


key: chroma:document
string_value:
CODE:

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = np.argmax(predictions, axis=1)


SUMMARY:
**Summary:**

`compute_metrics` function:

*   Takes `eval_pred` input and extracts predictions and labels from it.
*   Applies argmax to predictions along the first axis (i.e., predicts class with highest probability).
*   Calls `metric.compute()` with computed predictions and original labels, returning the result.

**Key Implementation Details:**

*   Used `np.argmax()` to get predicted classes.
*   Called `metric.compute()` function with specific arguments (`predictions=predictions`, `references=labels`).

**Notable Dependencies or Requirements:**

*   Uses NumPy (`np`) for array operations.
*   Calls `metric.compute()` function, which is not shown in the provided code snippet.


key: chroma:document
string_value:
CODE:

class CustomCallback(TrainerCallback):
    def __init__(self, trainer) -> None:
        super().__init__()
        self._trainer = trainer

    def on_epoch_end(self, args, state, control, **kwargs):
        if control.should_evaluate:
            control_copy = deepcopy(control)
            self._trainer.evaluate(eval_dataset=self._trainer.train_dataset, metric_key_prefix="train")


SUMMARY:
**Summary: Custom Callback Class**

* **Purpose:** Extends `TrainerCallback` to evaluate model on training dataset at the end of each epoch if evaluation is enabled.

**Key Implementation Details:**
- Uses `deepcopy` to create a copy of the `control` object before modifying it.
- Calls `_trainer.evaluate` method with `eval_dataset` set to `train_dataset` and `metric_key_prefix` set to `"train"`.

**Notable Dependencies and Requirements:**

* Inherited from `TrainerCallback`.
* Uses:
	+ `deepcopy` function.
	+ `_trainer` object (not shown in this code snippet).
* Assumes `control`, `eval_dataset`, and `metric_key_prefix` are defined elsewhere.


key: chroma:document
string_value:
CODE:

def main():
    args = get_args()
    set_seed(args.seed)

    dataset = load_dataset("codeparrot/codecomplex", split="train")
    train_test = dataset.train_test_split(test_size=0.2)
    test_validation = train_test["test"].train_test_split(test_size=0.5)
    train_test_validation = DatasetDict(
        {
            "train": train_test["train"],
            "test": test_validation["train"],
            "valid": test_validation["test"],
        }
    )

    print("Loading tokenizer and model")
    tokenizer = AutoTokenizer.from_pretrained(args.model_ckpt)
    tokenizer.pad_token = tokenizer.eos_token
    model = AutoModelForSequenceClassification.from_pretrained(args.model_ckpt, num_labels=7)
    model.config.pad_token_id = model.config.eos_token_id

    if args.freeze:
        for param in model.roberta.parameters():
            param.requires_grad = False

    labels = ClassLabel(num_classes=7, names=list(set(train_test_validation["train"]["complexity"])))

    def tokenize(example):
        inputs = tokenizer(example["src"], truncation=True, max_length=1024)
        label = labels.str2int(example["complexity"])
        return {
            "input_ids": inputs["input_ids"],
            "attention_mask": inputs["attention_mask"],
            "label": label,
        }

    tokenized_datasets = train_test_validation.map(
        tokenize,
        batched=True,
        remove_columns=train_test_validation["train"].column_names,
    )
    data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

    training_args = TrainingArguments(
        output_dir=args.output_dir,
        learning_rate=args.learning_rate,
        lr_scheduler_type=args.lr_scheduler_type,
        eval_strategy="epoch",
        save_strategy="epoch",
        logging_strategy="epoch",
        per_device_train_batch_size=args.batch_size,
        per_device_eval_batch_size=args.batch_size,
        num_train_epochs=args.num_epochs,
        gradient_accumulation_steps=args.gradient_accumulation_steps,
        weight_decay=0.01,
        metric_for_best_model="accuracy",
        run_name="complexity-java",
        report_to="wandb",
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["valid"],
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics,
    )

    print("Training...")
    trainer.add_callback(CustomCallback(trainer))


SUMMARY:
**Summary**

This code defines a training script for a sequence classification model using the Hugging Face library. The main purpose is to train a model on a dataset of coded complexity problems.

### Main Functionality

* Loads and preprocesses data from `codeparrot/codecomplex` dataset
* Trains an AutoModelForSequenceClassification model with frozen parameters when required

### Key Implementation Details

* Tokenization using `AutoTokenizer` and creating a `DatasetDict`
* Defining a custom `DataCollatorWithPadding` for padding
* Creating a training configuration (`TrainingArguments`) with various hyperparameters
* Instantiating the `Trainer` with the model, training dataset, evaluation dataset, and other parameters

### Dependencies/Requirements

* Related classes/functions:
	+ `load_dataset`, `train_test_split`, `DatasetDict`
	+ `AutoTokenizer`, `AutoModelForSequenceClassification`, `TrainingArguments`, `Trainer`
* External dependencies:
	+ TensorFlow 2.x (Hugging Face transformer models)
	+ PyTorch 1.9.x (transformer tokenizer)

Note: The code may not be comprehensive, as this analysis focus was on the provided function chunk.


key: chroma:document
string_value:
CODE:

def hans_data_collator(features: List[InputFeatures]) -> Dict[str, torch.Tensor]:
    """
    Data collator that removes the "pairID" key if present.
    """
    batch = default_data_collator(features)
    _ = batch.pop("pairID", None)


SUMMARY:
**Function Summary:**

`hans_data_collator`: A data collator that removes the "pairID" key from the input features, utilizing the default `default_data_collator`.

**Key Implementation Details:**

- Utilizes the `default_data_collator` function to create a batched dataset.
- Deletes the "pairID" key from the resulting batch using `batch.pop("pairID", None)`.

**Notable Dependencies and Requirements:**

- Uses `List[InputFeatures]` as input type hint, referencing `	InputFeatures`.
- Leans on an external dependency for `default_data_collator`, likely a custom or third-party implementation.
- Utilizes the PyTorch library (`torch.Tensor`) for output.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    if (
        os.path.exists(training_args.output_dir)
        and os.listdir(training_args.output_dir)
        and training_args.do_train
        and not training_args.overwrite_output_dir
    ):
        raise ValueError(
            f"Output directory ({training_args.output_dir}) already exists and is not empty. Use"
            " --overwrite_output_dir to overcome."
        )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO if training_args.local_rank in [-1, 0] else logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        training_args.local_rank,
        training_args.device,
        training_args.n_gpu,
        bool(training_args.local_rank != -1),
        training_args.fp16,
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    logger.info("Training/evaluation parameters %s", training_args)

    # Set seed
    set_seed(training_args.seed)

    try:
        num_labels = hans_tasks_num_labels[data_args.task_name]
    except KeyError:
        raise ValueError("Task not found: %s" % (data_args.task_name))

    # Load pretrained model and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.

    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        num_labels=num_labels,
        finetuning_task=data_args.task_name,
        cache_dir=model_args.cache_dir,
    )
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
    )
    model = AutoModelForSequenceClassification.from_pretrained(
        model_args.model_name_or_path,
        from_tf=bool(".ckpt" in model_args.model_name_or_path),
        config=config,
        cache_dir=model_args.cache_dir,
    )

    # Get datasets
    train_dataset = (
        HansDataset(
            data_dir=data_args.data_dir,
            tokenizer=tokenizer,
            task=data_args.task_name,
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
        )
        if training_args.do_train
        else None
    )
    eval_dataset = (
        HansDataset(
            data_dir=data_args.data_dir,
            tokenizer=tokenizer,
            task=data_args.task_name,
            max_seq_length=data_args.max_seq_length,
            overwrite_cache=data_args.overwrite_cache,
            evaluate=True,
        )
        if training_args.do_eval
        else None
    )

    # Initialize our Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        data_collator=hans_data_collator,
    )

    # Training
    if training_args.do_train:
        trainer.train(
            model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
        )
        trainer.save_model()
        # For convenience, we also re-save the tokenizer to the same directory,
        # so that you can share your model easily on huggingface.co/models =)
        if trainer.is_world_master():
            tokenizer.save_pretrained(training_args.output_dir)

    # Evaluation
    if training_args.do_eval:
        logger.info("*** Evaluate ***")

        output = trainer.predict(eval_dataset)
        preds = output.predictions
        preds = np.argmax(preds, axis=1)

        pair_ids = [ex.pairID for ex in eval_dataset]
        output_eval_file = os.path.join(training_args.output_dir, "hans_predictions.txt")
        label_list = eval_dataset.get_labels()
        if trainer.is_world_master():
            with open(output_eval_file, "w") as writer:
                writer.write("pairID,gold_label\n")
                for pid, pred in zip(pair_ids, preds):
                    writer.write("ex" + str(pid) + "," + label_list[int(pred)] + "\n")



SUMMARY:
**Summary**

This is the `main` function of a script for training a sequence classification model using the Hugging Face Transformers library. The main purpose is to train and evaluate a pre-trained model on a specific task, as defined by the user-provided arguments.

**Key Implementation Details**

* Loads pretrained model and tokenizer using `AutoConfig`, `AutoTokenizer`, and `AutoModelForSequenceClassification` from the Hugging Face library.
* Sets up logging with detailed level and format settings.
* Configures the training environment, including setting seed, distributed training, and 16-bit training.
* Creates datasets for training and evaluation using the `HansDataset` class.
* Initializes a `Trainer` object to perform the training and evaluation tasks.

**Notable Dependencies**

* Hugging Face Transformers library
* `AutoConfig`, `AutoTokenizer`, `AutoModelForSequenceClassification`
* `HansDataset` class
* `Trainer` class

**External Dependencies**

* `hans_tasks_num_labels` dictionary (not shown in this code snippet)
* `training_args` dataclasses (defined elsewhere in the script)


key: chroma:document
string_value:
CODE:

def _mp_fn(index):
    # For xla_spawn (TPUs)


SUMMARY:
**Summary:**

`_mp_fn` is an internal function with one argument `index`. It appears to be a multi-threading or multiprocessing wrapper, invoking the `main()` function when using XLA Spawn for parallel computing on TPUs. However, currently, it simply runs the `main()` function without any multithreading/multiprocessing logic.

**Key Details:**

* Function signature: `_mp_fn(index)`
* Main purpose: Invokes `main()` with XLA Spawn support
* Implementation detail:
	+ Lack of explicit multithreading or multiprocessing logic

**Notable Dependencies and Requirements:**

* Relies on XLA (Accelerated Linear Algebra) for parallel computing
* Utilizes TPUs (Tensor Processing Units) for execution
* No external dependencies listed
* No implemented or inherited interfaces specified


key: chroma:document
string_value:
CODE:

class HansProcessor(DataProcessor):
    """Processor for the HANS data set."""

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "heuristics_train_set.txt")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "heuristics_evaluation_set.txt")), "dev")

    def get_labels(self):
        """See base class.
        Note that we follow the standard three labels for MNLI
        (see :class:`~transformers.data.processors.utils.MnliProcessor`)
        but the HANS evaluation groups `contradiction` and `neutral` into `non-entailment` (label 0) while
        `entailment` is label 1."""
        return ["contradiction", "entailment", "neutral"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for i, line in enumerate(lines):
            if i == 0:
                continue
            guid = "%s-%s" % (set_type, line[0])
            text_a = line[5]
            text_b = line[6]
            pairID = line[7][2:] if line[7].startswith("ex") else line[7]
            label = line[0]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label, pairID=pairID))


SUMMARY:
**Class Summary: HansProcessor**

The `HansProcessor` class is a data processor for the HANS dataset. Its primary purpose is to tokenize and prepare data for training/evaluation.

**Key Functionality:**

* Reads and tokenizes heuristic train and evaluation sets
* Creates examples with predefined labels (contradiction, entailment, neutral) and pair IDs

**Implementation Details:**

* `_create_examples` method:
	+ Iterates over lines in the set files
	+ Constructs InputExample objects with text A, B, pair ID, label, and guid
* Primary purpose is to create examples for the training and dev sets, following a standard MNLi processing pattern

**Notable Dependencies:**

* Base class `DataProcessor`
* External dependencies:
	+ `os` (for file paths)
	+ `tsv` (data read functionality)

**Interfaces:**

* Inherited from `DataProcessor`
* `_create_examples` method is a custom implementation


key: chroma:document
string_value:
CODE:

def hans_convert_examples_to_features(
    examples: List[InputExample],
    label_list: List[str],
    max_length: int,
    tokenizer: PreTrainedTokenizer,
):
    """
    Loads a data file into a list of ``InputFeatures``

    Args:
        examples: List of ``InputExamples`` containing the examples.
        label_list: List of labels. Can be obtained from the processor using the ``processor.get_labels()`` method.
        max_length: Maximum example length.
        tokenizer: Instance of a tokenizer that will tokenize the examples.

    Returns:
        A list of task-specific ``InputFeatures`` which can be fed to the model.

    """

    label_map = {label: i for i, label in enumerate(label_list)}

    features = []
    for ex_index, example in tqdm.tqdm(enumerate(examples), desc="convert examples to features"):
        if ex_index % 10000 == 0:
            logger.info("Writing example %d" % (ex_index))

        inputs = tokenizer(
            example.text_a,
            example.text_b,
            add_special_tokens=True,
            max_length=max_length,
            padding="max_length",
            truncation=True,
            return_overflowing_tokens=True,
        )

        label = label_map[example.label] if example.label in label_map else 0

        pairID = int(example.pairID)

        features.append(InputFeatures(**inputs, label=label, pairID=pairID))

    for i, example in enumerate(examples[:5]):
        logger.info("*** Example ***")
        logger.info(f"guid: {example}")
        logger.info(f"features: {features[i]}")



SUMMARY:
**Function Summary**

`hans_convert_examples_to_features`
================================

Loads data examples into task-specific input features using the provided tokenizer.

### Purpose and Functionality

Converts list of `InputExamples` into a list of `InputFeatures` that can be fed to a model. It takes in:

*   `examples`: List of `InputExamples` containing text pairs.
*   `label_list`: List of labels for classification tasks.
*   `max_length`: Maximum example length.
*   `tokenizer`: Pre-trained tokenizer instance.

The function returns a list of `InputFeatures`, which are instances of the same class as in the processor.

### Key Implementation Details

*   Uses the provided `tokenizer` to tokenize text pairs and create input features.
*   Maps labels from `label_list` to integer indices using a dictionary.
*   Creates `InputFeatures` instances with tokenized inputs, label indices, and pair IDs.

### Notable Dependencies and Requirements

*   `PreTrainedTokenizer` instance for tokenization.
*   `processor` object for obtaining labels.

Example usage:
```python
examples = [...]  # List of InputExamples
label_list = [...]  # List of labels
tokenizer = ...  # Pretrained tokenizer instance

features = hans_convert_examples_to_features(examples, label_list, max_length=512, tokenizer=tokenizer)
```
Output:

*   A list of `InputFeatures` instances in the format `{ 'input_ids': [ tokenized input IDs ], 'attention_mask': [ attention mask tokens ], 'label': [ integer label index ], 'pair_id': [ pair ID ] }`.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/token-healing/run_token_healing.py


key: chroma:document
string_value:
CODE:

def generate(inputs, model, tokenizer, token_healing):
    input_ids = tokenizer(inputs, return_tensors="pt", padding=True, device_map="auto").input_ids
    generation_config = GenerationConfig(
        max_new_tokens=8,
        token_healing=token_healing,
        pad_token_id=model.config.pad_token_id,
        repetition_penalty=1.1,
    )
    output = model.generate(inputs=input_ids, generation_config=generation_config)


SUMMARY:
**Summary:**

The `generate` function generates text using a given sequence-to-sequence model and tokenizer. It takes in:

- `inputs`: Input string to generate text from
- `model`: Pre-trained sequence-to-sequence model (not shown)
- `tokenizer`: Tokenizer instance for converting inputs to format suitable for the model (not shown)
- `token_healing`: Flag to enable token healing functionality

The function performs the following steps:

1. Converts input string to input IDs using the tokenizer.
2. Sets up a generation configuration with custom parameters (max_new_tokens, token_healing, pad_token_id, repetition_penalty).
3. Uses the model's generate method to produce text based on the generated input IDs and configuration.
4. Decodes the output generated by the model into human-readable text using the tokenizer.

**Key Implementation Details:**

- Model's generate method (`model.generate`)
- Tokenization pipeline involving `tokenizer` instance
- Generation configuration settings (max_new_tokens, token_healing, pad_token_id, repetition_penalty)

**Notable Dependencies or Requirements:**

- Pre-trained sequence-to-sequence model (`model` instance, not shown)
- Tokenizer instance (`tokenizer`)
- External dependency on `pt` tensor format ( device_map="auto" option)


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--prompt", type=str)
    parser.add_argument("--model_name_or_path", type=str, default="TheBloke/deepseek-llm-7B-base-GPTQ")
    args = parser.parse_args()

    prompts = (
        [args.prompt]
        if args.prompt
        else [
            'An example ["like this"] and another example [',
            'The link is <a href="http:',
            'The link is <a href="http',  # test aggressive healing http->https
            "I read a book about ",  # test trailing whitespace
            "I read a book about",  # test nothing to heal
        ]
    )

    model_name_or_path = args.model_name_or_path
    completion_model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path,
        device_map="auto",
        use_cache=True,
    )
    tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)

    raw_output = generate(prompts, completion_model, tokenizer, token_healing=False)
    healed_output = generate(prompts, completion_model, tokenizer, token_healing=True)

    for p, a, b in zip(prompts, raw_output, healed_output):
        print(f"\nPrompt: {p}\nWithout healing:\n{a}\nWith healing:\n{b}")

    # You can also use token healing in isolation
    # This can be useful if you have other work to do before the generation
    # Or if you want to delegate generation to another process
    input_ids = tokenizer(prompts, return_tensors="pt", padding=True).input_ids.cuda()
    healed_ids = completion_model.heal_tokens(input_ids)
    healed_prompts = tokenizer.batch_decode(healed_ids, skip_special_tokens=True)
    print("\nhealed prompts:")
    for p in healed_prompts:


SUMMARY:
**Function Summary:**

This function generates text completions using a pre-trained GPTQ model. It takes two main approaches:

1. **With Token Healing:** Uses token healing (repairing broken links) to generate input prompts for the completion model.
2. **Without Healing:** Generates prompts without token healing, providing a comparison between the two approaches.

**Key Implementation Details:**

* Utilizes `argparse` to parse command-line arguments (`--prompt`, `--model_name_or_path`)
* Leverages pre-trained GPTQ models (e.g., "TheBloke/deepseek-llm-7B-base-GPTQ") for completion tasks
* Employs `AutoModelForCausalLM` and `AutoTokenizer` from the Hugging Face Transformers library
* Generates input prompts using either token healing or no healing approach

**Notable Dependencies:**

* `argparse`
* `transformers` (Hugging Face Transformers library)
* `torch.cuda`

Note that this summary focuses on the main purpose, key implementation details, and notable dependencies of the function.


key: chroma:document
string_value:
CODE:

class FSNERModel(torch.nn.Module):
    """
    The FSNER model implements a few-shot named entity recognition method from the paper `Example-Based Named Entity Recognition <https://arxiv.org/abs/2008.10570>`__ by
    Morteza Ziyadi, Yuting Sun, Abhishek Goswami, Jade Huang, Weizhu Chen. To identify entity spans in a new domain, it
    uses a train-free few-shot learning approach inspired by question-answering.
    """

    def __init__(self, pretrained_model_name_or_path="sayef/fsner-bert-base-uncased"):
        super(FSNERModel, self).__init__()

        self.bert = AutoModel.from_pretrained(pretrained_model_name_or_path, return_dict=True)
        self.cos = torch.nn.CosineSimilarity(3, 1e-08)
        self.softmax = torch.nn.Softmax(dim=1)

    def BERT(self, **inputs):
        return self.bert(**inputs).last_hidden_state

    def VectorSum(self, token_embeddings):
        return token_embeddings.sum(2, keepdim=True)

    def Atten(self, q_rep, S_rep, T=1):
        return self.softmax(T * self.cos(q_rep, S_rep))

    def forward(self, W_query, W_supports):
        """
        Find scores of each token being start and end token for an entity.
        Args:
            W_query (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
                Indices of query sequence tokens in the vocabulary.
            W_supports (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
                Indices of support sequence tokens in the vocabulary.
        Returns:
            p_start (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Scores of each token as
            being start token of an entity
            p_end (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Scores of each token as
            being end token of an entity
        """

        support_sizes = W_supports["sizes"].tolist()
        start_token_id = W_supports["start_token_id"].item()
        end_token_id = W_supports["end_token_id"].item()

        del W_supports["sizes"]
        del W_supports["start_token_id"]
        del W_supports["end_token_id"]

        q = self.BERT(**W_query)
        S = self.BERT(**W_supports)

        p_starts = None
        p_ends = None

        start_token_masks = W_supports["input_ids"] == start_token_id
        end_token_masks = W_supports["input_ids"] == end_token_id

        for i, size in enumerate(support_sizes):
            if i == 0:
                s = 0
            else:
                s = support_sizes[i - 1]

            s_start = S[s : s + size][start_token_masks[s : s + size]]
            s_end = S[s : s + size][end_token_masks[s : s + size]]

            p_start = torch.matmul(q[i], s_start.T).sum(1).softmax(0)
            p_end = torch.matmul(q[i], s_end.T).sum(1).softmax(0)

            if p_starts is not None:
                p_starts = torch.vstack((p_starts, p_start))
                p_ends = torch.vstack((p_ends, p_end))
            else:
                p_starts = p_start
                p_ends = p_end



SUMMARY:
Here is a concise and focused summary of the provided class code chunk:

**Main Purpose and Functionality**

The `FSNERModel` class implements a few-shot named entity recognition (NER) method based on the paper "Example-Based Named Entity Recognition". It uses a train-free few-shot learning approach inspired by question-answering to identify entity spans in a new domain. The primary purpose is to take in query and support sequences as input, produce scores for each token being start or end of an entity.

**Key Implementation Details**

* Uses a pre-trained BERT model (`AutoModel`) for embedding extraction.
* Implements a `BERT` method to retrieve the last hidden state of the input sequence.
* Defines three key attention functions: `VectorSum`, `Atten`, and computes token embeddings sum and attends over support representations.
* In the `forward` method, it performs the following:
	+ Extracts query and support representations using BERT.
	+ Computes scores for each token as start or end of an entity by attending over support representations.

**Notable Dependencies or Requirements**

* Pre-trained BERT model with a specific name (`sayef/fsner-bert-base-uncased`).
* `torch.nn.CosineSimilarity` and `torch.nn.Softmax` for attention computations.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/fsner/src/fsner/tokenizer_utils.py


key: chroma:document
string_value:
CODE:

class FSNERTokenizerUtils:
    def __init__(self, pretrained_model_name_or_path):
        self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path)

    def tokenize(self, x):
        """
        Wrapper function for tokenizing query and supports
        Args:
            x (`List[str] or List[List[str]]`):
                List of strings for query or list of lists of strings for supports.
        Returns:
            `transformers.tokenization_utils_base.BatchEncoding` dict with additional keys and values for start_token_id, end_token_id and sizes of example lists for each entity type
        """

        if isinstance(x, list) and all(isinstance(_x, list) for _x in x):
            d = None
            for l in x:
                t = self.tokenizer(
                    l,
                    padding="max_length",
                    max_length=384,
                    truncation=True,
                    return_tensors="pt",
                )
                t["sizes"] = torch.tensor([len(l)])
                if d is not None:
                    for k in d.keys():
                        d[k] = torch.cat((d[k], t[k]), 0)
                else:
                    d = t

            d["start_token_id"] = torch.tensor(self.tokenizer.convert_tokens_to_ids("[E]"))
            d["end_token_id"] = torch.tensor(self.tokenizer.convert_tokens_to_ids("[/E]"))

        elif isinstance(x, list) and all(isinstance(_x, str) for _x in x):
            d = self.tokenizer(
                x,
                padding="max_length",
                max_length=384,
                truncation=True,
                return_tensors="pt",
            )

        else:
            raise Exception(
                "Type of parameter x was not recognized! Only `list of strings` for query or `list of lists of"
                " strings` for supports are supported."
            )

        return d

    def extract_entity_from_scores(self, query, W_query, p_start, p_end, thresh=0.70):
        """
        Extracts entities from query and scores given a threshold.
        Args:
            query (`List[str]`):
                List of query strings.
            W_query (`torch.LongTensor` of shape `(batch_size, sequence_length)`):
                Indices of query sequence tokens in the vocabulary.
            p_start (`torch.FloatTensor` of shape `(batch_size, sequence_length)`):
                Scores of each token as being start token of an entity
            p_end (`torch.FloatTensor` of shape `(batch_size, sequence_length)`):
                Scores of each token as being end token of an entity
            thresh (`float`):
                Score threshold value
        Returns:
            A list of lists of tuples(decoded entity, score)
        """

        final_outputs = []
        for idx in range(len(W_query["input_ids"])):
            start_indexes = end_indexes = range(p_start.shape[1])

            output = []
            for start_id in start_indexes:
                for end_id in end_indexes:
                    if start_id < end_id:
                        output.append(
                            (
                                start_id,
                                end_id,
                                p_start[idx][start_id].item(),
                                p_end[idx][end_id].item(),
                            )
                        )

            output.sort(key=lambda tup: (tup[2] * tup[3]), reverse=True)
            temp = []
            for k in range(len(output)):
                if output[k][2] * output[k][3] >= thresh:
                    c_start_pos, c_end_pos = output[k][0], output[k][1]
                    decoded = self.tokenizer.decode(W_query["input_ids"][idx][c_start_pos:c_end_pos])
                    temp.append((decoded, output[k][2] * output[k][3]))

            final_outputs.append(temp)



SUMMARY:
**FSNERTokenizerUtils Class Summary**

**Purpose and Functionality:**
-------------------------

The `FSNERTokenizerUtils` class is designed to tokenize input data for entity recognition tasks. It provides methods for tokenizing queries, supports, and extracting entities from query-scoring outputs.

### Main Functions:

-   **tokenize**: Takes a list of strings or lists of strings as input and returns a dictionary with additional keys and values for start_token_id, end_token_id, and sizes of example lists for each entity type.
-   **extract_entity_from_scores**: Extracts entities from query-scoring outputs given a threshold score.

### Implementation Details:

#### tokenize
-   Uses `AutoTokenizer` to preprocess input data with padding and truncation. Supports tokenization in both list-of-list and list-of-string formats.
-   Manages output data across multiple lists by concatenating tensor values.

#### extract_entity_from_scores
-   Processes scoring outputs from query-scoring tasks, extracting tokens as potential entity boundaries based on score threshold `thresh`.

### Dependencies:
------------

-   Utilizes the `transformers` library for handling transformers models.
-   Uses PyTorch for tensor operations and computations (`torch.FloatTensor`, `torch.LongTensor`). 

       


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/rag-end2end-retriever/lightning_base.py


key: chroma:document
string_value:
CODE:

class BaseTransformer(pl.LightningModule):
    def __init__(
        self,
        hparams: argparse.Namespace,
        num_labels=None,
        mode="base",
        config=None,
        tokenizer=None,
        model=None,
        **config_kwargs,
    ):
        """Initialize a model, tokenizer and config."""
        super().__init__()
        # TODO: move to self.save_hyperparameters()
        # self.save_hyperparameters()
        # can also expand arguments into trainer signature for easier reading

        self.save_hyperparameters(hparams)
        self.step_count = 0
        self.output_dir = Path(self.hparams.output_dir)
        cache_dir = self.hparams.cache_dir if self.hparams.cache_dir else None
        if config is None:
            self.config = AutoConfig.from_pretrained(
                self.hparams.config_name if self.hparams.config_name else self.hparams.model_name_or_path,
                **({"num_labels": num_labels} if num_labels is not None else {}),
                cache_dir=cache_dir,
                **config_kwargs,
            )
        else:
            self.config: PretrainedConfig = config

        extra_model_params = ("encoder_layerdrop", "decoder_layerdrop", "dropout", "attention_dropout")
        for p in extra_model_params:
            if getattr(self.hparams, p, None):
                assert hasattr(self.config, p), f"model config doesn't have a `{p}` attribute"
                setattr(self.config, p, getattr(self.hparams, p))

        if tokenizer is None:
            self.tokenizer = AutoTokenizer.from_pretrained(
                self.hparams.tokenizer_name if self.hparams.tokenizer_name else self.hparams.model_name_or_path,
                cache_dir=cache_dir,
            )
        else:
            self.tokenizer: PreTrainedTokenizer = tokenizer
        self.model_type = MODEL_MODES[mode]
        if model is None:
            self.model = self.model_type.from_pretrained(
                self.hparams.model_name_or_path,
                from_tf=bool(".ckpt" in self.hparams.model_name_or_path),
                config=self.config,
                cache_dir=cache_dir,
            )
        else:
            self.model = model

    def load_hf_checkpoint(self, *args, **kwargs):
        self.model = self.model_type.from_pretrained(*args, **kwargs)

    def get_lr_scheduler(self):
        get_schedule_func = arg_to_scheduler[self.hparams.lr_scheduler]
        scheduler = get_schedule_func(
            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=self.total_steps()
        )
        scheduler = {"scheduler": scheduler, "interval": "step", "frequency": 1}
        return scheduler

    def configure_optimizers(self):
        """Prepare optimizer and schedule (linear warmup and decay)"""
        model = self.model
        no_decay = ["bias", "LayerNorm.weight"]
        optimizer_grouped_parameters = [
            {
                "params": [
                    p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)
                ],  # check this named paramters
                "weight_decay": self.hparams.weight_decay,
            },
            {
                "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
                "weight_decay": 0.0,
            },
        ]
        if self.hparams.adafactor:
            optimizer = Adafactor(
                optimizer_grouped_parameters, lr=self.hparams.learning_rate, scale_parameter=False, relative_step=False
            )

        else:
            optimizer = AdamW(
                optimizer_grouped_parameters, lr=self.hparams.learning_rate, eps=self.hparams.adam_epsilon
            )
        self.opt = optimizer

        scheduler = self.get_lr_scheduler()

        return [optimizer], [scheduler]

    def test_step(self, batch, batch_nb):
        return self.validation_step(batch, batch_nb)

    def test_epoch_end(self, outputs):
        return self.validation_end(outputs)

    def total_steps(self) -> int:
        """The number of total training steps that will be run. Used for lr scheduler purposes."""
        num_devices = max(1, self.hparams.gpus)  # TODO: consider num_tpu_cores
        effective_batch_size = self.hparams.train_batch_size * self.hparams.accumulate_grad_batches * num_devices
        return (self.dataset_size / effective_batch_size) * self.hparams.max_epochs

    def setup(self, stage):
        if stage == "test":
            self.dataset_size = len(self.test_dataloader().dataset)
        else:
            self.train_loader = self.get_dataloader("train", self.hparams.train_batch_size, shuffle=True)
            self.dataset_size = len(self.train_dataloader().dataset)

    def get_dataloader(self, type_path: str, batch_size: int, shuffle: bool = False):
        raise NotImplementedError("You must implement this for your task")

    def train_dataloader(self):
        return self.train_loader

    def val_dataloader(self):
        return self.get_dataloader("dev", self.hparams.eval_batch_size, shuffle=False)

    def test_dataloader(self):
        return self.get_dataloader("test", self.hparams.eval_batch_size, shuffle=False)

    def _feature_file(self, mode):
        return os.path.join(
            self.hparams.data_dir,
            "cached_{}_{}_{}".format(
                mode,
                list(filter(None, self.hparams.model_name_or_path.split("/"))).pop(),
                str(self.hparams.max_seq_length),
            ),
        )

    @pl.utilities.rank_zero_only
    def on_save_checkpoint(self, checkpoint: Dict[str, Any]) -> None:
        save_path = self.output_dir.joinpath("best_tfmr")
        self.model.config.save_step = self.step_count
        self.model.save_pretrained(save_path)
        self.tokenizer.save_pretrained(save_path)

    @staticmethod
    def add_model_specific_args(parser, root_dir):
        parser.add_argument(
            "--model_name_or_path",
            default=None,
            type=str,
            required=True,
            help="Path to pretrained model or model identifier from huggingface.co/models",
        )
        parser.add_argument(
            "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
        )
        parser.add_argument(
            "--tokenizer_name",
            default=None,
            type=str,
            help="Pretrained tokenizer name or path if not the same as model_name",
        )
        parser.add_argument(
            "--cache_dir",
            default=str(Path(__file__).parent / "test_run" / "cache"),
            type=str,
            help="Where do you want to store the pre-trained models downloaded from huggingface.co",
        )
        parser.add_argument(
            "--encoder_layerdrop",
            type=float,
            help="Encoder layer dropout probability (Optional). Goes into model.config",
        )
        parser.add_argument(
            "--decoder_layerdrop",
            type=float,
            help="Decoder layer dropout probability (Optional). Goes into model.config",
        )
        parser.add_argument(
            "--dropout",
            type=float,
            help="Dropout probability (Optional). Goes into model.config",
        )
        parser.add_argument(
            "--attention_dropout",
            type=float,
            help="Attention dropout probability (Optional). Goes into model.config",
        )
        parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
        parser.add_argument(
            "--lr_scheduler",
            default="linear",
            choices=arg_to_scheduler_choices,
            metavar=arg_to_scheduler_metavar,
            type=str,
            help="Learning rate scheduler",
        )
        parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
        parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
        parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
        parser.add_argument("--num_workers", default=4, type=int, help="kwarg passed to DataLoader")
        parser.add_argument("--num_train_epochs", dest="max_epochs", default=3, type=int)
        parser.add_argument("--train_batch_size", default=32, type=int)
        parser.add_argument("--eval_batch_size", default=32, type=int)


SUMMARY:
**Summary:**

The `BaseTransformer` class is a PyTorch Lightning module for transformers. It defines the architecture and training parameters for a transformer-based model.

**Main Purpose and Functionality:**

* Initializes the model, tokenizer, and configuration from pre-trained Hugging Face models
* Configures the optimizer and learning rate scheduler
* Supports multiple training modes (training, validation, testing)

**Key Implementation Details:**

* Uses `Argparse` for hyperparameter parsing and `HuggingFace` libraries for auto-loading model configurations and tokenizers.
* Implemented custom `get_dataloader`, `setup`, and `_feature_file` methods to support dataset loading.

**Hyperparameters and Trainable Parameters:**

* Model-related parameters:
	+ `model_name_or_path`
	+ `config_name`
	+ `tokenizer_name`
	+ `cache_dir`
	+ `encoder_layerdrop`
	+ `decoder_layerdrop`
	+ `dropout`
	+ `attention_dropout`
* Training-related parameters:
	+ `learning_rate`
	+ `lr_scheduler` (linear)
	+ `weight_decay`
	+ `adam_epsilon`
	+ `warmup_steps`
	+ `num_workers`
	+ `num_train_epochs`

**Training and Evaluation:**

The model is trained using the `train_dataloader` method, which returns a PyTorch DataLoader instance for training data. The model can also be evaluated on validation data using `val_dataloader`, while testing data can be used with `test_dataloader`.


key: chroma:document
string_value:
CODE:

class InitCallback(pl.Callback):
    # this process can also be done with PL ddp plugging.
    # But still it is experimental (check original RAG, I updated that with pluggin (shamanez))
    def on_sanity_check_start(self, trainer, pl_module):
        if (
            trainer.is_global_zero and trainer.global_rank == 0
        ):  # we initialize the retriever only on master worker with RAY. In new pytorch-lightning accelorators are removed.


SUMMARY:
**Class Summary: `InitCallback`**

The `InitCallback` class extends PL's `Callback` and provides an initialization step for the `rangefinder` module.

**Primary Purpose:**
Initializes the `rangefinder` module, specifically its retrieval component, when running on the master worker node (Ray).

**Key Implementation Details:**

* Checks if the training process is being run on Master Worker with Ray (`trainer.is_global_zero and trainer.global_rank == 0`)
* Calls the `init_retrieval()` method of the `rangefinder` model to initialize the retrieval component.

**Notable Dependencies:**
- PL's `Callback`
- PyTorch Lightning library (accelerator)
- Ray framework
- `rangefinder` module

Note that this code uses a comment to mention that it is experimental and can also be achieved through PL plugging.


key: chroma:document
string_value:
CODE:

class CheckParamCallback(pl.Callback):
    # check whether new added model paramters are differentiable
    def on_after_backward(self, trainer, pl_module):
        # print(pl_module.model.rag)
        for name, param in pl_module.model.rag.named_parameters():
            if param.grad is None:


SUMMARY:
**Class Summary:**

`CheckParamCallback`

A PL Callback class that checks whether added model parameters are differentiable.

**Purpose and Functionality:**
The primary purpose of this class is to inspect the addable parameters in the `rag` module of the trained model and print their names if they don't have a computed gradient (`param.grad` is None).

**Key Implementation Details:**

* Iterates over `rag.named_parameters()` of the `pl_module.model.rag`
* Checks for non-differentiable parameters (i.e., those with no gradient)
* Prints these parameter names on completion

**Notable Dependencies and Requirements:**

* Uses PyTorch Lightning's (`pl`) callback functionality
* Assumes access to a trained model's `rag` module.


key: chroma:document
string_value:
CODE:

class LoggingCallback(pl.Callback):
    def on_batch_end(self, trainer, pl_module):
        lr_scheduler = trainer.lr_schedulers[0]["scheduler"]
        lrs = {f"lr_group_{i}": lr for i, lr in enumerate(lr_scheduler.get_lr())}
        pl_module.logger.log_metrics(lrs)

    def on_validation_end(self, trainer: pl.Trainer, pl_module: pl.LightningModule):
        rank_zero_info("***** Validation results *****")
        metrics = trainer.callback_metrics
        # Log results
        for key in sorted(metrics):
            if key not in ["log", "progress_bar"]:
                rank_zero_info("{} = {}\n".format(key, str(metrics[key])))

    def on_test_end(self, trainer: pl.Trainer, pl_module: pl.LightningModule):
        rank_zero_info("***** Test results *****")
        metrics = trainer.callback_metrics
        # Log and save results to file
        output_test_results_file = os.path.join(pl_module.hparams.output_dir, "test_results.txt")
        with open(output_test_results_file, "w") as writer:
            for key in sorted(metrics):
                if key not in ["log", "progress_bar"]:
                    rank_zero_info("{} = {}\n".format(key, str(metrics[key])))


SUMMARY:
**Concise Summary**

The `LoggingCallback` class extends PyTorch Lightning's `pl.Callback` and provides three callback functions for logging training/validation/test results.

**Key Implementation Details:**

* `on_batch_end`: logs learning rate values (`lr_scheduler.get_lr()`) in a dictionary format.
	+ Method Signature: `on_batch_end(trainer, pl_module)`
* `on_validation_end`:
	+ Logs validation metrics (excluding "log" and "progress_bar") in sorted order.
	+ Uses `rank_zero_info` for logging output.
	+ File Output: saves results to a file named "test_results.txt" in the model's output directory.
* `on_test_end`: similar to `on_validation_end`, but logs results with additional context.

**Notable Dependencies and Requirements:**

* PyTorch Lightning (`pl.Trainer`, `pl.LightningModule`)
* PyTorch Tensors (for accessing learning rate values)
* `rank_zero_info` function for logging output
* External Dependency: None


key: chroma:document
string_value:
CODE:

def add_generic_args(parser, root_dir) -> None:
    #  To allow all pl args uncomment the following line
    #  parser = pl.Trainer.add_argparse_args(parser)
    parser.add_argument(
        "--output_dir",
        default=str(Path(__file__).parent / "test_run" / "model_checkpoints"),
        type=str,
        help="The output directory where the model predictions and checkpoints will be written.",
    )
    parser.add_argument(
        "--fp16",
        action="store_true",
        help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit",
    )

    parser.add_argument(
        "--fp16_opt_level",
        type=str,
        default="O2",
        help=(
            "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. "
            "See details at https://nvidia.github.io/apex/amp.html"
        ),
    )
    parser.add_argument("--n_tpu_cores", dest="tpu_cores", type=int)
    parser.add_argument("--max_grad_norm", dest="gradient_clip_val", default=1.0, type=float, help="Max gradient norm")
    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
    parser.add_argument("--do_predict", action="store_true", help="Whether to run predictions on the test set.")
    parser.add_argument(
        "--gradient_accumulation_steps",
        dest="accumulate_grad_batches",
        type=int,
        default=1,
        help="Number of updates steps to accumulate before performing a backward/update pass.",
    )
    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")
    parser.add_argument(
        "--data_dir",
        default=str(Path(__file__).parent / "test_run" / "dummy-train-data"),
        type=str,
        help="The input data dir. Should contain the training files for the CoNLL-2003 NER task.",


SUMMARY:
**Summary: `add_generic_args` Function**

This function adds various command-line arguments to a PyTorch Lightning parser.

**Purpose and Functionality**
The primary purpose of this function is to enable the configuration of several parameters for model training and prediction, including output directory, precision mode, optimization level, hyperparameters, and seed value.

**Key Implementation Details**

*   Adds 10 additional command-line arguments:
    *   `--output_dir`: Specifies the output directory for model predictions and checkpoints.
    *   `--fp16`/`--fp16_opt_level`: Enables 16-bit precision mode (mixed) with optimization level set to ["O0", "O1", "O2", or "O3"].
    *   `--n_tpu_cores`, `-max_grad_norm`, and `--seed`: Set the number of TPU cores, max gradient norm, and seed value for initialization, respectively.
    *   `--do_train` and `--do_predict`: Indicates whether to run training or predictions on the test set.
    *   `--gradient_accumulation_steps`: Accumulates gradients over a specified number of updates steps before performing a backward pass.

**Notable Dependencies**
The function does not explicitly depend on any external modules, but it relies on PyTorch Lightning and its command-line argument parser. Additionally, it utilizes the `Path` class from the `pathlib` module to resolve directory paths.


key: chroma:document
string_value:
CODE:

def generic_train(
    model: BaseTransformer,
    args: argparse.Namespace,
    early_stopping_callback=None,
    logger=True,  # can pass WandbLogger() here
    extra_callbacks=[],
    checkpoint_callback=None,
    logging_callback=None,
    **extra_train_kwargs,
):
    pl.seed_everything(args.seed)

    # init model
    odir = Path(model.hparams.output_dir)
    odir.mkdir(exist_ok=True)

    # add custom checkpoints
    if checkpoint_callback is None:
        checkpoint_callback = pl.callbacks.ModelCheckpoint(
            filepath=args.output_dir, prefix="checkpoint", monitor="val_loss", mode="min", save_top_k=1
        )
    if early_stopping_callback:
        extra_callbacks.append(early_stopping_callback)
    if logging_callback is None:
        logging_callback = LoggingCallback()

    train_params = {}

    if args.fp16:
        train_params["precision"] = 16

    if args.gpus > 1:
        train_params["accelerator"] = "auto"
        train_params["strategy"] = "ddp"

    train_params["accumulate_grad_batches"] = args.accumulate_grad_batches
    train_params["profiler"] = None
    train_params["devices"] = "auto"

    trainer = pl.Trainer.from_argparse_args(
        args,
        weights_summary=None,
        callbacks=[logging_callback] + extra_callbacks + [InitCallback()] + [checkpoint_callback],
        logger=logger,
        val_check_interval=1,
        num_sanity_val_steps=2,
        **train_params,
    )

    if args.do_train:
        trainer.fit(model)

    else:
        print("RAG modeling tests with new set functions successfully executed!")


SUMMARY:
**generic_train Function Summary**

### Main Purpose and Functionality

The `generic_train` function initializes and trains a `BaseTransformer` model using the PyTorch Lightning (PL) library. It takes various arguments to customize the training process, including early stopping callback, logging callback, checkpointing, and other hyperparameters.

**Key Implementation Details**

* Initializes model and output directory
* Configures checkpoints, logging, and accelerator settings based on input arguments
* Creates a `Trainer` object with specified callbacks and hyperparameters

### Notable Dependencies or Requirements

* PyTorch Lightning (PL) library
* `argparse.Namespace` for argument parsing
* WandbLogger for logging (optional)
* LoggingCallback for custom logging

### Important Method Signatures and Interfaces

* `pl.Trainer.from_argparse_args`: creates a trainer object from parsed arguments
* `pl.seed_everything`: sets the random seed for the training process
* `ModelCheckpoint` and `LoggingCallback` for monitoring progress


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/rag-end2end-retriever/callbacks_rag.py


key: chroma:document
string_value:
CODE:

def count_trainable_parameters(model):
    model_parameters = filter(lambda p: p.requires_grad, model.parameters())
    params = sum([np.prod(p.size()) for p in model_parameters])


SUMMARY:
**Summary:**

`count_trainable_parameters` function:

*   Returns the total count of trainable parameters in a given PyTorch neural network model.

**Key Details:**

*   Uses `filter` and lambda function to extract model parameters requiring gradient computation (`model.parameters()`).
*   Applies `np.prod(p.size())` to calculate parameter size for each trainable parameter.
*   Sums up the sizes of all trainable parameters using `sum`.

**Notable Dependencies/Requirements:**

*   PyTorch library (`torch`)
*   NumPy library (`numpy`, imported as `np`)


key: chroma:document
string_value:
CODE:

def get_checkpoint_callback(output_dir, metric):
    """Saves the best model by validation EM score."""
    if metric == "rouge2":
        exp = "{val_avg_rouge2:.4f}-{step_count}"
    elif metric == "bleu":
        exp = "{val_avg_bleu:.4f}-{step_count}"
    elif metric == "em":
        exp = "{val_avg_em:.4f}-{step_count}"
    elif metric == "loss":
        exp = "{val_avg_loss:.4f}-{step_count}"
    else:
        raise NotImplementedError(
            f"seq2seq callbacks only support rouge2 and bleu, got {metric}, You can make your own by adding to this"
            " function."
        )

    checkpoint_callback = ModelCheckpoint(
        dirpath=output_dir,
        filename=exp,
        monitor=f"val_{metric}",
        mode="max",
        save_top_k=1,
        every_n_epochs=1,  # works only with PL > 1.3
    )



SUMMARY:
**Summary:**

The `get_checkpoint_callback` function returns a `ModelCheckpoint` instance that saves the best model based on validation metrics.

**Main Purpose and Functionality:**
Saves the best model by validating different metrics (rouge2, bleu, em, loss).

**Key Implementation Details:**

* Uses `ModelCheckpoint` to save the best model
* Supports four metrics:
	+ rouge2
	+ bleu
	+ em
	+ loss
* Customizable filename template using `{val_metric:.4f}-{step_count}`

**Notable Dependencies or Requirements:**
None explicitly mentioned.

**Return Value:** A `ModelCheckpoint` instance.

**External Interfaces:**

* `ModelCheckpoint`: used for saving the best model
* PL ( PyTorch Lightning): >= 1.3 required for certain parameter settings (`every_n_epochs=1`)


key: chroma:document
string_value:
CODE:

def get_early_stopping_callback(metric, patience):
    return EarlyStopping(
        monitor=f"val_{metric}",  # does this need avg?
        mode="min" if "loss" in metric else "max",
        patience=patience,
        verbose=True,


SUMMARY:
**Summary:**

Creates an `EarlyStopping` callback to monitor the validation loss of a specific metric (`metric`) and stop training when improvement is not seen after a specified number of epochs (`patience`).

**Key Details:**

* Uses `val_<metric>` as the monitoring metric (e.g., `val_loss`)
* Adjusts observation mode based on whether the metric is "loss" or not
* Verbose mode is enabled

**Note:** The `avg` parameter is commented out, suggesting it may be optional or specific to a particular use case.


key: chroma:document
string_value:
CODE:

class Seq2SeqLoggingCallback(pl.Callback):
    def on_batch_end(self, trainer, pl_module):
        lrs = {f"lr_group_{i}": param["lr"] for i, param in enumerate(pl_module.trainer.optimizers[0].param_groups)}
        pl_module.logger.log_metrics(lrs)

    @rank_zero_only
    def _write_logs(
        self, trainer: pl.Trainer, pl_module: pl.LightningModule, type_path: str, save_generations=True
    ) -> None:
        logger.info(f"***** {type_path} results at step {trainer.global_step:05d} *****")
        metrics = trainer.callback_metrics
        trainer.logger.log_metrics({k: v for k, v in metrics.items() if k not in ["log", "progress_bar", "preds"]})
        # Log results
        od = Path(pl_module.hparams.output_dir)
        if type_path == "test":
            results_file = od / "test_results.txt"
            generations_file = od / "test_generations.txt"
        else:
            # this never gets hit. I prefer not to save intermediate generations, and results are in metrics.json
            # If people want this it will be easy enough to add back.
            results_file = od / f"{type_path}_results/{trainer.global_step:05d}.txt"
            generations_file = od / f"{type_path}_generations/{trainer.global_step:05d}.txt"
            results_file.parent.mkdir(exist_ok=True)
            generations_file.parent.mkdir(exist_ok=True)
        with open(results_file, "a+") as writer:
            for key in sorted(metrics):
                if key in ["log", "progress_bar", "preds"]:
                    continue
                val = metrics[key]
                if isinstance(val, torch.Tensor):
                    val = val.item()
                msg = f"{key}: {val:.6f}\n"
                writer.write(msg)

        if not save_generations:
            return

        if "preds" in metrics:
            content = "\n".join(metrics["preds"])
            generations_file.open("w+").write(content)

    @rank_zero_only
    def on_train_start(self, trainer, pl_module):
        try:
            npars = pl_module.model.model.num_parameters()
        except AttributeError:
            npars = pl_module.model.num_parameters()

        n_trainable_pars = count_trainable_parameters(pl_module)
        # mp stands for million parameters
        trainer.logger.log_metrics({"n_params": npars, "mp": npars / 1e6, "grad_mp": n_trainable_pars / 1e6})

    @rank_zero_only
    def on_test_end(self, trainer: pl.Trainer, pl_module: pl.LightningModule):
        save_json(pl_module.metrics, pl_module.metrics_save_path)
        return self._write_logs(trainer, pl_module, "test")

    @rank_zero_only
    def on_validation_end(self, trainer: pl.Trainer, pl_module):
        save_json(pl_module.metrics, pl_module.metrics_save_path)
        # Uncommenting this will save val generations


SUMMARY:
**Seq2SeqLoggingCallback Class Summary**

This class is a custom callback for PyTorch's Lightning framework. Its primary purpose is to log and store the progress of a sequence-to-sequence model during training and testing.

**Key Functionality:**

* Logs batch end metrics (e.g., loss, accuracy) to the console
* Saves test and validation results to files for later analysis
* Includes logging of number of parameters, gradients, and global step

**Notable Implementation Details:**
- Uses `pl.Callback` as the base class architecture
- Defines four callback methods:
  - `_write_logs` (test and validation logging)
  - `on_batch_end` (batch end metrics logging)
  - `on_train_start` (training start metrics logging)
  - `on_test_end` (testing end results to disk)

**External Dependencies:**
None

**Interfaces Implemented/Inherited:**
`pl.Callback`

This class provides a structured way for the model's performance to be tracked and saved during training, making it easier to monitor progress without manually modifying the model's execution flow.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py


key: chroma:document
string_value:
CODE:

class RayRetriever:
    def __init__(self):
        self.initialized = False

    def create_rag_retriever(self, config, question_encoder_tokenizer, generator_tokenizer, index):
        if not self.initialized:
            self.retriever = RagRetriever(
                config,
                question_encoder_tokenizer=question_encoder_tokenizer,
                generator_tokenizer=generator_tokenizer,
                index=index,
                init_retrieval=False,
            )
            self.initialized = True

    def init_retrieval(self):
        self.retriever.index.init_index()

    def clear_object(self):
        # delete the old self.retriever object before assigning the new index
        del self.retriever
        self.initialized = False

    def retrieve(self, question_hidden_states, n_docs):
        doc_ids, retrieved_doc_embeds = self.retriever._main_retrieve(question_hidden_states, n_docs)
        doc_dicts = self.retriever.index.get_doc_dicts(doc_ids)


SUMMARY:
**RayRetriever Class Summary**

The `RayRetriever` class is designed for efficient passage retrieval using the Rag (Rake-based) retrieval algorithm.

### Main Purpose and Functionality

* Instantiates and configures a `RagRetriever` object with the provided configuration, tokenizers, and index.
* Performs passage retrieval by calling the `_main_retrieve` method using the initialized `RagRetriever` instance.

### Key Implementation Details

* Initialization: Creates a new `RagRetriever` instance or updates an existing one if not already initialized (`create_rag_retriever`).
* Retrieval: Calls `_main_retrieve` to retrieve top documents based on the provided question hidden states and number of documents (`retrieve`).

### Notable Dependencies and Requirements

* `RagRetriever`: The base class used by the `RayRetriever` instance.
* `question_encoder_tokenizer`, `generator_tokenizer`: Tokenizers required for configuration.
* `index`: Index object used for indexing and searching passages.

### Interfaces and Functions

* `create_rag_retriever(config, question_encoder_tokenizer, generator_tokenizer, index)`:
	+ Configures the retreiver with the provided parameters.
	+ Returns the configured `RagRetriever` instance.
* `_main_retrieve(question_hidden_states, n_docs)`:
	+ The main retrieval function used to retrieve top documents.

### Class Variables and Functions

* `initialized`: Flag indicating whether the retreiver has been initialized.
* `retriever`: The instantiated `RagRetriever` object.


key: chroma:document
string_value:
CODE:

class RagRayDistributedRetriever(RagRetriever):
    """
    A distributed retriever built on top of the ``Ray`` API, a library
    for building distributed applications (https://docs.ray.io/en/master/).
    package. During training, all training workers initialize their own
    instance of a `RagRayDistributedRetriever`, and each instance of
    this distributed retriever shares a common set of Retrieval Ray
    Actors (https://docs.ray.io/en/master/walkthrough.html#remote
    -classes-actors) that load the index on separate processes. Ray
    handles the communication between the `RagRayDistributedRetriever`
    instances and the remote Ray actors. If training is done in a
    non-distributed setup, the index will simply be loaded in the same
    process as the training worker and Ray will not be used.

    Args:
        config (:class:`~transformers.RagConfig`):
            The configuration of the RAG model this Retriever is used with. Contains parameters indicating which ``Index`` to build.
        question_encoder_tokenizer (:class:`~transformers.PreTrainedTokenizer`):
            The tokenizer that was used to tokenize the question.
            It is used to decode the question and then use the generator_tokenizer.
        generator_tokenizer (:class:`~transformers.PreTrainedTokenizer`):
            The tokenizer used for the generator part of the RagModel.
        retrieval_workers (:obj:`List[ray.ActorClass(RayRetriever)]`): A list of already initialized `RayRetriever` actors.
            These actor classes run on remote processes and are responsible for performing the index lookup.
        index (:class:`~transformers.retrieval_rag.Index`, optional, defaults to the one defined by the configuration):
            If specified, use this index instead of the one built using the configuration
    """

    def __init__(self, config, question_encoder_tokenizer, generator_tokenizer, retrieval_workers, index=None):
        if index is not None and index.is_initialized() and len(retrieval_workers) > 0:
            raise ValueError(
                "When using Ray for distributed fine-tuning, "
                "you'll need to provide the paths instead, "
                "as the dataset and the index are loaded "
                "separately. More info in examples/rag/use_own_knowledge_dataset.py "
            )

        super().__init__(
            config,
            question_encoder_tokenizer=question_encoder_tokenizer,
            generator_tokenizer=generator_tokenizer,
            index=index,
            init_retrieval=False,
        )

        self.retrieval_workers = retrieval_workers
        self.question_encoder_tokenizer = question_encoder_tokenizer
        self.generator_tokenizer = generator_tokenizer
        if len(self.retrieval_workers) > 0:
            ray.get(
                [
                    worker.create_rag_retriever.remote(config, question_encoder_tokenizer, generator_tokenizer, index)
                    for worker in self.retrieval_workers
                ]
            )

    def init_retrieval(self):
        """
        Retriever initialization function, needs to be called from the
        training process. This function triggers retrieval initialization
        for all retrieval actors if using distributed setting, or loads
        index into current process if training is not distributed.
        """
        logger.info("initializing retrieval")

        if len(self.retrieval_workers) > 0:
            ray.get([worker.init_retrieval.remote() for worker in self.retrieval_workers])
        else:
            # Non-distributed training. Load index into this same process.
            self.index.init_index()

    def retrieve(self, question_hidden_states, n_docs):
        """
        Retrieves documents for specified ``question_hidden_states``. If
        running training with multiple workers, a random retrieval actor is
        selected to perform the index lookup and return the result.

        Args:
            question_hidden_states (:obj:`np.ndarray` of shape :obj:`(batch_size, vector_size)`):
                A batch of query vectors to retrieve with.
            n_docs (:obj:`int`):
                The number of docs retrieved per query.

        Output:
            retrieved_doc_embeds (:obj:`np.ndarray` of shape :obj:`(batch_size, n_docs, dim)`
                The retrieval embeddings of the retrieved docs per query.
            doc_ids (:obj:`np.ndarray` of shape :obj:`batch_size, n_docs`)
                The ids of the documents in the index
            doc_dicts (:obj:`List[dict]`):
                The retrieved_doc_embeds examples per query.
        """
        if len(self.retrieval_workers) > 0:
            # Select a random retrieval actor.
            random_worker = self.retrieval_workers[random.randint(0, len(self.retrieval_workers) - 1)]
            doc_ids, retrieved_doc_embeds, doc_dicts = ray.get(
                random_worker.retrieve.remote(question_hidden_states, n_docs)
            )
        else:
            doc_ids, retrieved_doc_embeds = self._main_retrieve(question_hidden_states, n_docs)
            doc_dicts = self.index.get_doc_dicts(doc_ids)
        return retrieved_doc_embeds, doc_ids, doc_dicts

    @classmethod
    def get_tokenizers(cls, retriever_name_or_path, indexed_dataset=None, **kwargs):
        return super(RagRayDistributedRetriever, cls).get_tokenizers(retriever_name_or_path, indexed_dataset, **kwargs)

    @classmethod
    def from_pretrained(cls, retriever_name_or_path, actor_handles, indexed_dataset=None, **kwargs):
        config = kwargs.pop("config", None) or RagConfig.from_pretrained(retriever_name_or_path, **kwargs)
        rag_tokenizer = RagTokenizer.from_pretrained(retriever_name_or_path, config=config)
        question_encoder_tokenizer = rag_tokenizer.question_encoder
        generator_tokenizer = rag_tokenizer.generator

        if indexed_dataset is not None:
            config.index_name = "custom"
            index = CustomHFIndex(config.retrieval_vector_size, indexed_dataset)
        else:
            index = cls._build_index(config)

        return cls(
            config,
            question_encoder_tokenizer=question_encoder_tokenizer,
            generator_tokenizer=generator_tokenizer,
            retrieval_workers=actor_handles,
            index=index,
        )

    def re_load(self):
        logger.info("re-loading the new dataset with embeddings")
        # access from the training loop

        ray.get([worker.clear_object.remote() for worker in self.retrieval_workers])

        # build the index object again
        index = self._build_index(self.config)

        ray.get(
            [
                worker.create_rag_retriever.remote(
                    self.config, self.question_encoder_tokenizer, self.generator_tokenizer, index
                )
                for worker in self.retrieval_workers
            ]


SUMMARY:
**RagRayDistributedRetriever Class Summary**

This class is a distributed retriever built on top of the `Ray` API. It inherits from `RagRetriever` and extends its functionality for parallel processing.

**Main Purpose:**
The main purpose of this class is to provide a distributed retrieval system for RAG models, allowing for scalability and parallelization of index loading and queryprocessing.

**Key Implementation Details:**

*   The retriever shares a common set of `Retrieval Ray` actors that load the index on separate processes.
*   It uses `Ray.get()` to remotely execute functions on worker processes.
*   The `init_retrieval()` function triggers retrieval initialization for all workers and loads the index into the current process.

**Notable Dependencies and Requirements:**

*   The class uses `transformers.RagConfig` for model configuration.
*   It relies on `transformers.PreTrainedTokenizer` for question encoder tokenizer and generator tokenizer.
*   The `ray.ActorClass(RayRetriever)` is used to create the retrieval workers.

**Methods and Interfaces:**

*   `__init__()`: Initializes the retriever with the given parameters.
*   `init_retrieval()`: Triggers retrieval initialization.
*   `retrieve(question_hidden_states, n_docs)`: Retrieves documents for specified query vectors.
*   `from_pretrained(retriever_name_or_path, actor_handles, indexed_dataset=None, **kwargs)`: Creates a new instance from pre-trained model.

**External Dependencies:**

*   [Ray](https://docs.ray.io/en/master/index.html): A library for building distributed applications.
*   [transformers](https://huggingface.co/docs/transformers/index): A Hugging Face library for Natural Language Processing.

Note that the `index` parameter is not always used, and its usage depends on whether a custom index is provided or the built-in one. The `re_load()` method ensures re-loading of the dataset with embeddings by clearing existing worker memory and rebuilding the index.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/rag-end2end-retriever/utils_rag.py


key: chroma:document
string_value:
CODE:

def encode_line(tokenizer, line, max_length, padding_side, pad_to_max_length=True, return_tensors="pt"):
    extra_kw = {"add_prefix_space": True} if isinstance(tokenizer, BartTokenizer) and not line.startswith(" ") else {}
    tokenizer.padding_side = padding_side
    return tokenizer(
        [line],
        max_length=max_length,
        padding="max_length" if pad_to_max_length else None,
        truncation=True,
        return_tensors=return_tensors,
        add_special_tokens=True,
        **extra_kw,


SUMMARY:
**Summary**

The `encode_line` function encodes a single input line of text into a format suitable for downstream NLP tasks. It takes in the following parameters:

*   `max_length`: The maximum length of the encoded output
*   `padding_side`: The side of the sequence to pad with [Padding Token]
*   `pad_to_max_length`: A flag indicating whether to pad the input sequence to the max length (Default: True), or truncate it if necessary
*   `return_tensors`: The tensor type to return (Default: "pt")

The function utilizes a tokenizer to preprocess the input line and returns the encoded output in a specified tensor format. Notable details include:

*   Padding side is set by default based on the provided tokenizer, but can be overridden using the `padding_side` parameter.
*   The `add_prefix_space` option is added or removed dynamically depending on the type of tokenizer used.

**Method Signature:**

`encode_line(tokenizer, line, max_length, padding_side=None, pad_to_max_length=True, return_tensors="pt") -> output`

**Dependencies:**

*   BARTTokenizer and other related transformers
*   `add_special_tokens` function

**Interfaces Implemented:**

1.  `__init_api__(...)` (implied from tokenizer initialization)
2.` __call__( line )`

This function is likely used for text encoding in a deep learning model, such as a natural language understanding pipeline.


key: chroma:document
string_value:
CODE:

def trim_batch(
    input_ids,
    pad_token_id,
    attention_mask=None,
):
    """Remove columns that are populated exclusively by pad_token_id"""
    keep_column_mask = input_ids.ne(pad_token_id).any(dim=0)
    if attention_mask is None:
        return input_ids[:, keep_column_mask]
    else:


SUMMARY:
**Summary: `trim_batch` Function**

The `trim_batch` function trims redundant columns from a batch of input IDs by removing rows populated exclusively with the pad token ID. It also handles optional `attention_mask` parameter.

**Key Features:**

* Conditionally includes/trims based on non-zero values in `input_ids` and `pad_token_id`.
* Accepts an `attention_mask` parameter for downstream processing.
* Returns two arrays: `input_ids` with redundant columns trimmed (`keep_column_mask`) and optionally `attention_mask` aligned to the same mask.

**Method Signatures:**

* `trim_batch(input_ids, pad_token_id, attention_mask=None)`

**Notable Dependencies/Requirements:**

* `pytorch` tensor library for support of `input_ids`, `pad_token_id`, and `attention_mask`.
* No external dependencies listed.


key: chroma:document
string_value:
CODE:

class Seq2SeqDataset(Dataset):
    def __init__(
        self,
        tokenizer,
        data_dir,
        max_source_length,
        max_target_length,
        type_path="train",
        n_obs=None,
        src_lang=None,
        tgt_lang=None,
        prefix="",
    ):
        super().__init__()
        self.src_file = Path(data_dir).joinpath(type_path + ".source")
        self.tgt_file = Path(data_dir).joinpath(type_path + ".target")
        self.src_lens = self.get_char_lens(self.src_file)
        self.max_source_length = max_source_length
        self.max_target_length = max_target_length
        assert min(self.src_lens) > 0, f"found empty line in {self.src_file}"
        self.tokenizer = tokenizer
        self.prefix = prefix
        if n_obs is not None:
            self.src_lens = self.src_lens[:n_obs]
        self.src_lang = src_lang
        self.tgt_lang = tgt_lang

    def __len__(self):
        return len(self.src_lens)

    def __getitem__(self, index) -> Dict[str, torch.Tensor]:
        index = index + 1  # linecache starts at 1
        source_line = self.prefix + linecache.getline(str(self.src_file), index).rstrip("\n")
        tgt_line = linecache.getline(str(self.tgt_file), index).rstrip("\n")
        assert source_line, f"empty source line for index {index}"
        assert tgt_line, f"empty tgt line for index {index}"

        # Need to add eos token manually for T5
        if isinstance(self.tokenizer, T5Tokenizer):
            source_line += self.tokenizer.eos_token
            tgt_line += self.tokenizer.eos_token

        # Pad source and target to the right
        source_tokenizer = (
            self.tokenizer.question_encoder if isinstance(self.tokenizer, RagTokenizer) else self.tokenizer
        )
        target_tokenizer = self.tokenizer.generator if isinstance(self.tokenizer, RagTokenizer) else self.tokenizer

        source_inputs = encode_line(source_tokenizer, source_line, self.max_source_length, "right")
        target_inputs = encode_line(target_tokenizer, tgt_line, self.max_target_length, "right")

        source_ids = source_inputs["input_ids"].squeeze()
        target_ids = target_inputs["input_ids"].squeeze()
        src_mask = source_inputs["attention_mask"].squeeze()
        return {
            "input_ids": source_ids,
            "attention_mask": src_mask,
            "decoder_input_ids": target_ids,
        }

    @staticmethod
    def get_char_lens(data_file):
        return [len(x) for x in Path(data_file).open().readlines()]

    def collate_fn(self, batch) -> Dict[str, torch.Tensor]:
        input_ids = torch.stack([x["input_ids"] for x in batch])
        masks = torch.stack([x["attention_mask"] for x in batch])
        target_ids = torch.stack([x["decoder_input_ids"] for x in batch])
        tgt_pad_token_id = (
            self.tokenizer.generator.pad_token_id
            if isinstance(self.tokenizer, RagTokenizer)
            else self.tokenizer.pad_token_id
        )
        src_pad_token_id = (
            self.tokenizer.question_encoder.pad_token_id
            if isinstance(self.tokenizer, RagTokenizer)
            else self.tokenizer.pad_token_id
        )
        y = trim_batch(target_ids, tgt_pad_token_id)
        source_ids, source_mask = trim_batch(input_ids, src_pad_token_id, attention_mask=masks)
        batch = {
            "input_ids": source_ids,
            "attention_mask": source_mask,
            "decoder_input_ids": y,
        }


SUMMARY:
**Class Summary: Seq2SeqDataset**

The `Seq2SeqDataset` class is a custom dataset implementation for sequence-to-sequence tasks. It extends the `Dataset` class and provides a specific data loader for training models.

**Main Purpose and Functionality:**

* Loads and preprocesses source and target sequences for sequence-to-sequence tasks
* Handles line numbering and padding for sequences of varying lengths

**Key Implementation Details:**

* Uses `linecache` to load source and target lines from files specified by `self.src_file` and `self.tgt_file`
* Applies padding and EOS token addition for T5 specifically using the `T5Tokenizer` class
* Utilizes `encode_line` function (not shown in this snippet) to preprocess input ids, attention masks, and encoder/decoder inputs

**Notable Dependencies or Requirements:**

* `RagTokenizer`, `T5Tokenizer`
* External dependencies:
	+ `linecache`
	+ `torch`

Note: The `get_char_lens` function is not part of the main dataset class implementation but rather a separate utility function. 

The RAG system notation (Referenced Abstract Glossary) for this summary would be:

- R1: Class name
- U1: Main purpose and functionality
- U2: Key implementation details 
- D1: Notable dependencies or requirements

Here's how the full RAG entry might look:

Seq2SeqDataset - R1 - Datasets 
U1 - Load and preprocess source and target sequences, applies line numbering and padding.


key: chroma:document
string_value:
CODE:

def flatten_list(summary_ids: List[List]):


SUMMARY:
**Summary**

The `flatten_list` function takes a 2D list of `summary_ids` as input and returns a single list of all the IDs by utilizing the `itertools.chain` function from the Python Standard Library.

**Key Details**

* The function uses `itertools.chain.from_iterable()` to concatenate all sublists in `(summary_ids)`.
* The resulting iterable is converted into a list using `list()`, which serves as the return type.

**Notable Dependencies**

* None explicitly imported, but implicitly relies on Python's Standard Library (`itertools`).


key: chroma:document
string_value:
CODE:

def save_git_info(folder_path: str) -> None:
    """Save git information to output_dir/git_log.json"""
    repo_infos = get_git_info()


SUMMARY:
**Summary**

```markdown
### `save_git_info` Function

Stores Git information in a JSON file.

#### Parameters

* `folder_path`: The path to the folder where the JSON file will be saved.

#### Implemented Logic

- Retrieves Git repository information using `get_git_info()`.
- Saves the retrieved information to `git_log.json` in the specified `folder_path` using `save_json()`.

#### Notable Dependencies

- `os` module for path operations.
```

**RAG System Label**: `save_git_info` - Stores Git information in a JSON file.


key: chroma:document
string_value:
CODE:

def save_json(content, path, indent=4, **json_dump_kwargs):
    with open(path, "w") as f:


SUMMARY:
**Function Summary: `save_json`**

**Purpose and Functionality**: Saves the provided `content` to a JSON file at the specified `path`, with optional customization using `indent` and additional keyword arguments (`json_dump_kwargs`).

**Key Implementation Details**:

* Uses `json.dump()` to serialize content and write to file
* Applicability to custom JSON dumping configuration via `indent` and `**kwargs`

**Notable Dependencies and Requirements**:
 
* Python's built-in `json` module


key: chroma:document
string_value:
CODE:

def load_json(path):
    with open(path) as f:


SUMMARY:
**Summary:**

The `load_json` function reads and parses JSON data from a file located at the specified `path`. 

**Key Implementation Details:**

* Loads JSON data from a file using `json.load()`.
* Returns the parsed JSON object.

**Notable Dependencies or Requirements:**

* The `json` module is imported, which suggests that Python 3.x is being used.


key: chroma:document
string_value:
CODE:

def get_git_info():
    repo = git.Repo(search_parent_directories=True)
    repo_infos = {
        "repo_id": str(repo),
        "repo_sha": str(repo.head.object.hexsha),
        "repo_branch": str(repo.active_branch),
        "hostname": str(socket.gethostname()),
    }


SUMMARY:
**Summary:**

This function, `get_git_info()`, retrieves repository information from the local Git version control system.

**Key Details:**

* Uses `git.Repo` to access the current repository, searching parent directories for multiple repositories.
* Extracts and returns various repository metadata:
	+ `repo_id`: The repository object as a string.
	+ `repo_sha`: The SHA checksum of the repository's head commit, as a hexadecimal string.
	+ `repo_branch`: The name of the currently active branch, as a string.
* Incorporates system hostname via `socket.gethostname()`.

**No notable dependencies or requirements are present in this snippet.**

This function appears to be intended for retrieving information about the current Git repository and the local machine's hostname, likely for logging or debugging purposes.


key: chroma:document
string_value:
CODE:

def lmap(f: Callable, x: Iterable) -> List:
    """list(map(f, x))"""


SUMMARY:
**Function Summary: `lmap`**

* Purpose: Applies a given function `f` to each element of an iterable `x`, equivalent to Python's built-in `map`.
* Implementation:
	+ Uses `list.map()` internally.
	+ Returns the result as a list.
* Key Details:
	+ Function signature: `lmap(f: Callable, x: Iterable) -> List`
* Dependencies/Collections:
	+ Utilizes built-in `map` and `list` collections/types.


key: chroma:document
string_value:
CODE:

def pickle_save(obj, path):
    """pickle.dump(obj, path)"""
    with open(path, "wb") as f:


SUMMARY:
**Summary:**

This is a wrapper function `pickle_save` that saves an object to a file. The primary purpose is to write the given object to the specified path using the Python `pickle.dump` method.

**Key Implementation Details:**

* Uses `pickle.dump` method to serialize and save the object to the file.
* Opens the file in binary write mode (`"wb"`) using a context manager.

**Notable Dependencies or Requirements:**

* The `pickle` module, which is a built-in Python library for serializing and deserializing objects.
* External dependency: The `pickle` module's functions are used, no external libraries besides the standard library is added. 

This code seems to be redundant with the standard pickle dump function as it simply re-uses the same method, does not add any new functionality or logic.


key: chroma:document
string_value:
CODE:

def normalize_answer(s):
    """Lower text and remove punctuation, articles and extra whitespace."""

    def remove_articles(text):
        return re.sub(r"\b(a|an|the)\b", " ", text)

    def white_space_fix(text):
        return " ".join(text.split())

    def remove_punc(text):
        exclude = set(string.punctuation)
        return "".join(ch for ch in text if ch not in exclude)

    def lower(text):
        return text.lower()



SUMMARY:
**Summary:**

This function, `normalize_answer`, takes a string input `s` and normalizes it by:

1. Removing articles ("a", "an", "the") using regular expression replacement
2. Normalizing whitespace to single spaces
3. Removing punctuation using character set exclusion
4. Converting text to lowercase

**Key Implementation Details:**

* Uses regular expressions for article removal (`re.sub`)
* Utilizes string manipulation functions `str.split()`, `str.join()`, and `str.lower`

**Notable Dependencies and Requirements:**

* Uses the `re` module (regular expression) and `string` module (punctuation set)
* No external dependencies or interfaces


key: chroma:document
string_value:
CODE:

def f1_score(prediction, ground_truth):
    prediction_tokens = normalize_answer(prediction).split()
    ground_truth_tokens = normalize_answer(ground_truth).split()
    common = Counter(prediction_tokens) & Counter(ground_truth_tokens)
    num_same = sum(common.values())
    if num_same == 0:
        return 0
    precision = 1.0 * num_same / len(prediction_tokens)
    recall = 1.0 * num_same / len(ground_truth_tokens)
    f1 = (2 * precision * recall) / (precision + recall)


SUMMARY:
**Concise Summary:**

### Function Name: `f1_score`

#### Purpose:
Calculates the F1 score of predictions against ground truth labels.

#### Key Details:

* **Algorithm:** Precision, Recall, and F1 score calculation using common elements between prediction tokens and ground truth tokens.
* **Method Signature:**
	+ `prediction` (input): predicted token sequence
	+ `ground_truth` (input): true labeled token sequence
* **Interfaces/Classes:**
	+ `normalize_answer`: normalization function for answer strings (not shown in this code snippet)
	+ `Counter`: collection class for counting elements in sequences (from `collections` module)
* **External Dependencies:**
	- `Python collections` module (for `Counter`)

#### Primary Purpose:
 Computes the F1 score, a measure of both precision and recall, to evaluate the accuracy of predictions against ground truth labels.


key: chroma:document
string_value:
CODE:

def exact_match_score(prediction, ground_truth):


SUMMARY:
**Summary**

The `exact_match_score` function calculates the exact match score between two prediction and ground truth answers. It normalizes both answers using the `normalize_answer` function and returns `True` if they are identical, indicating an exact match, or `False` otherwise.

**Key Implementation Details**

* The normalization of the prediction and ground truth answers is performed by calling the `normalize_answer` function twice on each input.
* The main logic for calculating the exact match score is a simple equality check between the two normalized values.

**Notable Requirements**

* Requires the `normalize_answer` function to be defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def calculate_exact_match(output_lns: List[str], reference_lns: List[str]) -> Dict:
    assert len(output_lns) == len(reference_lns)
    em = 0
    for hypo, pred in zip(output_lns, reference_lns):
        em += exact_match_score(hypo, pred)
    if len(output_lns) > 0:
        em /= len(output_lns)


SUMMARY:
**Function Summary:**

`calculate_exact_match`: Calculates the exact match score between two lists of lines (`output_lns`, `reference_lns`) and returns a dictionary with the average precision.

**Main Purpose:** Computes the average weighted precision for identifying exact matches between two sequences of strings.

**Implementation Details:**

* Uses `zip` to iterate over corresponding elements in both input lists.
* Calls `exact_match_score` function to compute the score, which is not defined within this snippet.
* Divides the sum of scores by the length of the output list to obtain the average precision.
* Returns a dictionary with the calculated EM (Expected Match) value.

**Notable Dependencies:**

* None mentioned in this snippet; however, `exact_match_score` function is assumed to be available and defined elsewhere.


key: chroma:document
string_value:
CODE:

def is_rag_model(model_prefix):


SUMMARY:
**Function Summary**

`is_rag_model` checks if a given `model_prefix` starts with the prefix `"rag"`, indicating it's a RAG (Rapid Application Generation) model. 

* **Purpose**: Verify RAG model prefix
* **Implementation Details**: Simple string comparison using `startswith()`
* **Return Value**: Boolean result (`True`/`False`)


key: chroma:document
string_value:
CODE:

def set_extra_model_params(extra_params, hparams, config):
    equivalent_param = {p: p for p in extra_params}
    # T5 models don't have `dropout` param, they have `dropout_rate` instead
    equivalent_param["dropout"] = "dropout_rate"
    for p in extra_params:
        if getattr(hparams, p, None):
            if not hasattr(config, p) and not hasattr(config, equivalent_param[p]):
                logger.info("config doesn't have a `{}` attribute".format(p))
                delattr(hparams, p)
                continue
            set_p = p if hasattr(config, p) else equivalent_param[p]
            setattr(config, set_p, getattr(hparams, p))
            delattr(hparams, p)


SUMMARY:
**Summary**

The `set_extra_model_params` function merges extra parameters from `extra_params` into `hparams` and updates the `config` dictionary accordingly. It handles differences in parameter names between T5 models and provides logging and attribute management.

**Key Details**

* Merges `extra_params` into `hparams` using a temporary mapping.
* Replaces `dropout` with `dropout_rate` for T5 models.
* Iterates over `param`s, updating `config` if missing attributes exist in either `hparams` or the equivalent mapping.
* Updates `config` attributes with values from `hparams`, and removes excess parameters from `hparams`.

**Notable Dependencies**

* `logger`: used for logging information about missing configuration attributes.
* External dependencies:
	+ Related classes/functions not explicitly listed, assume integration through standard libraries or package imports.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/rag-end2end-retriever/finetune_rag.py


key: chroma:document
string_value:
CODE:

class AttrDict(dict):
    def __init__(self, *args, **kwargs):
        super(AttrDict, self).__init__(*args, **kwargs)


SUMMARY:
**Class Summary:**

The `AttrDict` class is a custom dictionary implementation that allows attribute access on its instances. 

### Purpose and Functionality:

It extends the default Python dictionary (`dict`) by binding its instance to itself through the single underscore prefix (`self.__dict__`), enabling access to attributes like any other object in the instance's namespace.

### Key Implementation Details:

* The `__init__` method initializes the dictionary and sets up attribute access.
* The `super()` call ensures proper inheritance from the base `_dict` class.

### Notable Dependencies or Requirements:

None explicitly required; inherits from Python's built-in `dict`. 

### Interface/Methods:

* `__init__(self, *args, **kwargs)`: Initializes an `AttrDict` instance.
* `self.__dict__`: The bound dictionary attribute.


key: chroma:document
string_value:
CODE:

class GenerativeQAModule(BaseTransformer):
    mode = "generative_qa"
    loss_names = ["loss"]
    metric_names = ["em"]
    val_metric = "em"

    def __init__(self, hparams, **kwargs):
        # when loading from a pytorch lightning checkpoint, hparams are passed as dict
        if isinstance(hparams, dict):
            hparams = AttrDict(hparams)
        if hparams.model_type == "rag_sequence":
            self.model_class = RagSequenceForGeneration
        elif hparams.model_type == "rag_token":
            self.model_class = RagTokenForGeneration
        elif hparams.model_type == "bart":
            self.model_class = BartForConditionalGeneration
        else:
            self.model_class = T5ForConditionalGeneration
        self.is_rag_model = is_rag_model(hparams.model_type)

        config_class = RagConfig if self.is_rag_model else AutoConfig
        config = config_class.from_pretrained(hparams.model_name_or_path)

        # set retriever parameters
        config.index_name = hparams.index_name or config.index_name
        config.passages_path = hparams.passages_path or config.passages_path
        config.index_path = hparams.index_path or config.index_path
        config.use_dummy_dataset = hparams.use_dummy_dataset

        # set extra_model_params for generator configs and load_model
        extra_model_params = ("encoder_layerdrop", "decoder_layerdrop", "attention_dropout", "dropout")
        if self.is_rag_model:
            if hparams.prefix is not None:
                config.generator.prefix = hparams.prefix
            config.label_smoothing = hparams.label_smoothing
            hparams, config.generator = set_extra_model_params(extra_model_params, hparams, config.generator)
            if hparams.distributed_retriever == "ray":
                # The Ray retriever needs the handles to the retriever actors.
                retriever = RagRayDistributedRetriever.from_pretrained(
                    hparams.model_name_or_path, hparams.actor_handles, config=config
                )

                if hparams.end2end:
                    ctx_encoder_tokenizer = DPRContextEncoderTokenizerFast.from_pretrained(
                        "facebook/dpr-ctx_encoder-multiset-base"
                    )
                    retriever.set_ctx_encoder_tokenizer(ctx_encoder_tokenizer)
            else:
                logger.info("please use RAY as the distributed retrieval method")

            model = self.model_class.from_pretrained(hparams.model_name_or_path, config=config, retriever=retriever)
            if hparams.end2end:
                ctx_encoder = DPRContextEncoder.from_pretrained(hparams.context_encoder_name)
                model.set_context_encoder_for_training(ctx_encoder)
            prefix = config.question_encoder.prefix
        else:
            if hparams.prefix is not None:
                config.prefix = hparams.prefix
            hparams, config = set_extra_model_params(extra_model_params, hparams, config)
            model = self.model_class.from_pretrained(hparams.model_name_or_path, config=config)
            prefix = config.prefix

        tokenizer = (
            RagTokenizer.from_pretrained(hparams.model_name_or_path)
            if self.is_rag_model
            else AutoTokenizer.from_pretrained(hparams.model_name_or_path)
        )

        self.config_dpr = DPRConfig.from_pretrained(hparams.context_encoder_name)
        self.custom_config = hparams
        self.context_tokenizer = DPRContextEncoderTokenizerFast.from_pretrained(hparams.context_encoder_name)

        super().__init__(hparams, config=config, tokenizer=tokenizer, model=model)

        save_git_info(self.hparams.output_dir)
        self.output_dir = Path(self.hparams.output_dir)
        self.dpr_ctx_check_dir = str(Path(self.hparams.output_dir)) + "/dpr_ctx_checkpoint"
        self.metrics_save_path = Path(self.output_dir) / "metrics.json"
        self.hparams_save_path = Path(self.output_dir) / "hparams.pkl"
        pickle_save(self.hparams, self.hparams_save_path)
        self.step_count = 0
        self.metrics = defaultdict(list)

        self.dataset_kwargs: dict = {
            "data_dir": self.hparams.data_dir,
            "max_source_length": self.hparams.max_source_length,
            "prefix": prefix or "",
        }
        n_observations_per_split = {
            "train": self.hparams.n_train,
            "val": self.hparams.n_val,
            "test": self.hparams.n_test,
        }
        self.n_obs = {k: v if v >= 0 else None for k, v in n_observations_per_split.items()}
        self.target_lens = {
            "train": self.hparams.max_target_length,
            "val": self.hparams.val_max_target_length,
            "test": self.hparams.test_max_target_length,
        }
        assert self.target_lens["train"] <= self.target_lens["val"], f"target_lens: {self.target_lens}"
        assert self.target_lens["train"] <= self.target_lens["test"], f"target_lens: {self.target_lens}"

        self.hparams.git_sha = get_git_info()["repo_sha"]
        self.num_workers = hparams.num_workers
        self.distributed_port = self.hparams.distributed_port

        # For single GPU training, init_ddp_connection is not called.
        # So we need to initialize the retrievers here.
        if hparams.gpus <= 1:
            if hparams.distributed_retriever == "ray":
                self.model.retriever.init_retrieval()
            else:
                logger.info("please use RAY as the distributed retrieval method")

        self.distributed_retriever = hparams.distributed_retriever

    def forward(self, input_ids, **kwargs):
        return self.model(input_ids, **kwargs)

    def ids_to_clean_text(self, generated_ids: List[int]):
        gen_text = self.tokenizer.batch_decode(
            generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
        )
        return lmap(str.strip, gen_text)

    def _step(self, batch: dict) -> Tuple:
        source_ids, source_mask, target_ids = batch["input_ids"], batch["attention_mask"], batch["decoder_input_ids"]

        rag_kwargs = {}
        if isinstance(self.model, T5ForConditionalGeneration):
            decoder_input_ids = self.model._shift_right(target_ids)
            lm_labels = target_ids
        elif isinstance(self.model, BartForConditionalGeneration):
            decoder_input_ids = target_ids[:, :-1].contiguous()
            lm_labels = target_ids[:, 1:].clone()
        else:
            assert self.is_rag_model
            generator = self.model.rag.generator
            if isinstance(generator, T5ForConditionalGeneration):
                decoder_start_token_id = generator.config.decoder_start_token_id
                decoder_input_ids = (
                    torch.cat(
                        [torch.tensor([[decoder_start_token_id]] * target_ids.shape[0]).to(target_ids), target_ids],
                        dim=1,
                    )
                    if target_ids.shape[0] < self.target_lens["train"]
                    else generator._shift_right(target_ids)
                )
            elif isinstance(generator, BartForConditionalGeneration):
                decoder_input_ids = target_ids
            lm_labels = decoder_input_ids
            rag_kwargs["reduce_loss"] = True

        assert decoder_input_ids is not None

        outputs = self(
            source_ids,
            attention_mask=source_mask,
            decoder_input_ids=decoder_input_ids,
            use_cache=False,
            labels=lm_labels,
            **rag_kwargs,
        )
        loss = outputs["loss"]
        return (loss,)

    @property
    def pad(self) -> int:
        raise NotImplementedError("pad not implemented")

    def training_step(self, batch, batch_idx) -> Dict:
        global isEmUpdateBusy  # use to check whether the entire embedding update process is finished or not
        global isAddIndexBusy  # use to check whether the entire indexing process  is finished or not
        global processes  # use to keep threads embedding update processes
        global threadHandle_index  # use to keep thread in embedding indexing processes

        if (self.trainer.global_rank == 0) and (self.custom_config.end2end):
            if (not batch_idx == 0) and (batch_idx % self.custom_config.indexing_freq == 0):
                free_gpu_list = []
                nvmlInit()
                deviceCount = nvmlDeviceGetCount()

                my_list = json.loads(self.custom_config.gpu_order)

                for i in range(deviceCount):
                    handle = nvmlDeviceGetHandleByIndex(i)
                    info = nvmlDeviceGetMemoryInfo(handle)

                    if info.used / 1e6 < 15:
                        position = my_list.index(i)
                        free_gpu_list.append("cuda:" + str(position))

                if len(free_gpu_list) >= self.custom_config.index_gpus:
                    has_free_gpus = True

                else:
                    has_free_gpus = False

                if (not isEmUpdateBusy) and has_free_gpus:
                    model_copy = type(self.model.rag.ctx_encoder)(
                        self.config_dpr
                    )  # get a new instance  #this will be load in the CPU
                    model_copy.load_state_dict(self.model.rag.ctx_encoder.state_dict())  # copy weights

                    processes = []

                    if len(free_gpu_list) > self.custom_config.index_gpus:
                        cuda_devices = random.sample(free_gpu_list, self.custom_config.index_gpus)
                    else:
                        cuda_devices = free_gpu_list

                    num_processes = len(cuda_devices)

                    for rank in range(num_processes):
                        logger.info("Iniitializing  embedding calculation process rank{}".format(rank))
                        device = cuda_devices[rank]
                        p = multiprocessing.Process(
                            target=embed_update,
                            args=(
                                copy.deepcopy(model_copy),
                                num_processes,
                                device,
                                rank,
                                self.custom_config.shard_dir,
                                self.custom_config.csv_path,
                            ),
                        )
                        processes.append(p)

                    for p in processes:
                        p.start()

                    isEmUpdateBusy = True

            if isEmUpdateBusy and (not isAddIndexBusy):
                index_process_list = [processes[k].is_alive() for k in range(self.custom_config.index_gpus)]
                if (
                    sum(index_process_list) == 0
                ):  # If entire list is false, we can say all embedding calculation process has finished
                    logger.info("Start adding the index")
                    threadHandle_index = multiprocessing.Process(
                        target=add_index,
                        args=(
                            self.custom_config.shard_dir,
                            self.config.index_path,
                        ),
                    )
                    threadHandle_index.start()
                    isAddIndexBusy = True

            # check when index building has started
            if isAddIndexBusy:
                # check still the index_building process is happening
                if not threadHandle_index.is_alive():
                    logger.info("Merging the dataset shards")
                    saved_dataset_shards = []

                    for address in glob(str(self.custom_config.shard_dir) + "/*/"):
                        saved_dataset_shards.append(load_from_disk(address))

                    concat = concatenate_datasets(saved_dataset_shards)
                    concat.save_to_disk(self.config.passages_path)  # here we update the main passage file on the disk
                    logger.info("done updating the dataset")

                    # To Do (@Aaron) : Useful in the future dynamic memory implementation.
                    # if you load the index from the disk make sure to update the index file here, otherwise it is ok to update the index file from the worker.
                    # logger.info("then updating the index")
                    # shutil.copy(self.custom_config.temp_index, self.config.idex_path)

                    logger.info("Loading new passages and iniitalzing new index")
                    self.trainer.model.module.module.model.rag.retriever.re_load()
                    self.trainer.model.module.module.model.rag.retriever.init_retrieval()

                    isEmUpdateBusy = False
                    isAddIndexBusy = False
        self.trainer.strategy.barrier("barrier")

        loss_tensors = self._step(batch)

        logs = dict(zip(self.loss_names, loss_tensors))
        # tokens per batch
        tgt_pad_token_id = (
            self.tokenizer.generator.pad_token_id
            if isinstance(self.tokenizer, RagTokenizer)
            else self.tokenizer.pad_token_id
        )
        src_pad_token_id = (
            self.tokenizer.question_encoder.pad_token_id
            if isinstance(self.tokenizer, RagTokenizer)
            else self.tokenizer.pad_token_id
        )
        logs["tpb"] = (
            batch["input_ids"].ne(src_pad_token_id).sum() + batch["decoder_input_ids"].ne(tgt_pad_token_id).sum()
        )
        self.log("loss", loss_tensors[0])
        return loss_tensors[0]

    def validation_step(self, batch, batch_idx) -> Dict:
        return self._generative_step(batch)

    def validation_epoch_end(self, outputs, prefix="val") -> Dict:
        self.step_count += 1
        losses = {k: torch.stack([x[k] for x in outputs]).mean() for k in self.loss_names}
        loss = losses["loss"]
        gen_metrics = {
            k: np.array([x[k] for x in outputs]).mean() for k in self.metric_names + ["gen_time", "gen_len"]
        }
        metrics_tensor: torch.FloatTensor = torch.tensor(gen_metrics[self.val_metric]).type_as(loss)
        gen_metrics.update({k: v.item() for k, v in losses.items()})

        # fix for https://github.com/PyTorchLightning/pytorch-lightning/issues/2424
        if dist.is_initialized():
            dist.all_reduce(metrics_tensor, op=dist.ReduceOp.SUM)
            metrics_tensor = metrics_tensor / dist.get_world_size()
            gen_metrics.update({self.val_metric: metrics_tensor.item()})

        losses.update(gen_metrics)
        metrics = {f"{prefix}_avg_{k}": x for k, x in losses.items()}
        metrics["step_count"] = self.step_count
        self.save_metrics(metrics, prefix)  # writes to self.metrics_save_path

        log_dict = {
            f"{prefix}_avg_em": metrics[f"{prefix}_avg_em"],
            "step_count": metrics["step_count"],
            f"{prefix}_avg_loss": metrics[f"{prefix}_avg_loss"],
            f"{prefix}_loss": loss,
            f"{prefix}_em": metrics_tensor,
        }
        self.log_dict(log_dict)

    def save_metrics(self, latest_metrics, type_path) -> None:
        self.metrics[type_path].append(latest_metrics)
        save_json(self.metrics, self.metrics_save_path)

    def calc_generative_metrics(self, preds, target) -> Dict:
        return calculate_exact_match(preds, target)

    def _generative_step(self, batch: dict) -> dict:
        start_time = time.time()
        batch = BatchEncoding(batch).to(device=self.model.device)
        generated_ids = self.model.generate(
            batch["input_ids"],
            attention_mask=batch["attention_mask"],
            do_deduplication=False,  # rag specific parameter
            use_cache=True,
            min_length=1,
            max_length=self.target_lens["val"],
        )
        gen_time = (time.time() - start_time) / batch["input_ids"].shape[0]
        preds: List[str] = self.ids_to_clean_text(generated_ids)
        target: List[str] = self.ids_to_clean_text(batch["decoder_input_ids"])
        # print(preds,target)
        loss_tensors = self._step(batch)
        base_metrics = dict(zip(self.loss_names, loss_tensors))
        gen_metrics: Dict = self.calc_generative_metrics(preds, target)

        summ_len = np.mean(lmap(len, generated_ids))
        base_metrics.update(gen_time=gen_time, gen_len=summ_len, preds=preds, target=target, **gen_metrics)
        return base_metrics

    def test_step(self, batch, batch_idx):
        return self._generative_step(batch)

    def test_epoch_end(self, outputs):
        return self.validation_epoch_end(outputs, prefix="test")

    def get_dataset(self, type_path) -> Seq2SeqDataset:
        n_obs = self.n_obs[type_path]
        max_target_length = self.target_lens[type_path]
        dataset = Seq2SeqDataset(
            self.tokenizer,
            type_path=type_path,
            n_obs=n_obs,
            max_target_length=max_target_length,
            **self.dataset_kwargs,
        )
        return dataset

    def get_dataloader(self, type_path: str, batch_size: int, shuffle: bool = False) -> DataLoader:
        dataset = self.get_dataset(type_path)

        dataloader = DataLoader(
            dataset,
            batch_size=batch_size,
            collate_fn=dataset.collate_fn,
            shuffle=shuffle,
            num_workers=self.num_workers,
        )
        return dataloader

    def train_dataloader(self) -> DataLoader:
        dataloader = self.get_dataloader("train", batch_size=self.hparams.train_batch_size, shuffle=True)
        return dataloader

    def val_dataloader(self) -> DataLoader:
        return self.get_dataloader("val", batch_size=self.hparams.eval_batch_size)

    def test_dataloader(self) -> DataLoader:
        return self.get_dataloader("test", batch_size=self.hparams.eval_batch_size)

    @pl.utilities.rank_zero_only
    def on_save_checkpoint(self, checkpoint: Dict[str, Any]) -> None:
        save_path = self.output_dir.joinpath("checkpoint{}".format(self.step_count))
        self.model.config.save_step = self.step_count
        # self.model.save_pretrained(save_path)
        self.tokenizer.save_pretrained(save_path)

        if self.custom_config.end2end:
            modified_state_dict = self.model.state_dict()
            for key in self.model.state_dict().keys():
                if key.split(".")[1] == "ctx_encoder":
                    del modified_state_dict[key]
            self.model.save_pretrained(save_directory=save_path, state_dict=modified_state_dict)

            save_path_dpr = os.path.join(self.dpr_ctx_check_dir, "checkpoint{}".format(self.step_count))
            self.model.rag.ctx_encoder.save_pretrained(save_path_dpr)
            self.context_tokenizer.save_pretrained(save_path_dpr)

    @staticmethod
    def add_model_specific_args(parser, root_dir):
        BaseTransformer.add_model_specific_args(parser, root_dir)
        add_generic_args(parser, root_dir)
        parser.add_argument(
            "--max_source_length",
            default=128,
            type=int,
            help=(
                "The maximum total input sequence length after tokenization. Sequences longer "
                "than this will be truncated, sequences shorter will be padded."
            ),
        )
        parser.add_argument(
            "--max_target_length",
            default=25,
            type=int,
            help=(
                "The maximum total input sequence length after tokenization. Sequences longer "
                "than this will be truncated, sequences shorter will be padded."
            ),
        )
        parser.add_argument(
            "--val_max_target_length",
            default=25,
            type=int,
            help=(
                "The maximum total input sequence length after tokenization. Sequences longer "
                "than this will be truncated, sequences shorter will be padded."
            ),
        )
        parser.add_argument(
            "--test_max_target_length",
            default=25,
            type=int,
            help=(
                "The maximum total input sequence length after tokenization. Sequences longer "
                "than this will be truncated, sequences shorter will be padded."
            ),
        )
        parser.add_argument("--logger_name", type=str, choices=["default", "wandb", "wandb_shared"], default="default")
        parser.add_argument("--n_train", type=int, default=-1, required=False, help="# examples. -1 means use all.")
        parser.add_argument("--n_val", type=int, default=-1, required=False, help="# examples. -1 means use all.")
        parser.add_argument("--n_test", type=int, default=-1, required=False, help="# examples. -1 means use all.")
        parser.add_argument("--label_smoothing", type=float, default=0.0, required=False)
        parser.add_argument(
            "--prefix",
            type=str,
            default=None,
            help="Prefix added at the beginning of each text, typically used with T5-based models.",
        )
        parser.add_argument(
            "--early_stopping_patience",
            type=int,
            default=-1,
            required=False,
            help=(
                "-1 means never early stop. early_stopping_patience is measured in validation checks, not epochs. So"
                " val_check_interval will effect it."
            ),
        )
        parser.add_argument(
            "--distributed-port", type=int, default=-1, required=False, help="Port number for distributed training."
        )
        parser.add_argument(
            "--model_type",
            choices=["rag_sequence", "rag_token", "bart", "t5"],
            type=str,
            help=(
                "RAG model type: sequence or token, if none specified, the type is inferred from the"
                " model_name_or_path"
            ),
        )
        parser.add_argument(
            "--context_encoder_name",
            default="facebook/dpr-ctx_encoder-multiset-base",
            type=str,
            help="Name of the pre-trained context encoder checkpoint from the DPR",
        )
        parser.add_argument(
            "--csv_path",
            default=str(Path(__file__).parent / "test_run" / "dummy-kb" / "my_knowledge_dataset.csv"),
            type=str,
            help="path of the raw KB csv",
        )
        parser.add_argument("--end2end", action="store_true", help="whether to train the system end2end or not")
        parser.add_argument("--index_gpus", type=int, help="how many GPUs used in re-encoding process")
        parser.add_argument(
            "--shard_dir",
            type=str,
            default=str(Path(__file__).parent / "test_run" / "kb-shards"),
            help="directory used to keep temporary shards during the re-encode process",
        )

        parser.add_argument(
            "--gpu_order",
            type=str,
            help=(
                "order of the GPU used during the fine-tuning.  Used to finding free GPUs during the re-encode"
                " process. I do not have many GPUs :)"
            ),
        )

        parser.add_argument("--indexing_freq", type=int, help="frequency of re-encode process")
        return parser

    @staticmethod
    def add_retriever_specific_args(parser):
        parser.add_argument(
            "--index_name",
            type=str,
            default=None,
            help=(
                "Name of the index to use: 'hf' for a canonical dataset from the datasets library (default), 'custom'"
                " for a local index, or 'legacy' for the orignal one)"
            ),
        )
        parser.add_argument(
            "--passages_path",
            type=str,
            default=str(Path(__file__).parent / "test_run" / "dummy-kb" / "my_knowledge_dataset"),
            help=(
                "Path to the dataset of passages for custom index. More info about custom indexes in the RagRetriever"
                " documentation as well as in `examples/rag/use_own_knowledge_dataset.py`"
            ),
        )
        parser.add_argument(
            "--index_path",
            type=str,
            default=str(Path(__file__).parent / "test_run" / "dummy-kb" / "my_knowledge_dataset_hnsw_index.faiss"),
            help=(
                "Path to the faiss index for custom index. More info about custom indexes in the RagRetriever"
                " documentation as well as in `examples/rag/use_own_knowledge_dataset.py`"
            ),
        )
        parser.add_argument(
            "--distributed_retriever",
            choices=["ray", "pytorch"],
            type=str,
            default="ray",
            help=(
                "What implementation to use for distributed retriever? If "
                "pytorch is selected, the index is loaded on training "
                "worker 0, and torch.distributed is used to handle "
                "communication between training worker 0, and the other "
                "training workers. If ray is selected, the Ray library is "
                "used to create load the index on separate processes, "
                "and Ray handles the communication between the training "
                "workers and the retrieval actors."
            ),
        )
        parser.add_argument(
            "--use_dummy_dataset",
            type=bool,
            default=False,
            help=(
                "Whether to use the dummy version of the dataset index. More info about custom indexes in the"
                " RagRetriever documentation as well as in `examples/rag/use_own_knowledge_dataset.py`"
            ),
        )
        return parser

    @staticmethod
    def add_ray_specific_args(parser):
        # Ray cluster address.
        parser.add_argument(
            "--ray-address",
            default="auto",
            type=str,
            help=(
                "The address of the Ray cluster to connect to. If not "
                "specified, Ray will attempt to automatically detect the "
                "cluster. Has no effect if pytorch is used as the distributed "
                "retriever."
            ),
        )
        parser.add_argument(
            "--num_retrieval_workers",
            type=int,
            default=1,
            help=(
                "The number of retrieval actors to use when Ray is selected "
                "for the distributed retriever. Has no effect when "
                "distributed_retriever is set to pytorch."
            ),
        )


SUMMARY:
This code snippet appears to be part of a PyTorch model implementation, specifically for a text retrieval or question-answering system called RAG (Rearranged Architecture Generalist). The code is written in Python and includes various components such as data loading, model architecture, training loop, and optimization.

Here's a brief summary:

*   **Model Architecture**: The model has two main components: the RAG model and the context encoder model. The RAG model uses multiple attention mechanisms to process sequences, while the context encoder model is used for contextualized embeddings.
*   **Data Loading**: The code loads data from various text sources, including knowledge base files, passage files, and other supporting datasets.
*   **Training Loop**: The training loop involves an iterative process of optimizing the RAG model's parameters on the available dataset. The optimizer is typically a stochastic gradient descent (SGD) variant with regularization terms for controlling overfitting.
*   **Miscellaneous Parameters**: There are several other important parameters that control various aspects of the system, such as the number of GPUs to use in training (`index_gpus`), the indexing frequency of re-encoding processes (`indexing_freq`), and the model's type, including whether it should be used end-to-end or just for retriever tasks.


key: chroma:document
string_value:
CODE:

def main(args=None, model=None) -> GenerativeQAModule:
    parser = argparse.ArgumentParser()
    parser = pl.Trainer.add_argparse_args(parser)
    parser = GenerativeQAModule.add_model_specific_args(parser, os.getcwd())
    parser = GenerativeQAModule.add_retriever_specific_args(parser)
    args = args or parser.parse_args()

    Path(args.output_dir).mkdir(exist_ok=True)
    Path(args.output_dir + "/dpr_ctx_checkpoint").mkdir(
        exist_ok=True
    )  # save dpr_context encoder seprately for the future use
    print(args.shard_dir)
    if os.path.exists(args.shard_dir):  # we do not need previous kb shards used in dataset re-conding and re-indexing
        shutil.rmtree(args.shard_dir)
    Path(args.shard_dir).mkdir(exist_ok=True)

    if os.path.exists(
        args.cache_dir
    ):  # we do not need previous cache files used in dataset re-conding and re-indexing
        shutil.rmtree(args.cache_dir)
    Path(args.cache_dir).mkdir(exist_ok=True)

    named_actors = []
    if args.distributed_retriever == "ray" and args.gpus > 1:
        if not is_ray_available():
            raise RuntimeError("Please install Ray to use the Ray distributed retriever.")
        # Connect to an existing Ray cluster.
        try:
            ray.init(address=args.ray_address, namespace="rag")
        except (ConnectionError, ValueError):
            logger.warning(
                "Connection to Ray cluster failed. Make sure a Ray "
                "cluster is running by either using Ray's cluster "
                "launcher (`ray up`) or by manually starting Ray on "
                "each node via `ray start --head` for the head node "
                "and `ray start --address='<ip address>:6379'` for "
                "additional nodes. See "
                "https://docs.ray.io/en/master/cluster/index.html "
                "for more info."
            )
            raise

        # Create Ray actors only for rank 0.
        if ("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0) and (
            "NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0
        ):
            remote_cls = ray.remote(RayRetriever)
            named_actors = [
                remote_cls.options(name="retrieval_worker_{}".format(i)).remote()
                for i in range(args.num_retrieval_workers)
            ]
        else:
            logger.info(
                "Getting named actors for NODE_RANK {}, LOCAL_RANK {}".format(
                    os.environ["NODE_RANK"], os.environ["LOCAL_RANK"]
                )
            )
            named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
    args.actor_handles = named_actors
    assert args.actor_handles == named_actors

    if model is None:
        model: GenerativeQAModule = GenerativeQAModule(args)

    dataset = Path(args.data_dir).name
    if (
        args.logger_name == "default"
        or args.fast_dev_run
        or str(args.output_dir).startswith("/tmp")
        or str(args.output_dir).startswith("/var")
    ):
        training_logger = True  # don't pollute wandb logs unnecessarily
    elif args.logger_name == "wandb":
        from pytorch_lightning.loggers import WandbLogger

        project = os.environ.get("WANDB_PROJECT", dataset)
        training_logger = WandbLogger(name=model.output_dir.name, project=project)

    elif args.logger_name == "wandb_shared":
        from pytorch_lightning.loggers import WandbLogger

        training_logger = WandbLogger(name=model.output_dir.name, project=f"hf_{dataset}")

    es_callback = (
        get_early_stopping_callback(model.val_metric, args.early_stopping_patience)
        if args.early_stopping_patience >= 0
        else False
    )

    trainer: pl.Trainer = generic_train(
        model,
        args,
        logging_callback=Seq2SeqLoggingCallback(),
        checkpoint_callback=get_checkpoint_callback(args.output_dir, model.val_metric),
        early_stopping_callback=es_callback,
        logger=training_logger,
        profiler=pl.profiler.AdvancedProfiler() if args.profile else None,
    )

    pickle_save(model.hparams, model.output_dir / "hparams.pkl")
    if not args.do_predict:
        return model

    # test() without a model tests using the best checkpoint automatically
    trainer.test()


SUMMARY:
**Function Summary:**

`main` is the entry point for a Generative QA Model training and evaluation pipeline.

**Primary Purpose:**
The function initializes and configures the pipeline, including:

* Setting up command-line arguments and parsing
* Creating directories for output, shard data, and cache files
* Initializing Ray cluster (if using distributed retriever)
* Loading and setting up the model

**Key Implementation Details:**

* uses `argparse` for command-line argument parsing
* creates a file manager to handle directory creation and cleanup
* configures and initializes the Ray cluster (if using distributed retriever)
* sets up logging and early stopping callback
* enables profiling (if requested)

**Interfaces and Dependencies:**

* `GenerativeQAModule`: model class
* `argparse`: for command-line argument parsing
* `Ray`: for distributed computing
* `Shutil`: for directory cleanup
* `PyTorch Lightning`: for logging, training, and testing

Note that the function returns either the trained model or a message indicating testing (if `do_predict` is False).


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/rag-end2end-retriever/eval_rag.py


key: chroma:document
string_value:
CODE:

def infer_model_type(model_name_or_path):
    if "token" in model_name_or_path:
        return "rag_token"
    if "sequence" in model_name_or_path:
        return "rag_sequence"
    if "bart" in model_name_or_path:
        return "bart"


SUMMARY:
**Function Summary: `infer_model_type`**

Infer the type of a given model based on its name or path.

**Key Details:**

* **Primary Purpose:** Identify the model type by inspecting the `model_name_or_path`.
* **Implementation:
	+ Token-based models are identified as "rag_token".
	+ Sequence-based models are identified as "rag_sequence".
	+ Models containing "bart" in their name are identified as "bart".
* **Return Value:**
	+ Returns the model type as a string (`"rag_token"`, `"rag_sequence"`, or `"bart"`).
	+ Returns `None` if no matching model type can be inferred.

**Notable Dependencies:** None


key: chroma:document
string_value:
CODE:

def metric_max_over_ground_truths(metric_fn, prediction, ground_truths):


SUMMARY:
**Summary:**

The `metric_max_over_ground_truths` function calculates the maximum score metric across various ground truth labels. It takes:

*   A scoring function (`metric_fn`)
*   A prediction
*   A collection of corresponding ground truths (`ground_truths`)

The function returns the globally maximum score by iteratively applying the `metric_fn` to each ground truth label and its associated prediction, using the built-in `max` function.

**Key Details:**

*   Uses a generator expression to iterate over ground truths with their predictions.
*   Applies the `metric_fn` scoring function to individual predictions and ground truths.
*   Returns the global maximum score metric value.

**Example Usage:**
```
from sklearn.metrics import accuracy_score
ground_truths = [1, 0, 1]
prediction = [1, 1, 0]
max_accuracy = metric_max_over_ground_truths(accuracy_score, prediction, ground_truths)
print(max_accuracy)  # Output: 1.0
```


key: chroma:document
string_value:
CODE:

def get_scores(args, preds_path, gold_data_path):
    hypos = [line.strip() for line in open(preds_path, "r").readlines()]
    answers = []

    if args.gold_data_mode == "qa":
        data = pd.read_csv(gold_data_path, sep="\t", header=None)
        for answer_list in data[1]:
            ground_truths = ast.literal_eval(answer_list)
            answers.append(ground_truths)
    else:
        references = [line.strip() for line in open(gold_data_path, "r").readlines()]
        answers = [[reference] for reference in references]

    f1 = em = total = 0
    for prediction, ground_truths in zip(hypos, answers):
        total += 1
        em += metric_max_over_ground_truths(exact_match_score, prediction, ground_truths)
        f1 += metric_max_over_ground_truths(f1_score, prediction, ground_truths)

    em = 100.0 * em / total
    f1 = 100.0 * f1 / total

    logger.info(f"F1: {f1:.2f}")


SUMMARY:
**Function Summary**

* `get_scores`: computes and logs F1 score and exact match (EM) metrics for model predictions against gold standard data.
	+ Takes in:
		- `args`: command-line arguments
		- `preds_path`: path to file containing model predictions
		- `gold_data_path`: path to file containing gold standard answers
	+ Returns no value, logging metrics instead

**Key Implementation Details**

* Reads gold standard data from CSV or file, depending on `args.gold_data_mode`
* Computes F1 score and exact match (EM) using custom `metric_max_over_ground_truths` functions
* Calculates and logs F1 score and EM as percentages of total predictions
	+ F1: 100.0 * f1 / total
	+ EM: 100.0 * em / total

**Notable Dependencies/Requirements**

* Pandas (`pd`) for data reading
* NumPy (not explicitly used in this code snippet, but likely required by `ast` and `metric_max_over_ground_truths`)
* External dependency on `ast.literal_eval` for parsing gold standard answers
* Custom implementation of `metric_max_over_ground_truths` functions using unseen algorithm/technique


key: chroma:document
string_value:
CODE:

def get_precision_at_k(args, preds_path, gold_data_path):
    k = args.k
    hypos = [line.strip() for line in open(preds_path, "r").readlines()]
    references = [line.strip() for line in open(gold_data_path, "r").readlines()]

    em = total = 0
    for hypo, reference in zip(hypos, references):
        hypo_provenance = set(hypo.split("\t")[:k])
        ref_provenance = set(reference.split("\t"))
        total += 1
        em += len(hypo_provenance & ref_provenance) / k

    em = 100.0 * em / total


SUMMARY:
**Function Summary:** `get_precision_at_k`

**Purpose and Functionality:**

This function calculates the precision at k (P@k) for a given set of predicted values (`preds_path`) and gold data references (`gold_data_path`). It measures the percentage of true positive elements among the top k predictions, using element-wise set similarity.

**Key Implementation Details:**

- Uses `zip` to iterate over corresponding predictions and references.
- Calculates precision at k by dividing the number of matching elements (using set intersection) by k.
- Normalizes precision to a 100.0 value to obtain a scalar metric.

**Notable Dependencies/Requirements:**

* `args.k`: The integer threshold for top predictions.
* `preds_path` and `gold_data_path`: File paths for predicted values and gold data, respectively.
* External dependency:
	+ `open`: A built-in file access function.
	+ `strip`: A string manipulation technique.


key: chroma:document
string_value:
CODE:

def evaluate_batch_retrieval(args, rag_model, questions):
    def strip_title(title):
        if title.startswith('"'):
            title = title[1:]
        if title.endswith('"'):
            title = title[:-1]
        return title

    retriever_input_ids = rag_model.retriever.question_encoder_tokenizer.batch_encode_plus(
        questions,
        return_tensors="pt",
        padding=True,
        truncation=True,
    )["input_ids"].to(args.device)

    question_enc_outputs = rag_model.rag.question_encoder(retriever_input_ids)
    question_enc_pool_output = question_enc_outputs[0]

    result = rag_model.retriever(
        retriever_input_ids,
        question_enc_pool_output.cpu().detach().to(torch.float32).numpy(),
        prefix=rag_model.rag.generator.config.prefix,
        n_docs=rag_model.config.n_docs,
        return_tensors="pt",
    )
    all_docs = rag_model.retriever.index.get_doc_dicts(result.doc_ids)
    provenance_strings = []
    for docs in all_docs:
        provenance = [strip_title(title) for title in docs["title"]]
        provenance_strings.append("\t".join(provenance))


SUMMARY:
**Concise Summary:**

`evaluate_batch_retrieval` evaluates batch retrieval of questions and returns provenance strings.

**Main Purpose:** This function uses the `rag_model` to retrieve relevant documents for a set of input questions, extracts the titles from these documents, and combines them into a list of provenance strings.

**Key Implementation Details:**

- Utilizes `batch_encode_plus` to encode input questions into token IDs.
- Passes encoded question IDs through `question_encoder` to obtain pooled query representation.
- Uses `retriever` to perform document retrieval with the pooled query representation.

**Notable Dependencies and Requirements:**

*   **External dependencies:** `rag_model`, `torch`
*   **Inherited or implemented interfaces:** None


key: chroma:document
string_value:
CODE:

def evaluate_batch_e2e(args, rag_model, questions):
    with torch.no_grad():
        inputs_dict = rag_model.retriever.question_encoder_tokenizer.batch_encode_plus(
            questions, return_tensors="pt", padding=True, truncation=True
        )

        input_ids = inputs_dict.input_ids.to(args.device)
        attention_mask = inputs_dict.attention_mask.to(args.device)
        outputs = rag_model.generate(  # rag_model overwrites generate
            input_ids,
            attention_mask=attention_mask,
            num_beams=args.num_beams,
            min_length=args.min_length,
            max_length=args.max_length,
            early_stopping=False,
            num_return_sequences=1,
            bad_words_ids=[[0, 0]],  # BART likes to repeat BOS tokens, dont allow it to generate more than one
        )
        answers = rag_model.retriever.generator_tokenizer.batch_decode(outputs, skip_special_tokens=True)

        if args.print_predictions:
            for q, a in zip(questions, answers):
                logger.info("Q: {} - A: {}".format(q, a))



SUMMARY:
**Summary**

`evaluate_batch_e2e` function evaluates batched end-to-end for multiple questions.

**Functionality**

- Takes questions as input and encodes them using `question_encoder_tokenizer`.
- Passes pre-encoded inputs to `generate` method of `rag_model`, retrieving a batch of summaries.
- Decode the generated outputs using `generator_tokenizer`.

**Key Implementation Details**

- Uses `torch.no_grad()` mode.
- Configures `rag_model.generate` method with:
  - `num_beams` and length parameters controlled by `args`.
  - Early stopping disabled (`early_stopping=False`).
  - BART's bad word handling with one allowed repetition of BOS token.

**Notable Dependencies/Requirements**

- References `rag_model.retriever.question_encoder_tokenizer` and `input_ids` / `attention_mask` attributes.
- Expects `logging` module via `logger`.
- Uses external dependency: PyTorch.


key: chroma:document
string_value:
CODE:

def get_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_type",
        choices=["rag_sequence", "rag_token", "bart"],
        type=str,
        help=(
            "RAG model type: rag_sequence, rag_token or bart, if none specified, the type is inferred from the"
            " model_name_or_path"
        ),
    )
    parser.add_argument(
        "--index_name",
        default=None,
        choices=["exact", "compressed", "legacy"],
        type=str,
        help="RAG model retriever type",
    )
    parser.add_argument(
        "--index_path",
        default=None,
        type=str,
        help="Path to the retrieval index",
    )
    parser.add_argument("--n_docs", default=5, type=int, help="Number of retrieved docs")
    parser.add_argument(
        "--model_name_or_path",
        default=None,
        type=str,
        required=True,
        help="Path to pretrained checkpoints or model identifier from huggingface.co/models",
    )
    parser.add_argument(
        "--eval_mode",
        choices=["e2e", "retrieval"],
        default="e2e",
        type=str,
        help=(
            "Evaluation mode, e2e calculates exact match and F1 of the downstream task, retrieval calculates"
            " precision@k."
        ),
    )
    parser.add_argument("--k", default=1, type=int, help="k for the precision@k calculation")
    parser.add_argument(
        "--evaluation_set",
        default=None,
        type=str,
        required=True,
        help="Path to a file containing evaluation samples",
    )
    parser.add_argument(
        "--gold_data_path",
        default=None,
        type=str,
        required=True,
        help="Path to a tab-separated file with gold samples",
    )
    parser.add_argument(
        "--gold_data_mode",
        default="qa",
        type=str,
        choices=["qa", "ans"],
        help=(
            "Format of the gold data file"
            "qa - a single line in the following format: question [tab] answer_list"
            "ans - a single line of the gold file contains the expected answer string"
        ),
    )
    parser.add_argument(
        "--predictions_path",
        type=str,
        default="predictions.txt",
        help="Name of the predictions file, to be stored in the checkpoints directory",
    )
    parser.add_argument(
        "--eval_all_checkpoints",
        action="store_true",
        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
    )
    parser.add_argument(
        "--eval_batch_size",
        default=8,
        type=int,
        help="Batch size per GPU/CPU for evaluation.",
    )
    parser.add_argument(
        "--recalculate",
        help="Recalculate predictions even if the prediction file exists",
        action="store_true",
    )
    parser.add_argument(
        "--num_beams",
        default=4,
        type=int,
        help="Number of beams to be used when generating answers",
    )
    parser.add_argument("--min_length", default=1, type=int, help="Min length of the generated answers")
    parser.add_argument("--max_length", default=50, type=int, help="Max length of the generated answers")

    parser.add_argument(
        "--print_predictions",
        action="store_true",
        help="If True, prints predictions while evaluating.",
    )
    parser.add_argument(
        "--print_docs",
        action="store_true",
        help="If True, prints docs retried while generating.",
    )
    args = parser.parse_args()
    args.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")


SUMMARY:
**Function Summary:**

`get_args()` function creates and parses command-line arguments for the RAG (Rationale Guided Autoencoder) model. 
The main purpose is to facilitate configurable training, evaluation, and generation tasks.

**Key Implementation Details:**

- `argparse.ArgumentParser`: Utilizes `argparse` library for parsing command-line arguments.
- Argument validation using `choices`, `default`, `type`, and `help` attributes.
- Mandatory argument `model_name_or_path` ensures the path to a pretrained checkpoint or model identifier is provided.

**Notable Dependencies or Requirements:**

- `torch`: Imported implicitly as part of PyTorch's standard library. 
- `argparse`: Used for parsing command-line arguments.

**Interfaces and Inherited Dependences:**

- `argparse.ArgumentParser` inherits from ` argparse`, a built-in module in the Python Standard Library.
- Importing additional modules such as `torchvision`, `transformers`, etc., is indirectly implied through required and optional argument dependencies.


key: chroma:document
string_value:
CODE:

def main(args):
    model_kwargs = {}
    if args.model_type is None:
        args.model_type = infer_model_type(args.model_name_or_path)
        assert args.model_type is not None
    if args.model_type.startswith("rag"):
        model_class = RagTokenForGeneration if args.model_type == "rag_token" else RagSequenceForGeneration
        model_kwargs["n_docs"] = args.n_docs
        if args.index_name is not None:
            model_kwargs["index_name"] = args.index_name
        if args.index_path is not None:
            model_kwargs["index_path"] = args.index_path
    else:
        model_class = BartForConditionalGeneration

    checkpoints = (
        [f.path for f in os.scandir(args.model_name_or_path) if f.is_dir()]
        if args.eval_all_checkpoints
        else [args.model_name_or_path]
    )

    logger.info("Evaluate the following checkpoints: %s", checkpoints)

    score_fn = get_scores if args.eval_mode == "e2e" else get_precision_at_k
    evaluate_batch_fn = evaluate_batch_e2e if args.eval_mode == "e2e" else evaluate_batch_retrieval

    for checkpoint in checkpoints:
        if os.path.exists(args.predictions_path) and (not args.recalculate):
            logger.info("Calculating metrics based on an existing predictions file: {}".format(args.predictions_path))
            score_fn(args, args.predictions_path, args.gold_data_path)
            continue

        logger.info("***** Running evaluation for {} *****".format(checkpoint))
        logger.info("  Batch size = %d", args.eval_batch_size)
        logger.info("  Predictions will be stored under {}".format(args.predictions_path))

        if args.model_type.startswith("rag"):
            retriever = RagRetriever.from_pretrained(checkpoint, **model_kwargs)
            model = model_class.from_pretrained(checkpoint, retriever=retriever, **model_kwargs)
            model.retriever.init_retrieval()
        else:
            model = model_class.from_pretrained(checkpoint, **model_kwargs)
        model.to(args.device)

        with open(args.evaluation_set, "r") as eval_file, open(args.predictions_path, "w") as preds_file:
            questions = []
            for line in tqdm(eval_file):
                questions.append(line.strip())
                if len(questions) == args.eval_batch_size:
                    answers = evaluate_batch_fn(args, model, questions)
                    preds_file.write("\n".join(answers) + "\n")
                    preds_file.flush()
                    questions = []
            if len(questions) > 0:
                answers = evaluate_batch_fn(args, model, questions)
                preds_file.write("\n".join(answers))
                preds_file.flush()



SUMMARY:
**Function Summary**

`main` function evaluates the performance of a trained language model using various checkpoints. It supports two types of models: `RagTokenForGeneration` and `BartForConditionalGeneration`.

**Key Implementation Details**

* Uses `RagRetriever` and `model_class` from `transformers` library to load and evaluate the models.
* Evaluates a batch of questions for each checkpoint using `evaluate_batch_fn`, which calls either `get_scores` or `get_precision_at_k` depending on the evaluation mode.
* Writes predictions to a file specified by `args.predictions_path`.

**Notable Dependencies**

* `transformers`: library providing pre-trained models and their respective retrieval mechanisms.
* ` torch`: used for modeling and device execution.

**Primary Purpose**
Evaluate the performance of a trained language model using various checkpoints, with support for two types of models: Rag and Bart.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/rag-end2end-retriever/use_own_knowledge_dataset.py


key: chroma:document
string_value:
CODE:

def split_text(text: str, n=100, character=" ") -> List[str]:
    """Split the text every ``n``-th occurrence of ``character``"""
    text = text.split(character)


SUMMARY:
**Function Summary:**

`split_text` splits `text` into substrings of length `n`, where `character` defines the splitting delimiter.

**Implementation Details:**

* Splits `text` on `character` using `str.split()`.
* Returns a list of substrings, where each substring is:
	+ A join of `character` and adjacent segments from `i` to `i + n`.
	+ Joined with `strip()` for trimming.

**Notable Dependencies/Requirements:**

* `List[str]`: Returned type hint indicates a list of string values.
* `str.split()`, `str.join()`, `str.strip()`: Used library functions to implement substring extraction and joining.


key: chroma:document
string_value:
CODE:

def split_documents(documents: dict) -> dict:
    """Split documents into passages"""
    titles, texts = [], []
    for title, text in zip(documents["title"], documents["text"]):
        if text is not None:
            for passage in split_text(text):
                titles.append(title if title is not None else "")
                texts.append(passage)


SUMMARY:
**Summary**

The `split_documents` function splits a dictionary of documents into passages. It takes a document with a title and text as input, extracts passages from the text using the `split_text` function, and returns a new dictionary with split titles and texts.

**Key Implementation Details**

* Uses `zip` to iterate over both title and text fields in parallel.
* Extracts passages from text using `split_text(text)`.
* Handles cases where title is missing or None by appending an empty string.

**Dependencies/Requirements**

* `split_text` function (not shown in this code snippet)
* No external dependencies listed.


key: chroma:document
string_value:
CODE:

def embed(documents: dict, ctx_encoder: DPRContextEncoder, ctx_tokenizer: DPRContextEncoderTokenizerFast) -> dict:
    """Compute the DPR embeddings of document passages"""
    input_ids = ctx_tokenizer(
        documents["title"], documents["text"], truncation=True, padding="longest", return_tensors="pt"
    )["input_ids"]
    embeddings = ctx_encoder(input_ids.to(device=device), return_dict=True).pooler_output


SUMMARY:
**Summary**

Computes DPR embeddings for document passages using a context encoder and tokenizer.

**Key Implementation Details:**

* Utilizes `DPRContextEncoder` and `DPRContextEncoderTokenizerFast` to encode input documents into tensor embeddings.
* Transforms encoded embeddings with truncation, padding, and longest sequence length.
* Applies the `pooler_output` from the `DPRContextEncoder` model to obtain DPR embeddings.
* Returns a dictionary containing the computed DPR embeddings as NumPy arrays.

**Note:** The `device=device` parameter implies that the computation is being performed on a specified device (e.g., GPU).


key: chroma:document
string_value:
CODE:

def main(
    rag_example_args: "RagExampleArguments",
    processing_args: "ProcessingArguments",
    index_hnsw_args: "IndexHnswArguments",
):
    ######################################
    logger.info("Step 1 - Create the dataset")
    ######################################

    # The dataset needed for RAG must have three columns:
    # - title (string): title of the document
    # - text (string): text of a passage of the document
    # - embeddings (array of dimension d): DPR representation of the passage
    # Let's say you have documents in tab-separated csv files with columns "title" and "text"
    assert os.path.isfile(rag_example_args.csv_path), "Please provide a valid path to a csv file"

    # You can load a Dataset object this way
    dataset = load_dataset(
        "csv", data_files=[rag_example_args.csv_path], split="train", delimiter="\t", column_names=["title", "text"]
    )

    # More info about loading csv files in the documentation: https://huggingface.co/docs/datasets/loading_datasets?highlight=csv#csv-files

    # Then split the documents into passages of 100 words
    dataset = dataset.map(split_documents, batched=True, num_proc=processing_args.num_proc)

    # And compute the embeddings
    ctx_encoder = DPRContextEncoder.from_pretrained(rag_example_args.dpr_ctx_encoder_model_name).to(device=device)
    ctx_tokenizer = DPRContextEncoderTokenizerFast.from_pretrained(rag_example_args.dpr_ctx_encoder_model_name)
    new_features = Features(
        {"text": Value("string"), "title": Value("string"), "embeddings": Sequence(Value("float32"))}
    )  # optional, save as float32 instead of float64 to save space
    dataset = dataset.map(
        partial(embed, ctx_encoder=ctx_encoder, ctx_tokenizer=ctx_tokenizer),
        batched=True,
        batch_size=processing_args.batch_size,
        features=new_features,
    )

    # And finally save your dataset
    passages_path = os.path.join(rag_example_args.output_dir, "my_knowledge_dataset")
    dataset.save_to_disk(passages_path)
    # from datasets import load_from_disk
    # dataset = load_from_disk(passages_path)  # to reload the dataset

    ######################################
    logger.info("Step 2 - Index the dataset")
    ######################################

    # Let's use the Faiss implementation of HNSW for fast approximate nearest neighbor search
    index = faiss.IndexHNSWFlat(index_hnsw_args.d, index_hnsw_args.m, faiss.METRIC_INNER_PRODUCT)
    dataset.add_faiss_index("embeddings", custom_index=index)

    # And save the index
    index_path = os.path.join(rag_example_args.output_dir, "my_knowledge_dataset_hnsw_index.faiss")
    dataset.get_index("embeddings").save(index_path)


SUMMARY:
**Main Function Summary**

The `main` function generates and indexes a Dataset object for RAG (Readability Aware Graph) using Fast AI Library (FAIL). It takes three arguments:

*   `rag_example_args`: RagExampleArguments
*   `processing_args`: ProcessingArguments
*   `index_hnsw_args`: IndexHnswArguments

Main Purpose:
Create and index a Dataset object with DPR embeddings for approximately nearest neighbor search.

Key Implementation Details:

*   Loads a CSV dataset from the specified path, splitting it into passages of 100 words.
*   Computes DPR embeddings using an existing model and tokenizer.
*   Splits the dataset into batches and saves it to disk.
*   Creates an Faiss HNSW index for fast approximate nearest neighbor search.

Notable Dependencies:

*   `load_dataset` function from PyTorch Dataset
*   `DPRContextEncoder` and `DPRContextEncoderTokenizerFast` models from AI21 Labs DPR
*   `Dataset.add_faiss_index` function from datasets library
*   Faiss library for HNSW indexing

External Dependencies:

*   CSV files with "title" and "text" columns
*   FAI Library (FAIL) for DPR embeddings computation and data loading


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/rag-end2end-retriever/kb_encode_utils.py


key: chroma:document
string_value:
CODE:

def split_text(text, n=100, character=" "):
    """Split the text every ``n``-th occurrence of ``character``"""
    text = text.split(character)


SUMMARY:
**Function Summary:**

### `split_text`

Splits the input `text` into substrings of length `n`, separated by the specified `character`.

### Key Implementation Details:

* Splits the text using the `split()` method with the given `character`.
* Applies a list comprehension to create new substrings by joining every `n`-th occurrence of the character.

### Parameters:

* `text`: input string
* `n` (optional, default=100): substring length
* `character` (optional, default=" "): delimiter character

### Returns: A list of substring strings.


key: chroma:document
string_value:
CODE:

def split_documents(documents):
    """Split documents into passages"""
    titles, texts = [], []
    for title, text in zip(documents["title"], documents["text"]):
        if text is not None:
            for passage in split_text(text):
                titles.append(title if title is not None else "")
                texts.append(passage)


SUMMARY:
**Function Summary:**

`split_documents(documents)`
---------------------------

**Primary Purpose:** Splits documents into passages based on their text content.

**Implementation Details:**

* Utilizes `zip()` to iterate over title and text combinations from input dictionary
* Calls `split_text(text)` to split text into passages, storing results in separate lists for titles and texts
* Returns a dictionary containing the split titles and texts

**Key Method Signatures:**

`split_text(text)`

**Notable Dependencies/Requirements:**

None mentioned. However, it is assumed that `split_text()` is defined elsewhere in the codebase.

**External Interfaces:**

None referenced. This function appears to be self-contained within a single module or namespace.


key: chroma:document
string_value:
CODE:

def embed_update(ctx_encoder, total_processes, device, process_num, shard_dir, csv_path):
    kb_dataset = load_dataset(
        "csv", data_files=[csv_path], split="train", delimiter="\t", column_names=["title", "text"]
    )
    kb_dataset = kb_dataset.map(
        split_documents, batched=True, num_proc=1
    )  # if you want you can load already splitted csv.
    kb_list = [kb_dataset.shard(total_processes, i, contiguous=True) for i in range(total_processes)]
    data_shrad = kb_list[process_num]

    arrow_folder = "data_" + str(process_num)
    passages_path = os.path.join(shard_dir, arrow_folder)

    context_tokenizer = DPRContextEncoderTokenizerFast.from_pretrained("facebook/dpr-ctx_encoder-multiset-base")
    ctx_encoder = ctx_encoder.to(device=device)

    def embed(
        documents: dict, ctx_encoder: DPRContextEncoder, ctx_tokenizer: DPRContextEncoderTokenizerFast, device
    ) -> dict:
        """Compute the DPR embeddings of document passages"""
        input_ids = ctx_tokenizer(
            documents["title"], documents["text"], truncation=True, padding="longest", return_tensors="pt"
        )["input_ids"]
        embeddings = ctx_encoder(input_ids.to(device=device), return_dict=True).pooler_output
        return {"embeddings": embeddings.detach().cpu().numpy()}

    new_features = Features(
        {"text": Value("string"), "title": Value("string"), "embeddings": Sequence(Value("float32"))}
    )  # optional, save as float32 instead of float64 to save space

    dataset = data_shrad.map(
        partial(embed, ctx_encoder=ctx_encoder, ctx_tokenizer=context_tokenizer, device=device),
        batched=True,
        batch_size=16,
        features=new_features,
    )


SUMMARY:
**Function Summary:**

`embed_update`

Embeds text passages into a knowledge graph using DPR Context Encoder.

### Purpose and Functionality

The function `embed_update` combines two main tasks:

1.  **Data Preparation**: Loads a CSV dataset, splits it into shards based on the total number of processes, and assigns each shard to a specific process.
2.  **Feature Extraction**: Uses DPR Context Encoder to embed text passages from each document in the assigned shard.

### Key Implementation Details

*   **Context Encoder Configuration**:
    *   `ctx_encoder`: loaded pre-trained DPR Context Encoder model.
    *   `ctx_tokenizer`: used for tokenization and encoding.
*   **Feature Extraction Function** (`embed`):
    *   takes a document as input, tokenizes the text, and passes it through the context encoder to generate embeddings.
*   **Shard Shuffling**:
    *   uses the specified process number to access the corresponding shard in the dataset.

### Dependencies and Requirements

*   `huggingface-transformers` (v4.8.0+): for loading DPR Context Encoder model and tokenizer.
*   `numpy`: for converting output to NumPy arrays.
*   `pytorch` and `torchvision`: for device management and tensor operations.

**External Dependencies:**

*   `facebook/dpr-ctx_encoder-multiset-base`
*   Custom `Features` class (not shown in the code snippet)

This summary highlights the primary functionality, key implementation details, and dependencies required to run this function.


key: chroma:document
string_value:
CODE:

def add_index(shard_dir, index_path):
    data_shard_list = []

    for shard_address in glob(str(shard_dir) + "/*/"):
        data_shard_list.append(load_from_disk(shard_address))

    concat = concatenate_datasets(data_shard_list)
    faiss.omp_set_num_threads(96)

    index = faiss.IndexHNSWFlat(768, 128, faiss.METRIC_INNER_PRODUCT)
    concat.add_faiss_index("embeddings", custom_index=index)
    concat.get_index("embeddings").save(
        index_path


SUMMARY:
**Function Summary**

`add_index` function:

* Creates an Faiss-based index for multiple data shards and stores it on disk.

**Key Implementation Details**

* Concatenates datasets from `shard_dir` using `load_from_disk()` function
* Sets the number of threads for Faiss computation (96)
* Initializes an HNSW flat-index with 768 dimensions and 128 quantization levels
* Adds a custom index ("embeddings") to the concatenated dataset

**Notable Dependencies/Requirements**

* `glob` for path matching
* `load_from_disk` (not shown) for reading data from disk
* `concatenate_datasets` (not shown) for concatenating datasets
* `Faiss` library for indexing computations
* `METRIC_INNER_PRODUCT` for index type

**Primary Purpose**

The primary purpose of this function is to create an Faiss-based index for a dataset stored across multiple data shards.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/pplm/pplm_classification_head.py


key: chroma:document
string_value:
CODE:

class ClassificationHead(nn.Module):
    """Classification Head for  transformer encoders"""

    def __init__(self, class_size, embed_size):
        super().__init__()
        self.class_size = class_size
        self.embed_size = embed_size
        # self.mlp1 = nn.Linear(embed_size, embed_size)
        # self.mlp2 = (nn.Linear(embed_size, class_size))
        self.mlp = nn.Linear(embed_size, class_size)

    def forward(self, hidden_state):
        # hidden_state = nn.functional.relu(self.mlp1(hidden_state))
        # hidden_state = self.mlp2(hidden_state)
        logits = self.mlp(hidden_state)


SUMMARY:
**Summary:**

The `ClassificationHead` class is a neural network module for classification tasks in transformer encoders. It takes two inputs:

* `class_size`: the number of classes to classify
* `embed_size`: the embedding size used by the network

The primary functionality is defined in the `forward` method, which applies a linear transformation (`mlp`) to the input `hidden_state`, producing logits for classification.

**Key Implementation Details:**

* Applies a single linear layer (`nn.Linear(embed_size, class_size)`) for classification
* Does not use multiple intermediate layers (`self.mlp1` and `self.mlp2`)
* Returns logits for classification

**Notable Dependencies:**

* Requires PyTorch's `nn.Module`
* Utilizes `nn.Linear` for linear transformation
* Inherited from PyTorch's `nn.Module`

**Required Arguments:**

* `class_size`: int, number of classes to classify
* `embed_size`: int, embedding size used by the network


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/pplm/run_pplm_discrim_train.py


key: chroma:document
string_value:
CODE:

class Discriminator(nn.Module):
    """Transformer encoder followed by a Classification Head"""

    def __init__(self, class_size, pretrained_model="openai-community/gpt2-medium", cached_mode=False, device="cpu"):
        super().__init__()
        self.tokenizer = GPT2Tokenizer.from_pretrained(pretrained_model)
        self.encoder = GPT2LMHeadModel.from_pretrained(pretrained_model)
        self.embed_size = self.encoder.transformer.config.hidden_size
        self.classifier_head = ClassificationHead(class_size=class_size, embed_size=self.embed_size)
        self.cached_mode = cached_mode
        self.device = device

    def get_classifier(self):
        return self.classifier_head

    def train_custom(self):
        for param in self.encoder.parameters():
            param.requires_grad = False
        self.classifier_head.train()

    def avg_representation(self, x):
        mask = x.ne(0).unsqueeze(2).repeat(1, 1, self.embed_size).float().to(self.device).detach()
        hidden = self.encoder.transformer(x)["last_hidden_state"]
        masked_hidden = hidden * mask
        avg_hidden = torch.sum(masked_hidden, dim=1) / (torch.sum(mask, dim=1).detach() + EPSILON)
        return avg_hidden

    def forward(self, x):
        if self.cached_mode:
            avg_hidden = x.to(self.device)
        else:
            avg_hidden = self.avg_representation(x.to(self.device))

        logits = self.classifier_head(avg_hidden)
        probs = nn.functional.log_softmax(logits, dim=-1)



SUMMARY:
**Discriminator Class**
========================

*Class Purpose:*

The Discriminator class is designed as a transformer encoder followed by a classification head. It leverages pre-trained GPT2 weights and can be fine-tuned for custom tasks.

**Key Implementation Details:**

*   Uses pre-trained `GPT2Tokenizer` and `GPT2LMHeadModel`
*   Applies an average pooling mechanism (`avg_representation`) to extract a fixed-size representation
*   Employs a classification head (`ClassificationHead`) with a customizable output layer size
*   Offers toggling of cached mode for improved performance

**Notable Dependencies:**

*   `torch`: PyTorch library
*   `GPT2Tokenizer` and `GPT2LMHeadModel`: Pre-trained models from Hugging Face's Transformers library
*   `ClassificationHead`: Custom classification head class with a customizable output layer size


key: chroma:document
string_value:
CODE:

class Dataset(data.Dataset):
    def __init__(self, X, y):
        """Reads source and target sequences from txt files."""
        self.X = X
        self.y = y

    def __len__(self):
        return len(self.X)

    def __getitem__(self, index):
        """Returns one data pair (source and target)."""
        data = {}
        data["X"] = self.X[index]
        data["y"] = self.y[index]


SUMMARY:
**Class Summary: Dataset**

### Overview
A dataset class that extends PyTorch's `Dataset` and provides a custom implementation for storing and retrieving data.

### Main Purpose and Functionality
This class is designed to store and manage tabular data consisting of source (`X`) and target (`y`) sequences. It exposes methods for accessing the dataset's length and individual data points.

### Key Implementation Details

* The `__init__` method initializes the dataset with two inputs: `X` (source sequences) and `y` (target sequences).
* The `__len__` method returns the number of data points in the dataset, which is determined by the length of `X`.
* The `__getitem__` method retrieves a single data point (`index`) from the dataset, returning a dictionary containing the source sequence (`X[index]`) and target sequence (`y[index]`).

### Notable Dependencies or Requirements
This class does not include any external dependencies. However, it relies on PyTorch's `Dataset` and its associated interfaces.

Note: The unused codeblock following the `return data` line has been excluded from this summary as it is redundant and can be safely ignored in a production context.


key: chroma:document
string_value:
CODE:

def collate_fn(data):
    def pad_sequences(sequences):
        lengths = [len(seq) for seq in sequences]

        padded_sequences = torch.zeros(len(sequences), max(lengths)).long()  # padding value = 0

        for i, seq in enumerate(sequences):
            end = lengths[i]
            padded_sequences[i, :end] = seq[:end]

        return padded_sequences, lengths

    item_info = {}
    for key in data[0].keys():
        item_info[key] = [d[key] for d in data]

    x_batch, _ = pad_sequences(item_info["X"])
    y_batch = torch.tensor(item_info["y"], dtype=torch.long)



SUMMARY:
**Function Summary: `collate_fn`**

**Purpose and Functionality:**
The `collate_fn` function collates and pads batch data for training neural network models in PyTorch.

**Key Implementation Details:**

* Pads input sequences (`X`) to a maximum length using padding values of 0.
* Converts the padded sequences into a tensor format.

**Notable Dependencies and Requirements:**

* `torch`: required for tensor operations.
* `data` (collection): expected to be a list of dictionaries with keys "X" and "y".
	+ `item_info["X"]` is used as input data for padding.
	+ `item_info["y"]` is the corresponding label data.

**Key Method Signatures:**

* `pad_sequences(sequences)`: takes a list of sequences, pads them to a maximum length, and returns the padded sequences and their lengths.
* `torch.tensor()`: converts data into a tensor format.


key: chroma:document
string_value:
CODE:

def cached_collate_fn(data):
    item_info = {}
    for key in data[0].keys():
        item_info[key] = [d[key] for d in data]

    x_batch = torch.cat(item_info["X"], 0)
    y_batch = torch.tensor(item_info["y"], dtype=torch.long)



SUMMARY:
**Function Summary: `cached_collate_fn`**

This function implements the PyTorch collation function with caching. Its primary purpose is to batch and process data for use in training a neural network.

**Key Functionality:**

- Creates a dictionary `item_info` containing lists of values for each feature (X) across all samples.
- Concatenates the feature lists along their first dimension using `torch.cat`.
- Processes labels (`y`) into a tensor.
- Returns the batched feature data (`x_batch`) and corresponding labels (`y_batch`).

**Notable Implementation Details:**

- Uses list comprehensions for feature value extraction.
- Utilizes `torch.cat` for batching features.

**Dependencies/Requirements:**

* PyTorch library (specifically, its tensor functionality)
* The input data is expected to have a specific structure with the keys "X" and "y".
* The input data also should be iterable (possibly a list or some similar data structure).


key: chroma:document
string_value:
CODE:

def train_epoch(data_loader, discriminator, optimizer, epoch=0, log_interval=10, device="cpu"):
    samples_so_far = 0
    discriminator.train_custom()
    for batch_idx, (input_t, target_t) in enumerate(data_loader):
        input_t, target_t = input_t.to(device), target_t.to(device)

        optimizer.zero_grad()

        output_t = discriminator(input_t)
        loss = nn.functional.nll_loss(output_t, target_t)
        loss.backward(retain_graph=True)
        optimizer.step()

        samples_so_far += len(input_t)

        if batch_idx % log_interval == 0:
            print(
                "Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}".format(
                    epoch + 1,
                    samples_so_far,
                    len(data_loader.dataset),
                    100 * samples_so_far / len(data_loader.dataset),
                    loss.item(),
                )


SUMMARY:
**Function Summary**

`train_epoch`: A function responsible for training a neural network model using the training dataset.

**Main Purpose and Functionality**

Trains a discriminator model on a dataset loading in batches. The optimizer is used to update the weights of the model based on the loss value computed after each batch. The loss function used is cross-entropy loss (NLL).

**Key Implementation Details**

* Utilizes `nn.functional.nll_loss` for calculating loss
* Uses `retain_graph=True` when calling `loss.backward()` to maintain the computational graph
* Updates the optimizer's weights using `optimizer.step()`

**Notable Dependencies and Requirements**

* `data_loader`: A data loading object providing batches of input and target tensors.
* `discriminator`: The neural network model being trained.
* `optimizer`: An optimization algorithm (e.g., Adam, SGD) used to update the model weights.

**Notable Interface/Class Usage**

* `nn.functional.nll_loss`: Computes cross-entropy loss between predicted output and ground-truth target.
* `loss.backward()`: Computes gradients of loss with respect to model parameters
* `optimizer.step()`: Updates model weights based on computed gradients.


key: chroma:document
string_value:
CODE:

def evaluate_performance(data_loader, discriminator, device="cpu"):
    discriminator.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for input_t, target_t in data_loader:
            input_t, target_t = input_t.to(device), target_t.to(device)
            output_t = discriminator(input_t)
            # sum up batch loss
            test_loss += nn.functional.nll_loss(output_t, target_t, reduction="sum").item()
            # get the index of the max log-probability
            pred_t = output_t.argmax(dim=1, keepdim=True)
            correct += pred_t.eq(target_t.view_as(pred_t)).sum().item()

    test_loss /= len(data_loader.dataset)

    print(
        "Performance on test set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)".format(
            test_loss, correct, len(data_loader.dataset), 100.0 * correct / len(data_loader.dataset)
        )


SUMMARY:
**Summary**

Evaluates model performance on a test set.

**Main Purpose and Functionality**

* Evaluates the performance of a discriminator model using an `evaluate_performance` function.
* Computes average logarithmic loss and prediction accuracy on a test dataset loaded via a `data_loader`.

**Key Implementation Details**

* Sets the `eval` mode for the discriminator model.
* Sums up batch losses using `nn.functional.nll_loss`.
* Computes accuracy by comparing predicted labels with true labels.

**Notable Dependencies/Requirements**

* `torch`: No external dependencies listed, but references PyTorch classes such as `nn`, ` torch.no_grad()`.


