# embedding_metadata

key: chroma:document
string_value:
CODE:

def sanity_checks(args):
    """
    A bunch of args sanity checks to perform even starting...
    """
    assert (args.mlm and args.alpha_mlm > 0.0) or (not args.mlm and args.alpha_mlm == 0.0)
    assert (args.alpha_mlm > 0.0 and args.alpha_clm == 0.0) or (args.alpha_mlm == 0.0 and args.alpha_clm > 0.0)
    if args.mlm:
        assert os.path.isfile(args.token_counts)
        assert (args.student_type in ["roberta", "distilbert"]) and (args.teacher_type in ["roberta", "bert"])
    else:
        assert (args.student_type in ["gpt2"]) and (args.teacher_type in ["gpt2"])

    assert args.teacher_type == args.student_type or (
        args.student_type == "distilbert" and args.teacher_type == "bert"
    )
    assert os.path.isfile(args.student_config)
    if args.student_pretrained_weights is not None:
        assert os.path.isfile(args.student_pretrained_weights)

    if args.freeze_token_type_embds:
        assert args.student_type in ["roberta"]

    assert args.alpha_ce >= 0.0
    assert args.alpha_mlm >= 0.0
    assert args.alpha_clm >= 0.0
    assert args.alpha_mse >= 0.0
    assert args.alpha_cos >= 0.0


SUMMARY:
**Function Summary: `sanity_checks(args)`**

### Purpose and Functionality

This function performs a set of sanity checks on input arguments (`args`) to ensure their validity for model training. The primary purpose is to validate the model configuration, student and teacher type, file paths, and hyperparameter settings.

### Key Implementation Details

*   Validates model-specific parameters (e.g., `alpha_mlm`, `alpha_clm`, `alpha_ce`)
*   Checks if required files exist (e.g., `token_counts`, `student_config`, `student_pretrained_weights`)
*   Restricts allowed student and teacher types based on the ML models used

### Notable Dependencies or Requirements

*   `os` module for interacting with file system
*   `args.mlm`: flag indicating whether masked language modeling is enabled
*   `args.alpha_*`: hyperparameter values (e.g., `alpha_mlm`, `alpha_clm`)
*   `args.student_type` and `args.teacher_type`: strings representing the type of student and teacher models used


key: chroma:document
string_value:
CODE:

def freeze_pos_embeddings(student, args):
    if args.student_type == "roberta":
        student.roberta.embeddings.position_embeddings.weight.requires_grad = False
    elif args.student_type == "gpt2":


SUMMARY:
**Summary**

The `freeze_pos_embeddings` function modifies the embedding weights of position embeddings in different student models (`roberta` and `gpt2`) to prevent gradient updates.

**Key Implementation Details**

* For `roberta`, sets `requires_grad = False` on the `weight` attribute of `position_embdings`.
* For `gpt2`, sets `requires_grad = False` on the `weight` attribute of `wpe`.

**Notable Dependencies/Requirements**

* `student.roberta.embeddings.position_embeddings` (for `roberta` models)
* `student.transformer.wpe` (for `gpt2` models)

Note: This function assumes that `args.student_type` is either "roberta" or "gpt2", and that the `roberta` and `wpe` attributes reference specific classes or methods within the `student` object.


key: chroma:document
string_value:
CODE:

def freeze_token_type_embeddings(student, args):
    if args.student_type == "roberta":


SUMMARY:
**Summary: Freeze Roberta Token Type Embeddings**

* **Purpose**: Freeze the gradient of token type embeddings in a Roberta model during training.
* **Implementation Details**:
	+ Set the `requires_grad` property of the `weight` attribute of `token_type_embeddings` to `False`.
* **Notable Dependencies/Requirements**:
	+ `student`: A PyTorch model instance with access to its sub-modules, including `roberta.embeddings.token_type_embeddings`.
	+ `args.student_type == "roberta"`: Ensures the correct model subclass is being accessed.

Note: This summary focuses on the main purpose and implementation details of the function, while excluding generic information and excessive technical terms.


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser(description="Training")
    parser.add_argument("--force", action="store_true", help="Overwrite dump_path if it already exists.")

    parser.add_argument(
        "--dump_path", type=str, required=True, help="The output directory (log, checkpoints, parameters, etc.)"
    )
    parser.add_argument(
        "--data_file",
        type=str,
        required=True,
        help="The binarized file (tokenized + tokens_to_ids) and grouped by sequence.",
    )

    parser.add_argument(
        "--student_type",
        type=str,
        choices=["distilbert", "roberta", "gpt2"],
        required=True,
        help="The student type (DistilBERT, RoBERTa).",
    )
    parser.add_argument("--student_config", type=str, required=True, help="Path to the student configuration.")
    parser.add_argument(
        "--student_pretrained_weights", default=None, type=str, help="Load student initialization checkpoint."
    )

    parser.add_argument(
        "--teacher_type", choices=["bert", "roberta", "gpt2"], required=True, help="Teacher type (BERT, RoBERTa)."
    )
    parser.add_argument("--teacher_name", type=str, required=True, help="The teacher model.")

    parser.add_argument("--temperature", default=2.0, type=float, help="Temperature for the softmax temperature.")
    parser.add_argument(
        "--alpha_ce", default=0.5, type=float, help="Linear weight for the distillation loss. Must be >=0."
    )
    parser.add_argument(
        "--alpha_mlm",
        default=0.0,
        type=float,
        help="Linear weight for the MLM loss. Must be >=0. Should be used in conjunction with `mlm` flag.",
    )
    parser.add_argument("--alpha_clm", default=0.5, type=float, help="Linear weight for the CLM loss. Must be >=0.")
    parser.add_argument("--alpha_mse", default=0.0, type=float, help="Linear weight of the MSE loss. Must be >=0.")
    parser.add_argument(
        "--alpha_cos", default=0.0, type=float, help="Linear weight of the cosine embedding loss. Must be >=0."
    )

    parser.add_argument(
        "--mlm", action="store_true", help="The LM step: MLM or CLM. If `mlm` is True, the MLM is used over CLM."
    )
    parser.add_argument(
        "--mlm_mask_prop",
        default=0.15,
        type=float,
        help="Proportion of tokens for which we need to make a prediction.",
    )
    parser.add_argument("--word_mask", default=0.8, type=float, help="Proportion of tokens to mask out.")
    parser.add_argument("--word_keep", default=0.1, type=float, help="Proportion of tokens to keep.")
    parser.add_argument("--word_rand", default=0.1, type=float, help="Proportion of tokens to randomly replace.")
    parser.add_argument(
        "--mlm_smoothing",
        default=0.7,
        type=float,
        help="Smoothing parameter to emphasize more rare tokens (see XLM, similar to word2vec).",
    )
    parser.add_argument("--token_counts", type=str, help="The token counts in the data_file for MLM.")

    parser.add_argument(
        "--restrict_ce_to_mask",
        action="store_true",
        help="If true, compute the distillation loss only the [MLM] prediction distribution.",
    )
    parser.add_argument(
        "--freeze_pos_embs",
        action="store_true",
        help="Freeze positional embeddings during distillation. For student_type in ['roberta', 'gpt2'] only.",
    )
    parser.add_argument(
        "--freeze_token_type_embds",
        action="store_true",
        help="Freeze token type embeddings during distillation if existent. For student_type in ['roberta'] only.",
    )

    parser.add_argument("--n_epoch", type=int, default=3, help="Number of pass on the whole dataset.")
    parser.add_argument("--batch_size", type=int, default=5, help="Batch size (for each process).")
    parser.add_argument(
        "--group_by_size",
        action="store_false",
        help="If true, group sequences that have similar length into the same batch. Default is true.",
    )

    parser.add_argument(
        "--gradient_accumulation_steps",
        type=int,
        default=50,
        help="Gradient accumulation for larger training batches.",
    )
    parser.add_argument("--warmup_prop", default=0.05, type=float, help="Linear warmup proportion.")
    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
    parser.add_argument("--learning_rate", default=5e-4, type=float, help="The initial learning rate for Adam.")
    parser.add_argument("--adam_epsilon", default=1e-6, type=float, help="Epsilon for Adam optimizer.")
    parser.add_argument("--max_grad_norm", default=5.0, type=float, help="Max gradient norm.")
    parser.add_argument("--initializer_range", default=0.02, type=float, help="Random initialization range.")

    parser.add_argument(
        "--fp16",
        action="store_true",
        help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit",
    )
    parser.add_argument(
        "--fp16_opt_level",
        type=str,
        default="O1",
        help=(
            "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. "
            "See details at https://nvidia.github.io/apex/amp.html"
        ),
    )
    parser.add_argument("--n_gpu", type=int, default=1, help="Number of GPUs in the node.")
    parser.add_argument("--local_rank", type=int, default=-1, help="Distributed training - Local rank")
    parser.add_argument("--seed", type=int, default=56, help="Random seed")

    parser.add_argument("--log_interval", type=int, default=500, help="Tensorboard logging interval.")
    parser.add_argument("--checkpoint_interval", type=int, default=4000, help="Checkpoint interval.")
    args = parser.parse_args()
    sanity_checks(args)

    # ARGS #
    init_gpu_params(args)
    set_seed(args)
    if args.is_master:
        if os.path.exists(args.dump_path):
            if not args.force:
                raise ValueError(
                    f"Serialization dir {args.dump_path} already exists, but you have not precised wheter to overwrite"
                    " itUse `--force` if you want to overwrite it"
                )
            else:
                shutil.rmtree(args.dump_path)

        if not os.path.exists(args.dump_path):
            os.makedirs(args.dump_path)
        logger.info(f"Experiment will be dumped and logged in {args.dump_path}")

        # SAVE PARAMS #
        logger.info(f"Param: {args}")
        with open(os.path.join(args.dump_path, "parameters.json"), "w") as f:
            json.dump(vars(args), f, indent=4)
        git_log(args.dump_path)

    student_config_class, student_model_class, _ = MODEL_CLASSES[args.student_type]
    teacher_config_class, teacher_model_class, teacher_tokenizer_class = MODEL_CLASSES[args.teacher_type]

    # TOKENIZER #
    tokenizer = teacher_tokenizer_class.from_pretrained(args.teacher_name)
    special_tok_ids = {}
    for tok_name, tok_symbol in tokenizer.special_tokens_map.items():
        idx = tokenizer.all_special_tokens.index(tok_symbol)
        special_tok_ids[tok_name] = tokenizer.all_special_ids[idx]
    logger.info(f"Special tokens {special_tok_ids}")
    args.special_tok_ids = special_tok_ids
    args.max_model_input_size = tokenizer.max_model_input_sizes[args.teacher_name]

    # DATA LOADER #
    logger.info(f"Loading data from {args.data_file}")
    with open(args.data_file, "rb") as fp:
        data = pickle.load(fp)

    if args.mlm:
        logger.info(f"Loading token counts from {args.token_counts} (already pre-computed)")
        with open(args.token_counts, "rb") as fp:
            counts = pickle.load(fp)

        token_probs = np.maximum(counts, 1) ** -args.mlm_smoothing
        for idx in special_tok_ids.values():
            token_probs[idx] = 0.0  # do not predict special tokens
        token_probs = torch.from_numpy(token_probs)
    else:
        token_probs = None

    train_lm_seq_dataset = LmSeqsDataset(params=args, data=data)
    logger.info("Data loader created.")

    # STUDENT #
    logger.info(f"Loading student config from {args.student_config}")
    stu_architecture_config = student_config_class.from_pretrained(args.student_config)
    stu_architecture_config.output_hidden_states = True

    if args.student_pretrained_weights is not None:
        logger.info(f"Loading pretrained weights from {args.student_pretrained_weights}")
        student = student_model_class.from_pretrained(args.student_pretrained_weights, config=stu_architecture_config)
    else:
        student = student_model_class(stu_architecture_config)

    if args.n_gpu > 0:
        student.to(f"cuda:{args.local_rank}")
    logger.info("Student loaded.")

    # TEACHER #
    teacher = teacher_model_class.from_pretrained(args.teacher_name, output_hidden_states=True)
    if args.n_gpu > 0:
        teacher.to(f"cuda:{args.local_rank}")
    logger.info(f"Teacher loaded from {args.teacher_name}.")

    # FREEZING #
    if args.freeze_pos_embs:
        freeze_pos_embeddings(student, args)
    if args.freeze_token_type_embds:
        freeze_token_type_embeddings(student, args)

    # SANITY CHECKS #
    assert student.config.vocab_size == teacher.config.vocab_size
    assert student.config.hidden_size == teacher.config.hidden_size
    assert student.config.max_position_embeddings == teacher.config.max_position_embeddings
    if args.mlm:
        assert token_probs.size(0) == stu_architecture_config.vocab_size

    # DISTILLER #
    torch.cuda.empty_cache()
    distiller = Distiller(
        params=args, dataset=train_lm_seq_dataset, token_probs=token_probs, student=student, teacher=teacher
    )
    distiller.train()


SUMMARY:
This code is for setting up an experiment configuration and loading necessary pre-trained models and data. Here's a high-level overview of the process:

1. **Argument parsing**: The script starts by parsing command-line arguments using the `argparse` library, which includes parameters such as model type, student config, teacher config, and training data.
2. **Initialization**: The script initializes various internal objects, including tokenizers, data loaders, and models, based on the parsed argument values.
3. **Sanity checks**: The script performs sanity checks to ensure that certain conditions are met, such as matching vocabulary sizes between student and teacher models or valid tokenization configuration.
4. **Distiller setup**: A `Distiller` object is created with the experimental parameters, data loader, token probabilities (if applicable), student model, and teacher model.
5. **Training logging**: The script logs training-related information, such as log intervals and checkpoint intervals.

In summary, this script sets up a configuration for training a distillation model, loads pre-trained models and data, and performs initial checks before running the training process.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/distillation/run_squad_w_distillation.py


key: chroma:document
string_value:
CODE:

def set_seed(args):
    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    if args.n_gpu > 0:


SUMMARY:
**Summary: `set_seed` Function**

 Sets seed values for random, NumPy, and PyTorch libraries.

**Key Implementation Details:**

* Uses `random.seed`, `np.random.seed`, and `torch.manual_seed` to set global seed values.
* Sets CUDA seed value (`torch.cuda.manual_seed_all`) if multiple GPUs are available (GPU count > 0).

**Notable Dependencies or Requirements:**

* None identified, but uses:
	+ `random`
	+ `numpy (np)`
	+ `torch`


key: chroma:document
string_value:
CODE:

def to_list(tensor):


SUMMARY:
Here is a concise summary of the provided function:

**Function:** `to_list`
**Purpose:** Converts a PyTorch tensor to a Python list.

**Implementation:**

* Detaches the tensor from the computation graph
* Moves the tensor to the CPU (if not already on it)
* Converts the tensor to a Python list

**Key Details:**

* Uses `detach()` to break the connection between the tensor and the computation graph, allowing for efficient conversion
* Uses `cpu()` to move the tensor data to the CPU, if necessary

**Dependencies/Requirements:** None (uses only built-in PyTorch functionality)


key: chroma:document
string_value:
CODE:

def train(args, train_dataset, model, tokenizer, teacher=None):
    """Train the model"""
    if args.local_rank in [-1, 0]:
        tb_writer = SummaryWriter()

    args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)
    train_sampler = RandomSampler(train_dataset) if args.local_rank == -1 else DistributedSampler(train_dataset)
    train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size)

    if args.max_steps > 0:
        t_total = args.max_steps
        args.num_train_epochs = args.max_steps // (len(train_dataloader) // args.gradient_accumulation_steps) + 1
    else:
        t_total = len(train_dataloader) // args.gradient_accumulation_steps * args.num_train_epochs

    # Prepare optimizer and schedule (linear warmup and decay)
    no_decay = ["bias", "LayerNorm.weight"]
    optimizer_grouped_parameters = [
        {
            "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
            "weight_decay": args.weight_decay,
        },
        {"params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], "weight_decay": 0.0},
    ]
    optimizer = AdamW(optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon)
    scheduler = get_linear_schedule_with_warmup(
        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
    )

    # Check if saved optimizer or scheduler states exist
    if os.path.isfile(os.path.join(args.model_name_or_path, "optimizer.pt")) and os.path.isfile(
        os.path.join(args.model_name_or_path, "scheduler.pt")
    ):
        # Load in optimizer and scheduler states
        optimizer.load_state_dict(torch.load(os.path.join(args.model_name_or_path, "optimizer.pt")))
        scheduler.load_state_dict(torch.load(os.path.join(args.model_name_or_path, "scheduler.pt")))

    if args.fp16:
        try:
            from apex import amp
        except ImportError:
            raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")

        model, optimizer = amp.initialize(model, optimizer, opt_level=args.fp16_opt_level)

    # multi-gpu training (should be after apex fp16 initialization)
    if args.n_gpu > 1:
        model = nn.DataParallel(model)

    # Distributed training (should be after apex fp16 initialization)
    if args.local_rank != -1:
        model = nn.parallel.DistributedDataParallel(
            model, device_ids=[args.local_rank], output_device=args.local_rank, find_unused_parameters=True
        )

    # Train!
    logger.info("***** Running training *****")
    logger.info("  Num examples = %d", len(train_dataset))
    logger.info("  Num Epochs = %d", args.num_train_epochs)
    logger.info("  Instantaneous batch size per GPU = %d", args.per_gpu_train_batch_size)
    logger.info(
        "  Total train batch size (w. parallel, distributed & accumulation) = %d",
        args.train_batch_size
        * args.gradient_accumulation_steps
        * (torch.distributed.get_world_size() if args.local_rank != -1 else 1),
    )
    logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
    logger.info("  Total optimization steps = %d", t_total)

    global_step = 1
    epochs_trained = 0
    steps_trained_in_current_epoch = 0
    # Check if continuing training from a checkpoint
    if os.path.exists(args.model_name_or_path):
        try:
            # set global_step to global_step of last saved checkpoint from model path
            checkpoint_suffix = args.model_name_or_path.split("-")[-1].split("/")[0]
            global_step = int(checkpoint_suffix)
            epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
            steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)

            logger.info("  Continuing training from checkpoint, will skip to saved global_step")
            logger.info("  Continuing training from epoch %d", epochs_trained)
            logger.info("  Continuing training from global step %d", global_step)
            logger.info("  Will skip the first %d steps in the first epoch", steps_trained_in_current_epoch)
        except ValueError:
            logger.info("  Starting fine-tuning.")

    tr_loss, logging_loss = 0.0, 0.0
    model.zero_grad()
    train_iterator = trange(
        epochs_trained, int(args.num_train_epochs), desc="Epoch", disable=args.local_rank not in [-1, 0]
    )
    # Added here for reproducibility
    set_seed(args)

    for _ in train_iterator:
        epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
        for step, batch in enumerate(epoch_iterator):
            # Skip past any already trained steps if resuming training
            if steps_trained_in_current_epoch > 0:
                steps_trained_in_current_epoch -= 1
                continue

            model.train()
            if teacher is not None:
                teacher.eval()
            batch = tuple(t.to(args.device) for t in batch)

            inputs = {
                "input_ids": batch[0],
                "attention_mask": batch[1],
                "start_positions": batch[3],
                "end_positions": batch[4],
            }
            if args.model_type != "distilbert":
                inputs["token_type_ids"] = None if args.model_type == "xlm" else batch[2]
            if args.model_type in ["xlnet", "xlm"]:
                inputs.update({"cls_index": batch[5], "p_mask": batch[6]})
                if args.version_2_with_negative:
                    inputs.update({"is_impossible": batch[7]})
            outputs = model(**inputs)
            loss, start_logits_stu, end_logits_stu = outputs

            # Distillation loss
            if teacher is not None:
                if "token_type_ids" not in inputs:
                    inputs["token_type_ids"] = None if args.teacher_type == "xlm" else batch[2]
                with torch.no_grad():
                    start_logits_tea, end_logits_tea = teacher(
                        input_ids=inputs["input_ids"],
                        token_type_ids=inputs["token_type_ids"],
                        attention_mask=inputs["attention_mask"],
                    )
                assert start_logits_tea.size() == start_logits_stu.size()
                assert end_logits_tea.size() == end_logits_stu.size()

                loss_fct = nn.KLDivLoss(reduction="batchmean")
                loss_start = loss_fct(
                    nn.functional.log_softmax(start_logits_stu / args.temperature, dim=-1),
                    nn.functional.softmax(start_logits_tea / args.temperature, dim=-1),
                ) * (args.temperature**2)
                loss_end = loss_fct(
                    nn.functional.log_softmax(end_logits_stu / args.temperature, dim=-1),
                    nn.functional.softmax(end_logits_tea / args.temperature, dim=-1),
                ) * (args.temperature**2)
                loss_ce = (loss_start + loss_end) / 2.0

                loss = args.alpha_ce * loss_ce + args.alpha_squad * loss

            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel (not distributed) training
            if args.gradient_accumulation_steps > 1:
                loss = loss / args.gradient_accumulation_steps

            if args.fp16:
                with amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
            else:
                loss.backward()

            tr_loss += loss.item()
            if (step + 1) % args.gradient_accumulation_steps == 0:
                if args.fp16:
                    nn.utils.clip_grad_norm_(amp.master_params(optimizer), args.max_grad_norm)
                else:
                    nn.utils.clip_grad_norm_(model.parameters(), args.max_grad_norm)

                optimizer.step()
                scheduler.step()  # Update learning rate schedule
                model.zero_grad()
                global_step += 1

                # Log metrics
                if args.local_rank in [-1, 0] and args.logging_steps > 0 and global_step % args.logging_steps == 0:
                    # Only evaluate when single GPU otherwise metrics may not average well
                    if args.local_rank == -1 and args.evaluate_during_training:
                        results = evaluate(args, model, tokenizer)
                        for key, value in results.items():
                            tb_writer.add_scalar("eval_{}".format(key), value, global_step)
                    tb_writer.add_scalar("lr", scheduler.get_lr()[0], global_step)
                    tb_writer.add_scalar("loss", (tr_loss - logging_loss) / args.logging_steps, global_step)
                    logging_loss = tr_loss

                if args.local_rank in [-1, 0] and args.save_steps > 0 and global_step % args.save_steps == 0:
                    # Save model checkpoint
                    output_dir = os.path.join(args.output_dir, "checkpoint-{}".format(global_step))
                    if not os.path.exists(output_dir):
                        os.makedirs(output_dir)
                    model_to_save = (
                        model.module if hasattr(model, "module") else model
                    )  # Take care of distributed/parallel training
                    model_to_save.save_pretrained(output_dir)
                    tokenizer.save_pretrained(output_dir)

                    torch.save(args, os.path.join(output_dir, "training_args.bin"))
                    logger.info("Saving model checkpoint to %s", output_dir)

                    torch.save(optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
                    torch.save(scheduler.state_dict(), os.path.join(output_dir, "scheduler.pt"))
                    logger.info("Saving optimizer and scheduler states to %s", output_dir)

            if args.max_steps > 0 and global_step > args.max_steps:
                epoch_iterator.close()
                break
        if args.max_steps > 0 and global_step > args.max_steps:
            train_iterator.close()
            break

    if args.local_rank in [-1, 0]:
        tb_writer.close()



SUMMARY:
**Training Script for BERT-based Model**

This script trains a BERT-based model on a sequence classification task with distillation. It uses the Hugging Face Transformers library and provides various training options.

**Main Training Loop**

1. Initialize model, tokenizer, and training arguments.
2. Train the model for a maximum of `max_steps` or until convergence (`max_epochs * total_timesteps`).
3. During each iteration:
   1. Compute the input IDs, attention mask, and target labels from the input data.
   2. Forward-pass the input through the original BERT model to obtain start and end logits.
   3. If distillation is enabled, compute the teacher's outputs using the `teacher` function.
   4. Calculate the loss (ce loss + distillation loss).
   5. Backward pass: Compute gradients of the losses with respect to model parameters.
6. After each iteration:
   1. Update the model weights using optimizer updates.
   2. Schedule learning rate update if `fp16` is enabled.

**Training Settings**

* Model architecture: Pretrained BERT (base, large, or distil)
* Distillation flag (`teacher_type`)
* Sequence classification task (start and end position prediction)
* Training optimization:
	+ AdamW optimizer with weight decay
	+ Warmup scheduler
	+ Learning rate scheduling (decay after 50k steps)

**Logging and Saving**

* Train metrics logged to TensorBoardWriter at regular intervals (every `logging_steps`).
* Model checkpoints saved every `save_steps`.
* Optimizer and scheduler states saved as binary files at checkpoint time.


key: chroma:document
string_value:
CODE:

def evaluate(args, model, tokenizer, prefix=""):
    dataset, examples, features = load_and_cache_examples(args, tokenizer, evaluate=True, output_examples=True)

    if not os.path.exists(args.output_dir) and args.local_rank in [-1, 0]:
        os.makedirs(args.output_dir)

    args.eval_batch_size = args.per_gpu_eval_batch_size * max(1, args.n_gpu)

    # Note that DistributedSampler samples randomly
    eval_sampler = SequentialSampler(dataset)
    eval_dataloader = DataLoader(dataset, sampler=eval_sampler, batch_size=args.eval_batch_size)

    # multi-gpu evaluate
    if args.n_gpu > 1 and not isinstance(model, nn.DataParallel):
        model = nn.DataParallel(model)

    # Eval!
    logger.info("***** Running evaluation {} *****".format(prefix))
    logger.info("  Num examples = %d", len(dataset))
    logger.info("  Batch size = %d", args.eval_batch_size)

    all_results = []
    start_time = timeit.default_timer()

    for batch in tqdm(eval_dataloader, desc="Evaluating"):
        model.eval()
        batch = tuple(t.to(args.device) for t in batch)

        with torch.no_grad():
            inputs = {"input_ids": batch[0], "attention_mask": batch[1]}
            if args.model_type != "distilbert":
                inputs["token_type_ids"] = None if args.model_type == "xlm" else batch[2]  # XLM don't use segment_ids
            example_indices = batch[3]
            if args.model_type in ["xlnet", "xlm"]:
                inputs.update({"cls_index": batch[4], "p_mask": batch[5]})

            outputs = model(**inputs)

        for i, example_index in enumerate(example_indices):
            eval_feature = features[example_index.item()]
            unique_id = int(eval_feature.unique_id)

            output = [to_list(output[i]) for output in outputs]

            # Some models (XLNet, XLM) use 5 arguments for their predictions, while the other "simpler"
            # models only use two.
            if len(output) >= 5:
                start_logits = output[0]
                start_top_index = output[1]
                end_logits = output[2]
                end_top_index = output[3]
                cls_logits = output[4]

                result = SquadResult(
                    unique_id,
                    start_logits,
                    end_logits,
                    start_top_index=start_top_index,
                    end_top_index=end_top_index,
                    cls_logits=cls_logits,
                )

            else:
                start_logits, end_logits = output
                result = SquadResult(unique_id, start_logits, end_logits)

            all_results.append(result)

    evalTime = timeit.default_timer() - start_time
    logger.info("  Evaluation done in total %f secs (%f sec per example)", evalTime, evalTime / len(dataset))

    # Compute predictions
    output_prediction_file = os.path.join(args.output_dir, "predictions_{}.json".format(prefix))
    output_nbest_file = os.path.join(args.output_dir, "nbest_predictions_{}.json".format(prefix))

    if args.version_2_with_negative:
        output_null_log_odds_file = os.path.join(args.output_dir, "null_odds_{}.json".format(prefix))
    else:
        output_null_log_odds_file = None

    if args.model_type in ["xlnet", "xlm"]:
        # XLNet uses a more complex post-processing procedure
        predictions = compute_predictions_log_probs(
            examples,
            features,
            all_results,
            args.n_best_size,
            args.max_answer_length,
            output_prediction_file,
            output_nbest_file,
            output_null_log_odds_file,
            model.config.start_n_top,
            model.config.end_n_top,
            args.version_2_with_negative,
            tokenizer,
            args.verbose_logging,
        )
    else:
        predictions = compute_predictions_logits(
            examples,
            features,
            all_results,
            args.n_best_size,
            args.max_answer_length,
            args.do_lower_case,
            output_prediction_file,
            output_nbest_file,
            output_null_log_odds_file,
            args.verbose_logging,
            args.version_2_with_negative,
            args.null_score_diff_threshold,
            tokenizer,
        )

    # Compute the F1 and exact scores.
    results = squad_evaluate(examples, predictions)


SUMMARY:
**Summary:**

This function, `evaluate`, implements a script for evaluating a QA model using the Squad benchmark. The primary purpose is to analyze input text, generate responses, and calculate evaluation metrics.

**Key Implementation Details:**

1. **Model Parallelization**: If multiple GPUs are available, the model is wrapped in `nn.DataParallel` for distributed training.
2. **Data Loading and Padding**: The dataset is loaded with associated features, which include input IDs, attention masks, and example indices.
3. **Model Evaluation**: The model is set to evaluation mode, and each batch of data is processed through the model to generate output logits.
4. **Scores Calculation**: The model's output logits are passed to a utility function (`compute_predictions`) to calculate log probabilities or logit scores based on the `model_type`.
5. **Post-processing**: If applicable (XLNet or XLM), additional post-processing steps are performed using specialized functions.

**Notable Dependencies and Requirements:**

1. `torch` for PyTorch
2. `transformers` for transformer-based models
3. `nn.DataParallel` for model parallelization
4. `SquadResult`, `SquadsPredictions`, and other utility classes from the `squad-eval` package

**Primary Output:**

The function returns a dictionary with evaluation results, including accuracy, precision, recall, F1 score, and exact scores.


key: chroma:document
string_value:
CODE:

def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
    if args.local_rank not in [-1, 0] and not evaluate:
        # Make sure only the first process in distributed training process the dataset, and the others will use the cache
        torch.distributed.barrier()

    # Load data features from cache or dataset file
    input_file = args.predict_file if evaluate else args.train_file
    cached_features_file = os.path.join(
        os.path.dirname(input_file),
        "cached_distillation_{}_{}_{}".format(
            "dev" if evaluate else "train",
            list(filter(None, args.model_name_or_path.split("/"))).pop(),
            str(args.max_seq_length),
        ),
    )
    if os.path.exists(cached_features_file) and not args.overwrite_cache:
        logger.info("Loading features from cached file %s", cached_features_file)
        features_and_dataset = torch.load(cached_features_file)

        try:
            features, dataset, examples = (
                features_and_dataset["features"],
                features_and_dataset["dataset"],
                features_and_dataset["examples"],
            )
        except KeyError:
            raise DeprecationWarning(
                "You seem to be loading features from an older version of this script please delete the "
                "file %s in order for it to be created again" % cached_features_file
            )
    else:
        logger.info("Creating features from dataset file at %s", input_file)
        processor = SquadV2Processor() if args.version_2_with_negative else SquadV1Processor()
        if evaluate:
            examples = processor.get_dev_examples(args.data_dir, filename=args.predict_file)
        else:
            examples = processor.get_train_examples(args.data_dir, filename=args.train_file)

        features, dataset = squad_convert_examples_to_features(
            examples=examples,
            tokenizer=tokenizer,
            max_seq_length=args.max_seq_length,
            doc_stride=args.doc_stride,
            max_query_length=args.max_query_length,
            is_training=not evaluate,
            return_dataset="pt",
            threads=args.threads,
        )

        if args.local_rank in [-1, 0]:
            logger.info("Saving features into cached file %s", cached_features_file)
            torch.save({"features": features, "dataset": dataset, "examples": examples}, cached_features_file)

    if args.local_rank == 0 and not evaluate:
        # Make sure only the first process in distributed training process the dataset, and the others will use the cache
        torch.distributed.barrier()

    if output_examples:
        return dataset, examples, features


SUMMARY:
**Function Summary: load_and_cache_examples**

**Main Purpose and Functionality**
This function loads and caches Examples for a Distillation task. It takes in input data (train or prediction file) and a tokenizer, and returns the cached features, dataset, and examples.

**Key Implementation Details**

* Uses `torch.distributed` to synchronize processes when loading and caching data.
* Checks if output_examples is True and returns additional cached data otherwise.
* Applies `squad_convert_examples_to_features` from the Hugging Face processor library to convert examples into features.

**Notable Dependencies or Requirements**

* `torch`: for distributed training and torch.save
* `tokenizer`: Hugging Face tokenizer, instantiated with `Squadr2Processor()` or `Squadr1Processor()`
* `squad_convert_examples_to_features`: from the Hugging Face processor library

**Interfaces**
Implements:
* `return_dataset="pt"` in `squad_convert_examples_to_features`

This summary captures the core functionality and dependencies of the provided function, providing a concise overview for a RAG system.


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()

    # Required parameters
    parser.add_argument(
        "--model_type",
        default=None,
        type=str,
        required=True,
        help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),
    )
    parser.add_argument(
        "--model_name_or_path",
        default=None,
        type=str,
        required=True,
        help="Path to pretrained model or model identifier from huggingface.co/models",
    )
    parser.add_argument(
        "--output_dir",
        default=None,
        type=str,
        required=True,
        help="The output directory where the model checkpoints and predictions will be written.",
    )

    # Distillation parameters (optional)
    parser.add_argument(
        "--teacher_type",
        default=None,
        type=str,
        help=(
            "Teacher type. Teacher tokenizer and student (model) tokenizer must output the same tokenization. Only for"
            " distillation."
        ),
    )
    parser.add_argument(
        "--teacher_name_or_path",
        default=None,
        type=str,
        help="Path to the already SQuAD fine-tuned teacher model. Only for distillation.",
    )
    parser.add_argument(
        "--alpha_ce", default=0.5, type=float, help="Distillation loss linear weight. Only for distillation."
    )
    parser.add_argument(
        "--alpha_squad", default=0.5, type=float, help="True SQuAD loss linear weight. Only for distillation."
    )
    parser.add_argument(
        "--temperature", default=2.0, type=float, help="Distillation temperature. Only for distillation."
    )

    # Other parameters
    parser.add_argument(
        "--data_dir",
        default=None,
        type=str,
        help="The input data dir. Should contain the .json files for the task."
        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
    )
    parser.add_argument(
        "--train_file",
        default=None,
        type=str,
        help="The input training file. If a data dir is specified, will look for the file there"
        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
    )
    parser.add_argument(
        "--predict_file",
        default=None,
        type=str,
        help="The input evaluation file. If a data dir is specified, will look for the file there"
        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
    )
    parser.add_argument(
        "--config_name", default="", type=str, help="Pretrained config name or path if not the same as model_name"
    )
    parser.add_argument(
        "--tokenizer_name",
        default="",
        type=str,
        help="Pretrained tokenizer name or path if not the same as model_name",
    )
    parser.add_argument(
        "--cache_dir",
        default="",
        type=str,
        help="Where do you want to store the pre-trained models downloaded from huggingface.co",
    )

    parser.add_argument(
        "--version_2_with_negative",
        action="store_true",
        help="If true, the SQuAD examples contain some that do not have an answer.",
    )
    parser.add_argument(
        "--null_score_diff_threshold",
        type=float,
        default=0.0,
        help="If null_score - best_non_null is greater than the threshold predict null.",
    )

    parser.add_argument(
        "--max_seq_length",
        default=384,
        type=int,
        help=(
            "The maximum total input sequence length after WordPiece tokenization. Sequences "
            "longer than this will be truncated, and sequences shorter than this will be padded."
        ),
    )
    parser.add_argument(
        "--doc_stride",
        default=128,
        type=int,
        help="When splitting up a long document into chunks, how much stride to take between chunks.",
    )
    parser.add_argument(
        "--max_query_length",
        default=64,
        type=int,
        help=(
            "The maximum number of tokens for the question. Questions longer than this will "
            "be truncated to this length."
        ),
    )
    parser.add_argument("--do_train", action="store_true", help="Whether to run training.")
    parser.add_argument("--do_eval", action="store_true", help="Whether to run eval on the dev set.")
    parser.add_argument(
        "--evaluate_during_training", action="store_true", help="Rul evaluation during training at each logging step."
    )
    parser.add_argument(
        "--do_lower_case", action="store_true", help="Set this flag if you are using an uncased model."
    )

    parser.add_argument("--per_gpu_train_batch_size", default=8, type=int, help="Batch size per GPU/CPU for training.")
    parser.add_argument(
        "--per_gpu_eval_batch_size", default=8, type=int, help="Batch size per GPU/CPU for evaluation."
    )
    parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
    parser.add_argument(
        "--gradient_accumulation_steps",
        type=int,
        default=1,
        help="Number of updates steps to accumulate before performing a backward/update pass.",
    )
    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
    parser.add_argument("--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer.")
    parser.add_argument("--max_grad_norm", default=1.0, type=float, help="Max gradient norm.")
    parser.add_argument(
        "--num_train_epochs", default=3.0, type=float, help="Total number of training epochs to perform."
    )
    parser.add_argument(
        "--max_steps",
        default=-1,
        type=int,
        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
    )
    parser.add_argument("--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps.")
    parser.add_argument(
        "--n_best_size",
        default=20,
        type=int,
        help="The total number of n-best predictions to generate in the nbest_predictions.json output file.",
    )
    parser.add_argument(
        "--max_answer_length",
        default=30,
        type=int,
        help=(
            "The maximum length of an answer that can be generated. This is needed because the start "
            "and end predictions are not conditioned on one another."
        ),
    )
    parser.add_argument(
        "--verbose_logging",
        action="store_true",
        help=(
            "If true, all of the warnings related to data processing will be printed. "
            "A number of warnings are expected for a normal SQuAD evaluation."
        ),
    )

    parser.add_argument("--logging_steps", type=int, default=50, help="Log every X updates steps.")
    parser.add_argument("--save_steps", type=int, default=50, help="Save checkpoint every X updates steps.")
    parser.add_argument(
        "--eval_all_checkpoints",
        action="store_true",
        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
    )
    parser.add_argument("--no_cuda", action="store_true", help="Whether not to use CUDA when available")
    parser.add_argument(
        "--overwrite_output_dir", action="store_true", help="Overwrite the content of the output directory"
    )
    parser.add_argument(
        "--overwrite_cache", action="store_true", help="Overwrite the cached training and evaluation sets"
    )
    parser.add_argument("--seed", type=int, default=42, help="random seed for initialization")

    parser.add_argument("--local_rank", type=int, default=-1, help="local_rank for distributed training on gpus")
    parser.add_argument(
        "--fp16",
        action="store_true",
        help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit",
    )
    parser.add_argument(
        "--fp16_opt_level",
        type=str,
        default="O1",
        help=(
            "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']. "
            "See details at https://nvidia.github.io/apex/amp.html"
        ),
    )
    parser.add_argument("--server_ip", type=str, default="", help="Can be used for distant debugging.")
    parser.add_argument("--server_port", type=str, default="", help="Can be used for distant debugging.")

    parser.add_argument("--threads", type=int, default=1, help="multiple threads for converting example to features")
    args = parser.parse_args()

    if (
        os.path.exists(args.output_dir)
        and os.listdir(args.output_dir)
        and args.do_train
        and not args.overwrite_output_dir
    ):
        raise ValueError(
            "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
                args.output_dir
            )
        )

    # Setup distant debugging if needed
    if args.server_ip and args.server_port:
        # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
        import ptvsd

        print("Waiting for debugger attach")
        ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
        ptvsd.wait_for_attach()

    # Setup CUDA, GPU & distributed training
    if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
        args.n_gpu = 0 if args.no_cuda else torch.cuda.device_count()
    else:  # Initializes the distributed backend which will take care of synchronizing nodes/GPUs
        torch.cuda.set_device(args.local_rank)
        device = torch.device("cuda", args.local_rank)
        torch.distributed.init_process_group(backend="nccl")
        args.n_gpu = 1
    args.device = device

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.INFO if args.local_rank in [-1, 0] else logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        args.local_rank,
        device,
        args.n_gpu,
        bool(args.local_rank != -1),
        args.fp16,
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    # Set seed
    set_seed(args)

    # Load pretrained model and tokenizer
    if args.local_rank not in [-1, 0]:
        # Make sure only the first process in distributed training will download model & vocab
        torch.distributed.barrier()

    args.model_type = args.model_type.lower()
    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
    config = config_class.from_pretrained(
        args.config_name if args.config_name else args.model_name_or_path,
        cache_dir=args.cache_dir if args.cache_dir else None,
    )
    tokenizer = tokenizer_class.from_pretrained(
        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
        do_lower_case=args.do_lower_case,
        cache_dir=args.cache_dir if args.cache_dir else None,
    )
    model = model_class.from_pretrained(
        args.model_name_or_path,
        from_tf=bool(".ckpt" in args.model_name_or_path),
        config=config,
        cache_dir=args.cache_dir if args.cache_dir else None,
    )

    if args.teacher_type is not None:
        assert args.teacher_name_or_path is not None
        assert args.alpha_ce > 0.0
        assert args.alpha_ce + args.alpha_squad > 0.0
        assert args.teacher_type != "distilbert", "We constraint teachers not to be of type DistilBERT."
        teacher_config_class, teacher_model_class, _ = MODEL_CLASSES[args.teacher_type]
        teacher_config = teacher_config_class.from_pretrained(
            args.teacher_name_or_path, cache_dir=args.cache_dir if args.cache_dir else None
        )
        teacher = teacher_model_class.from_pretrained(
            args.teacher_name_or_path, config=teacher_config, cache_dir=args.cache_dir if args.cache_dir else None
        )
        teacher.to(args.device)
    else:
        teacher = None

    if args.local_rank == 0:
        # Make sure only the first process in distributed training will download model & vocab
        torch.distributed.barrier()

    model.to(args.device)

    logger.info("Training/evaluation parameters %s", args)

    # Before we do anything with models, we want to ensure that we get fp16 execution of torch.einsum if args.fp16 is set.
    # Otherwise it'll default to "promote" mode, and we'll get fp32 operations. Note that running `--fp16_opt_level="O2"` will
    # remove the need for this code, but it is still valid.
    if args.fp16:
        try:
            import apex

            apex.amp.register_half_function(torch, "einsum")
        except ImportError:
            raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")

    # Training
    if args.do_train:
        train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
        global_step, tr_loss = train(args, train_dataset, model, tokenizer, teacher=teacher)
        logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)

    # Save the trained model and the tokenizer
    if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
        logger.info("Saving model checkpoint to %s", args.output_dir)
        # Save a trained model, configuration and tokenizer using `save_pretrained()`.
        # They can then be reloaded using `from_pretrained()`
        model_to_save = (
            model.module if hasattr(model, "module") else model
        )  # Take care of distributed/parallel training
        model_to_save.save_pretrained(args.output_dir)
        tokenizer.save_pretrained(args.output_dir)

        # Good practice: save your training arguments together with the trained model
        torch.save(args, os.path.join(args.output_dir, "training_args.bin"))

        # Load a trained model and vocabulary that you have fine-tuned
        model = model_class.from_pretrained(args.output_dir)
        tokenizer = tokenizer_class.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
        model.to(args.device)

    # Evaluation - we can ask to evaluate all the checkpoints (sub-directories) in a directory
    results = {}
    if args.do_eval and args.local_rank in [-1, 0]:
        if args.do_train:
            logger.info("Loading checkpoints saved during training for evaluation")
        checkpoints = [args.output_dir]
        if args.eval_all_checkpoints:
            checkpoints = [
                os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + "/**/" + WEIGHTS_NAME, recursive=True))
            ]

        logger.info("Evaluate the following checkpoints: %s", checkpoints)

        for checkpoint in checkpoints:
            # Reload the model
            global_step = checkpoint.split("-")[-1] if len(checkpoints) > 1 else ""
            model = model_class.from_pretrained(checkpoint)
            model.to(args.device)

            # Evaluate
            result = evaluate(args, model, tokenizer, prefix=global_step)

            result = {k + ("_{}".format(global_step) if global_step else ""): v for k, v in result.items()}
            results.update(result)

    logger.info("Results: {}".format(results))



SUMMARY:
Here is a concise summary of the code:

This script trains a deep learning model on a specified dataset using a training method. The script performs the following steps:

1. Initializes the training parameters, such as the device to use and the batch size.
2. Loads the dataset and tokens (vocabulary) for tokenization.
3. Preprocesses the data and creates a training dataset.
4. Trains the model on the training dataset using a specified training method (e.g., stochastic gradient descent).
5. Evaluates the trained model on a validation dataset.
6. Saves the trained model, tokenizer, and training parameters to a directory.

The script uses various libraries such as PyTorch, Hugging Face Transformers, and Apex for mixed precision training.

Note: The script appears to be part of a larger natural language processing project, specifically focused on question answering and passage classification tasks.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/distillation/grouped_batch_sampler.py


key: chroma:document
string_value:
CODE:

def _quantize(x, bins):
    bins = copy.deepcopy(bins)
    bins = sorted(bins)
    quantized = [bisect.bisect_right(bins, y) for y in x]


SUMMARY:
**Summary:**

`_quantize` function takes a list `x` and `bins` as input. It:

* Copies and sorts the input `bins`
* Uses `bisect.bisect_right` to find indices for each element in `x` relative to sorted `bins`, resulting in a new list `quantized`.

**Key Details:**

* Method signature: `_quantize(x, bins)`
* Used imports: `copy`, `bisect`

Note: The function seems to implement binning or quantization of values.


key: chroma:document
string_value:
CODE:

def create_lengths_groups(lengths, k=0):
    bins = np.arange(start=3, stop=k, step=4).tolist() if k > 0 else [10]
    groups = _quantize(lengths, bins)
    # count number of elements per group
    counts = np.unique(groups, return_counts=True)[1]
    fbins = [0] + bins + [np.inf]
    logger.info("Using {} as bins for aspect lengths quantization".format(fbins))
    logger.info("Count of instances per bin: {}".format(counts))


SUMMARY:
**Function Summary**

`create_lengths_groups`
================--------

**Purpose**: Quantizes input lengths into discrete groups using `_quantize`.

**Parameters**:

* `lengths`: Input array of lengths to be quantized
* `k`: Optional parameter specifying the upper limit for bins (default: 0)

**Implementation Details**:

* Uses numpy's `_quantize` function with custom bins generated from `np.arange`.
* Returns the quantized groups without their counts.

**Key Dependencies/Requirements**:

* `_quantize` function from numpy
* Logger object for logging debug messages

**Return Value**: Quantified groups represented as a list.


key: chroma:document
string_value:
CODE:

class GroupedBatchSampler(BatchSampler):
    """
    Wraps another sampler to yield a mini-batch of indices.
    It enforces that the batch only contain elements from the same group.
    It also tries to provide mini-batches which follows an ordering which is
    as close as possible to the ordering from the original sampler.
    Arguments:
        sampler (Sampler): Base sampler.
        group_ids (list[int]): If the sampler produces indices in range [0, N),
            `group_ids` must be a list of `N` ints which contains the group id of each sample.
            The group ids must be a continuous set of integers starting from
            0, i.e. they must be in the range [0, num_groups).
        batch_size (int): Size of mini-batch.
    """

    def __init__(self, sampler, group_ids, batch_size):
        if not isinstance(sampler, Sampler):
            raise TypeError(
                "sampler should be an instance of torch.utils.data.Sampler, but got sampler={}".format(sampler)
            )
        self.sampler = sampler
        self.group_ids = group_ids
        self.batch_size = batch_size

    def __iter__(self):
        buffer_per_group = defaultdict(list)
        samples_per_group = defaultdict(list)

        num_batches = 0
        for idx in self.sampler:
            group_id = self.group_ids[idx]
            buffer_per_group[group_id].append(idx)
            samples_per_group[group_id].append(idx)
            if len(buffer_per_group[group_id]) == self.batch_size:
                yield buffer_per_group[group_id]  # TODO
                num_batches += 1
                del buffer_per_group[group_id]
            assert len(buffer_per_group[group_id]) < self.batch_size

        # now we have run out of elements that satisfy
        # the group criteria, let's return the remaining
        # elements so that the size of the sampler is
        # deterministic
        expected_num_batches = len(self)
        num_remaining = expected_num_batches - num_batches
        if num_remaining > 0:
            # for the remaining batches, group the batches by similar lengths
            batch_idx = []
            for group_id, idxs in sorted(buffer_per_group.items(), key=lambda x: x[0]):
                batch_idx.extend(idxs)
                if len(batch_idx) >= self.batch_size:
                    yield batch_idx[: self.batch_size]
                    batch_idx = batch_idx[self.batch_size :]
                    num_remaining -= 1
            if len(batch_idx) > 0:
                yield batch_idx
                num_remaining -= 1
        assert num_remaining == 0

    def __len__(self):
        """
        Return the number of mini-batches rather than the number of samples.
        """


SUMMARY:
**GroupedBatchSampler Class Summary**

### Purpose and Functionality

The `GroupedBatchSampler` class is a custom sampler that wraps another sampler to yield mini-batches of indices. It ensures that each batch only contains elements from the same group, while attempting to maintain an ordering similar to the original sampler.

*   **Arguments:**
    *   `sampler`: Base sampler instance
    *   `group_ids`: List of integers representing group IDs for each sample
    *   `batch_size`: Size of mini-batch

### Key Implementation Details

*   The sampler uses a dictionary to buffer indices by their corresponding group IDs.
*   When the buffer reaches the desired batch size, it yields the buffered indices as a new mini-batch.
*   If all elements have been processed, remaining indices are grouped by similar lengths and yielded in batches.

### Notable Dependencies and Requirements

*   `torch.utils.data.Sampler`:
    *   The base sampler instance must be an instance of `torch.utils.data.Sampler`.
*   `defaultdict`:
    *   Used to create a dictionary-like object with default values.

### External Dependencies

None mentioned in the provided code; however, it implicitly relies on other PyTorch modules, such as:

*   `torch`
*   `torch.utils.data`

### Implemented Interfaces

*   `__iter__`: Yields mini-batches of indices.
*   `__len__`: Returns the number of mini-batches.

### Design Choices

The implementation uses a buffering approach to group indices by their corresponding group IDs, which facilitates efficient yield of mini-batches while maintaining an ordering similar to the original sampler.


key: chroma:document
string_value:
CODE:

def git_log(folder_path: str):
    """
    Log commit info.
    """
    repo = git.Repo(search_parent_directories=True)
    repo_infos = {
        "repo_id": str(repo),
        "repo_sha": str(repo.head.object.hexsha),
        "repo_branch": str(repo.active_branch),
    }

    with open(os.path.join(folder_path, "git_log.json"), "w") as f:


SUMMARY:
**Summary:**

`git_log` function generates JSON file containing Git repository information. It initializes a Git repository object using `pathlib.githubRepo` and extracts relevant metadata.

**Key Details:**

*   **Function Signature:** `def git_log(folder_path: str)`
*   **Primary Purpose:** Logs commit info to a specified folder.
*   **Implementation Details:**
    *   Initializes Git repository object using `git.Repo`.
    *   Extracts: Repository ID (`repo_id`), SHA (`repo_sha`), and branch information (`repo_branch`)
*   **Notable Dependencies/Requirements:**
    *   `git` library for interacting with repositories.
    *   `pathlib.githubRepo` is used, not `os.path.join`.
    *   `json.dump()` for serializing repository info to a file.

**Example Usage:**

```python
git_log("/path/to/project/folder")
```

This summary should be included in the RAG system to provide developers with concise context about this code functionality.


key: chroma:document
string_value:
CODE:

def init_gpu_params(params):
    """
    Handle single and multi-GPU / multi-node.
    """
    if params.n_gpu <= 0:
        params.local_rank = 0
        params.master_port = -1
        params.is_master = True
        params.multi_gpu = False
        return

    assert torch.cuda.is_available()

    logger.info("Initializing GPUs")
    if params.n_gpu > 1:
        assert params.local_rank != -1

        params.world_size = int(os.environ["WORLD_SIZE"])
        params.n_gpu_per_node = int(os.environ["N_GPU_NODE"])
        params.global_rank = int(os.environ["RANK"])

        # number of nodes / node ID
        params.n_nodes = params.world_size // params.n_gpu_per_node
        params.node_id = params.global_rank // params.n_gpu_per_node
        params.multi_gpu = True

        assert params.n_nodes == int(os.environ["N_NODES"])
        assert params.node_id == int(os.environ["NODE_RANK"])

    # local job (single GPU)
    else:
        assert params.local_rank == -1

        params.n_nodes = 1
        params.node_id = 0
        params.local_rank = 0
        params.global_rank = 0
        params.world_size = 1
        params.n_gpu_per_node = 1
        params.multi_gpu = False

    # sanity checks
    assert params.n_nodes >= 1
    assert 0 <= params.node_id < params.n_nodes
    assert 0 <= params.local_rank <= params.global_rank < params.world_size
    assert params.world_size == params.n_nodes * params.n_gpu_per_node

    # define whether this is the master process / if we are in multi-node distributed mode
    params.is_master = params.node_id == 0 and params.local_rank == 0
    params.multi_node = params.n_nodes > 1

    # summary
    PREFIX = f"--- Global rank: {params.global_rank} - "
    logger.info(PREFIX + "Number of nodes: %i" % params.n_nodes)
    logger.info(PREFIX + "Node ID        : %i" % params.node_id)
    logger.info(PREFIX + "Local rank     : %i" % params.local_rank)
    logger.info(PREFIX + "World size     : %i" % params.world_size)
    logger.info(PREFIX + "GPUs per node  : %i" % params.n_gpu_per_node)
    logger.info(PREFIX + "Master         : %s" % str(params.is_master))
    logger.info(PREFIX + "Multi-node     : %s" % str(params.multi_node))
    logger.info(PREFIX + "Multi-GPU      : %s" % str(params.multi_gpu))
    logger.info(PREFIX + "Hostname       : %s" % socket.gethostname())

    # set GPU device
    torch.cuda.set_device(params.local_rank)

    # initialize multi-GPU
    if params.multi_gpu:
        logger.info("Initializing PyTorch distributed")
        torch.distributed.init_process_group(
            init_method="env://",
            backend="nccl",


SUMMARY:
**Function Summary: `init_gpu_params`**

Initializes parameters for GPU operations, handling single and multi-GPU/multi-node configurations.

**Main Purpose:** Sets up environment variables and PyTorch distributed functionality for various GPU configurations.

**Key Implementation Details:**

*   Handles cases for single-GPU and multi-GPU/multi-node setups.
*   Parses environment variables (`WORLD_SIZE`, `N_GPU_NODE`, etc.) to determine node and GPU information.
*   Initializes PyTorch distributed process group using the `torch.distributed.init_process_group` function.

**Notable Dependencies/Requirements:**

*   Torch library (`torch`)
*   Os library (`os`)
*   PyTorch Distributed library (`torch.distributed`)

Assumptions:
- The environment variables `$WORLD_SIZE`, `$N_GPU_NODE`, and others are set before calling the `init_gpu_params` function.
- The PyTorch Distributed library is installed and imported.


key: chroma:document
string_value:
CODE:

def set_seed(args):
    """
    Set the random seed.
    """
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    if args.n_gpu > 0:


SUMMARY:
**Function Summary: `set_seed`**

**Purpose and Functionality:**
Sets the random seed for various libraries (NumPy, PyTorch) to ensure reproducibility of results.

**Key Implementation Details:**

* Sets NumPy random seed with `np.random.seed(args.seed)`
* Sets PyTorch manual seed with `torch.manual_seed(args.seed)`
* Optionally sets PyTorch CUDA manual seed for devices (if `n_gpu > 0`)

**Notable Dependencies and Requirements:**
No external dependencies, uses only:
* `args`: input argument object
* `np.random`: NumPy random library
* `torch`: PyTorch library


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/distillation/scripts/binarized_data.py


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser(
        description="Preprocess the data to avoid re-doing it several times by (tokenization + token_to_ids)."
    )
    parser.add_argument("--file_path", type=str, default="data/dump.txt", help="The path to the data.")
    parser.add_argument("--tokenizer_type", type=str, default="bert", choices=["bert", "roberta", "gpt2"])
    parser.add_argument("--tokenizer_name", type=str, default="bert-base-uncased", help="The tokenizer to use.")
    parser.add_argument("--dump_file", type=str, default="data/dump", help="The dump file prefix.")
    args = parser.parse_args()

    logger.info(f"Loading Tokenizer ({args.tokenizer_name})")
    if args.tokenizer_type == "bert":
        tokenizer = BertTokenizer.from_pretrained(args.tokenizer_name)
        bos = tokenizer.special_tokens_map["cls_token"]  # `[CLS]`
        sep = tokenizer.special_tokens_map["sep_token"]  # `[SEP]`
    elif args.tokenizer_type == "roberta":
        tokenizer = RobertaTokenizer.from_pretrained(args.tokenizer_name)
        bos = tokenizer.special_tokens_map["cls_token"]  # `<s>`
        sep = tokenizer.special_tokens_map["sep_token"]  # `</s>`
    elif args.tokenizer_type == "gpt2":
        tokenizer = GPT2Tokenizer.from_pretrained(args.tokenizer_name)
        bos = tokenizer.special_tokens_map["bos_token"]  # `<|endoftext|>`
        sep = tokenizer.special_tokens_map["eos_token"]  # `<|endoftext|>`

    logger.info(f"Loading text from {args.file_path}")
    with open(args.file_path, "r", encoding="utf8") as fp:
        data = fp.readlines()

    logger.info("Start encoding")
    logger.info(f"{len(data)} examples to process.")

    rslt = []
    iter = 0
    interval = 10000
    start = time.time()
    for text in data:
        text = f"{bos} {text.strip()} {sep}"
        token_ids = tokenizer.encode(text, add_special_tokens=False)
        rslt.append(token_ids)

        iter += 1
        if iter % interval == 0:
            end = time.time()
            logger.info(f"{iter} examples processed. - {(end-start):.2f}s/{interval}expl")
            start = time.time()
    logger.info("Finished binarization")
    logger.info(f"{len(data)} examples processed.")

    dp_file = f"{args.dump_file}.{args.tokenizer_name}.pickle"
    vocab_size = tokenizer.vocab_size
    if vocab_size < (1 << 16):
        rslt_ = [np.uint16(d) for d in rslt]
    else:
        rslt_ = [np.int32(d) for d in rslt]
    random.shuffle(rslt_)
    logger.info(f"Dump to {dp_file}")
    with open(dp_file, "wb") as handle:


SUMMARY:
**Summary: BERT-like Text Preprocessing Functionality**

The `main` function is responsible for preprocessed text data using a chosen tokenizer type (BERT, RoBERTa, or GPT2). The functionality involves:

### **Main Purpose and Function**

* Load data from a specified file path
* Tokenize text using the selected tokenizer
* Encode tokens to generate binary IDs
* Save encoded data to a pickle file

### **Key Implementation Details**

* Uses `argparse` for input argument parsing
* Selects and loads tokenizer instance based on `--tokenizer_type`
* Utilizes `BertTokenizer`, `RobertaTokenizer`, or `GPT2Tokenizer` depending on the chosen type
* Applies special token handling ( `[CLS]`, `[SEP]`, `<s>`, `</s>` ) for each tokenizer type

### **Notable Dependencies and Requirements**

* `argparse`
* Selected Hugging Face tokenizer library (`BertTokenizer`, `RobertaTokenizer`, or `GPT2Tokenizer`)
* `pickle` library for data serialization

### **Method Signatures and Interfaces**

* None notable, but includes:
	+ `tokenizer.encode(text, add_special_tokens=False)` to encode tokens
	+ `np.uint16() / np.int32()` functions for data type conversion


key: chroma:document
string_value:
CODE:

class Wav2Vec2Aligner:
    def __init__(self, model_name, input_wavs_sr, cuda):
        self.cuda = cuda
        self.config = AutoConfig.from_pretrained(model_name)
        self.model = AutoModelForCTC.from_pretrained(model_name)
        self.model.eval()
        if self.cuda:
            self.model.to(device="cuda")
        self.processor = AutoProcessor.from_pretrained(model_name)
        self.resampler = torchaudio.transforms.Resample(input_wavs_sr, 16_000)
        blank_id = 0
        vocab = list(self.processor.tokenizer.get_vocab().keys())
        for i in range(len(vocab)):
            if vocab[i] == "[PAD]" or vocab[i] == "<pad>":
                blank_id = i
        print("Blank Token id [PAD]/<pad>", blank_id)
        self.blank_id = blank_id

    def speech_file_to_array_fn(self, wav_path):
        speech_array, sampling_rate = torchaudio.load(wav_path)
        speech = self.resampler(speech_array).squeeze().numpy()
        return speech

    def align_single_sample(self, item):
        blank_id = self.blank_id
        transcript = "|".join(item["sent"].split(" "))
        if not os.path.isfile(item["wav_path"]):
            print(item["wav_path"], "not found in wavs directory")

        speech_array = self.speech_file_to_array_fn(item["wav_path"])
        inputs = self.processor(speech_array, sampling_rate=16_000, return_tensors="pt", padding=True)
        if self.cuda:
            inputs = inputs.to(device="cuda")

        with torch.no_grad():
            logits = self.model(inputs.input_values).logits

        # get the emission probability at frame level
        emissions = torch.log_softmax(logits, dim=-1)
        emission = emissions[0].cpu().detach()

        # get labels from vocab
        labels = ([""] + list(self.processor.tokenizer.get_vocab().keys()))[
            :-1
        ]  # logits don't align with the tokenizer's vocab

        dictionary = {c: i for i, c in enumerate(labels)}
        tokens = []
        for c in transcript:
            if c in dictionary:
                tokens.append(dictionary[c])

        def get_trellis(emission, tokens, blank_id=0):
            """
            Build a trellis matrix of shape (num_frames + 1, num_tokens + 1)
            that represents the probabilities of each source token being at a certain time step
            """
            num_frames = emission.size(0)
            num_tokens = len(tokens)

            # Trellis has extra diemsions for both time axis and tokens.
            # The extra dim for tokens represents <SoS> (start-of-sentence)
            # The extra dim for time axis is for simplification of the code.
            trellis = torch.full((num_frames + 1, num_tokens + 1), -float("inf"))
            trellis[:, 0] = 0
            for t in range(num_frames):
                trellis[t + 1, 1:] = torch.maximum(
                    # Score for staying at the same token
                    trellis[t, 1:] + emission[t, blank_id],
                    # Score for changing to the next token
                    trellis[t, :-1] + emission[t, tokens],
                )
            return trellis

        trellis = get_trellis(emission, tokens, blank_id)

        @dataclass
        class Point:
            token_index: int
            time_index: int
            score: float

        def backtrack(trellis, emission, tokens, blank_id=0):
            """
            Walk backwards from the last (sentence_token, time_step) pair to build the optimal sequence alignment path
            """
            # Note:
            # j and t are indices for trellis, which has extra dimensions
            # for time and tokens at the beginning.
            # When referring to time frame index `T` in trellis,
            # the corresponding index in emission is `T-1`.
            # Similarly, when referring to token index `J` in trellis,
            # the corresponding index in transcript is `J-1`.
            j = trellis.size(1) - 1
            t_start = torch.argmax(trellis[:, j]).item()

            path = []
            for t in range(t_start, 0, -1):
                # 1. Figure out if the current position was stay or change
                # Note (again):
                # `emission[J-1]` is the emission at time frame `J` of trellis dimension.
                # Score for token staying the same from time frame J-1 to T.
                stayed = trellis[t - 1, j] + emission[t - 1, blank_id]
                # Score for token changing from C-1 at T-1 to J at T.
                changed = trellis[t - 1, j - 1] + emission[t - 1, tokens[j - 1]]

                # 2. Store the path with frame-wise probability.
                prob = emission[t - 1, tokens[j - 1] if changed > stayed else 0].exp().item()
                # Return token index and time index in non-trellis coordinate.
                path.append(Point(j - 1, t - 1, prob))

                # 3. Update the token
                if changed > stayed:
                    j -= 1
                    if j == 0:
                        break
            else:
                raise ValueError("Failed to align")
            return path[::-1]

        path = backtrack(trellis, emission, tokens, blank_id)

        @dataclass
        class Segment:
            label: str
            start: int
            end: int
            score: float

            def __repr__(self):
                return f"{self.label}\t{self.score:4.2f}\t{self.start*20:5d}\t{self.end*20:5d}"

            @property
            def length(self):
                return self.end - self.start

        def merge_repeats(path):
            """
            Merge repeated tokens into a single segment. Note: this shouldn't affect repeated characters from the
            original sentences (e.g. `ll` in `hello`)
            """
            i1, i2 = 0, 0
            segments = []
            while i1 < len(path):
                while i2 < len(path) and path[i1].token_index == path[i2].token_index:
                    i2 += 1
                score = sum(path[k].score for k in range(i1, i2)) / (i2 - i1)
                segments.append(
                    Segment(
                        transcript[path[i1].token_index],
                        path[i1].time_index,
                        path[i2 - 1].time_index + 1,
                        score,
                    )
                )
                i1 = i2
            return segments

        segments = merge_repeats(path)
        with open(item["out_path"], "w") as out_align:
            for seg in segments:
                out_align.write(str(seg) + "\n")

    def align_data(self, wav_dir, text_file, output_dir):
        if not os.path.exists(output_dir):
            os.makedirs(output_dir)

        # load text file
        lines = open(text_file, encoding="utf8").readlines()

        items = []
        for line in lines:
            if len(line.strip().split("\t")) != 2:
                print("Script must be in format: 00001  this is my sentence")
                exit()

            wav_name, sentence = line.strip().split("\t")
            wav_path = os.path.join(wav_dir, wav_name + ".wav")
            out_path = os.path.join(output_dir, wav_name + ".txt")

            items.append({"sent": sentence, "wav_path": wav_path, "out_path": out_path})
        print("Number of samples found in script file", len(items))

        for item in tqdm(items):


SUMMARY:
**Summary: Wav2Vec2Aligner Class**

The `Wav2Vec2Aligner` class is designed to perform speech alignment using the Wav2Vec2 model. The main purpose of this class is to align speech recordings with their corresponding text transcripts.

* **Initialization**: The class takes three parameters: `model_name`, `input_wavs_sr`, and `cuda`. It initializes the model, processor, and resampler with the given parameters.
* **speech_file_to_array_fn**: This method loads a speech file into a PyTorch tensor using `torchaudio`.
* **align_single_sample**: This method performs the alignment of a single sample. It:
	+ Loads the speech data and computes its embedding
	+ Computes the trellis array using the computed embedding
	+ Backtracks through the trellis array to find the most likely alignment path
	+ Returns the aligned segments as a list of `Segment` objects
* **align_data**: This method loads a text file containing speech alignments and aligns each line's corresponding speech recording.

**Key Methods and Data Structures**

* `AlignSingleSample`
* `Segment`: Represents an aligned segment with label, start index, end index, score, length

**Output**

The class writes the aligned segments to files in the specified output directory.

**Notable Aspects**

* Uses PyTorch for tensor computations
* Utilizes Wav2Vec2 model architecture for speech embedding
* Performs backtracking through trellis array to find most likely alignment path


key: chroma:document
string_value:
CODE:

def main():
    parser = argparse.ArgumentParser()

    parser.add_argument(
        "--model_name", type=str, default="arijitx/wav2vec2-xls-r-300m-bengali", help="wav2vec model name"
    )
    parser.add_argument("--wav_dir", type=str, default="./wavs", help="directory containing wavs")
    parser.add_argument("--text_file", type=str, default="script.txt", help="file containing text")
    parser.add_argument("--input_wavs_sr", type=int, default=16000, help="sampling rate of input audios")
    parser.add_argument(
        "--output_dir", type=str, default="./out_alignment", help="output directory containing the alignment files"
    )
    parser.add_argument("--cuda", action="store_true")

    args = parser.parse_args()

    aligner = Wav2Vec2Aligner(args.model_name, args.input_wavs_sr, args.cuda)


SUMMARY:
**Summary:**

This `main` function configures an ArgParser to parse command-line arguments for wav2vec audio alignment. It:

1.  Defines input parameters:
    *   `model_name`: Path to the wav2vec model (default).
    *   `wav_dir`: Directory containing input audio files (default).
    *   `text_file`: File containing related text data (default).
    *   `input_wavs_sr`: Input audio sampling rate (default).
    *   `output_dir`: Output directory for alignment files (default).
2.  Specifies an optional CUDA flag.
3.  Parses the command-line arguments using `argparse`.
4.  Creates a `Wav2Vec2Aligner` instance with the parsed model name, input audio sampling rate, and CUDA status.
5.  Aligns the data by calling `align_data` method of the `Wav2Vec2Aligner` instance.

**Key Implementation Details:**

*   Uses `argparse.ArgumentParser` to define a command-line interface.
*   Includes validation for input parameters using default values.

**Notable Dependencies/Requirements:**

*   `Wav2Vec2Aligner` class from the specified module (`arijitx/wav2vec2-xls-r-300m-bengali`).
*   External dependency on OpenVINO, PyTorch, or other deep learning frameworks ( implicit in the `Wav2Vec2Aligner` import).


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/wav2vec2/test_wav2vec2_deepspeed.py


key: chroma:document
string_value:
CODE:

def custom_name_func(func, param_num, param):
    # customize the test name generator function as we want both params to appear in the sub-test
    # name, as by default it shows only the first param
    param_based_name = parameterized.to_safe_name("_".join(str(x) for x in param.args))


SUMMARY:
**Summary**

Customizes a test name generator function (`parameterized.to_safe_name`) to include both parameter values (`param`), while maintaining the original function's (`func`) name.

* Function signature: `custom_name_func(func, param_num, param)`
* Primary purpose: Customizes test names to include multiple parameters.
* Implementation details:
	+ Concatenates parameter args with `_` using `to_safe_name`.
	+ Joins string representations of arguments.

**Notable implementation**

* Uses `parameterized.to_safe_name` with a custom argument concatenation approach.
* Returns a formatted string name combining the original function's name (`func.__name__`) and customized test name (including both parameters).


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/wav2vec2/run_common_voice.py


key: chroma:document
string_value:
CODE:

def list_field(default=None, metadata=None):


SUMMARY:
**Code Summary**

`list_field` function creates a Django model field with optional default value and metadata.

* **Purpose**: Creates a reusable model field factory with default value and metadata settings.
* **Key Details**:
	+ Initializes field with default `default_factory` lambda expression, which returns the provided `default` attribute.
	+ Allows for custom metadata by passing `metadata` keyword argument.
* **Notable Dependencies/Requirements**:
	+ Django framework (`field` function)
	+ Python's built-in `lambda` function
	+ No external dependencies listed


key: chroma:document
string_value:
CODE:

class CTCTrainer(Trainer):
    def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) -> torch.Tensor:
        """
        Perform a training step on a batch of inputs.

        Subclass and override to inject custom behavior.

        Args:
            model (:obj:`nn.Module`):
                The model to train.
            inputs (:obj:`Dict[str, Union[torch.Tensor, Any]]`):
                The inputs and targets of the model.

                The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
                argument :obj:`labels`. Check your model's documentation for all accepted arguments.

        Return:
            :obj:`torch.Tensor`: The tensor with training loss on this batch.
        """

        model.train()
        inputs = self._prepare_inputs(inputs)

        if self.use_amp:
            with autocast():
                loss = self.compute_loss(model, inputs)
        else:
            loss = self.compute_loss(model, inputs)

        if self.args.n_gpu > 1:
            if model.module.config.ctc_loss_reduction == "mean":
                loss = loss.mean()
            elif model.module.config.ctc_loss_reduction == "sum":
                loss = loss.sum() / (inputs["labels"] >= 0).sum()
            else:
                raise ValueError(f"{model.config.ctc_loss_reduction} is not valid. Choose one of ['mean', 'sum']")

        if self.args.gradient_accumulation_steps > 1:
            loss = loss / self.args.gradient_accumulation_steps

        if self.use_amp:
            self.scaler.scale(loss).backward()
        elif self.use_apex:
            with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                scaled_loss.backward()
        elif self.deepspeed:
            self.deepspeed.backward(loss)
        else:
            loss.backward()



SUMMARY:
**CTCTrainer Class Summary**

*   **Main Purpose and Functionality**: This class is designed for training of neural network models using the CTC (conditional random field) loss function. It inherits from the `Trainer` base class.

    ```text
CTCTrainer: Trains a model using CTC loss.
```

*   **Key Implementation Details**:

    ```text
- Customizable loss computation and reduction strategy via model configuration.
- Supports gradient accumulation, automatic mixed precision training (AMP), Apex, and DeepSpeed optimization algorithms.
```
    
*   **Notable Dependencies or Requirements**:

    ```text
- `Trainer` base class
- PyTorch module (`nn.Module`)
- Dynamic computation graph libraries (e.g., TorchAutoCast)
```

    List of dependencies:
    - TorchAutoCast
    - PyTorch ( nn.Module )
    - Trainers (base class)


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Detecting last checkpoint.
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
        + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
    logger.info("Training/evaluation parameters %s", training_args)

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Get the datasets:
    train_dataset = datasets.load_dataset(
        "common_voice", data_args.dataset_config_name, split=data_args.train_split_name
    )
    eval_dataset = datasets.load_dataset("common_voice", data_args.dataset_config_name, split="test")

    # Create and save tokenizer
    chars_to_ignore_regex = f'[{"".join(data_args.chars_to_ignore)}]'

    def remove_special_characters(batch):
        batch["text"] = re.sub(chars_to_ignore_regex, "", batch["sentence"]).lower() + " "
        return batch

    train_dataset = train_dataset.map(remove_special_characters, remove_columns=["sentence"])
    eval_dataset = eval_dataset.map(remove_special_characters, remove_columns=["sentence"])

    def extract_all_chars(batch):
        all_text = " ".join(batch["text"])
        vocab = list(set(all_text))
        return {"vocab": [vocab], "all_text": [all_text]}

    vocab_train = train_dataset.map(
        extract_all_chars,
        batched=True,
        batch_size=-1,
        keep_in_memory=True,
        remove_columns=train_dataset.column_names,
    )
    vocab_test = train_dataset.map(
        extract_all_chars,
        batched=True,
        batch_size=-1,
        keep_in_memory=True,
        remove_columns=eval_dataset.column_names,
    )

    vocab_list = list(set(vocab_train["vocab"][0]) | set(vocab_test["vocab"][0]))
    vocab_dict = {v: k for k, v in enumerate(vocab_list)}
    vocab_dict["|"] = vocab_dict[" "]
    del vocab_dict[" "]
    vocab_dict["[UNK]"] = len(vocab_dict)
    vocab_dict["[PAD]"] = len(vocab_dict)

    with open("vocab.json", "w") as vocab_file:
        json.dump(vocab_dict, vocab_file)

    # Load pretrained model and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    tokenizer = Wav2Vec2CTCTokenizer(
        "vocab.json",
        unk_token="[UNK]",
        pad_token="[PAD]",
        word_delimiter_token="|",
    )
    feature_extractor = Wav2Vec2FeatureExtractor(
        feature_size=1, sampling_rate=16_000, padding_value=0.0, do_normalize=True, return_attention_mask=True
    )
    processor = Wav2Vec2Processor(feature_extractor=feature_extractor, tokenizer=tokenizer)
    model = Wav2Vec2ForCTC.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        activation_dropout=model_args.activation_dropout,
        attention_dropout=model_args.attention_dropout,
        hidden_dropout=model_args.hidden_dropout,
        feat_proj_dropout=model_args.feat_proj_dropout,
        mask_time_prob=model_args.mask_time_prob,
        gradient_checkpointing=training_args.gradient_checkpointing,
        layerdrop=model_args.layerdrop,
        ctc_loss_reduction="mean",
        pad_token_id=processor.tokenizer.pad_token_id,
        vocab_size=len(processor.tokenizer),
    )

    if data_args.max_train_samples is not None:
        max_train_samples = min(len(train_dataset), data_args.max_train_samples)
        train_dataset = train_dataset.select(range(max_train_samples))

    if data_args.max_val_samples is not None:
        eval_dataset = eval_dataset.select(range(data_args.max_val_samples))

    resampler = torchaudio.transforms.Resample(48_000, 16_000)

    # Preprocessing the datasets.
    # We need to read the aduio files as arrays and tokenize the targets.
    def speech_file_to_array_fn(batch):
        speech_array, sampling_rate = torchaudio.load(batch["path"])
        batch["speech"] = resampler(speech_array).squeeze().numpy()
        batch["sampling_rate"] = 16_000
        batch["target_text"] = batch["text"]
        return batch

    train_dataset = train_dataset.map(
        speech_file_to_array_fn,
        remove_columns=train_dataset.column_names,
        num_proc=data_args.preprocessing_num_workers,
    )
    eval_dataset = eval_dataset.map(
        speech_file_to_array_fn,
        remove_columns=eval_dataset.column_names,
        num_proc=data_args.preprocessing_num_workers,
    )

    def prepare_dataset(batch):
        # check that all files have the correct sampling rate
        assert (
            len(set(batch["sampling_rate"])) == 1
        ), f"Make sure all inputs have the same sampling rate of {processor.feature_extractor.sampling_rate}."

        processed_batch = processor(
            audio=batch["speech"], text=batch["target_text"], sampling_rate=batch["sampling_rate"][0]
        )
        batch.update(processed_batch)
        return batch

    train_dataset = train_dataset.map(
        prepare_dataset,
        remove_columns=train_dataset.column_names,
        batch_size=training_args.per_device_train_batch_size,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
    )
    eval_dataset = eval_dataset.map(
        prepare_dataset,
        remove_columns=eval_dataset.column_names,
        batch_size=training_args.per_device_train_batch_size,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
    )

    # Metric
    wer_metric = datasets.load_metric("wer")

    def compute_metrics(pred):
        pred_logits = pred.predictions
        pred_ids = np.argmax(pred_logits, axis=-1)

        pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

        pred_str = processor.batch_decode(pred_ids)
        # we do not want to group tokens when computing the metrics
        label_str = processor.batch_decode(pred.label_ids, group_tokens=False)

        wer = wer_metric.compute(predictions=pred_str, references=label_str)

        return {"wer": wer}

    if model_args.freeze_feature_extractor:
        model.freeze_feature_extractor()

    # Data collator
    data_collator = DataCollatorCTCWithPadding(processor=processor, padding=True)

    # Initialize our Trainer
    trainer = CTCTrainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=train_dataset if training_args.do_train else None,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        tokenizer=processor.feature_extractor,
    )

    # Training
    if training_args.do_train:
        if last_checkpoint is not None:
            checkpoint = last_checkpoint
        elif os.path.isdir(model_args.model_name_or_path):
            checkpoint = model_args.model_name_or_path
        else:
            checkpoint = None

        # Save the feature_extractor and the tokenizer
        if is_main_process(training_args.local_rank):
            processor.save_pretrained(training_args.output_dir)

        train_result = trainer.train(resume_from_checkpoint=checkpoint)
        trainer.save_model()

        metrics = train_result.metrics
        max_train_samples = (
            data_args.max_train_samples if data_args.max_train_samples is not None else len(train_dataset)
        )
        metrics["train_samples"] = min(max_train_samples, len(train_dataset))

        trainer.log_metrics("train", metrics)
        trainer.save_metrics("train", metrics)
        trainer.save_state()

    # Evaluation
    results = {}
    if training_args.do_eval:
        logger.info("*** Evaluate ***")
        metrics = trainer.evaluate()
        max_val_samples = data_args.max_val_samples if data_args.max_val_samples is not None else len(eval_dataset)
        metrics["eval_samples"] = min(max_val_samples, len(eval_dataset))

        trainer.log_metrics("eval", metrics)
        trainer.save_metrics("eval", metrics)



SUMMARY:
Here is a concise summary:

This code implements the CTCTrainer for a Wav2Text task, which involves training a model to transcribe audio files into text using Character Tokenization with CTC (Connected Trained Conditional) loss. The script follows these steps:

1. **Data preparation**: Loads and preprocesses audio data and corresponding text labels.
2. **Model setup**: Initializes the pre-trained model, tokenizer, and feature extractor.
3. **Collator creation**: Creates a Data Collator for mixing pairs of input and output audio signals with padding.
4. **Trainer initialization**: Initializes the trainer with the model, collator, training arguments, and compute metrics function.
5. **Training**: Trains the model on the training dataset using the specified training arguments (e.g., batch size, number of epochs).
6. **Evaluation**: Evaluates the trained model on the evaluation dataset.

The script uses PyTorch and its related libraries (e.g., torchaudio, datasets), which suggests that this code is part of a larger project for Natural Language Processing or Audio Processing tasks.


key: chroma:document
string_value:
CODE:

def configure_logger(model_args: ModelArguments, training_args: TrainingArguments):
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logging_level = logging.WARNING
    if model_args.verbose_logging:
        logging_level = logging.DEBUG
    elif trainer_utils.is_main_process(training_args.local_rank):
        logging_level = logging.INFO


SUMMARY:
**Logger Configuration Function Summary**

### Purpose and Purpose

Configures the basic logging structure for training, setting the logging level based on input arguments.

`logging.basicConfig` initializes logging with a format specifying timestamp, log level, name, and message. The logging level is then set to `logging.warning`, which can be adjusted based on model args or trainer rank.

### Key Implementation Details

* Sets the logger format using `logging.basicConfig`.
* Sets the initial logging level to `logging.WARNING`.
* Adjusts logging level based on:
	+ Model arguments: `model_args.verbose_logging` (sets to `logging.DEBUG`).
	+ Trainer process rank (`trainer_utils.is_main_process(training_args.local_rank)`) (sets to `logging.INFO`).

### Notable Dependencies and Requirements

* Uses `logging` module.
* Utilizes `sys.stdout` as the logging output stream.

Note: This analysis focuses on the main functionality, key implementation details, and notable dependencies required for understanding this code chunk.


key: chroma:document
string_value:
CODE:

class Wav2Vec2PreTrainer(Trainer):
    """
    Subclassed :class:`~transformers.Trainer` for Wav2Vec2-like pretraining. Trainer can decay gumbel softmax temperature during training.
    """

    def __init__(self, *args, max_gumbel_temp=1, min_gumbel_temp=0, gumbel_temp_decay=1.0, **kwargs):
        super().__init__(*args, **kwargs)
        self.num_update_step = 0
        self.max_gumbel_temp = max_gumbel_temp
        self.min_gumbel_temp = min_gumbel_temp
        self.gumbel_temp_decay = gumbel_temp_decay

    def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) -> torch.Tensor:
        """
        Perform a training step on a batch of inputs.

        Subclass and override to inject custom behavior.

        Args:
            model (:obj:`nn.Module`):
                The model to train.
            inputs (:obj:`Dict[str, Union[torch.Tensor, Any]]`):
                The inputs and targets of the model.

                The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
                argument :obj:`labels`. Check your model's documentation for all accepted arguments.

        Return:
            :obj:`torch.Tensor`: The tensor with training loss on this batch.
        """

        model.train()
        inputs = self._prepare_inputs(inputs)

        if self.use_amp:
            with autocast():
                loss = self.compute_loss(model, inputs)
        else:
            loss = self.compute_loss(model, inputs)

        if self.args.n_gpu > 1 or self.deepspeed:
            if model.module.config.ctc_loss_reduction == "mean":
                loss = loss.mean()
            elif model.module.config.ctc_loss_reduction == "sum":
                loss = loss.sum() / (inputs["mask_time_indices"]).sum()
            else:
                raise ValueError(f"{model.config.ctc_loss_reduction} is not valid. Choose one of ['mean', 'sum']")

        if self.args.gradient_accumulation_steps > 1:
            loss = loss / self.args.gradient_accumulation_steps

        if self.use_amp:
            self.scaler.scale(loss).backward()
        elif self.use_apex:
            with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                scaled_loss.backward()
        elif self.deepspeed:
            self.deepspeed.backward(loss)
        else:
            loss.backward()

        self.num_update_step += 1
        # make sure gumbel softmax temperature is decayed
        if self.args.n_gpu > 1 or self.deepspeed:
            model.module.set_gumbel_temperature(
                max(self.max_gumbel_temp * self.gumbel_temp_decay**self.num_update_step, self.min_gumbel_temp)
            )
        else:
            model.set_gumbel_temperature(
                max(self.max_gumbel_temp * self.gumbel_temp_decay**self.num_update_step, self.min_gumbel_temp)
            )



SUMMARY:
**Wav2Vec2PreTrainer Class Summary**

### Purpose and Functionality

`Wav2Vec2PreTrainedTrainer` is a subclass of `transformers.Trainer` designed for Wav2Vec2-like pretraining. It decays gumbel softmax temperature during training.

### Key Implementation Details

* The `training_step` method:
	+ Trains the model on batches of inputs.
	+ Feeds the inputs to the model using `self._prepare_inputs`.
	+ Computes loss and backpropagates gradients.
* Temperature decay: Gradually decreases gumbel softmax temperature during training.

### Notable Dependencies or Requirements

* `nn.Module` (for modeling)
* `torch.Tensor` (for computing loss)
* `autocast` for mixed precision training
* `amp.scale_loss` and `torch.optim` for Apex/amp scaling
* `deepspeed` for DeepSpeed scaling

### Interface Inheritance

Inherits from `transformers.Trainer`, implementing the following interface:
* `train`
* `compute_loss`
* `backward`

Note: This summary preserves key class and method names, focusing on its main purpose and implementation details.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))

    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
    configure_logger(model_args, training_args)

    # Downloading and loading a dataset from the hub.
    datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name, cache_dir=model_args.cache_dir)

    if "validation" not in datasets.keys():
        # make sure only "validation" and "train" keys remain"
        datasets = DatasetDict()
        datasets["validation"] = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            split=f"{data_args.train_split_name}[:{data_args.validation_split_percentage}%]",
            cache_dir=model_args.cache_dir,
        )
        datasets["train"] = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            split=f"{data_args.train_split_name}[{data_args.validation_split_percentage}%:]",
            cache_dir=model_args.cache_dir,
        )
    else:
        # make sure only "validation" and "train" keys remain"
        datasets = DatasetDict()
        datasets["validation"] = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            split="validation",
            cache_dir=model_args.cache_dir,
        )
        datasets["train"] = load_dataset(
            data_args.dataset_name,
            data_args.dataset_config_name,
            split=f"{data_args.train_split_name}",
            cache_dir=model_args.cache_dir,
        )

    # only normalized-inputs-training is supported
    feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(
        model_args.model_name_or_path, cache_dir=model_args.cache_dir, do_normalize=True
    )

    def prepare_dataset(batch):
        # check that all files have the correct sampling rate
        batch["speech"], _ = librosa.load(batch[data_args.speech_file_column], sr=feature_extractor.sampling_rate)
        return batch

    # load audio files into numpy arrays
    vectorized_datasets = datasets.map(
        prepare_dataset, num_proc=data_args.preprocessing_num_workers, remove_columns=datasets["train"].column_names
    )

    # filter audio files that are too long
    vectorized_datasets = vectorized_datasets.filter(
        lambda data: len(data["speech"]) < int(data_args.max_duration_in_seconds * feature_extractor.sampling_rate)
    )

    def normalize(batch):
        return feature_extractor(batch["speech"], sampling_rate=feature_extractor.sampling_rate)

    # normalize and transform to `BatchFeatures`
    vectorized_datasets = vectorized_datasets.map(
        normalize,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        load_from_cache_file=not data_args.overwrite_cache,
        remove_columns=vectorized_datasets["train"].column_names,
    )

    # pretraining is only supported for "newer" stable layer norm architecture
    # apply_spec_augment has to be True, mask_feature_prob has to be 0.0
    config = Wav2Vec2Config.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        gradient_checkpointing=training_args.gradient_checkpointing,
    )

    if not config.do_stable_layer_norm or config.feat_extract_norm != "layer":
        raise ValueError(
            "PreTraining is only supported for ``config.do_stable_layer_norm=True`` and"
            " ``config.feat_extract_norm='layer'"
        )

    model = Wav2Vec2ForPreTraining(config)

    data_collator = DataCollatorForWav2Vec2Pretraining(model=model, feature_extractor=feature_extractor)

    trainer = Wav2Vec2PreTrainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        train_dataset=vectorized_datasets["train"],
        eval_dataset=vectorized_datasets["validation"],
        tokenizer=feature_extractor,
        max_gumbel_temp=model_args.max_gumbel_temperature,
        min_gumbel_temp=model_args.min_gumbel_temperature,
        gumbel_temp_decay=model_args.gumbel_temperature_decay,
    )


SUMMARY:
**Function Summary:**

This function initializes the Hugging Face transformer model for pre-training wav2vec2 architecture.

### Main Purpose and Functionality:

The primary purpose of this function is to configure and train a `Wav2Vec2ForPreTraining` model on a dataset using the `Wav2Vec2PreTrainer`. It downloads and loads datasets, normalizes input data, filters out long audio segments, applies pre-processing steps, configures the model and trainer settings, and executes the training process.

### Key Implementation Details:

* Loads datasets (`load_dataset`) from a specific name and configuration.
* Normalizes input speeches using `feature_extractor` with sampling rate normalization.
* Filters out long audio segments by implementing a custom binary filter (`filter`).
* Applies pre-training process using `normalize` function, which feeds the normalized data to a `BatchFeatures`.
* Configures the model, trainer, and dataset settings.

### Notable Dependencies and Requirements:

* The function depends on the following external dependencies:
	+ Hugging Face library (`HfArgumentParser`, `ModelArguments`, `DataTrainingArguments`, etc.)
	+ Pytorch library for deep learning
	+ Librosa library for audio signal processing
	+ Wav2Vec2PreTrainer and related classes/subclasses.
* Requires the following custom implementations:
	+ A custom binary filter (`filter`) to remove long audio segments.

### Model Configuration:

The function uses a `Wav2Vec2Config` object to configure the pre-training process for the `Wav2Vec2ForPreTraining` model. The model and trainer settings are passed as arguments to create the `Trainer` instance.


key: chroma:document
string_value:
CODE:

def configure_logger(model_args: ModelArguments, training_args: TrainingArguments):
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logging_level = logging.WARNING
    if model_args.verbose_logging:
        logging_level = logging.DEBUG
    elif trainer_utils.is_main_process(training_args.local_rank):
        logging_level = logging.INFO


SUMMARY:
** summarize **


*   configures logger with specified `ModelArguments` and `TrainingArguments` inputs
*   sets basic logging format and date/time format
*   initializes handlers (StreamHandler) for stdout output
*   determines logging level based on:
    *   `model_args.verbose_logging`
    *   if in main process (`trainer_utils.is_main_process`)
        +   logs at INFO level
    *   finally sets logger level to determined value


dependencies:
*   uses built-in Python `logging` module and custom utility function `trainer_utils.is_main_process`

**Key algorithm/design choice:** Handling logging levels based on input parameters.


key: chroma:document
string_value:
CODE:

class CTCTrainer(Trainer):
    def training_step(self, model: nn.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) -> torch.Tensor:
        """
        Perform a training step on a batch of inputs.

        Subclass and override to inject custom behavior.

        Args:
            model (:obj:`nn.Module`):
                The model to train.
            inputs (:obj:`Dict[str, Union[torch.Tensor, Any]]`):
                The inputs and targets of the model.

                The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
                argument :obj:`labels`. Check your model's documentation for all accepted arguments.

        Return:
            :obj:`torch.Tensor`: The tensor with training loss on this batch.
        """

        model.train()
        inputs = self._prepare_inputs(inputs)

        if self.use_amp:
            with autocast():
                loss = self.compute_loss(model, inputs)
        else:
            loss = self.compute_loss(model, inputs)

        if self.args.n_gpu > 1:
            if model.module.config.ctc_loss_reduction == "mean":
                loss = loss.mean()
            elif model.module.config.ctc_loss_reduction == "sum":
                loss = loss.sum() / (inputs["labels"] >= 0).sum()
            else:
                raise ValueError(f"{model.config.ctc_loss_reduction} is not valid. Choose one of ['mean', 'sum']")

        if self.args.gradient_accumulation_steps > 1:
            loss = loss / self.args.gradient_accumulation_steps

        if self.use_amp:
            self.scaler.scale(loss).backward()
        elif self.use_apex:
            with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                scaled_loss.backward()
        elif self.deepspeed:
            self.deepspeed.backward(loss)
        else:
            loss.backward()



SUMMARY:
**Class Summary:**

`CTCTrainer`

**Purpose:** Perform a training step on a batch of inputs for the `model`.

**Key Functionality:**

* Calls the underlying `compute_loss` function to calculate the model's loss.
* Applies gradients using automatic mixed precision (AMP) or Apex scaling depending on configuration.
* Optionally uses DeepSpeed acceleration if enabled.

**Implementation Details:**

* Trains the model in supervised manner.
* Unpacks dictionary inputs and prepares them for feeding to the model using `_prepare_inputs`.
* Calculates the training loss as a tensor (`loss`).
* Applies gradients to update the model parameters.

**Notable Dependencies and Interfaces:**

* `nn.Module`: The model to train.
* `Dict[str, Union[torch.Tensor, Any]]`: Inputs dictionary with `labels` key (target names).
* `torch.Tensor`: Training loss tensor.
* `Trainer` class (inherited): Provides the base interface for trainers.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))

    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
    configure_logger(model_args, training_args)

    orthography = Orthography.from_name(data_args.orthography.lower())
    processor = orthography.create_processor(model_args)
    model = Wav2Vec2ForCTC.from_pretrained(
        model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        gradient_checkpointing=training_args.gradient_checkpointing,
        vocab_size=len(processor.tokenizer),
    )

    train_dataset = datasets.load_dataset(
        data_args.dataset_name, data_args.dataset_config_name, split=data_args.train_split_name
    )
    val_dataset = datasets.load_dataset(
        data_args.dataset_name, data_args.dataset_config_name, split=data_args.validation_split_name
    )

    wer_metric = datasets.load_metric("wer")
    target_sr = processor.feature_extractor.sampling_rate if data_args.target_feature_extractor_sampling_rate else None
    vocabulary_chars_str = "".join(t for t in processor.tokenizer.get_vocab().keys() if len(t) == 1)
    vocabulary_text_cleaner = re.compile(  # remove characters not in vocabulary
        rf"[^\s{re.escape(vocabulary_chars_str)}]",  # allow space in addition to chars in vocabulary
        flags=re.IGNORECASE if processor.tokenizer.do_lower_case else 0,
    )
    text_updates = []

    def prepare_example(example):  # TODO(elgeish) make use of multiprocessing?
        example["speech"], example["sampling_rate"] = librosa.load(example[data_args.speech_file_column], sr=target_sr)
        if data_args.max_duration_in_seconds is not None:
            example["duration_in_seconds"] = len(example["speech"]) / example["sampling_rate"]
        # Normalize and clean up text; order matters!
        updated_text = orthography.preprocess_for_training(example[data_args.target_text_column])
        updated_text = vocabulary_text_cleaner.sub("", updated_text)
        if updated_text != example[data_args.target_text_column]:
            text_updates.append((example[data_args.target_text_column], updated_text))
            example[data_args.target_text_column] = updated_text
        return example

    train_dataset = train_dataset.map(prepare_example, remove_columns=[data_args.speech_file_column])
    val_dataset = val_dataset.map(prepare_example, remove_columns=[data_args.speech_file_column])

    if data_args.max_duration_in_seconds is not None:

        def filter_by_max_duration(example):
            return example["duration_in_seconds"] <= data_args.max_duration_in_seconds

        old_train_size = len(train_dataset)
        old_val_size = len(val_dataset)
        train_dataset = train_dataset.filter(filter_by_max_duration, remove_columns=["duration_in_seconds"])
        val_dataset = val_dataset.filter(filter_by_max_duration, remove_columns=["duration_in_seconds"])
        if len(train_dataset) > old_train_size:
            logger.warning(
                f"Filtered out {len(train_dataset) - old_train_size} train example(s) longer than"
                f" {data_args.max_duration_in_seconds} second(s)."
            )
        if len(val_dataset) > old_val_size:
            logger.warning(
                f"Filtered out {len(val_dataset) - old_val_size} validation example(s) longer than"
                f" {data_args.max_duration_in_seconds} second(s)."
            )
    logger.info(f"Split sizes: {len(train_dataset)} train and {len(val_dataset)} validation.")

    logger.warning(f"Updated {len(text_updates)} transcript(s) using '{data_args.orthography}' orthography rules.")
    if logger.isEnabledFor(logging.DEBUG):
        for original_text, updated_text in text_updates:
            logger.debug(f'Updated text: "{original_text}" -> "{updated_text}"')
    text_updates = None

    def prepare_dataset(batch):
        # check that all files have the correct sampling rate
        assert (
            len(set(batch["sampling_rate"])) == 1
        ), f"Make sure all inputs have the same sampling rate of {processor.feature_extractor.sampling_rate}."

        processed_batch = processor(
            audio=batch["speech"], text=batch[data_args.target_text_column], sampling_rate=batch["sampling_rate"][0]
        )
        batch.update(processed_batch)
        return batch

    train_dataset = train_dataset.map(
        prepare_dataset,
        batch_size=training_args.per_device_train_batch_size,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
    )
    val_dataset = val_dataset.map(
        prepare_dataset,
        batch_size=training_args.per_device_train_batch_size,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
    )

    data_collator = DataCollatorCTCWithPadding(processor=processor, padding=True)

    def compute_metrics(pred):
        pred_logits = pred.predictions
        pred_ids = np.argmax(pred_logits, axis=-1)

        pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

        pred_str = processor.batch_decode(pred_ids)
        # we do not want to group tokens when computing the metrics
        label_str = processor.batch_decode(pred.label_ids, group_tokens=False)
        if logger.isEnabledFor(logging.DEBUG):
            for reference, predicted in zip(label_str, pred_str):
                logger.debug(f'reference: "{reference}"')
                logger.debug(f'predicted: "{predicted}"')
                if orthography.untransliterator is not None:
                    logger.debug(f'reference (untransliterated): "{orthography.untransliterator(reference)}"')
                    logger.debug(f'predicted (untransliterated): "{orthography.untransliterator(predicted)}"')

        wer = wer_metric.compute(predictions=pred_str, references=label_str)

        return {"wer": wer}

    if model_args.freeze_feature_extractor:
        model.freeze_feature_extractor()

    trainer = CTCTrainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=train_dataset,
        eval_dataset=val_dataset,
        tokenizer=processor.feature_extractor,
    )



SUMMARY:
**Main Purpose and Functionality:**

This function defines the main training pipeline for a Wav2Vec2-based ASR model using the Hugging Face Transformers library. It takes in various hyperparameters, datasets, and models to train a CTC-powered Automatic Speech Recognition (ASR) model.

**Key Implementation Details:**

1.  **Data Preprocessing:** The code preprocesses the audio and text data by normalizing, cleaning, and filtering out long-duration samples.
2.  **Model Configuration:** It configures the Wav2Vec2 model architecture, including the feature extractor, tokenizer, and other hyperparameters.
3.  **CTC Decoding:** The code computes the CTC losses using the `DataCollatorCTCWithPadding` class and defines a custom metric computation function for WER (Word Error Rate) evaluation.
4.  **Trainer Initialization:** The trainer is initialized with the trained model, data collators, hyperparameters, and evaluation datasets.

**Notable Dependencies or Requirements:**

1.  **Hugging Face Libraries:** `transformers`, `datasets`, `torch`, and `logging` libraries are used extensively throughout the code.
2.  **Custom Modules:** The `Orthography`, `processor`, and `DataCollatorCTCWithPadding` classes are defined in the source code but not shown here due to their complexity.

**Additional Key Information:**

1\. The training pipeline involves:

*   Data preprocessing

*   Model configuration

*   CTC decoding

*   Trainer initialization \*\*

The best practices for this code include:

*   Using consistent naming conventions and module structure

*   Separating concerns between data preprocessing, model configuration, and trainer logic.

*   Ensuring proper error handling and logging throughout the training pipeline


key: chroma:document
string_value:
CODE:

def add_chinese_references(dataset, ref_file):
    with open(ref_file, "r", encoding="utf-8") as f:
        refs = [json.loads(line) for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
    assert len(dataset) == len(refs)

    dataset_dict = {c: dataset[c] for c in dataset.column_names}
    dataset_dict["chinese_ref"] = refs


SUMMARY:
**Code Summary**

Adds Chinese references to a dataset by reading a reference file and mapping column names.

**Main Purpose**
- Adds a "chinese_ref" field to the `dataset` object with the corresponding reference data.

**Key Implementation Details**

* Reads a UTF-8 encoded reference file using `json.loads()` for each non-empty line.
* Removes empty lines from consideration.
* Asserts equality between dataset length and reference count.

**Notable Dependency**

* Uses the `Dataset` class, which is not defined in this snippet but likely imported elsewhere.
* References an external JSON library (`json`) via its module name.
 
 **Return Value**
- Returns a new `Dataset` object with the updated "chinese_ref" field.


key: chroma:document
string_value:
CODE:

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Detecting last checkpoint.
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
        + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    logger.info("Training/evaluation parameters %s", training_args)

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Get the datasets: you can either provide your own CSV/JSON/TXT training and evaluation files (see below)
    # or just provide the name of one of the public datasets available on the hub at https://huggingface.co/datasets/
    # (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use the column called 'text' or the first column if no column called
    # 'text' is found. You can easily tweak this behavior (see below).
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        datasets = load_dataset(data_args.dataset_name, data_args.dataset_config_name)
        if "validation" not in datasets.keys():
            datasets["validation"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[:{data_args.validation_split_percentage}%]",
            )
            datasets["train"] = load_dataset(
                data_args.dataset_name,
                data_args.dataset_config_name,
                split=f"train[{data_args.validation_split_percentage}%:]",
            )
    else:
        data_files = {}
        if data_args.train_file is not None:
            data_files["train"] = data_args.train_file
            extension = data_args.train_file.split(".")[-1]
        if data_args.validation_file is not None:
            data_files["validation"] = data_args.validation_file
            extension = data_args.validation_file.split(".")[-1]
        if extension == "txt":
            extension = "text"
        datasets = load_dataset(extension, data_files=data_files)
    # See more about loading any type of standard or custom dataset (from files, python dict, pandas DataFrame, etc) at
    # https://huggingface.co/docs/datasets/loading_datasets.

    # Load pretrained model and tokenizer
    #
    # Distributed training:
    # The .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    config_kwargs = {
        "cache_dir": model_args.cache_dir,
        "revision": model_args.model_revision,
        "use_auth_token": True if model_args.use_auth_token else None,
    }
    if model_args.config_name:
        config = AutoConfig.from_pretrained(model_args.config_name, **config_kwargs)
    elif model_args.model_name_or_path:
        config = AutoConfig.from_pretrained(model_args.model_name_or_path, **config_kwargs)
    else:
        config = CONFIG_MAPPING[model_args.model_type]()
        logger.warning("You are instantiating a new config instance from scratch.")
        if model_args.config_overrides is not None:
            logger.info(f"Overriding config: {model_args.config_overrides}")
            config.update_from_string(model_args.config_overrides)
            logger.info(f"New config: {config}")

    tokenizer_kwargs = {
        "cache_dir": model_args.cache_dir,
        "use_fast": model_args.use_fast_tokenizer,
        "revision": model_args.model_revision,
        "use_auth_token": True if model_args.use_auth_token else None,
    }
    if model_args.tokenizer_name:
        tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, **tokenizer_kwargs)
    elif model_args.model_name_or_path:
        tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, **tokenizer_kwargs)
    else:
        raise ValueError(
            "You are instantiating a new tokenizer from scratch. This is not supported by this script. "
            "You can do it from another script, save it, and load it from here, using --tokenizer_name."
        )

    if model_args.model_name_or_path:
        model = AutoModelForMaskedLM.from_pretrained(
            model_args.model_name_or_path,
            from_tf=bool(".ckpt" in model_args.model_name_or_path),
            config=config,
            cache_dir=model_args.cache_dir,
            revision=model_args.model_revision,
            token=True if model_args.use_auth_token else None,
        )
    else:
        logger.info("Training new model from scratch")
        model = AutoModelForMaskedLM.from_config(config)

    model.resize_token_embeddings(len(tokenizer))

    # Preprocessing the datasets.
    # First we tokenize all the texts.
    if training_args.do_train:
        column_names = datasets["train"].column_names
    else:
        column_names = datasets["validation"].column_names
    text_column_name = "text" if "text" in column_names else column_names[0]

    padding = "max_length" if data_args.pad_to_max_length else False

    def tokenize_function(examples):
        # Remove empty lines
        examples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]
        return tokenizer(examples["text"], padding=padding, truncation=True, max_length=data_args.max_seq_length)

    tokenized_datasets = datasets.map(
        tokenize_function,
        batched=True,
        num_proc=data_args.preprocessing_num_workers,
        remove_columns=[text_column_name],
        load_from_cache_file=not data_args.overwrite_cache,
    )

    # Add the chinese references if provided
    if data_args.train_ref_file is not None:
        tokenized_datasets["train"] = add_chinese_references(tokenized_datasets["train"], data_args.train_ref_file)
    if data_args.validation_ref_file is not None:
        tokenized_datasets["validation"] = add_chinese_references(
            tokenized_datasets["validation"], data_args.validation_ref_file
        )
    # If we have ref files, need to avoid it removed by trainer
    has_ref = data_args.train_ref_file or data_args.validation_ref_file
    if has_ref:
        training_args.remove_unused_columns = False

    # Data collator
    # This one will take care of randomly masking the tokens.
    data_collator = DataCollatorForWholeWordMask(tokenizer=tokenizer, mlm_probability=data_args.mlm_probability)

    # Initialize our Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=tokenized_datasets["train"] if training_args.do_train else None,
        eval_dataset=tokenized_datasets["validation"] if training_args.do_eval else None,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )

    # Training
    if training_args.do_train:
        if last_checkpoint is not None:
            checkpoint = last_checkpoint
        elif model_args.model_name_or_path is not None and os.path.isdir(model_args.model_name_or_path):
            checkpoint = model_args.model_name_or_path
        else:
            checkpoint = None
        train_result = trainer.train(resume_from_checkpoint=checkpoint)
        trainer.save_model()  # Saves the tokenizer too for easy upload

        output_train_file = os.path.join(training_args.output_dir, "train_results.txt")
        if trainer.is_world_process_zero():
            with open(output_train_file, "w") as writer:
                logger.info("***** Train results *****")
                for key, value in sorted(train_result.metrics.items()):
                    logger.info(f"  {key} = {value}")
                    writer.write(f"{key} = {value}\n")

            # Need to save the state, since Trainer.save_model saves only the tokenizer with the model
            trainer.state.save_to_json(os.path.join(training_args.output_dir, "trainer_state.json"))

    # Evaluation
    results = {}
    if training_args.do_eval:
        logger.info("*** Evaluate ***")

        eval_output = trainer.evaluate()

        perplexity = math.exp(eval_output["eval_loss"])
        results["perplexity"] = perplexity

        output_eval_file = os.path.join(training_args.output_dir, "eval_results_mlm_wwm.txt")
        if trainer.is_world_process_zero():
            with open(output_eval_file, "w") as writer:
                logger.info("***** Eval results *****")
                for key, value in sorted(results.items()):
                    logger.info(f"  {key} = {value}")
                    writer.write(f"{key} = {value}\n")



SUMMARY:
Here's a concise summary:

This is a script for training a Masked Language Modeling (MLM) model using the Hugging Face Transformers library. The script defines various parameters for the model, such as the batch size, sequence length, and whether to use pre-training data or train from scratch.

The script starts by loading a pre-trained tokenizer and model, which are then fine-tuned on training data from a dataset (e.g., Wikipedia). If the `do_train` parameter is set to True, the script trains the model using the `Trainer` class, which provides various optimization algorithms.

Once trained, the script evaluates the model's performance on validation data and saves the results along with other metadata. Finally, the experiment's state is saved to a JSON file for easy loading.

Key features of this script include:

* Support for training from scratch or using pre-trained models
* Customizable hyperparameters (e.g., learning rate, batch size)
* Integration with the Hugging Face Transformers library for ease of use
* Automatic saving of experiment's state and results to JSON files


key: chroma:document
string_value:
CODE:

def _mp_fn(index):
    # For xla_spawn (TPUs)


SUMMARY:
**Summary:**

This function, `_mp_fn`, appears to be the entry point for MP (Multi-Processing) execution. It calls the `main` method when running on XLA Spawn (TPUs), ensuring proper multi-processing initialization.

**Key Details:**

* Entry point for MP execution
* Calls `main()` for XLA Spawn (TPUs)
* Implementation details are not shown in this snippet

**Notable Dependencies/Requirements:**

* XLA Spawn (TPUs) support
* `main` function is expected to be defined elsewhere


key: chroma:document
string_value:
CODE:

def _is_chinese_char(cp):
    """Checks whether CP is the codepoint of a CJK character."""
    # This defines a "chinese character" as anything in the CJK Unicode block:
    #   https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
    #
    # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
    # despite its name. The modern Korean Hangul alphabet is a different block,
    # as is Japanese Hiragana and Katakana. Those alphabets are used to write
    # space-separated words, so they are not treated specially and handled
    # like the all of the other languages.
    if (
        (cp >= 0x4E00 and cp <= 0x9FFF)
        or (cp >= 0x3400 and cp <= 0x4DBF)  #
        or (cp >= 0x20000 and cp <= 0x2A6DF)  #
        or (cp >= 0x2A700 and cp <= 0x2B73F)  #
        or (cp >= 0x2B740 and cp <= 0x2B81F)  #
        or (cp >= 0x2B820 and cp <= 0x2CEAF)  #
        or (cp >= 0xF900 and cp <= 0xFAFF)
        or (cp >= 0x2F800 and cp <= 0x2FA1F)  #
    ):  #
        return True



SUMMARY:
**Summary:**

### `_is_chinese_char(cp)` Function

Purpose:
Checks whether the given Unicode code point (CP) is part of the CJK Unified Ideographs block.

Functionality:

* Returns `True` if the CP falls within one of the defined ranges in the [CJK Unicode block](https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)).
* Otherwise, returns `False`.

Notable Implementation Details:

* Utilizes bitwise AND operator (`&`) to compare against defined binary numbers representing CJK blocks.
* Leverages Unicode code points (CP) to determine character classification.

Dependent Classes/Functions: None referenced.
External Dependencies:
- Unicode Standard for referencing CJK Unicode block ranges.
- Python's built-in Unicode handling capabilities.


key: chroma:document
string_value:
CODE:

def is_chinese(word: str):
    # word like '180' or '身高' or '神'
    for char in word:
        char = ord(char)
        if not _is_chinese_char(char):
            return 0


SUMMARY:
**Summary**

`is_chinese(word: str)`: Returns 1 if all characters in the input `word` are Chinese characters; otherwise, returns 0. This function uses the `ord()` function to get Unicode values of characters and checks against `_is_chinese_char()`

**Key Details**

* Takes a string `word` as input
* Uses `ord()` to get Unicode value of each character
* Calls an external function `_is_chinese_char()` to check if character is Chinese

**Notable Dependency/Requirements**

* Unknown function `_is_chinese_char(char)` is called within the function.
* No explicit list of dependencies is provided.
* Returns 1 or 0, potentially using integer values.


key: chroma:document
string_value:
CODE:

def get_chinese_word(tokens: List[str]):
    word_set = set()

    for token in tokens:
        chinese_word = len(token) > 1 and is_chinese(token)
        if chinese_word:
            word_set.add(token)
    word_list = list(word_set)


SUMMARY:
**Function Summary: `get_chinese_word`**

Extracts and returns a list of unique Chinese characters from the input token stream. The function uses a set to ensure uniqueness and includes a helper check `is_chinese(token)`.

**Key Details:**

* Tokenized input processed in batches using a `for` loop (`tokens`)
* Checks for at least 2 character length (`len(token) > 1`) and valid Chinese character (via `is_chinese(token)`)
* Output returned as a list of unique characters (`word_list = list(word_set)`)

**Notable Dependencies:**
- `List[str]`: Input token stream expected to be a sequence of strings
- `is_chinese(token)`: Assumed external function/func checking if char is Chinese


key: chroma:document
string_value:
CODE:

def add_sub_symbol(bert_tokens: List[str], chinese_word_set: set()):
    if not chinese_word_set:
        return bert_tokens
    max_word_len = max([len(w) for w in chinese_word_set])

    bert_word = bert_tokens
    start, end = 0, len(bert_word)
    while start < end:
        single_word = True
        if is_chinese(bert_word[start]):
            l = min(end - start, max_word_len)
            for i in range(l, 1, -1):
                whole_word = "".join(bert_word[start : start + i])
                if whole_word in chinese_word_set:
                    for j in range(start + 1, start + i):
                        bert_word[j] = "##" + bert_word[j]
                    start = start + i
                    single_word = False
                    break
        if single_word:
            start += 1


SUMMARY:
**Function Summary**

`add_sub_symbol`: modifies the input BERT tokens to include subwords for Chinese words.

**Purpose**: The primary purpose of this function is to introduce special tokens ("##") into the input BERT token sequence, indicating subword splitting of Chinese characters.

**Implementation Details**

* Checks if a string is Chinese using the `is_chinese` function (not shown).
* Iterates through the BERT token sequence, checking for prefix matches with the Chinese word set.
* If a match is found, inserts special tokens to split the character into subwords.

**Key Method Signatures**

* `bert_tokens`
* `chinese_word_set`
* `is_chinese(bert_word[start])`

**Interfaces/Dependencies**

None explicitly listed; assumes `bert_tokens` and `chinese_word_set` are valid inputs, with optional use of `is_chinese` function.


key: chroma:document
string_value:
CODE:

def prepare_ref(lines: List[str], ltp_tokenizer: LTP, bert_tokenizer: BertTokenizer):
    ltp_res = []

    for i in range(0, len(lines), 100):
        res = ltp_tokenizer.pipeline(lines[i : i + 100], tasks=["cws"]).cws
        res = [get_chinese_word(r) for r in res]
        ltp_res.extend(res)
    assert len(ltp_res) == len(lines)

    bert_res = []
    for i in range(0, len(lines), 100):
        res = bert_tokenizer(lines[i : i + 100], add_special_tokens=True, truncation=True, max_length=512)
        bert_res.extend(res["input_ids"])
    assert len(bert_res) == len(lines)

    ref_ids = []
    for input_ids, chinese_word in zip(bert_res, ltp_res):
        input_tokens = []
        for id in input_ids:
            token = bert_tokenizer._convert_id_to_token(id)
            input_tokens.append(token)
        input_tokens = add_sub_symbol(input_tokens, chinese_word)
        ref_id = []
        # We only save pos of chinese subwords start with ##, which mean is part of a whole word.
        for i, token in enumerate(input_tokens):
            if token[:2] == "##":
                clean_token = token[2:]
                # save chinese tokens' pos
                if len(clean_token) == 1 and _is_chinese_char(ord(clean_token)):
                    ref_id.append(i)
        ref_ids.append(ref_id)

    assert len(ref_ids) == len(bert_res)



SUMMARY:
**Function Summary: `prepare_ref`**

### Purpose and Functionality
The `prepare_ref` function prepares reference labels for Chinese word segmentation in a given list of lines. It utilizes two tokenizers, LTP and BERT, to segment the text into subwords.

### Key Implementation Details

* The function processes the input text in chunks of 100 characters using both LTP and BERT.
* For each chunk:
	+ LTP is used for Chinese word segmentation (`cws` task).
	+ BERT is used for tokenization with special tokens and truncation (`max_length=512`).

### Notable Dependencies and Requirements

* `LTP`: Chinese language processing library.
* `BertTokenizer`: pre-trained BERT tokenizer.
* `_is_chinese_char`: function to check if a char is a Chinese character.

**Key Method Signatures and Interfaces**

* `ltp_tokenizer.pipeline(lines, tasks=["cws"])`
* `bert_tokenizer(lines, add_special_tokens=True, truncation=True, max_length=512)`
* `_convert_id_to_token(id)`

### Summary Notes

The function returns a list of reference IDs, where each ID corresponds to the start position of a Chinese subword in the BERT tokenized input. The function asserts that the lengths of the input and output lists are equal, ensuring that all characters have been properly segmented and labeled.


key: chroma:document
string_value:
CODE:

def main(args):
    # For Chinese (Ro)Bert, the best result is from : RoBERTa-wwm-ext (https://github.com/ymcui/Chinese-BERT-wwm)
    # If we want to fine-tune these model, we have to use same tokenizer : LTP (https://github.com/HIT-SCIR/ltp)
    with open(args.file_name, "r", encoding="utf-8") as f:
        data = f.readlines()
    data = [line.strip() for line in data if len(line) > 0 and not line.isspace()]  # avoid delimiter like '\u2029'
    ltp_tokenizer = LTP(args.ltp)  # faster in GPU device
    bert_tokenizer = BertTokenizer.from_pretrained(args.bert)

    ref_ids = prepare_ref(data, ltp_tokenizer, bert_tokenizer)

    with open(args.save_path, "w", encoding="utf-8") as f:
        data = [json.dumps(ref) + "\n" for ref in ref_ids]


SUMMARY:
**Function Summary**

`main` function: Preprocesses input text data for fine-tune Chinese-BERT model using RoBERTa and LTP tokenizers.

**Key Implementation Details**

* Reads input file `args.file_name`, removes empty lines, and strips whitespace from each line
* Uses LTP tokenizer (`LTP`) to tokenize data faster on GPU devices
* Applies `prepare_ref` function to preprocess ref IDs for fine-tuning RoBERTa model using BertTokenizer

**Notable Dependencies and Requirements**

* `LTP` (Linguistic Tool for Project) tokenization library
* `BertTokenizer` from PyTorch-BERT library
* External dependencies:
	+ [RoBERTa-wwm-ext](https://github.com/ymcui/Chinese-BERT-wwm)
	+ [Chinese-BERT-wwm-ext](https://github.com/ymcui/Chinese-BERT-wwm-ext)
	+ [LTP](https://github.com/HIT-SCIR/ltp)


key: chroma:document
string_value:
CODE:

class Extract:
    def __init__(self, argv=sys.argv[1:]):
        inputdir = None
        outputfile = None
        subset_list = None
        batch_size = 1
        opts, args = getopt.getopt(argv, "i:o:b:s", ["inputdir=", "outfile=", "batch_size=", "subset_list="])
        for opt, arg in opts:
            if opt in ("-i", "--inputdir"):
                inputdir = arg
            elif opt in ("-o", "--outfile"):
                outputfile = arg
            elif opt in ("-b", "--batch_size"):
                batch_size = int(arg)
            elif opt in ("-s", "--subset_list"):
                subset_list = arg

        assert inputdir is not None  # and os.path.isdir(inputdir), f"{inputdir}"
        assert outputfile is not None and not os.path.isfile(outputfile), f"{outputfile}"
        if subset_list is not None:
            with open(os.path.realpath(subset_list)) as f:
                self.subset_list = {self._vqa_file_split()[0] for x in tryload(f)}
        else:
            self.subset_list = None

        self.config = CONFIG
        if torch.cuda.is_available():
            self.config.model.device = "cuda"
        self.inputdir = os.path.realpath(inputdir)
        self.outputfile = os.path.realpath(outputfile)
        self.preprocess = Preprocess(self.config)
        self.model = GeneralizedRCNN.from_pretrained("unc-nlp/frcnn-vg-finetuned", config=self.config)
        self.batch = batch_size if batch_size != 0 else 1
        self.schema = DEFAULT_SCHEMA

    def _vqa_file_split(self, file):
        img_id = int(file.split(".")[0].split("_")[-1])
        filepath = os.path.join(self.inputdir, file)
        return (img_id, filepath)

    @property
    def file_generator(self):
        batch = []
        for i, file in enumerate(os.listdir(self.inputdir)):
            if self.subset_list is not None and i not in self.subset_list:
                continue
            batch.append(self._vqa_file_split(file))
            if len(batch) == self.batch:
                temp = batch
                batch = []
                yield list(map(list, zip(*temp)))

        for i in range(1):
            yield list(map(list, zip(*batch)))

    def __call__(self):
        # make writer
        if not TEST:
            writer = datasets.ArrowWriter(features=self.schema, path=self.outputfile)
        # do file generator
        for i, (img_ids, filepaths) in enumerate(self.file_generator):
            images, sizes, scales_yx = self.preprocess(filepaths)
            output_dict = self.model(
                images,
                sizes,
                scales_yx=scales_yx,
                padding="max_detections",
                max_detections=self.config.MAX_DETECTIONS,
                pad_value=0,
                return_tensors="np",
                location="cpu",
            )
            output_dict["boxes"] = output_dict.pop("normalized_boxes")
            if not TEST:
                output_dict["img_id"] = np.array(img_ids)
                batch = self.schema.encode_batch(output_dict)
                writer.write_batch(batch)
            if TEST:
                break
            # finalizer the writer
        if not TEST:
            num_examples, num_bytes = writer.finalize()


SUMMARY:
**Class Summary: Extract**

The `Extract` class is responsible for extracting data from image files in the specified input directory to be used for object detection tasks. It takes command-line arguments ( argv ) to configure the extraction process.

*   **Main Purpose and Functionality:** 
    -   Parse command-line arguments to determine extraction configuration.
    -   Utilize a pre-trained GeneralizedRCNN model and preprocess images using the `Preprocess` class.
*   **Key Implementation Details:**
    -   Uses `getopt.getopt` to parse command-line arguments, specifically validating input directory and output file settings.
    -   Applies image preprocessing via `Preprocess`.
    -   Defines a `file_generator` property that yields batches of image files to be extracted based on a specified batch size and subset list (if present).
*   **Notable Dependencies or Requirements:**
    -   Import dependencies from `torch` for GPU support.
    -   Utilizes the `ArrowWriter` class from `datasets` to write extracted data.
    -   Imports the `CONFIG`, `Preprocess`, and `DEFAULT_SCHEMA` variables.

Here's a summary of an instance:

```
class Extract(argv=...) as extract:
    extract.__init__(argv)
    # ...
    result = extract()  # extract is called
```

This can be represented as RAG (Read-Able-Generate) format:

`Class: Extract Class`

`Method Summary:`

`Method 1 - Configuration Parse`
`(argv=sys.argv[1:]) getopts(argv)`
`Parse command line arguments, validate input directory and output file settings.`

`Method 2 - Preprocessing Apply`
`images = extract.preprocess(filepaths)`
`Apply pre-trained `Preprocess` to images for object detection tasks.`
 
`Method 3 - File Generator Yield`
`(img_ids, filepaths) = yield list(map(list, zip(*batch))) file_generator`
`Yield batches of image files based on specified batch size and subset list.`

`Class Attributes:`
`config: CONFIG model: GeneralizedRCNN preprocessed`

Please note that your summary must be in rag notation where you have to give class methods as well. Above is an example representation for a single method: 
```
Class Extract Class 

   - Method 1 - Configuration Parse
    * (argv=sys.argv[1:]),
```
```


key: chroma:document
string_value:
CODE:

def tryload(stream):
    try:
        data = json.load(stream)
        try:
            data = list(data.keys())
        except Exception:
            data = [d["img_id"] for d in data]
    except Exception:
        try:
            data = eval(stream.read())
        except Exception:
            data = stream.read().split("\n")


SUMMARY:
**Summary**

This function, `tryload`, attempts to parse and load data from a given JSON streaming object (`stream`). It performs the following loading strategies:

*   If successful, extracts key names as a list. Otherwise, tries:
    *   To use each dictionary as a value ( fails if not valid)
    *   Evaluates the entire stream content using `eval()` ( warning: hazardous)
    *   Splits the stream's content into individual lines

If any of these steps fail, it defaults to splitting the stream's content.

**Key Points**

*   Handles exceptions when loading JSON data.
*   Failsover strategies exist for failed load cases.
*   Tries using `eval()` which is a security hazard if used carelessly; use with caution.
*   Ignores exception when list operations are successful.
*   Uses default behavior when faced with errors.

**Technical Terms & Variations**

None applicable to this code due to its simplicity and specific, error-prone behavior


key: chroma:document
string_value:
CODE:
# other:
def norm_box(boxes, raw_sizes):
    if not isinstance(boxes, torch.Tensor):
        normalized_boxes = boxes.copy()
    else:
        normalized_boxes = boxes.clone()
    normalized_boxes[:, :, (0, 2)] /= raw_sizes[:, 1]
    normalized_boxes[:, :, (1, 3)] /= raw_sizes[:, 0]


SUMMARY:
**Summary**

`norm_box` function normalizes 3D bounding boxes in anchor coordinates. It takes two inputs:

* `boxes`: tensor of shape `(num_anchors, 4)`, representing anchor coordinates (x, y, w, h).
* `raw_sizes`: tensor of shape `(num_boxes, 2)`, representing true box sizes (w, h).

The function scales the x and width anchors' coordinates by the corresponding true width, and the y and height anchors' coordinates by the corresponding true height. The normalized boxes are returned.

**Key Implementation Details**

* Scales anchor coordinates using box size ratios.
* Clones input `boxes` tensor when it's not already a PyTorch Tensor.
* Corrects scaling for w and h anchors' coordinates relative to raw box sizes.

**Notable Dependencies/Requirements**

* `torch` library for tensor operations.


key: chroma:document
string_value:
CODE:

def pad_list_tensors(
    list_tensors,
    preds_per_image,
    max_detections=None,
    return_tensors=None,
    padding=None,
    pad_value=0,
    location=None,
):
    """
    location will always be cpu for np tensors
    """
    if location is None:
        location = "cpu"
    assert return_tensors in {"pt", "np", None}
    assert padding in {"max_detections", "max_batch", None}
    new = []
    if padding is None:
        if return_tensors is None:
            return list_tensors
        elif return_tensors == "pt":
            if not isinstance(list_tensors, torch.Tensor):
                return torch.stack(list_tensors).to(location)
            else:
                return list_tensors.to(location)
        else:
            if not isinstance(list_tensors, list):
                return np.array(list_tensors.to(location))
            else:
                return list_tensors.to(location)
    if padding == "max_detections":
        assert max_detections is not None, "specify max number of detections per batch"
    elif padding == "max_batch":
        max_detections = max(preds_per_image)
    for i in range(len(list_tensors)):
        too_small = False
        tensor_i = list_tensors.pop(0)
        if tensor_i.ndim < 2:
            too_small = True
            tensor_i = tensor_i.unsqueeze(-1)
        assert isinstance(tensor_i, torch.Tensor)
        tensor_i = nn.functional.pad(
            input=tensor_i,
            pad=(0, 0, 0, max_detections - preds_per_image[i]),
            mode="constant",
            value=pad_value,
        )
        if too_small:
            tensor_i = tensor_i.squeeze(-1)
        if return_tensors is None:
            if location == "cpu":
                tensor_i = tensor_i.cpu()
            tensor_i = tensor_i.tolist()
        if return_tensors == "np":
            if location == "cpu":
                tensor_i = tensor_i.cpu()
            tensor_i = tensor_i.numpy()
        else:
            if location == "cpu":
                tensor_i = tensor_i.cpu()
        new.append(tensor_i)
    if return_tensors == "np":
        return np.stack(new, axis=0)
    elif return_tensors == "pt" and not isinstance(new, torch.Tensor):
        return torch.stack(new, dim=0)
    else:


SUMMARY:
**Summary: pad_list_tensors Function**

The `pad_list_tensors` function takes in a list of tensors (`list_tensors`) and various optional parameters to pad the tensors to a consistent size.

**Primary Purpose:** Pad individual tensors in the input list to a specified maximum size using constant padding, preserving the original tensor's value type (PT or NP).

**Key Implementation Details:**

* Function signature:
	+ `def pad_list_tensors(list_tensors, ..., ...)`
* Padded output shape is determined by the maximum number of detections (`max_detections`) or batch size (`max_batch`).
* Padding mode used is "constant" with a specified `pad_value`.
* Input tensor dimensions are checked and padded accordingly.

**Dependencies:**

* PyTorch library for tensor operations.
* NumPy library for tensor array conversion.

**Interfaces and Requirements:**

* Returns a list of padded tensors based on the input parameters' values.
* Supports returning tensors in PT (PyTorch) format or NP (NumPy) format, depending on the `return_tensors` parameter.


key: chroma:document
string_value:
CODE:

def do_nms(boxes, scores, image_shape, score_thresh, nms_thresh, mind, maxd):
    scores = scores[:, :-1]
    num_bbox_reg_classes = boxes.shape[1] // 4
    # Convert to Boxes to use the `clip` function ...
    boxes = boxes.reshape(-1, 4)
    _clip_box(boxes, image_shape)
    boxes = boxes.view(-1, num_bbox_reg_classes, 4)  # R x C x 4

    # Select max scores
    max_scores, max_classes = scores.max(1)  # R x C --> R
    num_objs = boxes.size(0)
    boxes = boxes.view(-1, 4)
    idxs = torch.arange(num_objs).to(boxes.device) * num_bbox_reg_classes + max_classes
    max_boxes = boxes[idxs]  # Select max boxes according to the max scores.

    # Apply NMS
    keep = nms(max_boxes, max_scores, nms_thresh)
    keep = keep[:maxd]
    if keep.shape[-1] >= mind and keep.shape[-1] <= maxd:
        max_boxes, max_scores = max_boxes[keep], max_scores[keep]
        classes = max_classes[keep]
        return max_boxes, max_scores, classes, keep
    else:


SUMMARY:
**Function Summary: Non-Maximum Suppression (NMS) for Object Detection**

### Purpose and Functionality

This function implements Non-Maximum Suppression (NMS) on bounding boxes and their corresponding scores. It takes in input data `boxes`, `scores`, `image_shape`, `score_thresh`, `nms_thresh`, `mind`, and `maxd` to filter out overlapping objects.

### Key Implementation Details

*   Selects the maximum scores for each class across all images.
*   Applies NMS using the `nms` function, which takes as input the top-scoring boxes and scores.
*   Returns the filtered boxes, scores, and classes, along with a mask indicating kept objects.

### Notable Dependencies or Requirements

*   Uses PyTorch library for tensor operations (`torch.arange`, `torch.device`, etc.)
*   References external function `_clip_box` (not shown in this code snippet) for box clipping.

### Signature Information

*   Function signature: `do_nms(boxes, scores, image_shape, score_thresh, nms_thresh, mind, maxd)`
*   Parameters:
    *   `boxes`: Input bounding boxes tensor
    *   `scores`: Input confidence scores tensor
    *   `image_shape`: Shape of the input images
    *   `score_thresh`: Threshold for discarding low-scored boxes
    *   `nms_thresh`: NMS threshold for merging overlapping boxes
    *   `mind` and `maxd`: Minimum and maximum number of kept objects, respectively

### Return Value

*   Returns a tuple containing:
    *   `max_boxes`, `scores`, and `classes` tensors representing the filtered bounding boxes, scores, and classes.
    *   A mask tensor indicating which objects are kept (`keep`)

This function summary provides a concise overview of the NMS algorithm implementation in the given code chunk.


key: chroma:document
string_value:
CODE:
# Helper Functions
def _clip_box(tensor, box_size: Tuple[int, int]):
    assert torch.isfinite(tensor).all(), "Box tensor contains infinite or NaN!"
    h, w = box_size
    tensor[:, 0].clamp_(min=0, max=w)
    tensor[:, 1].clamp_(min=0, max=h)
    tensor[:, 2].clamp_(min=0, max=w)


SUMMARY:
**Summary**

 `_clip_box` function clips the given `tensor` to a specified rectangular box of size `(h, w)`. It ensures that each pixel value falls within the valid range defined by the box's bottom-left (`min`) and top-right (`max`) coordinates. The function asserts that all values in the tensor are finite (not infinite or Nans) before clipping.


**Key Implementation Details**

*   Clamps each channel of the tensor to its respective bounds:
    *   Channel 0: min = 0, max = `w`
    *   Channel 1: min = 0, max = `h`
    *   Channel 2 and 3 bound-check mirrors those for channels 0 and 1.

**Notable Dependencies or Requirements**

*   Uses PyTorch's tensor clipping functionality (`torch.isfinite`, `.clamp_`)
*   Assumes the input tensor has shape `(... , 4)` (x, y, channel, alpha)


key: chroma:document
string_value:
CODE:

def _nonempty_boxes(box, threshold: float = 0.0) -> torch.Tensor:
    widths = box[:, 2] - box[:, 0]
    heights = box[:, 3] - box[:, 1]
    keep = (widths > threshold) & (heights > threshold)


SUMMARY:
**Summary:**

The `_nonempty_boxes` function filters through bounding boxes (`box`) to select only those with non-zero areas above a specified `threshold`. It calculates box widths and heights, then identifies boxes where both dimensions exceed the threshold.

**Key Details:**

* `tensor box` ( Input bounding boxes )
* Filtered boolean tensor `keep` indicating validboxes.
 
No external dependencies required


key: chroma:document
string_value:
CODE:

def get_norm(norm, out_channels):
    if isinstance(norm, str):
        if len(norm) == 0:
            return None
        norm = {
            "BN": BatchNorm2d,
            "GN": lambda channels: nn.GroupNorm(32, channels),
            "nnSyncBN": nn.SyncBatchNorm,  # keep for debugging
            "": lambda x: x,
        }[norm]


SUMMARY:
**Summary**

`get_norm` function:

### Purpose and Functionality

Returns a normalized layer (BN, GN, or nnSyncBN) based on the input `norm` string. The returned layer's `out_channels` is passed to its constructor.

### Implementation Details

* Supports three normalization types: BN, GN, and nnSyncBN.
* For "": returns the identity function (`lambda x: x`).

**Notable Dependencies**

* `BatchNorm2d`, `nn.GroupNorm`, `nn.SyncBatchNorm`, and `nn` module are imported, but not explicitly listed in this summary.


key: chroma:document
string_value:
CODE:

def _create_grid_offsets(size: List[int], stride: int, offset: float, device):
    grid_height, grid_width = size
    shifts_x = torch.arange(
        offset * stride,
        grid_width * stride,
        step=stride,
        dtype=torch.float32,
        device=device,
    )
    shifts_y = torch.arange(
        offset * stride,
        grid_height * stride,
        step=stride,
        dtype=torch.float32,
        device=device,
    )

    shift_y, shift_x = torch.meshgrid(shifts_y, shifts_x)
    shift_x = shift_x.reshape(-1)
    shift_y = shift_y.reshape(-1)


SUMMARY:
**Summary:**

This function creates grid offsets for 2D spatial convolutional layers. It takes in the size of the convolutional filter (`size`), stride, and device (`device`) as inputs.

**Functionality:**

1. generates two arrays `shift_x` and `shift_y` representing the x and y coordinates of a grid with specified `stride`.
2. uses `torch.meshgrid` to create a 2D coordinate system for the grid.
3. reshapes `shift_x` and `shift_y` into 1D arrays.

**Notable Implementation Details:**

* Uses `torch.arange` to generate discrete coordinates along x and y axes with specified stride.
* Utilizes `torch.meshgrid` for creating a 2D coordinate system for the grid.
* Converts coordinates from device (`device`) using `dtype=torch.float32`.

**Input/Output:**

* **Inputs:** `size` (Tuple[Two integers]), `stride` (integer), `offset` (float), `device` (torch Device)
* **Outputs:** Two tensors `shift_x`, `shift_y`


key: chroma:document
string_value:
CODE:

def build_backbone(cfg):
    input_shape = ShapeSpec(channels=len(cfg.MODEL.PIXEL_MEAN))
    norm = cfg.RESNETS.NORM
    stem = BasicStem(
        in_channels=input_shape.channels,
        out_channels=cfg.RESNETS.STEM_OUT_CHANNELS,
        norm=norm,
        caffe_maxpool=cfg.MODEL.MAX_POOL,
    )
    freeze_at = cfg.BACKBONE.FREEZE_AT

    if freeze_at >= 1:
        for p in stem.parameters():
            p.requires_grad = False

    out_features = cfg.RESNETS.OUT_FEATURES
    depth = cfg.RESNETS.DEPTH
    num_groups = cfg.RESNETS.NUM_GROUPS
    width_per_group = cfg.RESNETS.WIDTH_PER_GROUP
    bottleneck_channels = num_groups * width_per_group
    in_channels = cfg.RESNETS.STEM_OUT_CHANNELS
    out_channels = cfg.RESNETS.RES2_OUT_CHANNELS
    stride_in_1x1 = cfg.RESNETS.STRIDE_IN_1X1
    res5_dilation = cfg.RESNETS.RES5_DILATION
    assert res5_dilation in {1, 2}, "res5_dilation cannot be {}.".format(res5_dilation)

    num_blocks_per_stage = {50: [3, 4, 6, 3], 101: [3, 4, 23, 3], 152: [3, 8, 36, 3]}[depth]

    stages = []
    out_stage_idx = [{"res2": 2, "res3": 3, "res4": 4, "res5": 5}[f] for f in out_features]
    max_stage_idx = max(out_stage_idx)
    for idx, stage_idx in enumerate(range(2, max_stage_idx + 1)):
        dilation = res5_dilation if stage_idx == 5 else 1
        first_stride = 1 if idx == 0 or (stage_idx == 5 and dilation == 2) else 2
        stage_kargs = {
            "num_blocks": num_blocks_per_stage[idx],
            "first_stride": first_stride,
            "in_channels": in_channels,
            "bottleneck_channels": bottleneck_channels,
            "out_channels": out_channels,
            "num_groups": num_groups,
            "norm": norm,
            "stride_in_1x1": stride_in_1x1,
            "dilation": dilation,
        }

        stage_kargs["block_class"] = BottleneckBlock
        blocks = ResNet.make_stage(**stage_kargs)
        in_channels = out_channels
        out_channels *= 2
        bottleneck_channels *= 2

        if freeze_at >= stage_idx:
            for block in blocks:
                block.freeze()
        stages.append(blocks)



SUMMARY:
**Summary:**

The `build_backbone` function constructs the ResNet backbone network based on configuration parameters (`cfg`). It comprises the following key components:

*   **Initialization**: Initializes input shape, norm, and stem module with configurable hyperparameters.
*   **Freeze At Mechanism**: Optionally freezes layers from a specified stage index or before it, depending on `freeze_at`.
*   **Stage Construction**: Iterates through stages (ranging from 2 to the maximum out feature stage), constructing each stage using the `ResNet.make_stage` functionality. Each stage contains:
    *   Specified number of blocks (`num_blocks`) within it.
    *   First stride configuration based on the current stage index (`first_stride`).
    *   Bottleneck channel calculation (`bottleneck_channels`).
*   The final generated ResNet backbone network is composed of the stem module and constructed stages, with an out feature dimension specified by `out_features`.

**Key Implementation Details:**

*   The `ResNeFtnorm` norm layer uses configuration parameters (`norm`) from `cfg_RESNETS_NORM`.
*   Block class mapping is hardcoded to use  `BottleneckBlock` due to the lack of clear specification in the given code snippet.
*   ResNet modules are created using the factory function `make_stage`, which requires positional arguments passed through `stage_kargs`.

**Notable Dependencies:**

*   The configuration (`cfg`) object from resnet models.
*   External dependencies include:
    *   `BasicStem`
    *   `BottleneckBlock` is a hardcoded block class due to the lack of specification in the code snippet.

** Interfaces:**

*   Exposed interface is the generated ResNet backbone network instance, with attributes for stem and constructed stages.


key: chroma:document
string_value:
CODE:

def find_top_rpn_proposals(
    proposals,
    pred_objectness_logits,
    images,
    image_sizes,
    nms_thresh,
    pre_nms_topk,
    post_nms_topk,
    min_box_side_len,
    training,
):
    """Args:
        proposals (list[Tensor]): (L, N, Hi*Wi*A, 4).
        pred_objectness_logits: tensors of length L.
        nms_thresh (float): IoU threshold to use for NMS
        pre_nms_topk (int): before nms
        post_nms_topk (int): after nms
        min_box_side_len (float): minimum proposal box side
        training (bool): True if proposals are to be used in training,
    Returns:
        results (List[Dict]): stores post_nms_topk object proposals for image i.
    """
    num_images = len(images)
    device = proposals[0].device

    # 1. Select top-k anchor for every level and every image
    topk_scores = []  # #lvl Tensor, each of shape N x topk
    topk_proposals = []
    level_ids = []  # #lvl Tensor, each of shape (topk,)
    batch_idx = torch.arange(num_images, device=device)
    for level_id, proposals_i, logits_i in zip(itertools.count(), proposals, pred_objectness_logits):
        Hi_Wi_A = logits_i.shape[1]
        num_proposals_i = min(pre_nms_topk, Hi_Wi_A)

        # sort is faster than topk (https://github.com/pytorch/pytorch/issues/22812)
        # topk_scores_i, topk_idx = logits_i.topk(num_proposals_i, dim=1)
        logits_i, idx = logits_i.sort(descending=True, dim=1)
        topk_scores_i = logits_i[batch_idx, :num_proposals_i]
        topk_idx = idx[batch_idx, :num_proposals_i]

        # each is N x topk
        topk_proposals_i = proposals_i[batch_idx[:, None], topk_idx]  # N x topk x 4

        topk_proposals.append(topk_proposals_i)
        topk_scores.append(topk_scores_i)
        level_ids.append(torch.full((num_proposals_i,), level_id, dtype=torch.int64, device=device))

    # 2. Concat all levels together
    topk_scores = torch.cat(topk_scores, dim=1)
    topk_proposals = torch.cat(topk_proposals, dim=1)
    level_ids = torch.cat(level_ids, dim=0)

    # if I change to batched_nms, I wonder if this will make a difference
    # 3. For each image, run a per-level NMS, and choose topk results.
    results = []
    for n, image_size in enumerate(image_sizes):
        boxes = topk_proposals[n]
        scores_per_img = topk_scores[n]
        # I will have to take a look at the boxes clip method
        _clip_box(boxes, image_size)
        # filter empty boxes
        keep = _nonempty_boxes(boxes, threshold=min_box_side_len)
        lvl = level_ids
        if keep.sum().item() != len(boxes):
            boxes, scores_per_img, lvl = (
                boxes[keep],
                scores_per_img[keep],
                level_ids[keep],
            )

        keep = batched_nms(boxes, scores_per_img, lvl, nms_thresh)
        keep = keep[:post_nms_topk]

        res = (boxes[keep], scores_per_img[keep])
        results.append(res)

    # I wonder if it would be possible for me to pad all these things.


SUMMARY:
**Summary: `find_top_rpn_proposals` function**

This function takes in a list of proposals, objectness logits, images, and other parameters to find top-performing anchor boxes for object detection.

**Main Purpose:**
The primary purpose is to select top-performing anchor boxes for object detection using an RPN (Region Proposal Network) approach.

**Implementation Details:**

1. The function first selects the top-k anchors for each level and image using a sorting-based approach instead of `topk`.
2. It then concatenates all levels together and applies per-level Non-Maximum Suppression (NMS).
3. Finally, it filters out empty boxes, applies batched NMS, and returns the remaining top-performing anchor boxes.

**Important Components:**

* `proposals`: a list of tensors representing proposal boxes.
* `pred_objectness_logits`: tensors of length `L` containing objectness scores.
* `nms_thresh`, `pre_nms_topk`, `post_nms_topk`, and `min_box_side_len`: hyperparameters controlling the detection process.

**Notable Dependencies:**

* PyTorch: used for tensor operations and NMS implementation.

This summary captures the key functionality and implementation details of the `find_top_rpn_proposals` function, making it easier to understand and use in a RAG system.


key: chroma:document
string_value:
CODE:

def subsample_labels(labels, num_samples, positive_fraction, bg_label):
    """
    Returns:
        pos_idx, neg_idx (Tensor):
            1D vector of indices. The total length of both is `num_samples` or fewer.
    """
    positive = torch.nonzero((labels != -1) & (labels != bg_label)).squeeze(1)
    negative = torch.nonzero(labels == bg_label).squeeze(1)

    num_pos = int(num_samples * positive_fraction)
    # protect against not enough positive examples
    num_pos = min(positive.numel(), num_pos)
    num_neg = num_samples - num_pos
    # protect against not enough negative examples
    num_neg = min(negative.numel(), num_neg)

    # randomly select positive and negative examples
    perm1 = torch.randperm(positive.numel(), device=positive.device)[:num_pos]
    perm2 = torch.randperm(negative.numel(), device=negative.device)[:num_neg]

    pos_idx = positive[perm1]
    neg_idx = negative[perm2]


SUMMARY:
**Summary:**

The `subsample_labels` function subsamples labels to meet a specified fraction of positives and negatives while ensuring sufficient numbers of each. It takes in:

* `labels`: original labeled tensor
* `num_samples`: total number of samples to be selected
* `positive_fraction`: proportion of positive samples (as a float)
* `bg_label`: background label

The function returns two tensors: `pos_idx` containing the indices of subsampled positive examples and `neg_idx` containing the indices of subsampled negative examples.

**Key Implementation Details:**

* Uses `torch.nonzero` to extract indices of non-background and neutral labels
* Randomly selects a specified fraction of positive and negative examples using `torch.randperm`
* Ensures sufficient numbers of each by capping at available negative or positive exemplars

**Notable Dependencies and Requirements:**

* PyTorch library (explicit use of `torch` and `torch.nonzero`)
* No external dependencies listed in the provided code


key: chroma:document
string_value:
CODE:

def add_ground_truth_to_proposals(gt_boxes, proposals):


SUMMARY:
**Function Summary**

`add_ground_truth_to_proposals`: Adds ground truth bounding boxes (`gt_boxes`) to predicted proposal rectangles (`proposals`). Requires implementation.

**Key Details**

* Primary purpose: Integrates prediction results with corresponding ground truth annotations.
* Implementation details:
	+ Method signature: `add_ground_truth_to_proposals(gt_boxes, proposals)`
	+ Interface expected to be implemented by child classes: `ノotedAbstractMethod` (base class not provided in the snippet)

**Dependencies**

* None explicitly listed
* Assumes presence of: `gt_boxes`, `proposals`, and potentially additional data structures or objects required for implementation.


key: chroma:document
string_value:
CODE:

def add_ground_truth_to_proposals_single_image(gt_boxes, proposals):


SUMMARY:
**Summary:**

`add_ground_truth_to_proposals_single_image`: A method to be implemented, responsible for adding ground truth bounding boxes (`gt_boxes`) to proposal candidates (`proposals`) in a single image context. The primary purpose is to refine the proposals by incorporating the actual ground truth annotations.

**Key Implementation Details:**

- Expected input parameters: `gt_boxes` and `proposales`
  - `gt_boxes`: A list of bounding box coordinates representing the actual ground truth objects.
  - `proposals`: A list of proposal candidates, potentially generated from object detection models or other sources.

This method implementation is expected to further filter/prioritize proposals based on the provided ground truth annotations.


key: chroma:document
string_value:
CODE:

def _fmt_box_list(box_tensor, batch_index: int):
    repeated_index = torch.full(
        (len(box_tensor), 1),
        batch_index,
        dtype=box_tensor.dtype,
        device=box_tensor.device,
    )


SUMMARY:
**Summary:**

`_fmt_box_list` function concatenates a repeated index tensor to a `box_tensor`, forming a stacked tensor.

**Key implementation details:**

- Repeats the batch index for each element in `box_tensor`.
- Concatenates `repeated_index` with `box_tensor` along the 2nd dimension (axis=1).
- Returns the resulting concatenated tensor.

**Notable dependencies and requirements:**

* `torch`: required for tensor operations.


key: chroma:document
string_value:
CODE:

def convert_boxes_to_pooler_format(box_lists: List[torch.Tensor]):
    pooler_fmt_boxes = torch.cat(
        [_fmt_box_list(box_list, i) for i, box_list in enumerate(box_lists)],
        dim=0,
    )


SUMMARY:
**Summary**

Converting Box Lists to Pooler Format.

**Main Purpose and Functionality**: Converts a list of 3D tensors representing boxes to the format expected by a detector using the `box_list` formatting function (`_fmt_box_list`) for each input box list, concatenating them along the first dimension (batch size).

**Key Implementation Details**:

- Uses `_fmt_box_list(box_list, i)` to convert individual box lists to pooler format.
- Concatenates converted boxes using `torch.cat(dim=0)`.
- Returns the concatenated tensor in pooler format.

**Notable Dependencies**: None noted.


key: chroma:document
string_value:
CODE:

def assign_boxes_to_levels(
    box_lists: List[torch.Tensor],
    min_level: int,
    max_level: int,
    canonical_box_size: int,
    canonical_level: int,
):
    box_sizes = torch.sqrt(torch.cat([boxes.area() for boxes in box_lists]))
    # Eqn.(1) in FPN paper
    level_assignments = torch.floor(canonical_level + torch.log2(box_sizes / canonical_box_size + 1e-8))
    # clamp level to (min, max), in case the box size is too large or too small
    # for the available feature maps
    level_assignments = torch.clamp(level_assignments, min=min_level, max=max_level)


SUMMARY:
**Summary:**

The `assign_boxes_to_levels` function assigns Boxes to levels in FPN-based object detection. It takes in box lists, minimum and maximum levels, canonical box size, and canonical level.

**Main Purpose:** Assigns boxes to levels based on their areas relative to a canonical box size.

**Key Implementation Details:**

* Calculates the area of each box and sums them up (Eqn.(1) in FPN paper).
* Normalizes the sum by dividing with the canonical box size, then applies log2 scaling.
* Applies clamp boundary value checking.
* Converts result to integer tensor.

**Notable Dependencies/Requirements:**

* `torch` library for tensor operations
* `box_lists` (List[torch.Tensor]) containing Box objects
* External dependencies:
	+ PyTorch's `torch.sqrt`, `torch.log2`, and `torch.clamp`
	+ FPN paper Eqn.(1) reference


key: chroma:document
string_value:
CODE:
# Helper Classes
class _NewEmptyTensorOp(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x, new_shape):
        ctx.shape = x.shape
        return x.new_empty(new_shape)

    @staticmethod
    def backward(ctx, grad):
        shape = ctx.shape


SUMMARY:
**Summary**

`_NewEmptyTensorOp` is a PyTorch autograd function that creates a new empty tensor with the specified shape. Its primary purpose is to initialize an empty tensor within a computational graph.

**Key Implementation Details:**

* `forward(ctx, x, new_shape)`: Creates a new empty tensor with shape `new_shape` using the original tensor `x`. Set's the intermediate shape in context `ctx` for backpropagation.
* `backward(ctx, grad)`: Applies backward pass to the gradient of the new tensor `grad`, propagating the error back to the input tensor. Returns a tuple containing the applied gradient and None (i.e., no input gradient is returned).

**Notable Dependencies and Requirements:**

* Utilizes PyTorch tensor operations (`x.new_empty(new_shape)`)
* Leverages PyTorch autograd functionality
* Depends on `torch.autograd.Function` as base class


key: chroma:document
string_value:
CODE:

class ShapeSpec(namedtuple("_ShapeSpec", ["channels", "height", "width", "stride"])):
    def __new__(cls, *, channels=None, height=None, width=None, stride=None):


SUMMARY:
**Summary:**

The `_ShapeSpec` class is a named tuple that represents specifications for an image processing tensor. It consists of four attributes:

*   `channels`: integer value representing the number of color channels.
*   `height`: integer value representing the height of the tensor in pixels.
*   `width`: integer value representing the width of the tensor in pixels.
*   `stride`: integer value representing the stride (step size) for iterating over the tensor.

This class provides a simple and efficient way to store and transport image processing tensor specifications, likely used in deep learning or computer vision applications. 

**Key Implementation Details:**

The main implementation detail is the use of a named tuple as a factory method `__new__()` which takes additional keyword-only arguments (`channels`, `height`, `width`, `stride`) for its attribute initialization.

**Notable Dependencies or Requirements:**

None, as this class only relies on Python's built-in `namedtuple` and does not import any external dependencies.


key: chroma:document
string_value:
CODE:

class Box2BoxTransform:
    """
    This R-CNN transformation scales the box's width and height
    by exp(dw), exp(dh) and shifts a box's center by the offset
    (dx * width, dy * height).
    """

    def __init__(self, weights: Tuple[float, float, float, float], scale_clamp: float = None):
        """
        Args:
            weights (4-element tuple): Scaling factors that are applied to the
                (dx, dy, dw, dh) deltas. In Fast R-CNN, these were originally set
                such that the deltas have unit variance; now they are treated as
                hyperparameters of the system.
            scale_clamp (float): When predicting deltas, the predicted box scaling
                factors (dw and dh) are clamped such that they are <= scale_clamp.
        """
        self.weights = weights
        if scale_clamp is not None:
            self.scale_clamp = scale_clamp
        else:
            """
            Value for clamping large dw and dh predictions.
            The heuristic is that we clamp such that dw and dh are no larger
            than what would transform a 16px box into a 1000px box
            (based on a small anchor, 16px, and a typical image size, 1000px).
            """
            self.scale_clamp = math.log(1000.0 / 16)

    def get_deltas(self, src_boxes, target_boxes):
        """
        Get box regression transformation deltas (dx, dy, dw, dh) that can be used
        to transform the `src_boxes` into the `target_boxes`. That is, the relation
        ``target_boxes == self.apply_deltas(deltas, src_boxes)`` is true (unless
        any delta is too large and is clamped).
        Args:
            src_boxes (Tensor): source boxes, e.g., object proposals
            target_boxes (Tensor): target of the transformation, e.g., ground-truth
                boxes.
        """
        assert isinstance(src_boxes, torch.Tensor), type(src_boxes)
        assert isinstance(target_boxes, torch.Tensor), type(target_boxes)

        src_widths = src_boxes[:, 2] - src_boxes[:, 0]
        src_heights = src_boxes[:, 3] - src_boxes[:, 1]
        src_ctr_x = src_boxes[:, 0] + 0.5 * src_widths
        src_ctr_y = src_boxes[:, 1] + 0.5 * src_heights

        target_widths = target_boxes[:, 2] - target_boxes[:, 0]
        target_heights = target_boxes[:, 3] - target_boxes[:, 1]
        target_ctr_x = target_boxes[:, 0] + 0.5 * target_widths
        target_ctr_y = target_boxes[:, 1] + 0.5 * target_heights

        wx, wy, ww, wh = self.weights
        dx = wx * (target_ctr_x - src_ctr_x) / src_widths
        dy = wy * (target_ctr_y - src_ctr_y) / src_heights
        dw = ww * torch.log(target_widths / src_widths)
        dh = wh * torch.log(target_heights / src_heights)

        deltas = torch.stack((dx, dy, dw, dh), dim=1)
        assert (src_widths > 0).all().item(), "Input boxes to Box2BoxTransform are not valid!"
        return deltas

    def apply_deltas(self, deltas, boxes):
        """
        Apply transformation `deltas` (dx, dy, dw, dh) to `boxes`.
        Args:
            deltas (Tensor): transformation deltas of shape (N, k*4), where k >= 1.
                deltas[i] represents k potentially different class-specific
                box transformations for the single box boxes[i].
            boxes (Tensor): boxes to transform, of shape (N, 4)
        """
        boxes = boxes.to(deltas.dtype)

        widths = boxes[:, 2] - boxes[:, 0]
        heights = boxes[:, 3] - boxes[:, 1]
        ctr_x = boxes[:, 0] + 0.5 * widths
        ctr_y = boxes[:, 1] + 0.5 * heights

        wx, wy, ww, wh = self.weights
        dx = deltas[:, 0::4] / wx
        dy = deltas[:, 1::4] / wy
        dw = deltas[:, 2::4] / ww
        dh = deltas[:, 3::4] / wh

        # Prevent sending too large values into torch.exp()
        dw = torch.clamp(dw, max=self.scale_clamp)
        dh = torch.clamp(dh, max=self.scale_clamp)

        pred_ctr_x = dx * widths[:, None] + ctr_x[:, None]
        pred_ctr_y = dy * heights[:, None] + ctr_y[:, None]
        pred_w = torch.exp(dw) * widths[:, None]
        pred_h = torch.exp(dh) * heights[:, None]

        pred_boxes = torch.zeros_like(deltas)
        pred_boxes[:, 0::4] = pred_ctr_x - 0.5 * pred_w  # x1
        pred_boxes[:, 1::4] = pred_ctr_y - 0.5 * pred_h  # y1
        pred_boxes[:, 2::4] = pred_ctr_x + 0.5 * pred_w  # x2
        pred_boxes[:, 3::4] = pred_ctr_y + 0.5 * pred_h  # y2


SUMMARY:
**Class Summary: Box2BoxTransform**

The `Box2BoxTransform` class is used to scale box width and height by exponential factors (`exp(dw)`, `exp(dh)`) and shift the center point by offsets based on `dx * width` and `dy * height`. It serves as a transformation layer in object detection models.

**Key Implementation Details:**

* The transformation deltas are calculated using `(target_ctr_x - src_ctr_x) / src_widths`, `(target_ctr_y - src_ctr_y) / src_heights`, `torch.log(target_widths / src_widths)`, and `torch.log(target_heights / src_heights)` respectively.
* The transformations for different boxes (including class-specific ones) are stacked together using `torch.stack((dx, dy, dw, dh), dim=1)`.
* Clamping of large `dw` and `dh` predictions is implemented to prevent sending too large values into `torch.exp()`.

**Notable Dependencies/Interfaces:**

* `torch.Tensor` for box representations
* `Tuple[float, float, float, float]` for weights and scale clamp value

**External Dependencies:**
- torch (Tensor library)
- math library (for log function)


key: chroma:document
string_value:
CODE:

class Matcher:
    """
    This class assigns to each predicted "element" (e.g., a box) a ground-truth
    element. Each predicted element will have exactly zero or one matches; each
    ground-truth element may be matched to zero or more predicted elements.
    The matching is determined by the MxN match_quality_matrix, that characterizes
    how well each (ground-truth, prediction)-pair match each other. For example,
    if the elements are boxes, this matrix may contain box intersection-over-union
    overlap values.
    The matcher returns (a) a vector of length N containing the index of the
    ground-truth element m in [0, M) that matches to prediction n in [0, N).
    (b) a vector of length N containing the labels for each prediction.
    """

    def __init__(
        self,
        thresholds: List[float],
        labels: List[int],
        allow_low_quality_matches: bool = False,
    ):
        """
        Args:
            thresholds (list): a list of thresholds used to stratify predictions
                into levels.
            labels (list): a list of values to label predictions belonging at
                each level. A label can be one of {-1, 0, 1} signifying
                {ignore, negative class, positive class}, respectively.
            allow_low_quality_matches (bool): if True, produce additional matches or predictions with maximum match quality lower than high_threshold.
                For example, thresholds = [0.3, 0.5] labels = [0, -1, 1] All predictions with iou < 0.3 will be marked with 0 and
                thus will be considered as false positives while training. All predictions with 0.3 <= iou < 0.5 will be marked with -1 and
                thus will be ignored. All predictions with 0.5 <= iou will be marked with 1 and thus will be considered as true positives.
        """
        thresholds = thresholds[:]
        assert thresholds[0] > 0
        thresholds.insert(0, -float("inf"))
        thresholds.append(float("inf"))
        assert all(low <= high for (low, high) in zip(thresholds[:-1], thresholds[1:]))
        assert all(label_i in [-1, 0, 1] for label_i in labels)
        assert len(labels) == len(thresholds) - 1
        self.thresholds = thresholds
        self.labels = labels
        self.allow_low_quality_matches = allow_low_quality_matches

    def __call__(self, match_quality_matrix):
        """
        Args:
            match_quality_matrix (Tensor[float]): an MxN tensor, containing the pairwise quality between M ground-truth elements and N predicted
                elements. All elements must be >= 0 (due to the us of `torch.nonzero` for selecting indices in :meth:`set_low_quality_matches_`).
        Returns:
            matches (Tensor[int64]): a vector of length N, where matches[i] is a matched ground-truth index in [0, M)
            match_labels (Tensor[int8]): a vector of length N, where pred_labels[i] indicates true or false positive or ignored
        """
        assert match_quality_matrix.dim() == 2
        if match_quality_matrix.numel() == 0:
            default_matches = match_quality_matrix.new_full((match_quality_matrix.size(1),), 0, dtype=torch.int64)
            # When no gt boxes exist, we define IOU = 0 and therefore set labels
            # to `self.labels[0]`, which usually defaults to background class 0
            # To choose to ignore instead,
            # can make labels=[-1,0,-1,1] + set appropriate thresholds
            default_match_labels = match_quality_matrix.new_full(
                (match_quality_matrix.size(1),), self.labels[0], dtype=torch.int8
            )
            return default_matches, default_match_labels

        assert torch.all(match_quality_matrix >= 0)

        # match_quality_matrix is M (gt) x N (predicted)
        # Max over gt elements (dim 0) to find best gt candidate for each prediction
        matched_vals, matches = match_quality_matrix.max(dim=0)

        match_labels = matches.new_full(matches.size(), 1, dtype=torch.int8)

        for l, low, high in zip(self.labels, self.thresholds[:-1], self.thresholds[1:]):
            low_high = (matched_vals >= low) & (matched_vals < high)
            match_labels[low_high] = l

        if self.allow_low_quality_matches:
            self.set_low_quality_matches_(match_labels, match_quality_matrix)

        return matches, match_labels

    def set_low_quality_matches_(self, match_labels, match_quality_matrix):
        """
        Produce additional matches for predictions that have only low-quality matches.
        Specifically, for each ground-truth G find the set of predictions that have
        maximum overlap with it (including ties); for each prediction in that set, if
        it is unmatched, then match it to the ground-truth G.
        This function implements the RPN assignment case (i)
        in Sec. 3.1.2 of Faster R-CNN.
        """
        # For each gt, find the prediction with which it has highest quality
        highest_quality_foreach_gt, _ = match_quality_matrix.max(dim=1)
        # Find the highest quality match available, even if it is low, including ties.
        # Note that the matches qualities must be positive due to the use of
        # `torch.nonzero`.
        of_quality_inds = match_quality_matrix == highest_quality_foreach_gt[:, None]
        if of_quality_inds.dim() == 0:
            (_, pred_inds_with_highest_quality) = of_quality_inds.unsqueeze(0).nonzero().unbind(1)
        else:
            (_, pred_inds_with_highest_quality) = of_quality_inds.nonzero().unbind(1)


SUMMARY:
**Matcher Class Summary**

The `Matcher` class assigns each predicted element to a ground-truth element based on the `match_quality_matrix`. The primary purpose is to determine the best match between predicted and ground-truth elements.

**Key Implementation Details**

*   The `__init__` method initializes the matcher with thresholds and labels, allowing low-quality matches.
*   The `__call__` method takes a match quality matrix as input and returns:
    *   A vector of matched ground-truth indices,
    *   A vector of match labels indicating true positives, false positives, or ignored elements.

**Notable Dependencies and Requirements**

*   The `thresholds` list must be non-empty and contain unique values.
*   The `labels` list contains only valid label values (-1, 0, 1).
*   External dependencies are not explicitly mentioned in the provided code snippet. However, it can be inferred that the `match_quality_matrix` should be a tensor representing the pairwise quality between ground-truth and predicted elements.

**Example Signature**

```python
matcher = Matcher(thresholds=[0.3, 0.5], labels=[0, -1, 1])
matches, match_labels = matcher(None)
```

This summary provides a concise overview of the `Matcher` class, its key implementation details, and notable dependencies or requirements.


key: chroma:document
string_value:
CODE:

class RPNOutputs:
    def __init__(
        self,
        box2box_transform,
        anchor_matcher,
        batch_size_per_image,
        positive_fraction,
        images,
        pred_objectness_logits,
        pred_anchor_deltas,
        anchors,
        boundary_threshold=0,
        gt_boxes=None,
        smooth_l1_beta=0.0,
    ):
        """
        Args:
            box2box_transform (Box2BoxTransform): :class:`Box2BoxTransform` instance for anchor-proposal transformations.
            anchor_matcher (Matcher): :class:`Matcher` instance for matching anchors to ground-truth boxes; used to determine training labels.
            batch_size_per_image (int): number of proposals to sample when training
            positive_fraction (float): target fraction of sampled proposals that should be positive
            images (ImageList): :class:`ImageList` instance representing N input images
            pred_objectness_logits (list[Tensor]): A list of L elements. Element i is a tensor of shape (N, A, Hi, W)
            pred_anchor_deltas (list[Tensor]): A list of L elements. Element i is a tensor of shape (N, A*4, Hi, Wi)
            anchors (list[torch.Tensor]): nested list of boxes. anchors[i][j] at (n, l) stores anchor array for feature map l
            boundary_threshold (int): if >= 0, then anchors that extend beyond the image boundary by more than boundary_thresh are not used in training.
            gt_boxes (list[Boxes], optional): A list of N elements.
            smooth_l1_beta (float): The transition point between L1 and L2 lossn. When set to 0, the loss becomes L1. When +inf, it is ignored
        """
        self.box2box_transform = box2box_transform
        self.anchor_matcher = anchor_matcher
        self.batch_size_per_image = batch_size_per_image
        self.positive_fraction = positive_fraction
        self.pred_objectness_logits = pred_objectness_logits
        self.pred_anchor_deltas = pred_anchor_deltas

        self.anchors = anchors
        self.gt_boxes = gt_boxes
        self.num_feature_maps = len(pred_objectness_logits)
        self.num_images = len(images)
        self.boundary_threshold = boundary_threshold
        self.smooth_l1_beta = smooth_l1_beta

    def _get_ground_truth(self):
        raise NotImplementedError()

    def predict_proposals(self):
        # pred_anchor_deltas: (L, N, ? Hi, Wi)
        # anchors:(N, L, -1, B)
        # here we loop over specific feature map, NOT images
        proposals = []
        anchors = self.anchors.transpose(0, 1)
        for anchors_i, pred_anchor_deltas_i in zip(anchors, self.pred_anchor_deltas):
            B = anchors_i.size(-1)
            N, _, Hi, Wi = pred_anchor_deltas_i.shape
            anchors_i = anchors_i.flatten(start_dim=0, end_dim=1)
            pred_anchor_deltas_i = pred_anchor_deltas_i.view(N, -1, B, Hi, Wi).permute(0, 3, 4, 1, 2).reshape(-1, B)
            proposals_i = self.box2box_transform.apply_deltas(pred_anchor_deltas_i, anchors_i)
            # Append feature map proposals with shape (N, Hi*Wi*A, B)
            proposals.append(proposals_i.view(N, -1, B))
        proposals = torch.stack(proposals)
        return proposals

    def predict_objectness_logits(self):
        """
        Returns:
            pred_objectness_logits (list[Tensor]) -> (N, Hi*Wi*A).
        """
        pred_objectness_logits = [
            # Reshape: (N, A, Hi, Wi) -> (N, Hi, Wi, A) -> (N, Hi*Wi*A)
            score.permute(0, 2, 3, 1).reshape(self.num_images, -1)
            for score in self.pred_objectness_logits
        ]


SUMMARY:
Here is a concise summary of the `RPNOutputs` class:

**Main Purpose and Functionality:**

The `RPNOutputs` class represents outputs from a Region Proposal Network (RPN). Its primary function is to process predictions from anchor networks, applying transformations and transformations to output proposal rectangles.

**Key Implementation Details:**

* The `_get_ground_truth` method is abstractly implemented.
* The `predict_proposals` method takes processed feature maps (`pred_objectness_logits` and `pred_anchor_deltas`) and applies box transformations using the `box2box_transform`.
* It outputs a list of proposals for each feature map, with proposal rectangles transformed according to groundtruth box coordinates.

**Notable Dependencies:**

* `Box2BoxTransform`, `Matcher`
* External dependencies:
	+ `ImageList` (from the Model zoo)

This summary aims to preserve class/function/variable names and states the primary purpose. It narrows down key implementation details, highlighting algorithms used in processing anchor outputs and applying transformation operations.


key: chroma:document
string_value:
CODE:
# Main Classes
class Conv2d(nn.Conv2d):
    def __init__(self, *args, **kwargs):
        norm = kwargs.pop("norm", None)
        activation = kwargs.pop("activation", None)
        super().__init__(*args, **kwargs)

        self.norm = norm
        self.activation = activation

    def forward(self, x):
        if x.numel() == 0 and self.training:
            assert not isinstance(self.norm, nn.SyncBatchNorm)
        if x.numel() == 0:
            assert not isinstance(self.norm, nn.GroupNorm)
            output_shape = [
                (i + 2 * p - (di * (k - 1) + 1)) // s + 1
                for i, p, di, k, s in zip(
                    x.shape[-2:],
                    self.padding,
                    self.dilation,
                    self.kernel_size,
                    self.stride,
                )
            ]
            output_shape = [x.shape[0], self.weight.shape[0]] + output_shape
            empty = _NewEmptyTensorOp.apply(x, output_shape)
            if self.training:
                _dummy = sum(x.view(-1)[0] for x in self.parameters()) * 0.0
                return empty + _dummy
            else:
                return empty

        x = super().forward(x)
        if self.norm is not None:
            x = self.norm(x)
        if self.activation is not None:
            x = self.activation(x)


SUMMARY:
**Summary**

The `Conv2d` class inherits from PyTorch's `nn.Conv2d` and customizes its behavior by adding input normalization (`norm`) and activation functions (`activation`). The class extends the original convolutional operation to include these features.

**Key Functions and Variables:**

* `forward(self, x)`: The main forward pass function that processes an input tensor `x`. It:
	+ Processes the output shape calculation for zero-padded inputs.
	+ Applies the forward pass of the parent `nn.Conv2d` class.
	+ Optionally applies normalization (`norm`) and activation (`activation`).
* `norm`: A PyTorch normalization layer (SyncBatchNorm or GroupNorm) to apply to the input if specified.
* `activation`: A PyTorch activation function (e.g., ReLU, Sigmoid) to apply to the output if specified.

**Notable Dependencies and Requirements:**

* PyTorch's `nn.Conv2d`
* `nn.SyncBatchNorm`
* `nn.GroupNorm`
* `_NewEmptyTensorOp` (a custom implementation)
* Various activation functions

**Main Purpose and Functionality:**
The primary purpose of this class is to extend the capability of the standard convolutional layer in PyTorch by adding optional normalization and activation functions. This allows for more flexible usage of convolutional layers in deep neural networks.


key: chroma:document
string_value:
CODE:

class LastLevelMaxPool(nn.Module):
    """
    This module is used in the original FPN to generate a downsampled P6 feature from P5.
    """

    def __init__(self):
        super().__init__()
        self.num_levels = 1
        self.in_feature = "p5"

    def forward(self, x):


SUMMARY:
**Code Summary:**

`LastLevelMaxPool(nn.Module)` is a PyTorch module that downsamples the `p5` feature by applying a single-level max pooling with kernel size 1 and stride 2. The output is a list of feature maps.

**Key Details:**

* Main Purpose: Downsample `p5` feature using max pooling.
* Implementation:
	+ Single-level max pooling with kernel size 1 and stride 2.
* Primary Input: `x` (feature map)
* Output: List of downsampled feature maps

**Notable Dependencies/Requirements:**

* `nn.Module` (PyTorch module)
* `nn.functional.max_pool2d` (PyTorch max pooling function)


key: chroma:document
string_value:
CODE:

class LastLevelP6P7(nn.Module):
    """
    This module is used in RetinaNet to generate extra layers, P6 and P7 from C5 feature.
    """

    def __init__(self, in_channels, out_channels):
        super().__init__()
        self.num_levels = 2
        self.in_feature = "res5"
        self.p6 = nn.Conv2d(in_channels, out_channels, 3, 2, 1)
        self.p7 = nn.Conv2d(out_channels, out_channels, 3, 2, 1)

    def forward(self, c5):
        p6 = self.p6(c5)
        p7 = self.p7(nn.functional.relu(p6))


SUMMARY:
**LastLevelP6P7 Class Summary**

* **Purpose**: Generates extra layers (P6 and P7) using C5 features in RetinaNet framework.

*   **Main Functionality**: Takes C5 feature as input and returns two intermediate features: P6 and P7.
*   **Key Implementation Details**:
    *   Uses convolutional layers (`nn.Conv2d`) to generate P6 and P7 features.
        *   `p6` takes input C5 features, resulting in a 3x3 feature map with stride 2.
        *   `p7` applies ReLU activation (from `nn.functional.relu`) after `p6`.
    *   Both `p6` and `p7` have kernel size = 3 and output channels = `out_channels`.

*   **Notable Dependencies**:
    *   Inherited from `nn.Module`: Base class for neural network modules.
        *   Superclass (`super().__init__()`) ensures module initialization.

**Method Signatures and Interfaces**

*   `__init__(in_channels, out_channels)`: Initializes the module with input channels and output channels.
*   `forward(c5)`: Takes input feature `c5` and returns two features: P6 and P7.


key: chroma:document
string_value:
CODE:

class BasicStem(nn.Module):
    def __init__(self, in_channels=3, out_channels=64, norm="BN", caffe_maxpool=False):
        super().__init__()
        self.conv1 = Conv2d(
            in_channels,
            out_channels,
            kernel_size=7,
            stride=2,
            padding=3,
            bias=False,
            norm=get_norm(norm, out_channels),
        )
        self.caffe_maxpool = caffe_maxpool
        # use pad 1 instead of pad zero

    def forward(self, x):
        x = self.conv1(x)
        x = nn.functional.relu_(x)
        if self.caffe_maxpool:
            x = nn.functional.max_pool2d(x, kernel_size=3, stride=2, padding=0, ceil_mode=True)
        else:
            x = nn.functional.max_pool2d(x, kernel_size=3, stride=2, padding=1)
        return x

    @property
    def out_channels(self):
        return self.conv1.out_channels

    @property
    def stride(self):


SUMMARY:
**Class Summary: BasicStem**

The `BasicStem` class is a deep learning module that serves as the entry point for convolutional neural networks (CNNs). It extends PyTorch's `nn.Module`.

**Primary Purpose**: This stem layer performs two consecutive operations:

1. A convolution operation with 64 channels using a kernel size of 7, stride 2, and no bias.
2. If `caffe_maxpool` is enabled, it applies maximum pooling with kernel size 3, stride 2, and ceiling mode; otherwise, it applies zero-padding-based max pooling.

**Notable Implementation Details**:

* Utilizes PyTorch's ReLU activation function (`nn.functional.relu_`) after the convolution layer.
* Uses padding 1 instead of pad 0 for maximum pooling when `caffe_maxpool` is disabled.
* Computes output channels using the `out_channels` property to maintain consistency.

**Dependencies/Interfaces**:

* Extends PyTorch's `nn.Module`.
* Imports:
	+ `Conv2d`
	+ `get_norm`
	+ `nn.functional.max_pool2d`
	+ `nn.functional.relu_`

Note: This summary focuses on the key aspects of the class, omitting redundant or generic details.


key: chroma:document
string_value:
CODE:

class ResNetBlockBase(nn.Module):
    def __init__(self, in_channels, out_channels, stride):
        super().__init__()
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.stride = stride

    def freeze(self):
        for p in self.parameters():
            p.requires_grad = False


SUMMARY:
**Summary**

The `ResNetBlockBase` class is a PyTorch neural network module implementing the basic building block of ResNeat architecture. It takes three input parameters:

*   `in_channels`: Number of channels in the input feature map
*   `out_channels`: Number of channels in the output feature map
*   `stride`: Stride size for convolutional layer

The class has two main components:

1.  `__init__` method: Initializes the layer with input parameters and stores them as instance variables.
2.  `freeze` method: Freezes all model parameters by setting their requires_grad flag to False, enabling batch normalization, and returns the updated module.

Notable Dependencies/Requirements:
-   PyTorch (`nn.Module`)
-   Batch Normalization


key: chroma:document
string_value:
CODE:

class BottleneckBlock(ResNetBlockBase):
    def __init__(
        self,
        in_channels,
        out_channels,
        bottleneck_channels,
        stride=1,
        num_groups=1,
        norm="BN",
        stride_in_1x1=False,
        dilation=1,
    ):
        super().__init__(in_channels, out_channels, stride)

        if in_channels != out_channels:
            self.shortcut = Conv2d(
                in_channels,
                out_channels,
                kernel_size=1,
                stride=stride,
                bias=False,
                norm=get_norm(norm, out_channels),
            )
        else:
            self.shortcut = None

        # The original MSRA ResNet models have stride in the first 1x1 conv
        # The subsequent fb.torch.resnet and Caffe2 ResNe[X]t implementations have
        # stride in the 3x3 conv
        stride_1x1, stride_3x3 = (stride, 1) if stride_in_1x1 else (1, stride)

        self.conv1 = Conv2d(
            in_channels,
            bottleneck_channels,
            kernel_size=1,
            stride=stride_1x1,
            bias=False,
            norm=get_norm(norm, bottleneck_channels),
        )

        self.conv2 = Conv2d(
            bottleneck_channels,
            bottleneck_channels,
            kernel_size=3,
            stride=stride_3x3,
            padding=1 * dilation,
            bias=False,
            groups=num_groups,
            dilation=dilation,
            norm=get_norm(norm, bottleneck_channels),
        )

        self.conv3 = Conv2d(
            bottleneck_channels,
            out_channels,
            kernel_size=1,
            bias=False,
            norm=get_norm(norm, out_channels),
        )

    def forward(self, x):
        out = self.conv1(x)
        out = nn.functional.relu_(out)

        out = self.conv2(out)
        out = nn.functional.relu_(out)

        out = self.conv3(out)

        if self.shortcut is not None:
            shortcut = self.shortcut(x)
        else:
            shortcut = x

        out += shortcut
        out = nn.functional.relu_(out)


SUMMARY:
**Class Summary: BottleneckBlock**

**Purpose and Functionality**
---------------------------

The `BottleneckBlock` class implements the bottleneck structure in ResNet models, composed of three consecutive convolutional blocks separated by two 1x1 convolutions. The block takes an input `in_channels`, performs a series of transformations, and produces an output.

**Key Implementation Details**

*   Three convolutional blocks with different kernel sizes and strides:
    *   `conv1`: 1x1 convolution with stride `_n` for downsampling.
    *   `conv2`: 3x3 convolution with stride `_m`, dilation `_n_(n)_`.
    *   `conv3`: 1x1 convolution for upsampling.

**Methods**

*   `forward(self, x)`: Takes input `x`, applies the bottleneck sequence of transformations and returns the output.

**Dependencies and Requirements**

*   Inherit from `ResNetBlockBase`.
*   Uses `Conv2d` and `nn.functional.ReLU_` modules.
*   Utilizes `get_norm` function for normalization.


**Method Signatures**
----------------------

*   `__init__: (in_channels, out_channels, bottleneck_channels, stride=1, ...)`
*   `forward: (x)`
```


key: chroma:document
string_value:
CODE:

class Backbone(nn.Module, metaclass=ABCMeta):
    def __init__(self):
        super().__init__()

    @abstractmethod
    def forward(self):
        pass

    @property
    def size_divisibility(self):
        """
        Some backbones require the input height and width to be divisible by a specific integer. This is
        typically true for encoder / decoder type networks with lateral connection (e.g., FPN) for which feature maps need to match
        dimension in the "bottom up" and "top down" paths. Set to 0 if no specific input size divisibility is required.
        """
        return 0

    def output_shape(self):
        return {
            name: ShapeSpec(
                channels=self._out_feature_channels[name],
                stride=self._out_feature_strides[name],
            )
            for name in self._out_features
        }

    @property
    def out_features(self):
        """deprecated"""
        return self._out_features

    @property
    def out_feature_strides(self):
        """deprecated"""
        return {f: self._out_feature_strides[f] for f in self._out_features}

    @property
    def out_feature_channels(self):
        """deprecated"""


SUMMARY:
**Backbone Class Summary**

The `Backbone` class ( inherits from `nn.Module`) is an abstract base class that requires implemenation of the `forward` method.

### Primary Purpose
Implementaion of feature maps and output shapes for neural networks.

### Key Implementation Details:

* Implementiation of the `forward` method, which should generate feature maps.
* Generation of output shape using the `_output_shape` method

### Dependencies/Requirements:
None explicit dependencies are mentioned. However, it assumes usage of 
- PyTorch's `nn.Module` and related classes.

### Notable Properties:

* `size_divisibility`: Returns an integer (0) indicating whether input size requires divisibility for lateral connections.
* `out_features`, `out_feature_strides`, and `out_feature_channels`: Return the list of feature names, strides and channels respectively.


key: chroma:document
string_value:
CODE:

class ResNet(Backbone):
    def __init__(self, stem, stages, num_classes=None, out_features=None):
        """
        Args:
            stem (nn.Module): a stem module
            stages (list[list[ResNetBlock]]): several (typically 4) stages, each contains multiple :class:`ResNetBlockBase`.
            num_classes (None or int): if None, will not perform classification.
            out_features (list[str]): name of the layers whose outputs should be returned in forward. Can be anything in:
            "stem", "linear", or "res2" ... If None, will return the output of the last layer.
        """
        super(ResNet, self).__init__()
        self.stem = stem
        self.num_classes = num_classes

        current_stride = self.stem.stride
        self._out_feature_strides = {"stem": current_stride}
        self._out_feature_channels = {"stem": self.stem.out_channels}

        self.stages_and_names = []
        for i, blocks in enumerate(stages):
            for block in blocks:
                assert isinstance(block, ResNetBlockBase), block
                curr_channels = block.out_channels
            stage = nn.Sequential(*blocks)
            name = "res" + str(i + 2)
            self.add_module(name, stage)
            self.stages_and_names.append((stage, name))
            self._out_feature_strides[name] = current_stride = int(
                current_stride * np.prod([k.stride for k in blocks])
            )
            self._out_feature_channels[name] = blocks[-1].out_channels

        if num_classes is not None:
            self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
            self.linear = nn.Linear(curr_channels, num_classes)

            # Sec 5.1 in "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour":
            # "The 1000-way fully-connected layer is initialized by
            # drawing weights from a zero-mean Gaussian with std of 0.01."
            nn.init.normal_(self.linear.weight, stddev=0.01)
            name = "linear"

        if out_features is None:
            out_features = [name]
        self._out_features = out_features
        assert len(self._out_features)
        children = [x[0] for x in self.named_children()]
        for out_feature in self._out_features:
            assert out_feature in children, "Available children: {}".format(", ".join(children))

    def forward(self, x):
        outputs = {}
        x = self.stem(x)
        if "stem" in self._out_features:
            outputs["stem"] = x
        for stage, name in self.stages_and_names:
            x = stage(x)
            if name in self._out_features:
                outputs[name] = x
        if self.num_classes is not None:
            x = self.avgpool(x)
            x = self.linear(x)
            if "linear" in self._out_features:
                outputs["linear"] = x
        return outputs

    def output_shape(self):
        return {
            name: ShapeSpec(
                channels=self._out_feature_channels[name],
                stride=self._out_feature_strides[name],
            )
            for name in self._out_features
        }

    @staticmethod
    def make_stage(
        block_class,
        num_blocks,
        first_stride=None,
        *,
        in_channels,
        out_channels,
        **kwargs,
    ):
        """
        Usually, layers that produce the same feature map spatial size
        are defined as one "stage".
        Under such definition, stride_per_block[1:] should all be 1.
        """
        if first_stride is not None:
            assert "stride" not in kwargs and "stride_per_block" not in kwargs
            kwargs["stride_per_block"] = [first_stride] + [1] * (num_blocks - 1)
        blocks = []
        for i in range(num_blocks):
            curr_kwargs = {}
            for k, v in kwargs.items():
                if k.endswith("_per_block"):
                    assert (
                        len(v) == num_blocks
                    ), f"Argument '{k}' of make_stage should have the same length as num_blocks={num_blocks}."
                    newk = k[: -len("_per_block")]
                    assert newk not in kwargs, f"Cannot call make_stage with both {k} and {newk}!"
                    curr_kwargs[newk] = v[i]
                else:
                    curr_kwargs[k] = v

            blocks.append(block_class(in_channels=in_channels, out_channels=out_channels, **curr_kwargs))
            in_channels = out_channels



SUMMARY:
**ResNet Class Summary:**

### Purpose and Functionality

The `ResNet` class is a PyTorch backbone network architecture, extending the `Backbone` base class. It takes in a stem module, stages (listed of `ResNetBlockBase` instances), number of output classes (`num_classes`), and optional out feature layers (`out_features`). The primary purpose is to perform classification tasks on image data.

### Key Implementation Details

*   **ResNet Block Staging**: The class uses a staging mechanism, where each stage contains multiple `ResNetBlockBase` instances. Each stage is processed sequentially, with intermediate outputs being returned.
*   **Module Addition and Naming**: The class adds the stages as modules using `self.add_module()`, naming them according to their position in the architecture (e.g., "res2").
*   **Output Shapes and Feature Channels**: The class computes the output shape and feature channel counts for each stage, using the `_out_feature_strides` and `_out_feature_channels` dictionaries.
*   **Linear Layer Initialization**: If specified, the linear layer is initialized with a mean of 0 and standard deviation of 0.01.

### Notable Dependencies or Requirements

*   `nn.Module`: The base class for the ResNet architecture.
*   `ResNetBlockBase`: A component required in each stage (usually an instance of `PyramidModule`).
*   `Linear` and `AdaptiveAvgPool2d`: Used for output classification layers.

### Additional Information

The `ResNet` class is commonly used in deep learning applications, such as image recognition, object detection, and segmentation tasks. The architecture is well-suited for efficient classification of large-scale images due to its ability to downsample the input signal through multiple levels of downsampling.


key: chroma:document
string_value:
CODE:

class ROIPooler(nn.Module):
    """
    Region of interest feature map pooler that supports pooling from one or more
    feature maps.
    """

    def __init__(
        self,
        output_size,
        scales,
        sampling_ratio,
        canonical_box_size=224,
        canonical_level=4,
    ):
        super().__init__()
        # assumption that stride is a power of 2.
        min_level = -math.log2(scales[0])
        max_level = -math.log2(scales[-1])

        # a bunch of testing
        assert math.isclose(min_level, int(min_level)) and math.isclose(max_level, int(max_level))
        assert len(scales) == max_level - min_level + 1, "not pyramid"
        assert 0 < min_level and min_level <= max_level
        if isinstance(output_size, int):
            output_size = (output_size, output_size)
        assert len(output_size) == 2 and isinstance(output_size[0], int) and isinstance(output_size[1], int)
        if len(scales) > 1:
            assert min_level <= canonical_level and canonical_level <= max_level
        assert canonical_box_size > 0

        self.output_size = output_size
        self.min_level = int(min_level)
        self.max_level = int(max_level)
        self.level_poolers = nn.ModuleList(RoIPool(output_size, spatial_scale=scale) for scale in scales)
        self.canonical_level = canonical_level
        self.canonical_box_size = canonical_box_size

    def forward(self, feature_maps, boxes):
        """
        Args:
            feature_maps: List[torch.Tensor(N,C,W,H)]
            box_lists: list[torch.Tensor])
        Returns:
            A tensor of shape(N*B, Channels, output_size, output_size)
        """
        x = list(feature_maps.values())
        num_level_assignments = len(self.level_poolers)
        assert len(x) == num_level_assignments and len(boxes) == x[0].size(0)

        pooler_fmt_boxes = convert_boxes_to_pooler_format(boxes)

        if num_level_assignments == 1:
            return self.level_poolers[0](x[0], pooler_fmt_boxes)

        level_assignments = assign_boxes_to_levels(
            boxes,
            self.min_level,
            self.max_level,
            self.canonical_box_size,
            self.canonical_level,
        )

        num_boxes = len(pooler_fmt_boxes)
        num_channels = x[0].shape[1]
        output_size = self.output_size[0]

        dtype, device = x[0].dtype, x[0].device
        output = torch.zeros(
            (num_boxes, num_channels, output_size, output_size),
            dtype=dtype,
            device=device,
        )

        for level, (x_level, pooler) in enumerate(zip(x, self.level_poolers)):
            inds = torch.nonzero(level_assignments == level).squeeze(1)
            pooler_fmt_boxes_level = pooler_fmt_boxes[inds]
            output[inds] = pooler(x_level, pooler_fmt_boxes_level)



SUMMARY:
**ROIPooler Class Summary**

The `ROIPooler` class is an implementation of a region of interest (ROI) feature map pooler in PyTorch. Its primary purpose is to support pooling from one or more feature maps.

**Key Functionality:**

* The class takes in multiple feature maps and boxes as input.
* It assigns each box to a level based on its spatial scale and returns a tensor with shape `(N*B, Channels, output_size, output_size)`.
* It uses a list of `RoIPool` modules to perform pyramid pooling.

**Notable Implementation Details:**

* The `forward` method iterates over multiple levels and assigns each box to the corresponding RoIPool module.
* It uses the `assign_boxes_to_levels` function to map boxes to levels based on their spatial scale.
* It uses list comprehension to zero-pad the output tensor with shape `(N*B, num_channels, output_size, output_size)`.

**Dependencies and Requirements:**

* The class inherits from PyTorch's `nn.Module`.
* It depends on the `RoIPool` module from torchvision.

**Method Signatures:**

* `__init__(output_size, scales, sampling_ratio, canonical_box_size, canonical_level)`: Initializes the ROIPooler with specified parameters.
* `forward(feature_maps, boxes)`: Performs ROI pooling on multiple feature maps and returns the pooled tensor.

Overall, this class is designed to efficiently compute ROI features from multiple feature maps using pyramid pooling.


key: chroma:document
string_value:
CODE:

class ROIOutputs:
    def __init__(self, cfg, training=False):
        self.smooth_l1_beta = cfg.ROI_BOX_HEAD.SMOOTH_L1_BETA
        self.box2box_transform = Box2BoxTransform(weights=cfg.ROI_BOX_HEAD.BBOX_REG_WEIGHTS)
        self.training = training
        self.score_thresh = cfg.ROI_HEADS.SCORE_THRESH_TEST
        self.min_detections = cfg.MIN_DETECTIONS
        self.max_detections = cfg.MAX_DETECTIONS

        nms_thresh = cfg.ROI_HEADS.NMS_THRESH_TEST
        if not isinstance(nms_thresh, list):
            nms_thresh = [nms_thresh]
        self.nms_thresh = nms_thresh

    def _predict_boxes(self, proposals, box_deltas, preds_per_image):
        num_pred = box_deltas.size(0)
        B = proposals[0].size(-1)
        K = box_deltas.size(-1) // B
        box_deltas = box_deltas.view(num_pred * K, B)
        proposals = torch.cat(proposals, dim=0).unsqueeze(-2).expand(num_pred, K, B)
        proposals = proposals.reshape(-1, B)
        boxes = self.box2box_transform.apply_deltas(box_deltas, proposals)
        return boxes.view(num_pred, K * B).split(preds_per_image, dim=0)

    def _predict_objs(self, obj_logits, preds_per_image):
        probs = nn.functional.softmax(obj_logits, dim=-1)
        probs = probs.split(preds_per_image, dim=0)
        return probs

    def _predict_attrs(self, attr_logits, preds_per_image):
        attr_logits = attr_logits[..., :-1].softmax(-1)
        attr_probs, attrs = attr_logits.max(-1)
        return attr_probs.split(preds_per_image, dim=0), attrs.split(preds_per_image, dim=0)

    @torch.no_grad()
    def inference(
        self,
        obj_logits,
        attr_logits,
        box_deltas,
        pred_boxes,
        features,
        sizes,
        scales=None,
    ):
        # only the pred boxes is the
        preds_per_image = [p.size(0) for p in pred_boxes]
        boxes_all = self._predict_boxes(pred_boxes, box_deltas, preds_per_image)
        obj_scores_all = self._predict_objs(obj_logits, preds_per_image)  # list of length N
        attr_probs_all, attrs_all = self._predict_attrs(attr_logits, preds_per_image)
        features = features.split(preds_per_image, dim=0)

        # fun for each image too, also I can experiment and do multiple images
        final_results = []
        zipped = zip(boxes_all, obj_scores_all, attr_probs_all, attrs_all, sizes)
        for i, (boxes, obj_scores, attr_probs, attrs, size) in enumerate(zipped):
            for nms_t in self.nms_thresh:
                outputs = do_nms(
                    boxes,
                    obj_scores,
                    size,
                    self.score_thresh,
                    nms_t,
                    self.min_detections,
                    self.max_detections,
                )
                if outputs is not None:
                    max_boxes, max_scores, classes, ids = outputs
                    break

            if scales is not None:
                scale_yx = scales[i]
                max_boxes[:, 0::2] *= scale_yx[1]
                max_boxes[:, 1::2] *= scale_yx[0]

            final_results.append(
                (
                    max_boxes,
                    classes,
                    max_scores,
                    attrs[ids],
                    attr_probs[ids],
                    features[i][ids],
                )
            )
        boxes, classes, class_probs, attrs, attr_probs, roi_features = map(list, zip(*final_results))
        return boxes, classes, class_probs, attrs, attr_probs, roi_features

    def training(self, obj_logits, attr_logits, box_deltas, pred_boxes, features, sizes):
        pass

    def __call__(
        self,
        obj_logits,
        attr_logits,
        box_deltas,
        pred_boxes,
        features,
        sizes,
        scales=None,
    ):
        if self.training:
            raise NotImplementedError()
        return self.inference(
            obj_logits,
            attr_logits,
            box_deltas,
            pred_boxes,
            features,
            sizes,
            scales=scales,


SUMMARY:
**Summary: ROIOutputs Class**

**Main Purpose and Functionality:**
The `ROIOutputs` class is designed for object detection and attribute classification. It takes in input data, such as object logos, attribute logits, box deltas, prediction boxes, features, sizes, and scales (optional), and produces the final detection results.

**Key Implementation Details:**

* `_predict_boxes`: Predicts bounding boxes from proposals and box deltas using `Box2BoxTransform`.
* `_predict_objs` and `_predict_attrs`: Softmaxes object and attribute logits to obtain probabilities.
* `inference`: Main inference function that coordinates prediction, non-maximum suppression (NMS), and feature extraction.
* `training`: A placeholder for future support.

**Notable Dependencies or Requirements:**

* `Box2BoxTransform`
* `do_nms` (Non-Max Suppression algorithm)
* `ROI_HEADS`, `ROI_BOX_HEAD`, and `MIN_DETECTIONS/MAX_DETECTIONS` from the configuration (`cfg`) object.
* External features are stored in a list of length equal to the number of predictions per image.

**Example Usage:**
```python
roi_outputs = ROIOutputs(cfg=your_config)
boxes, classes, class_probs, attrs, attr_probs, roi_features = roi_outputs(obj_logits=..., attr_logits=..., box_deltas=..., pred_boxes=...)
```


key: chroma:document
string_value:
CODE:

class Res5ROIHeads(nn.Module):
    """
    ROIHeads perform all per-region computation in an R-CNN.
    It contains logic of cropping the regions, extract per-region features
    (by the res-5 block in this case), and make per-region predictions.
    """

    def __init__(self, cfg, input_shape):
        super().__init__()
        self.batch_size_per_image = cfg.RPN.BATCH_SIZE_PER_IMAGE
        self.positive_sample_fraction = cfg.ROI_HEADS.POSITIVE_FRACTION
        self.in_features = cfg.ROI_HEADS.IN_FEATURES
        self.num_classes = cfg.ROI_HEADS.NUM_CLASSES
        self.proposal_append_gt = cfg.ROI_HEADS.PROPOSAL_APPEND_GT
        self.feature_strides = {k: v.stride for k, v in input_shape.items()}
        self.feature_channels = {k: v.channels for k, v in input_shape.items()}
        self.cls_agnostic_bbox_reg = cfg.ROI_BOX_HEAD.CLS_AGNOSTIC_BBOX_REG
        self.stage_channel_factor = 2**3  # res5 is 8x res2
        self.out_channels = cfg.RESNETS.RES2_OUT_CHANNELS * self.stage_channel_factor

        # self.proposal_matcher = Matcher(
        #     cfg.ROI_HEADS.IOU_THRESHOLDS,
        #     cfg.ROI_HEADS.IOU_LABELS,
        #     allow_low_quality_matches=False,
        # )

        pooler_resolution = cfg.ROI_BOX_HEAD.POOLER_RESOLUTION
        pooler_scales = (1.0 / self.feature_strides[self.in_features[0]],)
        sampling_ratio = cfg.ROI_BOX_HEAD.POOLER_SAMPLING_RATIO
        res5_halve = cfg.ROI_BOX_HEAD.RES5HALVE
        use_attr = cfg.ROI_BOX_HEAD.ATTR
        num_attrs = cfg.ROI_BOX_HEAD.NUM_ATTRS

        self.pooler = ROIPooler(
            output_size=pooler_resolution,
            scales=pooler_scales,
            sampling_ratio=sampling_ratio,
        )

        self.res5 = self._build_res5_block(cfg)
        if not res5_halve:
            """
            Modifications for VG in RoI heads:
            1. Change the stride of conv1 and shortcut in Res5.Block1 from 2 to 1
            2. Modifying all conv2 with (padding: 1 --> 2) and (dilation: 1 --> 2)
            """
            self.res5[0].conv1.stride = (1, 1)
            self.res5[0].shortcut.stride = (1, 1)
            for i in range(3):
                self.res5[i].conv2.padding = (2, 2)
                self.res5[i].conv2.dilation = (2, 2)

        self.box_predictor = FastRCNNOutputLayers(
            self.out_channels,
            self.num_classes,
            self.cls_agnostic_bbox_reg,
            use_attr=use_attr,
            num_attrs=num_attrs,
        )

    def _build_res5_block(self, cfg):
        stage_channel_factor = self.stage_channel_factor  # res5 is 8x res2
        num_groups = cfg.RESNETS.NUM_GROUPS
        width_per_group = cfg.RESNETS.WIDTH_PER_GROUP
        bottleneck_channels = num_groups * width_per_group * stage_channel_factor
        out_channels = self.out_channels
        stride_in_1x1 = cfg.RESNETS.STRIDE_IN_1X1
        norm = cfg.RESNETS.NORM

        blocks = ResNet.make_stage(
            BottleneckBlock,
            3,
            first_stride=2,
            in_channels=out_channels // 2,
            bottleneck_channels=bottleneck_channels,
            out_channels=out_channels,
            num_groups=num_groups,
            norm=norm,
            stride_in_1x1=stride_in_1x1,
        )
        return nn.Sequential(*blocks)

    def _shared_roi_transform(self, features, boxes):
        x = self.pooler(features, boxes)
        return self.res5(x)

    def forward(self, features, proposal_boxes, gt_boxes=None):
        if self.training:
            """
            see https://github.com/airsplay/py-bottom-up-attention/\
                    blob/master/detectron2/modeling/roi_heads/roi_heads.py
            """
            raise NotImplementedError()

        assert not proposal_boxes[0].requires_grad
        box_features = self._shared_roi_transform(features, proposal_boxes)
        feature_pooled = box_features.mean(dim=[2, 3])  # pooled to 1x1
        obj_logits, attr_logits, pred_proposal_deltas = self.box_predictor(feature_pooled)


SUMMARY:
**Class Summaries: Res5ROIHeads**

### Description
Res5ROIHeads is an NVIDIA Fairseq component responsible for ROI computations in R-CNNs. It contains logic of cropping regions, extracting per-region features via res-5 block, and making per-region predictions.

### Key Features

*   Computes and combines feature representations from RoI Pooler and Res5 Block
*   Applies class activation head and object detection layer for prediction outputs
*   Features multiple key architectural elements:
    *   **RoIPooler**: pools regions of interest using max pooling with strides defined by `pooler_resolution`
    *   **Res5 Block:**
        *   Implemented as a sequential block consisting of three bottleneck layers
        *   Each layer features batch normalization and 2x dilation for convolutional filters to increase receptive field size

### Key Parameters and Configurable Arguments:

*   `positive_sample_fraction`: fraction of positive samples on the training set.
*   `in_features`: indices into input shape defining feature resolution channels and strides.
*   `num_classes` : number of output classes (default equals dataset #classes-1).
*   `feature_strides` and `feature_channels`: specify resolution for RoI pooling along each spatial axis.

### Dependencies and Inherited Interfaces:

| Dependent Module            | Required Interface |
|-----------------------------|---------------------|
| `ROIPooler`                  | output\_size, scales, sampling\_ratio  |
| `BottleneckBlock`            | first\_stride, in\_channels, bottleneck\_channels, out\_channels, num\_groups, norm, stride\_in\_1x1           |
| `FastRCNNOutputLayers`      | out\_channels, num\_classes, cls\_agnostic\_bbox\_reg, use_attr=useattr,\#numattrs=numattrs                |

This class leverages these modules and inherits key interfaces for processing RoI pooling and Res5 block architecture.


key: chroma:document
string_value:
CODE:

class AnchorGenerator(nn.Module):
    """
    For a set of image sizes and feature maps, computes a set of anchors.
    """

    def __init__(self, cfg, input_shape: List[ShapeSpec]):
        super().__init__()
        sizes = cfg.ANCHOR_GENERATOR.SIZES
        aspect_ratios = cfg.ANCHOR_GENERATOR.ASPECT_RATIOS
        self.strides = [x.stride for x in input_shape]
        self.offset = cfg.ANCHOR_GENERATOR.OFFSET
        assert 0.0 <= self.offset < 1.0, self.offset

        """
        sizes (list[list[int]]): sizes[i] is the list of anchor sizes for feat map i
            1. given in absolute lengths in units of the input image;
            2. they do not dynamically scale if the input image size changes.
        aspect_ratios (list[list[float]])
        strides (list[int]): stride of each input feature.
        """

        self.num_features = len(self.strides)
        self.cell_anchors = nn.ParameterList(self._calculate_anchors(sizes, aspect_ratios))
        self._spacial_feat_dim = 4

    def _calculate_anchors(self, sizes, aspect_ratios):
        # If one size (or aspect ratio) is specified and there are multiple feature
        # maps, then we "broadcast" anchors of that single size (or aspect ratio)
        if len(sizes) == 1:
            sizes *= self.num_features
        if len(aspect_ratios) == 1:
            aspect_ratios *= self.num_features
        assert self.num_features == len(sizes)
        assert self.num_features == len(aspect_ratios)

        cell_anchors = [self.generate_cell_anchors(s, a).float() for s, a in zip(sizes, aspect_ratios)]

        return cell_anchors

    @property
    def box_dim(self):
        return self._spacial_feat_dim

    @property
    def num_cell_anchors(self):
        """
        Returns:
            list[int]: Each int is the number of anchors at every pixel location, on that feature map.
        """
        return [len(cell_anchors) for cell_anchors in self.cell_anchors]

    def grid_anchors(self, grid_sizes):
        anchors = []
        for size, stride, base_anchors in zip(grid_sizes, self.strides, self.cell_anchors):
            shift_x, shift_y = _create_grid_offsets(size, stride, self.offset, base_anchors.device)
            shifts = torch.stack((shift_x, shift_y, shift_x, shift_y), dim=1)

            anchors.append((shifts.view(-1, 1, 4) + base_anchors.view(1, -1, 4)).reshape(-1, 4))

        return anchors

    def generate_cell_anchors(self, sizes=(32, 64, 128, 256, 512), aspect_ratios=(0.5, 1, 2)):
        """
        anchors are continuous geometric rectangles
        centered on one feature map point sample.
        We can later build the set of anchors
        for the entire feature map by tiling these tensors
        """

        anchors = []
        for size in sizes:
            area = size**2.0
            for aspect_ratio in aspect_ratios:
                w = math.sqrt(area / aspect_ratio)
                h = aspect_ratio * w
                x0, y0, x1, y1 = -w / 2.0, -h / 2.0, w / 2.0, h / 2.0
                anchors.append([x0, y0, x1, y1])
        return nn.Parameter(torch.tensor(anchors))

    def forward(self, features):
        """
        Args:
            features List[torch.Tensor]: list of feature maps on which to generate anchors.
        Returns:
            torch.Tensor: a list of #image elements.
        """
        num_images = features[0].size(0)
        grid_sizes = [feature_map.shape[-2:] for feature_map in features]
        anchors_over_all_feature_maps = self.grid_anchors(grid_sizes)
        anchors_over_all_feature_maps = torch.stack(anchors_over_all_feature_maps)


SUMMARY:
**AnchorGenerator Class Summary**

**Purpose and Functionality**

The `AnchorGenerator` class is a neural network module (NN.Module) that computes a set of anchors for a given set of image sizes and feature maps. The primary purpose is to generate anchor boxes for object detection tasks.

**Main Components**

1. **Configuration and Initialization**: Initializes the module with configuration settings (`cfg`) and input shape information.
2. **Anchor Generation**: Calculates anchor sizes, aspect ratios, and cell anchors using the `_calculate_anchors` method.
3. **Grid Anchors**: Computes anchors for each feature map using the `grid_anchors` method.
4. **Forward Pass**: Takes in a list of feature maps and generates anchor boxes using the `forward` method.

**Key Implementation Details**

1. The `_calculate_anchors` method uses broadcasting to handle cases where only one size or aspect ratio is specified across multiple feature maps.
2. The `generate_cell_anchors` method calculates anchors for a single cell (grid) with given sizes and aspect ratios.
3. The `grid_anchors` method computes anchor boxes for each feature map using grid offsets.

**Notable Dependencies**

1. PyTorch (`torch`)
2. Math library (`math`)

**Interfaces and Inherited Dependencies**

1. NN.Module interface
2. torch.nn.ParameterList interface

**Key Method Signatures**

1. `__init__(cfg, input_shape)`
2. `_calculate_anchors(sizes, aspect_ratios)`
3. `generate_cell_anchors(sizes=(32, 64, 128, 256, 512), aspect_ratios=(0.5, 1, 2))`
4. `grid_anchors(grid_sizes)`
5. `forward(features)`


key: chroma:document
string_value:
CODE:

class RPNHead(nn.Module):
    """
    RPN classification and regression heads. Uses a 3x3 conv to produce a shared
    hidden state from which one 1x1 conv predicts objectness logits for each anchor
    and a second 1x1 conv predicts bounding-box deltas specifying how to deform
    each anchor into an object proposal.
    """

    def __init__(self, cfg, input_shape: List[ShapeSpec]):
        super().__init__()

        # Standard RPN is shared across levels:
        in_channels = [s.channels for s in input_shape]
        assert len(set(in_channels)) == 1, "Each level must have the same channel!"
        in_channels = in_channels[0]

        anchor_generator = AnchorGenerator(cfg, input_shape)
        num_cell_anchors = anchor_generator.num_cell_anchors
        box_dim = anchor_generator.box_dim
        assert len(set(num_cell_anchors)) == 1, "Each level must have the same number of cell anchors"
        num_cell_anchors = num_cell_anchors[0]

        if cfg.PROPOSAL_GENERATOR.HIDDEN_CHANNELS == -1:
            hid_channels = in_channels
        else:
            hid_channels = cfg.PROPOSAL_GENERATOR.HIDDEN_CHANNELS
            # Modifications for VG in RPN (modeling/proposal_generator/rpn.py)
            # Use hidden dim  instead fo the same dim as Res4 (in_channels)

        # 3x3 conv for the hidden representation
        self.conv = nn.Conv2d(in_channels, hid_channels, kernel_size=3, stride=1, padding=1)
        # 1x1 conv for predicting objectness logits
        self.objectness_logits = nn.Conv2d(hid_channels, num_cell_anchors, kernel_size=1, stride=1)
        # 1x1 conv for predicting box2box transform deltas
        self.anchor_deltas = nn.Conv2d(hid_channels, num_cell_anchors * box_dim, kernel_size=1, stride=1)

        for layer in [self.conv, self.objectness_logits, self.anchor_deltas]:
            nn.init.normal_(layer.weight, std=0.01)
            nn.init.constant_(layer.bias, 0)

    def forward(self, features):
        """
        Args:
            features (list[Tensor]): list of feature maps
        """
        pred_objectness_logits = []
        pred_anchor_deltas = []
        for x in features:
            t = nn.functional.relu(self.conv(x))
            pred_objectness_logits.append(self.objectness_logits(t))
            pred_anchor_deltas.append(self.anchor_deltas(t))


SUMMARY:
**RPNHead Class Summary**

This class implements the RPN (Region Proposal Network) classification and regression heads for object detection. It produces shared hidden states from feature maps using a 3x3 convolution, which are then used to predict objectness logits and box-2-box transform deltas.

**Key Implementation Details:**

*   A convolve-predict-convolve structure:
    *   `conv` (3x3) produces the shared hidden state.
    *   `objectness_logits` (1x1) predicts objectness logits for each anchor.
    *   `anchor_deltas` (1x1) predicts box-2-box transform deltas specifying how to deform each anchor into an object proposal.
*   Shared configuration and initialization between levels.

**Notable Dependencies:**

*   `nn.Conv2d`: Convolutional layer for producing shared hidden states.
*   `nn.functional.relu`: ReLU activation function used in the convolutional layer.
*   `cfg.PROPOSAL_GENERATOR.HIDDEN_CHANNELS`: Configuration parameter controlling the number of hidden channels.

**External Dependencies:**

*   `AnchorGenerator`: A class not shown in this code snippet, but assumed to generate cell anchors and anchor boundaries.
*   `ShapeSpec`, `List`: Assuming these are custom data structures for specifying input shapes, from the `torch` library.

**Interfaces and Interfaces Inherited:**

*   The `Module` interface is inherited directly.
*   `nn.init.normal_`: Initializes weights using a normal distribution.
*   `nn.init.constant_`: Initializes biases to zeros.


key: chroma:document
string_value:
CODE:

class RPN(nn.Module):
    """
    Region Proposal Network, introduced by the Faster R-CNN paper.
    """

    def __init__(self, cfg, input_shape: Dict[str, ShapeSpec]):
        super().__init__()

        self.min_box_side_len = cfg.PROPOSAL_GENERATOR.MIN_SIZE
        self.in_features = cfg.RPN.IN_FEATURES
        self.nms_thresh = cfg.RPN.NMS_THRESH
        self.batch_size_per_image = cfg.RPN.BATCH_SIZE_PER_IMAGE
        self.positive_fraction = cfg.RPN.POSITIVE_FRACTION
        self.smooth_l1_beta = cfg.RPN.SMOOTH_L1_BETA
        self.loss_weight = cfg.RPN.LOSS_WEIGHT

        self.pre_nms_topk = {
            True: cfg.RPN.PRE_NMS_TOPK_TRAIN,
            False: cfg.RPN.PRE_NMS_TOPK_TEST,
        }
        self.post_nms_topk = {
            True: cfg.RPN.POST_NMS_TOPK_TRAIN,
            False: cfg.RPN.POST_NMS_TOPK_TEST,
        }
        self.boundary_threshold = cfg.RPN.BOUNDARY_THRESH

        self.anchor_generator = AnchorGenerator(cfg, [input_shape[f] for f in self.in_features])
        self.box2box_transform = Box2BoxTransform(weights=cfg.RPN.BBOX_REG_WEIGHTS)
        self.anchor_matcher = Matcher(
            cfg.RPN.IOU_THRESHOLDS,
            cfg.RPN.IOU_LABELS,
            allow_low_quality_matches=True,
        )
        self.rpn_head = RPNHead(cfg, [input_shape[f] for f in self.in_features])

    def training(self, images, image_shapes, features, gt_boxes):
        pass

    def inference(self, outputs, images, image_shapes, features, gt_boxes=None):
        outputs = find_top_rpn_proposals(
            outputs.predict_proposals(),
            outputs.predict_objectness_logits(),
            images,
            image_shapes,
            self.nms_thresh,
            self.pre_nms_topk[self.training],
            self.post_nms_topk[self.training],
            self.min_box_side_len,
            self.training,
        )

        results = []
        for img in outputs:
            im_boxes, img_box_logits = img
            img_box_logits, inds = img_box_logits.sort(descending=True)
            im_boxes = im_boxes[inds]
            results.append((im_boxes, img_box_logits))

        (proposal_boxes, logits) = tuple(map(list, zip(*results)))
        return proposal_boxes, logits

    def forward(self, images, image_shapes, features, gt_boxes=None):
        """
        Args:
            images (torch.Tensor): input images of length `N`
            features (dict[str: Tensor])
            gt_instances
        """
        # features is dict, key = block level, v = feature_map
        features = [features[f] for f in self.in_features]
        pred_objectness_logits, pred_anchor_deltas = self.rpn_head(features)
        anchors = self.anchor_generator(features)
        outputs = RPNOutputs(
            self.box2box_transform,
            self.anchor_matcher,
            self.batch_size_per_image,
            self.positive_fraction,
            images,
            pred_objectness_logits,
            pred_anchor_deltas,
            anchors,
            self.boundary_threshold,
            gt_boxes,
            self.smooth_l1_beta,
        )
        # For RPN-only models, the proposals are the final output

        if self.training:
            raise NotImplementedError()
            return self.training(outputs, images, image_shapes, features, gt_boxes)
        else:


SUMMARY:
**RPN Module Summary**

**Purpose and Functionality:**
The `RPN` module implements the Region Proposal Network, introduced by the Faster R-CNN paper. Its primary function is to generate proposals for object detection tasks.

**Key Implementation Details:**

*   **Anchor Generation**: The anchor generator (`self.anchor_generator`) creates anchors based on the input features and their corresponding shapes.
*   **Box-Box Transformation**: The `Box2BoxTransform` weights are used to transform predicted box coordinates to normalized box coordinates. 
*   **Objectness Score Prediction**: The RPN head (`self.rpn_head`) predicts objectness scores (`pred_objectness_logits`) based on the input features and their corresponding shapes.  
*   **Anchor Matching**: The `Matcher` is used to match anchors with ground-truth boxes.
*   **NMS Top-K Selection**: NMS top-k selection is performed using `self.pre_nms_topk` and `self.post_nms_topk`.

**Notable Dependencies and Requirements:**

*   Required `torch` library for neural network functionality
*   Requires `cfg.RPN` configuration variables 
*   Uses external dependencies such as `AnchorGenerator`, `Box2BoxTransform`, `Matcher`, and `RPNHead`.
*   Inherents from the PyTorch module, specifically from the `nn.Module` class.

**Key Method Signatures:**

*   `__init__`: Initializes the RPN module based on the given configuration.
*   `forward`: Performs the inference or training operation.

Note that due to code length limit some part were omitted for better readability.


key: chroma:document
string_value:
CODE:

class FastRCNNOutputLayers(nn.Module):
    """
    Two linear layers for predicting Fast R-CNN outputs:
      (1) proposal-to-detection box regression deltas
      (2) classification scores
    """

    def __init__(
        self,
        input_size,
        num_classes,
        cls_agnostic_bbox_reg,
        box_dim=4,
        use_attr=False,
        num_attrs=-1,
    ):
        """
        Args:
            input_size (int): channels, or (channels, height, width)
            num_classes (int)
            cls_agnostic_bbox_reg (bool)
            box_dim (int)
        """
        super().__init__()

        if not isinstance(input_size, int):
            input_size = np.prod(input_size)

        # (do + 1 for background class)
        self.cls_score = nn.Linear(input_size, num_classes + 1)
        num_bbox_reg_classes = 1 if cls_agnostic_bbox_reg else num_classes
        self.bbox_pred = nn.Linear(input_size, num_bbox_reg_classes * box_dim)

        self.use_attr = use_attr
        if use_attr:
            """
            Modifications for VG in RoI heads
            Embedding: {num_classes + 1} --> {input_size // 8}
            Linear: {input_size + input_size // 8} --> {input_size // 4}
            Linear: {input_size // 4} --> {num_attrs + 1}
            """
            self.cls_embedding = nn.Embedding(num_classes + 1, input_size // 8)
            self.fc_attr = nn.Linear(input_size + input_size // 8, input_size // 4)
            self.attr_score = nn.Linear(input_size // 4, num_attrs + 1)

        nn.init.normal_(self.cls_score.weight, std=0.01)
        nn.init.normal_(self.bbox_pred.weight, std=0.001)
        for item in [self.cls_score, self.bbox_pred]:
            nn.init.constant_(item.bias, 0)

    def forward(self, roi_features):
        if roi_features.dim() > 2:
            roi_features = torch.flatten(roi_features, start_dim=1)
        scores = self.cls_score(roi_features)
        proposal_deltas = self.bbox_pred(roi_features)
        if self.use_attr:
            _, max_class = scores.max(-1)  # [b, c] --> [b]
            cls_emb = self.cls_embedding(max_class)  # [b] --> [b, 256]
            roi_features = torch.cat([roi_features, cls_emb], -1)  # [b, 2048] + [b, 256] --> [b, 2304]
            roi_features = self.fc_attr(roi_features)
            roi_features = nn.functional.relu(roi_features)
            attr_scores = self.attr_score(roi_features)
            return scores, attr_scores, proposal_deltas
        else:


SUMMARY:
**Class Summary: FastRCNNOutputLayers**

The `FastRCNNOutputLayers` class is a PyTorch nn.Module responsible for predicting Fast R-CNN outputs.

### Primary Purpose
Predicts two types of outputs:
1. **Classification scores**: `self.cls_score(roi_features)`, where `roi_features` represents the input features from Region-of-Interest (RoI).
2. **Box regression deltas**: `self.bbox_pred(roi_features)`.

### Implementation Details

*   Uses two linear layers (`self.cls_score` and `self.bbox_pred`) to compute classification scores and box regression deltas, respectively.
*   Optionally includes attribute information using an embedded layer (`self.cls_embedding`).
*   The `forward` method processes input features (`roi_features`) through the network, returning classification scores, box regression deltas, and attribute scores (if applicable).

### Notable Dependencies
None.

### Key Method Signatures

*   `__init__(input_size, num_classes, cls_agnostic_bbox_reg, box_dim, use_attr=False, num_attrs=-1)`: Initializes the layer with input size, number of classes, and other hyperparameters.
*   `forward(roi_features)`: Processes input features through the network.

### Key Interfaces
*   `nn.Module`
*   `nn.Linear`

Note: The class utilizes PyTorch nn.Module and Linear layers for its functionality.


key: chroma:document
string_value:
CODE:

class GeneralizedRCNN(nn.Module):
    def __init__(self, cfg):
        super().__init__()

        self.device = torch.device(cfg.MODEL.DEVICE)
        self.backbone = build_backbone(cfg)
        self.proposal_generator = RPN(cfg, self.backbone.output_shape())
        self.roi_heads = Res5ROIHeads(cfg, self.backbone.output_shape())
        self.roi_outputs = ROIOutputs(cfg)
        self.to(self.device)

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs):
        config = kwargs.pop("config", None)
        state_dict = kwargs.pop("state_dict", None)
        cache_dir = kwargs.pop("cache_dir", None)
        from_tf = kwargs.pop("from_tf", False)
        force_download = kwargs.pop("force_download", False)
        resume_download = kwargs.pop("resume_download", False)
        proxies = kwargs.pop("proxies", None)
        local_files_only = kwargs.pop("local_files_only", False)
        use_cdn = kwargs.pop("use_cdn", True)

        # Load config if we don't provide a configuration
        if not isinstance(config, Config):
            config_path = config if config is not None else pretrained_model_name_or_path
            # try:
            config = Config.from_pretrained(
                config_path,
                cache_dir=cache_dir,
                force_download=force_download,
                resume_download=resume_download,
                proxies=proxies,
                local_files_only=local_files_only,
            )

        # Load model
        if pretrained_model_name_or_path is not None:
            if os.path.isdir(pretrained_model_name_or_path):
                if os.path.isfile(os.path.join(pretrained_model_name_or_path, WEIGHTS_NAME)):
                    # Load from a PyTorch checkpoint
                    archive_file = os.path.join(pretrained_model_name_or_path, WEIGHTS_NAME)
                else:
                    raise EnvironmentError(
                        "Error no file named {} found in directory {} ".format(
                            WEIGHTS_NAME,
                            pretrained_model_name_or_path,
                        )
                    )
            elif os.path.isfile(pretrained_model_name_or_path) or is_remote_url(pretrained_model_name_or_path):
                archive_file = pretrained_model_name_or_path
            elif os.path.isfile(pretrained_model_name_or_path + ".index"):
                assert from_tf, "We found a TensorFlow checkpoint at {}, please set from_tf to True to load from this checkpoint".format(
                    pretrained_model_name_or_path + ".index"
                )
                archive_file = pretrained_model_name_or_path + ".index"
            else:
                archive_file = hf_bucket_url(
                    pretrained_model_name_or_path,
                    filename=WEIGHTS_NAME,
                    use_cdn=use_cdn,
                )

            try:
                # Load from URL or cache if already cached
                resolved_archive_file = cached_path(
                    archive_file,
                    cache_dir=cache_dir,
                    force_download=force_download,
                    proxies=proxies,
                    resume_download=resume_download,
                    local_files_only=local_files_only,
                )
                if resolved_archive_file is None:
                    raise EnvironmentError
            except EnvironmentError:
                msg = f"Can't load weights for '{pretrained_model_name_or_path}'."
                raise EnvironmentError(msg)

            if resolved_archive_file == archive_file:
                print("loading weights file {}".format(archive_file))
            else:
                print("loading weights file {} from cache at {}".format(archive_file, resolved_archive_file))
        else:
            resolved_archive_file = None

        # Instantiate model.
        model = cls(config)

        if state_dict is None:
            try:
                try:
                    state_dict = torch.load(resolved_archive_file, map_location="cpu")
                except Exception:
                    state_dict = load_checkpoint(resolved_archive_file)

            except Exception:
                raise OSError(
                    "Unable to load weights from pytorch checkpoint file. "
                    "If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. "
                )

        missing_keys = []
        unexpected_keys = []
        error_msgs = []

        # Convert old format to new format if needed from a PyTorch state_dict
        old_keys = []
        new_keys = []
        for key in state_dict.keys():
            new_key = None
            if "gamma" in key:
                new_key = key.replace("gamma", "weight")
            if "beta" in key:
                new_key = key.replace("beta", "bias")
            if new_key:
                old_keys.append(key)
                new_keys.append(new_key)
        for old_key, new_key in zip(old_keys, new_keys):
            state_dict[new_key] = state_dict.pop(old_key)

        # copy state_dict so _load_from_state_dict can modify it
        metadata = getattr(state_dict, "_metadata", None)
        state_dict = state_dict.copy()
        if metadata is not None:
            state_dict._metadata = metadata

        model_to_load = model
        model_to_load.load_state_dict(state_dict)

        if model.__class__.__name__ != model_to_load.__class__.__name__:
            base_model_state_dict = model_to_load.state_dict().keys()
            head_model_state_dict_without_base_prefix = [
                key.split(cls.base_model_prefix + ".")[-1] for key in model.state_dict().keys()
            ]
            missing_keys.extend(head_model_state_dict_without_base_prefix - base_model_state_dict)

        if len(unexpected_keys) > 0:
            print(
                f"Some weights of the model checkpoint at {pretrained_model_name_or_path} were not used when"
                f" initializing {model.__class__.__name__}: {unexpected_keys}\n- This IS expected if you are"
                f" initializing {model.__class__.__name__} from the checkpoint of a model trained on another task or"
                " with another architecture (e.g. initializing a BertForSequenceClassification model from a"
                " BertForPreTraining model).\n- This IS NOT expected if you are initializing"
                f" {model.__class__.__name__} from the checkpoint of a model that you expect to be exactly identical"
                " (initializing a BertForSequenceClassification model from a BertForSequenceClassification model)."
            )
        else:
            print(f"All model checkpoint weights were used when initializing {model.__class__.__name__}.\n")
        if len(missing_keys) > 0:
            print(
                f"Some weights of {model.__class__.__name__} were not initialized from the model checkpoint at"
                f" {pretrained_model_name_or_path} and are newly initialized: {missing_keys}\nYou should probably"
                " TRAIN this model on a down-stream task to be able to use it for predictions and inference."
            )
        else:
            print(
                f"All the weights of {model.__class__.__name__} were initialized from the model checkpoint at"
                f" {pretrained_model_name_or_path}.\nIf your task is similar to the task the model of the checkpoint"
                f" was trained on, you can already use {model.__class__.__name__} for predictions without further"
                " training."
            )
        if len(error_msgs) > 0:
            raise RuntimeError(
                "Error(s) in loading state_dict for {}:\n\t{}".format(
                    model.__class__.__name__, "\n\t".join(error_msgs)
                )
            )
        # Set model in evaluation mode to deactivate DropOut modules by default
        model.eval()

        return model

    def forward(
        self,
        images,
        image_shapes,
        gt_boxes=None,
        proposals=None,
        scales_yx=None,
        **kwargs,
    ):
        """
        kwargs:
            max_detections (int), return_tensors {"np", "pt", None}, padding {None,
            "max_detections"}, pad_value (int), location = {"cuda", "cpu"}
        """
        if self.training:
            raise NotImplementedError()
        return self.inference(
            images=images,
            image_shapes=image_shapes,
            gt_boxes=gt_boxes,
            proposals=proposals,
            scales_yx=scales_yx,
            **kwargs,
        )

    @torch.no_grad()
    def inference(
        self,
        images,
        image_shapes,
        gt_boxes=None,
        proposals=None,
        scales_yx=None,
        **kwargs,
    ):
        # run images through backbone
        original_sizes = image_shapes * scales_yx
        features = self.backbone(images)

        # generate proposals if none are available
        if proposals is None:
            proposal_boxes, _ = self.proposal_generator(images, image_shapes, features, gt_boxes)
        else:
            assert proposals is not None

        # pool object features from either gt_boxes, or from proposals
        obj_logits, attr_logits, box_deltas, feature_pooled = self.roi_heads(features, proposal_boxes, gt_boxes)

        # prepare FRCNN Outputs and select top proposals
        boxes, classes, class_probs, attrs, attr_probs, roi_features = self.roi_outputs(
            obj_logits=obj_logits,
            attr_logits=attr_logits,
            box_deltas=box_deltas,
            pred_boxes=proposal_boxes,
            features=feature_pooled,
            sizes=image_shapes,
            scales=scales_yx,
        )

        # will we pad???
        subset_kwargs = {
            "max_detections": kwargs.get("max_detections", None),
            "return_tensors": kwargs.get("return_tensors", None),
            "pad_value": kwargs.get("pad_value", 0),
            "padding": kwargs.get("padding", None),
        }
        preds_per_image = torch.tensor([p.size(0) for p in boxes])
        boxes = pad_list_tensors(boxes, preds_per_image, **subset_kwargs)
        classes = pad_list_tensors(classes, preds_per_image, **subset_kwargs)
        class_probs = pad_list_tensors(class_probs, preds_per_image, **subset_kwargs)
        attrs = pad_list_tensors(attrs, preds_per_image, **subset_kwargs)
        attr_probs = pad_list_tensors(attr_probs, preds_per_image, **subset_kwargs)
        roi_features = pad_list_tensors(roi_features, preds_per_image, **subset_kwargs)
        subset_kwargs["padding"] = None
        preds_per_image = pad_list_tensors(preds_per_image, None, **subset_kwargs)
        sizes = pad_list_tensors(image_shapes, None, **subset_kwargs)
        normalized_boxes = norm_box(boxes, original_sizes)
        return OrderedDict(
            {
                "obj_ids": classes,
                "obj_probs": class_probs,
                "attr_ids": attrs,
                "attr_probs": attr_probs,
                "boxes": boxes,
                "sizes": sizes,
                "preds_per_image": preds_per_image,
                "roi_features": roi_features,
                "normalized_boxes": normalized_boxes,
            }


SUMMARY:
Here is a summary of the provided PyTorch code snippet:

**Overview**

The code defines a class `FastRCNNModel` which implements a Fast R-CNN (Region-based Convolutional Neural Networks) model. The model takes in images as input and outputs bounding box predictions, object class labels, object probabilities, attribute classifications, and feature maps.

**Main Components**

1. **Backbone**: The code defines a backbone module that processes the input image features using a pre-trained PyTorch model (e.g., ResNet).
2. **Proposal Generator**: This module generates proposals (bounding box candidates) based on the output of the backbone.
3. **ROI Head**: This module processes the output of the proposal generator and produces bounding box predictions, object class labels, and attribute classifications.
4. **Inference Method**: The code defines an `inference` method that takes in images and outputs a dictionary containing all predicted values (object IDs, probabilities, boxes, sizes, feature maps).

**Key Functions**

* `forward`: This method processes input images through the entire model pipeline, including backing, proposal generation, ROI processing, and prediction.
* ` inference`: A wrapper function that encapsulates the backbone, proposal generator, ROI head, and prediction steps.

**Output Returns**

The code returns a dictionary containing:

* Object class IDs (obj_ids)
* Object probabilities
* Attribute classifications
* Bounding box predictions
* Feature maps

Overall, this implementation defines a PyTorch model for object detection, attribute classification, and region proposals.


key: chroma:document
string_value:
CODE:

def load_labels(objs=OBJECTS, attrs=ATTRIBUTES):
    vg_classes = []
    with open(objs) as f:
        for object in f.readlines():
            vg_classes.append(object.split(",")[0].lower().strip())

    vg_attrs = []
    with open(attrs) as f:
        for object in f.readlines():
            vg_attrs.append(object.split(",")[0].lower().strip())


SUMMARY:
**Concise Summary:**

```python
load_labels(objs=OBJECTS, attrs=ATTRIBUTES)
```

**Main Purpose and Functionality:** Loads classes and attributes from file paths `objs` and `attrs`, respectively, into lists (`vg_classes` and `vg_attrs`). Returns both lists.

**Key Implementation Details:**

* Reads comma-separated values (CSV) from files using `object.split(",")[0].lower().strip()`
* Splits files into class names and attribute names
* Uses `OBJECTS` and `ATTRIBUTES` variables as default file paths

**Notable Dependencies or Requirements:**

* External dependency on file system access for reading CSV files (`with open()` block)
* No references to specific classes, functions, or interfaces; all are part of the provided code chunk.


key: chroma:document
string_value:
CODE:

def load_checkpoint(ckp):
    r = OrderedDict()
    with open(ckp, "rb") as f:
        ckp = pkl.load(f)["model"]
    for k in copy.deepcopy(list(ckp.keys())):
        v = ckp.pop(k)
        if isinstance(v, np.ndarray):
            v = torch.tensor(v)
        else:
            assert isinstance(v, torch.tensor), type(v)
        r[k] = v


SUMMARY:
**Function Summary**

`load_checkpoint`: Loads and parses a saved model checkpoint file.

* Primary purpose: Load and deserialize the model architecture from a checkpoint file.

**Implementation Details**

* Uses `pickle` (pkl) to load the checkpoint file.
* Parses the loaded data using the "model" key, assuming it's an OrderedDict (`OrderedDict()`).
* Iterates through the keys in the loaded data and creates new tensors for any `numpy.ndarray` values, converting them to PyTorch tensors.

**Notable Dependencies/Requirements**

* External dependencies:
	+ `pickle` (pkl) module
	+ `numpy` (np)
	+ `torch`
* Assumes availability of `pytorch.util` namespace for tensor creation.


key: chroma:document
string_value:
CODE:

class Config:
    _pointer = {}

    def __init__(self, dictionary: dict, name: str = "root", level=0):
        self._name = name
        self._level = level
        d = {}
        for k, v in dictionary.items():
            if v is None:
                raise ValueError()
            k = copy.deepcopy(k)
            v = copy.deepcopy(v)
            if isinstance(v, dict):
                v = Config(v, name=k, level=level + 1)
            d[k] = v
            setattr(self, k, v)

        self._pointer = d

    def __repr__(self):
        return str(list((self._pointer.keys())))

    def __setattr__(self, key, val):
        self.__dict__[key] = val
        self.__dict__[key.upper()] = val
        levels = key.split(".")
        last_level = len(levels) - 1
        pointer = self._pointer
        if len(levels) > 1:
            for i, l in enumerate(levels):
                if hasattr(self, l) and isinstance(getattr(self, l), Config):
                    setattr(getattr(self, l), ".".join(levels[i:]), val)
                if l == last_level:
                    pointer[l] = val
                else:
                    pointer = pointer[l]

    def to_dict(self):
        return self._pointer

    def dump_yaml(self, data, file_name):
        with open(f"{file_name}", "w") as stream:
            dump(data, stream)

    def dump_json(self, data, file_name):
        with open(f"{file_name}", "w") as stream:
            json.dump(data, stream)

    @staticmethod
    def load_yaml(config):
        with open(config) as stream:
            data = load(stream, Loader=Loader)
        return data

    def __str__(self):
        t = "    "
        if self._name != "root":
            r = f"{t * (self._level-1)}{self._name}:\n"
        else:
            r = ""
        level = self._level
        for i, (k, v) in enumerate(self._pointer.items()):
            if isinstance(v, Config):
                r += f"{t * (self._level)}{v}\n"
                self._level += 1
            else:
                r += f"{t * (self._level)}{k}: {v} ({type(v).__name__})\n"
            self._level = level
        return r[:-1]

    @classmethod
    def from_pretrained(cls, pretrained_model_name_or_path: str, **kwargs):
        config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
        return cls(config_dict)

    @classmethod
    def get_config_dict(cls, pretrained_model_name_or_path: str, **kwargs):
        cache_dir = kwargs.pop("cache_dir", None)
        force_download = kwargs.pop("force_download", False)
        resume_download = kwargs.pop("resume_download", False)
        proxies = kwargs.pop("proxies", None)
        local_files_only = kwargs.pop("local_files_only", False)

        if os.path.isdir(pretrained_model_name_or_path):
            config_file = os.path.join(pretrained_model_name_or_path, CONFIG_NAME)
        elif os.path.isfile(pretrained_model_name_or_path) or is_remote_url(pretrained_model_name_or_path):
            config_file = pretrained_model_name_or_path
        else:
            config_file = hf_bucket_url(pretrained_model_name_or_path, filename=CONFIG_NAME, use_cdn=False)

        try:
            # Load from URL or cache if already cached
            resolved_config_file = cached_path(
                config_file,
                cache_dir=cache_dir,
                force_download=force_download,
                proxies=proxies,
                resume_download=resume_download,
                local_files_only=local_files_only,
            )
            # Load config dict
            if resolved_config_file is None:
                raise EnvironmentError

            config_file = Config.load_yaml(resolved_config_file)

        except EnvironmentError:
            msg = "Can't load config for"
            raise EnvironmentError(msg)

        if resolved_config_file == config_file:
            print("loading configuration file from path")
        else:
            print("loading configuration file cache")



SUMMARY:
**Class Summary: `Config`**

**Main Purpose and Functionality:**
The `Config` class represents a configuration for a specific model or data structure. It provides methods to load, dump, and manipulate the configuration data in various formats (e.g., YAML, JSON).

**Key Implementation Details:**

*   The class uses a nested dictionary (`_pointer`) to store the configuration data.
*   The `__init__` method initializes the configuration with a dictionary and recursively creates nested configurations for dictionaries values.
*   The `to_dict` method returns the configuration data as a dictionary.

**Notable Dependencies or Requirements:**

*   The class uses external dependencies such as:
    *   `copy`
    *   `json`
    *   `yaml`
    *   `torchvision` (for loading configurations from URLs)
    *   `tf.keras` (for compatibility with TensorFlow-based models)

**Notes:**


key: chroma:document
string_value:
CODE:
# quick compare tensors
def compare(in_tensor):
    out_tensor = torch.load("dump.pt", map_location=in_tensor.device)
    n1 = in_tensor.numpy()
    n2 = out_tensor.numpy()[0]
    print(n1.shape, n1[0, 0, :5])
    print(n2.shape, n2[0, 0, :5])
    assert np.allclose(n1, n2, rtol=0.01, atol=0.1), (
        f"{sum([1 for x in np.isclose(n1, n2, rtol=0.01, atol=0.1).flatten() if x is False])/len(n1.flatten())*100:.4f} %"
        " element-wise mismatch"
    )
    raise Exception("tensors are all good")



SUMMARY:
**Function Summary**

### `compare`

Compares two input tensors for element-wise equality.

**Purpose**
The main purpose of this function is to verify if the input tensor (`in_tensor`) and saved model tensor (`out_tensor`) are numerically equivalent, printing out their shapes and first few values for verification.

**Key Implementation Details**

* Loads the `out_tensor` from a file named "dump.pt" on the device specified by `in_tensor`. 
* Converts both tensors to numpy arrays and compares them element-wise using `np.allclose`.
* Prints tensor shapes and first 5 values for visual inspection.
* Raises an exception if the tensors are not identical.

**Notable Dependencies**

* PyTorch: Used for loading and manipulating tensors (`torch.load`, `tensor.device`).
* NumPy: Used for numerical computations and comparisons (`numpy.array`, `np.allclose`).


key: chroma:document
string_value:
CODE:

def is_remote_url(url_or_filename):
    parsed = urlparse(url_or_filename)


SUMMARY:
**Function Summary**

### `is_remote_url`

**Purpose**: Determines whether the input URL or filename represents a remote URL.

**Functionality**: Validates if the scheme of the provided URL is either 'http' or 'https'.

**Key Implementation Details**:

* Uses `urlparse` function to break down the input URL into its components.
* Checks if the scheme of the parsed URL matches either "http" or "https".
* Returns a boolean indicating whether the URL is remote.

### Notable Dependencies/Requirements

* `urllib.parse.urlparse`
* No dependencies on external libraries, assuming standard Python installation.


key: chroma:document
string_value:
CODE:

def hf_bucket_url(model_id: str, filename: str, use_cdn=True) -> str:
    endpoint = CLOUDFRONT_DISTRIB_PREFIX if use_cdn else S3_BUCKET_PREFIX
    legacy_format = "/" not in model_id
    if legacy_format:
        return f"{endpoint}/{model_id}-{filename}"
    else:


SUMMARY:
**Function Summary: `hf_bucket_url`**

Constructs a URL for storing/handling model files in either Cloudfront or S3, based on the provided configuration.

**Key Implementation Details:**

- Uses an endpoint prefix (_CLOUDFRONT_DISTRIB_PREFIX_ or _S3_BUCKET_PREFIX_) depending on whether CDN is enabled.
- Includes both the `model_id` and `filename` in the URL, with format choices between "legacy" and modern formats based on `_model_id_` containing a "/" character.

**Noteable Dependencies/Requirements:**

- Assumes presence of variables/_constants_;
- No explicit external dependencies or interface implementations.


key: chroma:document
string_value:
CODE:

def http_get(
    url,
    temp_file,
    proxies=None,
    resume_size=0,
    user_agent=None,
):
    ua = "python/{}".format(sys.version.split()[0])
    if _torch_available:
        ua += "; torch/{}".format(torch.__version__)
    if isinstance(user_agent, dict):
        ua += "; " + "; ".join("{}/{}".format(k, v) for k, v in user_agent.items())
    elif isinstance(user_agent, str):
        ua += "; " + user_agent
    headers = {"user-agent": ua}
    if resume_size > 0:
        headers["Range"] = "bytes=%d-" % (resume_size,)
    response = requests.get(url, stream=True, proxies=proxies, headers=headers)
    if response.status_code == 416:  # Range not satisfiable
        return
    content_length = response.headers.get("Content-Length")
    total = resume_size + int(content_length) if content_length is not None else None
    progress = tqdm(
        unit="B",
        unit_scale=True,
        total=total,
        initial=resume_size,
        desc="Downloading",
    )
    for chunk in response.iter_content(chunk_size=1024):
        if chunk:  # filter out keep-alive new chunks
            progress.update(len(chunk))
            temp_file.write(chunk)


SUMMARY:
**Summary**

`http_get` function performs a HTTP GET request with optional resume capability. It takes the following parameters:

* `url`: Request URL
* `temp_file`: Temporary file for saving response content
* `proxies`: Optional proxy configuration
* `resume_size`: Resumed download size (in bytes)
* `user_agent`: Custom user agent string

Key implementation details:

* Sets client user agent based on Python and PyTorch versions
* Handles resume capability by extracting Content-Range header
* Uses tqdm for progress tracking
* Writes chunked data to temporary file

**Dependencies**

* `requests`
* `tqdm` (via _torch_available condition)
* torch (if available)

**Interfaces Inherited/Implemented**

None


key: chroma:document
string_value:
CODE:

def get_from_cache(
    url,
    cache_dir=None,
    force_download=False,
    proxies=None,
    etag_timeout=10,
    resume_download=False,
    user_agent=None,
    local_files_only=False,
):
    if cache_dir is None:
        cache_dir = TRANSFORMERS_CACHE
    if isinstance(cache_dir, Path):
        cache_dir = str(cache_dir)

    os.makedirs(cache_dir, exist_ok=True)

    etag = None
    if not local_files_only:
        try:
            response = requests.head(url, allow_redirects=True, proxies=proxies, timeout=etag_timeout)
            if response.status_code == 200:
                etag = response.headers.get("ETag")
        except (EnvironmentError, requests.exceptions.Timeout):
            # etag is already None
            pass

    filename = url_to_filename(url, etag)

    # get cache path to put the file
    cache_path = os.path.join(cache_dir, filename)

    # etag is None = we don't have a connection, or url doesn't exist, or is otherwise inaccessible.
    # try to get the last downloaded one
    if etag is None:
        if os.path.exists(cache_path):
            return cache_path
        else:
            matching_files = [
                file
                for file in fnmatch.filter(os.listdir(cache_dir), filename + ".*")
                if not file.endswith(".json") and not file.endswith(".lock")
            ]
            if len(matching_files) > 0:
                return os.path.join(cache_dir, matching_files[-1])
            else:
                # If files cannot be found and local_files_only=True,
                # the models might've been found if local_files_only=False
                # Notify the user about that
                if local_files_only:
                    raise ValueError(
                        "Cannot find the requested files in the cached path and outgoing traffic has been"
                        " disabled. To enable model look-ups and downloads online, set 'local_files_only'"
                        " to False."
                    )
                return None

    # From now on, etag is not None.
    if os.path.exists(cache_path) and not force_download:
        return cache_path

    # Prevent parallel downloads of the same file with a lock.
    lock_path = cache_path + ".lock"
    with FileLock(lock_path):
        # If the download just completed while the lock was activated.
        if os.path.exists(cache_path) and not force_download:
            # Even if returning early like here, the lock will be released.
            return cache_path

        if resume_download:
            incomplete_path = cache_path + ".incomplete"

            @contextmanager
            def _resumable_file_manager():
                with open(incomplete_path, "a+b") as f:
                    yield f

            temp_file_manager = _resumable_file_manager
            if os.path.exists(incomplete_path):
                resume_size = os.stat(incomplete_path).st_size
            else:
                resume_size = 0
        else:
            temp_file_manager = partial(tempfile.NamedTemporaryFile, dir=cache_dir, delete=False)
            resume_size = 0

        # Download to temporary file, then copy to cache dir once finished.
        # Otherwise you get corrupt cache entries if the download gets interrupted.
        with temp_file_manager() as temp_file:
            print(
                "%s not found in cache or force_download set to True, downloading to %s",
                url,
                temp_file.name,
            )

            http_get(
                url,
                temp_file,
                proxies=proxies,
                resume_size=resume_size,
                user_agent=user_agent,
            )

        os.replace(temp_file.name, cache_path)

        meta = {"url": url, "etag": etag}
        meta_path = cache_path + ".json"
        with open(meta_path, "w") as meta_file:
            json.dump(meta, meta_file)



SUMMARY:
**Summary**

The `get_from_cache` function retrieves data from a cache directory based on the provided URL, with optional parameters for cache management, download behavior, and authentication. The primary purpose is to efficiently retrieve cached data while handling various edge cases such as concurrent downloads, incomplete files, and proxy settings.

**Key Implementation Details**

* Checks cache directory existence and creates it if necessary.
* Retrieves the ETag from the server response on first request (if applicable).
* Handles local file only mode to reduce outgoing traffic.
* Uses a lock to prevent parallel downloads of the same file.
* Resumes downloads by checking for incomplete files or using a resume token.

**Notable Dependencies and Requirements**

* `requests` for HTTP requests
* `os` for file system operations
* `Path` from `pathlib` for path manipulation
* `filelock` for locking file accesses
* `json` for writing metadata files

**Primary Purpose**
To efficiently retrieve cached data while handling various edge cases and ensuring cache integrity.


key: chroma:document
string_value:
CODE:

def url_to_filename(url, etag=None):
    url_bytes = url.encode("utf-8")
    url_hash = insecure_hashlib.sha256(url_bytes)
    filename = url_hash.hexdigest()

    if etag:
        etag_bytes = etag.encode("utf-8")
        etag_hash = insecure_hashlib.sha256(etag_bytes)
        filename += "." + etag_hash.hexdigest()

    if url.endswith(".h5"):
        filename += ".h5"



SUMMARY:
**Summary**

This function generates a unique filename from a given URL, optionally appending an ETag. The primary purpose is to hash and extend the URL to create a stable filename.

**Key Implementation Details:**
- Uses `sha256` hashing algorithm from `insecure_hashlib` for URL encoding.
- Optionally appends ETag's SHA256 hash using identical hashing function.
- Adds `.h5` extension if original URL ends with `.h5`.

**Notable Dependencies/Interfaces:**

* `insecure_hashlib`: External library for secure hashing (note: "insecure" suggests potential security concerns)
* No implemented or inherited interfaces.


key: chroma:document
string_value:
CODE:

def cached_path(
    url_or_filename,
    cache_dir=None,
    force_download=False,
    proxies=None,
    resume_download=False,
    user_agent=None,
    extract_compressed_file=False,
    force_extract=False,
    local_files_only=False,
):
    if cache_dir is None:
        cache_dir = TRANSFORMERS_CACHE
    if isinstance(url_or_filename, Path):
        url_or_filename = str(url_or_filename)
    if isinstance(cache_dir, Path):
        cache_dir = str(cache_dir)

    if is_remote_url(url_or_filename):
        # URL, so get it from the cache (downloading if necessary)
        output_path = get_from_cache(
            url_or_filename,
            cache_dir=cache_dir,
            force_download=force_download,
            proxies=proxies,
            resume_download=resume_download,
            user_agent=user_agent,
            local_files_only=local_files_only,
        )
    elif os.path.exists(url_or_filename):
        # File, and it exists.
        output_path = url_or_filename
    elif urlparse(url_or_filename).scheme == "":
        # File, but it doesn't exist.
        raise EnvironmentError("file {} not found".format(url_or_filename))
    else:
        # Something unknown
        raise ValueError("unable to parse {} as a URL or as a local path".format(url_or_filename))

    if extract_compressed_file:
        if not is_zipfile(output_path) and not tarfile.is_tarfile(output_path):
            return output_path

        # Path where we extract compressed archives
        # We avoid '.' in dir name and add "-extracted" at the end: "./model.zip" => "./model-zip-extracted/"
        output_dir, output_file = os.path.split(output_path)
        output_extract_dir_name = output_file.replace(".", "-") + "-extracted"
        output_path_extracted = os.path.join(output_dir, output_extract_dir_name)

        if os.path.isdir(output_path_extracted) and os.listdir(output_path_extracted) and not force_extract:
            return output_path_extracted

        # Prevent parallel extractions
        lock_path = output_path + ".lock"
        with FileLock(lock_path):
            shutil.rmtree(output_path_extracted, ignore_errors=True)
            os.makedirs(output_path_extracted)
            if is_zipfile(output_path):
                with ZipFile(output_path, "r") as zip_file:
                    zip_file.extractall(output_path_extracted)
                    zip_file.close()
            elif tarfile.is_tarfile(output_path):
                tar_file = tarfile.open(output_path)
                tar_file.extractall(output_path_extracted)
                tar_file.close()
            else:
                raise EnvironmentError("Archive format of {} could not be identified".format(output_path))

        return output_path_extracted



SUMMARY:
**Concise Summary:**

The `cached_path` function determines the output path for a file or URL, considering various cache-related options. Its primary purpose is to:

* Cache a file or retrieve it from storage if available
* Extract compressed archives (e.g., ZIP, tar) and create a new extraction directory

**Key Implementation Details:**

1.  **Cache Path Determination**: It checks whether the input `url_or_filename` is a URL, a local file path, or neither. Based on this, it either retrieves the file from storage if available, extracts compressed archives, or raises errors otherwise.
2.  **Compressed Archive Extraction**: If extraction is required, it creates an output directory with a unique name (replacing dots and appending "-extracted"). It then uses a lock to prevent parallel extractions and iterates through the archive using `zipfiles` and `tarifes`.
3.  **Notable Dependencies**:

    *   The functions `is_remote_url`, `get_from_cache`, `os.path.exists`, `urparse`, `ZipFile`, `tarfile`, `shutil`, `FileLock`
    *   External dependencies: None
4.  **Interfaces Implemented**:

    *   `zipfiles` and `tarfile` from Python's standard library (`tk` is not a required dependency I couldn't confirm its existence)

**Primary Purpose:**  
The main purpose of the `cached_path` function is to determine the output path for a file or URL while considering various cache-related options, including compressed archives extraction.


key: chroma:document
string_value:
CODE:

def get_data(query, delim=","):
    assert isinstance(query, str)
    if os.path.isfile(query):
        with open(query) as f:
            data = eval(f.read())
    else:
        req = requests.get(query)
        try:
            data = requests.json()
        except Exception:
            data = req.content.decode()
            assert data is not None, "could not connect"
            try:
                data = eval(data)
            except Exception:
                data = data.split("\n")
        req.close()


SUMMARY:
**Function Summary**

`get_data(query, delim="")`: fetches data from a query URL or local file.

### Key Implementation Details

*   Reads local file contents and evaluates as Python expression (`eval(f.read())`)
*   Retrieves JSON data from remote URL using `requests.get()` and attempts to parse via `requests.json()`
*   Fallbacks to decoding content with UTF-8 (`req.content.decode()`) if JSON fails
*   Further parses decoded string using `eval` in case of complete decode failure
*   Splits decoded string into lines for non-parsable cases

### Notable Dependencies and Requirements

*   `os.path.isfile()` for file check
*   `requests` library for network requests
*   `json` module for JSON parsing (imported implicitly)

### Assumptions and Notes

The function assumes that the query string provides a valid URL or path to a local file. If not, it attempts various methods of data retrieval before ultimately resorting to less reliable methods (`eval` on decoded content). This could raise security concerns if executed with untrusted input.


key: chroma:document
string_value:
CODE:

def get_image_from_url(url):
    response = requests.get(url)
    img = np.array(Image.open(BytesIO(response.content)))


SUMMARY:
**Summary**

### Function Purpose and Main Flow
The `get_image_from_url` function retrieves an image from a given URL, converts it to a numpy array representation.

### Implementation Details

* Uses `requests.get(url)` to send an HTTP GET request to the provided URL.
* Utilizes `Image.open(BytesIO(response.content))` to read the image data from the response content.
* Converts the image data to a numpy array representation using `np.array()`.
	+ Key Algorithm: JPEG and other format image decoding and conversion.

### Notable Dependencies and Requirements

* `requests`: External dependency for HTTP requests.
* `numpy` (`np`): Used for converting image data to a numpy array representation.
* `Pillow` (library used by `Image.open()`) and `BytesIO`: Used for reading and processing binary data from the response.

### Additional Notes

No public interfaces or parent/child class relationships are implemented in this code snippet.


key: chroma:document
string_value:
CODE:
# to load legacy frcnn checkpoint from detectron
def load_frcnn_pkl_from_url(url):
    fn = url.split("/")[-1]
    if fn not in os.listdir(os.getcwd()):
        wget.download(url)
    with open(fn, "rb") as stream:
        weights = pkl.load(stream)
    model = weights.pop("model")
    new = {}
    for k, v in model.items():
        new[k] = torch.from_numpy(v)
        if "running_var" in k:
            zero = torch.tensor([0])
            k2 = k.replace("running_var", "num_batches_tracked")
            new[k2] = zero


SUMMARY:
**Function Summary**

`load_frcnn_pkl_from_url(url)`: Loads FRCNN model weights from a URL and returns a structured dictionary.

**Main Purpose**
The function loads the FRCNN model weights from a given URL, checks if the local file exists, downloads if necessary, and loads the model's main weights and structure.

**Key Implementation Details**

* Uses `pkl.load` to load model weights from a binary stream.
* Updates model variables by mapping "running_var" backends to `num_batches_tracked` equivalents using torch tensor manipulation.

**Notable Dependencies/Requirements**

* External dependencies: `wget`, `torch`
* Related classes/functions:
	+ `pkl`: Python package for serializing/deserializing Python objects (i.e., model weights).
	+ `weights`: Dictionary containing loaded model weights.

**Return Value**
A structured dictionary containing populated FRCNN model variables, ready for use in a larger application.


key: chroma:document
string_value:
CODE:

def get_demo_path():


SUMMARY:
**Summary:**

The `get_demo_path` function returns the absolute path to a demo IPython notebook file located in the parent directory of the current project's root directory (`PATH`). The actual path is printed directly, without any additional processing.

**Key Details:**

* Function name: `get_demo_path`
* Purpose: Returns demo path
* Key methods:
	+ Uses `os.path.abspath` and `os.path.join` to construct the path.
	+ References `PATH`, which should be a project root directory variable.
	+ Prints the resulting absolute path.

**Dependencies:**

* `os` module, requiring specific module functions (`abspath`, `join`, `pardir`)
* External dependency: The actual value of `PATH` is expected to be defined elsewhere in the codebase.


key: chroma:document
string_value:
CODE:

def img_tensorize(im, input_format="RGB"):
    assert isinstance(im, str)
    if os.path.isfile(im):
        img = cv2.imread(im)
    else:
        img = get_image_from_url(im)
        assert img is not None, f"could not connect to: {im}"
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    if input_format == "RGB":
        img = img[:, :, ::-1]


SUMMARY:
**Summary**

The `img_tensorize` function transforms an image file into a tensor format for neural networks. It takes two inputs:

*   `im`: the path to the image file
*   `input_format`: the input format (default: "RGB")

The function performs the following operations:

1.  Reads the image file and converts it to RGB color space.
2.  Reorders the color channels in case of an RGB-only input.

**Key Implementation Details**

*   Uses OpenCV (`cv2`) for image reading and conversion.
*   Checks if the provided image path exists as a file or URL using `os.path.isfile` and `get_image_from_url`.

**Notable Dependencies/Requirements**

*   Requires OpenCV (`cv2`)
*   May require additional dependencies for `get_image_from_url`
*   Assumes `assert isinstance(im, str)` is valid due to the specification's focus on class/function naming.


key: chroma:document
string_value:
CODE:

def chunk(images, batch=1):


SUMMARY:
**Summary**

 Chunking Function:
```python
chunk(images, batch=1)
```

*Purpose*: Creates an iterator that yields batches of `images` with size `batch`.

*Functionality*: Takes in a list `images` and an optional parameter `batch` (default 1), and returns a generator that iterates over the input list in chunks of the specified size.

**Key details:**

* Uses Python's slice notation to create batches
* Returns a generator using tuple unpacking for mem efficiency

*Example Use Case*: Iterate over images with batch_size = 5```python
images = ['img1.jpg', 'img2.jpg', ..., 'img10.jpg']
chunked_images = chunk(images, batch=5)
for image_batch in chunked_images:
    # process each image in the batch
```
*Related classes/functions: `range()`


key: chroma:document
string_value:
CODE:

class SingleImageViz:
    def __init__(
        self,
        img,
        scale=1.2,
        edgecolor="g",
        alpha=0.5,
        linestyle="-",
        saveas="test_out.jpg",
        rgb=True,
        pynb=False,
        id2obj=None,
        id2attr=None,
        pad=0.7,
    ):
        """
        img: an RGB image of shape (H, W, 3).
        """
        if isinstance(img, torch.Tensor):
            img = img.numpy().astype("np.uint8")
        if isinstance(img, str):
            img = img_tensorize(img)
        assert isinstance(img, np.ndarray)

        width, height = img.shape[1], img.shape[0]
        fig = mplfigure.Figure(frameon=False)
        dpi = fig.get_dpi()
        width_in = (width * scale + 1e-2) / dpi
        height_in = (height * scale + 1e-2) / dpi
        fig.set_size_inches(width_in, height_in)
        ax = fig.add_axes([0.0, 0.0, 1.0, 1.0])
        ax.axis("off")
        ax.set_xlim(0.0, width)
        ax.set_ylim(height)

        self.saveas = saveas
        self.rgb = rgb
        self.pynb = pynb
        self.img = img
        self.edgecolor = edgecolor
        self.alpha = 0.5
        self.linestyle = linestyle
        self.font_size = int(np.sqrt(min(height, width)) * scale // 3)
        self.width = width
        self.height = height
        self.scale = scale
        self.fig = fig
        self.ax = ax
        self.pad = pad
        self.id2obj = id2obj
        self.id2attr = id2attr
        self.canvas = FigureCanvasAgg(fig)

    def add_box(self, box, color=None):
        if color is None:
            color = self.edgecolor
        (x0, y0, x1, y1) = box
        width = x1 - x0
        height = y1 - y0
        self.ax.add_patch(
            mpl.patches.Rectangle(
                (x0, y0),
                width,
                height,
                fill=False,
                edgecolor=color,
                linewidth=self.font_size // 3,
                alpha=self.alpha,
                linestyle=self.linestyle,
            )
        )

    def draw_boxes(self, boxes, obj_ids=None, obj_scores=None, attr_ids=None, attr_scores=None):
        if len(boxes.shape) > 2:
            boxes = boxes[0]
        if len(obj_ids.shape) > 1:
            obj_ids = obj_ids[0]
        if len(obj_scores.shape) > 1:
            obj_scores = obj_scores[0]
        if len(attr_ids.shape) > 1:
            attr_ids = attr_ids[0]
        if len(attr_scores.shape) > 1:
            attr_scores = attr_scores[0]
        if isinstance(boxes, torch.Tensor):
            boxes = boxes.numpy()
        if isinstance(boxes, list):
            boxes = np.array(boxes)
        assert isinstance(boxes, np.ndarray)
        areas = np.prod(boxes[:, 2:] - boxes[:, :2], axis=1)
        sorted_idxs = np.argsort(-areas).tolist()
        boxes = boxes[sorted_idxs] if boxes is not None else None
        obj_ids = obj_ids[sorted_idxs] if obj_ids is not None else None
        obj_scores = obj_scores[sorted_idxs] if obj_scores is not None else None
        attr_ids = attr_ids[sorted_idxs] if attr_ids is not None else None
        attr_scores = attr_scores[sorted_idxs] if attr_scores is not None else None

        assigned_colors = [self._random_color(maximum=1) for _ in range(len(boxes))]
        assigned_colors = [assigned_colors[idx] for idx in sorted_idxs]
        if obj_ids is not None:
            labels = self._create_text_labels_attr(obj_ids, obj_scores, attr_ids, attr_scores)
            for i in range(len(boxes)):
                color = assigned_colors[i]
                self.add_box(boxes[i], color)
                self.draw_labels(labels[i], boxes[i], color)

    def draw_labels(self, label, box, color):
        x0, y0, x1, y1 = box
        text_pos = (x0, y0)
        instance_area = (y1 - y0) * (x1 - x0)
        small = _SMALL_OBJ * self.scale
        if instance_area < small or y1 - y0 < 40 * self.scale:
            if y1 >= self.height - 5:
                text_pos = (x1, y0)
            else:
                text_pos = (x0, y1)

        height_ratio = (y1 - y0) / np.sqrt(self.height * self.width)
        lighter_color = self._change_color_brightness(color, brightness_factor=0.7)
        font_size = np.clip((height_ratio - 0.02) / 0.08 + 1, 1.2, 2)
        font_size *= 0.75 * self.font_size

        self.draw_text(
            text=label,
            position=text_pos,
            color=lighter_color,
        )

    def draw_text(
        self,
        text,
        position,
        color="g",
        ha="left",
    ):
        rotation = 0
        font_size = self.font_size
        color = np.maximum(list(mplc.to_rgb(color)), 0.2)
        color[np.argmax(color)] = max(0.8, np.max(color))
        bbox = {
            "facecolor": "black",
            "alpha": self.alpha,
            "pad": self.pad,
            "edgecolor": "none",
        }
        x, y = position
        self.ax.text(
            x,
            y,
            text,
            size=font_size * self.scale,
            family="sans-serif",
            bbox=bbox,
            verticalalignment="top",
            horizontalalignment=ha,
            color=color,
            zorder=10,
            rotation=rotation,
        )

    def save(self, saveas=None):
        if saveas is None:
            saveas = self.saveas
        if saveas.lower().endswith(".jpg") or saveas.lower().endswith(".png"):
            cv2.imwrite(
                saveas,
                self._get_buffer()[:, :, ::-1],
            )
        else:
            self.fig.savefig(saveas)

    def _create_text_labels_attr(self, classes, scores, attr_classes, attr_scores):
        labels = [self.id2obj[i] for i in classes]
        attr_labels = [self.id2attr[i] for i in attr_classes]
        labels = [
            f"{label} {score:.2f} {attr} {attr_score:.2f}"
            for label, score, attr, attr_score in zip(labels, scores, attr_labels, attr_scores)
        ]
        return labels

    def _create_text_labels(self, classes, scores):
        labels = [self.id2obj[i] for i in classes]
        if scores is not None:
            if labels is None:
                labels = ["{:.0f}%".format(s * 100) for s in scores]
            else:
                labels = ["{} {:.0f}%".format(li, s * 100) for li, s in zip(labels, scores)]
        return labels

    def _random_color(self, maximum=255):
        idx = np.random.randint(0, len(_COLORS))
        ret = _COLORS[idx] * maximum
        if not self.rgb:
            ret = ret[::-1]
        return ret

    def _get_buffer(self):
        if not self.pynb:
            s, (width, height) = self.canvas.print_to_buffer()
            if (width, height) != (self.width, self.height):
                img = cv2.resize(self.img, (width, height))
            else:
                img = self.img
        else:
            buf = io.BytesIO()  # works for cairo backend
            self.canvas.print_rgba(buf)
            width, height = self.width, self.height
            s = buf.getvalue()
            img = self.img

        buffer = np.frombuffer(s, dtype="uint8")
        img_rgba = buffer.reshape(height, width, 4)
        rgb, alpha = np.split(img_rgba, [3], axis=2)

        try:
            import numexpr as ne  # fuse them with numexpr

            visualized_image = ne.evaluate("img * (1 - alpha / 255.0) + rgb * (alpha / 255.0)")
        except ImportError:
            alpha = alpha.astype("float32") / 255.0
            visualized_image = img * (1 - alpha) + rgb * alpha

        return visualized_image.astype("uint8")

    def _change_color_brightness(self, color, brightness_factor):
        assert brightness_factor >= -1.0 and brightness_factor <= 1.0
        color = mplc.to_rgb(color)
        polygon_color = colorsys.rgb_to_hls(*mplc.to_rgb(color))
        modified_lightness = polygon_color[1] + (brightness_factor * polygon_color[1])
        modified_lightness = 0.0 if modified_lightness < 0.0 else modified_lightness
        modified_lightness = 1.0 if modified_lightness > 1.0 else modified_lightness
        modified_color = colorsys.hls_to_rgb(polygon_color[0], modified_lightness, polygon_color[2])


SUMMARY:
This code is a Python class that extends the functionality of Matplotlib's `FigureCanvas` and Cairo backend to display and save images in a visualized format. The class provides methods for drawing bounding boxes around detected objects, adding text labels with object names and confidence scores from an object detection network.

Key features:

* Supports both PIL and OpenCV for image processing
* Can be used with various visualization backends (Cairo, Agg)
* Allows for specifying the font family, size, color, and alignment for writing text
* Provides a way to change the brightness of colors while preserving their hue

Methodologies:
 
* The class uses Matplotlib's functions like `FigureCanvas`, `print_to_buffer`, and `draw_text` to combine display functionality with image filtering capabilities.
* Cairo backend is used to print PNG images to a byte buffer, then save them as JPEG files using OpenCV's image processing capabilities.

Data Sources:

* `_COLORS`: predefined list of color values
* _font_size: predefined value representing font size. 
*_alpha and *_pad: predefined variables defining alpha and padded values used for bounding boxes.
* `self.img`; image that is passed in through object detection networks, such as Faster R-CNN or YOLO.

The `save` method allows users to save the visualization as an image file with specified format (e.g., JPEG).


key: chroma:document
string_value:
CODE:

class ResizeShortestEdge:
    def __init__(self, short_edge_length, max_size=sys.maxsize):
        """
        Args:
            short_edge_length (list[min, max])
            max_size (int): maximum allowed longest edge length.
        """
        self.interp_method = "bilinear"
        self.max_size = max_size
        self.short_edge_length = short_edge_length

    def __call__(self, imgs):
        img_augs = []
        for img in imgs:
            h, w = img.shape[:2]
            # later: provide list and randomly choose index for resize
            size = np.random.randint(self.short_edge_length[0], self.short_edge_length[1] + 1)
            if size == 0:
                return img
            scale = size * 1.0 / min(h, w)
            if h < w:
                newh, neww = size, scale * w
            else:
                newh, neww = scale * h, size
            if max(newh, neww) > self.max_size:
                scale = self.max_size * 1.0 / max(newh, neww)
                newh = newh * scale
                neww = neww * scale
            neww = int(neww + 0.5)
            newh = int(newh + 0.5)

            if img.dtype == np.uint8:
                pil_image = Image.fromarray(img)
                pil_image = pil_image.resize((neww, newh), PILImageResampling.BILINEAR)
                img = np.asarray(pil_image)
            else:
                img = img.permute(2, 0, 1).unsqueeze(0)  # 3, 0, 1)  # hw(c) -> nchw
                img = nn.functional.interpolate(
                    img, (newh, neww), mode=self.interp_method, align_corners=False
                ).squeeze(0)
            img_augs.append(img)



SUMMARY:
**Class Summary: ResizeShortestEdge**

The `ResizeShortestEdge` class is designed to resize images while ensuring the shortest edge remains within a specified range. It takes two parameters:

- `short_edge_length`: a list of minimum and maximum allowed lengths for the shortest edge.
- `max_size`: the maximum allowed length for the longest edge (defaulting to the system's maximum integer value if not provided).

**Main Purpose and Functionality:**
The class aims to resize input images by randomly selecting a size within the specified range for the shortest edge, while scaling the other edges proportionally. It also resizes images with floating-point pixel values using bilinear interpolation.

**Key Implementation Details:**

- Uses `numpy` and `Pillow` libraries for image processing.
- Employs bilinear resampling (`PILImageResampling.BILINEAR`) for resizing images with integer pixel values.
- Supports interpolation for images with floating-point pixel values using `torch.nn.functional.interpolate`.

**Notable Dependencies:**

- Python's standard library (`numpy`)
- Pillow library (`Pillow` and related sub-imports)
- Torch library (`torch`)
- PyTorch Neural Networks (`nn.functional`)

**Interfaces/Behaviors:**

- `__init__`: Initializes the class with input parameters.
- `__call__`: Accepts a list of images as input, performs resizing operations, and returns the augmented image list.


key: chroma:document
string_value:
CODE:

class Preprocess:
    def __init__(self, cfg):
        self.aug = ResizeShortestEdge([cfg.INPUT.MIN_SIZE_TEST, cfg.INPUT.MIN_SIZE_TEST], cfg.INPUT.MAX_SIZE_TEST)
        self.input_format = cfg.INPUT.FORMAT
        self.size_divisibility = cfg.SIZE_DIVISIBILITY
        self.pad_value = cfg.PAD_VALUE
        self.max_image_size = cfg.INPUT.MAX_SIZE_TEST
        self.device = cfg.MODEL.DEVICE
        self.pixel_std = torch.tensor(cfg.MODEL.PIXEL_STD).to(self.device).view(len(cfg.MODEL.PIXEL_STD), 1, 1)
        self.pixel_mean = torch.tensor(cfg.MODEL.PIXEL_MEAN).to(self.device).view(len(cfg.MODEL.PIXEL_STD), 1, 1)
        self.normalizer = lambda x: (x - self.pixel_mean) / self.pixel_std

    def pad(self, images):
        max_size = tuple(max(s) for s in zip(*[img.shape for img in images]))
        image_sizes = [im.shape[-2:] for im in images]
        images = [
            nn.functional.pad(
                im,
                [0, max_size[-1] - size[1], 0, max_size[-2] - size[0]],
                value=self.pad_value,
            )
            for size, im in zip(image_sizes, images)
        ]

        return torch.stack(images), torch.tensor(image_sizes)

    def __call__(self, images, single_image=False):
        with torch.no_grad():
            if not isinstance(images, list):
                images = [images]
            if single_image:
                assert len(images) == 1
            for i in range(len(images)):
                if isinstance(images[i], torch.Tensor):
                    images.insert(i, images.pop(i).to(self.device).float())
                elif not isinstance(images[i], torch.Tensor):
                    images.insert(
                        i,
                        torch.as_tensor(img_tensorize(images.pop(i), input_format=self.input_format))
                        .to(self.device)
                        .float(),
                    )
            # resize smallest edge
            raw_sizes = torch.tensor([im.shape[:2] for im in images])
            images = self.aug(images)
            # transpose images and convert to torch tensors
            # images = [torch.as_tensor(i.astype("float32")).permute(2, 0, 1).to(self.device) for i in images]
            # now normalize before pad to avoid useless arithmetic
            images = [self.normalizer(x) for x in images]
            # now pad them to do the following operations
            images, sizes = self.pad(images)
            # Normalize

            if self.size_divisibility > 0:
                raise NotImplementedError()
            # pad
            scales_yx = torch.true_divide(raw_sizes, sizes)
            if single_image:
                return images[0], sizes[0], scales_yx[0]
            else:


SUMMARY:
**Concise Summary**

The `Preprocess` class is designed for image preprocessing in a deep learning pipeline. Its main purpose is to normalize and pad input images.

**Key Functionality:**

1. **Normalization:** Applies the normalization formula `(x - self.pixel_mean) / self.byte_std` to normalized pixel values using `normalizer`.
2. **Padding:** Pads input images with zero padding value, scaling images to a fixed size using `pad`.
3. **Resize Shortest Edge:** Resizes images to maintain aspect ratio with smallest edge using `ResizeShortestEdge`.

**Notable Dependencies and Requirements:**

1. **PyTorch**: Used for image tensorization, normalization, and device management.
2. **Configuration (cfg) object**: Provides input parameters such as resizing thresholds, pixel standards, and image format.

**Example Usage:**
```python
preprocessor = Preprocess(cfg)
preprocessed_image, image_size, scales_yx = preprocessor(preprocessed_image)
```


key: chroma:document
string_value:
CODE:

def _scale_box(boxes, scale_yx):
    boxes[:, 0::2] *= scale_yx[:, 1]
    boxes[:, 1::2] *= scale_yx[:, 0]


SUMMARY:
**Summary:**

 `_scale_box` function scales the x and y coordinates of bounding boxes (`boxes`) by proportional amounts based on an input scaling vector (`scale_yx`). Specifically, it applies transformations to even-indexed rows for `y` scaled and odd-indexed rows for `x` scaled axis. 

**Implementation Details:**

* Scales `boxes[:, 0::2]` using `scale_yx[:, 1]`
* Scales `boxes[:, 1::2]` using `scale_yx[:, 0]`

**Dependencies/Interfaces:**

* Uses NumPy indexing (`:`) and slice assignment (`*=`) for element-wise operations
* Assumes `scale_yx` has the same shape as or is compatible with `boxes`


key: chroma:document
string_value:
CODE:

def _clip_box(tensor, box_size: Tuple[int, int]):
    assert torch.isfinite(tensor).all(), "Box tensor contains infinite or NaN!"
    h, w = box_size
    tensor[:, 0].clamp_(min=0, max=w)
    tensor[:, 1].clamp_(min=0, max=h)
    tensor[:, 2].clamp_(min=0, max=w)


SUMMARY:
**Summary: Clip Box Operation**

This function clips a given 3D tensor (`tensor`) within the specified box bounds (`box_size`). The tensor's first three channels (representing x-coordinates) are clipped between 0 and `box_size[0]` (width), and the remaining two channels (representing y-coordinates) are clipped between 0 and `box_size[1]` (height).

**Key Details:**

* Clip all four channels (`h`, `w`, `H`, `W`) of the specified box size.
* Ensure all numerical values within the tensor are finite and do not contain NaN.

**Interfaces Implemented/Inherited:** None

**Notable Dependencies:**

* PyTorch (`torch`) library for finite type check and clamp operation.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/bertabs/test_utils_summarization.py


key: chroma:document
string_value:
CODE:

class SummarizationDataProcessingTest(unittest.TestCase):
    def setUp(self):
        self.block_size = 10

    def test_fit_to_block_sequence_too_small(self):
        """Pad the sequence with 0 if the sequence is smaller than the block size."""
        sequence = [1, 2, 3, 4]
        expected_output = [1, 2, 3, 4, 0, 0, 0, 0, 0, 0]
        self.assertEqual(truncate_or_pad(sequence, self.block_size, 0), expected_output)

    def test_fit_to_block_sequence_fit_exactly(self):
        """Do nothing if the sequence is the right size."""
        sequence = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
        expected_output = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
        self.assertEqual(truncate_or_pad(sequence, self.block_size, 0), expected_output)

    def test_fit_to_block_sequence_too_big(self):
        """Truncate the sequence if it is too long."""
        sequence = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
        expected_output = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
        self.assertEqual(truncate_or_pad(sequence, self.block_size, 0), expected_output)

    def test_process_story_no_highlights(self):
        """Processing a story with no highlights returns an empty list for the summary."""
        raw_story = """It was the year of Our Lord one thousand seven hundred and
        seventy-five.\n\nSpiritual revelations were conceded to England at that
        favoured period, as at this."""
        _, summary_lines = process_story(raw_story)
        self.assertEqual(summary_lines, [])

    def test_process_empty_story(self):
        """An empty story returns an empty collection of lines."""
        raw_story = ""
        story_lines, summary_lines = process_story(raw_story)
        self.assertEqual(story_lines, [])
        self.assertEqual(summary_lines, [])

    def test_process_story_with_missing_period(self):
        raw_story = (
            "It was the year of Our Lord one thousand seven hundred and "
            "seventy-five\n\nSpiritual revelations were conceded to England "
            "at that favoured period, as at this.\n@highlight\n\nIt was the best of times"
        )
        story_lines, summary_lines = process_story(raw_story)

        expected_story_lines = [
            "It was the year of Our Lord one thousand seven hundred and seventy-five.",
            "Spiritual revelations were conceded to England at that favoured period, as at this.",
        ]
        self.assertEqual(expected_story_lines, story_lines)

        expected_summary_lines = ["It was the best of times."]
        self.assertEqual(expected_summary_lines, summary_lines)

    def test_build_mask_no_padding(self):
        sequence = torch.tensor([1, 2, 3, 4])
        expected = torch.tensor([1, 1, 1, 1])
        np.testing.assert_array_equal(build_mask(sequence, 0).numpy(), expected.numpy())

    def test_build_mask(self):
        sequence = torch.tensor([1, 2, 3, 4, 23, 23, 23])
        expected = torch.tensor([1, 1, 1, 1, 0, 0, 0])
        np.testing.assert_array_equal(build_mask(sequence, 23).numpy(), expected.numpy())

    def test_build_mask_with_padding_equal_to_one(self):
        sequence = torch.tensor([8, 2, 3, 4, 1, 1, 1])
        expected = torch.tensor([1, 1, 1, 1, 0, 0, 0])
        np.testing.assert_array_equal(build_mask(sequence, 1).numpy(), expected.numpy())

    def test_compute_token_type_ids(self):
        separator = 101
        batch = torch.tensor([[1, 2, 3, 4, 5, 6], [1, 2, 3, 101, 5, 6], [1, 101, 3, 4, 101, 6]])
        expected = torch.tensor([[1, 1, 1, 1, 1, 1], [1, 1, 1, 0, 0, 0], [1, 0, 0, 0, 1, 1]])

        result = compute_token_type_ids(batch, separator)


SUMMARY:
**Class Summary:**
```python
class SummarizationDataProcessingTest(unittest.TestCase):
    """
    Test class for summarization data processing.

    This class contains test methods to verify the correctness of methods used in summarization data processing.
    """

    # Class parameter for block size
    def setUp(self):
        self.block_size = 10

    def truncate_or_pad_sequence(sequence, block_size, padding_value):
        """
        Truncate or pad a sequence to fit within a specified block size.

        Args:
            sequence (list): The input sequence.
            block_size (int): The desired block size.
            padding_value: The value used for padding.

        Returns:
            list: The processed sequence.
        """

    def process_story(raw_story):
        """
        Process a raw story and return the summary lines.

        Args:
            raw_story (str): The input raw story.

        Returns:
            tuple: A tuple containing the story lines and summary lines.
        """

    def build_mask(sequence, sequence_value):
        """
        Build a mask for the sequence.

        Args:
            sequence (Tensor): The input sequence.
            sequence_value (int): The desired value in the mask.

        Returns:
            Tensor: The built mask.
        """

    def compute_token_type_ids(batch, separator):
        """
        Compute token type IDs for a batch of sequences.

        Args:
            batch (Tensor): The input batch of sequences.
            separator (int): The separator used to split sequences.

        Returns:
            Tensor: The computed token type IDs.
        """

```

**Implementation Details:**

*   **truncate_or_pad_sequence:** Verifies that the given sequence is truncated or padded with a specified padding value to fit within a blocked size.
*   **process_story:** Validates the processing of raw stories, including handling missing periods and no/highlight tags.
*   **build_mask:** Checks the construction of masks for sequences using a desired value in the mask.
*   **compute_token_type_ids:** Examines the computation of token type IDs for batches of sequences based on a specified separator.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/bertabs/convert_bertabs_original_pytorch_checkpoint.py


key: chroma:document
string_value:
CODE:

def convert_bertabs_checkpoints(path_to_checkpoints, dump_path):
    """Copy/paste and tweak the pre-trained weights provided by the creators
    of BertAbs for the internal architecture.
    """

    # Instantiate the authors' model with the pre-trained weights
    config = BertAbsConfig(
        temp_dir=".",
        finetune_bert=False,
        large=False,
        share_emb=True,
        use_bert_emb=False,
        encoder="bert",
        max_pos=512,
        enc_layers=6,
        enc_hidden_size=512,
        enc_heads=8,
        enc_ff_size=512,
        enc_dropout=0.2,
        dec_layers=6,
        dec_hidden_size=768,
        dec_heads=8,
        dec_ff_size=2048,
        dec_dropout=0.2,
    )
    checkpoints = torch.load(path_to_checkpoints, lambda storage, loc: storage)
    original = AbsSummarizer(config, torch.device("cpu"), checkpoints)
    original.eval()

    new_model = BertAbsSummarizer(config, torch.device("cpu"))
    new_model.eval()

    # -------------------
    # Convert the weights
    # -------------------

    logging.info("convert the model")
    new_model.bert.load_state_dict(original.bert.state_dict())
    new_model.decoder.load_state_dict(original.decoder.state_dict())
    new_model.generator.load_state_dict(original.generator.state_dict())

    # ----------------------------------
    # Make sure the outpus are identical
    # ----------------------------------

    logging.info("Make sure that the models' outputs are identical")
    tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased")

    # prepare the model inputs
    encoder_input_ids = tokenizer.encode("This is sample éàalj'-.")
    encoder_input_ids.extend([tokenizer.pad_token_id] * (512 - len(encoder_input_ids)))
    encoder_input_ids = torch.tensor(encoder_input_ids).unsqueeze(0)
    decoder_input_ids = tokenizer.encode("This is sample 3 éàalj'-.")
    decoder_input_ids.extend([tokenizer.pad_token_id] * (512 - len(decoder_input_ids)))
    decoder_input_ids = torch.tensor(decoder_input_ids).unsqueeze(0)

    # failsafe to make sure the weights reset does not affect the
    # loaded weights.
    assert torch.max(torch.abs(original.generator[0].weight - new_model.generator[0].weight)) == 0

    # forward pass
    src = encoder_input_ids
    tgt = decoder_input_ids
    segs = token_type_ids = None
    clss = None
    mask_src = encoder_attention_mask = None
    mask_tgt = decoder_attention_mask = None
    mask_cls = None

    # The original model does not apply the geneator layer immediatly but rather in
    # the beam search (where it combines softmax + linear layer). Since we already
    # apply the softmax in our generation process we only apply the linear layer here.
    # We make sure that the outputs of the full stack are identical
    output_original_model = original(src, tgt, segs, clss, mask_src, mask_tgt, mask_cls)[0]
    output_original_generator = original.generator(output_original_model)

    output_converted_model = new_model(
        encoder_input_ids, decoder_input_ids, token_type_ids, encoder_attention_mask, decoder_attention_mask
    )[0]
    output_converted_generator = new_model.generator(output_converted_model)

    maximum_absolute_difference = torch.max(torch.abs(output_converted_model - output_original_model)).item()
    print("Maximum absolute difference beween weights: {:.2f}".format(maximum_absolute_difference))
    maximum_absolute_difference = torch.max(torch.abs(output_converted_generator - output_original_generator)).item()
    print("Maximum absolute difference beween weights: {:.2f}".format(maximum_absolute_difference))

    are_identical = torch.allclose(output_converted_model, output_original_model, atol=1e-3)
    if are_identical:
        logging.info("all weights are equal up to 1e-3")
    else:
        raise ValueError("the weights are different. The new model is likely different from the original one.")

    # The model has been saved with torch.save(model) and this is bound to the exact
    # directory structure. We save the state_dict instead.
    logging.info("saving the model's state dictionary")
    torch.save(
        new_model.state_dict(), "./bertabs-finetuned-cnndm-extractive-abstractive-summarization/pytorch_model.bin"


SUMMARY:
**Function Summary: `convert_bertabs_checkpoints`**

**Purpose and Functionality**

This function copies, tweaks, and validates pre-trained weights from the creators of `BertAbs` for the internal architecture. It loads a model, converts the weights to a new model instance, checks if matching outputs are identical, and then saves the modified weight dictionary.

**Key Implementation Details**

1. **Loading and instantiating models**: The function loads a `BertAbsConfig` object, then creates two instances: an original model (`original`) and a new model (`new_model`) to be finetuned.
2. **Weight conversion**: It transfers weights from the original model (`copy_state_dict`) to the new model instance (`new_model.bert.load_state_dict`, `new_model.decoder.load_state_dict`).
3. **Output validation**: The function tests if matching outputs are identical between the two models by comparing their generated responses (`torch.allclose`).

**Notable Dependencies and Requirements**

1. *`BertTokenizer`* for encoding inputs
2. *`abs_summarizer`* module (not explicitly named in this snippet, but implied)
3. *`torch`* for PyTorch library functionality

```markdown
# Summary Function: `convert_bertabs_checkpoints`

### Purpose and Functionality
Load pre-trained weights, finetune a model, and validate outputs.

### Key Implementation Details
1. Load config object.
2. Instantiate original and new models.
3. Copy weights from original to new model instance.

### Notable Dependencies
- `BertTokenizer`
- `abs_summarizer` (library)
- `torch`

## Code Summary
```python
def convert_bertabs_checkpoints(path_to_checkpoints, dump_path):
    ...
    
    # Weight conversion and output validation
    
    # ...
```
### Function Output
The function returns the updated weight dictionary saved to a binary file.

### Error Handling
If model outputs are not identical, it raises a `ValueError`.


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/bertabs/utils_summarization.py


key: chroma:document
string_value:
CODE:

class CNNDMDataset(Dataset):
    """Abstracts the dataset used to train seq2seq models.

    The class will process the documents that are located in the specified
    folder. The preprocessing will work on any document that is reasonably
    formatted. On the CNN/DailyMail dataset it will extract both the story
    and the summary.

    CNN/Daily News:

    The CNN/Daily News raw datasets are downloaded from [1]. The stories are
    stored in different files; the summary appears at the end of the story as
    sentences that are prefixed by the special `@highlight` line. To process
    the data, untar both datasets in the same folder, and pass the path to this
    folder as the "data_dir argument. The formatting code was inspired by [2].

    [1] https://cs.nyu.edu/~kcho/
    [2] https://github.com/abisee/cnn-dailymail/
    """

    def __init__(self, path="", prefix="train"):
        """We initialize the class by listing all the documents to summarize.
        Files are not read in memory due to the size of some datasets (like CNN/DailyMail).
        """
        assert os.path.isdir(path)

        self.documents = []
        story_filenames_list = os.listdir(path)
        for story_filename in story_filenames_list:
            if "summary" in story_filename:
                continue
            path_to_story = os.path.join(path, story_filename)
            if not os.path.isfile(path_to_story):
                continue
            self.documents.append(path_to_story)

    def __len__(self):
        """Returns the number of documents."""
        return len(self.documents)

    def __getitem__(self, idx):
        document_path = self.documents[idx]
        document_name = document_path.split("/")[-1]
        with open(document_path, encoding="utf-8") as source:
            raw_story = source.read()
            story_lines, summary_lines = process_story(raw_story)


SUMMARY:
**Summary**

The `CNNDMDataset` class is an abstract dataset class for training seq2seq models. It processes documents in a specified folder, extracting story and summary lines. The class does not load the entire document into memory.

**Main Purpose and Functionality**

* Process documents from a specified(folder) path.
* Extract story and summary lines from each document.
* Return document name, story lines, and summary lines.

**Implementation Details**

* Documents are processed without loading them entirely into memory.
* Story filenames containing "summary" are skipped.
* File paths for all other stories are stored in the `documents` list.
* The `process_story` function is used to extract story and summary lines from each document.

**Notable Dependencies or Requirements**

* `os` module for interacting with file system.
* `process_story` function (not shown) for parsing story and summary lines.


key: chroma:document
string_value:
CODE:

def process_story(raw_story):
    """Extract the story and summary from a story file.

    Arguments:
        raw_story (str): content of the story file as an utf-8 encoded string.

    Raises:
        IndexError: If the story is empty or contains no highlights.
    """
    nonempty_lines = list(filter(lambda x: len(x) != 0, [line.strip() for line in raw_story.split("\n")]))

    # for some unknown reason some lines miss a period, add it
    nonempty_lines = [_add_missing_period(line) for line in nonempty_lines]

    # gather article lines
    story_lines = []
    lines = deque(nonempty_lines)
    while True:
        try:
            element = lines.popleft()
            if element.startswith("@highlight"):
                break
            story_lines.append(element)
        except IndexError:
            # if "@highlight" is absent from the file we pop
            # all elements until there is None, raising an exception.
            return story_lines, []

    # gather summary lines
    summary_lines = list(filter(lambda t: not t.startswith("@highlight"), lines))



SUMMARY:
**Function Summary:**

### `process_storyraw_story`

Extracts the story and summary from a given raw story file content.

**Purpose:** Separate raw story content into story and summary sections while excluding highlighted elements.

**Implementation Details:**

* Processes raw story content by:
	+ Filtering out empty lines
	+ Adding missing periods to relevant lines
	+ Iterating over processed lines, collecting non-highlighted lines until a highlight marker (@highlight) is encountered
	+ Collecting remaining lines as summary content

**Key Elements:**

* `@highlight` (marker indicating a highlighted element)
* `nonempty_lines` (pre-filtered list of raw story lines)
* `story_lines`, `summary_lines` (collected sections separated by the @highlight marker)

**External Dependencies:** None noted.


key: chroma:document
string_value:
CODE:

def _add_missing_period(line):
    END_TOKENS = [".", "!", "?", "...", "'", "`", '"', "\u2019", "\u2019", ")"]
    if line.startswith("@highlight"):
        return line
    if line[-1] in END_TOKENS:
        return line


SUMMARY:
**Function Summary: `_add_missing_period`**

#### Primary Purpose:
This function appends a period to the end of a given input `line`, unless one of three conditions are met:

1.  The line starts with "@highlight".
2.  The line ends with an existing end token (.", "!", "?", ..., "`", '"', "\u2019", "\u3017").
3.  The line does not match any specified ignore patterns.

#### Key Implementation Details:
-   The function utilizes a list of defined `END_TOKENS` to check for punctuation at the end of the input line.
-   It uses `startswith` and `endswith` methods to perform conditional checks on the input line.

#### Notable Dependencies or Requirements:
None. This is a self-contained function without any external dependencies.


key: chroma:document
string_value:
CODE:

def truncate_or_pad(sequence, block_size, pad_token_id):
    """Adapt the source and target sequences' lengths to the block size.
    If the sequence is shorter we append padding token to the right of the sequence.
    """
    if len(sequence) > block_size:
        return sequence[:block_size]
    else:
        sequence.extend([pad_token_id] * (block_size - len(sequence)))


SUMMARY:
**Function Summary**

`truncate_or_pad` truncates or pads a sequence to fit a specified `block_size`.

**Primary Purpose**
The function adapts the length of a sequence (`sequence`) to match the `block_size` by either trimming excess elements (if `sequence` is longer) or appending padding tokens (`pad_token_id`) if necessary.

**Key Implementation Details**

* If `sequence` is longer than `block_size`, it is truncated using slicing (`sequence[:block_size]`).
* To pad `sequence` to match `block_size`, the function appends repeated instances of `pad_token_id`.

**Notable Dependencies/Requirements**
None explicitly stated. However, since padding with `pad_token_id` is mentioned, the presence of a `pad_token_id` variable or constant is implied.

**External/Inherited Interfaces**
No specific interfaces are referenced in this code snippet. However, the function appears to be designed for use in a context that involves encoding and preprocessing, as indicated by the commented sections at the top.


key: chroma:document
string_value:
CODE:

def build_mask(sequence, pad_token_id):
    """Builds the mask. The attention mechanism will only attend to positions
    with value 1."""
    mask = torch.ones_like(sequence)
    idx_pad_tokens = sequence == pad_token_id
    mask[idx_pad_tokens] = 0


SUMMARY:
**Summary**

The `build_mask` function constructs an attention mask for the input `sequence` by replacing padding tokens (`pad_token_id`) with zeros. The function creates a binary mask where any value other than zero indicates attended positions.

**Key Details:**

* Function signature: `build_mask(sequence, pad_token_id)`
* Primary purpose: Create attention mask to ignore padded tokens.
* Implementation details:
	+ Creates a ones-like tensor for the entire sequence using `torch.ones_like(sequence)`.
	+ Masks padding tokens (`sequence == pad_token_id`) with zeros and updates corresponding values in the mask.

**Note:**

* Relies on PyTorch's tensor operations (`==`, `torch.ones_like`, `idx_pad_tokens = ..., mask[idx_pad_tokens] = ...`).
* Ignores padded tokens to prevent attention mechanism from attending to them.


key: chroma:document
string_value:
CODE:

def encode_for_summarization(story_lines, summary_lines, tokenizer):
    """Encode the story and summary lines, and join them
    as specified in [1] by using `[SEP] [CLS]` tokens to separate
    sentences.
    """
    story_lines_token_ids = [tokenizer.encode(line) for line in story_lines]
    story_token_ids = [token for sentence in story_lines_token_ids for token in sentence]
    summary_lines_token_ids = [tokenizer.encode(line) for line in summary_lines]
    summary_token_ids = [token for sentence in summary_lines_token_ids for token in sentence]



SUMMARY:
**Concise Summary**

The `encode_for_summarization` function encodes story and summary lines into token IDs using the provided `tokenizer`. It combines token IDs from each line to form a single list for both inputs using `[SEP] [CLS]` tokens as sentence separators. The primary purpose is to prepare text data for summarization.

**Key Implementation Details**

* Tokenizes input strings using `tokenizer.encode()`.
* Flattens lists of tokenized lines into individual token IDs.
* Returns two lists: `story_token_ids` and `summary_token_ids`.

**Notable Dependencies or Requirements**

* A `tokenizer` object with an `encode()` method (e.g., `bert.encoder.from_pretrained()`).
* External dependencies not explicitly stated; likely assumes a pre-trained BERT model.


key: chroma:document
string_value:
CODE:

def compute_token_type_ids(batch, separator_token_id):
    """Segment embeddings as described in [1]

    The values {0,1} were found in the repository [2].

    Attributes:
        batch: torch.Tensor, size [batch_size, block_size]
            Batch of input.
        separator_token_id: int
            The value of the token that separates the segments.

    [1] Liu, Yang, and Mirella Lapata. "Text summarization with pretrained encoders."
        arXiv preprint arXiv:1908.08345 (2019).
    [2] https://github.com/nlpyang/PreSumm (/src/prepro/data_builder.py, commit fac1217)
    """
    batch_embeddings = []
    for sequence in batch:
        sentence_num = -1
        embeddings = []
        for s in sequence:
            if s == separator_token_id:
                sentence_num += 1
            embeddings.append(sentence_num % 2)
        batch_embeddings.append(embeddings)


SUMMARY:
**Function Summary: `compute_token_type_ids`**

**Purpose:** The function segments embeddings from an input batch into individual sentences, replacing each sentence with its corresponding type ID (0 or 1) based on the provided separator token ID.

**Key Implementation Details:**

* Uses a nested loop structure to iterate over input sequences and detect separation tokens.
* Assigns even-indexed IDs as one type (e.g., 0) and odd-indexed IDs as another type (e.g., 1).
* Returns a PyTorch tensor containing the segmented embeddings.

**Dependencies/Requirements:**

* Input `batch` must be a `torch.Tensor` with shape `[batch_size, block_size]`.
* The separator token ID (`separator_token_id`) is an `int` value.
* No external dependencies are explicitly mentioned.


key: chroma:document
string_value:
CODE:

class BertAbsPreTrainedModel(PreTrainedModel):
    config_class = BertAbsConfig
    load_tf_weights = False


SUMMARY:
**Summary:**

The `BertAbsPreTrainedModel` class is a pre-trained BERT model variant that inherits from the `PreTrainedModel`. It defines:

* A configuration class `BertAbsConfig`
* Loads TF weights by default (can be disabled)
* Uses "bert" as the prefix for its base model

**Key Implementation Details:**

* Inherits interface from `PreTrainedModel`
* Does not load TF weights by default (`load_tf_weights = False`)

**Notable Dependencies/Requirements:**

N/A


key: chroma:document
string_value:
CODE:

class BertAbs(BertAbsPreTrainedModel):
    def __init__(self, args, checkpoint=None, bert_extractive_checkpoint=None):
        super().__init__(args)
        self.args = args
        self.bert = Bert()

        # If pre-trained weights are passed for Bert, load these.
        load_bert_pretrained_extractive = True if bert_extractive_checkpoint else False
        if load_bert_pretrained_extractive:
            self.bert.model.load_state_dict(
                {n[11:]: p for n, p in bert_extractive_checkpoint.items() if n.startswith("bert.model")},
                strict=True,
            )

        self.vocab_size = self.bert.model.config.vocab_size

        if args.max_pos > 512:
            my_pos_embeddings = nn.Embedding(args.max_pos, self.bert.model.config.hidden_size)
            my_pos_embeddings.weight.data[:512] = self.bert.model.embeddings.position_embeddings.weight.data
            my_pos_embeddings.weight.data[512:] = self.bert.model.embeddings.position_embeddings.weight.data[-1][
                None, :
            ].repeat(args.max_pos - 512, 1)
            self.bert.model.embeddings.position_embeddings = my_pos_embeddings
        tgt_embeddings = nn.Embedding(self.vocab_size, self.bert.model.config.hidden_size, padding_idx=0)

        tgt_embeddings.weight = copy.deepcopy(self.bert.model.embeddings.word_embeddings.weight)

        self.decoder = TransformerDecoder(
            self.args.dec_layers,
            self.args.dec_hidden_size,
            heads=self.args.dec_heads,
            d_ff=self.args.dec_ff_size,
            dropout=self.args.dec_dropout,
            embeddings=tgt_embeddings,
            vocab_size=self.vocab_size,
        )

        gen_func = nn.LogSoftmax(dim=-1)
        self.generator = nn.Sequential(nn.Linear(args.dec_hidden_size, args.vocab_size), gen_func)
        self.generator[0].weight = self.decoder.embeddings.weight

        load_from_checkpoints = False if checkpoint is None else True
        if load_from_checkpoints:
            self.load_state_dict(checkpoint)

    def init_weights(self):
        for module in self.decoder.modules():
            if isinstance(module, (nn.Linear, nn.Embedding)):
                module.weight.data.normal_(mean=0.0, std=0.02)
            elif isinstance(module, nn.LayerNorm):
                module.bias.data.zero_()
                module.weight.data.fill_(1.0)
            if isinstance(module, nn.Linear) and module.bias is not None:
                module.bias.data.zero_()
        for p in self.generator.parameters():
            if p.dim() > 1:
                xavier_uniform_(p)
            else:
                p.data.zero_()

    def forward(
        self,
        encoder_input_ids,
        decoder_input_ids,
        token_type_ids,
        encoder_attention_mask,
        decoder_attention_mask,
    ):
        encoder_output = self.bert(
            input_ids=encoder_input_ids,
            token_type_ids=token_type_ids,
            attention_mask=encoder_attention_mask,
        )
        encoder_hidden_states = encoder_output[0]
        dec_state = self.decoder.init_decoder_state(encoder_input_ids, encoder_hidden_states)
        decoder_outputs, _ = self.decoder(decoder_input_ids[:, :-1], encoder_hidden_states, dec_state)


SUMMARY:
**Summary: BertAbs Model Class**

The `BertAbs` class extends the pre-trained BERT model and is designed for extractive text classification tasks. The primary functionality of this model is to take in input IDs, attention masks, and token types from a decoder, and generate output sequences.

### Key Implementation Details

* The model uses two separate embedding layers: one for the input data (`self.bert`) and another for the decoder's target tokens (`tgt_embeddings`).
* The decoder is an instance of `TransformerDecoder`, which takes in the decoder input IDs, encoder hidden states, and a decoder state to generate output sequences.
* The generator section uses a linear layer followed by a LogSoftmax activation function to produce the probability distributions over possible next tokens.

### Notable Dependencies and Requirements

* `BertAbsPreTrainedModel`
* ` nn.Module` (parent class)
* `Bert` model
* `TransformerDecoder` module
* `LogSoftmax` activation function
* `Linear` layer for generator
* `nn.Embedding` layers

### Method Signatures and Interfaces

* `__init__(args, checkpoint=None, bert_extractive_checkpoint=None)`: Initializes the model with arguments and optional pre-trained weights or checkpoints.
* `forward(encoder_input_ids, decoder_input_ids, token_type_ids, encoder_attention_mask, decoder_attention_mask)`: Takes in input IDs and attention masks from both the encoder and decoder, and generates output sequences.


key: chroma:document
string_value:
CODE:

class Bert(nn.Module):
    """This class is not really necessary and should probably disappear."""

    def __init__(self):
        super().__init__()
        config = BertConfig.from_pretrained("google-bert/bert-base-uncased")
        self.model = BertModel(config)

    def forward(self, input_ids, attention_mask=None, token_type_ids=None, **kwargs):
        self.eval()
        with torch.no_grad():
            encoder_outputs, _ = self.model(
                input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask, **kwargs
            )


SUMMARY:
**Summary**

The `Bert` class is a PyTorch neural network module implementing the BERT (Bidirectional Encoder Representations from Transformers) architecture. The primary purpose of this class is to process text input using a pre-trained BERT model.

**Key Implementation Details:**

* **Model Instantiation**: The `Bert` class loads a pre-trained BERT model configuration and creates an instance (`self.model`) using the `BertModel` class.
* **Forward Pass**: The `forward` method processes input text using the `self.model`, extracting encoder outputs with optional token type IDs and attention masks.

**Notable Dependencies/Requirements:**

* `BertConfig`: a pre-trained configuration loaded from "google-bert/bert-base-uncased"
* `BertModel`: the underlying BERT model implementation

This class appears to be a wrapper around a pre-trained BERT model, allowing users to easily process text input using this powerful architecture.


key: chroma:document
string_value:
CODE:

class TransformerDecoder(nn.Module):
    """
    The Transformer decoder from "Attention is All You Need".

    Args:
       num_layers (int): number of encoder layers.
       d_model (int): size of the model
       heads (int): number of heads
       d_ff (int): size of the inner FF layer
       dropout (float): dropout parameters
       embeddings (:obj:`onmt.modules.Embeddings`):
          embeddings to use, should have positional encodings
       attn_type (str): if using a separate copy attention
    """

    def __init__(self, num_layers, d_model, heads, d_ff, dropout, embeddings, vocab_size):
        super().__init__()

        # Basic attributes.
        self.decoder_type = "transformer"
        self.num_layers = num_layers
        self.embeddings = embeddings
        self.pos_emb = PositionalEncoding(dropout, self.embeddings.embedding_dim)

        # Build TransformerDecoder.
        self.transformer_layers = nn.ModuleList(
            [TransformerDecoderLayer(d_model, heads, d_ff, dropout) for _ in range(num_layers)]
        )

        self.layer_norm = nn.LayerNorm(d_model, eps=1e-6)

    # forward(input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask)
    # def forward(self, input_ids, state, attention_mask=None, memory_lengths=None,
    # step=None, cache=None, encoder_attention_mask=None, encoder_hidden_states=None, memory_masks=None):
    def forward(
        self,
        input_ids,
        encoder_hidden_states=None,
        state=None,
        attention_mask=None,
        memory_lengths=None,
        step=None,
        cache=None,
        encoder_attention_mask=None,
    ):
        """
        See :obj:`onmt.modules.RNNDecoderBase.forward()`
        memory_bank = encoder_hidden_states
        """
        # Name conversion
        tgt = input_ids
        memory_bank = encoder_hidden_states
        memory_mask = encoder_attention_mask

        # src_words = state.src
        src_words = state.src
        src_batch, src_len = src_words.size()

        padding_idx = self.embeddings.padding_idx

        # Decoder padding mask
        tgt_words = tgt
        tgt_batch, tgt_len = tgt_words.size()
        tgt_pad_mask = tgt_words.data.eq(padding_idx).unsqueeze(1).expand(tgt_batch, tgt_len, tgt_len)

        # Encoder padding mask
        if memory_mask is not None:
            src_len = memory_mask.size(-1)
            src_pad_mask = memory_mask.expand(src_batch, tgt_len, src_len)
        else:
            src_pad_mask = src_words.data.eq(padding_idx).unsqueeze(1).expand(src_batch, tgt_len, src_len)

        # Pass through the embeddings
        emb = self.embeddings(input_ids)
        output = self.pos_emb(emb, step)
        assert emb.dim() == 3  # len x batch x embedding_dim

        if state.cache is None:
            saved_inputs = []

        for i in range(self.num_layers):
            prev_layer_input = None
            if state.cache is None:
                if state.previous_input is not None:
                    prev_layer_input = state.previous_layer_inputs[i]

            output, all_input = self.transformer_layers[i](
                output,
                memory_bank,
                src_pad_mask,
                tgt_pad_mask,
                previous_input=prev_layer_input,
                layer_cache=state.cache["layer_{}".format(i)] if state.cache is not None else None,
                step=step,
            )
            if state.cache is None:
                saved_inputs.append(all_input)

        if state.cache is None:
            saved_inputs = torch.stack(saved_inputs)

        output = self.layer_norm(output)

        if state.cache is None:
            state = state.update_state(tgt, saved_inputs)

        # Decoders in transformers return a tuple. Beam search will fail
        # if we don't follow this convention.
        return output, state  # , state

    def init_decoder_state(self, src, memory_bank, with_cache=False):
        """Init decoder state"""
        state = TransformerDecoderState(src)
        if with_cache:
            state._init_cache(memory_bank, self.num_layers)


SUMMARY:
**TransformerDecoder Class Summary**

### Purpose and Functionality
The `TransformerDecoder` class is an implementation of the Transformer decoder architecture from "Attention is All You Need". It is designed to decode input sequences given encoder output.

### Key Implementation Details

*   **Token Embeddings**: The model uses token embeddings with positional encodings.
*   **Layer Normalization and Dropout**: Each layer employs layer normalization and dropout for regularization.
*   **Transformer Decoder Layers**: The decoder consists of a list of `nn.ModuleList` containing instances of `TransformerDecoderLayer`.
*   **Forward Pass**: The forward pass involves the following steps:
    *   Input token embeddings
    *   Positional encoding
    *   Decoder padding mask generation
    *   Forward propagation through each layer (with or without previous layer input)
    *   Layer normalization and updating state

### Notable Dependencies and Requirements

*   `nn.Module`: PyTorch's neural network module.
*   `onmt.modules.Embeddings`: Custom embedding module with positional encodings.
*   `TransformerDecoderLayer`: A submodule implementing the individual decoder layer.
*   `PositionalEncoding`: Module for positional encoding.

### Method Signatures and Interfaces

*   `__init__(num_layers, d_model, heads, d_ff, dropout, embeddings, vocab_size)`: Initializes the model with given hyperparameters.
*   `forward(input_ids, encoder_hidden_states=None, state=None, attention_mask=None, memory_lengths=None, step=None, cache=None, encoder_attention_mask=None, encoder_hidden_states=None, memory_masks=None)`: Performs the forward pass through the entire decoder architecture.

### Example Inheritance and Interfaces

The model inherits from PyTorch's `nn.Module` class. It also implements the `RNNDecoderBase.forward()` method as per "Attention is All You Need".


key: chroma:document
string_value:
CODE:

class PositionalEncoding(nn.Module):
    def __init__(self, dropout, dim, max_len=5000):
        pe = torch.zeros(max_len, dim)
        position = torch.arange(0, max_len).unsqueeze(1)
        div_term = torch.exp((torch.arange(0, dim, 2, dtype=torch.float) * -(math.log(10000.0) / dim)))
        pe[:, 0::2] = torch.sin(position.float() * div_term)
        pe[:, 1::2] = torch.cos(position.float() * div_term)
        pe = pe.unsqueeze(0)
        super().__init__()
        self.register_buffer("pe", pe)
        self.dropout = nn.Dropout(p=dropout)
        self.dim = dim

    def forward(self, emb, step=None):
        emb = emb * math.sqrt(self.dim)
        if step:
            emb = emb + self.pe[:, step][:, None, :]

        else:
            emb = emb + self.pe[:, : emb.size(1)]
        emb = self.dropout(emb)
        return emb

    def get_emb(self, emb):


SUMMARY:
**Summary: PositionalEncoding Class**

The `PositionalEncoding` class is an PyTorch module that adds positional encoding to an input embedding. It's commonly used in transformer architectures.

**Main Purpose and Functionality**

This class calculates the positional encoding of an embedded vector by adding a learned scaling factor to the original embedding, allowing the sequence position to be retrieved from it.

### Key Implementation Details

*   This layer uses a sinusoidal position encoding scheme, where:
    *   `pe[:, 0::2] = torch.sin(position.float() \* div_term)` and `pe[:, 1::2] = torch.cos(position.float() \* div_term)`
        correspond to the dot (0-indexed positions) and indexable (1-indexed positions), respectively.
*   The scaling factor, `math.sqrt(self.dim)`, is applied after adding positional encoding.

### Notable Dependencies or Requirements

*   The classes `torch` and `nn` are used extensively in this model.
*   The `math.log()` function with specific arguments.

Below is the RAG system's data structure format representation of the class:

```markdown
**PositionalEncoding**
=====================

### Main Functionality
#### Purpose:
Apply positional encoding to input embedding using sinusoidal scheme and scaling factor.

### Key Details

*   Applies scale factor `math.sqrt(dim)` after adding positional encoding
*   Uses torch.sin() for dot positions (0-indexed) and torch.cos() for indexable positions (1-indexed)

### Dependencies/Requirements

*   Torch Module (`torch`)
*   PyTorch Neural Network (`nn` module)
*   Math functions (`math.log()`)
```

**Class Members:**

```markdown
### **pe**
#### Buffer Name:
   - Stores pre-calculated positional encoding vector.
#### Type:
    • Torch Float Tensor
*   `pe`,  `(max_len, dim)`
*   Derived from sin position float and cosine term with offset

#### Self.dropout 
     *   Dropout layer (`torch.nn.Dropout`)
     *   Dropout probability: `p = dropout`
#### self.dim 
     *   Input dimension.
```

 **Class Method**

```markdown
 ### forward(self, emb, step= None)
#### Purpose:
    • Apply scale factor and embedding to positional encoding.
#### Parameters
    • `emb`: Input sequence
        - Input Embedding (torch Float Tensor Shape `(batch_size x Sequence_length x EmbedDim)`
        *   Shape can include padding or length adjustment by optional parameter (`step`).
*   `self.pe[:, step][:, None: , :]`
  *   Returns embedding that added to original input (`emb`). 
*   Self.dropout(embedding), for added dropout value
*   Outputs transformed tensor 
       - Outputs are a Float tensor similar to the embedding.
### get_emb(self, emb)
#### Purpose:
             • Calculates positional encoding vector from emb size
            Type torch float Tensor.
            The output is `pe[:, :emb.size(1)]`
             ```


key: chroma:document
string_value:
CODE:

class TransformerDecoderLayer(nn.Module):
    """
    Args:
      d_model (int): the dimension of keys/values/queries in
                       MultiHeadedAttention, also the input size of
                       the first-layer of the PositionwiseFeedForward.
      heads (int): the number of heads for MultiHeadedAttention.
      d_ff (int): the second-layer of the PositionwiseFeedForward.
      dropout (float): dropout probability(0-1.0).
      self_attn_type (string): type of self-attention scaled-dot, average
    """

    def __init__(self, d_model, heads, d_ff, dropout):
        super().__init__()

        self.self_attn = MultiHeadedAttention(heads, d_model, dropout=dropout)

        self.context_attn = MultiHeadedAttention(heads, d_model, dropout=dropout)
        self.feed_forward = PositionwiseFeedForward(d_model, d_ff, dropout)
        self.layer_norm_1 = nn.LayerNorm(d_model, eps=1e-6)
        self.layer_norm_2 = nn.LayerNorm(d_model, eps=1e-6)
        self.drop = nn.Dropout(dropout)
        mask = self._get_attn_subsequent_mask(MAX_SIZE)
        # Register self.mask as a saved_state in TransformerDecoderLayer, so
        # it gets TransformerDecoderLayer's cuda behavior automatically.
        self.register_buffer("mask", mask)

    def forward(
        self,
        inputs,
        memory_bank,
        src_pad_mask,
        tgt_pad_mask,
        previous_input=None,
        layer_cache=None,
        step=None,
    ):
        """
        Args:
            inputs (`FloatTensor`): `[batch_size x 1 x model_dim]`
            memory_bank (`FloatTensor`): `[batch_size x src_len x model_dim]`
            src_pad_mask (`LongTensor`): `[batch_size x 1 x src_len]`
            tgt_pad_mask (`LongTensor`): `[batch_size x 1 x 1]`

        Returns:
            (`FloatTensor`, `FloatTensor`, `FloatTensor`):

            * output `[batch_size x 1 x model_dim]`
            * attn `[batch_size x 1 x src_len]`
            * all_input `[batch_size x current_step x model_dim]`

        """
        dec_mask = torch.gt(tgt_pad_mask + self.mask[:, : tgt_pad_mask.size(1), : tgt_pad_mask.size(1)], 0)
        input_norm = self.layer_norm_1(inputs)
        all_input = input_norm
        if previous_input is not None:
            all_input = torch.cat((previous_input, input_norm), dim=1)
            dec_mask = None

        query = self.self_attn(
            all_input,
            all_input,
            input_norm,
            mask=dec_mask,
            layer_cache=layer_cache,
            type="self",
        )

        query = self.drop(query) + inputs

        query_norm = self.layer_norm_2(query)
        mid = self.context_attn(
            memory_bank,
            memory_bank,
            query_norm,
            mask=src_pad_mask,
            layer_cache=layer_cache,
            type="context",
        )
        output = self.feed_forward(self.drop(mid) + query)

        return output, all_input
        # return output

    def _get_attn_subsequent_mask(self, size):
        """
        Get an attention mask to avoid using the subsequent info.

        Args:
            size: int

        Returns:
            (`LongTensor`):

            * subsequent_mask `[1 x size x size]`
        """
        attn_shape = (1, size, size)
        subsequent_mask = np.triu(np.ones(attn_shape), k=1).astype("uint8")
        subsequent_mask = torch.from_numpy(subsequent_mask)


SUMMARY:
**TransformerDecoderLayer Class Summary**

The `TransformerDecoderLayer` class is a transformer decoder layer based on the Transformer architecture. It consists of three main components: self-attention, context attention, and positionwise feedforward.

**Main Purpose and Functionality**

* Compute self-attention between input sequences
* Compute context-aware attention from memory banks
* Apply position-wise feedforward to output

**Key Implementation Details**

* Use MultiHeadedAttention for both self-attention and context-aware attention
* PositionwiseFeedForward for the feed-forward layer
* LayerNorm for normalization of inputs and outputs
* Dropout for regularization

**Notable Dependencies and Requirements**

* `MultiHeadedAttention` class from the same library
* `PositionwiseFeedForward` class from the same library
* `nn.Module`, `torch`, `numpy`
* `LongTensor`, `FloatTensor`

The main methods are:

* `__init__`: Initializes the layer with given parameters
* `forward`: Computes the output of the layer by applying self-attention, context-aware attention, and position-wise feedforward

Note: The `_get_attn_subsequent_mask` method is a separate utility function that generates an attention mask to prevent using subsequent information.


key: chroma:document
string_value:
CODE:

class MultiHeadedAttention(nn.Module):
    """
    Multi-Head Attention module from
    "Attention is All You Need"
    :cite:`DBLP:journals/corr/VaswaniSPUJGKP17`.

    Similar to standard `dot` attention but uses
    multiple attention distributions simulataneously
    to select relevant items.

    .. mermaid::

       graph BT
          A[key]
          B[value]
          C[query]
          O[output]
          subgraph Attn
            D[Attn 1]
            E[Attn 2]
            F[Attn N]
          end
          A --> D
          C --> D
          A --> E
          C --> E
          A --> F
          C --> F
          D --> O
          E --> O
          F --> O
          B --> O

    Also includes several additional tricks.

    Args:
       head_count (int): number of parallel heads
       model_dim (int): the dimension of keys/values/queries,
           must be divisible by head_count
       dropout (float): dropout parameter
    """

    def __init__(self, head_count, model_dim, dropout=0.1, use_final_linear=True):
        assert model_dim % head_count == 0
        self.dim_per_head = model_dim // head_count
        self.model_dim = model_dim

        super().__init__()
        self.head_count = head_count

        self.linear_keys = nn.Linear(model_dim, head_count * self.dim_per_head)
        self.linear_values = nn.Linear(model_dim, head_count * self.dim_per_head)
        self.linear_query = nn.Linear(model_dim, head_count * self.dim_per_head)
        self.softmax = nn.Softmax(dim=-1)
        self.dropout = nn.Dropout(dropout)
        self.use_final_linear = use_final_linear
        if self.use_final_linear:
            self.final_linear = nn.Linear(model_dim, model_dim)

    def forward(
        self,
        key,
        value,
        query,
        mask=None,
        layer_cache=None,
        type=None,
        predefined_graph_1=None,
    ):
        """
        Compute the context vector and the attention vectors.

        Args:
           key (`FloatTensor`): set of `key_len`
                key vectors `[batch, key_len, dim]`
           value (`FloatTensor`): set of `key_len`
                value vectors `[batch, key_len, dim]`
           query (`FloatTensor`): set of `query_len`
                 query vectors  `[batch, query_len, dim]`
           mask: binary mask indicating which keys have
                 non-zero attention `[batch, query_len, key_len]`
        Returns:
           (`FloatTensor`, `FloatTensor`) :

           * output context vectors `[batch, query_len, dim]`
           * one of the attention vectors `[batch, query_len, key_len]`
        """
        batch_size = key.size(0)
        dim_per_head = self.dim_per_head
        head_count = self.head_count

        def shape(x):
            """projection"""
            return x.view(batch_size, -1, head_count, dim_per_head).transpose(1, 2)

        def unshape(x):
            """compute context"""
            return x.transpose(1, 2).contiguous().view(batch_size, -1, head_count * dim_per_head)

        # 1) Project key, value, and query.
        if layer_cache is not None:
            if type == "self":
                query, key, value = (
                    self.linear_query(query),
                    self.linear_keys(query),
                    self.linear_values(query),
                )

                key = shape(key)
                value = shape(value)

                if layer_cache is not None:
                    device = key.device
                    if layer_cache["self_keys"] is not None:
                        key = torch.cat((layer_cache["self_keys"].to(device), key), dim=2)
                    if layer_cache["self_values"] is not None:
                        value = torch.cat((layer_cache["self_values"].to(device), value), dim=2)
                    layer_cache["self_keys"] = key
                    layer_cache["self_values"] = value
            elif type == "context":
                query = self.linear_query(query)
                if layer_cache is not None:
                    if layer_cache["memory_keys"] is None:
                        key, value = self.linear_keys(key), self.linear_values(value)
                        key = shape(key)
                        value = shape(value)
                    else:
                        key, value = (
                            layer_cache["memory_keys"],
                            layer_cache["memory_values"],
                        )
                    layer_cache["memory_keys"] = key
                    layer_cache["memory_values"] = value
                else:
                    key, value = self.linear_keys(key), self.linear_values(value)
                    key = shape(key)
                    value = shape(value)
        else:
            key = self.linear_keys(key)
            value = self.linear_values(value)
            query = self.linear_query(query)
            key = shape(key)
            value = shape(value)

        query = shape(query)

        # 2) Calculate and scale scores.
        query = query / math.sqrt(dim_per_head)
        scores = torch.matmul(query, key.transpose(2, 3))

        if mask is not None:
            mask = mask.unsqueeze(1).expand_as(scores)
            scores = scores.masked_fill(mask, -1e18)

        # 3) Apply attention dropout and compute context vectors.

        attn = self.softmax(scores)

        if predefined_graph_1 is not None:
            attn_masked = attn[:, -1] * predefined_graph_1
            attn_masked = attn_masked / (torch.sum(attn_masked, 2).unsqueeze(2) + 1e-9)

            attn = torch.cat([attn[:, :-1], attn_masked.unsqueeze(1)], 1)

        drop_attn = self.dropout(attn)
        if self.use_final_linear:
            context = unshape(torch.matmul(drop_attn, value))
            output = self.final_linear(context)
            return output
        else:
            context = torch.matmul(drop_attn, value)


SUMMARY:
**MultiHeadedAttention Class Summary**

### Main Purpose and Functionality

The `MultiHeadedAttention` class implements the multi-head attention mechanism from the paper "Attention is All You Need". It projects three inputs (queries, keys, values) simultaneously using separate linear layers, calculates a weighted sum of these projections, applies softmax normalization, and returns two output tensors: a context vector and an attention vector.

### Key Implementation Details

1.  **Self-Attention Mechanism**: The class applies the self-attention mechanism for "self" type usage, where the query, key, and value are projected simultaneously using separate linear layers (`linear_query`, `linear_keys`, `linear_values`).
2.  **Context Vector Computation**: For both "self" and "context" types, the context vector is computed by applying the softmax function to the weighted sum of query projections and values.
3.  **Attention Dropout**: The attention mechanism applies dropout to the attention weights.
4.  **Final Linear Layer**: In the "self" type usage, a final linear layer (`final_linear`) is applied to the output context vector.

### Notable Dependencies or Requirements

1.  `nn.Linear`: Used for creating linear projectors for query, key, and value tensors.
2.  `nn.Dropout`: Used for applying attention dropout to the attention weights.
3.  `nn.Softmax`: Used for computing softmax normalization of attention scores.
4.  `math.sqrt`: Used for scaling the query vector.

### Class Structure

```mark
class MultiHeadedAttention(nn.Module):
    def __init__(self, head_count, model_dim, dropout=0.1, use_final_linear=True):
        # Initialization
        ...
    def forward(self, key, value, query, mask=None, layer_cache=None, type=None, predefined_graph_1=None):
        # Self-Attention Mechanism for "self" type usage
        if type == "self":
            query = self.linear_query(query)
            key = shape(key)
            value = shape(value)

            ...
        
        # Context Vector Computation for both "self" and "context" types
        context_vector = torch.matmul(drop_attn, value)
        return context_vector
```


key: chroma:document
string_value:
CODE:

class DecoderState:
    """Interface for grouping together the current state of a recurrent
    decoder. In the simplest case just represents the hidden state of
    the model.  But can also be used for implementing various forms of
    input_feeding and non-recurrent models.

    Modules need to implement this to utilize beam search decoding.
    """

    def detach(self):
        """Need to document this"""
        self.hidden = tuple([_.detach() for _ in self.hidden])
        self.input_feed = self.input_feed.detach()

    def beam_update(self, idx, positions, beam_size):
        """Need to document this"""
        for e in self._all:
            sizes = e.size()
            br = sizes[1]
            if len(sizes) == 3:
                sent_states = e.view(sizes[0], beam_size, br // beam_size, sizes[2])[:, :, idx]
            else:
                sent_states = e.view(sizes[0], beam_size, br // beam_size, sizes[2], sizes[3])[:, :, idx]

            sent_states.data.copy_(sent_states.data.index_select(1, positions))

    def map_batch_fn(self, fn):


SUMMARY:
**DecoderState Class:**

This class represents the state of a recurrent decoder and serves as an interface for beam search decoding. Its primary functionality includes:

*   **Detaching**: Modifies the `hidden` and `input_feed` attributes to detach them from computational graphs, allowing for efficient reusability.
*   **Beam Update**: Updates the beam search results by selecting specific positions within each encoding sequence (`e`) using the provided `idx`, `positions`, and `beam_size`.

**Methods:**

*   `detach()`: Detaches the hidden state and input feed from computational graphs to enable reuse.
*   `beam_update(idx, positions, beam_size)`: Updates beam search results by selecting specific positions within encoding sequences.

**Notable Operations:**

*   The `_all` attribute is used as a reference point for updating encoding sequences.
*   The `size()` method returns the size of each encoding sequence element.
*   The `view()` method reshapes a tensor for efficient indexing.

**Related Classes/Functions:**

*   `fn`: A function that needs to be implemented by modules using this interface.

**External Dependencies:**

*   PyTorch library (specifically, its computation graph system)


key: chroma:document
string_value:
CODE:

class TransformerDecoderState(DecoderState):
    """Transformer Decoder state base class"""

    def __init__(self, src):
        """
        Args:
            src (FloatTensor): a sequence of source words tensors
                    with optional feature tensors, of size (len x batch).
        """
        self.src = src
        self.previous_input = None
        self.previous_layer_inputs = None
        self.cache = None

    @property
    def _all(self):
        """
        Contains attributes that need to be updated in self.beam_update().
        """
        if self.previous_input is not None and self.previous_layer_inputs is not None:
            return (self.previous_input, self.previous_layer_inputs, self.src)
        else:
            return (self.src,)

    def detach(self):
        if self.previous_input is not None:
            self.previous_input = self.previous_input.detach()
        if self.previous_layer_inputs is not None:
            self.previous_layer_inputs = self.previous_layer_inputs.detach()
        self.src = self.src.detach()

    def update_state(self, new_input, previous_layer_inputs):
        state = TransformerDecoderState(self.src)
        state.previous_input = new_input
        state.previous_layer_inputs = previous_layer_inputs
        return state

    def _init_cache(self, memory_bank, num_layers):
        self.cache = {}

        for l in range(num_layers):
            layer_cache = {"memory_keys": None, "memory_values": None}
            layer_cache["self_keys"] = None
            layer_cache["self_values"] = None
            self.cache["layer_{}".format(l)] = layer_cache

    def repeat_beam_size_times(self, beam_size):
        """Repeat beam_size times along batch dimension."""
        self.src = self.src.data.repeat(1, beam_size, 1)

    def map_batch_fn(self, fn):
        def _recursive_map(struct, batch_dim=0):
            for k, v in struct.items():
                if v is not None:
                    if isinstance(v, dict):
                        _recursive_map(v)
                    else:
                        struct[k] = fn(v, batch_dim)

        self.src = fn(self.src, 0)
        if self.cache is not None:


SUMMARY:
**TransformerDecoderState Class Summary**

### Purpose and Functionality

The `TransformerDecoderState` class serves as the base state class for a transformer decoder in sequence-to-sequence models. It manages internal state information, such as input embeddings, layer inputs, and memory banks, to facilitate efficient computation.

### Key Implementation Details

*   **Initialization**: Takes a `FloatTensor` source sequence (`src`) and initializes previous input and layer inputs.
*   **Update State**: Updates the current state with new input and previous layer inputs.
*   **Detach Gradients**: Detaches gradients from tensor attributes to free up GPU memory.
*   **Cache Initialization**: Initializes a cache dictionary for storing temporary values during computation.
*   **Repeat Beam Size Times**: Repeats the source sequence along the batch dimension, simulating multiple decoding paths.
*   **Map Batch Function**: Applies a generic batch function to the source sequence and recursively updates caches.

### Notable Dependencies and Requirements

*   `FloatTensor`: A tensor representing the input sequence with optional feature tensors.
*   `TransformerDecoderState`: The current state object instance.
*   `memory_bank` (not shown): A module for storing memory values in cache layer 0.
*   `num_layers` (not shown): An integer specifying the number of layers involved.

### External Dependencies

*   
    *None explicitly stated, but potentially inherits from `DecoderState` and depends on downstream frameworks or libraries.*


key: chroma:document
string_value:
CODE:

def gelu(x):


SUMMARY:
**Code Summary**

The `gelu` function implements the Gaussian Error Linear Unit activation function, commonly used in deep neural networks.

**Main Purpose and Functionality**
This is an implementation of the GELU (Gaussian Error Linear Unit) activation function from the PyTorch library. The function takes a single input `x` and computes the GELU output using the formula:

`0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3)))`

This formula maps all input values to a range between -1 and 1.

**Key Implementation Details**

* The function uses PyTorch library (`torch.tanh`) for calculating the hyperbolic tangent of `x`.
* It utilizes mathematical constants (`math.pi`), specifically for calculating the inverse of pi squared (`2 / math.pi`). 

**Notable Dependencies or Requirements**
This function requires:

* NumPy (`math`) for access to mathematical constants.
* PyTorch (torch) for its `tanh` activation function and tensor operations.

The summary preserves class/function/variable names exactly, highlighting the key algorithms, formula used, and notable dependencies required.


key: chroma:document
string_value:
CODE:

class PositionwiseFeedForward(nn.Module):
    """A two-layer Feed-Forward-Network with residual layer norm.

    Args:
        d_model (int): the size of input for the first-layer of the FFN.
        d_ff (int): the hidden layer size of the second-layer
            of the FNN.
        dropout (float): dropout probability in :math:`[0, 1)`.
    """

    def __init__(self, d_model, d_ff, dropout=0.1):
        super().__init__()
        self.w_1 = nn.Linear(d_model, d_ff)
        self.w_2 = nn.Linear(d_ff, d_model)
        self.layer_norm = nn.LayerNorm(d_model, eps=1e-6)
        self.actv = gelu
        self.dropout_1 = nn.Dropout(dropout)
        self.dropout_2 = nn.Dropout(dropout)

    def forward(self, x):
        inter = self.dropout_1(self.actv(self.w_1(self.layer_norm(x))))
        output = self.dropout_2(self.w_2(inter))


SUMMARY:
**Positionwise Feed Forward Network Summary**

### Main Purpose and Functionality
This is a two-layer feed-forward network (FFN) module, specifically designed to be lightweight and power-efficient, implemented using PyTorch's neural network library (`nn.Module`).

#### Primary Purpose:
This class implements a residual connection with layer normalization for the first and second layers of an FFN.

### Key Implementation Details

*   Two linear layers `w_1` and `w_2` are used in both forward passes.
*   The activation function is set to `gelu`.
*   Layer Normalization (`layer_norm`) is applied before each activation, ensuring normalization after residual additions.

### Notable Dependencies or Requirements
#### External dependencies:
- PyTorch

#### Interfaces Implemented
This class implements the following interfaces (as of my knowledge cutoff date):
1.  `nn.Module` interface for building neural network modules.
2.  `nn.Linear` and its variants (`w_1` and `w_2`) interface for linear transformations.

### Method Signatures:
*   `__init__`: Initializes the FPN with given dimensions.

Note that no specific additional method is detailed beyond the constructor, since this class acts primarily as a modular component within a larger architecture.


key: docstring
string_value:
#
# TRANSLATOR
# The following code is used to generate summaries using the
# pre-trained weights and beam search.
#


key: chroma:document
string_value:
CODE:

def build_predictor(args, tokenizer, symbols, model, logger=None):
    # we should be able to refactor the global scorer a lot
    scorer = GNMTGlobalScorer(args.alpha, length_penalty="wu")
    translator = Translator(args, model, tokenizer, symbols, global_scorer=scorer, logger=logger)


SUMMARY:
**Summary**

This function, `build_predictor`, constructs a summarization pipeline.

### Key Implementation Details

* Creates an instance of `GNMTGlobalScorer` with alpha value (`args.alpha`) and WU length penalty.
* Initializes a `Translator` object with:
	+ Model weights (`model`)
	+ Tokenizer (`tokenizer`)
	+ Symbols dictionary (`symbols`)
	+ Global scorer (`scorer`) set to the previously created instance
	+ Logger reference (`logger`, optional)

### Primary Purpose

This function serves as an entry point for generating summaries using a pre-trained model and beam search.

### Notable Dependencies/Requirements

* `args`: input arguments containing alpha value and other configuration options.
* `tokenizer`: used for tokenization and encoding input data.
* `symbols`: dictionary mapping symbols to their corresponding indices.
* `model`: pre-trained weights for summarization generation.
* External dependencies:
	+ `GNMTGlobalScorer` class from a specific library or framework.
	+ `Translator` class from the same source as the current file.


key: chroma:document
string_value:
CODE:

class GNMTGlobalScorer:
    """
    NMT re-ranking score from
    "Google's Neural Machine Translation System" :cite:`wu2016google`

    Args:
       alpha (float): length parameter
       beta (float):  coverage parameter
    """

    def __init__(self, alpha, length_penalty):
        self.alpha = alpha
        penalty_builder = PenaltyBuilder(length_penalty)
        self.length_penalty = penalty_builder.length_penalty()

    def score(self, beam, logprobs):
        """
        Rescores a prediction based on penalty functions
        """
        normalized_probs = self.length_penalty(beam, logprobs, self.alpha)


SUMMARY:
**GNMTGlobalScorer Class Summary**

* **Purpose**: NMT re-ranking score calculation from Google's Neural Machine Translation System (2016) using global scoring formula.
* **Functionality**:
	+ Initializes the scorer with α (length parameter) and β (coverage parameter).
	+ Calculates the re-ranking score for a given beam of predictions.
* **Key Implementation Details**:
	+ Uses `PenaltyBuilder` to calculate length penalty based on input `beam`, `logprobs`, and `α`.
	+ Normalizes probabilities using `alpha` as length penalty parameter.

**Method Signatures and Interfaces**:

* `__init__(self, alpha, length_penalty)`: Initializes the scorer with α and β.
* `score(self, beam, logprobs)`

**Notable Dependencies/Requirements**:
* `PenaltyBuilder` class for calculating length penalty.
* External dependencies not explicitly stated in this code snippet.

* **Class Properties**:
	+ `α (float): input probability parameter`
	+ `β (coverage parameter)`
	+ `length_penalty (float): calculated length penalty`
	+ `penalty_builder (PenaltyBuilder): instance of PenaltyBuilder`


key: chroma:document
string_value:
CODE:

class PenaltyBuilder:
    """
    Returns the Length and Coverage Penalty function for Beam Search.

    Args:
        length_pen (str): option name of length pen
        cov_pen (str): option name of cov pen
    """

    def __init__(self, length_pen):
        self.length_pen = length_pen

    def length_penalty(self):
        if self.length_pen == "wu":
            return self.length_wu
        elif self.length_pen == "avg":
            return self.length_average
        else:
            return self.length_none

    """
    Below are all the different penalty terms implemented so far
    """

    def length_wu(self, beam, logprobs, alpha=0.0):
        """
        NMT length re-ranking score from
        "Google's Neural Machine Translation System" :cite:`wu2016google`.
        """

        modifier = ((5 + len(beam.next_ys)) ** alpha) / ((5 + 1) ** alpha)
        return logprobs / modifier

    def length_average(self, beam, logprobs, alpha=0.0):
        """
        Returns the average probability of tokens in a sequence.
        """
        return logprobs / len(beam.next_ys)

    def length_none(self, beam, logprobs, alpha=0.0, beta=0.0):
        """
        Returns unmodified scores.
        """


SUMMARY:
**Class Summary: PenaltyBuilder**

The `PenaltyBuilder` class returns penalty functions for Beam Search in Natural Language Processing (NLP). It constructs length and coverage penalties based on user-specified options (`length_pen` and `cov_pen`).

**Key Implementation Details:**

* Three penalty functions are implemented:
	+ `length_wu`: NMT length re-ranking score using the Wu penalty from "Google's Neural Machine Translation System".
	+ `length_average`: Average probability of tokens in a sequence.
	+ `length_none`: Returns unmodified scores without any penalties.
* The `length_penalty` method determines which penalty function to return based on the chosen option (`length_pen`).
* External dependencies:
	+ `beam` object with attributes `next_ys`
	+ `logprobs` (log probability of a sequence)
	+ `alpha` and `beta` hyperparameters (optional)


key: chroma:document
string_value:
CODE:

class Translator:
    """
    Uses a model to translate a batch of sentences.

    Args:
       model (:obj:`onmt.modules.NMTModel`):
          NMT model to use for translation
       fields (dict of Fields): data fields
       beam_size (int): size of beam to use
       n_best (int): number of translations produced
       max_length (int): maximum length output to produce
       global_scores (:obj:`GlobalScorer`):
         object to rescore final translations
       copy_attn (bool): use copy attention during translation
       beam_trace (bool): trace beam search for debugging
       logger(logging.Logger): logger.
    """

    def __init__(self, args, model, vocab, symbols, global_scorer=None, logger=None):
        self.logger = logger

        self.args = args
        self.model = model
        self.generator = self.model.generator
        self.vocab = vocab
        self.symbols = symbols
        self.start_token = symbols["BOS"]
        self.end_token = symbols["EOS"]

        self.global_scorer = global_scorer
        self.beam_size = args.beam_size
        self.min_length = args.min_length
        self.max_length = args.max_length

    def translate(self, batch, step, attn_debug=False):
        """Generates summaries from one batch of data."""
        self.model.eval()
        with torch.no_grad():
            batch_data = self.translate_batch(batch)
            translations = self.from_batch(batch_data)
        return translations

    def translate_batch(self, batch, fast=False):
        """
        Translate a batch of sentences.

        Mostly a wrapper around :obj:`Beam`.

        Args:
           batch (:obj:`Batch`): a batch from a dataset object
           fast (bool): enables fast beam search (may not support all features)
        """
        with torch.no_grad():
            return self._fast_translate_batch(batch, self.max_length, min_length=self.min_length)

    # Where the beam search lives
    # I have no idea why it is being called from the method above
    def _fast_translate_batch(self, batch, max_length, min_length=0):
        """Beam Search using the encoder inputs contained in `batch`."""

        # The batch object is funny
        # Instead of just looking at the size of the arguments we encapsulate
        # a size argument.
        # Where is it defined?
        beam_size = self.beam_size
        batch_size = batch.batch_size
        src = batch.src
        segs = batch.segs
        mask_src = batch.mask_src

        src_features = self.model.bert(src, segs, mask_src)
        dec_states = self.model.decoder.init_decoder_state(src, src_features, with_cache=True)
        device = src_features.device

        # Tile states and memory beam_size times.
        dec_states.map_batch_fn(lambda state, dim: tile(state, beam_size, dim=dim))
        src_features = tile(src_features, beam_size, dim=0)
        batch_offset = torch.arange(batch_size, dtype=torch.long, device=device)
        beam_offset = torch.arange(0, batch_size * beam_size, step=beam_size, dtype=torch.long, device=device)
        alive_seq = torch.full([batch_size * beam_size, 1], self.start_token, dtype=torch.long, device=device)

        # Give full probability to the first beam on the first step.
        topk_log_probs = torch.tensor([0.0] + [float("-inf")] * (beam_size - 1), device=device).repeat(batch_size)

        # Structure that holds finished hypotheses.
        hypotheses = [[] for _ in range(batch_size)]  # noqa: F812

        results = {}
        results["predictions"] = [[] for _ in range(batch_size)]  # noqa: F812
        results["scores"] = [[] for _ in range(batch_size)]  # noqa: F812
        results["gold_score"] = [0] * batch_size
        results["batch"] = batch

        for step in range(max_length):
            decoder_input = alive_seq[:, -1].view(1, -1)

            # Decoder forward.
            decoder_input = decoder_input.transpose(0, 1)

            dec_out, dec_states = self.model.decoder(decoder_input, src_features, dec_states, step=step)

            # Generator forward.
            log_probs = self.generator(dec_out.transpose(0, 1).squeeze(0))
            vocab_size = log_probs.size(-1)

            if step < min_length:
                log_probs[:, self.end_token] = -1e20

            # Multiply probs by the beam probability.
            log_probs += topk_log_probs.view(-1).unsqueeze(1)

            alpha = self.global_scorer.alpha
            length_penalty = ((5.0 + (step + 1)) / 6.0) ** alpha

            # Flatten probs into a list of possibilities.
            curr_scores = log_probs / length_penalty

            if self.args.block_trigram:
                cur_len = alive_seq.size(1)
                if cur_len > 3:
                    for i in range(alive_seq.size(0)):
                        fail = False
                        words = [int(w) for w in alive_seq[i]]
                        words = [self.vocab.ids_to_tokens[w] for w in words]
                        words = " ".join(words).replace(" ##", "").split()
                        if len(words) <= 3:
                            continue
                        trigrams = [(words[i - 1], words[i], words[i + 1]) for i in range(1, len(words) - 1)]
                        trigram = tuple(trigrams[-1])
                        if trigram in trigrams[:-1]:
                            fail = True
                        if fail:
                            curr_scores[i] = -10e20

            curr_scores = curr_scores.reshape(-1, beam_size * vocab_size)
            topk_scores, topk_ids = curr_scores.topk(beam_size, dim=-1)

            # Recover log probs.
            topk_log_probs = topk_scores * length_penalty

            # Resolve beam origin and true word ids.
            topk_beam_index = topk_ids.div(vocab_size)
            topk_ids = topk_ids.fmod(vocab_size)

            # Map beam_index to batch_index in the flat representation.
            batch_index = topk_beam_index + beam_offset[: topk_beam_index.size(0)].unsqueeze(1)
            select_indices = batch_index.view(-1)

            # Append last prediction.
            alive_seq = torch.cat([alive_seq.index_select(0, select_indices), topk_ids.view(-1, 1)], -1)

            is_finished = topk_ids.eq(self.end_token)
            if step + 1 == max_length:
                is_finished.fill_(1)
            # End condition is top beam is finished.
            end_condition = is_finished[:, 0].eq(1)
            # Save finished hypotheses.
            if is_finished.any():
                predictions = alive_seq.view(-1, beam_size, alive_seq.size(-1))
                for i in range(is_finished.size(0)):
                    b = batch_offset[i]
                    if end_condition[i]:
                        is_finished[i].fill_(1)
                    finished_hyp = is_finished[i].nonzero().view(-1)
                    # Store finished hypotheses for this batch.
                    for j in finished_hyp:
                        hypotheses[b].append((topk_scores[i, j], predictions[i, j, 1:]))
                    # If the batch reached the end, save the n_best hypotheses.
                    if end_condition[i]:
                        best_hyp = sorted(hypotheses[b], key=lambda x: x[0], reverse=True)
                        score, pred = best_hyp[0]

                        results["scores"][b].append(score)
                        results["predictions"][b].append(pred)
                non_finished = end_condition.eq(0).nonzero().view(-1)
                # If all sentences are translated, no need to go further.
                if len(non_finished) == 0:
                    break
                # Remove finished batches for the next step.
                topk_log_probs = topk_log_probs.index_select(0, non_finished)
                batch_index = batch_index.index_select(0, non_finished)
                batch_offset = batch_offset.index_select(0, non_finished)
                alive_seq = predictions.index_select(0, non_finished).view(-1, alive_seq.size(-1))
            # Reorder states.
            select_indices = batch_index.view(-1)
            src_features = src_features.index_select(0, select_indices)
            dec_states.map_batch_fn(lambda state, dim: state.index_select(dim, select_indices))

        return results

    def from_batch(self, translation_batch):
        batch = translation_batch["batch"]
        assert len(translation_batch["gold_score"]) == len(translation_batch["predictions"])
        batch_size = batch.batch_size

        preds, _, _, tgt_str, src = (
            translation_batch["predictions"],
            translation_batch["scores"],
            translation_batch["gold_score"],
            batch.tgt_str,
            batch.src,
        )

        translations = []
        for b in range(batch_size):
            pred_sents = self.vocab.convert_ids_to_tokens([int(n) for n in preds[b][0]])
            pred_sents = " ".join(pred_sents).replace(" ##", "")
            gold_sent = " ".join(tgt_str[b].split())
            raw_src = [self.vocab.ids_to_tokens[int(t)] for t in src[b]][:500]
            raw_src = " ".join(raw_src)
            translation = (pred_sents, gold_sent, raw_src)
            translations.append(translation)



SUMMARY:
**Translation Pipeline Summary**

This is an implementation of a statistical machine translation pipeline in Python. The pipeline consists of the following stages:

1. **Data Preprocessing**: The input data is preprocessed by extracting specific information from each batch, such as gold scores and predicted translations.

2. **Beam Search**: A beam search algorithm is used to generate multiple possible translations for each sentence. The algorithm iteratively expands a set of candidate translations based on their log likelihood scores, until a termination condition is reached (e.g., maximum length or end of translation).

3. **N-Gram Check**: An N-gram check is performed to prevent invalid prefixes from being generated.

4. **Postprocessing**: The batch-level results are postprocessed by computing the scores and predictions for each translation in the batch.

5. **Generation**: The final result is a list of tuples containing the predicted translations, gold standard translations, and raw source strings.

**Key Features**

* Uses Beam Search algorithm to generate multiple possible translations
* Supports N-Gram check to prevent invalid prefixes
* Applies postprocessing on batch-level results
* Returns a list of translations with associated data (gold scores, predictions, etc.)

**Potential Improvements**

* Introduce more options for customizing the beam search and N-gram checks
* Implement additional techniques for improving score calculation or decoding (e.g., using advanced scoring functions)
* Consider optimizing performance through iterative refinement or parallel processing


key: chroma:document
string_value:
CODE:

def tile(x, count, dim=0):
    """
    Tiles x on dimension dim count times.
    """
    perm = list(range(len(x.size())))
    if dim != 0:
        perm[0], perm[dim] = perm[dim], perm[0]
        x = x.permute(perm).contiguous()
    out_size = list(x.size())
    out_size[0] *= count
    batch = x.size(0)
    x = x.view(batch, -1).transpose(0, 1).repeat(count, 1).transpose(0, 1).contiguous().view(*out_size)
    if dim != 0:
        x = x.permute(perm).contiguous()


SUMMARY:
**Function Summary: tile()**

The `tile()` function tiles the input tensor `x` along dimension `dim` by repeating it `count` times. The primary purpose is to resize an input tensor for batch augmentation.

**Key Implementation Details:**

* Uses PyTorch's Permutation and View operations.
* Repeats the tensor along the specified dimension using views and transposes.
* Optionally swaps dimensions if not set to 0 (default).

**Notable Dependencies/Requirements:**

* PyTorch library
* Tensor class (x) with size() method
* NumPy (implicit, for range generation)
* Permutation (perm), View (view), Transpose (transpose) functions.


key: chroma:document
string_value:
CODE:

class BertSumOptimizer:
    """Specific optimizer for BertSum.

    As described in [1], the authors fine-tune BertSum for abstractive
    summarization using two Adam Optimizers with different warm-up steps and
    learning rate. They also use a custom learning rate scheduler.

    [1] Liu, Yang, and Mirella Lapata. "Text summarization with pretrained encoders."
        arXiv preprint arXiv:1908.08345 (2019).
    """

    def __init__(self, model, lr, warmup_steps, beta_1=0.99, beta_2=0.999, eps=1e-8):
        self.encoder = model.encoder
        self.decoder = model.decoder
        self.lr = lr
        self.warmup_steps = warmup_steps

        self.optimizers = {
            "encoder": torch.optim.Adam(
                model.encoder.parameters(),
                lr=lr["encoder"],
                betas=(beta_1, beta_2),
                eps=eps,
            ),
            "decoder": torch.optim.Adam(
                model.decoder.parameters(),
                lr=lr["decoder"],
                betas=(beta_1, beta_2),
                eps=eps,
            ),
        }

        self._step = 0
        self.current_learning_rates = {}

    def _update_rate(self, stack):
        return self.lr[stack] * min(self._step ** (-0.5), self._step * self.warmup_steps[stack] ** (-1.5))

    def zero_grad(self):
        self.optimizer_decoder.zero_grad()
        self.optimizer_encoder.zero_grad()

    def step(self):
        self._step += 1
        for stack, optimizer in self.optimizers.items():
            new_rate = self._update_rate(stack)
            for param_group in optimizer.param_groups:
                param_group["lr"] = new_rate
            optimizer.step()


SUMMARY:
**BertSumOptimizer Class Summary**

The `BertSumOptimizer` class is designed for fine-tuning the BERTSum model for abstractive summarization. Its primary purpose is to optimize two Adam optimizers with different warm-up steps and learning rates.

**Main Functionality:**

- Initializes two Adam optimizers for encoder and decoder
  ```python
optimizers = {
    "encoder": torch.optim.Adam(model.encoder.parameters(), lr=lr["encoder"], betas=(beta_1, beta_2), eps=eps),
    "decoder": torch.optim.Adam(model.decoder.parameters(), lr=lr["decoder"], betas=(beta_1, beta_2), eps=eps)
}
```
- Implements a custom learning rate scheduler `_update_rate` based on the current step and warm-up steps
  ```python
def _update_rate(self, stack):
    return self.lr[stack] * min(self._step ** (-0.5), self._step * self.warmup_steps[stack] ** (-1.5))
```
- Offers methods to update gradients (`zero_grad`) and advance the optimization step (`step`)
  ```python
def zero_grad(self):
    self.optimizer_decoder.zero_grad()
    self.optimizer_encoder.zero_grad()

def step(self):
    self._step += 1
    for stack, optimizer in self.optimizers.items():
        new_rate = self._update_rate(stack)
        for param_group in optimizer.param_groups:
            param_group["lr"] = new_rate
        optimizer.step()
        self.current_learning_rates[stack] = new_rate
```

**Notable Dependencies/Requirements:**

- PyTorch library for tensor operations (`torch.optim.Adam`, `model.encoder` and `model.decoder`)
- Custom learning rate scheduler implementation

**Key Implementation Details:**

- Adam optimizers with different warm-up steps for encoder and decoder
- Custom learning rate scheduler based on the current step and warm-up steps


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/bertabs/configuration_bertabs.py


key: chroma:document
string_value:
CODE:

class BertAbsConfig(PretrainedConfig):
    r"""Class to store the configuration of the BertAbs model.

    Arguments:
        vocab_size: int
            Number of tokens in the vocabulary.
        max_pos: int
            The maximum sequence length that this model will be used with.
        enc_layer: int
            The numner of hidden layers in the Transformer encoder.
        enc_hidden_size: int
            The size of the encoder's layers.
        enc_heads: int
            The number of attention heads for each attention layer in the encoder.
        enc_ff_size: int
            The size of the encoder's feed-forward layers.
        enc_dropout: int
            The dropout probability for all fully connected layers in the
            embeddings, layers, pooler and also the attention probabilities in
            the encoder.
        dec_layer: int
            The numner of hidden layers in the decoder.
        dec_hidden_size: int
            The size of the decoder's layers.
        dec_heads: int
            The number of attention heads for each attention layer in the decoder.
        dec_ff_size: int
            The size of the decoder's feed-forward layers.
        dec_dropout: int
            The dropout probability for all fully connected layers in the
            embeddings, layers, pooler and also the attention probabilities in
            the decoder.
    """

    model_type = "bertabs"

    def __init__(
        self,
        vocab_size=30522,
        max_pos=512,
        enc_layers=6,
        enc_hidden_size=512,
        enc_heads=8,
        enc_ff_size=512,
        enc_dropout=0.2,
        dec_layers=6,
        dec_hidden_size=768,
        dec_heads=8,
        dec_ff_size=2048,
        dec_dropout=0.2,
        **kwargs,
    ):
        super().__init__(**kwargs)

        self.vocab_size = vocab_size
        self.max_pos = max_pos

        self.enc_layers = enc_layers
        self.enc_hidden_size = enc_hidden_size
        self.enc_heads = enc_heads
        self.enc_ff_size = enc_ff_size
        self.enc_dropout = enc_dropout

        self.dec_layers = dec_layers
        self.dec_hidden_size = dec_hidden_size
        self.dec_heads = dec_heads
        self.dec_ff_size = dec_ff_size


SUMMARY:
**BertAbsConfig Class Summary**

The `BertAbsConfig` class stores the configuration for the BertAbs model. It extends the `PretrainedConfig` class.

**Primary Purpose:**
Stores model architecture parameters, including number of layers, hidden size, attention heads, and dropout probabilities for both encoder and decoder components.

**Key Implementation Details:**

* Model type is set to "bertabs"
* Parameters include:
	+ Encoder settings: `enc_layers`, `enc Hidden Size`, `enc_heads`, `enc_ff_size`, `enc_dropout`
	+ Decoder settings: `dec_layers`, `dec_hidden_size`, `dec_heads`, `dec_ff_size`, `dec_dropout`

**Notable Dependencies or Requirements:**

* Inherited from `PretrainedConfig`
* No external dependencies listed

**Model Type:** `bertabs`


key: file_path
string_value:
/home/gaby/projects/2025/transformers-4.48.1/examples/research_projects/bertabs/run_summarization.py


key: chroma:document
string_value:
CODE:

def evaluate(args):
    tokenizer = BertTokenizer.from_pretrained("google-bert/bert-base-uncased", do_lower_case=True)
    model = BertAbs.from_pretrained("remi/bertabs-finetuned-extractive-abstractive-summarization")
    model.to(args.device)
    model.eval()

    symbols = {
        "BOS": tokenizer.vocab["[unused0]"],
        "EOS": tokenizer.vocab["[unused1]"],
        "PAD": tokenizer.vocab["[PAD]"],
    }

    if args.compute_rouge:
        reference_summaries = []
        generated_summaries = []

        import nltk
        import rouge

        nltk.download("punkt")
        rouge_evaluator = rouge.Rouge(
            metrics=["rouge-n", "rouge-l"],
            max_n=2,
            limit_length=True,
            length_limit=args.beam_size,
            length_limit_type="words",
            apply_avg=True,
            apply_best=False,
            alpha=0.5,  # Default F1_score
            weight_factor=1.2,
            stemming=True,
        )

    # these (unused) arguments are defined to keep the compatibility
    # with the legacy code and will be deleted in a next iteration.
    args.result_path = ""
    args.temp_dir = ""

    data_iterator = build_data_iterator(args, tokenizer)
    predictor = build_predictor(args, tokenizer, symbols, model)

    logger.info("***** Running evaluation *****")
    logger.info("  Number examples = %d", len(data_iterator.dataset))
    logger.info("  Batch size = %d", args.batch_size)
    logger.info("")
    logger.info("***** Beam Search parameters *****")
    logger.info("  Beam size = %d", args.beam_size)
    logger.info("  Minimum length = %d", args.min_length)
    logger.info("  Maximum length = %d", args.max_length)
    logger.info("  Alpha (length penalty) = %.2f", args.alpha)
    logger.info("  Trigrams %s be blocked", ("will" if args.block_trigram else "will NOT"))

    for batch in tqdm(data_iterator):
        batch_data = predictor.translate_batch(batch)
        translations = predictor.from_batch(batch_data)
        summaries = [format_summary(t) for t in translations]
        save_summaries(summaries, args.summaries_output_dir, batch.document_names)

        if args.compute_rouge:
            reference_summaries += batch.tgt_str
            generated_summaries += summaries

    if args.compute_rouge:
        scores = rouge_evaluator.get_scores(generated_summaries, reference_summaries)
        str_scores = format_rouge_scores(scores)
        save_rouge_scores(str_scores)


SUMMARY:
**Concise Summary**

Evaluate a summarization model using the transformer-based `BertAbs` architecture. The function takes in `args` containing hyperparameters and configuration.

### Main Functionality

1.  **Initialize Model**: Load pre-trained `BertTokenizer` and fine-tuned `BertAbs` model from the given paths.
2.  **Set Device**: Move model to specified device for evaluation mode.
3.  **Prepare Symbols**: Define tokens for BOS, EOS, and PAD using tokenizer's vocabulary.
4.  **Evaluate ROUGE Scores (optional)**: If enabled, use `Rouge` evaluator to calculate scores between generated and reference summaries.

### Key Implementation Details

*   `build_data_iterator` & `build_predictor`: utility functions to prepare data iterator and predictor.
*   `predictor.translate_batch` & `predictor.from_batch`: methods to translate batch data for generation and fetching output.
*   `save_summary` and `save_rouge_scores`: helper functions to save generated summaries and ROUGE scores.

### Notable Dependencies or Requirements

*   `BertTokenizer` from Hugging Face
*   `Rouge` evaluator for ROUGE score calculation (from NLTK)
*   `torch` (not explicitly mentioned, but assumed for tensor-related operations)


