class_docstr: The PPOTrainer uses Proximal Policy Optimization to optimise language models.
  Note, this trainer is heavily inspired by the original OpenAI learning to summarize
  work here: https://github.com/openai/summarize-from-feedback
  
  Attributes:
      **cfg** (`LLM Studio Config`) -- Experiment configuration object. Check the
          documentation of `LLM Studio Config` for more details.
      **model** (`PreTrainedModelWrapper`) -- Model to be optimized, Hugging Face
          transformer model with a value head. Check the documentation of
          `PreTrainedModelWrapper` for more details.
      **tokenizer** (`Union[PreTrainedTokenizer, PreTrainedTokenizerFast]`)
          Tokenizer to be used for encoding the data. Check the documentation of
          `transformers.PreTrainedTokenizer` and
          `transformers.PreTrainedTokenizerFast` for more details.
      **optimizer** (`torch.optim.Optimizer`) -- Optimizer to be used for training.
      **lr_scheduler** (`torch.optim.lr_scheduler`) -- Learning rate scheduler to be
          used for training.
      **scaler** (`torch.cuda.amp.GradScaler`) -- Gradient scaler to be used for
          training.
class_docstr_tok: ["The","PPOTrainer","uses","Proximal","Policy","Optimization","to","optimise","language","models",".","Note",",","this","trainer","is","heavily","inspired","by","the","original","OpenAI","learning","to","summarize","work","here",":","https",":","\/\/github.com\/openai\/summarize-from-feedback","Attributes",":","*","*","cfg","*","*","(","`","LLM","Studio","Config","`",")","--","Experiment","configuration","object",".","Check","the","documentation","of","`","LLM","Studio","Config","`","for","more","details",".","*","*","model","*","*","(","`","PreTrainedModelWrapper","`",")","--","Model","to","be","optimized",",","Hugging","Face","transformer","model","with","a","value","head",".","Check","the","documentation","of","`","PreTrainedModelWrapper","`","for","more","details",".","*","*","tokenizer","*","*","(","`","Union","[","PreTrainedTokenizer",",","PreTrainedTokenizerFast","]","`",")","Tokenizer","to","be","used","for","encoding","the","data",".","Check","the","documentation","of","`","transformers.PreTrainedTokenizer","`","and","`","transformers.PreTrainedTokenizerFast","`","for","more","details",".","*","*","optimizer","*","*","(","`","torch.optim.Optimizer","`",")","--","Optimizer","to","be","used","for","training",".","*","*","lr_scheduler","*","*","(","`","torch.optim.lr_scheduler","`",")","--","Learning","rate","scheduler","to","be","used","for","training",".","*","*","scaler","*","*","(","`","torch.cuda.amp.GradScaler","`",")","--","Gradient","scaler","to","be","used","for","training","."]
docstr: Compute per token rewards from scores and KL-penalty.
  
  Args:
      scores (`torch.FloatTensor`):
          Scores from the reward model, shape (`batch_size`)
      logprobs (`torch.FloatTensor`):
          Log probabilities of the model, shape (`batch_size`, `response_length`)
      ref_logprobs (`torch.FloatTensor`):
          Log probabilities of the reference model, shape (`batch_size`,
          `response_length`)
code: def compute_rewards(
      self,
      scores: torch.FloatTensor,
      logprobs: torch.FloatTensor,
      ref_logprobs: torch.FloatTensor,
      masks: torch.LongTensor,
  ):
      """
      Compute per token rewards from scores and KL-penalty.
  
      Args:
          scores (`torch.FloatTensor`):
              Scores from the reward model, shape (`batch_size`)
          logprobs (`torch.FloatTensor`):
              Log probabilities of the model, shape (`batch_size`, `response_length`)
          ref_logprobs (`torch.FloatTensor`):
              Log probabilities of the reference model, shape (`batch_size`,
              `response_length`)
      """
      rewards, non_score_rewards = [], []
      for score, logprob, ref_logprob, mask in zip(
          scores, logprobs, ref_logprobs, masks
      ):
          # compute KL penalty (from difference in logprobs)
          kl = self._kl_penalty(logprob, ref_logprob)
          non_score_reward = -self.kl_ctl.value * kl
          non_score_rewards.append(non_score_reward)
          reward = non_score_reward.clone()
          last_non_masked_index = mask.nonzero()[-1]
  
          # reward is preference model score + KL penalty
          reward[last_non_masked_index] += score
          rewards.append(reward)
      return torch.stack(rewards), torch.stack(non_score_rewards)
  
imports: import math
  import time
  import warnings
  from collections.abc import Mapping
  from typing import Any, Dict, List, Optional, Union
  import numpy
  import torch
  import torch.nn.functional
  from datasets import Dataset
  from huggingface_hub import PyTorchModelHubMixin
  from torch.cuda.amp import autocast
  from torch.nn.utils.rnn import pad_sequence
  from torch.utils.data import DataLoader
  from transformers import DataCollatorForLanguageModeling, PreTrainedTokenizer, PreTrainedTokenizerFast
  from llm_studio.src.utils.modeling_utils import unwrap_model
label_desc: Use image node_id 6 for calling the PPOTrainer obj's underlying member method code with example usage: obj.compute_rewards(scores, logprobs, ref_logprobs, masks) and returns: torch,  torch