class_docstr: Callback to compute metrics at the end of every epoch. Unlike normal Keras metrics, these do not need to be
  compilable by TF. It is particularly useful for common NLP metrics like BLEU and ROUGE that require string
  operations or generation loops that cannot be compiled. Predictions (or generations) will be computed on the
  `eval_dataset` before being passed to the `metric_fn` in `np.ndarray` format. The `metric_fn` should compute
  metrics and return a dict mapping metric names to metric values.
  
  We provide an example of a suitable metric_fn that computes ROUGE scores for a summarization model below. Note that
  this example skips some post-processing for readability and simplicity, and should probably not be used as-is!
  
  ```py
  from datasets import load_metric
  
  rouge_metric = load_metric("rouge")
  
  
  def rouge_fn(predictions, labels):
      decoded_predictions = tokenizer.batch_decode(predictions, skip_special_tokens=True)
      decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
      result = rouge_metric.compute(predictions=decoded_predictions, references=decoded_labels)
      return {key: value.mid.fmeasure * 100 for key, value in result.items()}
  ```
  
  The above function will return a dict containing values which will be logged like any other Keras metric:
  
  ```
  {'rouge1': 37.4199, 'rouge2': 13.9768, 'rougeL': 34.361, 'rougeLsum': 35.0781
  ```
  
  Args:
      metric_fn (`Callable`):
          Metric function provided by the user. It will be called with two arguments - `predictions` and `labels`.
          These contain the model's outputs and matching labels from the dataset. It should return a dict mapping
          metric names to numerical values.
      eval_dataset (`tf.data.Dataset` or `dict` or `tuple` or `np.ndarray` or `tf.Tensor`):
          Validation data to be used to generate predictions for the `metric_fn`.
      output_cols (`List[str], *optional*):
          A list of columns to be retained from the model output as the predictions. Defaults to all.
      label_cols ('`List[str]`, *optional*'):
          A list of columns to be retained from the input dataset as the labels. Will be autodetected if this is not
          supplied.
      batch_size (`int`, *optional*):
          Batch size. Only used when the data is not a pre-batched `tf.data.Dataset`.
      predict_with_generate (`bool`, *optional*, defaults to `False`):
          Whether we should use `model.generate()` to get outputs for the model.
      use_xla_generation (`bool`, *optional*, defaults to `False`):
          If we're generating, whether to compile model generation with XLA. This can massively increase the speed of
          generation (up to 100X speedup) but will require a new XLA compilation for each input shape. When using XLA
          generation, it's a good idea to pad your inputs to the same size, or to use the `pad_to_multiple_of`
          argument in your `tokenizer` or `DataCollator`, which will reduce the number of unique input shapes and
          save a lot of compilation time. This option has no effect is `predict_with_generate` is `False`.
      generate_kwargs (`dict`, *optional*):
          Keyword arguments to pass to `model.generate()` when generating. Has no effect if `predict_with_generate`
          is `False`.
class_docstr_tok: ["Callback","to","compute","metrics","at","the","end","of","every","epoch",".","Unlike","normal","Keras","metrics",",","these","do","not","need","to","be","compilable","by","TF",".","It","is","particularly","useful","for","common","NLP","metrics","like","BLEU","and","ROUGE","that","require","string","operations","or","generation","loops","that","can","not","be","compiled",".","Predictions","(","or","generations",")","will","be","computed","on","the","`","eval_dataset","`","before","being","passed","to","the","`","metric_fn","`","in","`","np.ndarray","`","format",".","The","`","metric_fn","`","should","compute","metrics","and","return","a","dict","mapping","metric","names","to","metric","values",".","We","provide","an","example","of","a","suitable","metric_fn","that","computes","ROUGE","scores","for","a","summarization","model","below",".","Note","that","this","example","skips","some","post-processing","for","readability","and","simplicity",",","and","should","probably","not","be","used","as-is","!","``","`","py","from","datasets","import","load_metric","rouge_metric","=","load_metric","(","``","rouge","''",")","def","rouge_fn","(","predictions",",","labels",")",":","decoded_predictions","=","tokenizer.batch_decode","(","predictions",",","skip_special_tokens=True",")","decoded_labels","=","tokenizer.batch_decode","(","labels",",","skip_special_tokens=True",")","result","=","rouge_metric.compute","(","predictions=decoded_predictions",",","references=decoded_labels",")","return","{","key",":","value.mid.fmeasure","*","100","for","key",",","value","in","result.items","(",")","}","``","`","The","above","function","will","return","a","dict","containing","values","which","will","be","logged","like","any","other","Keras","metric",":","``","`","{","'rouge1","'",":","37.4199",",","'rouge2","'",":","13.9768",",","'rougeL","'",":","34.361",",","'rougeLsum","'",":","35.0781","``","`","Args",":","metric_fn","(","`","Callable","`",")",":","Metric","function","provided","by","the","user",".","It","will","be","called","with","two","arguments","-","`","predictions","`","and","`","labels","`",".","These","contain","the","model","'s","outputs","and","matching","labels","from","the","dataset",".","It","should","return","a","dict","mapping","metric","names","to","numerical","values",".","eval_dataset","(","`","tf.data.Dataset","`","or","`","dict","`","or","`","tuple","`","or","`","np.ndarray","`","or","`","tf.Tensor","`",")",":","Validation","data","to","be","used","to","generate","predictions","for","the","`","metric_fn","`",".","output_cols","(","`","List","[","str","]",",","*","optional","*",")",":","A","list","of","columns","to","be","retained","from","the","model","output","as","the","predictions",".","Defaults","to","all",".","label_cols","(","'","`","List","[","str","]","`",",","*","optional","*","'",")",":","A","list","of","columns","to","be","retained","from","the","input","dataset","as","the","labels",".","Will","be","autodetected","if","this","is","not","supplied",".","batch_size","(","`","int","`",",","*","optional","*",")",":","Batch","size",".","Only","used","when","the","data","is","not","a","pre-batched","`","tf.data.Dataset","`",".","predict_with_generate","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","Whether","we","should","use","`","model.generate","(",")","`","to","get","outputs","for","the","model",".","use_xla_generation","(","`","bool","`",",","*","optional","*",",","defaults","to","`","False","`",")",":","If","we","'re","generating",",","whether","to","compile","model","generation","with","XLA",".","This","can","massively","increase","the","speed","of","generation","(","up","to","100X","speedup",")","but","will","require","a","new","XLA","compilation","for","each","input","shape",".","When","using","XLA","generation",",","it","'s","a","good","idea","to","pad","your","inputs","to","the","same","size",",","or","to","use","the","`","pad_to_multiple_of","`","argument","in","your","`","tokenizer","`","or","`","DataCollator","`",",","which","will","reduce","the","number","of","unique","input","shapes","and","save","a","lot","of","compilation","time",".","This","option","has","no","effect","is","`","predict_with_generate","`","is","`","False","`",".","generate_kwargs","(","`","dict","`",",","*","optional","*",")",":","Keyword","arguments","to","pass","to","`","model.generate","(",")","`","when","generating",".","Has","no","effect","if","`","predict_with_generate","`","is","`","False","`","."]
code: def _concatenate_batches(batches, padding_index=-100):
      # If all batches are unidimensional or same length, do a simple concatenation
      if batches[0].ndim == 1 or all(
          [batch.shape[1] == batches[0].shape[1] for batch in batches]
      ):
          return np.concatenate(batches, axis=0)
  
      # Welp, they're not the same length. Let's do some padding
      max_len = max([batch.shape[1] for batch in batches])
      num_samples = sum([batch.shape[0] for batch in batches])
      output = np.full_like(
          batches[0],
          fill_value=padding_index,
          shape=[num_samples, max_len] + list(batches[0].shape[2:]),
      )
      # i keeps track of which part of the concatenated array we're writing the next batch to
      i = 0
      for batch in batches:
          output[i : i + len(batch), : batch.shape[1]] = batch
          i += len(batch)
      return output
  
imports: import logging
  import os
  from pathlib import Path
  from time import sleep
  from typing import Callable, List, Optional, Union
  import numpy
  import tensorflow
  from packaging.version import parse
  from tensorflow.keras.callbacks import Callback
  from huggingface_hub import Repository
  from .None import IntervalStrategy, PreTrainedTokenizerBase
  from .modelcard import TrainingSummary
  from .utils import get_full_repo_name
label_desc: Use image node_id 2 for calling the KerasMetricCallback obj's underlying member method code with example usage: obj._concatenate_batches(batches, padding_index) and returns: output, np