class_docstr: Holds a TokenizationDataset for use in a torch DataLoader
  
  The torch DataLoader is different from the DataLoader defined here
  and allows for cpu & gpu parallelism.  Updating output_predictions
  to use this class as a wrapper to a TokenizationDataset means the
  calculation of features can happen in parallel, saving quite a
  bit of time.
class_docstr_tok: ["Holds","a","TokenizationDataset","for","use","in","a","torch","DataLoader","The","torch","DataLoader","is","different","from","the","DataLoader","defined","here","and","allows","for","cpu","&","gpu","parallelism",".","Updating","output_predictions","to","use","this","class","as","a","wrapper","to","a","TokenizationDataset","means","the","calculation","of","features","can","happen","in","parallel",",","saving","quite","a","bit","of","time","."]
code: def collate(self, samples):
      if any(len(x) > 1 for x in samples):
          raise ValueError(
              "Expected all paragraphs to have no preset sentence splits!"
          )
      feat_size = samples[0][0][2].shape[-1]
      padid = self.dataset.vocab.unit2id("<PAD>")
  
      # +1 so that all samples end with at least one pad
      pad_len = max(len(x[0][3]) for x in samples) + 1
  
      units = torch.full(
          (len(samples), pad_len), padid, dtype=torch.int64
      )
      labels = torch.full(
          (len(samples), pad_len), -1, dtype=torch.int32
      )
      features = torch.zeros(
          (len(samples), pad_len, feat_size), dtype=torch.float32
      )
      raw_units = []
      for i, sample in enumerate(samples):
          u_, l_, f_, r_ = sample[0]
          units[i, : len(u_)] = torch.from_numpy(u_)
          labels[i, : len(l_)] = torch.from_numpy(l_)
          features[i, : len(f_), :] = torch.from_numpy(f_)
          raw_units.append(r_ + ["<PAD>"] * (pad_len - len(r_)))
  
      return units, labels, features, raw_units
  
imports: from bisect import bisect_right
  from copy import copy
  import numpy
  import random
  import logging
  import re
  import torch
  from torch.utils.data import Dataset
  from .vocab import Vocab
  from stanza.models.common.utils import sort_with_indices, unsort
label_desc: Use image node_id 5 for calling the SortedDataset obj's underlying member method code with example usage: obj.collate(samples) and returns: units,  labels,  features,  raw_units