class_docstr: Parent class for entity linking encoder training and index
  datasets
  
  Args:
      tokenizer (obj): huggingface tokenizer,
      data_file (str): path to tab separated column file where data 
          pairs apear in the format 
          concept_ID  concept_synonym1        concept_synonym2
  
      newline_idx_file (str): path to pickle file containing location
          of data_file newline characters
      max_seq_length (int): maximum length of a concept in tokens
      is_index_data (bool): Whether dataset will be used for building
                          a nearest neighbors index
class_docstr_tok: ["Parent","class","for","entity","linking","encoder","training","and","index","datasets","Args",":","tokenizer","(","obj",")",":","huggingface","tokenizer",",","data_file","(","str",")",":","path","to","tab","separated","column","file","where","data","pairs","apear","in","the","format","concept_ID","concept_synonym1","concept_synonym2","newline_idx_file","(","str",")",":","path","to","pickle","file","containing","location","of","data_file","newline","characters","max_seq_length","(","int",")",":","maximum","length","of","a","concept","in","tokens","is_index_data","(","bool",")",":","Whether","dataset","will","be","used","for","building","a","nearest","neighbors","index"]
docstr: collate batch of input_ids, segment_ids, input_mask, and label
  
  Args:
      batch:  A list of tuples of format (concept_ID, concept_synonym1, concept_synonym2).
docstr_tok: ["collate","batch","of","input_ids",",","segment_ids",",","input_mask",",","and","label","Args",":","batch",":","A","list","of","tuples","of","format","(","concept_ID",",","concept_synonym1",",","concept_synonym2",")","."]
code: def _collate_fn(self, batch):
      """collate batch of input_ids, segment_ids, input_mask, and label
  
      Args:
          batch:  A list of tuples of format (concept_ID, concept_synonym1, concept_synonym2).
      """
      if self.is_index_data:
          concept_ids, concepts = zip(*batch)
          concept_ids = list(concept_ids)
          concepts = list(concepts)
  
      else:
          concept_ids, concepts1, concepts2 = zip(*batch)
          concept_ids = list(concept_ids)
          concept_ids.extend(
              concept_ids
          )  # Need to double label list to match each concept
          concepts = list(concepts1)
          concepts.extend(concepts2)
  
      batch = self.tokenizer(
          concepts,
          add_special_tokens=True,
          padding=True,
          truncation=True,
          max_length=self.max_seq_length,
          return_token_type_ids=True,
          return_attention_mask=True,
          return_length=True,
      )
  
      return (
          torch.LongTensor(batch["input_ids"]),
          torch.LongTensor(batch["token_type_ids"]),
          torch.LongTensor(batch["attention_mask"]),
          torch.LongTensor(concept_ids),
      )
  
file_path: NeMo/nemo/collections/nlp/data/entity_linking/entity_linking_dataset.py
imports: import array
  import pickle
  from typing import Optional
  import torch
  from nemo.collections.nlp.data.data_utils.data_preprocessing import find_newlines, load_data_indices
  from nemo.core.classes import Dataset
  from nemo.utils import logging
label_desc: Use image node_id 4 for calling the EntityLinkingDataset obj's underlying member method code with example usage: obj._collate_fn(batch) and returns: torch,  torch,  torch,  torch