class_docstr: The Squad Feature
  
  Parameters
  ----------
  qas_id
      The unique query/answer ID in the squad dataset
  query_token_ids
      The tokenized query.
  context_text
      The original text of the context
  context_token_ids
      The tokenized context.
  context_token_offsets
      The offsets of the tokens in the original context string
  is_impossible
      Whether the sample is impossible.
  token_answer_mismatch
      If this value is True, it means that we cannot reconstruct the ground-truth answer with
      the tokenized version. Usually, the span-prediction-based approach won't be very
      accurate and we should rely on the encoder-decoder approach.
      For example:
          GT: "japan", Tokenized Version: "japanese"
               "six'                       "sixth"
               "one"                       "iPhone"
               "breed"                     "breeding"
               "emotion"                   "emotional"
  
  unreliable_span
      If this value is True, it means that we cannot rely on the gt_start_pos and gt_end_pos.
      In this scenario, we cannot utilize the span-prediction-based approach.
      One example is the question about "how many", the answer will spread across the
      whole document and there is no clear span.
  gt_answer_text
      The ground-truth answer text
  gt_start_pos
      The start position of the ground-truth span. None indicates that there is no valid
      ground-truth span.
  gt_end_pos
      The end position of the ground-truth span. None indicates that there is no valid
      ground-truth span.
class_docstr_tok: ["The","Squad","Feature","Parameters","--","--","--","--","--","qas_id","The","unique","query\/answer","ID","in","the","squad","dataset","query_token_ids","The","tokenized","query",".","context_text","The","original","text","of","the","context","context_token_ids","The","tokenized","context",".","context_token_offsets","The","offsets","of","the","tokens","in","the","original","context","string","is_impossible","Whether","the","sample","is","impossible",".","token_answer_mismatch","If","this","value","is","True",",","it","means","that","we","can","not","reconstruct","the","ground-truth","answer","with","the","tokenized","version",".","Usually",",","the","span-prediction-based","approach","wo","n't","be","very","accurate","and","we","should","rely","on","the","encoder-decoder","approach",".","For","example",":","GT",":","``","japan","''",",","Tokenized","Version",":","``","japanese","''","``","six","'","``","sixth","''","``","one","''","``","iPhone","''","``","breed","''","``","breeding","''","``","emotion","''","``","emotional","''","unreliable_span","If","this","value","is","True",",","it","means","that","we","can","not","rely","on","the","gt_start_pos","and","gt_end_pos",".","In","this","scenario",",","we","can","not","utilize","the","span-prediction-based","approach",".","One","example","is","the","question","about","``","how","many","''",",","the","answer","will","spread","across","the","whole","document","and","there","is","no","clear","span",".","gt_answer_text","The","ground-truth","answer","text","gt_start_pos","The","start","position","of","the","ground-truth","span",".","None","indicates","that","there","is","no","valid","ground-truth","span",".","gt_end_pos","The","end","position","of","the","ground-truth","span",".","None","indicates","that","there","is","no","valid","ground-truth","span","."]
docstr: Get a sequence of chunks for the squad feature.
  
  In reality, the document will be too long for the NLP model, and we will split it into
  multiple chunks.
  
  For example, consider the following
  Doc: the man went to the store and bought a gallon of milk
  
  We may divide it into four chunks:
  
  Chunk 1: the man went to the
  Chunk 2: to the store and bought
  Chunk 3: and bought a gallon of
  Chunk 4: gallon of milk
  
  We will use our network to extract features for each chunk,
  and do the aggregation afterwards. Here, one token may appear in multiple chunks.
  We can vote the output based on some heuristic score functions.
  
  Parameters
  ----------
  doc_stride
      The stride used when the context is too large and is split across several features.
  max_chunk_length
      The maximum size of the chunk
  
  Returns
  -------
  ret
      List of DocChunk objects
docstr_tok: ["Get","a","sequence","of","chunks","for","the","squad","feature",".","In","reality",",","the","document","will","be","too","long","for","the","NLP","model",",","and","we","will","split","it","into","multiple","chunks",".","For","example",",","consider","the","following","Doc",":","the","man","went","to","the","store","and","bought","a","gallon","of","milk","We","may","divide","it","into","four","chunks",":","Chunk","1",":","the","man","went","to","the","Chunk","2",":","to","the","store","and","bought","Chunk","3",":","and","bought","a","gallon","of","Chunk","4",":","gallon","of","milk","We","will","use","our","network","to","extract","features","for","each","chunk",",","and","do","the","aggregation","afterwards",".","Here",",","one","token","may","appear","in","multiple","chunks",".","We","can","vote","the","output","based","on","some","heuristic","score","functions",".","Parameters","--","--","--","--","--","doc_stride","The","stride","used","when","the","context","is","too","large","and","is","split","across","several","features",".","max_chunk_length","The","maximum","size","of","the","chunk","Returns","--","--","--","-","ret","List","of","DocChunk","objects"]
code: def get_chunks(self, doc_stride, max_chunk_length=None):
      """Get a sequence of chunks for the squad feature.
  
      In reality, the document will be too long for the NLP model, and we will split it into
      multiple chunks.
  
      For example, consider the following
      Doc: the man went to the store and bought a gallon of milk
  
      We may divide it into four chunks:
  
      Chunk 1: the man went to the
      Chunk 2: to the store and bought
      Chunk 3: and bought a gallon of
      Chunk 4: gallon of milk
  
      We will use our network to extract features for each chunk,
      and do the aggregation afterwards. Here, one token may appear in multiple chunks.
      We can vote the output based on some heuristic score functions.
  
      Parameters
      ----------
      doc_stride
          The stride used when the context is too large and is split across several features.
      max_chunk_length
          The maximum size of the chunk
  
      Returns
      -------
      ret
          List of DocChunk objects
      """
      doc_ptr = 0
      max_chunk_length = (
          max_chunk_length
          if max_chunk_length is not None
          else len(self.context_token_ids)
      )
      ret = []
      while doc_ptr < len(self.context_token_ids):
          chunk_length = min(
              max_chunk_length, len(self.context_token_ids) - doc_ptr
          )
          if self.gt_answer_text is None:
              chunk_gt_start_pos = None
              chunk_gt_end_pos = None
              chunk_is_impossible = True
          else:
              if (
                  self.gt_start_pos is not None
                  and self.gt_end_pos is not None
                  and self.gt_start_pos >= doc_ptr
                  and self.gt_end_pos < doc_ptr + chunk_length
              ):
                  # The chunk contains the ground-truth annotation
                  chunk_gt_start_pos = self.gt_start_pos - doc_ptr
                  chunk_gt_end_pos = self.gt_end_pos - doc_ptr
                  chunk_is_impossible = False
              else:
                  chunk_gt_start_pos = None
                  chunk_gt_end_pos = None
                  chunk_is_impossible = True
          ret.append(
              DocChunk(
                  start=doc_ptr,
                  length=chunk_length,
                  is_impossible=chunk_is_impossible,
                  gt_start_pos=chunk_gt_start_pos,
                  gt_end_pos=chunk_gt_end_pos,
              )
          )
          if doc_ptr + chunk_length == len(self.context_token_ids):
              break
          doc_ptr += doc_stride
      return ret
  
imports: from typing import Optional, List, Tuple
  from collections import namedtuple
  import dataclasses
  from dataclasses import dataclass
  import re
  import numpy
  import warnings
  import os
  from tqdm.auto import tqdm
  import json
  import string
  from gluonnlp.data.tokenizers import BaseTokenizerWithVocab
  from gluonnlp.utils.preprocessing import match_tokens_with_char_spans
  from mxnet.gluon.utils import download
  import mxnet
label_desc: Use image node_id 4 for calling the SquadFeature obj's underlying member method code with example usage: obj.get_chunks(doc_stride, max_chunk_length) and returns: ret