class_docstr: Make chunks from a document or a list of documents. Don't tear up sentences if needed.
  
  Args:
      sentencize_fn: a function for sentence segmentation
      keep_sentences: whether to tear up sentences between chunks or not
      tokens_limit: a number of tokens in a single chunk (usually this number corresponds to the squad model limit)
      flatten_result: whether to flatten the resulting list of lists of chunks
      paragraphs: whether to split document by paragrahs; if set to True, tokens_limit is ignored
  
  Attributes:
      keep_sentences: whether to tear up sentences between chunks or not
      tokens_limit: a number of tokens in a single chunk
      flatten_result: whether to flatten the resulting list of lists of chunks
      paragraphs: whether to split document by paragrahs; if set to True, tokens_limit is ignored
code: def __init__(
      self,
      sentencize_fn: Callable = sent_tokenize,
      keep_sentences: bool = True,
      tokens_limit: int = 400,
      flatten_result: bool = False,
      paragraphs: bool = False,
      number_of_paragraphs: int = -1,
      *args,
      **kwargs,
  ) -> None:
      self._sentencize_fn = sentencize_fn
      self.keep_sentences = keep_sentences
      self.tokens_limit = tokens_limit
      self.flatten_result = flatten_result
      self.paragraphs = paragraphs
      self.number_of_paragraphs = number_of_paragraphs
  
file_path: DeepPavlov/deeppavlov/models/preprocessors/odqa_preprocessors.py
imports: from itertools import chain
  from logging import getLogger
  from typing import List, Callable, Union, Tuple, Optional
  from nltk import sent_tokenize
  from deeppavlov.core.common.registry import register
  from deeppavlov.core.models.component import Component
label_desc: Use image node_id 1 to create a new DocumentChunker object from inherited base classes: Component with example: obj = DocumentChunker(sentencize_fn, keep_sentences, tokens_limit, flatten_result, paragraphs, number_of_paragraphs)