class_docstr: Tokenizer that uses the en_core_sci_sm Spacy model and some special heuristics.
  
  Implementation of :class:`Tokenizer` which uses the en_core_sci_sm Spacy model
  extended by special heuristics to consider characters such as "(", ")" "-" as
  additional token separators. The latter distinguishes this implementation from
  :class:`SpacyTokenizer`.
  
  Note, you if you want to use the "normal" SciSpacy tokenization just use
  :class:`SpacyTokenizer`.
class_docstr_tok: ["Tokenizer","that","uses","the","en_core_sci_sm","Spacy","model","and","some","special","heuristics",".","Implementation","of",":","class",":","`","Tokenizer","`","which","uses","the","en_core_sci_sm","Spacy","model","extended","by","special","heuristics","to","consider","characters","such","as","``","(","``",",","``",")","''","``","-","''","as","additional","token","separators",".","The","latter","distinguishes","this","implementation","from",":","class",":","`","SpacyTokenizer","`",".","Note",",","you","if","you","want","to","use","the","``","normal","''","SciSpacy","tokenization","just","use",":","class",":","`","SpacyTokenizer","`","."]
code: def __init__(self) -> None:
      super().__init__()
  
      try:
          import spacy
          from spacy.lang import char_classes
      except ImportError:
          raise ImportError(
              "  Please install scispacy version 0.5.1 (recommended) or higher before using the SciSpacy tokenizer, "
              "otherwise you can use SegtokTokenizer as alternative implementation.\n"
              "  You can install scispacy (version 0.5.1) by running:\n\n"
              "     pip install scispacy==0.5.1\n\n"
              "  By default HunFlair uses the `en_core_sci_sm` model. You can install the model by running:\n\n"
              "     pip install https://s3-us-west-2.amazonaws.com/ai2-s2-scispacy/releases/v0.5.1/en_core_sci_sm-0.5.1.tar.gz\n\n"
              "  Note that the scispacy version and the version of the model must match to work properly!"
          )
  
      def combined_rule_prefixes() -> List[str]:
          """Helper function that returns the prefix pattern for the tokenizer.
  
          It is a helper function to accommodate spacy tests that only test prefixes.
          """
          prefix_punct = char_classes.PUNCT.replace("|", " ")
  
          prefixes = [
              "§",
              "%",
              "=",
              "\\+",
              *char_classes.split_chars(prefix_punct),
              *char_classes.LIST_ELLIPSES,
              *char_classes.LIST_QUOTES,
              *char_classes.LIST_CURRENCY,
              *char_classes.LIST_ICONS,
          ]
          return prefixes
  
      infixes = (
          char_classes.LIST_ELLIPSES
          + char_classes.LIST_ICONS
          + [
              r"x",  # added this special x character to tokenize it separately
              r"[\(\)\[\]\{\}]",  # want to split at every bracket
              r"/",  # want to split at every slash
              r"(?<=[0-9])[+\-\*^](?=[0-9-])",
              rf"(?<=[{char_classes.ALPHA_LOWER}])\.(?=[{char_classes.ALPHA_UPPER}])",
              rf"(?<=[{char_classes.ALPHA}]),(?=[{char_classes.ALPHA}])",
              rf'(?<=[{char_classes.ALPHA}])[?";:=,.]*(?:{char_classes.HYPHENS})(?=[{char_classes.ALPHA}])',
              rf"(?<=[{char_classes.ALPHA}0-9])[:<>=/](?=[{char_classes.ALPHA}])",
          ]
      )
  
      prefix_re = spacy.util.compile_prefix_regex(
          combined_rule_prefixes()
      )
      infix_re = spacy.util.compile_infix_regex(infixes)
  
      self.model = spacy.load(
          "en_core_sci_sm",
          disable=["tagger", "ner", "parser", "textcat", "lemmatizer"],
      )
      self.model.tokenizer.prefix_search = prefix_re.search
      self.model.tokenizer.infix_finditer = infix_re.finditer
  
imports: import logging
  import sys
  from abc import ABC, abstractmethod
  from typing import Callable, List
  from segtok.segmenter import split_single
  from segtok.tokenizer import split_contractions, word_tokenizer
label_desc: Use image node_id 1 to create a new SciSpacyTokenizer object from inherited base classes: Tokenizer with example: obj = SciSpacyTokenizer()