docstr: Input a sentence, return a processed sentence: In order to support the Chinese whole word mask, the words that are separated will be marked with a special mark ("#"), so that the subsequent processing module can know which words belong to the same word.
  :param segment: a sentence
docstr_tok: ["Input","a","sentence",",","return","a","processed","sentence",":","In","order","to","support","the","Chinese","whole","word","mask",",","the","words","that","are","separated","will","be","marked","with","a","special","mark","(","``","#","''",")",",","so","that","the","subsequent","processing","module","can","know","which","words","belong","to","the","same","word",".",":","param","segment",":","a","sentence"]
code: def get_new_segment(self, segment):
      """
      Input a sentence, return a processed sentence: In order to support the Chinese whole word mask, the words that are separated will be marked with a special mark ("#"), so that the subsequent processing module can know which words belong to the same word.
      :param segment: a sentence
      """
      seq_cws = jieba.lcut("".join(segment))
      seq_cws_dict = {x: 1 for x in seq_cws}
      new_segment = []
      i = 0
      while i < len(segment):
          if len(self.rec.findall(segment[i])) == 0:
              new_segment.append(segment[i])
              i += 1
              continue
  
          has_add = False
          for length in range(3, 0, -1):
              if i + length > len(segment):
                  continue
              if "".join(segment[i : i + length]) in seq_cws_dict:
                  new_segment.append(segment[i])
                  for l in range(1, length):
                      new_segment.append("##" + segment[i + l])
                  i += length
                  has_add = True
                  break
          if not has_add:
              new_segment.append(segment[i])
              i += 1
      return new_segment
  
file_path: colossalai/examples/community/roberta/preprocessing/get_mask.py
imports: import collections
  import logging
  import random
  import jieba
  import re
  import mask
  import numpy
label_desc: Use image node_id 5 for calling the PreTrainingDataset obj's underlying member method code with example usage: obj.get_new_segment(segment) and returns: new_segment