class_docstr: Generate potential replacements for a word using a masked language
  model.
  
  Based off of following papers
      - "Robustness to Modification with Shared Words in Paraphrase Identification" (Shi et al., 2019) https://arxiv.org/abs/1909.02560
      - "BAE: BERT-based Adversarial Examples for Text Classification" (Garg et al., 2020) https://arxiv.org/abs/2004.01970
      - "BERT-ATTACK: Adversarial Attack Against BERT Using BERT" (Li et al, 2020) https://arxiv.org/abs/2004.09984
      - "CLARE: Contextualized Perturbation for Textual Adversarial Attack" (Li et al, 2020): https://arxiv.org/abs/2009.07502
  
  BAE and CLARE simply masks the word we want to replace and selects replacements predicted by the masked language model.
  
  BERT-Attack instead performs replacement on token level. For words that are consisted of two or more sub-word tokens,
      it takes the top-K replacements for seach sub-word token and produces all possible combinations of the top replacments.
      Then, it selects the top-K combinations based on their perplexity calculated using the masked language model.
  
  Choose which method to use by specifying "bae" or "bert-attack" for `method` argument.
  
  Args:
      method (str): the name of replacement method (e.g. "bae", "bert-attack")
      masked_language_model (Union[str|transformers.AutoModelForMaskedLM]): Either the name of pretrained masked language model from `transformers` model hub
          or the actual model. Default is `bert-base-uncased`.
      tokenizer (obj): The tokenizer of the corresponding model. If you passed in name of a pretrained model for `masked_language_model`,
          you can skip this argument as the correct tokenizer can be infered from the name. However, if you're passing the actual model, you must
          provide a tokenizer.
      max_length (int): the max sequence length the masked language model is designed to work with. Default is 512.
      window_size (int): The number of surrounding words to include when making top word prediction.
          For each word to swap, we take `window_size // 2` words to the left and `window_size // 2` words to the right and pass the text within the window
          to the masked language model. Default is `float("inf")`, which is equivalent to using the whole text.
      max_candidates (int): maximum number of candidates to consider as replacements for each word. Replacements are ranked by model's confidence.
      min_confidence (float): minimum confidence threshold each replacement word must pass.
      batch_size (int): Size of batch for "bae" replacement method.
class_docstr_tok: ["Generate","potential","replacements","for","a","word","using","a","masked","language","model",".","Based","off","of","following","papers","-","``","Robustness","to","Modification","with","Shared","Words","in","Paraphrase","Identification","''","(","Shi","et","al.",",","2019",")","https",":","\/\/arxiv.org\/abs\/1909.02560","-","``","BAE",":","BERT-based","Adversarial","Examples","for","Text","Classification","''","(","Garg","et","al.",",","2020",")","https",":","\/\/arxiv.org\/abs\/2004.01970","-","``","BERT-ATTACK",":","Adversarial","Attack","Against","BERT","Using","BERT","''","(","Li","et","al",",","2020",")","https",":","\/\/arxiv.org\/abs\/2004.09984","-","``","CLARE",":","Contextualized","Perturbation","for","Textual","Adversarial","Attack","''","(","Li","et","al",",","2020",")",":","https",":","\/\/arxiv.org\/abs\/2009.07502","BAE","and","CLARE","simply","masks","the","word","we","want","to","replace","and","selects","replacements","predicted","by","the","masked","language","model",".","BERT-Attack","instead","performs","replacement","on","token","level",".","For","words","that","are","consisted","of","two","or","more","sub-word","tokens",",","it","takes","the","top-K","replacements","for","seach","sub-word","token","and","produces","all","possible","combinations","of","the","top","replacments",".","Then",",","it","selects","the","top-K","combinations","based","on","their","perplexity","calculated","using","the","masked","language","model",".","Choose","which","method","to","use","by","specifying","``","bae","''","or","``","bert-attack","''","for","`","method","`","argument",".","Args",":","method","(","str",")",":","the","name","of","replacement","method","(","e.g",".","``","bae","''",",","``","bert-attack","''",")","masked_language_model","(","Union","[","str|transformers.AutoModelForMaskedLM","]",")",":","Either","the","name","of","pretrained","masked","language","model","from","`","transformers","`","model","hub","or","the","actual","model",".","Default","is","`","bert-base-uncased","`",".","tokenizer","(","obj",")",":","The","tokenizer","of","the","corresponding","model",".","If","you","passed","in","name","of","a","pretrained","model","for","`","masked_language_model","`",",","you","can","skip","this","argument","as","the","correct","tokenizer","can","be","infered","from","the","name",".","However",",","if","you","'re","passing","the","actual","model",",","you","must","provide","a","tokenizer",".","max_length","(","int",")",":","the","max","sequence","length","the","masked","language","model","is","designed","to","work","with",".","Default","is","512.","window_size","(","int",")",":","The","number","of","surrounding","words","to","include","when","making","top","word","prediction",".","For","each","word","to","swap",",","we","take","`","window_size","\/\/","2","`","words","to","the","left","and","`","window_size","\/\/","2","`","words","to","the","right","and","pass","the","text","within","the","window","to","the","masked","language","model",".","Default","is","`","float","(","``","inf","''",")","`",",","which","is","equivalent","to","using","the","whole","text",".","max_candidates","(","int",")",":","maximum","number","of","candidates","to","consider","as","replacements","for","each","word",".","Replacements","are","ranked","by","model","'s","confidence",".","min_confidence","(","float",")",":","minimum","confidence","threshold","each","replacement","word","must","pass",".","batch_size","(","int",")",":","Size","of","batch","for","``","bae","''","replacement","method","."]
docstr: Get replacement words for the word we want to replace using BERT-
  Attack method.
  
  Args:
      current_text (AttackedText): Text we want to get replacements for.
      index (int): index of word we want to replace
      id_preds (torch.Tensor): N x K tensor of top-K ids for each token-position predicted by the masked language model.
          N is equivalent to `self.max_length`.
      masked_lm_logits (torch.Tensor): N x V tensor of the raw logits outputted by the masked language model.
          N is equivlaent to `self.max_length` and V is dictionary size of masked language model.
docstr_tok: ["Get","replacement","words","for","the","word","we","want","to","replace","using","BERT-","Attack","method",".","Args",":","current_text","(","AttackedText",")",":","Text","we","want","to","get","replacements","for",".","index","(","int",")",":","index","of","word","we","want","to","replace","id_preds","(","torch.Tensor",")",":","N","x","K","tensor","of","top-K","ids","for","each","token-position","predicted","by","the","masked","language","model",".","N","is","equivalent","to","`","self.max_length","`",".","masked_lm_logits","(","torch.Tensor",")",":","N","x","V","tensor","of","the","raw","logits","outputted","by","the","masked","language","model",".","N","is","equivlaent","to","`","self.max_length","`","and","V","is","dictionary","size","of","masked","language","model","."]
code: def _bert_attack_replacement_words(
      self,
      current_text,
      index,
      id_preds,
      masked_lm_logits,
  ):
      """Get replacement words for the word we want to replace using BERT-
      Attack method.
  
      Args:
          current_text (AttackedText): Text we want to get replacements for.
          index (int): index of word we want to replace
          id_preds (torch.Tensor): N x K tensor of top-K ids for each token-position predicted by the masked language model.
              N is equivalent to `self.max_length`.
          masked_lm_logits (torch.Tensor): N x V tensor of the raw logits outputted by the masked language model.
              N is equivlaent to `self.max_length` and V is dictionary size of masked language model.
      """
      # We need to find which BPE tokens belong to the word we want to replace
      masked_text = current_text.replace_word_at_index(
          index, self._lm_tokenizer.mask_token
      )
      current_inputs = self._encode_text(masked_text.text)
      current_ids = current_inputs["input_ids"].tolist()[0]
      word_tokens = self._lm_tokenizer.encode(
          current_text.words[index], add_special_tokens=False
      )
  
      try:
          # Need try-except b/c mask-token located past max_length might be truncated by tokenizer
          masked_index = current_ids.index(
              self._lm_tokenizer.mask_token_id
          )
      except ValueError:
          return []
  
      # List of indices of tokens that are part of the target word
      target_ids_pos = list(
          range(
              masked_index,
              min(masked_index + len(word_tokens), self.max_length),
          )
      )
  
      if not len(target_ids_pos):
          return []
      elif len(target_ids_pos) == 1:
          # Word to replace is tokenized as a single word
          top_preds = id_preds[target_ids_pos[0]].tolist()
          replacement_words = []
          for id in top_preds:
              token = self._lm_tokenizer.convert_ids_to_tokens(id)
              if utils.is_one_word(
                  token
              ) and not utils.check_if_subword(
                  token,
                  self._language_model.config.model_type,
                  index == 0,
              ):
                  replacement_words.append(token)
          return replacement_words
      else:
          # Word to replace is tokenized as multiple sub-words
          top_preds = [id_preds[i] for i in target_ids_pos]
          products = itertools.product(*top_preds)
          combination_results = []
          # Original BERT-Attack implement uses cross-entropy loss
          cross_entropy_loss = torch.nn.CrossEntropyLoss(
              reduction="none"
          )
          target_ids_pos_tensor = torch.tensor(target_ids_pos)
          word_tensor = torch.zeros(
              len(target_ids_pos), dtype=torch.long
          )
          for bpe_tokens in products:
              for i in range(len(bpe_tokens)):
                  word_tensor[i] = bpe_tokens[i]
  
              logits = torch.index_select(
                  masked_lm_logits, 0, target_ids_pos_tensor
              )
              loss = cross_entropy_loss(logits, word_tensor)
              perplexity = torch.exp(torch.mean(loss, dim=0)).item()
              word = "".join(
                  self._lm_tokenizer.convert_ids_to_tokens(word_tensor)
              ).replace("##", "")
              if utils.is_one_word(word):
                  combination_results.append((word, perplexity))
          # Sort to get top-K results
          sorted(combination_results, key=lambda x: x[1])
          top_replacements = [
              x[0] for x in combination_results[: self.max_candidates]
          ]
          return top_replacements
  
file_path: TextAttack/textattack/transformations/word_swaps/word_swap_masked_lm.py
imports: import itertools
  import re
  import torch
  from transformers import AutoModelForMaskedLM, AutoTokenizer
  from textattack.shared import utils
  from .word_swap import WordSwap
label_desc: Use image node_id 4 for calling the WordSwapMaskedLM obj's underlying member method code with example usage: obj._bert_attack_replacement_words(current_text, index, id_preds, masked_lm_logits) and returns: list, list, replacement_words, top_replacements