class_docstr: This class calculates Viterbi alignments in the forward method.
  
  It also records alignments and creates batches of them for use
  in Viterbi training.
  
  Arguments
  ---------
  states_per_phoneme : int
      Number of hidden states to use per phoneme.
  output_folder : str
      It is the folder that the alignments will be stored in when
      saved to disk. Not yet implemented.
  neg_inf : float
      The float used to represent a negative infinite log probability.
      Using `-float("Inf")` tends to give numerical instability.
      A number more negative than -1e5 also sometimes gave errors when
      the `genbmm` library was used (currently not in use). (default: -1e5)
  batch_reduction : string
      One of "none", "sum" or "mean".
      What kind of batch-level reduction to apply to the loss calculated
      in the forward method.
  input_len_norm : bool
      Whether to normalize the loss in the forward method by the length of
      the inputs.
  target_len_norm : bool
      Whether to normalize the loss in the forward method by the length of
      the targets.
  lexicon_path : string
      The location of the lexicon.
  
  Example
  -------
  >>> log_posteriors = torch.tensor([[[ -1., -10., -10.],
  ...                                 [-10.,  -1., -10.],
  ...                                 [-10., -10.,  -1.]],
  ...
  ...                                [[ -1., -10., -10.],
  ...                                 [-10.,  -1., -10.],
  ...                                 [-10., -10., -10.]]])
  >>> lens = torch.tensor([1., 0.66])
  >>> phns = torch.tensor([[0, 1, 2],
  ...                      [0, 1, 0]])
  >>> phn_lens = torch.tensor([1., 0.66])
  >>> aligner = HMMAligner()
  >>> forward_scores = aligner(
  ...        log_posteriors, lens, phns, phn_lens, 'forward'
  ... )
  >>> forward_scores.shape
  torch.Size([2])
  >>> viterbi_scores, alignments = aligner(
  ...        log_posteriors, lens, phns, phn_lens, 'viterbi'
  ... )
  >>> alignments
  [[0, 1, 2], [0, 1]]
  >>> viterbi_scores.shape
  torch.Size([2])
docstr: Do processing using the lexicon to return a sequence of the possible
  phonemes, the transition/pi probabilities, and the possible final states.
  Inputs correspond to a single utterance, not a whole batch.
  
  Arguments
  ---------
  words : list
      List of the words in the transcript.
  interword_sils : bool
      If True, optional silences will be inserted between every word.
      If False, optional silences will only be placed at the beginning
      and end of each utterance.
  sample_pron : bool
      If True, it will sample a single possible sequence of phonemes.
      If False, it will return statistics for all possible sequences of
      phonemes.
  
  Returns
  -------
  poss_phns : torch.Tensor (phoneme)
      The phonemes that are thought to be in each utterance.
  log_transition_matrix : torch.Tensor (batch, from, to)
      Tensor containing transition (log) probabilities.
  start_states : list of ints
      A list of the possible starting states in each utterance.
  final_states : list of ints
      A list of the possible final states for each utterance.
docstr_tok: ["Do","processing","using","the","lexicon","to","return","a","sequence","of","the","possible","phonemes",",","the","transition\/pi","probabilities",",","and","the","possible","final","states",".","Inputs","correspond","to","a","single","utterance",",","not","a","whole","batch",".","Arguments","--","--","--","--","-","words",":","list","List","of","the","words","in","the","transcript",".","interword_sils",":","bool","If","True",",","optional","silences","will","be","inserted","between","every","word",".","If","False",",","optional","silences","will","only","be","placed","at","the","beginning","and","end","of","each","utterance",".","sample_pron",":","bool","If","True",",","it","will","sample","a","single","possible","sequence","of","phonemes",".","If","False",",","it","will","return","statistics","for","all","possible","sequences","of","phonemes",".","Returns","--","--","--","-","poss_phns",":","torch.Tensor","(","phoneme",")","The","phonemes","that","are","thought","to","be","in","each","utterance",".","log_transition_matrix",":","torch.Tensor","(","batch",",","from",",","to",")","Tensor","containing","transition","(","log",")","probabilities",".","start_states",":","list","of","ints","A","list","of","the","possible","starting","states","in","each","utterance",".","final_states",":","list","of","ints","A","list","of","the","possible","final","states","for","each","utterance","."]
code: def _use_lexicon(self, words, interword_sils, sample_pron):
      """Do processing using the lexicon to return a sequence of the possible
      phonemes, the transition/pi probabilities, and the possible final states.
      Inputs correspond to a single utterance, not a whole batch.
  
      Arguments
      ---------
      words : list
          List of the words in the transcript.
      interword_sils : bool
          If True, optional silences will be inserted between every word.
          If False, optional silences will only be placed at the beginning
          and end of each utterance.
      sample_pron : bool
          If True, it will sample a single possible sequence of phonemes.
          If False, it will return statistics for all possible sequences of
          phonemes.
  
      Returns
      -------
      poss_phns : torch.Tensor (phoneme)
          The phonemes that are thought to be in each utterance.
      log_transition_matrix : torch.Tensor (batch, from, to)
          Tensor containing transition (log) probabilities.
      start_states : list of ints
          A list of the possible starting states in each utterance.
      final_states : list of ints
          A list of the possible final states for each utterance.
      """
  
      number_of_states = 0
      words_prime = (
          []
      )  # This will contain one "word" for each optional silence and pronunciation.
      # structure of each "word_prime":
      # [word index, [[state sequence 1], [state sequence 2]], <is this an optional silence?>]
      word_index = 0
      phoneme_indices = []
      for word in words:
          if word_index == 0 or interword_sils is True:
              # optional silence
              word_prime = [
                  word_index,
                  [
                      [
                          number_of_states + i
                          for i in range(self.states_per_phoneme)
                      ]
                  ],
                  True,
              ]
              words_prime.append(word_prime)
              phoneme_indices += [
                  self.silence_index * self.states_per_phoneme + i
                  for i in range(self.states_per_phoneme)
              ]
              number_of_states += self.states_per_phoneme
              word_index += 1
  
          # word
          word_prime = [word_index, [], False]
          if sample_pron and len(self.lexicon[word]) > 1:
              random.shuffle(self.lexicon[word])
          for pron_idx in range(len(self.lexicon[word])):
              pronunciation = self.lexicon[word][pron_idx]
              phonemes = pronunciation.split()
              word_prime[1].append([])
              for p in phonemes:
                  phoneme_indices += [
                      self.lex_lab2ind[p] * self.states_per_phoneme + i
                      for i in range(self.states_per_phoneme)
                  ]
                  word_prime[1][pron_idx] += [
                      number_of_states + i
                      for i in range(self.states_per_phoneme)
                  ]
                  number_of_states += self.states_per_phoneme
              if sample_pron:
                  break
  
          words_prime.append(word_prime)
          word_index += 1
      # optional final silence
      word_prime = [
          word_index,
          [
              [
                  number_of_states + i
                  for i in range(self.states_per_phoneme)
              ]
          ],
          True,
      ]
      words_prime.append(word_prime)
      phoneme_indices += [
          self.silence_index * self.states_per_phoneme + i
          for i in range(self.states_per_phoneme)
      ]
      number_of_states += self.states_per_phoneme
      word_index += 1
  
      transition_matrix = 1.0 * torch.eye(
          number_of_states
      )  # diagonal = all states have a self-loop
      final_states = []
      for word_prime in words_prime:
          word_idx = word_prime[0]
          is_optional_silence = word_prime[-1]
          next_word_exists = word_idx < len(words_prime) - 2
          this_word_last_states = [
              word_prime[1][i][-1] for i in range(len(word_prime[1]))
          ]
  
          # create transitions to next state from previous state within each pronunciation
          for pronunciation in word_prime[1]:
              for state_idx in range(len(pronunciation) - 1):
                  state = pronunciation[state_idx]
                  next_state = pronunciation[state_idx + 1]
                  transition_matrix[state, next_state] = 1.0
  
          # create transitions to next word's starting states
          if next_word_exists:
              if is_optional_silence or not interword_sils:
                  next_word_idx = word_idx + 1
              else:
                  next_word_idx = word_idx + 2
              next_word_starting_states = [
                  words_prime[next_word_idx][1][i][0]
                  for i in range(len(words_prime[next_word_idx][1]))
              ]
  
              for this_word_last_state in this_word_last_states:
                  for (
                      next_word_starting_state
                  ) in next_word_starting_states:
                      transition_matrix[
                          this_word_last_state, next_word_starting_state
                      ] = 1.0
  
          else:
              final_states += this_word_last_states
  
          if not is_optional_silence:
              next_silence_idx = word_idx + 1
              next_silence_starting_state = words_prime[
                  next_silence_idx
              ][1][0][0]
              for this_word_last_state in this_word_last_states:
                  transition_matrix[
                      this_word_last_state, next_silence_starting_state
                  ] = 1.0
  
      log_transition_matrix = transition_matrix.log().log_softmax(1)
  
      start_states = [words_prime[0][1][0][0]]
      start_states += [
          words_prime[1][1][i][0] for i in range(len(words_prime[1][1]))
      ]
  
      poss_phns = torch.tensor(phoneme_indices)
  
      return (
          poss_phns,
          log_transition_matrix,
          start_states,
          final_states,
      )
  
imports: import torch
  import random
  from speechbrain.utils.checkpoints import register_checkpoint_hooks
  from speechbrain.utils.checkpoints import mark_as_saver
  from speechbrain.utils.checkpoints import mark_as_loader
  from speechbrain.utils.data_utils import undo_padding
label_desc: Use image node_id 2 for calling the HMMAligner obj's underlying member method code with example usage: obj._use_lexicon(words, interword_sils, sample_pron) and returns: poss_phns,  log_transition_matrix,  start_states,  final_states