class_docstr: An OutputSeq object is the result of running a language model on some input data. It contains not only the output
  sequence of words generated by the model, but also other data collecting during the generation process
  that is useful to analyze the model.
  
  In addition to the data, the object has methods to create plots
  and visualizations of that collected data. These include:
  
  - [layer_predictions()](./#ecco.output.OutputSeq.layer_predictions) <br/>
  Which tokens did the model consider as the best outputs for a specific position in the sequence?
  - [rankings()](./#ecco.output.OutputSeq.rankings) <br/>
  After the model chooses an output token for a specific position, this visual looks back at the ranking
  of this token at each layer of the model when it was generated (layers assign scores to candidate output tokens,
  the higher the "probability" score, the higher the ranking of the token).
  - [rankings_watch()](./#ecco.output.OutputSeq.rankings_watch) <br />
  Shows the rankings of multiple tokens as the model scored them for a single position. For example, if the input is
  "The cat \_\_\_", we use this method to observe how the model ranked the words "is", "are", "was" as candidates
  to fill in the blank.
  - [primary_attributions()](./#ecco.output.OutputSeq.primary_attributions) <br />
  How important was each input token in the selection of calculating the output token?
  
  
  To process neuron activations, OutputSeq has methods to reduce the dimensionality and reveal underlying patterns in
  neuron firings. These are:
  
  - [run_nmf()](./#ecco.output.OutputSeq.run_nmf)
class_docstr_tok: ["An","OutputSeq","object","is","the","result","of","running","a","language","model","on","some","input","data",".","It","contains","not","only","the","output","sequence","of","words","generated","by","the","model",",","but","also","other","data","collecting","during","the","generation","process","that","is","useful","to","analyze","the","model",".","In","addition","to","the","data",",","the","object","has","methods","to","create","plots","and","visualizations","of","that","collected","data",".","These","include",":","-","[","layer_predictions","(",")","]","(",".\/","#","ecco.output.OutputSeq.layer_predictions",")","<","br\/",">","Which","tokens","did","the","model","consider","as","the","best","outputs","for","a","specific","position","in","the","sequence","?","-","[","rankings","(",")","]","(",".\/","#","ecco.output.OutputSeq.rankings",")","<","br\/",">","After","the","model","chooses","an","output","token","for","a","specific","position",",","this","visual","looks","back","at","the","ranking","of","this","token","at","each","layer","of","the","model","when","it","was","generated","(","layers","assign","scores","to","candidate","output","tokens",",","the","higher","the","``","probability","''","score",",","the","higher","the","ranking","of","the","token",")",".","-","[","rankings_watch","(",")","]","(",".\/","#","ecco.output.OutputSeq.rankings_watch",")","<","br","\/",">","Shows","the","rankings","of","multiple","tokens","as","the","model","scored","them","for","a","single","position",".","For","example",",","if","the","input","is","''","The","cat","\\_\\_\\_","''",",","we","use","this","method","to","observe","how","the","model","ranked","the","words","``","is","''",",","``","are","''",",","``","was","''","as","candidates","to","fill","in","the","blank",".","-","[","primary_attributions","(",")","]","(",".\/","#","ecco.output.OutputSeq.primary_attributions",")","<","br","\/",">","How","important","was","each","input","token","in","the","selection","of","calculating","the","output","token","?","To","process","neuron","activations",",","OutputSeq","has","methods","to","reduce","the","dimensionality","and","reveal","underlying","patterns","in","neuron","firings",".","These","are",":","-","[","run_nmf","(",")","]","(",".\/","#","ecco.output.OutputSeq.run_nmf",")"]
docstr: Args:
      token_ids: The input token ids. Dimensions: (batch, position)
      n_input_tokens: Int. The number of input tokens in the sequence.
      tokenizer: huggingface tokenizer associated with the model generating this output
      output_text: The output text generated by the model (if processed with generate())
      tokens: A list of token text. Shorthand to passing the token ids by the tokenizer.
          dimensions are (batch, position)
      hidden_states: A tensor of  dimensions (layer, position, hidden_dimension).
          In layer, index 0 is for embedding hidden_state.
      attribution: A list of attributions. One element per generated token.
          Each element is a list giving a value for tokens from 0 to right before the generated token.
      activations: The activations collected from model processing.
          Shape is (batch, layer, neurons, position)
      collect_activations_layer_nums:
      attention: The attention tensor retrieved from the language model
      model_outputs: Raw return object returned by the model
      lm_head: The trained language model head from a language model projecting a
          hidden state to an output vocabulary associated with teh tokenizer.
      device: "cuda" or "cpu"
      config: The configuration dict of the language model
docstr_tok: ["Args",":","token_ids",":","The","input","token","ids",".","Dimensions",":","(","batch",",","position",")","n_input_tokens",":","Int",".","The","number","of","input","tokens","in","the","sequence",".","tokenizer",":","huggingface","tokenizer","associated","with","the","model","generating","this","output","output_text",":","The","output","text","generated","by","the","model","(","if","processed","with","generate","(",")",")","tokens",":","A","list","of","token","text",".","Shorthand","to","passing","the","token","ids","by","the","tokenizer",".","dimensions","are","(","batch",",","position",")","hidden_states",":","A","tensor","of","dimensions","(","layer",",","position",",","hidden_dimension",")",".","In","layer",",","index","0","is","for","embedding","hidden_state",".","attribution",":","A","list","of","attributions",".","One","element","per","generated","token",".","Each","element","is","a","list","giving","a","value","for","tokens","from","0","to","right","before","the","generated","token",".","activations",":","The","activations","collected","from","model","processing",".","Shape","is","(","batch",",","layer",",","neurons",",","position",")","collect_activations_layer_nums",":","attention",":","The","attention","tensor","retrieved","from","the","language","model","model_outputs",":","Raw","return","object","returned","by","the","model","lm_head",":","The","trained","language","model","head","from","a","language","model","projecting","a","hidden","state","to","an","output","vocabulary","associated","with","teh","tokenizer",".","device",":","``","cuda","''","or","``","cpu","''","config",":","The","configuration","dict","of","the","language","model"]
code: def __init__(
      self,
      token_ids=None,
      n_input_tokens=None,
      tokenizer=None,
      output_text=None,
      tokens=None,
      encoder_hidden_states=None,
      decoder_hidden_states=None,
      embedding_states=None,
      attribution=None,
      activations=None,
      collect_activations_layer_nums=None,
      attention=None,
      model_type: str = "mlm",
      lm_head=None,
      device="cpu",
      config=None,
  ):
      """
  
      Args:
          token_ids: The input token ids. Dimensions: (batch, position)
          n_input_tokens: Int. The number of input tokens in the sequence.
          tokenizer: huggingface tokenizer associated with the model generating this output
          output_text: The output text generated by the model (if processed with generate())
          tokens: A list of token text. Shorthand to passing the token ids by the tokenizer.
              dimensions are (batch, position)
          hidden_states: A tensor of  dimensions (layer, position, hidden_dimension).
              In layer, index 0 is for embedding hidden_state.
          attribution: A list of attributions. One element per generated token.
              Each element is a list giving a value for tokens from 0 to right before the generated token.
          activations: The activations collected from model processing.
              Shape is (batch, layer, neurons, position)
          collect_activations_layer_nums:
          attention: The attention tensor retrieved from the language model
          model_outputs: Raw return object returned by the model
          lm_head: The trained language model head from a language model projecting a
              hidden state to an output vocabulary associated with teh tokenizer.
          device: "cuda" or "cpu"
          config: The configuration dict of the language model
      """
      self.token_ids = token_ids
      self.tokenizer = tokenizer
      self.n_input_tokens = n_input_tokens
      self.output_text = output_text
      self.tokens = tokens
      self.encoder_hidden_states = encoder_hidden_states
      self.decoder_hidden_states = decoder_hidden_states
      self.embedding_states = embedding_states
      self.attribution = attribution
      self.activations = activations
      self.collect_activations_layer_nums = (
          collect_activations_layer_nums
      )
      self.attention_values = attention
      self.lm_head = lm_head
      self.device = device
      self.config = config
      self.model_type = model_type
      self._path = os.path.dirname(ecco.__file__)
  
imports: import os
  import json
  import ecco
  from IPython import display
  from ecco import util, lm_plots
  import random
  import matplotlib.pyplot
  import numpy
  import torch
  from torch.nn import functional
  from sklearn import decomposition
  from typing import Dict, Optional, List, Tuple, Union
  from ecco.util import strip_tokenizer_prefix, is_partial_token
label_desc: Use image node_id 1 to create a new OutputSeq object  with example: obj = OutputSeq(token_ids, n_input_tokens, tokenizer, output_text, tokens, encoder_hidden_states, decoder_hidden_states, embedding_states, attribution, activations, collect_activations_layer_nums, attention, model_type, lm_head, device, config)