trax/models/rnn.py
def LSTMSeq2SeqAttn(input_vocab_size=256,
                    target_vocab_size=256,
                    d_model=512,
                    n_encoder_layers=2,
                    n_decoder_layers=2,
                    n_attention_heads=1,
                    attention_dropout=0.0,
                    mode='train'):
  """Returns an LSTM sequence-to-sequence model with attention.

  The input to the model is a pair (input tokens, target tokens), e.g.,
  an English sentence (tokenized) and its translation into German (tokenized).

  The model works as follows:

  * Input encoder runs on the input tokens and creates activations that
    are used as both keys and values in attention.
  * Pre-attention decoder runs on the targets and creates
    activations that are used as queries in attention.
  * Attention runs on the queries, keys and values masking out input padding.
  * Decoder runs on the result, followed by a cross-entropy loss.

  Args:
    input_vocab_size: int: vocab size of the input
    target_vocab_size: int: vocab size of the target
    d_model: int:  depth of embedding (n_units in the LSTM cell)
    n_encoder_layers: int: number of LSTM layers in the encoder
    n_decoder_layers: int: number of LSTM layers in the decoder after attention
    n_attention_heads: int: number of attention heads
    attention_dropout: float, dropout for the attention layer
    mode: str: 'train', 'eval' or 'predict', predict mode is for fast inference

  Returns:
    An LSTM sequence-to-sequence model with attention.
  """
  input_encoder = tl.Serial(
      tl.Embedding(input_vocab_size, d_model),
      [tl.LSTM(d_model) for _ in range(n_encoder_layers)],
  )

  pre_attention_decoder = tl.Serial(
      tl.ShiftRight(mode=mode),
      tl.Embedding(target_vocab_size, d_model),
      tl.LSTM(d_model),
  )

  def PrepareAttentionInputs():
    """Layer that prepares queries, keys, values and mask for attention."""
    def F(encoder_activations, decoder_activations, input_tokens):
      keys = values = encoder_activations
      queries = decoder_activations
      # Mask is 1 where inputs are not padding (0) and 0 where they are padding.
      mask = (input_tokens != 0)
      # We need to add axes to the mask for attention heads and decoder length.
      mask = jnp.reshape(mask, (mask.shape[0], 1, 1, mask.shape[1]))
      # Broadcast so mask is [batch, 1 for heads, decoder-len, encoder-len].
      mask = mask + jnp.zeros((1, 1, decoder_activations.shape[1], 1))
      return queries, keys, values, mask
    return tl.Fn('PrepareAttentionInputs', F, n_out=4)

  return tl.Serial(              # in-toks, target-toks
      tl.Select([0, 1, 0, 1]),   # in-toks, target-toks, in-toks, target-toks
      tl.Parallel(input_encoder, pre_attention_decoder),
      PrepareAttentionInputs(),  # q, k, v, mask, target-toks
      tl.Residual(
          tl.AttentionQKV(d_model, n_heads=n_attention_heads,
                          dropout=attention_dropout, mode=mode)
      ),                         # decoder-vecs, mask, target-toks
      tl.Select([0, 2]),         # decoder-vecs, target-toks
      [tl.LSTM(d_model) for _ in range(n_decoder_layers)],
      tl.Dense(target_vocab_size),
      tl.LogSoftmax()
  )
Returns an LSTM sequence-to-sequence model with attention.

  The input to the model is a pair (input tokens, target tokens), e.g.,
  an English sentence (tokenized) and its translation into German (tokenized).

  The model works as follows:

  * Input encoder runs on the input tokens and creates activations that
    are used as both keys and values in attention.
  * Pre-attention decoder runs on the targets and creates
    activations that are used as queries in attention.
  * Attention runs on the queries, keys and values masking out input padding.
  * Decoder runs on the result, followed by a cross-entropy loss.

  Args:
    input_vocab_size: int: vocab size of the input
    target_vocab_size: int: vocab size of the target
    d_model: int:  depth of embedding (n_units in the LSTM cell)
    n_encoder_layers: int: number of LSTM layers in the encoder
    n_decoder_layers: int: number of LSTM layers in the decoder after attention
    n_attention_heads: int: number of attention heads
    attention_dropout: float, dropout for the attention layer
    mode: str: 'train', 'eval' or 'predict', predict mode is for fast inference

  Returns:
    An LSTM sequence-to-sequence model with attention.
  
Returns an LSTM sequence-to-sequence model with attention.
The input to the model is a pair (input tokens, target tokens), e.g.,
an English sentence (tokenized) and its translation into German (tokenized).

The model works as follows.

Input encoder runs on the input tokens and creates activations that
are used as both keys and values in attention.
Pre-attention decoder runs on the targets and creates
activations that are used as queries in attention.
Attention runs on the queries, keys and values masking out input padding.
Decoder runs on the result, followed by a cross-entropy loss.
Returns an LSTM sequence - to - sequence model with attention . The input to the model is a pair ( input tokens target tokens ) e . g . an English sentence ( tokenized ) and its translation into German ( tokenized ) . The model works as follows . Input encoder runs on the input tokens and creates activations that are used as both keys and values in attention . Pre - attention decoder runs on the targets and creates activations that are used as queries in attention . Attention runs on the queries keys and values masking out input padding . Decoder runs on the result followed by a cross - entropy loss .
def LSTMSeq2SeqAttn(input_vocab_size=256,
                    target_vocab_size=256,
                    d_model=512,
                    n_encoder_layers=2,
                    n_decoder_layers=2,
                    n_attention_heads=1,
                    attention_dropout=0.0,
                    mode='train'):
  input_encoder = tl.Serial(
      tl.Embedding(input_vocab_size, d_model),
      [tl.LSTM(d_model) for _ in range(n_encoder_layers)],
  )
  pre_attention_decoder = tl.Serial(
      tl.ShiftRight(mode=mode),
      tl.Embedding(target_vocab_size, d_model),
      tl.LSTM(d_model),
  )
  def PrepareAttentionInputs():
    def F(encoder_activations, decoder_activations, input_tokens):
      keys = values = encoder_activations
      queries = decoder_activations
      mask = (input_tokens != 0)
      mask = jnp.reshape(mask, (mask.shape[0], 1, 1, mask.shape[1]))
      mask = mask + jnp.zeros((1, 1, decoder_activations.shape[1], 1))
      return queries, keys, values, mask
    return tl.Fn('PrepareAttentionInputs', F, n_out=4)
  return tl.Serial(              
      tl.Select([0, 1, 0, 1]),   , in-toks, target-toks
      tl.Parallel(input_encoder, pre_attention_decoder),
      PrepareAttentionInputs(),  
      tl.Residual(
          tl.AttentionQKV(d_model, n_heads=n_attention_heads,
                          dropout=attention_dropout, mode=mode)
      ),                         
      tl.Select([0, 2]),         
      [tl.LSTM(d_model) for _ in range(n_decoder_layers)],
      tl.Dense(target_vocab_size),
      tl.LogSoftmax()
  )
def LSTMSeq2SeqAttn ( input_vocab_size = 256 , target_vocab_size = 256 , d_model = 512 , n_encoder_layers = 2 , n_decoder_layers = 2 , n_attention_heads = 1 , attention_dropout = 0.0 , mode = 'train' ) : input_encoder = tl . Serial ( tl . Embedding ( input_vocab_size , d_model ) , [ tl . LSTM ( d_model ) for _ in range ( n_encoder_layers ) ] , ) pre_attention_decoder = tl . Serial ( tl . ShiftRight ( mode = mode ) , tl . Embedding ( target_vocab_size , d_model ) , tl . LSTM ( d_model ) , ) def PrepareAttentionInputs ( ) : """Layer that prepares queries, keys, values and mask for attention.""" def F ( encoder_activations , decoder_activations , input_tokens ) : keys = values = encoder_activations queries = decoder_activations mask = ( input_tokens != 0 ) mask = jnp . reshape ( mask , ( mask . shape [ 0 ] , 1 , 1 , mask . shape [ 1 ] ) ) mask = mask + jnp . zeros ( ( 1 , 1 , decoder_activations . shape [ 1 ] , 1 ) ) return queries , keys , values , mask return tl . Fn ( 'PrepareAttentionInputs' , F , n_out = 4 ) return tl . Serial ( tl . Select ( [ 0 , 1 , 0 , 1 ] ) , tl . Parallel ( input_encoder , pre_attention_decoder ) , PrepareAttentionInputs ( ) , tl . Residual ( tl . AttentionQKV ( d_model , n_heads = n_attention_heads , dropout = attention_dropout , mode = mode ) ) , tl . Select ( [ 0 , 2 ] ) , [ tl . LSTM ( d_model ) for _ in range ( n_decoder_layers ) ] , tl . Dense ( target_vocab_size ) , tl . LogSoftmax ( ) )
Returns an LSTM sequence-to-sequence model with attention.
Returns an LSTM sequence - to - sequence model with attention .
"""Returns an LSTM sequence-to-sequence model with attention.

  The input to the model is a pair (input tokens, target tokens), e.g.,
  an English sentence (tokenized) and its translation into German (tokenized).

  The model works as follows:

  * Input encoder runs on the input tokens and creates activations that
    are used as both keys and values in attention.
  * Pre-attention decoder runs on the targets and creates
    activations that are used as queries in attention.
  * Attention runs on the queries, keys and values masking out input padding.
  * Decoder runs on the result, followed by a cross-entropy loss.

  Args:
    input_vocab_size: int: vocab size of the input
    target_vocab_size: int: vocab size of the target
    d_model: int:  depth of embedding (n_units in the LSTM cell)
    n_encoder_layers: int: number of LSTM layers in the encoder
    n_decoder_layers: int: number of LSTM layers in the decoder after attention
    n_attention_heads: int: number of attention heads
    attention_dropout: float, dropout for the attention layer
    mode: str: 'train', 'eval' or 'predict', predict mode is for fast inference

  Returns:
    An LSTM sequence-to-sequence model with attention.
  """ """Layer that prepares queries, keys, values and mask for attention.""" # Mask is 1 where inputs are not padding (0) and 0 where they are padding. # We need to add axes to the mask for attention heads and decoder length. # Broadcast so mask is [batch, 1 for heads, decoder-len, encoder-len]. # in-toks, target-toks # in-toks, target-toks, in-toks, target-toks # q, k, v, mask, target-toks # decoder-vecs, mask, target-toks # decoder-vecs, target-toks
An LSTM sequence-to-sequence model with attention.
An LSTM sequence - to - sequence model with attention .
input_vocab_size
vocab size of the input
vocab size of the input
target_vocab_size
vocab size of the target
vocab size of the target
d_model
depth of embedding (n_units in the LSTM cell)
depth of embedding ( n_units in the LSTM cell )
n_encoder_layers
number of LSTM layers in the encoder
number of LSTM layers in the encoder
n_decoder_layers
number of LSTM layers in the decoder after attention
number of LSTM layers in the decoder after attention
n_attention_heads
number of attention heads
number of attention heads
attention_dropout
float, dropout for the attention layer
float dropout for the attention layer
mode
'train', 'eval' or 'predict', predict mode is for fast inference
' train ' ' eval ' or ' predict ' predict mode is for fast inference