How to Find the Average of the Input Vectors I want to find the average of input vectors. I tried to use tf.math.reduce_mean, but it went error.

If I have to use keras.layers.Average, I have to make a list of the hidden states.

Does anyone know how to solve this?

Using tf.math.reduce_mean

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+1,
                  output_dim=embedding_size,
                  input_length=max_length,
                  name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size,
                             return_sequences=True,
                             dropout=dropout_rate,
                             name="LSTM"),
                        name="Bi-LSTM")

hidden_states = encoder(embed)

# Average
average = tf.math.reduce_mean(hidden_states)

# Outputs
outputs = Dense(n_tags,
                activation="softmax",
                name="Output")(average)


Error:

ValueError: Input 0 is incompatible with layer Output: expected min_ndim=2, found ndim=0


Using Average layer

# Input
inputs = Input(shape=(max_length,), name="Input")

# Embedding
embed = Embedding(input_dim=n_words+1,
                  output_dim=embedding_size,
                  input_length=max_length,
                  name="Embedding")(inputs)

# Bi-LSTM
encoder = Bidirectional(LSTM(units=hidden_state_encoder_size,
                             return_sequences=True,
                             dropout=dropout_rate,
                             name="LSTM"),
                        name="Bi-LSTM")

hidden_states = encoder(embed)

# Average
average = Average()(hidden_states)

# Outputs
outputs = Dense(n_tags,
                activation="softmax",
                name="Output")(average)


Error:

ValueError: A merge layer should be called on a list of inputs.
Does this work for you? After the Bi-Directional LSTM layer, I sum the tensor over the batch axis and then divide by the batch size.

from tensorflow.keras import layers, Sequential
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
from tensorflow.keras.utils import plot_model

inputs = layers.Input(batch_shape=(4, 500), name="Input")
embed = layers.Embedding(input_dim=500, output_dim=30, name="Embedding")(inputs)
encoder = layers.Bidirectional(
                  layers.LSTM(units=20,return_sequences=True,dropout=0.3,name="LSTM")
                  ,name="Bi-LSTM")

hidden_states = encoder(embed)
total = layers.Lambda(lambda x: K.sum(x, axis=0)/K.cast(K.shape(x)[0],"float32"))(hidden_states)

model = Model(inputs, total)
model.summary()