Adding BERT embeddings in LSTM embedding layer
Instead of using the Embedding() layer directly, you can create a new bertEmbedding() layer and use it instead.



# Sample code
# Model architecture

# Custom BERT layer
bert_output = BertLayer(n_fine_tune_layers=10)(bert_inputs)

# Build the rest of the classifier 
dense = tf.keras.layers.Dense(256, activation='relu')(bert_output)
pred = tf.keras.layers.Dense(1, activation='sigmoid')(dense)

model = tf.keras.models.Model(inputs=bert_inputs, outputs=pred)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(...)


This article will walk you through the entire process of creating the custom BERT layer along with example code. Give it a read.