title: How to save the best model of each epoch with transformers bert in tensorflow
body: <p>I uesd TFBertModel and Tensorflow model to combined and training with Hugging Face transformers.I want to save the best model of val_accuracy of each epoch.I used 'tensorflow checkpoint', but I got the error.How do I save the best model of each epoch with transformers bert in tensorflow?</p>
  <pre><code>from tensorflow.keras.callbacks import EarlyStopping
  from tensorflow.keras.initializers import TruncatedNormal
  from tensorflow.keras.losses import CategoricalCrossentropy,BinaryCrossentropy
  from tensorflow.keras.metrics import CategoricalAccuracy,BinaryAccuracy
  from tensorflow.keras.utils import to_categorical
  from tensorflow.keras.utils import plot_model
  from transformers import AutoTokenizer,TFBertModel
  import tensorflow as tf
  from tensorflow.keras.layers import Input, Dense
  tokenizer = AutoTokenizer.from_pretrained('bert-large-uncased')
  bert = TFBertModel.from_pretrained('bert-large-uncased')
  max_len = max_length
  input_ids = Input(shape=(max_len,), dtype=tf.int32, name=&quot;input_ids&quot;)
  input_mask = Input(shape=(max_len,), dtype=tf.int32, name=&quot;attention_mask&quot;)
  # embeddings = dbert_model(input_ids,attention_mask = input_mask)[0]
  
  
  embeddings = bert(input_ids,attention_mask = input_mask)[1] #(0 is the last hidden states,1 means pooler_output)
  # out = tf.keras.layers.GlobalMaxPool1D()(embeddings)
  out = tf.keras.layers.Dropout(0.1)(embeddings)
  
  out = Dense(128, activation='relu')(out)
  out = tf.keras.layers.Dropout(0.1)(out)
  out = Dense(32,activation = 'relu')(out)
  
  y = Dense(1,activation = 'sigmoid')(out)
      
  model = tf.keras.Model(inputs=[input_ids, input_mask], outputs=y)
  model.layers[2].trainable = True
  #model.save_weights('path/savefile')
  </code></pre>