Are all 110 million parameter in bert are trainable I am trying to understand are all these 110 million parameters trainable of bert uncased model. Is there any non trainable parameters in this image below?
By trainable I understand they are initialized with random weight and during pretraining these weights are backpropagated and updated.
All these parameters are trainable.
Note that in normal Transformers it is typical to have fixed (non-trainable) positional embeddings, but in BERT they are learned.
Note also the &quot;pooler&quot; component, which is an extra projection that was not mentioned in the paper, but which the authors commented on later.