im: int = 64
num_layers: int = 20
embedding_dim: int = 768
num_embeddings = 77
additional_embeddings = 4
dropout: float = 0.0

)


Parameters 

num_attention_heads (int, optional, defaults to 32) — The number of heads to use for multi-head attention.


attention_head_dim (int, optional, defaults to 64) — The number of channels in each head.


num_layers (int, optional, defaults to 20) — The number of layers of Transformer blocks to use.


embed