t (float, optional, defaults to 0.0) — The dropout probability to use.   act_fn (str, optional, defaults to "silu") — The activation function to use.   attention_head_dim (int, optional, defaults to 8) — The attention head dimension.   norm_num_groups (int, optional, defaults to 32) — The number of groups for normalization.   attn_norm_num_groups (int, optional, defaults to None) —
If set to an integer, a group norm layer will be created in the