In attention models with multiple layers, are weight matrices shared across layers?
Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples:
ALBERT is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only $\left\{ W_i^Q, W_i^K, W_i^V \right\}$ across all Transformer layers for large networks either result in the same accuracy or slightly improved accuracy, while significantly reducing model size. Sharing the position-wise FFN layer though hindered performance.
Text-to-Text Transfer Transformer shows that sharing the weights between encoder and decoder layer of the transformer (so e.g. layer 1 encoding weights = layer 1 decoding weights) barely affected accuracy (it dropped by 0.5%), but the model size is halved.
I'm sure there are more papers I have forgotten. Sharing weights is still an active area of research, but for vanilla transformers it is assumed they do not share weights.