How do the linear layers in the attention mechanism work?
Your understanding is not correct.
The relevant information is described in the original paper in section 3.2.2:
The three sets of projection matrices you are referring to are $W^Q_i \in \mathbb{R}^{d_{model} \times d_k}$ for the Queries, $W^K_i \in \mathbb{R}^{d_{model}\times d_k}$ for the Keys and $W^V_i \in \mathbb{R}^{d_{model}\times d_v}$ for the Values. Notice that the $i$ subindex in the matrix names refers to the attention head, indicating that there is a different matrix for each attention head.
The final projection matrix is $W^O \in \mathbb{R}^{hd_v \times d_{model}}$.
Given the number of attention heads, $h = 8$, the dimensions of the matrices are defined by $d_k=d_v=d_{model}/h=64$.
The three sets of matrices project the embedding dimensionality $d_{model}$ into a space with 8 times smaller dimensionality ($d_k=d_v=d_{model}/8$). However, note that for each of $W^K$, $W^V$ and $W^Q$ there are 8 matrices (one per attention head) and, analogously, 8 scaled dot products are computed. The results of the dot products are 8 vectors of dimensionality $d_{model}/8$; those 8 vectors are concatenated (see figure below), giving a tensor with the original dimensionality $d_{model}$.
Then, the final matrix multiplication by $W^O$ doesn't change the dimensionality, obtaining again the original one.


About your second question, either of the approaches you describe (averaging and using an RNN) is technically feasible, but what people normally do when using transformers for classification is to use BERT's approach, that is, adding a special token [CLS] at the beginning of the sequence and using the output at that position for the classification, just projecting with a matrix multiplication into a space with dimensionality equal to the number of classes.