In the [famous work on the Visual Transformers](https://arxiv.org/abs/2010.11929), the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform classification, a **CLS** token is added at the beginning of the resulting sequence:
$$
[\textbf{x}\_{class}, \textbf{x}\_{p}^{1}, \ldots, \textbf{x}\_{p}^{N}]
,$$
where $ \textbf{x}\_{p}^{i}$ are image patches. There multiple layers in the architecture and the state of the **CLS** token on the output layer is used for classification.

I think this architectural solution is done in the spirit of NLP problems (BERT in particular). However, for me, it would be more natural not to create a special token, but perform *1d Global Pooling* in the end, and attach an `nn.Linear(embedding_dim, num_classes)` as more conventional CV approach.

Why it is not done in this way? Or is there some intuition or evidence that this would perform worse than the approach used in the paper?