How does the trainable projection layer used in PRADO and pQRNN work?
The key idea in both the papers are word embeddings are replaced by trainable activations that are computed using a nueral network. The network (referred to as bottleneck layer) takes word projections as input. The parameters of this network is shared across tokens. This results in a trainable token representation. The projection itself is not trainable, but the representation derived from projection is trainable. Hope that clarifies.