Here, we report a new autoencoder (VQGAE) able to learn the vectorial representation of fragments in an unsupervised manner. Its architecture allows to obtained an atom order independent latent space. The latent vectors of VQGAE demonstrated high performance in both similarity search and QSAR benchmarking compared to fragment-based descriptors and latent vectors of previously developed architectures. A combination of VQGAE with the ordering network can be efficiently applied to the inverse QSAR task which was demonstrated on the example of A2A adenosine receptor inhibitors. We believe that this work opens new perspectives for exploring unsupervised learning of fragments. Follow-up studies are still needed to analyse learned fragment space and to improve generation ability of the proposed autoencoder.