How to JUST represent words as embeddings by pretrained BERT? I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding.
After the representation of words I want to clusterize it, find similar vectors (i.e. words). Maybe even then make a classification to a given classes (classification there unsupervised --- since I don't have labeled data to train on).
I know that almost any task can be solved &quot;inside&quot; BERT, i.e. using fine-tuning in final layer.
Since all described above, I have two QUESTIONS; answers/hints/anything really appreciated since i'm stuck on that:

How to just extract embeddings from BERT using some dictionary of words and use word representations for futher work?
Can we solve inside BERT using fine-tuning the next problem: a). Load dictionary of words into BERT b). Load given classes (words representing each class. E.g. &quot;fashion&quot;, &quot;nature&quot;). c) Make an unsupervised classification task?
BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a &quot;word-level&quot; representation.

You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-speech (POS) (e.g. this) tagging or named entity recognition (NER) (e.g. this). Normally, you associate the tag to either the first or the last subword token in the word. If you prepare a dataset that way, you could fine-tune BERT to perform word tagging with the classes you need. If you only have the words, you could find some text corpus (ideally of the intended domains) and apply the described data preparation process.