BERT Language Model and Gene Sequences - How Do I Relate Clusters of Sequences? I hope you'll indulge a question from a computer scientist with limited bioinformatics knowledge.   I've been working with the Google tool for language modeling called BERT.   It's generally regarded as state of the art when encoding language.   It allows you to say that the sentences "I went to the bank to cash my check." and "My wife is going to the Credit Union to pay the mortgage" are more similar than if compared to "he was fishing and fell off the bank."   In short, BERT takes sentences and encodes them so that related concepts are near each other and unrelated concepts are farther apart.  

BERT is an amazing tool and I wondered if it could successfully model the language of the genome if it were encoded into "sentences" (I chose 7-mer words and sentences of length 64, arbitrarily).  Below is the result of encoding the full genomes of human, bonobo, gorilla, chimp, and orangutan.  There are about 270K "sentences" here.   You can see there are four major blobs, with indications of small tight clusters.   BERT has decided that the points indicated in red are somehow semantically related.  I think is is significant, or at the very least interesting.   

  

My first thought was that clustering was trivial degenerate cases.  Maybe all the "AAAAAAA..." were put near each other, and all the "TTTTTTTTT...." etc.  I printed out samples from the clusters and visually that is not the case.  Also I've made histograms of the average Levenshtein distance in groups.  Within a tight group, the LD is statistically much lower (but not trivially low) as compared to unclustered regions (the big blob is all unclustered).



If there is a biological basis for the clustering, then new sequences could be similarly transformed, and classified according to cluster membership (and related biological function).  This exhausts my biological knowledge.  I've taken sequences from clusters and used BLAST to try and get some insight but I'm way out of my comfort zone with that.

My primary question is, given a set of sequences (in my case len=448), how do you figure out what they all have in common (given my hypothesis is that BERT says they are related based on proximity)?    And my backup question is, do you think this is interesting or a waste of time?
The UMAP results look very pretty :)

If I were to try to make sense of this blob, I'd try to do the following.


Pull out Gene Ontology (GO) terms for a list of candidate genes (you can try to subset Uniprot / Swissprot for some examples). Then I feed through some genes through your fragmented BERT kmer approach to see where they land. I'd be particularly interested in seeing where transmembrane proteins and enzymes land here.
I'd try to color this by introns vs exons (you should be able to extract this sort of information from your genome files). There is a good chance that the difference between introns / exons could explain some of these differences.
It maybe worth looking into genes that only encode RNAs vs genes that encode proteins.