How to find optimal token set for compression?
Are there existing algorithms for it? Are there any algorithms related to the problem?



Theoretical one: https://en.wikipedia.org/wiki/Sequitur_algorithm
Practical one: https://encode.ru/threads/1909-Tree-alpha-v0-1-download