How does Byte Pair Encoding work? I am using this to do some Byte Pair Encoding (BPE). My corpus looks like this. 

When I run the learn_bpe, I get a vocabulary that looks like this.

e r
r e
o n
o r
t i
) ;&lt;/w&gt;
a c
n t
' ,&lt;/w&gt;
er r
a l
r o
h e
m e


When I try to combine it again to see if it worked with ubword-nmt apply-bpe -c data/jsvocab.txt &lt; data/javascript.txt &gt; tst.txt, the resulting file has a lot of strange @ characters. 

const p@@ re@@ F@@ or@@ m@@ at@@ t@@ e@@ d@@ B@@ l@@ o@@ c@@ k@@ N@@ a@@ me@@ s = {
    '@@ ap@@ i@@ -@@ p@@ ro@@ j@@ ect@@ s@@ '@@ : '@@ A@@ P@@ I P@@ ro@@ j@@ ect@@ s@@ ',
    '@@ b@@ a@@ s@@ i@@ c@@ -@@ c@@ ss@@ '@@ : '@@ B@@ a@@ s@@ i@@ c C@@ S@@ S@@ ',
    '@@ b@@ a@@ s@@ i@@ c@@ -@@ h@@ t@@ m@@ l@@ -@@ and@@ -@@ h@@ t@@ m@@ l@@ 5@@ '@@ : '@@ B@@ a@@ s@@ i@@ c H@@ T@@ M@@ L an@@ d H@@ T@@ M@@ L@@ 5@@ ',
    '@@ c@@ ss@@ -@@ f@@ le@@ x@@ b@@ o@@ x@@ '@@ : '@@ C@@ S@@ S F@@ le@@ x@@ b@@ o@@ x@@ ',
    '@@ c@@ ss@@ -@@ g@@ r@@ i@@ d@@ '@@ : '@@ C@@ S@@ S G@@ r@@ i@@ d@@ ',
    de@@ v@@ o@@ p@@ s@@ : '@@ D@@ e@@ v@@ O@@ p@@ s@@ ',
    e@@ s@@ 6@@ : '@@ E@@ S@@ 6@@ ',
    '@@ in@@ f@@ or@@ m@@ ation@@ -@@ se@@ c@@ ur@@ i@@ t@@ y@@ -@@ w@@ i@@ th@@ -@@ he@@ l@@ me@@ t@@ j@@ s@@ '@@ : '@@ I@@ n@@ f@@ or@@ m@@ a@@ ti@@ o@@ n S@@ ec@@ ur@@ i@@ t@@ y w@@ i@@ t@@ h H@@ e@@ l@@ me@@ t@@ J@@ S@@ ',
    j@@ q@@ u@@ er@@ y@@ : '@@ j@@ Q@@ u@@ er@@ y@@ ',
    '@@ j@@ s@@ on@@ -@@ ap@@ i@@ s@@ -@@ and@@ -@@ a@@ j@@ a@@ x@@ '@@ : '@@ J@@ S@@ O@@ N A@@ P@@ I@@ s an@@ d A@@ j@@ a@@ x@@ ',
    '@@ m@@ on@@ g@@ o@@ d@@ b@@ -@@ and@@ -@@ m@@ on@@ g@@ o@@ o@@ se@@ '@@ : 'M@@ on@@ g@@ o@@ D@@ B an@@ d M@@ on@@ g@@ o@@ o@@ se@@ ',
    '@@ t@@ he@@ -@@ d@@ o@@ m'@@ : '@@ T@@ h@@ e D@@ O@@ M@@ ',
    '@@ ap@@ i@@ s@@ -@@ and@@ -@@ m@@ i@@ c@@ ro@@ serv@@ i@@ c@@ e@@ s@@ '@@ : '@@ A@@ P@@ I@@ s an@@ d M@@ i@@ c@@ ro@@ serv@@ i@@ c@@ e@@ s@@ ',
    '@@ ap@@ i@@ s@@ -@@ and@@ -@@ m@@ i@@ c@@ ro@@ serv@@ i@@ c@@ e@@ s@@ -@@ p@@ ro@@ j@@ ect@@ s@@ '@@ : '@@ A@@ P@@ I@@ s an@@ d M@@ i@@ c@@ ro@@ serv@@ i@@ c@@ e@@ s P@@ ro@@ j@@ ect@@ s@@ '
}@@ ;


And so on. I'm not sure what I'm missing, but it seems that it didn't fully reconstruct the text from the vocabulary?
Your BPE vocabulary is quite small given how the strings you want to segment look like. The biggest problem here is that BPE expects tokenized sentences with tokens separated by spaces. The BPE model is not aware of JS syntax and you feed it with JS code, so there is no wonder that it does not learn anything syntactically plausible.

You can interpret the BPE file a log of what subwords got merged and you can see that the longest string you get is something like:

m').valueOf
H:mm').valueOf


I would recommend some preprocessing, perhaps some JS lexer, so it respects the syntax and perhaps bigger vocabulary.