Are there any consistent rules as to how to distinguish words in Japanese?


I got involved with tokenization (mecab, kuromoji) and got confused as to what
a "word" is in the Japanese language.

**Scenario 1:**

"たべません" analyzed by a tokenizer, outputs the following tokens:

[code]

    たべ  動詞,自立,*,*   たべる
    ませ  助動詞,*,*,*   ます
    ん   助動詞,*,*,*   ん
    
[/code]

While the result outputs 3 token, I think generally it would be considered a
single word. In this case the word "たべません" would consist of a verb and 2
auxilary verbs (助動詞) attached to it.

**Scenario 2:**

"たべています" analyzed by a tokenizer, outputs the following tokens:

[code]

    たべ  動詞,自立,*,*   たべる
    て   助詞,接続助詞,*,* て
    い   動詞,非自立,*,*  いる
    ます  助動詞,*,*,*   ます
    
[/code]

I think usually "たべています" would be considered as two seperate words "たべて" and
"います". In this case the word "たべて" consists of a verb and a particle (助詞) and
"います" consists of a dependend (非自立) verb with an auxilary verb attached to it.

**Scenario 3:**

"たべてて" (a contraction of "たべていて") analyzed by a tokenizer, outputs the
following tokens:

[code]

    たべ  動詞,自立,*,*   たべる
    て   動詞,非自立,*,*  てる
    て   助詞,接続助詞,*,* て
    
[/code]

While I think that "たべていて" would be seperated as "たべて" and "いて", I have no
clue what to do with "たべてて". Following the logic from scenario 2, I would say
we should split at the てる as it is a dependent verb, just like the "います" was
in scenario 2. On the other hand it seems inconsistent to me, that we would
end up with "たべ" by itself...

**Scenario 4:**

"たべながら" analyzed by a tokenizer, outputs the following tokens:

[code]

    たべ  動詞,自立,*,*   たべる タベ  タベ
    ながら 助詞,接続助詞,*,* ながら ナガラ ナガラ
    
[/code]

I think usually "たべながら"would be considered as two seperate words "たべ" and
"ながら". However this would contradict the logic in scenario 1, since "ながら" is
classified as particle (助詞),just like "て".

**Conclusion and Problem**

As pointed out in scenario 3 and 4, I can't come up with a good and consistent
way to split "words" in Japanese.

I want to programmatically process big amounts of text with meaningful
results. If I consider every token a single word, the data would end up beeing
confusing and unpresentable. On the other hand if I include every particle and
helper verb into a single word, I would end up with far too many unique words.

I am hoping you can show me a middle ground between both extremes.

Is my intuition bad as far as the suggested "splits" go? Am I missing some
important rule here? Are the tokenizer results debatable? Or is there simply
no hope?




You are seeing the difference between 日本語文法 taught at Japanese-as-a-second-
language (JSL) school and 国語文法/学校文法 used by native Japanese speakers. The
outputs of tokenizers made by Japanese developers naturally follow the latter.
For instance, 食べません is generally considered a single (conjugated) word by JSL
learners, but it's generally considered as having three words in Japan.
Japanese monolingual dictionaries like 広辞苑 treat て/で, ます, ん and so on as
"words", not part of a conjugation. There is no such a thing as te-form in
国語文法.

  * [Auxiliary verbs in Japanese](https://japanese.stackexchange.com/q/63351/5010)
  * [Does "te-form" of a verb always include て/で? Why?](https://japanese.stackexchange.com/q/36310/5010)  
(This question was made by me, a native Japanese speaker, when I was ignorant
of the difference)

  * [Random Japanese / 活用](https://grapefruitcake.tumblr.com/post/145712193913/%E6%B4%BB%E7%94%A8-conjugation)
  * [Isn't a tag "auxiliary verbs" or "jyodoushi" necessary?](https://japanese.meta.stackexchange.com/q/1858/5010)

* * *

Regarding 食べてて, it's a contracted form of 食べていて, which is _four_ words
according to 国語文法:

  * 食べ: [連用形](https://japanese.stackexchange.com/a/65953/5010) of the verb 食べる
  * て: 接続助詞
  * い: 連用形 of the verb いる (used as a subsidiary verb to express progressive aspect)
  * て: 接続助詞

The contracted form 食べてて no longer includes い, but it is semantically
important, and a tokenizer cannot ignore it just because it's visually
nonexistent. To address this issue, the tokenizer introduced a "utility" or
"quasi-auxiliary" verb `てる`, which is essentially a single-word shortened
version of two-word `て + いる`. With this, 食べてて can be cleanly tokenized without
losing or adding something:

  * 食べ: 連用形 of the verb 食べる
  * て: 連用形 of the "utility verb" てる
  * て: 接続助詞

I think similar "utility verbs" are used to tokenize other contractions like
(食べ)とく, (食べ)ちゃう and (食べ)たげる. (I tentatively called them "utility verbs", but
there may be a better name.) Note that てる, とく, ちゃう, たげる and so on belong to no
traditional word class, just as you cannot classify English contractions like
_d'you_ , _it'll_ and _aren't_.




You are seeing the difference between 日本語文法 taught at Japanese-as-a-second-
language (JSL) school and 国語文法/学校文法 used by native Japanese speakers. The
outputs of tokenizers made by Japanese developers naturally follow the latter.
For instance, 食べません is generally considered a single (conjugated) word by JSL
learners, but it's generally considered as having three words in Japan.
Japanese monolingual dictionaries like 広辞苑 treat て/で, ます, ん and so on as
"words", not part of a conjugation. There is no such a thing as te-form in
国語文法.

  * [Auxiliary verbs in Japanese](https://japanese.stackexchange.com/q/63351/5010)
  * [Does "te-form" of a verb always include て/で? Why?](https://japanese.stackexchange.com/q/36310/5010)  
(This question was made by me, a native Japanese speaker, when I was ignorant
of the difference)

  * [Random Japanese / 活用](https://grapefruitcake.tumblr.com/post/145712193913/%E6%B4%BB%E7%94%A8-conjugation)
  * [Isn't a tag "auxiliary verbs" or "jyodoushi" necessary?](https://japanese.meta.stackexchange.com/q/1858/5010)

* * *

Regarding 食べてて, it's a contracted form of 食べていて, which is _four_ words
according to 国語文法:

  * 食べ: [連用形](https://japanese.stackexchange.com/a/65953/5010) of the verb 食べる
  * て: 接続助詞
  * い: 連用形 of the verb いる (used as a subsidiary verb to express progressive aspect)
  * て: 接続助詞

The contracted form 食べてて no longer includes い, but it is semantically
important, and a tokenizer cannot ignore it just because it's visually
nonexistent. To address this issue, the tokenizer introduced a "utility" or
"quasi-auxiliary" verb `てる`, which is essentially a single-word shortened
version of two-word `て + いる`. With this, 食べてて can be cleanly tokenized without
losing or adding something:

  * 食べ: 連用形 of the verb 食べる
  * て: 連用形 of the "utility verb" てる
  * て: 接続助詞

I think similar "utility verbs" are used to tokenize other contractions like
(食べ)とく, (食べ)ちゃう and (食べ)たげる. (I tentatively called them "utility verbs", but
there may be a better name.) Note that てる, とく, ちゃう, たげる and so on belong to no
traditional word class, just as you cannot classify English contractions like
_d'you_ , _it'll_ and _aren't_.

