Optimisation of vocabulary learning


I have a question related to the creation of a rationalised list of vocabulary
working in pairs with kanji learning. If one wants to learn Japanese, at some
point there is no other choice than learning big lists of vocabulary, simply
because there is not much to relate to the the learner's mother-tongue (at
least if one's native language is not Chinese). We also encounter in most
kanji books lists of vocabulary associated to each kanji, and the average
number of words related to one kanji varies from books (say **4** to **20** ).
Therefore I asked myself what would be the optimal number of words that shall
be actually used.

So we have our list of **_N_** kanji, and for all **_i_ = 1, ... , _N_** , let
**_m_** _i_ the number of words associated the kanji number **_i_**. Now one
should be careful to make a list without repetitions, but at the same time
minimising the average of numbers associated to each kanji. So we have
arranged all words, by relating each to only one kanji, such that the mean

![image](https://i.stack.imgur.com/k1y6v.png)

is minimal (this condition will appear clearly later). Then we look for an
integer **_n_** corresponding to the number of words we learn per kanji, such
that the expectation of knowing an arbitrary word is high, but **_n_** is not
too high either (for example, if one has to learn **100** words per kanji,
this system does not work). Therefore, for a word **_w_** associated to the
kanji **_i_** (let us denote **_A_** _i_ the set of words associated to the
kanji **_i_** ), the probability to know the word is

![image](https://i.stack.imgur.com/wH8Wt.png)

with obvious abuse of notations. Therefore

![image](https://i.stack.imgur.com/bmAJ6.png)

and one would know approximately **_n_** / ** _m ̅_** of the vocabulary. And
**_n_** is approximately equal to the average **_m ̅_**, which explains why we
had to minimise this quantity.

Therefore my question is two-sided : what is the minimal value of **_m ̅_**?
Indeed, if we compare a simple kanji like 人 which has more than **2000**
matches and some rare kanji with one or two occurrences, it is not _a priori_
obvious if this quantity ( ** _m ̅_**) is large ( **>100** say) or not. I
should add that obviously if we consider all words in Japanese language, this
system is unrealistic, but for smaller ambitions, it could actually be useful
if the number **_m ̅_** is not too large. One could for example think about
the kanji and vocabulary lists for standardized international tests.




I've long been wanting to create a list of _kanji_ (say, all _jōyō kanji_ )
ordered by weight, where the weight is determined by the frequency of the
_kanji_ itself and by the weight of all those _kanji_ in which this _kanji_
appears as a radical. (Also see the end of this post.)

If the average person has a vocabulary of [at least 30,000
words](http://www.kecl.ntt.co.jp/icl/lirg/resources/goitokusei/goi-test.html),
a list of words covering a larger range (say 95%) of people is obviously much
larger than this. Maybe more like 50,000–70,000.

Even though a good portion of words has no (or only obsolete) _kanji_
representations, it seems impractical to try to create a reasonably complete
vocabulary list of 50,000 words, or even an "average" list of 30,000. (This
would require around 15–25 words per _kanji_.)

Still, you could choose a number (say 5) and create a list of 10,000 words (
_kanji_ -by- _kanji_ in the order their "weights") by choosing the five most
common words containing a particular _kanji_ , eliminating words that you
chose previously.

This seems to be a reasonable way to learn _kanji_ , as well as creating
vocabulary lists that are more tuned to a gradually increasing knowledge of
_kanji_. (Otherwise pure frequency lists would do the job.)

I also think it gives you what you want, because the resulting vocabulary list
of 10,000 words is essentially just a reordering of the same list ordered by
frequency. (This is not entirely true, as there are extremely rare _kanji_ ,
like 訃 (least frequent _kanji_ in BCCWJ), which probably don't have 5 words in
the top 10,000 words containing this _kanji_. It's basically a reordering with
holes on those words whose _kanji_ already have been represented at least 5
times in the list by more frequent words with the holes having been patched by
less frequent words with _kanji_ who haven't been represented.)

* * *

There are several ways in which the weights could be adapted to account for
more than just frequency:

  1. One could also take into account the stage at which individual _kanji_ are taught in Japanese schools in order to optimize the list not only for usage (in newspapers/books), but also exposure in existing learning materials, which are mostly based on the [_kyōiku kanji_](https://ja.wikipedia.org/wiki/%E5%AD%A6%E5%B9%B4%E5%88%A5%E6%BC%A2%E5%AD%97%E9%85%8D%E5%BD%93%E8%A1%A8).

  2. I would also like to associate to each _kanji_ all _kanji_ which contain the same phonetic component, even if the _on'yomi_ is slightly different, like 番 幡 or 読 続. One could also consider that such _kanji_ are easier to learn in groups and thus allow for the weights to be increased.

  3. One might also give a small preference to pictographs over ideographs etc.

(I have started this and if I were only half as tech-savvy as the people here
on Stack Exchange, I might have finished already.)




I've long been wanting to create a list of _kanji_ (say, all _jōyō kanji_ )
ordered by weight, where the weight is determined by the frequency of the
_kanji_ itself and by the weight of all those _kanji_ in which this _kanji_
appears as a radical. (Also see the end of this post.)

If the average person has a vocabulary of [at least 30,000
words](http://www.kecl.ntt.co.jp/icl/lirg/resources/goitokusei/goi-test.html),
a list of words covering a larger range (say 95%) of people is obviously much
larger than this. Maybe more like 50,000–70,000.

Even though a good portion of words has no (or only obsolete) _kanji_
representations, it seems impractical to try to create a reasonably complete
vocabulary list of 50,000 words, or even an "average" list of 30,000. (This
would require around 15–25 words per _kanji_.)

Still, you could choose a number (say 5) and create a list of 10,000 words (
_kanji_ -by- _kanji_ in the order their "weights") by choosing the five most
common words containing a particular _kanji_ , eliminating words that you
chose previously.

This seems to be a reasonable way to learn _kanji_ , as well as creating
vocabulary lists that are more tuned to a gradually increasing knowledge of
_kanji_. (Otherwise pure frequency lists would do the job.)

I also think it gives you what you want, because the resulting vocabulary list
of 10,000 words is essentially just a reordering of the same list ordered by
frequency. (This is not entirely true, as there are extremely rare _kanji_ ,
like 訃 (least frequent _kanji_ in BCCWJ), which probably don't have 5 words in
the top 10,000 words containing this _kanji_. It's basically a reordering with
holes on those words whose _kanji_ already have been represented at least 5
times in the list by more frequent words with the holes having been patched by
less frequent words with _kanji_ who haven't been represented.)

* * *

There are several ways in which the weights could be adapted to account for
more than just frequency:

  1. One could also take into account the stage at which individual _kanji_ are taught in Japanese schools in order to optimize the list not only for usage (in newspapers/books), but also exposure in existing learning materials, which are mostly based on the [_kyōiku kanji_](https://ja.wikipedia.org/wiki/%E5%AD%A6%E5%B9%B4%E5%88%A5%E6%BC%A2%E5%AD%97%E9%85%8D%E5%BD%93%E8%A1%A8).

  2. I would also like to associate to each _kanji_ all _kanji_ which contain the same phonetic component, even if the _on'yomi_ is slightly different, like 番 幡 or 読 続. One could also consider that such _kanji_ are easier to learn in groups and thus allow for the weights to be increased.

  3. One might also give a small preference to pictographs over ideographs etc.

(I have started this and if I were only half as tech-savvy as the people here
on Stack Exchange, I might have finished already.)

