The development of rhotic vowels in Mandarin
BTW, I have Schuessler's book, and the transcription shown in parentheses after the character indicates the Middle Chinese rather than Old Chinese pronunciation, so ńźi refers to MC rather than OC pronunciation.

Back to your question, it's important to note that initial ńź- is being used as a transcription rather than an indication of the actual pronunciation.  According to A Handbook of Old Chinese Phonology, p. 55, Karlgren chose ńź- to transcribe this sound in order to reflect the fact that it evolved into a voiced fricative in descendants such as Mandarin (pinyin r- is often analyzed as IPA /ʐ/, a voiced retroflex sibilant); however, the book notes that "for Early Middle Chinese, however, it is widely agreed that it was simply a palatal nasal (IPA: /ɲ/)."

As you noted, this Early Middle Chinese /ɲ/ became /j/ in Cantonese, and /n/ in Japanese.  Such phonological changes are not too hard to imagine occurring, though EMC /ɲ/ changing to Mandarin's retroflex /ʐ/ may be a little harder to imagine.  However, it is interesting to note that in Late Middle Chinese, all palatal sibilants merged into the retroflex sibilants (e.g., /tɕ/ -> /ʈʂ/, /dʑ/ -> /ɖʐ/, etc.).  At this time the palatal /ɲ/, while not a sibilant, also got carried along with this change and became retroflex in LMC.  After that, metathesis occurred which changed /ʐɨ/ to /ɨʐ/ (see 《北京音系解析》, p. 24), which is realized as [əɻ] (pinyin: er) in modern Mandarin.

As an interesting aside, phonological changes often have exceptions, so a few words continued to retain the initial nasal in their modern descendants, the best example being 爾/尔 (MC: ńźje, Mand.: ěr, Cant.: ji5), whose colloquial pronunciation retained the nasal initial and became modern 你 (Mand.: nǐ, Cant.: nei5).