Thoretically, what makes multi-pitch detection so difficult?
First of all:


  mostly the frequency with the largest magnitude is desired frequency


This is a very naive and flawed approach, as what you are measuring here is very different from the perceptual notion of pitch. Not to mention that you hit the limited resolution of FFT...

Some of the challenges one has to face in multi-pitch detection:


Octaves are difficult to tell apart. Play two Cs one octave apart, and the harmonics of the highest note reinforces the harmonics of the lowest note, rather than create new peaks in the spectrum, to the point that one could hear one single "C" note with a different timbre. When mixing several notes together, some of their partials can reinforce or cancel each other.
Pitch detection can no longer be seen as a problem of period estimation. The most successful monophonic pitch detection algorithms are time-domain and based on metrics like the average magnitude (or square) difference function, and the underlying assumption is that the signal has a salient periodicity - it is almost similar to itself when translated in time. These methods cannot be applied to polyphonic signals, as the period of a mixture of periodic signals is the LCD of the components' periods. Knowing this LCD does not give away the factors...
The fact that the number of notes is not known in advance increases the size of the search space, and makes it an ill-posed problem. You are no longer ranking hypothesis like "this is a C3" vs "this is a C#3", but instead, "this is a C3 and a G4" vs "this is just a C3" vs "this is a C2 and a C3". For example, take a single piano note. One could describe its spectra as a "single C3 piano note". Or one could describe its spectra as "a C3, a C4, a G4, etc... all played together by an instrument producing purely sinusoidal notes". Distinguishing among these two options requires prior knowledge of the spectral shape of a piano note.