Discrete Fourier transform - finding the fundamental quickly?
What you are describing is very similar the the Harmonic Product Spectrum method of pitch estimation, as listed in this Stanford CCRMA paper.

An FFT does not give you an "infinite sum of amplitudes", but a finite number of result bins depending on the length of the FFT.

5 mS is only 1 period of a 200 Hz note, and only a fraction of a period below 200 Hz.  Musical pitch recognition usually requires hearing or analyzing a multiple number of periods of the periodicity of a pitched sound.  And a lot of music uses notes below G2. If you have a sufficient length of data, computing a pitch estimate from that data might take only on the order of microseconds rather than milliseconds on a modern PC or mobile device.