Neanderthal and modern human population sizes
Briggs et al. 2009, using mtDNA report an estimate of 3500 females. They failed to offer any estimate of the male population size. Their estimates is much  lower than previously thought. They report two hypotheses of why their estimate is so low:

Hypothesis 1 - Low population size

H. neanderthalis population size was indeed really low. After all, they mostly had to live in harsh ice age conditions.

Hypothesis 2 - Slow decline in population size

H. neanderthalis population size were quite high at some point and their population size slowly decreased in size up to the point of being completely wiped out. Such low decline in population size may explain the low level of genetic diversity observed (yielding to an estimate of small population size) in their sample. If the decline of H. neanderthalis was sudden, then even the last survivors of the lineage would have shown quite a high level of genetic diversity. Their decline may have started about 40,000 years with the arrival of Homo sapiens in Europe but this is just a speculation. This second hypothesis is congruent with the evidence of a rather rapid rate of evolution observed in H. neanderthalis

In short, we don't really know what the population size of H. neanderthalis was but it might have been as low as 3500 females only