Measuring power output of Thermoelectric Generator with multimeter
You might think you're measuring the open circuit voltage, and the short circuit current. Unfortunately, DMMs make much better voltmeters than they do ammeters. Depending on the selected range, your ammeter might have an input resistance comparable to the Peltier. Measure the short circuit current on several different ranges to see this effect.

It may be better to construct a load well below the Peltier resistance, say 100mohms, and measure the voltage across that load, at least then you know what your non-zero load resistance is. Better yet, make several low value loads, and extrapolate the readings down to zero ohms, to estimate the true s/c current.

Once you have the true o/c voltage and s/c current, your best power output will be at 25% of their product, into a load equal to the Peltier internal resistance.