Effect of capacitor size on output voltage of a rectifier circuit
The only time-window in which the capacitor can receive charge is when the AC voltage is positive and, is slightly higher in amplitude than the capacitor voltage. Only then can current pass through the diode.

If you have a small value capacitor (1uF say), it gets discharged by the load more easily and, when that capacitor gets recharged, that time-window begins earlier on in the positive AC waveform hence, the smaller value capacitor has the luxury of being charged for longer compared a larger value capacitor. Once the AC voltage has reached a positive peak charging is restrained because the diode rapidly gets reversed biased: -




Red is the AC waveform peaking at 5 volts
Blue is the rectified output slightly smoothed by a 1 uF capacitor (4.384 volts)
Green is the current through the diode peaking at 4.623 mA


For the same load resistor, the larger value capacitor (100 uF) hardly gets discharged (as expected) and hence, the time-window in which it can receive charge begins much later in the AC waveform and therefore, has to have a much shorter duration: -




The current (green) peaks at 69.5 mA and of course this level of current causes the diode to drop more voltage. Also notice how the time-window for recharging the capcitor has become much smaller.
Blue (due to the diode drop) peaks at 4.347 volts (previously 4.384 volts) i.e. 30 mV less.


My simulation uses a 1N4003 diode and other diodes can produce greater volt drops under heavier load (the OP talks about 4.4 volts dropping to 4.1 volts).

So, as said, during that diminished charge period, the demand for current to recharge the bigger capacitor is significantly greater and that produces a larger volt drop across the diode and voltage is lost. In other words it can never attain the same peak DC value due to diode losses.

This shortened time-window in which the larger capacitor receives charge also accounts for the extended length of time that it requires to reach full charge (apart from the fact that bigger capacitors take longer to charge from a limited or restricted energy source anyway).