Effect on output voltage when voltage regulator caps are undersized
That is actually a very broad question but here are a few pointers. 

Let us consider your typical regulator circuit.







simulate this circuit &ndash; Schematic created using CircuitLab

The voltage out of the regulator is defined for a specific range of input voltages. Too low a voltage and the output will basically just be a voltage drop over the input, above that, up to the destruction point of the regulator, the output will vary slightly with input voltage within the tolerance of the specified regulated voltage.

As such, the more stable the input voltage is, the more stable the output voltage. If the source on the left is a full wave rectified voltage, you want C1 to be large to minimize the amount of ripple that gets passed through the regulator. If however, it is coming from an already regulated supply, say your 12V rail, the capacitor can be much smaller.

The output capacitor C2 is more of a load leveler. It will help supply current when your load switches from low current to high current and gives the regulator time to adapt to the new demand. If your load is switching over a wide range, and often, you want this capacitor to be larger.

To complicate matters further, the regulator will pass that surge back to the input side, pulling C1 lower in turn. 

However, too large a capacitor on the output will cause the regulator to stay in current limit longer when the circuit initially powers up since the large charge current will be longer. Similarly, if the input capacitor is really large, it will take longer for the regulator to come up to the required output voltage. Both of these can be problematic if the attached circuitry partially functions during that transition.

As such, the balance of those capacitor sizes is, to a great extent, dependent on the nature of the application. The recommended minimums should always be adhered to "guarantee" stability, but you really need to revisit the design later with the functional prototype.