How to select proper capacitors and capacitance for LM78xx voltage regulators?
Within reason, too large is not a problem. Also usually this kind of regulator will actually work fine with no capacitors (advertised as such on some datasheets) under some conditions, but I don't recommend it (maybe if you're just powering an LED). Other parts (eg. LM7905, many LDOs) are about guaranteed to oscillate without the proper output cap. Most commercial designs will use a somewhat higher values than the datasheet example (not really 'recommended') because there is no cost penalty and 0.33uF is not very common. I don't think I've ever seen the 330nF outside of hobby/internet hacks. Similarly 1uF is perhaps more common on the output these days- there is little cost penalty and it's a bit better. On the other hand, these parts are mostly used at relatively low current these days because switching parts are better suited once you get into the hundreds of mA or higher, so 0.1uF has as much effect as 1uF at much higher current. 
Too big a capacitor on the output might affect startup because the regulator can only supply ~1.5A to charge the capacitor, otherwise larger is somewhat better all other things being equal. If the ESR is larger a large output cap may not be that much help. Low ESR is not a problem. A 1uF ceramic (or 0.1uF) works well and is almost free. Here's a plot of output Z vs frequency- at a few hundred kHz the output cap starts to have an effect. Below that, no capacitor is typically as good!





The 3300uF is probably there as a mains filter, not anything to do with the regulator directly. If the input voltage drops too low the regulator cannot regulate so the input reservoir capacitor has to be adequate for the lowest input mains voltage, highest load and end of life capacitance value. The input ripple also gets passed to the output to some degree (-54 to -80dB - see datasheet). For the regulator's stability purposes, a small ceramic or electrolytic capacitor close to the regulator (within a couple cm) is fine. A 47uF/50V 105°C electrolytic for through hole or another 1uF ceramic works fine. If the input voltage gets too close to the point where it drops out then performance will tend to suffer even before it drops out of regulation- 3V is good.
Low ESR on the output caps does not make the regulator more immune to HF oscillation. It will generally improve the transient response to high frequency load changes. It's not unusual to have several uF of low-ESR ceramic caps distributed around a board near various chips. Several dollars? $2.61 will buy 1,000 0.1uF 10V X5R capacitors at Digikey if you don't mind compact 0402 parts. 
Yes, both some negative and most LDO regulators are sensitive to ESR range- too high or too low and some will or may oscillate under some conditions. The LM780x is not an LDO regulator- it's actually quite difficult to convince it to oscillate. The LM790x, despite the similar name and the fact it's not an LDO is very easy to make oscillate (mostly because the output is taken from the collector of an NPN transistor rather than the emitter of a PNP as it would be if it was a true complementary part).