How to improve an elegant design for an LED dimmer, low-powered with LDO
Answering your questions in order:


and 2. Given I am using a battery, I figured there is no input capacitor necessary for the LDO. Is that correct?     


NO, that is not correct. You need both input and output capacitors for the regulator since it is an active device with loop gain. It is not like a resistor at all.    


Would the TP4056 (and/or solar panel) introduce noise anyway, which I would have to filter out with capacitors (input AND output)?      


It is unlikely to have appreciable noise from a solar panel, but the TP4056 IS an active device and specifies the use of input and output capacitors to control stability. 


Will the caps leak current all the time, wasting energy?    


YES, caps do leak, but for the values of ceramic caps you need we are talking pA. The self discharge rate of the battery you use is much much larger than the leakage current of the capacitors.   


Do you see a way of forgoing voltage regulation or to save energy in any other way in this circuit?     


A voltage regulator which tries to hold the voltage at a fixed level IS NOT the way to drive a high current LED. Given you are using a 350mA LED device, it is possible you will be using the short circuit limit in the regulator to drop the output voltage, especially when the LED gets hot. 

Suggestions:


Use a constant current driver to drive the LED directly from your battery. This will work even though you are using PWM to drive the LED for brightness control. You may be able to get by with just a series limiting resistor for the LED (especially if you are not approaching the 350mA LED limit), but the current will vary as the temperature of the LED increases.     
Change the 2N2222 to a FET, it will have a lower on voltage than the VCE(sat) of the transistor at 350mA. Right now the VCE(sat) of the 2N2222 is probably providing the current limitation of your circuit.


   

Try to understand the datasheet for the devices you use, clearly as shown above, VCE(sat) is the dominating concern at 350mA.


Don't second guess the datasheets. If they specify input an output capacitors, use them. If you are as you say new to electronics, the decisions you make may be poor and you simply don't understand the implications. In this case your understanding of the leakage of ceramic capacitors as a significant problem is a clear indication of your understanding level. You could start by reading this document from Murata.    


The worst capacitors for leakage are Aluminum Electrolytic, and here leakage current may be in the uA range, but still not significant for your application.