optimal resistor for TP4056 with AMS1117 input
1: Yes, I assume you are using 5V version of 1117, so that brings your power dissipation to (InputVoltage-OutputVoltage)*Current=(12V-5V)*580mA=4W and that's a lot of Heat and you'll need good Heat Sinking.
My suggestion: Use DC-DC converter instead of Linear Regulator, use something like this one
http://uk.farnell.com/xp-power/tr05s3v3/dc-dc-converter-0-5a-3-3v-sip/dp/2319829 (Note this is a 3.3V version you have to search for 5V). or you can use circuit from Car Mobile charger which mostly has MC64063 based buck Converter.

2: Now you can figure the optimum Current to bring power dissipation down, I don't know what will be safe dissipation in current setup. you may need to somehow add a heat sink.

3: Power dissipation will only depend upon current drawn and voltage drop across 1117, capacity of battery doesn't matter here.

Now if you don't want to go with DC-DC converter since you already made PCB, I suggest that you can divide Power dissipation by putting a high Watt resistor in series with the Input of 1117.
Lets see for Eg:
Dropout Voltage for 1117 is 1.2 So we need to keep Input Voltage to 1117 atleast 6.2 V for 5V ouput.
So if your Total Current draw is let's say 600mA. and you have 12V supply you can drop few watts across series resistor and max we can drop is 5.8V (since we need 6.2 at 1117 Input) and So resistor=5.8V/600mA=9.66Ohm, so we choose 10Ohm and P.D across this resistor will be= 10Ohm * (600mA)^2=3.6 Watts. Hence you'll need atleast 5W resistor. and voltage input to 1117 will be 6V(since we chose 10 ohm resistor and hence Voltage drop across it is 6V)
P.D across 1117=6V-5V * 600mA= 600 mW.

Edit1: 6V input to 1117 instead of aforesaid 6.2V is also fine since datasheet says 1.2V drop at 800mA and our current is less.