Beginner safety questions regarding voltage and current
I assume it's safe to use the second charger because the phone will only take the current that it needs?


The second charger's current specification is lower anyway, so this isn't an issue. But your assumption is right -- devices draw the current they require.


  If the second charger had a voltage higher than the phone's charger, should it be avoided?


No, because this is USB where smart devices negotiate the voltage to be supplied.


  In general, how would I determine if something is safe to use or not?
  When do things get to the point of damaging the device, or even worse, starting a fire?


In general, you need to match the voltage and use a supply that can supply at least as much current as the device requires. But this is a special case because USB has a specific charging protocol where the devices negotiate the voltage to be supplied and the current to be drawn. Unless you have a device, charger, or cable that doesn't follow the USB specification, you don't have to worry.