What should I do about a switch that feels warm and arcs when toggled?
It's good that you found this, in the worst case, this situation can lead to an electrical fire. 

Check Connections

First thing to do is turn off the power, and check all the mechanical connections at the switch. If this is aluminum wiring please comment and I'll update the post, as there are other considerations if so.

Are there wires doubled up on a screw (they should not be)? Is any of the wiring loose? The wire should be wrapped around the screw or connected to the pigtails using wire connectors (depending on type of switch) like this:

  

Check for overloading

Calculate the total current draw on this circuit (controlled by the switch) by adding up the wattage of each bulb. This must not exceed the rating of the circuit, switch and wiring. 

Most single pole switches are rated at 15A (1800W at 120V) but older ones may be less. Dimmer switches are also often less, often at 600W. 

If you calculate more than 1400W, post a comment and I'll add some additional detail to consider about overall circuit loading and wire sizes.

Replace switch

If the switch is particularly old or feels loose or "mushy", simply replace it with a new one. 

I'd actually replace it no matter what, since it's shown signs of overheating which could at least mean it's been damaged (even if the switch itself wasn't the initial cause).

You might also consider 'spec' or 'commercial' grade which are slightly more expensive but are much higher quality and will last much longer.