My chain is catching on a part for which I don't even know the name
The part is a front derailleur. It job is to shift the chain between the front gears. Change gears while looking at it, you'll notice it moves and pushes the chain between sprockets
You need to adjust it, removing it would actually require more effort since it will need you to delink the chain somewhere. Also, removing would remove your gear shifting functionality. 
What you're describing is a common thing that happens to drive trains over time. The cable attaching your gear shifter on your handle to the derailleur near the pedals, has either stretched if it was low quality, or slipped a little at the derailleur. The fix is quite simple. All you need to do is locate the point at which your cable attaches itself to the derailleur(you can do it by following the black tube coming out of your shifter on the handlebar, along the frame of your cycle and ending somewhere opposite the derailleur on the other side of the cycle) It looks somewhat like this- .
Shift your bike down to the first gear. Then locate the screw that holds the metallic cable coming out of the aforementioned black tube and loosen it enough that the cable can slide around just a little(In the image above its the screw which the right end of the double arrow is pointing at).
This is the tricky part, you might want to ask a friend to help if you don't have fancy tools. Move the derailleur like as if it were shifting the chain to the second gear.This might bend your chain a little, that's OK. Notice the stretch of cable visible and marked in the picture would have become noticeably slack. While still holding the derailleur in its forced position, pull your cable taught by pulling the end beyond the screw. Hold the point where the cable is attached with your fingers while letting go of the derailleur. Make sure you don't let that point move, it will take some effort, they make the springs rather tight. Now slowly let the cable loose again by moving the attachment point in the direction the spring is pulling it till the derailleur is exactly centered on the chain, a little closer to second gear position wouldn't hurt though. Holding the attachment point in position, tighten that screw holding the cable.
As a safety measure so that the next time your cable loosens, the chain doesn't fall off, you could adjust the minimum position screw as described below-
Flip the bike right side up. On the top of the derailleur attachment, you'll find two small identical screws right next to each other. Something like this-

These screws prevent your derailleur from moving beyond its maximum and minimum positions, it seems in this case one of them came loose and shifted letting the derailleur push the chain off the sprocket.
With the cycle in third gear, locate which one of the screws is closer to the obstruction below it (You can tell what this obstruction is by forcefully trying to shift past the third gear like as if there were a 4th gear. You'll notice the obstruction in question is pressed against the end of one of the screws when you do this). Once you've done so, pick the OTHER screw and tighten it till the point that it just touches the same obstruction when the cycle is in first gear.  


You're all set to go now 

EDIT:
Since I'm a noob to this forum and don't have commenting privileges yet, I'll put my comment to your update post here.
I only saw the original post which is why i was confused about what was wrong and I mistakenly assumed it to be the extremely common problem of the cable slipping a little due to use.
What you did in the update to your post was actually the correct solution, just not enough. Loosen the screw holding the derailleur to the frame and turn it a little more till the derailleur is inline with the chain(not just clearing it). If the inner edge of it is touching now, follow my instructions above for changing the lower limit, but change the upper limit instead (ie the screw which is closer to the obstruction in the top gear) till the derailleur is centered on the chain in third gear.