Gmail in iPhone: How to delete on swipe?
Go to the iPhone Home screen. 
Open Settings. 
Select Mail, Contacts, Calendars. 
Open the desired Gmail account under Accounts. 
Make sure Archive Messages is OFF.


To make swiping archive mail (and remove it from the current folder, but keep it under "All Mail"), make sure Archive Messages is ON. 

Just an addition for iOS6: there is an alternative for swiping:
When Archive Messages is OFF, the Archive button turns into a trash can.
When Archive Messages is ON you can press and hold the archive button in the right-hand corner to delete the message.