Why does my Mac Pro have two ethernet ports on the back of the computer?
If you wanted to connect to two different networks you could use this.  So say you had an internal network which you had your own computers all linked to but were not connected to the internet, you could use one port to connect to this network.  Then you could connect the other port to a broadband router/modem and access the internet on your Mac Pro.  It's really more of an interest to businesses than domestic users.
You could also use the second port to connect to Network Attached Storage (NAS) such as a MyBook World or similar.  This would allow you to use the full bandwidth of one port for backups to disk while not affecting the speed on the other network.
Backup interface in case one fails.  You could have each interface connect to different switches on your network in case of a failure on one you still have access to the network on the other.
There's also something called link aggregation which allows you to combine the ports to act as one port with twice the speed.  However, this depends on your other network equipment supporting this mode.