What is the Bonjour option in iMessage for?
When Bonjour is enabled in Messages it allows you to chat with other users on the same network who also have Bonjour enabled. As described in Bonjour For Developers:


  Bonjour, also known as zero-configuration networking, enables automatic discovery of devices and services on a local network using industry standard IP protocols.


This was developed some years ago to enable easy configuration of printers and other network services without needing to know ports, addresses, etc.  Messages simply ties into this service.

Jabber is a messaging protocol.  It underlies several protocols such as AIM and Google Chat.
iMessage is a messaging protocol. Messages is the chat application.