How to close Messages.app?
You can disable the notifications using Notification Center. Please note that they will be disabled whether the app is opened or closed.

Open System Preferences, then select Notifications (upper right corner). You will then see a list of Apps which use Notification Center. Find Messages in this list. Set the "Messages alert style" to None.