Muting Messages Audio Alerts
You can change this setting in the Messages app preferences.

Go to Messages-> Preferences -> Alerts. Select 'Message Received' in the Event menu and uncheck the 'Play a sound' box.

You may also have to do this for the 'Message Received in Active Chat' and 'Message Received in Chat Room' events, as well as any others that are triggering unwanted sounds.