itunes backup password
If you never set one, it then should be the passcode you used to unlock your phone (the 4-6 digit PIN) when you performed the back up.

If that doesn't work, look in your Keychain Access app, in your Utilities folder. Launch it, search for "iphone backup", and chances are you'll get a match. Open the item, click "show password" and enter your Mac system password.