What is a Bitcoin wallet? What is a wallet in the context of Bitcoin? 

What are wallets used for? 
What relation do wallets have to other concepts such as addresses, private-keys, client-software, and Bitcoin balances?
Bitcoin is like digital cash.
A bitcoin address is like an email account but for cash. 
A bitcoin address has a private key and a public key.
The public key is like your email (joesmith@gmail.com)
The private key is like your password to access your email (mypassword)
The main difference here is that a bitcoin private key holds all the information to access the account. So with a private key you can get the public key. 

A bitcoin wallet holds the private keys to bitcoin addresses - so instead of a wallet, think of it as a key chain with keys to different safes.
Wallets are used as an interface where you can access the bitcoins in your addresses without having to enter the private key in each time.
It makes the user experience a lot easier. So instead of entering in your private key each time you just might enter in a pin code instead.
Wallets also let you easily see the balance of each address, add and remove accounts (addresses) and send and receive bitcoin.
This is a very basic explanation but a good one to start with.
Hope this helps.