Why does my wallet need to download the whole transaction history?
The short answer to your question is "no". The reason is actually pretty simple: Without the entire block chain, there is no way to know that funds that are claimed to have been sent to you weren't actually sent to someone else. There is no way to prove the absence of a conflicting transaction, other than to check every valid transaction.

With a bank, the bank clears your checks for you. Here, you clear your own checks. Without knowing every check an account has received or sent out, how can you know the account has sufficient balance to cover your check?

Of course, you don't have to run the Bitcoin client. Your alternative is trust someone else with your coins.