Can I downgrade to drush 5.x using homebrew? I used brew install drush which installed the latest stable version (6.1), when I need the 5.x branch.

This GitHub issue indicates a drush 5.x package was made available for homebrew, but I'm unsure how to install it.
Homebrew install specific version of formula? - this answer on drupal.stackexchange might help you to get the overview of how specific formulas can be installed via homebrew.

Short version related to your specific question:


Update homebrew with brew update.
Get all the versions available for drush formula with brew versions drush
You will see a list of commits that point to specific drush versions, e.g., 


6.1.0    git checkout a188056 Library/Formula/drush.rb
6.0.0    git checkout 1264aaf Library/Formula/drush.rb
6.0-rc4  git checkout 39b2450 Library/Formula/drush.rb
6.0-rc2  git checkout 2a61472 Library/Formula/drush.rb
6.0-beta1 git checkout 54ba68d Library/Formula/drush.rb
5.9      git checkout 81108ad Library/Formula/drush.rb
5.8      git checkout 91cb127 Library/Formula/drush.rb

Go to the base brew directory by typing cd $( brew --prefix )
Uninstall your current drush installation with brew uninstall drush (this won't affect your settings in ~/.drush directory).
Checkout the drush formula for specific commit that you are interested in. If you want to have drush version 5.9, type git checkout 81108ad Library/Formula/drush.rb and you will have a brew formula available for this particular version.
Install drush (version 5.9) with brew install drush
Type drush --version to make sure you have the correct version installed.