How to install PHP library using Drush
You can actually use drush make for this purpose. Drush make files can add in third-party dependencies which can be useful for just this kind of thing. Unless you intend to share this module with the community, in which case, you can issue a wget from the module itself similar to how other contrib modules do it. (See JW Player for an older example of that)

This is a good post on the topic by basically a Drupal genius. (Not me, I swear)

http://davehall.com.au/blog/dave/2011/05/17/drush-make-and-module-dependencies