How to specify the version for drush site-install
Usually I use drush dl drupal then cd into the drupal directory and then drush si [other parameters].

That means, when you download drupal core you need to specify the version at that time -- not during site installation (which is just running the files).

You can do that by appending the version onto your dl argument, for example:
drush dl drupal-6 or drush dl drupal-6.x