Drush is always scanning the whole drupal directory?
There might be something wrong with your module... You say there are more than a thousand files scanned but are they placed in your module's directory ?

Drush is not supposed to scan your sites/default/files folder for instance or any other data folder you might have set. As a  best practice you should not place data in your module's path (obvious... isn't it ?). If you have done so, you should think about moving them somewhere else.

When executed, Drush scans all activated modules path (along with a few others such as sites/all/drush) for .drush.inc files. This is the expected behavior and that might cause your slow script execution.