Scrapy Username Spider
The main problem is that you are writing/appending to the file inside the spider's parse() method - this is not how this should be done in Scrapy - there is a special place - Item Pipelines. Check this answer to see how to include a custom pipeline when running Scrapy from a script.

Note that in the pipeline you should not be appending to a file, but instead collecting the data in the process_item() method and dumping it when the spider is finished in spider_closed() method. You may also do the writing in chunks to avoid potential memory problems on large output data sizes.

Other things to improve:


adjust/increase the number of concurrent requests
turn the logging off
enable HTTP cache
Scrapy running on PyPy?