Check the stock of references list
Auto-throttling extension may cause high download delays. Either turn the extension off to see what would the resulting time be, or limit the maximum delay via AUTOTHROTTLE_MAX_DELAY.



Also, you may issue multiple requests from the after_login() method instead of keeping the queue of references:

def after_login(self, response):
    for ref in references:
        yield scrapy.Request(url="url" + ref, callback=self.parse_stock)




Also, instead of writing to the file from a spider directly, you can use a CSV output pipeline:

import csv


class CSVWriterPipeline(object):
    def __init__(self):
        self.writer = csv.writer(open('file.txt', 'wb'))

    def process_item(self, item, spider):
        self.writer.writerow([item["field1"], item["field2"], item["field3"])
        return item


Where field1, field2, field3 are your item fields:

class MyItem(Item):
    field1 = Field()
    field2 = Field()
    field3 = Field()


Which should be set in the parse_stock() callback:

def parse_stock(self, response):
    item = MyItem()
    item["field1"] = response.xpath('//*[@id="priceDetails"]/form/div[2]/text()').extract_first()
    item["field2"] = response.xpath('//*[@id="priceDetails"]/form/div[8]/div[1]/span/span[2]/text()').extract_first()
    item["field3"] = response.xpath('//*[@id="priceDetails"]/form/div[8]/div[1]/span/span[1]/i/@style').extract_first()
    return item


Then, you would need to enable the pipeline:

process = CrawlerProcess({
    'ITEM_PIPELINES', {
         '__main__.CSVWriterPipeline': 100
    },
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
    'LOG_LEVEL': 'INFO',
    'AUTOTHROTTLE_ENABLED': 'True',
    'AUTOTHROTTLE_START_DELAY': '0.1',
    'AUTOTHROTTLE_TARGET_CONCURRENCY': '100'
})


Or course, improve the "field" and "item" class names to be more meaningful.



There are other things, like using local DNS cache, you may try to speed up your spider:


Speed up web scraper


And, as a "running Scrapy from script" reference topic, please see this post.