Trouble with downloading images using Scrapy
Question: I'm getting the following error when attempting to download images using a
spider with Scrapy.

    
    
    File "C:\Python27\lib\site-packages\scrapy\http\request\__init__.py",
    line 61, in _set_url
                raise ValueError('Missing scheme in request url: %s' % self._url)
            exceptions.ValueError: Missing scheme in request url: h
    

As best as I can understand it, it looks like I'm missing an "h" in a url
somewhere? But I can't for the life of me see where. Everything works fine if
I'm not trying to download images. But once I add the appropriate code to the
four files below, I can't get anything to work properly. Could anyone help me
make sense of this error?

**items.py**

    
    
    import scrapy
    
    class ProductItem(scrapy.Item):
        model = scrapy.Field()
        shortdesc = scrapy.Field()
        desc = scrapy.Field()
        series = scrapy.Field()
        imageorig = scrapy.Field()
        image_urls = scrapy.Field()
        images = scrapy.Field()
    

**settings.py**

    
    
    BOT_NAME = 'allenheath'
    
    SPIDER_MODULES = ['allenheath.spiders']
    NEWSPIDER_MODULE = 'allenheath.spiders'
    
    ITEM_PIPELINES = {'scrapy.contrib.pipeline.images.ImagesPipeline': 1}
    
    IMAGES_STORE = 'c:/allenheath/images'
    

**pipelines.py**

    
    
    class AllenheathPipeline(object):
        def process_item(self, item, spider):
            return item
    
    import scrapy
    from scrapy.contrib.pipeline.images import ImagesPipeline
    from scrapy.exceptions import DropItem
    
    class MyImagesPipeline(ImagesPipeline):
    
        def get_media_requests(self, item, info):
            for image_url in item['image_urls']:
                yield scrapy.Request(image_url)
    
        def item_completed(self, results, item, info):
            image_paths = [x['path'] for ok, x in results if ok]
            if not image_paths:
                raise DropItem("Item contains no images")
            item['image_paths'] = image_paths
            return item
    

**products.py** (my spider)

    
    
    import scrapy
    
    from allenheath.items import ProductItem
    from scrapy.selector import Selector
    from scrapy.http import HtmlResponse
    
    class productsSpider(scrapy.Spider):
        name = "products"
        allowed_domains = ["http://www.allen-heath.com/"]
        start_urls = [
            "http://www.allen-heath.com/ahproducts/ilive-80/",
            "http://www.allen-heath.com/ahproducts/ilive-112/"
        ]
    
        def parse(self, response):
            for sel in response.xpath('/html'):
                item = ProductItem()
                item['model'] = sel.css('#prodsingleouter > div > div > h2::text').extract()
                item['shortdesc'] = sel.css('#prodsingleouter > div > div > h3::text').extract()
                item['desc'] = sel.css('#tab1 #productcontent').extract()
                item['series'] = sel.css('#pagestrip > div > div > a:nth-child(3)::text').extract()
                item['imageorig'] = sel.css('#prodsingleouter > div > div > h2::text').extract()
                item['image_urls'] = sel.css('#tab1 #productcontent img').extract()[0]
                item['image_urls'] = 'http://www.allen-heath.com' + item['image_urls']
                yield item
    

Any help would be greatly appreciated.


Answer: The issue is here:

    
    
    def get_media_requests(self, item, info):
        for image_url in item['image_urls']:
            yield scrapy.Request(image_url)
    

and here:

    
    
    item['image_urls'] = sel.css('#tab1 #productcontent img').extract()[0]
    

You are extracting this field and taking the first element. Which means that
once you are iterating over it in the pipeline, you are in fact iterating over
characters in the URL, which begins with `http` \- explaining the error
message you are seeing, as soon as the first letter tries to be processed:

    
    
    Missing scheme in request url: h
    

Remove the `[0]` from the line. While you're at it, fetch the `src` of the
image, instead of the entire element:

    
    
    item['image_urls'] = sel.css('#tab1 #productcontent img').xpath('./@src').extract()
    

After that, you should update the next line also, in case the image url is
relative, to convert it to absolute:

    
    
    import urlparse  # put this at the top of the script
    item['image_urls'] = [urlparse.urljoin(response.url, url) for url in item['image_urls']]
    

But you don't need this last part if the image URL in `src` is actually
absolute, so just remove it.

