问:
Cant figure out what is wrong with this spider
So im strugging to determine what im doing wrong here.. Looking to scrape this site and extract some details for each item - category - title - price - url. But cant figure out why the spider will not give me the results im after.

If i remove the parse_details function in this spider, the spider will crawl the urls i want, yet when i call parse_details to grab some data, its breaking the crawl and not getting the full scope im looking for. (newb so please be gentle!)

without my yield request

```<code># -*- coding: utf-8 -*-
from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url))

    def parse_details(self, response):

        pass
</code>```

log

```<code>2020-02-16 14:52:21 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 14:52:21 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 113681,
 'downloader/request_count': 178,
 'downloader/request_method_count/GET': 178,
 'downloader/response_bytes': 9932059,
 'downloader/response_count': 178,
 'downloader/response_status_count/200': 178,
 'dupefilter/filtered': 212,
 'elapsed_time_seconds': 17.984344,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 14, 52, 21, 844590),
 'log_count/DEBUG': 179,
 'log_count/INFO': 10,
 'memusage/max': 53600256,
 'memusage/startup': 53600256,
 'request_depth_max': 4,
 'response_received_count': 178,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 177,
 'scheduler/dequeued/memory': 177,
 'scheduler/enqueued': 177,
 'scheduler/enqueued/memory': 177,
 'start_time': datetime.datetime(2020, 2, 16, 14, 52, 3, 860246)}
2020-02-16 14:52:21 [scrapy.core.engine] INFO: Spider closed (finished)
</code>```

with yield request

```<code># -*- coding: utf-8 -*-
from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url),
                         callback=self.parse_details)

    def parse_details(self, response):
        items = response.xpath('.//div[@class="products-list products-list-v2"]').extract()

        for item in items:
            category = response.xpath('.//h1/text()').extract_first()
            title = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = response.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            yield {
                'category': category,
                'title': title,
                'price': price,
                'url': url
            }
</code>```

log

```<code>2020-02-16 14:55:14 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 14:55:14 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 10801,
 'downloader/request_count': 19,
 'downloader/request_method_count/GET': 19,
 'downloader/response_bytes': 902364,
 'downloader/response_count': 19,
 'downloader/response_status_count/200': 19,
 'dupefilter/filtered': 17,
 'elapsed_time_seconds': 2.775066,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 14, 55, 14, 607396),
 'item_scraped_count': 2,
 'log_count/DEBUG': 22,
 'log_count/INFO': 10,
 'memusage/max': 53420032,
 'memusage/startup': 53420032,
 'request_depth_max': 1,
 'response_received_count': 19,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 18,
 'scheduler/dequeued/memory': 18,
 'scheduler/enqueued': 18,
 'scheduler/enqueued/memory': 18,
 'start_time': datetime.datetime(2020, 2, 16, 14, 55, 11, 832330)}
2020-02-16 14:55:14 [scrapy.core.engine] INFO: Spider closed (finished)
</code>```

答:
Maybe it will not resolve your problem but put it here.

I put all in single file script.py as standalone script with CrawlerProcess() and run as

```<code>python script.py
</code>```

and I get file output.csv with some data. But only on two pages it can find items. Other pages have only list of subcategories.

script.py

```<code>from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url),
                         callback=self.parse_details)

    def parse_details(self, response):
        # you can get it once - and use `strip()` to remove spaces
        category = response.xpath('//h1/text()').extract_first().strip()

        # you should search cards which keeps items separatelly - and don't extract it
        items = response.xpath('//div[@class="product-card__content"]') #.extract()
        print(len(items))

        for item in items:
            # you have to use `item.xpath` instead of `response.xpath`
            title = item.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = item.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = item.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            if title and price and url:
                yield {
                    'category': category,
                    'title': title.strip(),
                    'price': price.strip(),
                    'url': url.strip()
                }

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    #'USER_AGENT': 'Mozilla/5.0',
    'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0',
    # save in file CSV, JSON or XML
    'FEED_FORMAT': 'csv',     # csv, json, xml
    'FEED_URI': 'output.csv', #
})
c.crawl(WickesSpider)
c.start()
</code>```

Log on screen

```<code>2020-02-16 17:16:35 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: scrapybot)
2020-02-16 17:16:35 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.7.6 (default, Dec 19 2019, 23:50:13) - [GCC 7.4.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Linux-5.3.0-28-generic-x86_64-with-LinuxMint-19.3-tricia
2020-02-16 17:16:35 [scrapy.crawler] INFO: Overridden settings: {'FEED_FORMAT': 'csv', 'FEED_URI': 'output.csv', 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0'}
2020-02-16 17:16:35 [scrapy.extensions.telnet] INFO: Telnet Password: 7298b359c6ef9bcb
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-02-16 17:16:35 [scrapy.core.engine] INFO: Spider opened
2020-02-16 17:16:35 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-02-16 17:16:35 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-02-16 17:16:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/c/1000173> (referer: None)
2020-02-16 17:16:36 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.wickes.co.uk/Products/Building-Materials/Timber/c/1000175> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2020-02-16 17:16:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Timber/c/1000175> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Protective-Sheeting/c/1018000> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
9
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes 16mm Triplewall Polycarbonate Sheet - 980 x 4000mm', 'price': '£45.00', 'url': '/Wickes-16mm-Triplewall-Polycarbonate-Sheet---980-x-4000mm/p/243798'}
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Fascias+Soffits/c/1000183> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes Universal Edge Flashing for Polycarbonate Sheets - Brown 4m', 'price': '£1.50', 'url': '/Wickes-Universal-Edge-Flashing-for-Polycarbonate-Sheets---Brown-4m/p/164061'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes Universal Glazing Bar for Polycarbonate Sheets - Brown 4m', 'price': '£14.00', 'url': '/Wickes-Universal-Glazing-Bar-for-Polycarbonate-Sheets---Brown-4m/p/164057'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes White Universal Edge Flashing for Polycarbonate Sheets - 4m', 'price': '£2.00', 'url': '/Wickes-White-Universal-Edge-Flashing-for-Polycarbonate-Sheets---4m/p/164060'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes White Universal Glazing Bar for Polycarbonate Sheets - 4m', 'price': '£21.00', 'url': '/Wickes-White-Universal-Glazing-Bar-for-Polycarbonate-Sheets---4m/p/164056'}
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Bricks%2C-Blocks+Lintels/c/1000177> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Hardware-and-Metalwork/c/1000179> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Nails%2C-Screws+Adhesives/Sealants%2C-Adhesives+Fillers/c/1068000> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Water+Damp-Proofing/c/1000185> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
0
34
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Supermatt Emulsion Paint - White 10L', 'price': '£37.00', 'url': '/Dulux-Trade-Supermatt-Emulsion-Paint---White-10L/p/106478'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - White 10L', 'price': '£45.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---White-10L/p/106472'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Matt Emulsion Pure Brilliant White 10L', 'price': '£15.00', 'url': '/Wickes-Trade-Contract-Matt-Emulsion-Pure-Brilliant-White-10L/p/121732'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Pure Brilliant White 10L', 'price': '£48.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Pure-Brilliant-White-10L/p/106480'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Cappuccino 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Cappuccino-10L/p/121740'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Satinwood Paint - Pure Brilliant White 2.5L', 'price': '£30.00', 'url': '/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-2-5L/p/113082'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Matt Emulsion Paint - Ivory 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Matt-Emulsion-Paint---Ivory-10L/p/121725'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Classic Grey 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Classic-Grey-10L/p/145117'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Silk Emulsion Pure Brilliant White 10L', 'price': '£17.00', 'url': '/Wickes-Trade-Contract-Silk-Emulsion-Pure-Brilliant-White-10L/p/121737'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Matt Emulsion - White 10L', 'price': '£12.00', 'url': '/Wickes-Matt-Emulsion---White-10L/p/145119'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Emulsion Paint for New Plaster - Pure Brilliant White 10L', 'price': '£20.00', 'url': '/Wickes-Trade-Emulsion-Paint-for-New-Plaster---Pure-Brilliant-White-10L/p/122443'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Silk Emulsion Magnolia 10L', 'price': '£23.00', 'url': '/Wickes-Trade-Contract-Silk-Emulsion-Magnolia-10L/p/121743'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Basic Matt Emulsion Paint - Magnolia 10L', 'price': '£10.00', 'url': '/Wickes-Basic-Matt-Emulsion-Paint---Magnolia-10L/p/109394'}
0
0
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Basic Matt Emulsion Paint - Brilliant White 10L', 'price': '£10.00', 'url': '/Wickes-Basic-Matt-Emulsion-Paint---Brilliant-White-10L/p/104070'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Matt Emulsion Shadow Grey 10L', 'price': '£25.00', 'url': '/Wickes-Trade-Contract-Matt-Emulsion-Shadow-Grey-10L/p/191246'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Magnolia 10L', 'price': '£48.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-10L/p/106483'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Jasmine White 5L', 'price': '£30.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Jasmine-White-5L/p/173722'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Undercoat Paint - White 2.5L', 'price': '£25.00', 'url': '/Dulux-Trade-Undercoat-Paint---White-2-5L/p/106490'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade High Gloss Paint - Pure Brilliant White 2.5L', 'price': '£25.00', 'url': '/Dulux-Trade-High-Gloss-Paint---Pure-Brilliant-White-2-5L/p/106500'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Satinwood Paint - Pure Brilliant White 1L', 'price': '£18.00', 'url': '/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-1L/p/113079'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade High Gloss Paint - White 2.5L', 'price': '£20.00', 'url': '/Dulux-Trade-High-Gloss-Paint---White-2-5L/p/106496'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Weathershield Smooth Masonry Paint - Brilliant White 7.5L', 'price': '£38.00', 'url': '/Dulux-Trade-Weathershield-Smooth-Masonry-Paint---Brilliant-White-7-5L/p/119170'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Supermatt Emulsion Paint - Magnolia 10L', 'price': '£42.00', 'url': '/Dulux-Trade-Supermatt-Emulsion-Paint---Magnolia-10L/p/106481'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Magnolia 15L', 'price': '£29.99', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Magnolia-15L/p/121744'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Vinyl Silk Emulsion Paint - Pure Brilliant White 10L', 'price': '£32.00', 'url': '/Wickes-Trade-Vinyl-Silk-Emulsion-Paint---Pure-Brilliant-White-10L/p/121797'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Polished Pebble 5L', 'price': '£30.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Polished-Pebble-5L/p/173725'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Vinyl Matt Emulsion Paint - Magnolia 5L', 'price': '£18.00', 'url': '/Wickes-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-5L/p/121789'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Diamond Matt Emulsion Paint - Pure Brilliant White 5L', 'price': '£40.00', 'url': '/Dulux-Trade-Diamond-Matt-Emulsion-Paint---Pure-Brilliant-White-5L/p/113498'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Matt Emulsion Paint - Pure Brilliant White 15L', 'price': '£22.00', 'url': '/Wickes-Contract-Matt-Emulsion-Paint---Pure-Brilliant-White-15L/p/121733'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Diamond Eggshell Emulsion Paint - Pure Brilliant White 5L', 'price': '£60.00', 'url': '/Dulux-Trade-Diamond-Eggshell-Emulsion-Paint---Pure-Brilliant-White-5L/p/106487'}
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Decorating+Interiors/Mouldings/c/1000563> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Cladding/c/1000182> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Roofing/c/1000181> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Insulation/c/1000184> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Guttering+Drainage/c/1000180> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Cement+Aggregates/c/1000176> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Plaster+Plasterboard/c/1000178> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
0
0
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Sheet-Materials/c/1000174> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 17:16:39 [scrapy.extensions.feedexport] INFO: Stored csv feed (35 items) in: output.csv
2020-02-16 17:16:39 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 11242,
 'downloader/request_count': 18,
 'downloader/request_method_count/GET': 18,
 'downloader/response_bytes': 901446,
 'downloader/response_count': 18,
 'downloader/response_status_count/200': 18,
 'dupefilter/filtered': 17,
 'elapsed_time_seconds': 3.808066,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 16, 16, 39, 745213),
 'item_scraped_count': 35,
 'log_count/DEBUG': 54,
 'log_count/INFO': 11,
 'memusage/max': 50900992,
 'memusage/startup': 50900992,
 'request_depth_max': 1,
 'response_received_count': 18,
 'scheduler/dequeued': 18,
 'scheduler/dequeued/memory': 18,
 'scheduler/enqueued': 18,
 'scheduler/enqueued/memory': 18,
 'start_time': datetime.datetime(2020, 2, 16, 16, 16, 35, 937147)}
2020-02-16 17:16:39 [scrapy.core.engine] INFO: Spider closed (finished)
</code>```

output.csv

```<code>category,title,price,url
Building Materials Clearance,Wickes 16mm Triplewall Polycarbonate Sheet - 980 x 4000mm,£45.00,/Wickes-16mm-Triplewall-Polycarbonate-Sheet---980-x-4000mm/p/243798
Building Materials Clearance,Wickes Universal Edge Flashing for Polycarbonate Sheets - Brown 4m,£1.50,/Wickes-Universal-Edge-Flashing-for-Polycarbonate-Sheets---Brown-4m/p/164061
Building Materials Clearance,Wickes Universal Glazing Bar for Polycarbonate Sheets - Brown 4m,£14.00,/Wickes-Universal-Glazing-Bar-for-Polycarbonate-Sheets---Brown-4m/p/164057
Building Materials Clearance,Wickes White Universal Edge Flashing for Polycarbonate Sheets - 4m,£2.00,/Wickes-White-Universal-Edge-Flashing-for-Polycarbonate-Sheets---4m/p/164060
Building Materials Clearance,Wickes White Universal Glazing Bar for Polycarbonate Sheets - 4m,£21.00,/Wickes-White-Universal-Glazing-Bar-for-Polycarbonate-Sheets---4m/p/164056
Trade Paint,Dulux Trade Supermatt Emulsion Paint - White 10L,£37.00,/Dulux-Trade-Supermatt-Emulsion-Paint---White-10L/p/106478
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - White 10L,£45.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---White-10L/p/106472
Trade Paint,Wickes Trade Contract Matt Emulsion Pure Brilliant White 10L,£15.00,/Wickes-Trade-Contract-Matt-Emulsion-Pure-Brilliant-White-10L/p/121732
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Pure Brilliant White 10L,£48.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Pure-Brilliant-White-10L/p/106480
Trade Paint,Wickes Contract Silk Emulsion Paint - Cappuccino 10L,£25.00,/Wickes-Contract-Silk-Emulsion-Paint---Cappuccino-10L/p/121740
Trade Paint,Dulux Trade Satinwood Paint - Pure Brilliant White 2.5L,£30.00,/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-2-5L/p/113082
Trade Paint,Wickes Contract Matt Emulsion Paint - Ivory 10L,£25.00,/Wickes-Contract-Matt-Emulsion-Paint---Ivory-10L/p/121725
Trade Paint,Wickes Contract Silk Emulsion Paint - Classic Grey 10L,£25.00,/Wickes-Contract-Silk-Emulsion-Paint---Classic-Grey-10L/p/145117
Trade Paint,Wickes Trade Contract Silk Emulsion Pure Brilliant White 10L,£17.00,/Wickes-Trade-Contract-Silk-Emulsion-Pure-Brilliant-White-10L/p/121737
Trade Paint,Wickes Matt Emulsion - White 10L,£12.00,/Wickes-Matt-Emulsion---White-10L/p/145119
Trade Paint,Wickes Trade Emulsion Paint for New Plaster - Pure Brilliant White 10L,£20.00,/Wickes-Trade-Emulsion-Paint-for-New-Plaster---Pure-Brilliant-White-10L/p/122443
Trade Paint,Wickes Trade Contract Silk Emulsion Magnolia 10L,£23.00,/Wickes-Trade-Contract-Silk-Emulsion-Magnolia-10L/p/121743
Trade Paint,Wickes Basic Matt Emulsion Paint - Magnolia 10L,£10.00,/Wickes-Basic-Matt-Emulsion-Paint---Magnolia-10L/p/109394
Trade Paint,Wickes Basic Matt Emulsion Paint - Brilliant White 10L,£10.00,/Wickes-Basic-Matt-Emulsion-Paint---Brilliant-White-10L/p/104070
Trade Paint,Wickes Trade Contract Matt Emulsion Shadow Grey 10L,£25.00,/Wickes-Trade-Contract-Matt-Emulsion-Shadow-Grey-10L/p/191246
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Magnolia 10L,£48.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-10L/p/106483
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Jasmine White 5L,£30.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Jasmine-White-5L/p/173722
Trade Paint,Dulux Trade Undercoat Paint - White 2.5L,£25.00,/Dulux-Trade-Undercoat-Paint---White-2-5L/p/106490
Trade Paint,Dulux Trade High Gloss Paint - Pure Brilliant White 2.5L,£25.00,/Dulux-Trade-High-Gloss-Paint---Pure-Brilliant-White-2-5L/p/106500
Trade Paint,Dulux Trade Satinwood Paint - Pure Brilliant White 1L,£18.00,/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-1L/p/113079
Trade Paint,Dulux Trade High Gloss Paint - White 2.5L,£20.00,/Dulux-Trade-High-Gloss-Paint---White-2-5L/p/106496
Trade Paint,Dulux Trade Weathershield Smooth Masonry Paint - Brilliant White 7.5L,£38.00,/Dulux-Trade-Weathershield-Smooth-Masonry-Paint---Brilliant-White-7-5L/p/119170
Trade Paint,Dulux Trade Supermatt Emulsion Paint - Magnolia 10L,£42.00,/Dulux-Trade-Supermatt-Emulsion-Paint---Magnolia-10L/p/106481
Trade Paint,Wickes Contract Silk Emulsion Paint - Magnolia 15L,£29.99,/Wickes-Contract-Silk-Emulsion-Paint---Magnolia-15L/p/121744
Trade Paint,Wickes Trade Vinyl Silk Emulsion Paint - Pure Brilliant White 10L,£32.00,/Wickes-Trade-Vinyl-Silk-Emulsion-Paint---Pure-Brilliant-White-10L/p/121797
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Polished Pebble 5L,£30.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Polished-Pebble-5L/p/173725
Trade Paint,Wickes Trade Vinyl Matt Emulsion Paint - Magnolia 5L,£18.00,/Wickes-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-5L/p/121789
Trade Paint,Dulux Trade Diamond Matt Emulsion Paint - Pure Brilliant White 5L,£40.00,/Dulux-Trade-Diamond-Matt-Emulsion-Paint---Pure-Brilliant-White-5L/p/113498
Trade Paint,Wickes Contract Matt Emulsion Paint - Pure Brilliant White 15L,£22.00,/Wickes-Contract-Matt-Emulsion-Paint---Pure-Brilliant-White-15L/p/121733
Trade Paint,Dulux Trade Diamond Eggshell Emulsion Paint - Pure Brilliant White 5L,£60.00,/Dulux-Trade-Diamond-Eggshell-Emulsion-Paint---Pure-Brilliant-White-5L/p/106487
</code>```

EDIT: you have to search new links also in parse_details

```<code>    def parse(self, response):
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url), callback=self.parse_details)

    def parse_details(self, response):
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url), callback=self.parse_details)

        items = response.xpath('.//div[@class="products-list products-list-v2"]').extract()

        for item in items:
            category = response.xpath('.//h1/text()').extract_first()
            title = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = response.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            yield {
                'category': category.strip(),
                'title': title.strip(),
                'price': price.strip(),
                'url': url.strip()
            }
</code>```

Body:
So im strugging to determine what im doing wrong here.. Looking to scrape this site and extract some details for each item - category - title - price - url. But cant figure out why the spider will not give me the results im after.

If i remove the parse_details function in this spider, the spider will crawl the urls i want, yet when i call parse_details to grab some data, its breaking the crawl and not getting the full scope im looking for. (newb so please be gentle!)

without my yield request

# -*- coding: utf-8 -*-
from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url))

    def parse_details(self, response):

        pass

log

2020-02-16 14:52:21 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 14:52:21 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 113681,
 'downloader/request_count': 178,
 'downloader/request_method_count/GET': 178,
 'downloader/response_bytes': 9932059,
 'downloader/response_count': 178,
 'downloader/response_status_count/200': 178,
 'dupefilter/filtered': 212,
 'elapsed_time_seconds': 17.984344,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 14, 52, 21, 844590),
 'log_count/DEBUG': 179,
 'log_count/INFO': 10,
 'memusage/max': 53600256,
 'memusage/startup': 53600256,
 'request_depth_max': 4,
 'response_received_count': 178,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 177,
 'scheduler/dequeued/memory': 177,
 'scheduler/enqueued': 177,
 'scheduler/enqueued/memory': 177,
 'start_time': datetime.datetime(2020, 2, 16, 14, 52, 3, 860246)}
2020-02-16 14:52:21 [scrapy.core.engine] INFO: Spider closed (finished)

with yield request

# -*- coding: utf-8 -*-
from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url),
                         callback=self.parse_details)

    def parse_details(self, response):
        items = response.xpath('.//div[@class="products-list products-list-v2"]').extract()

        for item in items:
            category = response.xpath('.//h1/text()').extract_first()
            title = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = response.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            yield {
                'category': category,
                'title': title,
                'price': price,
                'url': url
            }

log

2020-02-16 14:55:14 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 14:55:14 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 10801,
 'downloader/request_count': 19,
 'downloader/request_method_count/GET': 19,
 'downloader/response_bytes': 902364,
 'downloader/response_count': 19,
 'downloader/response_status_count/200': 19,
 'dupefilter/filtered': 17,
 'elapsed_time_seconds': 2.775066,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 14, 55, 14, 607396),
 'item_scraped_count': 2,
 'log_count/DEBUG': 22,
 'log_count/INFO': 10,
 'memusage/max': 53420032,
 'memusage/startup': 53420032,
 'request_depth_max': 1,
 'response_received_count': 19,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 18,
 'scheduler/dequeued/memory': 18,
 'scheduler/enqueued': 18,
 'scheduler/enqueued/memory': 18,
 'start_time': datetime.datetime(2020, 2, 16, 14, 55, 11, 832330)}
2020-02-16 14:55:14 [scrapy.core.engine] INFO: Spider closed (finished)

Body:
Maybe it will not resolve your problem but put it here.

I put all in single file script.py as standalone script with CrawlerProcess() and run as

python script.py

and I get file output.csv with some data. But only on two pages it can find items. Other pages have only list of subcategories.

script.py

from scrapy import Spider
from scrapy.http import Request

# Define spider settings
class WickesSpider(Spider):
    name = 'wickes'
    allowed_domains = ['wickes.co.uk']
    start_urls = ['https://www.wickes.co.uk/Products/Building-Materials/c/1000173']

    def parse(self, response):
        # this will call self.parse by default for all your categories
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url),
                         callback=self.parse_details)

    def parse_details(self, response):
        # you can get it once - and use `strip()` to remove spaces
        category = response.xpath('//h1/text()').extract_first().strip()

        # you should search cards which keeps items separatelly - and don't extract it
        items = response.xpath('//div[@class="product-card__content"]') #.extract()
        print(len(items))

        for item in items:
            # you have to use `item.xpath` instead of `response.xpath`
            title = item.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = item.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = item.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            if title and price and url:
                yield {
                    'category': category,
                    'title': title.strip(),
                    'price': price.strip(),
                    'url': url.strip()
                }

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    #'USER_AGENT': 'Mozilla/5.0',
    'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0',
    # save in file CSV, JSON or XML
    'FEED_FORMAT': 'csv',     # csv, json, xml
    'FEED_URI': 'output.csv', #
})
c.crawl(WickesSpider)
c.start()

Log on screen

2020-02-16 17:16:35 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: scrapybot)
2020-02-16 17:16:35 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.7.6 (default, Dec 19 2019, 23:50:13) - [GCC 7.4.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1d  10 Sep 2019), cryptography 2.8, Platform Linux-5.3.0-28-generic-x86_64-with-LinuxMint-19.3-tricia
2020-02-16 17:16:35 [scrapy.crawler] INFO: Overridden settings: {'FEED_FORMAT': 'csv', 'FEED_URI': 'output.csv', 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64; rv:75.0) Gecko/20100101 Firefox/75.0'}
2020-02-16 17:16:35 [scrapy.extensions.telnet] INFO: Telnet Password: 7298b359c6ef9bcb
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-02-16 17:16:35 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-02-16 17:16:35 [scrapy.core.engine] INFO: Spider opened
2020-02-16 17:16:35 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-02-16 17:16:35 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-02-16 17:16:36 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/c/1000173> (referer: None)
2020-02-16 17:16:36 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.wickes.co.uk/Products/Building-Materials/Timber/c/1000175> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2020-02-16 17:16:37 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Timber/c/1000175> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Protective-Sheeting/c/1018000> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
9
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes 16mm Triplewall Polycarbonate Sheet - 980 x 4000mm', 'price': '£45.00', 'url': '/Wickes-16mm-Triplewall-Polycarbonate-Sheet---980-x-4000mm/p/243798'}
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Fascias+Soffits/c/1000183> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes Universal Edge Flashing for Polycarbonate Sheets - Brown 4m', 'price': '£1.50', 'url': '/Wickes-Universal-Edge-Flashing-for-Polycarbonate-Sheets---Brown-4m/p/164061'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes Universal Glazing Bar for Polycarbonate Sheets - Brown 4m', 'price': '£14.00', 'url': '/Wickes-Universal-Glazing-Bar-for-Polycarbonate-Sheets---Brown-4m/p/164057'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes White Universal Edge Flashing for Polycarbonate Sheets - 4m', 'price': '£2.00', 'url': '/Wickes-White-Universal-Edge-Flashing-for-Polycarbonate-Sheets---4m/p/164060'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Building-Materials/Building-Materials-Clearance/c/1022013>
{'category': 'Building Materials Clearance', 'title': 'Wickes White Universal Glazing Bar for Polycarbonate Sheets - 4m', 'price': '£21.00', 'url': '/Wickes-White-Universal-Glazing-Bar-for-Polycarbonate-Sheets---4m/p/164056'}
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Bricks%2C-Blocks+Lintels/c/1000177> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Hardware-and-Metalwork/c/1000179> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Nails%2C-Screws+Adhesives/Sealants%2C-Adhesives+Fillers/c/1068000> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Water+Damp-Proofing/c/1000185> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
0
34
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Supermatt Emulsion Paint - White 10L', 'price': '£37.00', 'url': '/Dulux-Trade-Supermatt-Emulsion-Paint---White-10L/p/106478'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - White 10L', 'price': '£45.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---White-10L/p/106472'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Matt Emulsion Pure Brilliant White 10L', 'price': '£15.00', 'url': '/Wickes-Trade-Contract-Matt-Emulsion-Pure-Brilliant-White-10L/p/121732'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Pure Brilliant White 10L', 'price': '£48.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Pure-Brilliant-White-10L/p/106480'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Cappuccino 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Cappuccino-10L/p/121740'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Satinwood Paint - Pure Brilliant White 2.5L', 'price': '£30.00', 'url': '/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-2-5L/p/113082'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Matt Emulsion Paint - Ivory 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Matt-Emulsion-Paint---Ivory-10L/p/121725'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Classic Grey 10L', 'price': '£25.00', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Classic-Grey-10L/p/145117'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Silk Emulsion Pure Brilliant White 10L', 'price': '£17.00', 'url': '/Wickes-Trade-Contract-Silk-Emulsion-Pure-Brilliant-White-10L/p/121737'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Matt Emulsion - White 10L', 'price': '£12.00', 'url': '/Wickes-Matt-Emulsion---White-10L/p/145119'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Emulsion Paint for New Plaster - Pure Brilliant White 10L', 'price': '£20.00', 'url': '/Wickes-Trade-Emulsion-Paint-for-New-Plaster---Pure-Brilliant-White-10L/p/122443'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Silk Emulsion Magnolia 10L', 'price': '£23.00', 'url': '/Wickes-Trade-Contract-Silk-Emulsion-Magnolia-10L/p/121743'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Basic Matt Emulsion Paint - Magnolia 10L', 'price': '£10.00', 'url': '/Wickes-Basic-Matt-Emulsion-Paint---Magnolia-10L/p/109394'}
0
0
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Basic Matt Emulsion Paint - Brilliant White 10L', 'price': '£10.00', 'url': '/Wickes-Basic-Matt-Emulsion-Paint---Brilliant-White-10L/p/104070'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Contract Matt Emulsion Shadow Grey 10L', 'price': '£25.00', 'url': '/Wickes-Trade-Contract-Matt-Emulsion-Shadow-Grey-10L/p/191246'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Magnolia 10L', 'price': '£48.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-10L/p/106483'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Jasmine White 5L', 'price': '£30.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Jasmine-White-5L/p/173722'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Undercoat Paint - White 2.5L', 'price': '£25.00', 'url': '/Dulux-Trade-Undercoat-Paint---White-2-5L/p/106490'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade High Gloss Paint - Pure Brilliant White 2.5L', 'price': '£25.00', 'url': '/Dulux-Trade-High-Gloss-Paint---Pure-Brilliant-White-2-5L/p/106500'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Satinwood Paint - Pure Brilliant White 1L', 'price': '£18.00', 'url': '/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-1L/p/113079'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade High Gloss Paint - White 2.5L', 'price': '£20.00', 'url': '/Dulux-Trade-High-Gloss-Paint---White-2-5L/p/106496'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Weathershield Smooth Masonry Paint - Brilliant White 7.5L', 'price': '£38.00', 'url': '/Dulux-Trade-Weathershield-Smooth-Masonry-Paint---Brilliant-White-7-5L/p/119170'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Supermatt Emulsion Paint - Magnolia 10L', 'price': '£42.00', 'url': '/Dulux-Trade-Supermatt-Emulsion-Paint---Magnolia-10L/p/106481'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Silk Emulsion Paint - Magnolia 15L', 'price': '£29.99', 'url': '/Wickes-Contract-Silk-Emulsion-Paint---Magnolia-15L/p/121744'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Vinyl Silk Emulsion Paint - Pure Brilliant White 10L', 'price': '£32.00', 'url': '/Wickes-Trade-Vinyl-Silk-Emulsion-Paint---Pure-Brilliant-White-10L/p/121797'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Vinyl Matt Emulsion Paint - Polished Pebble 5L', 'price': '£30.00', 'url': '/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Polished-Pebble-5L/p/173725'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Trade Vinyl Matt Emulsion Paint - Magnolia 5L', 'price': '£18.00', 'url': '/Wickes-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-5L/p/121789'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Diamond Matt Emulsion Paint - Pure Brilliant White 5L', 'price': '£40.00', 'url': '/Dulux-Trade-Diamond-Matt-Emulsion-Paint---Pure-Brilliant-White-5L/p/113498'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Wickes Contract Matt Emulsion Paint - Pure Brilliant White 15L', 'price': '£22.00', 'url': '/Wickes-Contract-Matt-Emulsion-Paint---Pure-Brilliant-White-15L/p/121733'}
2020-02-16 17:16:38 [scrapy.core.scraper] DEBUG: Scraped from <200 https://www.wickes.co.uk/Products/Decorating+Interiors/Paint/Trade-Paint/c/1006001>
{'category': 'Trade Paint', 'title': 'Dulux Trade Diamond Eggshell Emulsion Paint - Pure Brilliant White 5L', 'price': '£60.00', 'url': '/Dulux-Trade-Diamond-Eggshell-Emulsion-Paint---Pure-Brilliant-White-5L/p/106487'}
2020-02-16 17:16:38 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Decorating+Interiors/Mouldings/c/1000563> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Cladding/c/1000182> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Roofing/c/1000181> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Insulation/c/1000184> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Guttering+Drainage/c/1000180> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Cement+Aggregates/c/1000176> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Plaster+Plasterboard/c/1000178> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
0
0
0
2020-02-16 17:16:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.wickes.co.uk/Products/Building-Materials/Sheet-Materials/c/1000174> (referer: https://www.wickes.co.uk/Products/Building-Materials/c/1000173)
0
2020-02-16 17:16:39 [scrapy.core.engine] INFO: Closing spider (finished)
2020-02-16 17:16:39 [scrapy.extensions.feedexport] INFO: Stored csv feed (35 items) in: output.csv
2020-02-16 17:16:39 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 11242,
 'downloader/request_count': 18,
 'downloader/request_method_count/GET': 18,
 'downloader/response_bytes': 901446,
 'downloader/response_count': 18,
 'downloader/response_status_count/200': 18,
 'dupefilter/filtered': 17,
 'elapsed_time_seconds': 3.808066,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 2, 16, 16, 16, 39, 745213),
 'item_scraped_count': 35,
 'log_count/DEBUG': 54,
 'log_count/INFO': 11,
 'memusage/max': 50900992,
 'memusage/startup': 50900992,
 'request_depth_max': 1,
 'response_received_count': 18,
 'scheduler/dequeued': 18,
 'scheduler/dequeued/memory': 18,
 'scheduler/enqueued': 18,
 'scheduler/enqueued/memory': 18,
 'start_time': datetime.datetime(2020, 2, 16, 16, 16, 35, 937147)}
2020-02-16 17:16:39 [scrapy.core.engine] INFO: Spider closed (finished)

output.csv

category,title,price,url
Building Materials Clearance,Wickes 16mm Triplewall Polycarbonate Sheet - 980 x 4000mm,£45.00,/Wickes-16mm-Triplewall-Polycarbonate-Sheet---980-x-4000mm/p/243798
Building Materials Clearance,Wickes Universal Edge Flashing for Polycarbonate Sheets - Brown 4m,£1.50,/Wickes-Universal-Edge-Flashing-for-Polycarbonate-Sheets---Brown-4m/p/164061
Building Materials Clearance,Wickes Universal Glazing Bar for Polycarbonate Sheets - Brown 4m,£14.00,/Wickes-Universal-Glazing-Bar-for-Polycarbonate-Sheets---Brown-4m/p/164057
Building Materials Clearance,Wickes White Universal Edge Flashing for Polycarbonate Sheets - 4m,£2.00,/Wickes-White-Universal-Edge-Flashing-for-Polycarbonate-Sheets---4m/p/164060
Building Materials Clearance,Wickes White Universal Glazing Bar for Polycarbonate Sheets - 4m,£21.00,/Wickes-White-Universal-Glazing-Bar-for-Polycarbonate-Sheets---4m/p/164056
Trade Paint,Dulux Trade Supermatt Emulsion Paint - White 10L,£37.00,/Dulux-Trade-Supermatt-Emulsion-Paint---White-10L/p/106478
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - White 10L,£45.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---White-10L/p/106472
Trade Paint,Wickes Trade Contract Matt Emulsion Pure Brilliant White 10L,£15.00,/Wickes-Trade-Contract-Matt-Emulsion-Pure-Brilliant-White-10L/p/121732
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Pure Brilliant White 10L,£48.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Pure-Brilliant-White-10L/p/106480
Trade Paint,Wickes Contract Silk Emulsion Paint - Cappuccino 10L,£25.00,/Wickes-Contract-Silk-Emulsion-Paint---Cappuccino-10L/p/121740
Trade Paint,Dulux Trade Satinwood Paint - Pure Brilliant White 2.5L,£30.00,/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-2-5L/p/113082
Trade Paint,Wickes Contract Matt Emulsion Paint - Ivory 10L,£25.00,/Wickes-Contract-Matt-Emulsion-Paint---Ivory-10L/p/121725
Trade Paint,Wickes Contract Silk Emulsion Paint - Classic Grey 10L,£25.00,/Wickes-Contract-Silk-Emulsion-Paint---Classic-Grey-10L/p/145117
Trade Paint,Wickes Trade Contract Silk Emulsion Pure Brilliant White 10L,£17.00,/Wickes-Trade-Contract-Silk-Emulsion-Pure-Brilliant-White-10L/p/121737
Trade Paint,Wickes Matt Emulsion - White 10L,£12.00,/Wickes-Matt-Emulsion---White-10L/p/145119
Trade Paint,Wickes Trade Emulsion Paint for New Plaster - Pure Brilliant White 10L,£20.00,/Wickes-Trade-Emulsion-Paint-for-New-Plaster---Pure-Brilliant-White-10L/p/122443
Trade Paint,Wickes Trade Contract Silk Emulsion Magnolia 10L,£23.00,/Wickes-Trade-Contract-Silk-Emulsion-Magnolia-10L/p/121743
Trade Paint,Wickes Basic Matt Emulsion Paint - Magnolia 10L,£10.00,/Wickes-Basic-Matt-Emulsion-Paint---Magnolia-10L/p/109394
Trade Paint,Wickes Basic Matt Emulsion Paint - Brilliant White 10L,£10.00,/Wickes-Basic-Matt-Emulsion-Paint---Brilliant-White-10L/p/104070
Trade Paint,Wickes Trade Contract Matt Emulsion Shadow Grey 10L,£25.00,/Wickes-Trade-Contract-Matt-Emulsion-Shadow-Grey-10L/p/191246
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Magnolia 10L,£48.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-10L/p/106483
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Jasmine White 5L,£30.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Jasmine-White-5L/p/173722
Trade Paint,Dulux Trade Undercoat Paint - White 2.5L,£25.00,/Dulux-Trade-Undercoat-Paint---White-2-5L/p/106490
Trade Paint,Dulux Trade High Gloss Paint - Pure Brilliant White 2.5L,£25.00,/Dulux-Trade-High-Gloss-Paint---Pure-Brilliant-White-2-5L/p/106500
Trade Paint,Dulux Trade Satinwood Paint - Pure Brilliant White 1L,£18.00,/Dulux-Trade-Satinwood-Paint---Pure-Brilliant-White-1L/p/113079
Trade Paint,Dulux Trade High Gloss Paint - White 2.5L,£20.00,/Dulux-Trade-High-Gloss-Paint---White-2-5L/p/106496
Trade Paint,Dulux Trade Weathershield Smooth Masonry Paint - Brilliant White 7.5L,£38.00,/Dulux-Trade-Weathershield-Smooth-Masonry-Paint---Brilliant-White-7-5L/p/119170
Trade Paint,Dulux Trade Supermatt Emulsion Paint - Magnolia 10L,£42.00,/Dulux-Trade-Supermatt-Emulsion-Paint---Magnolia-10L/p/106481
Trade Paint,Wickes Contract Silk Emulsion Paint - Magnolia 15L,£29.99,/Wickes-Contract-Silk-Emulsion-Paint---Magnolia-15L/p/121744
Trade Paint,Wickes Trade Vinyl Silk Emulsion Paint - Pure Brilliant White 10L,£32.00,/Wickes-Trade-Vinyl-Silk-Emulsion-Paint---Pure-Brilliant-White-10L/p/121797
Trade Paint,Dulux Trade Vinyl Matt Emulsion Paint - Polished Pebble 5L,£30.00,/Dulux-Trade-Vinyl-Matt-Emulsion-Paint---Polished-Pebble-5L/p/173725
Trade Paint,Wickes Trade Vinyl Matt Emulsion Paint - Magnolia 5L,£18.00,/Wickes-Trade-Vinyl-Matt-Emulsion-Paint---Magnolia-5L/p/121789
Trade Paint,Dulux Trade Diamond Matt Emulsion Paint - Pure Brilliant White 5L,£40.00,/Dulux-Trade-Diamond-Matt-Emulsion-Paint---Pure-Brilliant-White-5L/p/113498
Trade Paint,Wickes Contract Matt Emulsion Paint - Pure Brilliant White 15L,£22.00,/Wickes-Contract-Matt-Emulsion-Paint---Pure-Brilliant-White-15L/p/121733
Trade Paint,Dulux Trade Diamond Eggshell Emulsion Paint - Pure Brilliant White 5L,£60.00,/Dulux-Trade-Diamond-Eggshell-Emulsion-Paint---Pure-Brilliant-White-5L/p/106487

EDIT: you have to search new links also in parse_details

    def parse(self, response):
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url), callback=self.parse_details)

    def parse_details(self, response):
        for url in response.xpath('.//ul[@class="aside-nav2__list"]/li/a/@href').extract():
            yield Request(response.urljoin(url), callback=self.parse_details)

        items = response.xpath('.//div[@class="products-list products-list-v2"]').extract()

        for item in items:
            category = response.xpath('.//h1/text()').extract_first()
            title = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/text()').extract_first()
            price = response.xpath('.//*[@class="product-card__price-value "]/text()').extract_first()
            url = response.xpath('.//*[@class="product-card__title product-card__title-v2"]/@href').extract_first()

            yield {
                'category': category.strip(),
                'title': title.strip(),
                'price': price.strip(),
                'url': url.strip()
            }


来源: stackexchange

问:
React: Change big calendar onNavigate trigger from clicking the date number to the whole cell date
I am currently using the Big Calendar on React. Currently, Big Calendar triggers its onNavigate callback when you click on the date in the cell.

What I want is to make onNavigate fire when I click the whole cell of date.

Is there a possible way to do this? Thanks.

答:
If you're using view, onView, date, and onNavigate, doing view and date as controlled props, then you could use your onSelectSlot prop to change your date.

```<code>const handleSlotSelect = slotInfo => {
  if (view === 'month') { // assumes you got 'view' from state
    // ... do some date stuff to get _just the date_ from the slotInfo.start
    handleNavigate(newDate);
    handleViewChange('day');
  }
  // ... any other slot selection functionality for other views
};
</code>```

Bear in mind that doing this will not call RBC's internal handleViewChange, onNavigate, or handleRangeChange methods. Also best if you're using function components with the useState hooks here, so that your state updates are bundled together, so date and view are updated simultaneously.

ContentLicense: CC BY-SA 4.0

Body:
If you're using view, onView, date, and onNavigate, doing view and date as controlled props, then you could use your onSelectSlot prop to change your date.

const handleSlotSelect = slotInfo => {
  if (view === 'month') { // assumes you got 'view' from state
    // ... do some date stuff to get _just the date_ from the slotInfo.start
    handleNavigate(newDate);
    handleViewChange('day');
  }
  // ... any other slot selection functionality for other views
};

Bear in mind that doing this will not call RBC's internal handleViewChange, onNavigate, or handleRangeChange methods. Also best if you're using function components with the useState hooks here, so that your state updates are bundled together, so date and view are updated simultaneously.


问:
getParameter in JSP sets null to href
I have the following situation, there's my index.jsp page:

```<code><%@ page contentType="text/html;charset=UTF-8" language="java" %>
<html>
<head>
    <title>Lab2</title>
</head>
<body>
<p>
    Period: <input type="number" name="period" size="50">
    <br>
    Faculty: <input type="text" name="faculty" size="50">
    <br>
    <br>
    <a href="${pageContext.request.contextPath}/calculatePaymentForSeveralSemesters?value=<%=request.getParameter("period")%>&faculty=<%=request.getParameter("faculty")%>">Calculate
        payment for several semesters</a>
    <a href="${pageContext.request.contextPath}/showTwoSmallestFaculties">Show two smallest faculties</a>
</p>
</body>
</html>
</code>```

So, I want to create the link with dynamical values which I get from inputs. But my result link is http://localhost:8080/Lab2_war_exploded/calculatePaymentForSeveralSemesters?value=null&faculty=null and I cannot understand why values from inputs are not added to this href. Can you help me to solve this problem? I will appreciate any help. Thanks in advance!

答:
Until the page is submitted, you won't get the value of period and faculty available in the request object.

For the purpose of demo, I have added a form with submit button. Enter some values for period and faculty and hit the submit button. Now check the link and you will find it to be populated with the desired values e.g. when you hit the submit button after entering 10 in the period and test in the semester field, you find the value of link as http://localhost:8080/TestDynamicProject/calculatePaymentForSeveralSemesters?value=10&faculty=test.

```<code><%@ page contentType="text/html;charset=UTF-8" language="java"%>
<html>
<head>
<title>Lab2</title>
</head>
<body>
    <p>
    <form>
        Period: <input type="number" name="period" size="50"> <br>
        Faculty: <input type="text" name="faculty" size="50"> <br>
        <a href="${pageContext.request.contextPath}/calculatePaymentForSeveralSemesters?value=<%=request.getParameter("period")%>&faculty=<%=request.getParameter("faculty")%>">Calculate payment for several semesters</a>
        <a href="${pageContext.request.contextPath}/showTwoSmallestFaculties">Show two smallest faculties</a>
        <input type="submit">
    </form>
    </p>
</body>
</html>
</code>```

Title: getParameter in JSP sets null to href

Body:
I have the following situation, there's my index.jsp page:

<%@ page contentType="text/html;charset=UTF-8" language="java" %>

    <title>Lab2</title>

    Period: <input type="number" name="period" size="50">

    Faculty: <input type="text" name="faculty" size="50">

    <a href="${pageContext.request.contextPath}/calculatePaymentForSeveralSemesters?value=<%=request.getParameter("period")%>&faculty=<%=request.getParameter("faculty")%>">Calculate
        payment for several semesters
    Show two smallest faculties

So, I want to create the link with dynamical values which I get from inputs. But my result link is http://localhost:8080/Lab2_war_exploded/calculatePaymentForSeveralSemesters?value=null&faculty=null and I cannot understand why values from inputs are not added to this href. Can you help me to solve this problem? I will appreciate any help. Thanks in advance!

Body:
Until the page is submitted, you won't get the value of period and faculty available in the request object.

For the purpose of demo, I have added a form with submit button. Enter some values for period and faculty and hit the submit button. Now check the link and you will find it to be populated with the desired values e.g. when you hit the submit button after entering 10 in the period and test in the semester field, you find the value of link as http://localhost:8080/TestDynamicProject/calculatePaymentForSeveralSemesters?value=10&faculty=test.

<%@ page contentType="text/html;charset=UTF-8" language="java"%>
<html>
<head>
<title>Lab2</title>
</head>
<body>
    <p>
    <form>
        Period: <input type="number" name="period" size="50"> <br>
        Faculty: <input type="text" name="faculty" size="50"> <br>
        <a href="${pageContext.request.contextPath}/calculatePaymentForSeveralSemesters?value=<%=request.getParameter("period")%>&faculty=<%=request.getParameter("faculty")%>">Calculate payment for several semesters</a>
        <a href="${pageContext.request.contextPath}/showTwoSmallestFaculties">Show two smallest faculties</a>
        <input type="submit">
    </form>
    </p>
</body>
</html>


问:
Reordering a daily data series into weekly rows with Mon, Tue, Wed, Thur and Fri as the weekly columns
I have a daily time series containing weekdays, excluding weekends.
I would like to reorder it so that each column represents a week and the subsequent five rows show the data for  Mon, Tue, Wed, Thur and Fri of that week.

I tried using cast (package: reshape) but am having problems in obtaining the above.

Thank you for your help.

example:

```<code>Date        Day Value
06/01/2020  mon 15
07/01/2020  tue 16
08/01/2020  wed 17
09/01/2020  thu 18
10/01/2020  fri 19
13/01/2020  mon 20
14/01/2020  tue 21
15/01/2020  wed 22
16/01/2020  thu 23
17/01/2020  fri 24
</code>```

to be reformated to:

```<code>Start of week   mon tue wed thu fri
06/01/2020      15  16  17  18  19
13/01/2020      20  21  22  23  24
</code>```

答:
Another option with the data.table-package:

```<code>library(data.table)

# convert to a 'data.table'
# set the 'Date' and 'Day' columns in the right format
setDT(mydf)[, `:=` (Date = as.Date(Date, format = "%d/%m/%Y"),
                    Day = factor(Day, levels = c("mon","tue","wed","thu","fri")))]

# create a 'start_of_week' column
# transform from long to wide format
res <- mydf[, start_of_week := Date[1], by = cumsum(Day == "mon")
            ][, dcast(.SD, start_of_week ~ Day, value.var = "Value")]
</code>```

which gives:

```<code>> res
   start_of_week mon tue wed thu fri
1:    06/01/2020  15  16  17  18  19
2:    13/01/2020  20  21  22  23  24
</code>```

Used data:

```<code>mydf <- read.table(text="Date        Day Value
06/01/2020  mon 15
07/01/2020  tue 16
08/01/2020  wed 17
09/01/2020  thu 18
10/01/2020  fri 19
13/01/2020  mon 20
14/01/2020  tue 21
15/01/2020  wed 22
16/01/2020  thu 23
17/01/2020  fri 24", header=TRUE, stringsAsFactors=FALSE)
</code>```

Body:
I have a daily time series containing weekdays, excluding weekends.
I would like to reorder it so that each column represents a week and the subsequent five rows show the data for  Mon, Tue, Wed, Thur and Fri of that week.

I tried using cast (package: reshape) but am having problems in obtaining the above.

Thank you for your help.

example:

Date        Day Value
06/01/2020  mon 15
07/01/2020  tue 16
08/01/2020  wed 17
09/01/2020  thu 18
10/01/2020  fri 19
13/01/2020  mon 20
14/01/2020  tue 21
15/01/2020  wed 22
16/01/2020  thu 23
17/01/2020  fri 24

to be reformated to:

Start of week   mon tue wed thu fri
06/01/2020      15  16  17  18  19
13/01/2020      20  21  22  23  24

Body:
Another option with the data.table-package:

library(data.table)

# convert to a 'data.table'
# set the 'Date' and 'Day' columns in the right format
setDT(mydf)[, `:=` (Date = as.Date(Date, format = "%d/%m/%Y"),
                    Day = factor(Day, levels = c("mon","tue","wed","thu","fri")))]

# create a 'start_of_week' column
# transform from long to wide format
res <- mydf[, start_of_week := Date[1], by = cumsum(Day == "mon")
            ][, dcast(.SD, start_of_week ~ Day, value.var = "Value")]

which gives:

> res
   start_of_week mon tue wed thu fri
1:    06/01/2020  15  16  17  18  19
2:    13/01/2020  20  21  22  23  24

Used data:

mydf <- read.table(text="Date        Day Value
06/01/2020  mon 15
07/01/2020  tue 16
08/01/2020  wed 17
09/01/2020  thu 18
10/01/2020  fri 19
13/01/2020  mon 20
14/01/2020  tue 21
15/01/2020  wed 22
16/01/2020  thu 23
17/01/2020  fri 24", header=TRUE, stringsAsFactors=FALSE)


问:
Whats wrong in this SpingMVC and Hibernate transaction?
I am creating a small application that manages two entities using hibernate and springMVC, one called User and the other one is Application. The problem is that when I try to enter values in the database (MySQL) I get the following error:
org.hibernate.HibernateException: Could not obtain transaction-synchronized Session for current thread.
Despite this error I have the necessary notes to make transactions, but it still does not work. Here I leave the code:
SpringConfig:

```<code>import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
import org.springframework.web.servlet.view.InternalResourceViewResolver;
import org.springframework.web.servlet.view.JstlView;

@Configuration
@EnableWebMvc
@ComponentScan(basePackages = "com.test.app")
@PropertySource("classpath:/application.properties")
public class AppConfig implements WebMvcConfigurer {
@Bean
public InternalResourceViewResolver resolver() {
    InternalResourceViewResolver resolver = new InternalResourceViewResolver();
    resolver.setViewClass(JstlView.class);
    resolver.setPrefix("/WEB-INF/jsp/");
    resolver.setSuffix(".jsp");
    return resolver;
    }
}
</code>```

Hiberate Config:

```<code>import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.core.env.Environment;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.orm.hibernate5.HibernateTransactionManager;
import org.springframework.orm.hibernate5.LocalSessionFactoryBean;
import org.springframework.transaction.annotation.EnableTransactionManagement;

import javax.sql.DataSource;
import java.util.Properties;

@Configuration
@EnableTransactionManagement
@PropertySource("classpath:/application.properties")
@ComponentScan(basePackages = {
    "com.test.app"
})
public class HibernateConfig {
@Autowired
private Environment environment;

@Bean
public LocalSessionFactoryBean sessionFactory() {
    LocalSessionFactoryBean sessionFactory = new LocalSessionFactoryBean();
    sessionFactory.setDataSource(dataSource());
    sessionFactory.setPackagesToScan("com.esliceu.examen.model");
    sessionFactory.setHibernateProperties(hibernateProperties());
    return sessionFactory;
    }

@Bean
public DataSource dataSource() {
    DriverManagerDataSource dataSource = new DriverManagerDataSource();
    dataSource.setDriverClassName(environment.getRequiredProperty("jdbc.driverClassName"));
    dataSource.setUrl(environment.getRequiredProperty("jdbc.url"));
    dataSource.setUsername(environment.getRequiredProperty("jdbc.username"));
    dataSource.setPassword(environment.getRequiredProperty("jdbc.password"));
    return dataSource;
    }

private Properties hibernateProperties() {
    Properties properties = new Properties();
    properties.put("hibernate.dialect", environment.getRequiredProperty("hibernate.dialect"));
    properties.put("hibernate.show_sql", environment.getRequiredProperty("hibernate.show_sql"));
    properties.put("hibernate.format_sql", environment.getRequiredProperty("hibernate.format_sql"));
    properties.put("hibernate.hbm2ddl.auto", environment.getRequiredProperty("hibernate.hbm2ddl.auto"));
    return properties;
    }

@Bean
    public HibernateTransactionManager getTransactionManager() {
    HibernateTransactionManager transactionManager = new HibernateTransactionManager();
    transactionManager.setSessionFactory(sessionFactory().getObject());
    return transactionManager;
    }
}
</code>```

AppController:

```<code>import com.test.app.model.Application;
import com.test.app.service.ApplicacionService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.GetMapping;

@Controller
@Transactional
public class AplicacioController {
@Autowired
private AplicationService aplicationService;

@GetMapping("/applications")
private String testInsert(){
    Application app = new Aplicacio();
    app.setId(1);
    app.setName("fooApp");
    aplicacioService.insertOrUpdate(app);
    return "applications";
    }
}
</code>```

ApplicationDao:

```<code>import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;

import javax.annotation.PostConstruct;
import javax.persistence.Query;
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.CriteriaQuery;
import javax.persistence.criteria.Root;
import java.util.ArrayList;
import java.util.List;

@Repository
public class AplicacioDao  implements AppDao{

@Autowired
private SessionFactory sessionFactory;
@Override
public void insertOrUpdate(Aplication app) {
    Session session = sessionFactory.getCurrentSession();
    session.saveOrUpdate(app);
    }
}
</code>```

Application Model:

```<code>import org.hibernate.envers.Audited;

import javax.persistence.*;
import java.util.List;
import java.util.Objects;

@Entity
@Table(name = "aplication")
@Audited
public class Aplicacio {
@Id
@Column(name = "idApp")
@GeneratedValue(strategy = GenerationType.IDENTITY)
private int id;

@Column(name = "name")
private String name;

@ManyToMany(mappedBy = "applications")
private List<User> ysers;

//...contructor, geters ands setters//
}
</code>```

And this is the service layer:

```<code>package com.test.app.service;

import com.test.app.dao.ApplicationDao;
import com.test.app.model.Aplicacio;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
public class ApplicationService implements AppService{
@Autowired
private AplicacioDao applicationDao;

public List<Application> findAll(){
    return applicationDao.findAll();
    }

@Override
public List<Application> findAllDatabase() {
    return applicationDao.findAllDataBase();
    }

@Override
public Application findById(int id) {
    return applicationDao.findById(id);
    }

@Override
public void insertOrUpdate(Application app) {
    applicationDao.insertOrUpdate(app);
    }

@Override
public void delete(Application app) {
    applicationDao.delete(app);
    }
}
</code>```

I am configuring everything through annotations as you can see, but I am unable to see where the error is. If someone could tell me where the error is, I would be very grateful

答:
Try this for service layer by removing @Transactional from DAO and Controller :

```<code>@Service
@Transactional
public class ApplicationService implements AppService{
@Autowired
private AplicacioDao applicationDao;

public List<Application> findAll(){
    return applicationDao.findAll();
    }

@Override
public List<Application> findAllDatabase() {
    return applicationDao.findAllDataBase();
    }

@Override
public Application findById(int id) {
    return applicationDao.findById(id);
    }

@Override
public void insertOrUpdate(Application app) {
    applicationDao.insertOrUpdate(app);
    }

@Override
public void delete(Application app) {
    applicationDao.delete(app);
    }
}
</code>```

Or problem is with HibernateConfig class, plz check that.

Body:
I am creating a small application that manages two entities using hibernate and springMVC, one called User and the other one is Application. The problem is that when I try to enter values in the database (MySQL) I get the following error:
org.hibernate.HibernateException: Could not obtain transaction-synchronized Session for current thread.
Despite this error I have the necessary notes to make transactions, but it still does not work. Here I leave the code:
SpringConfig:

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.web.servlet.config.annotation.EnableWebMvc;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
import org.springframework.web.servlet.view.InternalResourceViewResolver;
import org.springframework.web.servlet.view.JstlView;

@Configuration
@EnableWebMvc
@ComponentScan(basePackages = "com.test.app")
@PropertySource("classpath:/application.properties")
public class AppConfig implements WebMvcConfigurer {
@Bean
public InternalResourceViewResolver resolver() {
    InternalResourceViewResolver resolver = new InternalResourceViewResolver();
    resolver.setViewClass(JstlView.class);
    resolver.setPrefix("/WEB-INF/jsp/");
    resolver.setSuffix(".jsp");
    return resolver;
    }
}

Hiberate Config:

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.PropertySource;
import org.springframework.core.env.Environment;
import org.springframework.jdbc.datasource.DriverManagerDataSource;
import org.springframework.orm.hibernate5.HibernateTransactionManager;
import org.springframework.orm.hibernate5.LocalSessionFactoryBean;
import org.springframework.transaction.annotation.EnableTransactionManagement;

import javax.sql.DataSource;
import java.util.Properties;

@Configuration
@EnableTransactionManagement
@PropertySource("classpath:/application.properties")
@ComponentScan(basePackages = {
    "com.test.app"
})
public class HibernateConfig {
@Autowired
private Environment environment;

@Bean
public LocalSessionFactoryBean sessionFactory() {
    LocalSessionFactoryBean sessionFactory = new LocalSessionFactoryBean();
    sessionFactory.setDataSource(dataSource());
    sessionFactory.setPackagesToScan("com.esliceu.examen.model");
    sessionFactory.setHibernateProperties(hibernateProperties());
    return sessionFactory;
    }

@Bean
public DataSource dataSource() {
    DriverManagerDataSource dataSource = new DriverManagerDataSource();
    dataSource.setDriverClassName(environment.getRequiredProperty("jdbc.driverClassName"));
    dataSource.setUrl(environment.getRequiredProperty("jdbc.url"));
    dataSource.setUsername(environment.getRequiredProperty("jdbc.username"));
    dataSource.setPassword(environment.getRequiredProperty("jdbc.password"));
    return dataSource;
    }

private Properties hibernateProperties() {
    Properties properties = new Properties();
    properties.put("hibernate.dialect", environment.getRequiredProperty("hibernate.dialect"));
    properties.put("hibernate.show_sql", environment.getRequiredProperty("hibernate.show_sql"));
    properties.put("hibernate.format_sql", environment.getRequiredProperty("hibernate.format_sql"));
    properties.put("hibernate.hbm2ddl.auto", environment.getRequiredProperty("hibernate.hbm2ddl.auto"));
    return properties;
    }

@Bean
    public HibernateTransactionManager getTransactionManager() {
    HibernateTransactionManager transactionManager = new HibernateTransactionManager();
    transactionManager.setSessionFactory(sessionFactory().getObject());
    return transactionManager;
    }
}

AppController:

import com.test.app.model.Application;
import com.test.app.service.ApplicacionService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.transaction.annotation.Transactional;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.GetMapping;

@Controller
@Transactional
public class AplicacioController {
@Autowired
private AplicationService aplicationService;

@GetMapping("/applications")
private String testInsert(){
    Application app = new Aplicacio();
    app.setId(1);
    app.setName("fooApp");
    aplicacioService.insertOrUpdate(app);
    return "applications";
    }
}

ApplicationDao:

import org.hibernate.Session;
import org.hibernate.SessionFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;

import javax.annotation.PostConstruct;
import javax.persistence.Query;
import javax.persistence.criteria.CriteriaBuilder;
import javax.persistence.criteria.CriteriaQuery;
import javax.persistence.criteria.Root;
import java.util.ArrayList;
import java.util.List;

@Repository
public class AplicacioDao  implements AppDao{

@Autowired
private SessionFactory sessionFactory;
@Override
public void insertOrUpdate(Aplication app) {
    Session session = sessionFactory.getCurrentSession();
    session.saveOrUpdate(app);
    }
}

Application Model:

import org.hibernate.envers.Audited;

import javax.persistence.*;
import java.util.List;
import java.util.Objects;

@Entity
@Table(name = "aplication")
@Audited
public class Aplicacio {
@Id
@Column(name = "idApp")
@GeneratedValue(strategy = GenerationType.IDENTITY)
private int id;

@Column(name = "name")
private String name;

@ManyToMany(mappedBy = "applications")
private List<User> ysers;

//...contructor, geters ands setters//
}

And this is the service layer:

package com.test.app.service;

import com.test.app.dao.ApplicationDao;
import com.test.app.model.Aplicacio;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
public class ApplicationService implements AppService{
@Autowired
private AplicacioDao applicationDao;

public List<Application> findAll(){
    return applicationDao.findAll();
    }

@Override
public List<Application> findAllDatabase() {
    return applicationDao.findAllDataBase();
    }

@Override
public Application findById(int id) {
    return applicationDao.findById(id);
    }

@Override
public void insertOrUpdate(Application app) {
    applicationDao.insertOrUpdate(app);
    }

@Override
public void delete(Application app) {
    applicationDao.delete(app);
    }
}

I am configuring everything through annotations as you can see, but I am unable to see where the error is. If someone could tell me where the error is, I would be very grateful

Body:
Try this for service layer by removing @Transactional from DAO and Controller :

@Service
@Transactional
public class ApplicationService implements AppService{
@Autowired
private AplicacioDao applicationDao;

public List<Application> findAll(){
    return applicationDao.findAll();
    }

@Override
public List<Application> findAllDatabase() {
    return applicationDao.findAllDataBase();
    }

@Override
public Application findById(int id) {
    return applicationDao.findById(id);
    }

@Override
public void insertOrUpdate(Application app) {
    applicationDao.insertOrUpdate(app);
    }

@Override
public void delete(Application app) {
    applicationDao.delete(app);
    }
}

Or problem is with HibernateConfig class, plz check that.


问:
add column in table1 derived calculation from table2 's column
I have two tables

Employee

Finance

Employee contains 3 columns

id

Name

BaseSalary

Finance table contains 2 columns

id

BonusPercentage

I want to add TotalBonus column in Employee table. For calculation I am using BaseSalary from Employee table and BonusPercentage from Finance table.

TotalBonus = BaseSalary + (BaseSalary * BonusPercentage)

I can use this with stored procedure but i don't want. However, I created scalar function (calculateBonus) which returns TotalBonus but the problem is that i can't add column in table as i have dependency of BonusPercentage of Finance table.

In general i want employee table to have following:

id

Name

BaseSalary

TotalBonus

I know there are many ways but want in Employee table as I am using this table on C#.
Thanks

答:
Assuming that there can be a maximum of one row in finance for each employee, you want a (left) join:

```<code>SELECT e.id,
       e.name,
       e.basesalary,
       coalesce(f.bonuspercentage, 0) * e.basesalary totalbonus
       FROM employee e
            LEFT JOIN finance f
                      ON f.id = e.id;
</code>```

Body:
Assuming that there can be a maximum of one row in finance for each employee, you want a (left) join:

SELECT e.id,
       e.name,
       e.basesalary,
       coalesce(f.bonuspercentage, 0) * e.basesalary totalbonus
       FROM employee e
            LEFT JOIN finance f
                      ON f.id = e.id;


问:
How to center the ion-button inside ion-item?
How should I center ion-button inside ion-item?

```<code><ion-list>
  <ion-item>
    <ion-button>Sample</ion-button>
  </ion-item>
</ion-list>
</code>```

I tried some class from CSS Utilities but with no avail.

答:
You can do something like this:

CSS:

```<code>.center-button {
  position: relative;
  width: 100%;
  height: 100%;
  display: flex;
  justify-content: center;
  align-item: center;
}
</code>```

HTML:

```<code><ion-list>
  <ion-item>
    <div class='center-button'>
       <ion-button>Sample</ion-button>
    </div>
  </ion-item>
</ion-list>
</code>```

Body:
How should I center ion-button inside ion-item?

<ion-list>
  <ion-item>
    <ion-button>Sample</ion-button>
  </ion-item>
</ion-list>

I tried some class from CSS Utilities but with no avail.

Body:
You can do something like this:

CSS:

.center-button {
  position: relative;
  width: 100%;
  height: 100%;
  display: flex;
  justify-content: center;
  align-item: center;
}

HTML:

<ion-list>
  <ion-item>
    <div class='center-button'>
       <ion-button>Sample</ion-button>
    </div>
  </ion-item>
</ion-list>


问:
Question regarding choosing Cloud Provider
When creating an account and choose Azure option, does it mean all the services Cloud Services, Query Processing and Database Storage will all be from Azure? Or S3 for Storage and Cloud Services, Query Processing from Azure?

Does anyone know?

答: If you choose to create a Snowflake account on Azure, then everything related to Snowflake will exist on Azure (Storage, Compute, and Cloud Services) for that account.  Hope that helps.</p>


问:
How to get all the strings after a specific index?
Suppose I have this string: /ban 1d hello world which is essentially a command sended to a bot, meaning:

/ban specify to ban the user

1d represents the time

hello world is the reason

I need to get the reason, so I did:

let c = '/ban 1d hello world';
let arr = c.split(' ');
let result = c.replace(arr[0], '');
result = result.replace(arr[1], '')
alert(result);```

this is working, but I would like to ask if there is a better way to achieve this.

Kind regards.

答:
If you know that your reason will always be after the second space, you can do something like:</p>

```<code>const c = '/ban 1d hello world';
const reason = c.split(' ').slice(2).join(' ');
</code>```

Body:
Suppose I have this string: /ban 1d hello world which is essentially a command sended to a bot, meaning:

/ban specify to ban the user

1d represents the time

hello world is the reason

I need to get the reason, so I did:

let c = '/ban 1d hello world';
let arr = c.split(' ');
let result = c.replace(arr[0], '');
result = result.replace(arr[1], '')
alert(result);

this is working, but I would like to ask if there is a better way to achieve this.

Kind regards.

Body:
If you know that your reason will always be after the second space, you can do something like:

const c = '/ban 1d hello world';
const reason = c.split(' ').slice(2).join(' ');


问:
Add a generated column in postgres using to_char and a date
I have a date column created_at in a postgres table. I have a long running query that groups the data by week, so part of the query is

```<code>(SELECT TO_CHAR(score.created_at, 'IYYY-IW') AS week,
customer_id, AVG(score) AS avg_week_score FROM score
</code>```

This is inefficient, and I would like to use the new generated column functionality in Postgres 12 to rather store a text version of each date as it is inserted.

I tried to run

```<code>alter table score add column gen_week text generated always as (TO_CHAR(created_at, 'IYYY-IW')) stored;
</code>```

but got

```<code>ERROR:  generation expression is not immutable
</code>```

I guessed that this has to do with the fact that created_at is dependent on locale, and especially timezone, so I tried

```<code>alter table score add column week_gen text generated always as (TO_CHAR(created_at::timestamptz at time zone 'UTC', 'IYYY-IW')) stored;
</code>```

to make the timezone explicit, but this produces the same error.

Is there a way to make to_char behave immutably with a date object, or do I have to define my own immutable function based on to_char?

答:
The problem is that to_char() could take modifies from the locale -- and functions either are or are not immutable.

You can do the same thing using extract():

```<code>alter table score add column gen_week text generated always as
    ((extract(isoyear from created_at) * 100 + extract(week from created_at))::text ) stored;
</code>```

Actually, the above doesn't put the hyphen in, so:

```<code>alter table score add column gen_week text generated always as
    (extract(isoyear from created_at)::text || '-' || lpad(extract(week from created_at)::text, 2, '0')) stored;
</code>```

Here is a db<>fiddle.

Body:
I have a date column created_at in a postgres table. I have a long running query that groups the data by week, so part of the query is

(SELECT TO_CHAR(score.created_at, 'IYYY-IW') AS week,
customer_id, AVG(score) AS avg_week_score FROM score

This is inefficient, and I would like to use the new generated column functionality in Postgres 12 to rather store a text version of each date as it is inserted.

I tried to run

alter table score add column gen_week text generated always as (TO_CHAR(created_at, 'IYYY-IW')) stored;

but got

ERROR:  generation expression is not immutable

I guessed that this has to do with the fact that created_at is dependent on locale, and especially timezone, so I tried

alter table score add column week_gen text generated always as (TO_CHAR(created_at::timestamptz at time zone 'UTC', 'IYYY-IW')) stored;

to make the timezone explicit, but this produces the same error.

Is there a way to make to_char behave immutably with a date object, or do I have to define my own immutable function based on to_char?

Body:
The problem is that to_char() could take modifies from the locale -- and functions either are or are not immutable.

You can do the same thing using extract():

alter table score add column gen_week text generated always as
    ((extract(isoyear from created_at) * 100 + extract(week from created_at))::text ) stored;

Actually, the above doesn't put the hyphen in, so:

alter table score add column gen_week text generated always as
    (extract(isoyear from created_at)::text || '-' || lpad(extract(week from created_at)::text, 2, '0')) stored;

Here is a db<>fiddle.