<p dir="auto">I had a problem earlier when trying to scrape a site. I thought there was an issue with requests elsewhere but It turned out that I had to include the cookies from the headers as an argument in <code class="notranslate">scrapy.FormRequest()</code>. What is the reasoning behind this? Because, when using <code class="notranslate">request.post()</code> I can get a response 200 by just using the payload and headers. Why does scrapy differ to this for this situation? I had thought it follows the same structure as requests in the backend, but it seems like there's more to it.</p>
<p dir="auto">For example, here's what I had which gives a 404 error:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import scrapy



headers = {
    'authority': 'www.etsy.com',
    'sec-ch-ua': '&quot; Not A;Brand&quot;;v=&quot;99&quot;, &quot;Chromium&quot;;v=&quot;96&quot;, &quot;Google Chrome&quot;;v=&quot;96&quot;',
    'x-csrf-token': '3:1641383062:Exn8HMFDcc0UtitU6NOM3o3x8BGB:864dc90d926383d90686f37be56f69685b939f0f306b10a99bcd9016209f15d4',
    'sec-ch-ua-mobile': '?0',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
    'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'accept': '*/*',
    'x-requested-with': 'XMLHttpRequest',
    'x-page-guid': 'eeda48b359a.aa23cce28f31baac6f24.00',
    'x-detected-locale': 'GBP|en-GB|GB',
    'sec-ch-ua-platform': '&quot;Linux&quot;',
    'origin': 'https://www.etsy.com',
    'sec-fetch-site': 'same-origin',
    'sec-fetch-mode': 'cors',
    'sec-fetch-dest': 'empty',
    'referer': 'https://www.etsy.com/search/clothing/womens-clothing?q=20s&amp;explicit=1&amp;ship_to=GB&amp;page=2&amp;ref=pagination',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'cookie': 'uaid=G-_aWcvXqYHevnNO3ane9nOUmwNjZACCxCuVe2B0tVJpYmaKkpVSaVpUSoBZaGZVQL6Lj4mRv7ObrmmRR3F-aLyHp1ItAwA.; user_prefs=bNwL2wOEkWxqOSu2A1-CWlR6cr9jZACCxCuVe2B0tJK7U4CSTl5pTo6OUmqerruTko4SiACLGEEoXEQsAwA.; fve=1641314748.0; utm_lps=google__cpc; ua=531227642bc86f3b5fd7103a0c0b4fd6; p=eyJnZHByX3RwIjoxLCJnZHByX3AiOjF9; _gcl_au=1.1.1757627174.1641314793; _gid=GA1.2.1898390797.1641314793; __adal_cw=1641314793715; _pin_unauth=dWlkPVltVmtZemxoTldNdFpURXdPQzAwWkRWbUxXRTJOV1l0TTJGaE9URXdZVEEwTlRBeQ; last_browse_page=https%3A%2F%2Fwww.etsy.com%2Fuk%2F; __adal_ses=*; __adal_ca=so%3DGoogle%26me%3Dorganic%26ca%3D%28not%2520set%29%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; search_options={&quot;prev_search_term&quot;:&quot;20s&quot;,&quot;item_language&quot;:null,&quot;language_carousel&quot;:null}; _ga=GA1.2.559839679.1641314793; tsd=%7B%7D; __adal_id=952d43d7-5b80-4907-99d7-6f6baa9f4fe1.1641314794.3.1641383063.1641383059.2fe7a338-93bd-441f-b295-80549adbef7b; _tq_id.TV-27270909-1.a4d5=e2f6af8c27dee5e4.1641314794.0.1641383063..; _uetsid=dff577e06d7d11ec9617cbf4cc51b5b2; _uetvid=dff5f2706d7d11ec932fd3c5b816ab20; granify.uuid=bfd14e46-e8fa-4e7b-bce7-6f05dcb4b215; pla_spr=1; _ga_KR3J610VYM=GS1.1.1641383058.3.1.1641383118.60; exp_hangover=qk2fpkLi1lphuLsCKeq4gAe9BvxjZACCxCuVe8D01Zbb1UrlqUnxiUUlmWmZyZmJOfE5iSWpecmV8YUm8UYGhpZKVkqZeak5memZSTmpSrUMAA..; granify.session.QrsCf=-1',
}



class EtsySpider(scrapy.Spider):
    name = 'etit'
    start_urls = ['https://www.etsy.com/api/v3/ajax/bespoke/member/neu/specs/async_search_results']

    custom_settings = {
        'USER_AGENT':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'
    }
    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.FormRequest(
                url,
                method = &quot;POST&quot;,
                formdata = {
                    'log_performance_metrics': 'true',
                    'specs[async_search_results][]': 'Search2_ApiSpecs_WebSearch',
                    'specs[async_search_results][1][search_request_params][detected_locale][language]': 'en-GB',
                    'specs[async_search_results][1][search_request_params][detected_locale][currency_code]': 'GBP',
                    'specs[async_search_results][1][search_request_params][detected_locale][region]': 'GB',
                    'specs[async_search_results][1][search_request_params][locale][language]': 'en-GB',
                    'specs[async_search_results][1][search_request_params][locale][currency_code]': 'GBP',
                    'specs[async_search_results][1][search_request_params][locale][region]': 'GB',
                    'specs[async_search_results][1][search_request_params][name_map][query]': 'q',
                    'specs[async_search_results][1][search_request_params][name_map][query_type]': 'qt',
                    'specs[async_search_results][1][search_request_params][name_map][results_per_page]': 'result_count',
                    'specs[async_search_results][1][search_request_params][name_map][min_price]': 'min',
                    'specs[async_search_results][1][search_request_params][name_map][max_price]': 'max',
                    'specs[async_search_results][1][search_request_params][parameters][q]': '30s',
                    'specs[async_search_results][1][search_request_params][parameters][explicit]': '1',
                    'specs[async_search_results][1][search_request_params][parameters][locationQuery]': '2635167',
                    'specs[async_search_results][1][search_request_params][parameters][ship_to]': 'GB',
                    'specs[async_search_results][1][search_request_params][parameters][page]': '4',
                    'specs[async_search_results][1][search_request_params][parameters][ref]': 'pagination',
                    'specs[async_search_results][1][search_request_params][parameters][facet]': 'clothing/womens-clothing',
                    'specs[async_search_results][1][search_request_params][parameters][referrer]': 'https://www.etsy.com/search/clothing/womens-clothing?q=30s&amp;explicit=1locationQuery=2635167&amp;ship_to=GB&amp;page=3&amp;ref=pagination',
                    'specs[async_search_results][1][search_request_params][user_id]': '',
                    'specs[async_search_results][1][request_type]': 'pagination_preact',
                    'specs[async_search_results][1][is_eligible_for_spa_reformulations]': 'false',
                    'view_data_event_name': 'search_async_pagination_specview_rendered'
                },
                headers=headers,
                callback = self.parse
            )
    
    def parse(self, response):
        stuff = response.json().get('cssFiles')
        yield {
            'stuff':stuff
        }"><pre class="notranslate"><code class="notranslate">import scrapy



headers = {
    'authority': 'www.etsy.com',
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"',
    'x-csrf-token': '3:1641383062:Exn8HMFDcc0UtitU6NOM3o3x8BGB:864dc90d926383d90686f37be56f69685b939f0f306b10a99bcd9016209f15d4',
    'sec-ch-ua-mobile': '?0',
    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
    'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
    'accept': '*/*',
    'x-requested-with': 'XMLHttpRequest',
    'x-page-guid': 'eeda48b359a.aa23cce28f31baac6f24.00',
    'x-detected-locale': 'GBP|en-GB|GB',
    'sec-ch-ua-platform': '"Linux"',
    'origin': 'https://www.etsy.com',
    'sec-fetch-site': 'same-origin',
    'sec-fetch-mode': 'cors',
    'sec-fetch-dest': 'empty',
    'referer': 'https://www.etsy.com/search/clothing/womens-clothing?q=20s&amp;explicit=1&amp;ship_to=GB&amp;page=2&amp;ref=pagination',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'cookie': 'uaid=G-_aWcvXqYHevnNO3ane9nOUmwNjZACCxCuVe2B0tVJpYmaKkpVSaVpUSoBZaGZVQL6Lj4mRv7ObrmmRR3F-aLyHp1ItAwA.; user_prefs=bNwL2wOEkWxqOSu2A1-CWlR6cr9jZACCxCuVe2B0tJK7U4CSTl5pTo6OUmqerruTko4SiACLGEEoXEQsAwA.; fve=1641314748.0; utm_lps=google__cpc; ua=531227642bc86f3b5fd7103a0c0b4fd6; p=eyJnZHByX3RwIjoxLCJnZHByX3AiOjF9; _gcl_au=1.1.1757627174.1641314793; _gid=GA1.2.1898390797.1641314793; __adal_cw=1641314793715; _pin_unauth=dWlkPVltVmtZemxoTldNdFpURXdPQzAwWkRWbUxXRTJOV1l0TTJGaE9URXdZVEEwTlRBeQ; last_browse_page=https%3A%2F%2Fwww.etsy.com%2Fuk%2F; __adal_ses=*; __adal_ca=so%3DGoogle%26me%3Dorganic%26ca%3D%28not%2520set%29%26co%3D%28not%2520set%29%26ke%3D%28not%2520set%29; search_options={"prev_search_term":"20s","item_language":null,"language_carousel":null}; _ga=GA1.2.559839679.1641314793; tsd=%7B%7D; __adal_id=952d43d7-5b80-4907-99d7-6f6baa9f4fe1.1641314794.3.1641383063.1641383059.2fe7a338-93bd-441f-b295-80549adbef7b; _tq_id.TV-27270909-1.a4d5=e2f6af8c27dee5e4.1641314794.0.1641383063..; _uetsid=dff577e06d7d11ec9617cbf4cc51b5b2; _uetvid=dff5f2706d7d11ec932fd3c5b816ab20; granify.uuid=bfd14e46-e8fa-4e7b-bce7-6f05dcb4b215; pla_spr=1; _ga_KR3J610VYM=GS1.1.1641383058.3.1.1641383118.60; exp_hangover=qk2fpkLi1lphuLsCKeq4gAe9BvxjZACCxCuVe8D01Zbb1UrlqUnxiUUlmWmZyZmJOfE5iSWpecmV8YUm8UYGhpZKVkqZeak5memZSTmpSrUMAA..; granify.session.QrsCf=-1',
}



class EtsySpider(scrapy.Spider):
    name = 'etit'
    start_urls = ['https://www.etsy.com/api/v3/ajax/bespoke/member/neu/specs/async_search_results']

    custom_settings = {
        'USER_AGENT':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'
    }
    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.FormRequest(
                url,
                method = "POST",
                formdata = {
                    'log_performance_metrics': 'true',
                    'specs[async_search_results][]': 'Search2_ApiSpecs_WebSearch',
                    'specs[async_search_results][1][search_request_params][detected_locale][language]': 'en-GB',
                    'specs[async_search_results][1][search_request_params][detected_locale][currency_code]': 'GBP',
                    'specs[async_search_results][1][search_request_params][detected_locale][region]': 'GB',
                    'specs[async_search_results][1][search_request_params][locale][language]': 'en-GB',
                    'specs[async_search_results][1][search_request_params][locale][currency_code]': 'GBP',
                    'specs[async_search_results][1][search_request_params][locale][region]': 'GB',
                    'specs[async_search_results][1][search_request_params][name_map][query]': 'q',
                    'specs[async_search_results][1][search_request_params][name_map][query_type]': 'qt',
                    'specs[async_search_results][1][search_request_params][name_map][results_per_page]': 'result_count',
                    'specs[async_search_results][1][search_request_params][name_map][min_price]': 'min',
                    'specs[async_search_results][1][search_request_params][name_map][max_price]': 'max',
                    'specs[async_search_results][1][search_request_params][parameters][q]': '30s',
                    'specs[async_search_results][1][search_request_params][parameters][explicit]': '1',
                    'specs[async_search_results][1][search_request_params][parameters][locationQuery]': '2635167',
                    'specs[async_search_results][1][search_request_params][parameters][ship_to]': 'GB',
                    'specs[async_search_results][1][search_request_params][parameters][page]': '4',
                    'specs[async_search_results][1][search_request_params][parameters][ref]': 'pagination',
                    'specs[async_search_results][1][search_request_params][parameters][facet]': 'clothing/womens-clothing',
                    'specs[async_search_results][1][search_request_params][parameters][referrer]': 'https://www.etsy.com/search/clothing/womens-clothing?q=30s&amp;explicit=1locationQuery=2635167&amp;ship_to=GB&amp;page=3&amp;ref=pagination',
                    'specs[async_search_results][1][search_request_params][user_id]': '',
                    'specs[async_search_results][1][request_type]': 'pagination_preact',
                    'specs[async_search_results][1][is_eligible_for_spa_reformulations]': 'false',
                    'view_data_event_name': 'search_async_pagination_specview_rendered'
                },
                headers=headers,
                callback = self.parse
            )
    
    def parse(self, response):
        stuff = response.json().get('cssFiles')
        yield {
            'stuff':stuff
        }
</code></pre></div>
<p dir="auto">And when I include the following I can crawl the information:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=" cookies = {
            &quot;user_prefs&quot;: &quot;2sjEL59UUglDjNIW6TKc04MvLTVjZACCxJMbvsPoaKXQYBclnbzSnBwdpdQ83dBgJR2lUEeoiBGEwkXEMgAA&quot;,
            &quot;fve&quot;: &quot;1640607991.0&quot;,
            &quot;ua&quot;: &quot;531227642bc86f3b5fd7103a0c0b4fd6&quot;,
            &quot;_gcl_au&quot;: &quot;1.1.717562651.1640607992&quot;,
            &quot;uaid&quot;: &quot;E7bYwrWVwTy7YGe_b_ipYT3Avd9jZACCxJMbvoPpqwvzqpVKEzNTlKyUnLJ9Io3DTQt1k53MwiojXTLzvZPCS31yCoPC_JRqGQA.&quot;,
            &quot;pla_spr&quot;: &quot;0&quot;,
            &quot;_gid&quot;: &quot;GA1.2.1425785976.1641390447&quot;,
            &quot;_dc_gtm_UA-2409779-1&quot;: &quot;1&quot;,
            &quot;_pin_unauth&quot;: &quot;dWlkPU0yVTRaamxoTWpjdFlqTTVZUzAwT0RJeExXRmpNamt0WlROalpXTTVNREE0WkRVNQ&quot;,
            &quot;_ga&quot;: &quot;GA1.1.1730759327.1640607993&quot;,
            &quot;_uetsid&quot;: &quot;052ece906e2e11ecb56a0390ed629376&quot;,
            &quot;_uetvid&quot;: &quot;39de7550671011ec80d2dbfaa05c901b&quot;,
            &quot;exp_hangover&quot;: &quot;pB4zSokzfzMIT9Jzi7zIwmXybCJjZACCxJMbvoPpqwt7qpXKU5PiE4tKMtMykzMTc-JzEktS85Ir4wtN4o0MDC2VrJQy81JzMtMzk3JSlWoZAA..&quot;,
            &quot;_ga_KR3J610VYM&quot;: &quot;GS1.1.1641390446.2.1.1641390474.32&quot;

        }

        for url in self.start_urls:
            yield scrapy.FormRequest(
                'https://www.etsy.com/api/v3/ajax/bespoke/member/neu/specs/async_search_results',
                headers=headers,
                cookies=cookies,
                method=&quot;POST&quot;,
                formdata=data,
                callback = self.parse_res
            )"><pre class="notranslate"><code class="notranslate"> cookies = {
            "user_prefs": "2sjEL59UUglDjNIW6TKc04MvLTVjZACCxJMbvsPoaKXQYBclnbzSnBwdpdQ83dBgJR2lUEeoiBGEwkXEMgAA",
            "fve": "1640607991.0",
            "ua": "531227642bc86f3b5fd7103a0c0b4fd6",
            "_gcl_au": "1.1.717562651.1640607992",
            "uaid": "E7bYwrWVwTy7YGe_b_ipYT3Avd9jZACCxJMbvoPpqwvzqpVKEzNTlKyUnLJ9Io3DTQt1k53MwiojXTLzvZPCS31yCoPC_JRqGQA.",
            "pla_spr": "0",
            "_gid": "GA1.2.1425785976.1641390447",
            "_dc_gtm_UA-2409779-1": "1",
            "_pin_unauth": "dWlkPU0yVTRaamxoTWpjdFlqTTVZUzAwT0RJeExXRmpNamt0WlROalpXTTVNREE0WkRVNQ",
            "_ga": "GA1.1.1730759327.1640607993",
            "_uetsid": "052ece906e2e11ecb56a0390ed629376",
            "_uetvid": "39de7550671011ec80d2dbfaa05c901b",
            "exp_hangover": "pB4zSokzfzMIT9Jzi7zIwmXybCJjZACCxJMbvoPpqwt7qpXKU5PiE4tKMtMykzMTc-JzEktS85Ir4wtN4o0MDC2VrJQy81JzMtMzk3JSlWoZAA..",
            "_ga_KR3J610VYM": "GS1.1.1641390446.2.1.1641390474.32"

        }

        for url in self.start_urls:
            yield scrapy.FormRequest(
                'https://www.etsy.com/api/v3/ajax/bespoke/member/neu/specs/async_search_results',
                headers=headers,
                cookies=cookies,
                method="POST",
                formdata=data,
                callback = self.parse_res
            )
</code></pre></div>
<p dir="auto">I am new in scrapy, and I meet some problems which I can not get answer from google, so I post it here:</p>
<p dir="auto">1 Cookie not work even set in DEFAULT_REQUEST_HEADERS:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'accept-encoding': 'gzip, deflate, sdch',
    'cache-control': 'no-cache',
    'cookie': 'xx=yy',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
}"><pre class="notranslate"><code class="notranslate">DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'accept-encoding': 'gzip, deflate, sdch',
    'cache-control': 'no-cache',
    'cookie': 'xx=yy',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36'
}
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="class MySpider(scrapy.Spider):
    def make_requests_from_url(self, url):
        return scrapy.http.Request(url, headers=DEFAULT_REQUEST_HEADERS)"><pre class="notranslate"><code class="notranslate">class MySpider(scrapy.Spider):
    def make_requests_from_url(self, url):
        return scrapy.http.Request(url, headers=DEFAULT_REQUEST_HEADERS)
</code></pre></div>
<p dir="auto">I know the <code class="notranslate">make_requests_from_url</code> will only called once for the start_urls, and in my opinion, the first request will send the cookie I set in the <code class="notranslate">DEFAULT_REQUEST_HEADERS</code>, however it does not.</p>
<p dir="auto">2 Share settings between spiders.</p>
<p dir="auto">I have multiple spiders in the project which share most of the settings like <code class="notranslate">RandomAgentMiddleware</code> <code class="notranslate">RandomProxyMiddleware</code> <code class="notranslate">UserAgent</code> <code class="notranslate">DEFAULT_REQUEST_HEADERS</code> and etc, however they are configured inside the settings.py for each spider.</p>
<p dir="auto">Is it possible to share these settings?</p>
<hr>
<p dir="auto">The<br>
<code class="notranslate">COOKIES_ENABLED</code> is set to true.</p>