<h3 dir="auto">Description</h3>
<p dir="auto">I wrote my custom download middleware, which makes the request to my service(it lives on a different domain than a requested domain). After I receive a response from my service I replace URL in <code class="notranslate">response.url</code> and <code class="notranslate">response.request.url</code> to the original URL. But the issue is in <code class="notranslate">parse</code> function the URL in the <code class="notranslate">response.request</code> object is the URL from request to my service.</p>
<h3 dir="auto">Steps to Reproduce</h3>
<ol dir="auto">
<li>Original URL - <a href="http://example.com/some_page" rel="nofollow">http://example.com/some_page</a></li>
<li>URL of the cache request - http://my_service.com/?url=<a href="http://example.com/some_page" rel="nofollow">http://example.com/some_page</a></li>
<li>I replace the URL in middleware for <code class="notranslate">request</code> and <code class="notranslate">response</code> from step 2 to step 1</li>
<li>In <code class="notranslate">parse</code> function <code class="notranslate">response.url</code> is correct but <code class="notranslate">request.url</code> is from step 2</li>
</ol>
<p dir="auto"><strong>Expected behavior:</strong> request URL will be correct</p>
<p dir="auto"><strong>Actual behavior:</strong> request URL isn't correct</p>
<p dir="auto"><strong>Reproduces how often:</strong> each time</p>
<h3 dir="auto">Versions</h3>
<p dir="auto">1.8.0 - 2.1.0</p>
<h3 dir="auto">Additional context</h3>
<p dir="auto">Here is the code of my middleware</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="class MyServiceDownloaderMiddleware:

    def process_request(self, request, spider):
        if request.meta.get('__service_search') or request.meta.get('__service_action'):
            return None
        elif request.url.startswith(spider.eligible_url):
            qs = {
                'url': request.url,
            }
            request.meta['__service_search'] = True
            request.meta['__original_url'] = request.url

            new_request = request.replace(url=f'https://my_service.com/?{urlencode(qs)}')
            return new_request
        return None

    def process_response(self, request, response, spider):
        if request.meta.get('__service_action'):
            request = request.replace(url=request.meta['__original_url'])
            response = response.replace(url=request.meta['__original_url'], request=request)
            return response
        elif request.meta.get('__service_search'):
            b_attribute = response.css('div.b_attr:attr(u)').extract_first()
            if not b_attribute:
                request = request.replace(url=request.meta['__original_url'])
                request.meta.pop('__original_url')
                request.meta.pop('__service_search')
                response = response.replace(status=404, url=request.meta['__original_url'], request=request)
                
                return response
            _, _, d, w = b_attribute.split('|')
            qs = {
                'q': request.meta['__original_url'],
                'd': d,
                'w': w,
            }
            url = f'https://my_service.com/action/?{urlencode(qs)}'
            new_request = request.replace(url=url)
            new_request.meta['__service_action'] = True
            return new_request
        return response"><pre class="notranslate"><code class="notranslate">class MyServiceDownloaderMiddleware:

    def process_request(self, request, spider):
        if request.meta.get('__service_search') or request.meta.get('__service_action'):
            return None
        elif request.url.startswith(spider.eligible_url):
            qs = {
                'url': request.url,
            }
            request.meta['__service_search'] = True
            request.meta['__original_url'] = request.url

            new_request = request.replace(url=f'https://my_service.com/?{urlencode(qs)}')
            return new_request
        return None

    def process_response(self, request, response, spider):
        if request.meta.get('__service_action'):
            request = request.replace(url=request.meta['__original_url'])
            response = response.replace(url=request.meta['__original_url'], request=request)
            return response
        elif request.meta.get('__service_search'):
            b_attribute = response.css('div.b_attr:attr(u)').extract_first()
            if not b_attribute:
                request = request.replace(url=request.meta['__original_url'])
                request.meta.pop('__original_url')
                request.meta.pop('__service_search')
                response = response.replace(status=404, url=request.meta['__original_url'], request=request)
                
                return response
            _, _, d, w = b_attribute.split('|')
            qs = {
                'q': request.meta['__original_url'],
                'd': d,
                'w': w,
            }
            url = f'https://my_service.com/action/?{urlencode(qs)}'
            new_request = request.replace(url=url)
            new_request.meta['__service_action'] = True
            return new_request
        return response
</code></pre></div>
<h2 dir="auto">Summary</h2>
<p dir="auto">Currently requests are bound to responses in <a href="https://github.com/scrapy/scrapy/blob/2.1.0/scrapy/core/engine.py#L247">the engine</a> and/or in <a href="https://github.com/scrapy/scrapy/blob/2.1.0/scrapy/core/scraper.py#L149">the scraper</a>.<br>
I would like to start a conversation to figure out the implications of binding them only if the <code class="notranslate">response.request</code> attribute is not already set, i.e. something like this:</p>
<div class="highlight highlight-source-diff notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="diff --git scrapy/core/engine.py scrapy/core/engine.py
index 77d71846..72f1628a 100644
--- scrapy/core/engine.py
+++ scrapy/core/engine.py
@@ -244,7 +244,8 @@ class ExecutionEngine:
                     % (type(response), response)
                 )
             if isinstance(response, Response):
-                response.request = request  # tie request to response received
+                if response.request is None:
+                    response.request = request
                 logkws = self.logformatter.crawled(request, response, spider)
                 if logkws is not None:
                     logger.log(*logformatter_adapter(logkws), extra={'spider': spider})


diff --git scrapy/core/scraper.py scrapy/core/scraper.py
index edbb4dd6..256935da 100644
--- scrapy/core/scraper.py
+++ scrapy/core/scraper.py
@@ -146,7 +146,8 @@ class Scraper:
                 self._log_download_errors, request_result, request, spider)
 
     def call_spider(self, result, request, spider):
-        result.request = request
+        if not hasattr(result, &quot;request&quot;) or result.request is None:
+            result.request = request
         dfd = defer_result(result)
         callback = request.callback or spider.parse
         warn_on_generator_with_return_value(spider, callback)"><pre class="notranslate"><span class="pl-c1">diff --git scrapy/core/engine.py scrapy/core/engine.py</span>
index 77d71846..72f1628a 100644
<span class="pl-md">--- scrapy/core/engine.py</span>
<span class="pl-mi1">+++ scrapy/core/engine.py</span>
<span class="pl-mdr">@@ -244,7 +244,8 @@</span> class ExecutionEngine:
                     % (type(response), response)
                 )
             if isinstance(response, Response):
<span class="pl-md"><span class="pl-md">-</span>                response.request = request  # tie request to response received</span>
<span class="pl-mi1"><span class="pl-mi1">+</span>                if response.request is None:</span>
<span class="pl-mi1"><span class="pl-mi1">+</span>                    response.request = request</span>
                 logkws = self.logformatter.crawled(request, response, spider)
                 if logkws is not None:
                     logger.log(*logformatter_adapter(logkws), extra={'spider': spider})


<span class="pl-c1">diff --git scrapy/core/scraper.py scrapy/core/scraper.py</span>
index edbb4dd6..256935da 100644
<span class="pl-md">--- scrapy/core/scraper.py</span>
<span class="pl-mi1">+++ scrapy/core/scraper.py</span>
<span class="pl-mdr">@@ -146,7 +146,8 @@</span> class Scraper:
                 self._log_download_errors, request_result, request, spider)
 
     def call_spider(self, result, request, spider):
<span class="pl-md"><span class="pl-md">-</span>        result.request = request</span>
<span class="pl-mi1"><span class="pl-mi1">+</span>        if not hasattr(result, "request") or result.request is None:</span>
<span class="pl-mi1"><span class="pl-mi1">+</span>            result.request = request</span>
         dfd = defer_result(result)
         callback = request.callback or spider.parse
         warn_on_generator_with_return_value(spider, callback)</pre></div>
<h2 dir="auto">Motivation</h2>
<p dir="auto">Recently I've been working on a downloader middleware to transparently redirect requests to a service which takes an URL and returns its response body along with some other metadata. The <code class="notranslate">process_request</code> method replaces the URL, and the <code class="notranslate">process_response</code> decodes the response body and creates a new response, making it look like it came directly from the target website, not from the download service. At this point, I'd like to be able to restore the original request as well, but any request I return in the <code class="notranslate">Response.request</code> attribute gets overridden.</p>
<h2 dir="auto">Describe alternatives you've considered</h2>
<p dir="auto">ATM I don't see a way to modify this behaviour without altering the Scrapy core.</p>