title: Get proxy response in middleware
body: <p>i have the following problem with scrapy in my middleware:</p>
  
  <p>I do a request to a site with https and also use a proxy. When defining a middleware and using <code>process_response</code> in it, <code>response.headers</code> does only have the headers from the website. Is there any way to get the headers from the CONNECT request the proxy tunnel establish? The proxy we are using is adding some informations as headers in this response, we want to use it in the middleware.
  I found out that in <code>TunnelingTCP4ClientEndpoint.processProxyResponse</code> the parameter <code>rcvd_bytes</code> has all infos i need. I didn't find a way to get the <code>rcvd_bytes</code> in my middleware.</p>
  
  <p>Also i found a similiar (same) issue from a year ago which is not solved: <a href="https://stackoverflow.com/questions/46828207/not-receiving-headers-scrapy-proxymesh">Not receiving headers Scrapy ProxyMesh</a></p>
  
  <p>Here is the example from the proxy website:</p>
  
  <p>For HTTPS the IP is in the CONNECT response header x-hola-ip Example for Proxy Peer IP of 5.6.7.8:</p>
  
  <pre><code>Request
  CONNECT example.com:80 HTTP/1.1
  Host: example.com:80
  Accept: */*
  
  Response:
  HTTP/1.1 200 OK
  Content-Type: text/html
  x-hola-ip: 5.6.7.8
  </code></pre>
  
  <p>I want to get x-hola-ip in this example.</p>
  
  <p>When using curl like <code>curl --proxy mysuperproxy https://stackoverflow.com</code> i get also the right data in the CONNECT response.</p>
  
  <p>If this is not possible my possible solution is to monkey patch the class somehow so far, or maybe you know a better solution for that in python.</p>
  
  <p>Thanks in advance for your help.</p>
  
  <p>Note: I also posted this question on the github issues of scrapy, i will update both sites if i find any solution :)</p>
  
  <p>Working solution with help of Matthew:</p>
  
  <pre><code>from scrapy.core.downloader.handlers.http11 import (
      HTTP11DownloadHandler, ScrapyAgent, TunnelingTCP4ClientEndpoint, TunnelError, TunnelingAgent
  )
  from scrapy import twisted_version
  
  class MyHTTPDownloader(HTTP11DownloadHandler):
      i = ''
      def download_request(self, request, spider):
          # we're just overriding here to monkey patch the attribute
          agent = ScrapyAgent(contextFactory=self._contextFactory, pool=self._pool,
              maxsize=getattr(spider, 'download_maxsize', self._default_maxsize),
              warnsize=getattr(spider, 'download_warnsize', self._default_warnsize),
              fail_on_dataloss=self._fail_on_dataloss)
  
  
          agent._TunnelingAgent = MyTunnelingAgent
  
          return agent.download_request(request)
  
  class MyTunnelingAgent(TunnelingAgent):
      if twisted_version &gt;= (15, 0, 0):
          def _getEndpoint(self, uri):
              return MyTunnelingTCP4ClientEndpoint(
                  self._reactor, uri.host, uri.port, self._proxyConf,
                  self._contextFactory, self._endpointFactory._connectTimeout,
                  self._endpointFactory._bindAddress)
      else:
          def _getEndpoint(self, scheme, host, port):
              return MyTunnelingTCP4ClientEndpoint(
                  self._reactor, host, port, self._proxyConf,
                  self._contextFactory, self._connectTimeout,
                  self._bindAddress)
  
  class MyTunnelingTCP4ClientEndpoint(TunnelingTCP4ClientEndpoint):
      def processProxyResponse(self, rcvd_bytes):
          # log('hier rcvd_bytes')
          MyHTTPDownloader.i = rcvd_bytes
          return super(MyTunnelingTCP4ClientEndpoint, self).processProxyResponse(rcvd_bytes)
  </code></pre>
  
  <p>And in your settings:</p>
  
  <pre><code>DOWNLOAD_HANDLERS = {
      'http': 'crawler.MyHTTPDownloader.MyHTTPDownloader',
      'https': 'crawler.MyHTTPDownloader.MyHTTPDownloader',
  }
  </code></pre>