We are currently solving the following issue within our repository. Here is the issue text:
--- BEGIN ISSUE ---
Offsite middleware ignoring port
In my spider I have the following:

class MySpider(BaseSpider):

``` python
    allowed_domains = ['192.169.0.15:8080']
```

and in the parse method I do something like:

``` python
    yield Request('http://192.169.0.15:8080/mypage.html', self.my_callback_function)
```

the result when I run the code is that that scrapy reports:

 DEBUG: Filtered offsite request to '192.168.0.15': <GET http://192.168.0.15:8080/mypage.html>

Which is wrong - it seems to be ignoring the port. If I change the allowed_domains to:

``` python
    allowed_domains = ['192.169.0.15:8080', '192.16.0.15']
```

Then it works as you would expect it to. No big deal, can work around it but I think it is a bug. The problem being located in the should_follow method of the OffsiteMiddleware class in contrib/spidermiddleware/offsite.py

Fixed "Offsite middleware ignoring port #50" issue
ref: https://github.com/scrapy/scrapy/issues/50

Although a [patch](https://github.com/scrapy/scrapy/issues/50#issuecomment-289200833) was suggested, simply applying it fails another test. As shown here. 

```

============================================================ FAILURES ============================================================
____________________________________________________ EngineTest.test_crawler _____________________________________________________

result = None, g = <generator object test_crawler at 0x7f396bc25780>, deferred = <Deferred at 0x7f396b6a6a28 current result: None>

    def _inlineCallbacks(result, g, deferred):
        """
        See L{inlineCallbacks}.
        """
        # This function is complicated by the need to prevent unbounded recursion
        # arising from repeatedly yielding immediately ready deferreds.  This while
        # loop and the waiting variable solve that by manually unfolding the
        # recursion.

        waiting = [True, # waiting for result?
                   None] # result

        while 1:
            try:
                # Send the last result back as the result of the yield expression.
                isFailure = isinstance(result, failure.Failure)
                if isFailure:
                    result = result.throwExceptionIntoGenerator(g)
                else:
>                   result = g.send(result)

/scrapy/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py:1386:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/scrapy/tests/test_engine.py:167: in test_crawler
    self._assert_visited_urls()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <tests.test_engine.EngineTest testMethod=test_crawler>

    def _assert_visited_urls(self):
        must_be_visited = ["/", "/redirect", "/redirected",
                           "/item1.html", "/item2.html", "/item999.html"]
        urls_visited = set([rp[0].url for rp in self.run.respplug])
        urls_expected = set([self.run.geturl(p) for p in must_be_visited])
>       assert urls_expected <= urls_visited, "URLs not visited: %s" % list(urls_expected - urls_visited)
E       AssertionError: URLs not visited: ['http://localhost:42519/item2.html', 'http://localhost:42519/item999.html', 'http://localhost:42519/item1.html']

/scrapy/tests/test_engine.py:183: AssertionError
=============================================== 1 failed, 3 passed in 1.37 seconds ===============================================
ERROR: InvocationError: '/scrapy/.tox/py27/bin/py.test --cov=scrapy --cov-report= tests/test_engine.py'
____________________________________________________________ summary _____________________________________________________________

```

So, "/scrapy/tests/test_engine.py" is failing.
It looks the the test creates spiders by using  

```
allowed_domains = ["scrapytest.org", "localhost"] 
```

A spider fails to visit some urls with port number like 'http://localhost:37089/item1.html'.
Since test server is launched each time we run the test, it has different port number for local host each time. This means we cannot add a fixed port number to "localhost" in 'allowed_domain" list.

So I modified the patch to... 
```py
# scrapy/spidermiddlewares/offsite.py

    def should_follow(self, request, spider):
        regex = self.host_regex
        # hostname can be None for wrong urls (like javascript links)
        hostname = urlparse_cached(request).hostname or ''
        netloc = urlparse_cached(request).netloc or ''
        return bool(regex.search(hostname)) or bool(regex.search(netloc))
```




--- END ISSUE ---

Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
--- BEGIN FILES ---
Path: `scrapy/spidermiddlewares/offsite.py`
Content:
```
1 """
2 Offsite Spider Middleware
3 
4 See documentation in docs/topics/spider-middleware.rst
5 """
6 import re
7 import logging
8 import warnings
9 
10 from scrapy import signals
11 from scrapy.http import Request
12 from scrapy.utils.httpobj import urlparse_cached
13 
14 logger = logging.getLogger(__name__)
15 
16 
17 class OffsiteMiddleware(object):
18 
19     def __init__(self, stats):
20         self.stats = stats
21 
22     @classmethod
23     def from_crawler(cls, crawler):
24         o = cls(crawler.stats)
25         crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
26         return o
27 
28     def process_spider_output(self, response, result, spider):
29         for x in result:
30             if isinstance(x, Request):
31                 if x.dont_filter or self.should_follow(x, spider):
32                     yield x
33                 else:
34                     domain = urlparse_cached(x).hostname
35                     if domain and domain not in self.domains_seen:
36                         self.domains_seen.add(domain)
37                         logger.debug(
38                             "Filtered offsite request to %(domain)r: %(request)s",
39                             {'domain': domain, 'request': x}, extra={'spider': spider})
40                         self.stats.inc_value('offsite/domains', spider=spider)
41                     self.stats.inc_value('offsite/filtered', spider=spider)
42             else:
43                 yield x
44 
45     def should_follow(self, request, spider):
46         regex = self.host_regex
47         # hostname can be None for wrong urls (like javascript links)
48         host = urlparse_cached(request).hostname or ''
49         return bool(regex.search(host))
50 
51     def get_host_regex(self, spider):
52         """Override this method to implement a different offsite policy"""
53         allowed_domains = getattr(spider, 'allowed_domains', None)
54         if not allowed_domains:
55             return re.compile('')  # allow all by default
56         url_pattern = re.compile("^https?://.*$")
57         for domain in allowed_domains:
58             if url_pattern.match(domain):
59                 message = ("allowed_domains accepts only domains, not URLs. "
60                            "Ignoring URL entry %s in allowed_domains." % domain)
61                 warnings.warn(message, URLWarning)
62         domains = [re.escape(d) for d in allowed_domains if d is not None]
63         regex = r'^(.*\.)?(%s)$' % '|'.join(domains)
64         return re.compile(regex)
65 
66     def spider_opened(self, spider):
67         self.host_regex = self.get_host_regex(spider)
68         self.domains_seen = set()
69 
70 
71 class URLWarning(Warning):
72     pass
73
```
--- END FILES ---

Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.

Here is an example:

```diff
diff --git a/examples/server_async.py b/examples/server_async.py
--- a/examples/server_async.py
+++ b/examples/server_async.py
@@ -313,4 +313,4 @@
 
 
 if __name__ == "__main__":
-    asyncio.run(run_async_server("."), debug=True)
+    asyncio.run(run_async_server(), debug=True)
diff --git a/examples/server_sync.py b/examples/server_sync.py
--- a/examples/server_sync.py
+++ b/examples/server_sync.py
@@ -313,5 +313,5 @@
 
 
 if __name__ == "__main__":
-    server = run_sync_server(".")
+    server = run_sync_server()
     server.shutdown()

```

diff --git a/scrapy/spidermiddlewares/offsite.py b/scrapy/spidermiddlewares/offsite.py
--- a/scrapy/spidermiddlewares/offsite.py
+++ b/scrapy/spidermiddlewares/offsite.py
@@ -53,13 +53,22 @@
         allowed_domains = getattr(spider, 'allowed_domains', None)
         if not allowed_domains:
             return re.compile('')  # allow all by default
-        url_pattern = re.compile("^https?://.*$")
+        url_pattern = re.compile(r"^https?://.*$")
+        port_pattern = re.compile(r":\d+$")
+        domains = []
         for domain in allowed_domains:
-            if url_pattern.match(domain):
+            if domain is None:
+                continue
+            elif url_pattern.match(domain):
                 message = ("allowed_domains accepts only domains, not URLs. "
                            "Ignoring URL entry %s in allowed_domains." % domain)
                 warnings.warn(message, URLWarning)
-        domains = [re.escape(d) for d in allowed_domains if d is not None]
+            elif port_pattern.search(domain):
+                message = ("allowed_domains accepts only domains without ports. "
+                           "Ignoring entry %s in allowed_domains." % domain)
+                warnings.warn(message, PortWarning)
+            else:
+                domains.append(re.escape(domain))
         regex = r'^(.*\.)?(%s)$' % '|'.join(domains)
         return re.compile(regex)
 
@@ -70,3 +79,7 @@
 
 class URLWarning(Warning):
     pass
+
+
+class PortWarning(Warning):
+    pass

{"golden_diff": "diff --git a/scrapy/spidermiddlewares/offsite.py b/scrapy/spidermiddlewares/offsite.py\n--- a/scrapy/spidermiddlewares/offsite.py\n+++ b/scrapy/spidermiddlewares/offsite.py\n@@ -53,13 +53,22 @@\n         allowed_domains = getattr(spider, 'allowed_domains', None)\n         if not allowed_domains:\n             return re.compile('')  # allow all by default\n-        url_pattern = re.compile(\"^https?://.*$\")\n+        url_pattern = re.compile(r\"^https?://.*$\")\n+        port_pattern = re.compile(r\":\\d+$\")\n+        domains = []\n         for domain in allowed_domains:\n-            if url_pattern.match(domain):\n+            if domain is None:\n+                continue\n+            elif url_pattern.match(domain):\n                 message = (\"allowed_domains accepts only domains, not URLs. \"\n                            \"Ignoring URL entry %s in allowed_domains.\" % domain)\n                 warnings.warn(message, URLWarning)\n-        domains = [re.escape(d) for d in allowed_domains if d is not None]\n+            elif port_pattern.search(domain):\n+                message = (\"allowed_domains accepts only domains without ports. \"\n+                           \"Ignoring entry %s in allowed_domains.\" % domain)\n+                warnings.warn(message, PortWarning)\n+            else:\n+                domains.append(re.escape(domain))\n         regex = r'^(.*\\.)?(%s)$' % '|'.join(domains)\n         return re.compile(regex)\n \n@@ -70,3 +79,7 @@\n \n class URLWarning(Warning):\n     pass\n+\n+\n+class PortWarning(Warning):\n+    pass\n", "issue": "Offsite middleware ignoring port\nIn my spider I have the following:\n\nclass MySpider(BaseSpider):\n\n``` python\n    allowed_domains = ['192.169.0.15:8080']\n```\n\nand in the parse method I do something like:\n\n``` python\n    yield Request('http://192.169.0.15:8080/mypage.html', self.my_callback_function)\n```\n\nthe result when I run the code is that that scrapy reports:\n\n DEBUG: Filtered offsite request to '192.168.0.15': <GET http://192.168.0.15:8080/mypage.html>\n\nWhich is wrong - it seems to be ignoring the port. If I change the allowed_domains to:\n\n``` python\n    allowed_domains = ['192.169.0.15:8080', '192.16.0.15']\n```\n\nThen it works as you would expect it to. No big deal, can work around it but I think it is a bug. The problem being located in the should_follow method of the OffsiteMiddleware class in contrib/spidermiddleware/offsite.py\n\nFixed \"Offsite middleware ignoring port #50\" issue\nref: https://github.com/scrapy/scrapy/issues/50\r\n\r\nAlthough a [patch](https://github.com/scrapy/scrapy/issues/50#issuecomment-289200833) was suggested, simply applying it fails another test. As shown here. \r\n\r\n```\r\n\r\n============================================================ FAILURES ============================================================\r\n____________________________________________________ EngineTest.test_crawler _____________________________________________________\r\n\r\nresult = None, g = <generator object test_crawler at 0x7f396bc25780>, deferred = <Deferred at 0x7f396b6a6a28 current result: None>\r\n\r\n    def _inlineCallbacks(result, g, deferred):\r\n        \"\"\"\r\n        See L{inlineCallbacks}.\r\n        \"\"\"\r\n        # This function is complicated by the need to prevent unbounded recursion\r\n        # arising from repeatedly yielding immediately ready deferreds.  This while\r\n        # loop and the waiting variable solve that by manually unfolding the\r\n        # recursion.\r\n\r\n        waiting = [True, # waiting for result?\r\n                   None] # result\r\n\r\n        while 1:\r\n            try:\r\n                # Send the last result back as the result of the yield expression.\r\n                isFailure = isinstance(result, failure.Failure)\r\n                if isFailure:\r\n                    result = result.throwExceptionIntoGenerator(g)\r\n                else:\r\n>                   result = g.send(result)\r\n\r\n/scrapy/.tox/py27/local/lib/python2.7/site-packages/twisted/internet/defer.py:1386:\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n/scrapy/tests/test_engine.py:167: in test_crawler\r\n    self._assert_visited_urls()\r\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\r\n\r\nself = <tests.test_engine.EngineTest testMethod=test_crawler>\r\n\r\n    def _assert_visited_urls(self):\r\n        must_be_visited = [\"/\", \"/redirect\", \"/redirected\",\r\n                           \"/item1.html\", \"/item2.html\", \"/item999.html\"]\r\n        urls_visited = set([rp[0].url for rp in self.run.respplug])\r\n        urls_expected = set([self.run.geturl(p) for p in must_be_visited])\r\n>       assert urls_expected <= urls_visited, \"URLs not visited: %s\" % list(urls_expected - urls_visited)\r\nE       AssertionError: URLs not visited: ['http://localhost:42519/item2.html', 'http://localhost:42519/item999.html', 'http://localhost:42519/item1.html']\r\n\r\n/scrapy/tests/test_engine.py:183: AssertionError\r\n=============================================== 1 failed, 3 passed in 1.37 seconds ===============================================\r\nERROR: InvocationError: '/scrapy/.tox/py27/bin/py.test --cov=scrapy --cov-report= tests/test_engine.py'\r\n____________________________________________________________ summary _____________________________________________________________\r\n\r\n```\r\n\r\nSo, \"/scrapy/tests/test_engine.py\" is failing.\r\nIt looks the the test creates spiders by using  \r\n\r\n```\r\nallowed_domains = [\"scrapytest.org\", \"localhost\"] \r\n```\r\n\r\nA spider fails to visit some urls with port number like 'http://localhost:37089/item1.html'.\r\nSince test server is launched each time we run the test, it has different port number for local host each time. This means we cannot add a fixed port number to \"localhost\" in 'allowed_domain\" list.\r\n\r\nSo I modified the patch to... \r\n```py\r\n# scrapy/spidermiddlewares/offsite.py\r\n\r\n    def should_follow(self, request, spider):\r\n        regex = self.host_regex\r\n        # hostname can be None for wrong urls (like javascript links)\r\n        hostname = urlparse_cached(request).hostname or ''\r\n        netloc = urlparse_cached(request).netloc or ''\r\n        return bool(regex.search(hostname)) or bool(regex.search(netloc))\r\n```\r\n\r\n\r\n\n", "before_files": [{"content": "\"\"\"\nOffsite Spider Middleware\n\nSee documentation in docs/topics/spider-middleware.rst\n\"\"\"\nimport re\nimport logging\nimport warnings\n\nfrom scrapy import signals\nfrom scrapy.http import Request\nfrom scrapy.utils.httpobj import urlparse_cached\n\nlogger = logging.getLogger(__name__)\n\n\nclass OffsiteMiddleware(object):\n\n    def __init__(self, stats):\n        self.stats = stats\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        o = cls(crawler.stats)\n        crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)\n        return o\n\n    def process_spider_output(self, response, result, spider):\n        for x in result:\n            if isinstance(x, Request):\n                if x.dont_filter or self.should_follow(x, spider):\n                    yield x\n                else:\n                    domain = urlparse_cached(x).hostname\n                    if domain and domain not in self.domains_seen:\n                        self.domains_seen.add(domain)\n                        logger.debug(\n                            \"Filtered offsite request to %(domain)r: %(request)s\",\n                            {'domain': domain, 'request': x}, extra={'spider': spider})\n                        self.stats.inc_value('offsite/domains', spider=spider)\n                    self.stats.inc_value('offsite/filtered', spider=spider)\n            else:\n                yield x\n\n    def should_follow(self, request, spider):\n        regex = self.host_regex\n        # hostname can be None for wrong urls (like javascript links)\n        host = urlparse_cached(request).hostname or ''\n        return bool(regex.search(host))\n\n    def get_host_regex(self, spider):\n        \"\"\"Override this method to implement a different offsite policy\"\"\"\n        allowed_domains = getattr(spider, 'allowed_domains', None)\n        if not allowed_domains:\n            return re.compile('')  # allow all by default\n        url_pattern = re.compile(\"^https?://.*$\")\n        for domain in allowed_domains:\n            if url_pattern.match(domain):\n                message = (\"allowed_domains accepts only domains, not URLs. \"\n                           \"Ignoring URL entry %s in allowed_domains.\" % domain)\n                warnings.warn(message, URLWarning)\n        domains = [re.escape(d) for d in allowed_domains if d is not None]\n        regex = r'^(.*\\.)?(%s)$' % '|'.join(domains)\n        return re.compile(regex)\n\n    def spider_opened(self, spider):\n        self.host_regex = self.get_host_regex(spider)\n        self.domains_seen = set()\n\n\nclass URLWarning(Warning):\n    pass\n", "path": "scrapy/spidermiddlewares/offsite.py"}], "after_files": [{"content": "\"\"\"\nOffsite Spider Middleware\n\nSee documentation in docs/topics/spider-middleware.rst\n\"\"\"\nimport re\nimport logging\nimport warnings\n\nfrom scrapy import signals\nfrom scrapy.http import Request\nfrom scrapy.utils.httpobj import urlparse_cached\n\nlogger = logging.getLogger(__name__)\n\n\nclass OffsiteMiddleware(object):\n\n    def __init__(self, stats):\n        self.stats = stats\n\n    @classmethod\n    def from_crawler(cls, crawler):\n        o = cls(crawler.stats)\n        crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)\n        return o\n\n    def process_spider_output(self, response, result, spider):\n        for x in result:\n            if isinstance(x, Request):\n                if x.dont_filter or self.should_follow(x, spider):\n                    yield x\n                else:\n                    domain = urlparse_cached(x).hostname\n                    if domain and domain not in self.domains_seen:\n                        self.domains_seen.add(domain)\n                        logger.debug(\n                            \"Filtered offsite request to %(domain)r: %(request)s\",\n                            {'domain': domain, 'request': x}, extra={'spider': spider})\n                        self.stats.inc_value('offsite/domains', spider=spider)\n                    self.stats.inc_value('offsite/filtered', spider=spider)\n            else:\n                yield x\n\n    def should_follow(self, request, spider):\n        regex = self.host_regex\n        # hostname can be None for wrong urls (like javascript links)\n        host = urlparse_cached(request).hostname or ''\n        return bool(regex.search(host))\n\n    def get_host_regex(self, spider):\n        \"\"\"Override this method to implement a different offsite policy\"\"\"\n        allowed_domains = getattr(spider, 'allowed_domains', None)\n        if not allowed_domains:\n            return re.compile('')  # allow all by default\n        url_pattern = re.compile(r\"^https?://.*$\")\n        port_pattern = re.compile(r\":\\d+$\")\n        domains = []\n        for domain in allowed_domains:\n            if domain is None:\n                continue\n            elif url_pattern.match(domain):\n                message = (\"allowed_domains accepts only domains, not URLs. \"\n                           \"Ignoring URL entry %s in allowed_domains.\" % domain)\n                warnings.warn(message, URLWarning)\n            elif port_pattern.search(domain):\n                message = (\"allowed_domains accepts only domains without ports. \"\n                           \"Ignoring entry %s in allowed_domains.\" % domain)\n                warnings.warn(message, PortWarning)\n            else:\n                domains.append(re.escape(domain))\n        regex = r'^(.*\\.)?(%s)$' % '|'.join(domains)\n        return re.compile(regex)\n\n    def spider_opened(self, spider):\n        self.host_regex = self.get_host_regex(spider)\n        self.domains_seen = set()\n\n\nclass URLWarning(Warning):\n    pass\n\n\nclass PortWarning(Warning):\n    pass\n", "path": "scrapy/spidermiddlewares/offsite.py"}]}