Python's SimpleHTTPServer started in a thread won't close the port
Question: I have the following code:

    
    
    import os
    from ghost import Ghost
    import urlparse, urllib
    
    import SimpleHTTPServer
    import SocketServer
    
    import sys, traceback
    
    from threading import Thread, Event
    from time import sleep
    
    please_die = Event() # this is my enemy
    
    httpd = None
    PORT = 8001
    address = 'http://localhost:'+str(PORT)+'/'
    search_dir = './category'
    
    def main():
        """
          basic run script routine, 
          FIXME: is supossed to exits gracefully
        """
        thread = Thread(target = simpleServe)
        try:
          thread.start()
          run()
        except KeyboardInterrupt:
          print "Shutdown requested"
        except Exception:
          traceback.print_exc(file=sys.stdout)
    
        shutdown()
        sys.exit(0)
    
    def shutdown():
      global httpd
      global please_die
      print "Shutting down"
      # A try - except for the shutdown routine
      try:
        please_die.wait() # how do you do? 
    
        httpd.shutdown() # Please! I whant to run you multiple times. 
        print "Have you died?"
      except Exception:
        traceback.print_exc(file=sys.stdout)
    
    def path2url(path):
      """
      constructs an url from a relative path / concatenates the global address
      variable with the path given
      """
      global address
      return urlparse.urljoin(address, urllib.pathname2url(path))
    
    def simpleServe():
      global httpd, PORT
    
      please_die.set() # Attaching the event to this thread
    
      # Start the service
      Handler = SimpleHTTPServer.SimpleHTTPRequestHandler
    
      httpd = SocketServer.TCPServer(("", PORT), Handler)
    
      print "serving at port", PORT
      # And loop infinetly in the hope that I can stop you later
      httpd.serve_forever()
    
    def run():
      global search_dir;
    
      ghost = Ghost() # the webkit facade
    
      with ghost.start() as session:
    
        session.set_viewport_size(2560, 1600) # "retina" size
    
        for directory, subdirectories, files in os.walk(search_dir):
            for file in files:
                path = os.path.join(directory, file)
                urlPath = path2url(path)
                process(session, urlPath);
    
    def process(session, urlPath):
      page, resources = session.open(urlPath)
      assert page.http_status == 200
      # ... other asserts here 
    
    
    if __name__ == '__main__':
      main()
    

The idea is to make a script that starts a "simple http server", do some
requests on it and then exit.

First time it runs without any problems:

    
    
    ...
    127.0.0.1 - - [31/Jul/2015 13:16:17] "GET /category/52003.html HTTP/1.1" 200 -
    127.0.0.1 - - [31/Jul/2015 13:16:17] "GET /category/52003.html HTTP/1.1" 200 -
    127.0.0.1 - - [31/Jul/2015 13:16:17] "GET /category/52003.html HTTP/1.1" 200 -
    127.0.0.1 - - [31/Jul/2015 13:16:17] "GET /static/img/glyphicons-halflings.png HTTP/1.1" 200 -
    Shutting down
    Have you died?
    

Launching it the second time crashes saying that the:

> Address already in use
    
    
    Exception in thread Thread-1:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
        self.run()
      File "/usr/lib/python2.7/threading.py", line 763, in run
        self.__target(*self.__args, **self.__kwargs)
      File "download-images.py", line 51, in simpleServe
        httpd = SocketServer.TCPServer(("", PORT), Handler)
      File "/usr/lib/python2.7/SocketServer.py", line 420, in __init__
        self.server_bind()
      File "/usr/lib/python2.7/SocketServer.py", line 434, in server_bind
        self.socket.bind(self.server_address)
      File "/usr/lib/python2.7/socket.py", line 228, in meth
        return getattr(self._sock,name)(*args)
    error: [Errno 98] Address already in use
    

If I kill all python processes than the script runs again, and because of that
I'm assuming that I used the thread wrong, but I cannot find where.

## Update

Forgot to mention that,

my OS is :

    
    
    $ lsb_release -a
    No LSB modules are available.
    Distributor ID: Ubuntu
    Description:    Ubuntu 15.04
    Release:        15.04
    Codename:       vivid
    

The python that I'm using is :

    
    
    $ python --version
    Python 2.7.9
    

$ netstat -putelan | grep 8001 prints : 
    
    
    $ netstat -putelan | grep 8001
    (Not all processes could be identified, non-owned process info
        cp        0      0 127.0.0.1:34691         127.0.0.1:8001          TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:8001          127.0.0.1:34866         TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:34798         127.0.0.1:8001          TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:8001          127.0.0.1:34588         TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:34647         127.0.0.1:8001          TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:34915         127.0.0.1:8001          TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:34674         127.0.0.1:8001          TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:34451         127.0.0.1:8001          TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:8001          127.0.0.1:34930         TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:8001          127.0.0.1:34606         TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:34505         127.0.0.1:8001          TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:34717         127.0.0.1:8001          TIME_WAIT   0          0           -               
        tcp        0      0 127.0.0.1:8001          127.0.0.1:34670         0      0 127.0.0.1:8001          127.0.0.1:34626         
    ...
    

I can't post the whole sequence (due to the post limits of stackoverflow). The
rest is the same with 34*** port mixed with 8001 port in an uniform sequence.


Answer: As @LFJ say, this is probably due to the `allow_reuse_address` attribute of
the `TCPServer`.

    
    
    httpd = SocketServer.TCPServer(("", PORT), Handler, bind_and_activate=False)
    httpd.allow_reuse_address = True
    
    try:
        httpd.server_bind()
        httpd.server_activate()
    except:
        httpd.server_close()
        raise
    

Equivalent code :

    
    
    SocketServer.TCPServer.allow_reuse_address = True
    https = SocketServer.TCPServer(("", PORT), Handler)
    

Let's explain a bit why.

When you enable `TCPServer.allow_reuse_address`, it adds an option on the
socket :

    
    
    class TCPServer:
        [...]
        def server_bind(self):
            if self.allow_reuse_address:
                self.socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
            [...]
    

What is `socket.SO_REUSEADDR` ?

>
>     This socket option tells the kernel that even if this port is busy (in
>     the TIME_WAIT state), go ahead and reuse it anyway.  If it is busy,
>     but with another state, you will still get an address already in use
>     error.  It is useful if your server has been shut down, and then
>     restarted right away while sockets are still active on its port.  You
>     should be aware that if any unexpected data comes in, it may confuse
>     your server, but while this is possible, it is not likely.  
>  

In fact, it allows the reuse of your socket socket binding address. If another
process try to bind while the socket is not listening, the process will be
allowed to use this socket binding address.

The reason you need to enable that is because you don't shutdown properly your
`TCPServer`. In order to close it properly, you must run `shutdown` method,
which will close the thread launched by `server_forever` and then close the
socket properly by calling the `server_close` method.

    
    
    def shutdown():
        global httpd
        global please_die
        print "Shutting down"
    
        try:
            please_die.wait() # how do you do? 
            httpd.shutdown() # Stop the serve_forever
            httpd.server_close() # Close also the socket.
        except Exception:
            traceback.print_exc(file=sys.stdout)
    

