WAL-E: Cannot restart postgresql after backup-fetch
Question: This is on Ubuntu 14.04 LTS

**EDIT** :

WAL-E installed using pip, with secret keys managed by `envdir`, according to
the instructions <https://gist.github.com/elithrar/8682235> and
<https://github.com/wal-e/wal-e#dependencies>.

I am trying to restore a database using WAL-E, and everything appears to go
well initially, as I have Postgres installed and running, and can easily
create or restore a database and access it locally and remotely via pgadmin.
Where it goes bad is when I try to perform a restore from an S3 backup using
wal-e fetch-backup. It appears to go well up until the point of starting the
postgres.

There are a number of errors that come up, appearing to be either missing
packages or permissions issues, like the following:

    
    
    * Starting PostgreSQL 9.3 database server 
        * The PostgreSQL server failed to start. Please check the log output:
        2015-07-11 00:41:11 EDT LOG:  database system was interrupted; last known up at 2015-06-30 05:00:02 EDT
            2015-07-11 00:41:11 EDT LOG:  starting archive recovery
            Traceback (most recent call last):
              File "/usr/local/bin/wal-e", line 5, in <module>
                from pkg_resources import load_entry_point
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3084, in <module>
                @_call_aside
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3070, in _call_aside
                f(*args, **kwargs)
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3097, in _initialize_master_working_set
                working_set = WorkingSet._build_master()
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 651, in _build_master
                ws.require(__requires__)
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 952, in require
                needed = self.resolve(parse_requirements(requirements))
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 847, in resolve
                new_requirements = dist.requires(req.extras)[::-1]
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2602, in requires
                dm = self._dep_map
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2803, in _dep_map
                self.__dep_map = self._compute_dependencies()
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2825, in _compute_dependencies
                for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2794, in _parsed_pkg_info
                metadata = self.get_metadata(self.PKG_INFO)
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1617, in get_metadata
                return self._get(self._fn(self.egg_info, name))
              File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 1728, in _get
                with open(path, 'rb') as stream:
            IOError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/six-1.9.0.dist-info/METADATA'
            2015-07-11 00:41:11 EDT LOG:  invalid checkpoint record
            2015-07-11 00:41:11 EDT FATAL:  could not locate required checkpoint record
            2015-07-11 00:41:11 EDT HINT:  If you are not restoring from a backup, try removing the file "/var/lib/postgresql/9.3/main/backup_label".
            2015-07-11 00:41:11 EDT LOG:  startup process (PID 1693) exited with exit code 1
            2015-07-11 00:41:11 EDT LOG:  aborting startup due to startup process failure
    

I had several of these and was able to resolve them by changing the group and
modifying the permissions on the noted files to match others in the directory,
but I suspect the problem has more to do with how and/or where these packages
were installed. After resolving the above issue, postgres still fails to start
up, returning the following:

    
    
    * Starting PostgreSQL 9.3 database server
    * The PostgreSQL server failed to start. Please check the log output:
    2015-07-11 00:30:04 EDT LOG:  database system was interrupted; last known up at 2015-06-30 05:00:02 EDT
    2015-07-11 00:30:04 EDT LOG:  starting archive recovery
    Traceback (most recent call last):
      File "/usr/local/bin/wal-e", line 5, in <module>
        from pkg_resources import load_entry_point
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3084, in <module>
        @_call_aside
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3070, in _call_aside
        f(*args, **kwargs)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3097, in _initialize_master_working_set
        working_set = WorkingSet._build_master()
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 651, in _build_master
        ws.require(__requires__)
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 952, in require
        needed = self.resolve(parse_requirements(requirements))
      File "/usr/local/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 839, in resolve
        raise DistributionNotFound(req, requirers)
    pkg_resources.DistributionNotFound: The 'wal-e==0.8.1' distribution was not found and is required by the application
    2015-07-11 00:30:04 EDT LOG:  invalid checkpoint record
    2015-07-11 00:30:04 EDT FATAL:  could not locate required checkpoint record
    2015-07-11 00:30:04 EDT HINT:  If you are not restoring from a backup, try removing the file "/var/lib/postgresql/9.3/main/backup_label".
    2015-07-11 00:30:04 EDT LOG:  startup process (PID 1495) exited with exit code 1
    2015-07-11 00:30:04 EDT LOG:  aborting startup due to startup process failure
    

It is complaining about `the 'wal-e==0.8.1' distribution was not found...`,
but it is clearly installed and executable:

    
    
    ls -l /usr/local/lib/python2.7/dist-packages
    total 608
    drwxr-sr-x  2 root staff  4096 Jul 10 20:14 argparse-1.3.0.dist-info
    -rw-r--r--  1 root staff 88400 Jul 10 20:14 argparse.py
    -rw-r--r--  1 root staff 65659 Jul 10 20:14 argparse.pyc
    drwxr-sr-x  6 root staff  4096 Jul  9 22:36 azure
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 azure-0.11.1.egg-info
    drwxr-sr-x  5 root staff  4096 Jul  9 22:36 babel
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 Babel-1.3.egg-info
    drwxr-sr-x 57 root staff  4096 Jul  9 22:36 boto
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 boto-2.38.0.dist-info
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 concurrent
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 dateutil
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 debtcollector
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 debtcollector-0.5.0.dist-info
    -rw-r--r--  1 root staff   207 Jul 10 20:10 easy-install.pth
    -rw-r--r--  1 root staff   126 Jul 10 20:33 easy_install.py
    -rw-r--r--  1 root staff   315 Jul 10 20:33 easy_install.pyc
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 futures-3.0.3.dist-info
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 gevent
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 gevent-1.0.2.egg-info
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 greenlet-0.4.7.egg-info
    -rwxr-xr-x  1 root staff 82869 Jul  9 22:36 greenlet.so
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 iso8601
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 iso8601-0.1.10.egg-info
    drwxr-sr-x 15 root staff  4096 Jul  9 22:36 keystoneclient
    drwxr-sr-x  2 root staff  4096 Jul 10 20:33 _markerlib
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 msgpack
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 msgpack_python-0.4.6.egg-info
    drwxr-sr-x  5 root staff  4096 Jul  9 22:36 netaddr
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 netaddr-0.7.15.dist-info
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 netifaces-0.10.4.egg-info
    -rwxr-xr-x  1 root staff 58386 Jul  9 22:36 netifaces.so
    drwxr-sr-x  4 root staff  4096 Jul  9 22:36 oslo
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 oslo_config
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 oslo.config-1.14.0.dist-info
    -rw-r--r--  1 root root    299 Jul  9 22:35 oslo.config-1.14.0-py2.7-nspkg.pth
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 oslo_i18n
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 oslo.i18n-2.1.0.dist-info
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 oslo_serialization
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 oslo.serialization-1.7.0.dist-info
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 oslo_utils
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 oslo.utils-1.8.0.dist-info
    -rw-r--r--  1 root root    299 Jul  9 22:35 oslo.utils-1.8.0-py2.7-nspkg.pth
    drwxr-sr-x  5 root staff  4096 Jul 10 20:14 pbr
    drwxr-sr-x  2 root staff  4096 Jul 10 20:14 pbr-1.3.0.dist-info
    drwxr-sr-x  4 root staff  4096 Jul 10 20:10 pip-7.1.0-py2.7.egg
    drwxr-sr-x  3 root staff  4096 Jul 10 20:33 pkg_resources
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 python_dateutil-2.4.2.dist-info
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 python_keystoneclient-1.6.0.dist-info
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 python_swiftclient-2.4.0.dist-info
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 pytz
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 pytz-2015.4.dist-info
    drwxr-sr--  3 root staff  4096 Jul  9 23:25 requests
    drwxr-sr--  2 root staff  4096 Jul  9 23:25 requests-2.7.0.dist-info
    drwxr-sr-x  3 root staff  4096 Jul 10 20:33 setuptools
    drwxr-sr-x  2 root staff  4096 Jul 10 20:33 setuptools-18.0.1.dist-info
    drwxr-sr-x  3 root staff  4096 Jul  9 22:36 simplejson
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 simplejson-3.7.3.egg-info
    drwxr-sr--  2 root staff  4096 Jul  9 23:26 six-1.9.0.dist-info
    -rw-r--r--  1 root root  29664 Jul  9 23:26 six.py
    -rw-r--r--  1 root root  29006 Jul  9 23:26 six.pyc
    drwxr-sr-x  4 root staff  4096 Jul  9 22:36 stevedore
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 stevedore-1.6.0.dist-info
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 swiftclient
    drwxr-sr-x  7 root staff  4096 Jul  9 22:35 wal_e
    drwxr-sr-x  2 root staff  4096 Jul  9 22:35 wal_e-0.8.1.egg-info
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 wrapt
    drwxr-sr-x  2 root staff  4096 Jul  9 22:36 wrapt-1.10.5.egg-info
    

I have done quite a bit of searching, but haven't found anything that helps.
Any suggestions or points in the right direction are appreciated.

Additionally, it would be of great interest to solve this to ensure that my
primary installation will behave in the event that a restoration is necessary.
Backing up is pointless if I can't restore it.


Answer: I am not entirely sure what resolved the `The 'wal-e==0.8.1' distribution was
not found` errro, but on rebooting I never saw that again.

Aside from that, fixing this was rather straightforward.

The executable bit on a number of python distributions was not set.

Chowning these fixed the error:

    
    
          chmod o+x /usr/local/lib/python2.7/dist-packages/requests-2.7.0.dist-info/
          chmod o+x /usr/local/lib/python2.7/dist-packages/requests
    ...
    

