unable to run scrapy in ec2
Question: I'm trying to run a code on an ec2 server. It is a python scrapy project which
is executed fine on my own pc. when trying to run it in the ec2 i get this:

    
    
    Traceback (most recent call last):
    File "/usr/local/bin/scrapy", line 4, in <module>
    execute()
    File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 122, in execute
    cmds = _get_commands_dict(settings, inproject)
    File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
    cmds = _get_commands_from_module('scrapy.commands', inproject)
    File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
    for cmd in _iter_command_classes(module):
    File "/usr/local/lib/python2.7/site-packages/scrapy/cmdline.py", line 20, in _iter_command_classes
    for module in walk_modules(module_name):
    File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 68, in walk_modules
    submod = import_module(fullpath)
    File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
    File "/usr/local/lib/python2.7/site-packages/scrapy/commands/bench.py", line 3, in <module>
    from scrapy.tests.mockserver import MockServer
    File "/usr/local/lib/python2.7/site-packages/scrapy/tests/mockserver.py", line 6, in <module>
    from twisted.internet import reactor, defer, ssl
    File "/usr/local/lib/python2.7/site-packages/twisted/internet/ssl.py", line 59, in <module>
    from OpenSSL import SSL
    File "/usr/local/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
    File "/usr/local/lib/python2.7/site-packages/OpenSSL/rand.py", line 11, in <module>
    from OpenSSL._util import (
    File "/usr/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 4, in <module>
    binding = Binding()
    File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 87, in __init__
    self._ensure_ffi_initialized()
    File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 106, in _ensure_ffi_initialized
    libraries=libraries,
    File "/usr/local/lib/python2.7/site-packages/cryptography/hazmat/bindings/utils.py", line 39, in build_ffi
    ffi = cffi.FFI()
    File "/usr/local/lib/python2.7/site-packages/cffi/api.py", line 56, in __init__
    **import _cffi_backend as backend
    ImportError: libffi.so.5: cannot open shared object file: No such file or directory**
    

I've tried to update && reinstall all the mentioned libraries.

it always says it is most updated version.

this is my linux distro and default python is 2.7:

    
    
    #  cat /etc/*-release
    NAME="Amazon Linux AMI"
    VERSION="2015.03"
    ID="amzn"
    ID_LIKE="rhel fedora"
    VERSION_ID="2015.03"
    PRETTY_NAME="Amazon Linux AMI 2015.03"
    ANSI_COLOR="0;33"
    CPE_NAME="cpe:/o:amazon:linux:2015.03:ga"
    HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
    Amazon Linux AMI release 2015.03
    

UPDATE: it looks like some sort of problem in scrapy dependencies. trying

    
    
     pip install scrapy 
    

gave me the following error:

    
    
    creating /usr/lib/python2.6/dist-packages/cssselect
    error: could not create '/usr/lib/python2.6/dist-packages/cssselect':       Permission denied
    
    ----------------------------------------
    Command "/usr/bin/python2.6 -c "import setuptools,   
    tokenize;__file__='/tmp/pip-build- aoghBl/cssselect/setup.py'
    ;exec(compile(getattr(tokenize, 'open', open)
    (__file__).read().replace('\r\n', '\n'), __file__, 'exec'))"
    install --record /tmp/pip-1Qpx2a-record/install-record.txt
    --single-version-externally-managed --compile" 
    failed with error code 1 in /tmp/pip-build-aoghBl/cssselect
    


Answer: Your updated question is a permissions error. you likely need to run

    
    
    sudo pip install cssselect
    

May need to do that for other dependancies. It's strange though that it's
trying to install under python2.6...

Edit: Given that it's trying to run under 2.6, something is wrong with your
instance. I'd suggest terminating and recreating the instance. The following
commands are what I ran to get scrapy dependancies and my project up and
running

    
    
    sudo su
    yum groupinstall 'Development Tools'
    yum install -y libffi-devel libxslt-devel libxml2-devel openssl-devel python-devel mysql-devel
    

