How to configure Apache 2 to run a Python script from a Ruby CGI app with %x{}
Question: My Ruby CGI app is very simple, I would like to call a Python script, for
example `youtube-dl`, or `youtube-upload` with `%x{youtube-dl --help}`

For the sake of simplicity, I would like to print only the help page of the
`youtube-dl` Python script. So my Ruby script is also very simple:

    
    
    #!/usr/bin/ruby
    require "cgi"
    
    cgi=CGI.new(:accept_charset => "UTF-8")
    
    url=cgi['url']
    
    puts "Content-Type: text/plain; charset=\"UTF-8\""
    puts "Access-Control-Allow-Origin: *"
    puts
    
    puts url
    puts "-----------------------------------"
    puts %x{youtube-dl --help 2>&1}
    

Then I can invoke this CGI app called `ytdl` with <http://example.com/cgi-
bin/ytdl?url=a_youtube_url>. Unfortunately I get only a lot of error messages
from the Python interpreter which is unable to import some packages,
especially site-packages:

    
    
    Traceback (most recent call last):
      File "/usr/bin/youtube-dl", line 5, in <module>
        from pkg_resources import load_entry_point
      File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 36, in <module>
        import plistlib
      File "/usr/lib/python3.4/plistlib.py", line 65, in <module>
        from xml.parsers.expat import ParserCreate
      File "/usr/lib/python3.4/xml/parsers/expat.py", line 4, in <module>
        from pyexpat import *
    ImportError: /usr/lib/python3.4/lib-dynload/pyexpat.cpython-34m.so: undefined symbol: XML_SetHashSalt
    

I think I would configure my Apache 2 web server to provide some environment
variables for the Python interpereter, where it can find the packages.
Unfortunately I have no experience with Python, I only learned Ruby till now.
My Apache server runs CGI apps as user `daemon`, and when I login as `daemon`
I can run the `youtube-dl` script without error messages.

Apache provide these environment for my CGI apps (printed with `puts
%x{env}`):

    
    
    UNIQUE_ID=VZN9RyX3N7MAAAs95zsAAAAI
    RUBYOPT=rubygems
    GEM_HOME=/home/XXXXXX/.gem/ruby/2.2.0
    HTTP_USER_AGENT=Opera/9.80 (X11; Linux i686) Presto/2.12.388 Version/12.16
    HTTP_HOST=xx.xxx.xx.xxx
    HTTP_ACCEPT=text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
    HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.9
    HTTP_ACCEPT_ENCODING=gzip, deflate
    HTTP_CONNECTION=Keep-Alive
    PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/XXXXXX/.gem/ruby/2.2.0/bin:/home/XXXXXX/util
    LD_LIBRARY_PATH=/opt/lampp/lib:/opt/lampp/lib
    SERVER_SIGNATURE=
    SERVER_SOFTWARE=Apache/2.4.9 (Unix) OpenSSL/1.0.1g PHP/5.5.11 mod_perl/2.0.8-dev Perl/v5.16.3
    SERVER_NAME=xx.xxx.xx.xxx
    SERVER_ADDR=xx.xxx.xx.xxx
    SERVER_PORT=80
    REMOTE_ADDR=xx.x.xxx.xxx
    DOCUMENT_ROOT=/opt/lampp/htdocs
    REQUEST_SCHEME=http
    CONTEXT_PREFIX=/cgi-bin/
    CONTEXT_DOCUMENT_ROOT=/opt/lampp/cgi-bin/
    SERVER_ADMIN=you@example.com
    SCRIPT_FILENAME=/opt/lampp/cgi-bin/ytdl
    REMOTE_PORT=37798
    GATEWAY_INTERFACE=CGI/1.1
    SERVER_PROTOCOL=HTTP/1.1
    REQUEST_METHOD=GET
    QUERY_STRING=
    REQUEST_URI=/cgi-bin/ytdl
    SCRIPT_NAME=/cgi-bin/ytdl
    

Meanwhile I realized that when I set the environment variable (either in
Apache config httpd.conf with SetEnv directive, or with
`ENV['PYTHONHOME']=value` in my Ruby script) called PYTHONHOME to an empty
string "" or "/usr/lib" , the error message disappers, and I get another one:

    
    
    Fatal Python error: Py_Initialize: Unable to get the locale encoding
    ImportError: No module named 'encodings'
    

When I set:

    
    
    ENV['PYTHONHOME']="/usr/lib"
    ENV['PYTHONPATH']="/usr/lib/python3.4"
    

I get another error message:

    
    
    File "/usr/bin/youtube-dl", line 5, in <module>
        from pkg_resources import load_entry_point
    ImportError: No module named 'pkg_resources'
    

So my question is, how should I configure my Apache 2 webserver to make Python
scripts run and make them callable from other apps?


Answer: When you run an application in a subshell you have to be aware of the
environment being set up for that shell and application. The PATH and
variables will be nonexistent or very limited which can affect its ability to
find libraries.

Write a small CGI that outputs the Apache variables and environment passed to
the script, either as a web page or to the log or a file and examine the
results.

Also permissions and the user will be whatever the server sets up which is
usually very restricted.

Also, seriously consider _not_ using CGI. Instead use something more modern
and flexible. Sinatra works nicely with Apache using something like Passenger
to glue them together. Sinatra makes it really easy to do web-services, and
actually works so well you can often do without a heavyweight server like
Apache, especially during the development, test and early production stages. I
have an API used in-house that handles a lot of enterprise-critical requests
and it just cruises along.

