Run scrapy from Flask application
Question: I have a crawler which I want to run everytime a person goes to the link.
Since all the other modules are in Flask, I was told to build this in Flask
also. I have installed scrapy and selenium both in the virtual environment and
globally on the machine with root.

When I run the crawler through the terminal, everything works fine. When I
start the Flask application and visit `xx.xx.xx.xx:8080/whats` in the browser,
this also works fine and runs my crawler and gets me the file. But as soon as
I go live so that anytime a person goes to the link, it gives me internal
error in browser.

In order to run crawler, we have to type "scrapy crawl
whateverthespidernameis" in the terminal. I did this using Python's `os`
module.

Here is my flask code:

    
    
    import sys
    from flask import request, jsonify, render_template, url_for, redirect,   session, abort,render_template_string,send_file,send_from_directory
    from flask import *
    #from application1 import *
    from main import *
    from test123 import *
    import os
    app = Flask(__name__)
    
    filename = ''
    app = Flask(__name__)
    
    @app.route('/whats')
    def whats():
        os.getcwd()
        os.chdir("/var/www/myapp/whats")
        //cmd = "scrapy crawl whats"
        cmd = "sudo scrapy crawl whats"
        os.system(cmd)
        return send_file("/var/www/myapp/staticcsv/whats.csv", as_attachment =True)
    
    if __name__ == "__main__":
        app.run(host='0.0.0.0', port=8080,debug=True)
    

This is the error recorded in the log file when I run through live link:

    
    
    sh: 1: scrapy: not found**
    

This is the error recorded in the log file when I use sudo in the command
(variable `cmd`):

    
    
    sudo: no tty present and no askpass program specified**
    

I am using uwsgi and nginx.

How can I run this crawler so that when anyone goes to "xx.xx.xx.xx/whats" the
crawler runs and returns the csv file?


Answer: When you use `sudo` the shell this starts will ask for a password on the tty -
it specifically doesn't read standard input for this information. Since
`flask` and other web applications typically run detached from a terminal,
`sudo` has no way to ask for a password, so it looks for a program that can
provide the password. You can find more information on this topic in [this
answer](http://stackoverflow.com/questions/21659637/how-to-fix-sudo-no-tty-
present-and-no-askpass-program-specified-error).

The reason you aren't finding `scrapy` is most likely because of differences
in your `$PATH` between the interactive shells you used in testing and the
process that's running `flask`. The easiest way to get around this is to give
the full path to the `scrapy` program in your command.

