title: Run scrapy from Flask application
body: <p>I have a crawler which I want to run everytime a person goes to the link. Since all the other modules are in Flask, I was told to build this in Flask also. I have installed scrapy and selenium both in the virtual environment and globally on the machine with root.</p>
  
  <p>When I run the crawler through the terminal, everything works fine. When I start the Flask application and visit <code>xx.xx.xx.xx:8080/whats</code> in the browser, this also works fine and runs my crawler and gets me the file. But as soon as I go live so that anytime a person goes to the link, it gives me internal error in browser.</p>
  
  <p>In order to run crawler, we have to type "scrapy crawl whateverthespidernameis" in the terminal. I did this using Python's <code>os</code> module.</p>
  
  <p>Here is my flask code:</p>
  
  <pre><code>import sys
  from flask import request, jsonify, render_template, url_for, redirect,   session, abort,render_template_string,send_file,send_from_directory
  from flask import *
  #from application1 import *
  from main import *
  from test123 import *
  import os
  app = Flask(__name__)
  
  filename = ''
  app = Flask(__name__)
  
  @app.route('/whats')
  def whats():
      os.getcwd()
      os.chdir("/var/www/myapp/whats")
      //cmd = "scrapy crawl whats"
      cmd = "sudo scrapy crawl whats"
      os.system(cmd)
      return send_file("/var/www/myapp/staticcsv/whats.csv", as_attachment =True)
  
  if __name__ == "__main__":
      app.run(host='0.0.0.0', port=8080,debug=True)
  </code></pre>
  
  <p>This is the error recorded in the log file when I run through live link:</p>
  
  <pre><code>sh: 1: scrapy: not found**
  </code></pre>
  
  <p>This is the error recorded in the log file when I use sudo in the command (variable <code>cmd</code>):</p>
  
  <pre><code>sudo: no tty present and no askpass program specified**
  </code></pre>
  
  <p>I am using uwsgi and nginx.</p>
  
  <p>How can I run this crawler so that when anyone goes to "xx.xx.xx.xx/whats" the crawler runs and returns the csv file?</p>