Execute Python scripts with Selenium via Crontab
Question: I have several python scripts that use selenium webdriver on a Debian server.
If I run them manually from the terminal (usually as root) everything is ok
but every time I tried to run them via crontab I have an exception like this:

    
    
    WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpQ4vStP If you specified a log_file in the FirefoxBinary constructor, check it for details.
    

Try for example this script:

    
    
    from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
    from pyvirtualdisplay import Display
    from selenium import webdriver
    import datetime
    import logging
    
    FIREFOX_PATH = '/usr/bin/firefox'
    
    if __name__ == '__main__':
        cur_date = datetime.datetime.now().strftime('%Y-%m-%d')
        logging.basicConfig(filename="./logs/download_{0}.log".format(cur_date),
                            filemode='w',
                            level=logging.DEBUG,
                            format='%(asctime)s - %(levelname)s - %(message)s')
        try:
            display = Display(visible=0, size=(800, 600))
            display.start()
            print 'start'
            logging.info('start')
            binary = FirefoxBinary(FIREFOX_PATH,
                                   log_file='/home/egor/dev/test/logs/firefox_binary_log.log')
            driver = webdriver.Firefox()
            driver.get("http://google.com")
            logging.info('title: ' + driver.title)
            driver.quit()
            display.stop()
        except:
            logging.exception('')
        logging.info('finish')
        print 'finish'
    

The crontab command for it:

    
    
    0 13 * * * cd "/home/egor/dev/test" && python test.py
    

The log file for this script looks like this:

    
    
    2016-09-27 16:30:01,742 - DEBUG - param: "['Xvfb', '-help']" 
    2016-09-27 16:30:01,743 - DEBUG - command: ['Xvfb', '-help']
    2016-09-27 16:30:01,743 - DEBUG - joined command: Xvfb -help
    2016-09-27 16:30:01,745 - DEBUG - process was started (pid=23042)
    2016-09-27 16:30:01,747 - DEBUG - process has ended
    2016-09-27 16:30:01,748 - DEBUG - return code=0
    2016-09-27 16:30:01,748 - DEBUG - stdout=
    2016-09-27 16:30:01,751 - DEBUG - param: "['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1724']" 
    2016-09-27 16:30:01,751 - DEBUG - command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1724']
    2016-09-27 16:30:01,751 - DEBUG - joined command: Xvfb -br -nolisten tcp -screen 0 800x600x24 :1724
    2016-09-27 16:30:01,753 - DEBUG - param: "['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1725']" 
    2016-09-27 16:30:01,753 - DEBUG - command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1725']
    2016-09-27 16:30:01,753 - DEBUG - joined command: Xvfb -br -nolisten tcp -screen 0 800x600x24 :1725
    2016-09-27 16:30:01,755 - DEBUG - process was started (pid=23043)
    2016-09-27 16:30:01,755 - DEBUG - DISPLAY=:1725
    2016-09-27 16:30:01,855 - INFO - start
    2016-09-27 16:30:31,965 - ERROR - 
    Traceback (most recent call last):
      File "test.py", line 31, in <module>
        driver = webdriver.Firefox()
      File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 103, in __init__
        self.binary, timeout)
      File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__
        self.binary.launch_browser(self.profile, timeout=timeout)
      File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
        self._wait_until_connectable(timeout=timeout)
      File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 106, in _wait_until_connectable
        % (self.profile.path))
    WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpQ4vStP If you specified a log_file in the FirefoxBinary constructor, check it for details.
    
    2016-09-27 16:30:31,966 - INFO - finish
    

What I have tried:

  1. Ensure that the script file is owned by root
  2. Use export DISPLAY=:0; or export DISPLAY=:99; in crontab command
  3. set the HOME variable in the crontab to the path of the home directory of the user that the cronjob was being run as

I'm really stuck with this problem.

I have python 2.7.10, selenium 2.53.6 with Xvbf and Firefox 47.0.1 on Debian
7.7


Answer: Try a hardcoded Firefox binary
<https://seleniumhq.github.io/selenium/docs/api/py/webdriver_firefox/selenium.webdriver.firefox.firefox_binary.html>

    
    
    selenium.webdriver.firefox.firefox_binary.FirefoxBinary("/your/binary/location/firefox")
    driver = webdriver.Firefox(firefox_binary=binary)
    

