title: Scrapy and MySQLdb
body: <p>I'm using <code>Python 2.7</code> on <code>Mac OS X Lion 10.7.5</code>.</p>
  
  <p>I originally had an issue with installing <code>MySQLdb</code> using both <code>pip-2.7 install MySQL-python</code> as well as downloading and then running <code>python2.7 setup.py build</code> and <code>python2.7 setup.py install</code>.  I tried these different methods using both 32 and 64 bit installs of <code>MySQL</code> and corresponding architectures, but to no avail.</p>
  
  <p>My solution was to install <code>Macports</code>.  I then installed <code>MySQL</code> and <code>MySQL-python</code> (<code>MySQLdb</code>) using Macports.</p>
  
  <p>I use <code>Wing IDE</code> for developing code, so I switched over to the Macports version of <code>Python</code> - importing <code>MySQLdb</code> works.  I also switched the default terminal version of Python to this Macports version and verified that it was the default by calling <code>python</code> from the command line- the correct version started.</p>
  
  <p>So now the problem:  I'm using scrapy to scrape a movie web page for info.  My pipeline directs the scraped data to a database, which uses the previously mentioned <code>MySQLdb</code> module.  When I go to the command line, <code>cd</code> into my project and run <code>scrapy crawl MySpider</code>, I get the following error:</p>
  
  <pre><code> raise ImportError, "Error loading object '%s': %s" % (path, e)
   ImportError: Error loading object 'BoxOfficeMojo.pipelines.BoxofficemojoPipeline': No module named MySQLdb.cursors
  </code></pre>
  
  <p>I've checked and made sure that I can import MySQLdb.cursors from the python2.7 shell, so I think there is an issue with which version of Python scrapy is using...</p>
  
  <p>:::::UPDATE:::::</p>
  
  <p>Here is the full traceback:</p>
  
  <pre><code> Traceback (most recent call last):
     File "/usr/local/bin/scrapy", line 4, in &lt;module&gt;
  execute()
    File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 131, in execute
  _run_print_help(parser, _run_command, cmd, args, opts)
    File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 76, in _run_print_help
  func(*a, **kw)
    File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 138, in _run_command
  cmd.run(args, opts)
    File "/Library/Python/2.7/site-packages/scrapy/commands/crawl.py", line 43, in run
  spider = self.crawler.spiders.create(spname, **opts.spargs)
    File "/Library/Python/2.7/site-packages/scrapy/command.py", line 33, in crawler
  self._crawler.configure()
    File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 41, in configure
  self.engine = ExecutionEngine(self, self._spider_closed)
    File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 63, in __init__
  self.scraper = Scraper(crawler)
    File "/Library/Python/2.7/site-packages/scrapy/core/scraper.py", line 66, in __init__
  self.itemproc = itemproc_cls.from_crawler(crawler)
    File "/Library/Python/2.7/site-packages/scrapy/middleware.py", line 50, in from_crawler
  return cls.from_settings(crawler.settings, crawler)
    File "/Library/Python/2.7/site-packages/scrapy/middleware.py", line 29, in from_settings
  mwcls = load_object(clspath)
    File "/Library/Python/2.7/site-packages/scrapy/utils/misc.py", line 39, in load_object
      raise ImportError, "Error loading object '%s': %s" % (path, e)
      ImportError: Error loading object 'BoxOfficeMojo.pipelines.BoxofficemojoPipeline': No module named MySQLdb.cursors
  </code></pre>
  
  <p>:::::UPDATE 2:::::</p>
  
  <p>Here is my current path:</p>
  
  <pre><code> $PATH
    -bash:       /opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/local/sbin:~/bin:/Library/Frameworks/Python  .framework/Versions/3.3/bin:/Library/Frameworks/Python.framework/Versions/3.3/bin:/Library/Frameworks/Python.framework/Versions/3.3/bin:/Library/Frameworks/Python.framework/Versions/3.3/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin: No such file or directory
  </code></pre>
  
  <p>::ALSO::</p>
  
  <p>I added this to code in hopes of fixing things - it is the location of <code>py27-mysql</code> (<code>MySQLdb</code>), but the same error is returned:</p>
  
  <pre><code> import sys; sys.path.append("/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages")
  </code></pre>
  
  <p>::ALSO #2::</p>
  
  <p>Here is the code for my pipeline- I have no idea whether or not it works, as I keep getting the error regarding the <code>import</code>, but thought it might be helpful:</p>
  
  <pre><code> from scrapy import log
   from twisted.enterprise import adbapi
   import time
   import MySQLdb.cursors
   import sys; sys.path.append("/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages")
   class BoxofficemojoPipeline(object):
  
       def __init__(self):
           print ('init')
           self.dbpool = adbapi.ConnectionPool('MySQLdb', db = 'testdb', user='testuser', passwd='test', cursorclass=MySQLdb.cursors.DictCursor, charset='utf8', use_unicode=True)
  
  
  
        def process_item(self, item, spider):
           print('process')
           query = self.dbpool.runInteraction(self._conditional_insert, item)  #("""INSERT INTO Example_Movie (title, url, gross, release) VALUES (%s, %s, %s, %s)""", (item['title'].endcode('utf-8'), item['url'].encode('utf-8'), item['gross'].encode('utf-8'), item['release'].encode('utf-8')))
            query.addErrback(self.handle_error)#self.conn.commit()
  
           return item
  
        def _conditional_insert(self, tx, item):
           print ('conditional insert')
            #Create record if doesn't exist
            #all this block run on it's own thread
  
           tx.execute("select * from example_movie where url = %s", (item['url'], ))
           result = tx.fetchone()
           if result:
                log.msg("Item already stored in db: %s" % item, level = log.DEBUG)
           else:
                tx.execute("insert into example_movie (title, url, gross, release) values (%s, %s, %s, %s)", (item['title'].encode('utf-8'), item['url'].encode('utf-8'), item['gross'].encode('utf-8'), item['release'].encode('utf-8')))
                log.msg("Item stored in db: %s" %  item, level=log.DEBUG)
  
        def handle_error(self, e):
           print ('handle_error')
           log.err(e)
  </code></pre>