title: PySpark: No module named 'psycopg2._psycopg'
body: <p>I'm using <code>Psycopg2</code> in a PySpark job and I'm providing it in a zip file that contains all dependencies. The build script does this:</p>
  
  <pre><code>python -m pip install -t build/dependencies -r src/requirements.txt
  </code></pre>
  
  <p>where <code>requirements.txt</code> is</p>
  
  <pre><code>SQLAlchemy
  psycopg2
  </code></pre>
  
  <p>It then zips <code>dependencies</code> and I provide it to <code>spark-submit</code> through <code>--py-files</code>. When I run the job on EMR, SQLAlchemy is imported just fine, but I get this error:</p>
  
  <pre><code>ImportError: No module named 'psycopg2._psycopg'
  </code></pre>
  
  <p>If I removed <code>psycopg2</code> from requirements and installed it on EMR manually (through <code>pip</code>), it'd work just fine. Python version on both my machine and EMR is 3.4.7, but my machine is a Mac (whereas EMR is linux).</p>