title: Tracing down missing import from py4j on error "trying to call a package"
body: <p>Given the following python snippet:</p>
  
  <pre><code>from pyspark.mydb.mydb import *
  class HBaseTest:
  
    def __init__(self):
      from pyspark.context import SparkContext
      sc = SparkContext('local[4]', 'PythonTest')
      self._hbctx = MyDbSQLContext(sc)
  </code></pre>
  
  <p>The line of interest is the last one:  the MyDbSQLContext is a class in the org.apache.spark.mydb package. There is a python wrapper for it in the pyspark.mydb.mydb.py file</p>
  
  <p>In an attempt to make the imports known to the py4j java gateway, the following statements are executed:</p>
  
  <pre><code>java_import(self._sc._gateway.jvm,'org.apache.spark.sql.mydb.*')
  java_import(self._sc._gateway.jvm,'org.apache.hadoop.conf.Configuration')
  </code></pre>
  
  <p>However on the last line of the original snippet:</p>
  
  <pre><code>    self._hbctx = MyDbSQLContext(sc)
  </code></pre>
  
  <p>We get the following error:</p>
  
  <pre><code>sc=org.apache.spark.SparkContext@4925069a conf=&lt;py4j.java_gateway.JavaMember object at 0x10bdd5090&gt;
  Traceback (most recent call last):
    File "pyspark/hbase/hbase_tests.py", line 81, in &lt;module&gt;
      test._test()
    File "pyspark/hbase/hbase_tests.py", line 76, in _test
      self.create_test_tables()
    File "pyspark/hbase/hbase_tests.py", line 46, in create_test_tables
      self._ctx().sql(create_sql).toRdd().collect()
    File "/shared/pyjava/python/pyspark/sql.py", line 1620, in sql
      return SchemaRDD(self._ssql_ctx.sql(sqlQuery).toJavaSchemaRDD(), self)
    File "/shared/pyjava/python/pyspark/hbase/hbase.py", line 58, in _ssql_ctx
      self._scala_HBaseSQLContext = self._get_hbase_ctx()
    File "/shared/pyjava/python/pyspark/hbase/hbase.py", line 83, in _get_hbase_ctx
      return self._jvm.HBaseSQLContext(self._jsc.sc())
    File "/shared/pyjava/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 726, in __getattr__
  py4j.protocol.Py4JError: Trying to call a package.
  </code></pre>
  
  <p><strong>UPDATE</strong>  I have spent more time analyzing the way that imports are performed for hive. They are added via the spark-submit script that internally invokes <strong>compute-classpath.sh</strong>.  I had already correctly added the $MYDB_HOME/lib/* to the classpath in that script. The mydb jar files are available on the classpath:</p>
  
  <p>In addition, the java_imports are being printed - and here they do show the mydb imports:</p>
  
  <pre><code>java_imports: [u'org.apache.hadoop.conf.Configuration', u'org.apache.spark.SparkConf',
  u'org.apache.spark.sql.hive.LocalHiveContext', u'org.apache.spark.sql.hive.TestHiveContext', 
  u'org.apache.hadoop.mydb.client.Client', u'scala.Tuple2', u'org.apache.spark.sql.mydb.mydbSQLContext', 
  u'org.apache.spark.sql.hive.HiveContext', u'java.lang', u'org.apache.hadoop.mydb', u'org.apache.hadoop.mydb.client', 
  u'org.apache.hadoop.mydb.filter', u'org.apache.hadoop.mydb.util', u'org.apache.log4j', u'org.apache.spark.api.java', 
  u'org.apache.spark.api.python', u'org.apache.spark.mllib.api.python', u'org.apache.spark.sql', u'org.apache.spark.sql.mydb']
  </code></pre>
  
  <p>It is presently unclear what the source of this issue is. We have verified:</p>
  
  <ul>
  <li>mydb classes are in the path</li>
  <li>mydb classes are being imported by java_import</li>
  <li><p>AFAICT the patterns followed for importing and using hive classes are being similarly followed for mydb</p>
  
  <p>I am at a loss what else to examine to track this java class/classpath issue down.</p></li>
  </ul>