title: pyspark: call a custom java function from pyspark. Do I need Java_Gateway?
body: <p>I wrote the following MyPythonGateway.java so that I can call my custom java class from Python:</p>
  
  <pre><code>public class MyPythonGateway {
  
      public String findMyNum(String input) {
          return MyUtiltity.parse(input).getMyNum(); 
      }
  
      public static void main(String[] args) {
          GatewayServer server = new GatewayServer(new MyPythonGateway());
          server.start();
      }
  }
  </code></pre>
  
  <p>and here is how I used it in my Python code:</p>
  
  <pre><code>def main():
  
      gateway = JavaGateway()                   # connect to the JVM
      myObj = gateway.entry_point.findMyNum("1234 GOOD DAY")
      print(myObj)
  
  
  if __name__ == '__main__':
      main()
  </code></pre>
  
  <hr>
  
  <p>Now I want to use <code>MyPythonGateway.findMyNum()</code> function from PySpark, not just a standalone python script. I did the following:</p>
  
  <pre><code>myNum = sparkcontext._jvm.myPackage.MyPythonGateway.findMyNum("1234 GOOD DAY")
  print(myNum)
  </code></pre>
  
  <p>However, I got the following error:</p>
  
  <pre><code>... line 43, in main:
  myNum = sparkcontext._jvm.myPackage.MyPythonGateway.findMyNum("1234 GOOD DAY")
    File "/home/edamameQ/spark-1.5.2/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 726, in __getattr__
  py4j.protocol.Py4JError: Trying to call a package.
  </code></pre>
  
  <p>So what did I miss here? I don't know if I should run a separate JavaApplication of MyPythonGateway to start a gateway server when using pyspark. Please advice. Thanks!</p>
  
  <hr>
  
  <p>Below is exactly what I need:</p>
  
  <pre><code>input.map(f)
  
  def f(row):
     // call MyUtility.java 
     // x = MyUtility.parse(row).getMyNum()
     // return x
  </code></pre>
  
  <p>What would be the best way to approach this? Thanks!</p>