Unable to launch python scripts on the YARN Hadoop cluster via remote
Question: Since a few weeks i tried to submit python scripts via remote access or
connecting to the pyspark shell of the YARN cluster.

I am new to the HADOOP world. What I want is submitting spark scripts in my
local shell on the external HADOOP cluster.

My situation: External hadoop YARN cluster. Have access to the important
ports. I have Windows 7 64 Bit / Python 2.7.9 64 Bit / Spark 1.4.1. The HADOOP
cluster is running without any problems.

My problem: Submitting python scripts via remote access on the HADOOP cluster
doesnt work.

If i try spark-submit --master yarn-cluster --num-executors 2 --driver-memory
512m --executor-memory 512m --executor-cores 4 ... example.py

It says

    
    
    Error: Cluster deploy mode is not applicable to Spark shells.
    Exception: Java gateway process exited before sending the driver its port number
    

So as far as I understand the problem I think the question is

> > How do I set the yarn-config correctly to connect with my local client
> (not part of the cluster) to the external YARN cluster.


Answer: **SPARK VERSION 1.6.0 (which is the current version writing this).**

Python code cannot be executed in YARN-cluster mode. Python can only executed
in a cluster mode on a native spark cluster.

You can switch to use a spark cluster or you re-implement your code in Java or
Scala.

