What is the difference between spark-submit and pyspark?
Question: If I start up pyspark and then run this command:

    
    
    import my_script; spark = my_script.Sparker(sc); spark.collapse('./data/')
    

Everything is A-ok. If, however, I try to do the same thing through the
commandline and spark-submit, I get an error:

    
    
    Command: /usr/local/spark/bin/spark-submit my_script.py collapse ./data/
      File "/usr/local/spark/python/pyspark/rdd.py", line 352, in func
        return f(iterator)
      File "/usr/local/spark/python/pyspark/rdd.py", line 1576, in combineLocally
        merger.mergeValues(iterator)
      File "/usr/local/spark/python/pyspark/shuffle.py", line 245, in mergeValues
        for k, v in iterator:
      File "/.../my_script.py", line 173, in _json_args_to_arr
        js = cls._json(line)
    RuntimeError: uninitialized staticmethod object
    

my_script:

    
    
    ...
    if __name__ == "__main__":
        args = sys.argv[1:]
        if args[0] == 'collapse':
            directory = args[1]
            from pyspark import SparkContext
            sc = SparkContext(appName="Collapse")
            spark = Sparker(sc)
            spark.collapse(directory)
            sc.stop()
    

Why is this happening? What's the difference between running pyspark and
running spark-submit that would cause this divergence? And how can I make this
work in spark-submit?

EDIT: I tried running this from the bash shell by doing `pyspark my_script.py
collapse ./data/` and I got the same error. The only time when everything
works is when I am in a python shell and import the script.


Answer: pyspark-submit send your code to workers in a cluster to execute.

check: <http://spark.apache.org/docs/latest/submitting-applications.html>

