Python hadoop mapreduce job using mrjob subprocess.CalledProcessError
Question: I am trying to run basic example from the mrjob's website on my custom data. I
have run Hadoop map reduce successfully using streaming, I have also
successfully tried the script without Hadoop, but now I am trying to run it on
Hadoop via mrjob by following command.

    
    
    ./mapred.py -r hadoop --hadoop-bin /usr/bin/hadoop -o hdfs:///user/cloudera/wc_result_mrjob hdfs:///user/cloudera/books
    

Source code of mapred.py is following:

    
    
    #! /usr/bin/env python
    
    from mrjob.job import MRJob
    
    
    class MRWordFrequencyCount(MRJob):
    
        def mapper(self, _, line):
            yield "chars", len(line)
            yield "words", len(line.split())
            yield "lines", 1
    
        def reducer(self, key, values):
            yield key, sum(values)
    
    if __name__ == '__main__':
        MRWordFrequencyCount.run()
    

Unfortunately I am getting following error:

    
    
    no configs found; falling back on auto-configuration
    no configs found; falling back on auto-configuration
    creating tmp directory /tmp/mapred.cloudera.20140824.195414.420162
    writing wrapper script to /tmp/mapred.cloudera.20140824.195414.420162/setup-wrapper.sh
    STDERR: mkdir: `hdfs:///user/cloudera/tmp/mrjob/mapred.cloudera.20140824.195414.420162/files/': No such file or directory
    Traceback (most recent call last):
    File "./mapred.py", line 18, in <module>
    MRWordFrequencyCount.run()
    File "/usr/lib/python2.6/site-packages/mrjob/job.py", line 494, in run
    mr_job.execute()
    File "/usr/lib/python2.6/site-packages/mrjob/job.py", line 512, in execute
    super(MRJob, self).execute()
    File "/usr/lib/python2.6/site-packages/mrjob/launch.py", line 147, in execute
    self.run_job()
    File "/usr/lib/python2.6/site-packages/mrjob/launch.py", line 208, in run_job
    runner.run()
    File "/usr/lib/python2.6/site-packages/mrjob/runner.py", line 458, in run
    self._run()
    File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 238, in _run
    self._upload_local_files_to_hdfs()
    File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 265, in _upload_local_files_to_hdfs
    self._mkdir_on_hdfs(self._upload_mgr.prefix)
    File "/usr/lib/python2.6/site-packages/mrjob/hadoop.py", line 273, in _mkdir_on_hdfs
    self.invoke_hadoop(['fs', '-mkdir', path])
    File "/usr/lib/python2.6/site-packages/mrjob/fs/hadoop.py", line 109, in invoke_hadoop
    raise CalledProcessError(proc.returncode, args)
    subprocess.CalledProcessError: Command '['/usr/bin/hadoop', 'fs', '-mkdir', 'hdfs:///user/cloudera/tmp/mrjob/mapred.cloudera.20140824.195414.420162/files/']' returned non-zero exit status 1
    

It seems to me that mrjob is not able to create some directory in HDFS, but
unfortunately I don't know how to fix this problem.

My Hadoop is the cloudera CDH5.1. quickstart.

Thank you in advance for any suggestions and advices.

**EDIT:**

I have tried to run the same code on cloudera CDH4.7. quickstart and it worked
well. So my modified question is: Is cloudera CDH5.1. supported by mrjob
framework? If so, then how to run it?


Answer: I encountered the same error, the workaround I did was to change the :

`self.invoke_hadoop(['fs', '-mkdir', path])`

to

`self.invoke_hadoop(['fs', '-mkdir','-p', path])`

The file modified was : /usr/lib/python2.6/site-packages/mrjob/hadoop.py

My MRJOB have been running for couple of months now without any troubles, so
this looks fine to me.

I am myself keen to know alternative for the same.

