title: Hadoop streaming with python on Windows
body: <p>I'm using Hortonworks HDP for Windows and have it successfully configured with a master and 2 slaves.</p>
  
  <p>I'm using the following command;</p>
  
  <p><code>bin\hadoop jar contrib\streaming\hadoop-streaming-1.1.0-SNAPSHOT.jar -files file:///d:/dev/python/mapper.py,file:///d:/dev/python/reducer.py -mapper "python mapper.py" -reducer "python reduce.py" -input /flume/0424/userlog.MDAC-HD1.MDAC.local..20130424.1366789040945 -output /flume/o%1 -cmdenv PYTHONPATH=c:\python27</code></p>
  
  <p>The mapper runs through fine, but the log reports that the reduce.py file wasn't found. In the exception it looks like the hadoop taskrunner is creating the symlink for the reducer to the mapper.py file.</p>
  
  <p>When I check the job configuration file, I noticed that <strong>mapred.cache.files</strong> is set to;</p>
  
  <p>hdfs://MDAC-HD1:8020/mapred/staging/administrator/.staging/job_201304251054_0021/files/mapper.py#mapper.py</p>
  
  <p>It looks like although the reduce.py file is being added to the jar file, it's not being included in the configuration correctly and can't be found when the reducer tries to run.</p>
  
  <p>I think my command is correct, I've tried using -file parameters instead but then neither file is found.</p>
  
  <p>Can anyone see or know of an obvious reason?</p>
  
  <p>Please note, this is on Windows.</p>
  
  <p>EDIT- I've just run it locally and it worked, looks like my problem may be with the copying of the files round the cluster.</p>
  
  <p>Still welcome input!</p>