Processing XML With Hadoop Streaming failed
Question: i did

    
    
    bin/hadoop jar contrib/streaming/hadoop-streaming-1.0.4.jar -inputreader "StreamXmlRecordReader, begin=<metaData>,end=</metaData>" -input /user/root/xmlpytext/metaData.xml -mapper /Users/amrita/desktop/hadoop/pythonpractise/mapperxml.py -file /Users/amrita/desktop/hadoop/pythonpractise/mapperxml.py -reducer /Users/amrita/desktop/hadoop/pythonpractise/reducerxml.py  -file /Users/amrita/desktop/hadoop/pythonpractise/mapperxml.py -output /user/root/xmlpytext-output1 -numReduceTasks 1
    

but it shows

    
    
    13/03/22 09:38:48 INFO mapred.FileInputFormat: Total input paths to process : 1
    13/03/22 09:38:49 INFO streaming.StreamJob: getLocalDirs(): [/Users/amrita/desktop/hadoop/temp/mapred/local]
    13/03/22 09:38:49 INFO streaming.StreamJob: Running job: job_201303220919_0001
    13/03/22 09:38:49 INFO streaming.StreamJob: To kill this job, run:
    13/03/22 09:38:49 INFO streaming.StreamJob: /private/var/root/hadoop-1.0.4/libexec/../bin/hadoop job  -Dmapred.job.tracker=-kill job_201303220919_0001
    13/03/22 09:38:49 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp?jobid=job_201303220919_0001
    13/03/22 09:38:50 INFO streaming.StreamJob:  map 0%  reduce 0%
    13/03/22 09:39:26 INFO streaming.StreamJob:  map 100%  reduce 100%
    13/03/22 09:39:26 INFO streaming.StreamJob: To kill this job, run:
    13/03/22 09:39:26 INFO streaming.StreamJob: /private/var/root/hadoop-1.0.4/libexec/../bin/hadoop job  -Dmapred.job.tracker=-kill job_201303220919_0001
    13/03/22 09:39:26 INFO streaming.StreamJob: Tracking URL: http:///jobdetails.jsp?jobid=job_201303220919_0001
    13/03/22 09:39:26 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201303220919_0001_m_000000
    13/03/22 09:39:26 INFO streaming.StreamJob: killJob...
    Streaming Command Failed!
    

when i went through jobdetails.jsp there it shows

    
    
    java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at org.apache.hadoop.streaming.StreamInputFormat.getRecordReader(StreamInputFormat.java:77)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:197)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.streaming.StreamInputFormat.getRecordReader(StreamInputFormat.java:74)
        ... 8 more
    Caused by: java.io.IOException: JobConf: missing required property: stream.recordreader.begin
        at org.apache.hadoop.streaming.StreamXmlRecordReader.checkJobGet(StreamXmlRecordReader.java:278)
        at org.apache.hadoop.streaming.StreamXmlRecordReader.<init>(StreamXmlRecordReader.java:52)
        ... 13 more
    

my mapper

    
    
    #!/usr/bin/env python
    import sys
    import cStringIO
    import xml.etree.ElementTree as xml
    def cleanResult(element):
        result = None
        if element is not None:
            result = element.text
            result = result.strip()
        else:
            result = ""
        return result
    def process(val):
        root = xml.fromstring(val)
        sceneID = cleanResult(root.find('sceneID'))
        cc = cleanResult(root.find('cloudCover'))
        returnval = ("%s,%s") % (sceneID,cc)
        return returnval.strip()
    if __name__ == '__main__':
        buff = None
        intext = False
        for line in sys.stdin:
            line = line.strip()
            if line.find("<metaData>") != -1:
                intext = True
                buff = cStringIO.StringIO()
                 buff.write(line)
             elif line.find("</metaData>") != -1:
                  intext = False
                  buff.write(line)
                  val = buff.getvalue()
                  buff.close()
                  buff = None
                  print process(val)
            else:
                if intext:
                    buff.write(line)
    

and reducer

    
    
    #!/usr/bin/env python
    import sys
    if __name__ == '__main__':
         for line in sys.stdin:
             print line.strip()
    

can anyone tell me why this happens. I am using hadoop-1.0.4 im mac. Is there
any thing wrong. Should i change any thing . pls help me out.


Answer: Remove space in between comma and begin `, begin=<`.

The correct format is :

    
    
    hadoop jar hadoop-streaming.jar -inputreader
    "StreamXmlRecord,begin=BEGIN_STRING,end=END_STRING" ..... (rest of the command)
    

This is due to the code surrounding the following lines in
`org.apache.hadoop.streaming.StreamJob`

    
    
    for (int i = 1; i < args.length; i++) {
        String[] nv = args[i].split("=", 2);
        String k = "stream.recordreader." + nv[0];
        String v = (nv.length > 1) ? nv[1] : "";
        jobConf_.set(k, v);
      }
    

