Pig UDF running on AWS EMR with java.lang.NoClassDefFoundError: org/apache/pig/LoadFunc
Question: I am developing an application that try to read log file stored in S3 bucks
and parse it using Elastic MapReduce. Current the log file has following
format

    
    
    ------------------------------- 
    COLOR=Black 
    Date=1349719200 
    PID=23898 
    Program=Java 
    EOE 
    ------------------------------- 
    COLOR=White 
    Date=1349719234 
    PID=23828 
    Program=Python 
    EOE 
    

So I try to load the file into my Pig script, but the build-in Pig Loader
doesn't seems be able to load my data, so I have to create my own UDF. Since I
am pretty new to Pig and Hadoop, I want to try script that written by others
before I write my own, just to get a teast of how UDF works. I found one from
here <http://pig.apache.org/docs/r0.10.0/udf.html>, there is a
SimpleTextLoader. In order to compile this SimpleTextLoader, I have to add a
few imports, as

    
    
    import java.io.IOException; 
    import java.util.ArrayList;
    import org.apache.hadoop.io.Text; 
    import org.apache.hadoop.mapreduce.Job; 
    import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; 
    import org.apache.hadoop.mapreduce.InputFormat; 
    import org.apache.hadoop.mapreduce.RecordReader; 
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 
    import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSplit; 
    import org.apache.pig.backend.executionengine.ExecException; 
    import org.apache.pig.data.Tuple; 
    import org.apache.pig.data.TupleFactory;
    import org.apache.pig.data.DataByteArray; 
    import org.apache.pig.PigException; 
    import org.apache.pig.LoadFunc;
    

Then, I found out I need to compile this file. I have to download svn and pig
running

    
    
    sudo apt-get install subversion 
    svn co http://svn.apache.org/repos/asf/pig/trunk 
    ant
    

Now i have a pig.jar file, then I try to compile this file.

    
    
    javac -cp ./trunk/pig.jar SimpleTextLoader.java 
    jar -cf SimpleTextLoader.jar SimpleTextLoader.class 
    

It compiles successful, and i type in Pig entering grunt, in grunt i try to
load the file, using

    
    
    grunt> register file:/home/hadoop/myudfs.jar
    grunt> raw = LOAD 's3://mys3bucket/samplelogs/applog.log' USING myudfs.SimpleTextLoader('=') AS (key:chararray, value:chararray); 
    
    2012-12-05 00:08:26,737 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal error. org/apache/pig/LoadFunc Details at logfile: /home/hadoop/pig_1354666051892.log
    

Inside the pig_1354666051892.log, it has

    
    
    Pig Stack Trace
    ---------------
    ERROR 2998: Unhandled internal error. org/apache/pig/LoadFunc
    
    java.lang.NoClassDefFoundError: org/apache/pig/LoadFunc
    

I also try to use another UDF (UPPER.java) from
<http://wiki.apache.org/pig/UDFManual>, and I am still get the same error by
try to use UPPER method. Can you please help me out, what's the problem here?
Much thanks!

UPDATE: I did try EMR build-in Pig.jar at /home/hadoop/lib/pig/pig.jar, and
get the same problem.


Answer: Put the UDF jar in the /home/hadoop/lib/pig directory or copy the
pig-*-amzn.jar file to /home/hadoop/lib and it will work.

You would probably use a bootstrap action to do either of these.

