69713


Hadoopをインストールし、hadoop-mapreduce-examples-x.x.x.jarを使って、  
hdfsにwordcountの結果が出力されることを確認しました。

[code]

    hadoop jar hadoop-mapreduce-examples-x.x.x.jar wordcount /input /output
    
[/code]

次に自分でjarを作りたく、まずは上記サンプルのWordCount.javaをjarにして、それを実行しようとしています。  
サンプルは以下となり、packageだけ変更しました。  
WordCount.java

[code]

    /**
     * Licensed to the Apache Software Foundation (ASF) under one
     * or more contributor license agreements.  See the NOTICE file
     * distributed with this work for additional information
     * regarding copyright ownership.  The ASF licenses this file
     * to you under the Apache License, Version 2.0 (the
     * "License"); you may not use this file except in compliance
     * with the License.  You may obtain a copy of the License at
     *
     *     http://www.apache.org/licenses/LICENSE-2.0
     *
     * Unless required by applicable law or agreed to in writing, software
     * distributed under the License is distributed on an "AS IS" BASIS,
     * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     * See the License for the specific language governing permissions and
     * limitations under the License.
     */
    package hadooptest;
    
    import java.io.IOException;
    import java.util.StringTokenizer;
    
    import org.apache.hadoop.conf.Configuration;
    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.Mapper;
    import org.apache.hadoop.mapreduce.Reducer;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.hadoop.util.GenericOptionsParser;
    
    public class WordCount {
    
      public static class TokenizerMapper
           extends Mapper<Object, Text, Text, IntWritable>{
    
        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();
    
        public void map(Object key, Text value, Context context
                        ) throws IOException, InterruptedException {
          StringTokenizer itr = new StringTokenizer(value.toString());
          while (itr.hasMoreTokens()) {
            word.set(itr.nextToken());
            context.write(word, one);
          }
        }
      }
    
      public static class IntSumReducer
           extends Reducer<Text,IntWritable,Text,IntWritable> {
        private IntWritable result = new IntWritable();
    
        public void reduce(Text key, Iterable<IntWritable> values,
                           Context context
                           ) throws IOException, InterruptedException {
          int sum = 0;
          for (IntWritable val : values) {
            sum += val.get();
          }
          result.set(sum);
          context.write(key, result);
        }
      }
    
      public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
        if (otherArgs.length < 2) {
          System.err.println("Usage: wordcount <in> [<in>...] <out>");
          System.exit(2);
        }
        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        for (int i = 0; i < otherArgs.length - 1; ++i) {
          FileInputFormat.addInputPath(job, new Path(otherArgs[i]));
        }
        FileOutputFormat.setOutputPath(job,
          new Path(otherArgs[otherArgs.length - 1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
      }
    }
    
[/code]

コンパイル実行

[code]

    javac WordCount.java -cp $(/home/hadoop/hadoop-x.x.x/bin/hadoop classpath)
    
[/code]

実行結果

[code]

    WordCount.class
    WordCount$IntSumReducer.class
    WordCount$TokenizerMapper.class
    
[/code]

jar化

[code]

    jar -cf test.jar *.class
    
[/code]

上記で作成したjarを使ってHadoopコマンドを実行したのですが例外が発生してしまいます。

[code]

    $HADOOP_HOME/bin/hadoop jar test.jar WordCount /input /output
    Exception in thread "main" java.lang.ClassNotFoundException: WordCount
            at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
            at java.lang.Class.forName0(Native Method)
            at java.lang.Class.forName(Class.java:348)
            at org.apache.hadoop.util.RunJar.run(RunJar.java:316)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
    
[/code]

※同じような質問でフルパッケージ名にすればよいとあったので、以下を実施したのですが結果は変わらずでした。

[code]

    $HADOOP_HOME/bin/hadoop jar test.jar hadooptest.WordCount /input /output
    Exception in thread "main" java.lang.ClassNotFoundException: hadooptest.WordCount
            at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
            at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
            at java.lang.Class.forName0(Native Method)
            at java.lang.Class.forName(Class.java:348)
            at org.apache.hadoop.util.RunJar.run(RunJar.java:316)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
        
    
[/code]

何かアドバイスをいただけると幸いです。


CC BY-SA 4.0
2020-08-19T10:54:46.607
69695
2020-08-20T03:39:24.140
12842
question
hadoop
Hadoopのサンプルwordcountについて


ビルド時にJARに配置されないようにする必要があったようです。  
buildのディレクトリを作成しそこにClassファイルが作成されるようにコンパイル後、  
build配下で

[code]

     jar -cvf WordCount.jar .
    
[/code]

としたjarファイルを指定したところ、正常に実行されました。


CC BY-SA 4.0
2020-08-20T03:39:24.140
69713
2020-08-20T03:39:24.140
12842
69695
answer