From hadoop-commits-returnmillerjohn@example.net Mon Jul 03 10:35:19 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 44102 invoked from network); 3 Jul 2006 10:35:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Jul 2006 10:35:18 -0000 Received: (qmail 73602 invoked by uid 500); 3 Jul 2006 10:35:18 -0000 Delivered-To: millerjohn@example.net Received: (qmail 73581 invoked by uid 500); 3 Jul 2006 10:35:18 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 73571 invoked by uid 99); 3 Jul 2006 10:35:18 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jul 2006 03:35:18 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Jul 2006 03:35:17 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id 50BAB1A983A; Mon, 3 Jul 2006 03:34:57 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r418736 - /lucene/hadoop/nightly/nightly.sh Date: Mon, 03 Jul 2006 10:34:56 -0000 To: millerjohn@example.net From: millerjohn@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <millerjohn@example.net> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Mon Jul 3 03:34:53 2006 New Revision: 418736 URL: http://svn.apache.org/viewvc?rev=418736&view=rev Log: Use JDK 1.5 for nightly build. Modified: lucene/hadoop/nightly/nightly.sh Modified: lucene/hadoop/nightly/nightly.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/nightly/nightly.sh?rev=418736&r1=418735&r2=418736&view=diff ============================================================================== --- lucene/hadoop/nightly/nightly.sh (original) +++ lucene/hadoop/nightly/nightly.sh Mon Jul 3 03:34:53 2006 @@ -1,6 +1,6 @@ #!/bin/bash -vx -export JAVA_HOME=/usr/j2se +export JAVA_HOME=$HOME/local/jdk TRUNK=http://svn.apache.org/repos/asf/lucene/hadoop/trunk From hadoop-commits-returnmillerjohn@example.net Wed Jul 05 21:12:18 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 93275 invoked from network); 5 Jul 2006 21:12:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jul 2006 21:12:18 -0000 Received: (qmail 83166 invoked by uid 500); 5 Jul 2006 21:12:18 -0000 Delivered-To: millerjohn@example.net Received: (qmail 83128 invoked by uid 500); 5 Jul 2006 21:12:18 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 83119 invoked by uid 99); 5 Jul 2006 21:12:17 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2006 14:12:17 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2006 14:12:17 -0700 Received: from ajax.apache.org (localhost [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 6FBBFD498B for <millerjohn@example.net>; Wed, 5 Jul 2006 22:11:56 +0100 (BST) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <millerjohn@example.net> To: millerjohn@example.net Date: Wed, 05 Jul 2006 21:11:56 -0000 Message-ID: <millerjohn@example.net> Subject: [Lucene-hadoop Wiki] Update of "FrontPage" by OwenOMalley X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification. The following page has been changed by OwenOMalley: http://wiki.apache.org/lucene-hadoop/FrontPage ------------------------------------------------------------------------------ * Self:Roadmap, listing release plans. * HowToContribute * HowToRelease - + * HowManyMapsAndReduces == Examples == * WordCount From hadoop-commits-returnmillerjohn@example.net Wed Jul 05 22:05:56 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 11976 invoked from network); 5 Jul 2006 22:05:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jul 2006 22:05:56 -0000 Received: (qmail 61705 invoked by uid 500); 5 Jul 2006 22:05:56 -0000 Delivered-To: millerjohn@example.net Received: (qmail 61642 invoked by uid 500); 5 Jul 2006 22:05:56 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 61633 invoked by uid 99); 5 Jul 2006 22:05:55 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2006 15:05:55 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2006 15:05:55 -0700 Received: from ajax.apache.org (localhost [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 521E6D4973 for <millerjohn@example.net>; Wed, 5 Jul 2006 23:05:34 +0100 (BST) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <millerjohn@example.net> To: millerjohn@example.net Date: Wed, 05 Jul 2006 22:05:34 -0000 Message-ID: <millerjohn@example.net> Subject: [Lucene-hadoop Wiki] Update of "HowManyMapsAndReduces" by OwenOMalley X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification. The following page has been changed by OwenOMalley: http://wiki.apache.org/lucene-hadoop/HowManyMapsAndReduces New page: = Partitioning your job into maps and reduces = Picking the appropriate size for the tasks for your job can radically change the performance of Hadoop. Increasing the number of tasks increases the framework overhead, but increases load balancing and lowers the cost of failures. At one extreme is the 1 map/1 reduce case where nothing is distributed. The other extreme is to have 1,000,000 maps/ 1,000,000 reduces where the framework runs out of resources for the overhead. == Number of Maps == The number of maps is usually driven by the number of DFS blocks in the input files. Although that causes people to adjust their DFS block size to adjust the number of maps. The right level of parallelism for maps seems to be around 10-100 maps/node, although we have taken it up to 300 or so for very cpu-light map tasks. Task setup takes awhile, so it is best if the maps take at least a minute to execute. Actually controlling the number of maps is subtle. The mapred.map.tasks parameter is just a hint to the !InputFormat for the nubmer of maps. The default !InputFormat behavior is to split the total number of bytes into the right number of fragments. However, the DFS block size of the input files is treated as an upper bound for input splits. A lower bound on the split size can be set via mapred.min.split.size. Thus, if you expect 10TB of input data and have 128MB DFS blocks, you'll end up with 82k maps, unless your mapred.map.tasks is even larger. == Number of Reduces == The right number of reduces seems to be between 1.0 to 1.75 * (nodes * mapred.tasktracker.tasks.maximum). At 1.0 all of the reduces can launch immediately and start transfering map outputs as the maps finish. At 1.75 the faster nodes will finish their first round of reduces and launch a second round of reduces doing a much better job of load balancing. Currently the number of reduces is limited to roughly 1000 by the buffer size for the output files (io.buffer.size * 2 * numReduces << heapSize). This will be fixed at some point, but until it is it provides a pretty firm upper bound. The number of reduces also controls the number of output files in the output directory, but usually that is not important because the next map/reduce step will split them into even smaller splits for the maps. From hadoop-commits-returnmillerjohn@example.net Wed Jul 05 23:25:46 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 38203 invoked from network); 5 Jul 2006 23:25:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jul 2006 23:25:46 -0000 Received: (qmail 77047 invoked by uid 500); 5 Jul 2006 23:25:46 -0000 Delivered-To: millerjohn@example.net Received: (qmail 77022 invoked by uid 500); 5 Jul 2006 23:25:46 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 77013 invoked by uid 99); 5 Jul 2006 23:25:46 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2006 16:25:45 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2006 16:25:45 -0700 Received: from ajax.apache.org (localhost [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 8D54DD4973 for <millerjohn@example.net>; Thu, 6 Jul 2006 00:25:24 +0100 (BST) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <millerjohn@example.net> To: millerjohn@example.net Date: Wed, 05 Jul 2006 23:25:24 -0000 Message-ID: <millerjohn@example.net> Subject: [Lucene-hadoop Wiki] Update of "Sort" by OwenOMalley X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification. The following page has been changed by OwenOMalley: http://wiki.apache.org/lucene-hadoop/Sort ------------------------------------------------------------------------------ To run the program:[[BR]] bin/hadoop jar hadoop-*-examples.jar sort [-m <#maps>] [-r <#reduces>] <in-dir> <out-dir> + == Running Sort Benchmark == + + To use the sort example as a benchmark, generate 10GB/node of random data using RandomWriter. Then sort the data using ["Sort"]. This provides a sort benchmark that scales depending on the size of the cluster. By default, the ["Sort"] programs uses 1.0 * capacity for the number of reduces and depending on your cluster you may see better millerjohn@example.net * capacity. + From hadoop-commits-returnmillerjohn@example.net Wed Jul 05 23:36:47 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 41488 invoked from network); 5 Jul 2006 23:36:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jul 2006 23:36:47 -0000 Received: (qmail 95228 invoked by uid 500); 5 Jul 2006 23:36:47 -0000 Delivered-To: millerjohn@example.net Received: (qmail 95164 invoked by uid 500); 5 Jul 2006 23:36:47 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 95155 invoked by uid 99); 5 Jul 2006 23:36:47 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2006 16:36:47 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Jul 2006 16:36:46 -0700 Received: from ajax.apache.org (localhost [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id B55C4D4973 for <millerjohn@example.net>; Thu, 6 Jul 2006 00:36:25 +0100 (BST) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <millerjohn@example.net> To: millerjohn@example.net Date: Wed, 05 Jul 2006 23:36:25 -0000 Message-ID: <millerjohn@example.net> Subject: [Lucene-hadoop Wiki] Update of "Sort" by OwenOMalley X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification. The following page has been changed by OwenOMalley: http://wiki.apache.org/lucene-hadoop/Sort ------------------------------------------------------------------------------ == Running Sort Benchmark == - To use the sort example as a benchmark, generate 10GB/node of random data using RandomWriter. Then sort the data using ["Sort"]. This provides a sort benchmark that scales depending on the size of the cluster. By default, the ["Sort"] programs uses 1.0 * capacity for the number of reduces and depending on your cluster you may see better millerjohn@example.net * capacity. + To use the sort example as a benchmark, generate 10GB/node of random data using RandomWriter. Then sort the data using the sort example. This provides a sort benchmark that scales depending on the size of the cluster. By default, the sort example uses 1.0 * capacity for the number of reduces and depending on your cluster you may see better millerjohn@example.net * capacity. + The commands are: + % bin/hadoop jar hadoop-*-examples.jar randomwriter rand + + % bin/hadoop jar hadoop-*-examples.jar sort rand rand-sort + The first command will generate the unsorted data in the ''rand'' directory. The second command will read that data, sort it, and write into the ''rand-sort'' directory. + From hadoop-commits-returnmillerjohn@example.net Sat Jul 08 07:16:22 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 69851 invoked from network); 8 Jul 2006 07:16:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 8 Jul 2006 07:16:22 -0000 Received: (qmail 77306 invoked by uid 500); 8 Jul 2006 07:16:22 -0000 Delivered-To: millerjohn@example.net Received: (qmail 77273 invoked by uid 500); 8 Jul 2006 07:16:21 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 77263 invoked by uid 99); 8 Jul 2006 07:16:21 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Jul 2006 00:16:21 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 08 Jul 2006 00:16:21 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id D24C71A981A; Sat, 8 Jul 2006 00:16:00 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r420066 - /lucene/hadoop/trunk/conf/hadoop-default.xml Date: Sat, 08 Jul 2006 07:16:00 -0000 To: millerjohn@example.net From: millerjohn@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <millerjohn@example.net> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Sat Jul 8 00:16:00 2006 New Revision: 420066 URL: http://svn.apache.org/viewvc?rev=420066&view=rev Log: HADOOP-340. Document that non-existant directories in dfs.data.dir and mapred.local.dir are ignored. Modified: lucene/hadoop/trunk/conf/hadoop-default.xml Modified: lucene/hadoop/trunk/conf/hadoop-default.xml URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/conf/hadoop-default.xml?rev=420066&r1=420065&r2=420066&view=diff ============================================================================== --- lucene/hadoop/trunk/conf/hadoop-default.xml (original) +++ lucene/hadoop/trunk/conf/hadoop-default.xml Sat Jul 8 00:16:00 2006 @@ -129,7 +129,9 @@ <description>Determines where on the local filesystem an DFS data node should store its blocks. If this is a comma-delimited list of directories, then data will be stored in all named - directories, typically on different devices.</description> + directories, typically on different devices. + Directories that do not exist are ignored. + </description> </property> <property> @@ -216,6 +218,7 @@ <description>The local directory where MapReduce stores intermediate data files. May be a comma-separated list of directories on different devices in order to spread disk i/o. + Directories that do not exist are ignored. </description> </property> From hadoop-commits-returnmillerjohn@example.net Mon Jul 10 23:01:58 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 22139 invoked from network); 10 Jul 2006 23:01:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 10 Jul 2006 23:01:58 -0000 Received: (qmail 70748 invoked by uid 500); 10 Jul 2006 23:01:58 -0000 Delivered-To: millerjohn@example.net Received: (qmail 70719 invoked by uid 500); 10 Jul 2006 23:01:58 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 70702 invoked by uid 99); 10 Jul 2006 23:01:57 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jul 2006 16:01:57 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Jul 2006 16:01:57 -0700 Received: from ajax.apache.org (localhost [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 3272A6ACAA for <millerjohn@example.net>; Tue, 11 Jul 2006 00:01:36 +0100 (BST) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <millerjohn@example.net> To: millerjohn@example.net Date: Mon, 10 Jul 2006 23:01:36 -0000 Message-ID: <millerjohn@example.net> Subject: [Lucene-hadoop Wiki] Trivial Update of "HowToConfigure" by MichaelStack X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification. The following page has been changed by MichaelStack: http://wiki.apache.org/lucene-hadoop/HowToConfigure ------------------------------------------------------------------------------ Hadoop logs messages to Log4j by default. Log4j is configured via log4j.properties on the classpath. This file defines both what is logged and where. For applications, the default root logger is "INFO,console", which logs all message at level INFO and above to the console's stderr. Servers log to the "INFO,DRFA", which logs to a file that is rolled daily. Log files are named $HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-<server>.log. - For Hadoop developers, it is often convienent to get additional logging from particular classes. If you are working on the TaskTracker, for example, you would likely want + For Hadoop developers, it is often convenient to get additional logging from particular classes. If you are working on the TaskTracker, for example, you would likely want - log4.logger.org.apache.hadoop.mapred.!TaskTracker=DEBUG + log4j.logger.org.apache.hadoop.mapred.!TaskTracker=DEBUG in your log4j.properties. From hadoop-commits-returnmillerjohn@example.net Tue Jul 11 08:07:20 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 3575 invoked from network); 11 Jul 2006 08:07:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 11 Jul 2006 08:07:20 -0000 Received: (qmail 78694 invoked by uid 500); 11 Jul 2006 08:07:19 -0000 Delivered-To: millerjohn@example.net Received: (qmail 78667 invoked by uid 500); 11 Jul 2006 08:07:19 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 78658 invoked by uid 99); 11 Jul 2006 08:07:19 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jul 2006 01:07:19 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jul 2006 01:07:17 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id 488881A981A; Tue, 11 Jul 2006 01:06:57 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r420758 - in /lucene/hadoop/trunk: CHANGES.txt bin/hadoop bin/hadoop-daemon.sh bin/hadoop-daemons.sh bin/rcc bin/slaves.sh bin/start-all.sh bin/start-dfs.sh bin/start-mapred.sh bin/stop-all.sh bin/stop-dfs.sh bin/stop-mapred.sh Date: Tue, 11 Jul 2006 08:06:55 -0000 To: millerjohn@example.net From: millerjohn@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <millerjohn@example.net> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue Jul 11 01:06:54 2006 New Revision: 420758 URL: http://svn.apache.org/viewvc?rev=420758&view=rev Log: HADOOP-352. Fix shell scripts to use /bin/sh instead of /bin/bash for better portability. Contributed by Jean-Baptiste Quenot. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/bin/hadoop lucene/hadoop/trunk/bin/hadoop-daemon.sh lucene/hadoop/trunk/bin/hadoop-daemons.sh lucene/hadoop/trunk/bin/rcc lucene/hadoop/trunk/bin/slaves.sh lucene/hadoop/trunk/bin/start-all.sh lucene/hadoop/trunk/bin/start-dfs.sh lucene/hadoop/trunk/bin/start-mapred.sh lucene/hadoop/trunk/bin/stop-all.sh lucene/hadoop/trunk/bin/stop-dfs.sh lucene/hadoop/trunk/bin/stop-mapred.sh Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue Jul 11 01:06:54 2006 @@ -1,6 +1,13 @@ Hadoop Change Log +Trunk (unreleased changes) + + 1. HADOOP-352. Fix shell scripts to use /bin/sh instead of + /bin/bash, for better portability. + (Jean-Baptiste Quenot via cutting) + + Release 0.4.0 - 2006-06-28 1. HADOOP-298. Improved progress reports for CopyFiles utility, the Modified: lucene/hadoop/trunk/bin/hadoop URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/hadoop?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/hadoop (original) +++ lucene/hadoop/trunk/bin/hadoop Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # # The Hadoop command script # @@ -60,7 +60,7 @@ HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$HADOOP_HOME/conf}" if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then - source "${HADOOP_CONF_DIR}/hadoop-env.sh" + . "${HADOOP_CONF_DIR}/hadoop-env.sh" fi # some Java parameters Modified: lucene/hadoop/trunk/bin/hadoop-daemon.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/hadoop-daemon.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/hadoop-daemon.sh (original) +++ lucene/hadoop/trunk/bin/hadoop-daemon.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # # Runs a Hadoop command as a daemon. # @@ -44,7 +44,7 @@ HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$HADOOP_HOME/conf}" if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then - source "${HADOOP_CONF_DIR}/hadoop-env.sh" + . "${HADOOP_CONF_DIR}/hadoop-env.sh" fi # get log directory @@ -72,7 +72,7 @@ (start) if [ -f $pid ]; then - if [ -a /proc/`cat $pid` ]; then + if ps -p `cat $pid` > /dev/null 2>&1; then echo $command running as process `cat $pid`. Stop it first. exit 1 fi @@ -84,7 +84,7 @@ fi echo starting $command, logging to $log - nohup "$HADOOP_HOME"/bin/hadoop $command "$@" >& "$log" < /dev/null & + nohup "$HADOOP_HOME"/bin/hadoop $command "$@" > "$log" 2>&1 < /dev/null & echo $! > $pid sleep 1; head "$log" ;; @@ -92,7 +92,7 @@ (stop) if [ -f $pid ]; then - if [ -a /proc/`cat $pid` ]; then + if ps -p `cat $pid` > /dev/null 2>&1; then echo stopping $command kill `cat $pid` else Modified: lucene/hadoop/trunk/bin/hadoop-daemons.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/hadoop-daemons.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/hadoop-daemons.sh (original) +++ lucene/hadoop/trunk/bin/hadoop-daemons.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # # Run a Hadoop command on all slave hosts. Modified: lucene/hadoop/trunk/bin/rcc URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/rcc?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/rcc (original) +++ lucene/hadoop/trunk/bin/rcc Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # # The Hadoop record compiler # @@ -31,7 +31,7 @@ HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$HADOOP_HOME/conf}" if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then - source "${HADOOP_CONF_DIR}/hadoop-env.sh" + . "${HADOOP_CONF_DIR}/hadoop-env.sh" fi # some Java parameters Modified: lucene/hadoop/trunk/bin/slaves.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/slaves.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/slaves.sh (original) +++ lucene/hadoop/trunk/bin/slaves.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # # Run a shell command on all slave hosts. # @@ -38,7 +38,7 @@ HADOOP_CONF_DIR="${HADOOP_CONF_DIR:=$HADOOP_HOME/conf}" if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then - source "${HADOOP_CONF_DIR}/hadoop-env.sh" + . "${HADOOP_CONF_DIR}/hadoop-env.sh" fi if [ "$HADOOP_SLAVES" = "" ]; then Modified: lucene/hadoop/trunk/bin/start-all.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/start-all.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/start-all.sh (original) +++ lucene/hadoop/trunk/bin/start-all.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # Start all hadoop daemons. Run this on master node. Modified: lucene/hadoop/trunk/bin/start-dfs.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/start-dfs.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/start-dfs.sh (original) +++ lucene/hadoop/trunk/bin/start-dfs.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # Start hadoop dfs daemons. Run this on master node. Modified: lucene/hadoop/trunk/bin/start-mapred.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/start-mapred.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/start-mapred.sh (original) +++ lucene/hadoop/trunk/bin/start-mapred.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # Start hadoop map reduce daemons. Run this on master node. Modified: lucene/hadoop/trunk/bin/stop-all.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/stop-all.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/stop-all.sh (original) +++ lucene/hadoop/trunk/bin/stop-all.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # Stop all hadoop daemons. Run this on master node. Modified: lucene/hadoop/trunk/bin/stop-dfs.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/stop-dfs.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/stop-dfs.sh (original) +++ lucene/hadoop/trunk/bin/stop-dfs.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # Stop hadoop DFS daemons. Run this on master node. Modified: lucene/hadoop/trunk/bin/stop-mapred.sh URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/bin/stop-mapred.sh?rev=420758&r1=420757&r2=420758&view=diff ============================================================================== --- lucene/hadoop/trunk/bin/stop-mapred.sh (original) +++ lucene/hadoop/trunk/bin/stop-mapred.sh Tue Jul 11 01:06:54 2006 @@ -1,4 +1,4 @@ -#!/bin/bash +#!/bin/sh # Stop hadoop map reduce daemons. Run this on master node. From hadoop-commits-returnmillerjohn@example.net Tue Jul 11 08:16:53 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 8538 invoked from network); 11 Jul 2006 08:16:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 11 Jul 2006 08:16:52 -0000 Received: (qmail 97352 invoked by uid 500); 11 Jul 2006 08:16:52 -0000 Delivered-To: millerjohn@example.net Received: (qmail 97333 invoked by uid 500); 11 Jul 2006 08:16:52 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 97324 invoked by uid 99); 11 Jul 2006 08:16:52 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jul 2006 01:16:52 -0700 X-ASF-Spam-Status: No, hits=-8.6 required=10.0 tests=ALL_TRUSTED,INFO_TLD,NO_REAL_NAME X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jul 2006 01:16:51 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id 2F9491A981A; Tue, 11 Jul 2006 01:16:31 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r420760 - in /lucene/hadoop/trunk: ./ src/java/org/apache/hadoop/conf/ src/java/org/apache/hadoop/mapred/ Date: Tue, 11 Jul 2006 08:16:29 -0000 To: millerjohn@example.net From: millerjohn@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <millerjohn@example.net> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue Jul 11 01:16:28 2006 New Revision: 420760 URL: http://svn.apache.org/viewvc?rev=420760&view=rev Log: HADOOP-313. Permit task state to be saved so that single tasks may be manually re-executed when debugging. Contributed by Owen. Added: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/IsolationRunner.java Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/conf/Configuration.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobConf.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/Task.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskInProgress.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue Jul 11 01:16:28 2006 @@ -7,6 +7,9 @@ /bin/bash, for better portability. (Jean-Baptiste Quenot via cutting) + 2. HADOOP-313. Permit task state to be saved so that single tasks + may be manually re-executed when debugging. (omalley via cutting) + Release 0.4.0 - 2006-06-28 Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/conf/Configuration.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/conf/Configuration.java?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/conf/Configuration.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/conf/Configuration.java Tue Jul 11 01:16:28 2006 @@ -494,6 +494,14 @@ } } + /** + * Set the class loader that will be used to load the various objects. + * @param classLoader the new class loader + */ + public void setClassLoader(ClassLoader classLoader) { + this.classLoader = classLoader; + } + public String toString() { StringBuffer sb = new StringBuffer(); sb.append("Configuration: "); Added: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/IsolationRunner.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/IsolationRunner.java?rev=420760&view=auto ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/IsolationRunner.java (added) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/IsolationRunner.java Tue Jul 11 01:16:28 2006 @@ -0,0 +1,162 @@ +/** + * Copyright 2005 The Apache Software Foundation + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.mapred; + +import java.io.File; +import java.io.IOException; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.logging.*; +import org.apache.hadoop.fs.*; +import org.apache.hadoop.io.SequenceFile; + +public class IsolationRunner { + private static final Log LOG = + LogFactory.getLog(IsolationRunner.class.getName()); + + private static class FakeUmbilical implements TaskUmbilicalProtocol { + + public void done(String taskid) throws IOException { + LOG.info("Task " + taskid + " reporting done."); + } + + public void fsError(String message) throws IOException { + LOG.info("Task reporting file system error: " + message); + } + + public Task getTask(String taskid) throws IOException { + return null; + } + + public boolean ping(String taskid) throws IOException { + return true; + } + + public void progress(String taskid, float progress, String state + ) throws IOException { + StringBuffer buf = new StringBuffer("Task "); + buf.append(taskid); + buf.append(" making progress to "); + buf.append(progress); + if (state != null) { + buf.append(" and state of "); + buf.append(state); + } + LOG.info(buf.toString()); + } + + public void reportDiagnosticInfo(String taskid, String trace) throws IOException { + LOG.info("Task " + taskid + " has problem " + trace); + } + + } + + private static ClassLoader makeClassLoader(JobConf conf, + File workDir) throws IOException { + List cp = new ArrayList(); + + String jar = conf.getJar(); + if (jar != null) { // if jar exists, it into workDir + File[] libs = new File(workDir, "lib").listFiles(); + if (libs != null) { + for (int i = 0; i < libs.length; i++) { + cp.add(new URL("file:" + libs[i].toString())); + } + } + cp.add(new URL("file:" + new File(workDir, "classes/").toString())); + cp.add(new URL("file:" + workDir.toString() + "/")); + } + return new URLClassLoader((URL[]) cp.toArray(new URL[cp.size()])); + } + + /** + * Create empty sequence files for any of the map outputs that we don't have. + * @param fs the filesystem to create the files in + * @param dir the directory name to create the files in + * @param conf the jobconf + * @throws IOException if something goes wrong writing + */ + private static void fillInMissingMapOutputs(FileSystem fs, + String taskId, + int numMaps, + JobConf conf) throws IOException { + Class keyClass = conf.getMapOutputKeyClass(); + Class valueClass = conf.getMapOutputValueClass(); + MapOutputFile namer = new MapOutputFile(); + namer.setConf(conf); + for(int i=0; i<numMaps; i++) { + Path f = namer.getInputFile(i, taskId); + if(! fs.exists(f)) { + LOG.info("Create missing input: " + f); + SequenceFile.Writer out = new SequenceFile.Writer(fs, f, keyClass, + valueClass); + out.close(); + } + } + } + + /** + * Run a single task + * @param args the first argument is the task directory + */ + public static void main(String[] args) throws IOException { + if (args.length != 1) { + System.out.println("Usage: IsolationRunner <path>/job.xml"); + System.exit(1); + } + File jobFilename = new File(args[0]); + if (!jobFilename.exists() || !jobFilename.isFile()) { + System.out.println(jobFilename + " is not a valid job file."); + System.exit(1); + } + JobConf conf = new JobConf(new Path(jobFilename.toString())); + String taskId = conf.get("mapred.task.id"); + boolean isMap = conf.getBoolean("mapred.task.is.map", true); + String jobId = conf.get("mapred.job.id"); + int partition = conf.getInt("mapred.task.partition", 0); + + // setup the local and user working directories + FileSystem local = FileSystem.getNamed("local", conf); + File workDirName = new File(jobFilename.getParent(), "work"); + local.setWorkingDirectory(new Path(workDirName.toString())); + FileSystem.get(conf).setWorkingDirectory(conf.getWorkingDirectory()); + + // set up a classloader with the right classpath + ClassLoader classLoader = makeClassLoader(conf, workDirName); + Thread.currentThread().setContextClassLoader(classLoader); + conf.setClassLoader(classLoader); + + Task task; + if (isMap) { + FileSplit split = new FileSplit(new Path(conf.get("map.input.file")), + conf.getLong("map.input.start", 0), + conf.getLong("map.input.length", 0)); + task = new MapTask(jobId, jobFilename.toString(), taskId, partition, + split); + } else { + int numMaps = conf.getNumMapTasks(); + fillInMissingMapOutputs(local, taskId, numMaps, conf); + task = new ReduceTask(jobId, jobFilename.toString(), taskId, + partition, numMaps); + } + task.setConf(conf); + task.run(conf, new FakeUmbilical()); + } + +} Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobConf.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobConf.java?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobConf.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobConf.java Tue Jul 11 01:16:28 2006 @@ -194,6 +194,22 @@ public void setUser(String user) { set("user.name", user); } + + /** + * Set whether the framework shoul keep the intermediate files for + * failed tasks. + */ + public void setKeepFailedTaskFiles(boolean keep) { + setBoolean("keep.failed.task.files", keep); + } + + /** + * Should the temporary files for failed tasks be kept? + * @return should the files be kept? + */ + public boolean getKeepFailedTaskFiles() { + return getBoolean("keep.failed.task.files", false); + } /** * Set the current working directory for the default file system Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java Tue Jul 11 01:16:28 2006 @@ -78,13 +78,14 @@ // split input into minimum number of splits FileSplit[] splits; splits = job.getInputFormat().getSplits(fs, job, 1); - + String jobId = profile.getJobId(); // run a map task for each split job.setNumReduceTasks(1); // force a single reduce task for (int i = 0; i < splits.length; i++) { mapIds.add("map_" + newId()); - MapTask map = new MapTask(file, (String)mapIds.get(i), splits[i]); + MapTask map = new MapTask(jobId, file, (String)mapIds.get(i), i, + splits[i]); map.setConf(job); map_tasks += 1; map.run(job, this); @@ -104,8 +105,8 @@ } // run a single reduce task - ReduceTask reduce = new ReduceTask(profile.getJobId(), file, - reduceId, mapIds.size(),0); + ReduceTask reduce = new ReduceTask(jobId, file, + reduceId, 0, mapIds.size()); reduce.setConf(job); reduce_tasks += 1; reduce.run(job, this); Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/MapTask.java Tue Jul 11 01:16:28 2006 @@ -39,8 +39,9 @@ public MapTask() {} - public MapTask(String jobFile, String taskId, FileSplit split) { - super(jobFile, taskId); + public MapTask(String jobId, String jobFile, String taskId, + int partition, FileSplit split) { + super(jobId, jobFile, taskId, partition); this.split = split; } @@ -48,6 +49,13 @@ return true; } + public void localizeConfiguration(JobConf conf) { + super.localizeConfiguration(conf); + conf.set("map.input.file", split.getPath().toString()); + conf.setLong("map.input.start", split.getStart()); + conf.setLong("map.input.length", split.getLength()); + } + public TaskRunner createRunner(TaskTracker tracker) { return new MapTaskRunner(this, tracker, this.conf); } Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/ReduceTask.java Tue Jul 11 01:16:28 2006 @@ -36,9 +36,7 @@ }); } - private UTF8 jobId = new UTF8(); private int numMaps; - private int partition; private boolean sortComplete; { getProgress().setStatus("reduce"); } @@ -52,11 +50,9 @@ public ReduceTask() {} public ReduceTask(String jobId, String jobFile, String taskId, - int numMaps, int partition) { - super(jobFile, taskId); - this.jobId.set(jobId); + int partition, int numMaps) { + super(jobId, jobFile, taskId, partition); this.numMaps = numMaps; - this.partition = partition; } public TaskRunner createRunner(TaskTracker tracker) throws IOException { @@ -67,31 +63,26 @@ return false; } + public int getNumMaps() { return numMaps; } + /** - * Get the job name for this task. - * @return the job name + * Localize the given JobConf to be specific for this task. */ - public UTF8 getJobId() { - return jobId; + public void localizeConfiguration(JobConf conf) { + super.localizeConfiguration(conf); + conf.setNumMapTasks(numMaps); } - - public int getNumMaps() { return numMaps; } - public int getPartition() { return partition; } public void write(DataOutput out) throws IOException { super.write(out); - jobId.write(out); out.writeInt(numMaps); // write the number of maps - out.writeInt(partition); // write partition } public void readFields(DataInput in) throws IOException { super.readFields(in); - jobId.readFields(in); numMaps = in.readInt(); - this.partition = in.readInt(); // read partition } /** Iterates values while keys match in sorted input. */ @@ -215,7 +206,7 @@ // sort the input file SequenceFile.Sorter sorter = new SequenceFile.Sorter(lfs, comparator, valueClass, job); - sorter.sort(mapFiles, sortedFile, true); // sort + sorter.sort(mapFiles, sortedFile, !conf.getKeepFailedTaskFiles()); // sort } finally { sortComplete = true; Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/Task.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/Task.java?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/Task.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/Task.java Tue Jul 11 01:16:28 2006 @@ -35,15 +35,20 @@ private String jobFile; // job configuration file private String taskId; // unique, includes job id + private String jobId; // unique jobid + private int partition; // id within job + //////////////////////////////////////////// // Constructors //////////////////////////////////////////// public Task() {} - public Task(String jobFile, String taskId) { + public Task(String jobId, String jobFile, String taskId, int partition) { this.jobFile = jobFile; this.taskId = taskId; + this.jobId = jobId; + this.partition = partition; } //////////////////////////////////////////// @@ -52,6 +57,22 @@ public void setJobFile(String jobFile) { this.jobFile = jobFile; } public String getJobFile() { return jobFile; } public String getTaskId() { return taskId; } + + /** + * Get the job name for this task. + * @return the job name + */ + public String getJobId() { + return jobId; + } + + /** + * Get the index of this task within the job. + * @return the integer part of the task id + */ + public int getPartition() { + return partition; + } //////////////////////////////////////////// // Writable methods @@ -60,14 +81,28 @@ public void write(DataOutput out) throws IOException { UTF8.writeString(out, jobFile); UTF8.writeString(out, taskId); + UTF8.writeString(out, jobId); + out.writeInt(partition); } public void readFields(DataInput in) throws IOException { jobFile = UTF8.readString(in); taskId = UTF8.readString(in); + jobId = UTF8.readString(in); + partition = in.readInt(); } public String toString() { return taskId; } + /** + * Localize the given JobConf to be specific for this task. + */ + public void localizeConfiguration(JobConf conf) { + conf.set("mapred.task.id", taskId); + conf.setBoolean("mapred.task.is.map",isMapTask()); + conf.setInt("mapred.task.partition", partition); + conf.set("mapred.job.id", jobId); + } + /** Run this task as a part of the named job. This method is executed in the * child process and is what invokes user-supplied map, reduce, etc. methods. * @param umbilical for progress reports Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskInProgress.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskInProgress.java?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskInProgress.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskInProgress.java Tue Jul 11 01:16:28 2006 @@ -413,12 +413,12 @@ String taskid = (String) usableTaskIds.first(); usableTaskIds.remove(taskid); + String jobId = job.getProfile().getJobId(); if (isMapTask()) { - t = new MapTask(jobFile, taskid, split); + t = new MapTask(jobId, jobFile, taskid, partition, split); } else { - t = new ReduceTask(job.getProfile().getJobId(), jobFile, taskid, - numMaps, partition); + t = new ReduceTask(jobId, jobFile, taskid, partition, numMaps); } t.setConf(conf); Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java?rev=420760&r1=420759&r2=420760&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java Tue Jul 11 01:16:28 2006 @@ -652,6 +652,7 @@ boolean wasKilled = false; private JobConf defaultJobConf; private JobConf localJobConf; + private boolean keepFailedTaskFiles; /** */ @@ -682,7 +683,6 @@ t.setJobFile(localJobFile.toString()); localJobConf = new JobConf(localJobFile); - localJobConf.set("mapred.task.id", task.getTaskId()); localJobConf.set("mapred.local.dir", this.defaultJobConf.get("mapred.local.dir")); String jarFile = localJobConf.getJar(); @@ -690,6 +690,7 @@ fs.copyToLocalFile(new Path(jarFile), localJarFile); localJobConf.setJar(localJarFile.toString()); } + task.localizeConfiguration(localJobConf); FileSystem localFs = FileSystem.getNamed("local", fConf); OutputStream out = localFs.create(localJobFile); @@ -701,6 +702,7 @@ // set the task's configuration to the local job conf // rather than the default. t.setConf(localJobConf); + keepFailedTaskFiles = localJobConf.getKeepFailedTaskFiles(); } /** @@ -875,6 +877,9 @@ LOG.debug("Cleaning up " + taskId); synchronized (TaskTracker.this) { tasks.remove(taskId); + if (runstate == TaskStatus.FAILED && keepFailedTaskFiles) { + return; + } synchronized (this) { try { runner.close(); From hadoop-commits-returnmillerjohn@example.net Tue Jul 11 08:43:54 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 16418 invoked from network); 11 Jul 2006 08:43:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 11 Jul 2006 08:43:53 -0000 Received: (qmail 45262 invoked by uid 500); 11 Jul 2006 08:43:53 -0000 Delivered-To: millerjohn@example.net Received: (qmail 45203 invoked by uid 500); 11 Jul 2006 08:43:52 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 45194 invoked by uid 99); 11 Jul 2006 08:43:52 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jul 2006 01:43:52 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jul 2006 01:43:51 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id 90DE71A981A; Tue, 11 Jul 2006 01:43:31 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r420764 - in /lucene/hadoop/trunk: ./ src/java/org/apache/hadoop/mapred/ Date: Tue, 11 Jul 2006 08:43:30 -0000 To: millerjohn@example.net From: millerjohn@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <millerjohn@example.net> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue Jul 11 01:43:29 2006 New Revision: 420764 URL: http://svn.apache.org/viewvc?rev=420764&view=rev Log: HADOOP-339. Add a method to the JobClient API listing jobs that are not yet complete. Contributed by Mahadev. Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobStatus.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobSubmissionProtocol.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?rev=420764&r1=420763&r2=420764&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue Jul 11 01:43:29 2006 @@ -10,6 +10,10 @@ 2. HADOOP-313. Permit task state to be saved so that single tasks may be manually re-executed when debugging. (omalley via cutting) + 3. HADOOP-339. Add method to JobClient API listing jobs that are + not yet complete, i.e., that are queued or running. + (Mahadev Konar via cutting) + Release 0.4.0 - 2006-06-28 Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java?rev=420764&r1=420763&r2=420764&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobClient.java Tue Jul 11 01:43:29 2006 @@ -314,6 +314,10 @@ return jobSubmitClient.getClusterStatus(); } + public JobStatus[] jobsToComplete() throws IOException { + return jobSubmitClient.jobsToComplete(); + } + /** Utility that submits a job, then polls for progress until the job is * complete. */ public static void runJob(JobConf job) throws IOException { Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobStatus.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobStatus.java?rev=420764&r1=420763&r2=420764&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobStatus.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobStatus.java Tue Jul 11 01:43:29 2006 @@ -26,7 +26,7 @@ * * @author Mike Cafarella **************************************************/ -class JobStatus implements Writable { +public class JobStatus implements Writable { static { // register a ctor WritableFactories.setFactory @@ -45,13 +45,18 @@ float mapProgress; float reduceProgress; int runState; - + long startTime; /** */ public JobStatus() { } /** + * Create a job status object for a given jobid. + * @param jobid The jobid of the job + * @param mapProgress The progress made on the maps + * @param reduceProgress The progress made on the reduces + * @param runState The current state of the job */ public JobStatus(String jobid, float mapProgress, float reduceProgress, int runState) { this.jobid = jobid; @@ -61,14 +66,47 @@ } /** + * @return The jobid of the Job */ public String getJobId() { return jobid; } + + /** + * @return Percentage of progress in maps + */ public float mapProgress() { return mapProgress; } - public void setMapProgress(float p) { this.mapProgress = p; } + + /** + * Sets the map progress of this job + * @param p The value of map progress to set to + */ + void setMapProgress(float p) { this.mapProgress = p; } + + /** + * @return Percentage of progress in reduce + */ public float reduceProgress() { return reduceProgress; } - public void setReduceProgress(float p) { this.reduceProgress = p; } + + /** + * Sets the reduce progress of this Job + * @param p The value of reduce progress to set to + */ + void setReduceProgress(float p) { this.reduceProgress = p; } + + /** + * @return running state of the job + */ public int getRunState() { return runState; } - + + /** + * Set the start time of the job + * @param startTime The startTime of the job + */ + void setStartTime(long startTime) { this.startTime = startTime;}; + + /** + * @return start time of the job + */ + public long getStartTime() { return startTime;}; /////////////////////////////////////// // Writable /////////////////////////////////////// @@ -77,11 +115,13 @@ out.writeFloat(mapProgress); out.writeFloat(reduceProgress); out.writeInt(runState); + out.writeLong(startTime); } public void readFields(DataInput in) throws IOException { this.jobid = UTF8.readString(in); this.mapProgress = in.readFloat(); this.reduceProgress = in.readFloat(); this.runState = in.readInt(); + this.startTime = in.readLong(); } } Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobSubmissionProtocol.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobSubmissionProtocol.java?rev=420764&r1=420763&r2=420764&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobSubmissionProtocol.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobSubmissionProtocol.java Tue Jul 11 01:43:29 2006 @@ -17,6 +17,7 @@ package org.apache.hadoop.mapred; import java.io.*; +import java.util.*; /** * Protocol that a JobClient and the central JobTracker use to communicate. The @@ -64,4 +65,11 @@ * prior to submitting the job. */ public String getFilesystemName() throws IOException; + + /** + * Get the jobs that are not completed and not failed + * @return array of JobStatus for the running/to-be-run + * jobs. + */ + public JobStatus[] jobsToComplete() throws IOException; } Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java?rev=420764&r1=420763&r2=420764&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/JobTracker.java Tue Jul 11 01:43:29 2006 @@ -1038,6 +1038,20 @@ return (String) taskidToTrackerMap.get(taskId); } + public JobStatus[] jobsToComplete() { + Vector v = new Vector(); + for (Iterator it = jobs.values().iterator(); it.hasNext(); ) { + JobInProgress jip = (JobInProgress) it.next(); + JobStatus status = jip.getStatus(); + if (status.getRunState() == JobStatus.RUNNING + || status.getRunState() == JobStatus.PREP) { + status.setStartTime(jip.getStartTime()); + v.add(status); + } + } + return (JobStatus[]) v.toArray(new JobStatus[v.size()]); + } + /////////////////////////////////////////////////////////////// // JobTracker methods /////////////////////////////////////////////////////////////// Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java?rev=420764&r1=420763&r2=420764&view=diff ============================================================================== --- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java (original) +++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/LocalJobRunner.java Tue Jul 11 01:43:29 2006 @@ -210,4 +210,6 @@ public ClusterStatus getClusterStatus() { return new ClusterStatus(1, map_tasks, reduce_tasks, 1); } + + public JobStatus[] jobsToComplete() {return null;}; } From hadoop-commits-returnmillerjohn@example.net Tue Jul 11 08:58:33 2006 Return-Path: <hadoop-commits-returnmillerjohn@example.net> Delivered-To: millerjohn@example.net Received: (qmail 21508 invoked from network); 11 Jul 2006 08:58:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 11 Jul 2006 08:58:32 -0000 Received: (qmail 67415 invoked by uid 500); 11 Jul 2006 08:58:32 -0000 Delivered-To: millerjohn@example.net Received: (qmail 67370 invoked by uid 500); 11 Jul 2006 08:58:31 -0000 Mailing-List: contact millerjohn@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:millerjohn@example.net> List-Unsubscribe: <mailto:millerjohn@example.net> List-Post: <mailto:millerjohn@example.net> List-Id: <hadoop-commits.lucene.apache.org> Reply-To: millerjohn@example.net Delivered-To: mailing list millerjohn@example.net Received: (qmail 67361 invoked by uid 99); 11 Jul 2006 08:58:31 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jul 2006 01:58:31 -0700 X-ASF-Spam-Status: No, hits=-8.6 required=10.0 tests=ALL_TRUSTED,INFO_TLD,NO_REAL_NAME X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jul 2006 01:58:28 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id A75E21A981A; Tue, 11 Jul 2006 01:58:08 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r420768 - in /lucene/hadoop/trunk: ./ src/contrib/streaming/src/java/org/apache/hadoop/streaming/ Date: Tue, 11 Jul 2006 08:58:06 -0000 To: millerjohn@example.net From: millerjohn@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <millerjohn@example.net> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: cutting Date: Tue Jul 11 01:58:05 2006 New Revision: 420768 URL: http://svn.apache.org/viewvc?rev=420768&view=rev Log: HADOOP-355. Updates to streaming contrib module. Contributed by Michel. Added: lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamSequenceRecordReader.java Modified: lucene/hadoop/trunk/CHANGES.txt lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapper.java lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamBaseRecordReader.java lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamInputFormat.java lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamLineRecordReader.java lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamOutputFormat.java lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamXmlRecordReader.java Modified: lucene/hadoop/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?rev=420768&r1=420767&r2=420768&view=diff ============================================================================== --- lucene/hadoop/trunk/CHANGES.txt (original) +++ lucene/hadoop/trunk/CHANGES.txt Tue Jul 11 01:58:05 2006 @@ -14,6 +14,10 @@ not yet complete, i.e., that are queued or running. (Mahadev Konar via cutting) + 4. HADOOP-355. Updates to the streaming contrib module, including + API fixes, making reduce optional, and adding an input type for + StreamSequenceRecordReader. (Michel Tourn via cutting) + Release 0.4.0 - 2006-06-28 Modified: lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java?rev=420768&r1=420767&r2=420768&view=diff ============================================================================== --- lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java (original) +++ lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapRed.java Tue Jul 11 01:58:05 2006 @@ -35,11 +35,16 @@ import org.apache.hadoop.mapred.OutputCollector; import org.apache.hadoop.io.UTF8; +import org.apache.hadoop.io.BytesWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.WritableComparator; import org.apache.hadoop.io.WritableComparable; -import org.apache.hadoop.io.Writable; import org.apache.hadoop.io.LongWritable; +import org.apache.hadoop.io.Writable; + +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.FSDataOutputStream; /** Shared functionality for PipeMapper, PipeReducer. * @author Michel Tourn @@ -56,7 +61,13 @@ */ abstract String getKeyColPropName(); - + /** Write output as side-effect files rather than as map outputs. + This is useful to do "Map" tasks rather than "MapReduce" tasks. */ + boolean getUseSideEffect() + { + return false; + } + /** * @returns how many TABS before the end of the key part * usually: 1 or "ALL" @@ -154,7 +165,10 @@ String argv = getPipeCommand(job); keyCols_ = getKeyColsFromPipeCommand(argv); - doPipe_ = (argv != null); + job_ = job; + + // Currently: null is identity reduce. REDUCE_NONE is no-map-outputs. + doPipe_ = (argv != null) && !StreamJob.REDUCE_NONE.equals(argv); if(!doPipe_) return; setStreamJobDetails(job); @@ -169,29 +183,46 @@ new MustangFile(prog).setExecutable(true, true); } + + if(job_.getInputValueClass().equals(BytesWritable.class)) { + // TODO expose as separate config: + // job or semistandard inputformat property + optUseKey_ = false; + } + + optSideEffect_ = getUseSideEffect(); + + if(optSideEffect_) { + String fileName = job_.get("mapred.task.id"); + sideEffectPath_ = new Path(job_.getOutputPath(), fileName); + FileSystem fs = FileSystem.get(job_); + sideEffectOut_ = fs.create(sideEffectPath_); + } + // argvSplit[0]: // An absolute path should be a preexisting valid path on all TaskTrackers // A relative path should match in the unjarred Job data // In this case, force an absolute path to make sure exec finds it. argvSplit[0] = new File(argvSplit[0]).getAbsolutePath(); - log_.println("PipeMapRed exec " + Arrays.asList(argvSplit)); - + logprintln("PipeMapRed exec " + Arrays.asList(argvSplit)); + logprintln("sideEffectPath_=" + sideEffectPath_); Environment childEnv = (Environment)StreamUtil.env().clone(); - addEnvironment(childEnv, job.get("stream.addenvironment")); + addEnvironment(childEnv, job_.get("stream.addenvironment")); sim = Runtime.getRuntime().exec(argvSplit, childEnv.toArray()); /* // This way required jdk1.5 ProcessBuilder processBuilder = new ProcessBuilder(argvSplit); Map<String, String> env = processBuilder.environment(); - addEnvironment(env, job.get("stream.addenvironment")); + addEnvironment(env, job_.get("stream.addenvironment")); sim = processBuilder.start(); */ clientOut_ = new DataOutputStream(new BufferedOutputStream(sim.getOutputStream())); - clientIn_ = new BufferedReader(new InputStreamReader(sim.getInputStream())); - clientErr_ = new DataInputStream(sim.getErrorStream()); + clientIn_ = new DataInputStream(new BufferedInputStream(sim.getInputStream())); + clientErr_ = new DataInputStream(new BufferedInputStream(sim.getErrorStream())); doneLock_ = new Object(); + startTime_ = System.currentTimeMillis(); } catch(Exception e) { e.printStackTrace(); @@ -205,7 +236,7 @@ String s = job.get("stream.minRecWrittenToEnableSkip_"); if(s != null) { minRecWrittenToEnableSkip_ = Long.parseLong(s); - log_.println("JobConf set minRecWrittenToEnableSkip_ =" + minRecWrittenToEnableSkip_); + logprintln("JobConf set minRecWrittenToEnableSkip_ =" + minRecWrittenToEnableSkip_); } } @@ -222,6 +253,22 @@ } + void logprintln(String s) + { + if(log_ != null) { + log_.println(s); + } else { + System.err.println(s); // or LOG.info() + } + } + + void logflush() + { + if(log_ != null) { + log_.flush(); + } + } + void addEnvironment(Properties env, String nameVals) { // encoding "a=b c=d" from StreamJob @@ -230,9 +277,9 @@ for(int i=0; i<nv.length; i++) { String[] pair = nv[i].split("=", 2); if(pair.length != 2) { - log_.println("Skip ev entry:" + nv[i]); + logprintln("Skip ev entry:" + nv[i]); } else { - log_.println("Add ev entry:" + nv[i]); + logprintln("Add ev entry:" + nv[i]); env.put(pair[0], pair[1]); } } @@ -293,18 +340,23 @@ // 3/4 Tool to Hadoop while((answer = clientIn_.readLine()) != null) { // 4/4 Hadoop out - splitKeyVal(answer, key, val); - output.collect(key, val); - numRecWritten_++; - if(numRecWritten_ % 100 == 0) { - log_.println(numRecRead_+"/"+numRecWritten_); - log_.flush(); + if(optSideEffect_) { + sideEffectOut_.write(answer.getBytes()); + sideEffectOut_.write('\n'); + } else { + splitKeyVal(answer, key, val); + output.collect(key, val); + numRecWritten_++; + if(numRecWritten_ % 100 == 0) { + logprintln(numRecRead_+"/"+numRecWritten_); + logflush(); + } } } } catch(IOException io) { io.printStackTrace(log_); } - log_.println("MROutputThread done"); + logprintln("MROutputThread done"); } finally { outputDone_ = true; synchronized(doneLock_) { @@ -332,7 +384,7 @@ int bucket = 100; while((line=clientErr_.readLine()) != null) { num++; - log_.println(line); + logprintln(line); if(num < 10) { String hline = "MRErr: " + line; System.err.println(hline); @@ -353,10 +405,19 @@ public void mapRedFinished() { - log_.println("mapRedFinished"); + logprintln("mapRedFinished"); try { if(!doPipe_) return; try { + if(optSideEffect_) { + logprintln("closing " + sideEffectPath_); + sideEffectOut_.close(); + logprintln("closed " + sideEffectPath_); + } + } catch(IOException io) { + io.printStackTrace(); + } + try { if(clientOut_ != null) { clientOut_.close(); } @@ -385,10 +446,12 @@ void maybeLogRecord() { if(numRecRead_ >= nextRecReadLog_) { - log_.println(numRecInfo()); - log_.flush(); - nextRecReadLog_ *= 10; - //nextRecReadLog_ += 1000; + String info = numRecInfo(); + logprintln(info); + logflush(); + System.err.println(info); + //nextRecReadLog_ *= 10; + nextRecReadLog_ += 100; } } @@ -417,7 +480,15 @@ String numRecInfo() { - return "R/W/S=" + numRecRead_+"/"+numRecWritten_+"/"+numRecSkipped_; + long elapsed = (System.currentTimeMillis() - startTime_)/1000; + long total = numRecRead_+numRecWritten_+numRecSkipped_; + return "R/W/S=" + numRecRead_+"/"+numRecWritten_+"/"+numRecSkipped_ + + " in:" + safeDiv(numRecRead_, elapsed) + " [rec/s]" + + " out:" + safeDiv(numRecWritten_, elapsed) + " [rec/s]"; + } + String safeDiv(long n, long d) + { + return (d==0) ? "NA" : ""+n/d + "=" + n + "/" + d; } String logFailure(Exception e) { @@ -425,15 +496,15 @@ PrintWriter pw = new PrintWriter(sw); e.printStackTrace(pw); String msg = "log:" + jobLog_ + "\n" + getContext() + sw + "\n"; - log_.println(msg); + logprintln(msg); return msg; } + long startTime_; long numRecRead_ = 0; long numRecWritten_ = 0; long numRecSkipped_ = 0; - long nextRecReadLog_ = 1; long minRecWrittenToEnableSkip_ = Long.MAX_VALUE; @@ -441,6 +512,8 @@ int keyCols_; final static int ALL_COLS = Integer.MAX_VALUE; + JobConf job_; + // generic MapRed parameters passed on by hadoopStreaming String taskid_; int reportPortPlusOne_; @@ -455,24 +528,31 @@ boolean errorDone_; DataOutputStream clientOut_; DataInputStream clientErr_; - BufferedReader clientIn_; + DataInputStream clientIn_; String jobLog_; // set in PipeMapper/PipeReducer subclasses String mapredKey_; int numExceptions_; + boolean optUseKey_ = true; + + boolean optSideEffect_; + Path sideEffectPath_; + FSDataOutputStream sideEffectOut_; + String LOGNAME; PrintStream log_; + /* curr. going to stderr so that it is preserved { // instance initializer try { int id = (int)((System.currentTimeMillis()/2000) % 10); String sid = id+ "." + StreamUtil.env().get("USER"); LOGNAME = "/tmp/PipeMapRed." + sid + ".log"; log_ = new PrintStream(new FileOutputStream(LOGNAME)); - log_.println(new java.util.Date()); - log_.flush(); + logprintln(new java.util.Date()); + logflush(); } catch(IOException io) { System.err.println("LOGNAME=" + LOGNAME); io.printStackTrace(); @@ -482,5 +562,5 @@ } } } - + */ } Modified: lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapper.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapper.java?rev=420768&r1=420767&r2=420768&view=diff ============================================================================== --- lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapper.java (original) +++ lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/PipeMapper.java Tue Jul 11 01:58:05 2006 @@ -25,6 +25,7 @@ import org.apache.hadoop.mapred.Reporter; import org.apache.hadoop.mapred.OutputCollector; +import org.apache.hadoop.io.BytesWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.WritableComparator; import org.apache.hadoop.io.WritableComparable; @@ -47,6 +48,15 @@ return "mapKeyCols"; } + boolean getUseSideEffect() + { + String reduce = job_.get("stream.reduce.streamprocessor"); + if(StreamJob.REDUCE_NONE.equals(reduce)) { + return true; + } + return false; + } + // Do NOT declare default constructor // (MapRed creates it reflectively) @@ -61,16 +71,28 @@ } try { // 1/4 Hadoop in - mapredKey_ = key.toString(); + if(key instanceof BytesWritable) { + mapredKey_ = new String(((BytesWritable)key).get(), "UTF-8"); + } else { + mapredKey_ = key.toString(); + } numRecRead_++; maybeLogRecord(); // 2/4 Hadoop to Tool if(numExceptions_==0) { - clientOut_.writeBytes(mapredKey_); - clientOut_.writeBytes("\t"); - clientOut_.writeBytes(value.toString()); + String sval; + if(value instanceof BytesWritable) { + sval = new String(((BytesWritable)value).get(), "UTF-8"); + } else { + sval = value.toString(); + } + if(optUseKey_) { + clientOut_.writeBytes(mapredKey_); + clientOut_.writeBytes("\t"); + } + clientOut_.writeBytes(sval); clientOut_.writeBytes("\n"); clientOut_.flush(); } else { @@ -90,11 +112,10 @@ } } - public void close() { appendLogToJobLog("success"); mapRedFinished(); } - + } Modified: lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamBaseRecordReader.java URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamBaseRecordReader.java?rev=420768&r1=420767&r2=420768&view=diff ============================================================================== --- lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamBaseRecordReader.java (original) +++ lucene/hadoop/trunk/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamBaseRecordReader.java Tue Jul 11 01:58:05 2006 @@ -20,6 +20,7 @@ import org.apache.hadoop.io.Writable; import org.apache.hadoop.io.WritableComparable; +import org.apache.hadoop.fs.Path; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.mapred.Reporter; @@ -46,17 +47,18 @@ final String CONF_NS = "stream.recordreader."; public StreamBaseRecordReader( - FSDataInputStream in, long start, long end, - String splitName, Reporter reporter, JobConf job) + FSDataInputStream in, FileSplit split, Reporter reporter, JobConf job, FileSystem fs) throws IOException {
{'source': 'Apache Software Foundation Public Mail Archives'}