From common-commits-returnelliottmatthew@example.net Tue Sep 01 02:50:49 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 86831 invoked from network); 1 Sep 2009 02:50:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 02:50:49 -0000 Received: (qmail 10725 invoked by uid 500); 1 Sep 2009 02:50:49 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 10645 invoked by uid 500); 1 Sep 2009 02:50:49 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 10636 invoked by uid 500); 1 Sep 2009 02:50:49 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 10633 invoked by uid 99); 1 Sep 2009 02:50:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 02:50:49 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 02:50:38 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id 18389118AA for <elliottmatthew@example.net>; Tue, 1 Sep 2009 02:50:17 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <elliottmatthew@example.net> To: elliottmatthew@example.net Date: Tue, 01 Sep 2009 02:50:16 -0000 Message-ID: <elliottmatthew@example.net> Subject: [Hadoop Wiki] Trivial Update of "BristolHadoopWorkshop" by edwardyoon X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification. The following page has been changed by edwardyoon: http://wiki.apache.org/hadoop/BristolHadoopWorkshop ------------------------------------------------------------------------------ MS: dryad does not do graphs. Projects - * Apache Hamburg -proposed by Edward Yang, of Hamas, is it making progress? We need code. + * Apache Hamburg -proposed by Edward Yoon, of Hamas, is it making progress? We need code. + * Thread based [http://throb.googlecode.com/svn/trunk/src/java/org/apache/hadoop/hamburg/ prototype code of Hamburg] * Apache Common Graph. Dead, pre-MapReduce code. === Graph Algorithms === From common-commits-returnelliottmatthew@example.net Tue Sep 01 06:10:57 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 35521 invoked from network); 1 Sep 2009 06:10:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 06:10:56 -0000 Received: (qmail 8108 invoked by uid 500); 1 Sep 2009 06:10:56 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 8020 invoked by uid 500); 1 Sep 2009 06:10:56 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 8011 invoked by uid 99); 1 Sep 2009 06:10:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 06:10:56 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 06:10:54 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 8376523888DC; Tue, 1 Sep 2009 06:10:34 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r809840 - in /hadoop/common/branches/branch-0.20: CHANGES.txt src/test/org/apache/hadoop/mapred/TestTaskFail.java Date: Tue, 01 Sep 2009 06:10:34 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: sharad Date: Tue Sep 1 06:10:33 2009 New Revision: 809840 URL: http://svn.apache.org/viewvc?rev=809840&view=rev Log: MAPREDUCE-911. Fix a bug in TestTaskFail related to speculative execution. Contributed by Amareshwari Sriramadasu. Modified: hadoop/common/branches/branch-0.20/CHANGES.txt hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestTaskFail.java Modified: hadoop/common/branches/branch-0.20/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/CHANGES.txt?rev=809840&r1=809839&r2=809840&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20/CHANGES.txt (original) +++ hadoop/common/branches/branch-0.20/CHANGES.txt Tue Sep 1 06:10:33 2009 @@ -247,6 +247,9 @@ HDFS-525. The SimpleDateFormat object in ListPathsServlet is not thread safe. (Suresh Srinivas and cdouglas) + MAPREDUCE-911. Fix a bug in TestTaskFail related to speculative + execution. (Amareshwari Sriramadasu via sharad) + Release 0.20.0 - 2009-04-15 INCOMPATIBLE CHANGES Modified: hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestTaskFail.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestTaskFail.java?rev=809840&r1=809839&r2=809840&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestTaskFail.java (original) +++ hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestTaskFail.java Tue Sep 1 06:10:33 2009 @@ -101,6 +101,7 @@ conf.setNumReduceTasks(0); FileInputFormat.setInputPaths(conf, inDir); FileOutputFormat.setOutputPath(conf, outDir); + conf.setSpeculativeExecution(false); String TEST_ROOT_DIR = new Path(System.getProperty("test.build.data", "/tmp")).toString().replace(' ', '+'); conf.set("test.build.data", TEST_ROOT_DIR); From common-commits-returnelliottmatthew@example.net Tue Sep 01 06:33:39 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 59969 invoked from network); 1 Sep 2009 06:33:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 06:33:39 -0000 Received: (qmail 31672 invoked by uid 500); 1 Sep 2009 06:33:39 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 31591 invoked by uid 500); 1 Sep 2009 06:33:38 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 31582 invoked by uid 99); 1 Sep 2009 06:33:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 06:33:38 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 06:33:35 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id CC7B623888DC; Tue, 1 Sep 2009 06:33:14 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r809846 - in /hadoop/common/branches/branch-0.20: CHANGES.txt src/test/org/apache/hadoop/mapred/TestMiniMRMapRedDebugScript.java Date: Tue, 01 Sep 2009 06:33:14 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: sharad Date: Tue Sep 1 06:33:14 2009 New Revision: 809846 URL: http://svn.apache.org/viewvc?rev=809846&view=rev Log: MAPREDUCE-687. Fix an assertion in TestMiniMRMapRedDebugScript. Contributed by Amareshwari Sriramadasu. Modified: hadoop/common/branches/branch-0.20/CHANGES.txt hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestMiniMRMapRedDebugScript.java Modified: hadoop/common/branches/branch-0.20/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/CHANGES.txt?rev=809846&r1=809845&r2=809846&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20/CHANGES.txt (original) +++ hadoop/common/branches/branch-0.20/CHANGES.txt Tue Sep 1 06:33:14 2009 @@ -250,6 +250,9 @@ MAPREDUCE-911. Fix a bug in TestTaskFail related to speculative execution. (Amareshwari Sriramadasu via sharad) + MAPREDUCE-687. Fix an assertion in TestMiniMRMapRedDebugScript. + (Amareshwari Sriramadasu via sharad) + Release 0.20.0 - 2009-04-15 INCOMPATIBLE CHANGES Modified: hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestMiniMRMapRedDebugScript.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestMiniMRMapRedDebugScript.java?rev=809846&r1=809845&r2=809846&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestMiniMRMapRedDebugScript.java (original) +++ hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/TestMiniMRMapRedDebugScript.java Tue Sep 1 06:33:14 2009 @@ -207,7 +207,7 @@ outDir,debugDir, debugScript, input); // Assert the output of debug script. - assertEquals("Test Script\nBailing out", result); + assertTrue(result.contains("Test Script\nBailing out")); } finally { // close file system and shut down dfs and mapred cluster From common-commits-returnelliottmatthew@example.net Tue Sep 01 15:25:23 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 12870 invoked from network); 1 Sep 2009 15:25:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 15:25:23 -0000 Received: (qmail 28784 invoked by uid 500); 1 Sep 2009 15:25:22 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 28713 invoked by uid 500); 1 Sep 2009 15:25:22 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 28704 invoked by uid 99); 1 Sep 2009 15:25:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 15:25:22 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 15:25:19 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id C7FEE23888DB; Tue, 1 Sep 2009 15:24:57 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810097 - in /hadoop/common/trunk: CHANGES.txt src/java/org/apache/hadoop/conf/Configuration.java src/test/core/org/apache/hadoop/conf/TestConfiguration.java Date: Tue, 01 Sep 2009 15:24:57 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: yhemanth Date: Tue Sep 1 15:24:56 2009 New Revision: 810097 URL: http://svn.apache.org/viewvc?rev=810097&view=rev Log: HADOOP-6227. Fix Configuration to allow final parameters to be set to null and prevent them from being overridden. Contributed by Amareshwari Sriramadasu. Modified: hadoop/common/trunk/CHANGES.txt hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfiguration.java Modified: hadoop/common/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/CHANGES.txt?rev=810097&r1=810096&r2=810097&view=diff ============================================================================== --- hadoop/common/trunk/CHANGES.txt (original) +++ hadoop/common/trunk/CHANGES.txt Tue Sep 1 15:24:56 2009 @@ -951,6 +951,10 @@ HADOOP-6215. fix GenericOptionParser to deal with -D with '=' in the value. (Amar Kamat via sharad) + HADOOP-6227. Fix Configuration to allow final parameters to be set to null + and prevent them from being overridden. + (Amareshwari Sriramadasu via yhemanth) + Release 0.20.1 - Unreleased INCOMPATIBLE CHANGES Modified: hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java?rev=810097&r1=810096&r2=810097&view=diff ============================================================================== --- hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java (original) +++ hadoop/common/trunk/src/java/org/apache/hadoop/conf/Configuration.java Tue Sep 1 15:24:56 2009 @@ -1286,17 +1286,20 @@ } // Ignore this parameter if it has already been marked as 'final' - if (attr != null && value != null) { - if (!finalParameters.contains(attr)) { - properties.setProperty(attr, value); - if (storeResource) { - updatingResource.put(attr, name.toString()); - } - if (finalParameter) - finalParameters.add(attr); - } else { - LOG.warn(name+":a attempt to override final parameter: "+attr + if (attr != null) { + if (value != null) { + if (!finalParameters.contains(attr)) { + properties.setProperty(attr, value); + if (storeResource) { + updatingResource.put(attr, name.toString()); + } + } else { + LOG.warn(name+":a attempt to override final parameter: "+attr +"; Ignoring."); + } + } + if (finalParameter) { + finalParameters.add(attr); } } } Modified: hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfiguration.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfiguration.java?rev=810097&r1=810096&r2=810097&view=diff ============================================================================== --- hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfiguration.java (original) +++ hadoop/common/trunk/src/test/core/org/apache/hadoop/conf/TestConfiguration.java Tue Sep 1 15:24:56 2009 @@ -100,7 +100,28 @@ assertTrue(conf.getInt("intvar", -1) == 42); assertTrue(conf.getInt("my.int", -1) == 42); } - + + public void testFinalParam() throws IOException { + out=new BufferedWriter(new FileWriter(CONFIG)); + startConfig(); + declareProperty("my.var", "", "", true); + endConfig(); + Path fileResource = new Path(CONFIG); + Configuration conf1 = new Configuration(); + conf1.addResource(fileResource); + assertNull("my var is not null", conf1.get("my.var")); + + out=new BufferedWriter(new FileWriter(CONFIG2)); + startConfig(); + declareProperty("my.var", "myval", "myval", false); + endConfig(); + fileResource = new Path(CONFIG2); + + Configuration conf2 = new Configuration(conf1); + conf2.addResource(fileResource); + assertNull("my var is not final", conf2.get("my.var")); + } + public static void assertEq(Object a, Object b) { System.out.println("assertEq: " + a + ", " + b); assertEquals(a, b); From common-commits-returnelliottmatthew@example.net Tue Sep 01 15:39:28 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 18589 invoked from network); 1 Sep 2009 15:39:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 15:39:27 -0000 Received: (qmail 52880 invoked by uid 500); 1 Sep 2009 15:39:27 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 52820 invoked by uid 500); 1 Sep 2009 15:39:27 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 52811 invoked by uid 500); 1 Sep 2009 15:39:27 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 52808 invoked by uid 99); 1 Sep 2009 15:39:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 15:39:27 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 15:39:17 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id EAA52112A9 for <elliottmatthew@example.net>; Tue, 1 Sep 2009 15:38:56 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <elliottmatthew@example.net> To: elliottmatthew@example.net Date: Tue, 01 Sep 2009 15:38:56 -0000 Message-ID: <elliottmatthew@example.net> Subject: [Hadoop Wiki] Trivial Update of "BristolHadoopWorkshop" by SteveLoughran X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification. The following page has been changed by SteveLoughran: http://wiki.apache.org/hadoop/BristolHadoopWorkshop The comment on the change is: fix number list formatting ------------------------------------------------------------------------------ The talk looked at the options for long-haul-ness, of which there are two - WS-* : the big, comfortable, safe long-haul option, the Airbus A380. + 1. WS-* : the big, comfortable, safe long-haul option, the Airbus A380. You, the passenger, get looked after by the cabin crew. - The floatplane. Agile, can get around fast, but you read the location of + 1. The floatplane. Agile, can get around fast, but you read the location of the life vest instructions very carefully, make a note of the exit in the roof and hope that you aren't the one who has to get on the float to dock the plane with the boat. It isn't quite as comfy as the big plane, @@ -233, +233 @@ Projects * Apache Hamburg -proposed by Edward Yoon, of Hamas, is it making progress? We need code. - * Thread based [http://throb.googlecode.com/svn/trunk/src/java/org/apache/hadoop/hamburg/ prototype code of Hamburg] + * Thread based [http://throb.googlecode.com/svn/trunk/src/java/org/apache/hadoop/hamburg/ prototype code of Hamburg] * Apache Common Graph. Dead, pre-MapReduce code. === Graph Algorithms === @@ -271, +271 @@ * Y! are making Hadoop the core of the company; it is their LOB of datacentre. What are the risks of the Merger, and warning signs of trouble: - # silence: Y! developers do their own fork, it goes closed source. We have seen this happen in other OSS projects (Axis), where a single company suddenly disappears. There is no defence from this other than making sure development knowledge is widespread. The JIRA-based discussion/documentation is good here, + 1. silence: Y! developers do their own fork, it goes closed source. We have seen this happen in other OSS projects (Axis), where a single company suddenly disappears. There is no defence from this other than making sure development knowledge is widespread. The JIRA-based discussion/documentation is good here, as it preserves all knowledge, and makes decisions in the open. - - # staff departure. Key staff in the Hadoop team could leave, which would set things back. Moving into MS could be bad, but moving to Google would set back development the worst. + 1. staff departure. Key staff in the Hadoop team could leave, which would set things back. Moving into MS could be bad, but moving to Google would set back development the worst. - # slower development/rate of feature addition + 1. slower development/rate of feature addition - # reduced release rate. This can compensate for reduced testing resources. + 1. reduced release rate. This can compensate for reduced testing resources. - # reduced rate of bug fixes. We can assume that Y!s own problems will be addressed, then everything else is other people's problems. + 1. reduced rate of bug fixes. We can assume that Y!s own problems will be addressed, then everything else is other people's problems. - # Less testing, reduced quality + 1. Less testing, reduced quality Apparently under [http://community.cloudera.com] - number of messages/JIRA and infer activity, such as [http://community.cloudera.com/reports/47/contributors/ contributors] and [http://community.cloudera.com/reports/47/issues/ popular issues] With Yahoo! outsourcing searching to MS, it means that MS can take on a big project that -even if it isn't profitable to MS, can be subsidised by other parts of their business. It ensures Yahoo! continuing survival as an independent company, which is the best state for Hadoop development. It also frees up some Yahoo! datacentres for other projects. Those big datacentres are large, slowly-depreciating assets, and by offloading the indexing to someone else, there is now spare datacentre capacity for Yahoo! to use for other uses, uses that are highly likely to use Hadoop at the back end -because what else would they build a datacentre-scale application on? From common-commits-returnelliottmatthew@example.net Tue Sep 01 17:30:42 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 56740 invoked from network); 1 Sep 2009 17:30:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 17:30:42 -0000 Received: (qmail 16270 invoked by uid 500); 1 Sep 2009 17:30:42 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 16214 invoked by uid 500); 1 Sep 2009 17:30:42 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 16205 invoked by uid 99); 1 Sep 2009 17:30:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 17:30:42 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 17:30:40 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 74F512388892; Tue, 1 Sep 2009 17:30:20 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810136 - in /hadoop/common/branches/branch-0.20: CHANGES.txt src/test/org/apache/hadoop/mapred/pipes/TestPipes.java Date: Tue, 01 Sep 2009 17:30:20 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: ddas Date: Tue Sep 1 17:30:18 2009 New Revision: 810136 URL: http://svn.apache.org/viewvc?rev=810136&view=rev Log: MAPREDUCE-924. Fixes the TestPipes testcase to use Tool. Contributed by Amareshwari Sriramadasu. Modified: hadoop/common/branches/branch-0.20/CHANGES.txt hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/pipes/TestPipes.java Modified: hadoop/common/branches/branch-0.20/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/CHANGES.txt?rev=810136&r1=810135&r2=810136&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20/CHANGES.txt (original) +++ hadoop/common/branches/branch-0.20/CHANGES.txt Tue Sep 1 17:30:18 2009 @@ -253,6 +253,9 @@ MAPREDUCE-687. Fix an assertion in TestMiniMRMapRedDebugScript. (Amareshwari Sriramadasu via sharad) + MAPREDUCE-924. Fixes the TestPipes testcase to use Tool. + (Amareshwari Sriramadasu via sharad) + Release 0.20.0 - 2009-04-15 INCOMPATIBLE CHANGES Modified: hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/pipes/TestPipes.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/pipes/TestPipes.java?rev=810136&r1=810135&r2=810136&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/pipes/TestPipes.java (original) +++ hadoop/common/branches/branch-0.20/src/test/org/apache/hadoop/mapred/pipes/TestPipes.java Tue Sep 1 17:30:18 2009 @@ -40,6 +40,7 @@ import org.apache.hadoop.mapred.TestMiniMRWithDFS; import org.apache.hadoop.mapred.Counters.Counter; import org.apache.hadoop.util.StringUtils; +import org.apache.hadoop.util.ToolRunner; import junit.framework.TestCase; @@ -238,12 +239,14 @@ " -program " + dfs.getFileSystem().makeQualified(wordExec)); try { - Submitter.main(new String[]{"-conf", jobXml.toString(), + int ret = ToolRunner.run(new Submitter(), + new String[]{"-conf", jobXml.toString(), "-input", inDir.toString(), "-output", outDir.toString(), "-program", dfs.getFileSystem().makeQualified(wordExec).toString(), "-reduces", "2"}); + assertEquals(0, ret); } catch (Exception e) { assertTrue("got exception: " + StringUtils.stringifyException(e), false); } From common-commits-returnelliottmatthew@example.net Tue Sep 01 20:20:30 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 8453 invoked from network); 1 Sep 2009 20:20:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 20:20:30 -0000 Received: (qmail 15998 invoked by uid 500); 1 Sep 2009 20:20:30 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 15918 invoked by uid 500); 1 Sep 2009 20:20:30 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 15909 invoked by uid 99); 1 Sep 2009 20:20:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:20:30 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:20:27 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 721F42388896; Tue, 1 Sep 2009 20:20:06 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810213 - /hadoop/common/tags/release-0.20.1-rc1/ Date: Tue, 01 Sep 2009 20:20:06 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: omalley Date: Tue Sep 1 20:20:06 2009 New Revision: 810213 URL: http://svn.apache.org/viewvc?rev=810213&view=rev Log: Preparing for release 0.20.1-rc1 Added: hadoop/common/tags/release-0.20.1-rc1/ (props changed) - copied from r810212, hadoop/common/branches/branch-0.20/ Propchange: hadoop/common/tags/release-0.20.1-rc1/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Tue Sep 1 20:20:06 2009 @@ -0,0 +1,7 @@ +build +logs +.classpath +.project +.settings + +.externalToolBuilders Propchange: hadoop/common/tags/release-0.20.1-rc1/ ------------------------------------------------------------------------------ --- svn:mergeinfo (added) +++ svn:mergeinfo Tue Sep 1 20:20:06 2009 @@ -0,0 +1,2 @@ +/hadoop/core/branches/branch-0.19:713112 +/hadoop/core/trunk:727001,727117,727191,727212,727217,727228,727255,727869,728187,729052,729987,732385,732572,732613,732777,732838,732869,733887,734870,734916,736426,738328,738697,740077,740157,741703,741762,743745,743816,743892,744894,745180,746010,746206,746227,746233,746274,746338,746902-746903,746925,746944,746968,746970,747279,747289,747802,748084,748090,748783,749262,749318,749863,750533,752073,752609,752834,752836,752913,752932,753112-753113,753346,754645,754847,754927,755035,755226,755348,755370,755418,755426,755790,755905,755938,755960,755986,755998,756352,757448,757624,757849,758156,758180,759398,759932,760502,760783,761046,761482,761632,762216,762879,763107,763502,764967,765016,765809,765951,771607,771661,772844,772876,772884,772920,773889,776638,778962,778966,779893,781720,784661,785046,785569 From common-commits-returnelliottmatthew@example.net Tue Sep 01 20:30:17 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 10610 invoked from network); 1 Sep 2009 20:30:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 20:30:17 -0000 Received: (qmail 29546 invoked by uid 500); 1 Sep 2009 20:30:16 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 29505 invoked by uid 500); 1 Sep 2009 20:30:16 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 29455 invoked by uid 99); 1 Sep 2009 20:30:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:30:16 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:30:14 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 560A32388896; Tue, 1 Sep 2009 20:29:54 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810214 - /hadoop/common/tags/release-0.20.1-rc1/ Date: Tue, 01 Sep 2009 20:29:54 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: omalley Date: Tue Sep 1 20:29:53 2009 New Revision: 810214 URL: http://svn.apache.org/viewvc?rev=810214&view=rev Log: Forgot to update the release notes Removed: hadoop/common/tags/release-0.20.1-rc1/ From common-commits-returnelliottmatthew@example.net Tue Sep 01 20:31:09 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 10733 invoked from network); 1 Sep 2009 20:31:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 20:31:09 -0000 Received: (qmail 30284 invoked by uid 500); 1 Sep 2009 20:31:08 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 30207 invoked by uid 500); 1 Sep 2009 20:31:08 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 30198 invoked by uid 99); 1 Sep 2009 20:31:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:31:08 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:31:07 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id EEED42388896; Tue, 1 Sep 2009 20:30:46 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810215 - /hadoop/common/branches/branch-0.20/src/docs/releasenotes.html Date: Tue, 01 Sep 2009 20:30:46 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: omalley Date: Tue Sep 1 20:30:46 2009 New Revision: 810215 URL: http://svn.apache.org/viewvc?rev=810215&view=rev Log: Added MR-687 and MR-924 to the release notes Modified: hadoop/common/branches/branch-0.20/src/docs/releasenotes.html Modified: hadoop/common/branches/branch-0.20/src/docs/releasenotes.html URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/src/docs/releasenotes.html?rev=810215&r1=810214&r2=810215&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20/src/docs/releasenotes.html (original) +++ hadoop/common/branches/branch-0.20/src/docs/releasenotes.html Tue Sep 1 20:30:46 2009 @@ -171,6 +171,8 @@ </li> <li>[<a href='https://issues.apache.org/jira/browse/MAPREDUCE-657'>MAPREDUCE-657</a>] - CompletedJobStatusStore hardcodes filesystem to hdfs </li> +<li>[<a href='https://issues.apache.org/jira/browse/MAPREDUCE-687'>MAPREDUCE-687</a>] - TestMiniMRMapRedDebugScript fails sometimes +</li> <li>[<a href='https://issues.apache.org/jira/browse/MAPREDUCE-735'>MAPREDUCE-735</a>] - ArrayIndexOutOfBoundsException is thrown by KeyFieldBasedPartitioner </li> <li>[<a href='https://issues.apache.org/jira/browse/MAPREDUCE-745'>MAPREDUCE-745</a>] - TestRecoveryManager fails sometimes @@ -195,6 +197,8 @@ </li> <li>[<a href='https://issues.apache.org/jira/browse/MAPREDUCE-911'>MAPREDUCE-911</a>] - TestTaskFail fail sometimes </li> +<li>[<a href='https://issues.apache.org/jira/browse/MAPREDUCE-924'>MAPREDUCE-924</a>] - TestPipes crashes on trunk +</li> </ul> <h4> Improvement From common-commits-returnelliottmatthew@example.net Tue Sep 01 20:31:45 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 10834 invoked from network); 1 Sep 2009 20:31:45 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 20:31:45 -0000 Received: (qmail 30674 invoked by uid 500); 1 Sep 2009 20:31:45 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 30603 invoked by uid 500); 1 Sep 2009 20:31:44 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 30594 invoked by uid 99); 1 Sep 2009 20:31:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:31:44 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:31:42 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 409572388896; Tue, 1 Sep 2009 20:31:21 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810216 - /hadoop/common/tags/release-0.20.1-rc1/ Date: Tue, 01 Sep 2009 20:31:21 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: omalley Date: Tue Sep 1 20:31:20 2009 New Revision: 810216 URL: http://svn.apache.org/viewvc?rev=810216&view=rev Log: Retag the new release canidate Added: hadoop/common/tags/release-0.20.1-rc1/ (props changed) - copied from r810215, hadoop/common/branches/branch-0.20/ Propchange: hadoop/common/tags/release-0.20.1-rc1/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Tue Sep 1 20:31:20 2009 @@ -0,0 +1,7 @@ +build +logs +.classpath +.project +.settings + +.externalToolBuilders Propchange: hadoop/common/tags/release-0.20.1-rc1/ ------------------------------------------------------------------------------ --- svn:mergeinfo (added) +++ svn:mergeinfo Tue Sep 1 20:31:20 2009 @@ -0,0 +1,2 @@ +/hadoop/core/branches/branch-0.19:713112 +/hadoop/core/trunk:727001,727117,727191,727212,727217,727228,727255,727869,728187,729052,729987,732385,732572,732613,732777,732838,732869,733887,734870,734916,736426,738328,738697,740077,740157,741703,741762,743745,743816,743892,744894,745180,746010,746206,746227,746233,746274,746338,746902-746903,746925,746944,746968,746970,747279,747289,747802,748084,748090,748783,749262,749318,749863,750533,752073,752609,752834,752836,752913,752932,753112-753113,753346,754645,754847,754927,755035,755226,755348,755370,755418,755426,755790,755905,755938,755960,755986,755998,756352,757448,757624,757849,758156,758180,759398,759932,760502,760783,761046,761482,761632,762216,762879,763107,763502,764967,765016,765809,765951,771607,771661,772844,772876,772884,772920,773889,776638,778962,778966,779893,781720,784661,785046,785569 From common-commits-returnelliottmatthew@example.net Tue Sep 01 20:33:13 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 11185 invoked from network); 1 Sep 2009 20:33:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 20:33:13 -0000 Received: (qmail 31979 invoked by uid 500); 1 Sep 2009 20:33:13 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 31910 invoked by uid 500); 1 Sep 2009 20:33:13 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 31901 invoked by uid 99); 1 Sep 2009 20:33:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:33:13 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:33:11 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 690532388896; Tue, 1 Sep 2009 20:32:51 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810218 - /hadoop/common/tags/release-0.20.1-rc1/ Date: Tue, 01 Sep 2009 20:32:51 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: omalley Date: Tue Sep 1 20:32:51 2009 New Revision: 810218 URL: http://svn.apache.org/viewvc?rev=810218&view=rev Log: Untag the new release canidate Removed: hadoop/common/tags/release-0.20.1-rc1/ From common-commits-returnelliottmatthew@example.net Tue Sep 01 20:33:40 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 11315 invoked from network); 1 Sep 2009 20:33:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 20:33:40 -0000 Received: (qmail 33624 invoked by uid 500); 1 Sep 2009 20:33:40 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 33581 invoked by uid 500); 1 Sep 2009 20:33:40 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 33572 invoked by uid 99); 1 Sep 2009 20:33:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:33:40 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:33:38 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id BC4C523888E6; Tue, 1 Sep 2009 20:33:18 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810219 - /hadoop/common/branches/branch-0.20/CHANGES.txt Date: Tue, 01 Sep 2009 20:33:18 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: omalley Date: Tue Sep 1 20:33:18 2009 New Revision: 810219 URL: http://svn.apache.org/viewvc?rev=810219&view=rev Log: Update the CHANGES file with today's date for 0.20.1 Modified: hadoop/common/branches/branch-0.20/CHANGES.txt Modified: hadoop/common/branches/branch-0.20/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/CHANGES.txt?rev=810219&r1=810218&r2=810219&view=diff ============================================================================== --- hadoop/common/branches/branch-0.20/CHANGES.txt (original) +++ hadoop/common/branches/branch-0.20/CHANGES.txt Tue Sep 1 20:33:18 2009 @@ -1,6 +1,6 @@ Hadoop Change Log -Release 0.20.1 - 2009-08-27 +Release 0.20.1 - 2009-09-01 INCOMPATIBLE CHANGES From common-commits-returnelliottmatthew@example.net Tue Sep 01 20:34:01 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 11495 invoked from network); 1 Sep 2009 20:34:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Sep 2009 20:34:00 -0000 Received: (qmail 34646 invoked by uid 500); 1 Sep 2009 20:34:00 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 34585 invoked by uid 500); 1 Sep 2009 20:34:00 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 34576 invoked by uid 99); 1 Sep 2009 20:34:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:34:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Sep 2009 20:33:57 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id C67A323888E9; Tue, 1 Sep 2009 20:33:36 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810220 - /hadoop/common/tags/release-0.20.1-rc1/ Date: Tue, 01 Sep 2009 20:33:36 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: omalley Date: Tue Sep 1 20:33:36 2009 New Revision: 810220 URL: http://svn.apache.org/viewvc?rev=810220&view=rev Log: Tagging 0.20.1-rc1 Added: hadoop/common/tags/release-0.20.1-rc1/ (props changed) - copied from r810219, hadoop/common/branches/branch-0.20/ Propchange: hadoop/common/tags/release-0.20.1-rc1/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Tue Sep 1 20:33:36 2009 @@ -0,0 +1,7 @@ +build +logs +.classpath +.project +.settings + +.externalToolBuilders Propchange: hadoop/common/tags/release-0.20.1-rc1/ ------------------------------------------------------------------------------ --- svn:mergeinfo (added) +++ svn:mergeinfo Tue Sep 1 20:33:36 2009 @@ -0,0 +1,2 @@ +/hadoop/core/branches/branch-0.19:713112 +/hadoop/core/trunk:727001,727117,727191,727212,727217,727228,727255,727869,728187,729052,729987,732385,732572,732613,732777,732838,732869,733887,734870,734916,736426,738328,738697,740077,740157,741703,741762,743745,743816,743892,744894,745180,746010,746206,746227,746233,746274,746338,746902-746903,746925,746944,746968,746970,747279,747289,747802,748084,748090,748783,749262,749318,749863,750533,752073,752609,752834,752836,752913,752932,753112-753113,753346,754645,754847,754927,755035,755226,755348,755370,755418,755426,755790,755905,755938,755960,755986,755998,756352,757448,757624,757849,758156,758180,759398,759932,760502,760783,761046,761482,761632,762216,762879,763107,763502,764967,765016,765809,765951,771607,771661,772844,772876,772884,772920,773889,776638,778962,778966,779893,781720,784661,785046,785569 From common-commits-returnelliottmatthew@example.net Wed Sep 02 01:00:59 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 87990 invoked from network); 2 Sep 2009 01:00:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Sep 2009 01:00:59 -0000 Received: (qmail 93438 invoked by uid 500); 2 Sep 2009 01:00:59 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 93366 invoked by uid 500); 2 Sep 2009 01:00:59 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 93357 invoked by uid 99); 2 Sep 2009 01:00:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 01:00:59 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 01:00:55 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id DA4AA238887A; Wed, 2 Sep 2009 01:00:34 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810325 - in /hadoop/common/trunk: CHANGES.txt src/java/org/apache/hadoop/io/DataOutputBuffer.java src/java/org/apache/hadoop/io/Text.java Date: Wed, 02 Sep 2009 01:00:34 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: cdouglas Date: Wed Sep 2 01:00:34 2009 New Revision: 810325 URL: http://svn.apache.org/viewvc?rev=810325&view=rev Log: Revert HADOOP-6224 Modified: hadoop/common/trunk/CHANGES.txt hadoop/common/trunk/src/java/org/apache/hadoop/io/DataOutputBuffer.java hadoop/common/trunk/src/java/org/apache/hadoop/io/Text.java Modified: hadoop/common/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/CHANGES.txt?rev=810325&r1=810324&r2=810325&view=diff ============================================================================== --- hadoop/common/trunk/CHANGES.txt (original) +++ hadoop/common/trunk/CHANGES.txt Wed Sep 2 01:00:34 2009 @@ -507,10 +507,6 @@ HADOOP-6184. Provide an API to dump Configuration in a JSON format. (V.V.Chaitanya Krishna via yhemanth) - - HADOOP-6224. Adds methods to read strings safely, makes the Buffer class - in DataOutputBuffer public, and introduces public constructors there. These changes - are required for MAPREDUCE-318. (Jothi Padmanabhan and Arun Murthy via ddas) OPTIMIZATIONS Modified: hadoop/common/trunk/src/java/org/apache/hadoop/io/DataOutputBuffer.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/io/DataOutputBuffer.java?rev=810325&r1=810324&r2=810325&view=diff ============================================================================== --- hadoop/common/trunk/src/java/org/apache/hadoop/io/DataOutputBuffer.java (original) +++ hadoop/common/trunk/src/java/org/apache/hadoop/io/DataOutputBuffer.java Wed Sep 2 01:00:34 2009 @@ -20,8 +20,6 @@ import java.io.*; -import org.apache.hadoop.io.DataOutputBuffer.Buffer; - /** A reusable {@link DataOutput} implementation that writes to an in-memory * buffer. * @@ -43,7 +41,7 @@ */ public class DataOutputBuffer extends DataOutputStream { - public static class Buffer extends ByteArrayOutputStream { + private static class Buffer extends ByteArrayOutputStream { public byte[] getData() { return buf; } public int getLength() { return count; } @@ -55,10 +53,6 @@ super(size); } - public Buffer(byte[] buf) { - super.buf = buf; - } - public void write(DataInput in, int len) throws IOException { int newcount = count + len; if (newcount > buf.length) { @@ -82,10 +76,6 @@ this(new Buffer(size)); } - public DataOutputBuffer(byte[] buf) { - this(new Buffer(buf)); - } - private DataOutputBuffer(Buffer buffer) { super(buffer); this.buffer = buffer; Modified: hadoop/common/trunk/src/java/org/apache/hadoop/io/Text.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/io/Text.java?rev=810325&r1=810324&r2=810325&view=diff ============================================================================== --- hadoop/common/trunk/src/java/org/apache/hadoop/io/Text.java (original) +++ hadoop/common/trunk/src/java/org/apache/hadoop/io/Text.java Wed Sep 2 01:00:34 2009 @@ -35,7 +35,6 @@ import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; -import org.apache.hadoop.io.WritableUtils; /** This class stores text using standard UTF8 encoding. It provides methods * to serialize, deserialize, and compare texts at byte level. The type of @@ -404,30 +403,6 @@ in.readFully(bytes, 0, length); return decode(bytes); } - /** - * Read a string, but check it for sanity. The format consists of a vint - * followed by the given number of bytes. - * @param in the stream to read from - * @param maxLength the largest acceptable length of string - * @return the bytes as a string - * @throws IOException if reading from the DataInput fails - * @throws IllegalArgumentException if the string length is negative or - * larger than maxSize. Only the vint is read. - */ - public static String readStringSafely(DataInput in, - int maxLength - ) throws IOException, - IllegalArgumentException { - int length = WritableUtils.readVInt(in); - if (length < 0 || length > maxLength) { - throw new IllegalArgumentException("String size was " + length + - ", which is outside of 0.." + - maxLength); - } - byte [] bytes = new byte[length]; - in.readFully(bytes, 0, length); - return decode(bytes); - } /** Write a UTF8 encoded string to out */ From common-commits-returnelliottmatthew@example.net Wed Sep 02 01:27:03 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 91898 invoked from network); 2 Sep 2009 01:27:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Sep 2009 01:27:03 -0000 Received: (qmail 8696 invoked by uid 500); 2 Sep 2009 01:27:03 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 8617 invoked by uid 500); 2 Sep 2009 01:27:02 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 8608 invoked by uid 99); 2 Sep 2009 01:27:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 01:27:02 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 01:27:01 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id F33B7238887A; Wed, 2 Sep 2009 01:26:40 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810329 - in /hadoop/common/trunk: CHANGES.txt src/java/core-default.xml Date: Wed, 02 Sep 2009 01:26:40 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: cdouglas Date: Wed Sep 2 01:26:40 2009 New Revision: 810329 URL: http://svn.apache.org/viewvc?rev=810329&view=rev Log: HADOOP-6199. Move io.map.skip.index property to core-default from mapred. Contributed by Amareshwari Sriramadasu Modified: hadoop/common/trunk/CHANGES.txt hadoop/common/trunk/src/java/core-default.xml Modified: hadoop/common/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/CHANGES.txt?rev=810329&r1=810328&r2=810329&view=diff ============================================================================== --- hadoop/common/trunk/CHANGES.txt (original) +++ hadoop/common/trunk/CHANGES.txt Wed Sep 2 01:26:40 2009 @@ -951,6 +951,9 @@ and prevent them from being overridden. (Amareshwari Sriramadasu via yhemanth) + HADOOP-6199. Move io.map.skip.index property to core-default from mapred. + (Amareshwari Sriramadasu via cdouglas) + Release 0.20.1 - Unreleased INCOMPATIBLE CHANGES Modified: hadoop/common/trunk/src/java/core-default.xml URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/core-default.xml?rev=810329&r1=810328&r2=810329&view=diff ============================================================================== --- hadoop/common/trunk/src/java/core-default.xml (original) +++ hadoop/common/trunk/src/java/core-default.xml Wed Sep 2 01:26:40 2009 @@ -116,6 +116,14 @@ </description> </property> +<property> + <name>io.map.index.skip</name> + <value>0</value> + <description>Number of index entries to skip between each entry. + Zero by default. Setting this to values larger than zero can + facilitate opening large MapFiles using less memory.</description> +</property> + <!-- file system properties --> <property> From common-commits-returnelliottmatthew@example.net Wed Sep 02 07:07:44 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 60605 invoked from network); 2 Sep 2009 07:07:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Sep 2009 07:07:44 -0000 Received: (qmail 46351 invoked by uid 500); 2 Sep 2009 07:07:44 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 46274 invoked by uid 500); 2 Sep 2009 07:07:43 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 46265 invoked by uid 99); 2 Sep 2009 07:07:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 07:07:43 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 07:07:42 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id A92E923888AD; Wed, 2 Sep 2009 07:07:21 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810384 - in /hadoop/common/trunk: CHANGES.txt src/java/org/apache/hadoop/io/WritableUtils.java Date: Wed, 02 Sep 2009 07:07:21 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: cdouglas Date: Wed Sep 2 07:07:21 2009 New Revision: 810384 URL: http://svn.apache.org/viewvc?rev=810384&view=rev Log: HADOOP-6224. Add a method to WritableUtils supported a bounded read of an encoded String. Contributed by Jothi Padmanabhan Modified: hadoop/common/trunk/CHANGES.txt hadoop/common/trunk/src/java/org/apache/hadoop/io/WritableUtils.java Modified: hadoop/common/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/CHANGES.txt?rev=810384&r1=810383&r2=810384&view=diff ============================================================================== --- hadoop/common/trunk/CHANGES.txt (original) +++ hadoop/common/trunk/CHANGES.txt Wed Sep 2 07:07:21 2009 @@ -507,7 +507,10 @@ HADOOP-6184. Provide an API to dump Configuration in a JSON format. (V.V.Chaitanya Krishna via yhemanth) - + + HADOOP-6224. Add a method to WritableUtils performing a bounded read of an + encoded String. (Jothi Padmanabhan via cdouglas) + OPTIMIZATIONS HADOOP-5595. NameNode does not need to run a replicator to choose a Modified: hadoop/common/trunk/src/java/org/apache/hadoop/io/WritableUtils.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/io/WritableUtils.java?rev=810384&r1=810383&r2=810384&view=diff ============================================================================== --- hadoop/common/trunk/src/java/org/apache/hadoop/io/WritableUtils.java (original) +++ hadoop/common/trunk/src/java/org/apache/hadoop/io/WritableUtils.java Wed Sep 2 07:07:21 2009 @@ -415,4 +415,29 @@ } return out.getData(); } + + /** + * Read a string, but check it for sanity. The format consists of a vint + * followed by the given number of bytes. + * @param in the stream to read from + * @param maxLength the largest acceptable length of the encoded string + * @return the bytes as a string + * @throws IOException if reading from the DataInput fails + * @throws IllegalArgumentException if the encoded byte size for string + is negative or larger than maxSize. Only the vint is read. + */ + public static String readStringSafely(DataInput in, + int maxLength + ) throws IOException, + IllegalArgumentException { + int length = readVInt(in); + if (length < 0 || length > maxLength) { + throw new IllegalArgumentException("Encoded byte size for String was " + length + + ", which is outside of 0.." + + maxLength + " range."); + } + byte [] bytes = new byte[length]; + in.readFully(bytes, 0, length); + return Text.decode(bytes); + } } From common-commits-returnelliottmatthew@example.net Wed Sep 02 11:07:59 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 21903 invoked from network); 2 Sep 2009 11:07:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Sep 2009 11:07:59 -0000 Received: (qmail 54520 invoked by uid 500); 2 Sep 2009 11:07:59 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 54445 invoked by uid 500); 2 Sep 2009 11:07:58 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 54435 invoked by uid 99); 2 Sep 2009 11:07:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 11:07:58 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 11:07:49 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id E9EFB2388868; Wed, 2 Sep 2009 11:07:27 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810451 - in /hadoop/common/trunk: ./ src/java/org/apache/hadoop/io/ src/java/org/apache/hadoop/io/file/tfile/ src/test/core/org/apache/hadoop/io/ Date: Wed, 02 Sep 2009 11:07:27 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: ddas Date: Wed Sep 2 11:07:27 2009 New Revision: 810451 URL: http://svn.apache.org/viewvc?rev=810451&view=rev Log: HADOOP-6226. Moves BoundedByteArrayOutputStream from the tfile package to the io package and makes it available to other users (MAPREDUCE-318). Contributed by Jothi Padmanabhan. Added: hadoop/common/trunk/src/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java hadoop/common/trunk/src/test/core/org/apache/hadoop/io/TestBoundedByteArrayOutputStream.java Removed: hadoop/common/trunk/src/java/org/apache/hadoop/io/file/tfile/BoundedByteArrayOutputStream.java Modified: hadoop/common/trunk/CHANGES.txt hadoop/common/trunk/src/java/org/apache/hadoop/io/file/tfile/TFile.java Modified: hadoop/common/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/CHANGES.txt?rev=810451&r1=810450&r2=810451&view=diff ============================================================================== --- hadoop/common/trunk/CHANGES.txt (original) +++ hadoop/common/trunk/CHANGES.txt Wed Sep 2 11:07:27 2009 @@ -162,6 +162,10 @@ HADOOP-6120. Add support for Avro specific and reflect data. (sharad via cutting) + HADOOP-6226. Moves BoundedByteArrayOutputStream from the tfile package to + the io package and makes it available to other users (MAPREDUCE-318). + (Jothi Padmanabhan via ddas) + IMPROVEMENTS HADOOP-4565. Added CombineFileInputFormat to use data locality information Added: hadoop/common/trunk/src/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java?rev=810451&view=auto ============================================================================== --- hadoop/common/trunk/src/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java (added) +++ hadoop/common/trunk/src/java/org/apache/hadoop/io/BoundedByteArrayOutputStream.java Wed Sep 2 11:07:27 2009 @@ -0,0 +1,119 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with this + * work for additional information regarding copyright ownership. The ASF + * licenses this file to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + * License for the specific language governing permissions and limitations under + * the License. + */ + +package org.apache.hadoop.io; + +import java.io.EOFException; +import java.io.IOException; +import java.io.OutputStream; + +/** + * A byte array backed output stream with a limit. The limit should be smaller + * than the buffer capacity. The object can be reused through <code>reset</code> + * API and choose different limits in each round. + */ +public class BoundedByteArrayOutputStream extends OutputStream { + private final byte[] buffer; + private int limit; + private int count; + + /** + * Create a BoundedByteArrayOutputStream with the specified + * capacity + * @param capacity The capacity of the underlying byte array + */ + public BoundedByteArrayOutputStream(int capacity) { + this(capacity, capacity); + } + + /** + * Create a BoundedByteArrayOutputStream with the specified + * capacity and limit. + * @param capacity The capacity of the underlying byte array + * @param limit The maximum limit upto which data can be written + */ + public BoundedByteArrayOutputStream(int capacity, int limit) { + if ((capacity < limit) || (capacity | limit) < 0) { + throw new IllegalArgumentException("Invalid capacity/limit"); + } + this.buffer = new byte[capacity]; + this.limit = limit; + this.count = 0; + } + + @Override + public void write(int b) throws IOException { + if (count >= limit) { + throw new EOFException("Reaching the limit of the buffer."); + } + buffer[count++] = (byte) b; + } + + @Override + public void write(byte b[], int off, int len) throws IOException { + if ((off < 0) || (off > b.length) || (len < 0) || ((off + len) > b.length) + || ((off + len) < 0)) { + throw new IndexOutOfBoundsException(); + } else if (len == 0) { + return; + } + + if (count + len > limit) { + throw new EOFException("Reach the limit of the buffer"); + } + + System.arraycopy(b, off, buffer, count, len); + count += len; + } + + /** + * Reset the limit + * @param newlim New Limit + */ + public void reset(int newlim) { + if (newlim > buffer.length) { + throw new IndexOutOfBoundsException("Limit exceeds buffer size"); + } + this.limit = newlim; + this.count = 0; + } + + /** Reset the buffer */ + public void reset() { + this.limit = buffer.length; + this.count = 0; + } + + /** Return the current limit */ + public int getLimit() { + return limit; + } + + /** Returns the underlying buffer. + * Data is only valid to {@link #size()}. + */ + public byte[] getBuffer() { + return buffer; + } + + /** Returns the length of the valid data + * currently in the buffer. + */ + public int size() { + return count; + } +} Modified: hadoop/common/trunk/src/java/org/apache/hadoop/io/file/tfile/TFile.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/io/file/tfile/TFile.java?rev=810451&r1=810450&r2=810451&view=diff ============================================================================== --- hadoop/common/trunk/src/java/org/apache/hadoop/io/file/tfile/TFile.java (original) +++ hadoop/common/trunk/src/java/org/apache/hadoop/io/file/tfile/TFile.java Wed Sep 2 11:07:27 2009 @@ -34,6 +34,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.io.BoundedByteArrayOutputStream; import org.apache.hadoop.io.BytesWritable; import org.apache.hadoop.io.DataInputBuffer; import org.apache.hadoop.io.DataOutputBuffer; Added: hadoop/common/trunk/src/test/core/org/apache/hadoop/io/TestBoundedByteArrayOutputStream.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/test/core/org/apache/hadoop/io/TestBoundedByteArrayOutputStream.java?rev=810451&view=auto ============================================================================== --- hadoop/common/trunk/src/test/core/org/apache/hadoop/io/TestBoundedByteArrayOutputStream.java (added) +++ hadoop/common/trunk/src/test/core/org/apache/hadoop/io/TestBoundedByteArrayOutputStream.java Wed Sep 2 11:07:27 2009 @@ -0,0 +1,91 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.io; + +import java.io.IOException; + +import junit.framework.TestCase; +import java.util.Arrays; +import java.util.Random; + + +/** Unit tests for BoundedByteArrayOutputStream */ +public class TestBoundedByteArrayOutputStream extends TestCase { + + private static final int SIZE = 1024; + private static final byte[] INPUT = new byte[SIZE]; + static { + new Random().nextBytes(INPUT); + } + + public void testBoundedStream() throws IOException { + + BoundedByteArrayOutputStream stream = + new BoundedByteArrayOutputStream(SIZE); + + // Write to the stream, get the data back and check for contents + stream.write(INPUT, 0, SIZE); + assertTrue("Array Contents Mismatch", + Arrays.equals(INPUT, stream.getBuffer())); + + // Try writing beyond end of buffer. Should throw an exception + boolean caughtException = false; + + try { + stream.write(INPUT[0]); + } catch (Exception e) { + caughtException = true; + } + + assertTrue("Writing beyond limit did not throw an exception", + caughtException); + + //Reset the stream and try, should succeed + stream.reset(); + assertTrue("Limit did not get reset correctly", + (stream.getLimit() == SIZE)); + stream.write(INPUT, 0, SIZE); + assertTrue("Array Contents Mismatch", + Arrays.equals(INPUT, stream.getBuffer())); + + // Try writing one more byte, should fail + caughtException = false; + try { + stream.write(INPUT[0]); + } catch (Exception e) { + caughtException = true; + } + + // Reset the stream, but set a lower limit. Writing beyond + // the limit should throw an exception + stream.reset(SIZE - 1); + assertTrue("Limit did not get reset correctly", + (stream.getLimit() == SIZE -1)); + caughtException = false; + + try { + stream.write(INPUT, 0, SIZE); + } catch (Exception e) { + caughtException = true; + } + + assertTrue("Writing beyond limit did not throw an exception", + caughtException); + } +} From common-commits-returnelliottmatthew@example.net Wed Sep 02 14:31:01 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 80760 invoked from network); 2 Sep 2009 14:31:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Sep 2009 14:31:01 -0000 Received: (qmail 33414 invoked by uid 500); 2 Sep 2009 14:31:01 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 33357 invoked by uid 500); 2 Sep 2009 14:31:01 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 33348 invoked by uid 500); 2 Sep 2009 14:31:01 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 33345 invoked by uid 99); 2 Sep 2009 14:31:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 14:31:01 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 14:30:58 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id 69BA7118AA for <elliottmatthew@example.net>; Wed, 2 Sep 2009 14:30:37 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <elliottmatthew@example.net> To: elliottmatthew@example.net Date: Wed, 02 Sep 2009 14:30:37 -0000 Message-ID: <elliottmatthew@example.net> Subject: [Hadoop Wiki] Trivial Update of "BristolHadoopWorkshop" by SteveLoughran X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification. The following page has been changed by SteveLoughran: http://wiki.apache.org/hadoop/BristolHadoopWorkshop The comment on the change is: fix some sentences ------------------------------------------------------------------------------ There was discussion on handing big graphs in the system, ones where the graph itself is very large. Someone need's to take Paolo's PageRank-over-MapReduce code and test it on bigger data sets. - There was a good point on what is "efficient" in this world: If you can do it in O(n) on MR, and there is no single computer in the world that can work on a graph of that scale, then it's efficient. + There was a good point on what is "efficient" in this world. - The key point here being that yes, something done as a chain of MR jobs on a Hadoop cluster may seem an inefficient approach, but if there is no other way to store that much data, or run through it, then graph people will be happy. + Yes, something done as a chain of MR jobs on a Hadoop cluster may seem an inefficient approach, but if there is no other way to store that much data, or run through it, then graph people will be happy. = Yahoo! MS search deal = From common-commits-returnelliottmatthew@example.net Wed Sep 02 22:02:00 2009 Return-Path: <common-commits-returnelliottmatthew@example.net> Delivered-To: elliottmatthew@example.net Received: (qmail 40239 invoked from network); 2 Sep 2009 22:02:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Sep 2009 22:02:00 -0000 Received: (qmail 29869 invoked by uid 500); 2 Sep 2009 22:01:59 -0000 Delivered-To: elliottmatthew@example.net Received: (qmail 29784 invoked by uid 500); 2 Sep 2009 22:01:59 -0000 Mailing-List: contact elliottmatthew@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:elliottmatthew@example.net> List-Unsubscribe: <mailto:elliottmatthew@example.net> List-Post: <mailto:elliottmatthew@example.net> List-Id: <common-commits.hadoop.apache.org> Reply-To: elliottmatthew@example.net Delivered-To: mailing list elliottmatthew@example.net Received: (qmail 29774 invoked by uid 99); 2 Sep 2009 22:01:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 22:01:59 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 Sep 2009 22:01:56 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 5E03523888D2; Wed, 2 Sep 2009 22:01:35 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r810709 - in /hadoop/common/trunk: CHANGES.txt conf/hadoop-metrics.properties src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java Date: Wed, 02 Sep 2009 22:01:35 -0000 To: elliottmatthew@example.net From: elliottmatthew@example.net X-Mailer: svnmailer-1.0.8 Message-Id: <elliottmatthew@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Author: stack Date: Wed Sep 2 22:01:34 2009 New Revision: 810709 URL: http://svn.apache.org/viewvc?rev=810709&view=rev Log: HADOOP-4675 Current Ganglia metrics implementation is incompatible with Ganglia 3.1 Modified: hadoop/common/trunk/CHANGES.txt hadoop/common/trunk/conf/hadoop-metrics.properties hadoop/common/trunk/src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java Modified: hadoop/common/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/CHANGES.txt?rev=810709&r1=810708&r2=810709&view=diff ============================================================================== --- hadoop/common/trunk/CHANGES.txt (original) +++ hadoop/common/trunk/CHANGES.txt Wed Sep 2 22:01:34 2009 @@ -515,6 +515,9 @@ HADOOP-6224. Add a method to WritableUtils performing a bounded read of an encoded String. (Jothi Padmanabhan via cdouglas) + HADOOP-4675 Current Ganglia metrics implementation is incompatible with Ganglia 3.1 + (Brian Brockelman, Scott Beardsley via stack) + OPTIMIZATIONS HADOOP-5595. NameNode does not need to run a replicator to choose a Modified: hadoop/common/trunk/conf/hadoop-metrics.properties URL: http://svn.apache.org/viewvc/hadoop/common/trunk/conf/hadoop-metrics.properties?rev=810709&r1=810708&r2=810709&view=diff ============================================================================== --- hadoop/common/trunk/conf/hadoop-metrics.properties (original) +++ hadoop/common/trunk/conf/hadoop-metrics.properties Wed Sep 2 22:01:34 2009 @@ -7,7 +7,9 @@ #dfs.fileName=/tmp/dfsmetrics.log # Configuration of the "dfs" context for ganglia +# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter) # dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext +# dfs.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 # dfs.period=10 # dfs.servers=localhost:8649 @@ -21,13 +23,15 @@ #mapred.fileName=/tmp/mrmetrics.log # Configuration of the "mapred" context for ganglia +# Pick one: Ganglia 3.0 (former) or Ganglia 3.1 (latter) # mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext +# mapred.class=org.apache.hadoop.metrics.ganglia.GangliaContext31 # mapred.period=10 # mapred.servers=localhost:8649 # Configuration of the "jvm" context for null -jvm.class=org.apache.hadoop.metrics.spi.NullContext +#jvm.class=org.apache.hadoop.metrics.spi.NullContext # Configuration of the "jvm" context for file #jvm.class=org.apache.hadoop.metrics.file.FileContext Modified: hadoop/common/trunk/src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java URL: http://svn.apache.org/viewvc/hadoop/common/trunk/src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java?rev=810709&r1=810708&r2=810709&view=diff ============================================================================== --- hadoop/common/trunk/src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java (original) +++ hadoop/common/trunk/src/java/org/apache/hadoop/metrics/ganglia/GangliaContext.java Wed Sep 2 22:01:34 2009 @@ -71,16 +71,16 @@ typeTable.put(Float.class, "float"); } - private byte[] buffer = new byte[BUFFER_SIZE]; - private int offset; + protected byte[] buffer = new byte[BUFFER_SIZE]; + protected int offset; - private List<? extends SocketAddress> metricsServers; + protected List<? extends SocketAddress> metricsServers; private Map<String,String> unitsTable; private Map<String,String> slopeTable; private Map<String,String> tmaxTable; private Map<String,String> dmaxTable; - private DatagramSocket datagramSocket; + protected DatagramSocket datagramSocket; /** Creates
{'source': 'Apache Software Foundation Public Mail Archives'}