From common-user-nathanphillips@example.com Sun Aug 01 03:37:56 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 15947 invoked from network); 1 Aug 2010 03:37:56 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Aug 2010 03:37:56 -0000 Received: (qmail 89384 invoked by uid 500); 1 Aug 2010 03:37:54 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 88578 invoked by uid 500); 1 Aug 2010 03:37:48 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 88562 invoked by uid 99); 1 Aug 2010 03:37:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 03:37:47 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-qw0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 03:37:41 +0000 Received: by qwd7 with SMTP id 7so1344561qwd.35 for <multiple recipients>; Sat, 31 Jul 2010 20:37:19 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id s21mr2752245vcn.82.1280633839672; Sat, 31 Jul 2010 20:37:19 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sat, 31 Jul 2010 20:37:19 -0700 (PDT) In-Reply-To: <nathanphillips@example.com> References: <nathanphillips@example.com> Date: Sat, 31 Jul 2010 20:37:19 -0700 Message-ID: <nathanphillips@example.com> Subject: Re: How to get lzo compression library loaded? From: Alex Kozlov <nathanphillips@example.com> To: nathanphillips@example.com Cc: nathanphillips@example.com Content-Type: multipart/alternative; boundary=0016e64e9d2019dc1b048cbacd07 --0016e64e9d2019dc1b048cbacd07 Content-Type: text/plain; charset=ISO-8859-1 Hi Alex, org.apache.hadoop.io.compress.GzipCodec is in the hadoop-core-*.jar. You don't need to add /usr/local/hadoop/hadoop-0.20.2/lib/ to the classpath since hadoop shell script does it for you. What is exactly the command you are trying to run? Can you also give the output of 'env | grep -i hadoop' and 'env | grep -i classpath'. Alex K On Sat, Jul 31, 2010 at 1:38 AM, Alex Luya <nathanphillips@example.com> wrote: > Hello: > I have followed this link:http://code.google.com/p/hadoop-gpl- > compression/wiki/FAQ<http://code.google.com/p/hadoop-gpl-%0Acompression/wiki/FAQ>to install lzo compression library,and copy > hadoop-lzo-0.4.4.jar to $HADOOP_HOME/lib,and all files under > ..lib/native/Linux-amd64-64 to $HADOOP_HOME/lib/native/Linux-amd64-64, > and run example,but got this errors: > > ---------------------------------------------------------------------------------------------------------------- > Exception in thread "main" java.lang.IllegalArgumentException: Compression > codec > org.apache.hadoop.io.compress.GzipCodec not found. > at > > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96) > at > > org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134) > at > com.hadoop.compression.lzo.LzoIndex.createIndex(LzoIndex.java:202) > at > com.hadoop.compression.lzo.LzoIndexer.indexSingleFile(LzoIndexer.java:117) > at > com.hadoop.compression.lzo.LzoIndexer.indexInternal(LzoIndexer.java:98) > at com.hadoop.compression.lzo.LzoIndexer.index(LzoIndexer.java:52) > at com.hadoop.compression.lzo.LzoIndexer.main(LzoIndexer.java:137) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.io.compress.GzipCodec > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:762) > at > > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:89) > ... 11 more > > > > ---------------------------------------------------------------------------------------------------------------- > > and then I try to add this to hadoop-env.sh: > > export > HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/hadoop/hadoop-0.20.2/lib/ > > Same problem as before,this problem is killing me,because it has surrounded > me > for one month. > > > --0016e64e9d2019dc1b048cbacd07-- From common-user-nathanphillips@example.com Sun Aug 01 18:55:14 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 75428 invoked from network); 1 Aug 2010 18:55:14 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Aug 2010 18:55:14 -0000 Received: (qmail 96289 invoked by uid 500); 1 Aug 2010 18:55:11 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 96167 invoked by uid 500); 1 Aug 2010 18:55:11 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 96158 invoked by uid 99); 1 Aug 2010 18:55:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 18:55:10 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-wy0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 18:55:06 +0000 Received: by wyf23 with SMTP id 23so3722804wyf.35 for <nathanphillips@example.com>; Sun, 01 Aug 2010 11:54:45 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 45mr1864798wef.106.1280688884877; Sun, 01 Aug 2010 11:54:44 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sun, 1 Aug 2010 11:54:44 -0700 (PDT) In-Reply-To: <nathanphillips@example.com> References: <nathanphillips@example.com> <nathanphillips@example.com> <nathanphillips@example.com> <nathanphillips@example.com> Date: Sun, 1 Aug 2010 14:54:44 -0400 Message-ID: <nathanphillips@example.com> Subject: Re: Preferred Java version From: Eric Sammer <nathanphillips@example.com> To: nathanphillips@example.com Content-Type: multipart/alternative; boundary=0016364c75410d083a048cc79ec9 --0016364c75410d083a048cc79ec9 Content-Type: text/plain; charset=ISO-8859-1 Raymond: Both _16 and _20 are heavily used in production and seem to be very stable. I don't know that we have enough information about _21 yet to say how that does but it's worth testing if you have some extra time. On Fri, Jul 30, 2010 at 10:54 PM, Otis Gospodnetic < nathanphillips@example.com> wrote: > I don't have the answer, but I know there is a _21 update available, and > includes some HotSpot fixes. > > > Otis---- > Sematext 888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4 http://sematext.com/ 888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4 Solr - Lucene - Nutch > Hadoop ecosystem search 888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4 http://search-hadoop.com/ > > > > ----- Original Message ---- > > From: Raymond Jennings III <nathanphillips@example.com> > > To: nathanphillips@example.com > > Sent: Fri, July 16, 2010 12:44:03 PM > > Subject: Preferred Java version > > > > Is 1.6.0_17 or 1.6.0_20 preferred as the JRE for hadoop? Thank you. > > > > > > > > > > > -- Eric Sammer twitter: esammer data: www.cloudera.com --0016364c75410d083a048cc79ec9-- From common-user-nathanphillips@example.com Sun Aug 01 19:15:58 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 83776 invoked from network); 1 Aug 2010 19:15:57 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Aug 2010 19:15:57 -0000 Received: (qmail 4204 invoked by uid 500); 1 Aug 2010 19:15:55 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 4169 invoked by uid 500); 1 Aug 2010 19:15:55 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 4161 invoked by uid 500); 1 Aug 2010 19:15:54 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 4158 invoked by uid 99); 1 Aug 2010 19:15:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 19:15:54 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,SPF_HELO_PASS,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 19:15:48 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <nathanphillips@example.com>) id 1Ofe0N-0003Kv-30 for nathanphillips@example.com; Sun, 01 Aug 2010 12:15:27 -0700 Message-ID: <nathanphillips@example.com> Date: Sun, 1 Aug 2010 12:15:27 -0700 (PDT) From: Pete Tyler <nathanphillips@example.com> To: nathanphillips@example.com Subject: Where is Hadoop 20.3? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: nathanphillips@example.com X-Virus-Checked: Checked by ClamAV on apache.org Apologies for the newbie question but I think I'm a little lost. Hadoop 20.2 came out in Feb 2010 but the fix I'm looking for is in Hadoop 20.3, https://issues.apache.org/jira/browse/MAPREDUCE-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel However I can only find the somewhat mature 20.2 version under 'downloads'. Is it possible that 20.3 is not out yet? -- View this message in context: http://old.nabble.com/Where-is-Hadoop-20.3--tp29319909p29319909.html Sent from the Hadoop core-user mailing list nathanphillips@example.com. From common-user-nathanphillips@example.com Sun Aug 01 19:22:50 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 85021 invoked from network); 1 Aug 2010 19:22:49 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Aug 2010 19:22:49 -0000 Received: (qmail 8631 invoked by uid 500); 1 Aug 2010 19:22:47 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 8560 invoked by uid 500); 1 Aug 2010 19:22:47 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 8550 invoked by uid 99); 1 Aug 2010 19:22:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 19:22:46 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout3.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 19:22:40 +0000 Received: from [83.161.120.155] ([83.161.120.155]) by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o71JLxCS007555 for <nathanphillips@example.com>; Sun, 1 Aug 2010 12:21:59 -0700 (PDT) Message-ID: <nathanphillips@example.com> Date: Sun, 01 Aug 2010 12:21:59 -0700 From: Jakob Homan <nathanphillips@example.com> User-Agent: Thunderbird 83.161.120.155 (Macintosh/20080213) MIME-Version: 1.0 To: "nathanphillips@example.com" <nathanphillips@example.com> Subject: Re: Where is Hadoop 20.3? References: <nathanphillips@example.com> In-Reply-To: <nathanphillips@example.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Pete- You're correct, there's not yet been a 20.3 release. Although it has been a while and Owen's recent bugfix in HADOOP-6881 may be worth a new release soon. -Jakob Pete Tyler wrote: > Apologies for the newbie question but I think I'm a little lost. Hadoop 20.2 > came out in Feb 2010 but the fix I'm looking for is in Hadoop 20.3, > > > https://issues.apache.org/jira/browse/MAPREDUCE-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > > However I can only find the somewhat mature 20.2 version under 'downloads'. > Is it possible that 20.3 is not out yet? From common-user-nathanphillips@example.com Sun Aug 01 19:23:15 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 85245 invoked from network); 1 Aug 2010 19:23:14 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Aug 2010 19:23:14 -0000 Received: (qmail 10692 invoked by uid 500); 1 Aug 2010 19:23:12 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 10644 invoked by uid 500); 1 Aug 2010 19:23:12 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 10636 invoked by uid 99); 1 Aug 2010 19:23:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 19:23:12 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-wy0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 19:23:07 +0000 Received: by wyf23 with SMTP id 23so3740347wyf.35 for <nathanphillips@example.com>; Sun, 01 Aug 2010 12:22:45 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id k8mr4073544wbv.225.1280690565741; Sun, 01 Aug 2010 12:22:45 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sun, 1 Aug 2010 12:22:45 -0700 (PDT) In-Reply-To: <nathanphillips@example.com> References: <nathanphillips@example.com> Date: Sun, 1 Aug 2010 15:22:45 -0400 Message-ID: <nathanphillips@example.com> Subject: Re: Where is Hadoop 20.3? From: Eric Sammer <nathanphillips@example.com> To: nathanphillips@example.com Content-Type: multipart/alternative; boundary=0016362849a83d0980048cc80229 --0016362849a83d0980048cc80229 Content-Type: text/plain; charset=ISO-8859-1 Pete: You're correct. Hadoop 0.20.3 is not yet released. On Sun, Aug 1, 2010 at 3:15 PM, Pete Tyler <nathanphillips@example.com> wrote: > > Apologies for the newbie question but I think I'm a little lost. Hadoop > 20.2 > came out in Feb 2010 but the fix I'm looking for is in Hadoop 20.3, > > > > https://issues.apache.org/jira/browse/MAPREDUCE-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > > However I can only find the somewhat mature 20.2 version under 'downloads'. > Is it possible that 20.3 is not out yet? > -- > View this message in context: > http://old.nabble.com/Where-is-Hadoop-20.3--tp29319909p29319909.html > Sent from the Hadoop core-user mailing list nathanphillips@example.com. > > -- Eric Sammer twitter: esammer data: www.cloudera.com --0016362849a83d0980048cc80229-- From common-user-nathanphillips@example.com Sun Aug 01 21:24:15 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 19766 invoked from network); 1 Aug 2010 21:24:15 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Aug 2010 21:24:15 -0000 Received: (qmail 65049 invoked by uid 500); 1 Aug 2010 21:24:13 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 64951 invoked by uid 500); 1 Aug 2010 21:24:12 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 64943 invoked by uid 99); 1 Aug 2010 21:24:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 21:24:12 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO smtp0.epfl.ch) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Sun, 01 Aug 2010 21:24:03 +0000 Received: (qmail 22285 invoked by uid 107); 1 Aug 2010 21:23:43 -0000 X-Virus-Scanned: ClamAV Received: from adsl-84-227-223-160.adslplus.ch (83.161.120.155) (authenticated) by smtp0.epfl.ch (AngelmatoPhylax SMTP proxy); Sun, 01 Aug 2010 23:23:43 +0200 Message-ID: <nathanphillips@example.com> Date: Sun, 01 Aug 2010 23:23:42 +0200 From: Teodor Macicas <nathanphillips@example.com> User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:83.161.120.155) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: nathanphillips@example.com Subject: [HADOOP] Terasort for numbers Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi all, I am using hadoop 0.20.2 and I want to use sort huge amount of data. I've read about Terasort [from examples], but now it's using 10bytes char keys. Changing keys from char to integer wasn't a good solution as Terasort builds a trie for creating total order partitions. I got stuck when I tried to change the char trie to a one suitable for number keys. Then, I've given a try to Sort [also from examples] and it did work for integer keys, but without a total order partitioning. In the end of the day, the final result can not be created only by putting together all reducers' outputs. Each reducer sorts only a subset of data and no merging is occured between two reducers. Please can anyone advise me what and how to use in order to sort huge amount of real numbers ? Looking forward for your replies. Thank you. Best, Teodor From common-user-nathanphillips@example.com Sun Aug 01 22:14:59 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 33621 invoked from network); 1 Aug 2010 22:14:59 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Aug 2010 22:14:59 -0000 Received: (qmail 79502 invoked by uid 500); 1 Aug 2010 22:14:56 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 79246 invoked by uid 500); 1 Aug 2010 22:14:56 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 79229 invoked by uid 99); 1 Aug 2010 22:14:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 22:14:55 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-qw0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 22:14:49 +0000 Received: by qwd7 with SMTP id 7so1760502qwd.35 for <nathanphillips@example.com>; Sun, 01 Aug 2010 15:14:27 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id q18mr3567881vcn.113.1280700867621; Sun, 01 Aug 2010 15:14:27 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sun, 1 Aug 2010 15:14:27 -0700 (PDT) In-Reply-To: <nathanphillips@example.com> References: <nathanphillips@example.com> Date: Sun, 1 Aug 2010 15:14:27 -0700 Message-ID: <nathanphillips@example.com> Subject: Re: [HADOOP] Terasort for numbers From: Alex Kozlov <nathanphillips@example.com> To: nathanphillips@example.com Content-Type: multipart/alternative; boundary=0016e646410247337c048cca682f X-Virus-Checked: Checked by ClamAV on apache.org --0016e646410247337c048cca682f Content-Type: text/plain; charset=ISO-8859-1 Hi Teodor, I am not clear what you call 'real numbers'. Terasort does work on bytes (10 bytes key and 90 bytes payload). The actual 'meaning' of the bytes really does not matter as Hadoop uses binary comparators on the raw value. Total order partitioning should also work with any WritableComparable key (if it doesn't, it's a bug). My guess your problem is converting a char trie to WritableComparable. Can you provide more background? Are the strings of fixed length? Alex K On Sun, Aug 1, 2010 at 2:23 PM, Teodor Macicas <nathanphillips@example.com>wrote: > Hi all, > > > I am using hadoop 0.20.2 and I want to use sort huge amount of data. I've > read about Terasort [from examples], but now it's using 10bytes char keys. > Changing keys from char to integer wasn't a good solution as Terasort > builds a trie for creating total order partitions. I got stuck when I tried > to change the char trie to a one suitable for number keys. > > Then, I've given a try to Sort [also from examples] and it did work for > integer keys, but without a total order partitioning. In the end of the day, > the final result can not be created only by putting together all reducers' > outputs. Each reducer sorts only a subset of data and no merging is occured > between two reducers. > > Please can anyone advise me what and how to use in order to sort huge > amount of real numbers ? > Looking forward for your replies. > > > Thank you. > Best, > Teodor > --0016e646410247337c048cca682f-- From common-user-nathanphillips@example.com Sun Aug 01 23:11:40 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 50473 invoked from network); 1 Aug 2010 23:11:39 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Aug 2010 23:11:39 -0000 Received: (qmail 97286 invoked by uid 500); 1 Aug 2010 23:11:37 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 97202 invoked by uid 500); 1 Aug 2010 23:11:36 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 97194 invoked by uid 500); 1 Aug 2010 23:11:36 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 97191 invoked by uid 99); 1 Aug 2010 23:11:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 23:11:36 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,SPF_HELO_PASS,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Aug 2010 23:11:31 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <nathanphillips@example.com>) id 1OfhgU-0002gl-GW for nathanphillips@example.com; Sun, 01 Aug 2010 16:11:10 -0700 Message-ID: <nathanphillips@example.com> Date: Sun, 1 Aug 2010 16:11:10 -0700 (PDT) From: Pete Tyler <nathanphillips@example.com> To: nathanphillips@example.com Subject: Where is Hadoop 20.3? MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: nathanphillips@example.com Apologies for the newbie question but I think I'm a little lost. Hadoop 20.2 came out in Feb 2010 but the fix I'm looking for is in Hadoop 20.3, https://issues.apache.org/jira/browse/MAPREDUCE-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel However I can only find the somewhat mature 20.2 version under 'downloads'. Is it possible that 20.3 is not out yet? -- View this message in context: http://old.nabble.com/Where-is-Hadoop-20.3--tp29319909p29319909.html Sent from the Hadoop core-user mailing list nathanphillips@example.com. From common-user-nathanphillips@example.com Mon Aug 02 04:57:03 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 60098 invoked from network); 2 Aug 2010 04:57:02 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 04:57:02 -0000 Received: (qmail 98717 invoked by uid 500); 2 Aug 2010 04:57:00 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 98360 invoked by uid 500); 2 Aug 2010 04:56:55 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 98352 invoked by uid 99); 2 Aug 2010 04:56:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 04:56:54 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-qw0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 04:56:48 +0000 Received: by qwd7 with SMTP id 7so1951295qwd.35 for <nathanphillips@example.com>; Sun, 01 Aug 2010 21:56:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=tzCyd8w41HQQFs/ZE3bxIzR2EOgFMbyKSEu4mzL/bHU=; b=EJuL8RNdn+DBLiGOvcAYj6YCzvvzu6q4ScXZ8NAO9xMCrCRaQTrpoOIU6mkHrBL/V5 i1sflsLdAbto/ZUHUQJUzO4+OKm0O4loNK3vLaNCXM6ONExNlhZe423ZExa5XNUv6Shz yiPi9kVltUKWSTsCw0q/VDkW+Qv8lJ1U4wdBw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=R8zyibuuj7Fbl0tgUOnNpeAHyhyAOKVIRa9ags5CMPgylJnurghM7eUcw0R6GQWfZR c7y8hvfY3izmgyTKEWniOl+748dEh+pTx50z6JkKYO8KDvr9czFvRMTkmefar+iGoDkw zrXuT6UW5T22WuE5ahmnfIFVAFj8JPIubLKkM= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id y20mr3903114vcj.4.1280724985679; Sun, 01 Aug 2010 21:56:25 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sun, 1 Aug 2010 21:56:25 -0700 (PDT) In-Reply-To: <nathanphillips@example.com> References: <nathanphillips@example.com> <nathanphillips@example.com> <nathanphillips@example.com> <nathanphillips@example.com> Date: Mon, 2 Aug 2010 10:26:25 +0530 Message-ID: <nathanphillips@example.com> Subject: Re: reuse cached files From: Hemanth Yamijala <nathanphillips@example.com> To: nathanphillips@example.com Content-Type: text/plain; charset=ISO-8859-1 Hi, > Thanks Hemanth. Is there any way to invalidate the reuse and ask Hadoop to > resent exactly the same files to cache for every job? I may be able to answer this better if I understand the use case. If you need the same files for every job, why would you need to send them afresh each time ? If something is cached, it can be reused, no ? I am sure I must be missing something in your requirement ... Thanks Hemanth From common-user-nathanphillips@example.com Mon Aug 02 05:01:13 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 60653 invoked from network); 2 Aug 2010 05:01:12 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 05:01:12 -0000 Received: (qmail 1116 invoked by uid 500); 2 Aug 2010 05:01:10 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 927 invoked by uid 500); 2 Aug 2010 05:01:07 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 919 invoked by uid 99); 2 Aug 2010 05:01:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:01:06 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-qy0-f169.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:01:01 +0000 Received: by qyk9 with SMTP id 9so6902582qyk.14 for <nathanphillips@example.com>; Sun, 01 Aug 2010 22:00:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=ZbELM7AGfIKLZUuFudeKDUp1Da0/jZQb6n284bMWaEs=; b=RKwhPA6iHc2gvQqPmn18SGTR3r3leT8YpJKxZdV9DJzbU4IGOeUysLRV7tQse58OKw WwaOXYApziUDSsDeGTyvYiLWsVJJkrS6drZEnutHqmm8gv3B+ad020bHKxRTmL1dBkYq 2ClUjUOYhjHbZ5P8Ky1y91Jn9a4/lgHdnCdsI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=l4UJZYnqAQYCAzp7WJNLftXIcHf43uoz72+tn/88FD/M26rEnqX7AJpMIQK7PMI083 dt647jEABVJmP42HEl91/+OI/zANlcfK6cJ5su06L0qavXJDZ2lnU59WKzJalQnGhaJB 2rb0a2vVTmbEwyKri9dEGAqUNHKiZLHf4K7I8= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id m15mr3774595vcs.58.1280725240434; Sun, 01 Aug 2010 22:00:40 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sun, 1 Aug 2010 22:00:40 -0700 (PDT) In-Reply-To: <nathanphillips@example.com> References: <nathanphillips@example.com> <nathanphillips@example.com> Date: Mon, 2 Aug 2010 10:30:40 +0530 Message-ID: <nathanphillips@example.com> Subject: Re: jobtracker.jsp reports "GC overhead limit exceeded" From: Hemanth Yamijala <nathanphillips@example.com> To: nathanphillips@example.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, > Actually I enabled all level logs. But I didn't realize to check logs in = .out files and only looked at .log file and didn't see any error msgs. now = I opened the .out file and saw the following logged exception: > > Exception in thread "IPC Server handler 5 on 50002" java.lang.OutOfMemory= Error: Java heap space > =nathanphillips@example.com(Arrays.java:2786) > =nathanphillips@example.com(ByteArrayOutputStr= eam.java:133) > =nathanphillips@example.com(Server.java:1087) > =nathanphillips@example.com$2400(Server.java:77) > =nathanphillips@example.com$Handler.run(Server.java:984) > > Exception in thread "IPC Client (47) connection to servername/serverip:50= 001 from auser" java.lang.OutOfMemoryError: Java heap space > =nathanphillips@example.com(ByteBuffer.java:350) > =nathanphillips@example.com(SocketInputStre= am.java:128) > =nathanphillips@example.com(FilterInputStream.java:116) > =nathanphillips@example.com$Connection$PingInputStream.read= (Client.java:276) > =nathanphillips@example.com(BufferedInputStream.java:21= 8) > =nathanphillips@example.com(BufferedInputStream.java:23= 7) > =nathanphillips@example.com(DataInputStream.java:370) > =nathanphillips@example.com$Connection.receiveResponse(Clie= nt.java:501) > =nathanphillips@example.com$Connection.run(Client.java:446) > > Exception in thread "LeaseChecker" java.lang.OutOfMemoryError: GC overhea= d limit exceeded > Exception in thread "IPC Server handler 4 on 50002" java.lang.OutOfMemory= Error: Java heap space > Exception in thread "IPC Server handler 8 on 50002" java.lang.OutOfMemory= Error: GC overhead limit exceeded > Exception in thread "IPC Server handler 9 on 50002" java.lang.OutOfMemory= Error: GC overhead limit exceeded > Exception in thread "IPC Server handler 2 on 50002" java.lang.OutOfMemory= Error: GC overhead limit exceeded > Exception in thread "IPC Server handler 6 on 50002" java.lang.OutOfMemory= Error: GC overhead limit exceeded > Exception in thread "ResponseProcessor for block blk_-573819335330670501_= 1268378" java.lang.OutOfMemoryError: GC overhead limit exceeded > Exception in thread "IPC Server handler 7 on 50002" java.lang.OutOfMemory= Error: Java heap space > > 50001 and 50002 used for dfs and jobtracker. But it's normal that IPC thr= eads consume such big memory? > It may not be IPC threads that are causing the problem, though I remember at least one IPC buffer related OOM somewhere. A more typical case is that the daemon is running out of memory and the IPC threads are just manifesting the error because they're the entry-points into the application. I think we still need to go down the path that Alex was suggesting. What is the heap you have given the JobTracker ? What is number of Jobs / tasks / users ? How many nodes are there in the cluster ? > > --- On Fri, 7/30/10, Alex Loddengaard <nathanphillips@example.com> wrote: > > From: Alex Loddengaard <nathanphillips@example.com> > Subject: Re: jobtracker.jsp reports "GC overhead limit exceeded" > To: nathanphillips@example.com > Date: Friday, July 30, 2010, 2:19 PM > > What does "ps" show you?=A0 How much memory is being used by the jobtrack= er, > and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)?=A0= Also > consider turning on GC logging, which will find its way to the jobtracker > .out log in /var/log/hadoop: > > <http://java.sun.com/developer/technicalArticles/Programming/GCPortal/> > > Alex > > On Fri, Jul 30, 2010 at 3:10 PM, jiang licht <nathanphillips@example.com> wrot= e: > >> http://server:50030/jobtracker.jsp generates the following error message= : >> >> HTTP ERROR: 500 >> >> GC overhead limit exceeded >> >> RequestURI=3D/jobtracker.jsp >> Caused by: >> >> java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> Powered by Jetty:// >> >> The jobtracker is running below the limit. But "hadoop job -status" seem= s >> to halt and does not response ... >> >> The last 2 lines of jobtracker logs: >> >> 2010-07-30 13:53:18,482 DEBUG org.apache.hadoop.mapred.JobTracker: Got >> heartbeat from: tracker_host1:localhost.localdomain/83.161.120.155:53914(rest= arted: false initialContact: false acceptNewTasks: true) with >> responseId: -31252 >> 2010-07-30 13:55:32,917 DEBUG org.apache.hadoop.mapred.JobTracker: Start= ing >> launching task sweep >> >> Any thought about this? >> >> Thanks! >> --Michael >> >> >> > > > > From common-user-nathanphillips@example.com Mon Aug 02 05:12:54 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 67730 invoked from network); 2 Aug 2010 05:12:54 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 05:12:54 -0000 Received: (qmail 7230 invoked by uid 500); 2 Aug 2010 05:12:52 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 6959 invoked by uid 500); 2 Aug 2010 05:12:48 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 6951 invoked by uid 99); 2 Aug 2010 05:12:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:12:47 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-qy0-f169.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 05:12:41 +0000 Received: by qyk9 with SMTP id 9so6907913qyk.14 for <nathanphillips@example.com>; Sun, 01 Aug 2010 22:12:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=Q+jMw303UKwQ1CuSYjWlTiBMu1iPbj1IP0OQ8ROIlKU=; b=qgDgPYsirafJB2i3k1znDMwOFLG6U3KRuIP3FqHEgWQUL4gm4fxaWXnymmLf2Wc5jt DnDvLVeOzAjX/KH/jzzOuYtWtA1eBdvUIGwxWVaONQZw3iVN/gr5rh3hlrqk9/rFKy2I ys3hymMhXLbEcL2x6OqEE49cBA078aD/RpqR4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=m3SoCkAmq5d7DADNbBraObs/9UcqakXdYqXJ7Cn2yqTidXzeKPheKfWELyX2lhXQAO 7bPK1W6ytKL/rimQb0+Z43IZV/lrk8gMqfZKU2xqulNBcYu8V9tC6oyzjVJUEBdEWURc +uKqIvE4aL0PSVKKRykrknBMt+S56QUkkixp0= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id q11mr3909775vch.28.1280725940442; Sun, 01 Aug 2010 22:12:20 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sun, 1 Aug 2010 22:12:19 -0700 (PDT) Date: Mon, 2 Aug 2010 13:12:19 +0800 Message-ID: <nathanphillips@example.com> Subject: hadoop profile problem From: wenyang xu <nathanphillips@example.com> To: nathanphillips@example.com Content-Type: multipart/alternative; boundary=e0cb4e887b3bbc1763048cd03ea0 --e0cb4e887b3bbc1763048cd03ea0 Content-Type: text/plain; charset=ISO-8859-1 Hi, all. I am trying to run the terasort to collect some profile information on pseudo distributed mode of hadoop. I found hadoop could achive it by define the mapred.task.profile in conf/mapred-site.xml, so i add some properties to the conf/mapred-site.xml like this: ------conf/mapred-site.xml------------------ .... <property> <name>mapred.task.profile</name> <value>true</value> </property> <property> <name>keep.failed.task.files</name> <value>true</value> </property> <property> <name>keep.task.files.pattern</name> <value>.*</value> </property> .... --------------------------------------------- When I run the terasort, exception is throwed out like this: ------Exception showed on the screen---------------------------------------------------------- 10/08/02 04:35:05 INFO mapred.JobClient: Running job: job_201008020434_0001 10/08/02 04:35:06 INFO mapred.JobClient: map 0% reduce 0% java.io.FileNotFoundException: http://localhost:50060/tasklog?plaintextnathanphillips@example.com(HttpURLConnectionImpl.java:884) at org.apache.hadoop.mapred.JobClient.downloadProfile(JobClient.java:1225) at org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:1302) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1251) at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257) at java.lang.reflect.AccessibleObject.invokeV(AccessibleObject.java:197) at java.lang.reflect.Method.invoke(Method.java:238) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) at java.lang.reflect.AccessibleObject.invokeV(AccessibleObject.java:197) at java.lang.reflect.Method.invoke(Method.java:238) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) --------------------------------------------- -------------------------------------------------- I will be very thankful If someone could give me any sugguestion to solve this problem. Look forward to any reply. Best Regards, Cindy --e0cb4e887b3bbc1763048cd03ea0-- From common-user-nathanphillips@example.com Mon Aug 02 09:26:29 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 43317 invoked from network); 2 Aug 2010 09:26:29 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 09:26:29 -0000 Received: (qmail 51782 invoked by uid 500); 2 Aug 2010 09:26:27 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 51179 invoked by uid 500); 2 Aug 2010 09:26:23 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 51170 invoked by uid 99); 2 Aug 2010 09:26:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 09:26:21 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO smtp0.epfl.ch) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 Aug 2010 09:26:15 +0000 Received: (qmail 24874 invoked by uid 107); 2 Aug 2010 09:25:50 -0000 X-Virus-Scanned: ClamAV Received: from diaspriv1.epfl.ch (83.161.120.155) (authenticated) by smtp0.epfl.ch (AngelmatoPhylax SMTP proxy); Mon, 02 Aug 2010 11:25:50 +0200 Message-ID: <nathanphillips@example.com> Date: Mon, 02 Aug 2010 11:25:50 +0200 From: Teodor Macicas <nathanphillips@example.com> User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:83.161.120.155) Gecko/20100423 Thunderbird/3.0.4 MIME-Version: 1.0 To: "nathanphillips@example.com" <nathanphillips@example.com> Subject: Re: [HADOOP] Terasort for numbers References: <nathanphillips@example.com> <nathanphillips@example.com> In-Reply-To: <nathanphillips@example.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Alex, Thank you for your quick reply and sorry for not being so clear. The job I want to do is simple to sort data having numbers [doubles] as keys [0]. I noticed that Terasort is using 10b char key. How can I use this for my particular job ? Do I need to change the Terasort ? [0] example of workload: 123.45 payload1 -34.56 payload2 752.10 payload3 10.25 payload4 .... Does this make sense now ? Regards, Teodor On 08/02/2010 12:14 AM, Alex Kozlov wrote: > Hi Teodor, > > I am not clear what you call 'real numbers'. Terasort does work on bytes > (10 bytes key and 90 bytes payload). The actual 'meaning' of the bytes > really does not matter as Hadoop uses binary comparators on the raw value. > > Total order partitioning should also work with any WritableComparable key > (if it doesn't, it's a bug). > > My guess your problem is converting a char trie to WritableComparable. Can > you provide more background? Are the strings of fixed length? > > Alex K > > On Sun, Aug 1, 2010 at 2:23 PM, Teodor Macicas<nathanphillips@example.com>wrote: > > >> Hi all, >> >> >> I am using hadoop 0.20.2 and I want to use sort huge amount of data. I've >> read about Terasort [from examples], but now it's using 10bytes char keys. >> Changing keys from char to integer wasn't a good solution as Terasort >> builds a trie for creating total order partitions. I got stuck when I tried >> to change the char trie to a one suitable for number keys. >> >> Then, I've given a try to Sort [also from examples] and it did work for >> integer keys, but without a total order partitioning. In the end of the day, >> the final result can not be created only by putting together all reducers' >> outputs. Each reducer sorts only a subset of data and no merging is occured >> between two reducers. >> >> Please can anyone advise me what and how to use in order to sort huge >> amount of real numbers ? >> Looking forward for your replies. >> >> >> Thank you. >> Best, >> Teodor >> >> From common-user-nathanphillips@example.com Mon Aug 02 11:59:50 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 96912 invoked from network); 2 Aug 2010 11:59:50 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 11:59:50 -0000 Received: (qmail 79559 invoked by uid 500); 2 Aug 2010 11:59:48 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 79116 invoked by uid 500); 2 Aug 2010 11:59:43 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 79102 invoked by uid 99); 2 Aug 2010 11:59:42 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 11:59:42 +0000 X-ASF-Spam-Status: No, hits=4.4 required=10.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-wy0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 11:59:35 +0000 Received: by wyf23 with SMTP id 23so185513wyf.35 for <multiple recipients>; Mon, 02 Aug 2010 04:59:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=IRTVpZmsMFdfEZ49nrHz2esZEA9LhcdOgXRLSxa2O4s=; b=BJelFUKWGU/XytbeRtTFeAgdcIrfn7sDeOesLYOTAJvqto6f6eu4qVnuo5h69q0p+t l8Tlk74eT7Zf2KxGdBfaFpJvpzYECai7bhLX1qXE5/TBAx5rIHL64ssp1xeSWXWdIKo6 DnvRvYcI6qnplA5wedqNOm6/9Dc9dhsLl6mMI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=mll9G52xCmmyVRpmMhqew2NLFYonPSd2rMzQSgyW42TLQJPFQK53s3C+oxPtpvP+yR BKQQnDWK5JmbcYzDXA29yvNzeJSArpkmjzu4OQB1eJzIf32cuOfzimSvmR3xGEQTpR+g emUHuCFQaVEZ+1WhhVUFX/ttCY+ZhUy5G9UFI= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id t11mr4863533wbw.141.1280750354830; Mon, 02 Aug 2010 04:59:14 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Mon, 2 Aug 2010 04:59:14 -0700 (PDT) Date: Mon, 2 Aug 2010 21:59:14 +1000 Message-ID: <nathanphillips@example.com> Subject: how dfs.write.packet.size impact write through put of HDFS, strange result? From: elton sky <nathanphillips@example.com> To: common-user <nathanphillips@example.com>, nathanphillips@example.com Content-Type: multipart/alternative; boundary=0016365eec7ef214f9048cd5edcd X-Virus-Checked: Checked by ClamAV on apache.org --0016365eec7ef214f9048cd5edcd Content-Type: text/plain; charset=ISO-8859-1 Hello everyone, I am doing some evaluation on my 6 nodes mini cluster. Each node has 4 core Intel(R) Xeon(R) CPU 5130 @ 2.00GHz, 8GB memory, 500 GB disk, running Linux version 2.6.18-164.11.1.el5(Red Hat 4.1.2-46). I was trying to use different packet size (dfs.write.packet.size) and bytePerChunksum(bpc) (io.bytes.per.checksum), with default block size and buffer size, to see how they impact the performance of creating a 1GB file. The default packet size is 64K, and, surprisingly, if I increase this value to 256K, 1M, etc, the performance drops gradually. Below is some result I got: time(ms): 25742 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 1024 packet size: 1048576 time(ms): 24960 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 8192 packet size: 1048576 time(ms): 25412 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 65536 packet size: 1048576 time(ms): 20322 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 1024 packet size: 262144 time(ms): 19048 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 8192 packet size: 262144 time(ms): 19227 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 65536 packet size: 262144 time(ms): 15141 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 1024 packet size: 65536 time(ms): 14983 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 8192 packet size: 65536 time(ms): 14801 fileSize:1073741824 block size:67108864 bufferSize: 4096 bpc: 65536 packet size: 65536 ... >From these results I can see performance declined with packet size increases. bpc didn't affect much though. >From my understanding, I think if I use less packets then I introduces less overhead and less read/write roundtrip. I couldn't explain what happened here. Can anyone give me some idea here? --0016365eec7ef214f9048cd5edcd-- From common-user-nathanphillips@example.com Mon Aug 02 13:28:36 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 43726 invoked from network); 2 Aug 2010 13:28:36 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 13:28:36 -0000 Received: (qmail 54462 invoked by uid 500); 2 Aug 2010 13:28:33 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 54011 invoked by uid 500); 2 Aug 2010 13:28:29 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 54003 invoked by uid 99); 2 Aug 2010 13:28:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 13:28:27 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=FREEMAIL_FROM,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO web15907.mail.cnb.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 Aug 2010 13:28:21 +0000 Received: (qmail 11183 invoked by uid 60001); 2 Aug 2010 13:27:57 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1280755677; bh=N9COMFo1aPumiG5SGXKShq7pvRbkVieCnNgRMDteffw=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=tZgUxQ+X7oTvo1ZEqv24GssnFSRXNTdvGyrLbeHPIf6HLkt4yFxtyB1GoxnlnNQ2Rt84+VMqm5DFYOjgI7WE88Q3TqwTGDhIbUyRvxPOxkiR2laLTrN0x3fGLQuvLjw2bslz/sihJTf4ldFlvGTFdplao5fO91yL/b5BkSsEBRI= DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.cn; h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=iwcU3nEV4XTuI/pocwZW7htjya/5yIkcVh7aaMLH1PSjoB1eD/6hMYB2hPZDzF3Z8iBm4Pt51tOZO9fUuQ1m06VdiLh2xXBIqRHV/OHs8oJ8Z/hqsHvLl4DwtppPpZ7UYaiyL6KHjZYrrRvJEdo8DWQ8ZXPorBsyU8POF4e58UM=; Message-ID: <nathanphillips@example.com> X-YMail-OSG: q79VA7YVM1n5lKo5wJvP_AnqLcXGWlk9D9NbHnSqOpvO2li KUgyDEQtK5UXeVfa.GgYh.rA2sVsUcDF3PHJbFIStmWv_tOIniV4kKEqJuW5 P_LsLSXzIgJWuLfVcNSAcLJTNpWnsA5pSW0XJZaXtVb5snHM4BtGkLM2CvzA WnhV39DcU0CzQe_Ul3CjwtM_tG20lABm20or5da_2MVtOT6XqE5kbALr8WpQ 5P.4i9ttPXoCQq0Axj_5gVpvrtOYR0sZsMIDhqdRzoLBoSbm088jDUXmdf9Y iZhXvkgizdWVrRHA- Received: from [83.161.120.155] by web15907.mail.cnb.yahoo.com via HTTP; Mon, 02 Aug 2010 21:27:57 CST X-Mailer: YahooMailRC/420.4 YahooMailWebService/83.161.120.1559950 References: <nathanphillips@example.com> <nathanphillips@example.com> <nathanphillips@example.com> <nathanphillips@example.com> <nathanphillips@example.com> Date: Mon, 2 Aug 2010 21:27:57 +0800 (CST) From: Gang Luo <nathanphillips@example.com> Subject: Re: reuse cached files To: nathanphillips@example.com In-Reply-To: <nathanphillips@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=gb2312 Content-Transfer-Encoding: quoted-printable I am actually doing some test to see the performance. I want to eliminate t= he =0Ainterference of distributed cache. I find there is method in the api = to purge =0Athe cache. That might be what I want.=0A=0AThanks,=0A-Gang=0A= =0A=0A=0A----- =D4=AD=CA=BC=D3=CA=BC=FE ----=0A=B7=A2=BC=FE=C8=CB=A3=BA Hem= anth Yamijala <nathanphillips@example.com>=0A=CA=D5=BC=FE=C8=CB=A3=BA common-user@h= adoop.apache.org=0A=B7=A2=CB=CD=C8=D5=C6=DA=A3=BA 2010/8/2 (=D6=DC=D2=BB) 1= 2:56:25 =C9=CF=CE=E7=0A=D6=F7 =CC=E2=A3=BA Re: reuse cached files=0A=0AHi= ,=0A=0A> Thanks Hemanth. Is there any way to invalidate the reuse and ask H= adoop to=0A> resent exactly the same files to cache for every job?=0A=0AI m= ay be able to answer this better if I understand the use case. If=0Ayou nee= d the same files for every job, why would you need to send them=0Aafresh ea= ch time ? If something is cached, it can be reused, no ? I am=0Asure I must= be missing something in your requirement ...=0A=0AThanks=0AHemanth=0A=0A= =0A=0A From common-user-nathanphillips@example.com Mon Aug 02 13:43:10 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 47815 invoked from network); 2 Aug 2010 13:43:09 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 13:43:09 -0000 Received: (qmail 66506 invoked by uid 500); 2 Aug 2010 13:43:07 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 66107 invoked by uid 500); 2 Aug 2010 13:43:04 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 66099 invoked by uid 99); 2 Aug 2010 13:43:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 13:43:03 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_NEUTRAL,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO n1c.bullet.cnmail.cnb.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 Aug 2010 13:42:53 +0000 Received: from [83.161.120.155] by n1.bullet.cnmail.cnb.yahoo.com with NNFMP; 02 Aug 2010 13:41:53 -0000 Received: from [83.161.120.155] by t4.bullet.cnb.yahoo.com with NNFMP; 02 Aug 2010 13:41:53 -0000 Received: from [83.161.120.155] by omp104.mail.cnb.yahoo.com with NNFMP; 02 Aug 2010 13:41:53 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: nathanphillips@example.com Received: (qmail 17057 invoked by uid 60001); 2 Aug 2010 13:41:53 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.cn; s=s1024; t=1280756513; bh=/DhBXRYALsXUfUFFR/XoxQJsy81HNlw7Ph6Kj6I/aOc=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=GFlxWYK7HIK/IJelQm9HyuQyLG8QaDVqvt3kd9RS7RDxolHLHke8h3rKKHbvbIvmARjos+FB4ir1WgaefyK3CU4c1kkzSmgvhbzs7Zj12LNd0WOy8eXkqbDOhIMucULjMvVbZ8PBER0V9gTin5050wzbX7n4c+1rF6JQvtyzKqI= DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com.cn; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=Pt4v190hGeytv7vqN8FGbLB4RxPDXCAj7bBzTMinPKmh+HDqFPAhpSqaRXF1a0KNBZgm9O+HhHP+SDqXgpXIUNBQSaOub036k/waDwHKZEQ6qrpax3dZVG3WYNDPC42lj4Q+QzY9ABiDUxbL24SnrBSJ2dX/YAx+ZVA9Gm+fErY=; Message-ID: <nathanphillips@example.com> X-YMail-OSG: Yh8oSl8VM1mbhrC8zK36lci5f6R3H7367xUH1o_gBUJdm4F 6nW6VHjHMxxoO8QNpqMMR8J1GK3qL9JIk.kv7YGDJi5593L4jCBsdw9QcVJN AOcurgijuB_3t3vl1WWV08O1Fy4hWFxuRqsCSuJ4j4C3OgPiWn82lPmllktw MOpRZ2AcdgnILoTFOokSbAb.sf3fziQ9HCyBskkY2HrpbVlqWr6DD1Hc5I.w vGizDGUPu_ZbL6TkylvxTW7p99Go6n1PRZ2g8y2J8RQ-- Received: from [83.161.120.155] by web15205.mail.cnb.yahoo.com via HTTP; Mon, 02 Aug 2010 21:41:52 CST X-Mailer: YahooMailClassic/11.3.2 YahooMailWebService/83.161.120.1559950 Date: Mon, 2 Aug 2010 21:41:52 +0800 (CST) From: Dennis <nathanphillips@example.com> Subject: ERROR hdfs.DFSClient: Exception closing file To: nathanphillips@example.com MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-1320198121-1280756512=:17044" X-Virus-Checked: Checked by ClamAV on apache.org --0-1320198121-1280756512=:17044 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hi, there, I am trying to setup "Pseudo-Distributed" Hadoop-0.20.2 on my Ubuntu 9.10 b= y "file:///home/cloud/hadoop/docs/quickstart.html" exactly step by step, ev= erything was fine until I run "bin/hadoop fs -put conf input". The followin= g is the error. Any idea? Thanks Dennis cloud@cloud:~/hadoop$ bin/hadoop fs -put conf input 10/08/02 21:36:38 WARN hdfs.DFSClient: DataStreamer Exception: java.lang.Nu= mberFormatException: For input string: "888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a40" =nathanphillips@example.com(NumberFormatExc= eption.java:48) =nathanphillips@example.com(Integer.java:458) =nathanphillips@example.com(Integer.java:499) =nathanphillips@example.com(NetUtils.java:= 146) =nathanphillips@example.com(NetUtils.java:= 123) =nathanphillips@example.com$DFSOutputStream.createBlockOu= tputStream(DFSClient.java:2867) =nathanphillips@example.com$DFSOutputStream.nextBlockOutp= utStream(DFSClient.java:2826) =nathanphillips@example.com$DFSOutputStream.access$2000(D= FSClient.java:2102) =nathanphillips@example.com$DFSOutputStream$DataStreamer.= run(DFSClient.java:2288) 10/08/02 21:36:38 WARN hdfs.DFSClient: Error Recovery for block blk_-489554= 0561435221238_1037 bad datanode[0] nodes =3D=3D null 10/08/02 21:36:38 WARN hdfs.DFSClient: Could not get block locations. Sourc= e file "/user/cloud/input/mapred-site.xml" - Aborting... put: Could not get block locations. Source file "/user/cloud/input/mapred-s= ite.xml" - Aborting... 10/08/02 21:36:38 ERROR hdfs.DFSClient: Exception closing file /user/cloud/= input/mapred-site.xml : java.io.IOException: Could not get block locations.= Source file "/user/cloud/input/mapred-site.xml" - Aborting... java.io.IOException: Could not get block locations. Source file "/user/clou= d/input/mapred-site.xml" - Aborting... =nathanphillips@example.com$DFSOutputStream.processDatano= deError(DFSClient.java:2536) =nathanphillips@example.com$DFSOutputStream.access$1600(D= FSClient.java:2102) =nathanphillips@example.com$DFSOutputStream$DataStreamer.= run(DFSClient.java:2265) cloud@cloud:~/hadoop$=20 =0A=0A=0A --0-1320198121-1280756512=:17044-- From common-user-nathanphillips@example.com Mon Aug 02 15:39:41 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 1743 invoked from network); 2 Aug 2010 15:39:41 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 15:39:41 -0000 Received: (qmail 30170 invoked by uid 500); 2 Aug 2010 15:39:39 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 30135 invoked by uid 500); 2 Aug 2010 15:39:38 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 30127 invoked by uid 99); 2 Aug 2010 15:39:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 15:39:38 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ey0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 15:39:31 +0000 Received: by eyg5 with SMTP id 5so1922364eyg.35 for <nathanphillips@example.com>; Mon, 02 Aug 2010 08:39:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=MNUmKxaDUbVlXgjN7VQIQt4w8VdDlJRqzzK5Bcfb3cE=; b=bcLZPxKcUApNl5DhmuxKQ/o4jXfwAnwrSmMPxYtEvYSLmv0U7DvWdGxiWv2dl/sk2M upogxN9DFcCjW4IGEhI1XwEF4wcxfR6E7r0iHluFiDqO2Kyepxl3LxeWYs2dUFx8dSo0 kA35wpChyFv4QIQpjWkXckMG/Wa5VXQnqtJjo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=LuM6ClEqLQ0hrxfZ5ktWILy76p9MAnsc2K23LetCvBtuyz1SqWsmsBuG/Ubc9cndin Agt8rHunlPOKlStQsr+i74AwTBYoMvg+icJRf5DSd8p1V58HTyzLzwmK4xvN0ve+38lh 7IpxLGJYN/IgbO2aIEJQCogYqtWu4bv2YF01A= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id q14mr3528810ebd.27.1280763550600; Mon, 02 Aug 2010 08:39:10 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Mon, 2 Aug 2010 08:39:10 -0700 (PDT) Date: Mon, 2 Aug 2010 17:39:10 +0200 Message-ID: <nathanphillips@example.com> Subject: Combiner function From: Jackob Carlsson <nathanphillips@example.com> To: nathanphillips@example.com Content-Type: multipart/alternative; boundary=000e0cd1fbd8798fde048cd900eb --000e0cd1fbd8798fde048cd900eb Content-Type: text/plain; charset=ISO-8859-1 Hi everyone, Could anyone please help me to understand the function of combiner? Thanks in advance Jackob --000e0cd1fbd8798fde048cd900eb-- From common-user-nathanphillips@example.com Mon Aug 02 15:44:12 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 4631 invoked from network); 2 Aug 2010 15:44:11 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 15:44:11 -0000 Received: (qmail 47267 invoked by uid 500); 2 Aug 2010 15:44:09 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 47223 invoked by uid 500); 2 Aug 2010 15:44:09 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 47215 invoked by uid 99); 2 Aug 2010 15:44:08 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 15:44:08 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO VA3EHSOBE004.bigfish.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 15:43:58 +0000 Received: from mail128-va3-R.bigfish.com (83.161.120.155) by VA3EHSOBE004.bigfish.com (83.161.120.155) with Microsoft SMTP Server id 8.1.340.0; Mon, 2 Aug 2010 15:43:27 +0000 Received: from mail128-va3 (localhost.localdomain [83.161.120.155]) by mail128-va3-R.bigfish.com (Postfix) with ESMTP id 3F23A146810A for <nathanphillips@example.com>; Mon, 2 Aug 2010 15:43:21 +0000 (UTC) X-SpamScore: -42 X-BigFish: VPS-42(zz1432N14e3M98dN14ffO936eM1443N9371Pzz1202hzzz32i2a8h61h) X-Spam-TCS-SCL: 0:0 Received: from mail128-va3 (localhost.localdomain [83.161.120.155]) by mail128-va3 (MessageSwitch) id 1280763800305665_26683; Mon, 2 Aug 2010 15:43:20 +0000 (UTC) Received: from VA3EHSMHS018.bigfish.com (unknown [83.161.120.155]) by mail128-va3.bigfish.com (Postfix) with ESMTP id 3CC2B11C004F for <nathanphillips@example.com>; Mon, 2 Aug 2010 15:43:20 +0000 (UTC) Received: from ausb3extmailp02.amd.com (83.161.120.155) by VA3EHSMHS018.bigfish.com (83.161.120.155) with Microsoft SMTP Server (TLS) id 14.0.482.44; Mon, 2 Aug 2010 15:43:19 +0000 Received: from ausb3twp02.amd.com ([83.161.120.155]) by ausb3extmailp02.amd.com (Switch-3.2.7/Switch-3.2.7) with SMTP id o72Fhhn5007061 for <nathanphillips@example.com>; Mon, 2 Aug 2010 10:43:46 -0500 X-WSS-ID: 0L6J7NX-02-8TF-02 X-M-MSG: Received: from sausexhtp01.amd.com (sausexhtp01.amd.com [83.161.120.155]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by ausb3twp02.amd.com (Tumbleweed MailGate 3.7.2) with ESMTP id 20C3EC84D6 for <nathanphillips@example.com>; Mon, 2 Aug 2010 10:43:09 -0500 (CDT) Received: from [83.161.120.155] (83.161.120.155) by sausexhtp01.amd.com (83.161.120.155) with Microsoft SMTP Server id 83.161.120.155; Mon, 2 Aug 2010 10:43:13 -0500 Message-ID: <nathanphillips@example.com> Date: Mon, 2 Aug 2010 10:43:13 -0500 From: Nick Jones <nathanphillips@example.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:83.161.120.155) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5 MIME-Version: 1.0 To: "nathanphillips@example.com" <nathanphillips@example.com> Subject: Re: Combiner function References: <nathanphillips@example.com> In-Reply-To: <nathanphillips@example.com> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Reverse-DNS: ausb3extmailp02.amd.com X-Virus-Checked: Checked by ClamAV on apache.org Hi Jackob, A combiner acts a lot like a reduce step but it's executed on the mapper with in-memory data. I've seen a reduction in job execution time by adding one. The one caveat to keep in mind is that it may or may not run on a particular map attempt. Nick On 8/2/2010 10:39 AM, Jackob Carlsson wrote: > Hi everyone, > Could anyone please help me to understand the function of combiner? > > Thanks in advance > Jackob > From common-user-nathanphillips@example.com Mon Aug 02 15:44:21 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 4708 invoked from network); 2 Aug 2010 15:44:21 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 15:44:21 -0000 Received: (qmail 49348 invoked by uid 500); 2 Aug 2010 15:44:19 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 49303 invoked by uid 500); 2 Aug 2010 15:44:18 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 49295 invoked by uid 99); 2 Aug 2010 15:44:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 15:44:18 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-pw0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 15:44:12 +0000 Received: by pwj2 with SMTP id 2so2675044pwj.35 for <nathanphillips@example.com>; Mon, 02 Aug 2010 08:43:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=8GfvM8kIPRmdPWOYykJwReFoUTNMa5wizGt+Y6tWUC0=; b=Uj125DWwJlmc3mTsoRKJUeTG8km/Fdlwzw/a44u044kBvyniNvmnCDEKzomYMR6A5o U2ymIyfoKlX2T6t7rWLEwsZ+LaFAEUvALOXiTBHT66vZ+AburL4JmToBGjnBmXD9GCdP 71wTq7be9p/axjPFnAqHKYXvk1zuMiaVgIRJw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=SwWWHeRCzfCa07Eo1jmJPiep1Vmyc/zQB+YXCXZzJuDLSoTgxSc1C3QlDX+oWx9SC4 TjiWmfFz5XolyNAZt+3DuA+YPouAjvUwJtgxiiK6KLqjBSjM3iVjYuSv5/JW0NbvJcyp tDyh6mSQ1iQCwqc2IzaXx6Jxm9R6YLyC8Ik8o= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id d4mr4992612wfe.288.1280763831851; Mon, 02 Aug 2010 08:43:51 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Mon, 2 Aug 2010 08:43:51 -0700 (PDT) In-Reply-To: <nathanphillips@example.com> References: <nathanphillips@example.com> Date: Mon, 2 Aug 2010 11:43:51 -0400 Message-ID: <nathanphillips@example.com> Subject: Re: Combiner function From: zaki rahaman <nathanphillips@example.com> To: nathanphillips@example.com Content-Type: multipart/alternative; boundary=000e0cd14bea3d1ccb048cd91194 --000e0cd14bea3d1ccb048cd91194 Content-Type: text/plain; charset=ISO-8859-1 >From the Wiki: http://wiki.apache.org/hadoop/HadoopMapReduce In simple cases, your combiner may simply be your reduce function/code applied to your map output before it's shuffled, sorted, and available for reduce tasks. (This is often the case with counting/simple aggregation). On Mon, Aug 2, 2010 at 11:39 AM, Jackob Carlsson <nathanphillips@example.com>wrote: > Hi everyone, > Could anyone please help me to understand the function of combiner? > > Thanks in advance > Jackob > -- Zaki Rahaman --000e0cd14bea3d1ccb048cd91194-- From common-user-nathanphillips@example.com Mon Aug 02 15:54:19 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 7740 invoked from network); 2 Aug 2010 15:54:19 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 15:54:19 -0000 Received: (qmail 60502 invoked by uid 500); 2 Aug 2010 15:54:16 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 60351 invoked by uid 500); 2 Aug 2010 15:54:16 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 60343 invoked by uid 99); 2 Aug 2010 15:54:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 15:54:16 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-qy0-f169.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 15:54:09 +0000 Received: by qyk12 with SMTP id 12so98056qyk.14 for <nathanphillips@example.com>; Mon, 02 Aug 2010 08:53:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=kAD3yNV9A39JmVugu9Q3L5L9SMA5auxzGh1JwQwmXMs=; b=Gr/kf6w4q0ICbGd0FiADNcsRIBoUtliHDZnaQvbVZMpfOPnWLIoVPwUU6m6+1PZSnL FoF0ci5tJaaF/o5tN6gK3W523KfV7wixHEjzyron6aRTjZ+fcSh8zz0g/bPUMGuRxrMi suu+GOWRBTNd7L9xK0Lt8bSDwuII2AWCtgW4I= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=ww4qdieIU6RCw1H0hmGm0NdyFb5KXtdv2j+AE5yl9UEa3I9jfPlqVPmLMHgElfEi4S pi4LgJFv3RhqqlPdRK0kWflg1E7BXAq3sEJx4Ui6lGcdIznYZlkRK24U4MIQKCdx+VKf 4OnLnK2SKmVG1709eU3B0iQ4M+xE83yM0HFCM= Received: by 83.161.120.155 with SMTP id cb20mr221124qcb.126.1280764428352; Mon, 02 Aug 2010 08:53:48 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with HTTP; Mon, 2 Aug 2010 08:53:28 -0700 (PDT) In-Reply-To: <nathanphillips@example.com> References: <nathanphillips@example.com> From: Harsh J <nathanphillips@example.com> Date: Mon, 2 Aug 2010 21:23:28 +0530 Message-ID: <nathanphillips@example.com> Subject: Re: Combiner function To: nathanphillips@example.com Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org As others have pointed out, its mostly applied as an optimization step. In most cases one's 'Mapper' outputs carry at least a small group of similar keys that go on to the reducer after a copy and a sort phase. To reduce it locally (in-memory) via a 'Combiner' helps reduce data in the copy-sort stages until the 'Reducer' operation kicks-in. Do note that, implementation-wise, a 'combiner' class must always collect the same key-value pair types as the mapper function. On Mon, Aug 2, 2010 at 9:09 PM, Jackob Carlsson <nathanphillips@example.com> wrote: > Hi everyone, > Could anyone please help me to understand the function of combiner? > > Thanks in advance > Jackob > -- Harsh J www.harshj.com From common-user-nathanphillips@example.com Mon Aug 02 16:03:38 2010 Return-Path: <common-user-nathanphillips@example.com> Delivered-To: nathanphillips@example.com Received: (qmail 11284 invoked from network); 2 Aug 2010 16:03:37 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Aug 2010 16:03:37 -0000 Received: (qmail 80723 invoked by uid 500); 2 Aug 2010 16:03:34 -0000 Delivered-To: nathanphillips@example.com Received: (qmail 80694 invoked by uid 500); 2 Aug 2010 16:03:34 -0000 Mailing-List: contact nathanphillips@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:nathanphillips@example.com> List-Unsubscribe: <mailto:nathanphillips@example.com> List-Post: <mailto:nathanphillips@example.com> List-Id: <common-user.hadoop.apache.org> Reply-To: nathanphillips@example.com Delivered-To: mailing list nathanphillips@example.com Received: (qmail 80686 invoked by uid 99); 2 Aug 2010 16:03:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 16:03:34 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nathanphillips@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO edge.omniti.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Aug 2010 16:03:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; d=omniti.com; s=s1024; c=relaxed/relaxed; q=dns/txt; nathanphillips@example.com; t=1280764976; h=From:Subject:Date:To; bh=ZBUdPoTkYjXTQ+PfVQi4OWGgYV50VdOXaTT+oMwJw5E=; b=kc+L+ybaSiUHF5ZAGWfcTsdZCy6s4FprhbsGjzvonpeDxu1C5pP4/cDiC5LcV28F iyKEXnQw+lWkOsLs7oCLVVt0tJJBb8GgE+mQJpxsCrwe6ZATGZ/URB7YoucUylgq 4djgK5NjX5baDGAehIK9BfxWJAqJlACq6u6oTHPpicQ=; Authentication-Results: edge nathanphillips@example.com; auth=pass (LOGIN) Received: from [83.161.120.155] ([83.161.120.155:52270] helo=omniti.com) by edge (envelope-from <nathanphillips@example.com>) (ecelerity 83.161.120.155 r(37468M)) with ESMTPSA (cipher=AES256-SHA) id CA/D9-03560-03CE65C4; Mon, 02 Aug 2010 12:02:56 -0400 Date: Mon, 2 Aug 2010 12:02:53 -0400 From: Jason Dixon <nathanphillips@example.com> To: nathanphillips@example.com Subject: Register now for Surge 2010 Message-ID: <nathanphillips@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) X-Virus-Checked: Checked by ClamAV on apache.org Registration for Surge Scalability Conference 2010 is open for all attendees! We have an awesome lineup of leaders from across the various communities that support highly scalable architectures, as well as the companies that implement them. Here's a small sampling from our list of speakers: John Allspaw, Etsy Theo Schlossnagle, OmniTI Rasmus Lerdorf, creator of PHP Tom Cook, Facebook Benjamin Black, fast_ip Artur Bergman, Wikia Christopher Brown, Opscode Bryan Cantrill, Joyent Baron Schwartz, Percona Paul Querna, Cloudkick Surge 2010 focuses on real case studies from production environments; the lessons learned from failure and how to re-engineer your way to a successful, highly scalable Internet architecture. The conference takes place at the Tremont Grand Historic Venue on Sept 30 and Oct 1, 2010 in Baltimore, MD. Register now to enjoy the Early Bird discount and guarantee your seat
{'source': 'Apache Software Foundation Public Mail Archives'}