From mapreduce-user-zcarey@example.org Sat Aug 01 01:53:59 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 20879 invoked from network); 1 Aug 2009 01:53:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Aug 2009 01:53:59 -0000 Received: (qmail 23130 invoked by uid 500); 1 Aug 2009 01:54:03 -0000 Delivered-To: zcarey@example.org Received: (qmail 23047 invoked by uid 500); 1 Aug 2009 01:54:02 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 23038 invoked by uid 99); 1 Aug 2009 01:54:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Aug 2009 01:54:02 +0000 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO web31804.mail.mud.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Sat, 01 Aug 2009 01:53:50 +0000 Received: (qmail 82716 invoked by uid 60001); 1 Aug 2009 01:53:28 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.co.in; s=s1024; t=1249091608; bh=zRLe8x4viJnTmg32hrhJuhg8NpfAz94AvbhWPhGXsHE=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=w2HFJ5YcZvPQOu0PTia69BcrAUusoCKG2GY+kVLgkjapqLVL67WqIFbEwyLudBY1zzZZjqSHxD7cTnT6LFos0GFtwCuvcHCzOVBzkC8Y5tF/ao4NN6f9cRSclvcr8k+EwOm2tOfykXazH0/qK6qdnu+zGAcnhZJPv4kQDvOGNZc= DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.co.in; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type; b=qBvDIpbLBmsnSCLrq9N9AmDM9pYSm+kjLCWcc94LaRl72LUIHs0PMGx3MtOfugoieRmuW2HbuYfjsi0J0w9KgakGkc2om1l6O4aoAui0OWeynlRIiaPjpT5XFFwKdkuacCooRrNMgkZS5S1fsZELItvG8ga9a00ES83LkWWO8Lg=; Message-ID: <zcarey@example.org> X-YMail-OSG: tau1OxUVM1mNY18ZrgCN.ec4WrfQ7PhjUZwOMBFBMj72LUGhP1ft0WS1Z5eiFjU3S8ytcbhCBJmW0uwW4GE8pNBSrdZRkrwHjCH.TiLEN5PVyZT_7B5MpfydelmIFoR2eQNZVlzn1N7a.Z4TVviGSxAAb1ipEFb0F9.nwRBPTlAbZNmKZ0USG4FegpNbWNGSVAIS05N5eqeRnHN.FZAka.N32hfPVGlldg37ETruXREaAnz_QNZXUmGO35DIPsPVxmSZCXs80FTaMNa9mu1sg_QE1qvfG.e22XplGM_G8aEmLx7gjLclMYEJA7H70.8CQOQr Received: from [83.161.120.155] by web31804.mail.mud.yahoo.com via HTTP; Fri, 31 Jul 2009 18:53:28 PDT X-Mailer: YahooMailClassic/6.1.2 YahooMailWebService/0.7.338.2 Date: Fri, 31 Jul 2009 18:53:28 -0700 (PDT) From: zcarey@example.org Reply-To: zcarey@example.org Subject: using control characters to seperate key-value To: zcarey@example.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-890464030-1249091608=:82705" X-Virus-Checked: Checked by ClamAV on apache.org --0-890464030-1249091608=:82705 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Hi, =A0=A0 =A0I have a problem trying to customize the key-value seperator of the reducer output: =A0 My Hadoop version: 0.19.2 =A0 I'm trying to get the reducer to use ctrl-A to seperate key value pairs.=20 =A0 conf.set("mapred.textoutputformat.separator","\001"); // also tried conf.set("mapred.textoutputformat.separator","\u0001"); conf.setOutputFormat(TextOutputFormat.class);=20 =A0 =A0 When trying to run the map reduce process, I get the following error.=20 =A0 org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.lang.RuntimeException: org.xml.sax.SAXParseException: Character reference "&#1" is an invalid XML character. =A0=A0 =A0 =A0 As far as I know, xml does not permit control characters. Is there a trick to working around this ? =A0Rajeev --0-890464030-1249091608=:82705 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable <table cellspacing=3D"0" cellpadding=3D"0" border=3D"0" ><tr><td valign=3D"= top" style=3D"font: inherit;">Hi,<br>&nbsp;&nbsp; &nbsp;I have a problem tr= ying to customize the<br>key-value seperator of the reducer output:<br>&nbs= p;<br>My Hadoop version: 0.19.2<br>&nbsp;<br>I'm trying to get the reducer = to use ctrl-A to seperate key<br>value pairs. <br>&nbsp;<br>conf.set("mapre= d.textoutputformat.separator","\001");<br>// also tried<br>conf.set("mapred= .textoutputformat.separator","\u0001");<br>conf.setOutputFormat(TextOutputF= ormat.class); <br>&nbsp;<br>&nbsp;<br>When trying to run the map reduce pro= cess, I get the<br>following error. <br>&nbsp;<br>org.apache.hadoop.ipc.Rem= oteException: java.io.IOException:<br>java.lang.RuntimeException: org.xml.s= ax.SAXParseException:<br>Character reference "&amp;#1" is an invalid XML ch= aracter.<br>&nbsp;&nbsp; &nbsp;<br>&nbsp;<br>As far as I know, xml does not= permit control characters.<br>Is there a trick to working around this ?<br>&nbsp;<div>Rajeev<br><br></div></td></tr></table> --0-890464030-1249091608=:82705-- From mapreduce-user-zcarey@example.org Tue Aug 04 15:04:30 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 11009 invoked from network); 4 Aug 2009 15:04:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 15:04:30 -0000 Received: (qmail 97375 invoked by uid 500); 4 Aug 2009 15:04:35 -0000 Delivered-To: zcarey@example.org Received: (qmail 97295 invoked by uid 500); 4 Aug 2009 15:04:35 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 97286 invoked by uid 99); 4 Aug 2009 15:04:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 15:04:34 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 83.161.120.155 is neither permitted nor denied by domain of zcarey@example.org) Received: from [83.161.120.155] (HELO mail.blacknight.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 15:04:25 +0000 Received: (qmail 29937 invoked from network); 4 Aug 2009 15:03:59 -0000 Received: from unknown (HELO ?83.161.120.155?) (zcarey@example.org@[83.161.120.155]) by 83.161.120.155 with ESMTPS (DHE-RSA-AES256-SHA encrypted); 4 Aug 2009 15:03:59 -0000 Message-ID: <zcarey@example.org> Date: Tue, 04 Aug 2009 16:03:40 +0100 From: Lajos <zcarey@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20090605) MIME-Version: 1.0 To: zcarey@example.org Subject: Question about invoking Jobs Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi all, I'm looking for a way to invoke a Hadoop Job programmatically. Is there any sort of an interface for submitting Jobs to Hadoop besides just the command line or some sort of intermediate file hack? I see from previous posts that this is probably not the case, but has there been any discussion or interest in some sort of solution? Thanks, Lajos From mapreduce-user-zcarey@example.org Wed Aug 05 03:05:09 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 86977 invoked from network); 5 Aug 2009 03:05:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Aug 2009 03:05:09 -0000 Received: (qmail 4854 invoked by uid 500); 5 Aug 2009 03:05:16 -0000 Delivered-To: zcarey@example.org Received: (qmail 4774 invoked by uid 500); 5 Aug 2009 03:05:16 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 4545 invoked by uid 99); 5 Aug 2009 03:05:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 03:05:15 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of zcarey@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-yw0-f202.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Aug 2009 03:05:07 +0000 Received: by ywh40 with SMTP id 40so5666969ywh.29 for <multiple recipients>; Tue, 04 Aug 2009 20:04:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :x-goomoji-body:date:message-id:subject:from:to:cc:content-type; bh=PewwjDCsBpqw5PU8zQLDa+otG4S4X/KkibcewhiVLNM=; b=JlDU2qBPMYjr1jG/wd3UqtMFGP0SVVCLty1Lxf3OQMe8461aUxz9G6XusypOr2adBh 9hxwCZGcqswki4lhqchKrTgBIhepEiQ2OUwCAJOb5ogkJloq25WH4eLlQjhs0UQi/uTp hAJmBcHbGyYPpPtE7/HmYxTvcn48uZr4NRj5E= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:x-goomoji-body:date:message-id :subject:from:to:cc:content-type; b=P9IRZaAViJNvG0FGXVpWaIu2AtltF3VKbOv5scc4A2cXKw9U1QTMumVXKxfQ6/woDA fCkVi5vyirKt06Gbih0VwBiK4+kdJ8hZ6zKis2xYEfQDJS9pj7DFxbZSp3blZmfgGuQ3 NJo0WB3allH29oXBZdLanMN8nyil1LT5N+9Qk= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id o1mr13331888ybg.136.1249441486463; Tue, 04 Aug 2009 20:04:46 -0700 (PDT) In-Reply-To: <zcarey@example.org> References: <zcarey@example.org> X-Goomoji-Body: true Date: Wed, 5 Aug 2009 11:04:46 +0800 Message-ID: <zcarey@example.org> Subject: Re: Re: FYI X-RIME: Hadoop based large scale social network analysis released From: Bin Cai <zcarey@example.org> To: zcarey@example.org Cc: zcarey@example.org, zcarey@example.org, zcarey@example.org, zcarey@example.org Content-Type: multipart/related; boundary=000e0cd2e522f7e56504705c4351 X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd2e522f7e56504705c4351 Content-Type: multipart/alternative; boundary=000e0cd2e522f7e56104705c4350 --000e0cd2e522f7e56104705c4350 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi, Edward J. Yoon Sorry that I found I was not in hama-user maillist. Just joined[?] Xrime is based on Map/Reduce using HDFS to store graph information. It is distributed and parallel. We will release some documents and examples recently. I also noticed project Hamburg. It is interesting. The approach to store graph in HBase would be helpful to solve some issues we found in our solution. Thank you for your attention. Best Regards Cai Bin > That's really cool. BTW, Have you tried these algorithms on the > distributed environment? > > On Mon, Aug 3, 2009 at 6:33 PM, Bin Cai<zcarey@example.org> wrote: > > *X-RIM**E**(http://xrime.sourceforge.net/): Hadoop based large scale social > > network analysis* > > * > > Motivation* > > Today's telecom service providers and Internet-based social network sites > > possess huge user communities. They hold large amount of data about their > > users and want to generate core competency from the data. A key enabler for > > this is a cost efficient solution for social data management and social > > network analysis (SNA). > > > > Such a solution faces a few challenges. The most important one is that the > > solution should be able to handle massive and heterogeneous data sets. > > Facing this challenge, the traditional data warehouse based solutions are > > usually not cost efficient enough. On the other hand, existing SNA tools are > > mostly used in single workstation mode, and not scalable enough. To this > > end, low cost and highly scalable data management and processing > > technologies from cloud computing society should be brought in to help. > > > > However, most of existing cloud based data analysis solutions are trying to > > provide SQL-like general purpose query languages, and do not directly > > support social network analysis. This makes them hard to optimize and hard > > to use for SNA users. So, we came up with X-RIME to fix this gap. > > > > So, briefly speaking, X-RIME wants to provide a few value-added layers on > > top of existing cloud infrastructure, to support smart decision loops based > > on massive data sets and SNA. To end users, X-RIME is a library consists of > > Map-Reduce programs, which are used to do raw data pre-processing, > > transformation, SNA metrics and structures calculation, and graph / network > > visualization. The library could be integrated with other Hadoop based data > > warehouses (e.g., HIVE) to build more comprehensive solutions.> > > > > *Currently Supported SNA Metrics and Structures* > > vertex degree statistics > > weakly connected components (WCC) > > strongly connected components (SCC) > > bi-connected components (BCC) > > ego-centric density > > bread first search / single source shortest path (BFS/SSSP) > > K-core > > maximal cliques > > pagerank > > hyperlink-induced topic search (HITS) > > minimal spanning tree (MST) > > --000e0cd2e522f7e56104705c4350 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <div>Hi, Edward J. Yoon<br>=A0=A0=A0 Sorry that I found I was not in hama-u= ser maillist. Just joined<img style=3D"MARGIN: 0px 0.2ex; VERTICAL-ALIGN: m= iddle" height=3D"12" src=3D"cid:zcarey@example.org" width=3D"12" goomoji=3D"= 330"><br>=A0=A0=A0 Xrime is based on Map/Reduce using HDFS to store graph i= nformation. It is distributed and parallel. We will release some documents = and examples recently.<br> =A0=A0=A0 I also noticed project Hamburg. It is interesting. The approach t= o store graph in HBase would be helpful to solve some issues we found in ou= r solution.<br>=A0=A0=A0 Thank you for your attention.<br>=A0<br>Best Regar= ds<br>Cai Bin</div> <div><pre>&gt; That&#39;s really cool. BTW, Have you tried these algorithms= on the &gt; distributed environment? &gt;=20 &gt; On Mon, Aug 3, 2009 at 6:33 PM, Bin Cai&lt;<a href=3D"mailto:caibinbup= zcarey@example.org" target=3D"_blank">zcarey@example.org</a>&gt; wrote: &gt; &gt; *X-RIM**E**(<a href=3D"http://xrime.sourceforge.net/" target=3D"_= blank">http://xrime.sourceforge.net/</a>): Hadoop based large scale social &gt; &gt; network analysis* &gt; &gt; * &gt; &gt; Motivation* &gt; &gt; Today&#39;s telecom service providers and Internet-based social n= etwork sites &gt; &gt; possess huge user communities. They hold large amount of data abo= ut their &gt; &gt; users and want to generate core competency from the data. A key e= nabler for &gt; &gt; this is a cost efficient solution for social data management and = social &gt; &gt; network analysis (SNA). &gt; &gt; &gt; &gt; Such a solution faces a few challenges. The most important one is= that the &gt; &gt; solution should be able to handle massive and heterogeneous data = sets. &gt; &gt; Facing this challenge, the traditional data warehouse based solut= ions are &gt; &gt; usually not cost efficient enough. On the other hand, existing SN= A tools are &gt; &gt; mostly used in single workstation mode, and not scalable enough. = To this &gt; &gt; end, low cost and highly scalable data management and processing &gt; &gt; technologies from cloud computing society should be brought in to= help. &gt; &gt; &gt; &gt; However, most of existing cloud based data analysis solutions are= trying to &gt; &gt; provide SQL-like general purpose query languages, and do not dire= ctly &gt; &gt; support social network analysis. This makes them hard to optimize= and hard &gt; &gt; to use for SNA users. So, we came up with X-RIME to fix this gap. &gt; &gt; &gt; &gt; So, briefly speaking, X-RIME wants to provide a few value-added l= ayers on &gt; &gt; top of existing cloud infrastructure, to support smart decision l= oops based &gt; &gt; on massive data sets and SNA. To end users, X-RIME is a library c= onsists of &gt; &gt; Map-Reduce programs, which are used to do raw data pre-processing= , &gt; &gt; transformation, SNA metrics and structures calculation, and graph= / network &gt; &gt; visualization. The library could be integrated with other Hadoop = based data &gt; &gt; warehouses (e.g., HIVE) to build more comprehensive solutions.&gt= ;=20 &gt; &gt; &gt; &gt; *Currently Supported SNA Metrics and Structures* &gt; &gt; vertex degree statistics &gt; &gt; weakly connected components (WCC) &gt; &gt; strongly connected components (SCC) &gt; &gt; bi-connected components (BCC) &gt; &gt; ego-centric density &gt; &gt; bread first search / single source shortest path (BFS/SSSP) &gt; &gt; K-core &gt; &gt; maximal cliques &gt; &gt; pagerank &gt; &gt; hyperlink-induced topic search (HITS) &gt; &gt; minimal spanning tree (MST) &gt; &gt; </pre></div> --000e0cd2e522f7e56104705c4350-- --000e0cd2e522f7e56504705c4351 Content-Type: image/gif; name="330.gif" Content-Transfer-Encoding: base64 X-Attachment-Id: zcarey@example.org Content-ID: <zcarey@example.org> R0lGODlhDAAMAKIFAF5LAP/zxAAAANyuAP/gaP///wAAAAAAACH5BAEAAAUALAAAAAAMAAwAAAMl WLPcGjDKFYi9lxKBOaGcF35DhWHamZUW0K4mAbiwWtuf0uxFAgA7 --000e0cd2e522f7e56504705c4351-- From mapreduce-user-zcarey@example.org Thu Aug 06 15:38:03 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 45949 invoked from network); 6 Aug 2009 15:38:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 Aug 2009 15:38:02 -0000 Received: (qmail 45319 invoked by uid 500); 6 Aug 2009 14:58:09 -0000 Delivered-To: zcarey@example.org Received: (qmail 45303 invoked by uid 500); 6 Aug 2009 14:58:09 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 45293 invoked by uid 99); 6 Aug 2009 14:58:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 14:58:09 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of zcarey@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ey-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 14:58:00 +0000 Received: by ey-out-2122.google.com with SMTP id 22so346962eye.35 for <zcarey@example.org>; Thu, 06 Aug 2009 07:57:39 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id d1mr12040084ebd.62.1249570659801; Thu, 06 Aug 2009 07:57:39 -0700 (PDT) Date: Thu, 6 Aug 2009 10:57:39 -0400 Message-ID: <zcarey@example.org> Subject: break the cluster in regions for task-specialization From: Harold Valdivia Garcia <zcarey@example.org> To: zcarey@example.org Content-Type: multipart/alternative; boundary=0015174bdea04c9bac04707a57c3 X-Virus-Checked: Checked by ClamAV on apache.org --0015174bdea04c9bac04707a57c3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi, everyone. How can I split the cluster in subcluster?. I want to break my cluster in regions, for example I'd like to have a region for only sorting, other for only joins, other for only groupby, and so on..... If I submit a join-task the JobTracker should send this job to the join-region. Please. Can anyone give me a hint?. Thanks for all. Harold. --0015174bdea04c9bac04707a57c3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi, everyone. How can I split the cluster in subcluster?. I want to break m= y cluster in regions, for example I&#39;d like to have a region for only so= rting, other for only joins, other for only groupby, and so on..... If I su= bmit a join-task the JobTracker should send this job to the join-region. Pl= ease. Can anyone give me a hint?. Thanks for all.<br> Harold.<br><br> --0015174bdea04c9bac04707a57c3-- From mapreduce-user-zcarey@example.org Thu Aug 06 20:20:29 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 78930 invoked from network); 6 Aug 2009 20:20:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 Aug 2009 20:20:29 -0000 Received: (qmail 68129 invoked by uid 500); 6 Aug 2009 20:20:36 -0000 Delivered-To: zcarey@example.org Received: (qmail 68102 invoked by uid 500); 6 Aug 2009 20:20:36 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 68093 invoked by uid 99); 6 Aug 2009 20:20:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 20:20:36 +0000 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=FS_REPLICA,HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of zcarey@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ew0-f226.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Aug 2009 20:20:28 +0000 Received: by ewy26 with SMTP id 26so1196271ewy.29 for <zcarey@example.org>; Thu, 06 Aug 2009 13:20:07 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id k12mr374355eba.19.1249590007546; Thu, 06 Aug 2009 13:20:07 -0700 (PDT) Date: Thu, 6 Aug 2009 16:20:07 -0400 Message-ID: <zcarey@example.org> Subject: how to read replicated blocks with hdfs api? From: Harold Valdivia Garcia <zcarey@example.org> To: zcarey@example.org Content-Type: multipart/alternative; boundary=0015174bf1fe83c3b504707ed85d X-Virus-Checked: Checked by ClamAV on apache.org --0015174bf1fe83c3b504707ed85d Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi... I was reading the HDFS code, and I can't find a way to read the replicated blocks of a block-file. DFS.getFileBlockLocations returns all blocks of a file File = block-a, block-b, ..... block-n. each of these blocks has its replicated blocks. if for instance the replication factor is 3, how can I retrieve block-a1, block-a2, block-a3 in parallel from my user code? I did read DFSClient, DFSClient.DFInputStream to understand how hadoop retrieves data from blocks, but it is hard. There is no an easy way to do this? -- ****************************************** Harold Dwight Valdivia Garcia Graduate Student M.S Computer Engineering University of Puerto Rico, Mayaguez Campus ****************************************** --0015174bf1fe83c3b504707ed85d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi... I was reading the HDFS code, and I can&#39;t find a way to read the<b= r> replicated blocks of a block-file.<br> <br> DFS.getFileBlockLocations returns all blocks of a file<br> File =3D block-a, block-b, ..... block-n.<br> <br> each of these blocks has its replicated blocks. if for instance the<br> replication factor is 3, how can I retrieve block-a1, block-a2, block-a3 in= <br> parallel from my user code?<br> <br> I did read DFSClient, DFSClient.DFInputStream to understand how hadoop<br> retrieves data from blocks, but it is hard.<br> There is no an easy way to do this?<br>-- <br>*****************************= *************<br>Harold Dwight Valdivia Garcia<br>Graduate Student<br>M.S C= omputer Engineering<br>University of Puerto Rico, Mayaguez Campus<br>******= ************************************<br> --0015174bf1fe83c3b504707ed85d-- From mapreduce-user-zcarey@example.org Fri Aug 07 00:14:20 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 49045 invoked from network); 7 Aug 2009 00:14:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Aug 2009 00:14:20 -0000 Received: (qmail 19207 invoked by uid 500); 7 Aug 2009 00:14:27 -0000 Delivered-To: zcarey@example.org Received: (qmail 19151 invoked by uid 500); 7 Aug 2009 00:14:27 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 19129 invoked by uid 99); 7 Aug 2009 00:14:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 00:14:27 +0000 X-ASF-Spam-Status: No, hits=4.6 required=10.0 tests=FS_REPLICA,HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-yx0-f185.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Aug 2009 00:14:19 +0000 Received: by yxe15 with SMTP id 15so1569267yxe.5 for <zcarey@example.org>; Thu, 06 Aug 2009 17:13:58 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id s15mr369230aga.78.1249604038230; Thu, 06 Aug 2009 17:13:58 -0700 (PDT) In-Reply-To: <zcarey@example.org> References: <zcarey@example.org> From: Todd Lipcon <zcarey@example.org> Date: Thu, 6 Aug 2009 17:13:38 -0700 Message-ID: <zcarey@example.org> Subject: Re: how to read replicated blocks with hdfs api? To: zcarey@example.org Content-Type: multipart/alternative; boundary=00163630f0efcf01dd0470821c53 X-Virus-Checked: Checked by ClamAV on apache.org --00163630f0efcf01dd0470821c53 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit On Thu, Aug 6, 2009 at 1:20 PM, Harold Valdivia Garcia < zcarey@example.org> wrote: > Hi... I was reading the HDFS code, and I can't find a way to read the > replicated blocks of a block-file. > > DFS.getFileBlockLocations returns all blocks of a file > File = block-a, block-b, ..... block-n. > > each of these blocks has its replicated blocks. if for instance the > replication factor is 3, how can I retrieve block-a1, block-a2, block-a3 in > parallel from my user code? > > I did read DFSClient, DFSClient.DFInputStream to understand how hadoop > retrieves data from blocks, but it is hard. > There is no an easy way to do this? Correct - this is not a supported operation. People have discussed doing it, but no one has put in the work to get it done. I think I may have accidentally volunteered to do it at one point, but it hasn't been a priority quite yet - it's an odd mapreduce job that can process data faster than the datanode can serve it. -Todd > > -- > ****************************************** > Harold Dwight Valdivia Garcia > Graduate Student > M.S Computer Engineering > University of Puerto Rico, Mayaguez Campus > ****************************************** > --00163630f0efcf01dd0470821c53 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Thu, Aug 6, 2009 at 1:20 PM, Harold Valdivia Garcia <span dir=3D"ltr">&l= t;<a href=3D"mailto:zcarey@example.org">zcarey@example.org</a>&gt= ;</span> wrote:<br><div class=3D"gmail_quote"><blockquote class=3D"gmail_qu= ote" style=3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0p= t 0.8ex; padding-left: 1ex;"> Hi... I was reading the HDFS code, and I can&#39;t find a way to read the<b= r> replicated blocks of a block-file.<br> <br> DFS.getFileBlockLocations returns all blocks of a file<br> File =3D block-a, block-b, ..... block-n.<br> <br> each of these blocks has its replicated blocks. if for instance the<br> replication factor is 3, how can I retrieve block-a1, block-a2, block-a3 in= <br> parallel from my user code?<br> <br> I did read DFSClient, DFSClient.DFInputStream to understand how hadoop<br> retrieves data from blocks, but it is hard.<br> There is no an easy way to do this?</blockquote><div><br>Correct - this is = not a supported operation. People have discussed doing it, but no one has p= ut in the work to get it done. I think I may have accidentally volunteered = to do it at one point, but it hasn&#39;t been a priority quite yet - it&#39= ;s an odd mapreduce job that can process data faster than the datanode can = serve it.<br> <br>-Todd<br><br>=A0<br></div><blockquote class=3D"gmail_quote" style=3D"bo= rder-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding= -left: 1ex;"><br><font color=3D"#888888">-- <br>***************************= ***************<br> Harold Dwight Valdivia Garcia<br>Graduate Student<br>M.S Computer Engineeri= ng<br>University of Puerto Rico, Mayaguez Campus<br>***********************= *******************<br> </font></blockquote></div><br> --00163630f0efcf01dd0470821c53-- From mapreduce-user-zcarey@example.org Wed Aug 12 09:56:07 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 81428 invoked from network); 12 Aug 2009 09:56:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 12 Aug 2009 09:56:07 -0000 Received: (qmail 27512 invoked by uid 500); 12 Aug 2009 09:56:13 -0000 Delivered-To: zcarey@example.org Received: (qmail 27426 invoked by uid 500); 12 Aug 2009 09:56:13 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 27417 invoked by uid 99); 12 Aug 2009 09:56:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 09:56:13 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of zcarey@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail.vionto.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Aug 2009 09:56:02 +0000 Message-ID: <zcarey@example.org> Date: Wed, 12 Aug 2009 11:55:41 +0200 From: Thomas Thevis <zcarey@example.org> User-Agent: Thunderbird 83.161.120.155 (X11/20090320) MIME-Version: 1.0 To: zcarey@example.org Subject: Scheduling dependent jobs with hadoop-0.20.0 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Hi all, currently, we try to upgrade from hadoop-0.19 to hadoop-0.20. Therefore, we try to get rid of the deprecated JobConf class and switch to org.apache.hadoop.mapreduce.Job where possible. In several projects we have chained jobs which depend on each other. For these, we used the Job and JobControl classes of the org.apache.hadoop.mapred.jobcontrol package which rely on the deprecated JobConf class. How does chaining of dependent jobs work with hadoop-0.20+? Do we still have to use the org.apache.hadoop.mapred.jobcontrol package or is there a better way using only the org.apache.hadoop.mapreduce.Job class? Is there a plan to move org.apache.hadoop.mapred.jobcontrol to a package org.apache.hadoop.mapreduce.jobcontrol in the future? Regards, Thomas -- Thomas Thevis Software Developer ------------------------------------------------------------ vionto GmbH Karl-Marx-Allee 90a, D-10243 Berlin fon 218.435.3799 fax 218.435.3799 web http://www.vionto.com ------------------------------------------------------------ Geschäftsführer: Ralf von Grafenstein, Dr. Martin C. Hirsch Sitz der Gesellschaft: Berlin Amtsgericht Berlin Charlottenburg, HRB 108054B ------------------------------------------------------------ From mapreduce-user-zcarey@example.org Thu Aug 13 13:26:16 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 28624 invoked from network); 13 Aug 2009 13:26:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 13 Aug 2009 13:26:15 -0000 Received: (qmail 73521 invoked by uid 500); 13 Aug 2009 13:26:22 -0000 Delivered-To: zcarey@example.org Received: (qmail 73432 invoked by uid 500); 13 Aug 2009 13:26:22 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 73423 invoked by uid 99); 13 Aug 2009 13:26:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 13:26:21 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO ey-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 13:26:11 +0000 Received: by ey-out-2122.google.com with SMTP id 22so138334eye.35 for <zcarey@example.org>; Thu, 13 Aug 2009 06:25:50 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id d14mr171828wef.30.1250169949370; Thu, 13 Aug 2009 06:25:49 -0700 (PDT) In-Reply-To: <zcarey@example.org> References: <zcarey@example.org> Date: Thu, 13 Aug 2009 14:25:49 +0100 Message-ID: <zcarey@example.org> Subject: Re: Scheduling dependent jobs with hadoop-0.20.0 From: Tom White <zcarey@example.org> To: zcarey@example.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi Thomas, The job control feature has been ported to the new MapReduce API in https://issues.apache.org/jira/browse/MAPREDUCE-368. This will be in 0.21 I think. Cheers, Tom On Wed, Aug 12, 2009 at 10:55 AM, Thomas Thevis<zcarey@example.org> w= rote: > Hi all, > > currently, we try to upgrade from hadoop-0.19 to hadoop-0.20. Therefore, = we > try to get rid of the deprecated JobConf class and switch to > org.apache.hadoop.mapreduce.Job where possible. > In several projects we have chained jobs which depend on each other. For > these, we used the Job and JobControl classes of the > org.apache.hadoop.mapred.jobcontrol package which rely on the deprecated > =A0JobConf class. > How does chaining of dependent jobs work with hadoop-0.20+? Do we still h= ave > to use the org.apache.hadoop.mapred.jobcontrol package or is there a bett= er > way using only the org.apache.hadoop.mapreduce.Job class? > Is there a plan to move org.apache.hadoop.mapred.jobcontrol to a package > org.apache.hadoop.mapreduce.jobcontrol in the future? > > Regards, > Thomas > > -- > Thomas Thevis > Software Developer > ------------------------------------------------------------ > vionto GmbH > Karl-Marx-Allee 90a, D-10243 Berlin > > fon =A0 218.435.3799 > fax =A0 218.435.3799 > web =A0 http://www.vionto.com > ------------------------------------------------------------ > Gesch=E4ftsf=FChrer: Ralf von Grafenstein, Dr. Martin C. Hirsch > Sitz der Gesellschaft: Berlin > Amtsgericht Berlin Charlottenburg, HRB 108054B > ------------------------------------------------------------ > From mapreduce-user-zcarey@example.org Thu Aug 13 17:25:05 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 31791 invoked from network); 13 Aug 2009 17:25:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 13 Aug 2009 17:25:05 -0000 Received: (qmail 43061 invoked by uid 500); 13 Aug 2009 17:25:11 -0000 Delivered-To: zcarey@example.org Received: (qmail 43030 invoked by uid 500); 13 Aug 2009 17:25:11 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 43021 invoked by uid 99); 13 Aug 2009 17:25:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 17:25:11 +0000 X-ASF-Spam-Status: No, hits=1.6 required=10.0 tests=SPF_NEUTRAL,SUBJECT_FUZZY_TION X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout2-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Aug 2009 17:25:01 +0000 Received: from [83.161.120.155] (milkbuild-lm.corp.yahoo.com [83.161.120.155]) by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id n7DHMecK084500 for <zcarey@example.org>; Thu, 13 Aug 2009 10:22:42 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=user-agent:date:subject:from:to:message-id:thread-topic: thread-index:mime-version:content-type:content-transfer-encoding; b=zm8MIYD0GzVNOoF6tYkaXNmljPzNB1zlSrWXNJLZA9eb18eW713Ug38a6DitiKbK User-Agent: Microsoft-Entourage/83.161.120.155515 Date: Thu, 13 Aug 2009 10:22:38 -0700 Subject: org.apache.hadoop.mapreduce.Partitioner access to JobConfiguration From: Julien Le Dem <zcarey@example.org> To: <zcarey@example.org> Message-ID: <zcarey@example.org> Thread-Topic: org.apache.hadoop.mapreduce.Partitioner access to JobConfiguration Thread-Index: AcocOqeoG4biZmFH7EWEJMhQATudsQ== Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org I'm looking at hadoop 0.18 vs 0.20. Why doesn't Partitioner have a setup(JobContext) method ? It seem before it used to be a JobConfigurable. Parameters can be passed statically by the Mapper but it's not ideal. Someone can enlighten me ? Julien From mapreduce-user-zcarey@example.org Fri Aug 14 06:53:28 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 4883 invoked from network); 14 Aug 2009 06:53:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Aug 2009 06:53:28 -0000 Received: (qmail 28681 invoked by uid 500); 14 Aug 2009 06:53:35 -0000 Delivered-To: zcarey@example.org Received: (qmail 28616 invoked by uid 500); 14 Aug 2009 06:53:35 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 28607 invoked by uid 99); 14 Aug 2009 06:53:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 06:53:35 +0000 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-yw0-f172.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 06:53:25 +0000 Received: by ywh2 with SMTP id 2so1828642ywh.2 for <zcarey@example.org>; Thu, 13 Aug 2009 23:53:04 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id x5mr988989agl.28.1250232784257; Thu, 13 Aug 2009 23:53:04 -0700 (PDT) From: Todd Lipcon <zcarey@example.org> Date: Thu, 13 Aug 2009 23:52:44 -0700 Message-ID: <zcarey@example.org> Subject: Reduce input records >> Map output records To: zcarey@example.org Content-Type: multipart/alternative; boundary=0016e6460504fe05e40471148077 X-Virus-Checked: Checked by ClamAV on apache.org --0016e6460504fe05e40471148077 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hey all, Has anyone seen behavior where the number of reduce input records is significantly larger than the number of map output records? There's no combiner involved in the job at hand, and it's not particularly large (250GB in, about the same output). The numbers on one example job are: 2,202,290,092 map input records, 2,198,215,987 map output records, 2,200,081,377 reduce input records. The job in question had no failures or speculative task attempts killed. Running 0.18.3 on JVM 1.6.0u14. Anyone have any thoughts? Could a broken comparator trip up the merge in such a way that it would invent records? I searched JIRA and svn logs but nothing caught my eye. If no one has seen this before I'll keep digging and certainly open a JIRA if I can find some more useful data. Thanks -Todd --0016e6460504fe05e40471148077 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hey all,<br><br>Has anyone seen behavior where the number of reduce input r= ecords is significantly larger than the number of map output records? There= &#39;s no combiner involved in the job at hand, and it&#39;s not particular= ly large (250GB in, about the same output). The numbers on one example job = are: 2,202,290,092 map input records, 2,198,215,987 map output records, 2,2= 00,081,377 reduce input records. The job in question had no failures or spe= culative task attempts killed. Running 0.18.3 on JVM 1.6.0u14.<br> <br>Anyone have any thoughts? Could a broken comparator trip up the merge i= n such a way that it would invent records? I searched JIRA and svn logs but= nothing caught my eye. If no one has seen this before I&#39;ll keep diggin= g and certainly open a JIRA if I can find some more useful data.<br> <br>Thanks<br>-Todd<br> --0016e6460504fe05e40471148077-- From mapreduce-user-zcarey@example.org Fri Aug 14 16:31:10 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 35663 invoked from network); 14 Aug 2009 16:31:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Aug 2009 16:31:10 -0000 Received: (qmail 97832 invoked by uid 500); 14 Aug 2009 16:31:17 -0000 Delivered-To: zcarey@example.org Received: (qmail 97776 invoked by uid 500); 14 Aug 2009 16:31:17 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 97764 invoked by uid 99); 14 Aug 2009 16:31:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:31:17 +0000 X-ASF-Spam-Status: No, hits=1.6 required=10.0 tests=SPF_NEUTRAL,SUBJECT_FUZZY_TION X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 16:31:06 +0000 Received: from [83.161.120.155] (milkbuild-lm.corp.yahoo.com [83.161.120.155]) by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n7EGUHxl082971 for <zcarey@example.org>; Fri, 14 Aug 2009 09:30:17 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=user-agent:date:subject:from:to:message-id:thread-topic: thread-index:in-reply-to:mime-version:content-type:content-transfer-encoding; b=l9Q90uaEWjfImvZDMpM1tzfqsgitq8rpR1aoxyvtj2rgSk8krb2FbFPplCu0hjDa User-Agent: Microsoft-Entourage/83.161.120.155515 Date: Fri, 14 Aug 2009 09:30:17 -0700 Subject: Re: org.apache.hadoop.mapreduce.Partitioner access to JobConfiguration From: Julien Le Dem <zcarey@example.org> To: <zcarey@example.org> Message-ID: <zcarey@example.org> Thread-Topic: org.apache.hadoop.mapreduce.Partitioner access to JobConfiguration Thread-Index: AcocOqeoG4biZmFH7EWEJMhQATudsQAwdo7g In-Reply-To: <zcarey@example.org> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Replying to myself (I should read the archive): On Tue, Jul 14, 2009 at 5:28 AM, Tom White <zcarey@example.org> wrote: > Hi Jianmin, > > Sorry - I (incorrectly) assumed you were using the old API. > Partitioners don't yet work with the new API (see > https://issues.apache.org/jira/browse/MAPREDUCE-565). MAPREDUCE-565 just went in, so new API partitioners will be fixed in 0.20.1, which will likely come out next week. -- Owen On 8/13/09 10:22 AM, "Julien Le Dem" <zcarey@example.org> wrote: > I'm looking at hadoop 0.18 vs 0.20. > Why doesn't Partitioner have a setup(JobContext) method ? > It seem before it used to be a JobConfigurable. > Parameters can be passed statically by the Mapper but it's not ideal. > Someone can enlighten me ? > Julien > > From mapreduce-user-zcarey@example.org Fri Aug 14 19:09:50 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 10476 invoked from network); 14 Aug 2009 19:09:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Aug 2009 19:09:50 -0000 Received: (qmail 14646 invoked by uid 500); 14 Aug 2009 19:09:57 -0000 Delivered-To: zcarey@example.org Received: (qmail 14591 invoked by uid 500); 14 Aug 2009 19:09:57 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 14555 invoked by uid 99); 14 Aug 2009 19:09:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:09:57 +0000 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO qw-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Aug 2009 19:09:49 +0000 Received: by qw-out-2122.google.com with SMTP id 8so549575qwh.35 for <zcarey@example.org>; Fri, 14 Aug 2009 12:09:28 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id y15mr2684445qao.14.1250276968175; Fri, 14 Aug 2009 12:09:28 -0700 (PDT) In-Reply-To: <zcarey@example.org> References: <zcarey@example.org> <zcarey@example.org> From: Aaron Kimball <zcarey@example.org> Date: Fri, 14 Aug 2009 12:09:08 -0700 Message-ID: <zcarey@example.org> Subject: Re: which mailing listto send questions To: zcarey@example.org Content-Type: multipart/alternative; boundary=00c09f986a0d8f1b8e04711eca84 X-Virus-Checked: Checked by ClamAV on apache.org --00c09f986a0d8f1b8e04711eca84 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit To answer more precisely: you should read all three. In theory, mapreduce-specific questions should be directed to mapreduce-user, and hdfs-specific questions should be directed to hdfs-user. common-user is technically just for questions about the "common" code in Hadoop, e.g. the I/O serialization subsystem, things like SequenceFile, etc. In practice, common-user is sort of a catch-all where you'll find lots of questions about all aspects of Hadoop though. - Aaron On Fri, Jul 17, 2009 at 2:46 AM, Harish Mallipeddi < zcarey@example.org> wrote: > David, > > mapreduce-user@ and hdfs-user@ are new mailing lists created just recently > I believe. > > core-user@ has been renamed to common-user@ (as a result there's > significantly more traffic on common-user@). > > - Harish > > > On Fri, Jul 17, 2009 at 11:33 AM, David_ca <zcarey@example.org> wrote: > >> Hi, >> >> I just started following the hadoop discussion. >> I was wondering what is the mailing list to use. >> Is it this one: zcarey@example.org >> or zcarey@example.org? >> >> >> thanks >> David. >> > > > > -- > Harish Mallipeddi > http://blog.poundbang.in > --00c09f986a0d8f1b8e04711eca84 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable To answer more precisely: you should read all three. In theory, mapreduce-s= pecific questions should be directed to mapreduce-user, and hdfs-specific q= uestions should be directed to hdfs-user. common-user is technically just f= or questions about the &quot;common&quot; code in Hadoop, e.g. the I/O seri= alization subsystem, things like SequenceFile, etc. In practice, common-use= r is sort of a catch-all where you&#39;ll find lots of questions about all = aspects of Hadoop though.<br> <br>- Aaron<br><br><br><div class=3D"gmail_quote">On Fri, Jul 17, 2009 at 2= :46 AM, Harish Mallipeddi <span dir=3D"ltr">&lt;<a href=3D"mailto:harish.ma= zcarey@example.org">zcarey@example.org</a>&gt;</span> wrote:<br><b= lockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, 20= 4, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> David,<br><br>mapreduce-user@ and hdfs-user@ are new mailing lists created = just recently I believe.<br><br>core-user@ has been renamed to common-user@= (as a result there&#39;s significantly more traffic on common-user@).<br> <br>- Harish<div><div></div><div class=3D"h5"><br><br><div class=3D"gmail_q= uote">On Fri, Jul 17, 2009 at 11:33 AM, David_ca <span dir=3D"ltr">&lt;<a h= ref=3D"mailto:zcarey@example.org" target=3D"_blank">zcarey@example.org.= com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, = 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> Hi,<br><br>I just started following the hadoop discussion.<br>I was wonderi= ng what is the mailing list to use.<br>Is it this one: <a href=3D"mailto:ma= zcarey@example.org" target=3D"_blank">zcarey@example.org= che.org</a><br> or <a href=3D"mailto:zcarey@example.org" target=3D"_blank">commo= zcarey@example.org</a>?<br> <br><br>thanks <br><font color=3D"#888888">David.<br> </font></blockquote></div><br><br clear=3D"all"><br></div></div><font color= =3D"#888888">-- <br>Harish Mallipeddi<br><a href=3D"http://blog.poundbang.i= n" target=3D"_blank">http://blog.poundbang.in</a><br> </font></blockquote></div><br> --00c09f986a0d8f1b8e04711eca84-- From mapreduce-user-zcarey@example.org Sun Aug 16 02:01:10 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 11392 invoked from network); 16 Aug 2009 02:01:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 16 Aug 2009 02:01:10 -0000 Received: (qmail 54115 invoked by uid 500); 16 Aug 2009 02:01:17 -0000 Delivered-To: zcarey@example.org Received: (qmail 54042 invoked by uid 500); 16 Aug 2009 02:01:16 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 54033 invoked by uid 99); 16 Aug 2009 02:01:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 02:01:16 +0000 X-ASF-Spam-Status: No, hits=-1.8 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of zcarey@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mailout-sf2p.facebook.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 02:01:07 +0000 Received: from mail.thefacebook.com (intlb01.snat.snc1.facebook.com [83.161.120.155] (may be forged)) by pp02.snc1.tfbnw.net (8.14.1/8.14.1) with ESMTP id n7G20YAu024681 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT) for <zcarey@example.org>; Sat, 15 Aug 2009 19:00:34 -0700 Received: from SC-MBXC1.TheFacebook.com ([83.161.120.155]) by sc-hub01.TheFacebook.com ([83.161.120.155]) with mapi; Sat, 15 Aug 2009 19:00:45 -0700 From: Zheng Shao <zcarey@example.org> To: "zcarey@example.org" <zcarey@example.org> Date: Sat, 15 Aug 2009 19:00:41 -0700 Subject: merging multiple mapper's outputs Thread-Topic: merging multiple mapper's outputs Thread-Index: AcoeFIPOyUdXQ4CaSOCwuH8s08YnUgAALD0A Message-ID: <zcarey@example.org> References: <zcarey@example.org> In-Reply-To: <zcarey@example.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-puzzleid: {828B22AF-A60D-461C-81E3-331814BF0454} x-cr-hashedpuzzle: eg8= Aosy E8Gl Fe17 F67m GLX2 GXWy G335 G8b3 HAPJ HIwj H/+L IOD0 IRXG IsxN JTK8;1;bQBhAHAAcgBlAGQAdQBjAGUALQB1AHMAZQByAEAAaABhAGQAbwBvAHAALgBhAHAAYQBjAGgAZQAuAG8AcgBnAA==;Sosha1_v1;7;{828B22AF-A60D-461C-81E3-331814BF0454};egBzAGgAYQBvAEAAZgBhAGMAZQBiAG8AbwBrAC4AYwBvAG0A;Sun, 16 Aug 2009 02:00:41 GMT;bQBlAHIAZwBpAG4AZwAgAG0AdQBsAHQAaQBwAGwAZQAgAG0AYQBwAHAAZQByACcAcwAgAG8AdQB0AHAAdQB0AHMA acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_CD2E0A273BDAAD409C70BB3F48EE6A126CBDFD5D32SCMBXC1TheFac_" MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.8161:2.4.5,1.2.40,4.0.166 definitions=2009-08-15_01:2009-08-11,2009-08-15,2009-08-15 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-0907200000 definitions=main-0908150137 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CD2E0A273BDAAD409C70BB3F48EE6A126CBDFD5D32SCMBXC1TheFac_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Does hadoop have the capability of merging multiple mappers(on the same nod= e) output into a single one, to speed up the shuffling phase? Is there a JI= RA that I can find more information about it? Zheng --_000_CD2E0A273BDAAD409C70BB3F48EE6A126CBDFD5D32SCMBXC1TheFac_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:x=3D"urn:schemas-microsoft-com:office:excel" xmlns:p=3D"urn:schemas-m= icrosoft-com:office:powerpoint" xmlns:a=3D"urn:schemas-microsoft-com:office= :access" xmlns:dt=3D"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s=3D"= uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs=3D"urn:schemas-microsof= t-com:rowset" xmlns:z=3D"#RowsetSchema" xmlns:b=3D"urn:schemas-microsoft-co= m:office:publisher" xmlns:ss=3D"urn:schemas-microsoft-com:office:spreadshee= t" xmlns:c=3D"urn:schemas-microsoft-com:office:component:spreadsheet" xmlns= :odc=3D"urn:schemas-microsoft-com:office:odc" xmlns:oa=3D"urn:schemas-micro= soft-com:office:activation" xmlns:html=3D"http://www.w3.org/TR/REC-html40" = xmlns:q=3D"http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc=3D"http://m= icrosoft.com/officenet/conferencing" xmlns:D=3D"DAV:" xmlns:Repl=3D"http://= schemas.microsoft.com/repl/" xmlns:mt=3D"http://schemas.microsoft.com/share= point/soap/meetings/" xmlns:x2=3D"http://schemas.microsoft.com/office/excel= /2003/xml" xmlns:ppda=3D"http://www.passport.com/NameSpace.xsd" xmlns:ois= =3D"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir=3D"http://= schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds=3D"http://www.w3= .org/2000/09/xmldsig#" xmlns:dsp=3D"http://schemas.microsoft.com/sharepoint= /dsp" xmlns:udc=3D"http://schemas.microsoft.com/data/udc" xmlns:xsd=3D"http= ://www.w3.org/2001/XMLSchema" xmlns:sub=3D"http://schemas.microsoft.com/sha= repoint/soap/2002/1/alerts/" xmlns:ec=3D"http://www.w3.org/2001/04/xmlenc#"= xmlns:sp=3D"http://schemas.microsoft.com/sharepoint/" xmlns:sps=3D"http://= schemas.microsoft.com/sharepoint/soap/" xmlns:xsi=3D"http://www.w3.org/2001= /XMLSchema-instance" xmlns:udcs=3D"http://schemas.microsoft.com/data/udc/so= ap" xmlns:udcxf=3D"http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udc= p2p=3D"http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf=3D"http:/= /schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss=3D"http://sche= mas.microsoft.com/office/2006/digsig-setup" xmlns:dssi=3D"http://schemas.mi= crosoft.com/office/2006/digsig" xmlns:mdssi=3D"http://schemas.openxmlformat= s.org/package/2006/digital-signature" xmlns:mver=3D"http://schemas.openxmlf= ormats.org/markup-compatibility/2006" xmlns:m=3D"http://schemas.microsoft.c= om/office/2004/12/omml" xmlns:mrels=3D"http://schemas.openxmlformats.org/pa= ckage/2006/relationships" xmlns:spwp=3D"http://microsoft.com/sharepoint/web= partpages" xmlns:ex12t=3D"http://schemas.microsoft.com/exchange/services/20= 06/types" xmlns:ex12m=3D"http://schemas.microsoft.com/exchange/services/200= 6/messages" xmlns:pptsl=3D"http://schemas.microsoft.com/sharepoint/soap/Sli= deLibrary/" xmlns:spsl=3D"http://microsoft.com/webservices/SharePointPortal= Server/PublishedLinksService" xmlns:Z=3D"urn:schemas-microsoft-com:" xmlns:= st=3D"&#1;" xmlns=3D"http://www.w3.org/TR/REC-html40"> <head> <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; charset=3Dus-ascii"= > <meta name=3DGenerator content=3D"Microsoft Word 12 (filtered medium)"> <style> <!-- /* Font Definitions */ @font-face {font-family:SimSun; panose-1:2 1 6 0 3 1 1 1 1 1;} @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} @font-face {font-family:"\@SimSun"; panose-1:2 1 6 0 3 1 1 1 1 1;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal; font-family:"Calibri","sans-serif"; color:windowtext;} span.EmailStyle18 {mso-style-type:personal-reply; font-family:"Calibri","sans-serif"; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page Section1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.Section1 {page:Section1;} --> </style> <!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3DEN-US link=3Dblue vlink=3Dpurple> <div class=3DSection1> <p class=3DMsoNormal>Does hadoop have the capability of merging multiple ma= ppers(on the same node) output into a single one, to speed up the shuffling phase?<s= pan style=3D'color:#1F497D'> </span>Is there a JIRA that I can find more inform= ation about it?<o:p></o:p></p> <p class=3DMsoNormal><o:p>&nbsp;</o:p></p> <p class=3DMsoNormal>Zheng<o:p></o:p></p> <p class=3DMsoNormal><o:p>&nbsp;</o:p></p> </div> </body> </html> --_000_CD2E0A273BDAAD409C70BB3F48EE6A126CBDFD5D32SCMBXC1TheFac_-- From mapreduce-user-zcarey@example.org Sun Aug 16 15:42:31 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 52117 invoked from network); 16 Aug 2009 15:42:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 16 Aug 2009 15:42:31 -0000 Received: (qmail 8105 invoked by uid 500); 16 Aug 2009 15:42:38 -0000 Delivered-To: zcarey@example.org Received: (qmail 8025 invoked by uid 500); 16 Aug 2009 15:42:37 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 8016 invoked by uid 99); 16 Aug 2009 15:42:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 15:42:37 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of zcarey@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wa-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 15:42:28 +0000 Received: by wa-out-1112.google.com with SMTP id j32so377963waf.29 for <zcarey@example.org>; Sun, 16 Aug 2009 08:42:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=3a7ICT1G51AZhMLxaI3I4v9pgVM21OuEFL2xEHF1jqk=; b=xstLvsQpprd41lnJ8VwuSLns4ExxImbY83GWxK8u0rnvXoFpBhQFWh6kksIjj3FzFj u0j0u3zfdcSSmYA+/uf/8IiExgY9iqYqRdGTBI8yjYF8TcEvbx5ZDCUzpy3cAYulAE7r CoFtIaXcX6lCiLhg1J1+UJ8Oy4LtC7354Xa1k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=bRHz4uK8RI5W8CBAQIaXgQDpCcJg0ncy5BfjJyU12RRQDTUjbn2ozKKjCh48EW+dPz eXHb2acWcqPuI0zExpPgHIbeXyKF2NUOkzYRCmnxhNM/lk7pgGXvMo2mOoisw2ptwydG 3gmRWT56XhX7uQZa8jeTnA9u3Aal2YoMeQ6oU= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id c13mr3700628wam.65.1250437327195; Sun, 16 Aug 2009 08:42:07 -0700 (PDT) In-Reply-To: <zcarey@example.org> References: <zcarey@example.org> <zcarey@example.org> Date: Sun, 16 Aug 2009 23:42:07 +0800 Message-ID: <zcarey@example.org> Subject: Re: merging multiple mapper's outputs From: Zhong Wang <zcarey@example.org> To: zcarey@example.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org On Sun, Aug 16, 2009 at 10:00 AM, Zheng Shao<zcarey@example.org> wrote: > Does hadoop have the capability of merging multiple mappers(on the same > node) output into a single one, to speed up the shuffling phase? Is there a > JIRA that I can find more information about it? Do you mean outputs from multiple mapper tasks or multiple mapper functions? Could Combiner help? -- Zhong Wang From mapreduce-user-zcarey@example.org Sun Aug 16 22:05:58 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 67287 invoked from network); 16 Aug 2009 22:05:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 16 Aug 2009 22:05:58 -0000 Received: (qmail 18706 invoked by uid 500); 16 Aug 2009 22:06:04 -0000 Delivered-To: zcarey@example.org Received: (qmail 18646 invoked by uid 500); 16 Aug 2009 22:06:04 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 18637 invoked by uid 99); 16 Aug 2009 22:06:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 22:06:04 +0000 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of zcarey@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mailout-sf2p.facebook.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 16 Aug 2009 22:05:56 +0000 Received: from mail.thefacebook.com (intlb01.snat.snc1.facebook.com [83.161.120.155] (may be forged)) by pp02.snc1.tfbnw.net (8.14.1/8.14.1) with ESMTP id n7GM5OYC000543 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT) for <zcarey@example.org>; Sun, 16 Aug 2009 15:05:24 -0700 Received: from SC-MBXC1.TheFacebook.com ([83.161.120.155]) by sc-hub02.TheFacebook.com ([83.161.120.155]) with mapi; Sun, 16 Aug 2009 15:05:35 -0700 From: Zheng Shao <zcarey@example.org> To: "zcarey@example.org" <zcarey@example.org> Date: Sun, 16 Aug 2009 15:05:31 -0700 Subject: RE: merging multiple mapper's outputs Thread-Topic: merging multiple mapper's outputs Thread-Index: AcoeiC98xVLi/HNkRuq7wGjbM/KGgQANSw3w Message-ID: <zcarey@example.org> References: <zcarey@example.org> <zcarey@example.org> <zcarey@example.org> In-Reply-To: <zcarey@example.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.8161:2.4.5,1.2.40,4.0.166 definitions=2009-08-16_04:2009-08-11,2009-08-16,2009-08-16 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-0907200000 definitions=main-0908160156 X-Virus-Checked: Checked by ClamAV on apache.org TXVsdGlwbGUgbWFwcGVyIHRhc2tzLg0KDQpDb21iaW5lciBpcyBpbmRlcGVuZGVudCBmcm9tIHRo aXMgZnVuY3Rpb25hbGl0eSBJIHRoaW5rLiAgQ29tYmluZXIgbWVyZ2VzIHJvd3Mgd2l0aCB0aGUg c2FtZSBrZXkuIEl0IGNhbiB3b3JrIG9uIGJvdGggc2luZ2xlIG1hcHBlciBvdXRwdXQgYW5kIG11 bHRpcGxlIG1hcHBlciBvdXRwdXRzIHRvZ2V0aGVyLg0KDQpaaGVuZw0KLS0tLS1PcmlnaW5hbCBN ZXNzYWdlLS0tLS0NCkZyb206IFpob25nIFdhbmcgW21haWx0bzp3YW5nemhvbmcubmV1QGdtYWls LmNvbV0gDQpTZW50OiBTdW5kYXksIEF1Z3VzdCAxNiwgMjAwOSA4OjQyIEFNDQpUbzogbWFwcmVk dWNlLXVzZXJAaGFkb29wLmFwYWNoZS5vcmcNClN1YmplY3Q6IFJlOiBtZXJnaW5nIG11bHRpcGxl IG1hcHBlcidzIG91dHB1dHMNCg0KT24gU3VuLCBBdWcgMTYsIDIwMDkgYXQgMTA6MDAgQU0sIFpo ZW5nIFNoYW88enNoYW9AZmFjZWJvb2suY29tPiB3cm90ZToNCj4gRG9lcyBoYWRvb3AgaGF2ZSB0 aGUgY2FwYWJpbGl0eSBvZiBtZXJnaW5nIG11bHRpcGxlIG1hcHBlcnMob24gdGhlIHNhbWUNCj4g bm9kZSkgb3V0cHV0IGludG8gYSBzaW5nbGUgb25lLCB0byBzcGVlZCB1cCB0aGUgc2h1ZmZsaW5n IHBoYXNlPyBJcyB0aGVyZSBhDQo+IEpJUkEgdGhhdCBJIGNhbiBmaW5kIG1vcmUgaW5mb3JtYXRp b24gYWJvdXQgaXQ/DQoNCkRvIHlvdSBtZWFuIG91dHB1dHMgZnJvbSBtdWx0aXBsZSBtYXBwZXIg dGFza3Mgb3IgbXVsdGlwbGUgbWFwcGVyDQpmdW5jdGlvbnM/IENvdWxkIENvbWJpbmVyIGhlbHA/ DQoNCg0KDQotLSANClpob25nIFdhbmcNCg== From mapreduce-user-zcarey@example.org Mon Aug 17 12:38:25 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 85069 invoked from network); 17 Aug 2009 12:38:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 17 Aug 2009 12:38:25 -0000 Received: (qmail 38818 invoked by uid 500); 17 Aug 2009 12:38:31 -0000 Delivered-To: zcarey@example.org Received: (qmail 38776 invoked by uid 500); 17 Aug 2009 12:38:31 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 38760 invoked by uid 99); 17 Aug 2009 12:38:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 12:38:31 +0000 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=NO_RDNS_DOTCOM_HELO,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout2.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 12:38:19 +0000 Received: from EGL-EX07CAS01.ds.corp.yahoo.com (egl-ex07cas01.eglbp.corp.yahoo.com [83.161.120.155]) by mrout2.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n7HCaMa3009626 for <zcarey@example.org>; Mon, 17 Aug 2009 05:36:23 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:from:to:date:subject:thread-topic:thread-index: message-id:references:in-reply-to:accept-language: content-language:x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage: content-type:content-transfer-encoding:mime-version; b=eyjHIo0LEJv5ucOKP4jwk+JAxZVCKKz/cDGVMu0P122M43DySDD8UZ+nzUpIt22Y Received: from EGL-EX07VS01.ds.corp.yahoo.com ([83.161.120.155]) by EGL-EX07CAS01.ds.corp.yahoo.com ([83.161.120.155]) with mapi; Mon, 17 Aug 2009 18:06:22 +0530 From: Amogh Vasekar <zcarey@example.org> To: "zcarey@example.org" <zcarey@example.org> Date: Mon, 17 Aug 2009 18:05:17 +0530 Subject: RE: merging multiple mapper's outputs Thread-Topic: merging multiple mapper's outputs Thread-Index: AcoeiC98xVLi/HNkRuq7wGjbM/KGgQANSw3wAB5fqdA= Message-ID: <zcarey@example.org> References: <zcarey@example.org> <zcarey@example.org> <zcarey@example.org> <zcarey@example.org> In-Reply-To: <zcarey@example.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Same amount of data will have to be read and transferred over network, same= file or multiple files. If you do merge to a single file, the S&S phase ac= tually cant start till all mappers have finished, as opposed to fetching ou= tputs from individual mapper tasks which can be as soon as it has finished. Just my two cents. Amogh -----Original Message----- From: Zheng Shao [mailto:zcarey@example.org]=20 Sent: Monday, August 17, 2009 3:36 AM To: zcarey@example.org Subject: RE: merging multiple mapper's outputs Multiple mapper tasks. Combiner is independent from this functionality I think. Combiner merges r= ows with the same key. It can work on both single mapper output and multipl= e mapper outputs together. Zheng -----Original Message----- From: Zhong Wang [mailto:zcarey@example.org]=20 Sent: Sunday, August 16, 2009 8:42 AM To: zcarey@example.org Subject: Re: merging multiple mapper's outputs On Sun, Aug 16, 2009 at 10:00 AM, Zheng Shao<zcarey@example.org> wrote: > Does hadoop have the capability of merging multiple mappers(on the same > node) output into a single one, to speed up the shuffling phase? Is there= a > JIRA that I can find more information about it? Do you mean outputs from multiple mapper tasks or multiple mapper functions? Could Combiner help? --=20 Zhong Wang From mapreduce-user-zcarey@example.org Mon Aug 17 18:53:18 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 65169 invoked from network); 17 Aug 2009 18:53:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 17 Aug 2009 18:53:18 -0000 Received: (qmail 9991 invoked by uid 500); 17 Aug 2009 18:53:25 -0000 Delivered-To: zcarey@example.org Received: (qmail 9922 invoked by uid 500); 17 Aug 2009 18:53:25 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 9913 invoked by uid 99); 17 Aug 2009 18:53:25 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 18:53:25 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of zcarey@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-yx0-f185.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 18:53:17 +0000 Received: by yxe15 with SMTP id 15so4243381yxe.5 for <zcarey@example.org>; Mon, 17 Aug 2009 11:52:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=MNT0gMCHQoCC1G/FjmFSwLh9H77EXNxz0IwA09M00dI=; b=xU5fAFsBR3QhGJS2FLjzj7yUJ3reOjgK9A+4msgkzDA/QO6JcsI0TdQGEYOAJfMK4E 7ZxnUbqzTrXpkczTUTb7oKl18jL4/NeY8MZUlSkGWaxRg2PmVfCQKZ3K4gEsebHqIrJg S/O++fD1/1ruBIEoTu3InmZkVnCGk4YNJenMQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=DAIR3PH+0DtIt82Uio1kqhPlHJ1dgO0CeS2RA76lJ8qxxXyVLNm4Cww2HJiPwkJOFc yZa5Zzn6BKP9sEOEqAxVsRC+AhWAdbPdWognuNMwQgP/GGVIAY4aNtFvOnWxCnafIE6m oi6l5XsZgFiCyiYgZjox+5+Q0gEB39ArXe+0Q= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id y12mr2539573ibd.6.1250535176031; Mon, 17 Aug 2009 11:52:56 -0700 (PDT) Date: Mon, 17 Aug 2009 11:52:56 -0700 Message-ID: <zcarey@example.org> Subject: Shuffle phase not starting until 100% of maps are done? From: George Porter <zcarey@example.org> To: zcarey@example.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi everyone, I've come across a problem running Map/Reduce on an EC2 cluster, and was wondering if anyone here had any thoughts to what the issue was. I'm running a simple 'sort' M/R job on 40GB from the examples JAR on Hadoop 19.0 (using the Hadoop 19.0 AMI for Amazon EC2 on Extra-large images). When I run the sort job on a 4 or 16 node cluster, things work fine, and I notice that the shuffle phase begins when approx 45-50% of the maps are completed. However, when I run the sort job on an 8-node cluster, the shuffle doesn't begin until 100% of the maps are done. This causes the 8 node cluster to run much slower than I would have thought. There are over 2000 map tasks, and 16 map slots across those 8 nodes, and so a lot of map tasks have finished before the shuffle starts. Any thoughts on what would be delaying the start of the shuffle phase? Thanks, George From mapreduce-user-zcarey@example.org Mon Aug 17 19:21:51 2009 Return-Path: <mapreduce-user-zcarey@example.org> Delivered-To: zcarey@example.org Received: (qmail 72294 invoked from network); 17 Aug 2009 19:21:51 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 17 Aug 2009 19:21:51 -0000 Received: (qmail 32946 invoked by uid 500); 17 Aug 2009 19:21:57 -0000 Delivered-To: zcarey@example.org Received: (qmail 32891 invoked by uid 500); 17 Aug 2009 19:21:57 -0000 Mailing-List: contact zcarey@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:zcarey@example.org> List-Unsubscribe: <mailto:zcarey@example.org> List-Post: <mailto:zcarey@example.org> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: zcarey@example.org Delivered-To: mailing list zcarey@example.org Received: (qmail 32882 invoked by uid 99); 17 Aug 2009 19:21:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Aug 2009 19:21:57 +0000 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF:
{'source': 'Apache Software Foundation Public Mail Archives'}