From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 05:33:30 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 74311 invoked from network); 1 Feb 2011 05:33:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 05:33:30 -0000 Received: (qmail 12852 invoked by uid 500); 1 Feb 2011 05:33:29 -0000 Delivered-To: browndaniel@example.com Received: (qmail 12328 invoked by uid 500); 1 Feb 2011 05:33:27 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 12253 invoked by uid 99); 1 Feb 2011 05:33:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 05:33:26 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout3.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 05:33:20 +0000 Received: from l2tp-8-19.corp.yahoo.com (l2tp-8-19.corp.yahoo.com [83.161.120.155]) by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p115WmHW011618 for <browndaniel@example.com>; Mon, 31 Jan 2011 21:32:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com; s=cobra; t=1296538368; bh=jIbRtj1HZGQkLioKsXFtK2gwfrLfsMH0VkX9iJf+9fE=; h=Message-Id:From:To:In-Reply-To:Content-Type: Content-Transfer-Encoding:Mime-Version:Subject:Date:References; b=cwzPdAmKGJV5mrdTGkMUq+p5HL4Y9XWb0Db501OA9qKRfzenJt3CwThnJ7VdmIDNG HD+blNNsQj51CBR8qta2o+QZL9B5wWxWvFT1nTrpAdLAJvKUVWMeOYoNfgMcGzPiMf 4FHcRvnm/i/kFsiNhOkpcNbHafzR4CfEfkfhNuOw= Message-Id: <browndaniel@example.com> From: Arun C Murthy <browndaniel@example.com> To: "browndaniel@example.com" <browndaniel@example.com> In-Reply-To: <browndaniel@example.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: Map output in disk and memory at same time? Date: Mon, 31 Jan 2011 21:32:48 -0800 References: <browndaniel@example.com> X-Mailer: Apple Mail (2.936) X-Virus-Checked: Checked by ClamAV on apache.org On Jan 31, 2011, at 10:51 AM, Pedro Costa wrote: > Hi, > > When the reduce fetch from the mappers a map output of the size of 1GB > and do the merge, is it possible that part of the map output is saved > in disk and other part in memory? > Yes, the reduce tries to keep as much in memory as possible. If it's under memory pressure it merges and writes to disk. If there is excess memory it merges into memory. Then there are 3 kinds of merges going on, depending on available memory: memory-to-memory memory-to-disk disk-to-disk The final merge (from memory and/or disk) feeds the 'reduce' function. Arun From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 05:36:48 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 78797 invoked from network); 1 Feb 2011 05:36:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 05:36:48 -0000 Received: (qmail 13907 invoked by uid 500); 1 Feb 2011 05:36:47 -0000 Delivered-To: browndaniel@example.com Received: (qmail 13565 invoked by uid 500); 1 Feb 2011 05:36:45 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 13557 invoked by uid 99); 1 Feb 2011 05:36:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 05:36:44 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout3.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 05:36:38 +0000 Received: from l2tp-8-19.corp.yahoo.com (l2tp-8-19.corp.yahoo.com [83.161.120.155]) by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p115a1C3012545; Mon, 31 Jan 2011 21:36:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=yahoo-inc.com; s=cobra; t=1296538562; bh=fSqhvvqObqfaTwC9r0syaawe1rd2eeLatISGumV5YCE=; h=Cc:Message-Id:From:To:In-Reply-To:Content-Type:Mime-Version: Subject:Date:References; b=YP5b3oxCIuUBJwBjJ/91fJnNFArhSfnv0VDlZdrjQXEd4SwOISjN4LYcYwngDwDPz 58tTaIRTPL28iY3ZvXw+t0FRRASv4a4LRi/7cZC+AhWfPf8KL/W9x45aKWXJOtMOVN 9a+ozkN5fsZjnL03kLKngxOzwJohKMA8pYS0eZFA= Cc: CDH Users <browndaniel@example.com> Message-Id: <browndaniel@example.com> From: Arun C Murthy <browndaniel@example.com> To: "browndaniel@example.com" <browndaniel@example.com>, Kiss Tibor <browndaniel@example.com> In-Reply-To: <browndaniel@example.com> Content-Type: multipart/alternative; boundary=Apple-Mail-173-403781757 Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: What are the conditions or which is the status of re-scheduling feature for failed attempts caused by dying a node? Date: Mon, 31 Jan 2011 21:36:01 -0800 References: <browndaniel@example.com> X-Mailer: Apple Mail (2.936) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-173-403781757 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Please don't cross-post, CDH questions should go to their user lists. On Jan 31, 2011, at 6:15 AM, Kiss Tibor wrote: > Hi! > > I was running a Hadoop cluster on Amazon EC2 instances, then after 2 > days of work, one of the worker nodes just simply died (I cannot > connect to the instance either). That node also appears on the > dfshealth as dead node. > Until now everything is normal. > > Unfortunately the job it was running didn't survived. The cluster it > had 8 worker nodes, each with 4 mappers and 2 reducers. The job in > cause it had ~1200 map tasks and 10 reduce tasks. > One of the node died and I see around 31 failed attempts in the > jobtracker log. The log is very similar with the one somebody > placed it here: http://pastie.org/pastes/1270614 > > Some of the attempts (but not all!) has been retried and I saw at > least two of them which finally is getting in a successful state. > The following two lines appears several times in my jobtracker log: > 2011-01-29 15:50:34,956 WARN org.apache.hadoop.mapred.JobInProgress: > Running list for reducers missing!! Job details are missing. > 2011-01-29 15:50:34,956 WARN org.apache.hadoop.mapred.JobInProgress: > Failed cache for reducers missing!! Job details are missing. > > These pair of log lines could be the signals that the job couldn't > be finished by re-scheduling the failed attempts. > Nothing special I have seen in namenode logs. > > Of course I rerun the failed job which finished successfully. But my > problem is that I would like to understand the failover conditions. > What could be lost, which part of the hadoop is not fault tolerant > in this sense that it happens to see those warnings mentioned > earlier. Is there a chance to control such kind of situations? > > I am using CDH3b3 version, so it is a developing version of Hadoop. > Somebody knows about a special bug or fix which in the near future > can solve the problem? > > Regards > Tibor Kiss > > > --Apple-Mail-173-403781757 Content-Type: text/html; charset=US-ASCII Content-Transfer-Encoding: quoted-printable <html><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space; ">Please don't cross-post, CDH = questions should go to their user lists.<div><br><div><div>On Jan 31, = 2011, at 6:15 AM, Kiss Tibor wrote:</div><br = class=3D"Apple-interchange-newline"><blockquote type=3D"cite">Hi!<br><br>I= was running a Hadoop cluster on Amazon EC2 instances, then after 2 days = of work, one of the worker nodes just simply died (I cannot connect to = the instance either). That node also appears on the dfshealth as dead = node. <br> Until now everything is normal.<br><br>Unfortunately the job = it was running didn't survived. The cluster it had 8 worker nodes, each = with 4 mappers and 2 reducers. The job in cause it had ~1200 map tasks = and 10 reduce tasks. <br> One of the node died and I see around 31 = failed attempts in the jobtracker log.&nbsp; The log is very similar = with the one&nbsp; somebody placed it here: <a = href=3D"http://pastie.org/pastes/1270614">http://pastie.org/pastes/1270614= </a><br> <br>Some of the attempts (but not all!) has been retried and I = saw at least two of them which finally is getting in a successful = state.<br>The following two lines appears several times in my jobtracker = log:<br>2011-01-29 15:50:34,956 WARN = org.apache.hadoop.mapred.JobInProgress: Running list for reducers = missing!! Job details are missing.<br> 2011-01-29 15:50:34,956 WARN = org.apache.hadoop.mapred.JobInProgress: Failed cache for reducers = missing!! Job details are missing.<br><br>These pair of log lines could = be the signals that the job couldn't be finished by re-scheduling the = failed attempts.<br> Nothing special I have seen in namenode = logs.<br><br>Of course I rerun the failed job which finished = successfully. But my problem is that I would like to understand the = failover conditions. What could be lost, which part of the hadoop is not = fault tolerant in this sense that it happens to see those warnings = mentioned earlier. Is there a chance to control such kind of = situations?<br> <br>I am using CDH3b3 version, so it is a developing = version of Hadoop. <br>Somebody knows about a special bug or fix which = in the near future can solve the problem?<br><br>Regards<br>Tibor = Kiss<br><br><br><br></blockquote></div><br></div></body></html>= --Apple-Mail-173-403781757-- From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 11:43:44 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 44363 invoked from network); 1 Feb 2011 11:43:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 11:43:44 -0000 Received: (qmail 59903 invoked by uid 500); 1 Feb 2011 11:43:44 -0000 Delivered-To: browndaniel@example.com Received: (qmail 59543 invoked by uid 500); 1 Feb 2011 11:43:40 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 59530 invoked by uid 99); 1 Feb 2011 11:43:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 11:43:38 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of browndaniel@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-fx0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 11:43:31 +0000 Received: by fxm2 with SMTP id 2so7260226fxm.35 for <browndaniel@example.com>; Tue, 01 Feb 2011 03:43:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=drZP22+ep6jgpou8wfse/xcUQUtMtNLdnzld9ifzJbU=; b=Nw7gv5Tu2x51ALWrDvR5U6NEpXoboLfjNLuaGNDTASjKnbf2RuJINOtl+6iGYzwbEK uV5/XUncCISj9UhoQ84d2bBPlSjQb02awm4cUswBnqZxs59pdpx09XVeEyCqLx0ELbiK LRw4TIpgnlJJ1pTh8fKrdCXd0ye1vCGCljYc0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=FogAIWCoa5xP5qr0+E5/QWu2OA61wv+B6AxE4qILbxcgYIdH80G7DUJ89JAo/GlPtR NlfpyQhMv1+5oDuhNz/h1IdN4a592+SmZhSEvVvhIiJ4h//aMhEwBBOqeexVZdUJ0DeV 8+palbwHMjLFRIy7d5a8ESjEjhd+fTRN9TcOU= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id c12mr1371283fao.16.1296560590816; Tue, 01 Feb 2011 03:43:10 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 1 Feb 2011 03:43:10 -0800 (PST) Date: Tue, 1 Feb 2011 11:43:10 +0000 Message-ID: <browndaniel@example.com> Subject: raw length vs part length From: Pedro Costa <browndaniel@example.com> To: browndaniel@example.com Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org Hi, Hadoop uses the compressed length and the raw length. 1 - In my example, the RT is fetching a map output that shows that it has the raw length of 14 bytes and the partLength of 10 bytes. The map output doesn't use any compression. When I'm dealing with uncompressed data, the raw length should be 14 and the partlength 0? I'm saying this because, the data that is being transferred to the RT is uncompressed. 2 - The raw length of the map output is the size of the block (10 bytes) + header? 3 - part length means partition length? Thanks, -- Pedro From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 12:19:44 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 74245 invoked from network); 1 Feb 2011 12:19:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 12:19:42 -0000 Received: (qmail 14481 invoked by uid 500); 1 Feb 2011 12:19:41 -0000 Delivered-To: browndaniel@example.com Received: (qmail 14144 invoked by uid 500); 1 Feb 2011 12:19:38 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 14113 invoked by uid 99); 1 Feb 2011 12:19:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 12:19:37 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of browndaniel@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-fx0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 12:19:29 +0000 Received: by fxm2 with SMTP id 2so7294782fxm.35 for <browndaniel@example.com>; Tue, 01 Feb 2011 04:19:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=w/46KtDvGlU6Xa4zJgZWjNlmCmZJbuZTjIhcaX7FUig=; b=POmI46nv5DaczJ57PQvFG5H5iTQqb6gQLwr+we/vfKKbx/h9Hj5JJXBCl1J9+QN9sF TWgwUVzGp2wFFbYAFFQLDBbmoUrwGqQ3hQq587mPMoJKAS6Bi6mql9v4lkcROw7dZ+Ae yk9g7L0J+s4VbLVe39ifj/46b4IocFRwmqk10= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=E05k1ZTYkXeqz+IwVzLoD6jkF+9fqCIld+kIllCrlZxThI/qxHXFZsj3bYHR9ToQr1 +ERpjJiJny19H5Y1bHgBNWPp59L56Q9Nh41OPEIQ/pIggRCMW21k832T27vuG27S5U5w j6fgLtfqNsrOR6tb2xSpMRsVlmPEr6NSthV8k= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id a2mr7323508fap.92.1296562749146; Tue, 01 Feb 2011 04:19:09 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 1 Feb 2011 04:19:09 -0800 (PST) In-Reply-To: <browndaniel@example.com> References: <browndaniel@example.com> Date: Tue, 1 Feb 2011 12:19:09 +0000 Message-ID: <browndaniel@example.com> Subject: Fwd: raw length vs part length From: Pedro Costa <browndaniel@example.com> To: browndaniel@example.com Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org I just have to make a correction and add a new question. The correction: PartLength: 14 bytes Raw length: 10 bytes. Added question: Why the class IFileInputStream is created with a size of 14 bytes, if the data (segments) has the size of 10 bytes, despite of all the map-0.out file have the size of 14 bytes? ---------- Forwarded message ---------- From: Pedro Costa <browndaniel@example.com> Date: Tue, Feb 1, 2011 at 11:43 AM Subject: raw length vs part length To: browndaniel@example.com Hi, Hadoop uses the compressed length and the raw length. 1 - In my example, the RT is fetching a map output that shows that it has the raw length of 14 bytes and the partLength of 10 bytes. The map output doesn't use any compression. When I'm dealing with uncompressed data, the raw length should be 14 and the partlength 0? I'm saying this because, the data that is being transferred to the RT is uncompressed. 2 - The raw length of the map output is the size of the block (10 bytes) + header? 3 - part length means partition length? Thanks, -- Pedro -- Pedro From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 16:34:41 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 50466 invoked from network); 1 Feb 2011 16:34:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 16:34:41 -0000 Received: (qmail 8228 invoked by uid 500); 1 Feb 2011 16:34:41 -0000 Delivered-To: browndaniel@example.com Received: (qmail 8106 invoked by uid 500); 1 Feb 2011 16:34:38 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 8098 invoked by uid 99); 1 Feb 2011 16:34:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 16:34:37 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout2-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 16:34:29 +0000 Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [83.161.120.155]) by mrout2-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p11GXpKV064584; Tue, 1 Feb 2011 08:33:51 -0800 (PST) Received: from SP2-EX07VS04.ds.corp.yahoo.com ([83.161.120.155]) by SP2-EX07CAS04.ds.corp.yahoo.com ([83.161.120.155]) with mapi; Tue, 1 Feb 2011 08:33:50 -0800 From: Koji Noguchi <browndaniel@example.com> To: "browndaniel@example.com" <browndaniel@example.com>, Kiss Tibor <browndaniel@example.com> Date: Tue, 1 Feb 2011 08:33:47 -0800 Subject: Re: What are the conditions or which is the status of re-scheduling feature for failed attempts caused by dying a node? Thread-Topic: What are the conditions or which is the status of re-scheduling feature for failed attempts caused by dying a node? Thread-Index: AcvB0jW2SPOAwA6VQVCDeKNr8T9XSgAW5bDk Message-ID: <browndaniel@example.com> In-Reply-To: <browndaniel@example.com> Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C96D75EB67E46knoguchiyahooinccom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_C96D75EB67E46knoguchiyahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable (Bcc CDH alias) > Please don't cross-post, CDH questions should go to their user lists. > Was this CDH specific? Did the job show up as failed on the jobtracker webui? If yes, can you grep a jobtracker log to see something like 2011-02-01 00:06:41,510 INFO org.apache.hadoop.mapred.TaskInProgress: TaskI= nProgress task_201101040441_3330 49_r_000004 has failed 4 times. 2011-02-01 00:06:41,510 INFO org.apache.hadoop.mapred.JobInProgress: Aborti= ng job job_201101040441_333049 2011-02-01 00:06:41,510 INFO org.apache.hadoop.mapred.JobInProgress: Killin= g job 'job_201101040441_333049' which tells you which task failure caused the job to fail. Then you can look at the userlog of those task attempts to see why they fai= led. Ideally this info should show up on the webui. On the other hands, if the job just hang for hours, there's probably a bug = on the framework. Koji On 1/31/11 9:36 PM, "Arun C Murthy" <browndaniel@example.com> wrote: Please don't cross-post, CDH questions should go to their user lists. On Jan 31, 2011, at 6:15 AM, Kiss Tibor wrote: Hi! I was running a Hadoop cluster on Amazon EC2 instances, then after 2 days o= f work, one of the worker nodes just simply died (I cannot connect to the i= nstance either). That node also appears on the dfshealth as dead node. Until now everything is normal. Unfortunately the job it was running didn't survived. The cluster it had 8 = worker nodes, each with 4 mappers and 2 reducers. The job in cause it had ~= 1200 map tasks and 10 reduce tasks. One of the node died and I see around 31 failed attempts in the jobtracker= log. The log is very similar with the one somebody placed it here: http:= //pastie.org/pastes/1270614 Some of the attempts (but not all!) has been retried and I saw at least two= of them which finally is getting in a successful state. The following two lines appears several times in my jobtracker log: 2011-01-29 15:50:34,956 WARN org.apache.hadoop.mapred.JobInProgress: Runnin= g list for reducers missing!! Job details are missing. 2011-01-29 15:50:34,956 WARN org.apache.hadoop.mapred.JobInProgress: Faile= d cache for reducers missing!! Job details are missing. These pair of log lines could be the signals that the job couldn't be finis= hed by re-scheduling the failed attempts. Nothing special I have seen in namenode logs. Of course I rerun the failed job which finished successfully. But my proble= m is that I would like to understand the failover conditions. What could be= lost, which part of the hadoop is not fault tolerant in this sense that it= happens to see those warnings mentioned earlier. Is there a chance to cont= rol such kind of situations? I am using CDH3b3 version, so it is a developing version of Hadoop. Somebody knows about a special bug or fix which in the near future can solv= e the problem? Regards Tibor Kiss --_000_C96D75EB67E46knoguchiyahooinccom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <HTML> <HEAD> <TITLE>Re: What are the conditions or which is the status of re-scheduling = feature for failed attempts caused by dying a node?</TITLE> </HEAD> <BODY> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:= 11pt'>(Bcc CDH alias)<BR> <BR> &gt; Please don't cross-post, CDH questions should go to their user lists.<= BR> &gt;<BR> Was this CDH specific? &nbsp;<BR> <BR> Did the job show up as failed on the jobtracker webui? &nbsp;<BR> If yes, can you grep a jobtracker log to see something like <BR> <BR> 2011-02-01 00:06:41,510 INFO org.apache.hadoop.mapred.TaskInProgress: TaskI= nProgress task_201101040441_3330<BR> 49_r_000004 has failed 4 times.<BR> 2011-02-01 00:06:41,510 INFO org.apache.hadoop.mapred.JobInProgress: Aborti= ng job job_201101040441_333049<BR> 2011-02-01 00:06:41,510 INFO org.apache.hadoop.mapred.JobInProgress: Killin= g job 'job_201101040441_333049'<BR> <BR> which tells you which task failure caused the job to fail.<BR> Then you can look at the userlog of those task attempts to see why they fai= led.<BR> <BR> Ideally this info should show up on the webui. &nbsp;<BR> <BR> On the other hands, if the job just hang for hours, there&#8217;s probably = a bug on the framework.<BR> <BR> Koji<BR> <BR> On 1/31/11 9:36 PM, &quot;Arun C Murthy&quot; &lt;<a href=3D"browndaniel@example.com.= com">browndaniel@example.com</a>&gt; wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"= ><SPAN STYLE=3D'font-size:11pt'>Please don't cross-post, CDH questions shou= ld go to their user lists.<BR> <BR> On Jan 31, 2011, at 6:15 AM, Kiss Tibor wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"= ><SPAN STYLE=3D'font-size:11pt'>Hi!<BR> <BR> I was running a Hadoop cluster on Amazon EC2 instances, then after 2 days o= f work, one of the worker nodes just simply died (I cannot connect to the i= nstance either). That node also appears on the dfshealth as dead node. <BR> &nbsp;Until now everything is normal.<BR> <BR> Unfortunately the job it was running didn't survived. The cluster it had 8 = worker nodes, each with 4 mappers and 2 reducers. The job in cause it had ~= 1200 map tasks and 10 reduce tasks. <BR> &nbsp;One of the node died and I see around 31 failed attempts in the jobtr= acker log. &nbsp;The log is very similar with the one &nbsp;somebody placed= it here: <a href=3D"http://pastie.org/pastes/1270614">http://pastie.org/pa= stes/1270614</a><BR> &nbsp;<BR> Some of the attempts (but not all!) has been retried and I saw at least two= of them which finally is getting in a successful state.<BR> The following two lines appears several times in my jobtracker log:<BR> 2011-01-29 15:50:34,956 WARN org.apache.hadoop.mapred.JobInProgress: Runnin= g list for reducers missing!! Job details are missing.<BR> &nbsp;2011-01-29 15:50:34,956 WARN org.apache.hadoop.mapred.JobInProgress: = Failed cache for reducers missing!! Job details are missing.<BR> <BR> These pair of log lines could be the signals that the job couldn't be finis= hed by re-scheduling the failed attempts.<BR> &nbsp;Nothing special I have seen in namenode logs.<BR> <BR> Of course I rerun the failed job which finished successfully. But my proble= m is that I would like to understand the failover conditions. What could be= lost, which part of the hadoop is not fault tolerant in this sense that it= happens to see those warnings mentioned earlier. Is there a chance to cont= rol such kind of situations?<BR> &nbsp;<BR> I am using CDH3b3 version, so it is a developing version of Hadoop. <BR> Somebody knows about a special bug or fix which in the near future can solv= e the problem?<BR> <BR> Regards<BR> Tibor Kiss<BR> <BR> <BR> <BR> </SPAN></FONT></BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial= "><SPAN STYLE=3D'font-size:11pt'><BR> <BR> </SPAN></FONT></BLOCKQUOTE> </BODY> </HTML> --_000_C96D75EB67E46knoguchiyahooinccom_-- From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 17:58:45 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 91174 invoked from network); 1 Feb 2011 17:58:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 17:58:44 -0000 Received: (qmail 83692 invoked by uid 500); 1 Feb 2011 17:58:44 -0000 Delivered-To: browndaniel@example.com Received: (qmail 82755 invoked by uid 500); 1 Feb 2011 17:58:41 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 82435 invoked by uid 99); 1 Feb 2011 17:58:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 17:58:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of browndaniel@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ew0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 17:58:33 +0000 Received: by ewy20 with SMTP id 20so3249680ewy.35 for <browndaniel@example.com>; Tue, 01 Feb 2011 09:58:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=MWhtM2O4y2I5n4aENfi+pzQi60PPfgFhTxXyiy7G4co=; b=DRQL1Z69JNgFC7/vBNXf+w98YH8fLMnzASxHqjYrWT6fsN7BDV3BfB3aCgkxT4RYbC OQkN9Q3Vu+oFjcxawn5XVLiz9Mp6RuhLpG2HFZdwb7aozpZOYrazT/eUIPQXSuzuKBio S85NQl3pdXy9X52724FTIvWgjeZk7FuQcKlC8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=CjvqYCODjLm0UH4iJWAriYmdP/8HFdD6xuSm78l64Jq87GPulCgM3vcaYPPpqeL/Zl BDnWbft5a5Rm4EENFkFpdbZvib2QVDG/5M2VZa2QRf20blwJMUuD4u2pqhI+pVhQAvN/ NFk16PC7ipvB1AJK0UsI1bgwsvFyBvDr2vCDg= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id v16mr7018354fam.35.1296583093294; Tue, 01 Feb 2011 09:58:13 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 1 Feb 2011 09:58:13 -0800 (PST) Date: Tue, 1 Feb 2011 17:58:13 +0000 Message-ID: <browndaniel@example.com> Subject: map Output packet in Reduce side From: Pedro Costa <browndaniel@example.com> To: browndaniel@example.com Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org When the RT fetch a map output, the RT can save the map output in memory or in disk. 1 - The map output packet is different when it's saved to memory, or to disk? Does the header or the body of the saved map output packet contains different fields? 2 - What are the fields of the packet? Thanks, -- Pedro From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 18:49:40 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 23870 invoked from network); 1 Feb 2011 18:49:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 18:49:40 -0000 Received: (qmail 89959 invoked by uid 500); 1 Feb 2011 18:49:40 -0000 Delivered-To: browndaniel@example.com Received: (qmail 89685 invoked by uid 500); 1 Feb 2011 18:49:38 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 89677 invoked by uid 99); 1 Feb 2011 18:49:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:49:37 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of browndaniel@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:49:28 +0000 Received: by gxk4 with SMTP id 4so2891313gxk.35 for <browndaniel@example.com>; Tue, 01 Feb 2011 10:49:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:content-type:content-transfer-encoding :subject:date:message-id:to:mime-version:x-mailer; bh=BI+shvDXydyInDIax97M1jL74qW4/2rA3gy3TUEf5Y8=; b=Qt4S8dGP0gyZfBc4X7c3zlYgcY1SVlaL+wE/he3GwPkrzSFqmFNVhujFLX10z6iS3d msWQ595S9BNAVig4mj00bBBZrr21egOCeGeba6a7TolZHcR1beFk9xrx0fo4iH9b4kKb haWBwJGKBsxQo4I+vWiw5+hXp4HqEirOOJfFw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:subject:date:message-id :to:mime-version:x-mailer; b=FCPzRJwGorU48f30x89hEAKSIkcbhZfqCBIPd/LaqDI7zRT6lTJAHCBCMaZnzbo0pq Y94/U+zje/lykxYPHO7Xm3MVWc89L/GgUrEC6PGp3UNm9Sa+qnfU1o4Sjm46cuEHsdo7 HkLVUkI5d3eijU0FLyqxU3xouSHkVAIf0e/XY= Received: by 83.161.120.155 with SMTP id j6mr6370489ybd.142.1296586147884; Tue, 01 Feb 2011 10:49:07 -0800 (PST) Received: from [83.161.120.155] (sccc-606.214.9124.smartcity.com [83.161.120.155]) by mx.google.com with ESMTPS id u5sm2849103yba.10.283.161.120.155.49.06 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 01 Feb 2011 10:49:07 -0800 (PST) From: Jeremy Hanna <browndaniel@example.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: elastic mapreduce - custom outputformat? Date: Tue, 1 Feb 2011 10:49:03 -0800 Message-Id: <browndaniel@example.com> To: browndaniel@example.com Mime-Version: 1.0 (Apple Message framework v1082) X-Mailer: Apple Mail (2.1082) X-Virus-Checked: Checked by ClamAV on apache.org I wanted to input from s3 but output to someplace else in aws with = elastic mapreduce. Their docs seem to only suggest that they only read = from/write to s3. Is that correct?= From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 19:13:46 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 51842 invoked from network); 1 Feb 2011 19:13:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 19:13:38 -0000 Received: (qmail 44374 invoked by uid 500); 1 Feb 2011 19:13:37 -0000 Delivered-To: browndaniel@example.com Received: (qmail 44092 invoked by uid 500); 1 Feb 2011 19:13:35 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 44080 invoked by uid 99); 1 Feb 2011 19:13:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:13:34 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of browndaniel@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:13:26 +0000 Received: by bwz8 with SMTP id 8so7425194bwz.35 for <browndaniel@example.com>; Tue, 01 Feb 2011 11:13:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=o7xA51OfAXU/djSCuiN0BwxpC8Vb2PHyK6Jj2zE2EWc=; b=Uu4EwdLFv8TyIs0LPJoMuE26y8lO7uoN5tN1LGTf/pelumWQywM661zBZ7RHm9jtyj PsWJ3K8KC/QBVYyr2V1hBuIyDxAolysIHuN+7sKYnYDlVgUFxd/2nP7jBk0JKVX7V2UZ nOQ907EmIxfpmO2Xrbf1NJFO6+wHnEcx98Ips= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=gnWH02ChSrIsyjBnxcI1sNHggfHKXKLQVpyalUQ+t+8RepCpk+mDQN9w7wBZqOf/vs 19t22f0ciwP7K5sxWKZk2skgjG+xaBNl62SGHUhx0U9FI+q4uznFeQPEtfpJs6lgNBvU F39bjPqnMTstqnL0JqYVdp9D5+qxPDdEYsTts= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id a1mr6147094bkk.5.1296587586334; Tue, 01 Feb 2011 11:13:06 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 1 Feb 2011 11:13:06 -0800 (PST) Date: Tue, 1 Feb 2011 11:13:06 -0800 Message-ID: <browndaniel@example.com> Subject: elastic mapreduce - custom outputformat? From: Jeremy Hanna <browndaniel@example.com> To: browndaniel@example.com Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org I wanted to input from s3 but output to someplace else in aws with elastic mapreduce. Their docs seem to only suggest that they only read from/write to s3. Is that correct? From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 19:22:17 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 61727 invoked from network); 1 Feb 2011 19:22:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 19:22:16 -0000 Received: (qmail 68501 invoked by uid 500); 1 Feb 2011 19:22:16 -0000 Delivered-To: browndaniel@example.com Received: (qmail 68385 invoked by uid 500); 1 Feb 2011 19:22:15 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 68377 invoked by uid 99); 1 Feb 2011 19:22:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:22:15 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout1-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:22:07 +0000 Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [83.161.120.155]) by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p11JLA1Y005004 for <browndaniel@example.com>; Tue, 1 Feb 2011 11:21:10 -0800 (PST) Received: from SP2-EX07VS04.ds.corp.yahoo.com ([83.161.120.155]) by SP2-EX07CAS05.ds.corp.yahoo.com ([83.161.120.155]) with mapi; Tue, 1 Feb 2011 11:21:09 -0800 From: Koji Noguchi <browndaniel@example.com> To: "browndaniel@example.com" <browndaniel@example.com> Date: Tue, 1 Feb 2011 11:21:06 -0800 Subject: Re: elastic mapreduce - custom outputformat? Thread-Topic: elastic mapreduce - custom outputformat? Thread-Index: AcvCREvIdQzFah2mTB+Ob3i0zJ2IqgAAOBnl Message-ID: <browndaniel@example.com> In-Reply-To: <browndaniel@example.com> Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C96D9D2267F88knoguchiyahooinccom_" MIME-Version: 1.0 --_000_C96D9D2267F88knoguchiyahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Input and output filesystems don't have to match. You should be able to read from S3 and write to hdfs as long as you give a = path like s3:// and hdfs:// . Koji On 2/1/11 11:13 AM, "Jeremy Hanna" <browndaniel@example.com> wrote: I wanted to input from s3 but output to someplace else in aws with elastic mapreduce. Their docs seem to only suggest that they only read from/write to s3. Is that correct? --_000_C96D9D2267F88knoguchiyahooinccom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <HTML> <HEAD> <TITLE>Re: elastic mapreduce - custom outputformat?</TITLE> </HEAD> <BODY> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:= 11pt'>Input and output filesystems don&#8217;t have to match.<BR> <BR> You should be able to read from S3 and write to hdfs as long as you give a = path like &nbsp;s3:// and &nbsp;hdfs:// .<BR> <BR> Koji<BR> <BR> <BR> On 2/1/11 11:13 AM, &quot;Jeremy Hanna&quot; &lt;<a href=3D"jeremy.hanna123= browndaniel@example.com">browndaniel@example.com</a>&gt; wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"= ><SPAN STYLE=3D'font-size:11pt'>I wanted to input from s3 but output to som= eplace else in aws with<BR> elastic mapreduce. &nbsp;Their docs seem to only suggest that they only<BR> read from/write to s3. &nbsp;Is that correct?<BR> <BR> </SPAN></FONT></BLOCKQUOTE> </BODY> </HTML> --_000_C96D9D2267F88knoguchiyahooinccom_-- From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 19:25:12 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 62373 invoked from network); 1 Feb 2011 19:25:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 19:25:12 -0000 Received: (qmail 72942 invoked by uid 500); 1 Feb 2011 19:25:11 -0000 Delivered-To: browndaniel@example.com Received: (qmail 72745 invoked by uid 500); 1 Feb 2011 19:25:11 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 72737 invoked by uid 99); 1 Feb 2011 19:25:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:25:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of browndaniel@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-yi0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:25:03 +0000 Received: by yib17 with SMTP id 17so2886177yib.35 for <browndaniel@example.com>; Tue, 01 Feb 2011 11:24:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:content-type:mime-version:subject:from :in-reply-to:date:content-transfer-encoding:message-id:references:to :x-mailer; bh=xCdgPLWWHyjMFRBIYkrpEXz5WYqZViepFic9Y2ghHPI=; b=so4A0Md+PjV0JP5xoSm3JR5ics3jJV9IlVyFLcBzF1uSmsjIFv392LadrHokZoYLpM nEUmUpikIlgi96twNGpC2FLVqOLf5eQ2FlZx21wfyaZEIMuX4UN31pNfYk3fj69vdEm5 f4GfcU1TM214xqtjxPqUWJ0Q6fdFlNcKUILwI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; b=wIdKrweHxy5xH5JBpWUVBZ1YraAgr/W0Oxgv2jdPemXLRNUEeoyUeWVZniKULGleoK Ze6IQl/ktuq+EnqJWUIaNgYinfb2X5U1fy7+hcs3FXCSQYyCmU+cH6mHKGCu4Y2yDhmv alIBB3cb/D5wv8s6pfGYONOQgZLD82skaWluE= Received: by 83.161.120.155 with SMTP id o16mr11815597yan.35.1296588281953; Tue, 01 Feb 2011 11:24:41 -0800 (PST) Received: from [83.161.120.155] (sccc-606.214.9124.smartcity.com [83.161.120.155]) by mx.google.com with ESMTPS id i11sm5445852yhd.6.283.161.120.155.24.39 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 01 Feb 2011 11:24:41 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Apple Message framework v1082) Subject: Re: elastic mapreduce - custom outputformat? From: Jeremy Hanna <browndaniel@example.com> In-Reply-To: <browndaniel@example.com> Date: Tue, 1 Feb 2011 11:24:36 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <browndaniel@example.com> References: <browndaniel@example.com> To: browndaniel@example.com X-Mailer: Apple Mail (2.1082) Thanks! Okay - is there a way to specify a completely different outputformat - I = would like to output to my Cassandra cluster in ec2. On Feb 1, 2011, at 11:21 AM, Koji Noguchi wrote: > Input and output filesystems don=92t have to match. >=20 > You should be able to read from S3 and write to hdfs as long as you = give a path like s3:// and hdfs:// . >=20 > Koji >=20 >=20 > On 2/1/11 11:13 AM, "Jeremy Hanna" <browndaniel@example.com> wrote: >=20 > I wanted to input from s3 but output to someplace else in aws with > elastic mapreduce. Their docs seem to only suggest that they only > read from/write to s3. Is that correct? >=20 From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 21:04:54 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 20573 invoked from network); 1 Feb 2011 21:04:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 21:04:53 -0000 Received: (qmail 80119 invoked by uid 500); 1 Feb 2011 21:04:50 -0000 Delivered-To: browndaniel@example.com Received: (qmail 79222 invoked by uid 500); 1 Feb 2011 21:04:48 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 79163 invoked by uid 99); 1 Feb 2011 21:04:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 21:04:48 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 83.161.120.155 is neither permitted nor denied by domain of browndaniel@example.com) Received: from [83.161.120.155] (HELO mrout3.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 21:04:36 +0000 Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [83.161.120.155]) by mrout3.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id p11L3SPw087714; Tue, 1 Feb 2011 13:03:28 -0800 (PST) Received: from SP2-EX07VS04.ds.corp.yahoo.com ([83.161.120.155]) by SP2-EX07CAS04.ds.corp.yahoo.com ([83.161.120.155]) with mapi; Tue, 1 Feb 2011 13:03:28 -0800 From: Viraj Bhat <browndaniel@example.com> To: "browndaniel@example.com" <browndaniel@example.com>, "browndaniel@example.com" <browndaniel@example.com>, "browndaniel@example.com" <browndaniel@example.com>, "browndaniel@example.com" <browndaniel@example.com>, "browndaniel@example.com" <browndaniel@example.com>, "browndaniel@example.com" <browndaniel@example.com>, "browndaniel@example.com" <browndaniel@example.com>, "browndaniel@example.com" <browndaniel@example.com>, "browndaniel@example.com" <browndaniel@example.com> Date: Tue, 1 Feb 2011 13:02:23 -0800 Subject: CFP Fourth IEEE International Scalable Computing Challenge (SCALE 2011) - Deadline 28 Feb 2011 Thread-Topic: CFP Fourth IEEE International Scalable Computing Challenge (SCALE 2011) - Deadline 28 Feb 2011 Thread-Index: AcvCU1KxTll87Q1xT4aCw13P2r6CSQ== Message-ID: <browndaniel@example.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_45711125D3B37A479CFF56C0589AA1A2026523E8B6SP2EX07VS04ds_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_45711125D3B37A479CFF56C0589AA1A2026523E8B6SP2EX07VS04ds_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all, Please consider submitting to the: The Fourth IEEE International Scalable Computing Challenge (SCALE 2011), sp= onsored by the IEEE Computer Society Technical Committee on Scalable Comput= ing (TCSC). Objective and Focus: The objective of the Fourth IEEE International Scalable Computing Challenge= (SCALE 2011), sponsored by the IEEE Computer Society Technical Committee o= n Scalable Computing (TCSC), is to highlight and showcase real-world proble= m solving using computing that scales. Effective solutions to many scientific problems require applications that c= an scale. There are different dimensions to application scaling: for exampl= e, applications can scale-up to large number of cores or compute units, sca= le-out to utilize multiple distinct compute units, or scale-down to release= resources that are no longer needed. In order to scale, applications need = the support of tools, middleware, infrastructure, programming systems, etc.= SCALE is concerned with advances in application development and supporting= infrastructure that enable scaling. Call for Proposals: The Fourth IEEE International Scalable Computing Challenge (SCALE 2011) con= test will focus on end-to-end problem solving using concepts, technologies = and architectures (including Clusters, Grids and Clouds) that facilitate sc= aling. Participants in the challenge will be expected to identify significa= nt current real-world problems where scalable computing techniques can be e= ffectively used, and design, implement, evaluate and demonstrate solutions.= SCALE2011 will be held in conjunction with the 11th CCGrid Conference in N= ewport Beach, California, USA on 23-26 May, 2011. We invite teams to submit white papers outlining the problem addressed and = the technologies employed to enable applications to scale. White papers sho= uld be up to 4 pages long, 12-pt. font and single column, and in addition t= o listing team members and contact information, should clearly outline: 1. The problem being solved and the technology employed 2. The application scenario and its requirements 3. Performance data and a qualitative description of how the application sc= ales -- scale-up, scale-out or any other type of scaling 4. The solution -- architecture, underlying concepts and technologies used = -- highlighting the innovative aspects of the solution 5. Impact of the solution, including extensibility and uniqueness of result= s, and the extent to which the presented solution pushes the envelope in sc= alable computing 6. Analysis of solution and technology employed compared to related approac= hes In addition to the above, finalists will be judged on the quality of their = presentation, which shall include a 5-minute demonstration, as well as thei= r responses to questions, by a technical committee. Papers will be shortlisted using the above 6 points as merit criteria, and = up to 6 papers will be invited to compete in a final round at CCGrid 2011. = Selected teams will receive an award of up to $1000 to help with travel to = the conference. At least one member from each selected team will be expecte= d to present and demonstrate their project at CCGrid 2011. Participation f= rom students and young researchers, especially in leadership roles, is stro= ngly encouraged. Awards: First prize: Plaque + $1000 Second prize: Plaque + $500 Tentative timeline: The deadline for submitting proposals is 28 Feb, 2011. Decisions: 04 March 2011. Final presentation/demo: 23-26 May, 2011. Coordinator: Shantenu Jha, Rutgers and Louisiana State University, USA, browndaniel@example.com dot = lsu dot edu Regards Viraj --_000_45711125D3B37A479CFF56C0589AA1A2026523E8B6SP2EX07VS04ds_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" xmlns=3D"http://ww= w.w3.org/TR/REC-html40"> <head> <meta http-equiv=3DContent-Type content=3D"text/html; charset=3Dus-ascii"> <meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)"> <o:SmartTagType namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" name=3D"State"/> <o:SmartTagType namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" name=3D"country-region"/> <o:SmartTagType namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" name=3D"City"/> <o:SmartTagType namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" name=3D"place"/> <!--[if !mso]> <style> st1\:*{behavior:url(#default#ieooui) } </style> <![endif]--> <style> <!-- /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman";} a:link, span.MsoHyperlink {color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:Arial; color:windowtext;} @page Section1 {size:8.5in 11.0in; margin:1.0in 1.25in 1.0in 1.25in;} div.Section1 {page:Section1;} --> </style> <!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body lang=3DEN-US link=3Dblue vlink=3Dpurple> <div class=3DSection1> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Hi all,<o:p></o:p></sp= an></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Please consider submit= ting to the:<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>The Fourth IEEE International Scalable Computing Challenge (SCALE 2011), sponsored by the I= EEE Computer Society Technical Committee on Scalable Computing (TCSC).<o:p></o:= p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>&nbsp;<o:p></o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Objective and Focus: <= o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>The objective of the F= ourth IEEE International Scalable Computing Challenge (SCALE 2011), sponsored by = the IEEE Computer Society Technical Committee on Scalable Computing (TCSC), is = to highlight and showcase real-world problem solving using computing that scal= es.<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Effective solutions to= many scientific problems require applications that can scale. There are differen= t dimensions to application scaling: for example, applications can scale-up t= o large number of cores or compute units, scale-out to utilize multiple disti= nct compute units, or scale-down to release resources that are no longer needed= . In order to scale, applications need the support of tools, middleware, infrastructure, programming systems, etc. SCALE is concerned with advances = in application development and supporting infrastructure that enable scaling.<= o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Call for Proposals: <o= :p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>The Fourth IEEE International Scalable Computing Challenge (SCALE 2011) contest will focus = on end-to-end problem solving using concepts, technologies and architectures (including Clusters, Grids and Clouds) that facilitate scaling. Participant= s in the challenge will be expected to identify significant current real-world problems where scalable computing techniques can be effectively used, and design, implement, evaluate and demonstrate solutions. SCALE2011 will be he= ld in conjunction with the 11th CCGrid Conference in <st1:place w:st=3D"on"><s= t1:City w:st=3D"on">Newport Beach</st1:City>, <st1:State w:st=3D"on">California</s= t1:State>, <st1:country-region w:st=3D"on">USA</st1:country-region></st1:place> on 23= -26 May, 2011.<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>We invite teams to sub= mit white papers outlining the problem addressed and the technologies employed = to enable applications to scale. White papers should be up to 4 pages long, 12= -pt. font and single column, and in addition to listing team members and contact information, should clearly outline:<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>1. The problem being s= olved and the technology employed<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>2. The application sce= nario and its requirements<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>3. Performance data an= d a qualitative description of how the application scales -- scale-up, scale-ou= t or any other type of scaling<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>4. The solution -- architecture, underlying concepts and technologies used -- highlighting the innovative aspects of the solution<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>5. Impact of the solut= ion, including extensibility and uniqueness of results, and the extent to which = the presented solution pushes the envelope in scalable computing<o:p></o:p></sp= an></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>6. Analysis of solutio= n and technology employed compared to related approaches<o:p></o:p></span></font>= </p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>In addition to the abo= ve, finalists will be judged on the quality of their presentation, which shall include a 5-minute demonstration, as well as their responses to questions, = by a technical committee.<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Papers will be shortli= sted using the above 6 points as merit criteria, and up to 6 papers will be invi= ted to compete in a final round at CCGrid 2011. Selected teams will receive an award of up to $1000 to help with travel to the conference. At least one me= mber from each selected team will be expected to present and demonstrate their project at CCGrid 2011.&nbsp; Participation from students and young researchers, especially in leadership roles, is strongly encouraged.<o:p></= o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Awards:<o:p></o:p></sp= an></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>First&nbsp; prize:&nbsp;&nbsp; Plaque + $1000<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Second prize:&nbsp;&nb= sp; Plaque + $500<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Tentative timeline:<o:= p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>The deadline for submi= tting proposals is 28 Feb, 2011. <o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Decisions:&nbsp; 04 Ma= rch 2011.<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Final presentation/dem= o: 23-26 May, 2011.<o:p></o:p></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'><o:p>&nbsp;</o:p></spa= n></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Coordinator:<o:p></o:p= ></span></font></p> <p class=3DMsoNormal style=3D'text-autospace:none'><font size=3D2 face=3D"C= ourier New"><span style=3D'font-size:10.0pt;font-family:"Courier New"'>Shantenu Jha, Rutgers = and <st1:place w:st=3D"on"><st1:City w:st=3D"on">Louisiana State University</st1:City>, <s= t1:country-region w:st=3D"on">USA</st1:country-region></st1:place>, browndaniel@example.com dot = edu<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span style=3D'fon= t-size:10.0pt; font-family:"Courier New"'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span style=3D'fon= t-size:10.0pt; font-family:"Courier New"'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span style=3D'fon= t-size:10.0pt; font-family:"Courier New"'>Regards<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3D"Courier New"><span style=3D'fon= t-size:10.0pt; font-family:"Courier New"'>Viraj<o:p></o:p></span></font></p> </div> </body> </html> --_000_45711125D3B37A479CFF56C0589AA1A2026523E8B6SP2EX07VS04ds_-- From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 21:21:55 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 34063 invoked from network); 1 Feb 2011 21:21:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 21:21:54 -0000 Received: (qmail 9816 invoked by uid 500); 1 Feb 2011 21:21:53 -0000 Delivered-To: browndaniel@example.com Received: (qmail 9510 invoked by uid 500); 1 Feb 2011 21:21:53 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 9502 invoked by uid 99); 1 Feb 2011 21:21:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 21:21:52 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-wy0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 21:21:45 +0000 Received: by wye20 with SMTP id 20so7482371wye.35 for <browndaniel@example.com>; Tue, 01 Feb 2011 13:21:24 -0800 (PST) Received: by 83.161.120.155 with SMTP id j59mr8748567wea.39.1296595284402; Tue, 01 Feb 2011 13:21:24 -0800 (PST) MIME-Version: 1.0 Received: by 83.161.120.155 with HTTP; Tue, 1 Feb 2011 13:21:04 -0800 (PST) From: Zachary Kozick <browndaniel@example.com> Date: Tue, 1 Feb 2011 14:21:04 -0700 Message-ID: <browndaniel@example.com> Subject: Hadoop / HDFS equalivant but for realtime request handling / small files? To: browndaniel@example.com Content-Type: multipart/alternative; boundary=0016e6dd89f55829e1049b3f1d40 X-Virus-Checked: Checked by ClamAV on apache.org --0016e6dd89f55829e1049b3f1d40 Content-Type: text/plain; charset=ISO-8859-1 Hi all, I'm interested in creating a solution that leverages multiple computing nodes in an EC2 or Rackspace cloud environment in order to do massively parallelized processing in the context of serving HTTP requests, meaning I want results to be aggregated within 1-4 seconds. >From what I gather, Hadoop is designed for job-oriented tasks and the minimum job completion time is 30 seconds. Also HDFS is meant for storing few large files, as opposed to many small files. My question is there a framework similar to hadoop that is designed more for on-demand parallel computing? What about a technology similar to HDFS that is better at moving around small files and making them available to slave nodes on demand? --0016e6dd89f55829e1049b3f1d40 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable <meta charset=3D"utf-8"><font class=3D"Apple-style-span" face=3D"arial, san= s-serif"><span class=3D"Apple-style-span" style=3D"border-collapse: collaps= e; -webkit-border-horizontal-spacing: 2px; -webkit-border-vertical-spacing:= 2px;">Hi all,</span></font><div> <font class=3D"Apple-style-span" face=3D"arial, sans-serif"><span class=3D"= Apple-style-span" style=3D"border-collapse: collapse; -webkit-border-horizo= ntal-spacing: 2px; -webkit-border-vertical-spacing: 2px;"><br></span></font= ></div> <div><font class=3D"Apple-style-span" face=3D"arial, sans-serif"><span clas= s=3D"Apple-style-span" style=3D"border-collapse: collapse; -webkit-border-h= orizontal-spacing: 2px; -webkit-border-vertical-spacing: 2px;">I&#39;m inte= rested in creating a solution that leverages=A0multiple=A0computing nodes i= n an EC2 or Rackspace cloud environment in order to do=A0massively=A0parall= elized processing in the context of serving HTTP requests, meaning I want r= esults to be aggregated within 1-4 seconds. =A0</span></font></div> <div><font class=3D"Apple-style-span" face=3D"arial, sans-serif"><span clas= s=3D"Apple-style-span" style=3D"border-collapse: collapse; -webkit-border-h= orizontal-spacing: 2px; -webkit-border-vertical-spacing: 2px;"><br></span><= /font></div> <div><font class=3D"Apple-style-span" face=3D"arial, sans-serif"><span clas= s=3D"Apple-style-span" style=3D"border-collapse: collapse; -webkit-border-h= orizontal-spacing: 2px; -webkit-border-vertical-spacing: 2px;">From what I = gather, Hadoop is designed for job-oriented tasks and the minimum job compl= etion time is 30 seconds. =A0Also HDFS is meant for storing few large files= , as opposed to many small files.</span></font></div> <div><font class=3D"Apple-style-span" face=3D"arial, sans-serif"><span clas= s=3D"Apple-style-span" style=3D"border-collapse: collapse; -webkit-border-h= orizontal-spacing: 2px; -webkit-border-vertical-spacing: 2px;"><br></span><= /font></div> <div><font class=3D"Apple-style-span" face=3D"arial, sans-serif"><span clas= s=3D"Apple-style-span" style=3D"border-collapse: collapse; -webkit-border-h= orizontal-spacing: 2px; -webkit-border-vertical-spacing: 2px;">My question = is there a framework similar to hadoop that is designed more for on-demand = parallel computing? =A0What about a technology similar to HDFS that is bett= er at moving around small files and making them available to slave nodes on= demand?</span></font></div> --0016e6dd89f55829e1049b3f1d40-- From mapreduce-user-returnbrowndaniel@example.com Tue Feb 01 21:49:28 2011 Return-Path: <mapreduce-user-returnbrowndaniel@example.com> Delivered-To: browndaniel@example.com Received: (qmail 46491 invoked from network); 1 Feb 2011 21:49:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 21:49:27 -0000 Received: (qmail 66268 invoked by uid 500); 1 Feb 2011 21:49:27 -0000 Delivered-To: browndaniel@example.com Received: (qmail 66100 invoked by uid 500); 1 Feb 2011 21:49:26 -0000 Mailing-List: contact browndaniel@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:browndaniel@example.com> List-Unsubscribe: <mailto:browndaniel@example.com> List-Post: <mailto:browndaniel@example.com> List-Id: <mapreduce-user.hadoop.apache.org> Reply-To: browndaniel@example.com Delivered-To: mailing list browndaniel@example.com Received: (qmail 66092 invoked by uid 99); 1 Feb 2011 21:49:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 21:49:26 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of browndaniel@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ey0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 21:49:20 +0000 Received: by eyz10 with SMTP id 10so3479794eyz.35 for <browndaniel@example.com>; Tue, 01 Feb 2011 13:48:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=cB30fCik1bf885ismits4mqhslWoWVFe131FM9il+Rc=; b=lzCQybcaIPTtGW4nPQBViENf95Iid72597QkRhVP1RXtlaoDqbyITUeAUEzfuwLesb m4acrq2xlVwZ+7oPBb5wSpQrcalwDdLokdFZK3V8O/EzMy3+F8h92jWZ2zKbm/38vMjH EzvOztaGniHAQQGS4HWLKvFrIf28XKtIWgxPg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=G9IOb1COGpxuJ+UQEWkux9qQpKU6iSsVK8CyVxj2kQ82RV85q49/NsCkKRiLrX9r0k F2okl1LOeOF7dawccqRtauAfx4NkmHCnpfrdyqxhhfupFa4V0X6J+HYoyljUUqjdEMaG /ZkZ7WyGk0c+qiaCQ55aq35DBbEjvkTCOJinY= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id y50mr1207304web.28.1296596939654; Tue, 01 Feb 2011 13:48:59 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 1 Feb 2011 13:48:59 -0800 (PST) In-Reply-To: <browndaniel@example.com> References: <browndaniel@example.com> Date: Tue, 1 Feb 2011 13:48:59 -0800 Message-ID: <browndaniel@example.com> Subject: Re: Hadoop / HDFS equalivant but for realtime request handling / small files? From: Russ Ferriday <browndaniel@example.com> To: browndaniel@example.com Content-Type: multipart/alternative; boundary=0016e6db2ca4014835049b3f8063 X-Virus-Checked: Checked by ClamAV on apache.org --0016e6db2ca4014835049b3f8063 Content-Type: text/plain; charset=ISO-8859-1 Hi Zachary, Have you heard of Cassandra? You may be able to write processing nodes accessing data on Cassandra. Probably the easiest configuration is that on each node you have processing functions and a Cassandra node. Then as you expand your computing cluster, you also expand your cassandra bandwidth. This is not optimal, but very practical for a small project/small team. --r On Tue, Feb 1, 2011 at 1:21 PM, Zachary Kozick <browndaniel@example.com> wrote: > Hi all, > > I'm interested in creating a solution that leverages multiple computing > nodes in an EC2 or Rackspace cloud environment in order to > do massively parallelized processing in the context of serving HTTP > requests, meaning I want results to be aggregated within 1-4 seconds. > > From what I gather, Hadoop is designed for job-oriented tasks and the > minimum job completion time is 30 seconds. Also HDFS is meant for storing > few large files, as opposed to many small files. > > My question is there a framework similar to hadoop that is designed more > for on-demand parallel computing? What about a technology similar to HDFS > that is better at moving around small files and making them available to > slave nodes on demand? > --0016e6db2ca4014835049b3f8063 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Zachary,<br><br>Have you heard of Cassandra?<br>You may be able to write= processing nodes accessing data on Cassandra.<br>Probably the easiest conf= iguration is that on each node you have processing functions and a Cassandr= a node.=A0=A0 Then as you expand your computing cluster, you also expand yo= ur cassandra bandwidth.<br> This is not optimal, but very practical for a small project/small team.<br>= --r<br><br><br><div class=3D"gmail_quote">On Tue, Feb 1, 2011 at 1:21 PM, Z= achary Kozick <span dir=3D"ltr">&lt;<a href=3D"mailto:browndaniel@example.com">zach= @omniar.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; borde= r-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;"><font face=3D"ari= al, sans-serif"><span
{'source': 'Apache Software Foundation Public Mail Archives'}