From chukwa-user-returndustinhughes@example.org Fri Feb 04 17:34:17 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 26156 invoked from network); 4 Feb 2011 17:34:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Feb 2011 17:34:17 -0000 Received: (qmail 56918 invoked by uid 500); 4 Feb 2011 17:34:17 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 56783 invoked by uid 500); 4 Feb 2011 17:34:14 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Delivered-To: moderator for dustinhughes@example.org Received: (qmail 40104 invoked by uid 500); 4 Feb 2011 14:29:22 -0000 Delivered-To: dustinhughes@example.org X-ASF-Spam-Status: No, hits=2.3 required=5.0 tests=SPF_SOFTFAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: softfail (athena.apache.org: transitioning domain of dustinhughes@example.org does not designate 83.161.120.155 as permitted sender) Date: Fri, 4 Feb 2011 06:28:55 -0800 (PST) From: Arati Mahimane <dustinhughes@example.org> To: dustinhughes@example.org Message-ID: <dustinhughes@example.org> Subject: Query regarding adaptor for binary files MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi All, I want to send Binary files (larger than 10 MB) from Chukwa agent to collector. I have the following queries: 1) I read that the filetailer.LWFTAdaptor repeatedly tails a file, treating the file as a sequence of bytes, ignoring the content. But the Chunk boundaries are arbitrary. Is there any adaptor in which we can specify the chunk size ? If yes, how can the chunk size be configured? (e.g. I have a 10 MB file and need to divide it into 10 chunks of 1 MB each and send each chunk from agent to collector) 2) When binary files are sent, does chukwa ensure that they are not corrupted in anyway (i.e. Do the contents of the file remain unaltered) ? Any pointers ? Thanks, Arati Mahimane -- View this message in context: http://apache-chukwa.679492.n3.nabble.com/Query-regarding-adaptor-for-binary-files-tp2424540p2424540.html Sent from the Chukwa - Users mailing list dustinhughes@example.org. From chukwa-user-returndustinhughes@example.org Fri Feb 04 20:54:47 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 31974 invoked from network); 4 Feb 2011 20:54:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Feb 2011 20:54:47 -0000 Received: (qmail 55445 invoked by uid 500); 4 Feb 2011 20:54:47 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 55383 invoked by uid 500); 4 Feb 2011 20:54:46 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 55376 invoked by uid 99); 4 Feb 2011 20:54:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Feb 2011 20:54:46 +0000 X-ASF-Spam-Status: No, hits=0.6 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dustinhughes@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ww0-f43.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Feb 2011 20:54:41 +0000 Received: by wwi17 with SMTP id 17so2716371wwi.0 for <dustinhughes@example.org>; Fri, 04 Feb 2011 12:54:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=ksC2xroCzW2d3iWlXCrKVHUvSY6Z86lR3gHZD/DvUew=; b=MVaZHa5qU7UrrkrxCcpnK8kpMCuH0+fnBbaF7DVw8ERySjViKMkJ9dubRAeW6XEZ8/ cUJmqbUIBoGWBm+OJPNHWhyIlm0O+oVI9qPATE9hskiIcXrbd36/3zmOBsv55Ogp4bVF YFsL2+jdtZQ/XMSwHYjy4Z3LDq8tXafyRnKb8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=EYZxQ5pWsYTyJyy8pzpXfObN1kBXsdioC4+r4oqiR3UGNeJgHmxuJvXVZW7I28HTvZ j52kGIMKvR/YWaxfoyD7pOdeUIEOJfi1DnsvQBScfmTzIyTvdd+6sO8QqcbqeLbwtxdX yCp8tHK+Kz5s2DqgYOt9amKFG4fL9fjPgoR3Q= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id o2mr582939wbu.105.1296852859964; Fri, 04 Feb 2011 12:54:19 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 4 Feb 2011 12:54:19 -0800 (PST) In-Reply-To: <dustinhughes@example.org> References: <dustinhughes@example.org> Date: Fri, 4 Feb 2011 12:54:19 -0800 Message-ID: <dustinhughes@example.org> Subject: Re: Query regarding adaptor for binary files From: Ariel Rabkin <dustinhughes@example.org> To: dustinhughes@example.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Howdy. Yes, the chunk size is configurable. I think chukwaAgent.fileTailingAdaptor.maxReadSize is the option you want to change. Chukwa should not alter any of your data and should send every byte at least once, appropriately tagged so that you can recover the exact original file. --Ari On Fri, Feb 4, 2011 at 6:28 AM, Arati Mahimane <dustinhughes@example.org> wrote: > > Hi All, > > I want to send Binary files (larger than 10 MB) from Chukwa agent to > collector. > I have the following queries: > > 1) I read that the filetailer.LWFTAdaptor =A0repeatedly tails a file, tre= ating > the file as a sequence of bytes, ignoring the content. But the Chunk > boundaries are arbitrary. Is there any adaptor in which we can specify th= e > chunk size ? If yes, how can the chunk size be configured? > (e.g. I have a 10 MB file and need to divide it into 10 chunks of 1 MB ea= ch > and send each chunk from agent to collector) > > 2) When binary files are sent, does chukwa ensure that they are not > corrupted in anyway (i.e. Do the contents of the file remain unaltered) ? > > Any pointers ? > > Thanks, > Arati Mahimane > > -- > View this message in context: http://apache-chukwa.679492.n3.nabble.com/Q= uery-regarding-adaptor-for-binary-files-tp2424540p2424540.html > Sent from the Chukwa - Users mailing list dustinhughes@example.org. > --=20 Ari Rabkin dustinhughes@example.org UC Berkeley Computer Science Department From chukwa-user-returndustinhughes@example.org Wed Feb 09 19:10:54 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 66750 invoked from network); 9 Feb 2011 19:10:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 9 Feb 2011 19:10:54 -0000 Received: (qmail 52372 invoked by uid 500); 9 Feb 2011 19:10:52 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 50982 invoked by uid 500); 9 Feb 2011 19:10:48 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 50975 invoked by uid 500); 9 Feb 2011 19:10:47 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 50972 invoked by uid 99); 9 Feb 2011 19:10:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 19:10:47 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dustinhughes@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail1.swva.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Feb 2011 19:10:38 +0000 X-Default-Received-SPF: pass (skip=forwardok (res=PASS)) x-ip-name=83.161.120.155; Received: from [83.161.120.155] (unverified [83.161.120.155]) by swva.net (SurgeMail 5.0e4) with ESMTP id 59160883-1865996 for <dustinhughes@example.org>; Wed, 09 Feb 2011 14:10:17 -0500 From: Paul Tremblett <dustinhughes@example.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Chukwa Demux Priority Date: Wed, 9 Feb 2011 14:10:15 -0500 Message-Id: <dustinhughes@example.org> To: dustinhughes@example.org Mime-Version: 1.0 (Apple Message framework v1082) X-Mailer: Apple Mail (2.1082) X-Originating-IP: 83.161.120.155 X-Virus-Checked: Checked by ClamAV on apache.org Is there an easy (and dynamic) way to change the priority of the Chukwa = Demux job to something other than VERY_HIGH?= From chukwa-user-returndustinhughes@example.org Thu Feb 24 18:13:11 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 65862 invoked from network); 24 Feb 2011 18:13:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 24 Feb 2011 18:13:10 -0000 Received: (qmail 84704 invoked by uid 500); 24 Feb 2011 18:13:10 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 84646 invoked by uid 500); 24 Feb 2011 18:13:08 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 84639 invoked by uid 99); 24 Feb 2011 18:13:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 18:13:07 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 83.161.120.155 is neither permitted nor denied by domain of dustinhughes@example.org) Received: from [83.161.120.155] (HELO mail-ww0-f43.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 18:13:00 +0000 Received: by wwe15 with SMTP id 15so915526wwe.0 for <dustinhughes@example.org>; Thu, 24 Feb 2011 10:12:39 -0800 (PST) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id t3mr1133362wet.2.1298571159611; Thu, 24 Feb 2011 10:12:39 -0800 (PST) Received: by 83.161.120.155 with HTTP; Thu, 24 Feb 2011 10:12:39 -0800 (PST) Date: Thu, 24 Feb 2011 11:12:39 -0700 Message-ID: <dustinhughes@example.org> Subject: _SUCCESS files appearing in demuxOutput From: Corbin Hoenes <dustinhughes@example.org> To: dustinhughes@example.org Content-Type: multipart/alternative; boundary=0016364c7949af2b96049d0b28ca X-Virus-Checked: Checked by ClamAV on apache.org --0016364c7949af2b96049d0b28ca Content-Type: text/plain; charset=ISO-8859-1 Anyone seen this? /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS I clean them out and I keep getting the same file showing up and chukwa doesn't know how to handle it: postProcess.log: 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Cluster (_SUCCESS) 2011-02-21 06:51:55,027 INFO main MoveToRepository - processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS) 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in processDemuxOutput: java.io.IOException: hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS is not a dustinhughes@example.org(MoveToRepository.java:54) at org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(MoveToRepository.java:250) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.movetoMainRepository(PostProcessorManager.java:201) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.start(PostProcessorManager.java:146) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.main(PostProcessorManager.java:80) --0016364c7949af2b96049d0b28ca Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Anyone seen this?<br><br>/chukwa/postProcess/demuxOutputDir_1298061686862/_= SUCCESS<br><br>I clean them out and I keep getting the same file showing up= and chukwa doesn&#39;t know how to handle it:<br><br>postProcess.log:<br> 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Cluster= (_SUCCESS)<br>2011-02-21 06:51:55,027 INFO main MoveToRepository - process= ClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS)<br>2011-02-21 06:51:55,= 028 WARN main PostProcessorManager - Error in processDemuxOutput:<br> java.io.IOException: hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298= 061686862/_SUCCESS is not a directory!<br>=dustinhughes@example.org= ukwa.extraction.demux.MoveToRepository.processClutserDirectory(MoveToReposi= tory.java:54)<br> =dustinhughes@example.org= n(MoveToRepository.java:250)<br>=dustinhughes@example.org= ction.demux.PostProcessorManager.movetoMainRepository(PostProcessorManager.= java:201)<br> =dustinhughes@example.org= .start(PostProcessorManager.java:146)<br>=dustinhughes@example.org= kwa.extraction.demux.PostProcessorManager.main(PostProcessorManager.java:80= )<br><br> --0016364c7949af2b96049d0b28ca-- From chukwa-user-returndustinhughes@example.org Thu Feb 24 18:57:05 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 84843 invoked from network); 24 Feb 2011 18:57:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 24 Feb 2011 18:57:05 -0000 Received: (qmail 66363 invoked by uid 500); 24 Feb 2011 18:57:05 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 66351 invoked by uid 500); 24 Feb 2011 18:57:03 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 66344 invoked by uid 99); 24 Feb 2011 18:57:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 18:57:03 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout1-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 18:56:56 +0000 Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [83.161.120.155]) by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1OIu0L8057418 for <dustinhughes@example.org>; Thu, 24 Feb 2011 10:56:00 -0800 (PST) Received: from SP2-EX07VS05.ds.corp.yahoo.com ([83.161.120.155]) by SP2-EX07CAS05.ds.corp.yahoo.com ([83.161.120.155]) with mapi; Thu, 24 Feb 2011 10:56:00 -0800 From: Eric Yang <dustinhughes@example.org> To: "dustinhughes@example.org" <dustinhughes@example.org> Date: Thu, 24 Feb 2011 10:55:57 -0800 Subject: Re: _SUCCESS files appearing in demuxOutput Thread-Topic: _SUCCESS files appearing in demuxOutput Thread-Index: AcvUTopAquWVE6rcQu6bMi0TJN5LoQABe4BD Message-ID: <dustinhughes@example.org> In-Reply-To: <dustinhughes@example.org> Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C98BE9BDFE51eyangyahooinccom_" MIME-Version: 1.0 --_000_C98BE9BDFE51eyangyahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi Corbin, I have not seen this. What is the version of hadoop that you are using, ar= e you using 0.21? It looks like the _SUCCESS file is spill out after demux= mapreduce job. There are two possibilities leading to the creation of thi= s file. Demux is modified and it is doing something that is unexpected, or= the mapreduce framework 0.21 put that file there. If you are using 0.21, I would recommend to avoid it. A more stable version of Hadoop is 0.20.100 branch, and you can download it= from: http://people.apache.org/~eyang/ Regards, Eric On 2/24/11 10:12 AM, "Corbin Hoenes" <dustinhughes@example.org> wrote: Anyone seen this? /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS I clean them out and I keep getting the same file showing up and chukwa doe= sn't know how to handle it: postProcess.log: 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Cluster= (_SUCCESS) 2011-02-21 06:51:55,027 INFO main MoveToRepository - processClutserDirector= y (_SUCCESS,/chukwa/repos//_SUCCESS) 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in processDe= muxOutput: java.io.IOException: hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298= 061686862/_SUCCESS is not a dustinhughes@example.org= utserDirectory(MoveToRepository.java:54) at org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(Move= ToRepository.java:250) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.movet= oMainRepository(PostProcessorManager.java:201) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.start= (PostProcessorManager.java:146) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.main(= PostProcessorManager.java:80) --_000_C98BE9BDFE51eyangyahooinccom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <HTML> <HEAD> <TITLE>Re: _SUCCESS files appearing in demuxOutput</TITLE> </HEAD> <BODY> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:= 11pt'>Hi Corbin,<BR> <BR> I have not seen this. &nbsp;What is the version of hadoop that you are usin= g, are you using 0.21? &nbsp;It looks like the _SUCCESS file is spill out a= fter demux mapreduce job. &nbsp;There are two possibilities leading to the = creation of this file. &nbsp;Demux is modified and it is doing something th= at is unexpected, or the mapreduce framework 0.21 put that file there.<BR> If you are using 0.21, I would recommend to avoid it.<BR> <BR> A more stable version of Hadoop is 0.20.100 branch, and you can download it= from:<BR> <BR> <a href=3D"http://people.apache.org/~eyang/">http://people.apache.org/~eyan= g/</a><BR> <BR> Regards,<BR> Eric<BR> <BR> On 2/24/11 10:12 AM, &quot;Corbin Hoenes&quot; &lt;<a href=3D"dustinhughes@example.org= om">dustinhughes@example.org</a>&gt; wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"= ><SPAN STYLE=3D'font-size:11pt'>Anyone seen this?<BR> <BR> /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS<BR> <BR> I clean them out and I keep getting the same file showing up and chukwa doe= sn't know how to handle it:<BR> <BR> postProcess.log:<BR> 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Cluster= (_SUCCESS)<BR> 2011-02-21 06:51:55,027 INFO main MoveToRepository - processClutserDirector= y (_SUCCESS,/chukwa/repos//_SUCCESS)<BR> 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in processDe= muxOutput:<BR> java.io.IOException: hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298= 061686862/_SUCCESS is not a directory!<BR> =dustinhughes@example.org= cessClutserDirectory(MoveToRepository.java:54)<BR> =dustinhughes@example.org= n(MoveToRepository.java:250)<BR> =dustinhughes@example.org= .movetoMainRepository(PostProcessorManager.java:201)<BR> =dustinhughes@example.org= .start(PostProcessorManager.java:146)<BR> =dustinhughes@example.org= .main(PostProcessorManager.java:80)<BR> <BR> <BR> </SPAN></FONT></BLOCKQUOTE> </BODY> </HTML> --_000_C98BE9BDFE51eyangyahooinccom_-- From chukwa-user-returndustinhughes@example.org Thu Feb 24 19:20:58 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 99851 invoked from network); 24 Feb 2011 19:20:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 24 Feb 2011 19:20:58 -0000 Received: (qmail 2063 invoked by uid 500); 24 Feb 2011 19:20:57 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 1931 invoked by uid 500); 24 Feb 2011 19:20:56 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 1924 invoked by uid 99); 24 Feb 2011 19:20:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 19:20:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dustinhughes@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mx2.netflix.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 Feb 2011 19:20:48 +0000 Received: from ExchFE101.netflix.com (exchfe101.netflix.com [83.161.120.155]) by mx2.netflix.com (83.161.120.15560308/8.12.11) with ESMTP id p1OJKR8J002705 for <dustinhughes@example.org>; Thu, 24 Feb 2011 11:20:27 -0800 Received: from ExchMBX104.netflix.com ([888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4409a:eb58:3def:adb6]) by ExchFE101.netflix.com ([888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a46861:6a26:831f:e8c9%14]) with mapi; Thu, 24 Feb 2011 11:20:15 -0800 From: Jerome Boulon <dustinhughes@example.org> To: "dustinhughes@example.org" <dustinhughes@example.org> Date: Thu, 24 Feb 2011 11:20:27 -0800 Subject: Re: _SUCCESS files appearing in demuxOutput Thread-Topic: _SUCCESS files appearing in demuxOutput Thread-Index: AcvUV915dMZ/+YyYRTmWaR8kajYiHw== Message-ID: <dustinhughes@example.org> In-Reply-To: <dustinhughes@example.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/83.161.120.155115 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C98BEEE2187A3jboulonnetflixcom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_C98BEEE2187A3jboulonnetflixcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable This filename is coming from here: http://hadoop.apache.org/mapreduce/docs/= r0.21.0/api/constant-values.html In general for hadoop you may want to avoid looking at any "_*" file since = those are Hadoop related files like (_temporary, _log,=85) /Jerome. From: Eric Yang <dustinhughes@example.org<mailto:dustinhughes@example.org>> Reply-To: "dustinhughes@example.org<mailto:dustinhughes@example.org= ache.org>" <dustinhughes@example.org<mailto:dustinhughes@example.org= pache.org>> Date: Thu, 24 Feb 2011 10:55:57 -0800 To: "dustinhughes@example.org<mailto:dustinhughes@example.org= rg>" <dustinhughes@example.org<mailto:dustinhughes@example.org.= org>> Subject: Re: _SUCCESS files appearing in demuxOutput Hi Corbin, I have not seen this. What is the version of hadoop that you are using, ar= e you using 0.21? It looks like the _SUCCESS file is spill out after demux= mapreduce job. There are two possibilities leading to the creation of thi= s file. Demux is modified and it is doing something that is unexpected, or= the mapreduce framework 0.21 put that file there. If you are using 0.21, I would recommend to avoid it. A more stable version of Hadoop is 0.20.100 branch, and you can download it= from: http://people.apache.org/~eyang/ Regards, Eric On 2/24/11 10:12 AM, "Corbin Hoenes" <dustinhughes@example.org> wrote: Anyone seen this? /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS I clean them out and I keep getting the same file showing up and chukwa doe= sn't know how to handle it: postProcess.log: 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Cluster= (_SUCCESS) 2011-02-21 06:51:55,027 INFO main MoveToRepository - processClutserDirector= y (_SUCCESS,/chukwa/repos//_SUCCESS) 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in processDe= muxOutput: java.io.IOException: hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298= 061686862/_SUCCESS is not a dustinhughes@example.org= utserDirectory(MoveToRepository.java:54) at org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(Move= ToRepository.java:250) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.movet= oMainRepository(PostProcessorManager.java:201) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.start= (PostProcessorManager.java:146) at org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.main(= PostProcessorManager.java:80) --_000_C98BEEE2187A3jboulonnetflixcom_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html><head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"></head><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space;= -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14p= x; font-family: Calibri, sans-serif; "><div>This filename is coming from he= re:&nbsp;<a href=3D"http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/con= stant-values.html">http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/cons= tant-values.html</a></div><div>In general for hadoop you may want to avoid = looking at any &quot;_*&quot; file since those are Hadoop related files lik= e (<span class=3D"Apple-style-span" style=3D"font-family: monospace; font-s= ize: medium; ">_temporary</span>, _log,=85)</div><div><br></div><div>/Jerom= e.</div><span id=3D"OLK_SRC_BODY_SECTION"><div style=3D"font-family:Calibri= ; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none;= BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-= RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDI= NG-TOP: 3pt"><span style=3D"font-weight:bold">From: </span> Eric Yang &lt;<= a href=3D"mailto:dustinhughes@example.org">dustinhughes@example.org</a>&gt;<br><span = style=3D"font-weight:bold">Reply-To: </span> &quot;<a href=3D"mailto:chukwa= -dustinhughes@example.org">dustinhughes@example.org</a>&quot; &lt;= <a href=3D"mailto:dustinhughes@example.org">dustinhughes@example.org= pache.org</a>&gt;<br><span style=3D"font-weight:bold">Date: </span> Thu, 24= Feb 2011 10:55:57 -0800<br><span style=3D"font-weight:bold">To: </span> &q= uot;<a href=3D"mailto:dustinhughes@example.org">chukwa-user@incubat= or.apache.org</a>&quot; &lt;<a href=3D"mailto:dustinhughes@example.org.= org">dustinhughes@example.org</a>&gt;<br><span style=3D"font-weight= :bold">Subject: </span> Re: _SUCCESS files appearing in demuxOutput<br></di= v><div><br></div><div><div><font face=3D"Calibri,Verdana,Helvetica,Arial"><= span style=3D"font-size:11pt">Hi Corbin,<br><br> I have not seen this. &nbsp;What is the version of hadoop that you are usin= g, are you using 0.21? &nbsp;It looks like the _SUCCESS file is spill out a= fter demux mapreduce job. &nbsp;There are two possibilities leading to the = creation of this file. &nbsp;Demux is modified and it is doing something th= at is unexpected, or the mapreduce framework 0.21 put that file there.<br> If you are using 0.21, I would recommend to avoid it.<br><br> A more stable version of Hadoop is 0.20.100 branch, and you can download it= from:<br><br><a href=3D"http://people.apache.org/~eyang/">http://people.ap= ache.org/~eyang/</a><br><br> Regards,<br> Eric<br><br> On 2/24/11 10:12 AM, &quot;Corbin Hoenes&quot; &lt;<a href=3D"dustinhughes@example.org= om">dustinhughes@example.org</a>&gt; wrote:<br><br></span></font><blockquote><font f= ace=3D"Calibri,Verdana,Helvetica,Arial"><span style=3D"font-size:11pt">Anyo= ne seen this?<br><br> /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS<br><br> I clean them out and I keep getting the same file showing up and chukwa doe= sn't know how to handle it:<br><br> postProcess.log:<br> 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Cluster= (_SUCCESS)<br> 2011-02-21 06:51:55,027 INFO main MoveToRepository - processClutserDirector= y (_SUCCESS,/chukwa/repos//_SUCCESS)<br> 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in processDe= muxOutput:<br> java.io.IOException: hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298= 061686862/_SUCCESS is not a directory!<br> &nbsp;&nbsp;&nbsp; at org.apache.hadoop.chukwa.extraction.demux.MoveToRepos= itory.processClutserDirectory(MoveToRepository.java:54)<br> &nbsp;&nbsp;&nbsp; at org.apache.hadoop.chukwa.extraction.demux.MoveToRepos= itory.main(MoveToRepository.java:250)<br> &nbsp;&nbsp;&nbsp; at org.apache.hadoop.chukwa.extraction.demux.PostProcess= orManager.movetoMainRepository(PostProcessorManager.java:201)<br> &nbsp;&nbsp;&nbsp; at org.apache.hadoop.chukwa.extraction.demux.PostProcess= orManager.start(PostProcessorManager.java:146)<br> &nbsp;&nbsp;&nbsp; at org.apache.hadoop.chukwa.extraction.demux.PostProcess= orManager.main(PostProcessorManager.java:80)<br><br><br></span></font></blo= ckquote></div></div></span></body></html> --_000_C98BEEE2187A3jboulonnetflixcom_-- From chukwa-user-returndustinhughes@example.org Fri Feb 25 00:25:11 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 11276 invoked from network); 25 Feb 2011 00:25:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 25 Feb 2011 00:25:11 -0000 Received: (qmail 3900 invoked by uid 500); 25 Feb 2011 00:25:11 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 3819 invoked by uid 500); 25 Feb 2011 00:25:10 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 3812 invoked by uid 99); 25 Feb 2011 00:25:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 00:25:10 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: 83.161.120.155 is neither permitted nor denied by domain of dustinhughes@example.org) Received: from [83.161.120.155] (HELO mail-gy0-f175.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 00:25:02 +0000 Received: by gyg13 with SMTP id 13so512751gyg.6 for <dustinhughes@example.org>; Thu, 24 Feb 2011 16:24:41 -0800 (PST) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id h21mr2857677ybe.220.1298593481044; Thu, 24 Feb 2011 16:24:41 -0800 (PST) Received: by 83.161.120.155 with HTTP; Thu, 24 Feb 2011 16:24:41 -0800 (PST) In-Reply-To: <dustinhughes@example.org> References: <dustinhughes@example.org> <dustinhughes@example.org> Date: Thu, 24 Feb 2011 17:24:41 -0700 Message-ID: <dustinhughes@example.org> Subject: Re: _SUCCESS files appearing in demuxOutput From: Corbin Hoenes <dustinhughes@example.org> To: dustinhughes@example.org Cc: Jerome Boulon <dustinhughes@example.org> Content-Type: multipart/alternative; boundary=000e0cd731fa2532b0049d105bbb X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd731fa2532b0049d105bbb Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable We're using Cloudera's CDH3 beta 4 release. Maybe they've patched in the FileOutputCommitter stuff into their release as it's based on Hadoop 0.20.2 Looking at the source for Chukwa 0.3 (version we are on) the MoveToRepository class skips the _log and _temporary directories. Seems like Chukwa should skip the _SUCCESS directory as well? Or could a more general skip be used like skip anything starting with and underscore? (maybe too aggressive). Does Chukwa 0.4 or 0.5 fix this issue? (I'm probably going to just have to patch 0.3 but maybe just another reason to upgrade.) On Thu, Feb 24, 2011 at 12:20 PM, Jerome Boulon <dustinhughes@example.org> wrote= : > This filename is coming from here: > http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/constant-values.html > In general for hadoop you may want to avoid looking at any "_*" file sinc= e > those are Hadoop related files like (_temporary, _log,=85) > > /Jerome. > From: Eric Yang <dustinhughes@example.org> > Reply-To: "dustinhughes@example.org" < > dustinhughes@example.org> > Date: Thu, 24 Feb 2011 10:55:57 -0800 > To: "dustinhughes@example.org" <dustinhughes@example.org> > Subject: Re: _SUCCESS files appearing in demuxOutput > > Hi Corbin, > > I have not seen this. What is the version of hadoop that you are using, > are you using 0.21? It looks like the _SUCCESS file is spill out after > demux mapreduce job. There are two possibilities leading to the creation= of > this file. Demux is modified and it is doing something that is unexpecte= d, > or the mapreduce framework 0.21 put that file there. > If you are using 0.21, I would recommend to avoid it. > > A more stable version of Hadoop is 0.20.100 branch, and you can download = it > from: > > http://people.apache.org/~eyang/ > > Regards, > Eric > > On 2/24/11 10:12 AM, "Corbin Hoenes" <dustinhughes@example.org> wrote: > > Anyone seen this? > > /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS > > I clean them out and I keep getting the same file showing up and chukwa > doesn't know how to handle it: > > postProcess.log: > 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Clust= er > (_SUCCESS) > 2011-02-21 06:51:55,027 INFO main MoveToRepository - > processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS) > 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in > processDemuxOutput: > java.io.IOException: > hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS = is > not a directory! > at > org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.processClutser= Directory(MoveToRepository.java:54) > at > org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(MoveToRep= ository.java:250) > at > org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.movetoMain= Repository(PostProcessorManager.java:201) > at > org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.start(Post= ProcessorManager.java:146) > at > org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.main(PostP= rocessorManager.java:80) > > > --000e0cd731fa2532b0049d105bbb Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable We&#39;re using Cloudera&#39;s CDH3 beta 4 release.=A0=A0 Maybe they&#39;ve= patched in the FileOutputCommitter stuff into their release as it&#39;s ba= sed on Hadoop 0.20.2<br>Looking at the source for Chukwa 0.3 (version we ar= e on) the MoveToRepository class skips the _log and _temporary directories.= <br> <br>Seems like Chukwa should skip the _SUCCESS directory as well?=A0 Or cou= ld a more general skip be used like skip anything starting with and undersc= ore? (maybe too aggressive).<br><br>Does Chukwa 0.4 or 0.5 fix this issue? = (I&#39;m probably going to just have to patch 0.3 but maybe just another re= ason to upgrade.)<br> <br><div class=3D"gmail_quote">On Thu, Feb 24, 2011 at 12:20 PM, Jerome Bou= lon <span dir=3D"ltr">&lt;<a href=3D"mailto:dustinhughes@example.org">jboulon@ne= tflix.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style= =3D"margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); p= adding-left: 1ex;"> <div style=3D"word-wrap: break-word; color: rgb(0, 0, 0); font-size: 14px; = font-family: Calibri,sans-serif;"><div>This filename is coming from here:= =A0<a href=3D"http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/constant-= values.html" target=3D"_blank">http://hadoop.apache.org/mapreduce/docs/r0.2= 1.0/api/constant-values.html</a></div> <div>In general for hadoop you may want to avoid looking at any &quot;_*&qu= ot; file since those are Hadoop related files like (<span style=3D"font-fam= ily: monospace; font-size: medium;">_temporary</span>, _log,=85)</div><div> <br></div><div>/Jerome.</div><span><div style=3D"font-family: Calibri; font= -size: 11pt; text-align: left; color: black; border-width: 1pt medium mediu= m; border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use= -text-color -moz-use-text-color; padding: 3pt 0in 0in;"> <span style=3D"font-weight: bold;">From: </span> Eric Yang &lt;<a href=3D"m= ailto:dustinhughes@example.org" target=3D"_blank">dustinhughes@example.org</a>&gt;<br= ><span style=3D"font-weight: bold;">Reply-To: </span> &quot;<a href=3D"mail= to:dustinhughes@example.org" target=3D"_blank">chukwa-user@incubato= r.apache.org</a>&quot; &lt;<a href=3D"mailto:dustinhughes@example.org= rg" target=3D"_blank">dustinhughes@example.org</a>&gt;<br> <span style=3D"font-weight: bold;">Date: </span> Thu, 24 Feb 2011 10:55:57 = -0800<br><span style=3D"font-weight: bold;">To: </span> &quot;<a href=3D"ma= ilto:dustinhughes@example.org" target=3D"_blank">chukwa-user@incuba= tor.apache.org</a>&quot; &lt;<a href=3D"mailto:dustinhughes@example.org= .org" target=3D"_blank">dustinhughes@example.org</a>&gt;<br> <span style=3D"font-weight: bold;">Subject: </span> Re: _SUCCESS files appe= aring in demuxOutput<br></div><div><div></div><div class=3D"h5"><div><br></= div><div><div><font face=3D"Calibri,Verdana,Helvetica,Arial"><span style=3D= "font-size: 11pt;">Hi Corbin,<br> <br> I have not seen this. =A0What is the version of hadoop that you are using, = are you using 0.21? =A0It looks like the _SUCCESS file is spill out after d= emux mapreduce job. =A0There are two possibilities leading to the creation = of this file. =A0Demux is modified and it is doing something that is unexpe= cted, or the mapreduce framework 0.21 put that file there.<br> If you are using 0.21, I would recommend to avoid it.<br><br> A more stable version of Hadoop is 0.20.100 branch, and you can download it= from:<br><br><a href=3D"http://people.apache.org/%7Eeyang/" target=3D"_bla= nk">http://people.apache.org/~eyang/</a><br><br> Regards,<br> Eric<br><br> On 2/24/11 10:12 AM, &quot;Corbin Hoenes&quot; &lt;<a href=3D"http://corbin= @tynt.com" target=3D"_blank">dustinhughes@example.org</a>&gt; wrote:<br><br></span><= /font><blockquote><font face=3D"Calibri,Verdana,Helvetica,Arial"><span styl= e=3D"font-size: 11pt;">Anyone seen this?<br> <br> /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS<br><br> I clean them out and I keep getting the same file showing up and chukwa doe= sn&#39;t know how to handle it:<br><br> postProcess.log:<br> 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing Cluster= (_SUCCESS)<br> 2011-02-21 06:51:55,027 INFO main MoveToRepository - processClutserDirector= y (_SUCCESS,/chukwa/repos//_SUCCESS)<br> 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in processDe= muxOutput:<br> java.io.IOException: hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298= 061686862/_SUCCESS is not a directory!<br> =dustinhughes@example.org= cessClutserDirectory(MoveToRepository.java:54)<br> =dustinhughes@example.org= n(MoveToRepository.java:250)<br> =dustinhughes@example.org= .movetoMainRepository(PostProcessorManager.java:201)<br> =dustinhughes@example.org= .start(PostProcessorManager.java:146)<br> =dustinhughes@example.org= .main(PostProcessorManager.java:80)<br><br><br></span></font></blockquote><= /div></div></div></div></span></div> </blockquote></div><br> --000e0cd731fa2532b0049d105bbb-- From chukwa-user-returndustinhughes@example.org Fri Feb 25 00:27:49 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 12271 invoked from network); 25 Feb 2011 00:27:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 25 Feb 2011 00:27:49 -0000 Received: (qmail 5484 invoked by uid 500); 25 Feb 2011 00:27:49 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 5422 invoked by uid 500); 25 Feb 2011 00:27:48 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 5414 invoked by uid 99); 25 Feb 2011 00:27:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 00:27:48 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dustinhughes@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-wy0-f175.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 00:27:42 +0000 Received: by wyb38 with SMTP id 38so1124517wyb.6 for <dustinhughes@example.org>; Thu, 24 Feb 2011 16:27:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type:content-transfer-encoding; bh=3xa3gaXpck/NSoasfrlGoCnawrhXkHUzTPgI37TDwMs=; b=TTeL0xciaWEW1FPZpsQ8cBKUJoLm7Nh5KJTTn5nYZPIvE0aqwwWIC+colIM1mASQY8 7wqs3G5Y9yK36PFGnYotPtScTFgB/KWbJmbo8U046yI8ay6jkynp5UfkbHzV3S5seLGG CfDV3KXfbR96vrOjFTegjhsK3dKE6GhamMh2s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=mUwK/PtQBvcvI+TLTsMgOd/vvpVc6HUY95tBhVcRR36RKtVPOkcsPJj4mBPUwaA1YJ 2DzoV/61WCfNJz3QROIXwdFL9JoWMBjOWa19xTIl8CklTv7LRTC4g2h1TAOB8FLJE58A KqM5yGNoQO6ugHOnFjrsIMd57eJUZMpqqZrbY= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id j21mr1513038wbv.29.1298593642075; Thu, 24 Feb 2011 16:27:22 -0800 (PST) Received: by 83.161.120.155 with HTTP; Thu, 24 Feb 2011 16:27:21 -0800 (PST) In-Reply-To: <dustinhughes@example.org> References: <dustinhughes@example.org> <dustinhughes@example.org> <dustinhughes@example.org> Date: Thu, 24 Feb 2011 16:27:21 -0800 Message-ID: <dustinhughes@example.org> Subject: Re: _SUCCESS files appearing in demuxOutput From: Ariel Rabkin <dustinhughes@example.org> To: dustinhughes@example.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org I don't think this has been fixed yet in trunk, let alone 0.4. I would support skipping everything starting with _. Is there an actual use case this would break? --Ari On Thu, Feb 24, 2011 at 4:24 PM, Corbin Hoenes <dustinhughes@example.org> wrote: > We're using Cloudera's CDH3 beta 4 release.=A0=A0 Maybe they've patched i= n the > FileOutputCommitter stuff into their release as it's based on Hadoop 0.20= .2 > Looking at the source for Chukwa 0.3 (version we are on) the > MoveToRepository class skips the _log and _temporary directories. > > Seems like Chukwa should skip the _SUCCESS directory as well?=A0 Or could= a > more general skip be used like skip anything starting with and underscore= ? > (maybe too aggressive). > > Does Chukwa 0.4 or 0.5 fix this issue? (I'm probably going to just have t= o > patch 0.3 but maybe just another reason to upgrade.) > > On Thu, Feb 24, 2011 at 12:20 PM, Jerome Boulon <dustinhughes@example.org> wro= te: >> >> This filename is coming from >> here:=A0http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/constant-val= ues.html >> In general for hadoop you may want to avoid looking at any "_*" file sin= ce >> those are Hadoop related files like (_temporary, _log,=85) >> /Jerome. >> From: Eric Yang <dustinhughes@example.org> >> Reply-To: "dustinhughes@example.org" >> <dustinhughes@example.org> >> Date: Thu, 24 Feb 2011 10:55:57 -0800 >> To: "dustinhughes@example.org" <dustinhughes@example.org= > >> Subject: Re: _SUCCESS files appearing in demuxOutput >> >> Hi Corbin, >> >> I have not seen this. =A0What is the version of hadoop that you are usin= g, >> are you using 0.21? =A0It looks like the _SUCCESS file is spill out afte= r >> demux mapreduce job. =A0There are two possibilities leading to the creat= ion of >> this file. =A0Demux is modified and it is doing something that is unexpe= cted, >> or the mapreduce framework 0.21 put that file there. >> If you are using 0.21, I would recommend to avoid it. >> >> A more stable version of Hadoop is 0.20.100 branch, and you can download >> it from: >> >> http://people.apache.org/~eyang/ >> >> Regards, >> Eric >> >> On 2/24/11 10:12 AM, "Corbin Hoenes" <dustinhughes@example.org> wrote: >> >> Anyone seen this? >> >> /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS >> >> I clean them out and I keep getting the same file showing up and chukwa >> doesn't know how to handle it: >> >> postProcess.log: >> 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing >> Cluster (_SUCCESS) >> 2011-02-21 06:51:55,027 INFO main MoveToRepository - >> processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS) >> 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in >> processDemuxOutput: >> java.io.IOException: >> hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS= is >> not a directory! >> =A0=A0=A0 at >> org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.processClutse= rDirectory(MoveToRepository.java:54) >> =A0=A0=A0 at >> org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(MoveToRe= pository.java:250) >> =A0=A0=A0 at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.movetoMai= nRepository(PostProcessorManager.java:201) >> =A0=A0=A0 at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.start(Pos= tProcessorManager.java:146) >> =A0=A0=A0 at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.main(Post= ProcessorManager.java:80) >> >> > > --=20 Ari Rabkin dustinhughes@example.org UC Berkeley Computer Science Department From chukwa-user-returndustinhughes@example.org Fri Feb 25 00:33:35 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 14503 invoked from network); 25 Feb 2011 00:33:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 25 Feb 2011 00:33:35 -0000 Received: (qmail 9559 invoked by uid 500); 25 Feb 2011 00:33:35 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 9480 invoked by uid 500); 25 Feb 2011 00:33:34 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 9473 invoked by uid 99); 25 Feb 2011 00:33:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 00:33:34 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout2.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 00:33:26 +0000 Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [83.161.120.155]) by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p1P0Wpm0091863 for <dustinhughes@example.org>; Thu, 24 Feb 2011 16:32:51 -0800 (PST) Received: from SP2-EX07VS05.ds.corp.yahoo.com ([83.161.120.155]) by SP2-EX07CAS04.ds.corp.yahoo.com ([83.161.120.155]) with mapi; Thu, 24 Feb 2011 16:32:51 -0800 From: Eric Yang <dustinhughes@example.org> To: "dustinhughes@example.org" <dustinhughes@example.org> Date: Thu, 24 Feb 2011 16:32:49 -0800 Subject: Re: _SUCCESS files appearing in demuxOutput Thread-Topic: _SUCCESS files appearing in demuxOutput Thread-Index: AcvUgukXFY7d41rYSJ2dwcKHOpY0PQAAJ5xl Message-ID: <dustinhughes@example.org> In-Reply-To: <dustinhughes@example.org> Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C98C38B1FEAAeyangyahooinccom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_C98C38B1FEAAeyangyahooinccom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I also support for skipping _*. Regards, Eric On 2/24/11 4:27 PM, "Ariel Rabkin" <dustinhughes@example.org> wrote: I don't think this has been fixed yet in trunk, let alone 0.4. I would support skipping everything starting with _. Is there an actual use case this would break? --Ari On Thu, Feb 24, 2011 at 4:24 PM, Corbin Hoenes <dustinhughes@example.org> wrote: > We're using Cloudera's CDH3 beta 4 release. Maybe they've patched in th= e > FileOutputCommitter stuff into their release as it's based on Hadoop 0.20= .2 > Looking at the source for Chukwa 0.3 (version we are on) the > MoveToRepository class skips the _log and _temporary directories. > > Seems like Chukwa should skip the _SUCCESS directory as well? Or could a > more general skip be used like skip anything starting with and underscore= ? > (maybe too aggressive). > > Does Chukwa 0.4 or 0.5 fix this issue? (I'm probably going to just have t= o > patch 0.3 but maybe just another reason to upgrade.) > > On Thu, Feb 24, 2011 at 12:20 PM, Jerome Boulon <dustinhughes@example.org> wro= te: >> >> This filename is coming from >> here: http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/constant-value= s.html >> In general for hadoop you may want to avoid looking at any "_*" file sin= ce >> those are Hadoop related files like (_temporary, _log,...) >> /Jerome. >> From: Eric Yang <dustinhughes@example.org> >> Reply-To: "dustinhughes@example.org" >> <dustinhughes@example.org> >> Date: Thu, 24 Feb 2011 10:55:57 -0800 >> To: "dustinhughes@example.org" <dustinhughes@example.org= > >> Subject: Re: _SUCCESS files appearing in demuxOutput >> >> Hi Corbin, >> >> I have not seen this. What is the version of hadoop that you are using, >> are you using 0.21? It looks like the _SUCCESS file is spill out after >> demux mapreduce job. There are two possibilities leading to the creatio= n of >> this file. Demux is modified and it is doing something that is unexpect= ed, >> or the mapreduce framework 0.21 put that file there. >> If you are using 0.21, I would recommend to avoid it. >> >> A more stable version of Hadoop is 0.20.100 branch, and you can download >> it from: >> >> http://people.apache.org/~eyang/ >> >> Regards, >> Eric >> >> On 2/24/11 10:12 AM, "Corbin Hoenes" <dustinhughes@example.org> wrote: >> >> Anyone seen this? >> >> /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS >> >> I clean them out and I keep getting the same file showing up and chukwa >> doesn't know how to handle it: >> >> postProcess.log: >> 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing >> Cluster (_SUCCESS) >> 2011-02-21 06:51:55,027 INFO main MoveToRepository - >> processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS) >> 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in >> processDemuxOutput: >> java.io.IOException: >> hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS= is >> not a directory! >> at >> org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.processClutse= rDirectory(MoveToRepository.java:54) >> at >> org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(MoveToRe= pository.java:250) >> at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.movetoMai= nRepository(PostProcessorManager.java:201) >> at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.start(Pos= tProcessorManager.java:146) >> at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.main(Post= ProcessorManager.java:80) >> >> > > -- Ari Rabkin dustinhughes@example.org UC Berkeley Computer Science Department --_000_C98C38B1FEAAeyangyahooinccom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <HTML> <HEAD> <TITLE>Re: _SUCCESS files appearing in demuxOutput</TITLE> </HEAD> <BODY> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:= 11pt'>I also support for skipping _*.<BR> <BR> Regards,<BR> Eric<BR> <BR> On 2/24/11 4:27 PM, &quot;Ariel Rabkin&quot; &lt;<a href=3D"dustinhughes@example.org.= com">dustinhughes@example.org</a>&gt; wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"= ><SPAN STYLE=3D'font-size:11pt'>I don't think this has been fixed yet in tr= unk, let alone 0.4.<BR> <BR> I would support skipping everything starting with _. &nbsp;Is there an<BR> actual use case this would break?<BR> <BR> --Ari<BR> <BR> On Thu, Feb 24, 2011 at 4:24 PM, Corbin Hoenes &lt;<a href=3D"dustinhughes@example.org= om">dustinhughes@example.org</a>&gt; wrote:<BR> &gt; We're using Cloudera's CDH3 beta 4 release.=A0=A0 Maybe they've patche= d in the<BR> &gt; FileOutputCommitter stuff into their release as it's based on Hadoop 0= .20.2<BR> &gt; Looking at the source for Chukwa 0.3 (version we are on) the<BR> &gt; MoveToRepository class skips the _log and _temporary directories.<BR> &gt;<BR> &gt; Seems like Chukwa should skip the _SUCCESS directory as well?=A0 Or co= uld a<BR> &gt; more general skip be used like skip anything starting with and undersc= ore?<BR> &gt; (maybe too aggressive).<BR> &gt;<BR> &gt; Does Chukwa 0.4 or 0.5 fix this issue? (I'm probably going to just hav= e to<BR> &gt; patch 0.3 but maybe just another reason to upgrade.)<BR> &gt;<BR> &gt; On Thu, Feb 24, 2011 at 12:20 PM, Jerome Boulon &lt;<a href=3D"jboulon= @netflix.com">dustinhughes@example.org</a>&gt; wrote:<BR> &gt;&gt;<BR> &gt;&gt; This filename is coming from<BR> &gt;&gt; here:=A0<a href=3D"http://hadoop.apache.org/mapreduce/docs/r0.21.0= /api/constant-values.html">http://hadoop.apache.org/mapreduce/docs/r0.21.0/= api/constant-values.html</a><BR> &gt;&gt; In general for hadoop you may want to avoid looking at any &quot;_= *&quot; file since<BR> &gt;&gt; those are Hadoop related files like (_temporary, _log,&#8230;)<BR> &gt;&gt; /Jerome.<BR> &gt;&gt; From: Eric Yang &lt;<a href=3D"dustinhughes@example.org">eyang@yahoo-in= c.com</a>&gt;<BR> &gt;&gt; Reply-To: &quot;<a href=3D"dustinhughes@example.org">chukw= dustinhughes@example.org</a>&quot;<BR> &gt;&gt; &lt;<a href=3D"dustinhughes@example.org">chukwa-user@incub= ator.apache.org</a>&gt;<BR> &gt;&gt; Date: Thu, 24 Feb 2011 10:55:57 -0800<BR> &gt;&gt; To: &quot;<a href=3D"dustinhughes@example.org">chukwa-user= @incubator.apache.org</a>&quot; &lt;<a href=3D"dustinhughes@example.org= .org">dustinhughes@example.org</a>&gt;<BR> &gt;&gt; Subject: Re: _SUCCESS files appearing in demuxOutput<BR> &gt;&gt;<BR> &gt;&gt; Hi Corbin,<BR> &gt;&gt;<BR> &gt;&gt; I have not seen this. =A0What is the version of hadoop that you ar= e using,<BR> &gt;&gt; are you using 0.21? =A0It looks like the _SUCCESS file is spill ou= t after<BR> &gt;&gt; demux mapreduce job. =A0There are two possibilities leading to the= creation of<BR> &gt;&gt; this file. =A0Demux is modified and it is doing something that is = unexpected,<BR> &gt;&gt; or the mapreduce framework 0.21 put that file there.<BR> &gt;&gt; If you are using 0.21, I would recommend to avoid it.<BR> &gt;&gt;<BR> &gt;&gt; A more stable version of Hadoop is 0.20.100 branch, and you can do= wnload<BR> &gt;&gt; it from:<BR> &gt;&gt;<BR> &gt;&gt; <a href=3D"http://people.apache.org/~eyang/">http://people.apache.= org/~eyang/</a><BR> &gt;&gt;<BR> &gt;&gt; Regards,<BR> &gt;&gt; Eric<BR> &gt;&gt;<BR> &gt;&gt; On 2/24/11 10:12 AM, &quot;Corbin Hoenes&quot; &lt;<a href=3D"corb= dustinhughes@example.org">dustinhughes@example.org</a>&gt; wrote:<BR> &gt;&gt;<BR> &gt;&gt; Anyone seen this?<BR> &gt;&gt;<BR> &gt;&gt; /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS<BR> &gt;&gt;<BR> &gt;&gt; I clean them out and I keep getting the same file showing up and c= hukwa<BR> &gt;&gt; doesn't know how to handle it:<BR> &gt;&gt;<BR> &gt;&gt; postProcess.log:<BR> &gt;&gt; 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesin= g<BR> &gt;&gt; Cluster (_SUCCESS)<BR> &gt;&gt; 2011-02-21 06:51:55,027 INFO main MoveToRepository -<BR> &gt;&gt; processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS)<BR> &gt;&gt; 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in<= BR> &gt;&gt; processDemuxOutput:<BR> &gt;&gt; java.io.IOException:<BR> &gt;&gt; hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_S= UCCESS is<BR> &gt;&gt; not a directory!<BR> &gt;&gt; =A0=A0=A0 at<BR> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.process= ClutserDirectory(MoveToRepository.java:54)<BR> &gt;&gt; =A0=A0=A0 at<BR> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(Mo= veToRepository.java:250)<BR> &gt;&gt; =A0=A0=A0 at<BR> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.mov= etoMainRepository(PostProcessorManager.java:201)<BR> &gt;&gt; =A0=A0=A0 at<BR> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.sta= rt(PostProcessorManager.java:146)<BR> &gt;&gt; =A0=A0=A0 at<BR> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.mai= n(PostProcessorManager.java:80)<BR> &gt;&gt;<BR> &gt;&gt;<BR> &gt;<BR> &gt;<BR> <BR> <BR> <BR> --<BR> Ari Rabkin <a href=3D"dustinhughes@example.org">dustinhughes@example.org</a><BR> UC Berkeley Computer Science Department<BR> <BR> </SPAN></FONT></BLOCKQUOTE> </BODY> </HTML> --_000_C98C38B1FEAAeyangyahooinccom_-- From chukwa-user-returndustinhughes@example.org Fri Feb 25 00:38:14 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 16573 invoked from network); 25 Feb 2011 00:38:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 25 Feb 2011 00:38:14 -0000 Received: (qmail 13403 invoked by uid 500); 25 Feb 2011 00:38:14 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 13344 invoked by uid 500); 25 Feb 2011 00:38:14 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 13337 invoked by uid 99); 25 Feb 2011 00:38:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 00:38:13 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dustinhughes@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mx2.netflix.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 00:38:07 +0000 Received: from exchnmc101.netflix.com (exchnmc101.netflix.com [83.161.120.155]) by mx2.netflix.com (83.161.120.15560308/8.12.11) with ESMTP id p1P0bkOe016662 for <dustinhughes@example.org>; Thu, 24 Feb 2011 16:37:46 -0800 Received: from ExchMBX104.netflix.com ([888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4409a:eb58:3def:adb6]) by exchnmc101.netflix.com ([888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a44894:38ba:ac0c:e4bc%13]) with mapi; Thu, 24 Feb 2011 16:37:39 -0800 From: Jerome Boulon <dustinhughes@example.org> To: "dustinhughes@example.org" <dustinhughes@example.org> Date: Thu, 24 Feb 2011 16:37:48 -0800 Subject: Re: _SUCCESS files appearing in demuxOutput Thread-Topic: _SUCCESS files appearing in demuxOutput Thread-Index: AcvUhDTfDQhn/xsvR2ilvqQqfsccGw== Message-ID: <dustinhughes@example.org> In-Reply-To: <dustinhughes@example.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/83.161.120.155115 acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C98C39C818871jboulonnetflixcom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_C98C39C818871jboulonnetflixcom_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable +1 for skipping _* /Jerome From: Eric Yang <dustinhughes@example.org<mailto:dustinhughes@example.org>> Reply-To: "dustinhughes@example.org<mailto:dustinhughes@example.org= ache.org>" <dustinhughes@example.org<mailto:dustinhughes@example.org= pache.org>> Date: Thu, 24 Feb 2011 16:32:49 -0800 To: "dustinhughes@example.org<mailto:dustinhughes@example.org= rg>" <dustinhughes@example.org<mailto:dustinhughes@example.org.= org>> Subject: Re: _SUCCESS files appearing in demuxOutput I also support for skipping _*. Regards, Eric On 2/24/11 4:27 PM, "Ariel Rabkin" <dustinhughes@example.org> wrote: I don't think this has been fixed yet in trunk, let alone 0.4. I would support skipping everything starting with _. Is there an actual use case this would break? --Ari On Thu, Feb 24, 2011 at 4:24 PM, Corbin Hoenes <dustinhughes@example.org> wrote: > We're using Cloudera's CDH3 beta 4 release. Maybe they've patched in th= e > FileOutputCommitter stuff into their release as it's based on Hadoop 0.20= .2 > Looking at the source for Chukwa 0.3 (version we are on) the > MoveToRepository class skips the _log and _temporary directories. > > Seems like Chukwa should skip the _SUCCESS directory as well? Or could a > more general skip be used like skip anything starting with and underscore= ? > (maybe too aggressive). > > Does Chukwa 0.4 or 0.5 fix this issue? (I'm probably going to just have t= o > patch 0.3 but maybe just another reason to upgrade.) > > On Thu, Feb 24, 2011 at 12:20 PM, Jerome Boulon <dustinhughes@example.org> wro= te: >> >> This filename is coming from >> here: http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/constant-value= s.html >> In general for hadoop you may want to avoid looking at any "_*" file sin= ce >> those are Hadoop related files like (_temporary, _log,=85) >> /Jerome. >> From: Eric Yang <dustinhughes@example.org> >> Reply-To: "dustinhughes@example.org" >> <dustinhughes@example.org> >> Date: Thu, 24 Feb 2011 10:55:57 -0800 >> To: "dustinhughes@example.org" <dustinhughes@example.org= > >> Subject: Re: _SUCCESS files appearing in demuxOutput >> >> Hi Corbin, >> >> I have not seen this. What is the version of hadoop that you are using, >> are you using 0.21? It looks like the _SUCCESS file is spill out after >> demux mapreduce job. There are two possibilities leading to the creatio= n of >> this file. Demux is modified and it is doing something that is unexpect= ed, >> or the mapreduce framework 0.21 put that file there. >> If you are using 0.21, I would recommend to avoid it. >> >> A more stable version of Hadoop is 0.20.100 branch, and you can download >> it from: >> >> http://people.apache.org/~eyang/ >> >> Regards, >> Eric >> >> On 2/24/11 10:12 AM, "Corbin Hoenes" <dustinhughes@example.org> wrote: >> >> Anyone seen this? >> >> /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS >> >> I clean them out and I keep getting the same file showing up and chukwa >> doesn't know how to handle it: >> >> postProcess.log: >> 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing >> Cluster (_SUCCESS) >> 2011-02-21 06:51:55,027 INFO main MoveToRepository - >> processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS) >> 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in >> processDemuxOutput: >> java.io.IOException: >> hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS= is >> not a directory! >> at >> org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.processClutse= rDirectory(MoveToRepository.java:54) >> at >> org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(MoveToRe= pository.java:250) >> at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.movetoMai= nRepository(PostProcessorManager.java:201) >> at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.start(Pos= tProcessorManager.java:146) >> at >> org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.main(Post= ProcessorManager.java:80) >> >> > > -- Ari Rabkin dustinhughes@example.org UC Berkeley Computer Science Department --_000_C98C39C818871jboulonnetflixcom_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable <html><head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3DWindows-1= 252"></head><body style=3D"word-wrap: break-word; -webkit-nbsp-mode: space;= -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14p= x; font-family: Calibri, sans-serif; "><div>&#43;1&nbsp;<span class=3D"Appl= e-style-span" style=3D"font-size: 15px; font-family: Calibri, Verdana, Helv= etica, Arial; ">for skipping _*</span></div><div><span class=3D"Apple-style= -span" style=3D"font-size: 15px; font-family: Calibri, Verdana, Helvetica, = Arial; ">/Jerome</span></div><div><br></div><span id=3D"OLK_SRC_BODY_SECTIO= N"><div style=3D"font-family:Calibri; font-size:11pt; text-align:left; colo= r:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTT= OM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt sol= id; BORDER-RIGHT: medium none; PADDING-TOP: 3pt"><span style=3D"font-weight= :bold">From: </span> Eric Yang &lt;<a href=3D"mailto:dustinhughes@example.org">e= dustinhughes@example.org</a>&gt;<br><span style=3D"font-weight:bold">Reply-To: </= span> &quot;<a href=3D"mailto:dustinhughes@example.org">chukwa-user= @incubator.apache.org</a>&quot; &lt;<a href=3D"mailto:chukwa-user@incubator= .apache.org">dustinhughes@example.org</a>&gt;<br><span style=3D"fon= t-weight:bold">Date: </span> Thu, 24 Feb 2011 16:32:49 -0800<br><span style= =3D"font-weight:bold">To: </span> &quot;<a href=3D"mailto:chukwa-user@incub= ator.apache.org">dustinhughes@example.org</a>&quot; &lt;<a href=3D"= mailto:dustinhughes@example.org">dustinhughes@example.org</= a>&gt;<br><span style=3D"font-weight:bold">Subject: </span> Re: _SUCCESS fi= les appearing in demuxOutput<br></div><div><br></div><div><div><font face= =3D"Calibri,Verdana,Helvetica,Arial"><span style=3D"font-size:11pt">I also = support for skipping _*.<br><br> Regards,<br> Eric<br><br> On 2/24/11 4:27 PM, &quot;Ariel Rabkin&quot; &lt;<a href=3D"dustinhughes@example.org.= com">dustinhughes@example.org</a>&gt; wrote:<br><br></span></font><blockquote><fo= nt face=3D"Calibri,Verdana,Helvetica,Arial"><span style=3D"font-size:11pt">= I don't think this has been fixed yet in trunk, let alone 0.4.<br><br> I would support skipping everything starting with _. &nbsp;Is there an<br> actual use case this would break?<br><br> --Ari<br><br> On Thu, Feb 24, 2011 at 4:24 PM, Corbin Hoenes &lt;<a href=3D"dustinhughes@example.org= om">dustinhughes@example.org</a>&gt; wrote:<br> &gt; We're using Cloudera's CDH3 beta 4 release.&nbsp;&nbsp; Maybe they've = patched in the<br> &gt; FileOutputCommitter stuff into their release as it's based on Hadoop 0= .20.2<br> &gt; Looking at the source for Chukwa 0.3 (version we are on) the<br> &gt; MoveToRepository class skips the _log and _temporary directories.<br> &gt;<br> &gt; Seems like Chukwa should skip the _SUCCESS directory as well?&nbsp; Or= could a<br> &gt; more general skip be used like skip anything starting with and undersc= ore?<br> &gt; (maybe too aggressive).<br> &gt;<br> &gt; Does Chukwa 0.4 or 0.5 fix this issue? (I'm probably going to just hav= e to<br> &gt; patch 0.3 but maybe just another reason to upgrade.)<br> &gt;<br> &gt; On Thu, Feb 24, 2011 at 12:20 PM, Jerome Boulon &lt;<a href=3D"jboulon= @netflix.com">dustinhughes@example.org</a>&gt; wrote:<br> &gt;&gt;<br> &gt;&gt; This filename is coming from<br> &gt;&gt; here:&nbsp;<a href=3D"http://hadoop.apache.org/mapreduce/docs/r0.2= 1.0/api/constant-values.html">http://hadoop.apache.org/mapreduce/docs/r0.21= .0/api/constant-values.html</a><br> &gt;&gt; In general for hadoop you may want to avoid looking at any &quot;_= *&quot; file since<br> &gt;&gt; those are Hadoop related files like (_temporary, _log,=85)<br>&gt;= &gt; /Jerome.<br> &gt;&gt; From: Eric Yang &lt;<a href=3D"dustinhughes@example.org">eyang@yahoo-in= c.com</a>&gt;<br> &gt;&gt; Reply-To: &quot;<a href=3D"dustinhughes@example.org">chukw= dustinhughes@example.org</a>&quot;<br> &gt;&gt; &lt;<a href=3D"dustinhughes@example.org">chukwa-user@incub= ator.apache.org</a>&gt;<br> &gt;&gt; Date: Thu, 24 Feb 2011 10:55:57 -0800<br> &gt;&gt; To: &quot;<a href=3D"dustinhughes@example.org">chukwa-user= @incubator.apache.org</a>&quot; &lt;<a href=3D"dustinhughes@example.org= .org">dustinhughes@example.org</a>&gt;<br> &gt;&gt; Subject: Re: _SUCCESS files appearing in demuxOutput<br> &gt;&gt;<br> &gt;&gt; Hi Corbin,<br> &gt;&gt;<br> &gt;&gt; I have not seen this. &nbsp;What is the version of hadoop that you= are using,<br> &gt;&gt; are you using 0.21? &nbsp;It looks like the _SUCCESS file is spill= out after<br> &gt;&gt; demux mapreduce job. &nbsp;There are two possibilities leading to = the creation of<br> &gt;&gt; this file. &nbsp;Demux is modified and it is doing something that = is unexpected,<br> &gt;&gt; or the mapreduce framework 0.21 put that file there.<br> &gt;&gt; If you are using 0.21, I would recommend to avoid it.<br> &gt;&gt;<br> &gt;&gt; A more stable version of Hadoop is 0.20.100 branch, and you can do= wnload<br> &gt;&gt; it from:<br> &gt;&gt;<br> &gt;&gt; <a href=3D"http://people.apache.org/~eyang/">http://people.apache.= org/~eyang/</a><br> &gt;&gt;<br> &gt;&gt; Regards,<br> &gt;&gt; Eric<br> &gt;&gt;<br> &gt;&gt; On 2/24/11 10:12 AM, &quot;Corbin Hoenes&quot; &lt;<a href=3D"corb= dustinhughes@example.org">dustinhughes@example.org</a>&gt; wrote:<br> &gt;&gt;<br> &gt;&gt; Anyone seen this?<br> &gt;&gt;<br> &gt;&gt; /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS<br> &gt;&gt;<br> &gt;&gt; I clean them out and I keep getting the same file showing up and c= hukwa<br> &gt;&gt; doesn't know how to handle it:<br> &gt;&gt;<br> &gt;&gt; postProcess.log:<br> &gt;&gt; 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesin= g<br> &gt;&gt; Cluster (_SUCCESS)<br> &gt;&gt; 2011-02-21 06:51:55,027 INFO main MoveToRepository -<br> &gt;&gt; processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS)<br> &gt;&gt; 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in<= br> &gt;&gt; processDemuxOutput:<br> &gt;&gt; java.io.IOException:<br> &gt;&gt; hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_S= UCCESS is<br> &gt;&gt; not a directory!<br> &gt;&gt; &nbsp;&nbsp;&nbsp; at<br> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.process= ClutserDirectory(MoveToRepository.java:54)<br> &gt;&gt; &nbsp;&nbsp;&nbsp; at<br> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.main(Mo= veToRepository.java:250)<br> &gt;&gt; &nbsp;&nbsp;&nbsp; at<br> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.mov= etoMainRepository(PostProcessorManager.java:201)<br> &gt;&gt; &nbsp;&nbsp;&nbsp; at<br> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.sta= rt(PostProcessorManager.java:146)<br> &gt;&gt; &nbsp;&nbsp;&nbsp; at<br> &gt;&gt; org.apache.hadoop.chukwa.extraction.demux.PostProcessorManager.mai= n(PostProcessorManager.java:80)<br> &gt;&gt;<br> &gt;&gt;<br> &gt;<br> &gt;<br><br><br><br> --<br> Ari Rabkin <a href=3D"dustinhughes@example.org">dustinhughes@example.org</a><br> UC Berkeley Computer Science Department<br><br></span></font></blockquote><= /div></div></span></body></html> --_000_C98C39C818871jboulonnetflixcom_-- From chukwa-user-returndustinhughes@example.org Fri Feb 25 01:37:29 2011 Return-Path: <chukwa-user-returndustinhughes@example.org> Delivered-To: dustinhughes@example.org Received: (qmail 38051 invoked from network); 25 Feb 2011 01:37:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 25 Feb 2011 01:37:29 -0000 Received: (qmail 62560 invoked by uid 500); 25 Feb 2011 01:37:29 -0000 Delivered-To: dustinhughes@example.org Received: (qmail 62491 invoked by uid 500); 25 Feb 2011 01:37:28 -0000 Mailing-List: contact dustinhughes@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:dustinhughes@example.org> List-Unsubscribe: <mailto:dustinhughes@example.org> List-Post: <mailto:dustinhughes@example.org> List-Id: <chukwa-user.incubator.apache.org> Reply-To: dustinhughes@example.org Delivered-To: mailing list dustinhughes@example.org Received: (qmail 62484 invoked by uid 99); 25 Feb 2011 01:37:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 01:37:28 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 83.161.120.155 is neither permitted nor denied by domain of dustinhughes@example.org) Received: from [83.161.120.155] (HELO mail-fx0-f47.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Feb 2011 01:37:24 +0000 Received: by fxm8 with SMTP id 8so1173128fxm.6 for <dustinhughes@example.org>; Thu, 24 Feb 2011 17:37:02 -0800 (PST) Received: by 83.161.120.155 with SMTP id i3mr1920515faj.149.1298597822344; Thu, 24 Feb 2011 17:37:02 -0800 (PST) References: <dustinhughes@example.org> From: James Seigel <dustinhughes@example.org> In-Reply-To: <dustinhughes@example.org> Mime-Version: 1.0 (iPhone Mail 8C148) Date: Thu, 24 Feb 2011 18:36:59 -0700 Message-ID: <-8206573989064901689@unknownmsgid> Subject: Re: _SUCCESS files appearing in demuxOutput To: "dustinhughes@example.org" <dustinhughes@example.org> Content-Type: multipart/alternative; boundary=20cf3054a613e82f04049d115d9a --20cf3054a613e82f04049d115d9a Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable +1 Sent from my mobile. Please excuse the typos. On 2011-02-24, at 5:33 PM, Eric Yang <dustinhughes@example.org> wrote: I also support for skipping _*. Regards, Eric On 2/24/11 4:27 PM, "Ariel Rabkin" <dustinhughes@example.org> wrote: I don't think this has been fixed yet in trunk, let alone 0.4. I would support skipping everything starting with _. Is there an actual use case this would break? --Ari On Thu, Feb 24, 2011 at 4:24 PM, Corbin Hoenes <dustinhughes@example.org> wrote: > We're using Cloudera's CDH3 beta 4 release. Maybe they've patched in th= e > FileOutputCommitter stuff into their release as it's based on Hadoop 0.20.2 > Looking at the source for Chukwa 0.3 (version we are on) the > MoveToRepository class skips the _log and _temporary directories. > > Seems like Chukwa should skip the _SUCCESS directory as well? Or could a > more general skip be used like skip anything starting with and underscore= ? > (maybe too aggressive). > > Does Chukwa 0.4 or 0.5 fix this issue? (I'm probably going to just have t= o > patch 0.3 but maybe just another reason to upgrade.) > > On Thu, Feb 24, 2011 at 12:20 PM, Jerome Boulon <dustinhughes@example.org> wrote: >> >> This filename is coming from >> here: http://hadoop.apache.org/mapreduce/docs/r0.21.0/api/constant-values.html >> In general for hadoop you may want to avoid looking at any "_*" file since >> those are Hadoop related files like (_temporary, _log,=85) >> /Jerome. >> From: Eric Yang <dustinhughes@example.org> >> Reply-To: "dustinhughes@example.org" >> <dustinhughes@example.org> >> Date: Thu, 24 Feb 2011 10:55:57 -0800 >> To: "dustinhughes@example.org" <dustinhughes@example.org= > >> Subject: Re: _SUCCESS files appearing in demuxOutput >> >> Hi Corbin, >> >> I have not seen this. What is the version of hadoop that you are using, >> are you using 0.21? It looks like the _SUCCESS file is spill out after >> demux mapreduce job. There are two possibilities leading to the creatio= n of >> this file. Demux is modified and it is doing something that is unexpected, >> or the mapreduce framework 0.21 put that file there. >> If you are using 0.21, I would recommend to avoid it. >> >> A more stable version of Hadoop is 0.20.100 branch, and you can download >> it from: >> >> http://people.apache.org/~eyang/ >> >> Regards, >> Eric >> >> On 2/24/11 10:12 AM, "Corbin Hoenes" <dustinhughes@example.org> wrote: >> >> Anyone seen this? >> >> /chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS >> >> I clean them out and I keep getting the same file showing up and chukwa >> doesn't know how to handle it: >> >> postProcess.log: >> 2011-02-21 06:51:55,027 INFO main MoveToRepository - main procesing >> Cluster (_SUCCESS) >> 2011-02-21 06:51:55,027 INFO main MoveToRepository - >> processClutserDirectory (_SUCCESS,/chukwa/repos//_SUCCESS) >> 2011-02-21 06:51:55,028 WARN main PostProcessorManager - Error in >> processDemuxOutput: >> java.io.IOException: >> hdfs://cluster1/chukwa/postProcess/demuxOutputDir_1298061686862/_SUCCESS is >> not a directory! >> at >> org.apache.hadoop.chukwa.extraction.demux.MoveToRepository.processClutserDi=
{'source': 'Apache Software Foundation Public Mail Archives'}