From chukwa-usermatthew58@example.com Mon Jan 04 18:06:08 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 10700 invoked from network); 4 Jan 2010 18:06:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2010 18:06:08 -0000 Received: (qmail 37913 invoked by uid 500); 4 Jan 2010 18:06:08 -0000 Delivered-To: matthew58@example.com Received: (qmail 37853 invoked by uid 500); 4 Jan 2010 18:06:07 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 37844 invoked by uid 99); 4 Jan 2010 18:06:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 18:06:07 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of matthew58@example.com does not designate 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO sh1.exchange.ms) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jan 2010 18:05:57 +0000 Received: by sh1.exchange.ms (Postfix, from userid 115) id CDC532D84FD; Mon, 4 Jan 2010 13:08:06 -0500 (EST) Received: from mse16fe1.mse16.exchange.ms (unknown [83.161.120.155]) by sh1.exchange.ms (Postfix) with ESMTP id BB0F42D88C0 for <matthew58@example.com>; Mon, 4 Jan 2010 13:06:04 -0500 (EST) Received: from [83.161.120.155] ([83.161.120.155]) by mse16fe1.mse16.exchange.ms with Microsoft SMTPSVC9738655671); Mon, 4 Jan 2010 13:03:16 -0500 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1077) Subject: Re: Scribe vs. Chukwa From: Corbin Hoenes <matthew58@example.com> In-Reply-To: <matthew58@example.com> Date: Mon, 4 Jan 2010 11:03:15 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <matthew58@example.com> References: <matthew58@example.com> <matthew58@example.com> To: matthew58@example.com X-Mailer: Apple Mail (2.1077) X-OriginalArrivalTime: 04 Jan 2010 18:03:17.0145 (UTC) FILETIME=[30FC8090:01CA8D68] X-Virus-Checked: Checked by ClamAV on apache.org Ari, Thanks for the input. I found another thread on the general hadoop list = that pertains to the the discussion. I'll post here for future readers: = http://mail-archives.apache.org/mod_mbox/hadoop-general/200911.mbox/%3cC73= matthew58@example.com%3e Your explanation helps a lot though. I'd second the idea of have a = Chukwa-vs page on the website to help people understand what it can do = for them. We are trying to get our logs into HDFS so I think chukwa = will provide a lot of value in this regard. -- Corbin On Dec 24, 2009, at 9:51 AM, Ariel Rabkin wrote: > Howdy. >=20 > It's a good question, and maybe we should have a > Chukwa-vs-the-competition page on the website. >=20 > Chukwa is designed to help collect system logs, and store them in > HDFS. Scribe is designed as more of a syslog++. We went to some > trouble to make sure our metadata and output formats were MapReduce > friendly, whereas I think they leave that much more up to you. >=20 > My understanding is that Scribe exposes a Thrift service, only. So if > you want to use Scribe, you pretty much have to go all-in and modify > your app and deployment to send logs via Thrift. Chukwa can just read > whatever files or data sources are lying around. We can pretty easily > listen for incoming Thrift RPC calls -- I believe Jerome is currently > coding that up. >=20 > Chukwa comes with visualization and processing tools, whereas Scribe = doesn't. >=20 > Scribe might be slightly lower overhead -- I believe it's C++ and > Chukwa is Java. Chukwa isn't too big, either, so the absolute > difference might not be significant. >=20 > --Ari >=20 > On Wed, Dec 23, 2009 at 3:06 PM, Corbin Hoenes <matthew58@example.com> = wrote: >> Does anyone know of a comparison between Chukwa and Scribe( = http://sameerparwani.com/posts/facebook-scribe-server-documentation-and-tu= torials ) >> I am interested in why someone might choose Chukwa vs. Scribe and = their pros/cons. >>=20 >>=20 >>=20 >=20 >=20 >=20 > --=20 > Ari Rabkin matthew58@example.com > UC Berkeley Computer Science Department Corbin Hoenes matthew58@example.com skype: choenes From chukwa-usermatthew58@example.com Tue Jan 05 19:52:47 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 76998 invoked from network); 5 Jan 2010 19:52:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2010 19:52:47 -0000 Received: (qmail 38363 invoked by uid 500); 5 Jan 2010 19:52:47 -0000 Delivered-To: matthew58@example.com Received: (qmail 38309 invoked by uid 500); 5 Jan 2010 19:52:47 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 38300 invoked by uid 99); 5 Jan 2010 19:52:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 19:52:47 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (athena.apache.org: transitioning domain of matthew58@example.com does not designate 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO sh1.exchange.ms) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 19:52:38 +0000 Received: by sh1.exchange.ms (Postfix, from userid 115) id 900CE2D8611; Tue, 5 Jan 2010 14:54:51 -0500 (EST) Received: from mse16fe1.mse16.exchange.ms (unknown [83.161.120.155]) by sh1.exchange.ms (Postfix) with ESMTP id D84402D8513 for <matthew58@example.com>; Tue, 5 Jan 2010 14:53:59 -0500 (EST) Received: from [83.161.120.155] ([83.161.120.155]) by mse16fe1.mse16.exchange.ms with Microsoft SMTPSVC9738655671); Tue, 5 Jan 2010 14:51:09 -0500 From: Corbin Hoenes <matthew58@example.com> Content-Type: multipart/alternative; boundary=Apple-Mail-1-946027132 Subject: who transfers the data? Date: Tue, 5 Jan 2010 12:51:08 -0700 Message-Id: <matthew58@example.com> To: matthew58@example.com Mime-Version: 1.0 (Apple Message framework v1077) X-Mailer: Apple Mail (2.1077) X-OriginalArrivalTime: 05 Jan 2010 19:51:10.0119 (UTC) FILETIME=[6D97B370:01CA8E40] --Apple-Mail-1-946027132 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii It's a little unclear to me who is transferring the chunks to the = collectors. Does each adaptor have a connection or does the agent have = a single connection to the collector? For example if I have 10 log = files that I am tailing (an adaptor for each) do they all go to the same = collector or does it distribute those to any one of the collectors I = have listed in my collectors file? =20 http://hadoop.apache.org/chukwa/docs/current/design.html#Collectors "Rather than have each adaptor write directly to HDFS, data is sent = across the network to a collector process, that does the HDFS writes. = Each collector receives data from up to several hundred hosts, and = writes all this data to a single sink file, which is a Hadoop sequence = file of serialized Chunks. Periodically, collectors close their sink = files, rename them to mark them available for processing, and resume = writing a new file. Data is sent to collectors over HTTP." Corbin Hoenes matthew58@example.com skype: choenes --Apple-Mail-1-946027132 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; = "><div>It's a little unclear to me who is transferring the chunks to the = collectors. &nbsp;Does each adaptor have a connection or does the agent = have a single connection to the collector? &nbsp; For example if I have = 10 log files that I am tailing (an adaptor for each) do they all go to = the same collector or does it distribute those to any one of the = collectors I have listed in my collectors file? = &nbsp;</div><div><br></div><a = href=3D"http://hadoop.apache.org/chukwa/docs/current/design.html#Collector= s">http://hadoop.apache.org/chukwa/docs/current/design.html#Collectors</a>= <div><br></div><div><span class=3D"Apple-style-span" style=3D"font-family:= Verdana, Helvetica, sans-serif; font-size: 13px; line-height: 15px; = ">"Rather than have each adaptor write directly to HDFS, data is sent = across the network to a&nbsp;<em>collector</em>&nbsp;process, that does = the HDFS writes. Each collector receives data from up to several hundred = hosts, and writes all this data to a single&nbsp;<em>sink file</em>, = which is a Hadoop sequence file of serialized Chunks. Periodically, = collectors close their sink files, rename them to mark them available = for processing, and resume writing a new file. Data is sent to = collectors over HTTP."</span></div><div><font class=3D"Apple-style-span" = face=3D"Verdana, Helvetica, sans-serif" size=3D"3"><span = class=3D"Apple-style-span" style=3D"font-size: 13px; line-height: = 15px;"><br></span></font></div><div><font class=3D"Apple-style-span" = face=3D"Verdana, Helvetica, sans-serif" size=3D"3"><span = class=3D"Apple-style-span" style=3D"font-size: 13px; line-height: = 15px;"><br></span></font></div><div><font class=3D"Apple-style-span" = face=3D"Verdana, Helvetica, sans-serif" size=3D"3"><span = class=3D"Apple-style-span" style=3D"font-size: 13px; line-height: = 15px;"><br></span></font></div><div><font class=3D"Apple-style-span" = face=3D"Verdana, Helvetica, sans-serif" size=3D"3"><span = class=3D"Apple-style-span" style=3D"font-size: 13px; line-height: = 15px;"><br></span></font><div> <div>Corbin Hoenes</div><div>matthew58@example.com</div><div>skype: = choenes</div><div><br></div><br class=3D"Apple-interchange-newline"> </div> <br></div></body></html>= --Apple-Mail-1-946027132-- From chukwa-usermatthew58@example.com Tue Jan 05 20:07:42 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 80933 invoked from network); 5 Jan 2010 20:07:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2010 20:07:41 -0000 Received: (qmail 62066 invoked by uid 500); 5 Jan 2010 20:07:41 -0000 Delivered-To: matthew58@example.com Received: (qmail 62043 invoked by uid 500); 5 Jan 2010 20:07:41 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 62034 invoked by uid 99); 5 Jan 2010 20:07:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 20:07:41 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthew58@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-pw0-f50.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Jan 2010 20:07:34 +0000 Received: by pwi20 with SMTP id 20so11234757pwi.29 for <matthew58@example.com>; Tue, 05 Jan 2010 12:07:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=USkhy5fR9NmWc6XZxey9cF8CrdVxCIm+AtFBSrM2A8Q=; b=LroucSIDMyayvS9H2kXJYvOCjx38Qw6yBLA8KtNJx6YTv2BkrV4hOfDuOlw4vs5Edv aGTinFeJdi+KvMVnyrJFg2fwJ8HPhUhQBMYn0kGG5lQOmyVT6g1NTqGeGbWK8wQYovAP MerzzSH0FlA+ZY3fjZU97cHPIt1ykfJqMkUpU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=A6wZ5xYZxXPThunqy+fs1PwVjAYr7ygYNeIcViDS+Rp5mbk54K3FV4rNZ/2zw4+mOa EmZ2wda0pmPusboZzsdjQUo8VHggVFJtPaGl13AqGMUoGnmwEsmJEqrpwLFLWAHE4jSi cyq7bQb+EuFqLKnZVdsOpWTSvT2T1CIhEFpJY= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 20mr348391wff.262.1262722034582; Tue, 05 Jan 2010 12:07:14 -0800 (PST) In-Reply-To: <matthew58@example.com> References: <matthew58@example.com> Date: Tue, 5 Jan 2010 15:07:14 -0500 Message-ID: <matthew58@example.com> Subject: Re: who transfers the data? From: Ariel Rabkin <matthew58@example.com> To: matthew58@example.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable There is one connection from the agent to a collector, that carries all the data coming from that agent. A collector can saturate the single-writer bandwidth of HDFS -- call it 20-40 MB/sec. If you have a single agent writing a large fraction of that, say more than 4 MB/sec, then Chukwa is probably not solving your problem anyway. On Tue, Jan 5, 2010 at 2:51 PM, Corbin Hoenes <matthew58@example.com> wrote: > It's a little unclear to me who is transferring the chunks to the > collectors. =A0Does each adaptor have a connection or does the agent have= a > single connection to the collector? =A0 For example if I have 10 log file= s > that I am tailing (an adaptor for each) do they all go to the same collec= tor > or does it distribute those to any one of the collectors I have listed in= my > collectors file? > http://hadoop.apache.org/chukwa/docs/current/design.html#Collectors > "Rather than have each adaptor write directly to HDFS, data is sent acros= s > the network to a=A0collector=A0process, that does the HDFS writes. Each > collector receives data from up to several hundred hosts, and writes all > this data to a single=A0sink file, which is a Hadoop sequence file of > serialized Chunks. Periodically, collectors close their sink files, renam= e > them to mark them available for processing, and resume writing a new file= . > Data is sent to collectors over HTTP." > > > > Corbin Hoenes > matthew58@example.com > skype: choenes > > > --=20 Ari Rabkin matthew58@example.com UC Berkeley Computer Science Department From chukwa-usermatthew58@example.com Tue Jan 12 18:24:57 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 51587 invoked from network); 12 Jan 2010 18:24:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 12 Jan 2010 18:24:57 -0000 Received: (qmail 27137 invoked by uid 500); 12 Jan 2010 18:24:57 -0000 Delivered-To: matthew58@example.com Received: (qmail 27065 invoked by uid 500); 12 Jan 2010 18:24:56 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 27056 invoked by uid 99); 12 Jan 2010 18:24:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 18:24:56 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthew58@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-pw0-f50.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 18:24:50 +0000 Received: by pwi20 with SMTP id 20so2463065pwi.29 for <matthew58@example.com>; Tue, 12 Jan 2010 10:24:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:date:message-id :subject:from:to:content-type; bh=484OPWqogCNaHLFeyI8uIFH2DkqmFVkTW6+oCCygU6M=; b=oRSCAppVC+QOWR938hAtwovW7tboIdSXizxhkNIFKkJQ5jKRhhDvuM3X6iS3MtG8Ne 4nlj3WXQgvLx3JrbbT+Sce7ESR6xywinrxJcXcae8gE399GwwBjFBdEsy2OJ1UhUhp2f /jelsRMid4WvxzvoZBtb5oz5M5rkyDJRjSnC8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:date:message-id:subject:from:to:content-type; b=ZSZNSQ3ubc36UIb2OkimnzGxKHou8yHH3l3XQsWD+QbXq9Dl5uyz0XDwmUxWj/lSa9 EuWLy/pfEgDyKpBpkYQYQFumDxQq+hhKlrmCOZHjshRC9ePyavexRmjewvKOVvnyiN+w RC246XzWXWPQLTU1CSYMnygNR14Tl4483V0S4= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id t4mr1465074wfc.256.1263320669827; Tue, 12 Jan 2010 10:24:29 -0800 (PST) Reply-To: matthew58@example.com Date: Tue, 12 Jan 2010 10:24:29 -0800 Message-ID: <matthew58@example.com> Subject: suggested changes to tools/init.d scripts From: Bill Graham <matthew58@example.com> To: matthew58@example.com Content-Type: multipart/alternative; boundary=001636e90c3cc332e3047cfbc3e3 --001636e90c3cc332e3047cfbc3e3 Content-Type: text/plain; charset=ISO-8859-1 Hi, To get the scripts in the tools/init.d folder to run in our environment I had to make a few of the same tweaks to each of them. I'd like to suggest some refactoring of these scripts to make them more easy to use. These are the issues I came across: 1. The scripts try to write lock files to /var/lock/subsys, which is owned by root. Can we change this location to be somewhere that doesn't require root access? 2. The actual run command does an su to the CHUKWA_USER which also caused problems for us. It seems like it would be cleaner to not embed the su calls in the script, but instead allow the user to su when they run the script (which worked much better for us). That way everything done by the script would be done by the same user. 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. Hard coded defaults is ok, but the ability to override them without modifying the scripts would be ideal. Let me know if you think these changes make sense and I'll open a JIRA. thanks, Bill --001636e90c3cc332e3047cfbc3e3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi,<br><br>To get the scripts in the tools/init.d folder to run in our envi= ronment I had to make a few of the same tweaks to each of them. I&#39;d lik= e to suggest some refactoring of these scripts to make them more easy to us= e. These are the issues I came across:<br> <br>1. The scripts try to write lock files to /var/lock/subsys, which is ow= ned by root. Can we change this location to be somewhere that doesn&#39;t r= equire root access?<br><br>2. The actual run command does an su to the CHUK= WA_USER which also caused problems for us. It seems like it would be cleane= r to not embed the su calls in the script, but instead allow the user to su= when they run the script (which worked much better for us). That way every= thing done by the script would be done by the same user.<br> <br>3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard co= ded. Hard coded defaults is ok, but the ability to override them without mo= difying the scripts would be ideal.<br><br>Let me know if you think these c= hanges make sense and I&#39;ll open a JIRA.<br> <br>thanks,<br>Bill<br> --001636e90c3cc332e3047cfbc3e3-- From chukwa-usermatthew58@example.com Tue Jan 12 18:42:46 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 66098 invoked from network); 12 Jan 2010 18:42:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 12 Jan 2010 18:42:46 -0000 Received: (qmail 55973 invoked by uid 500); 12 Jan 2010 18:42:46 -0000 Delivered-To: matthew58@example.com Received: (qmail 55911 invoked by uid 500); 12 Jan 2010 18:42:46 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 55902 invoked by uid 99); 12 Jan 2010 18:42:46 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 18:42:46 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthew58@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-pz0-f198.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 18:42:39 +0000 Received: by pzk36 with SMTP id 36so14910190pzk.5 for <matthew58@example.com>; Tue, 12 Jan 2010 10:42:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=6bfa3kBdwPloHduP1QCoZ3vdZkeDveNyNoxJXcrZ3dM=; b=cOzSDY30ZSVoeB5bDuDescPYMdZUOPbHSAqgz5BDr6czdbwQFxAZCgUO1NKqXsLKAO qZNMwuCQkGdQuZKGZdCfW27iXjmS5kbg2Uwvaks7ugZNvuEe9kJXnBGZBun1CumAAEb2 tpHx9rWCTLGj78bt2zCcxMHMw0J8tDpHLJ4Bo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=LIPnPr54vOHZn6psuGB9lRYaZ+4NDSUnHzr9YVvW3YcEyQu82ds/N1As3ueUwrD12X uag2ZGFbFLRbx/F57RKpMel0zTzYEeu8NcUmYlJnfle/zDBPgsqvgCiqUqSgCtc7P7FZ cBvoU74j8XhQQkJcD6MYFHunAhC2lm2MAGU5c= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 2mr4051892wfa.243.1263321738809; Tue, 12 Jan 2010 10:42:18 -0800 (PST) In-Reply-To: <matthew58@example.com> References: <matthew58@example.com> Date: Tue, 12 Jan 2010 10:42:18 -0800 Message-ID: <matthew58@example.com> Subject: Re: suggested changes to tools/init.d scripts From: Ariel Rabkin <matthew58@example.com> To: matthew58@example.com, matthew58@example.com Content-Type: text/plain; charset=ISO-8859-1 Go ahead and open the JIRA. It's not clear to me how commonly used those scripts are. My site uses runit instead. But if improving them would make them more useful for you, go for it. --Ari On Tue, Jan 12, 2010 at 10:24 AM, Bill Graham <matthew58@example.com> wrote: > Hi, > > To get the scripts in the tools/init.d folder to run in our environment I > had to make a few of the same tweaks to each of them. I'd like to suggest > some refactoring of these scripts to make them more easy to use. These are > the issues I came across: > > 1. The scripts try to write lock files to /var/lock/subsys, which is owned > by root. Can we change this location to be somewhere that doesn't require > root access? > > 2. The actual run command does an su to the CHUKWA_USER which also caused > problems for us. It seems like it would be cleaner to not embed the su calls > in the script, but instead allow the user to su when they run the script > (which worked much better for us). That way everything done by the script > would be done by the same user. > > 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard coded. > Hard coded defaults is ok, but the ability to override them without > modifying the scripts would be ideal. > > Let me know if you think these changes make sense and I'll open a JIRA. > > thanks, > Bill > -- Ari Rabkin matthew58@example.com UC Berkeley Computer Science Department From chukwa-usermatthew58@example.com Tue Jan 12 18:49:48 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 70381 invoked from network); 12 Jan 2010 18:49:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 12 Jan 2010 18:49:48 -0000 Received: (qmail 77146 invoked by uid 500); 12 Jan 2010 18:49:48 -0000 Delivered-To: matthew58@example.com Received: (qmail 77105 invoked by uid 500); 12 Jan 2010 18:49:48 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 77096 invoked by uid 99); 12 Jan 2010 18:49:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 18:49:48 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthew58@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-pz0-f198.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 18:49:41 +0000 Received: by pzk36 with SMTP id 36so14915136pzk.5 for <matthew58@example.com>; Tue, 12 Jan 2010 10:49:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=67zZXF4KHH45QtqHumN9UO82iFHNpdUjrWPhnUhcoG8=; b=IAxKBk+LYPcpTrDk1NhNoF/QTz3PQsw+vNLAmb0QfVfE7AGZA4+ulXQ7oUFTTpwoxV y4BE/KOPRt37D+cgIKH4DPNQ15G6FdMQXkldphk2stMqK/H6bGMTE93XFo7hNarWiyIt xWBG4lpKD8CFACFZujafDH0+ZQ2n9PPpHoQyI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; b=a/evUiObmke67uj1wgjk0vlW3Fg9yrW8eQOntosBf0+jlXP/QBxRdX/WB0lwaPBdu+ ROhwt8iaiqiv7J8ay3qpYODLNdPu5YsnlEcsEV47uE/wEWNsuBLcndPoyNbCrBdNpvCg j7+ALaSKzJuoVZ+JK4rr2A8a9mee8G/UEfifg= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id c39mr5859679wfj.249.1263322161709; Tue, 12 Jan 2010 10:49:21 -0800 (PST) Reply-To: matthew58@example.com In-Reply-To: <matthew58@example.com> References: <matthew58@example.com> <matthew58@example.com> Date: Tue, 12 Jan 2010 10:49:21 -0800 Message-ID: <matthew58@example.com> Subject: Re: suggested changes to tools/init.d scripts From: Bill Graham <matthew58@example.com> To: Ariel Rabkin <matthew58@example.com> Cc: matthew58@example.com Content-Type: multipart/alternative; boundary=001636e0b61eaf8186047cfc1cc0 --001636e0b61eaf8186047cfc1cc0 Content-Type: text/plain; charset=ISO-8859-1 Thanks Ari, done. https://issues.apache.org/jira/browse/CHUKWA-446 On Tue, Jan 12, 2010 at 10:42 AM, Ariel Rabkin <matthew58@example.com> wrote: > Go ahead and open the JIRA. > > It's not clear to me how commonly used those scripts are. My site uses > runit instead. But if improving them would make them more useful for > you, go for it. > > --Ari > > On Tue, Jan 12, 2010 at 10:24 AM, Bill Graham <matthew58@example.com> > wrote: > > Hi, > > > > To get the scripts in the tools/init.d folder to run in our environment I > > had to make a few of the same tweaks to each of them. I'd like to suggest > > some refactoring of these scripts to make them more easy to use. These > are > > the issues I came across: > > > > 1. The scripts try to write lock files to /var/lock/subsys, which is > owned > > by root. Can we change this location to be somewhere that doesn't require > > root access? > > > > 2. The actual run command does an su to the CHUKWA_USER which also caused > > problems for us. It seems like it would be cleaner to not embed the su > calls > > in the script, but instead allow the user to su when they run the script > > (which worked much better for us). That way everything done by the script > > would be done by the same user. > > > > 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard > coded. > > Hard coded defaults is ok, but the ability to override them without > > modifying the scripts would be ideal. > > > > Let me know if you think these changes make sense and I'll open a JIRA. > > > > thanks, > > Bill > > > > > > -- > Ari Rabkin matthew58@example.com > UC Berkeley Computer Science Department > --001636e0b61eaf8186047cfc1cc0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Ari, done.<br><a href=3D"https://issues.apache.org/jira/browse/CHUKW= A-446">https://issues.apache.org/jira/browse/CHUKWA-446</a><br><br><div cla= ss=3D"gmail_quote">On Tue, Jan 12, 2010 at 10:42 AM, Ariel Rabkin <span dir= =3D"ltr">&lt;<a href=3D"mailto:matthew58@example.com">matthew58@example.com</a>&g= t;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, = 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Go ahead and open= the JIRA.<br> <br> It&#39;s not clear to me how commonly used those scripts are. My site uses<= br> runit instead. But if improving them would make them more useful for<br> you, go for it.<br> <br> --Ari<br> <div><div></div><div class=3D"h5"><br> On Tue, Jan 12, 2010 at 10:24 AM, Bill Graham &lt;<a href=3D"mailto:billgra= matthew58@example.com">matthew58@example.com</a>&gt; wrote:<br> &gt; Hi,<br> &gt;<br> &gt; To get the scripts in the tools/init.d folder to run in our environmen= t I<br> &gt; had to make a few of the same tweaks to each of them. I&#39;d like to = suggest<br> &gt; some refactoring of these scripts to make them more easy to use. These= are<br> &gt; the issues I came across:<br> &gt;<br> &gt; 1. The scripts try to write lock files to /var/lock/subsys, which is o= wned<br> &gt; by root. Can we change this location to be somewhere that doesn&#39;t = require<br> &gt; root access?<br> &gt;<br> &gt; 2. The actual run command does an su to the CHUKWA_USER which also cau= sed<br> &gt; problems for us. It seems like it would be cleaner to not embed the su= calls<br> &gt; in the script, but instead allow the user to su when they run the scri= pt<br> &gt; (which worked much better for us). That way everything done by the scr= ipt<br> &gt; would be done by the same user.<br> &gt;<br> &gt; 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard c= oded.<br> &gt; Hard coded defaults is ok, but the ability to override them without<br= > &gt; modifying the scripts would be ideal.<br> &gt;<br> &gt; Let me know if you think these changes make sense and I&#39;ll open a = JIRA.<br> &gt;<br> &gt; thanks,<br> &gt; Bill<br> &gt;<br> <br> <br> <br> </div></div><font color=3D"#888888">--<br> Ari Rabkin <a href=3D"mailto:matthew58@example.com">matthew58@example.com</a><br> UC Berkeley Computer Science Department<br> </font></blockquote></div><br> --001636e0b61eaf8186047cfc1cc0-- From chukwa-usermatthew58@example.com Tue Jan 12 18:51:21 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 70843 invoked from network); 12 Jan 2010 18:51:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 12 Jan 2010 18:51:21 -0000 Received: (qmail 77746 invoked by uid 500); 12 Jan 2010 18:51:21 -0000 Delivered-To: matthew58@example.com Received: (qmail 77695 invoked by uid 500); 12 Jan 2010 18:51:21 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 77686 invoked by uid 99); 12 Jan 2010 18:51:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 18:51:21 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 18:51:10 +0000 Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [83.161.120.155]) by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id o0CIolwv032439; Tue, 12 Jan 2010 10:50:47 -0800 (PST) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:user-agent:date:subject:from:to:message-id: thread-topic:thread-index:in-reply-to:mime-version:content-type: content-transfer-encoding:x-originalarrivaltime; b=xEZ8eYVa1QlYpBWrBmTuyy/CK7erZ3o2z9d8NdoeZqaxbwUjt12UCN8fhaIpro1g Received: from SNV-EXVS06.ds.corp.yahoo.com ([83.161.120.155]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC9738655671); Tue, 12 Jan 2010 10:50:47 -0800 Received: from 83.161.120.155 ([83.161.120.155]) by SNV-EXVS06.ds.corp.yahoo.com ([83.161.120.155]) via Exchange Front-End Server snv-webmail.corp.yahoo.com ([83.161.120.155]) with Microsoft Exchange Server HTTP-DAV ; Tue, 12 Jan 2010 18:50:47 +0000 User-Agent: Microsoft-Entourage/83.161.120.155001 Date: Tue, 12 Jan 2010 10:50:46 -0800 Subject: Re: suggested changes to tools/init.d scripts From: Eric Yang <matthew58@example.com> To: <matthew58@example.com>, <matthew58@example.com> Message-ID: <matthew58@example.com> Thread-Topic: suggested changes to tools/init.d scripts Thread-Index: AcqTuCZX51GKYEuJBkiO6+uQTw76AA== In-Reply-To: <matthew58@example.com> Mime-version: 1.0 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable X-OriginalArrivalTime: 12 Jan 2010 18:50:47.0426 (UTC) FILETIME=[2730DE20:01CA93B8] Hi Bill, The scripts are over due for a clean up. Majority of the scripts were designed to work in a specific environment for Yahoo. On 1/12/10 10:24 AM, "Bill Graham" <matthew58@example.com> wrote: > Hi, >=20 > To get the scripts in the tools/init.d folder to run in our environment I= had > to make a few of the same tweaks to each of them. I'd like to suggest som= e > refactoring of these scripts to make them more easy to use. These are the > issues I came across: >=20 > 1. The scripts try to write lock files to /var/lock/subsys, which is owne= d by > root. Can we change this location to be somewhere that doesn't require ro= ot > access? This was designed to run as /etc/init.d script. I would recommend to remov= e tools/init.d and /etc/init.d script, and focus on using start-*.sh stop-*.s= h commands like hadoop. > 2. The actual run command does an su to the CHUKWA_USER which also caused > problems for us. It seems like it would be cleaner to not embed the su ca= lls > in the script, but instead allow the user to su when they run the script > (which worked much better for us). That way everything done by the script > would be done by the same user. Base on previous experience, we had a user who were running everything as root. Su was put in there to safe guard the program from that user. It wa= s a pain to clean up hundred of machines with root owned files. In a production environment, it is probably not Chukwa's responsibility to safe guard permission. +1 on removing su. > 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard code= d. > Hard coded defaults is ok, but the ability to override them without modif= ying > the scripts would be ideal. >=20 Current documentation doesn=B9t talk about those properties are configurable during build time. If you compile from trunk or 0.3 branch, you can copy default.properties to build.properties, and specify the path and running user. However, the replace only applies to /etc/init.d/chukwa-* scripts, and install from RPM. (Chukwa has an "ant rpm" target.) tools/init.d was meant as a template, hence no replacement took place. > Let me know if you think these changes make sense and I'll open a JIRA. Yes, open a jira, and we can discuss in depth of the changes. I would like to see /etc/init.d and tools/init.d to be removed completely, and use start-*.sh and stop-*.sh command to be consistent with hadoop. Regards, Eric From chukwa-usermatthew58@example.com Tue Jan 12 20:16:17 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 97323 invoked from network); 12 Jan 2010 20:16:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 12 Jan 2010 20:16:17 -0000 Received: (qmail 5782 invoked by uid 500); 12 Jan 2010 20:16:17 -0000 Delivered-To: matthew58@example.com Received: (qmail 5735 invoked by uid 500); 12 Jan 2010 20:16:16 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 5725 invoked by uid 99); 12 Jan 2010 20:16:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 20:16:16 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthew58@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-px0-f201.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2010 20:16:11 +0000 Received: by pxi39 with SMTP id 39so17198148pxi.2 for <matthew58@example.com>; Tue, 12 Jan 2010 12:15:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=go+e4XBsNpc6UGhSAdCbh9uEKLY64IkFenZLsU3AUmY=; b=pHjjDcGwrqc5IZU7rK/3wyIN+kohqZkU9/fNhcOLC9jPM/1ingICK6tFUoDG+U8ETL /3mUTrq+cNQLJNbEIxX0uBHfSTBNXEBXz2bUx/cW/mpZb54tlu2YGPclfrka+V5IRwB7 3mFMFBNgAu4H5MWmuh8BeHY3JvV+3sNWyiVxY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; b=eOdOjJnRjaGdfcXmSP7L9TxszJlJa1o4SCYzZYRzRglpqTYqTxjERJ9NSOckgCfgT2 /fe2ywf5AnZegQ7Zg51yzwjNIqv6X8uQTv7IMdRCyDnsLdBesCgBsvfNm6f2O1EdxXbW 2iYkaWkEN+MoPU0yKbNe1UVt0pRABebyCxcUs= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id r7mr6073569wfc.261.1263327350820; Tue, 12 Jan 2010 12:15:50 -0800 (PST) Reply-To: matthew58@example.com In-Reply-To: <matthew58@example.com> References: <matthew58@example.com> <matthew58@example.com> Date: Tue, 12 Jan 2010 12:15:50 -0800 Message-ID: <matthew58@example.com> Subject: Re: suggested changes to tools/init.d scripts From: Bill Graham <matthew58@example.com> To: Eric Yang <matthew58@example.com> Cc: matthew58@example.com Content-Type: multipart/alternative; boundary=001636e0b6b7fb10b6047cfd5126 --001636e0b6b7fb10b6047cfd5126 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Eric, that makes sense. I'm going to use the Hadoop-style start-*.sh scripts instead, since they seem to work better for my needs. The agent and data processor start/stop scripts work for me, but the bin/start-collectors.sh scripts fails: $ sda bin/start-collectors.sh sed: -e expression #1, char 11: unknown option to `s' Has anyone else seen this? It also seems like this script ultimately invokes slaves.sh, which looks fo= r hostnames in conf/agents. Is this a bug or are collectors expected to run o= n the agents nodes? I thought they were more typically run on the Hadoop data nodes. On Tue, Jan 12, 2010 at 10:50 AM, Eric Yang <matthew58@example.com> wrote: > Hi Bill, > > The scripts are over due for a clean up. Majority of the scripts were > designed to work in a specific environment for Yahoo. > > On 1/12/10 10:24 AM, "Bill Graham" <matthew58@example.com> wrote: > > > Hi, > > > > To get the scripts in the tools/init.d folder to run in our environment= I > had > > to make a few of the same tweaks to each of them. I'd like to suggest > some > > refactoring of these scripts to make them more easy to use. These are t= he > > issues I came across: > > > > 1. The scripts try to write lock files to /var/lock/subsys, which is > owned by > > root. Can we change this location to be somewhere that doesn't require > root > > access? > > This was designed to run as /etc/init.d script. I would recommend to > remove > tools/init.d and /etc/init.d script, and focus on using start-*.sh > stop-*.sh > commands like hadoop. > > > 2. The actual run command does an su to the CHUKWA_USER which also caus= ed > > problems for us. It seems like it would be cleaner to not embed the su > calls > > in the script, but instead allow the user to su when they run the scrip= t > > (which worked much better for us). That way everything done by the scri= pt > > would be done by the same user. > > Base on previous experience, we had a user who were running everything as > root. Su was put in there to safe guard the program from that user. It > was > a pain to clean up hundred of machines with root owned files. In a > production environment, it is probably not Chukwa's responsibility to saf= e > guard permission. +1 on removing su. > > > 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard > coded. > > Hard coded defaults is ok, but the ability to override them without > modifying > > the scripts would be ideal. > > > > Current documentation doesn=B9t talk about those properties are configura= ble > during build time. If you compile from trunk or 0.3 branch, you can copy > default.properties to build.properties, and specify the path and running > user. However, the replace only applies to /etc/init.d/chukwa-* scripts, > and install from RPM. (Chukwa has an "ant rpm" target.) tools/init.d wa= s > meant as a template, hence no replacement took place. > > > Let me know if you think these changes make sense and I'll open a JIRA. > > Yes, open a jira, and we can discuss in depth of the changes. I would li= ke > to see /etc/init.d and tools/init.d to be removed completely, and use > start-*.sh and stop-*.sh command to be consistent with hadoop. > > Regards, > Eric > > --001636e0b6b7fb10b6047cfd5126 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Eric, that makes sense. I&#39;m going to use the Hadoop-style start-= *.sh scripts instead, since they seem to work better for my needs. <br><br>= The agent and data processor start/stop scripts work for me, but the bin/st= art-collectors.sh scripts fails:<br> <br>$ sda bin/start-collectors.sh<br>sed: -e expression #1, char 11: unknow= n option to `s&#39;<br><br>Has anyone else seen this?<br><br>It also seems = like this script ultimately invokes slaves.sh, which looks for hostnames in= conf/agents. Is this a bug or are collectors expected to run on the agents= nodes? I thought they were more typically run on the Hadoop data nodes.<br= > <br><br><div class=3D"gmail_quote">On Tue, Jan 12, 2010 at 10:50 AM, Eric Y= ang <span dir=3D"ltr">&lt;<a href=3D"mailto:matthew58@example.com">eyang@yaho= o-inc.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style= =3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; p= adding-left: 1ex;"> Hi Bill,<br> <br> The scripts are over due for a clean up. =A0Majority of the scripts were<br= > designed to work in a specific environment for Yahoo.<br> <div class=3D"im"><br> On 1/12/10 10:24 AM, &quot;Bill Graham&quot; &lt;<a href=3D"mailto:billgrah= matthew58@example.com">matthew58@example.com</a>&gt; wrote:<br> <br> &gt; Hi,<br> &gt;<br> &gt; To get the scripts in the tools/init.d folder to run in our environmen= t I had<br> &gt; to make a few of the same tweaks to each of them. I&#39;d like to sugg= est some<br> &gt; refactoring of these scripts to make them more easy to use. These are = the<br> &gt; issues I came across:<br> &gt;<br> &gt; 1. The scripts try to write lock files to /var/lock/subsys, which is o= wned by<br> &gt; root. Can we change this location to be somewhere that doesn&#39;t req= uire root<br> &gt; access?<br> <br> </div>This was designed to run as /etc/init.d script. =A0I would recommend = to remove<br> tools/init.d and /etc/init.d script, and focus on using start-*.sh stop-*.s= h<br> commands like hadoop.<br> <div class=3D"im"><br> &gt; 2. The actual run command does an su to the CHUKWA_USER which also cau= sed<br> &gt; problems for us. It seems like it would be cleaner to not embed the su= calls<br> &gt; in the script, but instead allow the user to su when they run the scri= pt<br> &gt; (which worked much better for us). That way everything done by the scr= ipt<br> &gt; would be done by the same user.<br> <br> </div>Base on previous experience, we had a user who were running everythin= g as<br> root. =A0Su was put in there to safe guard the program from that user. =A0I= t was<br> a pain to clean up hundred of machines with root owned files. =A0In a<br> production environment, it is probably not Chukwa&#39;s responsibility to s= afe<br> guard permission. =A0+1 on removing su.<br> <div class=3D"im"><br> &gt; 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard c= oded.<br> &gt; Hard coded defaults is ok, but the ability to override them without mo= difying<br> &gt; the scripts would be ideal.<br> &gt;<br> <br> </div>Current documentation doesn=B9t talk about those properties are confi= gurable<br> during build time. =A0If you compile from trunk or 0.3 branch, you can copy= <br> default.properties to build.properties, and specify the path and running<br= > user. =A0However, the replace only applies to /etc/init.d/chukwa-* scripts,= <br> and install from RPM. =A0(Chukwa has an &quot;ant rpm&quot; target.) =A0too= ls/init.d was<br> meant as a template, hence no replacement took place.<br> <div class=3D"im"><br> &gt; Let me know if you think these changes make sense and I&#39;ll open a = JIRA.<br> <br> </div>Yes, open a jira, and we can discuss in depth of the changes. =A0I wo= uld like<br> to see /etc/init.d and tools/init.d to be removed completely, and use<br> start-*.sh and stop-*.sh command to be consistent with hadoop.<br> <br> Regards,<br> <font color=3D"#888888">Eric<br> <br> </font></blockquote></div><br> --001636e0b6b7fb10b6047cfd5126-- From chukwa-usermatthew58@example.com Thu Jan 14 01:48:03 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 86925 invoked from network); 14 Jan 2010 01:48:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Jan 2010 01:48:03 -0000 Received: (qmail 80044 invoked by uid 500); 14 Jan 2010 01:48:03 -0000 Delivered-To: matthew58@example.com Received: (qmail 79986 invoked by uid 500); 14 Jan 2010 01:48:03 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 79977 invoked by uid 99); 14 Jan 2010 01:48:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2010 01:48:03 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthew58@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-px0-f187.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jan 2010 01:47:55 +0000 Received: by pxi17 with SMTP id 17so51428670pxi.30 for <matthew58@example.com>; Wed, 13 Jan 2010 17:47:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:date:message-id :subject:from:to:content-type; bh=JeM38qNTsZjllFWZJ0AOAlbDrR/eV5jlfn4ua8g4TEQ=; b=NWQEv6z9ufbrUTLnXnbtWGDORMmkVONkIDWRZzphY6oLGQ5hOUya1afru2v/UFrrwP RRa3qCRVsXr3tv4fG4TJNTAhMoK48QV3W4Il0kZXbykfaufjsgdlrgDcSnWc0XgjKxWS MUtNtiB3a8i9pqyhsV8a1Z3sGWIhXfKH/koR0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:date:message-id:subject:from:to:content-type; b=h+ITLMMWNhkVOyF8C+e0CFq+tfbOOVtXZvGhe6vjFQ3PclYrAbb4YBr3ox00iFJgaS qhopUfDsPX9BOvVXmW/XKcLWmM0UBsftOwr5Zc6/scTjXI+uUf24TcNEs5L7mM+PuTc0 lscwxhcWknegIZi410aCnHPd5r2slP+u3ea7U= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 42mr69719wfi.227.1263433655728; Wed, 13 Jan 2010 17:47:35 -0800 (PST) Reply-To: matthew58@example.com Date: Wed, 13 Jan 2010 17:47:35 -0800 Message-ID: <matthew58@example.com> Subject: Invoking Chukwa collectors through a load balancer From: Bill Graham <matthew58@example.com> To: matthew58@example.com Content-Type: multipart/alternative; boundary=00504502af753f3a66047d161284 --00504502af753f3a66047d161284 Content-Type: text/plain; charset=ISO-8859-1 Hi, When talking about the $CHUKWA_HOME/conf/collectors file, the admin guide says this: "Agents will pick a random collector from this list to try sending to, and will fail-over to another listed collector on error." What happens though in the case where there is only one collector configured? Will the agent continue to try the same collector? How many times will the agent retry before giving up? My use case is that we're looking to put a load balancer in front of our collectors, so my collectors file will contain only one entry. If a request fails, I'd like it to try again up to a certain point. I know the load balancer approach is counter to intent of the agent/collector design, but in our case managing active collector nodes via a load balancer is a lot easier that managing them in the configs of all of our agents. thanks, Bill --00504502af753f3a66047d161284 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi,<br><br>When talking about the <span class=3D"codefrag">$CHUKWA_HOME/con= f/collectors file, the admin guide says this:</span><br><br>&quot;Agents wi= ll pick a random collector from this list to try sending to, and will fail-over to another listed coll= ector on error.&quot;<br><br>What happens though in the case where there is only = one collector configured? Will the agent continue to try the same collector= ? How many times will the agent retry before giving up?<br><br>My use case = is that we&#39;re looking to put a load balancer in front of our collectors= , so my collectors file will contain only one entry. If a request fails, I&= #39;d like it to try again up to a certain point.<br> <br>I know the load balancer approach is counter to intent of the agent/col= lector design, but in our case managing active collector nodes via a load b= alancer is a lot easier that managing them in the configs of all of our age= nts.<br> <br>thanks,<br>Bill<br><br> --00504502af753f3a66047d161284-- From chukwa-usermatthew58@example.com Fri Jan 15 17:49:55 2010 Return-Path: <chukwa-usermatthew58@example.com> Delivered-To: matthew58@example.com Received: (qmail 41912 invoked from network); 15 Jan 2010 17:49:55 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 15 Jan 2010 17:49:55 -0000 Received: (qmail 1459 invoked by uid 500); 15 Jan 2010 17:49:55 -0000 Delivered-To: matthew58@example.com Received: (qmail 1411 invoked by uid 500); 15 Jan 2010 17:49:55 -0000 Mailing-List: contact matthew58@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:matthew58@example.com> List-Unsubscribe: <mailto:matthew58@example.com> List-Post: <mailto:matthew58@example.com> List-Id: <chukwa-user.hadoop.apache.org> Reply-To: matthew58@example.com Delivered-To: mailing list matthew58@example.com Received: (qmail 1402 invoked by uid 99); 15 Jan 2010 17:49:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Jan 2010 17:49:54 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of matthew58@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-pw0-f50.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Jan 2010 17:49:49 +0000 Received: by pwi20 with SMTP id 20so445586pwi.29 for <matthew58@example.com>; Fri, 15 Jan 2010 09:49:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=z+VzKL4Q9SwUGEF1fZdlulG52+4kzLorgOORrH7OXbk=; b=S/0JdwMCCv0LhH75h20B1gEi0CmtPNqRx7gziUFnWYdkH2rqydP7bcCo25/M9an6zG mfu2JDn5tE38BCdM0z8B2oEKPzWkykEgyzpXjWq/PHUSUL3lPgv3QsroQOmmm498bZSQ zH6JKWttKRUWSB5zxXMdWpobKPNYfTwnU+Tag= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; b=cKMSfUs1aLZIIE/tlp9ZwKYAZx5y/aE+oq1+17pFrnJnwz69Nqc3QTorTrEAC3D6F9 VjxH8LCStefary3j1ipy87Ay5um9HyPM6UenhdVTbr9+qPIDobdpl8k7YJlSqxra8uhC N3Rf6dx1nyybluD95QXE3pjTn/fhuNLl4lViY= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id f5mr1835169wfg.268.1263577769357; Fri, 15 Jan 2010 09:49:29 -0800 (PST) Reply-To: matthew58@example.com In-Reply-To: <matthew58@example.com> References: <matthew58@example.com> <matthew58@example.com> <matthew58@example.com> Date: Fri, 15 Jan 2010 09:49:29 -0800 Message-ID: <matthew58@example.com> Subject: Re: suggested changes to tools/init.d scripts From: Bill Graham <matthew58@example.com> To: Eric Yang <matthew58@example.com> Cc: matthew58@example.com Content-Type: multipart/alternative; boundary=000e0cd17e0216b293047d37a0a3 --000e0cd17e0216b293047d37a0a3 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Just to follow up on my own comment here, I've uncovered the issue. It seem= s bin/start-collectors.sh does correctly read the conf/collectors file for a list of hostnames to start the collectors on via ssh. The sed error I was getting upon running the start script was because the conf/collectors file didn't have hostnames in it. Instead it had HTTP URLs to the collectors, since what the agent expects. The problem is that conf/collectors file is used by two different processes= , and each requires a different format. bin/agents.sh expects URLs to the collectors it should communicate with, while bin/start-collectors.sh expect= s a list of hostname to start collectors on. I'm running each on the same hos= t in dev so I got bit by this issue. In our production environment this will be a non-issue, since we'll have RPMs with different configs on different hosts, but this is a source of confusion that I thought I should point out. thanks, Bill On Tue, Jan 12, 2010 at 12:15 PM, Bill Graham <matthew58@example.com> wrote: > Thanks Eric, that makes sense. I'm going to use the Hadoop-style start-*.= sh > scripts instead, since they seem to work better for my needs. > > The agent and data processor start/stop scripts work for me, but the > bin/start-collectors.sh scripts fails: > > $ sda bin/start-collectors.sh > sed: -e expression #1, char 11: unknown option to `s' > > Has anyone else seen this? > > It also seems like this script ultimately invokes slaves.sh, which looks > for hostnames in conf/agents. Is this a bug or are collectors expected to > run on the agents nodes? I thought they were more typically run on the > Hadoop data nodes. > > > > On Tue, Jan 12, 2010 at 10:50 AM, Eric Yang <matthew58@example.com> wrote: > >> Hi Bill, >> >> The scripts are over due for a clean up. Majority of the scripts were >> designed to work in a specific environment for Yahoo. >> >> On 1/12/10 10:24 AM, "Bill Graham" <matthew58@example.com> wrote: >> >> > Hi, >> > >> > To get the scripts in the tools/init.d folder to run in our environmen= t >> I had >> > to make a few of the same tweaks to each of them. I'd like to suggest >> some >> > refactoring of these scripts to make them more easy to use. These are >> the >> > issues I came across: >> > >> > 1. The scripts try to write lock files to /var/lock/subsys, which is >> owned by >> > root. Can we change this location to be somewhere that doesn't require >> root >> > access? >> >> This was designed to run as /etc/init.d script. I would recommend to >> remove >> tools/init.d and /etc/init.d script, and focus on using start-*.sh >> stop-*.sh >> commands like hadoop. >> >> > 2. The actual run command does an su to the CHUKWA_USER which also >> caused >> > problems for us. It seems like it would be cleaner to not embed the su >> calls >> > in the script, but instead allow the user to su when they run the scri= pt >> > (which worked much better for us). That way everything done by the >> script >> > would be done by the same user. >> >> Base on previous experience, we had a user who were running everything a= s >> root. Su was put in there to safe guard the program from that user. It >> was >> a pain to clean up hundred of machines with root owned files. In a >> production environment, it is probably not Chukwa's responsibility to sa= fe >> guard permission. +1 on removing su. >> >> > 3. Each script has CHUKWA_HOME, CHUKWA_CONF_DIR and CHUKWA_USER hard >> coded. >> > Hard coded defaults is ok, but the ability to override them without >> modifying >> > the scripts would be ideal. >> > >> >> Current documentation doesn=B9t talk about those properties are configur= able >> during build time. If you compile from trunk or 0.3 branch, you can cop= y >> default.properties to build.properties, and specify the path and running >> user. However, the replace only applies to /etc/init.d/chukwa-* scripts= , >> and install from RPM. (Chukwa has an "ant rpm" target.) tools/init.d w= as >> meant as a template, hence no replacement took place. >> >> > Let me know if you think these changes make sense and I'll open a JIRA= . >> >> Yes, open a jira, and we can discuss in depth of the changes. I would >> like >> to see /etc/init.d and tools/init.d to be removed completely, and use >> start-*.sh and stop-*.sh command to be consistent with hadoop. >> >> Regards, >> Eric >> >> > --000e0cd17e0216b293047d37a0a3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Just to follow up on my own comment here, I&#39;ve uncovered the issue. It = seems bin/start-collectors.sh does correctly read the conf/collectors file = for a list of hostnames to start the collectors on via ssh.<br><br>The sed = error I was getting upon running the start script was because the conf/coll= ectors file didn&#39;t have hostnames in it. Instead it had HTTP URLs to th= e collectors, since what the agent expects.<br> <br>The problem is that conf/collectors file is used by two different proce= sses, and each requires a different format. bin/agents.sh expects URLs to t= he collectors it should communicate with, while bin/start-collectors.sh exp= ects a list of hostname to start collectors on. I&#39;m running each on the= same host in dev so I got bit by this issue.<br> <br>In our production environment this will be a non-issue, since we&#39;ll= have RPMs with different configs on different hosts, but this is a source = of confusion that I thought I should point out.<br><br>thanks,<br>Bill<br> <br><div class=3D"gmail_quote">On Tue, Jan 12, 2010 at 12:15 PM, Bill Graha= m <span dir=3D"ltr">&lt;<a href=3D"mailto:matthew58@example.com">billgraham@= gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style= =3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; p= adding-left: 1ex;"> Thanks Eric, that makes sense. I&#39;m going to use the Hadoop-style start-= *.sh scripts instead, since they seem to work better for my needs. <br><br>= The agent and data processor start/stop scripts work for me, but the bin/st= art-collectors.sh scripts fails:<br> <br>$ sda bin/start-collectors.sh<br>sed: -e expression #1, char 11: unknow= n option to `s&#39;<br><br>Has anyone else seen this?<br><br>It also seems = like this script ultimately invokes slaves.sh, which looks for hostnames in= conf/agents. Is this a bug or are collectors expected to run on the agents= nodes? I thought they were more typically run on the Hadoop data nodes.<di= v> <div></div><div class=3D"h5"><br> <br><br><div class=3D"gmail_quote">On Tue, Jan 12, 2010 at 10:50 AM, Eric Y= ang <span dir=3D"ltr">&lt;<a href=3D"mailto:matthew58@example.com" target=3D"= _blank">matthew58@example.com</a>&gt;</span> wrote:<br><blockquote class=3D"g= mail_quote" style=3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt= 0pt 0pt 0.8ex; padding-left: 1ex;"> Hi Bill,<br> <br> The scripts are over due for a clean up. =A0Majority of the scripts were<br= > designed to work in a specific environment for Yahoo.<br> <div><br> On 1/12/10 10:24 AM, &quot;Bill Graham&quot; &lt;<a href=3D"mailto:billgrah= matthew58@example.com" target=3D"_blank">matthew58@example.com</a>&gt; wrote:<br> <br> &gt; Hi,<br> &gt;<br> &gt; To get the scripts in the tools/init.d folder to run in our environmen= t I had<br> &gt; to make a few of the same tweaks to each of them. I&#39;d like to sugg= est some<br> &gt; refactoring of these scripts to make them more easy to use. These are = the<br> &gt; issues I came across:<br> &gt;<br> &gt; 1. The scripts try to write lock files to /var/lock/subsys, which is o= wned by<br> &gt; root. Can we change this location to be somewhere that doesn&#39;t req= uire root<br> &gt; access?<br> <br> </div>This was designed to run as /etc/init.d script. =A0I would recommend = to remove<br> tools/init.d and /etc/init.d script, and focus on using start-*.sh stop-*.s= h<br> commands like hadoop.<br> <div><br> &gt; 2. The actual run command does an su to the CHUKWA_USER which also cau= sed<br> &gt; problems for us. It seems like it would be cleaner to not embed the su= calls<br> &gt; in the script, but instead allow the user to su when they run the scri= pt<br> &gt; (which worked much better for us). That way everything done by the scr= ipt<br> &gt; would be done by the same user.<br> <br> </div>Base on previous experience, we had a user who were running everythin= g as<br> root. =A0Su was
{'source': 'Apache Software Foundation Public Mail Archives'}