From hadoop-devwurobert@example.net Wed Mar 01 22:17:18 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 47811 invoked from network); 1 Mar 2006 22:17:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2006 22:17:18 -0000 Received: (qmail 85880 invoked by uid 500); 1 Mar 2006 22:18:05 -0000 Delivered-To: wurobert@example.net Received: (qmail 85857 invoked by uid 500); 1 Mar 2006 22:18:04 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 85848 invoked by uid 99); 1 Mar 2006 22:18:04 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 14:18:04 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO mslinux.media-style.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 14:18:03 -0800 Received: from localhost (localhost [83.161.120.155]) by mslinux.media-style.com (Postfix) with ESMTP id D33B214453C for <wurobert@example.net>; Wed, 1 Mar 2006 23:10:59 +0100 (CET) Received: from mslinux.media-style.com ([83.161.120.155]) by localhost (mslinux.media-style.com [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id 14634-07 for <wurobert@example.net>; Wed, 1 Mar 2006 23:10:59 +0100 (CET) Received: from [83.161.120.155] (dslb-084-059-201-047.pools.arcor-ip.net [83.161.120.155]) by mslinux.media-style.com (Postfix) with ESMTP id 494D51444F1 for <wurobert@example.net>; Wed, 1 Mar 2006 23:10:59 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v749) Content-Transfer-Encoding: 7bit Message-Id: <wurobert@example.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: wurobert@example.net From: Stefan Groschupf <wurobert@example.net> Subject: scalability limits getDetails, mapFile Readers? Date: Wed, 1 Mar 2006 23:17:37 +0100 X-Mailer: Apple Mail (2.749) X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at media-style.com X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Hi, We run into a problem with nutch using MapFileOutputFormat#getReaders and getEntry. In detail this happens until summary generation where we open for each segment as much readers as much parts (part-0000 to part-n) we have. Having 80 tasktracker and 80 segments means: 80 x 80 x 4 (parseData, parseText, content, crawl). A search server also needs to open as much files as required for the index searcher. So the problem is a FileNotFoundException, (Too many open files). Opening and closing Readers for each Detail makes no sense. We may can limit the number of readers somehow and close the readers that wasn't used since the longest time. But I'm not that happy with this solution, so any thoughts how we can solve this problem in general? Thanks. Stefan --------------------------------------------- blog: http://www.find23.org company: http://www.media-style.com From hadoop-devwurobert@example.net Wed Mar 01 23:08:23 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 78953 invoked from network); 1 Mar 2006 23:08:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2006 23:08:23 -0000 Received: (qmail 81326 invoked by uid 500); 1 Mar 2006 23:09:09 -0000 Delivered-To: wurobert@example.net Received: (qmail 81311 invoked by uid 500); 1 Mar 2006 23:09:09 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 81302 invoked by uid 99); 1 Mar 2006 23:09:09 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 15:09:07 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id D69ACDD for <wurobert@example.net>; Thu, 2 Mar 2006 00:08:44 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 00:08:44 +0100 (CET) From: "Mahadev konar (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-57) hadoop dfs -ls / does not show root of file system In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-57?page=comments#action_12368391 ] Mahadev konar commented on HADOOP-57: ------------------------------------- Here is a small patch that enables seeing the root file system. The getDFSPath() function in DistributedFileSystem returns empty string in case of an ls on /. The patch below modifies it to return /. 264d263 < path.append(DFSFile.DFS_FILE_SEPARATOR); 266a266 > path.append(DFSFile.DFS_FILE_SEPARATOR); 268,270d267 < if(i!=0){ < path.append(DFSFile.DFS_FILE_SEPARATOR); < } > hadoop dfs -ls / does not show root of file system > -------------------------------------------------- > > Key: HADOOP-57 > URL: http://issues.apache.org/jira/browse/HADOOP-57 > Project: Hadoop > Type: Bug > Components: dfs > Versions: 0.1 > Reporter: Yoram Arnon > Priority: Minor > > hadoop dfs -ls / does not show root of file system - it shows the user's home directory. > It's thus impossible to learn the contents of the root file system via the shell. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Wed Mar 01 23:25:30 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 85461 invoked from network); 1 Mar 2006 23:25:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2006 23:25:30 -0000 Received: (qmail 98648 invoked by uid 500); 1 Mar 2006 23:26:16 -0000 Delivered-To: wurobert@example.net Received: (qmail 98629 invoked by uid 500); 1 Mar 2006 23:26:16 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 98620 invoked by uid 99); 1 Mar 2006 23:26:16 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 15:26:16 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO ylpvm43.prodigy.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 15:26:16 -0800 Received: from pimout7-ext.prodigy.net (pimout7-int.prodigy.net [83.161.120.155]) by ylpvm43.prodigy.net (8.12.10 outbound/8.12.10) with ESMTP id k21NPvRR011681 for <wurobert@example.net>; Wed, 1 Mar 2006 18:25:57 -0500 X-ORBL: [83.161.120.155] Received: from [83.161.120.155] (adsl-69-228-231-148.dsl.pltn13.pacbell.net [83.161.120.155]) by pimout7-ext.prodigy.net (8.13.4 outbound domainkey aix/8.13.4) with ESMTP id k21NPnaW066438; Wed, 1 Mar 2006 18:25:51 -0500 Message-ID: <wurobert@example.net> Date: Wed, 01 Mar 2006 15:25:48 -0800 From: Doug Cutting <wurobert@example.net> User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051013) X-Accept-Language: en-us, en MIME-Version: 1.0 To: wurobert@example.net Subject: Re: scalability limits getDetails, mapFile Readers? References: <wurobert@example.net> In-Reply-To: <wurobert@example.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Stefan, I think you meant to send this to nutch-dev, not hadoop-dev. Doug Stefan Groschupf wrote: > Hi, > > We run into a problem with nutch using MapFileOutputFormat#getReaders > and getEntry. > In detail this happens until summary generation where we open for each > segment as much readers as much parts (part-0000 to part-n) we have. > Having 80 tasktracker and 80 segments means: > 80 x 80 x 4 (parseData, parseText, content, crawl). A search server > also needs to open as much files as required for the index searcher. > So the problem is a FileNotFoundException, (Too many open files). > > Opening and closing Readers for each Detail makes no sense. We may can > limit the number of readers somehow and close the readers that wasn't > used since the longest time. > But I'm not that happy with this solution, so any thoughts how we can > solve this problem in general? > > Thanks. > Stefan > > > --------------------------------------------- > blog: http://www.find23.org > company: http://www.media-style.com > > From hadoop-devwurobert@example.net Wed Mar 01 23:28:18 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 86081 invoked from network); 1 Mar 2006 23:28:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2006 23:28:18 -0000 Received: (qmail 736 invoked by uid 500); 1 Mar 2006 23:29:05 -0000 Delivered-To: wurobert@example.net Received: (qmail 716 invoked by uid 500); 1 Mar 2006 23:29:04 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 707 invoked by uid 99); 1 Mar 2006 23:29:04 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 15:29:04 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO mslinux.media-style.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 15:29:02 -0800 Received: from localhost (localhost [83.161.120.155]) by mslinux.media-style.com (Postfix) with ESMTP id 218D61444C9 for <wurobert@example.net>; Thu, 2 Mar 2006 00:21:59 +0100 (CET) Received: from mslinux.media-style.com ([83.161.120.155]) by localhost (mslinux.media-style.com [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id 15085-03 for <wurobert@example.net>; Thu, 2 Mar 2006 00:21:58 +0100 (CET) Received: from [83.161.120.155] (dslb-084-059-201-047.pools.arcor-ip.net [83.161.120.155]) by mslinux.media-style.com (Postfix) with ESMTP id 294631444C0 for <wurobert@example.net>; Thu, 2 Mar 2006 00:21:58 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v749) In-Reply-To: <wurobert@example.net> References: <wurobert@example.net> <wurobert@example.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <wurobert@example.net> Content-Transfer-Encoding: 7bit From: Stefan Groschupf <wurobert@example.net> Subject: Re: scalability limits getDetails, mapFile Readers? Date: Thu, 2 Mar 2006 00:28:35 +0100 To: wurobert@example.net X-Mailer: Apple Mail (2.749) X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at media-style.com X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N ups, sorry my mistake. I will post it in nutch- dev again. Am 02.03.2006 um 00:25 schrieb Doug Cutting: > Stefan, > > I think you meant to send this to nutch-dev, not hadoop-dev. > > Doug > > Stefan Groschupf wrote: >> Hi, >> We run into a problem with nutch using >> MapFileOutputFormat#getReaders and getEntry. >> In detail this happens until summary generation where we open for >> each segment as much readers as much parts (part-0000 to part-n) >> we have. >> Having 80 tasktracker and 80 segments means: >> 80 x 80 x 4 (parseData, parseText, content, crawl). A search >> server also needs to open as much files as required for the >> index searcher. >> So the problem is a FileNotFoundException, (Too many open files). >> Opening and closing Readers for each Detail makes no sense. We >> may can limit the number of readers somehow and close the readers >> that wasn't used since the longest time. >> But I'm not that happy with this solution, so any thoughts how we >> can solve this problem in general? >> Thanks. >> Stefan >> --------------------------------------------- >> blog: http://www.find23.org >> company: http://www.media-style.com > --------------------------------------------- blog: http://www.find23.org company: http://www.media-style.com From hadoop-devwurobert@example.net Wed Mar 01 23:51:26 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 97351 invoked from network); 1 Mar 2006 23:51:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2006 23:51:26 -0000 Received: (qmail 38966 invoked by uid 500); 1 Mar 2006 23:52:13 -0000 Delivered-To: wurobert@example.net Received: (qmail 38945 invoked by uid 500); 1 Mar 2006 23:52:13 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 38936 invoked by uid 99); 1 Mar 2006 23:52:13 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 15:52:11 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id B0B5BDD for <wurobert@example.net>; Thu, 2 Mar 2006 00:51:44 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 00:51:44 +0100 (CET) From: "Owen O'Malley (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Created: (HADOOP-62) can't get environment variables from HADOOP_CONF_DIR MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N can't get environment variables from HADOOP_CONF_DIR ---------------------------------------------------- Key: HADOOP-62 URL: http://issues.apache.org/jira/browse/HADOOP-62 Project: Hadoop Type: Bug Components: conf Versions: 0.1 Reporter: Owen O'Malley Assigned to: Owen O'Malley Priority: Minor The bin/hadoop script doesn't use the HADOOP_CONF_DIR variable to find hadoop-env.sh. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Wed Mar 01 23:53:26 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 97681 invoked from network); 1 Mar 2006 23:53:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2006 23:53:26 -0000 Received: (qmail 39488 invoked by uid 500); 1 Mar 2006 23:54:12 -0000 Delivered-To: wurobert@example.net Received: (qmail 39470 invoked by uid 500); 1 Mar 2006 23:54:12 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 39461 invoked by uid 99); 1 Mar 2006 23:54:12 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 15:54:10 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 29E9CDD for <wurobert@example.net>; Thu, 2 Mar 2006 00:53:45 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 00:53:45 +0100 (CET) From: "Owen O'Malley (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Updated: (HADOOP-62) can't get environment variables from HADOOP_CONF_DIR In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-62?page=all ] Owen O'Malley updated HADOOP-62: -------------------------------- Attachment: conf_dir.patch This patch fixes the problem by setting the HADOOP_CONF_DIR earlier in the script. > can't get environment variables from HADOOP_CONF_DIR > ---------------------------------------------------- > > Key: HADOOP-62 > URL: http://issues.apache.org/jira/browse/HADOOP-62 > Project: Hadoop > Type: Bug > Components: conf > Versions: 0.1 > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Priority: Minor > Attachments: conf_dir.patch > > The bin/hadoop script doesn't use the HADOOP_CONF_DIR variable to find hadoop-env.sh. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:12:20 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 2956 invoked from network); 2 Mar 2006 00:12:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:12:20 -0000 Received: (qmail 61145 invoked by uid 500); 2 Mar 2006 00:13:06 -0000 Delivered-To: wurobert@example.net Received: (qmail 61094 invoked by uid 500); 2 Mar 2006 00:13:06 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 61047 invoked by uid 99); 2 Mar 2006 00:13:06 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:13:03 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 99EA4E1 for <wurobert@example.net>; Thu, 2 Mar 2006 01:12:42 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 01:12:42 +0100 (CET) From: "Konstantin Shvachko (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-33) DF enhancement: performance and win XP support In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-33?page=comments#action_12368399 ] Konstantin Shvachko commented on HADOOP-33: ------------------------------------------- This patch (DF2.patch) covers DF caching and reuse of the same instance of DF in FSDataset. I removed main() from the DF class, and created class TestDF in the test directory. Additionally, for those who want Windows XP/2003 df functionality without cygwin I attach DF.java, which covers that and is ready for adding other OSs, if desired. Just replace committed DF.java with the one attached. > DF enhancement: performance and win XP support > ---------------------------------------------- > > Key: HADOOP-33 > URL: http://issues.apache.org/jira/browse/HADOOP-33 > Project: Hadoop > Type: Improvement > Components: fs, dfs > Environment: Unix, Cygwin, Win XP > Reporter: Konstantin Shvachko > Priority: Minor > Attachments: DF.patch, DFpatch.txt > > 1. DF is called twice for each heartbeat, which happens each 3 seconds. > There is a simple fix for that in the attached patch. > 2. cygwin is required to run df program in windows environment. > There is a class org.apache.commons.io.FileSystemUtils, which can return disk free space > for different OSs, but it does not have means to get disk capacity. > In general in windows there is no efficient and uniform way to calculate disk capacity > using a shell command. > The choices are 'chkdsk' and 'defrag -a', but both of them are too slow to be called > every 3 seconds. > WinXP and 2003 server have a new tool called fsutil, which provides all necessary info. > I implemented a call to fsutil in case df fails, and the OS is right. > Other win versions should still run cygwin. > I tested this fetaure for linux, winXP and cygwin. > See attached patch. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:14:17 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 3425 invoked from network); 2 Mar 2006 00:14:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:14:16 -0000 Received: (qmail 61951 invoked by uid 500); 2 Mar 2006 00:15:03 -0000 Delivered-To: wurobert@example.net Received: (qmail 61934 invoked by uid 500); 2 Mar 2006 00:15:03 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 61925 invoked by uid 99); 2 Mar 2006 00:15:03 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:15:03 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 47A3CDF for <wurobert@example.net>; Thu, 2 Mar 2006 01:14:42 +0100 (CET) Message-ID: <{{EMAIL+PHONE}}> Date: Thu, 2 Mar 2006 01:14:42 +0100 (CET) From: "Konstantin Shvachko (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Updated: (HADOOP-33) DF enhancement: performance and win XP support In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-33?page=all ] Konstantin Shvachko updated HADOOP-33: -------------------------------------- Attachment: DF2.patch > DF enhancement: performance and win XP support > ---------------------------------------------- > > Key: HADOOP-33 > URL: http://issues.apache.org/jira/browse/HADOOP-33 > Project: Hadoop > Type: Improvement > Components: fs, dfs > Environment: Unix, Cygwin, Win XP > Reporter: Konstantin Shvachko > Priority: Minor > Attachments: DF.patch, DF2.patch, DFpatch.txt > > 1. DF is called twice for each heartbeat, which happens each 3 seconds. > There is a simple fix for that in the attached patch. > 2. cygwin is required to run df program in windows environment. > There is a class org.apache.commons.io.FileSystemUtils, which can return disk free space > for different OSs, but it does not have means to get disk capacity. > In general in windows there is no efficient and uniform way to calculate disk capacity > using a shell command. > The choices are 'chkdsk' and 'defrag -a', but both of them are too slow to be called > every 3 seconds. > WinXP and 2003 server have a new tool called fsutil, which provides all necessary info. > I implemented a call to fsutil in case df fails, and the OS is right. > Other win versions should still run cygwin. > I tested this fetaure for linux, winXP and cygwin. > See attached patch. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:16:17 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 4576 invoked from network); 2 Mar 2006 00:16:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:16:17 -0000 Received: (qmail 65722 invoked by uid 500); 2 Mar 2006 00:17:04 -0000 Delivered-To: wurobert@example.net Received: (qmail 65664 invoked by uid 500); 2 Mar 2006 00:17:03 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 65655 invoked by uid 99); 2 Mar 2006 00:17:03 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:17:01 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 8E235DD for <wurobert@example.net>; Thu, 2 Mar 2006 01:16:40 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 01:16:40 +0100 (CET) From: "Konstantin Shvachko (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Updated: (HADOOP-33) DF enhancement: performance and win XP support In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-33?page=all ] Konstantin Shvachko updated HADOOP-33: -------------------------------------- Attachment: DF.java > DF enhancement: performance and win XP support > ---------------------------------------------- > > Key: HADOOP-33 > URL: http://issues.apache.org/jira/browse/HADOOP-33 > Project: Hadoop > Type: Improvement > Components: fs, dfs > Environment: Unix, Cygwin, Win XP > Reporter: Konstantin Shvachko > Priority: Minor > Attachments: DF.java, DF.patch, DF2.patch, DFpatch.txt > > 1. DF is called twice for each heartbeat, which happens each 3 seconds. > There is a simple fix for that in the attached patch. > 2. cygwin is required to run df program in windows environment. > There is a class org.apache.commons.io.FileSystemUtils, which can return disk free space > for different OSs, but it does not have means to get disk capacity. > In general in windows there is no efficient and uniform way to calculate disk capacity > using a shell command. > The choices are 'chkdsk' and 'defrag -a', but both of them are too slow to be called > every 3 seconds. > WinXP and 2003 server have a new tool called fsutil, which provides all necessary info. > I implemented a call to fsutil in case df fails, and the OS is right. > Other win versions should still run cygwin. > I tested this fetaure for linux, winXP and cygwin. > See attached patch. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:30:17 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 9860 invoked from network); 2 Mar 2006 00:30:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:30:17 -0000 Received: (qmail 88490 invoked by uid 500); 2 Mar 2006 00:31:04 -0000 Delivered-To: wurobert@example.net Received: (qmail 88469 invoked by uid 500); 2 Mar 2006 00:31:04 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 88460 invoked by uid 99); 2 Mar 2006 00:31:04 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:31:02 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id E7CD9DD for <wurobert@example.net>; Thu, 2 Mar 2006 01:30:41 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 01:30:41 +0100 (CET) From: "Doug Cutting (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-62) can't get environment variables from HADOOP_CONF_DIR In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-62?page=comments#action_12368402 ] Doug Cutting commented on HADOOP-62: ------------------------------------ Shouldn't we also make similar changes in the other scripts that source hadoop-env.sh? Also, wouldn't it be cleaner to assign & default HADOOP_CONF_DIR in a separate line, rather than as a side-effect the first time its used? Finally, why kill the comment about CLASSPATH? > can't get environment variables from HADOOP_CONF_DIR > ---------------------------------------------------- > > Key: HADOOP-62 > URL: http://issues.apache.org/jira/browse/HADOOP-62 > Project: Hadoop > Type: Bug > Components: conf > Versions: 0.1 > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Priority: Minor > Attachments: conf_dir.patch > > The bin/hadoop script doesn't use the HADOOP_CONF_DIR variable to find hadoop-env.sh. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:34:19 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 10639 invoked from network); 2 Mar 2006 00:34:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:34:18 -0000 Received: (qmail 91348 invoked by uid 500); 2 Mar 2006 00:35:04 -0000 Delivered-To: wurobert@example.net Received: (qmail 91290 invoked by uid 500); 2 Mar 2006 00:35:04 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 91127 invoked by uid 99); 2 Mar 2006 00:35:03 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:35:03 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 51CA6126 for <wurobert@example.net>; Thu, 2 Mar 2006 01:34:42 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 01:34:42 +0100 (CET) From: "wurobert@example.net (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-62) can't get environment variables from HADOOP_CONF_DIR In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-62?page=comments#action_12368404 ] wurobert@example.net commented on HADOOP-62: ----------------------------------------- HADOOP-60 does the above Owen patch and addresses the first Doug comment. > can't get environment variables from HADOOP_CONF_DIR > ---------------------------------------------------- > > Key: HADOOP-62 > URL: http://issues.apache.org/jira/browse/HADOOP-62 > Project: Hadoop > Type: Bug > Components: conf > Versions: 0.1 > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Priority: Minor > Attachments: conf_dir.patch > > The bin/hadoop script doesn't use the HADOOP_CONF_DIR variable to find hadoop-env.sh. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:43:17 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 12940 invoked from network); 2 Mar 2006 00:43:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:43:17 -0000 Received: (qmail 7498 invoked by uid 500); 2 Mar 2006 00:44:04 -0000 Delivered-To: wurobert@example.net Received: (qmail 7479 invoked by uid 500); 2 Mar 2006 00:44:04 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 7470 invoked by uid 99); 2 Mar 2006 00:44:04 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:44:02 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 0782EDD for <wurobert@example.net>; Thu, 2 Mar 2006 01:43:42 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 01:43:42 +0100 (CET) From: "Doug Cutting (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Resolved: (HADOOP-62) can't get environment variables from HADOOP_CONF_DIR In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-62?page=all ] Doug Cutting resolved HADOOP-62: -------------------------------- Fix Version: 0.1 Resolution: Duplicate > can't get environment variables from HADOOP_CONF_DIR > ---------------------------------------------------- > > Key: HADOOP-62 > URL: http://issues.apache.org/jira/browse/HADOOP-62 > Project: Hadoop > Type: Bug > Components: conf > Versions: 0.1 > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Priority: Minor > Fix For: 0.1 > Attachments: conf_dir.patch > > The bin/hadoop script doesn't use the HADOOP_CONF_DIR variable to find hadoop-env.sh. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:45:48 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 13874 invoked from network); 2 Mar 2006 00:45:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:45:47 -0000 Received: (qmail 9092 invoked by uid 500); 2 Mar 2006 00:46:33 -0000 Delivered-To: wurobert@example.net Received: (qmail 8951 invoked by uid 500); 2 Mar 2006 00:46:33 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 8857 invoked by uid 99); 2 Mar 2006 00:46:32 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:46:30 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id B2AE2126 for <wurobert@example.net>; Thu, 2 Mar 2006 01:46:09 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 01:46:09 +0100 (CET) From: "Doug Cutting (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-60) Specification of alternate conf. directory In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-60?page=comments#action_12368408 ] Doug Cutting commented on HADOOP-60: ------------------------------------ I should have committed this yesterday, before I made a bunch of other changes to these scripts. Now it no longer applies! Michael, can you please regenerate this against the current sources? Thanks! > Specification of alternate conf. directory > ------------------------------------------ > > Key: HADOOP-60 > URL: http://issues.apache.org/jira/browse/HADOOP-60 > Project: Hadoop > Type: Improvement > Reporter: wurobert@example.net > Priority: Minor > Attachments: HADOOP_CONF_DIR.patch > > Currently, hadoop configuration must be done by making edits and addition to ${HADOOP_HOME}/conf. Allowing specification of an alternate configuration directory will allow keeping configuration and binary distinct. Benefits include: Binary can be made read-only; or binary is blanket-updateable with configuration undisturbed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:55:38 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 17892 invoked from network); 2 Mar 2006 00:55:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:55:38 -0000 Received: (qmail 17575 invoked by uid 500); 2 Mar 2006 00:56:25 -0000 Delivered-To: wurobert@example.net Received: (qmail 17559 invoked by uid 500); 2 Mar 2006 00:56:24 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 17550 invoked by uid 99); 2 Mar 2006 00:56:24 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:56:22 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 6CEFCDD for <wurobert@example.net>; Thu, 2 Mar 2006 01:56:01 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 01:56:01 +0100 (CET) From: "Owen O'Malley (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-62) can't get environment variables from HADOOP_CONF_DIR In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-62?page=comments#action_12368409 ] Owen O'Malley commented on HADOOP-62: ------------------------------------- > Shouldn't we also make similar changes in the other scripts that source hadoop-env.sh? of course. I just missed the fact that they had similar mistakes in them. > Also, wouldn't it be cleaner to assign & default HADOOP_CONF_DIR in a separate line, rather than as > a side-effect the first time its used? I was just trying to copy your style, since it was your script. > Finally, why kill the comment about CLASSPATH? After the conditional assignment was removed, it seemed like a "add one to x" type comment. A better comment would be: # The configuration directory needs to be first in the classpath so that the config files are found by the # classloader. > can't get environment variables from HADOOP_CONF_DIR > ---------------------------------------------------- > > Key: HADOOP-62 > URL: http://issues.apache.org/jira/browse/HADOOP-62 > Project: Hadoop > Type: Bug > Components: conf > Versions: 0.1 > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Priority: Minor > Fix For: 0.1 > Attachments: conf_dir.patch > > The bin/hadoop script doesn't use the HADOOP_CONF_DIR variable to find hadoop-env.sh. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 00:57:18 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 18187 invoked from network); 2 Mar 2006 00:57:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 00:57:17 -0000 Received: (qmail 18013 invoked by uid 500); 2 Mar 2006 00:58:04 -0000 Delivered-To: wurobert@example.net Received: (qmail 17955 invoked by uid 500); 2 Mar 2006 00:58:04 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 17944 invoked by uid 99); 2 Mar 2006 00:58:04 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 16:58:01 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id A03BDDE for <wurobert@example.net>; Thu, 2 Mar 2006 01:57:40 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 01:57:40 +0100 (CET) From: "Owen O'Malley (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-60) Specification of alternate conf. directory In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-60?page=comments#action_12368411 ] Owen O'Malley commented on HADOOP-60: ------------------------------------- Looking at your patch, I would change the setting line from: HADOOP_CONF_DIR=${HADOOP_CONF_DIR:=$HADOOP_HOME/conf} to: HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-$HADOOP_HOME/conf} because you don't need the redundant assignment. You also need to surround all of the file/directory names with quotes so that you support spaces in directory names. > Specification of alternate conf. directory > ------------------------------------------ > > Key: HADOOP-60 > URL: http://issues.apache.org/jira/browse/HADOOP-60 > Project: Hadoop > Type: Improvement > Reporter: wurobert@example.net > Priority: Minor > Attachments: HADOOP_CONF_DIR.patch > > Currently, hadoop configuration must be done by making edits and addition to ${HADOOP_HOME}/conf. Allowing specification of an alternate configuration directory will allow keeping configuration and binary distinct. Benefits include: Binary can be made read-only; or binary is blanket-updateable with configuration undisturbed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 01:01:18 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 19665 invoked from network); 2 Mar 2006 01:01:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 01:01:17 -0000 Received: (qmail 20997 invoked by uid 500); 2 Mar 2006 01:02:04 -0000 Delivered-To: wurobert@example.net Received: (qmail 20914 invoked by uid 500); 2 Mar 2006 01:02:02 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 20415 invoked by uid 99); 2 Mar 2006 01:02:01 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 17:02:01 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 382EFDE for <wurobert@example.net>; Thu, 2 Mar 2006 02:01:40 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 02:01:40 +0100 (CET) From: "Doug Cutting (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-62) can't get environment variables from HADOOP_CONF_DIR In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-62?page=comments#action_12368414 ] Doug Cutting commented on HADOOP-62: ------------------------------------ It's not my script, it's *our* script! Which do you think is the better style? I don't think I wrote that line anyway... > can't get environment variables from HADOOP_CONF_DIR > ---------------------------------------------------- > > Key: HADOOP-62 > URL: http://issues.apache.org/jira/browse/HADOOP-62 > Project: Hadoop > Type: Bug > Components: conf > Versions: 0.1 > Reporter: Owen O'Malley > Assignee: Owen O'Malley > Priority: Minor > Fix For: 0.1 > Attachments: conf_dir.patch > > The bin/hadoop script doesn't use the HADOOP_CONF_DIR variable to find hadoop-env.sh. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 01:41:17 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 44066 invoked from network); 2 Mar 2006 01:41:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 01:41:17 -0000 Received: (qmail 65922 invoked by uid 500); 2 Mar 2006 01:42:04 -0000 Delivered-To: wurobert@example.net Received: (qmail 65906 invoked by uid 500); 2 Mar 2006 01:42:04 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 65897 invoked by uid 99); 2 Mar 2006 01:42:04 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Mar 2006 17:42:01 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id A7214DC for <wurobert@example.net>; Thu, 2 Mar 2006 02:41:40 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 02:41:40 +0100 (CET) From: "wurobert@example.net (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Updated: (HADOOP-60) Specification of alternate conf. directory In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-60?page=all ] wurobert@example.net updated HADOOP-60: ------------------------------------ Attachment: HADOOP_CONF_DIR-v2.patch Version of patch that will go against 382230. Thanks for the feedback Owen. I integrated your comments into this version of the patch. > Specification of alternate conf. directory > ------------------------------------------ > > Key: HADOOP-60 > URL: http://issues.apache.org/jira/browse/HADOOP-60 > Project: Hadoop > Type: Improvement > Reporter: wurobert@example.net > Priority: Minor > Attachments: HADOOP_CONF_DIR-v2.patch, HADOOP_CONF_DIR.patch > > Currently, hadoop configuration must be done by making edits and addition to ${HADOOP_HOME}/conf. Allowing specification of an alternate configuration directory will allow keeping configuration and binary distinct. Benefits include: Binary can be made read-only; or binary is blanket-updateable with configuration undisturbed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 17:40:27 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 26361 invoked from network); 2 Mar 2006 17:40:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 17:40:25 -0000 Received: (qmail 94448 invoked by uid 500); 2 Mar 2006 17:41:04 -0000 Delivered-To: wurobert@example.net Received: (qmail 94312 invoked by uid 500); 2 Mar 2006 17:41:03 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 94266 invoked by uid 99); 2 Mar 2006 17:41:03 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Mar 2006 09:41:01 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id DC66CDC for <wurobert@example.net>; Thu, 2 Mar 2006 18:40:39 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 18:40:39 +0100 (CET) From: "wurobert@example.net (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Updated: (HADOOP-60) Specification of alternate conf. directory In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-60?page=all ] wurobert@example.net updated HADOOP-60: ------------------------------------ Attachment: HADOOP_CONF_DIR-v3.patch Version 3 is version 2 plus edit of hadoop-env.template modifying defaults for HADOOP_SSH_OPTS. > Specification of alternate conf. directory > ------------------------------------------ > > Key: HADOOP-60 > URL: http://issues.apache.org/jira/browse/HADOOP-60 > Project: Hadoop > Type: Improvement > Reporter: wurobert@example.net > Priority: Minor > Attachments: HADOOP_CONF_DIR-v2.patch, HADOOP_CONF_DIR-v3.patch, HADOOP_CONF_DIR.patch > > Currently, hadoop configuration must be done by making edits and addition to ${HADOOP_HOME}/conf. Allowing specification of an alternate configuration directory will allow keeping configuration and binary distinct. Benefits include: Binary can be made read-only; or binary is blanket-updateable with configuration undisturbed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 18:52:21 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 62875 invoked from network); 2 Mar 2006 18:52:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 18:52:21 -0000 Received: (qmail 5152 invoked by uid 500); 2 Mar 2006 18:53:07 -0000 Delivered-To: wurobert@example.net Received: (qmail 5135 invoked by uid 500); 2 Mar 2006 18:53:07 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 5126 invoked by uid 99); 2 Mar 2006 18:53:07 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Mar 2006 10:53:07 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id A3504DC for <wurobert@example.net>; Thu, 2 Mar 2006 19:52:45 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 19:52:45 +0100 (CET) From: "Doug Cutting (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Commented: (HADOOP-60) Specification of alternate conf. directory In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-60?page=comments#action_12368567 ] Doug Cutting commented on HADOOP-60: ------------------------------------ I just committed this. I had to add a few more quotes to get things to work correctly with spaces in directory names, removed a spurious variable setting on the 'nohup' line of hadoop-daemon.sh, and updated a comment. > Specification of alternate conf. directory > ------------------------------------------ > > Key: HADOOP-60 > URL: http://issues.apache.org/jira/browse/HADOOP-60 > Project: Hadoop > Type: Improvement > Reporter: wurobert@example.net > Priority: Minor > Fix For: 0.1 > Attachments: HADOOP_CONF_DIR-v2.patch, HADOOP_CONF_DIR-v3.patch, HADOOP_CONF_DIR.patch > > Currently, hadoop configuration must be done by making edits and addition to ${HADOOP_HOME}/conf. Allowing specification of an alternate configuration directory will allow keeping configuration and binary distinct. Benefits include: Binary can be made read-only; or binary is blanket-updateable with configuration undisturbed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 18:52:25 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 62904 invoked from network); 2 Mar 2006 18:52:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 18:52:25 -0000 Received: (qmail 5249 invoked by uid 500); 2 Mar 2006 18:53:12 -0000 Delivered-To: wurobert@example.net Received: (qmail 5227 invoked by uid 500); 2 Mar 2006 18:53:11 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 5217 invoked by uid 99); 2 Mar 2006 18:53:11 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Mar 2006 10:53:09 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 97799DE for <wurobert@example.net>; Thu, 2 Mar 2006 19:52:47 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 19:52:47 +0100 (CET) From: "Doug Cutting (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Resolved: (HADOOP-60) Specification of alternate conf. directory In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-60?page=all ] Doug Cutting resolved HADOOP-60: -------------------------------- Fix Version: 0.1 Resolution: Fixed Fixed! > Specification of alternate conf. directory > ------------------------------------------ > > Key: HADOOP-60 > URL: http://issues.apache.org/jira/browse/HADOOP-60 > Project: Hadoop > Type: Improvement > Reporter: wurobert@example.net > Priority: Minor > Fix For: 0.1 > Attachments: HADOOP_CONF_DIR-v2.patch, HADOOP_CONF_DIR-v3.patch, HADOOP_CONF_DIR.patch > > Currently, hadoop configuration must be done by making edits and addition to ${HADOOP_HOME}/conf. Allowing specification of an alternate configuration directory will allow keeping configuration and binary distinct. Benefits include: Binary can be made read-only; or binary is blanket-updateable with configuration undisturbed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 19:46:22 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 90019 invoked from network); 2 Mar 2006 19:46:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 19:46:19 -0000 Received: (qmail 97737 invoked by uid 500); 2 Mar 2006 19:47:05 -0000 Delivered-To: wurobert@example.net Received: (qmail 97653 invoked by uid 500); 2 Mar 2006 19:47:05 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 97634 invoked by uid 99); 2 Mar 2006 19:47:05 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Mar 2006 11:47:04 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id CEA04182 for <wurobert@example.net>; Thu, 2 Mar 2006 20:46:43 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 20:46:43 +0100 (CET) From: "Konstantin Shvachko (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Updated: (HADOOP-33) DF enhancement: performance and win XP support In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-33?page=all ] Konstantin Shvachko updated HADOOP-33: -------------------------------------- Attachment: (was: DFpatch.txt) > DF enhancement: performance and win XP support > ---------------------------------------------- > > Key: HADOOP-33 > URL: http://issues.apache.org/jira/browse/HADOOP-33 > Project: Hadoop > Type: Improvement > Components: fs, dfs > Environment: Unix, Cygwin, Win XP > Reporter: Konstantin Shvachko > Priority: Minor > Attachments: DF.java, DF.patch, DF2.patch > > 1. DF is called twice for each heartbeat, which happens each 3 seconds. > There is a simple fix for that in the attached patch. > 2. cygwin is required to run df program in windows environment. > There is a class org.apache.commons.io.FileSystemUtils, which can return disk free space > for different OSs, but it does not have means to get disk capacity. > In general in windows there is no efficient and uniform way to calculate disk capacity > using a shell command. > The choices are 'chkdsk' and 'defrag -a', but both of them are too slow to be called > every 3 seconds. > WinXP and 2003 server have a new tool called fsutil, which provides all necessary info. > I implemented a call to fsutil in case df fails, and the OS is right. > Other win versions should still run cygwin. > I tested this fetaure for linux, winXP and cygwin. > See attached patch. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 20:02:17 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 96492 invoked from network); 2 Mar 2006 20:02:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 20:02:17 -0000 Received: (qmail 29131 invoked by uid 500); 2 Mar 2006 20:03:03 -0000 Delivered-To: wurobert@example.net Received: (qmail 29114 invoked by uid 500); 2 Mar 2006 20:03:03 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 29094 invoked by uid 99); 2 Mar 2006 20:03:03 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Mar 2006 12:03:02 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 02499E1 for <wurobert@example.net>; Thu, 2 Mar 2006 21:02:42 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 21:02:42 +0100 (CET) From: "Konstantin Shvachko (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Updated: (HADOOP-33) DF enhancement: performance and win XP support In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-33?page=all ] Konstantin Shvachko updated HADOOP-33: -------------------------------------- Attachment: (was: DF.java) > DF enhancement: performance and win XP support > ---------------------------------------------- > > Key: HADOOP-33 > URL: http://issues.apache.org/jira/browse/HADOOP-33 > Project: Hadoop > Type: Improvement > Components: fs, dfs > Environment: Unix, Cygwin, Win XP > Reporter: Konstantin Shvachko > Priority: Minor > Attachments: DF.patch, DF2.patch > > 1. DF is called twice for each heartbeat, which happens each 3 seconds. > There is a simple fix for that in the attached patch. > 2. cygwin is required to run df program in windows environment. > There is a class org.apache.commons.io.FileSystemUtils, which can return disk free space > for different OSs, but it does not have means to get disk capacity. > In general in windows there is no efficient and uniform way to calculate disk capacity > using a shell command. > The choices are 'chkdsk' and 'defrag -a', but both of them are too slow to be called > every 3 seconds. > WinXP and 2003 server have a new tool called fsutil, which provides all necessary info. > I implemented a call to fsutil in case df fails, and the OS is right. > Other win versions should still run cygwin. > I tested this fetaure for linux, winXP and cygwin. > See attached patch. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From hadoop-devwurobert@example.net Thu Mar 02 20:10:24 2006 Return-Path: <hadoop-devwurobert@example.net> Delivered-To: wurobert@example.net Received: (qmail 99988 invoked from network); 2 Mar 2006 20:10:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2006 20:10:24 -0000 Received: (qmail 43482 invoked by uid 500); 2 Mar 2006 20:11:10 -0000 Delivered-To: wurobert@example.net Received: (qmail 43461 invoked by uid 500); 2 Mar 2006 20:11:10 -0000 Mailing-List: contact wurobert@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:wurobert@example.net> List-Unsubscribe: <mailto:wurobert@example.net> List-Post: <mailto:wurobert@example.net> List-Id: <hadoop-dev.lucene.apache.org> Reply-To: wurobert@example.net Delivered-To: mailing list wurobert@example.net Received: (qmail 43452 invoked by uid 99); 2 Mar 2006 20:11:10 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Mar 2006 12:11:07 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id D7221182 for <wurobert@example.net>; Thu, 2 Mar 2006 21:10:46 +0100 (CET) Message-ID: <wurobert@example.net> Date: Thu, 2 Mar 2006 21:10:46 +0100 (CET) From: "Konstantin Shvachko (JIRA)" <wurobert@example.net> To: wurobert@example.net Subject: [jira] Updated: (HADOOP-33) DF enhancement: performance and win XP support In-Reply-To: <wurobert@example.net> MIME-Version: 1.0 Content-Type:
{'source': 'Apache Software Foundation Public Mail Archives'}