From chukwa-dev-returnnelsonheather@example.com Sun Mar 13 03:14:24 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 3748 invoked from network); 13 Mar 2011 03:14:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 13 Mar 2011 03:14:24 -0000
Received: (qmail 2426 invoked by uid 500); 13 Mar 2011 03:14:24 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 2383 invoked by uid 500); 13 Mar 2011 03:14:22 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 2375 invoked by uid 99); 13 Mar 2011 03:14:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 03:14:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 03:14:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155])
	by hel.zones.apache.org (Postfix) with ESMTP id 916A23A6950
	for <nelsonheather@example.com>; Sun, 13 Mar 2011 03:13:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 03:13:59 +0000 (UTC)
From: "Xiaoming Shi (JIRA)" <nelsonheather@example.com>
To: nelsonheather@example.com
Message-ID: <{{EMAIL+PHONE}}>
Subject: [jira] Created: (CHUKWA-584) Multiple replace function call can be
 replaced with a single for loop to improve performance
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394

Multiple replace function call can be replaced with a single for loop to improve performance 
---------------------------------------------------------------------------------------------

                 Key: CHUKWA-584
                 URL: https://issues.apache.org/jira/browse/CHUKWA-584
             Project: Chukwa
          Issue Type: Bug
          Components: data collection
    Affects Versions: 0.4.0
            Reporter: Xiaoming Shi


{noformat}
./chukwa-0.4.0/src/java/org/apache/hadoop/chukwa/datacollection/writer/SeqFileWriter.java  line: 179
./chukwa-0.4.0/src/java/org/apache/hadoop/chukwa/datacollection/writer/localfs/LocalWriter.java   line:285
{noformat}

3 consecutive replace() is called to remove the special characters.  It's 3+ times slower than using a for loop 
replace them all.

{noformat}
e.g.
 - str.replace('a', '#');
 - str.replace('b', '%');

 + StringBuilder sb = new StringBuilder( str.length() );
 + for (int i=0; i < str.length(); i++)
 +  {
 +           char c = str.charAt(i);
 +         if ( c == 'a' )
 +               sb.append('#');
 +       else if ( c== 'b' )
 +               sb.append('%');
 +       else 
 +              sb.append(c); 
 +  }
 +  str  = sb.toString();
{noformat}
This bug has the same problem as the MySQL bug : http://bugs.mysql.com/bug.php?id=45699

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From chukwa-dev-returnnelsonheather@example.com Sun Mar 13 04:04:25 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 75117 invoked from network); 13 Mar 2011 04:04:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 13 Mar 2011 04:04:24 -0000
Received: (qmail 20106 invoked by uid 500); 13 Mar 2011 04:04:24 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 20045 invoked by uid 500); 13 Mar 2011 04:04:23 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 20037 invoked by uid 99); 13 Mar 2011 04:04:22 -0000
Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 04:04:22 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 04:04:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155])
	by hel.zones.apache.org (Postfix) with ESMTP id 984C83A60E5
	for <nelsonheather@example.com>; Sun, 13 Mar 2011 04:03:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 04:03:59 +0000 (UTC)
From: "Ari Rabkin (JIRA)" <nelsonheather@example.com>
To: nelsonheather@example.com
Message-ID: <nelsonheather@example.com>
In-Reply-To: <{{EMAIL+PHONE}}>
Subject: [jira] Commented: (CHUKWA-584) Multiple replace function call can
 be replaced with a single for loop to improve performance
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/CHUKWA-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006137#comment-13006137 ] 

Ari Rabkin commented on CHUKWA-584:
-----------------------------------

I'm happy to commit a patch if you generate one. But this seems like a low priority; I'm skeptical that those calls use enough CPU to matter. 

> Multiple replace function call can be replaced with a single for loop to improve performance 
> ---------------------------------------------------------------------------------------------
>
>                 Key: CHUKWA-584
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-584
>             Project: Chukwa
>          Issue Type: Bug
>          Components: data collection
>    Affects Versions: 0.4.0
>            Reporter: Xiaoming Shi
>
> {noformat}
> ./chukwa-0.4.0/src/java/org/apache/hadoop/chukwa/datacollection/writer/SeqFileWriter.java  line: 179
> ./chukwa-0.4.0/src/java/org/apache/hadoop/chukwa/datacollection/writer/localfs/LocalWriter.java   line:285
> {noformat}
> 3 consecutive replace() is called to remove the special characters.  It's 3+ times slower than using a for loop 
> replace them all.
> {noformat}
> e.g.
>  - str.replace('a', '#');
>  - str.replace('b', '%');
>  + StringBuilder sb = new StringBuilder( str.length() );
>  + for (int i=0; i < str.length(); i++)
>  +  {
>  +           char c = str.charAt(i);
>  +         if ( c == 'a' )
>  +               sb.append('#');
>  +       else if ( c== 'b' )
>  +               sb.append('%');
>  +       else 
>  +              sb.append(c); 
>  +  }
>  +  str  = sb.toString();
> {noformat}
> This bug has the same problem as the MySQL bug : http://bugs.mysql.com/bug.php?id=45699

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From chukwa-dev-returnnelsonheather@example.com Sun Mar 13 04:12:24 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 90703 invoked from network); 13 Mar 2011 04:12:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 13 Mar 2011 04:12:24 -0000
Received: (qmail 22806 invoked by uid 500); 13 Mar 2011 04:12:24 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 22724 invoked by uid 500); 13 Mar 2011 04:12:23 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 22716 invoked by uid 99); 13 Mar 2011 04:12:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 04:12:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 04:12:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155])
	by hel.zones.apache.org (Postfix) with ESMTP id 5FACA3A61B0
	for <nelsonheather@example.com>; Sun, 13 Mar 2011 04:11:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 04:11:59 +0000 (UTC)
From: "Ari Rabkin (JIRA)" <nelsonheather@example.com>
To: nelsonheather@example.com
Message-ID: <nelsonheather@example.com>
In-Reply-To: <{{EMAIL+PHONE}}>
Subject: [jira] Updated: (CHUKWA-583) Copying data from inputStream to
 OuputStream needs appropriate buffer size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


     [ https://issues.apache.org/jira/browse/CHUKWA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ari Rabkin updated CHUKWA-583:
------------------------------

    Priority: Trivial  (was: Major)

This looks like an autogenerated bug report. I would caution you against this.  The code you're flagging is in the test suite. It is NOT performance critical, nor is it run on substantial data volumes. It's probably pointless to change. The human time invested in generating and committing a patch is probably more than the "fix" would ever save.

> Copying data from inputStream to OuputStream needs appropriate buffer size
> --------------------------------------------------------------------------
>
>                 Key: CHUKWA-583
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-583
>             Project: Chukwa
>          Issue Type: Bug
>          Components: Data Processors
>    Affects Versions: 0.4.0
>            Reporter: Xiaoming Shi
>            Priority: Trivial
>
> In the file 
> ./chukwa-0.4.0/src/test/org/apache/hadoop/chukwa/validationframework/util/DataOperations.java   line: 54-58
> In the function copyFile, the buffer size is fixed as 4096 bytes. With the size of the data varies, the performance can be damaged a lot.
> We need an appropriate buffer size which depends on the size of the data to be copied. 
> This is the same as the Appache Bug (https://issues.apache.org/bugzilla/show_bug.cgi?id=32546)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From chukwa-dev-returnnelsonheather@example.com Sun Mar 13 04:18:22 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 822 invoked from network); 13 Mar 2011 04:18:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 13 Mar 2011 04:18:22 -0000
Received: (qmail 23263 invoked by uid 500); 13 Mar 2011 04:18:22 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 23241 invoked by uid 500); 13 Mar 2011 04:18:21 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 23229 invoked by uid 99); 13 Mar 2011 04:18:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 04:18:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 04:18:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155])
	by hel.zones.apache.org (Postfix) with ESMTP id 8F3D73A628F
	for <nelsonheather@example.com>; Sun, 13 Mar 2011 04:17:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 04:17:59 +0000 (UTC)
From: "Xiaoming Shi (JIRA)" <nelsonheather@example.com>
To: nelsonheather@example.com
Message-ID: <nelsonheather@example.com>
In-Reply-To: <{{EMAIL+PHONE}}>
Subject: [jira] Commented: (CHUKWA-583) Copying data from inputStream to
 OuputStream needs appropriate buffer size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/CHUKWA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006140#comment-13006140 ] 

Xiaoming Shi commented on CHUKWA-583:
-------------------------------------

To copy 10M data with 1024-byte buffer it takes 338,591,180 nano-second, while it takes 105,384,689 nano-second with a 10240-byte buffer.

Maybe you can check the parent bug report, if you have time.

> Copying data from inputStream to OuputStream needs appropriate buffer size
> --------------------------------------------------------------------------
>
>                 Key: CHUKWA-583
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-583
>             Project: Chukwa
>          Issue Type: Bug
>          Components: Data Processors
>    Affects Versions: 0.4.0
>            Reporter: Xiaoming Shi
>            Priority: Trivial
>
> In the file 
> ./chukwa-0.4.0/src/test/org/apache/hadoop/chukwa/validationframework/util/DataOperations.java   line: 54-58
> In the function copyFile, the buffer size is fixed as 4096 bytes. With the size of the data varies, the performance can be damaged a lot.
> We need an appropriate buffer size which depends on the size of the data to be copied. 
> This is the same as the Appache Bug (https://issues.apache.org/bugzilla/show_bug.cgi?id=32546)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From chukwa-dev-returnnelsonheather@example.com Sun Mar 13 04:26:23 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 34445 invoked from network); 13 Mar 2011 04:26:23 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 13 Mar 2011 04:26:23 -0000
Received: (qmail 25614 invoked by uid 500); 13 Mar 2011 04:26:22 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 25561 invoked by uid 500); 13 Mar 2011 04:26:22 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 25553 invoked by uid 99); 13 Mar 2011 04:26:21 -0000
Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 04:26:21 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 04:26:20 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155])
	by hel.zones.apache.org (Postfix) with ESMTP id 606893A6358
	for <nelsonheather@example.com>; Sun, 13 Mar 2011 04:25:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 04:25:59 +0000 (UTC)
From: "Xiaoming Shi (JIRA)" <nelsonheather@example.com>
To: nelsonheather@example.com
Message-ID: <nelsonheather@example.com>
In-Reply-To: <{{EMAIL+PHONE}}>
Subject: [jira] Commented: (CHUKWA-584) Multiple replace function call can
 be replaced with a single for loop to improve performance
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/CHUKWA-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006141#comment-13006141 ] 

Xiaoming Shi commented on CHUKWA-584:
-------------------------------------

Well, it depends on the frequency the functions are called and the length of the input.  It can make a big difference if it's frequently used or the size of the input is large 

> Multiple replace function call can be replaced with a single for loop to improve performance 
> ---------------------------------------------------------------------------------------------
>
>                 Key: CHUKWA-584
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-584
>             Project: Chukwa
>          Issue Type: Bug
>          Components: data collection
>    Affects Versions: 0.4.0
>            Reporter: Xiaoming Shi
>
> {noformat}
> ./chukwa-0.4.0/src/java/org/apache/hadoop/chukwa/datacollection/writer/SeqFileWriter.java  line: 179
> ./chukwa-0.4.0/src/java/org/apache/hadoop/chukwa/datacollection/writer/localfs/LocalWriter.java   line:285
> {noformat}
> 3 consecutive replace() is called to remove the special characters.  It's 3+ times slower than using a for loop 
> replace them all.
> {noformat}
> e.g.
>  - str.replace('a', '#');
>  - str.replace('b', '%');
>  + StringBuilder sb = new StringBuilder( str.length() );
>  + for (int i=0; i < str.length(); i++)
>  +  {
>  +           char c = str.charAt(i);
>  +         if ( c == 'a' )
>  +               sb.append('#');
>  +       else if ( c== 'b' )
>  +               sb.append('%');
>  +       else 
>  +              sb.append(c); 
>  +  }
>  +  str  = sb.toString();
> {noformat}
> This bug has the same problem as the MySQL bug : http://bugs.mysql.com/bug.php?id=45699

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From chukwa-dev-returnnelsonheather@example.com Sun Mar 13 18:43:25 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 34305 invoked from network); 13 Mar 2011 18:43:24 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 13 Mar 2011 18:43:24 -0000
Received: (qmail 30430 invoked by uid 500); 13 Mar 2011 18:43:24 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 30399 invoked by uid 500); 13 Mar 2011 18:43:24 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 30391 invoked by uid 99); 13 Mar 2011 18:43:23 -0000
Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 18:43:23 +0000
X-ASF-Spam-Status: No, hits=-2000.0 required=5.0
	tests=ALL_TRUSTED,T_RP_MATCHES_RCVD
X-Spam-Check-By: apache.org
Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 13 Mar 2011 18:43:21 +0000
Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155])
	by hel.zones.apache.org (Postfix) with ESMTP id BDCC93A7F1E
	for <nelsonheather@example.com>; Sun, 13 Mar 2011 18:42:59 +0000 (UTC)
Date: Sun, 13 Mar 2011 18:42:59 +0000 (UTC)
From: "Eric Yang (JIRA)" <nelsonheather@example.com>
To: nelsonheather@example.com
Message-ID: <nelsonheather@example.com>
In-Reply-To: <{{EMAIL+PHONE}}>
Subject: [jira] Commented: (CHUKWA-583) Copying data from inputStream to
 OuputStream needs appropriate buffer size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/CHUKWA-583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006248#comment-13006248 ] 

Eric Yang commented on CHUKWA-583:
----------------------------------

This bug is in the test code, and we don't push large amount of data over the validation framework.  Hence, it is a minor bug.  Patch to increase the size is welcome.

> Copying data from inputStream to OuputStream needs appropriate buffer size
> --------------------------------------------------------------------------
>
>                 Key: CHUKWA-583
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-583
>             Project: Chukwa
>          Issue Type: Bug
>          Components: Data Processors
>    Affects Versions: 0.4.0
>            Reporter: Xiaoming Shi
>            Priority: Trivial
>
> In the file 
> ./chukwa-0.4.0/src/test/org/apache/hadoop/chukwa/validationframework/util/DataOperations.java   line: 54-58
> In the function copyFile, the buffer size is fixed as 4096 bytes. With the size of the data varies, the performance can be damaged a lot.
> We need an appropriate buffer size which depends on the size of the data to be copied. 
> This is the same as the Appache Bug (https://issues.apache.org/bugzilla/show_bug.cgi?id=32546)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

From chukwa-dev-returnnelsonheather@example.com Mon Mar 21 17:44:26 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 2712 invoked from network); 21 Mar 2011 17:44:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 21 Mar 2011 17:44:26 -0000
Received: (qmail 5714 invoked by uid 500); 21 Mar 2011 17:44:26 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 5695 invoked by uid 500); 21 Mar 2011 17:44:25 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 5687 invoked by uid 99); 21 Mar 2011 17:44:25 -0000
Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 17:44:25 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [83.161.120.155] (HELO mrout2.yahoo.com) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 17:44:15 +0000
Received: from SP2-EX07CAS05.ds.corp.yahoo.com (sp2-ex07cas05.corp.sp2.yahoo.com [83.161.120.155])
	by mrout2.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p2LHhKlt066462
	for <nelsonheather@example.com>; Mon, 21 Mar 2011 10:43:20 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([83.161.120.155]) by
 SP2-EX07CAS05.ds.corp.yahoo.com ([83.161.120.155]) with mapi; Mon, 21 Mar 2011
 10:43:20 -0700
From: Eric Yang <nelsonheather@example.com>
To: "nelsonheather@example.com" <nelsonheather@example.com>
Date: Mon, 21 Mar 2011 10:43:16 -0700
Subject: [Discussion] Move to maven to make chukwa more modular
Thread-Topic: [Discussion] Move to maven to make chukwa more modular
Thread-Index: Acvn73U408RMzkOtGEWR71NqQ4XHzA==
Message-ID: <nelsonheather@example.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9ACDC441087Beyangyahooinccom_"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org

--_000_C9ACDC441087Beyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,

For making chukwa more user friendly, it may be worthwhile to convert the p=
roject into multiple sub modules.  Hence, interested party could pick and c=
hoose which module to build and deploy.  Maven seems like a good choice to =
manage dependencies among submodules and also improve integration with hado=
op and hbase versions.

The layout could look like this:

Trunk
  /common
  /agent
  /collector
  /demux
  /analytics
  /hicc

Demux is the combination of data processors.  It could be named "extraction=
s" as well. Collector module depends on demux module for HBaseWriter.

Since there is no official release of hadoop + hbase that work together.  I=
t will take some time for both projects to stabilize.  I think we can make =
this change while they are evolving.  Chukwa 0.5 release can come in the ti=
me when both projects released official versions then we release 0.5.  Thou=
ghts?

Regards,
Eric

--_000_C9ACDC441087Beyangyahooinccom_--

From chukwa-dev-returnnelsonheather@example.com Mon Mar 21 21:10:47 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 59630 invoked from network); 21 Mar 2011 21:10:47 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 21 Mar 2011 21:10:47 -0000
Received: (qmail 42745 invoked by uid 500); 21 Mar 2011 21:10:47 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 42726 invoked by uid 500); 21 Mar 2011 21:10:47 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 42718 invoked by uid 99); 21 Mar 2011 21:10:47 -0000
Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 21:10:47 +0000
X-ASF-Spam-Status: No, hits=-0.7 required=5.0
	tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of nelsonheather@example.com designates 83.161.120.155 as permitted sender)
Received: from [83.161.120.155] (HELO mail-ww0-f41.google.com) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 21:10:41 +0000
Received: by wwi18 with SMTP id 18so3790452wwi.0
        for <nelsonheather@example.com>; Mon, 21 Mar 2011 14:10:21 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:in-reply-to:references:date
         :message-id:subject:from:to:content-type:content-transfer-encoding;
        bh=IlusyL/CXTpTSFiUP8ITCe2Zwl3wFijp/XRX5TPFXMM=;
        b=nEXnRR0nEYiVjPPLjOxZ9alImcUCYf4Fykom9gx3n4T6JgkcFN7mW//P7RF89TbA6U
         QXIEVF1lZRWCS0g5TEReDIfPCtcN1itIGNQcBYireyVCH2lbbTVKdNCtzaVqixyhkr2C
         4JAnpSwGQamhvKynHmigah6XY79LzBEnaQIUE=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=HX8kiOW4W8iZTmlGeqo1bybBR+kLT2zmqmJsHrnDHnQ+T8AVh2eeTsgyTLlCtip7KT
         rxfXvuq/4j2y9UuLZP6M3bW9nRbSP4wRigzMbPrPVG0B2LxggVAwar2/chLD11gHkVnn
         ybj5xSsZqbMOYzj8zSahxpL+LZaEt4ChRoVRk=
MIME-Version: 1.0
Received: by 83.161.120.155 with SMTP id g11mr4551987wbu.57.1300741820705; Mon,
 21 Mar 2011 14:10:20 -0700 (PDT)
Received: by 83.161.120.155 with HTTP; Mon, 21 Mar 2011 14:10:19 -0700 (PDT)
In-Reply-To: <nelsonheather@example.com>
References: <Acvn73U408RMzkOtGEWR71NqQ4XHzA==>
	<nelsonheather@example.com>
Date: Mon, 21 Mar 2011 14:10:19 -0700
Message-ID: <nelsonheather@example.com>
Subject: Re: [Discussion] Move to maven to make chukwa more modular
From: Ariel Rabkin <nelsonheather@example.com>
To: nelsonheather@example.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
X-Virus-Checked: Checked by ClamAV on apache.org

I think maven would be good but isn't essential.
My inclination is to take our current trunk and try releasing it, just
to get familiar with the incubator release process and to help build
momentum.

--Ari

On Mon, Mar 21, 2011 at 10:43 AM, Eric Yang <nelsonheather@example.com> wrote:
> Hi,
>
> For making chukwa more user friendly, it may be worthwhile to convert the=
 project into multiple sub modules. =A0Hence, interested party could pick a=
nd choose which module to build and deploy. =A0Maven seems like a good choi=
ce to manage dependencies among submodules and also improve integration wit=
h hadoop and hbase versions.
>
> The layout could look like this:
>
> Trunk
> =A0/common
> =A0/agent
> =A0/collector
> =A0/demux
> =A0/analytics
> =A0/hicc
>
> Demux is the combination of data processors. =A0It could be named "extrac=
tions" as well. Collector module depends on demux module for HBaseWriter.
>
> Since there is no official release of hadoop + hbase that work together. =
=A0It will take some time for both projects to stabilize. =A0I think we can=
 make this change while they are evolving. =A0Chukwa 0.5 release can come i=
n the time when both projects released official versions then we release 0.=
5. =A0Thoughts?
>
> Regards,
> Eric
>



--=20
Ari Rabkin nelsonheather@example.com
UC Berkeley Computer Science Department

From chukwa-dev-returnnelsonheather@example.com Mon Mar 21 22:08:46 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 54828 invoked from network); 21 Mar 2011 22:08:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 21 Mar 2011 22:08:46 -0000
Received: (qmail 43618 invoked by uid 500); 21 Mar 2011 22:08:46 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 43594 invoked by uid 500); 21 Mar 2011 22:08:46 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 43586 invoked by uid 99); 21 Mar 2011 22:08:46 -0000
Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 22:08:46 +0000
X-ASF-Spam-Status: No, hits=2.1 required=5.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLYTO,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of nelsonheather@example.com designates 83.161.120.155 as permitted sender)
Received: from [83.161.120.155] (HELO mail-wy0-f175.google.com) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 22:08:40 +0000
Received: by wyb40 with SMTP id 40so5900242wyb.6
        for <nelsonheather@example.com>; Mon, 21 Mar 2011 15:08:19 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=domainkey-signature:mime-version:reply-to:in-reply-to:references
         :from:date:message-id:subject:to:content-type
         :content-transfer-encoding;
        bh=262sdneOMmtpUJjycIjWztelmfAaMCm6geWud4v++5Y=;
        b=MFz8Vs8VCGXUynLjR6eAf1vNwjAIW2a6124sr6qw/GEMqxS9I6tGwGyq9TbclMeots
         HHRjEpYX7F74Bkypf2dNP2/uNbghMI3lTdmwGflERVbATYU4bWJh0gCBTgezoRmwRsSP
         jkElBKkahCdryVaCJF2lZn69IdSxiAlano2to=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:reply-to:in-reply-to:references:from:date:message-id
         :subject:to:content-type:content-transfer-encoding;
        b=yFrI+ASYwTopOxVngIX3ajp/L1eBYLVSNi4aw7gNpVEZEoK47PGdTSIjY3phdJWOaL
         71gEESDdoIobnWIr4NUFx6pxV608gnVK9x7hEQIEKmuU1wnOVLvQWIq7MmAZr1LhK0zQ
         0UJBMib9QctgVtgcC5ZrVTT7lwcj1ndTZtg2M=
Received: by 83.161.120.155 with SMTP id r20mr5069365weo.30.1300745299253; Mon,
 21 Mar 2011 15:08:19 -0700 (PDT)
MIME-Version: 1.0
Received: by 83.161.120.155 with HTTP; Mon, 21 Mar 2011 15:07:59 -0700 (PDT)
Reply-To: nelsonheather@example.com
In-Reply-To: <nelsonheather@example.com>
References: <nelsonheather@example.com> <nelsonheather@example.com>
From: Bill Graham <nelsonheather@example.com>
Date: Mon, 21 Mar 2011 15:07:59 -0700
Message-ID: <nelsonheather@example.com>
Subject: Re: [Discussion] Move to maven to make chukwa more modular
To: nelsonheather@example.com
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

+1 on being able to build specific modules in Chukwa. I think it would
be great to be able to build each separately, along with only the
config files that each needs (we went through some pains to do this).

It seems like we could just use ant for this without having to make
major changes to the directory layout. Maybe we split the configs up
into sub-dirs, but do we need much more? It seems like the source
packages are already fairly well delineated.

If we don't want to wait on Hadoop/HBase, we could always release a
developer build of 4.x.



On Mon, Mar 21, 2011 at 2:10 PM, Ariel Rabkin <nelsonheather@example.com> wrote:
> I think maven would be good but isn't essential.
> My inclination is to take our current trunk and try releasing it, just
> to get familiar with the incubator release process and to help build
> momentum.
>
> --Ari
>
> On Mon, Mar 21, 2011 at 10:43 AM, Eric Yang <nelsonheather@example.com> wrote:
>> Hi,
>>
>> For making chukwa more user friendly, it may be worthwhile to convert th=
e project into multiple sub modules. =A0Hence, interested party could pick =
and choose which module to build and deploy. =A0Maven seems like a good cho=
ice to manage dependencies among submodules and also improve integration wi=
th hadoop and hbase versions.
>>
>> The layout could look like this:
>>
>> Trunk
>> =A0/common
>> =A0/agent
>> =A0/collector
>> =A0/demux
>> =A0/analytics
>> =A0/hicc
>>
>> Demux is the combination of data processors. =A0It could be named "extra=
ctions" as well. Collector module depends on demux module for HBaseWriter.
>>
>> Since there is no official release of hadoop + hbase that work together.=
 =A0It will take some time for both projects to stabilize. =A0I think we ca=
n make this change while they are evolving. =A0Chukwa 0.5 release can come =
in the time when both projects released official versions then we release 0=
.5. =A0Thoughts?
>>
>> Regards,
>> Eric
>>
>
>
>
> --
> Ari Rabkin nelsonheather@example.com
> UC Berkeley Computer Science Department
>

From chukwa-dev-returnnelsonheather@example.com Mon Mar 21 23:04:29 2011
Return-Path: <chukwa-dev-returnnelsonheather@example.com>
Delivered-To: nelsonheather@example.com
Received: (qmail 39873 invoked from network); 21 Mar 2011 23:04:29 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 21 Mar 2011 23:04:29 -0000
Received: (qmail 10845 invoked by uid 500); 21 Mar 2011 23:04:29 -0000
Delivered-To: nelsonheather@example.com
Received: (qmail 10818 invoked by uid 500); 21 Mar 2011 23:04:29 -0000
Mailing-List: contact nelsonheather@example.com; run by ezmlm
Precedence: bulk
List-Help: <mailto:nelsonheather@example.com>
List-Unsubscribe: <mailto:nelsonheather@example.com>
List-Post: <mailto:nelsonheather@example.com>
List-Id: <chukwa-dev.incubator.apache.org>
Reply-To: nelsonheather@example.com
Delivered-To: mailing list nelsonheather@example.com
Received: (qmail 10810 invoked by uid 99); 21 Mar 2011 23:04:29 -0000
Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 23:04:29 +0000
X-ASF-Spam-Status: No, hits=3.3 required=5.0
	tests=HTML_MESSAGE,NO_RDNS_DOTCOM_HELO,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [83.161.120.155] (HELO mrout1-b.corp.re1.yahoo.com) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2011 23:04:22 +0000
Received: from SP2-EX07CAS04.ds.corp.yahoo.com (sp2-ex07cas04.corp.sp2.yahoo.com [83.161.120.155])
	by mrout1-b.corp.re1.yahoo.com (8.14.4/8.14.4/y.out) with ESMTP id p2LN3ZQh094368;
	Mon, 21 Mar 2011 16:03:35 -0700 (PDT)
Received: from SP2-EX07VS05.ds.corp.yahoo.com ([83.161.120.155]) by
 SP2-EX07CAS04.ds.corp.yahoo.com ([83.161.120.155]) with mapi; Mon, 21 Mar 2011
 16:03:34 -0700
From: Eric Yang <nelsonheather@example.com>
To: "nelsonheather@example.com" <nelsonheather@example.com>,
        "nelsonheather@example.com" <nelsonheather@example.com>
Date: Mon, 21 Mar 2011 16:03:33 -0700
Subject: Re: [Discussion] Move to maven to make chukwa more modular
Thread-Topic: [Discussion] Move to maven to make chukwa more modular
Thread-Index: AcvoFJrJGUhsV411STmXKrkF3vhSkwAB5iqE
Message-ID: <nelsonheather@example.com>
In-Reply-To: <nelsonheather@example.com>
Accept-Language: en-US
Content-Language: en
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C9AD2755108BBeyangyahooinccom_"
MIME-Version: 1.0

--_000_C9AD2755108BBeyangyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Advantage of maven, we will be able to publish our jar file into maven repo=
sitory for other projects to adopt our software.  It also cuts a cleaner se=
paration of module dependencies for packaging.  In the case of separating c=
onfig, maven can express the packaging once, and not repeating in super pom=
.  In ant, this needs to be setup manually, and build target dependency may=
 not seem clear over time.

Regards,
Eric

On 3/21/11 3:07 PM, "Bill Graham" <nelsonheather@example.com> wrote:

+1 on being able to build specific modules in Chukwa. I think it would
be great to be able to build each separately, along with only the
config files that each needs (we went through some pains to do this).

It seems like we could just use ant for this without having to make
major changes to the directory layout. Maybe we split the configs up
into sub-dirs, but do we need much more? It seems like the source
packages are already fairly well delineated.

If we don't want to wait on Hadoop/HBase, we could always release a
developer build of 4.x.



On Mon, Mar 21, 2011 at 2:10 PM, Ariel Rabkin <nelsonheather@example.com> wrote:
> I think maven would be good but isn't essential.
> My inclination is to take our current trunk and try releasing it, just
> to get familiar with the incubator release process and to help build
> momentum.
>
> --Ari
>
> On Mon, Mar 21, 2011 at 10:43 AM, Eric Yang <nelsonheather@example.com> wrote:
>> Hi,
>>
>> For making chukwa more user friendly, it may be worthwhile to convert th=
e project into multiple sub modules.  Hence, interested party could pick an=
d choose which module to build and deploy.  Maven seems like a good choice =
to manage dependencies among submodules and also improve integration with h=
adoop and hbase versions.
>>
>> The layout could look like this:
>>
>> Trunk
>>  /common
>>  /agent
>>  /collector
>>  /demux
>>  /analytics
>>  /hicc
>>
>> Demux is the combination of data processors.  It could be named "extract=
ions" as well. Collector module depends on demux module for HBaseWriter.
>>
>> Since there is no official release of hadoop + hbase that work together.=
  It will take some time for both projects to stabilize.  I think we can ma=
ke this change while they are evolving.  Chukwa 0.5 release can come in the=
 time when both projects released official versions then we release 0.5.  T=
houghts?
>>
>> Regards,
>> Eric
>>
>
>
>
> --
> Ari Rabkin nelsonheather@example.com
> UC Berkeley Computer Science Department
>


--_000_C9AD2755108BBeyangyahooinccom_--


{'source': 'Apache Software Foundation Public Mail Archives'}