From pigvshelton@example.com Wed Apr 01 00:20:12 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 31926 invoked from network); 1 Apr 2009 00:20:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 00:20:12 -0000 Received: (qmail 76257 invoked by uid 500); 1 Apr 2009 00:20:11 -0000 Delivered-To: vshelton@example.com Received: (qmail 76196 invoked by uid 500); 1 Apr 2009 00:20:11 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 76183 invoked by uid 99); 1 Apr 2009 00:20:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 00:20:11 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 00:20:10 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 667B4234C003 for <vshelton@example.com>; Tue, 31 Mar 2009 17:19:50 -0700 (PDT) Message-ID: <190370103.1238545190404.JavaMail.jira@brutus> Date: Tue, 31 Mar 2009 17:19:50 -0700 (PDT) From: "Viraj Bhat (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Created: (PIG-742) Spaces could be optional in Pig syntax MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org Spaces could be optional in Pig syntax -------------------------------------- Key: PIG-742 URL: https://issues.apache.org/jira/browse/PIG-742 Project: Pig Issue Type: Wish Components: grunt Affects Versions: 0.1.0 Reporter: Viraj Bhat Priority: Minor Fix For: 0.1.0 The following Pig statements generate an error if there is no space between A and "=" {code} A=load 'quf.txt' using PigStorage() as (q, u, f:long); B = group A by (q); C = foreach B { F = order A by f desc; generate F; }; describe C; dump C; {code} 2009-03-31 17:14:15,959 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Encountered " <PATH> "A=load "" at line 1, column 1. Was expecting one of: <EOF> "cat" ... "cd" ... "cp" ... "copyFromLocal" ... "copyToLocal" ... "dump" ... "describe" ... "aliases" ... "explain" ... "help" ... "kill" ... "ls" ... "mv" ... "mkdir" ... "pwd" ... "quit" ... "register" ... "rm" ... "rmf" ... "set" ... "illustrate" ... "run" ... "exec" ... "scriptDone" ... "" ... <EOL> ... ";" ... It would be nice if the parser would not expect these space requirements between an alias and "=" -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Wed Apr 01 02:15:27 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 91764 invoked from network); 1 Apr 2009 02:15:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 02:15:27 -0000 Received: (qmail 67559 invoked by uid 500); 1 Apr 2009 02:15:26 -0000 Delivered-To: vshelton@example.com Received: (qmail 67516 invoked by uid 500); 1 Apr 2009 02:15:26 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 67506 invoked by uid 99); 1 Apr 2009 02:15:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 02:15:26 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of vshelton@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f164.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 02:15:17 +0000 Received: by gxk8 with SMTP id 8so6194690gxk.5 for <vshelton@example.com>; Tue, 31 Mar 2009 19:14:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to:references :subject:date:mime-version:content-type:content-transfer-encoding :x-priority:x-msmail-priority:x-mailer:x-mimeole; bh=CD8PrDGTGAhn40Nv7POmEmnkCZOVRHhMl2GbVP7AZec=; b=kPGy7w3PvwuoEdDu8ku5Rw0dml8tncSlCgvSaVa3R796uCKd3H48ntBqENcLZskYiG h9irf4jkyEgn/fVa56shfAYq7zc6KQxlResGKGG44l8gAfyexPfDuEH36H12l6EtMLwc Q67RVPZdTllDB4oWEUidg6Rg6W1CJftUCkF3Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:references:subject:date:mime-version :content-type:content-transfer-encoding:x-priority:x-msmail-priority :x-mailer:x-mimeole; b=JxPX2iR8nhatDHuC9JtH76bb2laypXK6Cd5sDEGmh/42G0E9uEetUfVWHcsu1kk47I JyHQ4uUOnls46i01KczaTCRxqyoemFoCbpwAi4iC/NnTyybiRI3Cg5ffT1mz7AysO+K8 UAdK2EOMadM4GMtZDUEGUWOX7EPd0GNXYx6yU= Received: by 83.161.120.155 with SMTP id 3mr3787355agz.49.1238552096384; Tue, 31 Mar 2009 19:14:56 -0700 (PDT) Received: from daijypc ([83.161.120.155]) by mx.google.com with ESMTPS id 21sm1131855agd.27.283.161.120.155.14.54 (version=SSLv3 cipher=RC4-MD5); Tue, 31 Mar 2009 19:14:55 -0700 (PDT) Message-ID: <F0059B3E33A2476EA02BAC15F8594877@daijypc> From: "Daniel Dai" <vshelton@example.com> To: <vshelton@example.com> References: <vshelton@example.com>,<vshelton@example.com> <vshelton@example.com> Subject: Re: [VOTE] Release Pig 0.2.0 (candidate 2) Date: Tue, 31 Mar 2009 22:14:53 -0400 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5512 x-mimeole: Produced By Microsoft MimeOLE V6.00.2900.5579 X-Virus-Checked: Checked by ClamAV on apache.org Sorry for popping up so late, but I do find several unit test failures under cygwin (TestLogToPhyCompiler, TestLogicalOptimizer, TestParamSubPreproc). All of them are caused by distorted new line character of some golden data files. Simply retrieve the data file from trunk again will solve the problem. Here is a list of the data files need to be retrieved from trunk: dir test/org/apache/pig/test/data/GoldenFiles: Generate.gld IsNull2.gld Distinct.gld Cogroup.gld Arithmetic.gld Comparison.gld Limit.gld ComplexForeach.gld Sort.gld Union.gld Split1.gld Split2.gld dir test/org/apache/pig/test/data/DotFiles: optlimitplan1.dot optlimitplan2.dot ...... optlimitplan9.dot dir test/org/apache/pig/test/data generate_date.sh Daniel ----- Original Message ----- From: "Benjamin Reed" <vshelton@example.com> To: <vshelton@example.com> Sent: Tuesday, March 31, 2009 5:49 PM Subject: RE: [VOTE] Release Pig 0.2.0 (candidate 2) +1 ben ________________________________________ From: Santhosh Srinivasan [vshelton@example.com] Sent: Tuesday, March 31, 2009 10:50 AM To: vshelton@example.com Subject: RE: [VOTE] Release Pig 0.2.0 (candidate 2) +1 Santhosh -----Original Message----- From: Alan Gates [mailto:vshelton@example.com] Sent: Friday, March 27, 2009 6:08 PM To: vshelton@example.com Subject: [VOTE] Release Pig 0.2.0 (candidate 2) Pig Committers, I have created a candidate build for Pig 0.2.0. This is the same release as proposed by Olga last week, with the agreed upon change in the release number from 1.0 to 0.2. This release represents a major rewrite of Pig from the parser down. It also introduced a type system into Pig and greatly improves system performance. The rat report is attached inline. Note that there are many java files listed as being without a license header. All these files are generated by javacc. Keys used to sign the release are available at http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS?view=markup Please download, test, and try it out: http://people.apache.org/~gates/pig-0.2.0-candidate-2 Should we release this? Vote closes on Wednesday, April 1st. Alan. From pigvshelton@example.com Wed Apr 01 08:31:36 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 52267 invoked from network); 1 Apr 2009 08:31:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 08:31:36 -0000 Received: (qmail 93002 invoked by uid 500); 1 Apr 2009 08:31:36 -0000 Delivered-To: vshelton@example.com Received: (qmail 92972 invoked by uid 500); 1 Apr 2009 08:31:36 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 92962 invoked by uid 99); 1 Apr 2009 08:31:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 08:31:36 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 08:31:34 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 10822234C004 for <vshelton@example.com>; Wed, 1 Apr 2009 01:31:13 -0700 (PDT) Message-ID: <321781279.1238574673056.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 01:31:13 -0700 (PDT) From: "Gunther Hagleitner (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Updated: (PIG-627) PERFORMANCE: multi-query optimization In-Reply-To: <1915989148.1232498399494.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated PIG-627: ----------------------------------- Attachment: noop_filter_absolute_path_flag_0401.patch This one is the same as before, but: * Added some comments * Reversed the multiquery flag (on by default) * HBase stuff works without the "hbase://" but will print warning * Fixed problem in NoopStoreRemover > PERFORMANCE: multi-query optimization > ------------------------------------- > > Key: PIG-627 > URL: https://issues.apache.org/jira/browse/PIG-627 > Project: Pig > Issue Type: Improvement > Affects Versions: 0.2.0 > Reporter: Olga Natkovich > Attachments: file_cmds-0305.patch, fix_store_prob.patch, merge_741727_HEAD__0324.patch, merge_741727_HEAD__0324_2.patch, multi-store-0303.patch, multi-store-0304.patch, multiquery-phase2_0313.patch, multiquery-phase2_0323.patch, multiquery_0223.patch, multiquery_0224.patch, multiquery_0306.patch, multiquery_explain_fix.patch, noop_filter_absolute_path_flag.patch, noop_filter_absolute_path_flag_0401.patch > > > Currently, if your Pig script contains multiple stores and some shared computation, Pig will execute several independent queries. For instance: > A = load 'data' as (a, b, c); > B = filter A by a > 5; > store B into 'output1'; > C = group B by b; > store C into 'output2'; > This script will result in map-only job that generated output1 followed by a map-reduce job that generated output2. As the resuld data is read, parsed and filetered twice which is unnecessary and costly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Wed Apr 01 11:21:38 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 48781 invoked from network); 1 Apr 2009 11:21:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 11:21:37 -0000 Received: (qmail 12519 invoked by uid 500); 1 Apr 2009 11:21:37 -0000 Delivered-To: vshelton@example.com Received: (qmail 12469 invoked by uid 500); 1 Apr 2009 11:21:37 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 12457 invoked by uid 99); 1 Apr 2009 11:21:37 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 11:21:37 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 11:21:35 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id DEB23234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 04:21:12 -0700 (PDT) Message-ID: <213413569.1238584872897.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 04:21:12 -0700 (PDT) From: "Giridharan Kesavan (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Created: (PIG-743) to implement clover MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org to implement clover ------------------- Key: PIG-743 URL: https://issues.apache.org/jira/browse/PIG-743 Project: Pig Issue Type: Improvement Components: build Reporter: Giridharan Kesavan -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Wed Apr 01 18:03:24 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 90651 invoked from network); 1 Apr 2009 18:03:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 18:03:23 -0000 Received: (qmail 88673 invoked by uid 500); 1 Apr 2009 18:03:23 -0000 Delivered-To: vshelton@example.com Received: (qmail 88619 invoked by uid 500); 1 Apr 2009 18:03:23 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 88609 invoked by uid 99); 1 Apr 2009 18:03:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 18:03:23 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 18:03:11 +0000 Received: from wlanvpn-mc2e-246-120.corp.yahoo.com (wlanvpn-mc2e-246-120.corp.yahoo.com [83.161.120.155]) by mrout1.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id n31I2NYG066725 for <vshelton@example.com>; Wed, 1 Apr 2009 11:02:23 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=message-id:from:to:in-reply-to:content-type: content-transfer-encoding:mime-version:subject:x-priority:date:references:x-mailer; b=UP/VoevYKK92nB32a6KfHbDEjAty+jfNBmrAD5f3uk6aZ70DL0pslHk1JrV1QRaU Message-Id: <vshelton@example.com> From: Alan Gates <vshelton@example.com> To: vshelton@example.com In-Reply-To: <F0059B3E33A2476EA02BAC15F8594877@daijypc> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v930.3) Subject: Re: [VOTE] Release Pig 0.2.0 (candidate 2) X-Priority: 3 Date: Wed, 1 Apr 2009 11:02:23 -0700 References: <vshelton@example.com>,<vshelton@example.com> <vshelton@example.com> <F0059B3E33A2476EA02BAC15F8594877@daijypc> X-Mailer: Apple Mail (2.930.3) X-Virus-Checked: Checked by ClamAV on apache.org Daniel, We I diffed all of these files between the branch and the trunk and didn't see any difference. Are you seeing the tests pass in trunk and not in the branch? Alan. On Mar 31, 2009, at 7:14 PM, Daniel Dai wrote: > Sorry for popping up so late, but I do find several unit test > failures under > cygwin (TestLogToPhyCompiler, TestLogicalOptimizer, > TestParamSubPreproc). > All of them are caused by distorted new line character of some > golden data > files. Simply retrieve the data file from trunk again will solve the > problem. Here > is a list of the data files need to be retrieved from trunk: > > dir test/org/apache/pig/test/data/GoldenFiles: > Generate.gld > IsNull2.gld > Distinct.gld > Cogroup.gld > Arithmetic.gld > Comparison.gld > Limit.gld > ComplexForeach.gld > Sort.gld > Union.gld > Split1.gld > Split2.gld > > dir test/org/apache/pig/test/data/DotFiles: > optlimitplan1.dot > optlimitplan2.dot > ...... > optlimitplan9.dot > > dir test/org/apache/pig/test/data > generate_date.sh > > Daniel > > ----- Original Message ----- From: "Benjamin Reed" <vshelton@example.com > > > To: <vshelton@example.com> > Sent: Tuesday, March 31, 2009 5:49 PM > Subject: RE: [VOTE] Release Pig 0.2.0 (candidate 2) > > > +1 > > ben > > ________________________________________ > From: Santhosh Srinivasan [vshelton@example.com] > Sent: Tuesday, March 31, 2009 10:50 AM > To: vshelton@example.com > Subject: RE: [VOTE] Release Pig 0.2.0 (candidate 2) > > +1 > > Santhosh > > -----Original Message----- > From: Alan Gates [mailto:vshelton@example.com] > Sent: Friday, March 27, 2009 6:08 PM > To: vshelton@example.com > Subject: [VOTE] Release Pig 0.2.0 (candidate 2) > > Pig Committers, > > I have created a candidate build for Pig 0.2.0. This is the same > release as proposed by Olga last week, with the agreed upon change in > the release number from 1.0 to 0.2. > > This release represents a major rewrite of Pig from the parser down. > It > also introduced a type system into Pig and greatly improves system > performance. > > The rat report is attached inline. Note that there are many java files > listed > as being without a license header. All these files are generated by > javacc. > > Keys used to sign the release are available at > http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS?view=markup > > Please download, test, and try it out: > > http://people.apache.org/~gates/pig-0.2.0-candidate-2 > > > Should we release this? Vote closes on Wednesday, April 1st. > > Alan. > From pigvshelton@example.com Wed Apr 01 18:35:11 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 17946 invoked from network); 1 Apr 2009 18:35:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 18:35:11 -0000 Received: (qmail 58128 invoked by uid 500); 1 Apr 2009 18:35:10 -0000 Delivered-To: vshelton@example.com Received: (qmail 58079 invoked by uid 500); 1 Apr 2009 18:35:10 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 58065 invoked by uid 99); 1 Apr 2009 18:35:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 18:35:10 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of vshelton@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f164.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 18:35:02 +0000 Received: by gxk8 with SMTP id 8so421791gxk.5 for <vshelton@example.com>; Wed, 01 Apr 2009 11:34:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to:references :subject:date:mime-version:content-type:content-transfer-encoding :x-priority:x-msmail-priority:x-mailer:x-mimeole; bh=B0jxz3Apmity98sguuYZKPp3HuRVfTQvIpKn+EvFEqE=; b=ADE8SnV7aQ8H9u6r8jYzd5b654SIA9dJEBwdiSKsGfgkAiDI4GSA2237zL8cnIPHsn GCi2UQY00ip8W4YByisiwT7z2fqrey9Ef0qZPyFTpQaUFhob+hZr4ngmMySE7KoZCNOc jyE6RdXC0sDP55NFCn3Gok58Iy+L6wh1lBRD0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:references:subject:date:mime-version :content-type:content-transfer-encoding:x-priority:x-msmail-priority :x-mailer:x-mimeole; b=P3INT+2j/ge+/KJ5SkDfoNAgDuBj7qiVgk3lqUm8DcBB/00GXa1UFt+D3FyIjXq32y Ysuq/SZ8heJyFNoHmZ+b4BuEwJKoElZdmjFUhb2qhOSVThXggtr+I7SkEvSWEA+9ERl/ vdeWLpiR4uTli2joBnUcsc8SMHR4v7nqiAud8= Received: by 83.161.120.155 with SMTP id o14mr3276428aga.104.1238610880975; Wed, 01 Apr 2009 11:34:40 -0700 (PDT) Received: from daijypc ([83.161.120.155]) by mx.google.com with ESMTPS id 27sm392810agb.51.283.161.120.155.34.39 (version=SSLv3 cipher=RC4-MD5); Wed, 01 Apr 2009 11:34:39 -0700 (PDT) Message-ID: <BEBEB5878C094C51B9646CC7AC0D10F4@daijypc> From: "Daniel Dai" <vshelton@example.com> To: <vshelton@example.com> References: <vshelton@example.com>,<vshelton@example.com> <vshelton@example.com> <F0059B3E33A2476EA02BAC15F8594877@daijypc> <vshelton@example.com> Subject: Re: [VOTE] Release Pig 0.2.0 (candidate 2) Date: Wed, 1 Apr 2009 14:34:37 -0400 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5512 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 X-Virus-Checked: Checked by ClamAV on apache.org These files differ only in spaces. Some unit test fails if use the 0.2.0 tarball directly, but if you replace those data files with the one in trunk, then test success. ----- Original Message ----- From: "Alan Gates" <vshelton@example.com> To: <vshelton@example.com> Sent: Wednesday, April 01, 2009 2:02 PM Subject: Re: [VOTE] Release Pig 0.2.0 (candidate 2) > Daniel, > > We I diffed all of these files between the branch and the trunk and > didn't see any difference. Are you seeing the tests pass in trunk and > not in the branch? > > Alan. > > On Mar 31, 2009, at 7:14 PM, Daniel Dai wrote: > >> Sorry for popping up so late, but I do find several unit test failures >> under >> cygwin (TestLogToPhyCompiler, TestLogicalOptimizer, >> TestParamSubPreproc). >> All of them are caused by distorted new line character of some golden >> data >> files. Simply retrieve the data file from trunk again will solve the >> problem. Here >> is a list of the data files need to be retrieved from trunk: >> >> dir test/org/apache/pig/test/data/GoldenFiles: >> Generate.gld >> IsNull2.gld >> Distinct.gld >> Cogroup.gld >> Arithmetic.gld >> Comparison.gld >> Limit.gld >> ComplexForeach.gld >> Sort.gld >> Union.gld >> Split1.gld >> Split2.gld >> >> dir test/org/apache/pig/test/data/DotFiles: >> optlimitplan1.dot >> optlimitplan2.dot >> ...... >> optlimitplan9.dot >> >> dir test/org/apache/pig/test/data >> generate_date.sh >> >> Daniel >> >> ----- Original Message ----- From: "Benjamin Reed" <vshelton@example.com >> > >> To: <vshelton@example.com> >> Sent: Tuesday, March 31, 2009 5:49 PM >> Subject: RE: [VOTE] Release Pig 0.2.0 (candidate 2) >> >> >> +1 >> >> ben >> >> ________________________________________ >> From: Santhosh Srinivasan [vshelton@example.com] >> Sent: Tuesday, March 31, 2009 10:50 AM >> To: vshelton@example.com >> Subject: RE: [VOTE] Release Pig 0.2.0 (candidate 2) >> >> +1 >> >> Santhosh >> >> -----Original Message----- >> From: Alan Gates [mailto:vshelton@example.com] >> Sent: Friday, March 27, 2009 6:08 PM >> To: vshelton@example.com >> Subject: [VOTE] Release Pig 0.2.0 (candidate 2) >> >> Pig Committers, >> >> I have created a candidate build for Pig 0.2.0. This is the same >> release as proposed by Olga last week, with the agreed upon change in >> the release number from 1.0 to 0.2. >> >> This release represents a major rewrite of Pig from the parser down. It >> also introduced a type system into Pig and greatly improves system >> performance. >> >> The rat report is attached inline. Note that there are many java files >> listed >> as being without a license header. All these files are generated by >> javacc. >> >> Keys used to sign the release are available at >> http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS?view=markup >> >> Please download, test, and try it out: >> >> http://people.apache.org/~gates/pig-0.2.0-candidate-2 >> >> >> Should we release this? Vote closes on Wednesday, April 1st. >> >> Alan. >> > From pigvshelton@example.com Wed Apr 01 18:53:59 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 27621 invoked from network); 1 Apr 2009 18:53:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 18:53:59 -0000 Received: (qmail 83781 invoked by uid 500); 1 Apr 2009 18:53:59 -0000 Delivered-To: vshelton@example.com Received: (qmail 83715 invoked by uid 500); 1 Apr 2009 18:53:59 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 83705 invoked by uid 99); 1 Apr 2009 18:53:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 18:53:59 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of vshelton@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f164.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 18:53:49 +0000 Received: by gxk8 with SMTP id 8so442535gxk.5 for <vshelton@example.com>; Wed, 01 Apr 2009 11:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to:references :subject:date:mime-version:content-type:content-transfer-encoding :x-priority:x-msmail-priority:x-mailer:x-mimeole; bh=RWqFpsI2tO64PcWHLWGUs6R+kTETsauArJgMW61K1f0=; b=GA7Q1gcn+qjCzTsnJe3EWmj56pQeIbhvKYHnKtkOLeY3VnFixZxoT0rX1AaDRkbT/k yMRZJJ5lOlx6WiIPrrpHikEifxYBFNt6XSzDWX10DFM+7jKTGFYHjeM/gE4DeQiUWQHF I9+KCAZXVMbYHoBYTX8mz074bnkxZoX8fF1Tc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:references:subject:date:mime-version :content-type:content-transfer-encoding:x-priority:x-msmail-priority :x-mailer:x-mimeole; b=obrbM971hSsO5ZaP5C265c1jl8p6GH5l7n3FC4dGjX1iogWhzogBZT4QZH1yEPKGsB R83MGLU/4YaOJ11kr4zHK9Tfo/AlHAjSAO/R8fkAEO+TThjwv2XtM4Jc453rDGxn0KTV 7BxNhgJqBJqI0dmULCL0vKRpCak6pZlVR0GfY= Received: by 83.161.120.155 with SMTP id d17mr3577308agc.68.1238612008247; Wed, 01 Apr 2009 11:53:28 -0700 (PDT) Received: from daijypc ([83.161.120.155]) by mx.google.com with ESMTPS id 11sm391737aga.70.283.161.120.155.53.27 (version=SSLv3 cipher=RC4-MD5); Wed, 01 Apr 2009 11:53:27 -0700 (PDT) Message-ID: <2F921D52B8574767BF7B5BDDDBACDC28@daijypc> From: "Daniel Dai" <vshelton@example.com> To: <vshelton@example.com> References: <vshelton@example.com>,<vshelton@example.com> <vshelton@example.com> <F0059B3E33A2476EA02BAC15F8594877@daijypc> <vshelton@example.com> Subject: Re: [VOTE] Release Pig 0.2.0 (candidate 2) Date: Wed, 1 Apr 2009 14:53:25 -0400 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5512 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.5579 X-Virus-Checked: Checked by ClamAV on apache.org Hi, Alan, I find the problem. The tarball is good. The problem I met is when I untar it use Winzip, Winzip automatically convert CR/LR. When I untar it using command line "tar", it is good. So no problem for the 0.2.0 tarball. Sorry for the confusion. +1 for the release. Daniel ----- Original Message ----- From: "Alan Gates" <vshelton@example.com> To: <vshelton@example.com> Sent: Wednesday, April 01, 2009 2:02 PM Subject: Re: [VOTE] Release Pig 0.2.0 (candidate 2) > Daniel, > > We I diffed all of these files between the branch and the trunk and > didn't see any difference. Are you seeing the tests pass in trunk and > not in the branch? > > Alan. > > On Mar 31, 2009, at 7:14 PM, Daniel Dai wrote: > >> Sorry for popping up so late, but I do find several unit test failures >> under >> cygwin (TestLogToPhyCompiler, TestLogicalOptimizer, >> TestParamSubPreproc). >> All of them are caused by distorted new line character of some golden >> data >> files. Simply retrieve the data file from trunk again will solve the >> problem. Here >> is a list of the data files need to be retrieved from trunk: >> >> dir test/org/apache/pig/test/data/GoldenFiles: >> Generate.gld >> IsNull2.gld >> Distinct.gld >> Cogroup.gld >> Arithmetic.gld >> Comparison.gld >> Limit.gld >> ComplexForeach.gld >> Sort.gld >> Union.gld >> Split1.gld >> Split2.gld >> >> dir test/org/apache/pig/test/data/DotFiles: >> optlimitplan1.dot >> optlimitplan2.dot >> ...... >> optlimitplan9.dot >> >> dir test/org/apache/pig/test/data >> generate_date.sh >> >> Daniel >> >> ----- Original Message ----- From: "Benjamin Reed" <vshelton@example.com >> > >> To: <vshelton@example.com> >> Sent: Tuesday, March 31, 2009 5:49 PM >> Subject: RE: [VOTE] Release Pig 0.2.0 (candidate 2) >> >> >> +1 >> >> ben >> >> ________________________________________ >> From: Santhosh Srinivasan [vshelton@example.com] >> Sent: Tuesday, March 31, 2009 10:50 AM >> To: vshelton@example.com >> Subject: RE: [VOTE] Release Pig 0.2.0 (candidate 2) >> >> +1 >> >> Santhosh >> >> -----Original Message----- >> From: Alan Gates [mailto:vshelton@example.com] >> Sent: Friday, March 27, 2009 6:08 PM >> To: vshelton@example.com >> Subject: [VOTE] Release Pig 0.2.0 (candidate 2) >> >> Pig Committers, >> >> I have created a candidate build for Pig 0.2.0. This is the same >> release as proposed by Olga last week, with the agreed upon change in >> the release number from 1.0 to 0.2. >> >> This release represents a major rewrite of Pig from the parser down. It >> also introduced a type system into Pig and greatly improves system >> performance. >> >> The rat report is attached inline. Note that there are many java files >> listed >> as being without a license header. All these files are generated by >> javacc. >> >> Keys used to sign the release are available at >> http://svn.apache.org/viewvc/hadoop/pig/trunk/KEYS?view=markup >> >> Please download, test, and try it out: >> >> http://people.apache.org/~gates/pig-0.2.0-candidate-2 >> >> >> Should we release this? Vote closes on Wednesday, April 1st. >> >> Alan. >> > From pigvshelton@example.com Wed Apr 01 20:39:38 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 67072 invoked from network); 1 Apr 2009 20:39:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 20:39:37 -0000 Received: (qmail 30385 invoked by uid 500); 1 Apr 2009 20:39:36 -0000 Delivered-To: vshelton@example.com Received: (qmail 30316 invoked by uid 500); 1 Apr 2009 20:39:36 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 30181 invoked by uid 99); 1 Apr 2009 20:39:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 20:39:36 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 20:39:34 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 5728C234C48C for <vshelton@example.com>; Wed, 1 Apr 2009 13:39:13 -0700 (PDT) Message-ID: <220-228-4882.1238618353355.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 13:39:13 -0700 (PDT) From: "Pradeep Kamath (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Created: (PIG-744) PERFORMANCE: Bag creation can be more efficiently handled in order by MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org PERFORMANCE: Bag creation can be more efficiently handled in order by --------------------------------------------------------------------- Key: PIG-744 URL: https://issues.apache.org/jira/browse/PIG-744 Project: Pig Issue Type: Improvement Affects Versions: 0.2.0 Reporter: Pradeep Kamath Fix For: 0.3.0 Currently order by results in multiple map reduce jobs (2 or 3 depending on the script) of which the last one does the actual ordering. In this last map reduce job, we create a bag of values (each value being the entire tuple that is getting sorted) for each sort key(s) using POPackage in the reduce phase. Then we turn around and flatten the bag in the foreach following the package. So there is really no need for the bag. But to be generic and use the existing operators, we can be more efficient by tagging the POPackage to create bags which are backed by the Hadoop iterator itself. This way we do not create a bag by making a copy of each tuple from the hadoop iterator. This should help both performance and scalability by making better use of memory. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Wed Apr 01 20:45:34 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 69456 invoked from network); 1 Apr 2009 20:45:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 20:45:34 -0000 Received: (qmail 39124 invoked by uid 500); 1 Apr 2009 20:45:34 -0000 Delivered-To: vshelton@example.com Received: (qmail 39097 invoked by uid 500); 1 Apr 2009 20:45:34 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 39086 invoked by uid 99); 1 Apr 2009 20:45:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 20:45:34 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 20:45:33 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id E937A234C045 for <vshelton@example.com>; Wed, 1 Apr 2009 13:45:12 -0700 (PDT) Message-ID: <1656160341.1238618712954.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 13:45:12 -0700 (PDT) From: "David Ciemiewicz (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Created: (PIG-745) Please add DataTypes.toString() conversion function MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org Please add DataTypes.toString() conversion function --------------------------------------------------- Key: PIG-745 URL: https://issues.apache.org/jira/browse/PIG-745 Project: Pig Issue Type: Improvement Reporter: David Ciemiewicz I'm doing some work in string manipulation UDFs and I've found that it would be very convenient if I could always convert the argument to a chararray (internally a Java String). For example TOLOWERCASE(arg) shouldn't really care whether arg is a bytearray, chararray, int, long, double, or float, it should be treated as a string and operated on. The simplest and most foolproof method would be if the DataTypes added a static function of DataTypes.toString which did all of the argument type checking and provided consistent translation. I believe that this function might be coded as: public static String toString(Object o) throws ExecException { try { switch (findType(o)) { case BOOLEAN: if (((Boolean)o) == true) return new String('1'); else return new String('0'); case BYTE: return ((Byte)o).toString(); case INTEGER: return ((Integer)o).toString(); case LONG: return ((Long)o).toString(); case FLOAT: return ((Float)o).toString(); case DOUBLE: return ((Double)o).toString(); case BYTEARRAY: return ((DataByteArray)o).toString(); case CHARARRAY: return (String)o; case NULL: return null; case MAP: case TUPLE: case BAG: case UNKNOWN: default: int errCode = 1071; String msg = "Cannot convert a " + findTypeName(o) + " to an String"; throw new ExecException(msg, errCode, PigException.INPUT); } } catch (ExecException ee) { throw ee; } catch (Exception e) { int errCode = 2054; String msg = "Internal error. Could not convert " + o + " to String."; throw new ExecException(msg, errCode, PigException.BUG); } } -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Wed Apr 01 20:51:36 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 70624 invoked from network); 1 Apr 2009 20:51:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 20:51:36 -0000 Received: (qmail 53054 invoked by uid 500); 1 Apr 2009 20:51:35 -0000 Delivered-To: vshelton@example.com Received: (qmail 53020 invoked by uid 500); 1 Apr 2009 20:51:35 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 53010 invoked by uid 99); 1 Apr 2009 20:51:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 20:51:35 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 20:51:34 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id EE3B3234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 13:51:12 -0700 (PDT) Message-ID: <1840530469.1238619072961.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 13:51:12 -0700 (PDT) From: "Alan Gates (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Assigned: (PIG-741) Add LIMIT as a statement that works in nested FOREACH In-Reply-To: <1259159495.1238482310689.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned PIG-741: ------------------------------ Assignee: Alan Gates > Add LIMIT as a statement that works in nested FOREACH > ----------------------------------------------------- > > Key: PIG-741 > URL: https://issues.apache.org/jira/browse/PIG-741 > Project: Pig > Issue Type: New Feature > Reporter: David Ciemiewicz > Assignee: Alan Gates > > I'd like to compute the top 10 results in each group. > The natural way to express this in Pig would be: > {code} > A = load '...' using PigStorage() as ( > date: int, > count: int, > url: chararray > ); > B = group A by ( date ); > C = foreach B { > D = order A by count desc; > E = limit D 10; > generate > FLATTEN(E); > }; > dump C; > {code} > Yeah, I could write a UDF / PiggyBank function to take the top n results. But since LIMIT already exists as a statement, it seems like it should also work in the nested foreach context. > Example workaround code. > {code} > C = foreach B { > D = order A by count desc; > E = util.TOP(D, 10); > generate > FLATTEN(E); > }; > dump C; > {code} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Wed Apr 01 23:33:36 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 40100 invoked from network); 1 Apr 2009 23:33:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 23:33:36 -0000 Received: (qmail 90352 invoked by uid 500); 1 Apr 2009 23:33:36 -0000 Delivered-To: vshelton@example.com Received: (qmail 90276 invoked by uid 500); 1 Apr 2009 23:33:36 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 90266 invoked by uid 99); 1 Apr 2009 23:33:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 23:33:36 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 23:33:34 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id DE935234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 16:33:12 -0700 (PDT) Message-ID: <1790826171.1238628792897.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 16:33:12 -0700 (PDT) From: "Pradeep Kamath (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Updated: (PIG-733) Order by sampling dumps entire sample to hdfs which causes dfs "FileSystem closed" error on large input In-Reply-To: <1613399651.1238007958419.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-733?page=3Dcom.atlassian.j= ira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath updated PIG-733: ------------------------------- Fix Version/s: 0.3.0 Affects Version/s: 0.2.0 Status: Patch Available (was: Open) Attached patch which implements the fix described in the description of the= issue > Order by sampling dumps entire sample to hdfs which causes dfs "FileSyste= m closed" error on large input > -------------------------------------------------------------------------= ------------------------------ > > Key: PIG-733 > URL: https://issues.apache.org/jira/browse/PIG-733 > Project: Pig > Issue Type: Bug > Affects Versions: 0.2.0 > Reporter: Pradeep Kamath > Assignee: Pradeep Kamath > Fix For: 0.3.0 > > > Order by has a sampling job which samples the input and creates a sorted = list of sample items. CUrrently the number of items sampled is 100 per map = task. So if the input is large resulting in many maps (say 50,000) the samp= le is big. This sorted sample is stored on dfs. The WeightedRangePartitione= r computes quantile boundaries and weighted probabilities for repeating val= ues in each map by reading the samples file from DFS. In queries with many = maps (in the order of 50,000) the dfs read of the sample file fails with "F= ileSystem closed" error. This seems to point to a dfs issue wherein a big d= fs file being read simultaneously by many dfs clients (in this case all map= s) causes the clients to be closed. However on the pig side, loading the sa= mple from each map in the final map reduce job and computing the quantile b= oundaries and weighted probabilities is inefficient. We should do this comp= utation through a FindQuantiles udf in the same map reduce job which produc= es the sorted samples. This way lesser data is written to dfs and in the fi= nal map reduce job, the weightedRangePartitioner needs to just load the com= puted information. --=20 This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Wed Apr 01 23:35:35 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 40285 invoked from network); 1 Apr 2009 23:35:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2009 23:35:34 -0000 Received: (qmail 95149 invoked by uid 500); 1 Apr 2009 23:35:34 -0000 Delivered-To: vshelton@example.com Received: (qmail 95107 invoked by uid 500); 1 Apr 2009 23:35:34 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 95079 invoked by uid 99); 1 Apr 2009 23:35:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 23:35:33 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Apr 2009 23:35:33 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id E79DC234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 16:35:12 -0700 (PDT) Message-ID: <1486628507.1238628912933.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 16:35:12 -0700 (PDT) From: "Pradeep Kamath (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Updated: (PIG-733) Order by sampling dumps entire sample to hdfs which causes dfs "FileSystem closed" error on large input In-Reply-To: <1613399651.1238007958419.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-733?page=3Dcom.atlassian.j= ira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath updated PIG-733: ------------------------------- Attachment: PIG-733.patch > Order by sampling dumps entire sample to hdfs which causes dfs "FileSyste= m closed" error on large input > -------------------------------------------------------------------------= ------------------------------ > > Key: PIG-733 > URL: https://issues.apache.org/jira/browse/PIG-733 > Project: Pig > Issue Type: Bug > Affects Versions: 0.2.0 > Reporter: Pradeep Kamath > Assignee: Pradeep Kamath > Fix For: 0.3.0 > > Attachments: PIG-733.patch > > > Order by has a sampling job which samples the input and creates a sorted = list of sample items. CUrrently the number of items sampled is 100 per map = task. So if the input is large resulting in many maps (say 50,000) the samp= le is big. This sorted sample is stored on dfs. The WeightedRangePartitione= r computes quantile boundaries and weighted probabilities for repeating val= ues in each map by reading the samples file from DFS. In queries with many = maps (in the order of 50,000) the dfs read of the sample file fails with "F= ileSystem closed" error. This seems to point to a dfs issue wherein a big d= fs file being read simultaneously by many dfs clients (in this case all map= s) causes the clients to be closed. However on the pig side, loading the sa= mple from each map in the final map reduce job and computing the quantile b= oundaries and weighted probabilities is inefficient. We should do this comp= utation through a FindQuantiles udf in the same map reduce job which produc= es the sorted samples. This way lesser data is written to dfs and in the fi= nal map reduce job, the weightedRangePartitioner needs to just load the com= puted information. --=20 This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Thu Apr 02 01:13:38 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 69305 invoked from network); 2 Apr 2009 01:13:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2009 01:13:38 -0000 Received: (qmail 65431 invoked by uid 500); 2 Apr 2009 01:13:38 -0000 Delivered-To: vshelton@example.com Received: (qmail 65379 invoked by uid 500); 2 Apr 2009 01:13:38 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 65369 invoked by uid 99); 2 Apr 2009 01:13:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 01:13:38 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 01:13:34 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 46CB0234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 18:13:13 -0700 (PDT) Message-ID: <220-228-4882.1238634793274.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 18:13:13 -0700 (PDT) From: "Hadoop QA (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Commented: (PIG-733) Order by sampling dumps entire sample to hdfs which causes dfs "FileSystem closed" error on large input In-Reply-To: <1613399651.1238007958419.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-733?page=3Dcom.atlassian.ji= ra.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1269485= 5#action_12694855 ]=20 Hadoop QA commented on PIG-733: ------------------------------- -1 overall. Here are the results of testing the latest attachment=20 http://issues.apache.org/jira/secure/attachment/12404402/PIG-733.patch against trunk revision 759376. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modi= fied tests. Please justify why no tests are needed for this pat= ch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 207 javac compiler warnings (mor= e than the trunk's current 200 warnings). -1 findbugs. The patch appears to introduce 5 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number= of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva.a= pache.org/18/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-mine= rva.apache.org/18/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnin= gs.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-minerva= .apache.org/18/console This message is automatically generated. > Order by sampling dumps entire sample to hdfs which causes dfs "FileSyste= m closed" error on large input > -------------------------------------------------------------------------= ------------------------------ > > Key: PIG-733 > URL: https://issues.apache.org/jira/browse/PIG-733 > Project: Pig > Issue Type: Bug > Affects Versions: 0.2.0 > Reporter: Pradeep Kamath > Assignee: Pradeep Kamath > Fix For: 0.3.0 > > Attachments: PIG-733.patch > > > Order by has a sampling job which samples the input and creates a sorted = list of sample items. CUrrently the number of items sampled is 100 per map = task. So if the input is large resulting in many maps (say 50,000) the samp= le is big. This sorted sample is stored on dfs. The WeightedRangePartitione= r computes quantile boundaries and weighted probabilities for repeating val= ues in each map by reading the samples file from DFS. In queries with many = maps (in the order of 50,000) the dfs read of the sample file fails with "F= ileSystem closed" error. This seems to point to a dfs issue wherein a big d= fs file being read simultaneously by many dfs clients (in this case all map= s) causes the clients to be closed. However on the pig side, loading the sa= mple from each map in the final map reduce job and computing the quantile b= oundaries and weighted probabilities is inefficient. We should do this comp= utation through a FindQuantiles udf in the same map reduce job which produc= es the sorted samples. This way lesser data is written to dfs and in the fi= nal map reduce job, the weightedRangePartitioner needs to just load the com= puted information. --=20 This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Thu Apr 02 01:39:34 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 80700 invoked from network); 2 Apr 2009 01:39:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2009 01:39:34 -0000 Received: (qmail 86019 invoked by uid 500); 2 Apr 2009 01:39:33 -0000 Delivered-To: vshelton@example.com Received: (qmail 85983 invoked by uid 500); 2 Apr 2009 01:39:33 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 85973 invoked by uid 99); 2 Apr 2009 01:39:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 01:39:33 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 01:39:33 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id E64FB234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 18:39:12 -0700 (PDT) Message-ID: <1036032268.1238636352938.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 18:39:12 -0700 (PDT) From: "Pradeep Kamath (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Commented: (PIG-627) PERFORMANCE: multi-query optimization In-Reply-To: <1915989148.1232498399494.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694859#action_12694859 ] Pradeep Kamath commented on PIG-627: ------------------------------------ +1, patch committed - thanks for the contribution Gunther. > PERFORMANCE: multi-query optimization > ------------------------------------- > > Key: PIG-627 > URL: https://issues.apache.org/jira/browse/PIG-627 > Project: Pig > Issue Type: Improvement > Affects Versions: 0.2.0 > Reporter: Olga Natkovich > Attachments: file_cmds-0305.patch, fix_store_prob.patch, merge_741727_HEAD__0324.patch, merge_741727_HEAD__0324_2.patch, multi-store-0303.patch, multi-store-0304.patch, multiquery-phase2_0313.patch, multiquery-phase2_0323.patch, multiquery_0223.patch, multiquery_0224.patch, multiquery_0306.patch, multiquery_explain_fix.patch, noop_filter_absolute_path_flag.patch, noop_filter_absolute_path_flag_0401.patch > > > Currently, if your Pig script contains multiple stores and some shared computation, Pig will execute several independent queries. For instance: > A = load 'data' as (a, b, c); > B = filter A by a > 5; > store B into 'output1'; > C = group B by b; > store C into 'output2'; > This script will result in map-only job that generated output1 followed by a map-reduce job that generated output2. As the resuld data is read, parsed and filetered twice which is unnecessary and costly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Thu Apr 02 05:36:34 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 45419 invoked from network); 2 Apr 2009 05:36:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2009 05:36:34 -0000 Received: (qmail 10253 invoked by uid 500); 2 Apr 2009 05:36:34 -0000 Delivered-To: vshelton@example.com Received: (qmail 10207 invoked by uid 500); 2 Apr 2009 05:36:34 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 10197 invoked by uid 99); 2 Apr 2009 05:36:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 05:36:34 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 05:36:33 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id EFD96234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 22:36:12 -0700 (PDT) Message-ID: <1271977030.1238650572977.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 22:36:12 -0700 (PDT) From: "Giridharan Kesavan (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Assigned: (PIG-743) to implement clover In-Reply-To: <213413569.1238584872897.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan reassigned PIG-743: -------------------------------------- Assignee: Giridharan Kesavan > to implement clover > ------------------- > > Key: PIG-743 > URL: https://issues.apache.org/jira/browse/PIG-743 > Project: Pig > Issue Type: Improvement > Components: build > Reporter: Giridharan Kesavan > Assignee: Giridharan Kesavan > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Thu Apr 02 06:02:34 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 60986 invoked from network); 2 Apr 2009 06:02:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2009 06:02:34 -0000 Received: (qmail 21708 invoked by uid 500); 2 Apr 2009 06:02:34 -0000 Delivered-To: vshelton@example.com Received: (qmail 21652 invoked by uid 500); 2 Apr 2009 06:02:34 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 21641 invoked by uid 99); 2 Apr 2009 06:02:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 06:02:34 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 06:02:33 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 54F74234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 23:02:13 -0700 (PDT) Message-ID: <500987550.1238652133333.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 23:02:13 -0700 (PDT) From: "David Ciemiewicz (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Created: (PIG-746) Works in --exectype local, fails on grid - ERROR 2113: SingleTupleBag should never be serialized MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org Works in --exectype local, fails on grid - ERROR 2113: SingleTupleBag should never be serialized ------------------------------------------------------------------------------------------------ Key: PIG-746 URL: https://issues.apache.org/jira/browse/PIG-746 Project: Pig Issue Type: Bug Reporter: David Ciemiewicz The script below works on Pig 2.0 local mode but fails when I run the same program on the grid. I was attempting to create a workaround for PIG-710. Here's the error: {code} Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2113: SingleTupleBag should never be serialized or serialized. at org.apache.pig.data.SingleTupleBag.write(SingleTupleBag.java:129) at org.apache.pig.data.DataReaderWriter.writeDatum(DataReaderWriter.java:147) at org.apache.pig.data.DefaultTuple.write(DefaultTuple.java:291) at org.apache.pig.impl.io.PigNullableWritable.write(PigNullableWritable.java:83) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:439) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:101) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:219) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:208) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:86) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) {code} Here's the program: {code} A = load 'filterbug.data' using PigStorage() as ( id, str ); A = foreach A generate id, str, ( str matches 'hello' or str matches 'hello' ? 1 : 0 ) as matched; describe A; B = group A by ( id ); describe B; D = foreach B generate group, SUM(A.matched) as matchedcount, A; describe D; E = filter D by matchedcount > 0; describe E; F = foreach E generate FLATTEN(A); describe F; dump F; {code} Here's the data filterbug.data {code} a hello a goodbye b goodbye c hello c hello c hello e what {code} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigvshelton@example.com Thu Apr 02 06:18:36 2009 Return-Path: <pigvshelton@example.com> Delivered-To: vshelton@example.com Received: (qmail 71290 invoked from network); 2 Apr 2009 06:18:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2009 06:18:36 -0000 Received: (qmail 30814 invoked by uid 500); 2 Apr 2009 06:18:36 -0000 Delivered-To: vshelton@example.com Received: (qmail 30780 invoked by uid 500); 2 Apr 2009 06:18:36 -0000 Mailing-List: contact vshelton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:vshelton@example.com> List-Unsubscribe: <mailto:vshelton@example.com> List-Post: <mailto:vshelton@example.com> List-Id: <pig-dev.hadoop.apache.org> Reply-To: vshelton@example.com Delivered-To: mailing list vshelton@example.com Received: (qmail 30769 invoked by uid 99); 2 Apr 2009 06:18:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 06:18:36 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Apr 2009 06:18:34 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id E0784234C003 for <vshelton@example.com>; Wed, 1 Apr 2009 23:18:12 -0700 (PDT) Message-ID: <1344770844.1238653092903.JavaMail.jira@brutus> Date: Wed, 1 Apr 2009 23:18:12 -0700 (PDT) From: "David Ciemiewicz (JIRA)" <vshelton@example.com> To: vshelton@example.com Subject: [jira] Commented: (PIG-745) Please add DataTypes.toString() conversion function In-Reply-To: <1656160341.1238618712954.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694893#action_12694893 ] David Ciemiewicz commented on PIG-745: -------------------------------------- Actually, the proposed function DataTypes.toString() is the following: {code} public static String toString(Object o) throws ExecException { try { switch (findType(o)) { case BOOLEAN: if (((Boolean)o) == true) return new String('1'); else return new String('0'); case BYTE: return ((Byte)o).toString(); case INTEGER: return ((Integer)o).toString(); case LONG: return ((Long)o).toString(); case FLOAT: return ((Float)o).toString(); case DOUBLE: return ((Double)o).toString(); case BYTEARRAY: return ((DataByteArray)o).toString(); case CHARARRAY: return (String)o; case NULL: return null; case MAP: case TUPLE: case BAG: case UNKNOWN: default: int errCode = 1071; String msg = "Cannot convert a " + findTypeName(o) + " to an String"; throw new ExecException(msg, errCode, PigException.INPUT); } } catch (ExecException ee) { throw ee; } catch (Exception e) { int errCode = 2054; String msg = "Internal error. Could not
{'source': 'Apache Software Foundation Public Mail Archives'}