From rachel17@example.net Tue Feb 01 01:09:21 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 61677 invoked from network); 1 Feb 2011 01:09:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 01:09:21 -0000 Received: (qmail 36969 invoked by uid 500); 1 Feb 2011 01:09:21 -0000 Delivered-To: rachel17@example.net Received: (qmail 36812 invoked by uid 500); 1 Feb 2011 01:09:20 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 36804 invoked by uid 500); 1 Feb 2011 01:09:20 -0000 Delivered-To: rachel17@example.net Received: (qmail 36801 invoked by uid 99); 1 Feb 2011 01:09:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 01:09:20 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 01:09:19 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 1B8A51843B4 for <rachel17@example.net>; Tue, 1 Feb 2011 01:08:29 +0000 (UTC) Date: Tue, 1 Feb 2011 01:08:29 +0000 (UTC) From: "Xuefu Zhang (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> Subject: [jira] Updated: (PIG-1618) Switch to new parser generator technology MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated PIG-1618: ----------------------------- Attachment: NewParser-15.patch Includes: 1. Scalar support 2. Minor parser grammar fixes 3. Better schema validation and more test case. > Switch to new parser generator technology > ----------------------------------------- > > Key: PIG-1618 > URL: https://issues.apache.org/jira/browse/PIG-1618 > Project: Pig > Issue Type: Improvement > Affects Versions: 0.8.0 > Reporter: Alan Gates > Assignee: Xuefu Zhang > Fix For: 0.9.0 > > Attachments: NewParser-1.patch, NewParser-10.patch, NewParser-11.patch, NewParser-12.patch, NewParser-13.2.patch, NewParser-13.patch, NewParser-14.patch, NewParser-15.patch, NewParser-2.patch, NewParser-3.patch, NewParser-3.patch, NewParser-4.patch, NewParser-5.patch, NewParser-6.patch, NewParser-7.patch, NewParser-8.patches, NewParser-9.patch, antlr-3.2.jar, javadoc.patch > > > There are many bugs in Pig related to the parser, particularly to bad error messages. After review of Java CC we feel these will be difficult to address using that tool. Also, the .jjt files used by JavaCC are hard to understand and maintain. > ANTLR is being reviewed as the most likely choice to move to, but other parsers will be reviewed as well. > This JIRA will act as an umbrella issue for other parser issues. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 02:40:53 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 1614 invoked from network); 1 Feb 2011 02:40:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 02:40:53 -0000 Received: (qmail 4115 invoked by uid 500); 1 Feb 2011 02:40:53 -0000 Delivered-To: rachel17@example.net Received: (qmail 3912 invoked by uid 500); 1 Feb 2011 02:40:52 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 3904 invoked by uid 500); 1 Feb 2011 02:40:51 -0000 Delivered-To: rachel17@example.net Received: (qmail 3901 invoked by uid 99); 1 Feb 2011 02:40:51 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 02:40:51 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 02:40:50 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 2A82E1831BB for <rachel17@example.net>; Tue, 1 Feb 2011 02:40:30 +0000 (UTC) Date: Tue, 1 Feb 2011 02:40:30 +0000 (UTC) From: "Daniel Dai (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> In-Reply-To: <14809532.225091296076843827.JavaMail.jira@thor> Subject: [jira] Commented: (PIG-1826) Unexpected data type -1 found in stream error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989014#comment-12989014 ] Daniel Dai commented on PIG-1826: --------------------------------- What is MutableInt in UDF? What is your input file? > Unexpected data type -1 found in stream error > --------------------------------------------- > > Key: PIG-1826 > URL: https://issues.apache.org/jira/browse/PIG-1826 > Project: Pig > Issue Type: Bug > Affects Versions: 0.8.0 > Environment: This is pig 0.8.0 on a linux box > Reporter: Jonathan Coveney > Attachments: numgraph.java > > > When running the attached udf I get the title error. By inserting printlns extensively, the script is functioning properly and returning a DataBag, but for whatever reason, pig does not detect it as such. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 09:44:25 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 83866 invoked from network); 1 Feb 2011 09:44:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 09:44:25 -0000 Received: (qmail 23902 invoked by uid 500); 1 Feb 2011 09:44:25 -0000 Delivered-To: rachel17@example.net Received: (qmail 23692 invoked by uid 500); 1 Feb 2011 09:44:22 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 23684 invoked by uid 500); 1 Feb 2011 09:44:21 -0000 Delivered-To: rachel17@example.net Received: (qmail 23681 invoked by uid 99); 1 Feb 2011 09:44:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 09:44:20 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 09:44:20 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id B3E521842F3 for <rachel17@example.net>; Tue, 1 Feb 2011 09:43:29 +0000 (UTC) Date: Tue, 1 Feb 2011 09:43:29 +0000 (UTC) From: "Vivek Padmanabhan (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> Subject: [jira] Created: (PIG-1835) Pig 0.9 new logical plan throws class cast exception MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Pig 0.9 new logical plan throws class cast exception ---------------------------------------------------- Key: PIG-1835 URL: https://issues.apache.org/jira/browse/PIG-1835 Project: Pig Issue Type: Bug Affects Versions: 0.9.0 Reporter: Vivek Padmanabhan I have the below script which is throwing class cast exception while doing SUM. Even though all the fields are properly typed, while computing sum in m_agg0 and m_agg02 the record from tuple is coming as java.lang.Long instead of Double. The problem is happening in Pig 0.9. It works fine with 0.9 if I flag off new logical plan by -Dpig.usenewlogicalplan=false. {code} A0 = load 'inputA' using PigStorage('\t') as ( group_id, r_id:long, is_phase2:int, roi_value:double,roi_cost:double,ecpm, prob:double,pixel_id, pixel_type, val:long,f3, f4,type:long, amount:double,item_id:long); A0 = foreach A0 generate r_id, is_phase2, ((val==257 or val==258)? 1: 0) as imps, ((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id, ((val==257 or val==258)? roi_value: 0.0) as roi_value,((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or val==513)? ecpm: 0.0) as ecpm, ((val==257 or val==513)? prob: 0.0) as prob, ((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id,((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------- B0 = load 'inputB' using PigStorage('\t') as ( group_id:long, r_id:long, roi_value:double,roi_cost:double,receive_time, host_name,site_id,rm_has_cookies,rm_pearl_id, f1,f2,pixel_id:long,pixel_type:int, xcookie,val:long,f3, f4,type:long,amount:double,item_id:long); B0 = foreach B0 generate r_id, ((val==257 or val==258)? 1: 0) as B0,((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or val==258)? roi_value: 0.0) as roi_value, ((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------ C0 = load 'inputC' using PigStorage('\t') as ( group_id:long, r_id:long, roi_value:double, roi_cost:double, receive_time:long, host_name:chararray, site_id:long, rm_has_cookies:int,rm_pearl_id:long,f1,f2, pixel_id:long, pixel_type:int,rm_is_post_click:int, rm_conversion_id,xcookie:chararray,val:long,f3:long,f4:long,type:long,amount:double,item_id:long); C0 = foreach C0 generate r_id,((val==257 or val==258)? 1: 0) as C0, ((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------ m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ; m_agg01 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) as conv_pub_rev; store m_agg01 into 'out1' USING PigStorage(','); m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ; m_agg02 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) as conv_pub_rev; store m_agg02 into 'out2' USING PigStorage(','); {code} The below are the inputs to the script (all single record and tab seperated) inputA ------ 1111 1111 1 1.1 1.1 1.1 1.1 1111 1 1111 1111 1111 1111 1.1 1111 inputB ------ 1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1 1 1111 1 c1 1111 1111 1111 1111 1.1 1111 inputC ------ 1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1 1 1111 1 1 1111 c1 1111 1111 1111 1111 1.1 1111 Exception from reducers _______________________ org.apache.pig.backend.executionengine.ExecException: ERROR 2103: Problem while computing sum of doubles. at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:147) at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:46) at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:41) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:230) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:302) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.getNext(POBinCond.java:140) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:446) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:346) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:289) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:455) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:423) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:403) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:258) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) at org.apache.hadoop.mapred.Child.main(Child.java:236) Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to rachel17@example.net(DoubleSum.java:140) ... 20 more -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 09:48:23 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 84439 invoked from network); 1 Feb 2011 09:48:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 09:48:23 -0000 Received: (qmail 29127 invoked by uid 500); 1 Feb 2011 09:48:23 -0000 Delivered-To: rachel17@example.net Received: (qmail 28272 invoked by uid 500); 1 Feb 2011 09:48:21 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 28263 invoked by uid 500); 1 Feb 2011 09:48:20 -0000 Delivered-To: rachel17@example.net Received: (qmail 28257 invoked by uid 99); 1 Feb 2011 09:48:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 09:48:20 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 09:48:19 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 09133184636 for <rachel17@example.net>; Tue, 1 Feb 2011 09:47:29 +0000 (UTC) Date: Tue, 1 Feb 2011 09:47:29 +0000 (UTC) From: "Vivek Padmanabhan (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> In-Reply-To: <rachel17@example.net> Subject: [jira] Updated: (PIG-1835) Pig 0.9 new logical plan throws class cast exception MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vivek Padmanabhan updated PIG-1835: ----------------------------------- Description: I have the below script which is throwing class cast exception while doing SUM. Even though all the fields are properly typed, while computing sum in m_agg0 and m_agg02 the record from tuple is coming as java.lang.Long instead of Double. The problem is happening in Pig 0.9. It works fine with 0.9 if I flag off new logical plan by -Dpig.usenewlogicalplan=false. {code} A0 = load 'inputA' using PigStorage('\t') as ( group_id, r_id:long, is_phase2:int, roi_value:double,roi_cost:double,ecpm, prob:double,pixel_id, pixel_type, val:long,f3, f4,type:long, amount:double,item_id:long); A0 = foreach A0 generate r_id, is_phase2, ((val==257 or val==258)? 1: 0) as imps, ((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id, ((val==257 or val==258)? roi_value: 0.0) as roi_value,((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or val==513)? ecpm: 0.0) as ecpm, ((val==257 or val==513)? prob: 0.0) as prob, ((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id,((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------- B0 = load 'inputB' using PigStorage('\t') as ( group_id:long, r_id:long, roi_value:double,roi_cost:double,receive_time, host_name,site_id,rm_has_cookies,rm_pearl_id, f1,f2,pixel_id:long,pixel_type:int, xcookie,val:long,f3, f4,type:long,amount:double,item_id:long); B0 = foreach B0 generate r_id, ((val==257 or val==258)? 1: 0) as B0,((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or val==258)? roi_value: 0.0) as roi_value, ((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------ C0 = load 'inputC' using PigStorage('\t') as ( group_id:long, r_id:long, roi_value:double, roi_cost:double, receive_time:long, host_name:chararray, site_id:long, rm_has_cookies:int,rm_pearl_id:long,f1,f2, pixel_id:long, pixel_type:int,rm_is_post_click:int, rm_conversion_id,xcookie:chararray,val:long,f3:long,f4:long,type:long,amount:double,item_id:long); C0 = foreach C0 generate r_id,((val==257 or val==258)? 1: 0) as C0, ((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------ m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ; m_agg01 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) as conv_pub_rev; store m_agg01 into 'out1' USING PigStorage(','); m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ; m_agg02 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) as conv_pub_rev; store m_agg02 into 'out2' USING PigStorage(','); {code} The below are the inputs to the script (all single record and tab seperated) {code} inputA ------ 1111 1111 1 1.1 1.1 1.1 1.1 1111 1 1111 1111 1111 1111 1.1 1111 inputB ------ 1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1 1 1111 1 c1 1111 1111 1111 1111 1.1 1111 inputC ------ 1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1 1 1111 1 1 1111 c1 1111 1111 1111 1111 1.1 1111 {code} Exception from reducers _______________________ org.apache.pig.backend.executionengine.ExecException: ERROR 2103: Problem while computing sum of doubles. at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:147) at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:46) at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:41) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:230) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:302) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.getNext(POBinCond.java:140) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:446) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:346) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:289) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:455) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:423) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:403) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:258) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) at org.apache.hadoop.mapred.Child.main(Child.java:236) Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to rachel17@example.net(DoubleSum.java:140) ... 20 more was: I have the below script which is throwing class cast exception while doing SUM. Even though all the fields are properly typed, while computing sum in m_agg0 and m_agg02 the record from tuple is coming as java.lang.Long instead of Double. The problem is happening in Pig 0.9. It works fine with 0.9 if I flag off new logical plan by -Dpig.usenewlogicalplan=false. {code} A0 = load 'inputA' using PigStorage('\t') as ( group_id, r_id:long, is_phase2:int, roi_value:double,roi_cost:double,ecpm, prob:double,pixel_id, pixel_type, val:long,f3, f4,type:long, amount:double,item_id:long); A0 = foreach A0 generate r_id, is_phase2, ((val==257 or val==258)? 1: 0) as imps, ((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id, ((val==257 or val==258)? roi_value: 0.0) as roi_value,((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or val==513)? ecpm: 0.0) as ecpm, ((val==257 or val==513)? prob: 0.0) as prob, ((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id,((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------- B0 = load 'inputB' using PigStorage('\t') as ( group_id:long, r_id:long, roi_value:double,roi_cost:double,receive_time, host_name,site_id,rm_has_cookies,rm_pearl_id, f1,f2,pixel_id:long,pixel_type:int, xcookie,val:long,f3, f4,type:long,amount:double,item_id:long); B0 = foreach B0 generate r_id, ((val==257 or val==258)? 1: 0) as B0,((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or val==258)? roi_value: 0.0) as roi_value, ((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------ C0 = load 'inputC' using PigStorage('\t') as ( group_id:long, r_id:long, roi_value:double, roi_cost:double, receive_time:long, host_name:chararray, site_id:long, rm_has_cookies:int,rm_pearl_id:long,f1,f2, pixel_id:long, pixel_type:int,rm_is_post_click:int, rm_conversion_id,xcookie:chararray,val:long,f3:long,f4:long,type:long,amount:double,item_id:long); C0 = foreach C0 generate r_id,((val==257 or val==258)? 1: 0) as C0, ((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or val==513)? type: 0) as pub_pt; ------------------------------------------------------------------------------------------------------------------------------------------------ m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ; m_agg01 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) as conv_pub_rev; store m_agg01 into 'out1' USING PigStorage(','); m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ; m_agg02 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) as conv_pub_rev; store m_agg02 into 'out2' USING PigStorage(','); {code} The below are the inputs to the script (all single record and tab seperated) inputA ------ 1111 1111 1 1.1 1.1 1.1 1.1 1111 1 1111 1111 1111 1111 1.1 1111 inputB ------ 1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1 1 1111 1 c1 1111 1111 1111 1111 1.1 1111 inputC ------ 1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1 1 1111 1 1 1111 c1 1111 1111 1111 1111 1.1 1111 Exception from reducers _______________________ org.apache.pig.backend.executionengine.ExecException: ERROR 2103: Problem while computing sum of doubles. at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:147) at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:46) at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:41) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:230) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:302) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.getNext(POBinCond.java:140) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:446) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:346) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:289) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:455) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:423) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:403) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:258) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414) at org.apache.hadoop.mapred.Child$4.run(Child.java:242) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) at org.apache.hadoop.mapred.Child.main(Child.java:236) Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to rachel17@example.net(DoubleSum.java:140) ... 20 more > Pig 0.9 new logical plan throws class cast exception > ---------------------------------------------------- > > Key: PIG-1835 > URL: https://issues.apache.org/jira/browse/PIG-1835 > Project: Pig > Issue Type: Bug > Affects Versions: 0.9.0 > Reporter: Vivek Padmanabhan > > I have the below script which is throwing class cast exception while doing SUM. Even though all the fields are properly typed, while computing sum in m_agg0 and m_agg02 the record from tuple is coming as java.lang.Long instead of Double. > The problem is happening in Pig 0.9. It works fine with 0.9 if I flag off new logical plan by -Dpig.usenewlogicalplan=false. > {code} > A0 = load 'inputA' using PigStorage('\t') as ( group_id, r_id:long, is_phase2:int, roi_value:double,roi_cost:double,ecpm, prob:double,pixel_id, pixel_type, > val:long,f3, f4,type:long, amount:double,item_id:long); > A0 = foreach A0 generate r_id, is_phase2, ((val==257 or val==258)? 1: 0) as imps, > ((val==257 or val==258)? amount: 0.0) as a_out, ((val==257 or val==258)? item_id: 0) as a_item_id, > ((val==257 or val==258)? roi_value: 0.0) as roi_value,((val==257 or val==258)? roi_cost: 0.0) as roi_cost, > ((val==257 or val==513)? ecpm: 0.0) as ecpm, ((val==257 or val==513)? prob: 0.0) as prob, > ((val==257 or val==513)? amount: 0.0) as pub_rev, ((val==257 or val==513)? item_id: 0) as pub_line_id,((val==257 or val==513)? type: 0) as pub_pt; > ------------------------------------------------------------------------------------------------------------------------------------------------- > B0 = load 'inputB' using PigStorage('\t') as ( group_id:long, r_id:long, roi_value:double,roi_cost:double,receive_time, host_name,site_id,rm_has_cookies,rm_pearl_id, f1,f2,pixel_id:long,pixel_type:int, xcookie,val:long,f3, f4,type:long,amount:double,item_id:long); > B0 = foreach B0 generate r_id, ((val==257 or val==258)? 1: 0) as B0,((val==257 or val==258)? amount: 0.0) as a_out, > ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or val==258)? roi_value: 0.0) as roi_value, > ((val==257 or val==258)? roi_cost: 0.0) as roi_cost, ((val==257 or val==513)? amount: 0.0) as pub_rev, > ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or val==513)? type: 0) as pub_pt; > ------------------------------------------------------------------------------------------------------------------------------------------------ > C0 = load 'inputC' using PigStorage('\t') as ( group_id:long, r_id:long, roi_value:double, roi_cost:double, receive_time:long, host_name:chararray, site_id:long, rm_has_cookies:int,rm_pearl_id:long,f1,f2, pixel_id:long, pixel_type:int,rm_is_post_click:int, rm_conversion_id,xcookie:chararray,val:long,f3:long,f4:long,type:long,amount:double,item_id:long); > C0 = foreach C0 generate r_id,((val==257 or val==258)? 1: 0) as C0, ((val==257 or val==258)? amount: 0.0) as a_out, > ((val==257 or val==258)? item_id: 0) as a_item_id,((val==257 or val==513)? amount: 0.0) as pub_rev, > ((val==257 or val==513)? item_id: 0) as pub_line_id, ((val==257 or val==513)? type: 0) as pub_pt; > ------------------------------------------------------------------------------------------------------------------------------------------------ > m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ; > m_agg01 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) as conv_pub_rev; > store m_agg01 into 'out1' USING PigStorage(','); > m_all = cogroup A0 by (r_id) outer, B0 by (r_id) outer, C0 by (r_id) outer ; > m_agg02 = foreach m_all generate (double)(IsEmpty(C0) ? 0.0 : SUM(C0.pub_rev)) as conv_pub_rev; > store m_agg02 into 'out2' USING PigStorage(','); > {code} > The below are the inputs to the script (all single record and tab seperated) > {code} > inputA > ------ > 1111 1111 1 1.1 1.1 1.1 1.1 1111 1 1111 1111 1111 1111 1.1 1111 > inputB > ------ > 1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1 1 1111 1 c1 1111 1111 1111 1111 1.1 1111 > inputC > ------ > 1111 1111 1.1 1.1 1111 a1 1111 1 1111 b1 1 1111 1 1 1111 c1 1111 1111 1111 1111 1.1 1111 > {code} > Exception from reducers > _______________________ > org.apache.pig.backend.executionengine.ExecException: ERROR 2103: Problem while computing sum of doubles. > at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:147) > at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:46) > at org.apache.pig.builtin.DoubleSum.exec(DoubleSum.java:41) > at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:230) > at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:302) > at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.getNext(POBinCond.java:140) > at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:446) > at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:346) > at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:289) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.runPipeline(PigMapReduce.java:455) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.processOnePackageOutput(PigMapReduce.java:423) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:403) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Reduce.reduce(PigMapReduce.java:258) > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176) > at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:572) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414) > at org.apache.hadoop.mapred.Child$4.run(Child.java:242) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062) > at org.apache.hadoop.mapred.Child.main(Child.java:236) > Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Double > at org.apache.pig.builtin.DoubleSum.sum(DoubleSum.java:140) > ... 20 more -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 10:52:41 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 17507 invoked from network); 1 Feb 2011 10:52:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 10:52:41 -0000 Received: (qmail 96436 invoked by uid 500); 1 Feb 2011 10:52:41 -0000 Delivered-To: rachel17@example.net Received: (qmail 96211 invoked by uid 500); 1 Feb 2011 10:52:38 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 96203 invoked by uid 99); 1 Feb 2011 10:52:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 10:52:37 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO aegis.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 10:52:36 +0000 Received: from aegis (localhost [83.161.120.155]) by aegis.apache.org (Postfix) with ESMTP id 84424C0013 for <rachel17@example.net>; Tue, 1 Feb 2011 10:52:16 +0000 (UTC) Date: Tue, 1 Feb 2011 10:52:16 +0000 (UTC) From: Apache Hudson Server <rachel17@example.net> To: rachel17@example.net Message-ID: <947326006.13231296557536443.JavaMail.hudson@aegis> In-Reply-To: <(865)237-8273.12681296514229325.JavaMail.hudson@aegis> References: <(865)237-8273.12681296514229325.JavaMail.hudson@aegis> Subject: Hudson build is back to normal : Pig-trunk #918 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit See <https://hudson.apache.org/hudson/job/Pig-trunk/918/changes> From rachel17@example.net Tue Feb 01 11:47:25 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 45394 invoked from network); 1 Feb 2011 11:47:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 11:47:24 -0000 Received: (qmail 64317 invoked by uid 500); 1 Feb 2011 11:47:24 -0000 Delivered-To: rachel17@example.net Received: (qmail 63886 invoked by uid 500); 1 Feb 2011 11:47:21 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 63867 invoked by uid 500); 1 Feb 2011 11:47:20 -0000 Delivered-To: rachel17@example.net Received: (qmail 63859 invoked by uid 99); 1 Feb 2011 11:47:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 11:47:19 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 11:47:19 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 10E98184B78 for <rachel17@example.net>; Tue, 1 Feb 2011 11:46:29 +0000 (UTC) Date: Tue, 1 Feb 2011 11:46:29 +0000 (UTC) From: "Gerrit Jansen van Vuuren (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> In-Reply-To: <19550498.27791289490193681.JavaMail.jira@thor> Subject: [jira] Updated: (PIG-1717) pig needs to call setPartitionFilter if schema is null but getPartitionKeys is not MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerrit Jansen van Vuuren updated PIG-1717: ------------------------------------------ Attachment: PIG-1717.v2.patch > pig needs to call setPartitionFilter if schema is null but getPartitionKeys is not > ---------------------------------------------------------------------------------- > > Key: PIG-1717 > URL: https://issues.apache.org/jira/browse/PIG-1717 > Project: Pig > Issue Type: Improvement > Components: impl > Affects Versions: 0.9.0 > Reporter: Gerrit Jansen van Vuuren > Assignee: Gerrit Jansen van Vuuren > Priority: Minor > Fix For: 0.9.0 > > Attachments: PIG-1717.patch, PIG-1717.v1.patch, PIG-1717.v2.patch, patchReleaseAuditWarnings.txt.gz, testlog.tgz, trunkReleaseAuditWarnings.txt.gz > > > I'm writing a loader that works with hive style partitioning e.g. /logs/type1/daydate=2010-11-01 > The loader does not know the schema upfront and this is something that the user adds in the script using the AS clause. > The problem is that this user defined schema is not available to the loader, so the loader cannot return any schema, the Loader does know what the partition keys are and pig needs in some way to know about these partition keys. > Currently if the schema is null pig never calls the LoadMetaData:getPartitionKeys method or the setPartitionFilter method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 11:48:52 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 45663 invoked from network); 1 Feb 2011 11:48:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 11:48:52 -0000 Received: (qmail 66401 invoked by uid 500); 1 Feb 2011 11:48:52 -0000 Delivered-To: rachel17@example.net Received: (qmail 66293 invoked by uid 500); 1 Feb 2011 11:48:50 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 66285 invoked by uid 500); 1 Feb 2011 11:48:49 -0000 Delivered-To: rachel17@example.net Received: (qmail 66282 invoked by uid 99); 1 Feb 2011 11:48:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 11:48:49 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 11:48:49 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id F1DFD184C26 for <rachel17@example.net>; Tue, 1 Feb 2011 11:48:28 +0000 (UTC) Date: Tue, 1 Feb 2011 11:48:28 +0000 (UTC) From: "Gerrit Jansen van Vuuren (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> In-Reply-To: <19550498.27791289490193681.JavaMail.jira@thor> Subject: [jira] Updated: (PIG-1717) pig needs to call setPartitionFilter if schema is null but getPartitionKeys is not MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerrit Jansen van Vuuren updated PIG-1717: ------------------------------------------ Tags: PIG-1717.v2.patch (was: PIG-1717.v1.patch) Status: Patch Available (was: Open) Hi, finally I got to do this. What I did was add in a check in the LOLoad for conf == null, this will allow any current tests that pass the conf as null to succeed. > pig needs to call setPartitionFilter if schema is null but getPartitionKeys is not > ---------------------------------------------------------------------------------- > > Key: PIG-1717 > URL: https://issues.apache.org/jira/browse/PIG-1717 > Project: Pig > Issue Type: Improvement > Components: impl > Affects Versions: 0.9.0 > Reporter: Gerrit Jansen van Vuuren > Assignee: Gerrit Jansen van Vuuren > Priority: Minor > Fix For: 0.9.0 > > Attachments: PIG-1717.patch, PIG-1717.v1.patch, PIG-1717.v2.patch, patchReleaseAuditWarnings.txt.gz, testlog.tgz, trunkReleaseAuditWarnings.txt.gz > > > I'm writing a loader that works with hive style partitioning e.g. /logs/type1/daydate=2010-11-01 > The loader does not know the schema upfront and this is something that the user adds in the script using the AS clause. > The problem is that this user defined schema is not available to the loader, so the loader cannot return any schema, the Loader does know what the partition keys are and pig needs in some way to know about these partition keys. > Currently if the schema is null pig never calls the LoadMetaData:getPartitionKeys method or the setPartitionFilter method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 15:02:25 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 85052 invoked from network); 1 Feb 2011 15:02:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 15:02:25 -0000 Received: (qmail 71930 invoked by uid 500); 1 Feb 2011 15:02:25 -0000 Delivered-To: rachel17@example.net Received: (qmail 71697 invoked by uid 500); 1 Feb 2011 15:02:22 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 71593 invoked by uid 500); 1 Feb 2011 15:02:20 -0000 Delivered-To: rachel17@example.net Received: (qmail 71590 invoked by uid 99); 1 Feb 2011 15:02:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 15:02:20 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 15:02:19 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 1C147187642 for <rachel17@example.net>; Tue, 1 Feb 2011 15:01:29 +0000 (UTC) Date: Tue, 1 Feb 2011 15:01:29 +0000 (UTC) From: "Thomas Memenga (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> Subject: [jira] Commented: (PIG-240) Support launching concurrent Pig jobs from one VM MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989230#comment-12989230 ] Thomas Memenga commented on PIG-240: ------------------------------------ Any news on this? We are trying to trigger a lot of pig-jobs simultaneously from a single jvm and we are recieving "Unable to find clone for op" errors from time to time (pig 0.8.0). It's a pitty PigServer isn't threadsafe, spawning jvms for each script seems to be our only option. > Support launching concurrent Pig jobs from one VM > ------------------------------------------------- > > Key: PIG-240 > URL: https://issues.apache.org/jira/browse/PIG-240 > Project: Pig > Issue Type: Improvement > Components: impl > Reporter: Tom White > Assignee: Jeff Zhang > Attachments: patch_240.txt, pig-240-1.patch, pig-240-2.patch, pig-240.patch > > > For some applications it would be convenient to launch concurrent Pig jobs from a single VM. This is currently not possible since Pig has static mutable state. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 17:07:40 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 73322 invoked from network); 1 Feb 2011 17:07:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 17:07:36 -0000 Received: (qmail 67748 invoked by uid 500); 1 Feb 2011 17:07:36 -0000 Delivered-To: rachel17@example.net Received: (qmail 67555 invoked by uid 500); 1 Feb 2011 17:07:32 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 67424 invoked by uid 500); 1 Feb 2011 17:07:31 -0000 Delivered-To: rachel17@example.net Received: (qmail 67412 invoked by uid 99); 1 Feb 2011 17:07:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 17:07:31 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 17:07:30 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 2EA55187E4B for <rachel17@example.net>; Tue, 1 Feb 2011 17:06:29 +0000 (UTC) Date: Tue, 1 Feb 2011 17:06:29 +0000 (UTC) From: "Xuefu Zhang (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> Subject: [jira] Commented: (PIG-1618) Switch to new parser generator technology MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989280#comment-12989280 ] Xuefu Zhang commented on PIG-1618: ---------------------------------- Test-patch run result for patch NewParser-15.patch: (release warning is due to new files added.) [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 499 release audit warnings (more than the trunk's current 496 warnings). Ant test passed. > Switch to new parser generator technology > ----------------------------------------- > > Key: PIG-1618 > URL: https://issues.apache.org/jira/browse/PIG-1618 > Project: Pig > Issue Type: Improvement > Affects Versions: 0.8.0 > Reporter: Alan Gates > Assignee: Xuefu Zhang > Fix For: 0.9.0 > > Attachments: NewParser-1.patch, NewParser-10.patch, NewParser-11.patch, NewParser-12.patch, NewParser-13.2.patch, NewParser-13.patch, NewParser-14.patch, NewParser-15.patch, NewParser-2.patch, NewParser-3.patch, NewParser-3.patch, NewParser-4.patch, NewParser-5.patch, NewParser-6.patch, NewParser-7.patch, NewParser-8.patches, NewParser-9.patch, antlr-3.2.jar, javadoc.patch > > > There are many bugs in Pig related to the parser, particularly to bad error messages. After review of Java CC we feel these will be difficult to address using that tool. Also, the .jjt files used by JavaCC are hard to understand and maintain. > ANTLR is being reviewed as the most likely choice to move to, but other parsers will be reviewed as well. > This JIRA will act as an umbrella issue for other parser issues. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 18:11:23 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 4734 invoked from network); 1 Feb 2011 18:11:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 18:11:23 -0000 Received: (qmail 8945 invoked by uid 500); 1 Feb 2011 18:11:23 -0000 Delivered-To: rachel17@example.net Received: (qmail 8591 invoked by uid 500); 1 Feb 2011 18:11:20 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 8580 invoked by uid 500); 1 Feb 2011 18:11:20 -0000 Delivered-To: rachel17@example.net Received: (qmail 8577 invoked by uid 99); 1 Feb 2011 18:11:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:11:19 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:11:19 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 1231D186172 for <rachel17@example.net>; Tue, 1 Feb 2011 18:10:29 +0000 (UTC) Date: Tue, 1 Feb 2011 18:10:29 +0000 (UTC) From: "Thomas Memenga (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> Subject: [jira] Commented: (PIG-240) Support launching concurrent Pig jobs from one VM MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989305#comment-12989305 ] Thomas Memenga commented on PIG-240: ------------------------------------ Additional Note: Applied latest patch to 0.8.0 (release), but executing the following script results in nullpointerexception: {code} data = load 'hdfs://host.local:9000/user/hadoop/input/part-r-00000' using PigStorage() as (thed,ts:int,cId,ai,si); input_filtered = foreach data generate ts, ai, si; pipe_4965 = filter input_filtered by ts > 1288043999 and ts < 1288047599; pipe_4965_grouped = group pipe_4965 by (ai, si); pipe_4965_flat = foreach pipe_4965_grouped generate FLATTEN($0) , COUNT($1) , MIN(pipe_4965.ts) as ts_min, MAX(pipe_4965.ts) as ts_max; store pipe_4965_flat INTO '/user/hadoop/output/4965' USING PigStorage(); {code} {code} org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2042: Error in new logical plan. Try -Dpig.usenewlogicalplan=false. at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:309) at org.apache.pig.PigServer.compilePp(PigServer.java:1354) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1196) at org.apache.pig.PigServer.execute(PigServer.java:1190) at org.apache.pig.PigServer.access$100(PigServer.java:128) at org.apache.pig.PigServer$Graph.execute(PigServer.java:1517) at org.apache.pig.PigServer.executeBatchEx(PigServer.java:362) at org.apache.pig.PigServer.executeBatch(PigServer.java:329) at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:112) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90) at org.apache.pig.Main.run(Main.java:510) at org.apache.pig.Main.main(Main.java:107) Caused by: rachel17@example.net(ReverseDependencyOrderWalker.java:70) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50) at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:105) at org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:229) at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75) at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:94) at org.apache.pig.newplan.logical.relational.LOForEach.accept(LOForEach.java:71) at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75) at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50) at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:261) ... 13 more {code} Script works fine with unpatched 0.8.0 > Support launching concurrent Pig jobs from one VM > ------------------------------------------------- > > Key: PIG-240 > URL: https://issues.apache.org/jira/browse/PIG-240 > Project: Pig > Issue Type: Improvement > Components: impl > Reporter: Tom White > Assignee: Jeff Zhang > Attachments: patch_240.txt, pig-240-1.patch, pig-240-2.patch, pig-240.patch > > > For some applications it would be convenient to launch concurrent Pig jobs from a single VM. This is currently not possible since Pig has static mutable state. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 18:52:53 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 26489 invoked from network); 1 Feb 2011 18:52:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 18:52:53 -0000 Received: (qmail 96531 invoked by uid 500); 1 Feb 2011 18:52:53 -0000 Delivered-To: rachel17@example.net Received: (qmail 96357 invoked by uid 500); 1 Feb 2011 18:52:51 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 96349 invoked by uid 500); 1 Feb 2011 18:52:50 -0000 Delivered-To: rachel17@example.net Received: (qmail 96346 invoked by uid 99); 1 Feb 2011 18:52:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:52:49 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:52:49 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id E7F7F187B26 for <rachel17@example.net>; Tue, 1 Feb 2011 18:52:28 +0000 (UTC) Date: Tue, 1 Feb 2011 18:52:28 +0000 (UTC) From: "Alan Gates (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> In-Reply-To: <19550498.27791289490193681.JavaMail.jira@thor> Subject: [jira] Commented: (PIG-1717) pig needs to call setPartitionFilter if schema is null but getPartitionKeys is not MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989327#comment-12989327 ] Alan Gates commented on PIG-1717: --------------------------------- Re-running test-patch and unit tests. > pig needs to call setPartitionFilter if schema is null but getPartitionKeys is not > ---------------------------------------------------------------------------------- > > Key: PIG-1717 > URL: https://issues.apache.org/jira/browse/PIG-1717 > Project: Pig > Issue Type: Improvement > Components: impl > Affects Versions: 0.9.0 > Reporter: Gerrit Jansen van Vuuren > Assignee: Gerrit Jansen van Vuuren > Priority: Minor > Fix For: 0.9.0 > > Attachments: PIG-1717.patch, PIG-1717.v1.patch, PIG-1717.v2.patch, patchReleaseAuditWarnings.txt.gz, testlog.tgz, trunkReleaseAuditWarnings.txt.gz > > > I'm writing a loader that works with hive style partitioning e.g. /logs/type1/daydate=2010-11-01 > The loader does not know the schema upfront and this is something that the user adds in the script using the AS clause. > The problem is that this user defined schema is not available to the loader, so the loader cannot return any schema, the Loader does know what the partition keys are and pig needs in some way to know about these partition keys. > Currently if the schema is null pig never calls the LoadMetaData:getPartitionKeys method or the setPartitionFilter method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 18:56:52 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 27156 invoked from network); 1 Feb 2011 18:56:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 18:56:52 -0000 Received: (qmail 98348 invoked by uid 500); 1 Feb 2011 18:56:52 -0000 Delivered-To: rachel17@example.net Received: (qmail 97906 invoked by uid 500); 1 Feb 2011 18:56:50 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 97896 invoked by uid 500); 1 Feb 2011 18:56:49 -0000 Delivered-To: rachel17@example.net Received: (qmail 97893 invoked by uid 99); 1 Feb 2011 18:56:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:56:49 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:56:48 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id E6A9D187CE0 for <rachel17@example.net>; Tue, 1 Feb 2011 18:56:28 +0000 (UTC) Date: Tue, 1 Feb 2011 18:56:28 +0000 (UTC) From: "Alan Gates (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> In-Reply-To: <19550498.27791289490193681.JavaMail.jira@thor> Subject: [jira] Commented: (PIG-1717) pig needs to call setPartitionFilter if schema is null but getPartitionKeys is not MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989329#comment-12989329 ] Alan Gates commented on PIG-1717: --------------------------------- [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 7 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 509 release audit warnings (more than the trunk's current 507 warnings). [exec] As far as I can tell the release audit complaints are just because two new files got added which made changes in the jdiff output, so we can ignore those. > pig needs to call setPartitionFilter if schema is null but getPartitionKeys is not > ---------------------------------------------------------------------------------- > > Key: PIG-1717 > URL: https://issues.apache.org/jira/browse/PIG-1717 > Project: Pig > Issue Type: Improvement > Components: impl > Affects Versions: 0.9.0 > Reporter: Gerrit Jansen van Vuuren > Assignee: Gerrit Jansen van Vuuren > Priority: Minor > Fix For: 0.9.0 > > Attachments: PIG-1717.patch, PIG-1717.v1.patch, PIG-1717.v2.patch, patchReleaseAuditWarnings.txt.gz, testlog.tgz, trunkReleaseAuditWarnings.txt.gz > > > I'm writing a loader that works with hive style partitioning e.g. /logs/type1/daydate=2010-11-01 > The loader does not know the schema upfront and this is something that the user adds in the script using the AS clause. > The problem is that this user defined schema is not available to the loader, so the loader cannot return any schema, the Loader does know what the partition keys are and pig needs in some way to know about these partition keys. > Currently if the schema is null pig never calls the LoadMetaData:getPartitionKeys method or the setPartitionFilter method. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 18:59:22 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 29408 invoked from network); 1 Feb 2011 18:59:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 18:59:22 -0000 Received: (qmail 6107 invoked by uid 500); 1 Feb 2011 18:59:21 -0000 Delivered-To: rachel17@example.net Received: (qmail 6012 invoked by uid 500); 1 Feb 2011 18:59:20 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 6002 invoked by uid 500); 1 Feb 2011 18:59:19 -0000 Delivered-To: rachel17@example.net Received: (qmail 5999 invoked by uid 99); 1 Feb 2011 18:59:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:59:19 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 18:59:19 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id 1F163187E46 for <rachel17@example.net>; Tue, 1 Feb 2011 18:58:29 +0000 (UTC) Date: Tue, 1 Feb 2011 18:58:29 +0000 (UTC) From: "Alan Gates (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> In-Reply-To: <27415072.224181296073364582.JavaMail.jira@thor> Subject: [jira] Commented: (PIG-1825) ability to turn off the write ahead log for pig's HBaseStorage MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989330#comment-12989330 ] Alan Gates commented on PIG-1825: --------------------------------- Dmitriy, is this something we should check in? You seemed to indicate that this was no longer necessary after we moved to HBase 0.89 or above. > ability to turn off the write ahead log for pig's HBaseStorage > -------------------------------------------------------------- > > Key: PIG-1825 > URL: https://issues.apache.org/jira/browse/PIG-1825 > Project: Pig > Issue Type: Improvement > Affects Versions: 0.8.0 > Reporter: Corbin Hoenes > Priority: Minor > Fix For: 0.8.0 > > Attachments: HBaseStorage_noWAL.patch > > > Added an option to allow a caller of HBaseStorage to turn off the WriteAheadLog feature while doing bulk loads into hbase. > From the performance tuning wikipage: http://wiki.apache.org/hadoop/PerformanceTuning > "To speed up the inserts in a non critical job (like an import job), you can use Put.writeToWAL(false) to bypass writing to the write ahead log." > We've tested this on HBase 0.20.6 and it helps dramatically. > The -noWAL options is passed in just like other options for hbase storage: > STORE myalias INTO 'MyTable' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('mycolumnfamily:field1 mycolumnfamily:field2','-noWAL'); > This would be my first patch so please educate me with any steps I need to do. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 19:07:00 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 41949 invoked from network); 1 Feb 2011 19:06:55 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 19:06:55 -0000 Received: (qmail 23347 invoked by uid 500); 1 Feb 2011 19:06:55 -0000 Delivered-To: rachel17@example.net Received: (qmail 21976 invoked by uid 500); 1 Feb 2011 19:06:51 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 21968 invoked by uid 500); 1 Feb 2011 19:06:50 -0000 Delivered-To: rachel17@example.net Received: (qmail 21960 invoked by uid 99); 1 Feb 2011 19:06:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:06:49 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED,T_RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO hel.zones.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:06:49 +0000 Received: from hel.zones.apache.org (hel.zones.apache.org [83.161.120.155]) by hel.zones.apache.org (Postfix) with ESMTP id ECFB61882EA for <rachel17@example.net>; Tue, 1 Feb 2011 19:06:28 +0000 (UTC) Date: Tue, 1 Feb 2011 19:06:28 +0000 (UTC) From: "Daniel Dai (JIRA)" <rachel17@example.net> To: rachel17@example.net Message-ID: <rachel17@example.net> In-Reply-To: <14991496.263671296209023787.JavaMail.jira@thor> Subject: [jira] Updated: (PIG-1831) Indeterministic behavior in local mode due to static variable PigMapReduce.sJobConf MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/PIG-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-1831: ---------------------------- Attachment: PIG-1831-1.patch Richard suggest a better fix using ThreadLocal variable instead of static variable. Still keep static sJobConf for backward compatibility though it is already marked as deprecate in 0.7. In theory, if UDF still use deprecated sJobConf, they might see the same issue. But the chance of it should be very low. > Indeterministic behavior in local mode due to static variable PigMapReduce.sJobConf > ----------------------------------------------------------------------------------- > > Key: PIG-1831 > URL: https://issues.apache.org/jira/browse/PIG-1831 > Project: Pig > Issue Type: Bug > Affects Versions: 0.8.0 > Reporter: Vivek Padmanabhan > Assignee: Daniel Dai > Attachments: PIG-1831-0.patch, PIG-1831-1.patch > > > The below script when run in local mode gives me a different output. It looks like in local mode I have to store a relation obtained through streaming in order to use it afterwards. > For example consider the below script : > DEFINE MySTREAMUDF `test.sh`; > A = LOAD 'myinput' USING PigStorage() AS (myId:chararray, data2, data3,data4 ); > B = STREAM A THROUGH MySTREAMUDF AS (wId:chararray, num:int); > --STORE B into 'output.B'; > C = JOIN B by wId LEFT OUTER, A by myId; > D = FOREACH C GENERATE 888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4wId,888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4num,data4 ; > D = STREAM D THROUGH MySTREAMUDF AS (f1:chararray,f2:int); > --STORE D into 'output.D'; > E = foreach B GENERATE wId,num; > F = DISTINCT E; > G = GROUP F ALL; > H = FOREACH G GENERATE COUNT_STAR(F) as TotalCount; > I = CROSS D,H; > STORE I into 'output.I'; > test.sh > --------- > #/bin/bash > cut -f1,3 > And input is > abcd label1 11 feature1 > acbd label2 22 feature2 > adbc label3 33 feature3 > Here if I store relation B and D then everytime i get the result : > acbd 3 > abcd 3 > adbc 3 > But if i dont store relations B and D then I get an empty output. Here again I have observed that this behaviour is random ie sometimes like 1out of 5 runs there will be output. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira From rachel17@example.net Tue Feb 01 19:07:56 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 44138 invoked from network); 1 Feb 2011 19:07:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 19:07:56 -0000 Received: (qmail 28014 invoked by uid 500); 1 Feb 2011 19:07:55 -0000 Delivered-To: rachel17@example.net Received: (qmail 27860 invoked by uid 500); 1 Feb 2011 19:07:53 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 27851 invoked by uid 99); 1 Feb 2011 19:07:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:07:52 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rachel17@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ex0.corp.w3data.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:07:45 +0000 Received: from ex2.corp.w3data.com ([83.161.120.155]) by ex0.corp.w3data.com ([83.161.120.155]) with mapi; Tue, 1 Feb 2011 11:07:24 -0800 From: Andrew Musselman <rachel17@example.net> To: "rachel17@example.net" <rachel17@example.net> Date: Tue, 1 Feb 2011 11:07:22 -0800 Subject: How to debug Pig exception Thread-Topic: How to debug Pig exception Thread-Index: AcvCQ0HBmEiGoxJNQUOvjyapVX9FDg== Message-ID: <rachel17@example.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Hi List, How can I set up Eclipse or IntelliJ to debug Pig like a non-just-in-time-c= ompiled Java app? Thanks, Andrew From rachel17@example.net Tue Feb 01 19:08:52 2011 Return-Path: <rachel17@example.net> Delivered-To: rachel17@example.net Received: (qmail 44420 invoked from network); 1 Feb 2011 19:08:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2011 19:08:52 -0000 Received: (qmail 29904 invoked by uid 500); 1 Feb 2011 19:08:52 -0000 Delivered-To: rachel17@example.net Received: (qmail 29809 invoked by uid 500); 1 Feb 2011 19:08:49 -0000 Mailing-List: contact rachel17@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rachel17@example.net> List-Unsubscribe: <mailto:rachel17@example.net> List-Post: <mailto:rachel17@example.net> List-Id: <dev.pig.apache.org> Reply-To: rachel17@example.net Delivered-To: mailing list rachel17@example.net Received: (qmail 29788 invoked by uid 99); 1 Feb 2011 19:08:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:08:49 +0000 X-ASF-Spam-Status: No, hits=-1997.8 required=5.0 tests=ALL_TRUSTED,FREEMAIL_FROM,HTML_MESSAGE,MIME_QP_LONG_LINE X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO reviews.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Feb 2011 19:08:45 +0000 Received: from reviews.apache.org (localhost [83.161.120.155]) by reviews.apache.org (8.14.3/8.14.3/Debian-9.1ubuntu1) with ESMTP id p11J8Mqp022259; Tue, 1 Feb 2011 19:08:23 GMT Content-Type: multipart/alternative; boundary="===============4296129173518051486==" MIME-Version: 1.0 Subject: Review Request: Indeterministic behavior in local mode due to static variable PigMapReduce.sJobConf From: "Daniel Dai" <rachel17@example.net> To: "Richard Ding" <rachel17@example.net> Date: Tue, 01 Feb 2011 19:08:23 -0000 Message-ID: <rachel17@example.net> X-ReviewBoard-URL: https://reviews.apache.org X-ReviewRequest-URL: https://reviews.apache.org/r/376/ Cc: "pig" <rachel17@example.net>, "Daniel Dai" <rachel17@example.net> X-Virus-Checked: Checked by ClamAV on apache.org --===============4296129173518051486== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/376/ ----------------------------------------------------------- Review request for pig and Richard Ding. Summary ------- The below script when run in local mode gives me a different output. It loo= ks like in local mode I have to store a relation obtained through streaming= in order to use it afterwards. For example consider the
{'source': 'Apache Software Foundation Public Mail Archives'}