From pigashleyjenkins@example.net Sun Feb 01 19:40:23 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 92881 invoked from network); 1 Feb 2009 19:40:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2009 19:40:23 -0000 Received: (qmail 98404 invoked by uid 500); 1 Feb 2009 19:40:23 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 98385 invoked by uid 500); 1 Feb 2009 19:40:22 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 98374 invoked by uid 99); 1 Feb 2009 19:40:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 11:40:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 19:40:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 93DCD234C48D for <ashleyjenkins@example.net>; Sun, 1 Feb 2009 11:39:59 -0800 (PST) Message-ID: <1770740100.1233517199590.JavaMail.jira@brutus> Date: Sun, 1 Feb 2009 11:39:59 -0800 (PST) From: "Daniel Dai (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Commented: (PIG-631) 4 Unit test failures on Windows In-Reply-To: <358127891.1232749799558.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669432#action_12669432 ] Daniel Dai commented on PIG-631: -------------------------------- The problem is caused by [HBASE-1063|http://issues.apache.org/jira/browse/HBASE-1063]. It is fixed in hbase 0.19.0. However, hbase 0.19.0 works with hadoop 0.19.0 only, so we cannot upgrade hbase right now. I think we can disable TestHBaseStorage under Windows for now. Once we upgraded to hadoop 19, we can also upgrade hbase to 19. This issue is fixed automatically once we use hbase 19 (in my test, one additional action is needed, which is removing /tmp/hbase manually to let hbase recreate its file system). So once we move to hadoop 19 and upgrade to hbase 19, reverse the temporary patch, and close this issue. > 4 Unit test failures on Windows > -------------------------------- > > Key: PIG-631 > URL: https://issues.apache.org/jira/browse/PIG-631 > Project: Pig > Issue Type: Bug > Affects Versions: types_branch > Environment: Windows > Reporter: Lee Tucker > Attachments: PIG-631.temp.patch > > > 4 Windows unit test failures. All timeouts. Errors occur at tip of branch and have for several days. > [junit] Running org.apache.pig.test.TestAlgebraicEval > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] Test org.apache.pig.test.TestAlgebraicEval FAILED (timeout) > [junit] Running org.apache.pig.test.TestEvalPipeline > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] Test org.apache.pig.test.TestEvalPipeline FAILED (timeout) > [junit] Running org.apache.pig.test.TestHBaseStorage > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] Test org.apache.pig.test.TestHBaseStorage FAILED (timeout) > [junit] Running org.apache.pig.test.TestMapReduce > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] Test org.apache.pig.test.TestMapReduce FAILED (timeout) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Sun Feb 01 19:44:23 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 93439 invoked from network); 1 Feb 2009 19:44:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2009 19:44:23 -0000 Received: (qmail 99354 invoked by uid 500); 1 Feb 2009 19:44:22 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 99327 invoked by uid 500); 1 Feb 2009 19:44:22 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 99316 invoked by uid 99); 1 Feb 2009 19:44:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 11:44:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 19:44:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 869DC234C48D for <ashleyjenkins@example.net>; Sun, 1 Feb 2009 11:43:59 -0800 (PST) Message-ID: <543828513.1233517439536.JavaMail.jira@brutus> Date: Sun, 1 Feb 2009 11:43:59 -0800 (PST) From: "Daniel Dai (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Assigned: (PIG-631) 4 Unit test failures on Windows In-Reply-To: <358127891.1232749799558.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai reassigned PIG-631: ------------------------------ Assignee: Daniel Dai > 4 Unit test failures on Windows > -------------------------------- > > Key: PIG-631 > URL: https://issues.apache.org/jira/browse/PIG-631 > Project: Pig > Issue Type: Bug > Affects Versions: types_branch > Environment: Windows > Reporter: Lee Tucker > Assignee: Daniel Dai > Attachments: PIG-631.temp.patch > > > 4 Windows unit test failures. All timeouts. Errors occur at tip of branch and have for several days. > [junit] Running org.apache.pig.test.TestAlgebraicEval > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] Test org.apache.pig.test.TestAlgebraicEval FAILED (timeout) > [junit] Running org.apache.pig.test.TestEvalPipeline > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] Test org.apache.pig.test.TestEvalPipeline FAILED (timeout) > [junit] Running org.apache.pig.test.TestHBaseStorage > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] Test org.apache.pig.test.TestHBaseStorage FAILED (timeout) > [junit] Running org.apache.pig.test.TestMapReduce > [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 0 sec > [junit] Test org.apache.pig.test.TestMapReduce FAILED (timeout) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Sun Feb 01 20:40:28 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 5609 invoked from network); 1 Feb 2009 20:40:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2009 20:40:27 -0000 Received: (qmail 31140 invoked by uid 500); 1 Feb 2009 20:40:27 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 31107 invoked by uid 500); 1 Feb 2009 20:40:27 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 31096 invoked by uid 99); 1 Feb 2009 20:40:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 12:40:27 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 20:40:19 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id A3DCF234C4C2 for <ashleyjenkins@example.net>; Sun, 1 Feb 2009 12:39:59 -0800 (PST) Message-ID: <358315105.1233520799670.JavaMail.jira@brutus> Date: Sun, 1 Feb 2009 12:39:59 -0800 (PST) From: "Daniel Dai (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Resolved: (PIG-642) Limit after FRJ causes problems In-Reply-To: <234774831.1233088859667.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai resolved PIG-642. ---------------------------- Resolution: Fixed Fix Version/s: types_branch > Limit after FRJ causes problems > ------------------------------- > > Key: PIG-642 > URL: https://issues.apache.org/jira/browse/PIG-642 > Project: Pig > Issue Type: Bug > Reporter: Olga Natkovich > Assignee: Daniel Dai > Fix For: types_branch > > Attachments: PIG-642.patch > > > Script: > a = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age,gpa); > b = load '/user/pig/tests/data/singlefile/studenttab10k' as (name, age,gpa); > c = join a by name, b by name using "replicated"; > d = limit c 10; > dump d; > Error: ERROR 2013: Moving LOLimit in front of LOFRJoin is not implemented > It is fine not to move limit and apply it after the join. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Sun Feb 01 20:44:23 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 6039 invoked from network); 1 Feb 2009 20:44:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2009 20:44:23 -0000 Received: (qmail 32832 invoked by uid 500); 1 Feb 2009 20:44:23 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 32810 invoked by uid 500); 1 Feb 2009 20:44:22 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 32799 invoked by uid 99); 1 Feb 2009 20:44:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 12:44:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 20:44:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 7FE1E234C48D for <ashleyjenkins@example.net>; Sun, 1 Feb 2009 12:43:59 -0800 (PST) Message-ID: <114465223.1233521039509.JavaMail.jira@brutus> Date: Sun, 1 Feb 2009 12:43:59 -0800 (PST) From: "Daniel Dai (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Resolved: (PIG-569) Inconsistency with Hadoop in Pig load statements involving globs with subdirectories In-Reply-To: <543443801.1229545124274.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai resolved PIG-569. ---------------------------- Resolution: Duplicate It is a duplicate of [PIG-252|https://issues.apache.org/jira/browse/PIG-252] > Inconsistency with Hadoop in Pig load statements involving globs with subdirectories > ------------------------------------------------------------------------------------ > > Key: PIG-569 > URL: https://issues.apache.org/jira/browse/PIG-569 > Project: Pig > Issue Type: Bug > Components: impl > Affects Versions: types_branch > Environment: FC Linux x86/64, Pig revision 724576 > Reporter: Kevin Weil > Fix For: types_branch > > > Pig cannot handle LOAD statements with Hadoop globs where the globs have subdirectories. For example, > A = LOAD 'dir/{dir1/subdir1,dir2/subdir2,dir3/subdir3}' USING ... > A similar statement in Hadoop, hadoop dfs -ls dir/{dir1/subdir1,dir2/subdir2,dir3/subdir3}, does work correctly. > The output of running the above load statement in pig, built from svn revision 724576, is: > 2008-12-17 12:02:28,480 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete > 2008-12-17 12:02:28,480 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Map reduce job failed > 2008-12-17 12:02:28,480 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - java.io.IOException: Unable to get collect for pattern dir/{dir1/subdir1,dir2/subdir2,dir3/subdir3}} [Failed to obtain glob for dir/{dir1/subdir1,dir2/subdir2,dir3/subdir3}] > at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asCollection(HDataStorage.java:231) > at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asCollection(HDataStorage.java:40) > at org.apache.pig.impl.io.FileLocalizer.globMatchesFiles(FileLocalizer.java:486) > at org.apache.pig.impl.io.FileLocalizer.fileExists(FileLocalizer.java:455) > at org.apache.pig.backend.executionengine.PigSlicer.validate(PigSlicer.java:108) > at org.apache.pig.impl.io.ValidatingInputFileSpec.validate(ValidatingInputFileSpec.java:59) > at org.apache.pig.impl.io.ValidatingInputFileSpec.<init>(ValidatingInputFileSpec.java:44) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:200) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:742) > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:370) > at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247) > at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.pig.backend.datastorage.DataStorageException: Failed to obtain glob for dir/{dir1/subdir1,dir2/subdir2,dir3/subdir3} > ... 13 more > Caused by: java.io.IOException: Illegal file pattern: Expecting set closure character or end of range, or } for glob {dir1 at 5 > at org.apache.hadoop.fs.FileSystem$GlobFilter.error(FileSystem.java:1084) > at org.apache.hadoop.fs.FileSystem$GlobFilter.setRegex(FileSystem.java:1069) > at org.apache.hadoop.fs.FileSystem$GlobFilter.<init>(FileSystem.java:987) > at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:953) > at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:962) > at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:962) > at org.apache.hadoop.fs.FileSystem.globPathsLevel(FileSystem.java:962) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:902) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:862) > at org.apache.pig.backend.hadoop.datastorage.HDataStorage.asCollection(HDataStorage.java:215) > ... 12 more -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Sun Feb 01 21:04:23 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 8511 invoked from network); 1 Feb 2009 21:04:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2009 21:04:22 -0000 Received: (qmail 51934 invoked by uid 500); 1 Feb 2009 21:04:22 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 51912 invoked by uid 500); 1 Feb 2009 21:04:22 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 51901 invoked by uid 99); 1 Feb 2009 21:04:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 13:04:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 21:04:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 81AA3234C48D for <ashleyjenkins@example.net>; Sun, 1 Feb 2009 13:03:59 -0800 (PST) Message-ID: <643884498.1233522239516.JavaMail.jira@brutus> Date: Sun, 1 Feb 2009 13:03:59 -0800 (PST) From: "Daniel Dai (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Created: (PIG-654) Optimize build.xml MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Optimize build.xml ------------------ Key: PIG-654 URL: https://issues.apache.org/jira/browse/PIG-654 Project: Pig Issue Type: Improvement Components: impl Affects Versions: types_branch Reporter: Daniel Dai Assignee: Daniel Dai Priority: Trivial Fix For: types_branch Build process can be faster. Here is the issues slow down the build process: 1. Should change test/org/apache/pig/test/utils/dotGraph/Dot.jjt to DOTParser.jjt. [jjtree] assumes the output file name the same with the input file name, otherwise, it will recompile every time 2. Delete src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPostCombinerPackage.java and src/org/apache/pig/impl/plan/SplitIntroducer.java, both classes are empty and generates no class file. Ant will recompile them every time since it do not see the output class file. I do not know how to create a patch with file deletion and rename, so I describe the actions below: 1. mv test/org/apache/pig/test/utils/dotGraph/Dot.jjt test/org/apache/pig/test/utils/dotGraph/DOTParser.jjt 2. rm src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPostCombinerPackage.java 3. rm src/org/apache/pig/impl/plan/SplitIntroducer.java 4. apply patch to change build.xml (Dot.jjt -> DOTParser.jjt) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Sun Feb 01 21:10:22 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 9268 invoked from network); 1 Feb 2009 21:10:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2009 21:10:22 -0000 Received: (qmail 55060 invoked by uid 500); 1 Feb 2009 21:10:22 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 55040 invoked by uid 500); 1 Feb 2009 21:10:22 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 55029 invoked by uid 99); 1 Feb 2009 21:10:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 13:10:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Feb 2009 21:10:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 88509234C48D for <ashleyjenkins@example.net>; Sun, 1 Feb 2009 13:09:59 -0800 (PST) Message-ID: <1624258253.1233522599543.JavaMail.jira@brutus> Date: Sun, 1 Feb 2009 13:09:59 -0800 (PST) From: "Daniel Dai (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Updated: (PIG-654) Optimize build.xml In-Reply-To: <643884498.1233522239516.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-654: --------------------------- Attachment: PIG-654.patch > Optimize build.xml > ------------------ > > Key: PIG-654 > URL: https://issues.apache.org/jira/browse/PIG-654 > Project: Pig > Issue Type: Improvement > Components: impl > Affects Versions: types_branch > Reporter: Daniel Dai > Assignee: Daniel Dai > Priority: Trivial > Fix For: types_branch > > Attachments: PIG-654.patch > > > Build process can be faster. Here is the issues slow down the build process: > 1. Should change test/org/apache/pig/test/utils/dotGraph/Dot.jjt to DOTParser.jjt. [jjtree] assumes the output file name the same with the input file name, otherwise, it will recompile every time > 2. Delete src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPostCombinerPackage.java and src/org/apache/pig/impl/plan/SplitIntroducer.java, both classes are empty and generates no class file. Ant will recompile them every time since it do not see the output class file. > I do not know how to create a patch with file deletion and rename, so I describe the actions below: > 1. mv test/org/apache/pig/test/utils/dotGraph/Dot.jjt test/org/apache/pig/test/utils/dotGraph/DOTParser.jjt > 2. rm src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPostCombinerPackage.java > 3. rm src/org/apache/pig/impl/plan/SplitIntroducer.java > 4. apply patch to change build.xml (Dot.jjt -> DOTParser.jjt) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Mon Feb 02 19:00:24 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 26789 invoked from network); 2 Feb 2009 19:00:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Feb 2009 19:00:23 -0000 Received: (qmail 71974 invoked by uid 500); 2 Feb 2009 19:00:23 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 71852 invoked by uid 500); 2 Feb 2009 19:00:23 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 71841 invoked by uid 99); 2 Feb 2009 19:00:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Feb 2009 11:00:23 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Feb 2009 19:00:21 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 96BA3234C48D for <ashleyjenkins@example.net>; Mon, 2 Feb 2009 10:59:59 -0800 (PST) Message-ID: <1223260125.1233601199602.JavaMail.jira@brutus> Date: Mon, 2 Feb 2009 10:59:59 -0800 (PST) From: "Pradeep Kamath (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Commented: (PIG-428) TypeCastInserter does not replace projects in inner plans correctly MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12669687#action_12669687 ] Pradeep Kamath commented on PIG-428: ------------------------------------ Have you tried your query with top of trunk? The original issue fixed in this issue was when the TypeCastInserter was involved in the query. That is the case only when the load statement has a schema like " a = load 'bla' as (x:int, y:float);". In your query in the previous comment the load statement does not have a schema. I am wondering if the issues is somewhere else in that query but the error message is the same. > TypeCastInserter does not replace projects in inner plans correctly > ------------------------------------------------------------------- > > Key: PIG-428 > URL: https://issues.apache.org/jira/browse/PIG-428 > Project: Pig > Issue Type: Bug > Affects Versions: types_branch > Reporter: Pradeep Kamath > Fix For: types_branch > > Attachments: PIG-428.patch > > > The TypeCastInserter tries to replace the Project's input operator in inner plans with the new foreach operator it adds. However it should replace only those Projects' input where the new Foreach has been added after the operator which was earlier the input to Project. > Here is a query which fails due to this: > {code} > a = load 'st10k' as (name:chararray,age:int, gpa:double); > another = load 'st10k'; > c = foreach another generate $0, $1+ 10, $2 + 10; > d = join a by $0, c by $0; > dump d; > {code} > Here is the error: > {noformat} > 2008-09-11 23:34:28,169 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Error message from task (map) tip_200809051428_0045_m_000000java.io.IOException: Type mismatch in key from map: expected org.apache.pig.impl.io.NullableText, recieved org.apache.pig.impl.io.NullableBytesWritable > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:419) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.collect(PigMapReduce.java:83) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:172) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:158) > at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapReduce$Map.map(PigMapReduce.java:75) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:219) > at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124) > {noformat} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Tue Feb 03 18:52:22 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 89427 invoked from network); 3 Feb 2009 18:52:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Feb 2009 18:52:22 -0000 Received: (qmail 57645 invoked by uid 500); 3 Feb 2009 18:52:22 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 57514 invoked by uid 500); 3 Feb 2009 18:52:21 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 57486 invoked by uid 99); 3 Feb 2009 18:52:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2009 10:52:21 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Feb 2009 18:52:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 0B649234C4BD for <ashleyjenkins@example.net>; Tue, 3 Feb 2009 10:52:00 -0800 (PST) Message-ID: <1140431322.1233687120045.JavaMail.jira@brutus> Date: Tue, 3 Feb 2009 10:52:00 -0800 (PST) From: "Alan Gates (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Commented: (PIG-654) Optimize build.xml In-Reply-To: <643884498.1233522239516.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670056#action_12670056 ] Alan Gates commented on PIG-654: -------------------------------- +1 > Optimize build.xml > ------------------ > > Key: PIG-654 > URL: https://issues.apache.org/jira/browse/PIG-654 > Project: Pig > Issue Type: Improvement > Components: impl > Affects Versions: types_branch > Reporter: Daniel Dai > Assignee: Daniel Dai > Priority: Trivial > Fix For: types_branch > > Attachments: PIG-654.patch > > > Build process can be faster. Here is the issues slow down the build process: > 1. Should change test/org/apache/pig/test/utils/dotGraph/Dot.jjt to DOTParser.jjt. [jjtree] assumes the output file name the same with the input file name, otherwise, it will recompile every time > 2. Delete src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPostCombinerPackage.java and src/org/apache/pig/impl/plan/SplitIntroducer.java, both classes are empty and generates no class file. Ant will recompile them every time since it do not see the output class file. > I do not know how to create a patch with file deletion and rename, so I describe the actions below: > 1. mv test/org/apache/pig/test/utils/dotGraph/Dot.jjt test/org/apache/pig/test/utils/dotGraph/DOTParser.jjt > 2. rm src/org/apache/pig/backend/hadoop/executionengine/physicalLayer/relationalOperators/POPostCombinerPackage.java > 3. rm src/org/apache/pig/impl/plan/SplitIntroducer.java > 4. apply patch to change build.xml (Dot.jjt -> DOTParser.jjt) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Wed Feb 04 23:42:23 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 60046 invoked from network); 4 Feb 2009 23:42:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Feb 2009 23:42:23 -0000 Received: (qmail 84450 invoked by uid 500); 4 Feb 2009 23:42:22 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 84431 invoked by uid 500); 4 Feb 2009 23:42:22 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 84420 invoked by uid 99); 4 Feb 2009 23:42:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2009 15:42:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2009 23:42:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 9AA7F234C48B for <ashleyjenkins@example.net>; Wed, 4 Feb 2009 15:41:59 -0800 (PST) Message-ID: <643323788.1233790919619.JavaMail.jira@brutus> Date: Wed, 4 Feb 2009 15:41:59 -0800 (PST) From: "Santhosh Srinivasan (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Created: (PIG-655) Comparison of schemas of bincond operands is flawed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Comparison of schemas of bincond operands is flawed --------------------------------------------------- Key: PIG-655 URL: https://issues.apache.org/jira/browse/PIG-655 Project: Pig Issue Type: Bug Components: impl Affects Versions: types_branch Reporter: Santhosh Srinivasan Assignee: Santhosh Srinivasan Fix For: types_branch The comparison of schemas of bincond is flawed. Instead of comparing the field schemas, the type checker is comparing the schemas. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 01:34:21 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 28514 invoked from network); 5 Feb 2009 01:34:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 01:34:21 -0000 Received: (qmail 91303 invoked by uid 500); 5 Feb 2009 01:34:20 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 91283 invoked by uid 500); 5 Feb 2009 01:34:20 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 91267 invoked by uid 99); 5 Feb 2009 01:34:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2009 17:34:20 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 01:34:19 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 8B863234C48C for <ashleyjenkins@example.net>; Wed, 4 Feb 2009 17:33:59 -0800 (PST) Message-ID: <850693639.1233797639556.JavaMail.jira@brutus> Date: Wed, 4 Feb 2009 17:33:59 -0800 (PST) From: "Viraj Bhat (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Created: (PIG-656) Use of eval word in the package hierarchy of a UDF causes parse exception MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Use of eval word in the package hierarchy of a UDF causes parse exception ------------------------------------------------------------------------- Key: PIG-656 URL: https://issues.apache.org/jira/browse/PIG-656 Project: Pig Issue Type: Bug Components: documentation, grunt Affects Versions: types_branch Reporter: Viraj Bhat Fix For: types_branch Consider a Pig script which does something similar to a word count. It uses the built-in TOKENIZE function, but packages it inside a class hierarchy such as "mypackage.eval" {code} register TOKENIZE.jar my_src = LOAD '/user/viraj/mywordcount.txt' USING PigStorage('\t') AS (mlist: chararray); modules = FOREACH my_src GENERATE FLATTEN(mypackage.eval.TOKENIZE(mlist)); describe modules; grouped = GROUP modules BY $0; describe grouped; counts = FOREACH grouped GENERATE COUNT(modules), group; ordered = ORDER counts BY $0; dump ordered; {code} The parser complains: =========================================================================================================================== 2009-02-05 01:17:29,231 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Invalid alias: mypackage in {mlist: chararray} =========================================================================================================================== I looked at the following source code at (src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt) and it seems that : EVAL is a keyword in Pig. Here are some clarifications: 1) Is there documentation on what the EVAL keyword actually is? 2) Is EVAL keyword actually implemented? Viraj -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 01:38:21 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 31186 invoked from network); 5 Feb 2009 01:38:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 01:38:20 -0000 Received: (qmail 97989 invoked by uid 500); 5 Feb 2009 01:38:20 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 97974 invoked by uid 500); 5 Feb 2009 01:38:20 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 97963 invoked by uid 99); 5 Feb 2009 01:38:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2009 17:38:20 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 01:38:19 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 88B85234C48C for <ashleyjenkins@example.net>; Wed, 4 Feb 2009 17:37:59 -0800 (PST) Message-ID: <630178694.1233797879545.JavaMail.jira@brutus> Date: Wed, 4 Feb 2009 17:37:59 -0800 (PST) From: "Viraj Bhat (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Updated: (PIG-656) Use of eval word in the package hierarchy of a UDF causes parse exception In-Reply-To: <850693639.1233797639556.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Bhat updated PIG-656: --------------------------- Attachment: TOKENIZE.jar TOKENIZE.jar with java source file included > Use of eval word in the package hierarchy of a UDF causes parse exception > ------------------------------------------------------------------------- > > Key: PIG-656 > URL: https://issues.apache.org/jira/browse/PIG-656 > Project: Pig > Issue Type: Bug > Components: documentation, grunt > Affects Versions: types_branch > Reporter: Viraj Bhat > Fix For: types_branch > > Attachments: TOKENIZE.jar > > > Consider a Pig script which does something similar to a word count. It uses the built-in TOKENIZE function, but packages it inside a class hierarchy such as "mypackage.eval" > {code} > register TOKENIZE.jar > my_src = LOAD '/user/viraj/mywordcount.txt' USING PigStorage('\t') AS (mlist: chararray); > modules = FOREACH my_src GENERATE FLATTEN(mypackage.eval.TOKENIZE(mlist)); > describe modules; > grouped = GROUP modules BY $0; > describe grouped; > counts = FOREACH grouped GENERATE COUNT(modules), group; > ordered = ORDER counts BY $0; > dump ordered; > {code} > The parser complains: > =========================================================================================================================== > 2009-02-05 01:17:29,231 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Invalid alias: mypackage in {mlist: chararray} > =========================================================================================================================== > I looked at the following source code at (src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt) and it seems that : EVAL is a keyword in Pig. Here are some clarifications: > 1) Is there documentation on what the EVAL keyword actually is? > 2) Is EVAL keyword actually implemented? > Viraj -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 01:40:21 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 32081 invoked from network); 5 Feb 2009 01:40:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 01:40:20 -0000 Received: (qmail 98886 invoked by uid 500); 5 Feb 2009 01:40:20 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 98867 invoked by uid 500); 5 Feb 2009 01:40:20 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 98856 invoked by uid 99); 5 Feb 2009 01:40:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2009 17:40:20 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 01:40:19 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 8BD8E234C48C for <ashleyjenkins@example.net>; Wed, 4 Feb 2009 17:39:59 -0800 (PST) Message-ID: <609850489.1233797999558.JavaMail.jira@brutus> Date: Wed, 4 Feb 2009 17:39:59 -0800 (PST) From: "Viraj Bhat (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Updated: (PIG-656) Use of eval word in the package hierarchy of a UDF causes parse exception In-Reply-To: <850693639.1233797639556.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Bhat updated PIG-656: --------------------------- Attachment: mywordcount.txt Test data > Use of eval word in the package hierarchy of a UDF causes parse exception > ------------------------------------------------------------------------- > > Key: PIG-656 > URL: https://issues.apache.org/jira/browse/PIG-656 > Project: Pig > Issue Type: Bug > Components: documentation, grunt > Affects Versions: types_branch > Reporter: Viraj Bhat > Fix For: types_branch > > Attachments: mywordcount.txt, TOKENIZE.jar > > > Consider a Pig script which does something similar to a word count. It uses the built-in TOKENIZE function, but packages it inside a class hierarchy such as "mypackage.eval" > {code} > register TOKENIZE.jar > my_src = LOAD '/user/viraj/mywordcount.txt' USING PigStorage('\t') AS (mlist: chararray); > modules = FOREACH my_src GENERATE FLATTEN(mypackage.eval.TOKENIZE(mlist)); > describe modules; > grouped = GROUP modules BY $0; > describe grouped; > counts = FOREACH grouped GENERATE COUNT(modules), group; > ordered = ORDER counts BY $0; > dump ordered; > {code} > The parser complains: > =========================================================================================================================== > 2009-02-05 01:17:29,231 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000: Error during parsing. Invalid alias: mypackage in {mlist: chararray} > =========================================================================================================================== > I looked at the following source code at (src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt) and it seems that : EVAL is a keyword in Pig. Here are some clarifications: > 1) Is there documentation on what the EVAL keyword actually is? > 2) Is EVAL keyword actually implemented? > Viraj -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 12:24:21 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 10918 invoked from network); 5 Feb 2009 12:24:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 12:24:21 -0000 Received: (qmail 37747 invoked by uid 500); 5 Feb 2009 12:24:20 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 37729 invoked by uid 500); 5 Feb 2009 12:24:20 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 37718 invoked by uid 99); 5 Feb 2009 12:24:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 04:24:20 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 12:24:19 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id A0B52234C48D for <ashleyjenkins@example.net>; Thu, 5 Feb 2009 04:23:59 -0800 (PST) Message-ID: <1693253430.1233836639643.JavaMail.jira@brutus> Date: Thu, 5 Feb 2009 04:23:59 -0800 (PST) From: "Shravan Matthur Narayanamurthy (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Updated: (PIG-545) PERFORMANCE: Sampler for order bys does not produce a good distribution In-Reply-To: <1195389451.1227660764249.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shravan Matthur Narayanamurthy updated PIG-545: ----------------------------------------------- Status: Patch Available (was: Open) > PERFORMANCE: Sampler for order bys does not produce a good distribution > ----------------------------------------------------------------------- > > Key: PIG-545 > URL: https://issues.apache.org/jira/browse/PIG-545 > Project: Pig > Issue Type: Bug > Components: impl > Affects Versions: types_branch > Reporter: Alan Gates > Assignee: Amir Youssefi > Fix For: types_branch > > > In running tests on actual data, I've noticed that the final reduce of an order by has skewed partitions. Some reduces finish in a few seconds while some run for 20 minutes. Getting a better distribution should lead to much better performance for order by. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 12:36:30 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 17054 invoked from network); 5 Feb 2009 12:36:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 12:36:29 -0000 Received: (qmail 52696 invoked by uid 500); 5 Feb 2009 12:36:24 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 52675 invoked by uid 500); 5 Feb 2009 12:36:24 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 52362 invoked by uid 99); 5 Feb 2009 12:36:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 04:36:23 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 12:36:22 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id DC646234C4AC for <ashleyjenkins@example.net>; Thu, 5 Feb 2009 04:36:01 -0800 (PST) Message-ID: <318730390.1233837361901.JavaMail.jira@brutus> Date: Thu, 5 Feb 2009 04:36:01 -0800 (PST) From: "Shravan Matthur Narayanamurthy (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Updated: (PIG-545) PERFORMANCE: Sampler for order bys does not produce a good distribution In-Reply-To: <1195389451.1227660764249.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shravan Matthur Narayanamurthy updated PIG-545: ----------------------------------------------- Attachment: WRP.patch This patch implements the Weighted Range Partitioner as detailed in the Dewitt et. al. paper on Practical Skew Handling in Parallel Joins. The JobControlCompiler has been modified to use the new partitioner for order by. So the old unit tests should be valid. One caveat is that we need to mention the number of reducers via the parallel keyword when doing order by. Currently, if you don't specify it by default there will just be one partition and it messes up the distribution. We need to do something about this. Another thing is when the Partitioner gets configured it reads the entire sample file from HDFS but it currently doesn't do any reporting as I could not think of a way to do it right now > PERFORMANCE: Sampler for order bys does not produce a good distribution > ----------------------------------------------------------------------- > > Key: PIG-545 > URL: https://issues.apache.org/jira/browse/PIG-545 > Project: Pig > Issue Type: Bug > Components: impl > Affects Versions: types_branch > Reporter: Alan Gates > Assignee: Amir Youssefi > Fix For: types_branch > > Attachments: WRP.patch > > > In running tests on actual data, I've noticed that the final reduce of an order by has skewed partitions. Some reduces finish in a few seconds while some run for 20 minutes. Getting a better distribution should lead to much better performance for order by. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 22:44:30 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 81314 invoked from network); 5 Feb 2009 22:44:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 22:44:30 -0000 Received: (qmail 42164 invoked by uid 500); 5 Feb 2009 22:44:26 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 42149 invoked by uid 500); 5 Feb 2009 22:44:26 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 42121 invoked by uid 99); 5 Feb 2009 22:44:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 14:44:26 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 22:44:19 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id D308C234C4AB for <ashleyjenkins@example.net>; Thu, 5 Feb 2009 14:43:59 -0800 (PST) Message-ID: <540122434.1233873839863.JavaMail.jira@brutus> Date: Thu, 5 Feb 2009 14:43:59 -0800 (PST) From: "Alex Newman (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Issue Comment Edited: (PIG-6) Addition of Hbase Storage Option In Load/Store Statement MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670929#action_12670929 ] posix4e edited comment on PIG-6 at 2/5/09 2:43 PM: ------------------------------------------------------- Ok I have this "working" with hbase-0.19, you need to stick your hbase-conf files in the hadoop conf directory however. It is also poorly tested http://pastebin.com/m34813f5 you also need the hbase and hadoop 0.19 jars in the right place. I also added an optional start row and stop row when you create the hbase query was (Author: posix4e): Ok I have this "working" with hbase-0.19, you need to stick your hbase-conf files in the hadoop conf directory however. It is also poorly tested http://pastebin.com/m34813f5 > Addition of Hbase Storage Option In Load/Store Statement > -------------------------------------------------------- > > Key: PIG-6 > URL: https://issues.apache.org/jira/browse/PIG-6 > Project: Pig > Issue Type: New Feature > Environment: all environments > Reporter: Edward J. Yoon > Fix For: types_branch > > Attachments: hbase-0.18.1-test.jar, hbase-0.18.1.jar, PIG-6.patch, PIG-6_V01.patch > > > It needs to be able to load full table in hbase. (maybe ... difficult? i'm not sure yet.) > Also, as described below, > It needs to compose an abstract 2d-table only with certain data filtered from hbase array structure using arbitrary query-delimited. > {code} > A = LOAD table('hbase_table'); > or > B = LOAD table('hbase_table') Using HbaseQuery('Query-delimited by attributes & timestamp') as (f1, f2[, f3]); > {code} > Once test is done on my local machines, > I will clarify the grammars and give you more examples to help you explain more storage options. > Any advice welcome. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 22:44:31 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 81348 invoked from network); 5 Feb 2009 22:44:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 22:44:31 -0000 Received: (qmail 42552 invoked by uid 500); 5 Feb 2009 22:44:28 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 42535 invoked by uid 500); 5 Feb 2009 22:44:28 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 42496 invoked by uid 99); 5 Feb 2009 22:44:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 14:44:28 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 22:44:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id A8EFF234C48C for <ashleyjenkins@example.net>; Thu, 5 Feb 2009 14:43:59 -0800 (PST) Message-ID: <1284579643.1233873839684.JavaMail.jira@brutus> Date: Thu, 5 Feb 2009 14:43:59 -0800 (PST) From: "Alex Newman (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Commented: (PIG-6) Addition of Hbase Storage Option In Load/Store Statement MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-6?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670929#action_12670929 ] Alex Newman commented on PIG-6: ------------------------------- Ok I have this "working" with hbase-0.19, you need to stick your hbase-conf files in the hadoop conf directory however. It is also poorly tested http://pastebin.com/m34813f5 > Addition of Hbase Storage Option In Load/Store Statement > -------------------------------------------------------- > > Key: PIG-6 > URL: https://issues.apache.org/jira/browse/PIG-6 > Project: Pig > Issue Type: New Feature > Environment: all environments > Reporter: Edward J. Yoon > Fix For: types_branch > > Attachments: hbase-0.18.1-test.jar, hbase-0.18.1.jar, PIG-6.patch, PIG-6_V01.patch > > > It needs to be able to load full table in hbase. (maybe ... difficult? i'm not sure yet.) > Also, as described below, > It needs to compose an abstract 2d-table only with certain data filtered from hbase array structure using arbitrary query-delimited. > {code} > A = LOAD table('hbase_table'); > or > B = LOAD table('hbase_table') Using HbaseQuery('Query-delimited by attributes & timestamp') as (f1, f2[, f3]); > {code} > Once test is done on my local machines, > I will clarify the grammars and give you more examples to help you explain more storage options. > Any advice welcome. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 23:04:21 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 87503 invoked from network); 5 Feb 2009 23:04:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 23:04:20 -0000 Received: (qmail 86922 invoked by uid 500); 5 Feb 2009 23:04:20 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 86905 invoked by uid 500); 5 Feb 2009 23:04:20 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 86893 invoked by uid 99); 5 Feb 2009 23:04:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 15:04:20 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 23:04:19 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 9FE48234C4A7 for <ashleyjenkins@example.net>; Thu, 5 Feb 2009 15:03:59 -0800 (PST) Message-ID: <1565581374.1233875039653.JavaMail.jira@brutus> Date: Thu, 5 Feb 2009 15:03:59 -0800 (PST) From: "Alan Gates (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Assigned: (PIG-545) PERFORMANCE: Sampler for order bys does not produce a good distribution In-Reply-To: <1195389451.1227660764249.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned PIG-545: ------------------------------ Assignee: Pradeep Kamath (was: Amir Youssefi) > PERFORMANCE: Sampler for order bys does not produce a good distribution > ----------------------------------------------------------------------- > > Key: PIG-545 > URL: https://issues.apache.org/jira/browse/PIG-545 > Project: Pig > Issue Type: Bug > Components: impl > Affects Versions: types_branch > Reporter: Alan Gates > Assignee: Pradeep Kamath > Fix For: types_branch > > Attachments: WRP.patch > > > In running tests on actual data, I've noticed that the final reduce of an order by has skewed partitions. Some reduces finish in a few seconds while some run for 20 minutes. Getting a better distribution should lead to much better performance for order by. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 23:04:22 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 87532 invoked from network); 5 Feb 2009 23:04:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 23:04:22 -0000 Received: (qmail 87075 invoked by uid 500); 5 Feb 2009 23:04:22 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 87052 invoked by uid 500); 5 Feb 2009 23:04:22 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 87041 invoked by uid 99); 5 Feb 2009 23:04:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 15:04:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 23:04:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id ACAD8234C4AC for <ashleyjenkins@example.net>; Thu, 5 Feb 2009 15:03:59 -0800 (PST) Message-ID: <672125412.1233875039706.JavaMail.jira@brutus> Date: Thu, 5 Feb 2009 15:03:59 -0800 (PST) From: "Alan Gates (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Assigned: (PIG-460) PERFORMANCE: Order by done in 3 MR jobs, could be done in 2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned PIG-460: ------------------------------ Assignee: Pradeep Kamath (was: Amir Youssefi) Assigning to Pradeep for review. > PERFORMANCE: Order by done in 3 MR jobs, could be done in 2 > ------------------------------------------------------------ > > Key: PIG-460 > URL: https://issues.apache.org/jira/browse/PIG-460 > Project: Pig > Issue Type: Bug > Affects Versions: types_branch > Reporter: Alan Gates > Assignee: Pradeep Kamath > Fix For: types_branch > > Attachments: sampler.patch, sampler2.patch > > > Currently order by is done in three MR jobs: > job 1: read data in whatever loader the user requests, store using BinStorage > job 2: load using RandomSampleLoader, find quantiles > job 3: load data again and sort > It is done this way because RandomSampleLoader extends BinStorage, and so needs the data in that format to read it. > If the logic in RandomSampleLoader was made into an operator instead of being in a loader then jobs 1 and 2 could be merged. On average job 1 takes about 15% of the time of an order by script. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 23:04:24 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 87564 invoked from network); 5 Feb 2009 23:04:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 23:04:24 -0000 Received: (qmail 87329 invoked by uid 500); 5 Feb 2009 23:04:24 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 87206 invoked by uid 500); 5 Feb 2009 23:04:24 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 87195 invoked by uid 99); 5 Feb 2009 23:04:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 15:04:24 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 23:04:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 8D3D8234C48C for <ashleyjenkins@example.net>; Thu, 5 Feb 2009 15:03:59 -0800 (PST) Message-ID: <585.376.8353.1233875039563.JavaMail.jira@brutus> Date: Thu, 5 Feb 2009 15:03:59 -0800 (PST) From: "Alan Gates (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Commented: (PIG-545) PERFORMANCE: Sampler for order bys does not produce a good distribution In-Reply-To: <1195389451.1227660764249.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12670938#action_12670938 ] Alan Gates commented on PIG-545: -------------------------------- I ran the pigmix L9 (order by of single field) and L10 (order by of multiple fields). L9 went from 14 minutes to 8, so this patch holds huge promise. But L10 went from 8 minutes to 11, so it doesn't seem to be working well in the multiple field case. (It could also be related to the fact that L10 uses descending on one of the columns, I don't know if the new partitioner can handle that or not.) I also ran our end to end order by tests on it, and all passed, except bigdata_1, which fails with an IndexOutOfBounds exception in the new WeightedRangePartitioner class. As for the caveat that it needs to know the number of reducers up front, I believe in cases where the user doesn't say parallel, that we can determine the parallelism of the reduces using JobClient.getDefaultReduces(). We need to double check that this will give us the right information in both the hod and non-hod cases. > PERFORMANCE: Sampler for order bys does not produce a good distribution > ----------------------------------------------------------------------- > > Key: PIG-545 > URL: https://issues.apache.org/jira/browse/PIG-545 > Project: Pig > Issue Type: Bug > Components: impl > Affects Versions: types_branch > Reporter: Alan Gates > Assignee: Amir Youssefi > Fix For: types_branch > > Attachments: WRP.patch > > > In running tests on actual data, I've noticed that the final reduce of an order by has skewed partitions. Some reduces finish in a few seconds while some run for 20 minutes. Getting a better distribution should lead to much better performance for order by. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From pigashleyjenkins@example.net Thu Feb 05 23:36:22 2009 Return-Path: <pigashleyjenkins@example.net> Delivered-To: ashleyjenkins@example.net Received: (qmail 97755 invoked from network); 5 Feb 2009 23:36:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Feb 2009 23:36:22 -0000 Received: (qmail 23887 invoked by uid 500); 5 Feb 2009 23:36:22 -0000 Delivered-To: ashleyjenkins@example.net Received: (qmail 23764 invoked by uid 500); 5 Feb 2009 23:36:22 -0000 Mailing-List: contact ashleyjenkins@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:ashleyjenkins@example.net> List-Unsubscribe: <mailto:ashleyjenkins@example.net> List-Post: <mailto:ashleyjenkins@example.net> List-Id: <pig-dev.hadoop.apache.org> Reply-To: ashleyjenkins@example.net Delivered-To: mailing list ashleyjenkins@example.net Received: (qmail 23753 invoked by uid 99); 5 Feb 2009 23:36:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 15:36:22 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Feb 2009 23:36:20 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 8EECC234C48C for <ashleyjenkins@example.net>; Thu, 5 Feb 2009 15:35:59 -0800 (PST) Message-ID: <1335304286.1233876959570.JavaMail.jira@brutus> Date: Thu, 5 Feb 2009 15:35:59 -0800 (PST) From: "Laukik Chitnis (JIRA)" <ashleyjenkins@example.net> To: ashleyjenkins@example.net Subject: [jira] Created: (PIG-657) splitsize is ignored in PigInputFormat MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org splitsize is ignored in PigInputFormat -------------------------------------- Key: PIG-657 URL: https://issues.apache.org/jira/browse/PIG-657 Project: Pig Issue Type: Bug Reporter: Laukik Chitnis The way to control the number of mappers in Hadoop has been to specify a mapred.min.split.size parameter in the job conf. For eg. mapred.min.split.size=1073741824,mapred.map.tasks=10 However, even if this parameter is specified, Pig creates the number of mappers depending only on the number of blocks in the file. This is because the parameter is not used in the PigInputFormat. The parameter can actually be extracted from the job conf object. So, one way of doing this would be to pass an handle to the job conf object to the PigInputFormat or the custom slicer.
{'source': 'Apache Software Foundation Public Mail Archives'}