From pig-user-returneric91@example.com Tue Jun 03 23:51:09 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 55085 invoked from network); 3 Jun 2008 23:51:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Jun 2008 23:51:09 -0000 Received: (qmail 89743 invoked by uid 500); 3 Jun 2008 23:51:11 -0000 Delivered-To: eric91@example.com Received: (qmail 89727 invoked by uid 500); 3 Jun 2008 23:51:11 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 89716 invoked by uid 99); 3 Jun 2008 23:51:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jun 2008 16:51:11 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO outbound.mse16.exchange.ms) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jun 2008 23:50:15 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Subject: a builtin function to return number of fields? Date: Tue, 3 Jun 2008 19:49:35 -0400 Message-ID: <eric91@example.com> In-Reply-To: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: a builtin function to return number of fields? Thread-Index: AcjFdTewhJz4Ngs0RO+n3YzlMJHyGQAXoYaw References: <eric91@example.com> From: "Haijun Cao" <eric91@example.com> To: <eric91@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Hi,=20 I am grouping by $1, but got IndexOutofBoundException because some rows are mal-formed, they only have 1 field. Is there a built-in function to return the number of fields for each row? So I can ignore those rows myself.=20 Or, even better, automatically ignore the mal-formed rows for me?=20 Thanks Haijun From pig-user-returneric91@example.com Wed Jun 04 00:00:26 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 58800 invoked from network); 4 Jun 2008 00:00:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 00:00:24 -0000 Received: (qmail 97878 invoked by uid 500); 4 Jun 2008 00:00:27 -0000 Delivered-To: eric91@example.com Received: (qmail 97869 invoked by uid 500); 4 Jun 2008 00:00:27 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 97858 invoked by uid 99); 4 Jun 2008 00:00:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jun 2008 17:00:26 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout2-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jun 2008 23:59:28 +0000 Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [83.161.120.155]) by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m53NwZwW072543 for <eric91@example.com>; Tue, 3 Jun 2008 16:58:38 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:x-mimeole:content-class:mime-version: content-type:content-transfer-encoding:subject:date:message-id: in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic: thread-index:references:from:to:return-path:x-originalarrivaltime; b=YNeS0ri9B1ZQ2zGpKHtqIQyMHxePMcUKGBQoyX+utg4OnCMOXVM9cRoT26YpgiI3 Received: from SNV-EXVS10.ds.corp.yahoo.com ([83.161.120.155]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(539)789-3517); Tue, 3 Jun 2008 16:58:32 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: a builtin function to return number of fields? Date: Tue, 3 Jun 2008 16:57:52 -0700 Message-ID: <eric91@example.com> In-Reply-To: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: a builtin function to return number of fields? Thread-Index: AcjFdTewhJz4Ngs0RO+n3YzlMJHyGQAXoYawAABzdsA= References: <eric91@example.com> <eric91@example.com> From: "Amir Youssefi" <eric91@example.com> To: <eric91@example.com> X-OriginalArrivalTime: 03 Jun 2008 23:58:32.0916 (UTC) FILETIME=[BA960940:01C8C5D5] X-Virus-Checked: Checked by ClamAV on apache.org ARITY - computes the number of fields in a tuple (also known as the "arity" of a tuple)=20 Amir -----Original Message----- From: Haijun Cao [mailto:eric91@example.com]=20 Sent: Tuesday, June 03, 2008 4:50 PM To: eric91@example.com Subject: a builtin function to return number of fields? Hi,=20 I am grouping by $1, but got IndexOutofBoundException because some rows are mal-formed, they only have 1 field. Is there a built-in function to return the number of fields for each row? So I can ignore those rows myself.=20 Or, even better, automatically ignore the mal-formed rows for me?=20 Thanks Haijun From pig-user-returneric91@example.com Wed Jun 04 00:07:40 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 77841 invoked from network); 4 Jun 2008 00:07:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 00:07:39 -0000 Received: (qmail 2763 invoked by uid 500); 4 Jun 2008 00:07:41 -0000 Delivered-To: eric91@example.com Received: (qmail 2743 invoked by uid 500); 4 Jun 2008 00:07:41 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 2732 invoked by uid 99); 4 Jun 2008 00:07:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jun 2008 17:07:41 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric91@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0708.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 00:06:53 +0000 Received: by rv-out-0708.google.com with SMTP id k29so1692684rvb.0 for <eric91@example.com>; Tue, 03 Jun 2008 17:07:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:mime-version:content-type:x-google-sender-auth; bh=vcpOVFcFb8TUufnBbTowXIpHFLyrP0wHlqOWYllBx30=; b=jJZjRMYLJpnnf27/yokhdDGd/pWUecQ2f8HYAlHSPeBVA8+AMbOphIvOHr8F6GldFONe+YSTYpZUUZdiwjmmGcXQDzeZcqizhu5y0noyLd1JI6cgp113NaNci8V/A76FNPhEd0knNY6sKMra3m0xSlkxIrsGI080VIz2Pv/4fD8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:mime-version:content-type:x-google-sender-auth; b=aGLiGh9bZc8oUqFS4PigvIIVZLahsQjJDclznlJCb5waQkVxk/2WyM76Y2vy/PUsNA1BrELcFBgRNr+q52yzQE96LKBoixE50TSstb0EeCj9oQrjmQN89aLmbkm2TZogWsfILcoQ0auD9nIfnCn95HliYFwHTDspvf7WsK08Y6s= Received: by 83.161.120.155 with SMTP id s12mr6237000rve.101.1212537613596; Tue, 03 Jun 2008 17:00:13 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Tue, 3 Jun 2008 17:00:13 -0700 (PDT) Message-ID: <eric91@example.com> Date: Tue, 3 Jun 2008 17:00:13 -0700 From: "Prashanth Pappu" <eric91@example.com> Sender: eric91@example.com To: eric91@example.com Subject: PIG performance MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_267_19367677.1212537613594" X-Google-Sender-Auth: c5091e33ab63c834 X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_267_19367677.1212537613594 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I just upgraded my PIG src to top of svn and see really poor performance with group and cogroup queries. Is there a recommended svn version to be used with hadoop 1.17? Prashanth ------=_Part_267_19367677.1212537613594-- From pig-user-returneric91@example.com Wed Jun 04 05:51:03 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 78999 invoked from network); 4 Jun 2008 05:51:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 05:51:03 -0000 Received: (qmail 32338 invoked by uid 500); 4 Jun 2008 05:51:05 -0000 Delivered-To: eric91@example.com Received: (qmail 32309 invoked by uid 500); 4 Jun 2008 05:51:05 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 32298 invoked by uid 99); 4 Jun 2008 05:51:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jun 2008 22:51:05 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO outbound.mse16.exchange.ms) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 05:50:16 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----_=_NextPart_001_01C8C606.E6D47D06" Subject: RE: a builtin function to return number of fields? Date: Wed, 4 Jun 2008 01:47:49 -0400 Message-ID: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: <eric91@example.com> Thread-Topic: a builtin function to return number of fields? Thread-Index: AcjFdTewhJz4Ngs0RO+n3YzlMJHyGQAXoYawAABzdsAADD5+ig== References: <eric91@example.com> <eric91@example.com> <eric91@example.com> From: "Haijun Cao" <eric91@example.com> To: <eric91@example.com> X-Virus-Checked: Checked by ClamAV on apache.org ------_=_NextPart_001_01C8C606.E6D47D06 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Amir, Thanks for the pointer. It works for me. I need to look at the twiki = more closely. :-) Haijun -----Original Message----- From: Amir Youssefi [mailto:eric91@example.com] Sent: Tue 6/3/2008 7:57 PM To: eric91@example.com Subject: RE: a builtin function to return number of fields? =20 ARITY - computes the number of fields in a tuple (also known as the "arity" of a tuple)=20 Amir -----Original Message----- From: Haijun Cao [mailto:eric91@example.com]=20 Sent: Tuesday, June 03, 2008 4:50 PM To: eric91@example.com Subject: a builtin function to return number of fields? Hi,=20 I am grouping by $1, but got IndexOutofBoundException because some rows are mal-formed, they only have 1 field. Is there a built-in function to return the number of fields for each row? So I can ignore those rows myself.=20 Or, even better, automatically ignore the mal-formed rows for me?=20 Thanks Haijun ------_=_NextPart_001_01C8C606.E6D47D06-- From pig-user-returneric91@example.com Wed Jun 04 14:56:25 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 92984 invoked from network); 4 Jun 2008 14:56:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 14:56:25 -0000 Received: (qmail 68724 invoked by uid 500); 4 Jun 2008 14:56:22 -0000 Delivered-To: eric91@example.com Received: (qmail 68654 invoked by uid 500); 4 Jun 2008 14:56:21 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 68632 invoked by uid 99); 4 Jun 2008 14:56:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 07:56:21 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric91@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO py-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 14:55:33 +0000 Received: by py-out-1112.google.com with SMTP id u52so84005pyb.13 for <eric91@example.com>; Wed, 04 Jun 2008 07:55:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=GR5L9TIC7JMPI70WzeedXZuQXBwYJRvtonhx4ioMLpg=; b=xdvtVfx4NU8AAGtygsHC0qnzr8Rd9DCC7z97cK0I4P//kpjS/L5ggrLrJ9rwQlph+e m5R83WQD7esLn1y6lXQyau/T2a4JYwM60Zk9hNupJgnXScSYKf261b3r7T8KNyvHjzog vBqLjeHm+RmiGe0g66aRZhjaN0bJddiZFi9jk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=nVYOqXg2UhxcWlcuQ+50dH4rVLXZSV4ox4yTh3IY2flBhomL1YVD8zvI//z2eOLIYM 99o19bx/8qddbK56sw1KI5WffXhx11vNUGYyNk8R60z5aF+FwDC+Oj1lhM6Ta9ZW8xaL uZGbpMplBY89WMkO7TqF0q410/hn09VnNkdq4= Received: by 83.161.120.155 with SMTP id z1mr6658054rvi.277.1212591347411; Wed, 04 Jun 2008 07:55:47 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 4 Jun 2008 07:55:47 -0700 (PDT) Message-ID: <eric91@example.com> Date: Wed, 4 Jun 2008 15:55:47 +0100 From: "Tom White" <eric91@example.com> To: eric91@example.com Subject: Specifying multiple input paths MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Virus-Checked: Checked by ClamAV on apache.org I;m having a problem loading data from multiple paths in Pig. What I'm trying to do is to load data from a range of dates, so I would like to specify an input of two globbed paths: x = LOAD '2008/05/{26,27,28,29,30,31},2008/06/{1,2}' Pig doesn't seem to like this though as it's trying to interpret it as a single path. The best I can do it to use UNION: x1 = LOAD '2008/05/{26,27,28,29,30,31}' x2 = LOAD '2008/06/{1,2}' x = UNION x1, x2 The downside to this is that I want to parameterize my paths, and having separate script for each number of paths in the input is cumbersome. Is there a better way of doing this? Are there any plans to support multiple paths, and/or PathFilters? Thanks, Tom From pig-user-returneric91@example.com Wed Jun 04 17:52:24 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 24797 invoked from network); 4 Jun 2008 17:52:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 17:52:24 -0000 Received: (qmail 13778 invoked by uid 500); 4 Jun 2008 17:52:27 -0000 Delivered-To: eric91@example.com Received: (qmail 13757 invoked by uid 500); 4 Jun 2008 17:52:27 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 13746 invoked by uid 99); 4 Jun 2008 17:52:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 10:52:27 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout1-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 17:51:28 +0000 Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [83.161.120.155]) by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m54Hp8mK027335 for <eric91@example.com>; Wed, 4 Jun 2008 10:51:15 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:x-mimeole:content-class:mime-version: content-type:content-transfer-encoding:subject:date:message-id: in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic: thread-index:references:from:to:return-path:x-originalarrivaltime; b=FA0zP86lQ/u8BaHqi/XsSKmap7Fzx59wvJpaM2xt8s7A/fcUtuqL15KCoIFpc+Ih Received: from SNV-EXVS09.ds.corp.yahoo.com ([83.161.120.155]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(539)789-3517); Wed, 4 Jun 2008 10:51:08 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: PIG performance Date: Wed, 4 Jun 2008 10:51:02 -0700 Message-ID: <eric91@example.com> In-Reply-To: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: PIG performance Thread-Index: AcjF1xhcVqaV59upQ9mdl1QH2F11PgAlFImg References: <eric91@example.com> From: "Olga Natkovich" <eric91@example.com> To: <eric91@example.com> X-OriginalArrivalTime: 04 Jun 2008 17:51:08.0719 (UTC) FILETIME=[91A243F0:01C8C66B] X-Virus-Checked: Checked by ClamAV on apache.org Do you mean that the same query ran faster with some previous code you ran? Which code was that? Olga=20 > -----Original Message----- > From: eric91@example.com=20 > [mailto:eric91@example.com] On Behalf Of Prashanth Pappu > Sent: Tuesday, June 03, 2008 5:00 PM > To: eric91@example.com > Subject: PIG performance >=20 > I just upgraded my PIG src to top of svn and see really poor=20 > performance with group and cogroup queries. >=20 > Is there a recommended svn version to be used with hadoop 1.17? >=20 > Prashanth >=20 From pig-user-returneric91@example.com Wed Jun 04 18:16:17 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 34698 invoked from network); 4 Jun 2008 18:16:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 18:16:17 -0000 Received: (qmail 49056 invoked by uid 500); 4 Jun 2008 18:16:20 -0000 Delivered-To: eric91@example.com Received: (qmail 48918 invoked by uid 500); 4 Jun 2008 18:16:20 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 48906 invoked by uid 99); 4 Jun 2008 18:16:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 11:16:20 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric91@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0708.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 18:15:31 +0000 Received: by rv-out-0708.google.com with SMTP id k29so249842rvb.0 for <eric91@example.com>; Wed, 04 Jun 2008 11:15:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:in-reply-to:mime-version:content-type:references :x-google-sender-auth; bh=N/ePXfmsty6/m7AHmbMxDiphA4T8nF3w6hza1+1DcXc=; b=KLofO+9pbT/8X+Zb/rCJ/MnrJYregRkoBupSN0Zg1hCr4ebuJtkCnk0WAZJxyKlasn 0UMR6+OUWS0sgItK9vqpIHf5H6K8mX7xl+K0CcTALZGG2RuRz5+VsShm1IgRhS3w6end 03K3H9V7uO/m/44GsqNd7nHNnO9e4PEjjZp6k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:in-reply-to:mime-version :content-type:references:x-google-sender-auth; b=XiM/hs2YWjLd41C91N1Zu6Dvvx7xAas/1v09/t5ql2jvIJRbWYTNGVVi0jO5fAYX/B iDpEE2HS+E3oZ6hDY2Nijt3Lhmq5mEFSj6SFpU1Jclte25CGj+wvCmrZ49OWbo8PaZHx HFN8B8EJ65bEFhBPdchiAoYLCfhbX8lKIj1gE= Received: by 83.161.120.155 with SMTP id y1mr188860rvh.195.1212603346136; Wed, 04 Jun 2008 11:15:46 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 4 Jun 2008 11:15:46 -0700 (PDT) Message-ID: <eric91@example.com> Date: Wed, 4 Jun 2008 11:15:46 -0700 From: "Prashanth Pappu" <eric91@example.com> Sender: eric91@example.com To: eric91@example.com Subject: Re: PIG performance In-Reply-To: <eric91@example.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_4418_27596262.1212603346122" References: <eric91@example.com> <eric91@example.com> X-Google-Sender-Auth: 37ef5d6af4935a62 X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_4418_27596262.1212603346122 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Yes, the same queries were substantially slower with hadoop 1.17 and the latest svn of PIG (compared to hadoop1.16 and svn 653894). It is hard for me to copy the exact query as there is a lot of context but I zeroed it down to cogroup statements. I can try and create an example. But I wanted to check if there is a specific svn version that others use with hadoop1.17. Or does everyone use top of svn trunk? Because upgrading from hadoop1.16 and older svn version to latest svn version did degrade the performance quite a bit. Prashanth On Wed, Jun 4, 2008 at 10:51 AM, Olga Natkovich <eric91@example.com> wrote: > Do you mean that the same query ran faster with some previous code you > ran? Which code was that? > > Olga > > > -----Original Message----- > > From: eric91@example.com > > [mailto:eric91@example.com] On Behalf Of Prashanth Pappu > > Sent: Tuesday, June 03, 2008 5:00 PM > > To: eric91@example.com > > Subject: PIG performance > > > > I just upgraded my PIG src to top of svn and see really poor > > performance with group and cogroup queries. > > > > Is there a recommended svn version to be used with hadoop 1.17? > > > > Prashanth > > > ------=_Part_4418_27596262.1212603346122-- From pig-user-returneric91@example.com Wed Jun 04 19:07:21 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 20763 invoked from network); 4 Jun 2008 19:07:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 19:07:21 -0000 Received: (qmail 31947 invoked by uid 500); 4 Jun 2008 19:07:20 -0000 Delivered-To: eric91@example.com Received: (qmail 31919 invoked by uid 500); 4 Jun 2008 19:07:20 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 31887 invoked by uid 99); 4 Jun 2008 19:07:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 12:07:20 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout2-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 19:06:21 +0000 Received: from SNV-EXBH01.ds.corp.yahoo.com (snv-exbh01.ds.corp.yahoo.com [83.161.120.155]) by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m54J5l4S044592 for <eric91@example.com>; Wed, 4 Jun 2008 12:06:03 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:x-mimeole:content-class:mime-version: content-type:content-transfer-encoding:subject:date:message-id: in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic: thread-index:references:from:to:return-path:x-originalarrivaltime; b=oCVI3S2cEqvbbotq7/m3x/DdxEKhhMa3ImV7y8WFPume/JQcVWbugFca2JapAS1P Received: from SNV-EXVS09.ds.corp.yahoo.com ([83.161.120.155]) by SNV-EXBH01.ds.corp.yahoo.com with Microsoft SMTPSVC(539)789-3517); Wed, 4 Jun 2008 12:05:59 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: PIG performance Date: Wed, 4 Jun 2008 12:05:26 -0700 Message-ID: <eric91@example.com> In-Reply-To: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: PIG performance Thread-Index: AcjGb1PBlA+9+bYhR0Shp8aB2hYeRQABpP5w References: <eric91@example.com> <eric91@example.com> <eric91@example.com> From: "Olga Natkovich" <eric91@example.com> To: <eric91@example.com> X-OriginalArrivalTime: 04 Jun 2008 19:05:59.0034 (UTC) FILETIME=[0611F5A0:01C8C676] X-Virus-Checked: Checked by ClamAV on apache.org We saw some slowdown in the unit tests but not in our end-to-end tests. Olga=20 > -----Original Message----- > From: eric91@example.com=20 > [mailto:eric91@example.com] On Behalf Of Prashanth Pappu > Sent: Wednesday, June 04, 2008 11:16 AM > To: eric91@example.com > Subject: Re: PIG performance >=20 > Yes, the same queries were substantially slower with hadoop=20 > 1.17 and the latest svn of PIG (compared to hadoop1.16 and=20 > svn 653894). >=20 > It is hard for me to copy the exact query as there is a lot=20 > of context but I zeroed it down to cogroup statements. I can=20 > try and create an example. But I wanted to check if there is=20 > a specific svn version that others use with hadoop1.17. Or=20 > does everyone use top of svn trunk? > Because upgrading from hadoop1.16 and older svn version to=20 > latest svn version did degrade the performance quite a bit. >=20 > Prashanth >=20 > On Wed, Jun 4, 2008 at 10:51 AM, Olga Natkovich=20 > <eric91@example.com> wrote: >=20 > > Do you mean that the same query ran faster with some=20 > previous code you=20 > > ran? Which code was that? > > > > Olga > > > > > -----Original Message----- > > > From: eric91@example.com > > > [mailto:eric91@example.com] On Behalf Of Prashanth Pappu > > > Sent: Tuesday, June 03, 2008 5:00 PM > > > To: eric91@example.com > > > Subject: PIG performance > > > > > > I just upgraded my PIG src to top of svn and see really poor=20 > > > performance with group and cogroup queries. > > > > > > Is there a recommended svn version to be used with hadoop 1.17? > > > > > > Prashanth > > > > > >=20 From pig-user-returneric91@example.com Wed Jun 04 21:21:28 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 33402 invoked from network); 4 Jun 2008 21:21:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 21:21:27 -0000 Received: (qmail 27455 invoked by uid 500); 4 Jun 2008 21:21:30 -0000 Delivered-To: eric91@example.com Received: (qmail 27311 invoked by uid 500); 4 Jun 2008 21:21:30 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 27300 invoked by uid 99); 4 Jun 2008 21:21:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 14:21:30 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout2-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 21:20:38 +0000 Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [83.161.120.155]) by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m54LJLcq094927 for <eric91@example.com>; Wed, 4 Jun 2008 14:19:35 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:x-mimeole:content-class:mime-version: content-type:content-transfer-encoding:subject:date:message-id: in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic: thread-index:references:from:to:return-path:x-originalarrivaltime; b=XJRoTCNI1Yd2BuY1ODWoiBXPT8NSw6I9g3yH8XOBYFhmy5MoAQRTmEbvOqFXH/ed Received: from SNV-EXVS09.ds.corp.yahoo.com ([83.161.120.155]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(539)789-3517); Wed, 4 Jun 2008 14:19:26 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: Specifying multiple input paths Date: Wed, 4 Jun 2008 14:18:40 -0700 Message-ID: <eric91@example.com> In-Reply-To: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Specifying multiple input paths Thread-Index: AcjGUzfpne9k83IYQAKcyaqLyhiQPAANNXJw References: <eric91@example.com> From: "Olga Natkovich" <eric91@example.com> To: <eric91@example.com> X-OriginalArrivalTime: 04 Jun 2008 21:19:26.0113 (UTC) FILETIME=[AAA94110:01C8C688] X-Virus-Checked: Checked by ClamAV on apache.org Tom, You are correct that currently we only allow a single glob in the load statement. It would not be hard to extend it to multiple globs. I have created a JIRA for it: https://issues.apache.org/jira/browse/PIG-252; maybe somebody will be interested to contribute a patch.=20 Olga > -----Original Message----- > From: Tom White [mailto:eric91@example.com]=20 > Sent: Wednesday, June 04, 2008 7:56 AM > To: eric91@example.com > Subject: Specifying multiple input paths >=20 > I;m having a problem loading data from multiple paths in Pig.=20 > What I'm trying to do is to load data from a range of dates,=20 > so I would like to specify an input of two globbed paths: >=20 > x =3D LOAD '2008/05/{26,27,28,29,30,31},2008/06/{1,2}' >=20 > Pig doesn't seem to like this though as it's trying to=20 > interpret it as a single path. The best I can do it to use UNION: >=20 > x1 =3D LOAD '2008/05/{26,27,28,29,30,31}' > x2 =3D LOAD '2008/06/{1,2}' > x =3D UNION x1, x2 >=20 > The downside to this is that I want to parameterize my paths,=20 > and having separate script for each number of paths in the=20 > input is cumbersome. >=20 > Is there a better way of doing this? Are there any plans to=20 > support multiple paths, and/or PathFilters? >=20 > Thanks, >=20 > Tom >=20 From pig-user-returneric91@example.com Wed Jun 04 22:02:12 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 51634 invoked from network); 4 Jun 2008 22:02:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 22:02:11 -0000 Received: (qmail 97405 invoked by uid 500); 4 Jun 2008 22:02:08 -0000 Delivered-To: eric91@example.com Received: (qmail 97323 invoked by uid 500); 4 Jun 2008 22:02:08 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 97289 invoked by uid 99); 4 Jun 2008 22:02:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 15:02:08 -0700 X-ASF-Spam-Status: No, hits=3.2 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout1-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 22:01:16 +0000 Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [83.161.120.155]) by mrout1-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id m54M0oc6018739; Wed, 4 Jun 2008 15:00:53 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:x-mimeole:content-class:mime-version: content-type:subject:date:message-id:x-ms-has-attach: x-ms-tnef-correlator:thread-topic:thread-index:from:to:cc:return-path:x-originalarrivaltime; b=fJQXuiDb70eq0/Ei9alJX1rtAqird2nDNYWFk8bGiR8dglnN/6JikSGu5AyxLVuK Received: from SNV-EXVS02.ds.corp.yahoo.com ([83.161.120.155]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(539)789-3517); Wed, 4 Jun 2008 15:00:49 -0700 x-mimeole: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C8C68E.569BDB3C" Subject: Monthly user group meeting Date: Wed, 4 Jun 2008 15:00:01 -0700 Message-ID: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Monthly user group meeting Thread-Index: AcjGjlZI4DSB1mopQ9+2VHBAe1smpQ== From: "Ajay Anand" <eric91@example.com> To: <eric91@example.com>, <eric91@example.com>, <eric91@example.com> Cc: "Jeff Hammerbacher" <eric91@example.com>, "Chad Walters" <eric91@example.com>, "Owen O'Malley" <eric91@example.com> X-OriginalArrivalTime: 04 Jun 2008 22:00:49.0254 (UTC) FILETIME=[72BA8060:01C8C68E] X-Virus-Checked: Checked by ClamAV on apache.org ------_=_NextPart_001_01C8C68E.569BDB3C Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable The next user group meeting is scheduled for June 18th from 6-7:30 pm at the Yahoo! Mission College campus (2821 Mission College, Santa Clara). Registration, driving directions etc are at http://upcoming.yahoo.com/event/760573/ =20 Agenda: 1) Hadoop at Facebook, Hive - Jeff Hammerbacher 2) Using Zookeeper - Ted Dunning, Ben Reed ... And plenty of networking, as always. =20 Ajay =20 ------_=_NextPart_001_01C8C68E.569BDB3C-- From pig-user-returneric91@example.com Wed Jun 04 23:37:52 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 92878 invoked from network); 4 Jun 2008 23:37:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jun 2008 23:37:52 -0000 Received: (qmail 25572 invoked by uid 500); 4 Jun 2008 23:37:54 -0000 Delivered-To: eric91@example.com Received: (qmail 25430 invoked by uid 500); 4 Jun 2008 23:37:54 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 25419 invoked by uid 99); 4 Jun 2008 23:37:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 16:37:54 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric91@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0708.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 23:37:05 +0000 Received: by rv-out-0708.google.com with SMTP id k29so384035rvb.0 for <eric91@example.com>; Wed, 04 Jun 2008 16:37:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:mime-version:content-type:x-google-sender-auth; bh=OYsPwCyyb67DdJ/kSBWtqtw0Kj1Y9cSe1lFBB8AejQE=; b=T4G28C0Dv6wPI0T/JeRByn5I4vuzQJgJo8B2C5B6dWP1KoS/520lEMZ0JIJtANgB7c f0aFAb3Si0xYR/99w0CZDZCW5sHDfEY5v7hCn/b4zuEyqxtT1YKXNcVX3AdHtg+8AhJ6 +cGN4QAdiCmrh3zyKpmj/EsubS1yGIVqbjj1s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:mime-version:content-type :x-google-sender-auth; b=hm0XVEeu+Ka0Wj3mjjXjl1R6P7gV5rz5iJb5mJHn0ONFlgaLjTtlfxD0QE0hyKBzG8 igiHYJpSrNlMQ1RIinIUNyUvO76YNur4PYoeBUPfO8NsnVjxE6HnJPahkwCLHOu59Zgn 77D5XCpRjmbeqqsKlGyB/fd5mwB69B/7brW+Q= Received: by 83.161.120.155 with SMTP id z21mr430742rvf.81.1212622642480; Wed, 04 Jun 2008 16:37:22 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 4 Jun 2008 16:37:22 -0700 (PDT) Message-ID: <eric91@example.com> Date: Wed, 4 Jun 2008 16:37:22 -0700 From: "Prashanth Pappu" <eric91@example.com> Sender: eric91@example.com To: eric91@example.com Subject: Spaces (' ') in data atoms MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_6029_14178922.1212622642429" X-Google-Sender-Auth: 245552b48430613c X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_6029_14178922.1212622642429 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I think there are a few unresolved issues due to lack of explicit type declarations in PIG. Especially with data atoms that have leading or trialing spaces (' '), implicit typecast into strings and integers/floats can lead to unexpected results. Consider the following example - grunt> a = load '/test' using PigStorage(',') as (x,y); grunt> dump a; (1, 2 ) (2, 3 ) (3, 4) (4, 5) grunt> b = load '/test' using PigStorage(',') as (x,y); grunt> dump b; (1, 2 ) (2, 3 ) (3, 4) (4, 5) grunt> a1 = filter a by x==2; grunt> dump a1; (2, 3 ) grunt> b1 = filter b by y==2; grunt> dump b1; (1, 2 ) So, both a and b have tuples that can be filtered with x==2 and y==2 respectively. But, what do we get when we cogroup them? grunt> c = cogroup a by (x) INNER, b by (y) INNER; grunt> dump c; [NOTHING!] This is because COGROUP is treating x and y as strings but FILTER was explicitly asked to treat them as integers/floats with the '==' comparator. This can be quite confusing since COGROUP seems to be using the comparator 'eq' by default! Hence, the following result when we cogroup without the keywords 'INNER' grunt> c2 = cogroup a by (x), b by (y); grunt> dump c2; (1, {(1, 2 )}, {}) (2, {(2, 3 )}, {}) (3, {(3, 4)}, {}) (4, {(4, 5)}, {(3, 4)}) (5, {}, {(4, 5)}) ( 2 , {}, {(1, 2 )}) ( 3 , {}, {(2, 3 )}) I think that we can avoid the confusion by one of two ways (a) Forcing the rule that all data atoms will be stripped of leading and trailing spaces (' '). This is important because data atoms have no explicit type declaration. (b) Or atleast throwing up an error when a string with a leading/trailing space is typecast (implicitly or explicitly) to an integer or float. I.e, only the output of STRING(FLOAT) or STRING(INTEGER) should be allowed as the input of FLOAT(STRING) or INTEGER(STRING) conversions. I see that, if unresolved, this can be an annoying problem as fields in input files often tend to have leading/trailing spaces that are not part of the field separator. Prashanth ------=_Part_6029_14178922.1212622642429-- From pig-user-returneric91@example.com Thu Jun 05 05:45:31 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 4011 invoked from network); 5 Jun 2008 05:45:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jun 2008 05:45:31 -0000 Received: (qmail 8607 invoked by uid 500); 5 Jun 2008 05:45:34 -0000 Delivered-To: eric91@example.com Received: (qmail 8471 invoked by uid 500); 5 Jun 2008 05:45:33 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 8458 invoked by uid 99); 5 Jun 2008 05:45:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2008 22:45:33 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO outbound.mse16.exchange.ms) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 05:44:38 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable x-cr-hashedpuzzle: BJUm BlI9 Bs0/ B0nI CVIa C6uu EGIS Eife E2Ak FXMU GKGO GbX0 IjYy IsTE JPj8 KA/2;1;cABpAGcALQB1AHMAZQByAEAAaQBuAGMAdQBiAGEAdABvAHIALgBhAHAAYQBjAGgAZQAuAG8AcgBnAA==;Sosha1_v1;7;{B529DABA-4810-4C93-B22C-CB9D1F94B0AA};aABhAGkAagB1AG4AQABrAGkAbgBkAHMAaQBnAGgAdAAuAG4AZQB0AA==;Thu, 05 Jun 2008 05:43:55 GMT;bABvAGMAYQBsACAAYgB5AHQAZQBzACAAcgBlAGEAZAAvAHcAcgBpAHQAdABlAG4A x-cr-puzzleid: {B529DABA-4810-4C93-B22C-CB9D1F94B0AA} Content-class: urn:content-classes:message Subject: local bytes read/written Date: Thu, 5 Jun 2008 01:43:55 -0400 Message-ID: <eric91@example.com> In-Reply-To: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: local bytes read/written Thread-Index: AcjF1xhcVqaV59upQ9mdl1QH2F11PgAlFImgABjFG5A= References: <eric91@example.com> <eric91@example.com> From: "Haijun Cao" <eric91@example.com> To: <eric91@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Hi, I just started using pig, it is really fun to write pig query. I noticed in the map reduce job page, it reports bytes read/written from/to local file system, and the number is 2x, 3x of the bytes read/write to hadoop. Just want to understand the internal working of pig a little bit better, what operations read/write to local fs? For what purpose? Is it to the local fs of the data nodes? which directory? Thanks Haijun=20 From pig-user-returneric91@example.com Thu Jun 05 11:23:27 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 4446 invoked from network); 5 Jun 2008 11:23:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jun 2008 11:23:27 -0000 Received: (qmail 43738 invoked by uid 500); 5 Jun 2008 11:23:30 -0000 Delivered-To: eric91@example.com Received: (qmail 43598 invoked by uid 500); 5 Jun 2008 11:23:29 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 43581 invoked by uid 99); 5 Jun 2008 11:23:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 04:23:29 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric91@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wa-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 11:22:40 +0000 Received: by wa-out-1112.google.com with SMTP id m16so405773waf.6 for <eric91@example.com>; Thu, 05 Jun 2008 04:22:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:reply-to :to:subject:in-reply-to:mime-version:content-type:references; bh=sO1CA0R+6vcQYJxfQuqpQDJsq1y1BVIIIKFJZxSKMi4=; b=Ucqej00GTQ3Lr1LmOLjf09I2p86+mPM3HxFx/mzh1CY9bE+vKIy2IgB64Q6UtY7KnA 7vTegTU9p9lCJM/8PKQmIaNKgKoVXls+NGG5GUl4UkZsMqOEGqKV8KmJjA+YOPwE0nen HkSl1xMVE8F6AUYu3ovrsz51IL/GQ/GVn3Qq0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:reply-to:to:subject:in-reply-to:mime-version :content-type:references; b=YQROBoCw7GQxDnI9NsX+gEFn9mjvc2g+ROjH8BxNTtF4wepMTKySLDVLbwcNCkdBG+ qJHCdUnAA5NB85bZxc6nSTU2Uvn882pX+Nu8thhV+kHe2oxPLkzKa2V/tqi6mk/sSex3 bKf/wfSxLSgolcHVNGyBeUIRLNfpJcOD1T8qs= Received: by 83.161.120.155 with SMTP id s1mr1345088wab.13.1212664977600; Thu, 05 Jun 2008 04:22:57 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Thu, 5 Jun 2008 04:22:57 -0700 (PDT) Message-ID: <eric91@example.com> Date: Thu, 5 Jun 2008 21:22:57 +1000 From: "pi song" <eric91@example.com> Reply-To: eric91@example.com To: eric91@example.com Subject: Re: PIG performance In-Reply-To: <eric91@example.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1369_25045167.1212664977592" References: <eric91@example.com> <eric91@example.com> <eric91@example.com> <eric91@example.com> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_1369_25045167.1212664977592 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I think saying "slow down" is too subjective. When we get performance boost in most areas, some smaller areas might perform poorer than before. I saw a Jira issue that discussed about introducing a standard way to benchmark before but didn't see any progress. On Thu, Jun 5, 2008 at 5:05 AM, Olga Natkovich <eric91@example.com> wrote: > We saw some slowdown in the unit tests but not in our end-to-end tests. > > Olga > > > -----Original Message----- > > From: eric91@example.com > > [mailto:eric91@example.com] On Behalf Of Prashanth Pappu > > Sent: Wednesday, June 04, 2008 11:16 AM > > To: eric91@example.com > > Subject: Re: PIG performance > > > > Yes, the same queries were substantially slower with hadoop > > 1.17 and the latest svn of PIG (compared to hadoop1.16 and > > svn 653894). > > > > It is hard for me to copy the exact query as there is a lot > > of context but I zeroed it down to cogroup statements. I can > > try and create an example. But I wanted to check if there is > > a specific svn version that others use with hadoop1.17. Or > > does everyone use top of svn trunk? > > Because upgrading from hadoop1.16 and older svn version to > > latest svn version did degrade the performance quite a bit. > > > > Prashanth > > > > On Wed, Jun 4, 2008 at 10:51 AM, Olga Natkovich > > <eric91@example.com> wrote: > > > > > Do you mean that the same query ran faster with some > > previous code you > > > ran? Which code was that? > > > > > > Olga > > > > > > > -----Original Message----- > > > > From: eric91@example.com > > > > [mailto:eric91@example.com] On Behalf Of Prashanth Pappu > > > > Sent: Tuesday, June 03, 2008 5:00 PM > > > > To: eric91@example.com > > > > Subject: PIG performance > > > > > > > > I just upgraded my PIG src to top of svn and see really poor > > > > performance with group and cogroup queries. > > > > > > > > Is there a recommended svn version to be used with hadoop 1.17? > > > > > > > > Prashanth > > > > > > > > > > ------=_Part_1369_25045167.1212664977592-- From pig-user-returneric91@example.com Thu Jun 05 12:48:02 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 12255 invoked from network); 5 Jun 2008 12:48:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jun 2008 12:48:02 -0000 Received: (qmail 66424 invoked by uid 500); 5 Jun 2008 12:48:05 -0000 Delivered-To: eric91@example.com Received: (qmail 66407 invoked by uid 500); 5 Jun 2008 12:48:05 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 66395 invoked by uid 99); 5 Jun 2008 12:48:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 05:48:05 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric91@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO py-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 12:47:15 +0000 Received: by py-out-1112.google.com with SMTP id u52so330698pyb.13 for <eric91@example.com>; Thu, 05 Jun 2008 05:47:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:reply-to :to:subject:in-reply-to:mime-version:content-type:references; bh=433ncehavY5ZoZo7pnRdA/73quxgwyLIODV6d+UGmrs=; b=jHuQSRGcpfssqsdLaIjnmf5q5iPCRapiROgF56q7bC2XV1XWHC4U6qCzjwhWYUpZeM pTGs0CkWs5RG0OL/AWJJr9gpIgDJeeA6Ls+eiFUvr9T8wJSq8UwlhxjY68btEFlGCkEB iBtKotCzLRQlGS75iTY29erpM3Fp6PfVa54JQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:reply-to:to:subject:in-reply-to:mime-version :content-type:references; b=Vuh+cwAX9dRfkq6wPqA+JWys8GMXaE5Ru0581Y4Eo2i3fS3mmGhfK42X193gR/b7Sm 960rqS2gSe6VrYPsTjJHJm2kya5fFgnN5FxXSY6pu7d/5mMnWCabZHW4lbSm4lY+lcu0 oPQQpQU2PQUzwg4o23iz9oNz2r9Y3mwZtRQcM= Received: by 83.161.120.155 with SMTP id g1mr1345840waf.208.1212670052293; Thu, 05 Jun 2008 05:47:32 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Thu, 5 Jun 2008 05:47:32 -0700 (PDT) Message-ID: <eric91@example.com> Date: Thu, 5 Jun 2008 22:47:32 +1000 From: "pi song" <eric91@example.com> Reply-To: eric91@example.com To: eric91@example.com Subject: Re: Spaces (' ') in data atoms In-Reply-To: <eric91@example.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1820_29576853.1212670052281" References: <eric91@example.com> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_1820_29576853.1212670052281 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline The main theme in Pig development at the moment is developing a new type system which means you will be able to stick typing information to your data. Basically, the problem you mention will not happen if you specify all the metadata. However if you don't, this might happen again and I think dealing with whitespaces is users' responsibility. Do you think this will resolve your problem? Pi On Thu, Jun 5, 2008 at 9:37 AM, Prashanth Pappu <eric91@example.com> wrote: > I think there are a few unresolved issues due to lack of explicit type > declarations in PIG. Especially with data atoms that have leading or > trialing spaces (' '), implicit typecast into strings and integers/floats > can lead to unexpected results. > > Consider the following example - > > grunt> a = load '/test' using PigStorage(',') as (x,y); > grunt> dump a; > (1, 2 ) > (2, 3 ) > (3, 4) > (4, 5) > > grunt> b = load '/test' using PigStorage(',') as (x,y); > grunt> dump b; > (1, 2 ) > (2, 3 ) > (3, 4) > (4, 5) > > grunt> a1 = filter a by x==2; > grunt> dump a1; > (2, 3 ) > > grunt> b1 = filter b by y==2; > grunt> dump b1; > (1, 2 ) > > So, both a and b have tuples that can be filtered with x==2 and y==2 > respectively. > But, what do we get when we cogroup them? > > grunt> c = cogroup a by (x) INNER, b by (y) INNER; > grunt> dump c; > > [NOTHING!] > > This is because COGROUP is treating x and y as strings but FILTER was > explicitly asked to treat them as integers/floats with the '==' comparator. > This can be quite confusing since COGROUP seems to be using the comparator > 'eq' by default! > > Hence, the following result when we cogroup without the keywords 'INNER' > > grunt> c2 = cogroup a by (x), b by (y); > grunt> dump c2; > (1, {(1, 2 )}, {}) > (2, {(2, 3 )}, {}) > (3, {(3, 4)}, {}) > (4, {(4, 5)}, {(3, 4)}) > (5, {}, {(4, 5)}) > ( 2 , {}, {(1, 2 )}) > ( 3 , {}, {(2, 3 )}) > > I think that we can avoid the confusion by one of two ways > > (a) Forcing the rule that all data atoms will be stripped of leading and > trailing spaces (' '). This is important because data atoms have no > explicit > type declaration. > (b) Or atleast throwing up an error when a string with a leading/trailing > space is typecast (implicitly or explicitly) to an integer or float. I.e, > only the output of STRING(FLOAT) or STRING(INTEGER) should be allowed as > the > input of FLOAT(STRING) or INTEGER(STRING) conversions. > > I see that, if unresolved, this can be an annoying problem as fields in > input files often tend to have leading/trailing spaces that are not part of > the field separator. > > Prashanth > ------=_Part_1820_29576853.1212670052281-- From pig-user-returneric91@example.com Thu Jun 05 17:55:05 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 13175 invoked from network); 5 Jun 2008 17:55:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jun 2008 17:55:05 -0000 Received: (qmail 51197 invoked by uid 500); 5 Jun 2008 17:55:08 -0000 Delivered-To: eric91@example.com Received: (qmail 51177 invoked by uid 500); 5 Jun 2008 17:55:08 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 51166 invoked by uid 99); 5 Jun 2008 17:55:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 10:55:08 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of eric91@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0708.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 17:54:18 +0000 Received: by rv-out-0708.google.com with SMTP id k29so821344rvb.0 for <eric91@example.com>; Thu, 05 Jun 2008 10:54:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:in-reply-to:mime-version:content-type:references :x-google-sender-auth; bh=Z6H8YiLyO581Zi/RLXoaDFVH/9eYQfDFsYPwJqIq9WA=; b=FabwgKgnNE5Al2ee2iMvpGzSVpnfJoYTXPaVG69ILHTvsaOsnmAKLsCU4fYSXNbhmx XRq+cqOCGYSvhCdtMg1zO6KvhGexbGH76lOH8pHI50gpbiF034PGKuALbdWPWB2SPeFA bcCjoO5GDK+77UKgVm2Z31m7Jk8FMWTcUbaXU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:in-reply-to:mime-version :content-type:references:x-google-sender-auth; b=xWpFGQ3H315xRB1WPsIsjbpAA/n/yuXWXETK5+NLuThCaouDLjpR8gWokY07j2Q8BP IojHOEZuOv1xe3HbDU7pUcneBL4EoDIFOfuKciXKQdFVNehUIoUB9ot8jyV9TaZEf1UC CKQrmust8Ok4cT9gikN0MqBoHaapSGQItG+70= Received: by 83.161.120.155 with SMTP id g18mr1068138rvj.298.1212688474657; Thu, 05 Jun 2008 10:54:34 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Thu, 5 Jun 2008 10:54:34 -0700 (PDT) Message-ID: <eric91@example.com> Date: Thu, 5 Jun 2008 10:54:34 -0700 From: "Prashanth Pappu" <eric91@example.com> Sender: eric91@example.com To: eric91@example.com, eric91@example.com Subject: Re: Spaces (' ') in data atoms In-Reply-To: <eric91@example.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2499_29833136.1212688474638" References: <eric91@example.com> <eric91@example.com> X-Google-Sender-Auth: eb09b828caaa68dc X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_2499_29833136.1212688474638 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Explicit type definitions will definitely help. But I still think that in this example - grunt> b1 = filter b by y==2; grunt> dump b1; (1, 2 ) Either we should throw an error or not return any results. Because the '==' operator is implicitly stripping the spaces in ' 2 ' and determining that ' 2 ' == 2! (The result of (' 2 ' ==2) should be the same as say, ('X2X' ==2)) And cogroup isn't removing the spaces and hence ' 2 ' ne 2. Overall, where we do not have explicit type declarations, the behavior of FILTER, COGROUP etc should be consistent - either they all remove the spaces or none of them do. The rest, definitely, is users' responsibility. Prashanth On Thu, Jun 5, 2008 at 5:47 AM, pi song <eric91@example.com> wrote: > The main theme in Pig development at the moment is developing a new type > system which means you will be able to stick typing information to your > data. Basically, the problem you mention will not happen if you specify all > the metadata. However if you don't, this might happen again and I think > dealing with whitespaces is users' responsibility. > > Do you think this will resolve your problem? > > Pi > > On Thu, Jun 5, 2008 at 9:37 AM, Prashanth Pappu <eric91@example.com> > wrote: > > > I think there are a few unresolved issues due to lack of explicit type > > declarations in PIG. Especially with data atoms that have leading or > > trialing spaces (' '), implicit typecast into strings and integers/floats > > can lead to unexpected results. > > > > Consider the following example - > > > > grunt> a = load '/test' using PigStorage(',') as (x,y); > > grunt> dump a; > > (1, 2 ) > > (2, 3 ) > > (3, 4) > > (4, 5) > > > > grunt> b = load '/test' using PigStorage(',') as (x,y); > > grunt> dump b; > > (1, 2 ) > > (2, 3 ) > > (3, 4) > > (4, 5) > > > > grunt> a1 = filter a by x==2; > > grunt> dump a1; > > (2, 3 ) > > > > grunt> b1 = filter b by y==2; > > grunt> dump b1; > > (1, 2 ) > > > > So, both a and b have tuples that can be filtered with x==2 and y==2 > > respectively. > > But, what do we get when we cogroup them? > > > > grunt> c = cogroup a by (x) INNER, b by (y) INNER; > > grunt> dump c; > > > > [NOTHING!] > > > > This is because COGROUP is treating x and y as strings but FILTER was > > explicitly asked to treat them as integers/floats with the '==' > comparator. > > This can be quite confusing since COGROUP seems to be using the > comparator > > 'eq' by default! > > > > Hence, the following result when we cogroup without the keywords 'INNER' > > > > grunt> c2 = cogroup a by (x), b by (y); > > grunt> dump c2; > > (1, {(1, 2 )}, {}) > > (2, {(2, 3 )}, {}) > > (3, {(3, 4)}, {}) > > (4, {(4, 5)}, {(3, 4)}) > > (5, {}, {(4, 5)}) > > ( 2 , {}, {(1, 2 )}) > > ( 3 , {}, {(2, 3 )}) > > > > I think that we can avoid the confusion by one of two ways > > > > (a) Forcing the rule that all data atoms will be stripped of leading and > > trailing spaces (' '). This is important because data atoms have no > > explicit > > type declaration. > > (b) Or atleast throwing up an error when a string with a leading/trailing > > space is typecast (implicitly or explicitly) to an integer or float. I.e, > > only the output of STRING(FLOAT) or STRING(INTEGER) should be allowed as > > the > > input of FLOAT(STRING) or INTEGER(STRING) conversions. > > > > I see that, if unresolved, this can be an annoying problem as fields in > > input files often tend to have leading/trailing spaces that are not part > of > > the field separator. > > > > Prashanth > > > ------=_Part_2499_29833136.1212688474638-- From pig-user-returneric91@example.com Thu Jun 05 18:50:29 2008 Return-Path: <pig-user-returneric91@example.com> Delivered-To: eric91@example.com Received: (qmail 64372 invoked from network); 5 Jun 2008 18:50:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jun 2008 18:50:29 -0000 Received: (qmail 97082 invoked by uid 500); 5 Jun 2008 18:50:32 -0000 Delivered-To: eric91@example.com Received: (qmail 97072 invoked by uid 500); 5 Jun 2008 18:50:32 -0000 Mailing-List: contact eric91@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:eric91@example.com> List-Unsubscribe: <mailto:eric91@example.com> List-Post: <mailto:eric91@example.com> List-Id: <pig-user.incubator.apache.org> Reply-To: eric91@example.com Delivered-To: mailing list eric91@example.com Received: (qmail 97056 invoked by uid 99); 5 Jun 2008 18:50:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 11:50:32 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mbx1.veoh) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Jun 2008 18:49:36 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C8C73C.F3033C86" Subject: RE: Spaces (' ') in data atoms Date: Thu, 5 Jun 2008 11:47:29 -0700 Message-ID: <eric91@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Spaces (' ') in data atoms Thread-Index: AcjHNUnKyiZC0I+zRCa9CD+wXPmwnwAB1FT3 References: <eric91@example.com><eric91@example.com> <eric91@example.com> From: "Ted Dunning" <eric91@example.com> To: <eric91@example.com>, <eric91@example.com>, <eric91@example.com> X-Virus-Checked: Checked by ClamAV on apache.org ------_=_NextPart_001_01C8C73C.F3033C86 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable is it possible to ask cogroup to group on a function such as a cast to = integer? I think that cogroup is pretty strongly wedded to eq, but I would also = think that if there is some sort of way to coerce the group key to a = desired type, then you can force eq to give you the behavior you want. Pig isn't the first language to have confusion between objects and their = string representation. I pretty commonly have unit tests that fail with = messages like "Expected 5 but got 5" where one 5 is 5 and the other is = "5".=20 -----Original Message----- From: eric91@example.com on behalf of Prashanth Pappu Sent: Thu 6/5/2008 10:54 AM To: eric91@example.com; eric91@example.com Subject: Re: Spaces (' ') in data atoms =20 Explicit type definitions will definitely help. But I still think that = in this example - grunt> b1 =3D filter b by y=3D=3D2; grunt> dump b1; (1, 2 ) Either we should throw an error or not return any results. Because the = '=3D=3D' operator is implicitly stripping the spaces in ' 2 ' and determining = that ' 2 ' =3D=3D 2! (The result of (' 2 ' =3D=3D2) should be the same as say, = ('X2X' =3D=3D2)) And cogroup isn't removing the spaces and hence ' 2 ' ne 2. Overall, where we do not have explicit type declarations, the behavior = of FILTER, COGROUP etc should be consistent - either they all remove the = spaces or none of them do. The rest, definitely, is users' responsibility. Prashanth On Thu, Jun 5, 2008 at 5:47 AM, pi song <eric91@example.com> wrote: > The main theme in Pig development at the moment is developing a new = type > system which means you will be able to stick typing information to = your > data. Basically, the problem you mention will not happen if you = specify all > the metadata. However if you don't, this might happen again and I = think > dealing with whitespaces is users' responsibility. > > Do you think this will resolve your problem? > > Pi > > On Thu, Jun 5, 2008 at 9:37 AM, Prashanth Pappu = <eric91@example.com> > wrote: > > > I think there are a few unresolved issues due to lack of explicit = type > > declarations in PIG. Especially with data atoms that have leading or > > trialing spaces (' '), implicit typecast into strings and = integers/floats > > can lead to unexpected results. > > > > Consider the following example - > > > > grunt> a =3D load '/test' using PigStorage(',') as (x,y); > > grunt> dump a; > > (1, 2 ) > > (2, 3 ) > > (3, 4) > > (4, 5) > > > > grunt> b =3D load '/test' using PigStorage(',') as (x,y); > > grunt> dump b; > > (1, 2 ) > > (2, 3 ) > > (3, 4) > > (4, 5) > > > > grunt> a1 =3D filter a by x=3D=3D2; > > grunt> dump a1; > > (2, 3 ) > > > > grunt> b1 =3D filter b by y=3D=3D2; > > grunt> dump b1; > > (1, 2 ) > > > > So, both a and b have tuples that can be filtered with x=3D=3D2 and = y=3D=3D2 > > respectively. > > But, what do we get when we cogroup them? > > > > grunt> c =3D cogroup a by (x) INNER, b by (y) INNER; > > grunt> dump c; > > > > [NOTHING!] > > > > This is because COGROUP is treating x and y as strings but FILTER = was > > explicitly asked to treat them as integers/floats with the '=3D=3D' > comparator. > > This can be quite confusing since COGROUP seems to be using the > comparator > > 'eq' by default! > > > > Hence, the following result when we cogroup without the keywords = 'INNER' > > > > grunt> c2 =3D cogroup a by (x), b by (y); > > grunt> dump c2; > > (1, {(1, 2 )}, {}) > > (2, {(2, 3 )}, {}) > > (3, {(3, 4)}, {}) > > (4, {(4, 5)}, {(3, 4)}) > > (5, {}, {(4, 5)}) > > ( 2 , {}, {(1, 2 )}) > > ( 3 , {}, {(2, 3 )}) > > > > I think that we can avoid the confusion by one of two ways > > > > (a) Forcing the rule that all data atoms will be stripped of leading = and >
{'source': 'Apache Software Foundation Public Mail Archives'}