From hive-userrobert49@example.com Sat Aug 01 01:28:38 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 6815 invoked from network); 1 Aug 2009 01:28:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Aug 2009 01:28:38 -0000 Received: (qmail 99453 invoked by uid 500); 1 Aug 2009 01:02:01 -0000 Delivered-To: robert49@example.com Received: (qmail 99418 invoked by uid 500); 1 Aug 2009 01:02:01 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 99409 invoked by uid 99); 1 Aug 2009 01:02:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Aug 2009 01:02:01 +0000 X-ASF-Spam-Status: No, hits=3.0 required=10.0 tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mx2.netflix.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Aug 2009 01:01:52 +0000 Received: from bmfilter01.netflix.com ([83.161.120.155]) by mx2.netflix.com (83.161.120.15560308/8.12.11) with ESMTP id n7111VQt002705 for <robert49@example.com>; Fri, 31 Jul 2009 18:01:31 -0700 X-AuditID: 0a402051-b7cabae00000094d-00-4a7393ea872a Received: from message.netflix.com (message.netflix.com [83.161.120.155]) by (Symantec Mail Security) with SMTP id 3A.08.02381.AE3937A4; Fri, 31 Jul 2009 18:01:30 -0700 (PDT) Received: from Superfly.netflix.com ([83.161.120.155]) by message.netflix.com with Microsoft SMTPSVC(448)341-0601); Fri, 31 Jul 2009 18:01:30 -0700 Received: from 83.161.120.155 ([83.161.120.155]) by superfly.netflix.com ([83.161.120.155]) with Microsoft Exchange Server HTTP-DAV ; Sat, 1 Aug 2009 01:01:29 +0000 User-Agent: Microsoft-Entourage/83.161.120.155605 Date: Fri, 31 Jul 2009 18:01:29 -0700 Subject: Load data inpath w/o overwrite does not move the files From: Eva Tse <robert49@example.com> To: "robert49@example.com" <robert49@example.com> Message-ID: <robert49@example.com> Thread-Topic: Load data inpath w/o overwrite does not move the files Thread-Index: AcoSQ5oKYcXvxaUfVkGrGKzcoDYVdA== Mime-version: 1.0 Content-type: multipart/alternative; boundary="B_3331908090_13942913" X-OriginalArrivalTime: 01 Aug 2009 01:01:30.0188 (UTC) FILETIME=[9AC040C0:01CA1243] X-Brightmail-Tracker: AAAAAA== X-Virus-Checked: Checked by ClamAV on apache.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3331908090_13942913 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable We discovered a problem where loading into a new partition in hive w/o specifying =8Coverwrite=B9 doesn=B9t work. If the specified partition does not exist yet, running the following statement would not move the data files, but only create the directory on HDFS. hive -e "load data inpath '/user/output/nccp/P20090731T233339/sessionsFacts_L20090718T03-r-00000' int= o table nccp_session_facts partition (dateint =3D 20090718, hour =3D 4) If we run the same statement one more time, it will move the file. However, if we run the same statement with =8Coverwrite=B9, it will create the directory and move the file just fine. We don=B9t think it is expected behavior. Hence, would like to know if this i= s a bug. Hive version: r795559 with 487.4 patch. Hadoop: 0.20 Thanks a lot! Eva. --B_3331908090_13942913 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable <HTML> <HEAD> <TITLE>Load data inpath w/o overwrite does not move the files</TITLE> </HEAD> <BODY> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:11pt= '><BR> </SPAN></FONT><FONT SIZE=3D"2"><FONT FACE=3D"Arial"><SPAN STYLE=3D'font-size:10pt= '><BR> We discovered a problem where loading into a new partition in hive w/o spec= ifying &#8216;overwrite&#8217; doesn&#8217;t work. <BR> <BR> If the specified partition does not exist yet, running the following statem= ent would not move the data files, but only create the directory on HDFS.<BR= > hive -e &quot;load data inpath '/user/output/nccp/P20090731T233339/sessions= Facts_L20090718T03-r-00000' into table nccp_session_facts partition (dateint= =3D 20090718, hour =3D 4)<BR> <BR> If we run the same statement one more time, it will move the file.<BR> <BR> However, if we run the same statement with &#8216;overwrite&#8217;, it will= create the directory and move the file just fine.<BR> <BR> We don&#8217;t think it is expected behavior. Hence, would like to know if = this is a bug.<BR> <BR> <BR> Hive version: r795559 with 487.4 patch.<BR> Hadoop: 0.20<BR> <BR> <BR> Thanks a lot!<BR> Eva.<BR> </SPAN></FONT></FONT> </BODY> </HTML> --B_3331908090_13942913-- From hive-userrobert49@example.com Sun Aug 02 15:57:53 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 65001 invoked from network); 2 Aug 2009 15:57:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Aug 2009 15:57:53 -0000 Received: (qmail 77774 invoked by uid 500); 2 Aug 2009 15:57:58 -0000 Delivered-To: robert49@example.com Received: (qmail 77687 invoked by uid 500); 2 Aug 2009 15:57:58 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 77678 invoked by uid 99); 2 Aug 2009 15:57:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Aug 2009 15:57:58 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wf-out-1314.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 02 Aug 2009 15:57:48 +0000 Received: by wf-out-1314.google.com with SMTP id 23so843720wfg.2 for <robert49@example.com>; Sun, 02 Aug 2009 08:57:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=61LH5tSspIHIuGj1y2hbq+Wlxgh62+FbKqWULyRj4qs=; b=iYWVRTnYdON85jwZRTiu4uCD942dBchvg/y2Q33OeMpw2T1ikhyXHRjgAuVJr8qdw8 tNzjiqVT4fGN+mKmx3ljk0+2g5lfMM/iH6eQrpSJiqu1VRQQPQzAcWB2HzrQglO8INqr j/KZ5aNFRrV/Bnz/3ffSWWwjV4mRbrG7OrXcA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=a0ZDOvbOZUa9Cz6HJUKnrmTfigA8z2/9b+R1iI3Zl+kCW8Di62xLiC571OW0HNBWbv bFx01l65L/D7EaOctbqOtCbHkEcEJwj8b1ybgr/zSOYKBUCVmTjYKW+3P/ZxlVxNtgsw aA64Mvtn0OIeJTahpehCxybHeju4XamAmuZps= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 2mr208597wfk.283.1249228647117; Sun, 02 Aug 2009 08:57:27 -0700 (PDT) Date: Sun, 2 Aug 2009 08:57:27 -0700 Message-ID: <robert49@example.com> Subject: How Can I debug Hive in local enviroment ? From: zhang jianfeng <robert49@example.com> To: robert49@example.com Content-Type: multipart/alternative; boundary=000e0cd3313ac1338b04702ab54a X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd3313ac1338b04702ab54a Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi all, I=E2=80=99d like to use my own serde to load my data, so I want to first te= st it in local mode. Does Hive has a local mode like Pig ? So that I can debug the program in eclipse ? Thank you. Jeff zhang --000e0cd3313ac1338b04702ab54a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p><span lang=3D"EN-US">Hi all,</span></p> <p><span lang=3D"EN-US">=C2=A0</span></p> <p><span lang=3D"EN-US">I=E2=80=99d like to use my own serde to load my data, so I want to first test it in local mode. Does Hive has a local mode = like Pig ?</span></p> <p><span lang=3D"EN-US">So that I can debug the program in eclipse ?</span></p> <p><span lang=3D"EN-US">Thank you.</span></p> <p><span lang=3D"EN-US">=C2=A0</span></p> <p><span lang=3D"EN-US">Jeff zhang</span></p> <p><span lang=3D"EN-US">=C2=A0</span></p> --000e0cd3313ac1338b04702ab54a-- From hive-userrobert49@example.com Mon Aug 03 16:59:09 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 77693 invoked from network); 3 Aug 2009 16:59:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Aug 2009 16:59:09 -0000 Received: (qmail 38534 invoked by uid 500); 3 Aug 2009 16:59:14 -0000 Delivered-To: robert49@example.com Received: (qmail 38476 invoked by uid 500); 3 Aug 2009 16:59:14 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 38467 invoked by uid 99); 3 Aug 2009 16:59:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 16:59:13 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-yw0-f202.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 16:59:04 +0000 Received: by ywh40 with SMTP id 40so4013038ywh.29 for <robert49@example.com>; Mon, 03 Aug 2009 09:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=gZcU9UVuhG0g6GIU9vUiGuyxaMJj6TLpW6CnbFOx/nA=; b=Dmpy161ZIjPDwGkDXuy46K17ftreAPj0SkjsmE9ll7VQfqIsI1pwn5KIIk9dq04gPV ugt5GmEngSk0tENBp3AMwyR3MpJRHupuQV2w2nUR8pEfQykbjQVwBgBDuIaq9jV6COH5 oIh+xloVkU9aKeMWUsjAOXa8w2R96fU/qF3qg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=LTz4HwAq9k6roH+bWlTuezhn6jD2Pzhh0kwMOxFw6Z9UcLygSsWVWM6y6/MhsFWITO xGtyaFhR2bfDuWwCpnckIq7IUtQj0RovYJXhhqd8y3ElkISwjk0jMoMbZfjcXgKam40x nFQFHyp78k7dviTW6KkJZZyXsXCPjfUsX0kyU= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id i16mr1768385and.85.1249318723851; Mon, 03 Aug 2009 09:58:43 -0700 (PDT) In-Reply-To: <robert49@example.com> References: <robert49@example.com> Date: Mon, 3 Aug 2009 09:58:43 -0700 Message-ID: <robert49@example.com> Subject: Re: How Can I debug Hive in local enviroment ? From: Zheng Shao <robert49@example.com> To: robert49@example.com Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi Jianfeng, That is not currently supported, because Hive starts a new JVM to run the local map-reduce job in local mode. If you are interested in more details on how to modify it, you can take a look at the TaskFactory.java:98-102. We just need to always use ExecDriver (instead of MapRedTask) for that to happen. The recommended way to test a serde is to write a unit test for it. You can look for TestBinarySortableSerDe.java in the code base. Zheng On Sun, Aug 2, 2009 at 8:57 AM, zhang jianfeng<robert49@example.com> wrote: > Hi all, > > > > I=92d like to use my own serde to load my data, so I want to first test i= t in > local mode. Does Hive has a local mode like Pig ? > > So that I can debug the program in eclipse ? > > Thank you. > > > > Jeff zhang > > --=20 Yours, Zheng From hive-userrobert49@example.com Mon Aug 03 19:08:29 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 57259 invoked from network); 3 Aug 2009 19:08:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Aug 2009 19:08:29 -0000 Received: (qmail 15523 invoked by uid 500); 3 Aug 2009 19:08:34 -0000 Delivered-To: robert49@example.com Received: (qmail 15481 invoked by uid 500); 3 Aug 2009 19:08:34 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 15472 invoked by uid 99); 3 Aug 2009 19:08:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 19:08:34 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-px0-f198.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Aug 2009 19:08:25 +0000 Received: by pxi36 with SMTP id 36so2681498pxi.2 for <robert49@example.com>; Mon, 03 Aug 2009 12:08:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=HS8PU6gamnYzroUroJGXJBxGJAsHRGB+Kfateb0GYIE=; b=QfLBSPJKYAHW0xWLLWcv24yWjU1B65FJwQV+p+x5TNtON41oo+Aj2cEKEqKnI8uq3E TzONBQ59W7yur22j1HOLEP63MFyygUEJe95SBMU0HTFmb3AE+E0AaTyqNbvPBDQSPgkp 76rDz9OLpHiZBKuHogZc+3kfqVC0frv548lxM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=h0Jc/rHYBGF2nkIEimBDF+P9gIeVThEdZ4137YQ+AwODaYvVH7Jx2iUldS+JuKt2Eh 3Vk/NV/F6h2x0a9jaBrhyw6givBjazORNq4deWsatrMTdT9ns9rAcvlv8V03FCe53urb whOwxAxY4/dlyP9mWnotoa6fx1ZbWjBw3cTR4= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id w11mr8606611wal.121.1249326483860; Mon, 03 Aug 2009 12:08:03 -0700 (PDT) In-Reply-To: <robert49@example.com> References: <AcoSQ5oKYcXvxaUfVkGrGKzcoDYVdA==> <robert49@example.com> Date: Mon, 3 Aug 2009 12:08:03 -0700 Message-ID: <robert49@example.com> Subject: Re: Load data inpath w/o overwrite does not move the files From: Zheng Shao <robert49@example.com> To: robert49@example.com Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org I reproduced the bug and opened HIVE-718. Zheng On Fri, Jul 31, 2009 at 6:01 PM, Eva Tse<robert49@example.com> wrote: > > > We discovered a problem where loading into a new partition in hive w/o > specifying =91overwrite=92 doesn=92t work. > > If the specified partition does not exist yet, running the following > statement would not move the data files, but only create the directory on > HDFS. > hive -e "load data inpath > '/user/output/nccp/P20090731T233339/sessionsFacts_L20090718T03-r-00000' i= nto > table nccp_session_facts partition (dateint =3D 20090718, hour =3D 4) > > If we run the same statement one more time, it will move the file. > > However, if we run the same statement with =91overwrite=92, it will creat= e the > directory and move the file just fine. > > We don=92t think it is expected behavior. Hence, would like to know if th= is is > a bug. > > > Hive version: r795559 with 487.4 patch. > Hadoop: 0.20 > > > Thanks a lot! > Eva. > --=20 Yours, Zheng From hive-userrobert49@example.com Tue Aug 04 03:02:56 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 52052 invoked from network); 4 Aug 2009 03:02:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 03:02:56 -0000 Received: (qmail 86624 invoked by uid 500); 4 Aug 2009 03:03:01 -0000 Delivered-To: robert49@example.com Received: (qmail 86574 invoked by uid 500); 4 Aug 2009 03:03:01 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 86565 invoked by uid 99); 4 Aug 2009 03:03:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 03:03:01 +0000 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=HTML_MESSAGE,SPF_PASS,SUBJECT_FUZZY_TION X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f210.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 03:02:50 +0000 Received: by bwz6 with SMTP id 6so2991968bwz.29 for <robert49@example.com>; Mon, 03 Aug 2009 20:02:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=IYyh9LvmCDUoMV+yv8AYajOA4J0K4GHYraQQWdJVnyE=; b=U9TvjmYzeaRnc7n7dJ1Ud3K3BaQKg2kT7x0YjiNl4NHyTpEGUF+16rskowN5DC6uEF CTLo+wNSV+kSwd+S9E5ZM0gBwlGYZ/RdLWXZBCSp5KNbW+QxW/q2+zEGnxepjpQyfOa8 EHCipst/BPy/iBH7xO3hp/v5vi8tMLNQqpQpA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=CVf0+smfkt/2z5pSVgvijT+W8lf1hBIM9JNvdO9M2arAz2ahVU8aL1nGScj8DTA9l6 f0E4ZbR4ujaQ3QUntDtIUWgVyCZT1rkbtiH8Tv2/pLk4r2VvqXCqPFyrRfwLiIpeHCoZ iS7egUFWXgYbS0TqhoTw+wCUNlJOtf1OWX2CQ= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id d12mr677119hbb.152.1249354948606; Mon, 03 Aug 2009 20:02:28 -0700 (PDT) Date: Tue, 4 Aug 2009 11:02:28 +0800 Message-ID: <robert49@example.com> Subject: why insert overwrite table tmp partition(dt=1) select bar, foo from pokes NEEDS 2 MR JOBS? From: Min Zhou <robert49@example.com> To: hive-user <robert49@example.com> Content-Type: multipart/alternative; boundary=001485f7721ce8fa430470481d44 X-Virus-Checked: Checked by ClamAV on apache.org --001485f7721ce8fa430470481d44 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit I thought one map only job is ok. try hive> explain insert overwrite table tmp partition(dt=1) select bar, foo from pokes; Thanks, Min -- My research interests are distributed systems, parallel computing and bytecode based virtual machine. My profile: http://www.linkedin.com/in/coderplay My blog: http://coderplay.javaeye.com --001485f7721ce8fa430470481d44 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit I thought one map only job is ok. try<br>hive&gt; explain insert overwrite table tmp partition(dt=1) select bar, foo from pokes;<br><br><div><br> Thanks,<br>Min</div>-- <br>My research interests are distributed systems, parallel computing and bytecode based virtual machine.<br><br>My profile:<br><a href="http://www.linkedin.com/in/coderplay">http://www.linkedin.com/in/coderplay</a><br> My blog:<br><a href="http://coderplay.javaeye.com">http://coderplay.javaeye.com</a><br> --001485f7721ce8fa430470481d44-- From hive-userrobert49@example.com Tue Aug 04 03:58:30 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 61416 invoked from network); 4 Aug 2009 03:58:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 03:58:30 -0000 Received: (qmail 28328 invoked by uid 500); 4 Aug 2009 03:58:35 -0000 Delivered-To: robert49@example.com Received: (qmail 28269 invoked by uid 500); 4 Aug 2009 03:58:35 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 28259 invoked by uid 99); 4 Aug 2009 03:58:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 03:58:35 +0000 X-ASF-Spam-Status: No, hits=0.4 required=10.0 tests=SPF_PASS,SUBJECT_FUZZY_TION X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-yw0-f202.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 03:58:23 +0000 Received: by ywh40 with SMTP id 40so4551537ywh.29 for <robert49@example.com>; Mon, 03 Aug 2009 20:58:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=SLSicICAC2yOay4mjFgwTRyMneoMWHIGJNWlNiQUNw8=; b=uXd8ZyFFVrN2JLMOmRXzT0r2sYrUX4lMSa0dlGs+8l+kxlw4k8WXWph6RIwGTxOW/X PrDPE23w7jG0n2I4YkgTY5OxEk5TaDTxE8M7K+d7Pmbyf6+ZRuVL3LBQqinlqvFtzwjj 6xuKA9cpHU/Gf13EO3XztdvVsWurXXwBbDrvg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=RUReUIGEiHMW0Hn13+d7s5LcjnkNluy8v7oegG1y2dP86Kp6j7itvyefmLXdeDkFps 7JH2OqTwT5dUSpkxC0/FN7KkqDIyR20dm0trXlZHeWkeOi5TOrM9vm6a5mEVf5DSagDI o0ENL4ZZEWKSClOrhroSqqA1KjQ7wAGTUqBnY= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id x17mr8232509ank.25.1249358282034; Mon, 03 Aug 2009 20:58:02 -0700 (PDT) In-Reply-To: <robert49@example.com> References: <robert49@example.com> Date: Mon, 3 Aug 2009 20:58:01 -0700 Message-ID: <robert49@example.com> Subject: Re: why insert overwrite table tmp partition(dt=1) select bar, foo from pokes NEEDS 2 MR JOBS? From: Zheng Shao <robert49@example.com> To: robert49@example.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi Min, We recently added a capability to Hive to merge small output files. You can do the following to disable that feature: set hive.merge.mapfiles=false; OR you can adjust the following parameter to determine when the additional merge job should run: set hive.merge.size.per.task=256000000; By default it's 256MB which means if the average output of a mapper is smaller than 256MB, an additional job will run. You can set that number to something like 64MB if you want. Zheng On Mon, Aug 3, 2009 at 8:02 PM, Min Zhou<robert49@example.com> wrote: > I thought one map only job is ok. try > hive> explain insert overwrite table tmp partition(dt=1) select bar, foo > from pokes; > > > Thanks, > Min > -- > My research interests are distributed systems, parallel computing and > bytecode based virtual machine. > > My profile: > http://www.linkedin.com/in/coderplay > My blog: > http://coderplay.javaeye.com > -- Yours, Zheng From hive-userrobert49@example.com Tue Aug 04 07:48:01 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 21435 invoked from network); 4 Aug 2009 07:48:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 07:48:01 -0000 Received: (qmail 4294 invoked by uid 500); 4 Aug 2009 07:48:06 -0000 Delivered-To: robert49@example.com Received: (qmail 4264 invoked by uid 500); 4 Aug 2009 07:48:06 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 4243 invoked by uid 99); 4 Aug 2009 07:48:02 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 07:48:02 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-vw0-f202.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 07:47:52 +0000 Received: by vws40 with SMTP id 40so1296963vws.2 for <robert49@example.com>; Tue, 04 Aug 2009 00:47:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=L9/m/jVp9dUgUtyBiyvBIEqn/IS3H2Xq5kPcpiJ+RmE=; b=H4EXKxgstPPk+1ukA02vKyvGbOy9WlEPbJnp7c4iRijwEFWCNc56B8jrxORfA4dQwq v+TlmlvkAQUmRb5g4E3OP1NvC79mxUdzrK6SVrUQ4fpY5AuhSP/PN4V15SwbUVT30IAj 5TeaQIFYZR2Y+n4dcXZ7Zc8wJ5f3eeIRshumw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=iCOsQOaxyJ6dpfJRv6ZP6wgi+ZJbIelAItiY5u8Do6qf7a8fvD4OtrizBv1UtH/aVn gAJowTc8gVsddmkVT2+tIK+vDmuaSK/PelMeGiUjTINjKJSsCUDX6y8ctChocyD2lRH5 gi5aeRHkYA9N6hC5gM5GuQVzlK7N3U8Vmsaww= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id m20mr6159553vcn.40.1249372051713; Tue, 04 Aug 2009 00:47:31 -0700 (PDT) Date: Tue, 4 Aug 2009 13:17:31 +0530 Message-ID: <robert49@example.com> Subject: JDBC: Infinite while(rs.next()) loop From: Saurabh Nanda <robert49@example.com> To: robert49@example.com Content-Type: multipart/alternative; boundary=0016e645fabc55d01b04704c190f X-Virus-Checked: Checked by ClamAV on apache.org --0016e645fabc55d01b04704c190f Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Here's what I'm trying: ResultSet rs=st.executeQuery("show tables"); while(rs.next()) { System.out.println(rs.getString(1)); } The while loop never terminates, after going through the list of tables, it keeps printing the last table name over & over again. Am I doing something wrong over here, or have I hit a bug? I'm on hive-0.3.0-hadoop-0.18.0-bin Saurabh. -- http://nandz.blogspot.com http://foodieforlife.blogspot.com --0016e645fabc55d01b04704c190f Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Here&#39;s what I&#39;m trying:<br><br>=A0=A0=A0=A0=A0=A0 ResultSet rs=3Dst= .executeQuery(&quot;show tables&quot;);<br>=A0=A0=A0=A0=A0=A0=A0 while(rs.n= ext()) {<br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 System.out.println(rs.getStri= ng(1));<br>=A0=A0=A0=A0=A0=A0=A0 }<br clear=3D"all"><br>The while loop neve= r terminates, after going through the list of tables, it keeps printing the= last table name over &amp; over again. Am I doing something wrong over her= e, or have I hit a bug? I&#39;m on hive-0.3.0-hadoop-0.18.0-bin<br> <br>Saurabh.<br>-- <br><a href=3D"http://nandz.blogspot.com">http://nandz.b= logspot.com</a><br><a href=3D"http://foodieforlife.blogspot.com">http://foo= dieforlife.blogspot.com</a><br> --0016e645fabc55d01b04704c190f-- From hive-userrobert49@example.com Tue Aug 04 12:44:00 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 27068 invoked from network); 4 Aug 2009 12:43:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 12:43:59 -0000 Received: (qmail 56249 invoked by uid 500); 4 Aug 2009 12:44:04 -0000 Delivered-To: robert49@example.com Received: (qmail 56196 invoked by uid 500); 4 Aug 2009 12:44:04 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 56187 invoked by uid 99); 4 Aug 2009 12:44:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 12:44:04 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO qw-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 12:43:55 +0000 Received: by qw-out-2122.google.com with SMTP id 8so1898094qwh.35 for <robert49@example.com>; Tue, 04 Aug 2009 05:43:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=pafy4OSE0tSgm9ivWi/SvlzOL+hHk5jiWMk3awzB9WU=; b=SObvlCGOh0CPnz1fFVguBDt19w81Sjru+neQGeCPbzgxBtZeWfqHYfsx/qz3rf0xmf b7tm/1tWcxNcZugXCKQNafObiAyAsROVrjSsUlyrKaMZVpqRaNPkfKWzQAJ9wsnAC8XE Awu5+drAewcACYpAH/y9YlKcsm/kq9kmJHpCU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=S1u/zX2hp3+y41foC3Lbsr+KzNaUjavxKIdMVoeVUty+speBvFw1+gXmhtFml3LEV+ zeZShQRFH60eRLLEKLZ/qZ2N9y6VF1FNgZPu4tIkCgN8wqAerlZ7CxUmXpZ4JwT4ZFWd iAfw3t0sPglbxo4roPk5YxcrABXxZVQimloow= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id b4mr3728429vck.119.1249389814146; Tue, 04 Aug 2009 05:43:34 -0700 (PDT) Date: Tue, 4 Aug 2009 18:13:34 +0530 Message-ID: <robert49@example.com> Subject: Passing parameters to custom map/reduce scripts From: Saurabh Nanda <robert49@example.com> To: robert49@example.com Content-Type: multipart/alternative; boundary=0016e64698200ef05e0470503ca1 X-Virus-Checked: Checked by ClamAV on apache.org --0016e64698200ef05e0470503ca1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Can I pass parameters to custom map/reduce scripts like this: from try_hits select transform ip_address, aid, uid using 'parse_logs.rb MY_PARAM_HERE' as ip_address, aid, uid I've tried it, and it seems to work, but I just want to be sure whether this is guaranteed to work under all circumstances. A large part of my current project depends on this :-) PS: Here's what I'm trying to do -- based on a profile ID (which is passed as a parameter), the map/reduce script is going to look up a DB and fetch a bunch of regular expressions from there. Based on these dynamically fetched regular expressions it'll process the input. Saurabh. -- http://nandz.blogspot.com http://foodieforlife.blogspot.com --0016e64698200ef05e0470503ca1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Can I pass parameters to custom map/reduce scripts like this:<br><br>=A0=A0= =A0 from try_hits <br>=A0=A0=A0 select transform ip_address, aid, uid<br>= =A0=A0=A0 using &#39;parse_logs.rb MY_PARAM_HERE&#39; as ip_address, aid, u= id<br><br>I&#39;ve tried it, and it seems to work, but I just want to be su= re whether this is guaranteed to work under all circumstances. A large part= of my current project depends on this :-)<br> <br>PS: Here&#39;s what I&#39;m trying to do -- based on a profile ID (whic= h is passed as a parameter), the map/reduce script is going to look up a DB= and fetch a bunch of regular expressions from there. Based on these dynami= cally fetched regular expressions it&#39;ll process the input.<br> <br>Saurabh.<br>-- <br><a href=3D"http://nandz.blogspot.com">http://nandz.b= logspot.com</a><br><a href=3D"http://foodieforlife.blogspot.com">http://foo= dieforlife.blogspot.com</a><br> --0016e64698200ef05e0470503ca1-- From hive-userrobert49@example.com Tue Aug 04 13:14:23 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 41862 invoked from network); 4 Aug 2009 13:14:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 13:14:23 -0000 Received: (qmail 8987 invoked by uid 500); 4 Aug 2009 13:14:27 -0000 Delivered-To: robert49@example.com Received: (qmail 8965 invoked by uid 500); 4 Aug 2009 13:14:27 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 8956 invoked by uid 99); 4 Aug 2009 13:14:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 13:14:27 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ew0-f226.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 13:14:19 +0000 Received: by ewy26 with SMTP id 26so3832696ewy.29 for <robert49@example.com>; Tue, 04 Aug 2009 06:13:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=XIIPXgP4TsNS4BKFCBR23pJ1cDSs4Oriy4liEvdrIR8=; b=UQYsWkQUdEQvfZ7C4N21kBG0J4jBEHMw7dUsCi6TjdCv0Q8aDXGH+tVVsoBYAYXgJP TtRX7yyMxPjJf8s55HwkqW+AGMWWkrYqKFURPDXWZNrKGGIA0MLpHeN+1XE39xqL0cNA VkQywTAns6R3a2J86FyTLjQv0g69A/d03MP+A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=ULSbuj0jYXvqpopv8p4FTl08KCsWkObUHJXS1exCtWH/HGrWgQLhBMjiZm6vBLqD4I R99+DasODri3x4H7H/CE69WsP0khRXqhSVWgtxft2J1jPpHTjgPppl2AWbClsC5Od1u6 OjM04hKJL9ioyBhk4Ly1C1MIe+fLNGUGiJR3g= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id e14mr6706996ebq.42.1249391638627; Tue, 04 Aug 2009 06:13:58 -0700 (PDT) In-Reply-To: <robert49@example.com> References: <robert49@example.com> Date: Tue, 4 Aug 2009 16:13:58 +0300 Message-ID: <robert49@example.com> Subject: Does Hive support HBase (like) column based storage? From: Haggai Roitman <robert49@example.com> To: robert49@example.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi to all Hivers out there! Does Hive currently support column-based storage? For example, can we use HBase tables as Hive table inputs for HQL? Thanks in advance, Haggai From hive-userrobert49@example.com Tue Aug 04 14:19:54 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 85538 invoked from network); 4 Aug 2009 14:19:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 14:19:54 -0000 Received: (qmail 19325 invoked by uid 500); 4 Aug 2009 14:19:59 -0000 Delivered-To: robert49@example.com Received: (qmail 19262 invoked by uid 500); 4 Aug 2009 14:19:59 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 19253 invoked by uid 99); 4 Aug 2009 14:19:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:19:59 +0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail.software.ict.ac.cn) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 04 Aug 2009 14:19:50 +0000 Received: (qmail 10636 invoked from network); 4 Aug 2009 13:31:23 -0000 Received: from unknown (HELO ?83.161.120.155?) (robert49@example.com) by 83.161.120.155 with SMTP; 4 Aug 2009 13:31:23 -0000 User-Agent: Microsoft-Entourage/83.161.120.155130 Date: Tue, 04 Aug 2009 22:13:29 +0800 Subject: Re: Does Hive support HBase (like) column based storage? From: He Yongqiang <robert49@example.com> To: <robert49@example.com> Message-ID: <robert49@example.com> Thread-Topic: Does Hive support HBase (like) column based storage? Thread-Index: AcoVDb1oDpHByKLWPUCH6xhpWX8RGA== In-Reply-To: <robert49@example.com> Mime-version: 1.0 Content-type: text/plain; charset="ISO-2022-JP" Content-transfer-encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org > Does Hive currently support column-based storage? Y, hive support column-based storage. RCFile. You can create a table using command like : Create table tbl(cols) .... stored as RCFile to try Hive's column storage. > For example, can we use HBase tables as Hive table inputs for HQL? Right now, Hive does not support it, but is working on it. See https://issues.apache.org/jira/browse/HIVE-705 Please feel free to add your comment on the jira page. On 09-8-4 $B2<8a(B9:13, "Haggai Roitman" <robert49@example.com> wrote: > Hi to all Hivers out there! > > Does Hive currently support column-based storage? > For example, can we use HBase tables as Hive table inputs for HQL? > > Thanks in advance, > > Haggai > > From hive-userrobert49@example.com Tue Aug 04 14:22:03 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 89644 invoked from network); 4 Aug 2009 14:22:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 14:22:03 -0000 Received: (qmail 26417 invoked by uid 500); 4 Aug 2009 14:22:08 -0000 Delivered-To: robert49@example.com Received: (qmail 26394 invoked by uid 500); 4 Aug 2009 14:22:08 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 26385 invoked by uid 99); 4 Aug 2009 14:22:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:22:08 +0000 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-fx0-f225.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:21:56 +0000 Received: by fxm25 with SMTP id 25so3319836fxm.29 for <robert49@example.com>; Tue, 04 Aug 2009 07:21:35 -0700 (PDT) Received: by 83.161.120.155 with SMTP id v12mr1680007mup.93.1249395692777; Tue, 04 Aug 2009 07:21:32 -0700 (PDT) Received: from ?83.161.120.155? ([83.161.120.155]) by mx.google.com with ESMTPS id j9sm47611457mue.26.283.161.120.155.21.31 (version=SSLv3 cipher=RC4-MD5); Tue, 04 Aug 2009 07:21:31 -0700 (PDT) Subject: RE: Group by week? From: Andraz Tori <robert49@example.com> To: robert49@example.com Content-Type: multipart/alternative; boundary="=-os+hXUoe941eVgHyZbea" Date: Tue, 04 Aug 2009 16:21:29 +0200 Message-Id: <1249395689.4039.266.camel@minmax-laptop> Mime-Version: 1.0 X-Mailer: Evolution 2.26.1 X-Virus-Checked: Checked by ClamAV on apache.org --=-os+hXUoe941eVgHyZbea Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit Hi Ashish, thanks for help! 1) regarding the group-by-week, it was entirely my fault... I miswrote the function... to get grouping by week one needs to have FIELD: date_add('2007-12-31', (datediff(date_field, '2007-12-31') / 7) * 7) GROUP BY: datediff(date_field, '2007-12-31') / 7 This then works and you get both the grouping and the first day of the month as the field value 2) However this is not so simple for the month: > For getting numeric sorting you can do an order by CAST(month(pdate) AS INT). That will probably work for you? This won't work since pdate by itself is a date string, not a number, and even if it was then the month dates would be: 2008-4-20 which comes after 2008-10-20 when sorting lexically Therefore the question is how to get zero padding with hive? I have this expression: concat(concat(concat(CAST(year(date_field) as STRING), '-'), CAST(month(date_field) as STRING)), '-1') which produces dates like: 2008-4-1 but I want zero-padding, which means 2008-04-1 How does one do that with hive? Is there a function that would take separate year,month and day values and return properly formated date? [btw from my experience support for handling dates in hive is extremely weak, and code becomes heavily convoluted to do it properly] -- Andraz Tori, CTO Zemanta Ltd, New York, London, Ljubljana www.zemanta.com mail: robert49@example.com tel: (448)341-0601 twitter: andraz, skype: minmax_test --=-os+hXUoe941eVgHyZbea Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 7bit <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN"> <HTML> <HEAD> <META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8"> <META NAME="GENERATOR" CONTENT="GtkHTML/3.26.0"> </HEAD> <BODY> Hi Ashish, thanks for help!<BR> <BR> 1) regarding the group-by-week, it was entirely my fault... I miswrote the function... to get grouping by week one needs to have<BR> FIELD: date_add('2007-12-31', (datediff(date_field, '2007-12-31') / 7) * 7)<BR> GROUP BY: datediff(date_field, '2007-12-31') / 7<BR> <BR> This then works and you get both the grouping and the first day of the month as the field value<BR> <BR> 2) However this is not so simple for the month:<BR> <BR> &gt; For getting numeric sorting you can do an order by CAST(month(pdate) AS INT). That will probably work for you?<BR> <BR> This won't work since pdate by itself is a date string, not a number, and even if it was then the month dates would be:<BR> 2008-4-20 which comes after 2008-10-20 when sorting lexically<BR> Therefore the question is how to get zero padding with hive?<BR> I have this expression:<BR> concat(concat(concat(CAST(year(date_field) as STRING), '-'), CAST(month(date_field) as STRING)), '-1')<BR> <BR> which produces dates like:<BR> 2008-4-1<BR> but I want zero-padding, which means<BR> 2008-04-1<BR> <BR> How does one do that with hive?<BR> <BR> Is there a function that would take separate year,month and day values and return properly formated date?<BR> <BR> [btw from my experience support for handling dates in hive is extremely weak, and code becomes heavily convoluted to do it properly]<BR> <BR> <BR> <TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%"> <TR> <TD> <PRE> -- Andraz Tori, CTO Zemanta Ltd, New York, London, Ljubljana <A HREF="http://www.zemanta.com">www.zemanta.com</A> mail: <A HREF="mailto:robert49@example.com">robert49@example.com</A> tel: (448)341-0601 &#65279;twitter: andraz, skype: minmax_test </PRE> </TD> </TR> </TABLE> </BODY> </HTML> --=-os+hXUoe941eVgHyZbea-- From hive-userrobert49@example.com Tue Aug 04 14:36:22 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 699 invoked from network); 4 Aug 2009 14:36:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 14:36:22 -0000 Received: (qmail 55318 invoked by uid 500); 4 Aug 2009 14:36:27 -0000 Delivered-To: robert49@example.com Received: (qmail 55267 invoked by uid 500); 4 Aug 2009 14:36:27 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 55258 invoked by uid 99); 4 Aug 2009 14:36:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:36:27 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-qy0-f188.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:36:19 +0000 Received: by qyk26 with SMTP id 26so5034571qyk.5 for <robert49@example.com>; Tue, 04 Aug 2009 07:35:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=EUqFFJSOJz5P27hXzPIJg5dTTt525QFIeuPFlA3mDxA=; b=dZaOOhPetYPw0tBNHty98KbJjEaVcEaYYoPl/PV/CsIBrY9tp8cYhIBy+kiwJmlU4S RUzwYxY/oSpA3lxOJrGc3D8KFHEproXjR6P4TN84Bph4cr0UGoij8RThujA5tuTRGma4 PHDXRuftoVo43zypoOXFaBnCMSCTl2Ldh0v90= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:content-type; b=Wavth4l+q6PNDbo+QCnRPjhZLWqZW3Ej5p+P8rFBzchAF+pN8rs8ecRsU60ROzUNrL T4x2eHyIS04wb3cF8fypirLEfHRbOxlBP70d1+aCpxg5BpYeHdSejEv2fTQrBvoBDeXm DaLxSeJhljStfyAGccvyC5YzuKm59b/p3xUiE= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id c21mr5931160qab.103.1249396558073; Tue, 04 Aug 2009 07:35:58 -0700 (PDT) Reply-To: robert49@example.com In-Reply-To: <robert49@example.com> References: <robert49@example.com> Date: Tue, 4 Aug 2009 07:35:58 -0700 Message-ID: <robert49@example.com> Subject: Re: JDBC: Infinite while(rs.next()) loop From: Bill Graham <robert49@example.com> To: robert49@example.com Content-Type: multipart/alternative; boundary=00151748db7207109c047051ce4b X-Virus-Checked: Checked by ClamAV on apache.org --00151748db7207109c047051ce4b Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit This bug has been fixed on the trunk. Check out the hive trunk and build the JDBC driver and you should be fine. On Tue, Aug 4, 2009 at 12:47 AM, Saurabh Nanda <robert49@example.com>wrote: > Here's what I'm trying: > > ResultSet rs=st.executeQuery("show tables"); > while(rs.next()) { > System.out.println(rs.getString(1)); > } > > The while loop never terminates, after going through the list of tables, it > keeps printing the last table name over & over again. Am I doing something > wrong over here, or have I hit a bug? I'm on hive-0.3.0-hadoop-0.18.0-bin > > Saurabh. > -- > http://nandz.blogspot.com > http://foodieforlife.blogspot.com > --00151748db7207109c047051ce4b Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable This bug has been fixed on the trunk. Check out the hive trunk and build th= e JDBC driver and you should be fine.<br><br><div class=3D"gmail_quote">On = Tue, Aug 4, 2009 at 12:47 AM, Saurabh Nanda <span dir=3D"ltr">&lt;<a href= =3D"mailto:robert49@example.com">robert49@example.com</a>&gt;</span> wr= ote:<br> <blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, = 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Here&#39;s what I= &#39;m trying:<br><br>=A0=A0=A0=A0=A0=A0 ResultSet rs=3Dst.executeQuery(&qu= ot;show tables&quot;);<br> =A0=A0=A0=A0=A0=A0=A0 while(rs.next()) {<br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 System.out.println(rs.getString(1));<br>=A0=A0=A0=A0=A0=A0=A0 }<br clea= r=3D"all"><br>The while loop never terminates, after going through the list= of tables, it keeps printing the last table name over &amp; over again. Am= I doing something wrong over here, or have I hit a bug? I&#39;m on hive-0.= 3.0-hadoop-0.18.0-bin<br> <br>Saurabh.<br><font color=3D"#888888">-- <br><a href=3D"http://nandz.blog= spot.com" target=3D"_blank">http://nandz.blogspot.com</a><br><a href=3D"htt= p://foodieforlife.blogspot.com" target=3D"_blank">http://foodieforlife.blog= spot.com</a><br> </font></blockquote></div><br> --00151748db7207109c047051ce4b-- From hive-userrobert49@example.com Tue Aug 04 14:40:15 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 3495 invoked from network); 4 Aug 2009 14:40:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 14:40:15 -0000 Received: (qmail 65203 invoked by uid 500); 4 Aug 2009 14:40:20 -0000 Delivered-To: robert49@example.com Received: (qmail 65179 invoked by uid 500); 4 Aug 2009 14:40:20 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 65170 invoked by uid 99); 4 Aug 2009 14:40:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:40:20 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO qw-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:40:12 +0000 Received: by qw-out-2122.google.com with SMTP id 8so1936924qwh.35 for <robert49@example.com>; Tue, 04 Aug 2009 07:39:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=3UHLOT+TcPTBjMexbJpPTGkMN12nscugtR3NNarVvmU=; b=X5Pq7dOgJwBi2ACiHIdPhPy3i8imP0lJL84ZCf4zX80tqKiqHrh+ezi15j7jL3x+Qy sSLqStCpD0eYOLXDuTanZ4eyjk0vbjjmr74z5GethkOqEh8jChYtr4nHR8qLFFz85wmY CM4mOLpKw63w5n0DJF/t1SKGrXabaq9/y63x4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=NAeIzmFVqXqbCAl0sha2kKc5NPiZYXjwKoJLyC+32+Q03j3DTnzNkXBDtdsJGZfOHs 2dptN8Uvdj9Ygg6sWGSB5YwiKwcDcYZPWwjp/9BuSWDC8CNrOYZ2p8+ibmNbSGoC7OH0 Yq6mSz5Uk3loI6MVVf/RYUBSZqfToDhg/aVg4= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id u1mr6570671vcm.59.1249396790908; Tue, 04 Aug 2009 07:39:50 -0700 (PDT) In-Reply-To: <robert49@example.com> References: <robert49@example.com> <robert49@example.com> Date: Tue, 4 Aug 2009 20:09:50 +0530 Message-ID: <robert49@example.com> Subject: Re: JDBC: Infinite while(rs.next()) loop From: Saurabh Nanda <robert49@example.com> To: robert49@example.com, robert49@example.com Content-Type: multipart/alternative; boundary=0016364ed386e7d61b047051db8d X-Virus-Checked: Checked by ClamAV on apache.org --0016364ed386e7d61b047051db8d Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit I was dreading this response! Are there any plans to push out a new Hive build with the latest features & bug fixes? Building from trunk is not everyone's cup of tea, you know :-) Any nightly builds that I can pick up? Saurabh. On Tue, Aug 4, 2009 at 8:05 PM, Bill Graham <robert49@example.com> wrote: > This bug has been fixed on the trunk. Check out the hive trunk and build > the JDBC driver and you should be fine. > > > On Tue, Aug 4, 2009 at 12:47 AM, Saurabh Nanda <robert49@example.com>wrote: > >> Here's what I'm trying: >> >> ResultSet rs=st.executeQuery("show tables"); >> while(rs.next()) { >> System.out.println(rs.getString(1)); >> } >> >> The while loop never terminates, after going through the list of tables, >> it keeps printing the last table name over & over again. Am I doing >> something wrong over here, or have I hit a bug? I'm on >> hive-0.3.0-hadoop-0.18.0-bin >> >> Saurabh. >> -- >> http://nandz.blogspot.com >> http://foodieforlife.blogspot.com >> > > -- http://nandz.blogspot.com http://foodieforlife.blogspot.com --0016364ed386e7d61b047051db8d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable I was dreading this response! Are there any plans to push out a new Hive bu= ild with the latest features &amp; bug fixes? Building from trunk is not ev= eryone&#39;s cup of tea, you know :-)<br><br>Any nightly builds that I can = pick up?<br> <br>Saurabh.<br><br><div class=3D"gmail_quote">On Tue, Aug 4, 2009 at 8:05 = PM, Bill Graham <span dir=3D"ltr">&lt;<a href=3D"mailto:robert49@example.com= m">robert49@example.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail= _quote" style=3D"border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt= 0pt 0.8ex; padding-left: 1ex;"> This bug has been fixed on the trunk. Check out the hive trunk and build th= e JDBC driver and you should be fine.<div><div></div><div class=3D"h5"><br>= <br><div class=3D"gmail_quote">On Tue, Aug 4, 2009 at 12:47 AM, Saurabh Nan= da <span dir=3D"ltr">&lt;<a href=3D"mailto:robert49@example.com" target= =3D"_blank">robert49@example.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, = 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Here&#39;s what I= &#39;m trying:<br><br>=A0=A0=A0=A0=A0=A0 ResultSet rs=3Dst.executeQuery(&qu= ot;show tables&quot;);<br> =A0=A0=A0=A0=A0=A0=A0 while(rs.next()) {<br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 System.out.println(rs.getString(1));<br>=A0=A0=A0=A0=A0=A0=A0 }<br clea= r=3D"all"><br>The while loop never terminates, after going through the list= of tables, it keeps printing the last table name over &amp; over again. Am= I doing something wrong over here, or have I hit a bug? I&#39;m on hive-0.= 3.0-hadoop-0.18.0-bin<br> <br>Saurabh.<br><font color=3D"#888888">-- <br><a href=3D"http://nandz.blog= spot.com" target=3D"_blank">http://nandz.blogspot.com</a><br><a href=3D"htt= p://foodieforlife.blogspot.com" target=3D"_blank">http://foodieforlife.blog= spot.com</a><br> </font></blockquote></div><br> </div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><a href=3D"= http://nandz.blogspot.com">http://nandz.blogspot.com</a><br><a href=3D"http= ://foodieforlife.blogspot.com">http://foodieforlife.blogspot.com</a><br> --0016364ed386e7d61b047051db8d-- From hive-userrobert49@example.com Tue Aug 04 14:44:24 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 4645 invoked from network); 4 Aug 2009 14:44:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 14:44:24 -0000 Received: (qmail 72109 invoked by uid 500); 4 Aug 2009 14:44:28 -0000 Delivered-To: robert49@example.com Received: (qmail 72050 invoked by uid 500); 4 Aug 2009 14:44:28 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 72041 invoked by uid 99); 4 Aug 2009 14:44:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:44:28 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-qy0-f188.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 14:44:19 +0000 Received: by qyk26 with SMTP id 26so5042415qyk.5 for <robert49@example.com>; Tue, 04 Aug 2009 07:43:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=hrRzDwuRCpq8ySrBuic6DIMd3pW3R6MnH4/JHJnOexw=; b=ZB7F0pNLIgyrRAs4ThcNbfhWZCcuAaHqM9hwe4TumiI/9doftIBundi/1Gqid/tn3l QvCmncvUXVd86Zu4z+9PUCQY7zCecjxKFE9Hl085Qus2Ldq6Pi/1YNAXb5ruitT8nGRd ++a9Vv3ZxQrYYTKOfUAYrRqI8xVrHjoEgvfOU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; b=BCcIDMAmVQiCzdJgF/BWf/J2WbeUGddOUxKVUXkfFEKffMt8k7ujQS3rKSJGi+hUmb Zit0+pxS3GfHI7jRamWeLP0yt3SddGJvn5NPh6JWrAHzgRPC4CInCEGQ6+ptaudtlArG pkwdPEA+zfDw0fdh44D4PzgnN9qptYF2ehfkE= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id c4mr5965135qab.145.1249397038032; Tue, 04 Aug 2009 07:43:58 -0700 (PDT) Reply-To: robert49@example.com In-Reply-To: <robert49@example.com> References: <robert49@example.com> <robert49@example.com> <robert49@example.com> Date: Tue, 4 Aug 2009 07:43:58 -0700 Message-ID: <robert49@example.com> Subject: Re: JDBC: Infinite while(rs.next()) loop From: Bill Graham <robert49@example.com> To: Saurabh Nanda <robert49@example.com> Cc: robert49@example.com Content-Type: multipart/alternative; boundary=0015175cd694a2a8ea047051eacc X-Virus-Checked: Checked by ClamAV on apache.org --0015175cd694a2a8ea047051eacc Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit +1 I agree. I do not know the answer to that one. Can anyone comment on the future Hive release schedule? On Tue, Aug 4, 2009 at 7:39 AM, Saurabh Nanda <robert49@example.com>wrote: > I was dreading this response! Are there any plans to push out a new Hive > build with the latest features & bug fixes? Building from trunk is not > everyone's cup of tea, you know :-) > > Any nightly builds that I can pick up? > > Saurabh. > > > On Tue, Aug 4, 2009 at 8:05 PM, Bill Graham <robert49@example.com> wrote: > >> This bug has been fixed on the trunk. Check out the hive trunk and build >> the JDBC driver and you should be fine. >> >> >> On Tue, Aug 4, 2009 at 12:47 AM, Saurabh Nanda <robert49@example.com>wrote: >> >>> Here's what I'm trying: >>> >>> ResultSet rs=st.executeQuery("show tables"); >>> while(rs.next()) { >>> System.out.println(rs.getString(1)); >>> } >>> >>> The while loop never terminates, after going through the list of tables, >>> it keeps printing the last table name over & over again. Am I doing >>> something wrong over here, or have I hit a bug? I'm on >>> hive-0.3.0-hadoop-0.18.0-bin >>> >>> Saurabh. >>> -- >>> http://nandz.blogspot.com >>> http://foodieforlife.blogspot.com >>> >> >> > > > -- > http://nandz.blogspot.com > http://foodieforlife.blogspot.com > --0015175cd694a2a8ea047051eacc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable +1 <br><br>I agree. I do not know the answer to that one. Can anyone commen= t on the future Hive release schedule?<br><br><br><div class=3D"gmail_quote= ">On Tue, Aug 4, 2009 at 7:39 AM, Saurabh Nanda <span dir=3D"ltr">&lt;<a hr= ef=3D"mailto:robert49@example.com">robert49@example.com</a>&gt;</span> = wrote:<br> <blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, = 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">I was dreading th= is response! Are there any plans to push out a new Hive build with the late= st features &amp; bug fixes? Building from trunk is not everyone&#39;s cup = of tea, you know :-)<br> <br>Any nightly builds that I can pick up?<br><font color=3D"#888888"> <br>Saurabh.</font><div><div></div><div class=3D"h5"><br><br><div class=3D"= gmail_quote">On Tue, Aug 4, 2009 at 8:05 PM, Bill Graham <span dir=3D"ltr">= &lt;<a href=3D"mailto:robert49@example.com" target=3D"_blank">billgraham@gm= ail.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, = 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> This bug has been fixed on the trunk. Check out the hive trunk and build th= e JDBC driver and you should be fine.<div><div></div><div><br><br><div clas= s=3D"gmail_quote">On Tue, Aug 4, 2009 at 12:47 AM, Saurabh Nanda <span dir= =3D"ltr">&lt;<a href=3D"mailto:robert49@example.com" target=3D"_blank">sa= robert49@example.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, = 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Here&#39;s what I= &#39;m trying:<br><br>=A0=A0=A0=A0=A0=A0 ResultSet rs=3Dst.executeQuery(&qu= ot;show tables&quot;);<br> =A0=A0=A0=A0=A0=A0=A0 while(rs.next()) {<br>=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 System.out.println(rs.getString(1));<br>=A0=A0=A0=A0=A0=A0=A0 }<br clea= r=3D"all"><br>The while loop never terminates, after going through the list= of tables, it keeps printing the last table name over &amp; over again. Am= I doing something wrong over here, or have I hit a bug? I&#39;m on hive-0.= 3.0-hadoop-0.18.0-bin<br> <br>Saurabh.<br><font color=3D"#888888">-- <br><a href=3D"http://nandz.blog= spot.com" target=3D"_blank">http://nandz.blogspot.com</a><br><a href=3D"htt= p://foodieforlife.blogspot.com" target=3D"_blank">http://foodieforlife.blog= spot.com</a><br> </font></blockquote></div><br> </div></div></blockquote></div><br><br clear=3D"all"><br>-- <br><a href=3D"= http://nandz.blogspot.com" target=3D"_blank">http://nandz.blogspot.com</a><= br><a href=3D"http://foodieforlife.blogspot.com" target=3D"_blank">http://f= oodieforlife.blogspot.com</a><br> </div></div></blockquote></div><br> --0015175cd694a2a8ea047051eacc-- From hive-userrobert49@example.com Tue Aug 04 15:44:10 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 25944 invoked from network); 4 Aug 2009 15:44:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 15:44:10 -0000 Received: (qmail 84917 invoked by uid 500); 4 Aug 2009 15:44:15 -0000 Delivered-To: robert49@example.com Received: (qmail 84882 invoked by uid 500); 4 Aug 2009 15:44:14 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 84873 invoked by uid 99); 4 Aug 2009 15:44:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 15:44:14 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-fx0-f225.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 15:44:06 +0000 Received: by fxm25 with SMTP id 25so3374046fxm.29 for <robert49@example.com>; Tue, 04 Aug 2009 08:43:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=NfwHvTyDb98J5E6CvJkzoNWdFAOTMOy4TD9CgLc3uh0=; b=RBk9ww2taaAxyJrcs41ulbmlkJdj7td72kvzj7SpKz2U+X3bUwXf566YBIxN2LLyaS lFHtq5HviTjdaniV7Miz8S685vNlMcLXo8w1ZNyRY8BGlM2k4qlxlM9G3Tg9PkfG2Gs6 txSkVNsRM1GUWIGHFl+OAdO4TGpOO9yT0h5FM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=BozhnsSn2Vr0t+VNmzqjuMzr0eerzYG4KGApXHDQSYFh4ymiep95sUt0FMaKK3Q0KJ +xzxEHdHCLLGRx1sNltbPMZ6V7zusX4sI7WbORPVfnzJvVz6QL0WwefIAn6Jeti2Zx1F buArx2BekD02GT9YXeftSuJ1H+Eps+1lPN8z4= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id h8mr480209hbc.128.1249400625738; Tue, 04 Aug 2009 08:43:45 -0700 (PDT) In-Reply-To: <robert49@example.com> References: <robert49@example.com> <robert49@example.com> <robert49@example.com> <robert49@example.com> Date: Tue, 4 Aug 2009 11:43:45 -0400 Message-ID: <robert49@example.com> Subject: Re: JDBC: Infinite while(rs.next()) loop From: Edward Capriolo <robert49@example.com> To: robert49@example.com, robert49@example.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On Tue, Aug 4, 2009 at 10:43 AM, Bill Graham<robert49@example.com> wrote: > +1 > > I agree. I do not know the answer to that one. Can anyone comment on the > future Hive release schedule? > > > On Tue, Aug 4, 2009 at 7:39 AM, Saurabh Nanda <robert49@example.com> > wrote: >> >> I was dreading this response! Are there any plans to push out a new Hive >> build with the latest features & bug fixes? Building from trunk is not >> everyone's cup of tea, you know :-) >> >> Any nightly builds that I can pick up? >> >> Saurabh. >> >> On Tue, Aug 4, 2009 at 8:05 PM, Bill Graham <robert49@example.com> wrote= : >>> >>> This bug has been fixed on the trunk. Check out the hive trunk and buil= d >>> the JDBC driver and you should be fine. >>> >>> On Tue, Aug 4, 2009 at 12:47 AM, Saurabh Nanda <robert49@example.com> >>> wrote: >>>> >>>> Here's what I'm trying: >>>> >>>> =A0=A0=A0=A0=A0=A0 ResultSet rs=3Dst.executeQuery("show tables"); >>>> =A0=A0=A0=A0=A0=A0=A0 while(rs.next()) { >>>> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 System.out.println(rs.getString(1)); >>>> =A0=A0=A0=A0=A0=A0=A0 } >>>> >>>> The while loop never terminates, after going through the list of table= s, >>>> it keeps printing the last table name over & over again. Am I doing >>>> something wrong over here, or have I hit a bug? I'm on >>>> hive-0.3.0-hadoop-0.18.0-bin >>>> >>>> Saurabh. >>>> -- >>>> http://nandz.blogspot.com >>>> http://foodieforlife.blogspot.com >>> >> >> >> >> -- >> http://nandz.blogspot.com >> http://foodieforlife.blogspot.com > > Hive 4.0 will be a release candidate soon. The largest major blocker that I know of is dealing with Hadoop 0.20. See: https://issues.apache.org/jira/browse/HIVE-487 Soon after that their should be a release candidate, then voting. Edward From hive-userrobert49@example.com Tue Aug 04 16:26:12 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 38679 invoked from network); 4 Aug 2009 16:26:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 16:26:12 -0000 Received: (qmail 36528 invoked by uid 500); 4 Aug 2009 16:26:16 -0000 Delivered-To: robert49@example.com Received: (qmail 36460 invoked by uid 500); 4 Aug 2009 16:26:16 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 36440 invoked by uid 99); 4 Aug 2009 16:26:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 16:26:11 +0000 X-ASF-Spam-Status: No, hits=-1.8 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mailout-sf2p.facebook.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 16:26:01 +0000 Received: from mail.thefacebook.com (intlb01.snat.snc1.facebook.com [83.161.120.155] (may be forged)) by pp02.snc1.tfbnw.net (8.14.1/8.14.1) with ESMTP id n74GPTnO020367 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT) for <robert49@example.com>; Tue, 4 Aug 2009 09:25:29 -0700 Received: from SC-MBXC1.TheFacebook.com ([83.161.120.155]) by sc-hub01.TheFacebook.com ([83.161.120.155]) with mapi; Tue, 4 Aug 2009 09:25:39 -0700 From: Prasad Chakka <robert49@example.com> To: "robert49@example.com" <robert49@example.com> Date: Tue, 4 Aug 2009 09:25:37 -0700 Subject: Re: Passing parameters to custom map/reduce scripts Thread-Topic: Passing parameters to custom map/reduce scripts Thread-Index: AcoVAUCAOlsnPYvLQaSuvY9hNwLdJwAHvJfk Message-ID: <robert49@example.com> In-Reply-To: <robert49@example.com> Accept-Language: en-US Content-Language: en X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_C69DAF111D701prasadfacebookcom_" MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.8161:2.4.5,1.2.40,4.0.166 definitions=2009-08-04_09:2009-07-24,2009-08-04,2009-08-04 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-0907200000 definitions=main-0908040090 X-Virus-Checked: Checked by ClamAV on apache.org --_000_C69DAF111D701prasadfacebookcom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Quotes should work. Prasad ________________________________ From: Saurabh Nanda <robert49@example.com> Reply-To: <robert49@example.com> Date: Tue, 4 Aug 2009 05:43:34 -0700 To: <robert49@example.com> Subject: Passing parameters to custom map/reduce scripts Can I pass parameters to custom map/reduce scripts like this: from try_hits select transform ip_address, aid, uid using 'parse_logs.rb MY_PARAM_HERE' as ip_address, aid, uid I've tried it, and it seems to work, but I just want to be sure whether thi= s is guaranteed to work under all circumstances. A large part of my current= project depends on this :-) PS: Here's what I'm trying to do -- based on a profile ID (which is passed = as a parameter), the map/reduce script is going to look up a DB and fetch a= bunch of regular expressions from there. Based on these dynamically fetche= d regular expressions it'll process the input. Saurabh. -- http://nandz.blogspot.com http://foodieforlife.blogspot.com --_000_C69DAF111D701prasadfacebookcom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <HTML> <HEAD> <TITLE>Re: Passing parameters to custom map/reduce scripts</TITLE> </HEAD> <BODY> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:= 11pt'>Quotes should work.<BR> <BR> Prasad<BR> <BR> <BR> <HR ALIGN=3DCENTER SIZE=3D"3" WIDTH=3D"95%"><B>From: </B>Saurabh Nanda &lt;= <a href=3D"robert49@example.com">robert49@example.com</a>&gt;<BR> <B>Reply-To: </B>&lt;<a href=3D"robert49@example.com">hive-user@hado= op.apache.org</a>&gt;<BR> <B>Date: </B>Tue, 4 Aug 2009 05:43:34 -0700<BR> <B>To: </B>&lt;<a href=3D"robert49@example.com">robert49@example.com= che.org</a>&gt;<BR> <B>Subject: </B>Passing parameters to custom map/reduce scripts<BR> <BR> Can I pass parameters to custom map/reduce scripts like this:<BR> <BR> =A0=A0=A0 from try_hits <BR> =A0=A0=A0 select transform ip_address, aid, uid<BR> =A0=A0=A0 using 'parse_logs.rb MY_PARAM_HERE' as ip_address, aid, uid<BR> <BR> I've tried it, and it seems to work, but I just want to be sure whether thi= s is guaranteed to work under all circumstances. A large part of my current= project depends on this :-)<BR> <BR> PS: Here's what I'm trying to do -- based on a profile ID (which is passed = as a parameter), the map/reduce script is going to look up a DB and fetch a= bunch of regular expressions from there. Based on these dynamically fetche= d regular expressions it'll process the input.<BR> <BR> Saurabh.<BR> -- <BR> <a href=3D"http://nandz.blogspot.com">http://nandz.blogspot.com</a><BR> <a href=3D"http://foodieforlife.blogspot.com">http://foodieforlife.blogspot= .com</a><BR> <BR> </SPAN></FONT> </BODY> </HTML> --_000_C69DAF111D701prasadfacebookcom_-- From hive-userrobert49@example.com Tue Aug 04 17:21:11 2009 Return-Path: <hive-userrobert49@example.com> Delivered-To: robert49@example.com Received: (qmail 53657 invoked from network); 4 Aug 2009 17:21:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Aug 2009 17:21:11 -0000 Received: (qmail 27295 invoked by uid 500); 4 Aug 2009 17:21:16 -0000 Delivered-To: robert49@example.com Received: (qmail 27220 invoked by uid 500); 4 Aug 2009 17:21:16 -0000 Mailing-List: contact robert49@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:robert49@example.com> List-Unsubscribe: <mailto:robert49@example.com> List-Post: <mailto:robert49@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: robert49@example.com Delivered-To: mailing list robert49@example.com Received: (qmail 27211 invoked by uid 99); 4 Aug 2009 17:21:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 17:21:16 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of robert49@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-vw0-f202.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Aug 2009 17:21:08 +0000 Received: by vws40 with SMTP id 40so1455334vws.2 for <robert49@example.com>; Tue, 04 Aug 2009 10:20:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=/1i2WBmHPOS6UuHgE/reCtHD0ihkm0wC17DCyRuL7Tw=; b=ImBg6E+V/i5cYjQDOkX1hvPZfqrU1/bWWpjNaOg+IgW0qAKK2qSWFqdkpu5X5lF982 TEqm2DEdODA7pLuHdcFgLVe99LhFW0aA89scAyxXf1imbYRvDfYx7AhNMns8Va/eNgP1 G8sXp58DuJa6Rv0vqhokFBUOqQU71qPZw7pTw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=T4kOPbUwUB1orbOJHXA2a6aV31PRA8feYgi3NaaAh9iR9l+3w7ajrbQrfOQvzhHaz9 97DHYM5BzONTSKSZep9qz63i3cKQI6fbnGUZ037PHuBNwaMHtkH1CMKVKGWixY4RbPRA 6MLOYNBRiz5U2Sht/husLVgyR4hGObQtAOLzc= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id w5mr6858723vck.23.1249406447581; Tue, 04 Aug 2009 10:20:47 -0700 (PDT) In-Reply-To: <robert49@example.com> References: <robert49@example.com> <robert49@example.com> <robert49@example.com> <robert49@example.com> <robert49@example.com> Date: Tue, 4 Aug 2009 22:50:47 +0530 Message-ID: <robert49@example.com> Subject: Re: JDBC: Infinite while(rs.next()) loop From: Saurabh Nanda <robert49@example.com> To: robert49@example.com Cc: robert49@example.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Is there any possibility of having a nightly build off the trunk, before hive 0.4 is officially released? On 8/4/09, Edward Capriolo <robert49@example.com> wrote: > On Tue, Aug 4, 2009 at 10:43 AM, Bill Graham<robert49@example.com> wrote: >> +1 >> >> I agree. I do not know the answer to that one. Can anyone comment on the >> future Hive release schedule? >> >> >> On Tue, Aug 4, 2009 at 7:39 AM, Saurabh Nanda <robert49@example.com> >> wrote: >>> >>> I was dreading this response! Are there any plans to push out a new Hiv= e >>> build with the latest features & bug fixes? Building from trunk is not >>> everyone's cup of tea, you know :-) >>> >>> Any nightly builds that I can pick up? >>> >>> Saurabh. >>> >>> On Tue, Aug 4, 2009 at 8:05 PM, Bill Graham <robert49@example.com> wrot= e: >>>> >>>> This bug has been fixed on the trunk. Check out the hive trunk and bui= ld >>>> the JDBC driver and you should be fine. >>>> >>>> On Tue, Aug 4, 2009 at 12:47 AM, Saurabh Nanda <robert49@example.com= > >>>> wrote: >>>>> >>>>> Here's what I'm trying: >>>>> >>>>> =A0=A0=A0=A0=A0=A0 ResultSet rs=3Dst.executeQuery("show tables"); >>>>> =A0=A0=A0=A0=A0=A0=A0 while(rs.next()) { >>>>> =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 System.out.println(rs.getString(1))= ; >>>>> =A0=A0=A0=A0=A0=A0=A0 } >>>>> >>>>> The while loop never terminates, after going through the list of >>>>> tables, >>>>> it keeps printing the last
{'source': 'Apache Software Foundation Public Mail Archives'}