From hive-userdavidcunningham@example.com Thu Jul 01 16:18:38 2010 Return-Path: <hive-userdavidcunningham@example.com> Delivered-To: davidcunningham@example.com Received: (qmail 89722 invoked from network); 1 Jul 2010 16:18:35 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Jul 2010 16:18:35 -0000 Received: (qmail 99134 invoked by uid 500); 1 Jul 2010 16:18:34 -0000 Delivered-To: davidcunningham@example.com Received: (qmail 98955 invoked by uid 500); 1 Jul 2010 16:18:33 -0000 Mailing-List: contact davidcunningham@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:davidcunningham@example.com> List-Unsubscribe: <mailto:davidcunningham@example.com> List-Post: <mailto:davidcunningham@example.com> List-Id: <hive-user.hadoop.apache.org> Reply-To: davidcunningham@example.com Delivered-To: mailing list davidcunningham@example.com Received: (qmail 98945 invoked by uid 99); 1 Jul 2010 16:18:33 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 16:18:33 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrout2-b.corp.re1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jul 2010 16:18:17 +0000 Received: from SNV-EXPF01.ds.corp.yahoo.com (snv-expf01.ds.corp.yahoo.com [83.161.120.155]) by mrout2-b.corp.re1.yahoo.com (8.13.8/8.13.8/y.out) with ESMTP id o61GFhPu035561 for <davidcunningham@example.com>; Thu, 1 Jul 2010 09:15:47 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:x-mimeole:content-class:mime-version: content-type:subject:date:message-id:in-reply-to:x-ms-has-attach: x-ms-tnef-correlator:thread-topic:thread-index:references:from:to:x-originalarrivaltime; b=fYTAYzAYn45qqKf3LEu4dewFdnUMa9DTQtkrN3AMFa60qpp47u5KE05jSiccRRob Received: from SNV-EXVS06.ds.corp.yahoo.com ([83.161.120.155]) by SNV-EXPF01.ds.corp.yahoo.com with Microsoft SMTPSVC(480)894-8247); Thu, 1 Jul 2010 09:15:42 -0700 x-mimeole: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CB1938.A1B3DB42" Subject: RE: alter table add partition error Date: Thu, 1 Jul 2010 09:15:32 -0700 Message-ID: <davidcunningham@example.com> In-Reply-To: <davidcunningham@example.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: alter table add partition error Thread-Index: AcsSL4ic79JbaupASoOX4pV8KQbj9gAPoVwAAB/XxzAAbil8EAEknnQw References: <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com>! ! ! <4C2 0F131.708080 davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> <davidcunningham@example.com> From: "Pradeep Kamath" <davidcunningham@example.com> To: <davidcunningham@example.com> X-OriginalArrivalTime: 01 Jul 2010 16:15:42.0955 (UTC) FILETIME=[A784C3B0:01CB1938] X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. ------_=_NextPart_001_01CB1938.A1B3DB42 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I am having difficulty starting a thrift service instance in the junit test for this patch - https://issues.apache.org/jira/browse/HIVE-1428?focusedCommentId=3D128836= 1 4&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpan= e l#action_12883614 - any pointers would be appreciated. =20 Thanks, Pradeep =20 ________________________________ From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Friday, June 25, 2010 1:37 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 I have posted a patch for this - not sure how to write unit tests for it - so patch doesn't have tests. =20 ________________________________ From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Wednesday, June 23, 2010 9:04 AM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 In the second step of=20 http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code =20 "Use the approved version of thrift. This is currently thrift-instant-r790732, which you can obtain from http://instant.thrift-rpc.org" =20 Looks like that site is down since it is not responding to requests - is there an alternate url I can try? Also, is the approved version still thrift-instant-r790732? =20 ________________________________ From: Paul Yang [mailto:davidcunningham@example.com]=20 Sent: Tuesday, June 22, 2010 6:06 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Hey Pradeep, =20 Yep, that's on the right path :) Since you made a change to hive_metastore.thrift, you should run the thrift compiler by calling 'ant thriftif' within trunk/metastore. But before you can do that, the thrift compiler has to be in your path. JVS has a nice write up on how to do this: =20 http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code =20 Also, I've set up a JIRA for this issue at: =20 https://issues.apache.org/jira/browse/HIVE-1428 =20 Can you post your patch there? It makes it a lot easier to track issues. Thanks! =20 -Paul =20 From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Tuesday, June 22, 2010 10:22 AM To: davidcunningham@example.com Subject: Re: alter table add partition error =20 Hi Paul, I have attached a draft of a patch (incomplete). The problem I am facing is that to make this change I seem to need to change the thrift idl (hive_metastore.thrift). Looking at the code structure it seems like the generate java files are checked in rather than generated through the build (is this so that the build does not depend on the thrift compiler?). I tried downloading and building thrift but hit issues in the make install - wanted to check with you if the patch looks correct so far or if I am on the wrong path. Also if the direction is right, I need some guidance on generating the java files from the modified idl. Thanks, Pradeep Paul Yang wrote:=20 Well, the section of code you are referring to is automatically generated by the thrift compiler so we can't modify it. As the inability to return null is a limitation of Thrift, we can work around that issue by throwing an exception instead. =20 Check out this thread: http://publists.facebook.com/pipermail/thrift/2007-September/000085.html =20 =20 From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Monday, June 21, 2010 5:46 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Hi Paul, Yes I don't mind working on a patch though I have no knowledge of how thrift code works. I was also confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() - to first check if(result.o1 !=3D null) throw result.o1, then to just return result.success and not throw the "unknown result" exception. I tried something along these lines and the alter table worked. Am I missing something? =20 Thanks, Pradeep =20 ________________________________ From: Paul Yang [mailto:davidcunningham@example.com]=20 Sent: Monday, June 21, 2010 2:35 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Ah okay. So what's going on is that getPartition() is supposed to return null if the specified partition doesn't exist. But because Thrift can't handle null return values, we are seeing this exception. This isn't a problem with a local metastore because Thrift isn't used. =20 One solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent partitions. Then Hive.getPartition() can catch the exception and return null to retain the original behavior. =20 Pradeep, are you interested in working on a patch for this problem? =20 From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Monday, June 21, 2010 2:11 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 I tried debugging in code a little more. Here is what I found: The code in ThriftHiveMetaStore eventually makes a call - get_partition() passing the partition key values for the partition I am trying to add using alter table. I assume this is to check that the partition doesn't already exist. I added a debug line in the following code: public Partition recv_get_partition() throws MetaException, TException { TMessage msg =3D iprot_.readMessageBegin(); if (msg.type =3D=3D TMessageType.EXCEPTION) { TApplicationException x =3D TApplicationException.read(iprot_); iprot_.readMessageEnd(); throw x; } get_partition_result result =3D new get_partition_result(); =20 result.read(iprot_); System.err.println("XXX: result:" + result); iprot_.readMessageEnd(); if (result.isSetSuccess()) { return result.success; } if (result.o1 !=3D null) { throw result.o1; } throw new TApplicationException(TApplicationException.MISSING_RESULT, "get_partition failed: unknown result"); } =20 I also put a debug statements in the read() method: public void read(TProtocol iprot) throws TException { TField field; iprot.readStructBegin(); System.err.println("XXX: Reading TProtocol object:"); while (true) { field =3D iprot.readFieldBegin(); System.err.println("XXX: field just read:" + field); if (field.type =3D=3D TType.STOP) {=20 break; =20 I got=20 XXX: Reading TProtocol object: XXX: field just read:<TField name:'' type:0 field-id:0> XXX: result:get_partition_result(success:null, o1:null) =20 The field read in the thrift response message is of type "STOP" and with id of type SUCCESS. This seems right since there are no existing partitions. But the way the rest of the code handles this, results in the exception. =20 Any pointers? =20 TIA, Pradeep ________________________________ From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Friday, June 18, 2010 2:51 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Looks like the standalone script works fine against the existing partition: ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=3D20100602/srcid=3D100/action=3Dview/testid=3D10 Partition(parameters=3D{'transient_lastDdlTime': '1276881277'}, tableName=3D'dummy', createTime=3D1276881277, lastAccessTime=3D0, values=3D['20100602', '100', 'view', '10'], dbName=3D'default', sd=3DStorageDescriptor(outputFormat=3D'org.apache.hadoop.hive.ql.io.HiveI= gno reKeyTextOutputFormat', sortCols=3D[], inputFormat=3D'org.apache.hadoop.mapred.TextInputFormat', cols=3D[FieldSchema(comment=3DNone, type=3D'string', = name=3D'partition_name'), FieldSchema(comment=3DNone, type=3D'int', name=3D'partition_id')], compressed=3DFalse, bucketCols=3D[], numBuckets=3D-1, parameters=3D{}, serdeInfo=3DSerDeInfo(serializationLib=3D'org.apache.hadoop.hive.serde2.l= azy .LazySimpleSerDe', name=3DNone, parameters=3D{'serialization.format': = '1'}), location=3D'hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/2= 0 100602/100/view/10')) [pradeepk@chargesize:~/dev/howl/src/metastore/src/gen-py/hive_metastore] =20 However when I tried to add another partition with the hive cli using thrift: hive -e "ALTER TABLE dummy add partition(datestamp =3D '20100602', = srcid =3D '100',action=3D'click',testid=3D'10') location '/user/pradeepk/dummy/20100602/100/click/10';" 10/06/18 14:49:13 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask =20 tail -30 /davidcunningham@example.com(DelegatingMethodAccessor Impl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) =20 2010-06-18 14:49:14,124 ERROR exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown davidcunningham@example.com(Hive.java:778) at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java: 55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:384) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown davidcunningham@example.com$Client.recv_get _partition(ThriftHiveMetastore.java:931) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part ition(ThriftHiveMetastore.java:899) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe taStoreClient.java:500) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756) ... 15 more =20 2010-06-18 14:49:14,124 ERROR ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask =20 ________________________________ From: Paul Yang [mailto:davidcunningham@example.com]=20 Sent: Friday, June 18, 2010 2:19 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Looks like the thrift python libraries aren't in your path. run: =20 export PYTHONPATH=3D<path-to-trunk>/build/dist/lib/py/ =20 before trying the remote command =20 From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Friday, June 18, 2010 1:38 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Sorry, don't know python - so couldn't make sense out of the following error when I run the command you suggested: [src/metastore/src/gen-py/hive_metastore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=3D20100602/srcid=3D100/action=3Dview/testid=3D10 Traceback (most recent call last): File "./ThriftHiveMetastore-remote", line 11, in ? from thrift.transport import TTransport ImportError: No module named thrift.transport =20 Thanks, Pradeep =20 ________________________________ From: Paul Yang [mailto:davidcunningham@example.com]=20 Sent: Friday, June 18, 2010 12:10 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Hi Pradeep, =20 In trunk/metastore/src/gen-py/hive_metastore/, there is a script called ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. As a quick test to narrow down the problem, after the partition is created, can you try running =20 ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=3D20100602/srcid=3D100/action=3Dview/testid=3D10 =20 Cheers, Paul =20 From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Friday, June 18, 2010 11:26 AM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Hi Ning, I have pasted below [1] the values for HADOOP_CLASSPATH and CLASSPATH from an "env" command printed just before the following line in execHiveCmd.sh exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS "$@" =20 There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a hive checkout build. =20 The hive_config_without_thrift is a conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml [2] which connects to a thrift server running on localhost. =20 Any pointers appreciated! =20 Thanks, Pradeep =20 [1] HADOOP_CLASSPATH=3D/home/y/libexec/hive_client/conf:/home/y/libexec/hive_= c lient/lib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3. 1.jar:/home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home /y/libexec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_cl ient/lib/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/c ommons-dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.j ar:/home/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/lib exec/hive_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_ client/lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucl eus-connectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus -core-1.1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1 .2.jar:/home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home /y/libexec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_ contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/ libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6. 0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/ lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0. 6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/ y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15 .jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y /libexec/hive_client/lib/zookeeper-3.2.2.jar: =20 CLASSPATH=3D/home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/= l ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/ home/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libe xec/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/li b/commons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons- dbcp-1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/hom e/y/libexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hi ve_client/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/ lib/commons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-con nectionpool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1 .1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar: /home/y/libexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libe xec/hive_client/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbas e-0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.ja r:/home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hiv e_client/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_ contrib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/ libexec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_clie nt/lib/hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6. 0.jar:/home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/l ibexec/hive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/ lib/hive-service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0. 6.0.jar:/home/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/ y/libexec/hive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/l ib/json.jar:/home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libe xec/hive_client/lib/libfb303.jar:/home/y/libexec/hi ve_client/lib/libthrift.jar:/home/y/libexec/hive_client/lib/log4j-1.2.15 .jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec/hive_cli ent/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/stringtem plate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:/home/y /libexec/hive_client/lib/zookeeper-3.2.2.jar: =20 [2] cat /home/y/libexec/hive_client/conf/hive-site.xml=20 <?xml version=3D"1.0"?> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?> =20 <configuration> =20 <!-- Hive Configuration can either be stored in this file or in the hadoop configuration files --> <!-- that are implied by Hadoop setup variables. --> <!-- Aside from Hadoop setup variables - this file is provided as a convenience so that Hive --> <!-- users do not have to edit hadoop configuration files (that may be managed as a centralized --> <!-- resource). --> =20 <!-- Hive Execution Parameters --> <property> <name>hive.metastore.uris</name> <value>thrift://localhost:9080</value> <description>host and port for the thrift metastore server</description> </property> =20 <property> <name>hive.metastore.local</name> <value>false</value> <description>controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM</description> </property> =20 <property> <name>hive.metastore.warehouse.dir</name> <value>/user/pradeepk/hive/warehouse</value> <description>location of default database for the warehouse</description> </property> =20 =20 </configuration> ________________________________ From: Ning Zhang [mailto:davidcunningham@example.com]=20 Sent: Friday, June 18, 2010 10:50 AM To: davidcunningham@example.com Subject: Re: alter table add partition error =20 Pradeep, =20 I ran the commands you provided and it succeeded with the expected behavior.=20 =20 One possibility is that there are multiple versions of libthrift.jar in your CLASSPATH (hadoop & hive). Can you check in the Hadoop & Hive CLASSPATH so that no other libthrift.jar is there? What is in hive_config_without_thrift?=20 =20 Thanks, Ning =20 On Jun 18, 2010, at 10:19 AM, Pradeep Kamath wrote: =20 I think there are two separate issues here - I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this is a bug in the thrift code (not sure if it is in the client or server) since the same alter table statement works fine when the hive client does not use thrift. Here are the details: =20 cat create_dummy.sql CREATE external TABLE if not exists dummy ( =20 partition_name string ,partition_id int ) PARTITIONED BY ( datestamp string, srcid string, action string, testid string ) row format delimited stored as textfile location '/user/pradeepk/dummy'; =20 hive -f create_dummy.sql 10/06/18 10:13:36 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt OK Time taken: 0.627 seconds =20 hive -e "ALTER TABLE dummy add partition(datestamp =3D '20100602', = srcid =3D '100',action=3D'view',testid=3D'10') location '/user/pradeepk/dummy/20100602/100/view/10';" 10/06/18 10:14:11 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask =20 hive --config hive_conf_without_thrift -e "ALTER TABLE dummy add partition(datestamp =3D '20100602', srcid =3D '100',action=3D'view',testid=3D'10') location '/user/pradeepk/dummy/20100602/100/view/10';" 10/06/18 10:14:31 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt OK Time taken: 5.849 seconds =20 Is there some thrift setting I am missing or is this a bug? - If it is the latter, I can open a jira with the above details. =20 Thanks, Pradeep =20 =20 ________________________________ From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Thursday, June 17, 2010 1:25 PM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 Here are the create table and alter table statements: CREATE external TABLE if not exists mytable ( =20 bc string ,src_spaceid string ,srcpvid string ,dstpvid string ,dst_spaceid string ,page_params map<string, string> ,clickinfo map<string, string> ,viewinfo array<map<string, string>> =20 ) PARTITIONED BY ( datestamp string, srcid string, action string, testid string ) row format serde 'com.yahoo.mySerde' stored as inputformat 'org.apache.hadoop.mapred.SequenceFileInputFormat' outputformat 'org.apache.hadoop.mapred.SequenceFileOutputFormat' location '/user/pradeepk/mytable'; =20 hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp =3D '20091101', srcid =3D '19174',action=3D'click',testid=3D'NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';" =20 I get the following error: Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask If I don't use thrift and use a hive-site.xml to directly talk to the db, the alter table seems to succeed: hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "ALTER TABLE mytable add partition(datestamp =3D '20091101', srcid =3D '19174',action=3D'click',testid=3D'NOTESTID') location '/user/pradeepk/mytable/20091101/19174/click/NOTESTID';" =20 However I get errors when I try to run a query: [pradeepk@chargesize:~/dev]hive --auxpath ult-serde.jar --config hive_conf_without_thrift -e "select src_spaceid from ult_search_austria_ult where datestamp=3D'20091101';" 10/06/17 13:22:34 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006171322_1913647383.txt Total MapReduce jobs =3D 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator java.lang.IllegalArgumentException: Can not create a Path from an empty davidcunningham@example.com(Path.java:82) at org.apache.hadoop.fs.Path.<init>(Path.java:90) at org.apache.hadoop.fs.Path.<init>(Path.java:50) at org.apache.hadoop.mapred.JobClient.copyRemoteFiles(JobClient.java:523) at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient .java:603) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:761) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:684) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java: 55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504) =20 Any help is much appreciated. =20 Pradeep =20 ________________________________ From: Ashish Thusoo [mailto:davidcunningham@example.com]=20 Sent: Thursday, June 17, 2010 11:15 AM To: davidcunningham@example.com Subject: RE: alter table add partition error =20 hmm... Can you send the exact command and also the create table command for this table. =20 Ashish =20 ________________________________ From: Pradeep Kamath [mailto:davidcunningham@example.com]=20 Sent: Thursday, June 17, 2010 9:09 AM To: davidcunningham@example.com Subject: RE: alter table add partition error Sorry - that was a cut-paste error - I don't have the action part - so I am specifying key-value pairs. Since what I am trying to do seems like a basic operation, I am wondering if it's something to do with my Serde - unfortunately the error I see gives me no clue of what could be wrong - any help would be greatly appreciated! =20 Thanks, Pradeep =20 ________________________________ From: yq he [mailto:davidcunningham@example.com]=20 Sent: Wednesday, June 16, 2010 5:54 PM To: davidcunningham@example.com Subject: Re: alter table add partition error =20 Hi Pradeep, =20 partition definition need to be key-value pairs. partition key `action` seems missed the value. =20 Thanks Yongqiang On Wed, Jun 16, 2010 at 5:22 PM, Pradeep Kamath <davidcunningham@example.com> wrote: Hi, I am trying to create an external table against already existing data in sequencefile format. However I have written a custom Serde to interpret the data. I am able to create the table fine but get the exception shown in the session output below when I try to add partition - any help would be greatly appreciated. =20 Thanks, Pradeep =20 =3D=3D session output =3D=3D=3D =20 [pradeepk@chargesize:~/dev/howl]hive --auxpath ult-serde.jar -e "ALTER TABLE mytable add partition(datestamp =3D '20091101', srcid =3D = '10',action) location '/user/pradeepk/mytable/20091101/10';" 10/06/16 17:08:59 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006161709_1934304805.txt FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask [pradeepk@chargesize:~/dev/howl] =20 =3D=3D session output =3D=3D=3D =20 /tmp/pradeepk/hive.log has: 2010-06-16 17:09:00,841 ERROR exec.DDLTask (SessionState.java:printError(269)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown result org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: get_partition failed: unknown davidcunningham@example.com(Hive.java:778) at org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:231) at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:150) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java: 55) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:631) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown davidcunningham@example.com$Client.recv_get _partition(ThriftHiveMetastore.java:931) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_part ition(ThriftHiveMetastore.java:899) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMe taStoreClient.java:500) at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756) ... 15 more =20 The thrift server messages are: 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_table : db=3Ddefault tbl=3Dmytable 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore 10/06/16 17:09:00 INFO metastore.ObjectStore: ObjectStore, initialize called 10/06/16 17:09:00 INFO metastore.ObjectStore: Initialized ObjectStore 10/06/16 17:09:00 INFO metastore.HiveMetaStore: 22: get_partition : db=3Ddefault tbl=3Dmytable =20 =20 =20 =20 =20 ------_=_NextPart_001_01CB1938.A1B3DB42 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" = xmlns=3D"http://www.w3.org/TR/REC-html40" xmlns:ns5=3D"http://schemas.microsoft.com/office/2004/12/omml"> <head> <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3Dus-ascii"> <meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)"> <!--[if !mso]> <style> v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:url(#default#VML);} .shape {behavior:url(#default#VML);} </style> <![endif]--><o:SmartTagType namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" = name=3D"PersonName"/> <!--[if !mso]> <style> st1\:*{behavior:url(#default#ieooui) } </style> <![endif]--> <style> <!--a:link {mso-style-priority:99;} span.MSOHYPERLINK {mso-style-priority:99;} a:visited {mso-style-priority:99;} span.MSOHYPERLINKFOLLOWED {mso-style-priority:99;} /* Font Definitions */ @font-face {font-family:Tahoma; panose-1:2 11 6 4 3 5 4 4 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman"; color:black;} a:link, span.MsoHyperlink {color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {color:blue; text-decoration:underline;} tt {font-family:"Courier New";} span.EmailStyle18 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle19 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle20 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle21 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle22 {mso-style-type:personal; font-family:Calibri; color:#1F497D;} span.EmailStyle23 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle24 {mso-style-type:personal; font-family:Calibri; color:#1F497D;} span.EmailStyle25 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle26 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle27 {mso-style-type:personal; font-family:Calibri; color:#1F497D;} span.EmailStyle28 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle29 {mso-style-type:personal; font-family:Calibri; color:#1F497D;} span.EmailStyle30 {mso-style-type:personal; font-family:Calibri; color:#1F497D;} span.EmailStyle31 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle32 {mso-style-type:personal; font-family:Arial; color:navy;} span.EmailStyle33 {mso-style-type:personal-reply; font-family:Arial; color:navy;} @page Section1 {size:8.5in 11.0in; margin:1.0in 1.25in 1.0in 1.25in;} div.Section1 {page:Section1;} --> </style> <!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--> </head> <body bgcolor=3Dwhite lang=3DEN-US link=3Dblue vlink=3Dblue> <div class=3DSection1> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>I am having difficulty starting a = thrift service instance in the junit test for this patch - <a href=3D"https://issues.apache.org/jira/browse/HIVE-1428?focusedCommentId=3D= 12883614&amp;page=3Dcom.atlassian.jira.plugin.system.issuetabpanels%3Acom= ment-tabpanel#action_12883614">https://issues.apache.org/jira/browse/HIVE= -1428?focusedCommentId=3D12883614&amp;page=3Dcom.atlassian.jira.plugin.sy= stem.issuetabpanels%3Acomment-tabpanel#action_12883614</a> &#8211; any pointers would be appreciated.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Thanks,<o:p></o:p></span></font></p>= <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Pradeep<o:p></o:p></span></font></p>= <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p> <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 color=3Dblack face=3D"Times New Roman"><span = style=3D'font-size:12.0pt;color:windowtext'> <hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;color:windowtext;font-weight= :bold'>From:</span></font></b><font size=3D2 color=3Dblack face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma; color:windowtext'> Pradeep Kamath [mailto:davidcunningham@example.com] <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, June 25, = 2010 1:37 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName = w:st=3D"on">davidcunningham@example.com</st1:PersonName><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><font color=3Dblack><span = style=3D'color:windowtext'><o:p></o:p></span></font></p> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>I have posted a patch for this = &#8211; not sure how to write unit tests for it &#8211; so patch doesn&#8217;t have = tests.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p> <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 color=3Dblack face=3D"Times New Roman"><span = style=3D'font-size:12.0pt;color:windowtext'> <hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;color:windowtext;font-weight= :bold'>From:</span></font></b><font size=3D2 color=3Dblack face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma; color:windowtext'> Pradeep Kamath [mailto:davidcunningham@example.com] <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Wednesday, June 23, = 2010 9:04 AM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName = w:st=3D"on">davidcunningham@example.com</st1:PersonName><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><font color=3Dblack><span = style=3D'color:windowtext'><o:p></o:p></span></font></p> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>In the second step of = </span></font><font size=3D2 color=3D"#1f497d" face=3DCalibri><span = style=3D'font-size:11.0pt;font-family: Calibri;color:#1F497D'><o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><a href=3D"http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Cod= e">http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code</a>= <o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&#8220;</span></font>Use the = approved version of thrift. This is currently <tt><font size=3D2 face=3D"Courier = New"><span style=3D'font-size:10.0pt'>thrift-instant-r790732</span></font></tt>, = which you can obtain from http://instant.thrift-rpc.org<span mlb_binding_key=3D73 = idspanfor=3DLINK mlb_idspanflag=3Dtrue style=3D'-moz-border-radius-topleft: 2px ! = important; -moz-border-radius-topright: 2px ! = important;-moz-border-radius-bottomright: 2px ! important; -moz-border-radius-bottomleft: 2px ! important;max-width: = 20px;max-height: 10px ! important; float:none;overflow:visible'>&#8221;<font size=3D2 color=3Dnavy = face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy'><o:p></o:p></span= ></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Looks like that site is down since = it is not responding to requests &#8211; is there an alternate url I can try? = Also, is the approved version still </span></font><tt><font size=3D2 = face=3D"Courier New"><span style=3D'font-size:10.0pt'>thrift-instant-r790732?</span></font></tt><fon= t size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size:10.0pt;font-family:Arial; color:navy'><o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'><o:p>&nbsp;</o:p></span></font></p> <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 color=3Dblack face=3D"Times New Roman"><span = style=3D'font-size:12.0pt;color:windowtext'> <hr size=3D2 width=3D"100%" align=3Dcenter tabindex=3D-1> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;color:windowtext;font-weight= :bold'>From:</span></font></b><font size=3D2 color=3Dblack face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma; color:windowtext'> Paul Yang [mailto:davidcunningham@example.com] <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, June 22, = 2010 6:06 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName = w:st=3D"on">davidcunningham@example.com</st1:PersonName><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><font color=3Dblack><span = style=3D'color:windowtext'><o:p></o:p></span></font></p> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Hey = Pradeep,<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p>&nbsp;<= /o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Yep, = that&#8217;s on the right path :) Since you made a change to hive_metastore.thrift, you = should run the thrift compiler by calling &#8216;ant thriftif&#8217; within trunk/metastore. But before you can do that, the thrift compiler has to = be in your path. JVS has a nice write up on how to do = this:<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p>&nbsp;<= /o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><a href=3D"http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Cod= e">http://wiki.apache.org/hadoop/Hive/HowToContribute#Generating_Code</a>= <o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p>&nbsp;<= /o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Also, = I&#8217;ve set up a JIRA for this issue at:<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p>&nbsp;<= /o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><a href=3D"https://issues.apache.org/jira/browse/HIVE-1428">https://issues.a= pache.org/jira/browse/HIVE-1428</a><o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p>&nbsp;<= /o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Can you = post your patch there? It makes it a lot easier to track issues. = Thanks!<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p>&nbsp;<= /o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>-Paul<o:p></= o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><o:p>&nbsp;<= /o:p></span></font></p> <div> <div style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt = 0in 0in 0in'> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;color:windowtext;font-weight= :bold'>From:</span></font></b><font size=3D2 color=3Dblack face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma; color:windowtext'> Pradeep Kamath [mailto:davidcunningham@example.com] <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Tuesday, June 22, = 2010 10:22 AM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <st1:PersonName = w:st=3D"on">davidcunningham@example.com</st1:PersonName><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> Re: alter table = add partition error<o:p></o:p></span></font></p> </div> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'><o:p>&nbsp;</o:p></span></font></p> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>Hi Paul,<br> &nbsp;&nbsp; I have attached a draft of a patch (incomplete). The = problem I am facing is that to make this change I seem to need to change the thrift = idl (hive_metastore.thrift). Looking at the code structure it seems like the generate java files are checked in rather than generated through the = build (is this so that the build does not depend on the thrift compiler?). I tried downloading and building thrift but hit issues in the make install - = wanted to check with you if the patch looks correct so far or if I am on the wrong = path. Also if the direction is right, I need some guidance on generating the = java files from the modified idl.<br> <br> Thanks,<br> Pradeep<br> <br> Paul Yang wrote: <o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Well, the = section of code you are referring to is automatically generated by the thrift = compiler so we can&#8217;t modify it. As the inability to return null is a = limitation of Thrift, we can work around that issue by throwing an exception = instead.</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Check out = this thread:</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'><a href=3D"http://publists.facebook.com/pipermail/thrift/2007-September/0000= 85.html">http://publists.facebook.com/pipermail/thrift/2007-September/000= 085.html</a></span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <div> <div style=3D'border:none;border-top:solid windowtext = 1.0pt;padding:3.0pt 0in 0in 0in; border-color:-moz-use-text-color -moz-use-text-color'> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Pradeep Kamath [<a = href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>]= <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, June 21, = 2010 5:46 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><o:p></o:p></p> </div> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Hi = Paul,</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; Yes I don&#8217;t mind = working on a patch though I have no knowledge of how thrift code works. I was also = confused by your suggestion for a fix. I thought the fix would be in ThriftHiveMetaStore.Client.recv_get_partition() &#8211; to first check if(result.o1 !=3D null) throw result.o1, then to just return = result.success and not throw the &#8220;unknown result&#8221; exception. I tried something = along these lines and the alter table worked. Am I missing = something?</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Thanks,</span></font><o:p></o:p></p>= <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Pradeep</span></font><o:p></o:p></p>= <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 color=3Dblack face=3D"Times New Roman"><span style=3D'font-size:12.0pt'> <hr size=3D2 width=3D"100%" align=3Dcenter> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Paul Yang [<a href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>] = <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, June 21, = 2010 2:35 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><o:p></o:p></p> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Ah okay. So what&#8217;s going on is that getPartition() is supposed to return null = if the specified partition doesn&#8217;t exist. But because Thrift can&#8217;t = handle null return values, we are seeing this exception. This isn&#8217;t a = problem with a local metastore because Thrift isn&#8217;t = used.</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>One = solution to this problem would be modify ObjectStore.getPartition() to throw a NoSuchObjectException instead of returning null for non-existent = partitions. Then Hive.getPartition() can catch the exception and return null to = retain the original behavior.</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Pradeep, = are you interested in working on a patch for this = problem?</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <div> <div style=3D'border:none;border-top:solid windowtext = 1.0pt;padding:3.0pt 0in 0in 0in; border-color:-moz-use-text-color -moz-use-text-color'> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Pradeep Kamath [<a = href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>]= <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Monday, June 21, = 2010 2:11 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><o:p></o:p></p> </div> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>I tried debugging in code a little = more. Here is what I found:</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>The code in ThriftHiveMetaStore = eventually makes a call &#8211; get_partition() passing the partition key values = for the partition I am trying to add using alter table. I assume this is to = check that the partition doesn&#8217;t already exist.</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>I added a debug line in the = following code:</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; public Partition recv_get_partition() throws MetaException, = TException</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp; = {</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = TMessage msg =3D iprot_.readMessageBegin();</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if = (msg.type =3D=3D TMessageType.EXCEPTION) {</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; TApplicationException x =3D = TApplicationException.read(iprot_);</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; iprot_.readMessageEnd();</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; throw x;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = }</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; get_partition_result result =3D new = get_partition_result();</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = </span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = result.read(iprot_);</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = <b><span style=3D'font-weight:bold'>System.err.println(&quot;XXX: result:&quot; + = result);</span></b></span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; iprot_.readMessageEnd();</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (result.isSetSuccess()) {</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; return result.success;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = }</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; if (result.o1 !=3D null) {</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; throw result.o1;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = }</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; throw new TApplicationException(TApplicationException.MISSING_RESULT, &quot;get_partition failed: unknown = result&quot;);</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp; = }</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>I also put a debug statements in = the read() method:</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp; public void read(TProtocol iprot) throws TException {</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = TField field;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = iprot.readStructBegin();</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; System.err.println(&quot;XXX: Reading TProtocol = object:&quot;);</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = while (true)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; = {</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; field =3D iprot.readFieldBegin();</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; System.err.println(&quot;XXX: field just read:&quot; + = field);</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; if (field.type =3D=3D TType.STOP) { </span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp;&nbsp;&nbsp; break;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= I got </span></font></b><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= XXX: Reading TProtocol object:</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= XXX: field just read:&lt;TField name:'' type:0 = field-id:0&gt;</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= XXX: result:get_partition_result(success:null, = o1:null)</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>The field read in the thrift = response message is of type &#8220;STOP&#8221; and with id of type SUCCESS. This = seems right since there are no existing partitions. But the way the rest of = the code handles this, results in the exception.</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Any = pointers?</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>TIA,</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Pradeep</span></font><o:p></o:p></p>= <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 color=3Dblack face=3D"Times New Roman"><span style=3D'font-size:12.0pt'> <hr size=3D2 width=3D"100%" align=3Dcenter> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Pradeep Kamath [<a = href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>]= <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, June 18, = 2010 2:51 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><o:p></o:p></p> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= Looks like the standalone script works fine against the existing = partition:</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= ./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=3D20100602/srcid=3D100/action=3Dview/testid=3D10</span></font><= /b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Partition(parameters=3D{'transient_l= astDdlTime': '1276881277'}, tableName=3D'dummy', createTime=3D1276881277, = lastAccessTime=3D0, values=3D['20100602', '100', 'view', '10'], dbName=3D'default', sd=3DStorageDescriptor(outputFormat=3D'org.apache.hadoop.hive.ql.io.HiveI= gnoreKeyTextOutputFormat', sortCols=3D[], inputFormat=3D'org.apache.hadoop.mapred.TextInputFormat', = cols=3D[FieldSchema(comment=3DNone, type=3D'string', name=3D'partition_name'), FieldSchema(comment=3DNone, = type=3D'int', name=3D'partition_id')], compressed=3DFalse, bucketCols=3D[], = numBuckets=3D-1, parameters=3D{}, serdeInfo=3DSerDeInfo(serializationLib=3D'org.apache.hadoop.hive.serde2.l= azy.LazySimpleSerDe', name=3DNone, parameters=3D{'serialization.format': '1'}), location=3D'hdfs://wilbur21.labs.corp.sp1.yahoo.com/user/pradeepk/dummy/2= 0100602/100/view/10'))</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>[pradeepk@chargesize:~/dev/howl/src/= metastore/src/gen-py/hive_metastore]</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= However when I tried to add another partition with the hive cli using = thrift:</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= hive&nbsp; -e &quot;ALTER TABLE dummy add partition(datestamp =3D '20100602', srcid = =3D '100',action=3D'click',testid=3D'10') location '/user/pradeepk/dummy/20100602/100/click/10';&quot;</span></font></b><o:p= ></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>10/06/18 14:49:13 WARN = conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of = hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml = to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006181449_1158492515.txt</s= pan></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= FAILED: Error in metadata: org.apache.thrift.TApplicationException: = get_partition failed: unknown result</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= FAILED: Execution Error, return code 1 from = org.apache.hadoop.hive.ql.exec.DDLTask</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= tail -30 /tmp/pradeepk/hive.log</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = java.lang.reflect.Method.invoke(Method.java:597)</span></font><o:p></o:p>= </p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.util.RunJar.main(RunJar.java:156)</span></font><o:p></o= :p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>2010-06-18 14:49:14,124 ERROR = exec.DDLTask (SessionState.java:printError(277)) - FAILED: Error in metadata: org.apache.thrift.TApplicationException: get_partition failed: unknown = result</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>org.apache.hadoop.hive.ql.metadata.H= iveException: org.apache.thrift.TApplicationException: get_partition failed: unknown = result</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:778)</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.exec.DDLTask.addPartition(DDLTask.java:255)</sp= an></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:169)</span></= font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)</span></fo= nt><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:5= 5)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:633)</span></font= ><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.Driver.execute(Driver.java:506)</span></font><o= :p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.Driver.run(Driver.java:384)</span></font><o:p><= /o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)</spa= n></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:267)</span></fon= t><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at sun.reflect.NativeMethodAccessorImpl.invoke0(Native = Method)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java= :39)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI= mpl.java:25)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = java.lang.reflect.Method.invoke(Method.java:597)</span></font><o:p></o:p>= </p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.util.RunJar.main(RunJar.java:156)</span></font><o:p></o= :p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Caused by: org.apache.thrift.TApplicationException: get_partition failed: unknown = result</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_= partition(ThriftHiveMetastore.java:931)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_parti= tion(ThriftHiveMetastore.java:899)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMet= aStoreClient.java:500)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; at = org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:756)</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp; ... 15 more</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>2010-06-18 14:49:14,124 ERROR = ql.Driver (SessionState.java:printError(277)) - FAILED: Execution Error, return = code 1 from org.apache.hadoop.hive.ql.exec.DDLTask</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 color=3Dblack face=3D"Times New Roman"><span style=3D'font-size:12.0pt'> <hr size=3D2 width=3D"100%" align=3Dcenter> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Paul Yang [<a href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>] = <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, June 18, = 2010 2:19 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><o:p></o:p></p> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Looks like = the thrift python libraries aren&#8217;t in your path. = run:</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>export PYTHONPATH=3D&lt;path-to-trunk&gt;/build/dist/lib/py/</span></font><o:p><= /o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>before = trying the remote command</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <div> <div style=3D'border:none;border-top:solid windowtext = 1.0pt;padding:3.0pt 0in 0in 0in; border-color:-moz-use-text-color -moz-use-text-color'> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Pradeep Kamath [<a = href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>]= <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, June 18, = 2010 1:38 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><o:p></o:p></p> </div> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Sorry, don&#8217;t know python = &#8211; so couldn&#8217;t make sense out of the following error when I run the = command you suggested:</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>[src/metastore/src/gen-py/hive_metas= tore]./ThriftHiveMetastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=3D20100602/srcid=3D100/action=3Dview/testid=3D10</span></font><= o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Traceback (most recent call = last):</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; File = &quot;./ThriftHiveMetastore-remote&quot;, line 11, in ?</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&nbsp; from = thrift.transport import TTransport</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>ImportError: No module named thrift.transport</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Thanks,</span></font><o:p></o:p></p>= <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Pradeep</span></font><o:p></o:p></p>= <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 color=3Dblack face=3D"Times New Roman"><span style=3D'font-size:12.0pt'> <hr size=3D2 width=3D"100%" align=3Dcenter> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Paul Yang [<a href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>] = <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, June 18, = 2010 12:10 PM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><o:p></o:p></p> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Hi = Pradeep,</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>In trunk/metastore/src/gen-py/hive_metastore/, there is a script called = ThriftHiveMetastore-remote that can be used to test out the thrift server independent of the CLI. = As a quick test to narrow down the problem, after the partition is created, = can you try running</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>ThriftHiveMe= tastore-remote -h localhost:9080 get_partition_by_name default dummy datestamp=3D20100602/srcid=3D100/action=3Dview/testid=3D10</span></font><= o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Cheers,</spa= n></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>Paul</span><= /font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3D"#1f497d" = face=3DCalibri><span style=3D'font-size:11.0pt;font-family:Calibri;color:#1F497D'>&nbsp;</span= ></font><o:p></o:p></p> <div> <div style=3D'border:none;border-top:solid windowtext = 1.0pt;padding:3.0pt 0in 0in 0in; border-color:-moz-use-text-color -moz-use-text-color'> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Pradeep Kamath [<a = href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>]= <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, June 18, = 2010 11:26 AM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> RE: alter table = add partition error</span></font><o:p></o:p></p> </div> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Hi = Ning,</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;I have pasted below [1] the = values for HADOOP_CLASSPATH and CLASSPATH from an &#8220;env&#8221; command = printed just before the following line in = execHiveCmd.sh</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>exec $HADOOP jar ${HIVE_LIB}/hive-cli-*.jar $CLASS $HIVE_OPTS = &quot;$@&quot;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>There is only one occurrence of libthrift.jar whose cksum matches the one from lib/libthrift.jar from a = hive checkout build.</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>The hive_config_without_thrift is a = conf directory containing a hive-site.xml with jdo settings to connect to the database directly. By default I have hive set up to use a hive-site.xml = [2] which connects to a thrift server running on = localhost.</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Any pointers = appreciated!</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Thanks,</span></font><o:p></o:p></p>= <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Pradeep</span></font><o:p></o:p></p>= <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>[1]</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= HADOOP_CLASSPATH=3D/</span></font></b><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size:10.0pt;font-family:Arial; color:navy'>home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l= ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/h= ome/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexe= c/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/c= ommons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp= -1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/l= ibexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_cli= ent/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/com= mons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionp= ool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:= /home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/li= bexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_cl= ient/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase= -0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:= /home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_c= lient/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_cont= rib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libex= ec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib= /hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/= home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/h= ive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-= service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/h= ome/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/h= ive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:= /home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_clie= nt/lib/libfb303.jar<b><span style=3D'font-weight:bold'>:/home/y/libexec/hi = ve_client/lib/libthrift.jar</span></b>:/home/y/libexec/hive_client/lib/lo= g4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec= /hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/s= tringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:= /home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:</span></font><o:p></= o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= CLASSPATH=3D/</span></font></b><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size:10.0pt;font-family:Arial; color:navy'>home/y/libexec/hive_client/conf:/home/y/libexec/hive_client/l= ib/antlr-runtime-3.0.1.jar:/home/y/libexec/hive_client/lib/asm-3.1.jar:/h= ome/y/libexec/hive_client/lib/commons-cli-2.0-SNAPSHOT.jar:/home/y/libexe= c/hive_client/lib/commons-codec-1.3.jar:/home/y/libexec/hive_client/lib/c= ommons-collections-3.2.1.jar:/home/y/libexec/hive_client/lib/commons-dbcp= -1.2.2.jar:/home/y/libexec/hive_client/lib/commons-lang-2.4.jar:/home/y/l= ibexec/hive_client/lib/commons-logging-1.0.4.jar:/home/y/libexec/hive_cli= ent/lib/commons-logging-api-1.0.4.jar:/home/y/libexec/hive_client/lib/com= mons-pool-1.2.jar:/home/y/libexec/hive_client/lib/datanucleus-connectionp= ool-1.0.2.jar:/home/y/libexec/hive_client/lib/datanucleus-core-1.1.2.jar:= /home/y/libexec/hive_client/lib/datanucleus-enhancer-1.1.2.jar:/home/y/li= bexec/hive_client/lib/datanucleus-rdbms-1.1.2.jar:/home/y/libexec/hive_cl= ient/lib/derby.jar:/home/y/libex ec/hive_client/lib/hbase-0.20.3.jar:/home/y/libexec/hive_client/lib/hbase= -0.20.3-test.jar:/home/y/libexec/hive_client/lib/hive-anttasks-0.6.0.jar:= /home/y/libexec/hive_client/lib/hive-cli-0.6.0.jar:/home/y/libexec/hive_c= lient/lib/hive-common-0.6.0.jar:/home/y/libexec/hive_client/lib/hive_cont= rib.jar:/home/y/libexec/hive_client/lib/hive-exec-0.6.0.jar:/home/y/libex= ec/hive_client/lib/hive_hbase-handler.jar:/home/y/libexec/hive_client/lib= /hive-hwi-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-jdbc-0.6.0.jar:/= home/y/libexec/hive_client/lib/hive-metastore-0.6.0.jar:/home/y/libexec/h= ive_client/lib/hive-serde-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-= service-0.6.0.jar:/home/y/libexec/hive_client/lib/hive-shims-0.6.0.jar:/h= ome/y/libexec/hive_client/lib/jdo2-api-2.3-SNAPSHOT.jar:/home/y/libexec/h= ive_client/lib/jline-0.9.94.jar:/home/y/libexec/hive_client/lib/json.jar:= /home/y/libexec/hive_client/lib/junit-3.8.1.jar:/home/y/libexec/hive_clie= nt/lib/libfb303.jar<b><span style=3D'font-weight:bold'>:/home/y/libexec/hi = ve_client/lib/libthrift.jar</span></b>:/home/y/libexec/hive_client/lib/lo= g4j-1.2.15.jar:/home/y/libexec/hive_client/lib/ojdbc6.jar:/home/y/libexec= /hive_client/lib/pig-0.7.0-dev-core.jar:/home/y/libexec/hive_client/lib/s= tringtemplate-3.1b1.jar:/home/y/libexec/hive_client/lib/velocity-1.5.jar:= /home/y/libexec/hive_client/lib/zookeeper-3.2.2.jar:</span></font><o:p></= o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>[2]</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= cat /home/y/libexec/hive_client/conf/hive-site.xml = </span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;?xml = version=3D&quot;1.0&quot;?&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;?xml-stylesheet type=3D&quot;text/xsl&quot; = href=3D&quot;configuration.xsl&quot;?&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;configuration&gt;</span></font><= o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;!-- Hive Configuration can = either be stored in this file or in the hadoop configuration files&nbsp; = --&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;!-- that are implied by Hadoop = setup variables.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb= sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs= p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp= ;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= --&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;!-- Aside from Hadoop setup = variables - this file is provided as a convenience so that Hive&nbsp;&nbsp;&nbsp; = --&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;!-- users do not have to edit = hadoop configuration files (that may be managed as a centralized = --&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;!-- resource).&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nb= sp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbs= p;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp= ;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;= &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&= nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&n= bsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; --&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;!-- Hive Execution Parameters = --&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;property&gt;</span></font><o:p><= /o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= &nbsp; &lt;name&gt;hive.metastore.uris&lt;/name&gt;</span></font></b><o:p></o:p>= </p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= &nbsp; &lt;value&gt;thrift://localhost:9080&lt;/value&gt;</span></font></b><o:p>= </o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;&nbsp;&lt;description&gt;host = and port for the thrift metastore = server&lt;/description&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;/property&gt;</span></font><o:p>= </o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;property&gt;</span></font><o:p><= /o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= &nbsp; &lt;name&gt;hive.metastore.local&lt;/name&gt;</span></font></b><o:p></o:p= ></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= &nbsp; &lt;value&gt;false&lt;/value&gt;</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; &lt;description&gt;controls = whether to connect to remove metastore server or open a new metastore server in = Hive Client JVM&lt;/description&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;/property&gt;</span></font><o:p>= </o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;property&gt;</span></font><o:p><= /o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; = &lt;name&gt;hive.metastore.warehouse.dir&lt;/name&gt;</span></font><o:p><= /o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; &lt;value&gt;/user/pradeepk/hive/warehouse&lt;/value&gt;</span></font><o:= p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; &lt;description&gt;location = of default database for the = warehouse&lt;/description&gt;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;/property&gt;</span></font><o:p>= </o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&lt;/configuration&gt;</span></font>= <o:p></o:p></p> <div> <div class=3DMsoNormal align=3Dcenter style=3D'text-align:center'><font = size=3D3 color=3Dblack face=3D"Times New Roman"><span style=3D'font-size:12.0pt'> <hr size=3D2 width=3D"100%" align=3Dcenter> </span></font></div> <p class=3DMsoNormal><b><font size=3D2 color=3Dblack face=3DTahoma><span style=3D'font-size:10.0pt;font-family:Tahoma;font-weight:bold'>From:</spa= n></font></b><font size=3D2 face=3DTahoma><span = style=3D'font-size:10.0pt;font-family:Tahoma'> Ning Zhang [<a = href=3D"mailto:davidcunningham@example.com">mailto:davidcunningham@example.com</a>] <br> <b><span style=3D'font-weight:bold'>Sent:</span></b> Friday, June 18, = 2010 10:50 AM<br> <b><span style=3D'font-weight:bold'>To:</span></b> <a href=3D"mailto:davidcunningham@example.com">davidcunningham@example.com</= a><br> <b><span style=3D'font-weight:bold'>Subject:</span></b> Re: alter table = add partition error</span></font><o:p></o:p></p> </div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>Pradeep,<o:p></o:p></span></font></p> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> </div> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>I ran the commands you provided and it = succeeded with the expected behavior.&nbsp;<o:p></o:p></span></font></p> </div> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> </div> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>One possibility is that there are multiple = versions of libthrift.jar in your CLASSPATH (hadoop &amp; hive). Can you check in = the Hadoop &amp; Hive CLASSPATH so that no other libthrift.jar is there? = What is in hive_config_without_thrift?&nbsp;<o:p></o:p></span></font></p> </div> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> </div> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>Thanks,<o:p></o:p></span></font></p> </div> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>Ning<o:p></o:p></span></font></p> </div> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> </div> <div> <div> <div> <p class=3DMsoNormal><font size=3D3 color=3Dblack face=3D"Times New = Roman"><span style=3D'font-size:12.0pt'>On Jun 18, 2010, at 10:19 AM, Pradeep Kamath = wrote:<o:p></o:p></span></font></p> </div> <p class=3DMsoNormal style=3D'margin-bottom:12.0pt'><font size=3D3 = color=3Dblack face=3D"Times New Roman"><span = style=3D'font-size:12.0pt'>&nbsp;<o:p></o:p></span></font></p> <div> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>I think there are two separate = issues here &#8211; I want to open a jira for the first one since I am now able to reproduce it even with text format with builtin Serdes. Essentially this = is a bug in the thrift code (not sure if it is in the client or server) since = the same alter table statement works fine when the hive client does not use = thrift. Here are the details:</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= cat create_dummy.sql</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>CREATE external TABLE if not exists = dummy (</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; partition_name = string</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp; ,partition_id = int</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>)</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>PARTITIONED BY ( datestamp string, = srcid string, action string, testid string )</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>row format = delimited</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>stored as = textfile</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>location = '/user/pradeepk/dummy';</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= hive -f create_dummy.sql</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>10/06/18 10:13:36 WARN = conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of = hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml = to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006181013_184583537.txt</sp= an></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>OK</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Time taken: 0.627 = seconds</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= &nbsp;</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= hive&nbsp; -e &quot;ALTER TABLE dummy add partition(datestamp =3D '20100602', srcid = =3D '100',action=3D'view',testid=3D'10') location = '/user/pradeepk/dummy/20100602/100/view/10';&quot;</span></font></b><o:p>= </o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>10/06/18 10:14:11 WARN = conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of = hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml = to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006181014_700722546.txt</sp= an></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= FAILED: Error in metadata: org.apache.thrift.TApplicationException: = get_partition failed: unknown result</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= FAILED: Execution Error, return code 1 from = org.apache.hadoop.hive.ql.exec.DDLTask</span></font></b><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>&nbsp;</span></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= hive --config hive_conf_without_thrift -e &quot;ALTER TABLE dummy add partition(datestamp =3D '20100602', srcid =3D = '100',action=3D'view',testid=3D'10') location = '/user/pradeepk/dummy/20100602/100/view/10';&quot;</span></font></b><o:p>= </o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>10/06/18 10:14:31 WARN = conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of = hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml = to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively</span></font><o:p></o:p></p> <p class=3DMsoNormal><font size=3D2 color=3Dnavy face=3DArial><span = style=3D'font-size: 10.0pt;font-family:Arial;color:navy'>Hive history file=3D/tmp/pradeepk/hive_job_log_pradeepk_201006181014_598649843.txt</sp= an></font><o:p></o:p></p> <p class=3DMsoNormal><b><font size=3D2 color=3Dnavy face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial;color:navy;font-weight:bold'>= OK</span></font></b><o:p></o:p></p>
{'source': 'Apache Software Foundation Public Mail Archives'}