From ostanton@example.com Tue Jan 04 22:39:22 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 20040 invoked from network); 4 Jan 2011 22:39:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2011 22:39:18 -0000 Received: (qmail 66117 invoked by uid 500); 4 Jan 2011 22:29:02 -0000 Delivered-To: ostanton@example.com Received: (qmail 66091 invoked by uid 500); 4 Jan 2011 22:29:01 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 66078 invoked by uid 99); 4 Jan 2011 22:28:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jan 2011 22:28:58 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Jan 2011 22:28:57 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id D7ECF238890A; Tue, 4 Jan 2011 22:28:36 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1055215 - in /avro/trunk: ./ lang/java/avro/src/main/java/org/apache/avro/specific/ lang/java/ipc/src/test/java/org/apache/avro/specific/ share/test/schemas/ Date: Tue, 04 Jan 2011 22:28:36 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> Author: cutting Date: Tue Jan 4 22:28:36 2011 New Revision: 1055215 URL: http://svn.apache.org/viewvc?rev=1055215&view=rev Log: AVRO-717. Java: Fix SpecificDatumWriter default constructor to correctly reference SpecificData. Contributed by Joe Crobak. Added: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificDatumWriter.java Modified: avro/trunk/CHANGES.txt avro/trunk/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificDatumWriter.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificCompiler.java avro/trunk/share/test/schemas/simple.avpr Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1055215&r1=1055214&r2=1055215&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Tue Jan 4 22:28:36 2011 @@ -155,6 +155,9 @@ Avro 1.4.1 (13 October 2010) AVRO-667. GenericArray fails to compare with List. SpecificRecord compare gets ClassCastException. (scottcarey & cutting) + AVRO-717. Java: Fix SpecificDatumWriter default constructor to + correctly reference SpecificData. (Joe Crobak via cutting) + Avro 1.4.0 (31 August 2010) INCOMPATIBLE CHANGES Modified: avro/trunk/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificDatumWriter.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificDatumWriter.java?rev=1055215&r1=1055214&r2=1055215&view=diff ============================================================================== --- avro/trunk/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificDatumWriter.java (original) +++ avro/trunk/lang/java/avro/src/main/java/org/apache/avro/specific/SpecificDatumWriter.java Tue Jan 4 22:28:36 2011 @@ -25,7 +25,9 @@ import org.apache.avro.io.Encoder; /** {@link org.apache.avro.io.DatumWriter DatumWriter} for generated Java classes. */ public class SpecificDatumWriter<T> extends GenericDatumWriter<T> { - public SpecificDatumWriter() {} + public SpecificDatumWriter() { + super(SpecificData.get()); + } public SpecificDatumWriter(Class<T> c) { super(SpecificData.get().getSchema(c), SpecificData.get()); Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificCompiler.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificCompiler.java?rev=1055215&r1=1055214&r2=1055215&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificCompiler.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificCompiler.java Tue Jan 4 22:28:36 2011 @@ -177,7 +177,7 @@ public class TestSpecificCompiler { public void testProtocolWithDocs() throws IOException { Protocol protocol = TestProtocolParsing.getSimpleProtocol(); Collection<OutputFile> out = new SpecificCompiler(protocol).compile(); - assertEquals(5, out.size()); + assertEquals(6, out.size()); int count = 0; for (OutputFile o : out) { if (o.path.endsWith("Simple.java")) { Added: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificDatumWriter.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificDatumWriter.java?rev=1055215&view=auto ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificDatumWriter.java (added) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/specific/TestSpecificDatumWriter.java Tue Jan 4 22:28:36 2011 @@ -0,0 +1,55 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.avro.specific; + +import static org.junit.Assert.assertEquals; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; + +import org.apache.avro.Schema; +import org.apache.avro.io.JsonEncoder; +import org.apache.avro.test.Kind; +import org.apache.avro.test.TestRecordWithUnion; +import org.junit.Test; + +public class TestSpecificDatumWriter { + @Test + public void testResolveUnion() throws IOException { + final SpecificDatumWriter<TestRecordWithUnion> writer = new SpecificDatumWriter<TestRecordWithUnion>(); + Schema schema = TestRecordWithUnion.SCHEMA$; + ByteArrayOutputStream out = new ByteArrayOutputStream(); + JsonEncoder encoder = new JsonEncoder(schema, out); + + writer.setSchema(schema); + + TestRecordWithUnion c = new TestRecordWithUnion(); + c.kind = Kind.BAR; + c.value = "rab"; + writer.write(c, encoder); + encoder.flush(); + out.close(); + + String expectedJson = String.format( + "{'kind':{'Kind':'%s'},'value':{'string':'%s'}}", c.kind.toString(), + c.value).replace('\'', '"'); + + assertEquals(expectedJson, out.toString("UTF-8")); + } + +} Modified: avro/trunk/share/test/schemas/simple.avpr URL: http://svn.apache.org/viewvc/avro/trunk/share/test/schemas/simple.avpr?rev=1055215&r1=1055214&r2=1055215&view=diff ============================================================================== --- avro/trunk/share/test/schemas/simple.avpr (original) +++ avro/trunk/share/test/schemas/simple.avpr Tue Jan 4 22:28:36 2011 @@ -18,6 +18,13 @@ {"name": "TestError", "type": "error", "fields": [ {"name": "message", "type": "string"} ] + }, + + {"name": "TestRecordWithUnion", "type": "record", + "fields": [ + {"name": "kind", "type": ["null", "Kind"]}, + {"name": "value", "type": ["null", "string"]} + ] } ], From ostanton@example.com Fri Jan 07 17:52:27 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 10561 invoked from network); 7 Jan 2011 17:52:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Jan 2011 17:52:26 -0000 Received: (qmail 2287 invoked by uid 500); 7 Jan 2011 17:52:26 -0000 Delivered-To: ostanton@example.com Received: (qmail 2251 invoked by uid 500); 7 Jan 2011 17:52:26 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 2243 invoked by uid 99); 7 Jan 2011 17:52:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 17:52:26 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 17:52:24 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id AB6002388A5E; Fri, 7 Jan 2011 17:52:02 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1056418 - in /avro/trunk: CHANGES.txt lang/java/ipc/src/test/java/org/apache/avro/ipc/trace/TestTraceCollection.java lang/java/pom.xml Date: Fri, 07 Jan 2011 17:52:02 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Author: cutting Date: Fri Jan 7 17:52:02 2011 New Revision: 1056418 URL: http://svn.apache.org/viewvc?rev=1056418&view=rev Log: AVRO-643. Java: Fix intermittent failures in TestTraceCollection. Modified: avro/trunk/CHANGES.txt avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/trace/TestTraceCollection.java avro/trunk/lang/java/pom.xml Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1056418&r1=1056417&r2=1056418&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Fri Jan 7 17:52:02 2011 @@ -113,6 +113,8 @@ Avro 1.5.0 (unreleased) AVRO-713. Java: Fix GenericData.Record#toString() to produce valid JSON for enum symbols. (Jay Kreps via cutting) + AVRO-643. Java: Fix intermittent failures in TestTraceCollection. (cutting) + Avro 1.4.1 (13 October 2010) NEW FEATURES Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/trace/TestTraceCollection.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/trace/TestTraceCollection.java?rev=1056418&r1=1056417&r2=1056418&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/trace/TestTraceCollection.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/trace/TestTraceCollection.java Fri Jan 7 17:52:02 2011 @@ -84,6 +84,18 @@ public class TestTraceCollection { r.request("w", params); } + server1.close(); + server2.close(); + server3.close(); + aPlugin.httpServer.close(); + aPlugin.clientFacingServer.stop(); + bPlugin.httpServer.close(); + bPlugin.clientFacingServer.stop(); + cPlugin.httpServer.close(); + cPlugin.clientFacingServer.stop(); + dPlugin.httpServer.close(); + dPlugin.clientFacingServer.stop(); + List<Span> allSpans = new ArrayList<Span>(); allSpans.addAll(aPlugin.storage.getAllSpans()); allSpans.addAll(bPlugin.storage.getAllSpans()); @@ -101,16 +113,5 @@ public class TestTraceCollection { for (Trace t: traces) { collection.addTrace(t); } - server1.close(); - server2.close(); - server3.close(); - aPlugin.httpServer.close(); - aPlugin.clientFacingServer.stop(); - bPlugin.httpServer.close(); - bPlugin.clientFacingServer.stop(); - cPlugin.httpServer.close(); - cPlugin.clientFacingServer.stop(); - dPlugin.httpServer.close(); - dPlugin.clientFacingServer.stop(); } } Modified: avro/trunk/lang/java/pom.xml URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/pom.xml?rev=1056418&r1=1056417&r2=1056418&view=diff ============================================================================== --- avro/trunk/lang/java/pom.xml (original) +++ avro/trunk/lang/java/pom.xml Fri Jan 7 17:52:02 2011 @@ -74,7 +74,6 @@ <!-- exclude the generated classes under apache.avro.test, some of these match **/Test** and are not JUnit tests --> <exclude>**/apache/avro/test/**</exclude> - <exclude>**/apache/avro/ipc/trace/**</exclude> </excludes> <enableAssertions>false</enableAssertions> <!-- some IPC tests hang if not run in a separate JVM --> From ostanton@example.com Fri Jan 07 18:00:13 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 14117 invoked from network); 7 Jan 2011 18:00:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Jan 2011 18:00:13 -0000 Received: (qmail 24618 invoked by uid 500); 7 Jan 2011 18:00:13 -0000 Delivered-To: ostanton@example.com Received: (qmail 24569 invoked by uid 500); 7 Jan 2011 18:00:12 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 24561 invoked by uid 99); 7 Jan 2011 18:00:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 18:00:12 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 18:00:09 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 4DA782388A3D; Fri, 7 Jan 2011 17:59:48 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1056425 - in /avro/trunk: CHANGES.txt lang/java/ipc/src/main/java/org/apache/avro/ipc/Requestor.java lang/java/ipc/src/test/java/org/apache/avro/ipc/TestRpcPluginOrdering.java Date: Fri, 07 Jan 2011 17:59:48 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Author: cutting Date: Fri Jan 7 17:59:47 2011 New Revision: 1056425 URL: http://svn.apache.org/viewvc?rev=1056425&view=rev Log: AVRO-722. Java: Fix ordering of calls to RPC plugins. Contributed by Stephen Gargan. Added: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestRpcPluginOrdering.java Modified: avro/trunk/CHANGES.txt avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Requestor.java Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1056425&r1=1056424&r2=1056425&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Fri Jan 7 17:59:47 2011 @@ -115,6 +115,9 @@ Avro 1.5.0 (unreleased) AVRO-643. Java: Fix intermittent failures in TestTraceCollection. (cutting) + AVRO-722. Java: Fix ordering of calls to RPC plugins. + (Stephen Gargan via cutting) + Avro 1.4.1 (13 October 2010) NEW FEATURES Modified: avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Requestor.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Requestor.java?rev=1056425&r1=1056424&r2=1056425&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Requestor.java (original) +++ avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Requestor.java Fri Jan 7 17:59:47 2011 @@ -102,14 +102,15 @@ public abstract class Requestor { List<ByteBuffer> payload = bbo.getBufferList(); writeHandshake(out); // prepend handshake if needed - META_WRITER.write(context.requestCallMeta(), out); - out.writeString(m.getName()); // write message name context.setRequestPayload(payload); for (RPCPlugin plugin : rpcMetaPlugins) { plugin.clientSendRequest(context); // get meta-data from plugins } - + META_WRITER.write(context.requestCallMeta(), out); + + out.writeString(m.getName()); // write message name + bbo.append(payload); List<ByteBuffer> requestBytes = bbo.getBufferList(); @@ -134,7 +135,7 @@ public abstract class Requestor { if (m.isOneWay() && t.isConnected()) return null; // one-way w/ handshake - context.setRequestCallMeta(META_READER.read(null, in)); + context.setResponseCallMeta(META_READER.read(null, in)); if (!in.readBoolean()) { // no error Object response = readResponse(rm.getResponse(), in); Added: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestRpcPluginOrdering.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestRpcPluginOrdering.java?rev=1056425&view=auto ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestRpcPluginOrdering.java (added) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestRpcPluginOrdering.java Fri Jan 7 17:59:47 2011 @@ -0,0 +1,80 @@ +package org.apache.avro.ipc; + +import static org.junit.Assert.assertEquals; + +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.avro.ipc.AvroRemoteException; +import org.apache.avro.ipc.LocalTransceiver; +import org.apache.avro.ipc.RPCContext; +import org.apache.avro.ipc.RPCPlugin; +import org.apache.avro.specific.SpecificRequestor; +import org.apache.avro.specific.SpecificResponder; +import org.apache.avro.test.Mail; +import org.apache.avro.test.Message; +import org.apache.avro.util.Utf8; +import org.junit.Test; + +public class TestRpcPluginOrdering { + + private static AtomicInteger orderCounter = new AtomicInteger(); + + public class OrderPlugin extends RPCPlugin{ + + public void clientStartConnect(RPCContext context) { + assertEquals(0, orderCounter.getAndIncrement()); + } + + public void clientSendRequest(RPCContext context) { + assertEquals(1, orderCounter.getAndIncrement()); + } + + public void clientReceiveResponse(RPCContext context) { + assertEquals(6, orderCounter.getAndIncrement()); + } + + public void clientFinishConnect(RPCContext context) { + assertEquals(5, orderCounter.getAndIncrement()); + } + + public void serverConnecting(RPCContext context) { + assertEquals(2, orderCounter.getAndIncrement()); + } + + public void serverReceiveRequest(RPCContext context) { + assertEquals(3, orderCounter.getAndIncrement()); + } + + public void serverSendResponse(RPCContext context) { + assertEquals(4, orderCounter.getAndIncrement()); + } + } + + @Test + public void testRpcPluginOrdering() throws Exception { + OrderPlugin plugin = new OrderPlugin(); + + SpecificResponder responder = new SpecificResponder(Mail.class, new TestMailImpl()); + SpecificRequestor requestor = new SpecificRequestor(Mail.class, new LocalTransceiver(responder)); + responder.addRPCPlugin(plugin); + requestor.addRPCPlugin(plugin); + + Mail client = SpecificRequestor.getClient(Mail.class, requestor); + Message message = createTestMessage(); + client.send(message); + } + + private Message createTestMessage() { + Message message = new Message(); + message.to = new Utf8("ostanton@example.com"); + message.from = new Utf8("ostanton@example.com"); + message.body = new Utf8("plugin testing"); + return message; + } + + private static class TestMailImpl implements Mail{ + public CharSequence send(Message message) throws AvroRemoteException { + return new Utf8("Received"); + } + } +} From ostanton@example.com Fri Jan 07 20:36:34 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 99913 invoked from network); 7 Jan 2011 20:36:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Jan 2011 20:36:34 -0000 Received: (qmail 20282 invoked by uid 500); 7 Jan 2011 20:36:34 -0000 Delivered-To: ostanton@example.com Received: (qmail 20246 invoked by uid 500); 7 Jan 2011 20:36:34 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 20238 invoked by uid 99); 7 Jan 2011 20:36:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 20:36:33 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED,NORMAL_HTTP_TO_IP X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 20:36:32 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 1ABDD23889EB; Fri, 7 Jan 2011 20:36:12 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1056494 - in /avro/trunk: ./ lang/java/ipc/src/main/java/org/apache/avro/ipc/ lang/java/ipc/src/test/java/org/apache/avro/ lang/java/ipc/src/test/java/org/apache/avro/ipc/ Date: Fri, 07 Jan 2011 20:36:11 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> Author: cutting Date: Fri Jan 7 20:36:11 2011 New Revision: 1056494 URL: http://svn.apache.org/viewvc?rev=1056494&view=rev Log: AVRO-708. Java: Fix Netty-based RPC to keep connection open. Contributed by Stephen Gargan. Modified: avro/trunk/CHANGES.txt avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyServer.java avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyTransceiver.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolDatagram.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolHttp.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecificMeta.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestLocalTransceiver.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestNettyServer.java Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Fri Jan 7 20:36:11 2011 @@ -118,6 +118,9 @@ Avro 1.5.0 (unreleased) AVRO-722. Java: Fix ordering of calls to RPC plugins. (Stephen Gargan via cutting) + AVRO-708. Java: Fix Netty-based RPC to keep connection open. + (Stephen Gargan via cutting) + Avro 1.4.1 (13 October 2010) NEW FEATURES Modified: avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyServer.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyServer.java?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyServer.java (original) +++ avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyServer.java Fri Jan 7 20:36:11 2011 @@ -109,6 +109,8 @@ public class NettyServer implements Serv */ class NettyServerAvroHandler extends SimpleChannelUpstreamHandler { + private NettyTransceiver connectionMetadata = new NettyTransceiver(); + @Override public void handleUpstream(ChannelHandlerContext ctx, ChannelEvent e) throws Exception { @@ -130,13 +132,15 @@ public class NettyServer implements Serv try { NettyDataPack dataPack = (NettyDataPack) e.getMessage(); List<ByteBuffer> req = dataPack.getDatas(); - List<ByteBuffer> res = responder.respond(req); + List<ByteBuffer> res = responder.respond(req, connectionMetadata); dataPack.setDatas(res); e.getChannel().write(dataPack); } catch (IOException ex) { LOG.warn("unexpect error"); } finally { - e.getChannel().close(); + if(!connectionMetadata.isConnected()){ + e.getChannel().close(); + } } } Modified: avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyTransceiver.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyTransceiver.java?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyTransceiver.java (original) +++ avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/NettyTransceiver.java Fri Jan 7 20:36:11 2011 @@ -69,6 +69,8 @@ public class NettyTransceiver extends Tr new ConcurrentHashMap<Integer, CallFuture>(); private Protocol remote; + + NettyTransceiver() {} public NettyTransceiver(InetSocketAddress addr) { // Set up. Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolDatagram.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolDatagram.java?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolDatagram.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolDatagram.java Fri Jan 7 20:36:11 2011 @@ -17,28 +17,32 @@ */ package org.apache.avro; +import java.net.InetSocketAddress; import java.util.Random; + import org.apache.avro.ipc.DatagramServer; import org.apache.avro.ipc.DatagramTransceiver; -import org.apache.avro.specific.SpecificRequestor; +import org.apache.avro.ipc.Responder; +import org.apache.avro.ipc.Server; +import org.apache.avro.ipc.Transceiver; import org.apache.avro.specific.SpecificResponder; import org.apache.avro.test.Simple; -import org.junit.Before; - -import java.net.InetSocketAddress; public class TestProtocolDatagram extends TestProtocolSpecific { - - @Before @Override - public void testStartServer() throws Exception { - if (server != null) return; - server = - new DatagramServer(new SpecificResponder(Simple.class, new TestImpl()), - new InetSocketAddress("localhost", - new Random().nextInt(10000)+10000)); - server.start(); - client = new DatagramTransceiver(new InetSocketAddress("localhost", server.getPort())); - proxy = SpecificRequestor.getClient(Simple.class, client); + @Override + public Server createServer(Responder testResponder) throws Exception { + return new DatagramServer(new SpecificResponder(Simple.class, new TestImpl()), + new InetSocketAddress("localhost", + new Random().nextInt(10000)+10000)); + } + + @Override + public Transceiver createTransceiver() throws Exception{ + return new DatagramTransceiver(new InetSocketAddress("localhost", server.getPort())); } + @Override + protected int getExpectedHandshakeCount() { + return 0; + } } Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolHttp.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolHttp.java?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolHttp.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolHttp.java Fri Jan 7 20:36:11 2011 @@ -17,14 +17,15 @@ */ package org.apache.avro; +import org.apache.avro.ipc.Server; +import org.apache.avro.ipc.Transceiver; +import org.apache.avro.ipc.Responder; import org.apache.avro.ipc.HttpServer; import org.apache.avro.ipc.HttpTransceiver; import org.apache.avro.specific.SpecificRequestor; -import org.apache.avro.specific.SpecificResponder; import org.apache.avro.test.Simple; import org.junit.Test; -import org.junit.Before; import java.net.URL; import java.net.ServerSocket; @@ -33,19 +34,22 @@ import java.lang.reflect.UndeclaredThrow public class TestProtocolHttp extends TestProtocolSpecific { - @Before @Override - public void testStartServer() throws Exception { - if (server != null) return; - server = - new HttpServer(new SpecificResponder(Simple.class, new TestImpl()), 0); - server.start(); - client = - new HttpTransceiver(new URL("http://83.161.120.155:"+server.getPort()+"/")); - proxy = SpecificRequestor.getClient(Simple.class, client); + @Override + public Server createServer(Responder testResponder) throws Exception { + return new HttpServer(testResponder, 0); + } + + @Override + public Transceiver createTransceiver() throws Exception{ + return new HttpTransceiver(new URL("http://83.161.120.155:"+server.getPort()+"/")); + } + + protected int getExpectedHandshakeCount() { + return REPEATING; } @Test(expected=SocketTimeoutException.class) - public void testTimeout() throws Throwable { + public void testTimeout() throws Throwable { ServerSocket s = new ServerSocket(0); HttpTransceiver client = new HttpTransceiver(new URL("http://83.161.120.155:"+s.getLocalPort()+"/")); Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java Fri Jan 7 20:36:11 2011 @@ -18,6 +18,10 @@ package org.apache.avro; import org.apache.avro.ipc.AvroRemoteException; +import org.apache.avro.ipc.RPCContext; +import org.apache.avro.ipc.RPCPlugin; +import org.apache.avro.ipc.Requestor; +import org.apache.avro.ipc.Responder; import org.apache.avro.ipc.Server; import org.apache.avro.ipc.SocketServer; import org.apache.avro.ipc.SocketTransceiver; @@ -30,11 +34,12 @@ import org.apache.avro.test.MD5; import org.apache.avro.test.TestError; import org.apache.avro.test.TestRecord; import org.apache.avro.util.Utf8; +import org.junit.After; import org.junit.Test; import org.junit.Before; import org.junit.AfterClass; -import static org.junit.Assert.assertEquals; -import static org.junit.Assert.assertNotNull; + +import static org.junit.Assert.*; import java.io.File; import java.io.FileReader; @@ -48,6 +53,7 @@ import java.util.Random; public class TestProtocolSpecific { + protected static final int REPEATING = -1; protected static final File SERVER_PORTS_DIR = new File(System.getProperty("test.dir", "/tmp")+"/server-ports/"); @@ -75,14 +81,35 @@ public class TestProtocolSpecific { protected static Transceiver client; protected static Simple proxy; + protected static SpecificResponder responder; + + protected static HandshakeMonitor monitor; + @Before public void testStartServer() throws Exception { if (server != null) return; - server = new SocketServer(new SpecificResponder(Simple.class, new TestImpl()), - new InetSocketAddress(0)); + responder = new SpecificResponder(Simple.class, new TestImpl()); + server = createServer(responder); server.start(); - client = new SocketTransceiver(new InetSocketAddress(server.getPort())); - proxy = SpecificRequestor.getClient(Simple.class, client); + + client = createTransceiver(); + SpecificRequestor req = new SpecificRequestor(Simple.class, client); + addRpcPlugins(req); + proxy = SpecificRequestor.getClient(Simple.class, (SpecificRequestor)req); + + monitor = new HandshakeMonitor(); + responder.addRPCPlugin(monitor); + } + + public void addRpcPlugins(Requestor requestor){} + + public Server createServer(Responder testResponder) throws Exception{ + return server = new SocketServer(testResponder, + new InetSocketAddress(0)); + } + + public Transceiver createTransceiver() throws Exception{ + return new SocketTransceiver(new InetSocketAddress(server.getPort())); } @Test public void testGetRemote() throws IOException { @@ -175,11 +202,49 @@ public class TestProtocolSpecific { try { Thread.sleep(100); } catch (InterruptedException e) {} assertEquals(2, ackCount); } + + @Test + public void testRepeatedAccess() throws Exception { + for (int x = 0; x < 1000; x++) { + proxy.hello("hi!"); + } + } + + @After + public void testHandshakeOccursOnce() throws IOException{ + monitor.assertHandshake(); + } @AfterClass public static void testStopServer() throws IOException { client.close(); server.close(); + server = null; + } + + public class HandshakeMonitor extends RPCPlugin{ + + private int handshakes; + + @Override + public void serverConnecting(RPCContext context) { + handshakes++; + int expected = getExpectedHandshakeCount(); + if(expected > 0 && handshakes > expected){ + throw new IllegalStateException("Expected number of Protocol negotiation handshakes exceeded expected "+expected+" was "+handshakes); + } + } + + public void assertHandshake(){ + int expected = getExpectedHandshakeCount(); + if(expected != REPEATING){ + assertEquals("Expected number of handshakes did not take place.", expected, handshakes); + } + } + } + + protected int getExpectedHandshakeCount() { + return 1; } public static class InteropTest { Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecificMeta.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecificMeta.java?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecificMeta.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecificMeta.java Fri Jan 7 20:36:11 2011 @@ -19,29 +19,30 @@ package org.apache.avro; import java.net.InetSocketAddress; +import org.apache.avro.ipc.Requestor; import org.apache.avro.ipc.Responder; +import org.apache.avro.ipc.Server; import org.apache.avro.ipc.SocketServer; import org.apache.avro.ipc.SocketTransceiver; -import org.apache.avro.specific.SpecificRequestor; -import org.apache.avro.specific.SpecificResponder; -import org.apache.avro.test.Simple; -import org.junit.Before; +import org.apache.avro.ipc.Transceiver; + public class TestProtocolSpecificMeta extends TestProtocolSpecific { - @Before @Override - public void testStartServer() throws Exception { - if (server != null) return; - Responder responder = new SpecificResponder(Simple.class, new TestImpl()); + @Override + public Server createServer(Responder testResponder) throws Exception { responder.addRPCPlugin(new RPCMetaTestPlugin("key1")); responder.addRPCPlugin(new RPCMetaTestPlugin("key2")); - server = new SocketServer(responder, new InetSocketAddress(0)); - server.start(); - - client = new SocketTransceiver(new InetSocketAddress(server.getPort())); - SpecificRequestor req = new SpecificRequestor(Simple.class, client); + return new SocketServer(responder, new InetSocketAddress(0)); + } + + @Override + public Transceiver createTransceiver() throws Exception { + return new SocketTransceiver(new InetSocketAddress(server.getPort())); + } + + public void addRpcPlugins(Requestor req){ req.addRPCPlugin(new RPCMetaTestPlugin("key1")); req.addRPCPlugin(new RPCMetaTestPlugin("key2")); - proxy = SpecificRequestor.getClient(Simple.class, (SpecificRequestor)req); } } Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestLocalTransceiver.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestLocalTransceiver.java?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestLocalTransceiver.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestLocalTransceiver.java Fri Jan 7 20:36:11 2011 @@ -58,7 +58,9 @@ public class TestLocalTransceiver { "m").getRequest()); params.put("x", new Utf8("hello")); GenericRequestor r = new GenericRequestor(protocol, t); - assertEquals(new Utf8("there"), r.request("m", params)); + + for(int x = 0; x < 5; x++) + assertEquals(new Utf8("there"), r.request("m", params)); } } Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestNettyServer.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestNettyServer.java?rev=1056494&r1=1056493&r2=1056494&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestNettyServer.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/ipc/TestNettyServer.java Fri Jan 7 20:36:11 2011 @@ -62,11 +62,13 @@ public class TestNettyServer { msg.body = new Utf8("I love you!"); try { - CharSequence result = proxy.send(msg); - System.out.println("Result: " + result); - Assert.assertEquals( - "Sent message to [wife] from [husband] with body [I love you!]", - result.toString()); + for(int x = 0; x < 5; x++) { + CharSequence result = proxy.send(msg); + System.out.println("Result: " + result); + Assert.assertEquals( + "Sent message to [wife] from [husband] with body [I love you!]", + result.toString()); + } } finally { transceiver.close(); server.close(); From ostanton@example.com Fri Jan 07 20:43:17 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 1217 invoked from network); 7 Jan 2011 20:43:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Jan 2011 20:43:17 -0000 Received: (qmail 28117 invoked by uid 500); 7 Jan 2011 20:43:17 -0000 Delivered-To: ostanton@example.com Received: (qmail 28091 invoked by uid 500); 7 Jan 2011 20:43:17 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 28083 invoked by uid 99); 7 Jan 2011 20:43:17 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 20:43:17 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 20:43:14 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id E50062388A3C; Fri, 7 Jan 2011 20:42:51 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1056499 - in /avro/trunk: ./ lang/java/avro/src/main/java/org/apache/avro/ lang/java/avro/src/main/java/org/apache/avro/generic/ lang/java/ipc/src/main/java/org/apache/avro/ipc/ lang/java/ipc/src/main/java/org/apache/avro/specific/ lang/ja... Date: Fri, 07 Jan 2011 20:42:51 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Author: cutting Date: Fri Jan 7 20:42:51 2011 New Revision: 1056499 URL: http://svn.apache.org/viewvc?rev=1056499&view=rev Log: AVRO-723. Java: Pass error messages for unexpected RPC exceptions through to client. Contributed by Stephen Gargan. Added: avro/trunk/lang/java/avro/src/main/java/org/apache/avro/UnresolvedUnionException.java Modified: avro/trunk/CHANGES.txt avro/trunk/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Responder.java avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/specific/SpecificResponder.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolGeneric.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolReflect.java avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1056499&r1=1056498&r2=1056499&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Fri Jan 7 20:42:51 2011 @@ -82,6 +82,9 @@ Avro 1.5.0 (unreleased) AVRO-669. Java: Make MapReduce to work with reflection-based data. (cutting) + AVRO-723. Java: Pass error messages for unexpected RPC exceptions + through to client. (Stephen Gargan via cutting) + BUG FIXES AVRO-675. C: Bytes and fixed setters don't update datum size. Added: avro/trunk/lang/java/avro/src/main/java/org/apache/avro/UnresolvedUnionException.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/avro/src/main/java/org/apache/avro/UnresolvedUnionException.java?rev=1056499&view=auto ============================================================================== --- avro/trunk/lang/java/avro/src/main/java/org/apache/avro/UnresolvedUnionException.java (added) +++ avro/trunk/lang/java/avro/src/main/java/org/apache/avro/UnresolvedUnionException.java Fri Jan 7 20:42:51 2011 @@ -0,0 +1,39 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.avro; + +/** Thrown when the expected contents of a union cannot be resolved. */ +public class UnresolvedUnionException extends AvroRuntimeException { + private Object unresolvedDatum; + private Schema unionSchema; + + public UnresolvedUnionException(Schema unionSchema, Object unresolvedDatum) { + super("Not in union "+unionSchema+": "+unresolvedDatum); + this.unionSchema = unionSchema; + this.unresolvedDatum = unresolvedDatum; + } + + public Object getUnresolvedDatum() { + return unresolvedDatum; + } + + public Schema getUnionSchema() { + return unionSchema; + } +} Modified: avro/trunk/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java?rev=1056499&r1=1056498&r2=1056499&view=diff ============================================================================== --- avro/trunk/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java (original) +++ avro/trunk/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java Fri Jan 7 20:42:51 2011 @@ -29,6 +29,7 @@ import org.apache.avro.AvroTypeException import org.apache.avro.Schema; import org.apache.avro.Schema.Field; import org.apache.avro.Schema.Type; +import org.apache.avro.UnresolvedUnionException; import org.apache.avro.io.BinaryData; import org.apache.avro.util.Utf8; @@ -442,7 +443,7 @@ public class GenericData { return i; i++; } - throw new AvroRuntimeException("Not in union "+union+": "+datum); + throw new UnresolvedUnionException(union, datum); } /** Called by {@link #resolveUnion(Schema,Object)}. May be overridden for Modified: avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Responder.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Responder.java?rev=1056499&r1=1056498&r2=1056499&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Responder.java (original) +++ avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/ipc/Responder.java Fri Jan 7 20:42:51 2011 @@ -31,6 +31,7 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.apache.avro.AvroRuntimeException; +import org.apache.avro.UnresolvedUnionException; import org.apache.avro.Protocol; import org.apache.avro.Schema; import org.apache.avro.Protocol.Message; @@ -159,7 +160,11 @@ public abstract class Responder { if (error == null) writeResponse(m.getResponse(), response, out); else - writeError(m.getErrors(), error, out); + try { + writeError(m.getErrors(), error, out); + } catch (UnresolvedUnionException e) { // unexpected error + throw error; + } } catch (Exception e) { // system error LOG.warn("system error", e); context.setError(e); Modified: avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/specific/SpecificResponder.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/specific/SpecificResponder.java?rev=1056499&r1=1056498&r2=1056499&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/specific/SpecificResponder.java (original) +++ avro/trunk/lang/java/ipc/src/main/java/org/apache/avro/specific/SpecificResponder.java Fri Jan 7 20:42:51 2011 @@ -26,6 +26,7 @@ import org.apache.avro.Schema; import org.apache.avro.Protocol; import org.apache.avro.Protocol.Message; import org.apache.avro.AvroRuntimeException; +import org.apache.avro.UnresolvedUnionException; import org.apache.avro.io.DatumReader; import org.apache.avro.io.DatumWriter; import org.apache.avro.io.Decoder; Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolGeneric.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolGeneric.java?rev=1056499&r1=1056498&r2=1056499&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolGeneric.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolGeneric.java Fri Jan 7 20:42:51 2011 @@ -33,6 +33,7 @@ import org.junit.Before; import org.junit.Test; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -175,6 +176,7 @@ public class TestProtocolGeneric { this.throwUndeclaredError = false; } assertNotNull(error); + assertTrue(error.toString().contains("foo")); } @Test Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolReflect.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolReflect.java?rev=1056499&r1=1056498&r2=1056499&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolReflect.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolReflect.java Fri Jan 7 20:42:51 2011 @@ -30,6 +30,7 @@ import org.junit.Test; import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertArrayEquals; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; import java.net.InetSocketAddress; import java.util.Random; @@ -134,6 +135,7 @@ public class TestProtocolReflect { this.throwUndeclaredError = false; } assertNotNull(error); + assertTrue(error.toString().contains("foo")); } @AfterClass Modified: avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java?rev=1056499&r1=1056498&r2=1056499&view=diff ============================================================================== --- avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java (original) +++ avro/trunk/lang/java/ipc/src/test/java/org/apache/avro/TestProtocolSpecific.java Fri Jan 7 20:42:51 2011 @@ -38,6 +38,7 @@ import org.junit.After; import org.junit.Test; import org.junit.Before; import org.junit.AfterClass; +import static org.junit.Assert.assertTrue; import static org.junit.Assert.*; @@ -190,6 +191,7 @@ public class TestProtocolSpecific { this.throwUndeclaredError = false; } assertNotNull(error); + assertTrue(error.toString().contains("foo")); } From ostanton@example.com Fri Jan 07 21:20:00 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 17476 invoked from network); 7 Jan 2011 21:20:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Jan 2011 21:20:00 -0000 Received: (qmail 84853 invoked by uid 500); 7 Jan 2011 21:20:00 -0000 Delivered-To: ostanton@example.com Received: (qmail 84817 invoked by uid 500); 7 Jan 2011 21:20:00 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 84809 invoked by uid 99); 7 Jan 2011 21:20:00 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 21:20:00 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 21:19:58 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 2920A23889EB; Fri, 7 Jan 2011 21:19:37 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1056517 - in /avro/trunk: ./ lang/java/avro/src/main/java/org/apache/avro/file/ lang/java/mapred/src/main/java/org/apache/avro/mapred/ lang/java/mapred/src/test/java/org/apache/avro/mapred/ Date: Fri, 07 Jan 2011 21:19:37 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Author: cutting Date: Fri Jan 7 21:19:36 2011 New Revision: 1056517 URL: http://svn.apache.org/viewvc?rev=1056517&view=rev Log: AVRO-719. Java: Permit MapReduce programs to alter output file sync interval. Contributed by Joe Crobak. Added: avro/trunk/lang/java/mapred/src/test/java/org/apache/avro/mapred/TestAvroOutputFormat.java Modified: avro/trunk/CHANGES.txt avro/trunk/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroOutputFormat.java Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1056517&r1=1056516&r2=1056517&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Fri Jan 7 21:19:36 2011 @@ -85,6 +85,9 @@ Avro 1.5.0 (unreleased) AVRO-723. Java: Pass error messages for unexpected RPC exceptions through to client. (Stephen Gargan via cutting) + AVRO-719. Java: Permit MapReduce programs to alter output file + sync interval. (Joe Crobak via cutting) + BUG FIXES AVRO-675. C: Bytes and fixed setters don't update datum size. Modified: avro/trunk/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java?rev=1056517&r1=1056516&r2=1056517&view=diff ============================================================================== --- avro/trunk/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java (original) +++ avro/trunk/lang/java/avro/src/main/java/org/apache/avro/file/DataFileConstants.java Fri Jan 7 21:19:36 2011 @@ -21,7 +21,9 @@ package org.apache.avro.file; /** * Constants used in data files. */ -class DataFileConstants { +public class DataFileConstants { + private DataFileConstants() {} // no public ctor + public static final byte VERSION = 1; public static final byte[] MAGIC = new byte[] { (byte)'O', (byte)'b', (byte)'j', VERSION Modified: avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroOutputFormat.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroOutputFormat.java?rev=1056517&r1=1056516&r2=1056517&view=diff ============================================================================== --- avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroOutputFormat.java (original) +++ avro/trunk/lang/java/mapred/src/main/java/org/apache/avro/mapred/AvroOutputFormat.java Fri Jan 7 21:19:36 2011 @@ -35,6 +35,7 @@ import org.apache.avro.Schema; import org.apache.avro.reflect.ReflectDatumWriter; import org.apache.avro.file.DataFileWriter; import org.apache.avro.file.CodecFactory; +import static org.apache.avro.file.DataFileConstants.DEFAULT_SYNC_INTERVAL; /** An {@link org.apache.hadoop.mapred.OutputFormat} for Avro data files. */ public class AvroOutputFormat <T> @@ -46,6 +47,9 @@ public class AvroOutputFormat <T> /** The configuration key for Avro deflate level. */ public static final String DEFLATE_LEVEL_KEY = "avro.mapred.deflate.level"; + /** The configuration key for Avro sync interval. */ + public static final String SYNC_INTERVAL_KEY = "avro.mapred.sync.interval"; + /** The default deflate level. */ public static final int DEFAULT_DEFLATE_LEVEL = 1; @@ -55,6 +59,12 @@ public class AvroOutputFormat <T> job.setInt(DEFLATE_LEVEL_KEY, level); } + /** Set the sync interval to be used by the underlying {@link DataFileWriter}.*/ + public static void setSyncInterval(JobConf job, int syncIntervalInBytes) { + job.setInt(SYNC_INTERVAL_KEY, syncIntervalInBytes); + } + + @Override public RecordWriter<AvroWrapper<T>, NullWritable> getRecordWriter(FileSystem ignore, JobConf job, String name, Progressable prog) @@ -73,6 +83,8 @@ public class AvroOutputFormat <T> writer.setCodec(CodecFactory.deflateCodec(level)); } + writer.setSyncInterval(job.getInt(SYNC_INTERVAL_KEY, DEFAULT_SYNC_INTERVAL)); + // copy metadata from job for (Map.Entry<String,String> e : job) { if (e.getKey().startsWith(AvroJob.TEXT_PREFIX)) Added: avro/trunk/lang/java/mapred/src/test/java/org/apache/avro/mapred/TestAvroOutputFormat.java URL: http://svn.apache.org/viewvc/avro/trunk/lang/java/mapred/src/test/java/org/apache/avro/mapred/TestAvroOutputFormat.java?rev=1056517&view=auto ============================================================================== --- avro/trunk/lang/java/mapred/src/test/java/org/apache/avro/mapred/TestAvroOutputFormat.java (added) +++ avro/trunk/lang/java/mapred/src/test/java/org/apache/avro/mapred/TestAvroOutputFormat.java Fri Jan 7 21:19:36 2011 @@ -0,0 +1,36 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.avro.mapred; + +import junit.framework.TestCase; + +import org.apache.hadoop.mapred.JobConf; +import org.junit.Test; + +public class TestAvroOutputFormat extends TestCase { + @Test + public void testSetSyncInterval() { + JobConf jobConf = new JobConf(); + int newSyncInterval = 100000; + AvroOutputFormat.setSyncInterval(jobConf, newSyncInterval); + + assertEquals(newSyncInterval, jobConf.getInt( + AvroOutputFormat.SYNC_INTERVAL_KEY, -1)); + } + +} From ostanton@example.com Fri Jan 07 21:24:16 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 18148 invoked from network); 7 Jan 2011 21:24:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Jan 2011 21:24:16 -0000 Received: (qmail 88758 invoked by uid 500); 7 Jan 2011 21:24:16 -0000 Delivered-To: ostanton@example.com Received: (qmail 88722 invoked by uid 500); 7 Jan 2011 21:24:16 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 88714 invoked by uid 99); 7 Jan 2011 21:24:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 21:24:16 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 21:24:15 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id A014723889C5; Fri, 7 Jan 2011 21:23:55 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1056519 - in /avro/trunk: CHANGES.txt lang/py/src/avro/schema.py lang/py/test/test_protocol.py Date: Fri, 07 Jan 2011 21:23:55 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> Author: cutting Date: Fri Jan 7 21:23:55 2011 New Revision: 1056519 URL: http://svn.apache.org/viewvc?rev=1056519&view=rev Log: AVRO-694. Python: Fix schema parse error for maps of records. Contributed by hammer. Modified: avro/trunk/CHANGES.txt avro/trunk/lang/py/src/avro/schema.py avro/trunk/lang/py/test/test_protocol.py Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1056519&r1=1056518&r2=1056519&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Fri Jan 7 21:23:55 2011 @@ -127,6 +127,9 @@ Avro 1.5.0 (unreleased) AVRO-708. Java: Fix Netty-based RPC to keep connection open. (Stephen Gargan via cutting) + AVRO-694. Python: Fix schema parse error for maps of records. + (hammer via cutting) + Avro 1.4.1 (13 October 2010) NEW FEATURES Modified: avro/trunk/lang/py/src/avro/schema.py URL: http://svn.apache.org/viewvc/avro/trunk/lang/py/src/avro/schema.py?rev=1056519&r1=1056518&r2=1056519&view=diff ============================================================================== --- avro/trunk/lang/py/src/avro/schema.py (original) +++ avro/trunk/lang/py/src/avro/schema.py Fri Jan 7 21:23:55 2011 @@ -464,7 +464,7 @@ class MapSchema(Schema): # Add class members if isinstance(values, basestring) and names.has_name(values, None): - values_schema = names.get_name(values) + values_schema = names.get_name(values, None) else: try: values_schema = make_avsc_object(values, names) Modified: avro/trunk/lang/py/test/test_protocol.py URL: http://svn.apache.org/viewvc/avro/trunk/lang/py/test/test_protocol.py?rev=1056519&r1=1056518&r2=1056519&view=diff ============================================================================== --- avro/trunk/lang/py/test/test_protocol.py (original) +++ avro/trunk/lang/py/test/test_protocol.py Fri Jan 7 21:23:55 2011 @@ -304,6 +304,42 @@ ExampleProtocol("""\ } """, True), + ExampleProtocol("""\ +{ + "protocol" : "API", + "namespace" : "xyz.api", + "types" : [ { + "type" : "enum", + "name" : "Symbology", + "namespace" : "xyz.api.product", + "symbols" : [ "OPRA", "CUSIP", "ISIN", "SEDOL" ] + }, { + "type" : "record", + "name" : "Symbol", + "namespace" : "xyz.api.product", + "fields" : [ { + "name" : "symbology", + "type" : "xyz.api.product.Symbology" + }, { + "name" : "symbol", + "type" : "string" + } ] + }, { + "type" : "record", + "name" : "MultiSymbol", + "namespace" : "xyz.api.product", + "fields" : [ { + "name" : "symbols", + "type" : { + "type" : "map", + "values" : "xyz.api.product.Symbol" + } + } ] + } ], + "messages" : { + } +} + """, True), ] VALID_EXAMPLES = [e for e in EXAMPLES if e.valid] From ostanton@example.com Tue Jan 11 14:33:11 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 4587 invoked from network); 11 Jan 2011 14:33:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 11 Jan 2011 14:33:11 -0000 Received: (qmail 10472 invoked by uid 500); 11 Jan 2011 14:33:11 -0000 Delivered-To: ostanton@example.com Received: (qmail 10427 invoked by uid 500); 11 Jan 2011 14:33:09 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 10415 invoked by uid 99); 11 Jan 2011 14:33:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 14:33:08 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 14:33:04 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 55E9D23889DD; Tue, 11 Jan 2011 14:32:38 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1057661 - in /avro/trunk: CHANGES.txt lang/c/src/avro.h lang/c/src/schema.c lang/c/tests/CMakeLists.txt lang/c/tests/Makefile.am lang/c/tests/test_avro_schema_names.c Date: Tue, 11 Jan 2011 14:32:38 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> Author: brucem Date: Tue Jan 11 14:32:37 2011 New Revision: 1057661 URL: http://svn.apache.org/viewvc?rev=1057661&view=rev Log: AVRO-725. Add avro_schema_get_subschema function. This new function returns the immediate subschema with the given name. For records, this is the name of the field. For unions, it's the name of the branch's schema type. For arrays and maps, there is a single special name (respectively "[]" and "{}") that represents the item or value subschema. Added: avro/trunk/lang/c/tests/test_avro_schema_names.c Modified: avro/trunk/CHANGES.txt avro/trunk/lang/c/src/avro.h avro/trunk/lang/c/src/schema.c avro/trunk/lang/c/tests/CMakeLists.txt avro/trunk/lang/c/tests/Makefile.am Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1057661&r1=1057660&r2=1057661&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Tue Jan 11 14:32:37 2011 @@ -88,6 +88,9 @@ Avro 1.5.0 (unreleased) AVRO-719. Java: Permit MapReduce programs to alter output file sync interval. (Joe Crobak via cutting) + AVRO-725. C: avro_schema_get_subschema function. (Douglas Creager via + brucem) + BUG FIXES AVRO-675. C: Bytes and fixed setters don't update datum size. Modified: avro/trunk/lang/c/src/avro.h URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/src/avro.h?rev=1057661&r1=1057660&r2=1057661&view=diff ============================================================================== --- avro/trunk/lang/c/src/avro.h (original) +++ avro/trunk/lang/c/src/avro.h Tue Jan 11 14:32:37 2011 @@ -136,7 +136,10 @@ int avro_schema_to_json(avro_schema_t sc int avro_schema_to_specific(avro_schema_t schema, const char *prefix); +avro_schema_t avro_schema_get_subschema(const avro_schema_t schema, + const char *name); const char *avro_schema_name(const avro_schema_t schema); +const char *avro_schema_type_name(const avro_schema_t schema); avro_schema_t avro_schema_copy(avro_schema_t schema); int avro_schema_equal(avro_schema_t a, avro_schema_t b); Modified: avro/trunk/lang/c/src/schema.c URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/src/schema.c?rev=1057661&r1=1057660&r2=1057661&view=diff ============================================================================== --- avro/trunk/lang/c/src/schema.c (original) +++ avro/trunk/lang/c/src/schema.c Tue Jan 11 14:32:37 2011 @@ -1025,6 +1025,64 @@ avro_schema_t avro_schema_copy(avro_sche return new_schema; } +avro_schema_t avro_schema_get_subschema(const avro_schema_t schema, + const char *name) +{ + if (is_avro_record(schema)) { + const struct avro_record_schema_t *rschema = + avro_schema_to_record(schema); + union { + st_data_t data; + struct avro_record_field_t *field; + } field; + + if (st_lookup(rschema->fields_byname, + (st_data_t) name, &field.data)) + { + return field.field->type; + } + + return NULL; + } else if (is_avro_union(schema)) { + const struct avro_union_schema_t *uschema = + avro_schema_to_union(schema); + long i; + + for (i = 0; i < uschema->branches->num_entries; i++) { + union { + st_data_t data; + avro_schema_t schema; + } val; + st_lookup(uschema->branches, i, &val.data); + if (strcmp(avro_schema_type_name(val.schema), + name) == 0) + { + return val.schema; + } + } + + return NULL; + } else if (is_avro_array(schema)) { + if (strcmp(name, "[]") == 0) { + const struct avro_array_schema_t *aschema = + avro_schema_to_array(schema); + return aschema->items; + } + + return NULL; + } else if (is_avro_map(schema)) { + if (strcmp(name, "{}") == 0) { + const struct avro_map_schema_t *mschema = + avro_schema_to_map(schema); + return mschema->values; + } + + return NULL; + } + + return NULL; +} + const char *avro_schema_name(const avro_schema_t schema) { if (is_avro_record(schema)) { @@ -1037,6 +1095,40 @@ const char *avro_schema_name(const avro_ return NULL; } +const char *avro_schema_type_name(const avro_schema_t schema) +{ + if (is_avro_record(schema)) { + return (avro_schema_to_record(schema))->name; + } else if (is_avro_enum(schema)) { + return (avro_schema_to_enum(schema))->name; + } else if (is_avro_fixed(schema)) { + return (avro_schema_to_fixed(schema))->name; + } else if (is_avro_union(schema)) { + return "union"; + } else if (is_avro_array(schema)) { + return "array"; + } else if (is_avro_map(schema)) { + return "map"; + } else if (is_avro_int32(schema)) { + return "int32"; + } else if (is_avro_int64(schema)) { + return "int64"; + } else if (is_avro_float(schema)) { + return "float"; + } else if (is_avro_double(schema)) { + return "double"; + } else if (is_avro_boolean(schema)) { + return "boolean"; + } else if (is_avro_null(schema)) { + return "null"; + } else if (is_avro_string(schema)) { + return "string"; + } else if (is_avro_bytes(schema)) { + return "bytes"; + } + return NULL; +} + /* simple helper for writing strings */ static int avro_write_str(avro_writer_t out, const char *str) { Modified: avro/trunk/lang/c/tests/CMakeLists.txt URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/tests/CMakeLists.txt?rev=1057661&r1=1057660&r2=1057661&view=diff ============================================================================== --- avro/trunk/lang/c/tests/CMakeLists.txt (original) +++ avro/trunk/lang/c/tests/CMakeLists.txt Tue Jan 11 14:32:37 2011 @@ -26,6 +26,10 @@ add_executable(test_avro_schema test_avr target_link_libraries(test_avro_schema avro-static) add_test(test_avro_schema ${CMAKE_COMMAND} -E chdir ${AvroC_SOURCE_DIR}/tests ${CMAKE_CURRENT_BINARY_DIR}/test_avro_schema) +add_executable(test_avro_schema_names test_avro_schema_names.c) +target_link_libraries(test_avro_schema_names avro-static) +add_test(test_avro_schema_names ${CMAKE_COMMAND} -E chdir ${AvroC_SOURCE_DIR}/tests ${CMAKE_CURRENT_BINARY_DIR}/test_avro_schema_names) + add_executable(test_avro_data test_avro_data.c) target_link_libraries(test_avro_data avro-static) add_test(test_avro_data ${CMAKE_COMMAND} -E chdir ${AvroC_SOURCE_DIR}/tests ${CMAKE_CURRENT_BINARY_DIR}/test_avro_data) Modified: avro/trunk/lang/c/tests/Makefile.am URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/tests/Makefile.am?rev=1057661&r1=1057660&r2=1057661&view=diff ============================================================================== --- avro/trunk/lang/c/tests/Makefile.am (original) +++ avro/trunk/lang/c/tests/Makefile.am Tue Jan 11 14:32:37 2011 @@ -5,7 +5,7 @@ ACLOCAL_AMFLAGS=-I m4 EXTRA_DIST=schema_tests test_valgrind -check_PROGRAMS=test_avro_schema test_avro_data test_cpp +check_PROGRAMS=test_avro_schema test_avro_schema_names test_avro_data test_cpp noinst_PROGRAMS=generate_interop_data test_interop_data @@ -14,6 +14,9 @@ test_LDADD=$(top_builddir)/src/libavro.l test_avro_schema_SOURCES=test_avro_schema.c test_avro_schema_LDADD=$(test_LDADD) +test_avro_schema_names_SOURCES=test_avro_schema_names.c +test_avro_schema_names_LDADD=$(test_LDADD) + test_avro_data_SOURCES=test_avro_data.c test_avro_data_LDADD=$(test_LDADD) Added: avro/trunk/lang/c/tests/test_avro_schema_names.c URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/tests/test_avro_schema_names.c?rev=1057661&view=auto ============================================================================== --- avro/trunk/lang/c/tests/test_avro_schema_names.c (added) +++ avro/trunk/lang/c/tests/test_avro_schema_names.c Tue Jan 11 14:32:37 2011 @@ -0,0 +1,133 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to you under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + * implied. See the License for the specific language governing + * permissions and limitations under the License. + */ + +#include "avro_private.h" +#include <stdio.h> +#include <stdlib.h> + +int test_cases = 0; +avro_writer_t avro_stderr; + +static void test_helper(const char *json, + const char *name, + avro_schema_t expected) +{ + int rc; + avro_schema_t base; + avro_schema_error_t serror; + + rc = avro_schema_from_json(json, strlen(json), &base, &serror); + if (rc != 0) + { + fprintf(stderr, + "Error parsing Avro schema:\n%s\n", + json); + exit(EXIT_FAILURE); + } + + avro_schema_t actual = + avro_schema_get_subschema(base, name); + + if (actual == NULL) + { + fprintf(stderr, + "No subschema named \"%s\" in %s\n", + name, avro_schema_type_name(base)); + exit(EXIT_FAILURE); + } + + if (!avro_schema_equal(actual, expected)) + { + fprintf(stderr, + "Subschema \"%s\" should be %s, " + "is actually %s\n", + name, + avro_schema_type_name(expected), + avro_schema_type_name(actual)); + exit(EXIT_FAILURE); + } + + avro_schema_decref(base); + avro_schema_decref(expected); +} + +static void test_array_schema_01() +{ + static char *JSON = + "{" + " \"type\": \"array\"," + " \"items\": \"long\"" + "}"; + + test_helper(JSON, "[]", avro_schema_long()); +} + +static void test_map_schema_01() +{ + static char *JSON = + "{" + " \"type\": \"map\"," + " \"values\": \"long\"" + "}"; + + test_helper(JSON, "{}", avro_schema_long()); +} + +static void test_record_schema_01() +{ + static char *JSON = + "{" + " \"type\": \"record\"," + " \"name\": \"test\"," + " \"fields\": [" + " { \"name\": \"a\", \"type\": \"long\" }" + " ]" + "}"; + + test_helper(JSON, "a", avro_schema_long()); +} + +static void test_union_schema_01() +{ + static char *JSON = + "[" + " \"long\"," + " {" + " \"type\": \"record\"," + " \"name\": \"test\"," + " \"fields\": [" + " { \"name\": \"a\", \"type\": \"long\" }" + " ]" + " }" + "]"; + + test_helper(JSON, "long", avro_schema_long()); +} + +int main(int argc, char *argv[]) +{ + AVRO_UNUSED(argc); + AVRO_UNUSED(argv); + + test_array_schema_01(); + + test_map_schema_01(); + + test_record_schema_01(); + + return EXIT_SUCCESS; +} From ostanton@example.com Tue Jan 11 14:48:56 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 8979 invoked from network); 11 Jan 2011 14:48:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 11 Jan 2011 14:48:56 -0000 Received: (qmail 35015 invoked by uid 500); 11 Jan 2011 14:48:56 -0000 Delivered-To: ostanton@example.com Received: (qmail 34976 invoked by uid 500); 11 Jan 2011 14:48:54 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 34968 invoked by uid 99); 11 Jan 2011 14:48:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 14:48:53 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 14:48:52 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id DCEB123889E7; Tue, 11 Jan 2011 14:48:26 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1057667 - in /avro/trunk: CHANGES.txt lang/c/src/avro.h lang/c/src/datum.c lang/c/tests/test_avro_data.c Date: Tue, 11 Jan 2011 14:48:26 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> Author: brucem Date: Tue Jan 11 14:48:26 2011 New Revision: 1057667 URL: http://svn.apache.org/viewvc?rev=1057667&view=rev Log: AVRO-630. Size accessors for array and map data. Modified: avro/trunk/CHANGES.txt avro/trunk/lang/c/src/avro.h avro/trunk/lang/c/src/datum.c avro/trunk/lang/c/tests/test_avro_data.c Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1057667&r1=1057666&r2=1057667&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Tue Jan 11 14:48:26 2011 @@ -91,6 +91,9 @@ Avro 1.5.0 (unreleased) AVRO-725. C: avro_schema_get_subschema function. (Douglas Creager via brucem) + AVRO-630. C: Add size accessors for map and list data. (Douglas Creager + via brucem) + BUG FIXES AVRO-675. C: Bytes and fixed setters don't update datum size. Modified: avro/trunk/lang/c/src/avro.h URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/src/avro.h?rev=1057667&r1=1057666&r2=1057667&view=diff ============================================================================== --- avro/trunk/lang/c/src/avro.h (original) +++ avro/trunk/lang/c/src/avro.h Tue Jan 11 14:48:26 2011 @@ -212,7 +212,9 @@ int avro_record_get(const avro_datum_t r avro_datum_t * value); int avro_map_get(const avro_datum_t datum, const char *key, avro_datum_t * value); +size_t avro_map_size(const avro_datum_t datum); int avro_array_get(const avro_datum_t datum, int64_t index, avro_datum_t * value); +size_t avro_array_size(const avro_datum_t datum); /* setters */ int avro_string_set(avro_datum_t datum, const char *p); Modified: avro/trunk/lang/c/src/datum.c URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/src/datum.c?rev=1057667&r1=1057666&r2=1057667&view=diff ============================================================================== --- avro/trunk/lang/c/src/datum.c (original) +++ avro/trunk/lang/c/src/datum.c Tue Jan 11 14:48:26 2011 @@ -621,6 +621,13 @@ avro_datum_t avro_map(void) return &datum->obj; } +size_t +avro_map_size(const avro_datum_t datum) +{ + const struct avro_map_datum_t *map = avro_datum_to_map(datum); + return map->map->num_entries; +} + int avro_map_get(const avro_datum_t datum, const char *key, avro_datum_t * value) { @@ -704,6 +711,13 @@ avro_array_get(const avro_datum_t array_ return EINVAL; } +size_t +avro_array_size(const avro_datum_t datum) +{ + const struct avro_array_datum_t *array = avro_datum_to_array(datum); + return array->els->num_entries; +} + int avro_array_append_datum(const avro_datum_t array_datum, const avro_datum_t datum) Modified: avro/trunk/lang/c/tests/test_avro_data.c URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/tests/test_avro_data.c?rev=1057667&r1=1057666&r2=1057667&view=diff ============================================================================== --- avro/trunk/lang/c/tests/test_avro_data.c (original) +++ avro/trunk/lang/c/tests/test_avro_data.c Tue Jan 11 14:48:26 2011 @@ -279,6 +279,11 @@ static int test_array(void) } } + if (avro_array_size(datum) != 10) { + fprintf(stderr, "Unexpected array size"); + exit(EXIT_FAILURE); + } + write_read_check(schema, NULL, datum, "array"); avro_datum_decref(datum); avro_schema_decref(schema); @@ -298,6 +303,12 @@ static int test_map(void) avro_datum_decref(i_datum); i++; } + + if (avro_array_size(datum) != 7) { + fprintf(stderr, "Unexpected array size"); + exit(EXIT_FAILURE); + } + write_read_check(schema, NULL, datum, "map"); avro_datum_decref(datum); avro_schema_decref(schema); From ostanton@example.com Tue Jan 11 14:57:26 2011 Return-Path: <ostanton@example.com> Delivered-To: ostanton@example.com Received: (qmail 11905 invoked from network); 11 Jan 2011 14:57:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 11 Jan 2011 14:57:26 -0000 Received: (qmail 50111 invoked by uid 500); 11 Jan 2011 14:57:26 -0000 Delivered-To: ostanton@example.com Received: (qmail 50073 invoked by uid 500); 11 Jan 2011 14:57:25 -0000 Mailing-List: contact ostanton@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:ostanton@example.com> List-Unsubscribe: <mailto:ostanton@example.com> List-Post: <mailto:ostanton@example.com> List-Id: <commits.avro.apache.org> Reply-To: ostanton@example.com Delivered-To: mailing list ostanton@example.com Received: (qmail 50065 invoked by uid 99); 11 Jan 2011 14:57:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 14:57:24 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Jan 2011 14:57:20 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 7B0FB2388A44; Tue, 11 Jan 2011 14:56:52 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1057676 - in /avro/trunk: CHANGES.txt lang/c/src/avro.h lang/c/src/datum.c lang/c/src/datum.h lang/c/src/schema.c lang/c/src/schema.h lang/c/tests/test_avro_data.c lang/c/tests/test_avro_schema.c Date: Tue, 11 Jan 2011 14:56:52 -0000 To: ostanton@example.com From: ostanton@example.com X-Mailer: svnmailer-1.0.8 Message-Id: <ostanton@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Author: brucem Date: Tue Jan 11 14:56:51 2011 New Revision: 1057676 URL: http://svn.apache.org/viewvc?rev=1057676&view=rev Log: AVRO-727. This patchs several missing accessor and mutator methods for Avro schemas and data instances. For the most part, their definitions are straightforward. Some notes for the more interesting cases: - You can create the appropriate avro_datum_t instance for an arbitrary schema with the new avro_datum_from_schema() function. For records, this includes creating a datum instance for each of the record's fields. Arrays and maps are initially created empty, while a union is created with none of its branches initially selected. This prevents the function from going into an endless loop for recursive schemas. - The active branch of a union can now be changed. If the desired branch is already active, this has no result. Otherwise, a new avro_datum_t is created for the new branch. - Map values can now be accessed by index as well as by key. The index of a map entry is based on the order that it was added to the map. - Enum values can be retrieved or set either by integer value or by symbol name. A couple of the new functions need access to the schema of a datum in order to work. These functions take in the schema instance as an extra parameter. A better solution might be to include a reference to a schema object in each avro_datum_t instance. Modified: avro/trunk/CHANGES.txt avro/trunk/lang/c/src/avro.h avro/trunk/lang/c/src/datum.c avro/trunk/lang/c/src/datum.h avro/trunk/lang/c/src/schema.c avro/trunk/lang/c/src/schema.h avro/trunk/lang/c/tests/test_avro_data.c avro/trunk/lang/c/tests/test_avro_schema.c Modified: avro/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/avro/trunk/CHANGES.txt?rev=1057676&r1=1057675&r2=1057676&view=diff ============================================================================== --- avro/trunk/CHANGES.txt (original) +++ avro/trunk/CHANGES.txt Tue Jan 11 14:56:51 2011 @@ -94,6 +94,9 @@ Avro 1.5.0 (unreleased) AVRO-630. C: Add size accessors for map and list data. (Douglas Creager via brucem) + AVRO-727. C: Add many new accessor and mutator functions. (Douglas Creager + via brucem) + BUG FIXES AVRO-675. C: Bytes and fixed setters don't update datum size. Modified: avro/trunk/lang/c/src/avro.h URL: http://svn.apache.org/viewvc/avro/trunk/lang/c/src/avro.h?rev=1057676&r1=1057675&r2=1057676&view=diff ============================================================================== --- avro/trunk/lang/c/src/avro.h (original) +++ avro/trunk/lang/c/src/avro.h Tue Jan 11 14:56:51 2011 @@ -110,21 +110,38 @@ avro_schema_t avro_schema_null(void); avro_schema_t avro_schema_record(const char *name, const char *space); avro_schema_t avro_schema_record_field_get(const avro_schema_t record, const char *field_name); +const char *avro_schema_record_field_name(const avro_schema_t schema, int index); +avro_schema_t avro_schema_record_field_get_by_index +(const avro_schema_t record, int index); int avro_schema_record_field_append(const avro_schema_t record, const char *field_name, const avro_schema_t type); +size_t avro_schema_record_size(const avro_schema_t record); avro_schema_t avro_schema_enum(const char *name); +const char *avro_schema_enum_get(const avro_schema_t enump, + int index); +int avro_schema_enum_get_by_name(const avro_schema_t enump, + const char *symbol_name); int avro_schema_enum_symbol_append(const avro_schema_t enump, const char *symbol); avro_schema_t avro_schema_fixed(const char *name, const int64_t len); +int64_t avro_schema_fixed_size(const avro_schema_t fixed); + avro_schema_t avro_schema_map(const avro_schema_t values); +avro_schema_t avro_schema_map_values(avro_schema_t map); + avro_schema_t avro_schema_array(const avro_schema_t items); +avro_schema_t avro_schema_array_items(avro_schema_t array); avro_schema_t avro_schema_union(void); int avro_schema_union_append(const avro_schema_t union_schema, const avro_schema_t schema); +avro_schema_t avro_schema_union_branch(avro_schema_t union_schema, + int branch_index); +avro_schema_t avro_schema_union_branch_by_name +(avro_schema_t union_schema, int *branch_index, const char *name); avro_schema_t avro_schema_link(avro_schema_t schema); @@ -198,6 +215,13 @@ avro_datum_t avro_map(void); avro_datum_t avro_array(void); avro_datum_t avro_union(int64_t discriminant, const avro_datum_t datum); +/* + * Constructs a new avro_datum_t instance that's appropriate for holding + * values of the given schema. + */ + +avro_datum_t avro_datum_from_schema(const avro_schema_t schema); + /* getters */ int avro_string_get(avro_datum_t datum, char **p); int avro_bytes_get(avro_datum_t datum, char **bytes, int64_t * size); @@ -207,15 +231,34 @@ int avro_float_get(avro_datum_t datum, f int avro_double_get(avro_datum_t datum, double *d); int avro_boolean_get(avro_datum_t datum, int8_t * i); +int avro_enum_get(const avro_datum_t datum); +const char *avro_enum_get_name(const avro_datum_t datum, + const avro_schema_t schema); int avro_fixed_get(avro_datum_t datum, char **bytes, int64_t * size); int avro_record_get(const avro_datum_t record, const char *field_name, avro_datum_t * value); int avro_map_get(const avro_datum_t datum, const char *key, avro_datum_t * value); +/* + * For maps, the "index" for each entry is based on the order that they + * were added to the map. + */ +int avro_map_get_key(const avro_datum_t datum, int index, + const char **key); size_t avro_map_size(const avro_datum_t datum); int avro_array_get(const avro_datum_t datum, int64_t index, avro_datum_t * value); size_t avro_array_size(const avro_datum_t datum); +/* + * These accessors allow you to query the current branch of a union + * value, returning either the branch's discriminant value or the + * avro_datum_t of the branch. A union value can be uninitialized, in + * which case the discriminant will be -1 and the datum NULL. + */ + +int64_t avro_union_discriminant(const avro_datum_t datum); +avro_datum_t avro_union_current_branch(avro_datum_t datum); + /* setters */ int avro_string_set(avro_datum_t datum, const char *p); int avro_givestring_set(avro_datum_t datum, const char *p); @@ -233,18 +276,37 @@ int avro_float_set(avro_datum_t datum, c int avro_double_set(avro_datum_t datum, const double d); int avro_boolean_set(avro_datum_t datum, const int8_t i); +int avro_enum_set(avro_datum_t datum, const int symbol_value); +int avro_enum_set_name(avro_datum_t datum, avro_schema_t schema, + const char *symbol_name); int avro_fixed_set(avro_datum_t datum, const char *bytes, const int64_t size); int avro_givefixed_set(avro_datum_t datum, const char *bytes, const int64_t size); int avro_wrapfixed_set(avro_datum_t datum, const char *bytes, const int64_t size); -int avro_record_set(const avro_datum_t record, const char *field_name, - const avro_datum_t value); -int
{'source': 'Apache Software Foundation Public Mail Archives'}