From lucyrobertsjustin@example.org Fri Oct 13 04:47:16 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 3452 invoked from network); 13 Oct 2006 04:47:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 13 Oct 2006 04:47:16 -0000 Received: (qmail 1528 invoked by uid 500); 13 Oct 2006 04:47:16 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 1503 invoked by uid 500); 13 Oct 2006 04:47:15 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 1494 invoked by uid 99); 13 Oct 2006 04:47:15 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Oct 2006 21:47:15 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Oct 2006 21:47:14 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GYExQ-0008BC-7c for robertsjustin@example.org; Thu, 12 Oct 2006 21:47:40 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) Content-Transfer-Encoding: 7bit Message-Id: <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: robertsjustin@example.org From: Marvin Humphrey <robertsjustin@example.org> Subject: Creation and Destruction Date: Thu, 12 Oct 2006 21:46:50 -0700 X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Greets, I've been working on the problem of on generalizing native object wrapping, in the hopes that we can apply the same set of C functions/ macros regardless of what language we're targeting -- but define them differently for each binding. Perl manages memory using reference counting. When a Perl SV* (that's scalar value, Perl's basic data type) has its reference count drop to 0 (most commonly because a lexical variable goes out of scope) it gets reclaimed. If that scalar references any other Perl data structures, their reference counts get decremented at that point. { my $foo = "foo yoo"; # $foo has a refcount of 1 { my $foo_ref = \$foo; # now $foo a refcount of 2 } # $foo_ref's count drops to 0 and it gets reclaimed # now $foo has a refcount of 1 again } # $foo's refcount drops to 0 and it gets reclaimed. If the scalar is an object, the object's DESTROY method is called. { my $foo = Foo->new; } # $foo->DESTROY gets called. Usually, Perl just cleans up its own data structures and there's no need to write a specific DESTROY method. However, it doesn't know how to deal with things like foreign C structs wrapped in Perl objects, so DESTROY is where custom cleanup stuff goes. One problem with objects that have to be accessible from both Perl and C is that if a C struct has to be wrapped in a Perl object so that it can travel trough Perl-space, it's hard to stop it from being destroyed when it leaves Perl space. Say that we have a TokenBatch struct, wrapped in a Perl object, which holds a bunch of little Token structs. Say we want to do something like this... while ( $token_batch->next ) { my $token = $token_batch->get_token; transform_token_somehow($token); } We'll have to wrap each Token in a Perl object as it leaves the TokenBatch. Then, when $token goes out of scope, $token->DESTROY will get called. We don't really want that to happen, though -- the TokenBatch isn't done with the Token struct yet. The solution for most cases is to assign every struct a Perl object at creation time if the struct will have to pass through Perl space at some point, and to keep track of that native object by assigning it to a struct member. self->ref = lucy_Obj_create_ref(self, class_name); Historically, such functionality has resided in KinoSearch's XS wrapper code and all constructors have been called from Perl space. But as I've been Ferret-izing KinoSearch, I've been writing more and more C code, and it's becoming desirable to have C constructors call C constructors. That means some native objects have to be created from C space. It's possible to quarantine all the actual perlapi C routines in a single module. Let's say it's Lucy/Util/Object. Object.h might declare these functions: /* Create a native ref with a refcount of 1. */ void* lucy_Obj_create_ref(void *ptr, const char *class); /* Decrement an opaque reference's native refcount. */ void lucy_Obj_refcount_dec(void *ref); The Perl implementation in Object.c probably looks like this: #ifdef LUCY_PERL void* lucy_Obj_create_ref(void *ptr, const char *class) { /* I'll explain what this means some other time. */ SV *obj_sv = newSViv( PTR2IV(ptr) ); HV *const stash = gv_stashpv(class, true); SvRV_set( (SV*)Obj_scratch_ref, obj_sv ); sv_bless( (SV*)Obj_scratch_ref, stash ); return (void*)obj_sv; } void lucy_Obj_refcount_dec(void *ref) { SvREFCNT_dec( (SV*)ref ); } #endif /* LUCY_PERL */ That scheme is working fine in KinoSearch, but there's a problem which is getting in the way of full generalization. Perl has to know the classname, and it spells it using double colons as separators. instream->ref = lucy_Obj_create_ref(instream, "Lucy888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Stor888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Instream"); That string isn't going to be useful in other implementations. If all object creation is handled from native space, no big deal, because the calls to lucy_Obj_refcount_dec all happen from the XS binding code, not from the C modules. But once you get C constructors calling C constructors, you have problems. For now in KinoSearch, I'm just pressing ahead and typing in the class names as Perl expects them, but we'll have to solve this problem for Lucy. Does Ruby have analogous quirks for object creation and destruction? How about other languages? Another problem we have to solve is how to make memory management work under multiple systems. We can't really do mark-and-sweep with Perl, because Perl doesn't provide an event we can cue collection off of. We'd have to write our own tracing garbage collector, our own malloc() and free(), and manage our own memory pool -- yikes. So... how can we make this work everywhere? How would you implement lucy_Obj_create_ref and lucy_Obj_refcount_dec? Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Fri Oct 13 06:22:03 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 24372 invoked from network); 13 Oct 2006 06:22:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 13 Oct 2006 06:22:02 -0000 Received: (qmail 86196 invoked by uid 500); 13 Oct 2006 06:22:02 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 86162 invoked by uid 500); 13 Oct 2006 06:22:02 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 86147 invoked by uid 99); 13 Oct 2006 06:22:02 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Oct 2006 23:22:02 -0700 X-ASF-Spam-Status: No, hits=0.5 required=10.0 tests=DNS_FROM_RFC_ABUSE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of robertsjustin@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wx-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Oct 2006 23:22:00 -0700 Received: by wx-out-0506.google.com with SMTP id s17so798058wxc for <robertsjustin@example.org>; Thu, 12 Oct 2006 23:21:40 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=n1n7Na7P3QTKEhsyaFfo0oRg545RaYQi3YzOkPZy8O200J74X+eBX4YgakMboL5zK5IB5HbJ+ybP1MGHUOQfGj83x+T+q70zdaoqBLh9GjGOG95SxZpzJQVLnq7xq8g24WX1LgCLM/rDsADjGInTqJIheDe2lnz04eJkwHm7w6g= Received: by 83.161.120.155 with SMTP id z18mr1935291agz; Thu, 12 Oct 2006 23:21:39 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Thu, 12 Oct 2006 23:21:39 -0700 (PDT) Message-ID: <robertsjustin@example.org> Date: Fri, 13 Oct 2006 15:21:39 +0900 From: "David Balmain" <robertsjustin@example.org> To: robertsjustin@example.org Subject: Re: Creation and Destruction In-Reply-To: <robertsjustin@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <robertsjustin@example.org> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On 10/13/06, Marvin Humphrey <robertsjustin@example.org> wrote: > Greets, > > I've been working on the problem of on generalizing native object > wrapping, in the hopes that we can apply the same set of C functions/ > macros regardless of what language we're targeting -- but define them > differently for each binding. > > Perl manages memory using reference counting. When a Perl SV* > (that's scalar value, Perl's basic data type) has its reference count > drop to 0 (most commonly because a lexical variable goes out of > scope) it gets reclaimed. If that scalar references any other Perl > data structures, their reference counts get decremented at that point. > > { > my $foo = "foo yoo"; # $foo has a refcount of 1 > { > my $foo_ref = \$foo; # now $foo a refcount of 2 > } # $foo_ref's count drops to 0 and it gets reclaimed > # now $foo has a refcount of 1 again > } # $foo's refcount drops to 0 and it gets reclaimed. > > > If the scalar is an object, the object's DESTROY method is called. > > { > my $foo = Foo->new; > } # $foo->DESTROY gets called. > > Usually, Perl just cleans up its own data structures and there's no > need to write a specific DESTROY method. However, it doesn't know > how to deal with things like foreign C structs wrapped in Perl > objects, so DESTROY is where custom cleanup stuff goes. > > One problem with objects that have to be accessible from both Perl > and C is that if a C struct has to be wrapped in a Perl object so > that it can travel trough Perl-space, it's hard to stop it from being > destroyed when it leaves Perl space. Say that we have a TokenBatch > struct, wrapped in a Perl object, which holds a bunch of little Token > structs. Say we want to do something like this... > > while ( $token_batch->next ) { > my $token = $token_batch->get_token; > transform_token_somehow($token); > } > > We'll have to wrap each Token in a Perl object as it leaves the > TokenBatch. Then, when $token goes out of scope, $token->DESTROY > will get called. We don't really want that to happen, though -- the > TokenBatch isn't done with the Token struct yet. > > The solution for most cases is to assign every struct a Perl object > at creation time if the struct will have to pass through Perl space > at some point, and to keep track of that native object by assigning > it to a struct member. > > self->ref = lucy_Obj_create_ref(self, class_name); > > Historically, such functionality has resided in KinoSearch's XS > wrapper code and all constructors have been called from Perl space. > But as I've been Ferret-izing KinoSearch, I've been writing more and > more C code, and it's becoming desirable to have C constructors call > C constructors. That means some native objects have to be created > from C space. > > It's possible to quarantine all the actual perlapi C routines in a > single module. Let's say it's Lucy/Util/Object. Object.h might > declare these functions: > > /* Create a native ref with a refcount of 1. > */ > void* > lucy_Obj_create_ref(void *ptr, const char *class); > > /* Decrement an opaque reference's native refcount. > */ > void > lucy_Obj_refcount_dec(void *ref); > > > The Perl implementation in Object.c probably looks like this: > > #ifdef LUCY_PERL > > void* > lucy_Obj_create_ref(void *ptr, const char *class) > { > /* I'll explain what this means some other time. */ > SV *obj_sv = newSViv( PTR2IV(ptr) ); > HV *const stash = gv_stashpv(class, true); > SvRV_set( (SV*)Obj_scratch_ref, obj_sv ); > sv_bless( (SV*)Obj_scratch_ref, stash ); > return (void*)obj_sv; > } > > void > lucy_Obj_refcount_dec(void *ref) > { > SvREFCNT_dec( (SV*)ref ); > } > > #endif /* LUCY_PERL */ > > That scheme is working fine in KinoSearch, but there's a problem > which is getting in the way of full generalization. Perl has to know > the classname, and it spells it using double colons as separators. > > instream->ref = lucy_Obj_create_ref(instream, > "Lucy888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Stor888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Instream"); > > That string isn't going to be useful in other implementations. If > all object creation is handled from native space, no big deal, > because the calls to lucy_Obj_refcount_dec all happen from the XS > binding code, not from the C modules. But once you get C > constructors calling C constructors, you have problems. > > For now in KinoSearch, I'm just pressing ahead and typing in the > class names as Perl expects them, but we'll have to solve this > problem for Lucy. > > Does Ruby have analogous quirks for object creation and destruction? Hi Marvin, I don't know how much of Ferret's binding code you've looked at but you may have noticed a lot of my structs have a ref_cnt variable. That way, every time a struct gets wrapped in a ruby object, it's ref_cnt is incremented. When the object goes out of scope and is garbage collected the ref_cnt is decremented (and the object deleted if ref_cnt = 0). I think as far as supporting multiple languages like we are attempting to do with Lucy, this is the easiest way to go, since it should work for all languages. The bigger problem I had was when Ruby objects are going into C space. What happens when all references are removed in Ruby and the object gets deleted while it is still being used in C space. Actually, I think reference counting languages make this a lot easier to deal with, although I just discovered recently I can deregister an object for garbage collection in Ruby which is very helpful. > How about other languages? The only other language I've actually written an extension for is Python which is reference counted like Perl. As far as I know, so is PHP. Lua and IO are mark-and-sweep interpreters like Ruby. I think that is basically what it is going to come down to; the garbage collection algorithm used. > Another problem we have to solve is how to make memory management > work under multiple systems. We can't really do mark-and-sweep with > Perl, because Perl doesn't provide an event we can cue collection off > of. We'd have to write our own tracing garbage collector, our own > malloc() and free(), and manage our own memory pool -- yikes. > > So... how can we make this work everywhere? How would you implement > lucy_Obj_create_ref and lucy_Obj_refcount_dec? If I really had to do it this way lucy_Obj_create_ref would add the object to a giant hash-table which keeps track of reference counts and is visible to Ruby's garbage collector . lucy_Obj_refcount_dec would decrement the reference count in the hash-table and delete the object for the hash-table if the ref-count is 0. Cheers, Dave PS: you may have read on the Java mailing list that I'm working on building an object database with Lucene-like search capabilities. This is what people really need in the Ruby community since most people are just using Ferret to add full-text search to their database. Everyone seems to be struggling with trying to keep their indexes in synch with the database, not to mention the performance implications of having a separate index and data-store. This also makes it much easier to have multiple processes adding data to the index. And creating bindings will be greatly simplified because the binding simply need to know about an SQL like query language and how to turn the results sets into the correct objects for the language in question. Obviously there are downsides to this solution, the biggest being extensibility in the native language which is why I am still going to go ahead with Lucy development. From lucyrobertsjustin@example.org Fri Oct 13 13:42:34 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 82723 invoked from network); 13 Oct 2006 13:42:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 13 Oct 2006 13:42:33 -0000 Received: (qmail 25185 invoked by uid 500); 13 Oct 2006 13:42:33 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 25152 invoked by uid 500); 13 Oct 2006 13:42:33 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 25143 invoked by uid 99); 13 Oct 2006 13:42:33 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Oct 2006 06:42:33 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO cenn.mc.mpls.visi.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Oct 2006 06:42:29 -0700 Received: from [83.161.120.155] (karman.dsl.visi.com [83.161.120.155]) by cenn.mc.mpls.visi.com (Postfix) with ESMTP id 97F438399 for <robertsjustin@example.org>; Fri, 13 Oct 2006 08:42:07 -0500 (CDT) Message-ID: <robertsjustin@example.org> Date: Fri, 13 Oct 2006 08:42:02 -0500 From: Peter Karman <robertsjustin@example.org> Reply-To: robertsjustin@example.org User-Agent: Thunderbird 83.161.120.155 (Macintosh/20060909) MIME-Version: 1.0 To: robertsjustin@example.org Subject: Re: Creation and Destruction References: <robertsjustin@example.org> <robertsjustin@example.org> In-Reply-To: <robertsjustin@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N David Balmain scribbled on 10/13/06 1:21 AM: > I don't know how much of Ferret's binding code you've looked at but > you may have noticed a lot of my structs have a ref_cnt variable. That > way, every time a struct gets wrapped in a ruby object, it's ref_cnt > is incremented. When the object goes out of scope and is garbage > collected the ref_cnt is decremented (and the object deleted if > ref_cnt = 0). I think as far as supporting multiple languages like we > are attempting to do with Lucy, this is the easiest way to go, since > it should work for all languages. > Hi folks, Interesting thread. fwiw, I have used the above approach in Perl myself. I just increm/decrem the refcnt in XS (Perl) space, in the DESTROY method. -- Peter Karman . http://peknet.com/ . robertsjustin@example.org From lucyrobertsjustin@example.org Sat Oct 14 05:03:35 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 8434 invoked from network); 14 Oct 2006 05:03:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Oct 2006 05:03:35 -0000 Received: (qmail 69009 invoked by uid 500); 14 Oct 2006 05:03:35 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 68989 invoked by uid 500); 14 Oct 2006 05:03:35 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 68980 invoked by uid 99); 14 Oct 2006 05:03:35 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Oct 2006 22:03:35 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Oct 2006 22:03:32 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GYbh4-000Acd-Cd for robertsjustin@example.org; Fri, 13 Oct 2006 22:04:18 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <robertsjustin@example.org> References: <robertsjustin@example.org> <robertsjustin@example.org> <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <robertsjustin@example.org> Content-Transfer-Encoding: 7bit From: Marvin Humphrey <robertsjustin@example.org> Subject: Re: Creation and Destruction Date: Fri, 13 Oct 2006 22:03:08 -0700 To: robertsjustin@example.org X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On Oct 13, 2006, at 6:42 AM, Peter Karman wrote: > David Balmain scribbled on 10/13/06 1:21 AM: >> I don't know how much of Ferret's binding code you've looked at but >> you may have noticed a lot of my structs have a ref_cnt variable. >> That >> way, every time a struct gets wrapped in a ruby object, it's ref_cnt >> is incremented. When the object goes out of scope and is garbage >> collected the ref_cnt is decremented (and the object deleted if >> ref_cnt = 0). I think as far as supporting multiple languages like we >> are attempting to do with Lucy, this is the easiest way to go, since >> it should work for all languages. > fwiw, I have used the above approach in Perl myself. I just increm/ > decrem the refcnt in XS (Perl) space, in the DESTROY method. I like this approach better, and I've now changed KinoSearch to use it. One minor theoretical problem with subclassing. Say you have a struct that holds a TermDocs. typedef struct lucy_SomeStruct { struct lucy_TermDocs *term_docs; /* ... */ } lucy_SomeStruct; A Perl object created off of that has to take the class "Lucy888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Index888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4TermDocs", rather than "Lucy888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Index888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4SegTermDocs" or "Lucy888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Index888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4MultiTermDocs". That's not quite right. /* This is an XS binding */ lucy_TermDocs* get_term_docs(self) lucy_SomeStruct *self; CODE: RETVAL = self->term_docs; OUTPUT: RETVAL However, you can only fake OOP so well with C, and this limitation hasn't triggered any glitches yet. We'll see what happens when I port IndexReader, SegReader, and MultiReader to C -- they'll all appear to be "IndexReader" from Perl-space. Dunno if that will break something. I also had to jettison a scheme I'd devised for generalizing object- method callbacks to [your_language_here], but that might be for the best. :) Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Sat Oct 14 06:28:38 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 18960 invoked from network); 14 Oct 2006 06:28:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Oct 2006 06:28:38 -0000 Received: (qmail 1313 invoked by uid 500); 14 Oct 2006 06:28:37 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 1291 invoked by uid 500); 14 Oct 2006 06:28:37 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 1280 invoked by uid 99); 14 Oct 2006 06:28:37 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Oct 2006 23:28:37 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Oct 2006 23:28:36 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GYd1Q-000Akd-Bc for robertsjustin@example.org; Fri, 13 Oct 2006 23:29:24 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) Content-Transfer-Encoding: 7bit Message-Id: <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: robertsjustin@example.org From: Marvin Humphrey <robertsjustin@example.org> Subject: Charmonizer revisited Date: Fri, 13 Oct 2006 23:28:12 -0700 X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Greets, I've been using a version of Charmonizer with KinoSearch for a little while now. No public release, and it's a work in progress, but the basic approach seems to be sound. I bailed on the idea of implementing Charmonizer as a domain-specific language with a full-on custom parser. Too much hassle, when another idea presented itself... Dave had suggested that we might consider using C to write C. However, we agreed that embedding C code in C string literals was just too dang gnarly. The solution: a source filter. Here's a description taken from the README file: <readme> METAQUOTE UTILITY Charmonizer is a C program which writes C code. Writing C from C is ordinarily quite awkward, because when you try to embed more than a line or two of C source code inside C string literals, things get ugly fast. To get around this problem, Charmonizer uses a source filter. In the "real" Charmonizer source code, longer code fragments are surrounded by METAQUOTE tags instead of literal double quotes. char hello_source[] = METAQUOTE #include <stdio.h> int main() { printf("Hello, world.\n"); return 0; } METAQUOTE; These source files are run through the "metaquote" utility -- which transforms everything between paired METAQUOTE tags into concatenated string literals. The result is hideous but valid C. </readme> Technically, Charmonizer ain't C -- it really is a domain-specific language which compiles to C. But this is a lot more straightforward than the syntax in the original proposal. And unless you're writing custom modules and you want to take advantage of metaquote, you just write your main configurator app in C and #include post-filter Charmonizer files. http://www.rectangular.com/svn/kinosearch/trunk/charmonizer/ http://www.rectangular.com/svn/kinosearch/trunk/charmonize.c Sound good? metaquote is implemented in C, BTW. Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Sat Oct 14 07:30:48 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 33229 invoked from network); 14 Oct 2006 07:30:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Oct 2006 07:30:47 -0000 Received: (qmail 23403 invoked by uid 500); 14 Oct 2006 07:30:47 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 23369 invoked by uid 500); 14 Oct 2006 07:30:47 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 23358 invoked by uid 99); 14 Oct 2006 07:30:47 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 00:30:47 -0700 X-ASF-Spam-Status: No, hits=0.5 required=10.0 tests=DNS_FROM_RFC_ABUSE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of robertsjustin@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 00:30:46 -0700 Received: by nf-out-0910.google.com with SMTP id b2so1615531nfe for <robertsjustin@example.org>; Sat, 14 Oct 2006 00:30:25 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=aY4STapz4Y/fX4fNP/OqUwe9C+ggQo3qMtzBv1pPScb3A1XiFD2BDuKe/GwTjToukMw1NRnO/6IaYbTmqSEEPXvYCxYoWpDSwfUuueQF7UV9vSe9/D75jwI3Tdg9Iracb0fbU7JIDOYuRL23bWuDwI63d4x16V9f3QJRjrvcjqc= Received: by 83.161.120.155 with SMTP id s15mr704856buc; Sat, 14 Oct 2006 00:30:24 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sat, 14 Oct 2006 00:30:24 -0700 (PDT) Message-ID: <robertsjustin@example.org> Date: Sat, 14 Oct 2006 16:30:24 +0900 From: "David Balmain" <robertsjustin@example.org> To: robertsjustin@example.org Subject: Re: Charmonizer revisited In-Reply-To: <robertsjustin@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <robertsjustin@example.org> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On 10/14/06, Marvin Humphrey <robertsjustin@example.org> wrote: > Greets, > > I've been using a version of Charmonizer with KinoSearch for a little > while now. No public release, and it's a work in progress, but the > basic approach seems to be sound. > > I bailed on the idea of implementing Charmonizer as a domain-specific > language with a full-on custom parser. Too much hassle, when another > idea presented itself... > > Dave had suggested that we might consider using C to write C. > However, we agreed that embedding C code in C string literals was > just too dang gnarly. > > The solution: a source filter. > > Here's a description taken from the README file: > > <readme> > > METAQUOTE UTILITY > > Charmonizer is a C program which writes C code. Writing C from > C is > ordinarily quite awkward, because when you try to embed more > than a line > or two of C source code inside C string literals, things get > ugly fast. > > To get around this problem, Charmonizer uses a source filter. > In the > "real" Charmonizer source code, longer code fragments are > surrounded by > METAQUOTE tags instead of literal double quotes. > > char hello_source[] = METAQUOTE > #include <stdio.h> > int main() { > printf("Hello, world.\n"); > return 0; > } > METAQUOTE; > > These source files are run through the "metaquote" utility -- which > transforms everything between paired METAQUOTE tags into > concatenated > string literals. The result is hideous but valid C. > > </readme> > > Technically, Charmonizer ain't C -- it really is a domain-specific > language which compiles to C. But this is a lot more straightforward > than the syntax in the original proposal. And unless you're writing > custom modules and you want to take advantage of metaquote, you just > write your main configurator app in C and #include post-filter > Charmonizer files. > > http://www.rectangular.com/svn/kinosearch/trunk/charmonizer/ > http://www.rectangular.com/svn/kinosearch/trunk/charmonize.c > > Sound good? > > metaquote is implemented in C, BTW. > > Marvin Humphrey > Rectangular Research > http://www.rectangular.com/ > Hi Marvin, Looks great to me. The only problem I had when playing around with this months ago was the ISO C89 string length limit of 509: t.c:33: warning: string length '1836' is greater than the length '509' ISO C89 compilers are required to support I guess it shouldn't be too hard to keep all strings under that limit. Cheers, Dave From lucyrobertsjustin@example.org Sat Oct 14 07:36:58 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 33796 invoked from network); 14 Oct 2006 07:36:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Oct 2006 07:36:58 -0000 Received: (qmail 26235 invoked by uid 500); 14 Oct 2006 07:36:57 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 26205 invoked by uid 500); 14 Oct 2006 07:36:57 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 26196 invoked by uid 99); 14 Oct 2006 07:36:57 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 00:36:57 -0700 X-ASF-Spam-Status: No, hits=0.5 required=10.0 tests=DNS_FROM_RFC_ABUSE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of robertsjustin@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 00:36:57 -0700 Received: by nf-out-0910.google.com with SMTP id b2so1616251nfe for <robertsjustin@example.org>; Sat, 14 Oct 2006 00:36:35 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=DBRDHOT44uOSL+uN2+vL9vAUJCdD4ti7/aI3nR6HMNDbohOG60YgjA/VFw/Xmy0lHwwWAecf749kjcs60erBVaF29HsCa3Gkb6niCIsDdS5ZtyzB0KPpOChBId3gISZvbW4Yf2lXbUWXFfgB28nt1yKqYkZaPs5JI2xY88ktXWA= Received: by 83.161.120.155 with SMTP id z16mr704709bud; Sat, 14 Oct 2006 00:36:35 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sat, 14 Oct 2006 00:36:34 -0700 (PDT) Message-ID: <robertsjustin@example.org> Date: Sat, 14 Oct 2006 16:36:35 +0900 From: "David Balmain" <robertsjustin@example.org> To: robertsjustin@example.org Subject: Re: Creation and Destruction In-Reply-To: <robertsjustin@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <robertsjustin@example.org> <robertsjustin@example.org> <robertsjustin@example.org> <robertsjustin@example.org> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On 10/14/06, Marvin Humphrey <robertsjustin@example.org> wrote: > > On Oct 13, 2006, at 6:42 AM, Peter Karman wrote: > > > David Balmain scribbled on 10/13/06 1:21 AM: > >> I don't know how much of Ferret's binding code you've looked at but > >> you may have noticed a lot of my structs have a ref_cnt variable. > >> That > >> way, every time a struct gets wrapped in a ruby object, it's ref_cnt > >> is incremented. When the object goes out of scope and is garbage > >> collected the ref_cnt is decremented (and the object deleted if > >> ref_cnt = 0). I think as far as supporting multiple languages like we > >> are attempting to do with Lucy, this is the easiest way to go, since > >> it should work for all languages. > > > fwiw, I have used the above approach in Perl myself. I just increm/ > > decrem the refcnt in XS (Perl) space, in the DESTROY method. > > I like this approach better, and I've now changed KinoSearch to use it. > > One minor theoretical problem with subclassing. Say you have a > struct that holds a TermDocs. > > typedef struct lucy_SomeStruct { > struct lucy_TermDocs *term_docs; > /* ... */ > } lucy_SomeStruct; > > A Perl object created off of that has to take the class > "Lucy888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Index888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4TermDocs", rather than "Lucy888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Index888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4SegTermDocs" or > "Lucy888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4Index888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4MultiTermDocs". That's not quite right. > > /* This is an XS binding */ > lucy_TermDocs* > get_term_docs(self) > lucy_SomeStruct *self; > CODE: > RETVAL = self->term_docs; > OUTPUT: RETVAL > > However, you can only fake OOP so well with C, and this limitation > hasn't triggered any glitches yet. We'll see what happens when I > port IndexReader, SegReader, and MultiReader to C -- they'll all > appear to be "IndexReader" from Perl-space. Dunno if that will break > something. It has worked fine for me. After all SegReader and MultiReader are just implementations of IndexReader and the calling code shouldn't ever need to know what it is calling as long as it implements the IndexReader interface. > I also had to jettison a scheme I'd devised for generalizing object- > method callbacks to [your_language_here], but that might be for the > best. :) That would be ambitious. :) I think we'll just have to define a list of callbacks that each languages binding needs to implement. From lucyrobertsjustin@example.org Sat Oct 14 08:18:32 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 38773 invoked from network); 14 Oct 2006 08:18:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Oct 2006 08:18:32 -0000 Received: (qmail 41806 invoked by uid 500); 14 Oct 2006 08:18:32 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 41785 invoked by uid 500); 14 Oct 2006 08:18:32 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 41772 invoked by uid 99); 14 Oct 2006 08:18:32 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 01:18:32 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 01:18:31 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GYejp-000Auz-Ss for robertsjustin@example.org; Sat, 14 Oct 2006 01:19:21 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <robertsjustin@example.org> References: <robertsjustin@example.org> <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <robertsjustin@example.org> Content-Transfer-Encoding: 7bit From: Marvin Humphrey <robertsjustin@example.org> Subject: Re: Charmonizer revisited Date: Sat, 14 Oct 2006 01:18:07 -0700 To: robertsjustin@example.org X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On Oct 14, 2006, at 12:30 AM, David Balmain wrote: > Looks great to me. Groovy. Suggestions before I add the requisite Apache licenses and commit? > The only problem I had when playing around with > this months ago was the ISO C89 string length limit of 509: > > t.c:33: warning: string length '1836' is greater than the length > '509' ISO C89 compilers are required to support > > I guess it shouldn't be too hard to keep all strings under that limit. If we need longer snips -- and it wouldn't surprise me if we do at some point -- we can turn chaz_Core_capture_output into a varargs function. We can also macrofy stuff that's common to each snip and save a couple characters that way. <before> static char iso_func_code[] = METAQUOTE #include <stdio.h> int main() { freopen("_charmonizer_target", "w", stdout); printf("%s", __func__); return 0; } METAQUOTE; </before> <after> static char iso_func_code[] = METAQUOTE #include "_ch.h" int main() { Redirect_Stdout; printf("%s", __func__); return 0; } METAQUOTE; </after> The "_ch.h" header file, we'd write out at the beginning of each Charmonizer session -- probably cued by the invocation of a required chaz_init() function defined in Charmonizer.c. And maybe metaquote should collapse whitespace outside of string literals. Dunno if that's worth it. We'll probably need the varargs function eventually, which solves the problem. And string manipulation from C is such a nightmare. Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Sat Oct 14 15:42:47 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 27777 invoked from network); 14 Oct 2006 15:42:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Oct 2006 15:42:47 -0000 Received: (qmail 35383 invoked by uid 500); 14 Oct 2006 15:42:47 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 35351 invoked by uid 500); 14 Oct 2006 15:42:46 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 35342 invoked by uid 99); 14 Oct 2006 15:42:46 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 08:42:46 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 08:42:46 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GYlfp-000Bhe-Tu for robertsjustin@example.org; Sat, 14 Oct 2006 08:43:41 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <robertsjustin@example.org> References: <robertsjustin@example.org> <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <robertsjustin@example.org> Content-Transfer-Encoding: 7bit From: Marvin Humphrey <robertsjustin@example.org> Subject: Re: Charmonizer revisited Date: Sat, 14 Oct 2006 08:42:23 -0700 To: robertsjustin@example.org X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Greets, I've added a "ToDo" page to the Lucy Wiki, and started it off with a bunch of Charmonizer items: http://wiki.apache.org/lucy/ToDo These refer to the version in the KinoSearch repository, rather than the stuff in the Lucy repository that's about to get replaced. Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Sat Oct 14 16:34:30 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 36376 invoked from network); 14 Oct 2006 16:34:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Oct 2006 16:34:29 -0000 Received: (qmail 81470 invoked by uid 500); 14 Oct 2006 16:34:29 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 81452 invoked by uid 500); 14 Oct 2006 16:34:29 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 81443 invoked by uid 99); 14 Oct 2006 16:34:29 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 09:34:29 -0700 X-ASF-Spam-Status: No, hits=0.5 required=10.0 tests=DNS_FROM_RFC_ABUSE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of robertsjustin@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 09:34:28 -0700 Received: by nf-out-0910.google.com with SMTP id b2so1701212nfe for <robertsjustin@example.org>; Sat, 14 Oct 2006 09:34:07 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=jCVXrwFdDn3jcFIGV4pZMr9iHzygEsjxSuO3nIXOiyaLa7ErivTmKOjWD24Px6f4laiWST/OkbWDVFpAFQOnqz1jDgzpnw31tJofxDQHdU1i5VI3pGEplf5olBaOo/Zd251DiMbFgHBZv1qOzqY4AFPaBoJvEXmR2Y1mb6Nayxs= Received: by 83.161.120.155 with SMTP id m3mr734575buc; Sat, 14 Oct 2006 09:34:06 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sat, 14 Oct 2006 09:34:06 -0700 (PDT) Message-ID: <robertsjustin@example.org> Date: Sun, 15 Oct 2006 01:34:06 +0900 From: "David Balmain" <robertsjustin@example.org> To: robertsjustin@example.org Subject: Re: Charmonizer revisited In-Reply-To: <robertsjustin@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <robertsjustin@example.org> <robertsjustin@example.org> <robertsjustin@example.org> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On 10/15/06, Marvin Humphrey <robertsjustin@example.org> wrote: > Greets, > > I've added a "ToDo" page to the Lucy Wiki, and started it off with a > bunch of Charmonizer items: > > http://wiki.apache.org/lucy/ToDo > > These refer to the version in the KinoSearch repository, rather than > the stuff in the Lucy repository that's about to get replaced. > Great. I still don't have a lot of time but I'll try and put some work into some of these ToDos. From lucyrobertsjustin@example.org Sat Oct 14 16:53:13 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 39165 invoked from network); 14 Oct 2006 16:53:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Oct 2006 16:53:13 -0000 Received: (qmail 97650 invoked by uid 500); 14 Oct 2006 16:53:12 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 97621 invoked by uid 500); 14 Oct 2006 16:53:12 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 97612 invoked by uid 99); 14 Oct 2006 16:53:12 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 09:53:12 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 14 Oct 2006 09:53:10 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GYmm0-000Boo-OO for robertsjustin@example.org; Sat, 14 Oct 2006 09:54:08 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <robertsjustin@example.org> References: <robertsjustin@example.org> <robertsjustin@example.org> <robertsjustin@example.org> <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <robertsjustin@example.org> Content-Transfer-Encoding: 7bit From: Marvin Humphrey <robertsjustin@example.org> Subject: Re: Charmonizer revisited Date: Sat, 14 Oct 2006 09:52:48 -0700 To: robertsjustin@example.org X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On Oct 14, 2006, at 9:34 AM, David Balmain wrote: > Great. I still don't have a lot of time but I'll try and put some work > into some of these ToDos. I'm able to dedicate a fair amount of time to this in the immediate future. The flags problem has to get solved for KinoSearch 0.20. If I can request that you focus on one item, it would be thinking about the documentation from a high level. Since I'm immersed in the coding, lack of distance somewhat impedes my ability to handle that one. I'll go ahead and commit the new material. Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Sun Oct 15 18:01:53 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 67938 invoked from network); 15 Oct 2006 18:01:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 15 Oct 2006 18:01:52 -0000 Received: (qmail 9310 invoked by uid 500); 15 Oct 2006 18:01:52 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 9289 invoked by uid 500); 15 Oct 2006 18:01:52 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 9277 invoked by uid 99); 15 Oct 2006 18:01:52 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Oct 2006 11:01:52 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Oct 2006 11:01:51 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GZAKL-000EwB-Vh for robertsjustin@example.org; Sun, 15 Oct 2006 11:03:10 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) Content-Transfer-Encoding: 7bit Message-Id: <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: robertsjustin@example.org From: Marvin Humphrey <robertsjustin@example.org> Subject: Input to ./charmonize Date: Sun, 15 Oct 2006 11:01:27 -0700 X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Greets, I think the best way to communicate a string representing compiler flags is to have the native build script write a file and have the charmonize app read it. Right now usage for charmonize looks like this: ./charmonize --outpath=OUTPATH [--cc=COMPILER] An input file is more extensible, though: ./charmonize INFILE I'm thinking something XML-ish for the input format. This isn't part of the Charmonizer library, it's specific to our build scripts, and only people who write bindings are going to have to deal with it. <charm_compiler>gcc</charm_compiler> <charm_ccflags>-g -pipe -fno-common -DPERL_DARWIN -no-cpp- precomp -fno-strict-aliasing -I/usr/local/include</charm_ccflags> <charm_outpath>lucyconf.h</charm_outpath> That's cake to parse from C and human-readable. Prepending 'charm_' to each tag name makes the delimiters robust. Will there ever be a situation where we'd need to make the compiler flags anything other than a fixed string? There might be for other potential uses of Charmonizer down the road, but for Lucy, I think the fixed string suffices. Each of these vars would be a char* in Charmonizer/Core, with a public setter in Charmonizer/Main: /* this is in charmonize.c */ compiler = extract_delimited(infile_contents, "charm_compiler"); chaz_Main_set_compiler(compiler); Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Sun Oct 15 18:13:36 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 69128 invoked from network); 15 Oct 2006 18:13:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 15 Oct 2006 18:13:36 -0000 Received: (qmail 14087 invoked by uid 500); 15 Oct 2006 18:13:36 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 14046 invoked by uid 500); 15 Oct 2006 18:13:36 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 14028 invoked by uid 99); 15 Oct 2006 18:13:36 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Oct 2006 11:13:36 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 15 Oct 2006 11:13:34 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GZAVj-000Exw-5t for robertsjustin@example.org; Sun, 15 Oct 2006 11:14:55 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) Content-Transfer-Encoding: 7bit Message-Id: <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: robertsjustin@example.org From: Marvin Humphrey <robertsjustin@example.org> Subject: Large File Support Date: Sun, 15 Oct 2006 11:13:12 -0700 X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Greets, It would be possible to have Charmonizer's LargeFiles module probe whether configuring the _LARGEFILE_SOURCE and _FILE_OFFSET_BITS macros succeeds in providing us with a 64-bit offset type when the current probes fail (as they do on Linux). However, I think that sets us up for a possible confict with the host environment in the case that it wasn't compiled with large file support. Here's a discussion of the issue: http://www.ohse.de/uwe/articles/lfs.html I think what we ought to do is just leave the LargeFiles module as it is now and count on the ccflags to define those macros when necessary (-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE). Simpler, and failsafe. Is it reasonable to make large file support a required prerequisite for Lucy? Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Wed Oct 18 15:45:32 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 95665 invoked from network); 18 Oct 2006 15:45:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 18 Oct 2006 15:45:32 -0000 Received: (qmail 55381 invoked by uid 500); 18 Oct 2006 15:45:32 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 55357 invoked by uid 500); 18 Oct 2006 15:45:32 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 55348 invoked by uid 99); 18 Oct 2006 15:45:32 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Oct 2006 08:45:32 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=URIBL_OB_SURBL X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Oct 2006 08:45:27 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1GaDdx-0000Ap-Br for robertsjustin@example.org; Wed, 18 Oct 2006 08:47:45 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) Content-Transfer-Encoding: 7bit Message-Id: <robertsjustin@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: robertsjustin@example.org From: Marvin Humphrey <robertsjustin@example.org> Subject: Directory layout Date: Wed, 18 Oct 2006 08:45:03 -0700 X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Greets, Perl wants to have a bunch of stuff in the root directory of the distro. In addition to Build.PL it wants MANIFEST, META.yml, Changes... and there are some other optional things that would make life more pleasant too, like MANIFEST.SKIP. Presumably Ruby wants analogous items, and other languages would as well. That's a recipe for an incoherent root directory. So instead of the current layout, I think we ought to have one folder per language at the root dir. The Perl modules will will move out of $DIST_ROOT/src/perl and into $DIST_ROOT/perl/lib. Ditto with Ruby: $DIST_ROOT/ruby/lib. When the perl distro gets built for CPAN, stuff will have to move around some so that the CPAN automated tools find a Build.PL and so on in the expected location. No big deal, that. How does this look? ./c_src ./c_test ./charmonizer ./perl ./ruby ./README Marvin Humphrey Rectangular Research http://www.rectangular.com/ From lucyrobertsjustin@example.org Wed Oct 18 15:54:56 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 99522 invoked from network); 18 Oct 2006 15:54:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 18 Oct 2006 15:54:56 -0000 Received: (qmail 81078 invoked by uid 500); 18 Oct 2006 15:54:55 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 81062 invoked by uid 500); 18 Oct 2006 15:54:55 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To: mailing list robertsjustin@example.org Received: (qmail 81053 invoked by uid 99); 18 Oct 2006 15:54:55 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Oct 2006 08:54:55 -0700 X-ASF-Spam-Status: No, hits=3.1 required=10.0 tests=DNS_FROM_RFC_ABUSE,SPF_PASS,URIBL_OB_SURBL X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of robertsjustin@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Oct 2006 08:54:53 -0700 Received: by nf-out-0910.google.com with SMTP id b2so673815nfe for <robertsjustin@example.org>; Wed, 18 Oct 2006 08:54:31 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=XIdGMDydhaF9xumBDAAUnPL5Jmavl1RaQfFrhnjWXgN2tYb6Cw7ww5eDuUvL6I1VaLrBEOGwLqvm6mse+YzOiF74Xhq0C6A7PNjmtrcAarPgQL9XHFVnDRXAfI0SLNUZ0btkNp/diF5el5HAZYYpyr/YV3isTwXSQHjdD7ueows= Received: by 83.161.120.155 with SMTP id q2mr2293870buc; Wed, 18 Oct 2006 08:54:31 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 18 Oct 2006 08:54:31 -0700 (PDT) Message-ID: <robertsjustin@example.org> Date: Thu, 19 Oct 2006 00:54:31 +0900 From: "David Balmain" <robertsjustin@example.org> To: robertsjustin@example.org Subject: Re: Directory layout In-Reply-To: <robertsjustin@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <robertsjustin@example.org> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On 10/19/06, Marvin Humphrey <robertsjustin@example.org> wrote: > Greets, > > Perl wants to have a bunch of stuff in the root directory of the > distro. In addition to Build.PL it wants MANIFEST, META.yml, > Changes... and there are some other optional things that would make > life more pleasant too, like MANIFEST.SKIP. Presumably Ruby wants > analogous items, and other languages would as well. That's a recipe > for an incoherent root directory. > > So instead of the current layout, I think we ought to have one folder > per language at the root dir. The Perl modules will will move out of > $DIST_ROOT/src/perl and into $DIST_ROOT/perl/lib. Ditto with Ruby: > $DIST_ROOT/ruby/lib. > > When the perl distro gets built for CPAN, stuff will have to move > around some so that the CPAN automated tools find a Build.PL and so > on in the expected location. No big deal, that. > > How does this look? > > ./c_src > ./c_test > ./charmonizer > ./perl > ./ruby > ./README > Looks good to me. From lucyrobertsjustin@example.org Wed Oct 18 17:18:01 2006 Return-Path: <lucyrobertsjustin@example.org> Delivered-To: robertsjustin@example.org Received: (qmail 29944 invoked from network); 18 Oct 2006 17:18:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 18 Oct 2006 17:18:01 -0000 Received: (qmail 86965 invoked by uid 500); 18 Oct 2006 17:18:01 -0000 Delivered-To: robertsjustin@example.org Received: (qmail 86945 invoked by uid 500); 18 Oct 2006 17:18:01 -0000 Mailing-List: contact robertsjustin@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:robertsjustin@example.org> List-Unsubscribe: <mailto:robertsjustin@example.org> List-Post: <mailto:robertsjustin@example.org> List-Id: <lucy-dev.lucene.apache.org> Reply-To: robertsjustin@example.org Delivered-To:
{'source': 'Apache Software Foundation Public Mail Archives'}