From java-devrebeccamedina@example.net Sun Jan 01 20:10:33 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 8770 invoked from network); 1 Jan 2006 20:10:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2006 20:10:28 -0000 Received: (qmail 29946 invoked by uid 500); 1 Jan 2006 20:10:24 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 29917 invoked by uid 500); 1 Jan 2006 20:10:24 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 29905 invoked by uid 99); 1 Jan 2006 20:10:23 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jan 2006 12:10:22 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id D9A66DE for <rebeccamedina@example.net>; Sun, 1 Jan 2006 21:10:01 +0100 (CET) Message-ID: <rebeccamedina@example.net> Date: Sun, 1 Jan 2006 21:10:01 +0100 (CET) From: "paul.elschot (JIRA)" <rebeccamedina@example.net> To: rebeccamedina@example.net Subject: [jira] Updated: (LUCENE-330) [PATCH] Use filter bits for next() and skipTo() in FilteredQuery Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/LUCENE-330?page=all ] paul.elschot updated LUCENE-330: -------------------------------- Attachment: (was: IterFilter.java) > [PATCH] Use filter bits for next() and skipTo() in FilteredQuery > ---------------------------------------------------------------- > > Key: LUCENE-330 > URL: http://issues.apache.org/jira/browse/LUCENE-330 > Project: Lucene - Java > Type: Improvement > Components: Search > Versions: CVS Nightly - Specify date in submission > Environment: Operating System: other > Platform: Other > Reporter: paul.elschot > Assignee: Lucene Developers > Priority: Minor > Attachments: FilteredQuery.java, FilteredQuery.java, FilteredQuery.java, FilteredQuery.java, FilteredQueryPatch1.txt, IndexSearcherPatch2.txt, SkipFilter.java, SkipFilter.java > > This improves performance of FilteredQuery by not calling score() > on documents that do not pass the filter. > This passes the current tests for FilteredQuery, but these tests > have not been adapted/extended. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Sun Jan 01 20:34:34 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 23467 invoked from network); 1 Jan 2006 20:34:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2006 20:34:34 -0000 Received: (qmail 48522 invoked by uid 500); 1 Jan 2006 20:34:22 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 48506 invoked by uid 500); 1 Jan 2006 20:34:22 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 48495 invoked by uid 99); 1 Jan 2006 20:34:22 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jan 2006 12:34:22 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id E9C07DE for <rebeccamedina@example.net>; Sun, 1 Jan 2006 21:34:00 +0100 (CET) Message-ID: <rebeccamedina@example.net> Date: Sun, 1 Jan 2006 21:34:00 +0100 (CET) From: "paul.elschot (JIRA)" <rebeccamedina@example.net> To: rebeccamedina@example.net Subject: [jira] Commented: (LUCENE-328) Some utilities for a compact sparse filter Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/LUCENE-328?page=comments#action_12361490 ] paul.elschot commented on LUCENE-328: ------------------------------------- > 1. Any particular reason for SortedVIntList not to implement DocNrSkipper interface, the method getDocNrSkipper() is there, but declaration is missing. The object returned by the getDocNrSkipper() method implements the interface by adding a bit of state for the iteration over the document numbers. This allows more than one iterator on the (non modifiable) SortedVIntList. > 2. Should getDocNrSkipper() DocNrSkipper interface throw IOException? I have tried to add TermDocsSortedIntList to the family, but all methods in TermDocs are throwing IOException, and it is not nice to eat silently this exception too early in DocNrSkipper. Better ideas to deal with that? I have no problem with the addition of throwing an IOException to the DocNrSkipper interface. The idea is to filter query results from RAM from which one would not normally expect an IOException , so one could also consider rethrowing the IOException wrapped in an Error. OTOH, the ability to obtain a DocNrSkipper directly from an index is certainly appealing, and then IOException is unavoidable. > 3. Paul, why SkipFilter exists (here I refer to the JIRA-330 )? Wouldn't be better to use DocNrSkipper interface instead (SkipFilter does nothing but wrapping this interface). Also, the same question applies to IterFilter. Did I get something wrong here? The presence of class BitSet in the bits() method of Filter makes it impossible to provide another implementation of a Filter. SkipFilter/DocNrSkipper are intended to be parallel to Filter/BitSet, and the DocNrSkipper interface allows alternative implementations. Both SkipFilter and Filter are interfaces for factories/caches of for filter data structures. I'd like to somehow have these parallel paths merged, but I don't now how to do that. Perhaps SkipFilter could allow backward compatibility by also providing a bits() method, and use that method when it does not throw for example an UnsupportedOperationException. Another way would be to deprecate Filter in favour of SkipFilter, but that would have a lot of backward compatibility issues, and perhaps also some performance issues. The FilteredQuery of LUCENE-330 allows for both paths to be used, both paths are joined at line 211 in this FilteredQuery. The IterFilter of LUCENE-330 was replaced by SkipFilter, I forgot to indicate that when I downloaded the replacement. I have just deleted IterFilter there. > Must say, excelent work! Thanks. I should add that most of the hard work had already been done in org.apache.lucene.store.InputStream.readVInt() and org.apache.lucene.store.OutputStream.writeVInt(). Regards, Paul Elschot > Some utilities for a compact sparse filter > ------------------------------------------ > > Key: LUCENE-328 > URL: http://issues.apache.org/jira/browse/LUCENE-328 > Project: Lucene - Java > Type: Improvement > Components: Search > Versions: CVS Nightly - Specify date in submission > Environment: Operating System: other > Platform: Other > Reporter: paul.elschot > Assignee: Lucene Developers > Priority: Minor > Attachments: AndDocNrSkipper.java, AndDocNrSkipper.java, BitSetSortedIntList.java, DocNrSkipper.java, DocNrSkipper.java, IntArraySortedIntList.java, IntArraySortedIntList.java, OrDocNrSkipper.java, OrDocNrSkipper.java, SortedVIntList.java, SortedVIntList.java, SortedVIntList.java, TestDocNrSkippers.java, TestDocNrSkippers.java, TestSortedVIntList.java, TestSortedVIntList.java, TestSortedVIntList.java, intIterator.java > > Two files are attached that might form the basis for an alternative > filter implementation that is more memory efficient than one bit > per doc when less than about 1/8 of the docs pass through the filter. > > The document numbers are stored in RAM as VInt's from the Lucene index > format. These VInt's encode the difference between two successive > document numbers, much like a PositionDelta in the Positions: > http://jakarta.apache.org/lucene/docs/fileformats.html > > The getByteSize() method can be used to verify the compression > once a SortedVIntList is constructed. > The precise conditions under which this is more memory efficient than > one bit per document are not easy to specify in advance. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Sun Jan 01 21:23:28 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 32922 invoked from network); 1 Jan 2006 21:23:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2006 21:23:28 -0000 Received: (qmail 72560 invoked by uid 500); 1 Jan 2006 21:23:27 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 72017 invoked by uid 500); 1 Jan 2006 21:23:25 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 72006 invoked by uid 99); 1 Jan 2006 21:23:25 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jan 2006 13:23:23 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id 6EDC7E0 for <rebeccamedina@example.net>; Sun, 1 Jan 2006 22:23:02 +0100 (CET) Message-ID: <{{EMAIL+PHONE}}> Date: Sun, 1 Jan 2006 22:23:02 +0100 (CET) From: "Daniel Naber (JIRA)" <rebeccamedina@example.net> To: rebeccamedina@example.net Subject: [jira] Commented: (LUCENE-478) CJK char list In-Reply-To: <rebeccamedina@example.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/LUCENE-478?page=comments#action_12361493 ] Daniel Naber commented on LUCENE-478: ------------------------------------- This is how the code looks currently: | < CJ: // Chinese, Japanese [ "\u3040"-"\u318f", "\u3300"-"\u337f", "\u3400"-"\u3d2d", "\u4e00"-"\u9fff", "\uf900"-"\ufaff" ] > | < KOREAN: // Korean [ "\uac00"-"\ud7af" ] > Are your suggested changes still needed and if so, where should which range be added (Chinese/Japanese or Korean)? > CJK char list > ------------- > > Key: LUCENE-478 > URL: http://issues.apache.org/jira/browse/LUCENE-478 > Project: Lucene - Java > Type: Bug > Components: Analysis > Versions: 1.4 > Reporter: John Wang > Priority: Minor > > Seems the character list in the CJK section of the StandardTokenizer.jj is not quite complete. Following is a more complete list: > < CJK: // non-alphabets > [ > "\u1100"-"\u11ff", > "\u3040"-"\u30ff", > "\u3130"-"\u318f", > "\u31f0"-"\u31ff", > "\u3300"-"\u337f", > "\u3400"-"\u4dbf", > "\u4e00"-"\u9fff", > "\uac00"-"\ud7a3", > "\uf900"-"\ufaff", > "\uff65"-"\uffdc" > ] > > -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Mon Jan 02 01:02:27 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 4634 invoked from network); 2 Jan 2006 01:02:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2006 01:02:27 -0000 Received: (qmail 93547 invoked by uid 500); 2 Jan 2006 01:02:25 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 93514 invoked by uid 500); 2 Jan 2006 01:02:25 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 93503 invoked by uid 99); 2 Jan 2006 01:02:24 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jan 2006 17:02:22 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id B7805E0 for <rebeccamedina@example.net>; Mon, 2 Jan 2006 02:02:01 +0100 (CET) Message-ID: <rebeccamedina@example.net> Date: Mon, 2 Jan 2006 02:02:01 +0100 (CET) From: "John Wang (JIRA)" <rebeccamedina@example.net> To: rebeccamedina@example.net Subject: [jira] Commented: (LUCENE-478) CJK char list In-Reply-To: <rebeccamedina@example.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/LUCENE-478?page=comments#action_12361497 ] John Wang commented on LUCENE-478: ---------------------------------- Yes I am. Our i18n team has provided a more up-to-date list and I thought I'd contribute it back. -John > CJK char list > ------------- > > Key: LUCENE-478 > URL: http://issues.apache.org/jira/browse/LUCENE-478 > Project: Lucene - Java > Type: Bug > Components: Analysis > Versions: 1.4 > Reporter: John Wang > Priority: Minor > > Seems the character list in the CJK section of the StandardTokenizer.jj is not quite complete. Following is a more complete list: > < CJK: // non-alphabets > [ > "\u1100"-"\u11ff", > "\u3040"-"\u30ff", > "\u3130"-"\u318f", > "\u31f0"-"\u31ff", > "\u3300"-"\u337f", > "\u3400"-"\u4dbf", > "\u4e00"-"\u9fff", > "\uac00"-"\ud7a3", > "\uf900"-"\ufaff", > "\uff65"-"\uffdc" > ] > > -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Mon Jan 02 22:15:28 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 81851 invoked from network); 2 Jan 2006 22:15:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2006 22:15:28 -0000 Received: (qmail 78768 invoked by uid 500); 2 Jan 2006 22:15:27 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 78434 invoked by uid 500); 2 Jan 2006 22:15:25 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 78423 invoked by uid 99); 2 Jan 2006 22:15:25 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jan 2006 14:15:22 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id E3E10DE for <rebeccamedina@example.net>; Mon, 2 Jan 2006 23:15:01 +0100 (CET) Message-ID: <rebeccamedina@example.net> Date: Mon, 2 Jan 2006 23:15:01 +0100 (CET) From: "paul.elschot (JIRA)" <rebeccamedina@example.net> To: rebeccamedina@example.net Subject: [jira] Updated: (LUCENE-328) Some utilities for a compact sparse filter Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/LUCENE-328?page=all ] paul.elschot updated LUCENE-328: -------------------------------- Attachment: (was: intIterator.java) > Some utilities for a compact sparse filter > ------------------------------------------ > > Key: LUCENE-328 > URL: http://issues.apache.org/jira/browse/LUCENE-328 > Project: Lucene - Java > Type: Improvement > Components: Search > Versions: CVS Nightly - Specify date in submission > Environment: Operating System: other > Platform: Other > Reporter: paul.elschot > Assignee: Lucene Developers > Priority: Minor > Attachments: AndDocNrSkipper.java, AndDocNrSkipper.java, BitSetSortedIntList.java, DocNrSkipper.java, DocNrSkipper.java, IntArraySortedIntList.java, IntArraySortedIntList.java, OrDocNrSkipper.java, OrDocNrSkipper.java, SortedVIntList.java, SortedVIntList.java, SortedVIntList.java, TestDocNrSkippers.java, TestDocNrSkippers.java, TestSortedVIntList.java, TestSortedVIntList.java, TestSortedVIntList.java > > Two files are attached that might form the basis for an alternative > filter implementation that is more memory efficient than one bit > per doc when less than about 1/8 of the docs pass through the filter. > > The document numbers are stored in RAM as VInt's from the Lucene index > format. These VInt's encode the difference between two successive > document numbers, much like a PositionDelta in the Positions: > http://jakarta.apache.org/lucene/docs/fileformats.html > > The getByteSize() method can be used to verify the compression > once a SortedVIntList is constructed. > The precise conditions under which this is more memory efficient than > one bit per document are not easy to specify in advance. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Tue Jan 03 00:09:12 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 18903 invoked from network); 3 Jan 2006 00:09:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Jan 2006 00:09:12 -0000 Received: (qmail 74261 invoked by uid 500); 3 Jan 2006 00:09:05 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 74233 invoked by uid 500); 3 Jan 2006 00:09:05 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 74222 invoked by uid 99); 3 Jan 2006 00:09:05 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jan 2006 16:09:05 -0800 X-ASF-Spam-Status: No, hits=0.5 required=10.0 tests=DNS_FROM_RFC_ABUSE X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO smtp009.mail.ukl.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 Jan 2006 16:09:04 -0800 Received: (qmail 19426 invoked from network); 3 Jan 2006 00:08:42 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.co.uk; h=Received:Message-ID:Date:From:User-Agent:X-Accept-Language:MIME-Version:To:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=GtCUdU1Ox5Vhd2obiV+V3oVUpE60blMUw+z9rWk/BXfDaQcEA8RxwhUcu1mhy6Z0NCRrzC0CCpJPnNiGf9H5fxQP0YxZcuWm0h//Bm0OwrHcyApEjDDgkdMMmjDj4JGgpIBI01VIJD3A0tv98Ty0m6AKDo9kmHbtfzQ0b+bKRp8= ; Received: from unknown (HELO ?83.161.120.155?) (rebeccamedina@example.net with plain) by smtp009.mail.ukl.yahoo.com with SMTP; 3 Jan 2006 00:08:42 -0000 Message-ID: <rebeccamedina@example.net> Date: Tue, 03 Jan 2006 00:08:38 +0000 From: markharw00d <rebeccamedina@example.net> User-Agent: Mozilla Thunderbird 1.0.5 (Windows/20050711) X-Accept-Language: en-us, en MIME-Version: 1.0 To: rebeccamedina@example.net Subject: Re: "Advanced" query language References: <rebeccamedina@example.net> <rebeccamedina@example.net> In-Reply-To: <rebeccamedina@example.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N >I thought >you said you "didn't really want to have to design a general API for >parsing XML as part of this project" ? :) > > Having grown tired of messing with my own solution I tried using commons Digester with my example XML but ran into issues so I'm back looking at a custom solution. I'd still like to keep the parser core reasonably generic (ie java.lang.Object rather than Query or Filter) because I can see it being used for instantiating many different types of objects eg requests for GroupBy , highlighting, indexing, testing etc. As for your type-safety requirement, one approach I considered which supports an extensible list of types with type safety was to use a reflection-based API like this: class MyObjectBuilder ..... parser.processChildElements(new Object(){ public void consume(Query query) { //do something with query } } ); .... class Parser public void processChildElements(Object consumer) ... This is how an ObjectBuilder could hand-off to the parser to get something nested built eg a BooleanQuery builder wanting to get a clause's choice of Query built. The generic parser knows nothing about the "Query" type - it introspects the supplied consumer and (by convention) sees what type of object the "consume" method takes. It then ensures that any child elements are built by a registered ObjectBuilder that supports that type and then passes the resulting object to the consumer by dynamically invoking the consume method. However, I'm not sure that that really buys us a lot more than the existing approach where java.lang.Object is used in ObjectConsumer and the calling ObjectBuilder has to explicitly cast a java.lang.Object to the required type. We still don't find out until runtime that something doesn't work. This type-safety requirement and the general-purpose-parser requirement seem to be at odds with each other. ___________________________________________________________ To help you stay safe and secure online, we've developed the all new Yahoo! Security Centre. http://uk.security.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Tue Jan 03 00:11:39 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 20292 invoked from network); 3 Jan 2006 00:11:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Jan 2006 00:11:39 -0000 Received: (qmail 77201 invoked by uid 500); 3 Jan 2006 00:11:38 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 76658 invoked by uid 500); 3 Jan 2006 00:11:37 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 76644 invoked by uid 99); 3 Jan 2006 00:11:36 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jan 2006 16:11:36 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO mail.app.aconex.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jan 2006 16:11:36 -0800 Received: from page.mel.office.aconex.com (page.mel.office.aconex.com [83.161.120.155]) by mail.app.aconex.com (Postfix) with ESMTP id 109AC5B804C for <rebeccamedina@example.net>; Tue, 3 Jan 2006 11:11:13 +1100 (EST) Received: from localhost (page.mel.aconex.com [83.161.120.155]) by page.mel.office.aconex.com (Postfix) with ESMTP id EEDE5534155 for <rebeccamedina@example.net>; Tue, 3 Jan 2006 11:11:12 +1100 (EST) Received: from page.mel.office.aconex.com ([83.161.120.155]) by localhost (mail.aconex.com [83.161.120.155]) (amavisd-new, port 10024) with LMTP id 00910-03-5 for <rebeccamedina@example.net>; Tue, 3 Jan 2006 11:11:12 +1100 (EST) Received: from [83.161.120.155] (unknown [83.161.120.155]) by page.mel.office.aconex.com (Postfix) with ESMTP id 2C7B5534154 for <rebeccamedina@example.net>; Tue, 3 Jan 2006 11:11:12 +1100 (EST) Mime-Version: 1.0 (Apple Message framework v746.2) In-Reply-To: <rebeccamedina@example.net> References: <rebeccamedina@example.net> <rebeccamedina@example.net> <rebeccamedina@example.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <rebeccamedina@example.net> Content-Transfer-Encoding: 7bit From: Paul Smith <rebeccamedina@example.net> Subject: Re: "Advanced" query language Date: Tue, 3 Jan 2006 11:11:11 +1100 To: rebeccamedina@example.net X-Mailer: Apple Mail (2.746.2) X-Virus-Scanned: rebeccamedina@example.net X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N On 03/01/2006, at 11:08 AM, markharw00d wrote: > >> I thought >> you said you "didn't really want to have to design a general API for >> parsing XML as part of this project" ? :) >> > > Having grown tired of messing with my own solution I tried using > commons Digester with my example XML but ran into issues so I'm > back looking at a custom solution. Seriously... Did you try out Xstream? Digester is just too hard, Xstream will work so easily you'll be pleasantly suprised.. Paul --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Tue Jan 03 10:43:23 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 44760 invoked from network); 3 Jan 2006 10:43:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Jan 2006 10:43:22 -0000 Received: (qmail 60072 invoked by uid 500); 3 Jan 2006 10:43:19 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 60053 invoked by uid 500); 3 Jan 2006 10:43:19 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 60026 invoked by uid 99); 3 Jan 2006 10:43:19 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jan 2006 02:43:19 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=HTML_MESSAGE X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy includes SPF rebeccamedina@example.net) Received: from [83.161.120.155] (HELO smtp103.biz.mail.re2.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 03 Jan 2006 02:43:18 -0800 Received: (qmail 88604 invoked from network); 3 Jan 2006 10:42:56 -0000 Received: from unknown (HELO jupiter) (rebeccamedina@example.net@83.161.120.155 with login) by smtp103.biz.mail.re2.yahoo.com with SMTP; 3 Jan 2006 10:42:54 -0000 Reply-To: <rebeccamedina@example.net> From: "Daniel Feinstein" <rebeccamedina@example.net> To: <rebeccamedina@example.net> Subject: LogFormatter Date: Tue, 3 Jan 2006 12:42:52 +0200 Organization: RawSugar MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0036_01C61063.3774B540" X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.2180 Thread-index: AcYQUnIk2fWYVxZuSpmmjBEL2Rc68w== X-Virus-Checked: Checked by ClamAV on apache.org Message-Id: <rebeccamedina@example.net> X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N ------=_NextPart_000_0036_01C61063.3774B540 Content-Type: multipart/alternative; boundary="----=_NextPart_001_0037_01C61063.37772640" ------=_NextPart_001_0037_01C61063.37772640 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi, This is my first email to the community. In our (rawsugar.com) project we integrated multiple lucene indexes and a nutch package. We use the packages for about two years. Lucene is working great and we could use it as a "black box". Unfortunately, we had to modify Nutch package for our usage because of some leaks and a design problem in LogFormatter class. This fact force us to use a customized version of Nutch and avoid to update versions. At the last week we decided to move to the latest version of Nutch but discovered that the LogFormatter still have the same problem (leaks I believe has been fixed). The problem: In Nutch project LogFormatter has duplicated functionality: 1) Logger records format and 2) Severe error handler The first usage is standard and usually could be overwritten by a user of the package by modifying logging.properties file. The second usage is much more problematic because it affects the behavior of the whole application (not only Nutch package). To support the error handling LogFormatter enforce usage of the formatter class by all classes of the whole application which uses Nutch package. This is done by overwriting all the system handlers (class java.util.logging.Handler). This operation prevents the application to use its own log formatter. Also this cause LogFormatter.hasLoggedSevere() to be sensitive to all severe records in the big system but not only to relevant. More than that this flag, LogFormatter.loggedSevere is never cleaned what means if an application had one, even unrelated severe record, tools like Fetcher will never run until the application will be restarted. I would like to suggest the following solutions: 1) To separate the functionality of log formatting and error handling or 2) Change LogFormatter class to be affected only by nutch package functions For my opinion the first solution is much better especially if error handling will be encapsulated for each task. I have found the following usages of LogFormatter.hasLoggedSevere(): - Fetcher - URLFilterChecker - ParseSegment Unfortunately I'm not familiar enough with the usages above to implement this solution that why I suggest the second one. The attached file is our implementation of LogFormatter class which we use for more than a year. I hope this change will be accepted by the community. Daniel _____________________ Daniel Feinstein Lead Software Engineer www.rawsugar.com <http://www.rawsugar.com/> __________________________ CONFIDENTIALITY NOTICE This email may contain confidential and privileged material for the sole use of the intended recipient. Any review or distribution by others is strictly prohibited. If you are not the intended recipient please contact the sender and delete all copies. ------=_NextPart_001_0037_01C61063.37772640 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"> <HTML><HEAD> <META http-equiv=3DContent-Type content=3D"text/html; = charset=3Dus-ascii"> <META content=3D"MSHTML 6.00.2900.2802" name=3DGENERATOR></HEAD> <BODY dir=3Dltr> <DIV><FONT face=3DArial size=3D2><SPAN=20 class=3D969154109-03012006>Hi,</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN=20 class=3D969154109-03012006></SPAN></FONT>&nbsp;</DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>This = is my first=20 email to the community. In our (rawsugar.com) project we integrated = multiple=20 lucene indexes and a nutch package. We use the&nbsp;packages&nbsp;for = about two=20 years. Lucene is working great and we could use it as a "black=20 box".&nbsp;Unfortunately, we had to modify Nutch package for our usage = because=20 of some leaks and a design problem in LogFormatter class.&nbsp;This fact = force=20 us to use a customized version of Nutch and avoid to update versions. At = the=20 last week we decided to move to the latest version of Nutch but = discovered that=20 the LogFormatter still have the same problem (leaks I believe has been=20 fixed).</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN=20 class=3D969154109-03012006></SPAN></FONT>&nbsp;</DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>The=20 problem:</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>In = Nutch project=20 LogFormatter has duplicated functionality:</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>1) = Logger records=20 format and</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>2) = Severe error=20 handler</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN=20 class=3D969154109-03012006></SPAN></FONT>&nbsp;</DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>The = first usage is=20 standard and usually could be overwritten by a user of the package by = modifying=20 logging.properties file.</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>The = second usage is=20 much more problematic because it affects the behavior of the whole = application=20 (not only Nutch package). To support the error handling LogFormatter = enforce=20 usage of the formatter class by all classes of the whole application = which uses=20 Nutch package. This is done by overwriting all the system handlers = (class=20 java.util.logging.Handler). This operation prevents the application to = use its=20 own log formatter. Also this cause LogFormatter.hasLoggedSevere() to be=20 sensitive to all severe records in the big system but not only to = relevant. More=20 than&nbsp;that this flag, LogFormatter.loggedSevere is never cleaned = what means=20 if an application had one, even unrelated severe record, tools like = Fetcher will=20 never run until the application will be restarted.</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN=20 class=3D969154109-03012006></SPAN></FONT>&nbsp;</DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>I = would like to=20 suggest the following solutions:</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>1) To = separate the=20 functionality of log formatting and error handling = or</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>2) = Change=20 LogFormatter class to be affected only by nutch package=20 functions</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN=20 class=3D969154109-03012006></SPAN></FONT>&nbsp;</DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>For my = opinion the=20 first solution is much better especially if error handling will be = encapsulated=20 for each task. I have found the following usages of=20 LogFormatter.hasLoggedSevere():</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>-=20 Fetcher</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>-=20 URLFilterChecker</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN class=3D969154109-03012006>-=20 ParseSegment</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2><SPAN = class=3D969154109-03012006>Unfortunately I'm=20 not familiar enough with the usages above to implement this solution = that why I=20 suggest the second one. The attached file is our implementation of = LogFormatter=20 class which we use for more than a year. I hope this change will be = accepted by=20 the community.</SPAN></FONT></DIV> <DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV> <DIV><SPAN class=3D969154109-03012006><FONT face=3DArial=20 size=3D2>Daniel</FONT></SPAN></DIV> <DIV><FONT face=3DArial size=3D2>_____________________</FONT></DIV> <DIV><FONT face=3DArial><STRONG><EM>Daniel = Feinstein</EM></STRONG></FONT></DIV> <DIV><FONT face=3DArial size=3D2>Lead Software Engineer</FONT></DIV> <DIV><FONT face=3DArial size=3D2><A=20 href=3D"http://www.rawsugar.com/">www.rawsugar.com</A></FONT></DIV> <DIV>&nbsp;</DIV> <DIV><FONT face=3DArial size=3D2>__________________________</FONT></DIV> <DIV> <P style=3D"MARGIN-TOP: 0.1in; MARGIN-BOTTOM: 0.1in"><FONT = face=3DCourier><FONT=20 size=3D2>CONFIDENTIALITY NOTICE</FONT></FONT></P> <P style=3D"MARGIN-TOP: 0.1in; MARGIN-BOTTOM: 0.1in"><FONT = face=3DCourier><FONT=20 size=3D2>This email may contain confidential and privileged material for = the sole=20 use of the intended recipient. Any review or distribution by others is = strictly=20 prohibited. If you are not the intended recipient please contact the = sender and=20 delete all copies.</FONT></FONT></P></DIV> <DIV>&nbsp;</DIV></BODY></HTML> ------=_NextPart_001_0037_01C61063.37772640-- ------=_NextPart_000_0036_01C61063.3774B540 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net ------=_NextPart_000_0036_01C61063.3774B540-- From java-devrebeccamedina@example.net Tue Jan 03 13:20:10 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 20066 invoked from network); 3 Jan 2006 13:20:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Jan 2006 13:20:09 -0000 Received: (qmail 44026 invoked by uid 500); 3 Jan 2006 13:20:08 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 43700 invoked by uid 500); 3 Jan 2006 13:20:06 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 43689 invoked by uid 99); 3 Jan 2006 13:20:06 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jan 2006 05:20:06 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO ehatchersolutions.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jan 2006 05:20:05 -0800 Received: by ehatchersolutions.com (Postfix, from userid 504) id 6C6AF13E201D; Tue, 3 Jan 2006 08:19:44 -0500 (EST) Received: from [83.161.120.155] (d-128-167-108.bootp.Virginia.EDU [83.161.120.155]) by ehatchersolutions.com (Postfix) with ESMTP id AAC1A13E201A for <rebeccamedina@example.net>; Tue, 3 Jan 2006 08:19:33 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v746.2) In-Reply-To: <rebeccamedina@example.net> References: <rebeccamedina@example.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <rebeccamedina@example.net> Content-Transfer-Encoding: 7bit From: Erik Hatcher <rebeccamedina@example.net> Subject: Re: LogFormatter Date: Tue, 3 Jan 2006 08:19:32 -0500 To: rebeccamedina@example.net X-Mailer: Apple Mail (2.746.2) X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on javelina X-Spam-Level: X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No, score=-2.6 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.0.1 X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Daniel, Please post this to the Nutch list, not the Lucene one. Erik On Jan 3, 2006, at 5:42 AM, Daniel Feinstein wrote: > Hi, > > This is my first email to the community. In our (rawsugar.com) > project we integrated multiple lucene indexes and a nutch package. > We use the packages for about two years. Lucene is working great > and we could use it as a "black box". Unfortunately, we had to > modify Nutch package for our usage because of some leaks and a > design problem in LogFormatter class. This fact force us to use a > customized version of Nutch and avoid to update versions. At the > last week we decided to move to the latest version of Nutch but > discovered that the LogFormatter still have the same problem (leaks > I believe has been fixed). > > The problem: > In Nutch project LogFormatter has duplicated functionality: > 1) Logger records format and > 2) Severe error handler > > The first usage is standard and usually could be overwritten by a > user of the package by modifying logging.properties file. > The second usage is much more problematic because it affects the > behavior of the whole application (not only Nutch package). To > support the error handling LogFormatter enforce usage of the > formatter class by all classes of the whole application which uses > Nutch package. This is done by overwriting all the system handlers > (class java.util.logging.Handler). This operation prevents the > application to use its own log formatter. Also this cause > LogFormatter.hasLoggedSevere() to be sensitive to all severe > records in the big system but not only to relevant. More than that > this flag, LogFormatter.loggedSevere is never cleaned what means if > an application had one, even unrelated severe record, tools like > Fetcher will never run until the application will be restarted. > > I would like to suggest the following solutions: > 1) To separate the functionality of log formatting and error > handling or > 2) Change LogFormatter class to be affected only by nutch package > functions > > For my opinion the first solution is much better especially if > error handling will be encapsulated for each task. I have found the > following usages of LogFormatter.hasLoggedSevere(): > - Fetcher > - URLFilterChecker > - ParseSegment > Unfortunately I'm not familiar enough with the usages above to > implement this solution that why I suggest the second one. The > attached file is our implementation of LogFormatter class which we > use for more than a year. I hope this change will be accepted by > the community. > > Daniel > _____________________ > Daniel Feinstein > Lead Software Engineer > www.rawsugar.com > > __________________________ > CONFIDENTIALITY NOTICE > This email may contain confidential and privileged material for the > sole use of the intended recipient. Any review or distribution by > others is strictly prohibited. If you are not the intended > recipient please contact the sender and delete all copies. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: rebeccamedina@example.net > For additional commands, e-mail: rebeccamedina@example.net --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Tue Jan 03 21:42:53 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 3384 invoked from network); 3 Jan 2006 21:42:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Jan 2006 21:42:52 -0000 Received: (qmail 7787 invoked by uid 500); 3 Jan 2006 21:42:49 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 7723 invoked by uid 500); 3 Jan 2006 21:42:48 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 7637 invoked by uid 99); 3 Jan 2006 21:42:47 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO ajax.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Jan 2006 13:42:47 -0800 Received: from ajax.apache.org (ajax.apache.org [83.161.120.155]) by ajax.apache.org (Postfix) with ESMTP id E17D8126 for <rebeccamedina@example.net>; Tue, 3 Jan 2006 22:42:26 +0100 (CET) Message-ID: <rebeccamedina@example.net> Date: Tue, 3 Jan 2006 22:42:26 +0100 (CET) From: "Daniel Naber (JIRA)" <rebeccamedina@example.net> To: rebeccamedina@example.net Subject: [jira] Commented: (LUCENE-478) CJK char list In-Reply-To: <rebeccamedina@example.net> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/LUCENE-478?page=comments#action_12361679 ] Daniel Naber commented on LUCENE-478: ------------------------------------- John, I'm not sure I understand: do you think that this issue can be closed now? If not, could you ask your i18n experts how your changes could be integrated into the current code (the one where K/Korean and CJ are separate things)? > CJK char list > ------------- > > Key: LUCENE-478 > URL: http://issues.apache.org/jira/browse/LUCENE-478 > Project: Lucene - Java > Type: Bug > Components: Analysis > Versions: 1.4 > Reporter: John Wang > Priority: Minor > > Seems the character list in the CJK section of the StandardTokenizer.jj is not quite complete. Following is a more complete list: > < CJK: // non-alphabets > [ > "\u1100"-"\u11ff", > "\u3040"-"\u30ff", > "\u3130"-"\u318f", > "\u31f0"-"\u31ff", > "\u3300"-"\u337f", > "\u3400"-"\u4dbf", > "\u4e00"-"\u9fff", > "\uac00"-"\ud7a3", > "\uf900"-"\ufaff", > "\uff65"-"\uffdc" > ] > > -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Wed Jan 04 09:29:23 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 11336 invoked from network); 4 Jan 2006 09:29:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2006 09:29:23 -0000 Received: (qmail 92912 invoked by uid 500); 4 Jan 2006 09:29:20 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 92873 invoked by uid 500); 4 Jan 2006 09:29:19 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 92862 invoked by uid 99); 4 Jan 2006 09:29:19 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2006 01:29:19 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO rescomp.berkeley.edu) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2006 01:29:18 -0800 Received: by rescomp.berkeley.edu (Postfix, from userid 1007) id DAD3E5B76C; Wed, 4 Jan 2006 01:29:06 -0800 (PST) Received: from localhost (localhost [83.161.120.155]) by rescomp.berkeley.edu (Postfix) with ESMTP id D2F887F459 for <rebeccamedina@example.net>; Wed, 4 Jan 2006 01:29:06 -0800 (PST) Date: Wed, 4 Jan 2006 01:29:06 -0800 (PST) From: Chris Hostetter <rebeccamedina@example.net> Sender: rebeccamedina@example.net To: rebeccamedina@example.net Subject: Re: "Advanced" query language In-Reply-To: <rebeccamedina@example.net> Message-ID: <rebeccamedina@example.net> References: <rebeccamedina@example.net> <rebeccamedina@example.net> <rebeccamedina@example.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N : I'd still like to keep the parser core reasonably generic (ie : java.lang.Object rather than Query or Filter) because I can see it being : used for instantiating many different types of objects eg requests for : GroupBy , highlighting, indexing, testing etc. : As for your type-safety requirement, one approach I considered which : supports an extensible list of types with type safety was to use a : reflection-based API like this: Interesting approach. I'm generally leary of reflection, but it's just my prejudice, i have no legitimate objection to the idea. : However, I'm not sure that that really buys us a lot more than the : existing approach where java.lang.Object is used in ObjectConsumer and : the calling ObjectBuilder has to explicitly cast a java.lang.Object to : the required type. We still don't find out until runtime that something : doesn't work. well, ultimately what we're doing is parsing text supplied by the user, there's really no way to know for certain if it's going to work untill runtime anyway -- my main concern was in eliminating ClassCastExceptions, without requiring every ObjectBuilder to do their own little instanceof/exception/cast dance... ... Object o = parser.processChild(...); if (! o instanceof FooBar) { throw new ParseException("child element is not a FooBar", ...); } FooBar f = (FooBar)obj; o = parser.processChild(...); if (! o instanceof Query) { throw new ParseException("child element is not a Query", ...); } Query q = (Query)obj ... That said, I think it's possible to eliminate some suprises at compile time using a variation of the approach i described before (where a seperate interface/register/process method is used for each type). If the ObjectBuilder's have init methods that take in the specific interface of the parser they expect to deal with, then at compile time you'll at least know if you're trying to deal with a parser that can only handle Query and Filter objects and an ObjectHandler for building test indexes that's expecting it's child nodes to be Documents. Unfortunately I can't really picture in my mind how it could work without having a buttload of marker interfaces and methods on the parsers. Perhaps some logic inversion is in order? ... For the record: it's very late, and i'm totally winging this off the cuff, so forgive me if it makes no sense what so ever. ...what if instead of registering ObjectBuilders with the parser, the ObjectBuilders are explicitly registered with eachother in a parent child relationship -- possibly with wild carded names for "any child"; and the "parser" just reads the InputStream and produced an intermediate representation (either generic SAX like events or generic dom like objects) and the user then interacts with the ObjectBuilder of the outermost type they expect. The interfaces the ObjectBuilders impliment can be organized in a hierarchy that mirrors the hierarchy of the obejcts they produce, and generic ObjectBuilder classes for high level interfaces can be made to "shortcut" common sets of Builders (ie: all supported Query types) Something like... public interface FilterObjectBuilder extends ObjectBuilder { public Filter process(Node n); } public class FOB impliments FilterObjectBuilder { public FOB(/* no children allowed */) { ... } public Filter process(Node n) { ... } } public interface QueryObjectBuilder extends ObjectBuilder { public Query process(Node n) throws UnexpectedNodeNameException; } public interface TermQueryObjectBuilder extends QueryObjectBuilder; public class TQOB impliments TermQueryObjectBuilder { public TQOB(/* no children allowed */) { ... } public Query process(Node n) { ... } } public interface FilterQueryObjectBuilder extends QueryObjectBuilder; public class FQOB impliments FilterQueryObjectBuilder { public FQOB(FilterObjectBuilder fkid, QueryObjectBuilder qkid) { ... } public Query process(Node n) { ... } } public interface BooleanQueryObjectBuilder extends QueryObjectBuilder; public class BQOB impliments BooleanQueryObjectBuilder { public BQOB(BooleanClauseObjectBuilder childBuilder) { ... } public Query process(Node n) { ... } } public interfae BooleanClauseObjectBuilder extends ObjectBuilder { public BooleanClause process(Node n); } public class BCOB impliments BooleanClauseObjectBuilder { public BCOB(QueryObjectBuilder childBuilder) { ... } public BooleanClause process(Node n) { ... } } ... public class QOB impliments QueryObjectBuilder { public QOB() { ... } /** adds a node name => QueryObjectBuilder that can be delegated to */ public void addDelegate(String name, QueryObjectBuilder builder) {...} /** throws exception if n.getName() isn't a known delegate */ public Query process(Node n) throws UnexpectedNodeNameException { ... } } ... FilterObjectBuilder f = new FOB(); QueryObjectBuilder q = new QOB(); q.addDelegate(new BQOB(new BCOB(q))); q.addDelegate(new FQOB(f, q)); q.addDelegate(new TQOB()); ... /* how people get a Query object from some XML... */ Node n = parser.parse(myXmlInputStream); Query myQuery = q.process(n); /* how people get a Filter object from some XML... */ Node n = parser.parse(myXmlInputStream); Filter myFilter = f.process(n); ...even better, people could subclass the Parser, and put those constructer/addDelegate calls in their parser's constructor, and expose new type specific "parse" methods. (NOTE: I've glossed over the other issues i mentioned in my earlier email about passing up/down state, and being able "decorate" existing ObjectBuilder's so things like <BooleanClause> could be replaced with clause="required" in any sub tag of a <BooleanQuery> ... but i *think* all of those things could still be done with an approach like this.) -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Wed Jan 04 14:02:26 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 36798 invoked from network); 4 Jan 2006 14:02:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2006 14:02:26 -0000 Received: (qmail 26742 invoked by uid 500); 4 Jan 2006 14:02:23 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 26701 invoked by uid 500); 4 Jan 2006 14:02:22 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 26680 invoked by uid 99); 4 Jan 2006 14:02:21 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2006 06:02:21 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO smtp-vbr4.xs4all.nl) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2006 06:02:20 -0800 Received: from [83.161.120.155] (a82-93-70-124.adsl.xs4all.nl [83.161.120.155]) by smtp-vbr4.xs4all.nl (8.13.3/8.13.3) with ESMTP id k04E26cY034048 for <rebeccamedina@example.net>; Wed, 4 Jan 2006 15:02:07 +0100 (CET) (envelope-from rebeccamedina@example.net) Mime-Version: 1.0 (Apple Message framework v746.2) Content-Transfer-Encoding: 7bit Message-Id: <rebeccamedina@example.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: rebeccamedina@example.net From: karl wettin <rebeccamedina@example.net> Subject: Search agents Date: Wed, 4 Jan 2006 15:03:28 +0100 X-Mailer: Apple Mail (2.746.2) X-Virus-Scanned: by XS4ALL Virus Scanner X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Hello list, I wrote a search agent thingy for Lucene. It was built to handle huge amounts of agents. Rather than one query per agent to find out if the new document is interesting or not, agent trigger queries are stored in an index that is queried with the tokens of a new document. Since it uses the index a bit backwards the agent trigger queries are somewhat limited: At least one token in a OR or FUZZY OR per agent field must match the new document. Any NOT token in agent must not match the new document. It is fairly easy to add more query types, but is limited to single token and non-wildcard types since the query if created from the new document tokens. Agents are clustered by required fields by agent, and each cluster is stored in an own index. When a new document is sent to the AgentManager it creates one query per possible cluster. I'm not sure this actually speeds things up, just a gut feeling. Example agents in psuedo trigger query language: Possible agent: AND (OR ("category","media")) AND (OR ("name", "hotel") OR ("name","rowanda")) AND (NOT("name", "paradise")) Impossible agent: AND (OR ("category","media")) AND (("name", "hotel") AND ("name","rowanda")) AND (NOT("name", "paradise")) In effect the agents can't trigger on AND queries of the same field. One could of couse place a more complex query on the new document as the agent triggers, use some classifier or whatever if speed is not a big deal. The agent triggers could then be built from the original query. I probably won't implement such a thing my self. Should I post the code to the sandbox when I've tested it? Are there any restrictions to the code if I do that? -- karl --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Wed Jan 04 14:42:49 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 62485 invoked from network); 4 Jan 2006 14:42:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2006 14:42:49 -0000 Received: (qmail 4452 invoked by uid 500); 4 Jan 2006 14:42:46 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 4415 invoked by uid 500); 4 Jan 2006 14:42:46 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 4397 invoked by uid 99); 4 Jan 2006 14:42:46 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2006 06:42:46 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO ehatchersolutions.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2006 06:42:45 -0800 Received: by ehatchersolutions.com (Postfix, from userid 504) id DC65F13E2005; Wed, 4 Jan 2006 09:42:10 -0500 (EST) Received: from [83.161.120.155] (d-128-167-108.bootp.Virginia.EDU [83.161.120.155]) by ehatchersolutions.com (Postfix) with ESMTP id 3560813E2005 for <rebeccamedina@example.net>; Wed, 4 Jan 2006 09:41:10 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v746.2) In-Reply-To: <rebeccamedina@example.net> References: <rebeccamedina@example.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <rebeccamedina@example.net> Content-Transfer-Encoding: 7bit From: Erik Hatcher <rebeccamedina@example.net> Subject: Re: Search agents Date: Wed, 4 Jan 2006 09:41:09 -0500 To: rebeccamedina@example.net X-Mailer: Apple Mail (2.746.2) X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on javelina X-Spam-Level: X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No, score=-2.6 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.0.1 X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Karl, Have you considered the MemoryIndex for this sort of thing? I've thought that it would make for an elegant way to handle this sort of "agent" or notification service such that new documents get indexed normally, but also a single document goes into a MemoryIndex and is matched against many queries. It would be great to see the code you've developed. You are free to contribute it to the Lucene contrib codebase. If it is a substantial contribution it needs further discussion and perhaps incubation for it to be accepted. There are no restrictions other than the Apache Software License on the code in the contrib area. The "sandbox" is a deprecated term for what we now call "contrib". Erik On Jan 4, 2006, at 9:03 AM, karl wettin wrote: > Hello list, > > I wrote a search agent thingy for Lucene. It was built to handle > huge amounts of agents. > > Rather than one query per agent to find out if the new document is > interesting or not, agent trigger queries are stored in an index > that is queried with the tokens of a new document. > > Since it uses the index a bit backwards the agent trigger queries > are somewhat limited: > > At least one token in a OR or FUZZY OR per agent field must match > the new document. > Any NOT token in agent must not match the new document. > > It is fairly easy to add more query types, but is limited to single > token and non-wildcard types since the query if created from the > new document tokens. > > Agents are clustered by required fields by agent, and each cluster > is stored in an own index. When a new document is sent to the > AgentManager it creates one query per possible cluster. I'm not > sure this actually speeds things up, just a gut feeling. > > Example agents in psuedo trigger query language: > > Possible agent: > > AND (OR ("category","media")) > AND (OR ("name", "hotel") OR ("name","rowanda")) > AND (NOT("name", "paradise")) > > Impossible agent: > > AND (OR ("category","media")) > AND (("name", "hotel") AND ("name","rowanda")) > AND (NOT("name", "paradise")) > > In effect the agents can't trigger on AND queries of the same field. > > One could of couse place a more complex query on the new document > as the agent triggers, use some classifier or whatever if speed is > not a big deal. The agent triggers could then be built from the > original query. I probably won't implement such a thing my self. > > Should I post the code to the sandbox when I've tested it? Are > there any restrictions to the code if I do that? > > -- > karl > > --------------------------------------------------------------------- > To unsubscribe, e-mail: rebeccamedina@example.net > For additional commands, e-mail: rebeccamedina@example.net --------------------------------------------------------------------- To unsubscribe, e-mail: rebeccamedina@example.net For additional commands, e-mail: rebeccamedina@example.net From java-devrebeccamedina@example.net Wed Jan 04 14:51:16 2006 Return-Path: <java-devrebeccamedina@example.net> Delivered-To: rebeccamedina@example.net Received: (qmail 68180 invoked from network); 4 Jan 2006 14:51:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2006 14:51:15 -0000 Received: (qmail 30273 invoked by uid 500); 4 Jan 2006 14:51:02 -0000 Delivered-To: rebeccamedina@example.net Received: (qmail 29789 invoked by uid 500); 4 Jan 2006 14:50:59 -0000 Mailing-List: contact rebeccamedina@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:rebeccamedina@example.net> List-Unsubscribe: <mailto:rebeccamedina@example.net> List-Post: <mailto:rebeccamedina@example.net> List-Id: <java-dev.lucene.apache.org> Reply-To: rebeccamedina@example.net Delivered-To: mailing list rebeccamedina@example.net Received: (qmail 29612 invoked by uid 99); 4 Jan 2006 14:50:56 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jan 2006 06:50:55 -0800 X-ASF-Spam-Status: No, hits=0.5 required=10.0 tests=DNS_FROM_RFC_ABUSE X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy includes SPF rebeccamedina@example.net) Received: from [83.161.120.155] (HELO web26014.mail.ukl.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 04 Jan 2006 06:50:53 -0800 Received: (qmail 93259 invoked by uid 60001); 4 Jan 2006 14:50:41 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.co.uk; h=Message-ID:Received:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=GRx18MgQ3BzSlVn4IXZ4Q5TtERnhB3rgThW0yPN3fruyZSaZHM6hQEs6WUBoXrp4K/lC3c6dAh9VoJB9ENX6aXMzxLt/d6MhGfUcm4LxtonorzCvUQfBWNQFtEyONmaVSUVEqwumDzKmuiAvF2beiB5JU5pno4j0KAgNo7kVmvs= ; Message-ID: <rebeccamedina@example.net> Received: from [83.161.120.155] by web26014.mail.ukl.yahoo.com via HTTP; Wed, 04 Jan 2006 14:50:41 GMT Date: Wed, 4
{'source': 'Apache Software Foundation Public Mail Archives'}