From java-devjohnwilliams@example.org Sun Apr 01 07:56:26 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 79792 invoked from network); 1 Apr 2007 07:56:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2007 07:56:26 -0000 Received: (qmail 33611 invoked by uid 500); 1 Apr 2007 07:56:31 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 33569 invoked by uid 500); 1 Apr 2007 07:56:31 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 33557 invoked by uid 99); 1 Apr 2007 07:56:31 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Apr 2007 00:56:31 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO rescomp.berkeley.edu) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Apr 2007 00:56:21 -0700 Received: by rescomp.berkeley.edu (Postfix, from userid 1007) id 71BB45B764; Sun, 1 Apr 2007 00:55:59 -0700 (PDT) Received: from localhost (localhost [83.161.120.155]) by rescomp.berkeley.edu (Postfix) with ESMTP id 686E67F403 for <johnwilliams@example.org>; Sun, 1 Apr 2007 00:55:59 -0700 (PDT) Date: Sun, 1 Apr 2007 00:55:59 -0700 (PDT) From: Chris Hostetter <johnwilliams@example.org> To: johnwilliams@example.org Subject: Re: Hits.hitDoc(int) In-Reply-To: <johnwilliams@example.org> Message-ID: <johnwilliams@example.org> References: <johnwilliams@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org 1) if you have questions about using the java lucene library, please send them to the java-user@lucene mailing list ... java-dev if for discussing hte development of the library. 2) ... : Since Hits.hitDoc(int) is private, the only way to access nth documet in a : returnd Hits object is to iterate over it n times. Why don't you make it : public? i don't understand why you think you need to iterate to get the nth match ... just call hits.doc(n) -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Sun Apr 01 10:19:42 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 106 invoked from network); 1 Apr 2007 10:19:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2007 10:19:40 -0000 Received: (qmail 15603 invoked by uid 500); 1 Apr 2007 10:19:40 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 15549 invoked by uid 500); 1 Apr 2007 10:19:40 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 15538 invoked by uid 99); 1 Apr 2007 10:19:40 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Apr 2007 03:19:40 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of johnwilliams@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO talk.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Apr 2007 03:19:32 -0700 Received: from [83.161.120.155] (helo=jubjub.nabble.com) by talk.nabble.com with esmtp (Exim 4.50) id 1HXx9U-0000Bc-4R for johnwilliams@example.org; Sun, 01 Apr 2007 03:19:12 -0700 Message-ID: <johnwilliams@example.org> Date: Sun, 1 Apr 2007 03:19:12 -0700 (PDT) From: Mohsen Saboorian <johnwilliams@example.org> To: johnwilliams@example.org Subject: Re: Hits.hitDoc(int) In-Reply-To: <johnwilliams@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: johnwilliams@example.org References: <johnwilliams@example.org> <johnwilliams@example.org> X-Virus-Checked: Checked by ClamAV on apache.org I was just *incorrectly* thinking that this is an API problem, and that's why I sent my mail to this list. Anyway, sorry for disturbance. Chris Hostetter wrote: > > > 1) if you have questions about using the java lucene library, please send > them to the java-user@lucene mailing list ... java-dev if for discussing > hte development of the library. > > 2) ... > > : Since Hits.hitDoc(int) is private, the only way to access nth documet in > a > : returnd Hits object is to iterate over it n times. Why don't you make it > : public? > > i don't understand why you think you need to iterate to get the nth match > ... just call hits.doc(n) > > > > -Hoss > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: johnwilliams@example.org > For additional commands, e-mail: johnwilliams@example.org > > > -- View this message in context: http://www.nabble.com/Hits.hitDoc%28int%29-tf3500174.html#a9776352 Sent from the Lucene - Java Developer mailing list johnwilliams@example.org. --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Sun Apr 01 18:52:06 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 20141 invoked from network); 1 Apr 2007 18:52:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2007 18:52:05 -0000 Received: (qmail 85311 invoked by uid 500); 1 Apr 2007 18:52:06 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 85280 invoked by uid 500); 1 Apr 2007 18:52:06 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 85269 invoked by uid 99); 1 Apr 2007 18:52:06 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Apr 2007 11:52:06 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO rescomp.berkeley.edu) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Apr 2007 11:51:57 -0700 Received: by rescomp.berkeley.edu (Postfix, from userid 1007) id DFC8B5B764; Sun, 1 Apr 2007 11:51:36 -0700 (PDT) Received: from localhost (localhost [83.161.120.155]) by rescomp.berkeley.edu (Postfix) with ESMTP id D98E87F403 for <johnwilliams@example.org>; Sun, 1 Apr 2007 11:51:36 -0700 (PDT) Date: Sun, 1 Apr 2007 11:51:36 -0700 (PDT) From: Chris Hostetter <johnwilliams@example.org> To: johnwilliams@example.org Subject: Re: Hits.hitDoc(int) In-Reply-To: <johnwilliams@example.org> Message-ID: <johnwilliams@example.org> References: <johnwilliams@example.org> <johnwilliams@example.org> <johnwilliams@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org : I was just *incorrectly* thinking that this is an API problem, and that's why well, even if you think there is a problem with the API, the best approach is to start by emailing question to java-user to make sure you understand the correct usage of the API, and then once you are sure that you are, follow up with a thread on java-dev about chaging the API. : I sent my mail to this list. Anyway, sorry for disturbance. there's no reason to be sorry - it's just a question of audience; we want to make sure people get good anssers to their questions, and you are more likely to get helpful answers from the (larger) java-user audience. -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Sun Apr 01 19:26:58 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 33463 invoked from network); 1 Apr 2007 19:26:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Apr 2007 19:26:57 -0000 Received: (qmail 10064 invoked by uid 500); 1 Apr 2007 19:27:03 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 9621 invoked by uid 500); 1 Apr 2007 19:27:01 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 9610 invoked by uid 99); 1 Apr 2007 19:27:01 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Apr 2007 12:27:01 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Apr 2007 12:26:53 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id DAFBD71406C for <johnwilliams@example.org>; Sun, 1 Apr 2007 12:26:32 -0700 (PDT) Message-ID: <7176367.1175455592893.JavaMail.jira@brutus> Date: Sun, 1 Apr 2007 12:26:32 -0700 (PDT) From: =?utf-8?Q?Nicolas_Lalev=C3=A9e_=28JIRA=29?= <johnwilliams@example.org> To: johnwilliams@example.org Subject: [jira] Updated: (LUCENE-662) Extendable writer and reader of field data In-Reply-To: <19231794.1156264813957.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-662?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Lalev=C3=A9e updated LUCENE-662: ----------------------------------- Attachment: indexFormat.patch indexFormat-only.patch Synchronized with the trunk, so with the payload feature. It allowed me to = refactor in one class the payload writing which is in two places today : it= is now in the DefaultPostingWriter class. >From my last update, the TODO list is still to do, nothing has been fixed. = Furthermore there is a regression in the new patch : the ensureOpen() is no= t correctly handled for lazy loaded fields : a test fail. This is due to th= e fact that the FieldsReader doesn't handle it anymore in my patch. As the = data struture can be customized, lazy loading is exported to the FieldData = created by the FieldsReader. So the both instance have to communicate about= the closing of the streams. So a new item in the TODO list. As discussed in java-dev, here is a light patch with only the index format = handling, without the possibility to redefine how data and postings are sto= re/retreived. > Extendable writer and reader of field data > ------------------------------------------ > > Key: LUCENE-662 > URL: https://issues.apache.org/jira/browse/LUCENE-662 > Project: Lucene - Java > Issue Type: Improvement > Components: Store > Reporter: Nicolas Lalev=C3=A9e > Priority: Minor > Attachments: entrytable.patch, generic-fieldIO-2.patch, generic-f= ieldIO-3.patch, generic-fieldIO-4.patch, generic-fieldIO-5.patch, generic-f= ieldIO.patch, indexFormat-only.patch, indexFormat.patch, indexFormat.patch,= indexFormat.patch > > > As discussed on the dev mailing list, I have modified Lucene to allow to = define how the data of a field is writen and read in the index. > Basically, I have introduced the notion of IndexFormat. It is in fact a f= actory of FieldsWriter and FieldsReader. So the IndexReader, the indexWrite= r and the SegmentMerger are using this factory and not doing a "new FieldsR= eader/Writer()". > I have also introduced the notion of FieldData. It handles every data of = a field, and also the writing and the reading in a stream. I have done this= way because in the current design of Lucene, Fiedable is an interface, so = methods with a protected or package visibility cannot be defined. > A FieldsWriter just writes data into a stream via the FieldData of the fi= eld. > A FieldsReader instanciates a FieldData depending on the field name. Then= it use the field data to read the stream. And finnaly it instanciates a Fi= eld with the field data. > About compatibility, I think it is kept, as I have writen a DefaultIndexF= ormat that provides some DefaultFieldsWriter and DefaultFieldsReader. These= implementations do the exact job that is done today. > To acheive this modification, some classes and methods had to be moved fr= om private and/or final to public or protected. > About the lazy fields, I have implemented them in a more general way in t= he implementation of the abstract class FieldData, so it will be totally tr= ansparent for the Lucene user that will extends FieldData. The stream is ke= pt in the fieldData and used as soon as the stringValue (or something else)= is called. Implementing this way allowed me to handle the recently introdu= ced LOAD_FOR_MERGE; it is just a lazy field data, and when read() is called= on this lazy field data, the saved input stream is directly copied in the = output stream. > I have a last issue with this patch. The current design allow to read an = index in an old format, and just do a writer.addIndexes() into a new format= . With the new design, you cannot, because the writer will use the FieldDat= a.write provided by the reader. > enjoy ! --=20 This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 08:58:37 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 22255 invoked from network); 2 Apr 2007 08:58:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 08:58:35 -0000 Received: (qmail 75147 invoked by uid 500); 2 Apr 2007 08:58:37 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 75073 invoked by uid 500); 2 Apr 2007 08:58:37 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 74967 invoked by uid 99); 2 Apr 2007 08:58:36 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 01:58:36 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of johnwilliams@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO talk.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 01:58:27 -0700 Received: from [83.161.120.155] (helo=jubjub.nabble.com) by talk.nabble.com with esmtp (Exim 4.50) id 1HYIMW-0007mI-V8 for johnwilliams@example.org; Mon, 02 Apr 2007 01:58:04 -0700 Message-ID: <johnwilliams@example.org> Date: Mon, 2 Apr 2007 01:58:04 -0700 (PDT) From: deva_java <johnwilliams@example.org> To: johnwilliams@example.org Subject: Invalid Sort results when column contains link names of like "*.html,*.xls,*.doc etc." MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_7264_23460019.1175504284962" X-Nabble-From: johnwilliams@example.org X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_7264_23460019.1175504284962 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Hi, In my application,I wanna sort out index based on different column names in ascending or descending order but it doesn't do correctly For example: it does well as its column contains single word or phrase but in the phrase type contains differnet links like .html or .doc or .xls it does sorting based on .extension so that i got invalid sorting results like "A series in middle part". How can i solve it ,Please help me Regards devanadan -- View this message in context: http://www.nabble.com/Invalid-Sort-results-when-column-contains-link-names-of-like-%22*.html%2C*.xls%2C*.doc-etc.%22-tf3504432.html#a9787099 Sent from the Lucene - Java Developer mailing list johnwilliams@example.org. ------=_Part_7264_23460019.1175504284962-- From java-devjohnwilliams@example.org Mon Apr 02 14:23:42 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 34952 invoked from network); 2 Apr 2007 14:23:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 14:23:40 -0000 Received: (qmail 2009 invoked by uid 500); 2 Apr 2007 14:23:43 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 1920 invoked by uid 500); 2 Apr 2007 14:23:43 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 1880 invoked by uid 99); 2 Apr 2007 14:23:43 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 07:23:43 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO mailer.syr.edu) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 07:23:34 -0700 Received: from [83.161.120.155] (ist-h335-d03.syr.edu) by mailer.syr.edu (LSMTP for Windows NT v1.1b) with SMTP id <johnwilliams@example.org>; Mon, 2 Apr 2007 10:23:13 -0400 Message-ID: <johnwilliams@example.org> Date: Mon, 02 Apr 2007 10:24:03 -0400 From: Steven Rowe <johnwilliams@example.org> User-Agent: Mail/News 83.161.120.155-GroupWise-IMAP-fix (Windows/20060619) MIME-Version: 1.0 To: johnwilliams@example.org Subject: Re: Invalid Sort results when column contains link names of like "*.html,*.xls,*.doc etc." References: <johnwilliams@example.org> In-Reply-To: <johnwilliams@example.org> X-Enigmail-Version: 83.161.120.155 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi devanadan, deva_java wrote: > In my application,I wanna sort out index based on different column names in > ascending or descending order but it doesn't do correctly > > For example: > > it does well as its column contains single word or phrase but in the > phrase type contains differnet links like .html or .doc or .xls it > does sorting based on .extension so that i got invalid sorting > results like "A series in middle part". This question, one of Lucene usage rather than of development, really belongs on the java-user list instead of java-dev. I'm guessing that the sort field you're using is tokenized - if so, this is a problem. >From the org.apache.lucene.search.Sort API documentation[1]: The fields used to determine sort order must be carefully chosen. Documents must contain a single term in such a field .... The field must be indexed, but should not be tokenized, and does not need to be stored (unless you happen to want it back with the rest of your document data). In other words: document.add(new Fieldable("byNumber", Integer.toString(x), Fieldable.Store.NO, Fieldable.Index.UN_TOKENIZED)); Your options (in order of likely quickest results): 1. Switch the field to be sorted so that it is not tokenized. Things will likely "just work" if you do this. 2. If the field you're using must remain tokenized, add a new field that is a duplicate of the current sort field, but make it untokenized, and then make this new field the one you sort on. 3. There is an ongoing discussion at issue #252 in JIRA[2] that may be of interest to you if neither of the above options is feasible. Steve [1] <http://lucene.apache.org/java/docs/api/org/apache/lucene/search/Sort.html> [2] <http://issues.apache.org/jira/browse/LUCENE-252> --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 14:43:59 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 43843 invoked from network); 2 Apr 2007 14:43:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 14:43:58 -0000 Received: (qmail 45105 invoked by uid 500); 2 Apr 2007 14:44:01 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 45059 invoked by uid 500); 2 Apr 2007 14:44:01 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 44972 invoked by uid 99); 2 Apr 2007 14:44:01 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 07:44:01 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 07:43:52 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id B049071406C for <johnwilliams@example.org>; Mon, 2 Apr 2007 07:43:32 -0700 (PDT) Message-ID: <6715534.1175525012718.JavaMail.jira@brutus> Date: Mon, 2 Apr 2007 07:43:32 -0700 (PDT) From: "Michael McCandless (JIRA)" <johnwilliams@example.org> To: johnwilliams@example.org Subject: [jira] Updated: (LUCENE-843) improve how IndexWriter uses RAM to buffer added documents In-Reply-To: <16648050.1174583194037.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-843: -------------------------------------- Attachment: LUCENE-843.take4.patch Another rev of the patch. All tests pass except disk full tests. The code is still rather "dirty" and not well commented. I think I'm close to finishing optimizing and now I will focus on error handling (eg disk full), adding some deeper unit tests, more testing on corner cases like massive docs or docs with massive terms, etc., flushing pending norms to disk, cleaning up / commenting the code and various other smaller items. Here are the changes in this rev: * A proposed backwards compatible change to the Token API to also allow the term text to be delivered as a slice (offset & length) into a char[] array instead of String. With an analyzer/tokenizer that takes advantage of this, this was a decent performance gain in my local testing. I've created a SimpleSpaceAnalyzer that only splits words at the space character to test this. * Added more asserts (run java -ea to enable asserts). The asserts are quite useful and now often catch a bug I've introduced before the unit tests do. * Changed to custom int[] block buffering for postings to store freq, prox's and offsets. With this buffering we no longer have to double the size of int[] arrays while adding positions, nor do we have to copy ints whenever we needs more space for these arrays. Instead I allocate larger slices out of the shared int[] arrays. This reduces memory and improves performance. * Changed to custom char[] block buffering for postings to store term text. This also reduces memory and improves performance. * Changed to single file for RAM & flushed partial segments (was 3 separate files before) * Changed how I merge flushed partial segments to match what's described in LUCENE-854 * Reduced memory usage when indexing large docs (25 MB plain text each). I'm still consuming more RAM in this case than the baseline (trunk) so I'm still working on this one ... * Fixed a slow memory leak when building large (20+ GB) indices > improve how IndexWriter uses RAM to buffer added documents > ---------------------------------------------------------- > > Key: LUCENE-843 > URL: https://issues.apache.org/jira/browse/LUCENE-843 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Affects Versions: 2.2 > Reporter: Michael McCandless > Assigned To: Michael McCandless > Priority: Minor > Attachments: LUCENE-843.patch, LUCENE-843.take2.patch, LUCENE-843.take3.patch, LUCENE-843.take4.patch > > > I'm working on a new class (MultiDocumentWriter) that writes more than > one document directly into a single Lucene segment, more efficiently > than the current approach. > This only affects the creation of an initial segment from added > documents. I haven't changed anything after that, eg how segments are > merged. > The basic ideas are: > * Write stored fields and term vectors directly to disk (don't > use up RAM for these). > * Gather posting lists & term infos in RAM, but periodically do > in-RAM merges. Once RAM is full, flush buffers to disk (and > merge them later when it's time to make a real segment). > * Recycle objects/buffers to reduce time/stress in GC. > * Other various optimizations. > Some of these changes are similar to how KinoSearch builds a segment. > But, I haven't made any changes to Lucene's file format nor added > requirements for a global fields schema. > So far the only externally visible change is a new method > "setRAMBufferSize" in IndexWriter (and setMaxBufferedDocs is > deprecated) so that it flushes according to RAM usage and not a fixed > number documents added. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 18:17:03 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 37216 invoked from network); 2 Apr 2007 18:17:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 18:17:00 -0000 Received: (qmail 36199 invoked by uid 500); 2 Apr 2007 18:17:05 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 36160 invoked by uid 500); 2 Apr 2007 18:17:04 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 36145 invoked by uid 99); 2 Apr 2007 18:17:04 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 11:17:04 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO web50301.mail.re2.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 Apr 2007 11:16:56 -0700 Received: (qmail 87339 invoked by uid 60001); 2 Apr 2007 18:16:35 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=B22oW3U4gfmaWKNkfqYFTLtQ5/++j9s0dHDBw/P5Udt6pGfX+e6gQ7LrVCp0QcxhIpZeiigXNvWn4al/vTk6tUFY946ynkXUxLqGvBFwr/Go8CrAro6dC1Np6O8zZrBRMICERxdZ9rUKPGoXzTpmfnZOcOrt0vtPN3H0yLg8AB8=; X-YMail-OSG: wn.D4IEVM1n6F7TmaUNUwM8QyOzPbBhMl8_XWZFcXEcO3sL5 Received: from [83.161.120.155] by web50301.mail.re2.yahoo.com via HTTP; Mon, 02 Apr 2007 11:16:35 PDT X-Mailer: YahooMailRC/476 YahooMailWebService/83.161.120.155 Date: Mon, 2 Apr 2007 11:16:35 -0700 (PDT) From: Otis Gospodnetic <johnwilliams@example.org> Subject: Re: Commit and Review (was "Is Lucene Java trunk still stable for production code?") To: johnwilliams@example.org MIME-Version: 1.0 Content-Type: text/plain; charset=ascii Message-ID: <johnwilliams@example.org> X-Virus-Checked: Checked by ClamAV on apache.org I don't think anyone followed-up on this. I think I prefer our current approach, where contributors provide patches, everyone is free to review and comment on them, and commiters finally commit it. Feels a bit slower than commit first, review second, but I think that makes it easier to control the quality of the code that goes in. There seems to be enough momentum/active developers that even with the current patch first approach we could release more often. Otis ----- Original Message ---- From: Grant Ingersoll <johnwilliams@example.org> To: johnwilliams@example.org Sent: Saturday, March 17, 2007 12:02:55 PM Subject: Commit and Review (was "Is Lucene Java trunk still stable for production code?") Hoss wrote: > (or in short: we're moving more towards a *true* commit and review model) I'm curious as to what you think are the practical implications are for committers for this model? Do you imagine a change in the workflow whereby we commit and then review or do we stick to the patch approach as committers (contributors will always submit patches)? It has always been a gray area, where we all kind of know what we can commit w/o creating patches for versus what we should put up patches for. Just curious, I'm working on the payloads stuff and I know that as long as it compiles, it isn't going to break anything, so in some sense I could commit b/c I know it would make it easier for Michael B. and others to update and review w/o going through the patch process. On the other hand, the patch approach makes you take one extra good look at what you are doing. What do others think? -Grant --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 19:42:29 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 81534 invoked from network); 2 Apr 2007 19:42:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 19:42:29 -0000 Received: (qmail 74351 invoked by uid 500); 2 Apr 2007 19:42:33 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 74305 invoked by uid 500); 2 Apr 2007 19:42:32 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 74294 invoked by uid 99); 2 Apr 2007 19:42:32 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 12:42:32 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO esseff.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 12:42:23 -0700 Received: from localhost (localhost [83.161.120.155]) by esseff.org (Postfix) with ESMTP id 488C5647FD3 for <johnwilliams@example.org>; Mon, 2 Apr 2007 12:43:31 -0700 (PDT) X-Quarantine-ID: <g+S9YzE57ryW> X-Virus-Scanned: johnwilliams@example.org Received: from esseff.org ([83.161.120.155]) by localhost (esseff.org [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id g+S9YzE57ryW for <johnwilliams@example.org>; Mon, 2 Apr 2007 12:43:27 -0700 (PDT) Received: from professorville.windows.esseff.org (professorville.windows.esseff.org [83.161.120.155]) by esseff.org (Postfix) with ESMTP id 7658F647FCB for <johnwilliams@example.org>; Mon, 2 Apr 2007 12:43:27 -0700 (PDT) Subject: RE: [jira] Updated: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-class: urn:content-classes:message Date: Mon, 2 Apr 2007 12:41:58 -0700 X-MIMEOLE: Produced By Microsoft Exchange V6.5 Message-ID: <johnwilliams@example.org> In-Reply-To: <johnwilliams@example.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [jira] Updated: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff Thread-Index: AcdxX/d07NsoO3O2SY2rchqRXoJpHQD/qcnQ References: <24021458.1175101765409.JavaMail.jira@brutus> <johnwilliams@example.org> From: "Steven Parkes" <johnwilliams@example.org> To: <johnwilliams@example.org> X-Virus-Checked: Checked by ClamAV on apache.org I checked and there are escape sequences in there. If it was ever debatable, I think that tips it in favor of SAX. xerces? The contrib/gdata stuff seems to use it. I suppose if I'm careful and creative enough, we could share a lot of the code amongst benchmark ingesters that use XML, should there be more ...=20 -----Original Message----- From: Grant Ingersoll [mailto:johnwilliams@example.org]=20 Sent: Wednesday, March 28, 2007 10:44 AM To: johnwilliams@example.org Subject: Re: [jira] Updated: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff On Mar 28, 2007, at 1:09 PM, Steven Parkes (JIRA) wrote: > > [ https://issues.apache.org/jira/browse/LUCENE-848?=20 > page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] > > Steven Parkes updated LUCENE-848: > --------------------------------- > > Description: Add support for using Wikipedia for =20 > benchmarking. (was: Add support for using Wikipedia for =20 > benchmarking. If no one is working on this, I'll start soon.) > Lucene Fields: (was: [New]) > Summary: Add supported for Wikipedia English as a corpus =20 > in the benchmarker stuff (was: Add supported for Wikipediea =20 > English as a corpus in the benchmarker stuff) > > Can't leave the typo in the title. It's bugging me. > > Karl, it looks like your stuff grabs individual articles, right? =20 > I'm gong to have it download the bzip2 snapshots they provide (and =20 > that they prefer you use, if you're getting much). > > Question (for Doron and anyone else): the file is xml and it's big, =20 > so DOM isn't going to work. I could still use something SAX based =20 > but since the format is so tightly controlled, I'm thinking regular =20 > expressions would be sufficient and have less dependences. Anyone =20 > have opinions on this? Personally, I think SAX is the way to go, as you'll get handling of =20 escape sequences, etc. out of the box. And seems like it is easier =20 to read/maintain???? > >> Add supported for Wikipedia English as a corpus in the benchmarker =20 >> stuff >> --------------------------------------------------------------------- >> --- >> >> Key: LUCENE-848 >> URL: https://issues.apache.org/jira/browse/LUCENE-848 >> Project: Lucene - Java >> Issue Type: New Feature >> Components: contrib/benchmark >> Reporter: Steven Parkes >> Assigned To: Steven Parkes >> Priority: Minor >> Fix For: 2.2 >> >> Attachments: WikipediaHarvester.java >> >> >> Add support for using Wikipedia for benchmarking. > > --=20 > This message is automatically generated by JIRA. > - > You can reply to this email to add a comment to the issue online. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: johnwilliams@example.org > For additional commands, e-mail: johnwilliams@example.org > ------------------------------------------------------ Grant Ingersoll http://www.grantingersoll.com/ http://lucene.grantingersoll.com http://www.paperoftheweek.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 20:01:42 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 94609 invoked from network); 2 Apr 2007 20:01:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 20:01:40 -0000 Received: (qmail 12219 invoked by uid 500); 2 Apr 2007 20:01:45 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 11793 invoked by uid 500); 2 Apr 2007 20:01:44 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 11779 invoked by uid 99); 2 Apr 2007 20:01:44 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 13:01:44 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of johnwilliams@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wx-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 13:01:36 -0700 Received: by wx-out-0506.google.com with SMTP id i29so1271979wxd for <johnwilliams@example.org>; Mon, 02 Apr 2007 13:01:15 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:mime-version:in-reply-to:references:content-type:message-id:content-transfer-encoding:from:subject:date:to:x-mailer; b=r4Hy+MuZ4M7hbvl8ZSO/oezeBIHSz9FbLv14mUAFFeBkLok5SoS9PGQ6Yyg5yx2tJPFOsdroQlRjiMoFUVerbCGDqyTv/GaFCXAEg01Ri6YSlC7ToMyLvQx5Y08h5XGDEO3WfcdDtVvaSzaA77Msfbra/l+bMzxxGkfvgydFLdE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:mime-version:in-reply-to:references:content-type:message-id:content-transfer-encoding:from:subject:date:to:x-mailer; b=oZYcV9AEB4neqvyHW2/v0WbijT3ssnpWj75C3tFHxy9MoEHG5RP52LyrA69+jUrn4PtF/cSqTtw8FqlPi/jREBIBbMSFzccorH11t0rzBSa5hHRc6PGoC+La6H7AlMA/UJikdHPfYCkFYU1I8Xc4ljenQzYzBE5TAncCJlV2mEY= Received: by 83.161.120.155 with SMTP id l20mr3853676agc.1175544075296; Mon, 02 Apr 2007 13:01:15 -0700 (PDT) Received: from ?83.161.120.155? ( [83.161.120.155]) by mx.google.com with ESMTP id 7sm5555877wrh.283.161.120.155.01.09; Mon, 02 Apr 2007 13:01:10 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <johnwilliams@example.org> References: <24021458.1175101765409.JavaMail.jira@brutus> <johnwilliams@example.org> <johnwilliams@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <johnwilliams@example.org> Content-Transfer-Encoding: 7bit From: Grant Ingersoll <johnwilliams@example.org> Subject: Re: [jira] Updated: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff Date: Mon, 2 Apr 2007 16:01:07 -0400 To: johnwilliams@example.org X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org On Apr 2, 2007, at 3:41 PM, Steven Parkes wrote: > I checked and there are escape sequences in there. If it was ever > debatable, I think that tips it in favor of SAX. xerces? The > contrib/gdata stuff seems to use it. Xerces should be fine, I think. > > I suppose if I'm careful and creative enough, we could share a lot of > the code amongst benchmark ingesters that use XML, should there be > more > ... > Yes, indeed. May not be necessary initially, but we could support XPath or something down the road to allow us to specify what things we are interested in. I wouldn't worry about generalizing too much to start with. Once we have a couple collections then we can go that route. > -----Original Message----- > From: Grant Ingersoll [mailto:johnwilliams@example.org] > Sent: Wednesday, March 28, 2007 10:44 AM > To: johnwilliams@example.org > Subject: Re: [jira] Updated: (LUCENE-848) Add supported for Wikipedia > English as a corpus in the benchmarker stuff > > > On Mar 28, 2007, at 1:09 PM, Steven Parkes (JIRA) wrote: > >> >> [ https://issues.apache.org/jira/browse/LUCENE-848? >> page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] >> >> Steven Parkes updated LUCENE-848: >> --------------------------------- >> >> Description: Add support for using Wikipedia for >> benchmarking. (was: Add support for using Wikipedia for >> benchmarking. If no one is working on this, I'll start soon.) >> Lucene Fields: (was: [New]) >> Summary: Add supported for Wikipedia English as a corpus >> in the benchmarker stuff (was: Add supported for Wikipediea >> English as a corpus in the benchmarker stuff) >> >> Can't leave the typo in the title. It's bugging me. >> >> Karl, it looks like your stuff grabs individual articles, right? >> I'm gong to have it download the bzip2 snapshots they provide (and >> that they prefer you use, if you're getting much). >> >> Question (for Doron and anyone else): the file is xml and it's big, >> so DOM isn't going to work. I could still use something SAX based >> but since the format is so tightly controlled, I'm thinking regular >> expressions would be sufficient and have less dependences. Anyone >> have opinions on this? > > > Personally, I think SAX is the way to go, as you'll get handling of > escape sequences, etc. out of the box. And seems like it is easier > to read/maintain???? > >> >>> Add supported for Wikipedia English as a corpus in the benchmarker >>> stuff >>> -------------------------------------------------------------------- >>> - > >>> --- >>> >>> Key: LUCENE-848 >>> URL: https://issues.apache.org/jira/browse/ >>> LUCENE-848 >>> Project: Lucene - Java >>> Issue Type: New Feature >>> Components: contrib/benchmark >>> Reporter: Steven Parkes >>> Assigned To: Steven Parkes >>> Priority: Minor >>> Fix For: 2.2 >>> >>> Attachments: WikipediaHarvester.java >>> >>> >>> Add support for using Wikipedia for benchmarking. >> >> -- >> This message is automatically generated by JIRA. >> - >> You can reply to this email to add a comment to the issue online. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: johnwilliams@example.org >> For additional commands, e-mail: johnwilliams@example.org >> > > ------------------------------------------------------ > Grant Ingersoll > http://www.grantingersoll.com/ > http://lucene.grantingersoll.com > http://www.paperoftheweek.com/ > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: johnwilliams@example.org > For additional commands, e-mail: johnwilliams@example.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: johnwilliams@example.org > For additional commands, e-mail: johnwilliams@example.org > ------------------------------------------------------ Grant Ingersoll http://www.grantingersoll.com/ http://lucene.grantingersoll.com http://www.paperoftheweek.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 21:51:14 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 51307 invoked from network); 2 Apr 2007 21:51:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 21:51:13 -0000 Received: (qmail 10680 invoked by uid 500); 2 Apr 2007 21:51:17 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 10644 invoked by uid 500); 2 Apr 2007 21:51:17 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 10629 invoked by uid 99); 2 Apr 2007 21:51:17 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 14:51:17 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO esseff.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 14:51:08 -0700 Received: from localhost (localhost [83.161.120.155]) by esseff.org (Postfix) with ESMTP id A819264802E for <johnwilliams@example.org>; Mon, 2 Apr 2007 14:52:17 -0700 (PDT) X-Quarantine-ID: <3wY61Ob1jqxU> X-Virus-Scanned: johnwilliams@example.org Received: from esseff.org ([83.161.120.155]) by localhost (esseff.org [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id 3wY61Ob1jqxU for <johnwilliams@example.org>; Mon, 2 Apr 2007 14:52:15 -0700 (PDT) Received: from professorville.windows.esseff.org (professorville.windows.esseff.org [83.161.120.155]) by esseff.org (Postfix) with ESMTP id 8CDF4647E27 for <johnwilliams@example.org>; Mon, 2 Apr 2007 14:52:14 -0700 (PDT) Subject: RE: [jira] Updated: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-class: urn:content-classes:message Date: Mon, 2 Apr 2007 14:50:44 -0700 X-MIMEOLE: Produced By Microsoft Exchange V6.5 Message-ID: <johnwilliams@example.org> In-Reply-To: <johnwilliams@example.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [jira] Updated: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff Thread-Index: Acd1YcBTirDgdF8PQsaxRix/Nq4MowACDDSA References: <24021458.1175101765409.JavaMail.jira@brutus> <johnwilliams@example.org> <johnwilliams@example.org> <johnwilliams@example.org> From: "Steven Parkes" <johnwilliams@example.org> To: <johnwilliams@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Yes, indeed. May not be necessary initially, but we could support =20 XPath or something down the road to allow us to specify what things =20 > I wouldn't worry about generalizing too much =20 > to start with. Once we have a couple collections then we can go that > route. My thoughts, too. I've been looking at the Reuters stuff. It uncompressed the distribution and then creates per-article files. I can't decide if I think that's a good idea for Wikipedia. It's big (about 10G uncompressed) and has about 1.2M files (so I've heard; unverified). On the one hand, creating separate per-article files is "clean" in that when you then ingest, you only have disk i/o that's going to affect the ingest performance (as opposed to, say, uncompressing/parsing). On the other hand, that's a lot of disk i/o (compresses by about 5X) and a lot of directory lookups. Anybody have any opinions/relevant past experience? --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 21:54:02 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 52880 invoked from network); 2 Apr 2007 21:53:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 21:53:58 -0000 Received: (qmail 15072 invoked by uid 500); 2 Apr 2007 21:54:00 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 15002 invoked by uid 500); 2 Apr 2007 21:54:00 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 14969 invoked by uid 99); 2 Apr 2007 21:54:00 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 14:54:00 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 14:53:52 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 8755E714066 for <johnwilliams@example.org>; Mon, 2 Apr 2007 14:53:32 -0700 (PDT) Message-ID: <18716273.1175550812550.JavaMail.jira@brutus> Date: Mon, 2 Apr 2007 14:53:32 -0700 (PDT) From: "Andy Liu (JIRA)" <johnwilliams@example.org> To: johnwilliams@example.org Subject: [jira] Created: (LUCENE-855) MemoryCachedRangeFilter to boost performance of Range queries MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org MemoryCachedRangeFilter to boost performance of Range queries ------------------------------------------------------------- Key: LUCENE-855 URL: https://issues.apache.org/jira/browse/LUCENE-855 Project: Lucene - Java Issue Type: Improvement Components: Search Affects Versions: 2.1 Reporter: Andy Liu Currently RangeFilter uses TermEnum and TermDocs to find documents that fall within the specified range. This requires iterating through every single term in the index and can get rather slow for large document sets. MemoryCachedRangeFilter reads all <docId, value> pairs of a given field, sorts by value, and stores in a SortedFieldCache. During bits(), binary searches are used to find the start and end indices of the lower and upper bound values. The BitSet is populated by all the docId values that fall in between the start and end indices. TestMemoryCachedRangeFilterPerformance creates a 100K RAMDirectory-backed index with random date values within a 5 year range. Executing bits() 1000 times on standard RangeQuery using random date intervals took 63904ms. Using MemoryCachedRangeFilter, it took 876ms. Performance increase is less dramatic when you have less unique terms in a field or using less number of documents. Currently MemoryCachedRangeFilter only works with numeric values (values are stored in a long[] array) but it can be easily changed to support Strings. A side "benefit" of storing the values are stored as longs, is that there's no longer the need to make the values lexographically comparable, i.e. padding numeric values with zeros. The downside of using MemoryCachedRangeFilter is there's a fairly significant memory requirement. So it's designed to be used in situations where range filter performance is critical and memory consumption is not an issue. The memory requirements are: (sizeof(int) + sizeof(long)) * numDocs. MemoryCachedRangeFilter also requires a warmup step which can take a while to run in large datasets (it took 40s to run on a 3M document corpus). Warmup can be called explicitly or is automatically called the first time MemoryCachedRangeFilter is applied using a given field. So in summery, MemoryCachedRangeFilter can be useful when: - Performance is critical - Memory is not an issue - Field contains many unique numeric values - Index contains large amount of documents -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 21:55:58 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 55794 invoked from network); 2 Apr 2007 21:55:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 21:55:56 -0000 Received: (qmail 24800 invoked by uid 500); 2 Apr 2007 21:56:00 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 24753 invoked by uid 500); 2 Apr 2007 21:56:00 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 24738 invoked by uid 99); 2 Apr 2007 21:56:00 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 14:56:00 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 14:55:52 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 661A8714049 for <johnwilliams@example.org>; Mon, 2 Apr 2007 14:55:32 -0700 (PDT) Message-ID: <18276217.1175550932415.JavaMail.jira@brutus> Date: Mon, 2 Apr 2007 14:55:32 -0700 (PDT) From: "Andy Liu (JIRA)" <johnwilliams@example.org> To: johnwilliams@example.org Subject: [jira] Updated: (LUCENE-855) MemoryCachedRangeFilter to boost performance of Range queries In-Reply-To: <18716273.1175550812550.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Liu updated LUCENE-855: ---------------------------- Attachment: MemoryCachedRangeFilter.patch Patch produced from latest from SVN > MemoryCachedRangeFilter to boost performance of Range queries > ------------------------------------------------------------- > > Key: LUCENE-855 > URL: https://issues.apache.org/jira/browse/LUCENE-855 > Project: Lucene - Java > Issue Type: Improvement > Components: Search > Affects Versions: 2.1 > Reporter: Andy Liu > Attachments: MemoryCachedRangeFilter.patch > > > Currently RangeFilter uses TermEnum and TermDocs to find documents that fall within the specified range. This requires iterating through every single term in the index and can get rather slow for large document sets. > MemoryCachedRangeFilter reads all <docId, value> pairs of a given field, sorts by value, and stores in a SortedFieldCache. During bits(), binary searches are used to find the start and end indices of the lower and upper bound values. The BitSet is populated by all the docId values that fall in between the start and end indices. > TestMemoryCachedRangeFilterPerformance creates a 100K RAMDirectory-backed index with random date values within a 5 year range. Executing bits() 1000 times on standard RangeQuery using random date intervals took 63904ms. Using MemoryCachedRangeFilter, it took 876ms. Performance increase is less dramatic when you have less unique terms in a field or using less number of documents. > Currently MemoryCachedRangeFilter only works with numeric values (values are stored in a long[] array) but it can be easily changed to support Strings. A side "benefit" of storing the values are stored as longs, is that there's no longer the need to make the values lexographically comparable, i.e. padding numeric values with zeros. > The downside of using MemoryCachedRangeFilter is there's a fairly significant memory requirement. So it's designed to be used in situations where range filter performance is critical and memory consumption is not an issue. The memory requirements are: (sizeof(int) + sizeof(long)) * numDocs. > MemoryCachedRangeFilter also requires a warmup step which can take a while to run in large datasets (it took 40s to run on a 3M document corpus). Warmup can be called explicitly or is automatically called the first time MemoryCachedRangeFilter is applied using a given field. > So in summery, MemoryCachedRangeFilter can be useful when: > - Performance is critical > - Memory is not an issue > - Field contains many unique numeric values > - Index contains large amount of documents -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 21:59:39 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 57992 invoked from network); 2 Apr 2007 21:59:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 21:59:37 -0000 Received: (qmail 31369 invoked by uid 500); 2 Apr 2007 21:59:43 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 30715 invoked by uid 500); 2 Apr 2007 21:59:42 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 30704 invoked by uid 99); 2 Apr 2007 21:59:42 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 14:59:42 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO rectangular.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 14:59:34 -0700 Received: from [83.161.120.155] (helo=[83.161.120.155]) by rectangular.com with esmtpa (Exim 4.44) id 1HYUgj-000JEt-BK for johnwilliams@example.org; Mon, 02 Apr 2007 15:07:45 -0700 Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <johnwilliams@example.org> References: <24021458.1175101765409.JavaMail.jira@brutus> <johnwilliams@example.org> <johnwilliams@example.org> <johnwilliams@example.org> <johnwilliams@example.org> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <johnwilliams@example.org> Content-Transfer-Encoding: 7bit From: Marvin Humphrey <johnwilliams@example.org> Subject: Re: [jira] Updated: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff Date: Mon, 2 Apr 2007 14:59:09 -0700 To: johnwilliams@example.org X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org On Apr 2, 2007, at 2:50 PM, Steven Parkes wrote: > On the one hand, creating separate per-article files is "clean" in > that > when you then ingest, you only have disk i/o that's going to affect > the > ingest performance (as opposed to, say, uncompressing/parsing). On the > other hand, that's a lot of disk i/o (compresses by about 5X) and a > lot > of directory lookups. One reason I was expanding the elements into individual files was so that I could compare different libraries against Lucene, including those in other languages. It was important to measure the engines themselves, not SGML parsers. Marvin Humphrey Rectangular Research http://www.rectangular.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 22:47:06 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 77702 invoked from network); 2 Apr 2007 22:47:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 22:47:05 -0000 Received: (qmail 5341 invoked by uid 500); 2 Apr 2007 22:47:10 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 5301 invoked by uid 500); 2 Apr 2007 22:47:10 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 5290 invoked by uid 99); 2 Apr 2007 22:47:09 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 15:47:09 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO web50309.mail.re2.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 02 Apr 2007 15:47:02 -0700 Received: (qmail 21160 invoked by uid 60001); 2 Apr 2007 22:46:40 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=GqE60JrhBg8jXZZENu+ujHdtg8Gg5liqAVjUkj81s2fUPbHTgveYOv96z4FjpryQRGaAnTEvFSwKSjF1vUabzZrT+KyVPWWolzMs2isWbD237R6hB1DIS4O4HWKQN+Zb+OvMuHmEMvE4N0D6D+WjGHwVH92Ncf2A2tvm7x+ZH8I=; X-YMail-OSG: LW5V0X4VM1mXBzbuBGMu8tBifx3DoT7b0trcDEnVN3wo9kt3hplkAPk5tnk6HiG92LhoR_YlgduxmaqAVSfMets7Jg-- Received: from [83.161.120.155] by web50309.mail.re2.yahoo.com via HTTP; Mon, 02 Apr 2007 15:46:40 PDT X-Mailer: YahooMailRC/476 YahooMailWebService/83.161.120.155 Date: Mon, 2 Apr 2007 15:46:40 -0700 (PDT) From: Otis Gospodnetic <johnwilliams@example.org> Subject: contrib/benchmark - DOS line endings To: johnwilliams@example.org MIME-Version: 1.0 Content-Type: text/plain; charset=ascii Message-ID: <johnwilliams@example.org> X-Virus-Checked: Checked by ClamAV on apache.org I noticed some contrib/benchmark files have DOS-style line endings. Tried this, but failed: $ svn propset svn:eol-style native ./src/java/org/apache/lucene/benchmark/byTask/tasks/SearchTravTask.java subversion/libsvn_wc/props.c:1460: (apr_err=200009) svn: File 'src/java/org/apache/lucene/benchmark/byTask/tasks/SearchTravTask.java' has inconsistent newlines subversion/libsvn_subr/subst.c:643: (apr_err=135000) svn: Inconsistent line ending style If anyone knows a SVN way of fixing this, I'm all eyes! Otis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simpy -- http://www.simpy.com/ - Tag - Search - Share --------------------------------------------------------------------- To unsubscribe, e-mail: johnwilliams@example.org For additional commands, e-mail: johnwilliams@example.org From java-devjohnwilliams@example.org Mon Apr 02 23:15:30 2007 Return-Path: <java-devjohnwilliams@example.org> Delivered-To: johnwilliams@example.org Received: (qmail 84718 invoked from network); 2 Apr 2007 23:15:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Apr 2007 23:15:28 -0000 Received: (qmail 39121 invoked by uid 500); 2 Apr 2007 23:15:35 -0000 Delivered-To: johnwilliams@example.org Received: (qmail 38466 invoked by uid 500); 2 Apr 2007 23:15:33 -0000 Mailing-List: contact johnwilliams@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:johnwilliams@example.org> List-Unsubscribe: <mailto:johnwilliams@example.org> List-Post: <mailto:johnwilliams@example.org> List-Id: <java-dev.lucene.apache.org> Reply-To: johnwilliams@example.org Delivered-To: mailing list johnwilliams@example.org Received: (qmail 38455 invoked by uid 99); 2 Apr 2007 23:15:33 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 16:15:33 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO esseff.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Apr 2007 16:15:24 -0700 Received: from localhost (localhost [83.161.120.155]) by esseff.org (Postfix) with ESMTP id B43BB648066 for <johnwilliams@example.org>; Mon, 2 Apr 2007 16:16:33 -0700 (PDT) X-Quarantine-ID: <beqOzLzF-h-1> X-Virus-Scanned: johnwilliams@example.org Received: from esseff.org ([83.161.120.155]) by localhost (esseff.org [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id beqOzLzF-h-1 for <johnwilliams@example.org>; Mon, 2 Apr 2007 16:16:30 -0700 (PDT) Received: from professorville.windows.esseff.org (professorville.windows.esseff.org [83.161.120.155]) by esseff.org (Postfix) with ESMTP id 410B934A8E for <johnwilliams@example.org>; Mon, 2 Apr 2007 16:16:30 -0700 (PDT) Subject:
{'source': 'Apache Software Foundation Public Mail Archives'}