From java-devdtorres@example.com Sun Jul 01 08:53:58 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 25694 invoked from network); 1 Jul 2007 08:53:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jul 2007 08:53:56 -0000 Received: (qmail 23508 invoked by uid 500); 1 Jul 2007 08:53:56 -0000 Delivered-To: dtorres@example.com Received: (qmail 23456 invoked by uid 500); 1 Jul 2007 08:53:56 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 23445 invoked by uid 99); 1 Jul 2007 08:53:56 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jul 2007 01:53:56 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of dtorres@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mtagate8.de.ibm.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jul 2007 01:53:51 -0700 Received: from d12nrmr1607.megacenter.de.ibm.com (d12nrmr1607.megacenter.de.ibm.com [83.161.120.155]) by mtagate8.de.ibm.com (8.13.8/8.13.8) with ESMTP id l618rThW129744 for <dtorres@example.com>; Sun, 1 Jul 2007 08:53:29 GMT Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com [83.161.120.155]) by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id l618rTfq2121784 for <dtorres@example.com>; Sun, 1 Jul 2007 10:53:29 +0200 Received: from d12av02.megacenter.de.ibm.com (loopback [83.161.120.155]) by d12av02.megacenter.de.ibm.com (83.161.120.15560308/8.13.3) with ESMTP id l618rTl4020456 for <dtorres@example.com>; Sun, 1 Jul 2007 10:53:29 +0200 Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com [83.161.120.155]) by d12av02.megacenter.de.ibm.com (83.161.120.15560308/8.12.11) with ESMTP id l618rTAL020453 for <dtorres@example.com>; Sun, 1 Jul 2007 10:53:29 +0200 In-Reply-To: <dtorres@example.com> Subject: Re: search quality - assessment & improvements To: dtorres@example.com X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006 Message-ID: <dtorres@example.com> From: Doron Cohen <dtorres@example.com> Date: Sun, 1 Jul 2007 01:49:45 -0700 X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 | November 3, 2006) at 01/07/2007 11:53:28 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org Thanks for your comments Chris, and sorry for the delayed response - you raised some tough questions for me, and I felt I have to clear my thoughts on this before replying. (Well, as you'll see below they are not too clear now either, but I am going to be off-line for the next ~10 days, so decided to not wait more with this...) Chris Hostetter wrote: > for the record: SweetSpotSimilarity was designed to let you > use the average length as the "sweet spot" (both min and max) > but as you say: you have to know/guess the average length in > advance (and explicitly set it using > SweetSpotSimilarity.setLengthNormFactors(yourAvg,yourAvg, > steepness)) I didn't try this - passing the computed avg doc length to SweetSpotSimilarity (SSS) - it would be interesting to try. I wonder how this would perform comparing to the variation of pivoted (unique) length normalization that I tried. The difference is that SSS punishes docs above and below the range, while with pivoted normalization docs above the pivot are punished and those below the pivot boosted. Pivoted normalization makes more sense to me than SSS. Assuming this proves to be useful in Lucene as a general improvement (collection independent, largely) - question is how to compute/store/retrieve this data. The way I experimented with it was not focused on efficiency but rather on flexibility at search time, my custom analyzer counted the number of unique tokens in the document, and finally a field was added to the document with this number. At search time this field was loaded (for all docs), the average was computed (the pivot), and both the pivot and the unique length were used for normalization with some slop. One way to do something like this more efficiently would be to store the average (unique) length in the segment, and to update this value when segments are merged (need to think about doc deletions). This works nice with the single doc in a segment, not sure how it would work with the new code in LUCENE-843. This perhaps goes too far too early with implementation details, and I wonder if flexible index format can satisfy/represent adding this type of data to the index. > if i understand your argument, the pure "IR'ish" model makes > no difference between finding input in differnet fields, so > it should use a single document length -- but that assumes > you are searching for the input in all fields right? if you > have a title field and a body field but you are only searching > for "title:word" then why should the length of hte body fieild > factor into the score at all? > > my basic understanding of IRish relevancy judgements of the > idf/tf model is that "documents" are generic lumps of text .. > while Lucene has a "Document" class that supports multipel > Fields, the most direct mapping of IR "documents" and > Lucene objects would be a Document with onlyone Field > ... but that doesn't mean that if a Document has more > then one field that the length of all fields should be a > facor in searching for a "field1:word" (any more then the > tf used should be the frequency of "word" in all fields) These are good arguments. I am not 100% sure here. My use case was of boosting certain fields - say you have just {title} and {body} and want to make title words worth 3 times more. A natural way to do this is to have two fields "body" and "title", set their boosts 1 for "body" and 3 for "title", and then, when one searches the entire document (without specifying a field), create a multi field query. Things should work fine, - boosts are ok, tf() is by field, so is norm. But empirically it doesn't work well. When I modified my index to have a single field in which I just multiplied the title 3 times, I got better results. There are questions - is there a pure mathematical explanation, that the sum of the scores/statistics by separate fields yielded poorer quality than the single score/statistics of a single field containing all the text? Perhaps the loss of information in the norms was more damaging with more (smaller) fields? (I had four btw) - I don't know. I just saw that a single field is better (when this is possible), and I went on with it. Now, the new payloads allow to specify boost at token level. So it is possible (although not yet very easy) to set that for the title words, and not separate to fields just for boosts. This would remove the original motivation I started with - because in my case there really was no need to separate to fields except for boosts. Looking at the general case - multi-field Lucene document and a user query on only part of those fields. Currently Lucene does it separately for each field. When I first noticed this behavior in Lucene, (which was different than what I was used to), I was surprised. Then I thought that this is really the right thing - assume docs with two fields: "friends" and "enemies" - what sense is there to consider statistics of "friends" when searching in "enemies"? (I am taking your side here). This is a weird example, having two fields that negate each other, but it makes the question clearer. So, for query "Mr. Jones and Mrs. James" in "friends" field, a document would be considered only for matches in the "friends" field. If both doc1 and doc2 contain "Mr. Jones", in "friends", and their length of "friends" is equal, but doc2's "enemies" field is longer, does it make sense to punish doc2? I think that for this example, if anything, doc2 should have been boosted, because it has more enemies, but even so, one of the friends is "Mr. Jones". So theoretically I agree with you. In reality, I don't know if we often get to see examples as that last one. And, it would be expensive to maintain the (accurate) information needed for such pivot normalization for each field of each document. But maintaining just once for the entire document may be possible... Mmm... adding this to segment's field-info we can do it per field to... should probably be optional. (If the above reads uncertain and somewhat confused it is because currently this is how it is..) > Note: for the record, i think it would be nice if the Score > computation "engine" of Lucene allowed for more hooks to let > people controlstuff like this, so that it is possible in use > cases where it makes sense -- i'm just not convinced it makes > sense as the general case. (ie: even if it were > possible, i don't think it would make sense as the default) Agree. > : a nice problem is that current Similarity API does not > : even allow to consider this info (the average term frequency > : in the specific document) because Similarity.tf(int/float freq) > : takes only the frequency param. One way to open way for such > : computation is to add an "int docid" param to the > : Similarity class, but then the implementation of that class > : becomes IndexReader aware. > this strikes me as as not being a deficiency in the existing > Query/Scorer/Similaity APIs as much as perhaps an opportunity for new > Query types (with new Scorer types) that take advantage of the Payloads > API (i say this having almost 0 knowledge of what the payloads API looks > like) to take advantage of more term, doc, and index level payloads. Any > of this type of logic could be put into new Query subclasses, the trick > becomes allowing the common match of those new Query classes to be > refactored out and controled by classes at runtime -- this is really what > the current Similarity class does, provide a refactoring of common types > of calculations in such a way that those calculations can be replaced at > run time (at least: that's how i view the Similarity class) > > i'm just typing out loud here without thinkng it through, but > one appraoch > we could take to allow Similaity to be extended in special uses cases, > would be if each new Query's getSimilarity() method tested the similarity > from it's parent to see if it implemented an interface indicating that it > contained the method(s) this query needed to perform it's custom > calculations ... if it does: hurray! ... if not, then wrap it in the > "default" version of that marker interface. > > ie... > > public class DocRelativeTfTermQuery extends TermQuery { > ... > public Similarity getSimilarity(Searcher searcher) { > Similarity s = super.getSimilarity(Searcher searcher); > if (s instanceOf DocRelativeTfSimilaity) return s; > return AverageTfSimilaityWrapper(s); > } > public static interface DocRelativeTfSimilaity { > public float tf(int doc, float freq); > } > public static class AverageTfSimilaityWrapper > extends SimilarityDelegator > implements DocRelativeTfSimilaity { > public DocRelativeTfSimilarityWrapper(Similarity s) { super(s) } > public float tf(int doc, float freq) { > return tf(freq - averageFreq(doc)); > } > protected averageFreq(int doc) { > ...get from document payload ... > } > } > > > ...so now by default the DocRelativeTfTermQuery computes the score for > that term based on the average tf of all terms in that field ... but if > you construct a Similarity of your own that implements > DocRelativeTfSimilaity you can make it use any document specific > calculation you want. > > The performance of simple queries doesn't change, the api for > simple users > of Similaity doesnt' get more complicated, and we can start supporting > arbitrarily complex scoring calculations in Scorers and still allow the > "client" to supply the meat of those calculations atrun time. This is nice. I didn't think of it. It may be nice if instead of creating new types of queries the existing ones (Span, Boolean, Phrase, Wild) could be somehow "set to" use DocRelativeTfTermQuery instead of TermQuery. ? > -Hoss Thanks again Hoss for the detailed comments and great ideas. Doron --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Sun Jul 01 15:06:10 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 44580 invoked from network); 1 Jul 2007 15:06:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jul 2007 15:06:09 -0000 Received: (qmail 80882 invoked by uid 500); 1 Jul 2007 15:06:11 -0000 Delivered-To: dtorres@example.com Received: (qmail 80240 invoked by uid 500); 1 Jul 2007 15:06:10 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 80225 invoked by uid 99); 1 Jul 2007 15:06:10 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jul 2007 08:06:10 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO spunkymail-a16.dreamhost.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jul 2007 08:06:06 -0700 Received: from [83.161.120.155] (adsl-074-229-189-244.sip.rmo.bellsouth.net [83.161.120.155]) by spunkymail-a16.dreamhost.com (Postfix) with ESMTP id 850467CD23 for <dtorres@example.com>; Sun, 1 Jul 2007 08:05:45 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <dtorres@example.com> References: <dtorres@example.com> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <dtorres@example.com> Content-Transfer-Encoding: 7bit From: Grant Ingersoll <dtorres@example.com> Subject: Re: [jira] Created: (LUCENE-945) contrib/benchmark tests fail find data dirs Date: Sun, 1 Jul 2007 11:05:43 -0400 To: dtorres@example.com X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org I think legally we are fine, since we aren't actually shipping it. I just mean that people may not want to wait however long it takes to download it. Of course, I don't know a work around other than to have some smaller set. On Jun 30, 2007, at 10:10 AM, Doron Cohen wrote: > Grant Ingersoll <dtorres@example.com> wrote on 30/06/2007 05:20:34: > >> Does this imply it is going to download the test collection for >> people when they don't have it when running tests? I don't know if >> that is something people are going to want to happen. > > Yes it does.. and so would auto-build-bots - I am also using > the Reuters collection for TestQualityRun in LUCENE-836. Do > you mean legal wise? I should probably add a "DOWNLOAD" > warning here. Is there another issue with this? > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dtorres@example.com > For additional commands, e-mail: dtorres@example.com > --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 00:41:30 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 91177 invoked from network); 2 Jul 2007 00:41:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 00:41:29 -0000 Received: (qmail 19963 invoked by uid 500); 2 Jul 2007 00:41:30 -0000 Delivered-To: dtorres@example.com Received: (qmail 19515 invoked by uid 500); 2 Jul 2007 00:41:29 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 19499 invoked by uid 99); 2 Jul 2007 00:41:29 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jul 2007 17:41:29 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jul 2007 17:41:25 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id DFD367141EA for <dtorres@example.com>; Sun, 1 Jul 2007 17:41:04 -0700 (PDT) Message-ID: <26873213.1183336864860.JavaMail.jira@brutus> Date: Sun, 1 Jul 2007 17:41:04 -0700 (PDT) From: "Michael McCandless (JIRA)" <dtorres@example.com> To: dtorres@example.com Subject: [jira] Commented: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff In-Reply-To: <5130574.1174845812186.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12509473 ] Michael McCandless commented on LUCENE-848: ------------------------------------------- I think somehow the wrong version (1.4.4) of Xerces was committed and named as lib/xerces-2.9.0.jar. I'm hitting what I think is the same issue Steven is referring to above (https://issues.apache.org/jira/browse/LUCENE-848#action_1248988), this exception after ~350K docs: Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Internal Error: fPreviousChunk == dtorres@example.com(ExtractWikipedia.java:184) at org.apache.lucene.benchmark.utils.ExtractWikipedia.main(ExtractWikipedia.java:199) Caused by: java.lang.RuntimeException: Internal Error: fPreviousChunk == dtorres@example.com(XMLParser.java:1111) at org.apache.lucene.benchmark.utils.ExtractWikipedia.extract(ExtractWikipedia.java:181) ... 1 more I downloaded 1.4.4 of xerces.jar and "cmp" says it's the same file that's now checked in as lib/xerces-2.9.0.jar. When I download xerces 2.9.0 myself and overwrite this one in lib, the extraction finishes without errors. > Add supported for Wikipedia English as a corpus in the benchmarker stuff > ------------------------------------------------------------------------ > > Key: LUCENE-848 > URL: https://issues.apache.org/jira/browse/LUCENE-848 > Project: Lucene - Java > Issue Type: New Feature > Components: contrib/benchmark > Reporter: Steven Parkes > Assignee: Grant Ingersoll > Priority: Minor > Attachments: LUCENE-848-build.patch, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, WikipediaHarvester.java, xerces.jar, xerces.jar, xml-apis.jar > > > Add support for using Wikipedia for benchmarking. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 01:00:34 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 94942 invoked from network); 2 Jul 2007 01:00:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 01:00:33 -0000 Received: (qmail 36812 invoked by uid 500); 2 Jul 2007 01:00:34 -0000 Delivered-To: dtorres@example.com Received: (qmail 36786 invoked by uid 500); 2 Jul 2007 01:00:33 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 36775 invoked by uid 99); 2 Jul 2007 01:00:33 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jul 2007 18:00:33 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Jul 2007 18:00:30 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id A99F57141EA for <dtorres@example.com>; Sun, 1 Jul 2007 18:00:07 -0700 (PDT) Message-ID: <11886279.1183338007689.JavaMail.jira@brutus> Date: Sun, 1 Jul 2007 18:00:07 -0700 (PDT) From: "Grant Ingersoll (JIRA)" <dtorres@example.com> To: dtorres@example.com Subject: [jira] Commented: (LUCENE-848) Add supported for Wikipedia English as a corpus in the benchmarker stuff In-Reply-To: <5130574.1174845812186.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12509474 ] Grant Ingersoll commented on LUCENE-848: ---------------------------------------- OK, go ahead and commit it. -------------------------- Grant Ingersoll Center for Natural Language Processing http://www.cnlp.org/tech/lucene.asp Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ > Add supported for Wikipedia English as a corpus in the benchmarker stuff > ------------------------------------------------------------------------ > > Key: LUCENE-848 > URL: https://issues.apache.org/jira/browse/LUCENE-848 > Project: Lucene - Java > Issue Type: New Feature > Components: contrib/benchmark > Reporter: Steven Parkes > Assignee: Grant Ingersoll > Priority: Minor > Attachments: LUCENE-848-build.patch, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, LUCENE-848.txt, WikipediaHarvester.java, xerces.jar, xerces.jar, xml-apis.jar > > > Add support for using Wikipedia for benchmarking. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 13:35:40 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 11269 invoked from network); 2 Jul 2007 13:35:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 13:35:37 -0000 Received: (qmail 6623 invoked by uid 500); 2 Jul 2007 13:35:37 -0000 Delivered-To: dtorres@example.com Received: (qmail 6582 invoked by uid 500); 2 Jul 2007 13:35:36 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 6571 invoked by uid 99); 2 Jul 2007 13:35:36 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 06:35:36 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO out2.smtp.messagingengine.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 06:35:32 -0700 Received: from compute2.internal (compute2.internal [83.161.120.155]) by out1.messagingengine.com (Postfix) with ESMTP id 223C254FB for <dtorres@example.com>; Mon, 2 Jul 2007 09:35:12 -0400 (EDT) Received: from web8.messagingengine.com ([83.161.120.155]) by compute2.internal (MEProxy); Mon, 02 Jul 2007 09:35:12 -0400 Received: by web8.messagingengine.com (Postfix, from userid 99) id 064A41E5CB; Mon, 2 Jul 2007 09:35:12 -0400 (EDT) Message-Id: <dtorres@example.com> X-Sasl-Enc: yEfwlmQO5K75syQOTjrw3K8LGCV/8WyiFzxh3H9zXpin 1183383311 From: "Michael McCandless" <dtorres@example.com> To: dtorres@example.com Content-Disposition: inline Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="ISO-8859-1" MIME-Version: 1.0 X-Mailer: MessagingEngine.com Webmail Interface Subject: [VOTE] Commit LUCENE-843 (IndexWriter performance gains) Date: Mon, 02 Jul 2007 09:35:11 -0400 X-Virus-Checked: Checked by ClamAV on apache.org Hi, I'd like to commit LUCENE-843. The patch has gone through a number of iterations but the final version that's there now (take9) is quite a bit cleaner & simpler than the ones leading up to it and I believe ready. It provides solid indexing performance gains (between 2X-8X), but, it is somewhat more complex than the current "single doc per segment" approach and it does introduce a change to the index format (only when autoCommit=false) whereby multiple segments can share a single set of term vector & stored fields files. Given that it's such a big change I think (?) it's appropriate to ask for a vote (only PMC member votes are binding) to make sure we have consensus that this is net/net a good change for Lucene. Mike --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 13:42:30 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 13104 invoked from network); 2 Jul 2007 13:42:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 13:42:28 -0000 Received: (qmail 18474 invoked by uid 500); 2 Jul 2007 13:42:28 -0000 Delivered-To: dtorres@example.com Received: (qmail 18424 invoked by uid 500); 2 Jul 2007 13:42:28 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 18413 invoked by uid 99); 2 Jul 2007 13:42:28 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 06:42:28 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 06:42:24 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 859CB7140D2 for <dtorres@example.com>; Mon, 2 Jul 2007 06:42:04 -0700 (PDT) Message-ID: <5573700.1183383724539.JavaMail.jira@brutus> Date: Mon, 2 Jul 2007 06:42:04 -0700 (PDT) From: "Michael McCandless (JIRA)" <dtorres@example.com> To: dtorres@example.com Subject: [jira] Created: (LUCENE-947) Some improvements to contrib/benchmark MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Some improvements to contrib/benchmark -------------------------------------- Key: LUCENE-947 URL: https://issues.apache.org/jira/browse/LUCENE-947 Project: Lucene - Java Issue Type: Improvement Components: contrib/benchmark Reporter: Michael McCandless Assignee: Michael McCandless Priority: Minor I've made some small improvements to the contrib/benchmark, mostly merging in the ad-hoc benchmarking code I've been using in LUCENE-843: - Fixed thread safety of DirDocMaker's usage of SimpleDateFormat - Print the props in sorted order - Added new config "autocommit=true|false" to CreateIndexTask - Added new config "ram.flush.mb=int" to AddDocTask - Added new configs "doc.term.vector.positions=true|false" and "doc.term.vector.offsets=true|false" to BasicDocMaker - Added WriteLineDocTask.java, so you can make an alg that uses this to build up a single file containing one document per line in a single file. EG this alg converts the reuters-out tree into a single file that has ~1000 bytes per body field, saved to work/reuters.1000.txt: docs.dir=reuters-out doc.maker=org.apache.lucene.benchmark.byTask.feeds.DirDocMaker line.file.out=work/reuters.1000.txt doc.maker.forever=false {WriteLineDoc(1000)}: * Each line has tab-separted TITLE, DATE, BODY fields. - Created feeds/LineDocMaker.java that creates documents read from the file created by WriteLineDocTask.java. EG this alg indexes all documents created above: analyzer=org.apache.lucene.analysis.SimpleAnalyzer directory=FSDirectory doc.add.log.step=500 docs.file=work/reuters.1000.txt doc.maker=org.apache.lucene.benchmark.byTask.feeds.LineDocMaker doc.tokenized=true doc.maker.forever=false ResetSystemErase CreateIndex {AddDoc}: * CloseIndex RepSumByPref AddDoc I'll attach initial patch shortly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 13:44:34 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 13610 invoked from network); 2 Jul 2007 13:44:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 13:44:32 -0000 Received: (qmail 20621 invoked by uid 500); 2 Jul 2007 13:44:28 -0000 Delivered-To: dtorres@example.com Received: (qmail 20578 invoked by uid 500); 2 Jul 2007 13:44:28 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 20564 invoked by uid 99); 2 Jul 2007 13:44:28 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 06:44:28 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 06:44:25 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id EE7A07141EA for <dtorres@example.com>; Mon, 2 Jul 2007 06:44:04 -0700 (PDT) Message-ID: <25168612.1183383844965.JavaMail.jira@brutus> Date: Mon, 2 Jul 2007 06:44:04 -0700 (PDT) From: "Michael McCandless (JIRA)" <dtorres@example.com> To: dtorres@example.com Subject: [jira] Updated: (LUCENE-947) Some improvements to contrib/benchmark In-Reply-To: <5573700.1183383724539.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-947: -------------------------------------- Attachment: LUCENE-947.patch First cut patch. > Some improvements to contrib/benchmark > -------------------------------------- > > Key: LUCENE-947 > URL: https://issues.apache.org/jira/browse/LUCENE-947 > Project: Lucene - Java > Issue Type: Improvement > Components: contrib/benchmark > Reporter: Michael McCandless > Assignee: Michael McCandless > Priority: Minor > Attachments: LUCENE-947.patch > > > I've made some small improvements to the contrib/benchmark, mostly > merging in the ad-hoc benchmarking code I've been using in LUCENE-843: > - Fixed thread safety of DirDocMaker's usage of SimpleDateFormat > - Print the props in sorted order > - Added new config "autocommit=true|false" to CreateIndexTask > - Added new config "ram.flush.mb=int" to AddDocTask > - Added new configs "doc.term.vector.positions=true|false" and > "doc.term.vector.offsets=true|false" to BasicDocMaker > - Added WriteLineDocTask.java, so you can make an alg that uses this > to build up a single file containing one document per line in a > single file. EG this alg converts the reuters-out tree into a > single file that has ~1000 bytes per body field, saved to > work/reuters.1000.txt: > docs.dir=reuters-out > doc.maker=org.apache.lucene.benchmark.byTask.feeds.DirDocMaker > line.file.out=work/reuters.1000.txt > doc.maker.forever=false > {WriteLineDoc(1000)}: * > Each line has tab-separted TITLE, DATE, BODY fields. > - Created feeds/LineDocMaker.java that creates documents read from > the file created by WriteLineDocTask.java. EG this alg indexes > all documents created above: > analyzer=org.apache.lucene.analysis.SimpleAnalyzer > directory=FSDirectory > doc.add.log.step=500 > docs.file=work/reuters.1000.txt > doc.maker=org.apache.lucene.benchmark.byTask.feeds.LineDocMaker > doc.tokenized=true > doc.maker.forever=false > ResetSystemErase > CreateIndex > {AddDoc}: * > CloseIndex > RepSumByPref AddDoc > I'll attach initial patch shortly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 14:16:51 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 25909 invoked from network); 2 Jul 2007 14:16:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 14:16:30 -0000 Received: (qmail 85198 invoked by uid 500); 2 Jul 2007 14:16:29 -0000 Delivered-To: dtorres@example.com Received: (qmail 85159 invoked by uid 500); 2 Jul 2007 14:16:29 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 85136 invoked by uid 99); 2 Jul 2007 14:16:29 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 07:16:29 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 07:16:25 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 45C007141EB for <dtorres@example.com>; Mon, 2 Jul 2007 07:16:05 -0700 (PDT) Message-ID: <16535440.1183385765282.JavaMail.jira@brutus> Date: Mon, 2 Jul 2007 07:16:05 -0700 (PDT) From: "Michael McCandless (JIRA)" <dtorres@example.com> To: dtorres@example.com Subject: [jira] Commented: (LUCENE-856) Optimize segment merging In-Reply-To: <29717292.1175722292226.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12509576 ] Michael McCandless commented on LUCENE-856: ------------------------------------------- I ran a new performance comparison here to test the merging cost of autoCommit false vs true, this time using Wikipedia content and contrib/benchmark. I indexed all of Wikipedia using the patch from LUCENE-843 and the patch from LUCENE-947, once with autoCommit=true and once with autoCommit=false. I used this alg (and just changed autocommit=true to false for the second test): max.field.length=+1-321-281-6863 compound=false analyzer=org.apache.lucene.analysis.SimpleAnalyzer directory=FSDirectory ram.flush.mb=32 max.buffered=20000 autocommit=true doc.stored=true doc.tokenized=true doc.term.vector=true doc.term.vector.offsets=true doc.term.vector.positions=true doc.add.log.step=500 docs.dir=enwiki doc.maker=org.apache.lucene.benchmark.byTask.feeds.DirDocMaker doc.maker.forever=false ResetSystemErase CreateIndex [{AddDoc}: *] : 4 CloseIndex RepSumByPref AddDoc Which means: use 4 threads to index all text from each of the 3.2 million wikipedia docs, with stored fields & term vectors turned on, using SimpleAnalyzer, flushing when RAM usage hits 32 MB. The index size is 20 GB. Report from autoCommit=true: ------------> Report Sum By Prefix (AddDoc) (1 about 3204066 out of 3204073) Operation round runCnt recsPerRun rec/s elapsedSec avgUsedMem avgTotalMem AddDoc 0 3204066 1 226.3 14,159.22 282,843,296 373,480,960 Net elapsed time = 87 minutes 18 seconds Report from autoCommit=false: ------------> Report Sum By Prefix (AddDoc) (1 about 3204066 out of 3204073) Operation round runCnt recsPerRun rec/s elapsedSec avgUsedMem avgTotalMem AddDoc 0 3204066 1 407.6 7,860.63 252,046,000 329,962,048 Net elapsed time = 60 minutes 5 seconds Some comments: * According to net elapsed time, autoCommit=false is 31% faster than autoCommit=true. * According to "rec/s" it's actually 44% faster; this is because rec/s only measures the actual addDocument time and not eg the IO cost of retrieving the document contents. * The speedup is due entirely to the fact that the "doc stores" (vectors & stored fields) do not need to be merged when autoCommit=false. This is a major win because these files are enormous if you turn on stored fields & term vectors with offsets & positions. * The basic conclusion is the same as before: if you want to build up a large index, and, it's not necessary to be searching this index while you are building it, the fastest way to do so is with LUCENE-843 patch and with autoCommit=false. > Optimize segment merging > ------------------------ > > Key: LUCENE-856 > URL: https://issues.apache.org/jira/browse/LUCENE-856 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Affects Versions: 2.1 > Reporter: Michael McCandless > Assignee: Michael McCandless > Priority: Minor > > With LUCENE-843, the time spent indexing documents has been > substantially reduced and now the time spent merging is a sizable > portion of indexing time. > I ran a test using the patch for LUCENE-843, building an index of 10 > million docs, each with ~5,500 byte plain text, with term vectors > (positions + offsets) on and with 2 small stored fields per document. > RAM buffer size was 32 MB. I didn't optimize the index in the end, > though optimize speed would also improve if we optimize segment > merging. Index size is 86 GB. > Total time to build the index was 8 hrs 38 minutes, 5 hrs 40 minutes > of which was spent merging. That's 65.6% of the time! > Most of this time is presumably IO which probably can't be reduced > much unless we improve overall merge policy and experiment with values > for mergeFactor / buffer size. > These tests were run on a Mac Pro with 2 dual-core Intel CPUs. The IO > system is RAID 0 of 4 drives, so, these times are probably better than > the more common case of a single hard drive which would likely be > slower IO. > I think there are some simple things we could do to speed up merging: > * Experiment with buffer sizes -- maybe larger buffers for the > IndexInputs used during merging could help? Because at a default > mergeFactor of 10, the disk heads must do alot of seeking back and > forth between these 10 files (and then to the 11th file where we > are writing). > * Use byte copying when possible, eg if there are no deletions on a > segment we can almost (I think?) just copy things like prox > postings, stored fields, term vectors, instead of full parsing to > Jave objects and then re-serializing them. > * Experiment with mergeFactor / different merge policies. For > example I think LUCENE-854 would reduce time spend merging for a > given index size. > This is currently just a place to list ideas for optimizing segment > merges. I don't plan on working on this until after LUCENE-843. > Note that for "autoCommit=false", this optimization is somewhat less > important, depending on how often you actually close/open a new > IndexWriter. In the extreme case, if you open a writer, add 100 MM > docs, close the writer, then no segment merges dtorres@example.com. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 14:30:13 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 34355 invoked from network); 2 Jul 2007 14:30:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 14:30:10 -0000 Received: (qmail 9508 invoked by uid 500); 2 Jul 2007 14:30:10 -0000 Delivered-To: dtorres@example.com Received: (qmail 9465 invoked by uid 500); 2 Jul 2007 14:30:10 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 9454 invoked by uid 99); 2 Jul 2007 14:30:10 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 07:30:10 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of dtorres@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO nz-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 07:30:05 -0700 Received: by nz-out-0506.google.com with SMTP id i28so531195nzi for <dtorres@example.com>; Mon, 02 Jul 2007 07:29:44 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=XOSPC1mdUH+ZadOO6y8mqHUz9xnTXpjhhtzJ+fjw277/iUQA6L2omnsqzztjLkDJTM0MBs5fhRx5QcwD6F9oKL31S70d0Ncuzp+7dgxLRfa7SoUKY0vsEM/m80H1mvkWAYMPfBwySKITolbTh62CePv+oVWyEYKwLH83W6jOPRs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=h9Tqxrs3+bCMSEe/ejgj5N+1DbBsCeUKE8ap1iZefRFzO/aKUs3xDImsN/dVgT2cy0Jv9GVTnAvlCYLgl1xpiHo5eRnaysDGZKECUGrTmRF76dM1xOO+eV0HX9kQeynttftzlGCk+KnZRXjM0xl/wpDMVlXBSMtlG07CDrxMUXk= Received: by 83.161.120.155 with SMTP id q6mr359081wfj.1183386584185; Mon, 02 Jul 2007 07:29:44 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Mon, 2 Jul 2007 07:29:44 -0700 (PDT) Message-ID: <dtorres@example.com> Date: Mon, 2 Jul 2007 10:29:44 -0400 From: "Yonik Seeley" <dtorres@example.com> Sender: dtorres@example.com To: dtorres@example.com Subject: Re: [VOTE] Commit LUCENE-843 (IndexWriter performance gains) In-Reply-To: <dtorres@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <dtorres@example.com> X-Google-Sender-Auth: 7a7506231d9c3c48 X-Virus-Checked: Checked by ClamAV on apache.org On 7/2/07, Michael McCandless <dtorres@example.com> wrote: > I'd like to commit LUCENE-843. +1 Awesome job! > The patch has gone through a number of iterations but the final > version that's there now (take9) is quite a bit cleaner & simpler than > the ones leading up to it and I believe ready. > > It provides solid indexing performance gains (between 2X-8X), but, it > is somewhat more complex than the current "single doc per segment" > approach and it does introduce a change to the index format (only when > autoCommit=false) whereby multiple segments can share a single set of > term vector & stored fields files. I'll miss the elegant single doc approach that's been with us for so long, but one can't ignore the magnitude of these performance gains. > Given that it's such a big change I think (?) it's appropriate to ask > for a vote (only PMC member votes are binding) to make sure we have > consensus that this is net/net a good change for Lucene. IMO, there's no need to be that formal. A simple vote on the dev list (non-committer votes are welcome and carry weight too), and if there's a consensus then everything is good. -Yonik --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 15:41:43 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 37834 invoked from network); 2 Jul 2007 15:41:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 15:41:40 -0000 Received: (qmail 91400 invoked by uid 500); 2 Jul 2007 15:41:40 -0000 Delivered-To: dtorres@example.com Received: (qmail 91353 invoked by uid 500); 2 Jul 2007 15:41:40 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 91342 invoked by uid 99); 2 Jul 2007 15:41:40 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 08:41:40 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO out2.smtp.messagingengine.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 08:41:36 -0700 Received: from compute2.internal (compute2.internal [83.161.120.155]) by out1.messagingengine.com (Postfix) with ESMTP id C5EFB5831 for <dtorres@example.com>; Mon, 2 Jul 2007 11:41:15 -0400 (EDT) Received: from web8.messagingengine.com ([83.161.120.155]) by compute2.internal (MEProxy); Mon, 02 Jul 2007 11:41:15 -0400 Received: by web8.messagingengine.com (Postfix, from userid 99) id 9F1F71599D; Mon, 2 Jul 2007 11:41:15 -0400 (EDT) Message-Id: <dtorres@example.com> X-Sasl-Enc: H+ZuN9kMzVaviKtV7ODoSAxzZQp9NJmGqC0dIB6nPv/o 1183390875 From: "Michael McCandless" <dtorres@example.com> To: dtorres@example.com Content-Disposition: inline Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="ISO-8859-1" MIME-Version: 1.0 X-Mailer: MessagingEngine.com Webmail Interface Subject: ant jar-core is slow Date: Mon, 02 Jul 2007 11:41:15 -0400 X-Virus-Checked: Checked by ClamAV on apache.org I've noticed that "ant jar-core" walks the full subtree; this is very slow and in fact is deadly when wikipedia XML has been unpacked (~3.2 million files under contrib/benchmark/work/enwiki!). I've tracked it down to this nested element in the jar task in common-build.xml: <metainf dir="."> <patternset refid="metainf.includes"/> </metainf> which references this "metainf.includes" patternset: <patternset id="metainf.includes"> <exclude name="**/*"/> </patternset> Does this rule just exclude everything? (I'm not very familiar w/ the syntax here). EG I don't see anything besides LICENSE.txt and NOTICE.txt and MANIFEST.MF under the "META-INF" dir in the released 2.2.0 core JAR, so it seems like this rule isn't doing anything? Mike --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 17:26:22 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 84585 invoked from network); 2 Jul 2007 17:26:20 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 17:26:20 -0000 Received: (qmail 17425 invoked by uid 500); 2 Jul 2007 17:25:34 -0000 Delivered-To: dtorres@example.com Received: (qmail 17370 invoked by uid 500); 2 Jul 2007 17:25:33 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 17299 invoked by uid 99); 2 Jul 2007 17:25:29 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 10:25:29 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 10:25:25 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id AD9C07141EA for <dtorres@example.com>; Mon, 2 Jul 2007 10:25:04 -0700 (PDT) Message-ID: <2371042.1183397104708.JavaMail.jira@brutus> Date: Mon, 2 Jul 2007 10:25:04 -0700 (PDT) From: "Michael McCandless (JIRA)" <dtorres@example.com> To: dtorres@example.com Subject: [jira] Updated: (LUCENE-892) CompoundFileReader's openInput produces streams that may do an extra buffer copy In-Reply-To: <2202978.1180119856144.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-892: -------------------------------------- Attachment: LUCENE-892.patch Attached patch that implements the design above (make an IndexInput.readBytes that let's you specify whether or not a buffer should be used). All tests pass. > CompoundFileReader's openInput produces streams that may do an extra buffer copy > -------------------------------------------------------------------------------- > > Key: LUCENE-892 > URL: https://issues.apache.org/jira/browse/LUCENE-892 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Affects Versions: 2.1 > Reporter: Michael McCandless > Assignee: Michael McCandless > Priority: Minor > Attachments: LUCENE-892.patch > > > Spinoff of LUCENE-888. > The class for reading from a compound file (CompoundFileReader) has a > primary stream which is a BufferedIndexInput when that stream is from > an FSDirectory (which is the norm). That is one layer of buffering. > Then, when its openInput is called, a CSIndexInput is created which > also subclasses from BufferedIndexInput. That's a second layer of > buffering. > When a consumer actually uses that CSIndexInput to read, and a call to > readByte or readBytes runs out of what's in the first buffer, it will > go to refill its buffer. But that refill calls the first > BufferedIndexInput which in turn may refill its buffer (a double > copy) by reading the underlying stream. > Not sure how to fix it yet but we should change things to not do the > extra buffer copy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 17:26:33 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 84706 invoked from network); 2 Jul 2007 17:26:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 17:26:24 -0000 Received: (qmail 17495 invoked by uid 500); 2 Jul 2007 17:25:34 -0000 Delivered-To: dtorres@example.com Received: (qmail 17448 invoked by uid 500); 2 Jul 2007 17:25:34 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 17314 invoked by uid 99); 2 Jul 2007 17:25:30 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 10:25:30 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 10:25:25 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 29F187141F0 for <dtorres@example.com>; Mon, 2 Jul 2007 10:25:05 -0700 (PDT) Message-ID: <22213462.1183397105169.JavaMail.jira@brutus> Date: Mon, 2 Jul 2007 10:25:05 -0700 (PDT) From: "Michael McCandless (JIRA)" <dtorres@example.com> To: dtorres@example.com Subject: [jira] Updated: (LUCENE-892) CompoundFileReader's openInput produces streams that may do an extra buffer copy In-Reply-To: <2202978.1180119856144.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/LUCENE-892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-892: -------------------------------------- Lucene Fields: [New, Patch Available] (was: [New]) > CompoundFileReader's openInput produces streams that may do an extra buffer copy > -------------------------------------------------------------------------------- > > Key: LUCENE-892 > URL: https://issues.apache.org/jira/browse/LUCENE-892 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Affects Versions: 2.1 > Reporter: Michael McCandless > Assignee: Michael McCandless > Priority: Minor > Attachments: LUCENE-892.patch > > > Spinoff of LUCENE-888. > The class for reading from a compound file (CompoundFileReader) has a > primary stream which is a BufferedIndexInput when that stream is from > an FSDirectory (which is the norm). That is one layer of buffering. > Then, when its openInput is called, a CSIndexInput is created which > also subclasses from BufferedIndexInput. That's a second layer of > buffering. > When a consumer actually uses that CSIndexInput to read, and a call to > readByte or readBytes runs out of what's in the first buffer, it will > go to refill its buffer. But that refill calls the first > BufferedIndexInput which in turn may refill its buffer (a double > copy) by reading the underlying stream. > Not sure how to fix it yet but we should change things to not do the > extra buffer copy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 17:31:43 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 87986 invoked from network); 2 Jul 2007 17:31:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 17:31:41 -0000 Received: (qmail 34070 invoked by uid 500); 2 Jul 2007 17:31:41 -0000 Delivered-To: dtorres@example.com Received: (qmail 34016 invoked by uid 500); 2 Jul 2007 17:31:41 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 34005 invoked by uid 99); 2 Jul 2007 17:31:41 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 10:31:41 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO spunkymail-a15.dreamhost.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 10:31:37 -0700 Received: from [83.161.120.155] (adsl-074-229-189-244.sip.rmo.bellsouth.net [83.161.120.155]) by spunkymail-a15.dreamhost.com (Postfix) with ESMTP id D11707F053 for <dtorres@example.com>; Mon, 2 Jul 2007 10:31:14 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v752.2) In-Reply-To: <dtorres@example.com> References: <dtorres@example.com> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <dtorres@example.com> Content-Transfer-Encoding: 7bit From: Grant Ingersoll <dtorres@example.com> Subject: Re: [VOTE] Commit LUCENE-843 (IndexWriter performance gains) Date: Mon, 2 Jul 2007 13:31:10 -0400 To: dtorres@example.com X-Mailer: Apple Mail (2.752.2) X-Virus-Checked: Checked by ClamAV on apache.org On Jul 2, 2007, at 9:35 AM, Michael McCandless wrote: > Hi, > > I'd like to commit LUCENE-843. > > The patch has gone through a number of iterations but the final > version that's there now (take9) is quite a bit cleaner & simpler than > the ones leading up to it and I believe ready. > > It provides solid indexing performance gains (between 2X-8X), but, it > is somewhat more complex than the current "single doc per segment" > approach and it does introduce a change to the index format (only when > autoCommit=false) whereby multiple segments can share a single set of > term vector & stored fields files. > +0 for now, I will try to review tonight or tomorrow night. From what I gather from reading the issue, etc. it sounds great and you and others have put a lot of hard work into it. Also, from some benchmarking I have done, it seems to sit well with the notion of optimizing merge factor, etc. based on the amount of memory available. --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 18:09:05 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 3515 invoked from network); 2 Jul 2007 18:09:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 18:09:03 -0000 Received: (qmail 16199 invoked by uid 500); 2 Jul 2007 18:09:04 -0000 Delivered-To: dtorres@example.com Received: (qmail 15895 invoked by uid 500); 2 Jul 2007 18:09:03 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 15884 invoked by uid 99); 2 Jul 2007 18:09:03 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 11:09:03 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO rescomp.berkeley.edu) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 11:08:59 -0700 Received: by rescomp.berkeley.edu (Postfix, from userid 1007) id 1EA995B766; Mon, 2 Jul 2007 11:08:39 -0700 (PDT) Received: from localhost (localhost [83.161.120.155]) by rescomp.berkeley.edu (Postfix) with ESMTP id 02D257F403 for <dtorres@example.com>; Mon, 2 Jul 2007 11:08:39 -0700 (PDT) Date: Mon, 2 Jul 2007 11:08:38 -0700 (PDT) From: Chris Hostetter <dtorres@example.com> To: dtorres@example.com Subject: Re: ant jar-core is slow In-Reply-To: <dtorres@example.com> Message-ID: <dtorres@example.com> References: <dtorres@example.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org : which references this "metainf.includes" patternset: : : <patternset id="metainf.includes"> : <exclude name="**/*"/> : </patternset> : : Does this rule just exclude everything? (I'm not very familiar w/ the : syntax here). EG I don't see anything besides LICENSE.txt and NOTICE.txt : and MANIFEST.MF under the "META-INF" dir in the released 2.2.0 core JAR, so : it seems like this rule isn't doing anything? that's the patternset as defined in common-build.xml, but it can be overridden in the individual build files -- it looks like it was put in place for the snowball contirb... chrish@asimov:~/svn/lucene-clean$ find contrib -name \*.xml | xargs grep -A3 metainf.includes contrib/snowball/build.xml: <patternset id="metainf.includes"> contrib/snowball/build.xml- <include name="SNOWBALL-LICENSE.txt"/> contrib/snowball/build.xml- </patternset> contrib/snowball/build.xml- ...we could eliminate that patternset and change the jarify macro to take in nested element instead, but before trying that i'm curious: does adding... <exclude name="*" /> ..do the default patternset improve the performance? (i'm not actaully sure how to tell which files ant walks so i haven't tried it myself) -Hoss --------------------------------------------------------------------- To unsubscribe, e-mail: dtorres@example.com For additional commands, e-mail: dtorres@example.com From java-devdtorres@example.com Mon Jul 02 19:02:57 2007 Return-Path: <java-devdtorres@example.com> Delivered-To: dtorres@example.com Received: (qmail 25032 invoked from network); 2 Jul 2007 19:02:55 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jul 2007 19:02:55 -0000 Received: (qmail 23565 invoked by uid 500); 2 Jul 2007 19:02:56 -0000 Delivered-To: dtorres@example.com Received: (qmail 23199 invoked by uid 500); 2 Jul 2007 19:02:55 -0000 Mailing-List: contact dtorres@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:dtorres@example.com> List-Unsubscribe: <mailto:dtorres@example.com> List-Post: <mailto:dtorres@example.com> List-Id: <java-dev.lucene.apache.org> Reply-To: dtorres@example.com Delivered-To: mailing list dtorres@example.com Received: (qmail 23188 invoked by uid 99); 2 Jul 2007 19:02:55 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 12:02:55 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO out2.smtp.messagingengine.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2007 12:02:50 -0700 Received: from compute1.internal (compute1.internal [83.161.120.155]) by out1.messagingengine.com (Postfix) with ESMTP id 84B5C51A5 for <dtorres@example.com>; Mon, 2 Jul 2007 15:02:29 -0400 (EDT) Received: from web8.messagingengine.com ([83.161.120.155]) by compute1.internal (MEProxy); Mon, 02 Jul 2007 15:02:29 -0400
{'source': 'Apache Software Foundation Public Mail Archives'}