From java-usernunezjerry@example.org Fri Oct 01 05:39:25 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 82878 invoked from network); 1 Oct 2010 05:39:25 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Oct 2010 05:39:25 -0000 Received: (qmail 97474 invoked by uid 500); 1 Oct 2010 05:39:23 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 97046 invoked by uid 500); 1 Oct 2010 05:39:20 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Delivered-To: moderator for nunezjerry@example.org Received: (qmail 3906 invoked by uid 99); 30 Sep 2010 20:58:58 -0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1285880306; bh=FcwJkIM0X+DS/xd2+EFatMlkLrODBdlASw0b5TRUNdI=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=PVViZmAUe8zdysGEqB7VTiRBFaww7G0wrm1/sDRPNOhxe7WnquIOIXZwHAaUeg9c3NBPT4bEhhGgfKZOAxZ2gHcRvEeP1P+BywFGC04lghFosK76gam95W7qmWGyfFMh/erLQX+K9LWIQgQ9qte+1GQdbC5ITcIpU5IcTp0UuFE= DomainKey-Signature:a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type; b=f2vRawNAYNa+pN6jPKLn3juGKPqdYsHM4Dw1+GgvXuZmxMcjTvDP/ieDxzkzfFKxaca994oNIpjfqpM+HqtGtNJxX13q4ak0NQrvPsH9BCV6tqcy6WXpT9SMLMNWaSahDToEWM1lmXSARJyNMbwwhpbvzNrPSbgm1YsDF/tkMz0=; Message-ID: <nunezjerry@example.org> X-YMail-OSG: LV0ls80VM1kMZqO2mpkcJTUmkmzF2VV4WJL7zkfkOQGuzVr AEzUFmTuSIWYbl7rlhyh9u.o10.0A7LBRgl_V2Txl8ZnHceO7xPEWUHzSX_i 5pBxdRXzb3IXWBTD5QX9foqKnvJJehvRMlLWX1va7f7Rf_DghooIVcBXgcvf P0NtVYtw5KxM3BAHpCZhG_7rc2yGVYXcVyX1Q9yH_WdnvIlI8wC7DfJZuhyX 6guoO5NqPfA0JnuK2A7jg3.EY8qQ4.2.ec.YzM3cYBzTPy5zlg_ZtmXmxDFe wU9jsjyMn X-Mailer: YahooMailRC/497 YahooMailWebService/83.161.120.1559950 Date: Thu, 30 Sep 2010 13:58:26 -0700 (PDT) From: Jonathan Ciampi <nunezjerry@example.org> Subject: Looking for advice on using Lucene to semantically compare two documents To: nunezjerry@example.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-2046786559-1285880306=:89208" X-Virus-Checked: Checked by ClamAV on apache.org --0-2046786559-1285880306=:89208 Content-Type: text/plain; charset=us-ascii Advice on comparing two documents. Summary This project is not a search engine but a semantic comparison between two documents. The purpose of this application is to assist users in modifying the text in a document to improve the relevancy rank of the document to another document. For example, the user would want to compare Document A to Document B to identify the text in Document A that has relevancy to Document B. Then, the user would want the ability to identify the text to modify to improve the relevancy rating. Description: Both documents are XML with tags identifying the keywords or blocks of text in the document. Sample Structure Document A <DocumentName>DocumentA</DocumentName> <Keyword>This is keyword 1</Keyword> <Keyword>Keywords can be any length</Keyword> <Keyword>Some keywords will match Document B</Keyword> <Keyword>Some keywords will not match</Keyword> <Keyword>Keywords can contain text, numbers, and symbols</Keyword> Document B <DocumentName>DocumentB</DocumentName> <Keyword>This is Document B keyword 1</Keyword> <Keyword>Document B serves as the basis or standard for comparing</Keyword> <Keyword>Document A will be modified by the user to match the keywords in Document B</Keyword> <Keyword>Document A and Document B will always be compared to each other</Keyword> <Keyword>This application is to help users add text, numbers and symbols to improve their relevancy ranking</Keyword> We believe we need to use Lucene to do semantic searches to determine relevance. Our preferred output would be to show a user the words from each document with their relevancy. To remove excessive data, the output would show all keywords from Document B, and only those with a relevancy ranking from Document A. Sample Output Document B Document A Relevancy This is Document B keyword 1 This is keyword 1 .25 This is Document B keyword 1 Keywords can be any length .25 This is Document B keyword 1 Some keywords will match Document B .25 This is Document B keyword 1 Some keywords will not match .25 This is Document B keyword 1 Keywords can contain text, numbers, and symbols .25 Document B serves as the basis or standard for comparing Some keywords will match Document B .5 Document A will be modified by the user to match the keywords in Document B This is keyword 1 .1 Document A will be modified by the user to match the keywords in Document B Keywords can be any length .1 Document A will be modified by the user to match the keywords in Document B Some keywords will not match .1 Document A will be modified by the user to match the keywords in Document B Some keywords will match Document B .75 Document A will be modified by the user to match the keywords in Document B Keywords can contain text, numbers, and symbols .1 This application is to help users add text, numbers and symbols to improve their relevancy ranking Keywords can contain text, numbers, and symbols .9 Jon Ciampi Mobile 602.971.5981 --0-2046786559-1285880306=:89208-- From java-usernunezjerry@example.org Fri Oct 01 05:42:13 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 83331 invoked from network); 1 Oct 2010 05:42:13 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Oct 2010 05:42:13 -0000 Received: (qmail 4394 invoked by uid 500); 1 Oct 2010 05:42:11 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 4186 invoked by uid 500); 1 Oct 2010 05:42:08 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 4178 invoked by uid 99); 1 Oct 2010 05:42:07 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 05:42:07 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO serve.stimulussoft.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 05:42:00 +0000 Received: from serve.stimulussoft.com (localhost.localdomain [83.161.120.155]) by serve.stimulussoft.com (Postfix) with ESMTP id 463A210747 for <nunezjerry@example.org>; Thu, 30 Sep 2010 22:41:40 -0700 (PDT) Received: by serve.stimulussoft.com (Postfix, from userid 5001) id 328DF1078C; Thu, 30 Sep 2010 22:41:40 -0700 (PDT) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on serve.stimulussoft.com X-Spam-Level: Received: from Jamie-Bands-MacBook-Pro.local (196-210-169-220.dynamic.isadsl.co.za [83.161.120.155]) by serve.stimulussoft.com (Postfix) with ESMTPA id 5DDBD10747 for <nunezjerry@example.org>; Thu, 30 Sep 2010 22:41:37 -0700 (PDT) Message-ID: <nunezjerry@example.org> Date: Fri, 01 Oct 2010 07:41:13 +0200 From: Jamie <nunezjerry@example.org> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:83.161.120.155) Gecko/20100915 Thunderbird/3.1.4 MIME-Version: 1.0 To: nunezjerry@example.org Subject: Re: File Handle Leaks During Lucene 3.0.2 Merge References: <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> In-Reply-To: <nunezjerry@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP X-Old-Spam-Status: No, score=-4.5 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.4 Hi Mike I managed to get hold of a copy of your book through Safari Books. Quite an impressive online reading system they have there! I integrated your SearchManager class into our code, but I am still seeing file handles marked deleted in the index directory. I am running the following command on Linux: sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l" Every 0.1s: lsof | grep /var/index | grep deleted |... Fri Oct 1 09:37:36 2010 54 The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32, etc. They seem stable though. Is this to be expected when using NRT search? I am pretty certain that all Searchers are released at the end of every search. I double checked it at least twenty times. Jamie On 2010/09/30 11:56 PM, Michael McCandless wrote: > On Thu, Sep 30, 2010 at 5:59 AM, Jamie<nunezjerry@example.org> wrote: >> Hi Michael / Uwe >> >>> It's good to cache the reader, but, finalize would worry me too since >>> you have no control over when GC gets around to calling it... you risk >>> tying up resources for longer than necessary. >> I did it this way, as I didn't want to over complicate the code by >> introducing mechanisms to track the number of search threads using a shared >> indexreader. Admittedly, its not a very clean solution but in my case it >> does work. Is there a particular technique for knowing when to a close a >> reader when there are multiple search threads using that reader? Should I >> keep some kind of counter and override the close method of the reader such >> that the underlying reader is only closed when everyone's done with it? > See Uwe's response (or SearcherManager). > >>> IndexWriter has a reader pool, internally, where it holds open >>> SegmentReaders for the still-live segments in the index. This is used >>> by IndexReader.reopen to share open SegmentReaders. >>> >>> But the open files should correspond only to segments still "live" in >>> the index. After segments are merged away, these readers are dropped. >>> Is this what you are seeing? >>> >> I dont fully understand your explanation/question. When I run lsof, I am >> seeing the following: >> >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyr.cfs >> (deleted) >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyp.cfs >> (deleted) >> >> I assume these are left by the OS after the merge operation tried to delete >> old segments. The OS is unable to delete the files. I think its because our >> new code never closes the indexwriter, but rather uses the >> indexwriter.commit() method to apply the changes. Is this correct? > Ahh I see they are deleted but held open... hmmm. > > Though this is also what you'd see if there were still a reader open. > Are you certain all readers were closed (finalized) when you ran lsof? > > Mike > > --------------------------------------------------------------------- > To unsubscribe, e-mail: nunezjerry@example.org > For additional commands, e-mail: nunezjerry@example.org > --------------------------------------------------------------------- To unsubscribe, e-mail: nunezjerry@example.org For additional commands, e-mail: nunezjerry@example.org From java-usernunezjerry@example.org Fri Oct 01 06:00:22 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 85849 invoked from network); 1 Oct 2010 06:00:21 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Oct 2010 06:00:21 -0000 Received: (qmail 20288 invoked by uid 500); 1 Oct 2010 06:00:20 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 19818 invoked by uid 500); 1 Oct 2010 06:00:16 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 19810 invoked by uid 99); 1 Oct 2010 06:00:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 06:00:15 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mail.troja.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 06:00:07 +0000 Received: from localhost (localhost.localdomain [83.161.120.155]) by mail.troja.net (Postfix) with ESMTP id 9734745F1D8 for <nunezjerry@example.org>; Fri, 1 Oct 2010 07:59:45 +0200 (CEST) X-Virus-Scanned: Debian nunezjerry@example.org Received: from mail.troja.net ([83.161.120.155]) by localhost (megaira.troja.net [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id FW3bKuJqO+fU for <nunezjerry@example.org>; Fri, 1 Oct 2010 07:59:33 +0200 (CEST) Received: from VEGA (port-83-236-62-54.dynamic.qsc.de [83.161.120.155]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.troja.net (Postfix) with ESMTPSA id BA20F45F1C8 for <nunezjerry@example.org>; Fri, 1 Oct 2010 07:59:32 +0200 (CEST) From: "Uwe Schindler" <nunezjerry@example.org> To: <nunezjerry@example.org> References: <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> In-Reply-To: <nunezjerry@example.org> Subject: RE: File Handle Leaks During Lucene 3.0.2 Merge Date: Fri, 1 Oct 2010 08:00:04 +0200 Message-ID: <nunezjerry@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQK1IASPuHD7fWK4tGfTUhRYh1Mz8AJQ2/brAf/iy38B6P+tvQKZsL4xAcbXWDIBzpQnwgHBVo+kAsiDC1OQ0IXGYA== Content-Language: de X-Virus-Checked: Checked by ClamAV on apache.org Hi Jamie, YES, ist expected for the reasons described above (segments are still referenced by the open IndexReaders, but files were already deleted by IndexWriter). The approx. number of open, but already deleted files should be approx. stable. Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: nunezjerry@example.org > -----Original Message----- > From: Jamie [mailto:nunezjerry@example.org] > Sent: Friday, October 01, 2010 7:41 AM > To: nunezjerry@example.org > Subject: Re: File Handle Leaks During Lucene 3.0.2 Merge > > Hi Mike > > I managed to get hold of a copy of your book through Safari Books. > Quite an impressive online reading system they have there! I integrated your > SearchManager class into our code, but I am still seeing file handles marked > deleted in the index directory. I am running the following command on Linux: > > sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l" > > Every 0.1s: lsof | grep /var/index | grep deleted |... Fri Oct 1 > 09:37:36 2010 > > 54 > > The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32, etc. They > seem stable though. Is this to be expected when using NRT search? > > I am pretty certain that all Searchers are released at the end of every search. > I double checked it at least twenty times. > > Jamie > > > > On 2010/09/30 11:56 PM, Michael McCandless wrote: > > On Thu, Sep 30, 2010 at 5:59 AM, Jamie<nunezjerry@example.org> wrote: > >> Hi Michael / Uwe > >> > >>> It's good to cache the reader, but, finalize would worry me too > >>> since you have no control over when GC gets around to calling it... > >>> you risk tying up resources for longer than necessary. > >> I did it this way, as I didn't want to over complicate the code by > >> introducing mechanisms to track the number of search threads using a > >> shared indexreader. Admittedly, its not a very clean solution but in > >> my case it does work. Is there a particular technique for knowing > >> when to a close a reader when there are multiple search threads using > >> that reader? Should I keep some kind of counter and override the > >> close method of the reader such that the underlying reader is only closed > when everyone's done with it? > > See Uwe's response (or SearcherManager). > > > >>> IndexWriter has a reader pool, internally, where it holds open > >>> SegmentReaders for the still-live segments in the index. This is > >>> used by IndexReader.reopen to share open SegmentReaders. > >>> > >>> But the open files should correspond only to segments still "live" > >>> in the index. After segments are merged away, these readers are dropped. > >>> Is this what you are seeing? > >>> > >> I dont fully understand your explanation/question. When I run lsof, I > >> am seeing the following: > >> > >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyr.cf > >> s > >> (deleted) > >> /usr/local/mailarchiva/server/webapps/ROOT/WEB-INF/logs/index/_jyp.cf > >> s > >> (deleted) > >> > >> I assume these are left by the OS after the merge operation tried to > >> delete old segments. The OS is unable to delete the files. I think > >> its because our new code never closes the indexwriter, but rather > >> uses the > >> indexwriter.commit() method to apply the changes. Is this correct? > > Ahh I see they are deleted but held open... hmmm. > > > > Though this is also what you'd see if there were still a reader open. > > Are you certain all readers were closed (finalized) when you ran lsof? > > > > Mike > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: nunezjerry@example.org > > For additional commands, e-mail: nunezjerry@example.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: nunezjerry@example.org > For additional commands, e-mail: nunezjerry@example.org --------------------------------------------------------------------- To unsubscribe, e-mail: nunezjerry@example.org For additional commands, e-mail: nunezjerry@example.org From java-usernunezjerry@example.org Fri Oct 01 09:48:37 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 46612 invoked from network); 1 Oct 2010 09:48:37 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Oct 2010 09:48:37 -0000 Received: (qmail 99597 invoked by uid 500); 1 Oct 2010 09:48:35 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 99155 invoked by uid 500); 1 Oct 2010 09:48:31 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 99147 invoked by uid 99); 1 Oct 2010 09:48:29 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 09:48:29 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-ww0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 09:48:23 +0000 Received: by wwb22 with SMTP id 22so1579523wwb.5 for <nunezjerry@example.org>; Fri, 01 Oct 2010 02:48:02 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id i44mr1890007wed.53.1285926482522; Fri, 01 Oct 2010 02:48:02 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 1 Oct 2010 02:48:02 -0700 (PDT) In-Reply-To: <nunezjerry@example.org> References: <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> Date: Fri, 1 Oct 2010 05:48:02 -0400 Message-ID: <nunezjerry@example.org> Subject: Re: File Handle Leaks During Lucene 3.0.2 Merge From: Michael McCandless <nunezjerry@example.org> To: nunezjerry@example.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On Fri, Oct 1, 2010 at 1:41 AM, Jamie <nunezjerry@example.org> wrote: > =A0I managed to get hold of a copy of your book through Safari Books. Qui= te an > impressive online reading system they have there! Neat! I had never heard of Safari Books until now, but it looks cool. > I integrated your > SearchManager class into our code, but I am still seeing file handles mar= ked > deleted in the index directory. I am running the following command on Lin= ux: > > sudo watch -n 0 "lsof | grep /var/index | grep deleted | wc -l" > > Every 0.1s: lsof | grep /var/index | grep deleted |... =A0Fri Oct =A01 09= :37:36 > 2010 > > 54 > > The deleted file handles fluctuate up and down. 54 -> 102 -> 64 -> 32, et= c. > They seem stable though. Is this to be expected when using NRT search? > > =A0I am pretty certain that all Searchers are released at the end of ever= y > search. I double checked it at least twenty times. SearcherManager always keeps one IndexReader open (the current "live" one), so it's expected that this reader is holding open references to deleted files if in fact IndexWriter has completed merges since the reader was opened. Though.. I would expect you to sometimes see 0 deleted files in your lsof... Maybe try this: 1) pause your indexing, 2) reopen the reader (call SearcherManager.reopen), and then 3) run lsof again, at which point you should see no deleted files still held open. But note that to truly pause your indexing, you should switch (temporarily, for this test) your IndexWriter to the SerialMergeScheduler, and then ensure you stop calling IndexWriter ops while you run steps 2 and 3. Mike --------------------------------------------------------------------- To unsubscribe, e-mail: nunezjerry@example.org For additional commands, e-mail: nunezjerry@example.org From java-usernunezjerry@example.org Fri Oct 01 15:33:48 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 7072 invoked from network); 1 Oct 2010 15:33:47 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 1 Oct 2010 15:33:47 -0000 Received: (qmail 68583 invoked by uid 500); 1 Oct 2010 15:33:45 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 68213 invoked by uid 500); 1 Oct 2010 15:33:42 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 68190 invoked by uid 99); 1 Oct 2010 15:33:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 15:33:41 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nunezjerry@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ww0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 Oct 2010 15:33:33 +0000 Received: by wwb22 with SMTP id 22so1856118wwb.5 for <nunezjerry@example.org>; Fri, 01 Oct 2010 08:33:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=WifJ2yy+kZifLuUPzahyaqcuMvQmhmBtAsDxDO4/AQQ=; b=UnTvgvYVPYnDe3x+dnJXZvl07qVwBvsDqfx4/VByZt1b7JKXLY3OajLFm3OoZp/a8X h/RiRTu72kMlj7oXZRaijuY0nOTdI5qwQFLzmXTimwGCJFa7AR02EDS/U69Odpwv9jYc gOiViRY0nQgHrgzkSyzHHkRReQDmIi4yaTasY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=EL4Uo1BA/9LCoIrPtKicQTELSEr4Uc3FuhlmO7iYvjp6HGM2Rl2bnmU8YJDd5I1HlC Rm34qWpZHLrTbjsu6o//ORz8nHN6ESwXFD9gwz35uWz3vggwOMkuRDuYyJmpoMtBPdn7 jtzn4lF0K9QKLigYnNLi58nbb1TxsIRmRFlfQ= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id g33mr960973weg.40.1285947193607; Fri, 01 Oct 2010 08:33:13 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 1 Oct 2010 08:33:13 -0700 (PDT) Date: Fri, 1 Oct 2010 11:33:13 -0400 Message-ID: <nunezjerry@example.org> Subject: How to get the score of a term in a document? From: Sahin Buyrukbilen <nunezjerry@example.org> To: nunezjerry@example.org Content-Type: multipart/alternative; boundary=0015175909b0acc4ef04918fe962 X-Virus-Checked: Checked by ClamAV on apache.org --0015175909b0acc4ef04918fe962 Content-Type: text/plain; charset=ISO-8859-1 Hi all, I need to retrieve the score of a term in a document? I dont want to play different scoring schemes. I just checked my index with Luke and it shows me a score for each term in each document the term exists. So, I need just to get that score. Can anybody help me? Thank you in advance. Sahin. --0015175909b0acc4ef04918fe962-- From java-usernunezjerry@example.org Sat Oct 02 01:22:39 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 48226 invoked from network); 2 Oct 2010 01:22:39 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Oct 2010 01:22:39 -0000 Received: (qmail 12642 invoked by uid 500); 2 Oct 2010 01:22:37 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 12538 invoked by uid 500); 2 Oct 2010 01:22:36 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 12530 invoked by uid 99); 2 Oct 2010 01:22:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 01:22:36 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nunezjerry@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-qw0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 01:22:28 +0000 Received: by qwk3 with SMTP id 3so2435743qwk.35 for <nunezjerry@example.org>; Fri, 01 Oct 2010 18:22:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=0c8op+qNKIrMi6xdaeWWdvw+WR24aNjSl3EJCfKgNrQ=; b=KTAqgky9rQ9PMJobQu7WHMssjFJsTRUlVDKZl91EeEFPJEi39s+cv4J1IhvRmLhDlL Lx0lIJIl93B7KWd8QBQsRXBBSw5YGeadHi3zgcOBfPyqDinMDHLIGlbCbZJmvXKvEPJu dG2u5dqlYyhaJQ9EaOIU3kKXjrPSxMVtSBhbE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=Hzf/Jv5ivmwyBVXbnRqOq1EekPwpsX5Pxpzxut68ClpdnlC5AGw9M5vYsiT5uWVkaa N7Ep6f6rSa0G9yrW1uu4YxR2zjwZ8tnQgUGbYsTgvAtLATVtRBe2otHs2DvoU2D0el56 /JgdxNzIi1Teq6U+WYpFNhY67kAVA+W7czGTQ= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id bw12mr3193635qcb.249.1285982528055; Fri, 01 Oct 2010 18:22:08 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 1 Oct 2010 18:22:07 -0700 (PDT) In-Reply-To: <nunezjerry@example.org> References: <nunezjerry@example.org> Date: Fri, 1 Oct 2010 21:22:07 -0400 Message-ID: <nunezjerry@example.org> Subject: Re: How to get the score of a term in a document? From: Erick Erickson <nunezjerry@example.org> To: nunezjerry@example.org Content-Type: multipart/alternative; boundary=0016361e7fe4c5aa1304919823f6 X-Virus-Checked: Checked by ClamAV on apache.org --0016361e7fe4c5aa1304919823f6 Content-Type: text/plain; charset=ISO-8859-1 I'm not sure what you're asking for. "Score of a term in a document"? Do you mean the amount a term contributed to a search for a particular document? The frequency of a term in a document? ??? Could you elaborate on what you're trying to do? If you describe the problem you're trying to solve, people can provide better answers. Best Erick On Fri, Oct 1, 2010 at 11:33 AM, Sahin Buyrukbilen < nunezjerry@example.org> wrote: > Hi all, > > I need to retrieve the score of a term in a document? I dont want to play > different scoring schemes. I just checked my index with Luke and it shows > me > a score for each term in each document the term exists. So, I need just to > get that score. > > Can anybody help me? > > Thank you in advance. > > Sahin. > --0016361e7fe4c5aa1304919823f6-- From java-usernunezjerry@example.org Sat Oct 02 03:50:20 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 80423 invoked from network); 2 Oct 2010 03:50:19 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Oct 2010 03:50:19 -0000 Received: (qmail 84832 invoked by uid 500); 2 Oct 2010 03:50:17 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 84399 invoked by uid 500); 2 Oct 2010 03:50:14 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 84391 invoked by uid 99); 2 Oct 2010 03:50:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 03:50:13 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nunezjerry@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-wy0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 03:50:03 +0000 Received: by wye20 with SMTP id 20so501385wye.35 for <nunezjerry@example.org>; Fri, 01 Oct 2010 20:49:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=h40emQFcP46TMHaxZMDQ2r3ug0uRrdfcizwUi0vX8KA=; b=EuUKpIZABDweyc+NxyZh0hAZkRkFMvzdTlSLIwgTLG8bqvk783I8R3gcCwsf75cL1r FFs/u/EaZJTN6PsusMyUxb+AwzwbLpAQzCx6REC05U1IyjPdzaOLhmbeFbG+fZ8aFADU sEei8AYSmits+x9DAW/e+5y1RUSSyX3SpBNOY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=XL5yRjJKjrGnb+2ootrdUA3dYjpbkANLuE01qeV7BSwa4cGutJkEpUPjUYosfPkve5 rnJMqPkz4GmuiJJObpdp3xSDpk/lQySxecL370RkOiNeZ7eDwsIScy9jtMSj8RVDjBOf v/BjSmYiE8U+zMEajt6HhrTk2SzbX1x7+9Iww= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id t3mr5244028wet.39.1285991383067; Fri, 01 Oct 2010 20:49:43 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 1 Oct 2010 20:49:42 -0700 (PDT) In-Reply-To: <nunezjerry@example.org> References: <nunezjerry@example.org> <nunezjerry@example.org> Date: Fri, 1 Oct 2010 23:49:42 -0400 Message-ID: <nunezjerry@example.org> Subject: Re: How to get the score of a term in a document? From: Sahin Buyrukbilen <nunezjerry@example.org> To: nunezjerry@example.org Content-Type: multipart/mixed; boundary=0016364c7949926da504919a3301 X-Virus-Checked: Checked by ClamAV on apache.org --0016364c7949926da504919a3301 Content-Type: multipart/alternative; boundary=0016364c7949926d9904919a33ff --0016364c7949926d9904919a33ff Content-Type: text/plain; charset=ISO-8859-1 Hi Erick, I mean the score of a term in a document (we can think this as a one word query) which is calculated by using "Default Similarity". Actually, when I walk through my index term-by-term, Luke shows me the number of documents in which the term exists. And for each document there is a score field. please check the attachment for the screenshot. I am very new to the jargon of Lucene, so I am sorry if I explain things in an incorrect way. My question is: For a term in the index, can we retrieve the value (here I say score) calculated by using default similarity? Is this a value which is already stored in the index or is it calculated on the fly by Luke (since I can only see by using Luke)? My goal is to create an inverted index and write it into a text file in the following form: Term t ft Inverted list for t ---------------------------------------------------------------------------------- big 2 <2, 0.148> <3, 0.088> in 5 <6, 0.159> <2, 0.143> <5, 0.088> <1, 0.076> <4, 0.065> - - - - - so on for all terms. Here ft is the total frequency of term t in the whole index, <docID , score > pairs are ID of the document in which term t has a score, and these pairs are listed according to the decreasing order of scores. I checked through the documentation, and found scorer class but couldnt understand how to use it. I hope this is a kind of better explanation. Best. Sahin. On Fri, Oct 1, 2010 at 9:22 PM, Erick Erickson <nunezjerry@example.org>wrote: > I'm not sure what you're asking for. "Score of a term in a document"? Do > you > mean the amount a term contributed to a search for a particular document? > The frequency of a term in a document? ??? > > Could you elaborate on what you're trying to do? If you describe the > problem > you're trying to solve, people can provide better answers. > > Best > Erick > > On Fri, Oct 1, 2010 at 11:33 AM, Sahin Buyrukbilen < > nunezjerry@example.org> wrote: > > > Hi all, > > > > I need to retrieve the score of a term in a document? I dont want to play > > different scoring schemes. I just checked my index with Luke and it shows > > me > > a score for each term in each document the term exists. So, I need just > to > > get that score. > > > > Can anybody help me? > > > > Thank you in advance. > > > > Sahin. > > > --0016364c7949926d9904919a33ff Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Erick,<br><br>I mean the score of a term in a document (we can think thi= s as a one word query) which is calculated by using &quot;Default Similarit= y&quot;. Actually, when I walk through my index term-by-term, Luke shows me= the number of documents in which the term exists. And for each document th= ere is a score field. please check the attachment for the screenshot. I am = very new to the jargon of Lucene, so I am sorry if I explain things in an i= ncorrect way.<br> <br>My question is: For a term in the index, can we retrieve the value (her= e I say score) calculated by using default similarity? Is this a value whic= h is already stored in the index or is it calculated on the fly by Luke (si= nce I can only see by using Luke)?<br> <br>My goal is to create an inverted index and write it into a text file in= the following form:<br><br>Term t=A0=A0=A0=A0=A0=A0=A0 ft=A0=A0=A0=A0=A0= =A0=A0=A0 Inverted list for t<br>------------------------------------------= ----------------------------------------<br> big=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0=A0=A0=A0=A0 &lt;2, 0.= 148&gt; &lt;3, 0.088&gt;<br>in=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 5=A0=A0=A0=A0=A0=A0=A0 &lt;6, 0.159&gt; &lt;2, 0.143&gt; &lt;5, 0.088&gt; = &lt;1, 0.076&gt; &lt;4, 0.065&gt;<br>-<br>-<br>-<br>-<br>-<br>so on for all= terms. Here ft is the total frequency of term t in the whole index, &lt;do= cID , score &gt; pairs are ID of the document in which term t has a score, = and these pairs are listed according to the decreasing order of scores.<br> <br><br>I checked through the documentation, and found scorer class but cou= ldnt understand how to use it.<br><br>I hope this is a kind of better expla= nation.<br><br>Best.<br>Sahin.<br><br><br><div class=3D"gmail_quote">On Fri= , Oct 1, 2010 at 9:22 PM, Erick Erickson <span dir=3D"ltr">&lt;<a href=3D"m= ailto:nunezjerry@example.org">nunezjerry@example.org</a>&gt;</span> wrote= :<br> <blockquote class=3D"gmail_quote" style=3D"margin: 0pt 0pt 0pt 0.8ex; borde= r-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">I&#39;m not sure = what you&#39;re asking for. &quot;Score of a term in a document&quot;? Do y= ou<br> mean the amount a term contributed to a search for a particular document?<b= r> The frequency of a term in a document? ???<br> <br> Could you elaborate on what you&#39;re trying to do? If you describe the pr= oblem<br> you&#39;re trying to solve, people can provide better answers.<br> <br> Best<br> <font color=3D"#888888">Erick<br> </font><div><div></div><div class=3D"h5"><br> On Fri, Oct 1, 2010 at 11:33 AM, Sahin Buyrukbilen &lt;<br> <a href=3D"mailto:nunezjerry@example.org">nunezjerry@example.org<= /a>&gt; wrote:<br> <br> &gt; Hi all,<br> &gt;<br> &gt; I need to retrieve the score of a term in a document? I dont want to p= lay<br> &gt; different scoring schemes. I just checked my index with Luke and it sh= ows<br> &gt; me<br> &gt; a score for each term in each document the term exists. So, I need jus= t to<br> &gt; get that score.<br> &gt;<br> &gt; Can anybody help me?<br> &gt;<br> &gt; Thank you in advance.<br> &gt;<br> &gt; Sahin.<br> &gt;<br> </div></div></blockquote></div><br> --0016364c7949926d9904919a33ff-- --0016364c7949926da504919a3301 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: nunezjerry@example.org For additional commands, e-mail: nunezjerry@example.org --0016364c7949926da504919a3301-- From java-usernunezjerry@example.org Sat Oct 02 09:31:15 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 51604 invoked from network); 2 Oct 2010 09:31:15 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Oct 2010 09:31:15 -0000 Received: (qmail 7938 invoked by uid 500); 2 Oct 2010 09:31:13 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 7435 invoked by uid 500); 2 Oct 2010 09:31:08 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 7427 invoked by uid 99); 2 Oct 2010 09:31:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 09:31:07 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-wy0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 09:30:58 +0000 Received: by wye20 with SMTP id 20so639167wye.35 for <nunezjerry@example.org>; Sat, 02 Oct 2010 02:30:38 -0700 (PDT) MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id d16mr5461491wel.0.1286011838163; Sat, 02 Oct 2010 02:30:38 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sat, 2 Oct 2010 02:30:38 -0700 (PDT) In-Reply-To: <nunezjerry@example.org> References: <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> Date: Sat, 2 Oct 2010 05:30:38 -0400 Message-ID: <nunezjerry@example.org> Subject: Re: How to get the score of a term in a document? From: Michael McCandless <nunezjerry@example.org> To: nunezjerry@example.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org It sounds like you can just use Lucene's enum APIs (IndexReader.terms, IndexReader.termDocs) to walk the entire index, converting it to your format? I'm not sure how Luke computes the "score"... but maybe you could, for every term, make a TermQuery and then directly walk its matching docs & scores? You'd have to do something like: Scorer s =3D TermQuery.weight(searcher).scorer(reader, true, false); int docID; while((docID =3D s.nextDoc()) !=3D Scorer.NO_MORE_DOCS) { float score =3D s.score(); } I think? Mike On Fri, Oct 1, 2010 at 11:49 PM, Sahin Buyrukbilen <nunezjerry@example.org> wrote: > Hi Erick, > > I mean the score of a term in a document (we can think this as a one word > query) which is calculated by using "Default Similarity". Actually, when = I > walk through my index term-by-term, Luke shows me the number of documents= in > which the term exists. And for each document there is a score field. plea= se > check the attachment for the screenshot. I am very new to the jargon of > Lucene, so I am sorry if I explain things in an incorrect way. > > My question is: For a term in the index, can we retrieve the value (here = I > say score) calculated by using default similarity? Is this a value which = is > already stored in the index or is it calculated on the fly by Luke (since= I > can only see by using Luke)? > > My goal is to create an inverted index and write it into a text file in t= he > following form: > > Term t=A0=A0=A0=A0=A0=A0=A0 ft=A0=A0=A0=A0=A0=A0=A0=A0 Inverted list for = t > -------------------------------------------------------------------------= --------- > big=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 2=A0=A0=A0=A0=A0=A0=A0 <2, 0.1= 48> <3, 0.088> > in=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 5=A0=A0=A0=A0=A0=A0=A0 <6= , 0.159> <2, 0.143> <5, 0.088> <1, 0.076> <4, > 0.065> > - > - > - > - > - > so on for all terms. Here ft is the total frequency of term t in the whol= e > index, <docID , score > pairs are ID of the document in which term t has = a > score, and these pairs are listed according to the decreasing order of > scores. > > > I checked through the documentation, and found scorer class but couldnt > understand how to use it. > > I hope this is a kind of better explanation. > > Best. > Sahin. > > > On Fri, Oct 1, 2010 at 9:22 PM, Erick Erickson <nunezjerry@example.org> > wrote: >> >> I'm not sure what you're asking for. "Score of a term in a document"? Do >> you >> mean the amount a term contributed to a search for a particular document= ? >> The frequency of a term in a document? ??? >> >> Could you elaborate on what you're trying to do? If you describe the >> problem >> you're trying to solve, people can provide better answers. >> >> Best >> Erick >> >> On Fri, Oct 1, 2010 at 11:33 AM, Sahin Buyrukbilen < >> nunezjerry@example.org> wrote: >> >> > Hi all, >> > >> > I need to retrieve the score of a term in a document? I dont want to >> > play >> > different scoring schemes. I just checked my index with Luke and it >> > shows >> > me >> > a score for each term in each document the term exists. So, I need jus= t >> > to >> > get that score. >> > >> > Can anybody help me? >> > >> > Thank you in advance. >> > >> > Sahin. >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: nunezjerry@example.org > For additional commands, e-mail: nunezjerry@example.org > --------------------------------------------------------------------- To unsubscribe, e-mail: nunezjerry@example.org For additional commands, e-mail: nunezjerry@example.org From java-usernunezjerry@example.org Sat Oct 02 13:43:34 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 6657 invoked from network); 2 Oct 2010 13:43:33 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Oct 2010 13:43:33 -0000 Received: (qmail 26410 invoked by uid 500); 2 Oct 2010 13:43:32 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 25906 invoked by uid 500); 2 Oct 2010 13:43:28 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 25898 invoked by uid 99); 2 Oct 2010 13:43:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 13:43:27 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of nunezjerry@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ww0-f48.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 13:43:20 +0000 Received: by wwb22 with SMTP id 22so2483227wwb.5 for <nunezjerry@example.org>; Sat, 02 Oct 2010 06:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=JS5q+jKjRomAeo/f8KoJB3jkHgLaLfOj2CN24jMa94A=; b=c+n6xV8JlqxKuC+cntofi39uSRHqjOk+F4ohzlfKd31WarqFT2vKQqr2nMed3zSUJG ff8GrWYh1L/PvjTy5kw3WbxUbjkea04GM+J1WAduPNBvD4DbkHTfG2vTMkI49xwhTroB QpSkVNDAY5DGbRWgGpKyLzDNSs7YbrzFQMEro= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=McUKtnOfVh0ki/07lQ2FY2FnNHxPxvRu/l/HDwt7MfPY/gTTJgXy+ohgdmUP3vin6A TRGtvDeACom/oUXTUDhwUSG847RVz2bItzpMrbTwPQpYcBkZS50VADR3wxug7IyT+Zz0 EBAWLVGNUKY2XoAIMGLZC6c5NAgO9o4LVSEOQ= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id s52mr3249982wes.11.1286026979963; Sat, 02 Oct 2010 06:42:59 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sat, 2 Oct 2010 06:42:59 -0700 (PDT) In-Reply-To: <nunezjerry@example.org> References: <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> Date: Sat, 2 Oct 2010 09:42:59 -0400 Message-ID: <nunezjerry@example.org> Subject: Re: How to get the score of a term in a document? From: Sahin Buyrukbilen <nunezjerry@example.org> To: nunezjerry@example.org Content-Type: multipart/alternative; boundary=0016364d1b8d4ff24f0491a27dfb X-Virus-Checked: Checked by ClamAV on apache.org --0016364d1b8d4ff24f0491a27dfb Content-Type: text/plain; charset=ISO-8859-1 Hi Mike, I am already done with walking through the terms, frequencies and the docs by using termenum, termdocs, and indexreader,. The only thing left is the scores. I will try your suggestion. hope it works. Thank you. Sahin. On Sat, Oct 2, 2010 at 5:30 AM, Michael McCandless < nunezjerry@example.org> wrote: > It sounds like you can just use Lucene's enum APIs (IndexReader.terms, > IndexReader.termDocs) to walk the entire index, converting it to your > format? > > I'm not sure how Luke computes the "score"... but maybe you could, for > every term, make a TermQuery and then directly walk its matching docs > & scores? You'd have to do something like: > > Scorer s = TermQuery.weight(searcher).scorer(reader, true, false); > > int docID; > while((docID = s.nextDoc()) != Scorer.NO_MORE_DOCS) { > float score = s.score(); > } > > I think? > > Mike > > On Fri, Oct 1, 2010 at 11:49 PM, Sahin Buyrukbilen > <nunezjerry@example.org> wrote: > > Hi Erick, > > > > I mean the score of a term in a document (we can think this as a one word > > query) which is calculated by using "Default Similarity". Actually, when > I > > walk through my index term-by-term, Luke shows me the number of documents > in > > which the term exists. And for each document there is a score field. > please > > check the attachment for the screenshot. I am very new to the jargon of > > Lucene, so I am sorry if I explain things in an incorrect way. > > > > My question is: For a term in the index, can we retrieve the value (here > I > > say score) calculated by using default similarity? Is this a value which > is > > already stored in the index or is it calculated on the fly by Luke (since > I > > can only see by using Luke)? > > > > My goal is to create an inverted index and write it into a text file in > the > > following form: > > > > Term t ft Inverted list for t > > > ---------------------------------------------------------------------------------- > > big 2 <2, 0.148> <3, 0.088> > > in 5 <6, 0.159> <2, 0.143> <5, 0.088> <1, 0.076> > <4, > > 0.065> > > - > > - > > - > > - > > - > > so on for all terms. Here ft is the total frequency of term t in the > whole > > index, <docID , score > pairs are ID of the document in which term t has > a > > score, and these pairs are listed according to the decreasing order of > > scores. > > > > > > I checked through the documentation, and found scorer class but couldnt > > understand how to use it. > > > > I hope this is a kind of better explanation. > > > > Best. > > Sahin. > > > > > > On Fri, Oct 1, 2010 at 9:22 PM, Erick Erickson <nunezjerry@example.org> > > wrote: > >> > >> I'm not sure what you're asking for. "Score of a term in a document"? Do > >> you > >> mean the amount a term contributed to a search for a particular > document? > >> The frequency of a term in a document? ??? > >> > >> Could you elaborate on what you're trying to do? If you describe the > >> problem > >> you're trying to solve, people can provide better answers. > >> > >> Best > >> Erick > >> > >> On Fri, Oct 1, 2010 at 11:33 AM, Sahin Buyrukbilen < > >> nunezjerry@example.org> wrote: > >> > >> > Hi all, > >> > > >> > I need to retrieve the score of a term in a document? I dont want to > >> > play > >> > different scoring schemes. I just checked my index with Luke and it > >> > shows > >> > me > >> > a score for each term in each document the term exists. So, I need > just > >> > to > >> > get that score. > >> > > >> > Can anybody help me? > >> > > >> > Thank you in advance. > >> > > >> > Sahin. > >> > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: nunezjerry@example.org > > For additional commands, e-mail: nunezjerry@example.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: nunezjerry@example.org > For additional commands, e-mail: nunezjerry@example.org > > --0016364d1b8d4ff24f0491a27dfb-- From java-usernunezjerry@example.org Sat Oct 02 18:49:31 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 99144 invoked from network); 2 Oct 2010 18:49:31 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 2 Oct 2010 18:49:31 -0000 Received: (qmail 31235 invoked by uid 500); 2 Oct 2010 18:49:29 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 31111 invoked by uid 500); 2 Oct 2010 18:49:28 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 31103 invoked by uid 99); 2 Oct 2010 18:49:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 18:49:28 +0000 X-ASF-Spam-Status: No, hits=4.7 required=10.0 tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of nunezjerry@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-wy0-f176.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Oct 2010 18:49:23 +0000 Received: by wye20 with SMTP id 20so975473wye.35 for <nunezjerry@example.org>; Sat, 02 Oct 2010 11:48:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=3v1QU/m03wmV+icLn7sVvrI1o0oZzvDgPNARzV3BJDE=; b=X9skWyuDeo3WuVidisVw69r8Je0tuQtHKBR6yxwmmzX5NdHHF7wJqkP3qnOHG5MJhO 5LPvKAw652OHuU/OahOXSF1Ik8k7qSJmVctKwoxfyfmCdj3bqFd8m7CAs+W57+y0LkPy z0I7NAWEZT2Kv7NK1VYTvoWylgHkBFA3v8rRI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=VFwQ5lDefQj+Hg3sEWqzm+FX2VZNc3dgT4GAczN8Cnaflez/alaeMwBoIq1fDy0utA rMwj8l/kiftEyUjFVSzUYEnmW3dd5hnvmZudUFOEBDXyJW7fcPVVMfUXOmsemv9O1dHn 6oNRCCaY2t24lGJ/ddiEzwBTqPCFUWiKGUsSI= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id o10mr5897659wed.83.1286045338079; Sat, 02 Oct 2010 11:48:58 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sat, 2 Oct 2010 11:48:57 -0700 (PDT) In-Reply-To: <nunezjerry@example.org> References: <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> <nunezjerry@example.org> Date: Sat, 2 Oct 2010 14:48:57 -0400 Message-ID: <nunezjerry@example.org> Subject: Re: How to get the score of a term in a document? From: Sahin Buyrukbilen <nunezjerry@example.org> To: nunezjerry@example.org Content-Type: multipart/alternative; boundary=0016e6d388cf8a93170491a6c353 --0016e6d388cf8a93170491a6c353 Content-Type: text/plain; charset=ISO-8859-1 Hi Mike and all other friends, the code below does almost what I want. The only thing I need to do is to write the <docID, score> pairs in the order with respect to score, not docID. Current implementation does it wrt docID. I will try to solve that problem, if anybody knows how to do that I appreciate if he/she shares with me. I am adding my code just in case anybody needs it. Regards. Sahin. public class YilmazTest { public static void main(String[] args) { try{ BufferedWriter out = new BufferedWriter(new FileWriter("/home/guardian/Lucene/output")); // output file Directory dir = FSDirectory.open(new File("/home/guardian/Lucene/Indexes")); IndexReader reader = IndexReader.open(dir); TermEnum termEnum = reader.terms(); IndexSearcher searcher; searcher = new IndexSearcher(dir, true); System.out.println(reader.numDocs()); while(termEnum.next()){ TermDocs termDocs = reader.termDocs(termEnum.term()); TermQuery tq = new TermQuery(new Term(termEnum.term().field(), termEnum.term().text())); Scorer s = tq.weight(searcher).scorer(reader, true, false); boolean once = true; while(termDocs.next()){ if (once){ out.write(termEnum.term().text() + " "); out.write(termEnum.docFreq() + " "); } s.nextDoc(); out.write("<" + termDocs.doc() + "," + s.score() + "> "); once = false; } out.newLine(); } out.close(); } catch(IOException ex){} } } On Sat, Oct 2, 2010 at 9:42 AM, Sahin Buyrukbilen < nunezjerry@example.org> wrote: > Hi Mike, > > I am already done with walking through the terms, frequencies and the docs > by using termenum, termdocs, and indexreader,. The only thing left is the > scores. I will try your suggestion. hope it works. > > Thank you. > > Sahin. > > On Sat, Oct 2, 2010 at 5:30 AM, Michael McCandless < > nunezjerry@example.org> wrote: > >> It sounds like you can just use Lucene's enum APIs (IndexReader.terms, >> IndexReader.termDocs) to walk the entire index, converting it to your >> format? >> >> I'm not sure how Luke computes the "score"... but maybe you could, for >> every term, make a TermQuery and then directly walk its matching docs >> & scores? You'd have to do something like: >> >> Scorer s = TermQuery.weight(searcher).scorer(reader, true, false); >> >> int docID; >> while((docID = s.nextDoc()) != Scorer.NO_MORE_DOCS) { >> float score = s.score(); >> } >> >> I think? >> >> Mike >> >> On Fri, Oct 1, 2010 at 11:49 PM, Sahin Buyrukbilen >> <nunezjerry@example.org> wrote: >> > Hi Erick, >> > >> > I mean the score of a term in a document (we can think this as a one >> word >> > query) which is calculated by using "Default Similarity". Actually, when >> I >> > walk through my index term-by-term, Luke shows me the number of >> documents in >> > which the term exists. And for each document there is a score field. >> please >> > check the attachment for the screenshot. I am very new to the jargon of >> > Lucene, so I am sorry if I explain things in an incorrect way. >> > >> > My question is: For a term in the index, can we retrieve the value (here >> I >> > say score) calculated by using default similarity? Is this a value which >> is >> > already stored in the index or is it calculated on the fly by Luke >> (since I >> > can only see by using Luke)? >> > >> > My goal is to create an inverted index and write it into a text file in >> the >> > following form: >> > >> > Term t ft Inverted list for t >> > >> ---------------------------------------------------------------------------------- >> > big 2 <2, 0.148> <3, 0.088> >> > in 5 <6, 0.159> <2, 0.143> <5, 0.088> <1, 0.076> >> <4, >> > 0.065> >> > - >> > - >> > - >> > - >> > - >> > so on for all terms. Here ft is the total frequency of term t in the >> whole >> > index, <docID , score > pairs are ID of the document in which term t has >> a >> > score, and these pairs are listed according to the decreasing order of >> > scores. >> > >> > >> > I checked through the documentation, and found scorer class but couldnt >> > understand how to use it. >> > >> > I hope this is a kind of better explanation. >> > >> > Best. >> > Sahin. >> > >> > >> > On Fri, Oct 1, 2010 at 9:22 PM, Erick Erickson <nunezjerry@example.org >> > >> > wrote: >> >> >> >> I'm not sure what you're asking for. "Score of a term in a document"? >> Do >> >> you >> >> mean the amount a term contributed to a search for a particular >> document? >> >> The frequency of a term in a document? ??? >> >> >> >> Could you elaborate on what you're trying to do? If you describe the >> >> problem >> >> you're trying to solve, people can provide better answers. >> >> >> >> Best >> >> Erick >> >> >> >> On Fri, Oct 1, 2010 at 11:33 AM, Sahin Buyrukbilen < >> >> nunezjerry@example.org> wrote: >> >> >> >> > Hi all, >> >> > >> >> > I need to retrieve the score of a term in a document? I dont want to >> >> > play >> >> > different scoring schemes. I just checked my index with Luke and it >> >> > shows >> >> > me >> >> > a score for each term in each document the term exists. So, I need >> just >> >> > to >> >> > get that score. >> >> > >> >> > Can anybody help me? >> >> > >> >> > Thank you in advance. >> >> > >> >> > Sahin. >> >> > >> > >> > >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: nunezjerry@example.org >> > For additional commands, e-mail: nunezjerry@example.org >> > >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: nunezjerry@example.org >> For additional commands, e-mail: nunezjerry@example.org >> >> > --0016e6d388cf8a93170491a6c353-- From java-usernunezjerry@example.org Mon Oct 04 01:59:34 2010 Return-Path: <java-usernunezjerry@example.org> Delivered-To: nunezjerry@example.org Received: (qmail 2216 invoked from network); 4 Oct 2010 01:59:33 -0000 Received: from unknown (HELO mail.apache.org) (83.161.120.155) by 83.161.120.155 with SMTP; 4 Oct 2010 01:59:33 -0000 Received: (qmail 47603 invoked by uid 500); 4 Oct 2010 01:59:31 -0000 Delivered-To: nunezjerry@example.org Received: (qmail 47382 invoked by uid 500); 4 Oct 2010 01:59:31 -0000 Mailing-List: contact nunezjerry@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:nunezjerry@example.org> List-Unsubscribe: <mailto:nunezjerry@example.org> List-Post: <mailto:nunezjerry@example.org> List-Id: <java-user.lucene.apache.org> Reply-To: nunezjerry@example.org Delivered-To: mailing list nunezjerry@example.org Received: (qmail 47374 invoked by uid 99); 4 Oct 2010 01:59:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Oct 2010 01:59:30
{'source': 'Apache Software Foundation Public Mail Archives'}