From lucene-user-hannahhall@example.com Tue Apr 01 08:18:07 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 12309 invoked from network); 1 Apr 2003 08:18:05 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 08:18:05 -0000 Received: (qmail 17634 invoked by uid 97); 1 Apr 2003 08:20:05 -0000 Delivered-To: hannahhall@example.com Received: (qmail 17627 invoked from network); 1 Apr 2003 08:20:04 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 1 Apr 2003 08:20:04 -0000 Received: (qmail 12042 invoked by uid 500); 1 Apr 2003 08:18:01 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 12028 invoked from network); 1 Apr 2003 08:18:00 -0000 Received: from innocence.interface-business.de (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 08:18:00 -0000 Received: from rathole (rathole-srv.interface-business.de [83.161.120.155]) by innocence.interface-business.de (8.12.9/8.12.6/ifb evision: 1.24 $) with SMTP id h318IB1Q001174 for <hannahhall@example.com>; Tue, 1 Apr 2003 10:18:11 +0200 (CEST) (envelope-from hannahhall@example.com) Received: from chuck.interface-business.de ([83.161.120.155]) by rathole; Tue, 01 Apr 2003 10:18:49 +0200 (MEST) Received: from buddelflink (buddelflink.interface-business.de [83.161.120.155]) by chuck.interface-business.de (8.12.9/8.12.5) with ESMTP id h318I62K006688 for <hannahhall@example.com>; Tue, 1 Apr 2003 10:18:06 +0200 (CEST) (envelope-from hannahhall@example.com) MIME-Version: 1.0 To: hannahhall@example.com Subject: Re: Problems on massive indexing References: <hannahhall@example.com> Message-ID: <oprmxyofaahawgj6@mail> Content-Type: text/plain; charset=iso-8859-15; format=flowed From: Kristian Hermsdorf <hannahhall@example.com> Organization: Interface Business Date: Tue, 01 Apr 2003 10:18:05 +0200 In-Reply-To: <hannahhall@example.com> User-Agent: Opera7.03/Win32 M2 build 2670 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by chuck.interface-business.de id h318I62K006688 X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Hi =A0 I also got the IndexOutOfBoundException while optimizing the index (index= -=20 size about 1GB, 50 Docs with 25 fields each). (optimizing was called via merging of RamDirectoy to FSDirectory). The problem was that the FieldsReader tried to read more fields than=20 existed ... .I've no glue how to fix it ... =A0 bye =A0 Kristian On Tue, 01 Apr 2003 08:55:23 +0200, Omar Cal <hannahhall@example.com> wrote= : > Hello, i'm a newby of Lucene. > > I've the following scenario: > -450.000 xml files and text files > -5 indexes, two stored and three unstored > -lucene library 1.2 (tested also 1.3RC) > > When i try to index the material i've an IndexOutOfBoundException in th= e=20 > call to the index.optimize() after two hours of indexing.I know there i= s=20 > the bug 14355 and i think it could be the responsable for that exceptio= n. > > I've tried also to index the whole material in subsequent runs but the=20 > problem seems to depend on the number of the documents. > > I've tried to set the maxFieldLength at its maximum but nothing appened. > > If i split the material in "trunks" of about 20.000 - 30.000 documents = in=20 > each directory, the problem doesn't appear. Obviously i've to repeat th= e=20 > searches for each "trunk" (directory). > > Anyone out there with a similar scenario? Other solutions? > > Thanks, Omar > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hannahhall@example.com > For additional commands, e-mail: hannahhall@example.com > > > --=20 ACRONYM: Acronym Causing Recursion, Obviously Numbing Your Mind =A0 Kristian Hermsdorf interface:projects gmbh =09 Tollkewitzer Stra=DFe 49 =09 01277 Dresden =09 tel.: 985.967.7188 mail: hannahhall@example.com priv: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Tue Apr 01 20:04:05 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 85030 invoked from network); 1 Apr 2003 20:04:04 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:04:04 -0000 Received: (qmail 15256 invoked by uid 97); 1 Apr 2003 20:05:57 -0000 Delivered-To: hannahhall@example.com Received: (qmail 15249 invoked from network); 1 Apr 2003 20:05:56 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 1 Apr 2003 20:05:56 -0000 Received: (qmail 84748 invoked by uid 500); 1 Apr 2003 20:04:01 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 84736 invoked from network); 1 Apr 2003 20:04:00 -0000 Received: from mail.ideorlando.org (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:04:00 -0000 Received: by mail.ideorlando.org (Postfix, from userid 8) id 1D7D3367D5; Tue, 1 Apr 2003 20:04:50 +0000 (UTC) X-Scanned-By: AMaViS at a badly configured site. Received: from ROUTARDBW (unknown [83.161.120.155]) by mail.ideorlando.org (Postfix) with SMTP id E09E0367CE for <hannahhall@example.com>; Tue, 1 Apr 2003 20:04:48 +0000 (UTC) From: "Rob Outar" <hannahhall@example.com> To: "Lucene Users List" <hannahhall@example.com> Subject: Indexing Growth Date: Tue, 1 Apr 2003 15:05:28 -0500 Message-ID: <hannahhall@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 In-Reply-To: <hannahhall@example.com> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Hi all, Will the index grow based on queries alone? I build my index, then run several queries against it and afterwards I check the size of the index and in some cases it has grown quite a bit although I did not add anything??? Anyhow please let me know the cases when the index will grow. Thanks, Rob --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Tue Apr 01 20:07:08 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 93235 invoked from network); 1 Apr 2003 20:07:07 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:07:07 -0000 Received: (qmail 15325 invoked by uid 97); 1 Apr 2003 20:09:00 -0000 Delivered-To: hannahhall@example.com Received: (qmail 15318 invoked from network); 1 Apr 2003 20:08:59 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 1 Apr 2003 20:08:59 -0000 Received: (qmail 92922 invoked by uid 500); 1 Apr 2003 20:07:04 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 92911 invoked from network); 1 Apr 2003 20:07:04 -0000 Received: from web12708.mail.yahoo.com (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:07:04 -0000 Message-ID: <hannahhall@example.com> Received: from [83.161.120.155] by web12708.mail.yahoo.com via HTTP; Tue, 01 Apr 2003 12:07:09 PST Date: Tue, 1 Apr 2003 12:07:09 -0800 (PST) From: Otis Gospodnetic <hannahhall@example.com> Subject: Re: Indexing Growth To: Lucene Users List <hannahhall@example.com> In-Reply-To: <hannahhall@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Only when you add new documents to it. Otis --- Rob Outar <hannahhall@example.com> wrote: > Hi all, > > Will the index grow based on queries alone? I build my index, then > run > several queries against it and afterwards I check the size of the > index and > in some cases it has grown quite a bit although I did not add > anything??? > > Anyhow please let me know the cases when the index will grow. > > Thanks, > > Rob > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hannahhall@example.com > For additional commands, e-mail: hannahhall@example.com > __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - File online, calculators, forms, and more http://platinum.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Tue Apr 01 20:11:59 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 7593 invoked from network); 1 Apr 2003 20:11:59 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:11:59 -0000 Received: (qmail 15482 invoked by uid 97); 1 Apr 2003 20:13:51 -0000 Delivered-To: hannahhall@example.com Received: (qmail 15475 invoked from network); 1 Apr 2003 20:13:50 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 1 Apr 2003 20:13:50 -0000 Received: (qmail 7325 invoked by uid 500); 1 Apr 2003 20:11:55 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 7311 invoked from network); 1 Apr 2003 20:11:55 -0000 Received: from mail.ideorlando.org (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:11:55 -0000 Received: by mail.ideorlando.org (Postfix, from userid 8) id 93818367D2; Tue, 1 Apr 2003 20:12:44 +0000 (UTC) X-Scanned-By: AMaViS at a badly configured site. Received: from ROUTARDBW (unknown [83.161.120.155]) by mail.ideorlando.org (Postfix) with SMTP id 49D0F367CE for <hannahhall@example.com>; Tue, 1 Apr 2003 20:12:43 +0000 (UTC) From: "Rob Outar" <hannahhall@example.com> To: "Lucene Users List" <hannahhall@example.com> Subject: RE: Indexing Growth Date: Tue, 1 Apr 2003 15:13:22 -0500 Message-ID: <hannahhall@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 In-Reply-To: <hannahhall@example.com> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Dang I must be doing something crazy cause all my client app does is search and the index size increases. I do not add anything. Thanks, Rob -----Original Message----- From: Otis Gospodnetic [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:07 PM To: Lucene Users List Subject: Re: Indexing Growth Only when you add new documents to it. Otis --- Rob Outar <hannahhall@example.com> wrote: > Hi all, > > Will the index grow based on queries alone? I build my index, then > run > several queries against it and afterwards I check the size of the > index and > in some cases it has grown quite a bit although I did not add > anything??? > > Anyhow please let me know the cases when the index will grow. > > Thanks, > > Rob > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hannahhall@example.com > For additional commands, e-mail: hannahhall@example.com > __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - File online, calculators, forms, and more http://platinum.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Tue Apr 01 20:22:03 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 26989 invoked from network); 1 Apr 2003 20:22:02 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:22:02 -0000 Received: (qmail 15714 invoked by uid 97); 1 Apr 2003 20:23:55 -0000 Delivered-To: hannahhall@example.com Received: (qmail 15707 invoked from network); 1 Apr 2003 20:23:55 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 1 Apr 2003 20:23:55 -0000 Received: (qmail 26765 invoked by uid 500); 1 Apr 2003 20:21:59 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 26744 invoked from network); 1 Apr 2003 20:21:59 -0000 Received: from unknown (HELO www18.dixiesys.com) (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:21:59 -0000 Received: from Lissus (s160.dial2.sne.nac.net [83.161.120.155]) (authenticated (0 bits)) by www18.dixiesys.com (8.11.6/8.11.6) with ESMTP id h31KM1d31094 for <hannahhall@example.com>; Tue, 1 Apr 2003 14:22:02 -0600 Reply-To: <hannahhall@example.com> From: "Alex Murzaku" <hannahhall@example.com> To: "'Lucene Users List'" <hannahhall@example.com> Subject: RE: Indexing Growth Date: Tue, 1 Apr 2003 15:21:52 -0500 Organization: LISSUS llc Message-ID: <000001c2f88c$599a27a0$6501000a@Lissus> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.4024 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 In-Reply-To: <hannahhall@example.com> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N I don't know if I remember this correctly: I think for every query (term) is created a file but the file should disappear after the query is completed. -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:13 PM To: Lucene Users List Subject: RE: Indexing Growth Dang I must be doing something crazy cause all my client app does is search and the index size increases. I do not add anything. Thanks, Rob -----Original Message----- From: Otis Gospodnetic [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:07 PM To: Lucene Users List Subject: Re: Indexing Growth Only when you add new documents to it. Otis --- Rob Outar <hannahhall@example.com> wrote: > Hi all, > > Will the index grow based on queries alone? I build my index, then > run several queries against it and afterwards I check the size of the > index and > in some cases it has grown quite a bit although I did not add > anything??? > > Anyhow please let me know the cases when the index will grow. > > Thanks, > > Rob > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hannahhall@example.com > For additional commands, e-mail: hannahhall@example.com > __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - File online, calculators, forms, and more http://platinum.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Tue Apr 01 20:31:16 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 52075 invoked from network); 1 Apr 2003 20:31:16 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:31:16 -0000 Received: (qmail 15960 invoked by uid 97); 1 Apr 2003 20:33:09 -0000 Delivered-To: hannahhall@example.com Received: (qmail 15953 invoked from network); 1 Apr 2003 20:33:08 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 1 Apr 2003 20:33:08 -0000 Received: (qmail 49545 invoked by uid 500); 1 Apr 2003 20:30:41 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 49478 invoked from network); 1 Apr 2003 20:30:41 -0000 Received: from mail.ideorlando.org (83.161.120.155) by daedalus.apache.org with SMTP; 1 Apr 2003 20:30:41 -0000 Received: by mail.ideorlando.org (Postfix, from userid 8) id 13B89367CE; Tue, 1 Apr 2003 20:31:31 +0000 (UTC) X-Scanned-By: AMaViS at a badly configured site. Received: from ROUTARDBW (unknown [83.161.120.155]) by mail.ideorlando.org (Postfix) with SMTP id D4B26367C4; Tue, 1 Apr 2003 20:31:29 +0000 (UTC) From: "Rob Outar" <hannahhall@example.com> To: "Lucene Users List" <hannahhall@example.com>, <hannahhall@example.com> Subject: RE: Indexing Growth Date: Tue, 1 Apr 2003 15:32:08 -0500 Message-ID: <hannahhall@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 In-Reply-To: <000001c2f88c$599a27a0$6501000a@Lissus> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N I reuse the same searcher, analyzer and Query object I don't think that should cause the problem. Thanks, Rob -----Original Message----- From: Alex Murzaku [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:22 PM To: 'Lucene Users List' Subject: RE: Indexing Growth I don't know if I remember this correctly: I think for every query (term) is created a file but the file should disappear after the query is completed. -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:13 PM To: Lucene Users List Subject: RE: Indexing Growth Dang I must be doing something crazy cause all my client app does is search and the index size increases. I do not add anything. Thanks, Rob -----Original Message----- From: Otis Gospodnetic [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:07 PM To: Lucene Users List Subject: Re: Indexing Growth Only when you add new documents to it. Otis --- Rob Outar <hannahhall@example.com> wrote: > Hi all, > > Will the index grow based on queries alone? I build my index, then > run several queries against it and afterwards I check the size of the > index and > in some cases it has grown quite a bit although I did not add > anything??? > > Anyhow please let me know the cases when the index will grow. > > Thanks, > > Rob > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hannahhall@example.com > For additional commands, e-mail: hannahhall@example.com > __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - File online, calculators, forms, and more http://platinum.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Wed Apr 02 00:15:33 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 73259 invoked from network); 2 Apr 2003 00:15:32 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 00:15:32 -0000 Received: (qmail 28992 invoked by uid 97); 2 Apr 2003 00:17:28 -0000 Delivered-To: hannahhall@example.com Received: (qmail 28985 invoked from network); 2 Apr 2003 00:17:27 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Apr 2003 00:17:27 -0000 Received: (qmail 72965 invoked by uid 500); 2 Apr 2003 00:15:29 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 72952 invoked from network); 2 Apr 2003 00:15:28 -0000 Received: from mta.micromuse.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 00:15:28 -0000 Received: from micromuse.com ([83.161.120.155]) (authenticated (0 bits)) by mta.micromuse.com (Switch-2.2.5/Switch-2.2.5) with ESMTP id h320FUj03085 (using TLSv1/SSLv3 with cipher RC4-MD5 (128 bits) verified NO) for <hannahhall@example.com>; Wed, 2 Apr 2003 01:15:32 +0100 Message-ID: <hannahhall@example.com> Date: Tue, 01 Apr 2003 16:15:30 -0800 From: David Spencer <hannahhall@example.com> Organization: Micromuse User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3) Gecko/20030312 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Lucene Users List <hannahhall@example.com> Subject: Re: I need a list of the indexed words References: <000801c2f7c2$314fe1b0$1a35ae86@fido> In-Reply-To: <000801c2f7c2$314fe1b0$1a35ae86@fido> Content-Type: multipart/alternative; boundary="------------020009060901090508090402" X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N --------------020009060901090508090402 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit jcrowell wrote: >Thanks for responding. Are you referring to the solution under the title: > >"How do I retrieve all the values of a particular field that exists within >an index, across all documents" ? > Here's some code that might do what you want. It's shows the frequency of each term also. Args are "-i INDEX" and optional "-f FIELD". I haven't tested it outside my tree (to see if it's clean of dependencies) but it looks reasonable at a glance. package com.tropo.lucene; import org.apache.lucene.analysis.*; import org.apache.lucene.index.*; import org.apache.lucene.document.*; import java.io.*; import java.util.*; /** * Call IndexReader.terms(), termPositions(). */ public class DumpTerms { private DumpTerms() { } /** * */ public static void main(String[] args) throws Throwable { String name = "index"; String field = null; for ( int i = 0; i< args.length; i++) { if ( args[i].equals( "-i")) name = args[ ++i]; else if ( args[i].equals( "-f")) field = args[ ++i]; /* else if ( args[i].equals( "-v")) v = args[ ++i]; */ } o.println( "Opening "+ name); final IndexReader r = IndexReader.open( Rammer.convertMaybe( name)); final TermEnum te = r.terms(); final Set s =new TreeSet(); final List lis = new LinkedList(); final int docs = r.numDocs(); o.println( "Docs: " + docs); while ( te.next()) { Term t = te.term(); if ( field != null && ! field.equals( t.field())) continue; s.add( t.field()); /* o.println( "field=" +t.field() + " text=" + t.text() + " freq="+ te.docFreq()); */ lis.add( new Object[] { t, new Integer( te.docFreq())}); /* TermPositions tp = r.termPositions( t); while ( tp.next()) { int f = tp.freq(); o.println( "\t" + "doc=" + tp.doc() + " freq=" +f); for ( int i = 0; i < f; i++) o.println( "\t\tnextp=" + tp.nextPosition()); } tp.close(); */ } o.println( "Terms: " + lis.size()); Collections.sort( lis, new ICmp()); Iterator it = lis.iterator(); int[] lens = new int[ 30]; while ( it.hasNext()) { Object[] ar = (Object[]) it.next(); Term t = (Term) ar[0]; int freq = ((Integer)ar[1]).intValue(); int len = t.text().length(); if ( len >= lens.length) lens[ lens.length-1]++; else lens[ len]++; o.print( fill( t.text(), 16) + " " + fill( "" + freq, 6)); int pc = (freq * 100); pc += (docs/2); pc /= docs; o.print( " " +pc + "%"); if ( ! t.field().equals( "contents")) o.println( " " + t.field()); else o.println(); } o.println(); int tot_terms = lis.size(); for ( int i = 0; i< lens.length; i++) { int pc = lens[ i] * 100; pc += tot_terms/2; pc /= tot_terms; o.println( "" + i + ". " + lens[ i] + " " + pc + "%"); } r.close(); /* o.println(); Iterator it = s.iterator(); while ( it.hasNext()) o.println( it.next()); */ } /** * */ private static String fill( String s, int w) { int len = s.length(); for ( int i = len; i < w; i++) s += " "; return s; } private static class ICmp implements Comparator { public int compare( Object a, Object b) { Object[] a1 = (Object[]) a; Object[] b1 = (Object[]) b; int d1 = ((Integer)a1[1]).intValue(); int d2 = ((Integer)b1[1]).intValue(); if ( d1 < d2) return 1; if ( d1 > d2) return -1; return 0; } } static PrintStream o = System.out; } > >Thanks, > >Jon > > > >-----Original Message----- >From: Otis Gospodnetic [mailto:hannahhall@example.com] >Sent: Monday, March 31, 2003 3:02 PM >To: Lucene Users List >Subject: Re: I need a list of the indexed words > > >Please see the FAQ at jGuru, the answer is there. > >Otis > >--- jcrowell <hannahhall@example.com> wrote: > > >>I have a very simple problem: I need to get a list of the words that >>will result in a hit if searched on. Should be simple, but I'm not >>quite sure >>where to start. >> >>Thanks, >> >>Jon >> >> >>--------------------------------------------------------------------- >>To unsubscribe, e-mail: hannahhall@example.com >>For additional commands, e-mail: hannahhall@example.com >> >> >> > > >__________________________________________________ >Do you Yahoo!? >Yahoo! Platinum - Watch CBS' NCAA March Madness, live on your desktop! >http://platinum.yahoo.com > >--------------------------------------------------------------------- >To unsubscribe, e-mail: hannahhall@example.com >For additional commands, e-mail: hannahhall@example.com > > >--------------------------------------------------------------------- >To unsubscribe, e-mail: hannahhall@example.com >For additional commands, e-mail: hannahhall@example.com > > > --------------020009060901090508090402-- From lucene-user-hannahhall@example.com Wed Apr 02 09:44:27 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 41222 invoked from network); 2 Apr 2003 09:44:26 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 09:44:26 -0000 Received: (qmail 9981 invoked by uid 97); 2 Apr 2003 09:46:27 -0000 Delivered-To: hannahhall@example.com Received: (qmail 9974 invoked from network); 2 Apr 2003 09:46:26 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Apr 2003 09:46:26 -0000 Received: (qmail 40546 invoked by uid 500); 2 Apr 2003 09:44:18 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 40520 invoked from network); 2 Apr 2003 09:44:17 -0000 Received: from unknown (HELO newgen.co.in) (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 09:44:17 -0000 Received: from nhweb.newgen.co.in ([83.161.120.155]) by newgen.co.in (8.11.6/8.11.6) with ESMTP id h329hcG01085; Wed, 2 Apr 2003 15:13:40 +0530 Received: from kamal (ws132.newgen.co.in [83.161.120.155]) by nhweb.newgen.co.in (8.12.3/8.12.3/Debian -4) with SMTP id h32A1H8V032149; Wed, 2 Apr 2003 15:31:21 +0530 Message-ID: <002701c2f8fc$5a25a9e0$8405a8c0@kamal> From: "Amit Kapur" <hannahhall@example.com> To: <hannahhall@example.com> Cc: <hannahhall@example.com> Subject: Behaviour of Lucene during Stress/Scalability Test Date: Wed, 2 Apr 2003 15:13:40 +0530 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0024_01C2F92A.710C9D70" X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 5.00.2919.6700 X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N ------=_NextPart_000_0024_01C2F92A.710C9D70 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable hi everybody I am trying to index documents using Lucene generating about 30 MB of = index (Optimized) which can be raised to about 100 MB or More ( but that = would be on a high end server machine). Description of Current Case: #---Each Document has four fields (One Text field, and 3 other Keyword = Fields).=20 #---The analyzer is based on a StopFilter and a PorterStemFilter. #---I am using a Compaq PIII, 128 MB RAM, 650 MHz.=20 #---mergeFactor is set to 25, and I am optimizing the index after adding = about 20 Documents. #---Using Lucene Release 1.2 Problem Faced After adding about 4000 Documents generating an index of 30 MB, I = initially got an error saying, **** couldn't rename segments.new to = segments **** after which the IndexReader or the IndexWriter to the = current index couldnot be opened. Then I changed a couple of settings,=20 #---mergeFactor=3D20 and Optimize was called after ever 10 documents. #---Using Lucene Release 1.3 Problem Faced After adding about 1500 Documents generating an index of 10 MB, I = initially got an error saying, **** F:\Program Files\OmniDocs = Server\ftstest\_3cf.fnm (Too many open files)**** after which the = IndexWriter to the current index couldnot be opened. Now my requirement needs to have a much much larger index (practically) = and I am actually at the point where these errors are coming = unpredictably.=20 Please if anyone could guide me on this ASAP. Thanx in advance Regards Amit PS: I have already read articles in the mail archieve = http://hannahhall@example.com/msg02815.html.= ------=_NextPart_000_0024_01C2F92A.710C9D70-- From lucene-user-hannahhall@example.com Wed Apr 02 13:49:37 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 22614 invoked from network); 2 Apr 2003 13:49:37 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 13:49:37 -0000 Received: (qmail 17848 invoked by uid 97); 2 Apr 2003 13:51:26 -0000 Delivered-To: hannahhall@example.com Received: (qmail 17841 invoked from network); 2 Apr 2003 13:51:25 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Apr 2003 13:51:25 -0000 Received: (qmail 22336 invoked by uid 500); 2 Apr 2003 13:49:33 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 22325 invoked from network); 2 Apr 2003 13:49:33 -0000 Received: from mail.ideorlando.org (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 13:49:33 -0000 Received: by mail.ideorlando.org (Postfix, from userid 8) id 6EFA0367D5; Wed, 2 Apr 2003 13:50:14 +0000 (UTC) X-Scanned-By: AMaViS at a badly configured site. Received: from ROUTARDBW (unknown [83.161.120.155]) by mail.ideorlando.org (Postfix) with SMTP id 0F84F367D4 for <hannahhall@example.com>; Wed, 2 Apr 2003 13:50:13 +0000 (UTC) From: "Rob Outar" <hannahhall@example.com> To: "Lucene Users List" <hannahhall@example.com> Subject: RE: Indexing Growth Date: Wed, 2 Apr 2003 08:50:56 -0500 Message-ID: <hannahhall@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 1 (Highest) X-MSMail-Priority: High X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: High In-Reply-To: <hannahhall@example.com> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Hi all, This is too odd and I do not even know where to start. We built a Windows Explorer type tool that indexes all files in a "sabdboxed" file system. Each Lucene document contains stuff like path, parent directory, last modified date, file_lock etc.. When we display the files in a given directory through the tool we query the index about 5 times for each file in the repository, this is done so we can display all attributes in the index about that file. So for example if there are 5 files in the directory, each file has 6 attributes that means about 30 term queries are executed. The initial index when build it about 10.4megs, after accessing about 3 or 4 directories the index size increased to over 100megs, and we did not add anything!! All we are doing is querying!! Yesterday after querying became ungodly slow, we looked at the index size it had grown from 10megs to 1.5GB (granted we tested the tool all morning). But I have no idea why the index is growing like this. ANY help would be greatly appreciated. Thanks, Rob -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:32 PM To: Lucene Users List; hannahhall@example.com Subject: RE: Indexing Growth I reuse the same searcher, analyzer and Query object I don't think that should cause the problem. Thanks, Rob -----Original Message----- From: Alex Murzaku [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:22 PM To: 'Lucene Users List' Subject: RE: Indexing Growth I don't know if I remember this correctly: I think for every query (term) is created a file but the file should disappear after the query is completed. -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:13 PM To: Lucene Users List Subject: RE: Indexing Growth Dang I must be doing something crazy cause all my client app does is search and the index size increases. I do not add anything. Thanks, Rob -----Original Message----- From: Otis Gospodnetic [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:07 PM To: Lucene Users List Subject: Re: Indexing Growth Only when you add new documents to it. Otis --- Rob Outar <hannahhall@example.com> wrote: > Hi all, > > Will the index grow based on queries alone? I build my index, then > run several queries against it and afterwards I check the size of the > index and > in some cases it has grown quite a bit although I did not add > anything??? > > Anyhow please let me know the cases when the index will grow. > > Thanks, > > Rob > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hannahhall@example.com > For additional commands, e-mail: hannahhall@example.com > __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - File online, calculators, forms, and more http://platinum.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Wed Apr 02 14:19:53 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 2363 invoked from network); 2 Apr 2003 14:19:53 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:19:53 -0000 Received: (qmail 18858 invoked by uid 97); 2 Apr 2003 14:21:42 -0000 Delivered-To: hannahhall@example.com Received: (qmail 18851 invoked from network); 2 Apr 2003 14:21:41 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Apr 2003 14:21:41 -0000 Received: (qmail 1263 invoked by uid 500); 2 Apr 2003 14:19:32 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 1188 invoked from network); 2 Apr 2003 14:19:29 -0000 Received: from mail.ideorlando.org (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:19:29 -0000 Received: by mail.ideorlando.org (Postfix, from userid 8) id 0DF20367DD; Wed, 2 Apr 2003 14:19:43 +0000 (UTC) X-Scanned-By: AMaViS at a badly configured site. Received: from ROUTARDBW (unknown [83.161.120.155]) by mail.ideorlando.org (Postfix) with SMTP id 6EBAF367DD for <hannahhall@example.com>; Wed, 2 Apr 2003 14:19:41 +0000 (UTC) From: "Rob Outar" <hannahhall@example.com> To: "Lucene Users List" <hannahhall@example.com> Subject: RE: Indexing Growth Date: Wed, 2 Apr 2003 09:20:24 -0500 Message-ID: <hannahhall@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: Normal In-Reply-To: <hannahhall@example.com> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Additional info on the problem, the index contains several 1kb files and several files that have different names, but the same file size. It looks like the files that comprise the index are being duplicated causing the index to become huge. Thanks, Rob -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Wednesday, April 02, 2003 8:51 AM To: Lucene Users List Subject: RE: Indexing Growth Importance: High Hi all, This is too odd and I do not even know where to start. We built a Windows Explorer type tool that indexes all files in a "sabdboxed" file system. Each Lucene document contains stuff like path, parent directory, last modified date, file_lock etc.. When we display the files in a given directory through the tool we query the index about 5 times for each file in the repository, this is done so we can display all attributes in the index about that file. So for example if there are 5 files in the directory, each file has 6 attributes that means about 30 term queries are executed. The initial index when build it about 10.4megs, after accessing about 3 or 4 directories the index size increased to over 100megs, and we did not add anything!! All we are doing is querying!! Yesterday after querying became ungodly slow, we looked at the index size it had grown from 10megs to 1.5GB (granted we tested the tool all morning). But I have no idea why the index is growing like this. ANY help would be greatly appreciated. Thanks, Rob -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:32 PM To: Lucene Users List; hannahhall@example.com Subject: RE: Indexing Growth I reuse the same searcher, analyzer and Query object I don't think that should cause the problem. Thanks, Rob -----Original Message----- From: Alex Murzaku [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:22 PM To: 'Lucene Users List' Subject: RE: Indexing Growth I don't know if I remember this correctly: I think for every query (term) is created a file but the file should disappear after the query is completed. -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:13 PM To: Lucene Users List Subject: RE: Indexing Growth Dang I must be doing something crazy cause all my client app does is search and the index size increases. I do not add anything. Thanks, Rob -----Original Message----- From: Otis Gospodnetic [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:07 PM To: Lucene Users List Subject: Re: Indexing Growth Only when you add new documents to it. Otis --- Rob Outar <hannahhall@example.com> wrote: > Hi all, > > Will the index grow based on queries alone? I build my index, then > run several queries against it and afterwards I check the size of the > index and > in some cases it has grown quite a bit although I did not add > anything??? > > Anyhow please let me know the cases when the index will grow. > > Thanks, > > Rob > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hannahhall@example.com > For additional commands, e-mail: hannahhall@example.com > __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - File online, calculators, forms, and more http://platinum.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Wed Apr 02 14:20:21 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 3900 invoked from network); 2 Apr 2003 14:20:20 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:20:20 -0000 Received: (qmail 18878 invoked by uid 97); 2 Apr 2003 14:22:08 -0000 Delivered-To: hannahhall@example.com Received: (qmail 18871 invoked from network); 2 Apr 2003 14:22:08 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Apr 2003 14:22:08 -0000 Received: (qmail 2526 invoked by uid 500); 2 Apr 2003 14:19:55 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 2477 invoked from network); 2 Apr 2003 14:19:54 -0000 Received: from mail2.cos.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:19:54 -0000 Received: from ntmailer.cos.com (ntmailer [83.161.120.155]) by mail2.cos.com (Postfix) with ESMTP id 3E586222EF2 for <hannahhall@example.com>; Wed, 2 Apr 2003 09:19:55 -0500 (EST) Received: by ntmailer.cos.com with Internet Mail Service (5.5.2653.19) id <HYNZA2BH>; Wed, 2 Apr 2003 09:21:59 -0500 Received: from cos.com (barry.cos.com [83.161.120.155]) by ntmailer.cos.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id HYNZA2BG; Wed, 2 Apr 2003 09:21:54 -0500 From: Michael Barry <hannahhall@example.com> To: Lucene Users List <hannahhall@example.com> Message-ID: <hannahhall@example.com> Date: Wed, 02 Apr 2003 09:19:46 -0500 User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4a) Gecko/20030401 X-Accept-Language: en-us, en MIME-Version: 1.0 Subject: Re: Indexing Growth References: <hannahhall@example.com> In-Reply-To: <hannahhall@example.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Sounds like you either have an indexer that's run amok (maybe a background process that's continually re-indexing your sandbox - or expanding outside your sandbox) or your Query code is doing more than querying. It's not behaviour I've seen. Without a snippet of Query code, it's going to be hard to help. Rob Outar wrote: >Hi all, > > This is too odd and I do not even know where to start. We built a Windows >Explorer type tool that indexes all files in a "sabdboxed" file system. >Each Lucene document contains stuff like path, parent directory, last >modified date, file_lock etc.. When we display the files in a given >directory through the tool we query the index about 5 times for each file in >the repository, this is done so we can display all attributes in the index >about that file. So for example if there are 5 files in the directory, each >file has 6 attributes that means about 30 term queries are executed. The >initial index when build it about 10.4megs, after accessing about 3 or 4 >directories the index size increased to over 100megs, and we did not add >anything!! All we are doing is querying!! Yesterday after querying became >ungodly slow, we looked at the index size it had grown from 10megs to 1.5GB >(granted we tested the tool all morning). But I have no idea why the index >is growing like this. ANY help would be greatly appreciated. > > >Thanks, > >Rob > > --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Wed Apr 02 14:24:17 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 16952 invoked from network); 2 Apr 2003 14:24:17 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:24:17 -0000 Received: (qmail 19042 invoked by uid 97); 2 Apr 2003 14:26:06 -0000 Delivered-To: hannahhall@example.com Received: (qmail 19035 invoked from network); 2 Apr 2003 14:26:05 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Apr 2003 14:26:05 -0000 Received: (qmail 10546 invoked by uid 500); 2 Apr 2003 14:22:17 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 10491 invoked from network); 2 Apr 2003 14:22:16 -0000 Received: from unknown (HELO www18.dixiesys.com) (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:22:16 -0000 Received: from Lissus (s87.dial2.sne.nac.net [83.161.120.155]) (authenticated (0 bits)) by www18.dixiesys.com (8.11.6/8.11.6) with ESMTP id h32EMFp24405 for <hannahhall@example.com>; Wed, 2 Apr 2003 08:22:15 -0600 Reply-To: <hannahhall@example.com> From: "Alex Murzaku" <hannahhall@example.com> To: "'Lucene Users List'" <hannahhall@example.com> Subject: RE: Indexing Growth Date: Wed, 2 Apr 2003 09:22:07 -0500 Organization: LISSUS llc Message-ID: <000201c2f923$41cb4f90$6501000a@Lissus> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.4024 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: Normal In-Reply-To: <hannahhall@example.com> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Hi Rob, After you build the initial index (10.4M) you can see that the directory contains some files - those are the index files. I am sure that the size and number of those initial files doesn't change while you query Lucene. When you say the index grows, I assume you are referring to other files that are added to the index directory therefore increasing the size of the directory and not the initial index files. One experiment you could do is to inspect the index directory after each query. This should give you an idea of what is going on. Also, it might help sending some code snippets to the list to get more informed help... Alex -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Wednesday, April 02, 2003 8:51 AM To: Lucene Users List Subject: RE: Indexing Growth Importance: High Hi all, This is too odd and I do not even know where to start. We built a Windows Explorer type tool that indexes all files in a "sabdboxed" file system. Each Lucene document contains stuff like path, parent directory, last modified date, file_lock etc.. When we display the files in a given directory through the tool we query the index about 5 times for each file in the repository, this is done so we can display all attributes in the index about that file. So for example if there are 5 files in the directory, each file has 6 attributes that means about 30 term queries are executed. The initial index when build it about 10.4megs, after accessing about 3 or 4 directories the index size increased to over 100megs, and we did not add anything!! All we are doing is querying!! Yesterday after querying became ungodly slow, we looked at the index size it had grown from 10megs to 1.5GB (granted we tested the tool all morning). But I have no idea why the index is growing like this. ANY help would be greatly appreciated. Thanks, Rob -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:32 PM To: Lucene Users List; hannahhall@example.com Subject: RE: Indexing Growth I reuse the same searcher, analyzer and Query object I don't think that should cause the problem. Thanks, Rob -----Original Message----- From: Alex Murzaku [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:22 PM To: 'Lucene Users List' Subject: RE: Indexing Growth I don't know if I remember this correctly: I think for every query (term) is created a file but the file should disappear after the query is completed. -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:13 PM To: Lucene Users List Subject: RE: Indexing Growth Dang I must be doing something crazy cause all my client app does is search and the index size increases. I do not add anything. Thanks, Rob -----Original Message----- From: Otis Gospodnetic [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:07 PM To: Lucene Users List Subject: Re: Indexing Growth Only when you add new documents to it. Otis --- Rob Outar <hannahhall@example.com> wrote: > Hi all, > > Will the index grow based on queries alone? I build my index, then > run several queries against it and afterwards I check the size of the > index and in some cases it has grown quite a bit although I did not > add anything??? > > Anyhow please let me know the cases when the index will grow. > > Thanks, > > Rob > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: hannahhall@example.com > For additional commands, e-mail: hannahhall@example.com > __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - File online, calculators, forms, and more http://platinum.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Wed Apr 02 14:27:18 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 26474 invoked from network); 2 Apr 2003 14:27:18 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:27:18 -0000 Received: (qmail 19174 invoked by uid 97); 2 Apr 2003 14:29:07 -0000 Delivered-To: hannahhall@example.com Received: (qmail 19167 invoked from network); 2 Apr 2003 14:29:07 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Apr 2003 14:29:07 -0000 Received: (qmail 18823 invoked by uid 500); 2 Apr 2003 14:24:50 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 18780 invoked from network); 2 Apr 2003 14:24:50 -0000 Received: from mail.ideorlando.org (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:24:50 -0000 Received: by mail.ideorlando.org (Postfix, from userid 8) id 8D9DB367D8; Wed, 2 Apr 2003 14:25:31 +0000 (UTC) X-Scanned-By: AMaViS at a badly configured site. Received: from ROUTARDBW (unknown [83.161.120.155]) by mail.ideorlando.org (Postfix) with SMTP id 2401B367D5 for <hannahhall@example.com>; Wed, 2 Apr 2003 14:25:30 +0000 (UTC) From: "Rob Outar" <hannahhall@example.com> To: "Lucene Users List" <hannahhall@example.com> Subject: RE: Indexing Growth Date: Wed, 2 Apr 2003 09:26:13 -0500 Message-ID: <hannahhall@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2911.0) X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: Normal In-Reply-To: <hannahhall@example.com> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Just about everything calls getValue: public synchronized String getValue(String key, File file) throws ParseException, IOException { Document doc = getDocument(file); return doc.get(key.toLowerCase()); } which calls get document: private synchronized Document getDocument(File file) throws MalformedURLException, IOException { checkForIndexChange(); Term t = new Term(PATH, file.toURI().toString().toLowerCase()); TermQuery tQ = new TermQuery(t); Hits hits = this.searcher.search(tQ); if (hits.length() == 1) { return hits.doc(0); } //this should never happen, cannot have a URL that returns 2 hits //that would mean the same file has been indexed twice else { return null; } Thanks, Rob -----Original Message----- From: Michael Barry [mailto:hannahhall@example.com] Sent: Wednesday, April 02, 2003 9:20 AM To: Lucene Users List Subject: Re: Indexing Growth Sounds like you either have an indexer that's run amok (maybe a background process that's continually re-indexing your sandbox - or expanding outside your sandbox) or your Query code is doing more than querying. It's not behaviour I've seen. Without a snippet of Query code, it's going to be hard to help. Rob Outar wrote: >Hi all, > > This is too odd and I do not even know where to start. We built a Windows >Explorer type tool that indexes all files in a "sabdboxed" file system. >Each Lucene document contains stuff like path, parent directory, last >modified date, file_lock etc.. When we display the files in a given >directory through the tool we query the index about 5 times for each file in >the repository, this is done so we can display all attributes in the index >about that file. So for example if there are 5 files in the directory, each >file has 6 attributes that means about 30 term queries are executed. The >initial index when build it about 10.4megs, after accessing about 3 or 4 >directories the index size increased to over 100megs, and we did not add >anything!! All we are doing is querying!! Yesterday after querying became >ungodly slow, we looked at the index size it had grown from 10megs to 1.5GB >(granted we tested the tool all morning). But I have no idea why the index >is growing like this. ANY help would be greatly appreciated. > > >Thanks, > >Rob > > --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com --------------------------------------------------------------------- To unsubscribe, e-mail: hannahhall@example.com For additional commands, e-mail: hannahhall@example.com From lucene-user-hannahhall@example.com Wed Apr 02 14:28:24 2003 Return-Path: <lucene-user-hannahhall@example.com> Delivered-To: hannahhall@example.com Received: (qmail 29952 invoked from network); 2 Apr 2003 14:28:24 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:28:24 -0000 Received: (qmail 19215 invoked by uid 97); 2 Apr 2003 14:30:13 -0000 Delivered-To: hannahhall@example.com Received: (qmail 19208 invoked from network); 2 Apr 2003 14:30:12 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Apr 2003 14:30:12 -0000 Received: (qmail 19474 invoked by uid 500); 2 Apr 2003 14:25:06 -0000 Mailing-List: contact hannahhall@example.com; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:hannahhall@example.com> List-Subscribe: <mailto:hannahhall@example.com> List-Help: <mailto:hannahhall@example.com> List-Post: <mailto:hannahhall@example.com> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <hannahhall@example.com> Delivered-To: mailing list hannahhall@example.com Received: (qmail 19419 invoked from network); 2 Apr 2003 14:25:05 -0000 Received: from insws8502.gs.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Apr 2003 14:25:05 -0000 Received: from insvs8502.inz.gs.com (unknown [83.161.120.155]) by insws8502.gs.com (Postfix) with ESMTP id 2830B1C5A6 for <hannahhall@example.com>; Wed, 2 Apr 2003 09:25:06 -0500 (EST) Received: from gsnmp02es.ny.fw.gs.com by insvs8502.inz.gs.com with ESMTP for hannahhall@example.com; Wed, 2 Apr 2003 09:25:06 -0500 Received: by gsnmp02es.ny.fw.gs.com with Internet Mail Service (5.5.2655.55) id <2D50P48A>; Wed, 2 Apr 2003 09:25:13 -0500 Message-Id: <hannahhall@example.com> From: "Lichtner, Guglielmo" <hannahhall@example.com> To: 'Lucene Users List' <hannahhall@example.com> Subject: RE: Indexing Growth Date: Wed, 2 Apr 2003 09:25:11 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2655.55) Content-Type: text/plain; charset="iso-8859-1" X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Sounds pretty scary. Please send the code because it's hard to believe (but then most good bugs are ..) -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Wednesday, April 02, 2003 8:51 AM To: Lucene Users List Subject: RE: Indexing Growth Importance: High Hi all, This is too odd and I do not even know where to start. We built a Windows Explorer type tool that indexes all files in a "sabdboxed" file system. Each Lucene document contains stuff like path, parent directory, last modified date, file_lock etc.. When we display the files in a given directory through the tool we query the index about 5 times for each file in the repository, this is done so we can display all attributes in the index about that file. So for example if there are 5 files in the directory, each file has 6 attributes that means about 30 term queries are executed. The initial index when build it about 10.4megs, after accessing about 3 or 4 directories the index size increased to over 100megs, and we did not add anything!! All we are doing is querying!! Yesterday after querying became ungodly slow, we looked at the index size it had grown from 10megs to 1.5GB (granted we tested the tool all morning). But I have no idea why the index is growing like this. ANY help would be greatly appreciated. Thanks, Rob -----Original Message----- From: Rob Outar [mailto:hannahhall@example.com] Sent: Tuesday, April 01, 2003 3:32 PM To: Lucene Users List; hannahhall@example.com Subject: RE: Indexing Growth I reuse the same searcher, analyzer and Query object I don't think that should cause the problem. Thanks, Rob -----Original Message----- From: Alex Murzaku
{'source': 'Apache Software Foundation Public Mail Archives'}