From lucene-user-carterdavid@example.net Tue Jul 01 14:18:58 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 53967 invoked from network); 1 Jul 2003 14:18:56 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 1 Jul 2003 14:18:56 -0000 Received: (qmail 11599 invoked by uid 97); 1 Jul 2003 14:21:21 -0000 Delivered-To: carterdavid@example.net Received: (qmail 11592 invoked from network); 1 Jul 2003 14:21:20 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 1 Jul 2003 14:21:20 -0000 Received: (qmail 53690 invoked by uid 500); 1 Jul 2003 14:18:53 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 53665 invoked from network); 1 Jul 2003 14:18:52 -0000 Received: from gateway.aidministrator.nl (HELO aidministrator.nl) (83.161.120.155) by daedalus.apache.org with SMTP; 1 Jul 2003 14:18:52 -0000 Received: from home.aidministrator.nl by aidministrator.nl (SMI-8.6/SMI-SVR4) id QAA01888; Tue, 1 Jul 2003 16:19:42 +0200 Received: from suzuka (suzuka [83.161.120.155]) by home.aidministrator.nl (8.11.6+Sun/8.11.6) with SMTP id h61EGGD03763 for <carterdavid@example.net>; Tue, 1 Jul 2003 16:16:16 +0200 (MEST) Message-Id: <carterdavid@example.net> Date: Tue, 1 Jul 2003 16:16:16 +0200 (MEST) From: Jos van der Meer <carterdavid@example.net> Reply-To: Jos van der Meer <carterdavid@example.net> Subject: Lucene + controlled vocabulary + guided exploration To: carterdavid@example.net MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: L4X1r5z0OySfEXh9pgLf2g== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.4.2 SunOS 5.8 sun4u sparc X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N We use Lucene in a way in which we provide a controlled vocabulary to the end-users + continuous feedback about all combinations of concepts that occur. Creating a controlled vocabulary is not harder for you than it is for the average user to construct a good, selective query. However, the benefits of having a controlled vocabulary are much bigger. This is the setup: - have of lot of unstructured documents (locally or on the web) - index them by Lucene. So far nothing new, but now: - create a controlled vocabulary: * For each concept: write down a corresponding Lucene query * Group related concepts An example controlled vocabulary is "abnamro.xml" http://spectacle.aidministrator.nl/spectacle/channel/lucabn/files/abnamro.xml (by default the concept name itself is the Lucene query.) Download Spectacle:Server from http://spectacle.aidministrator.nl/ and offer guided exploration (see below) over the results. You and your end-users will be more succesful in exploration ("find valuable results you were not looking for") and regular finding ("find what you were looking for"). The results on a small set of documents is in: http://spectacle.aidministrator.nl/spectacle/channel/lucabn It's not the best demo ever (..), but it is a start. Do it on your own data, create your own controlled vocabulary and try for yourself. You'll find all source code you'll need in the Spectacle:Server download. You're invited to forward this mail, or to reply to: carterdavid@example.net aidministrator nederland bv - http://www.aidministrator.nl/ prinses julianaplein 14-b, 3817 cs amersfoort, the netherlands tel. 859-312-1121 fax. 859-312-1121 ============================================================================= Background: ============================================================================= * In fact we are using Lucene as a poor man's classifier. A document is considered to contain a concept "A" if the document matches the query specified with "A". This mechanism is not only cheap because Lucene is for free, it is also cheap because the creation the characteristics (in this case a query) is simple and elegant. It takes a lot less time than e.g. assembling a training set for each concept. * Even though it's a poor man's solution, it is surprisingly well performing. The reason is that the query can deviate a lot from the concept itself and a lot of human expertise can be 'implemented' in the query definition. ============================================================================= Guided exploration consists of: ============================================================================= - continuous feedback to the user about the options which limit the amount of matching documents. -- there is no need for the user to type-in anything. Spectacle shows what it knows. -- eliminates the famous "no hits" behaviour of so many traditional sites -- enables guided specialization whenever the current result set is too large. - continuous feedback to the user about the options which lead to alternatives for the current set of matching documents. --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Wed Jul 02 05:22:54 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 94504 invoked from network); 2 Jul 2003 05:22:53 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 05:22:53 -0000 Received: (qmail 8603 invoked by uid 97); 2 Jul 2003 05:25:27 -0000 Delivered-To: carterdavid@example.net Received: (qmail 8596 invoked from network); 2 Jul 2003 05:25:26 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Jul 2003 05:25:26 -0000 Received: (qmail 94106 invoked by uid 500); 2 Jul 2003 05:22:48 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 94086 invoked from network); 2 Jul 2003 05:22:47 -0000 Received: from mail.gmhwh.org (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 05:22:47 -0000 X-Server-Uuid: 1ea73c7c-c7d8-11d5-bae0-0002a564cf8c Message-ID: <carterdavid@example.net> Date: Tue, 01 Jul 2003 23:22:07 -0600 From: "Jim Hargrave" <carterdavid@example.net> To: carterdavid@example.net Subject: Re: Geting exact term positions for each document inside acollect method... MIME-Version: 1.0 X-WSS-ID: 131CB3A0392051-01-02 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Our application indexes and retreieves sentences from a large database. Our= = terms are overlapping characters (n-grams). In order to calculate our custo= m= score we need to know the (relative) position of each n-gram in the matche= d= sentences. I'm currently using a boolen query (each n-ngram in a big 'OR' = statement). I will investigate customizing the query as you suggest.=20 Basically we are using Lucene as a Translation Memeory tool! Pretty cool. = Lucene is wonderful and I think we can use it in many of our linguistic = projects (Terminlogy, concordance, TM etc.). Jim >>> carterdavid@example.net 06/30/03 10:56 AM >>> Jim Hargrave wrote: > I've defined my own collector (I want the raw score before it is = normalized between 1.0 and 0.0). For each document I need to know the the = matching term positions in the document. I've seen the methods in = IndexReader, but how can I access them inside my collect method? Are there = other methods I am missing?=20 No, this information is not available to the hit collector. Why do you need this? If it is only for summaries, then you're probably=20 better off re-tokenizing those few documents that you wish to summarize.=20 If it is for query evaluation, then you're probably better off writing=20 a new class of query (which is non-trivial). Doug --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net =46or additional commands, e-mail: carterdavid@example.net ---------------------------------------------------------------------------= --- This message may contain confidential information, and is intended only for= = the use of the individual(s) to whom it is addressed. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Wed Jul 02 11:09:57 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 11891 invoked from network); 2 Jul 2003 11:09:57 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 11:09:57 -0000 Received: (qmail 16303 invoked by uid 97); 2 Jul 2003 11:12:20 -0000 Delivered-To: carterdavid@example.net Received: (qmail 16296 invoked from network); 2 Jul 2003 11:12:19 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Jul 2003 11:12:19 -0000 Received: (qmail 11577 invoked by uid 500); 2 Jul 2003 11:09:54 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 11564 invoked from network); 2 Jul 2003 11:09:53 -0000 Received: from bay7-f94.bay7.hotmail.com (HELO hotmail.com) (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 11:09:53 -0000 Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Wed, 2 Jul 2003 04:09:54 -0700 Received: from 83.161.120.155 by by7fd.bay7.hotmail.msn.com with HTTP; Wed, 02 Jul 2003 11:09:53 GMT X-Originating-IP: [83.161.120.155] X-Originating-Email: [carterdavid@example.net] From: "anumodh narayanan kutty" <carterdavid@example.net> To: carterdavid@example.net Bcc: Subject: problem - pointing to last list in xml Date: Wed, 02 Jul 2003 11:09:53 +0000 Mime-Version: 1.0 Content-Type: text/plain; format=flowed Message-ID: <carterdavid@example.net> X-OriginalArrivalTime: 02 Jul 2003 11:09:54.0138 (UTC) FILETIME=[76EC7BA0:01C3408A] X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N I am using Lucene for implementing search, My xml file looks like this --------------------------------------------- <?xml version='1.0' encoding='utf-8'?> <address-book> <contact type="individual"> <name>Zane Pasolini</name> <address>999 W. Prince St.</address> <city>New York</city> <province>NY</province> <postalcode>10013</postalcode> <country>USA</country> <telephone>859-312-1121</telephone> </contact> <contact type="business"> <name>SAMOFIX d.o.o.</name> <address>Ilica 47-2</address> <city>Zagreb</city> <province></province> <postalcode>10000</postalcode> <country>Croatia</country> <telephone>385-1-123-4567</telephone> </contact> <contact type="personal"> <name>anumodh</name> <address>Madrid</address> <city>Azber</city> <province></province> <postalcode>680 594</postalcode> <country>Spain</country> <telephone>385-1-123-4567</telephone> </contact> <contact type="personal"> <name>anumodh</name> <address>Madrid</address> <city>Azber</city> <province></province> <postalcode>680 594</postalcode> <country>Spain</country> <telephone>385-1-123-4567</telephone> </contact> </address-book> --------------------------------------------- * I implemented app using Lucene, and if i give search for "Zane" it finds that document , but i am not able retrieve the contents ie "Zane's city etc". Because it is always pointing to the last entity of the xml docs, ie in this case it is pointing to "anumodh's details". * It is not recognzing duplicate or similar records, plz help me to solve this prob, any good example of xml indexing & search Happy Hacking Anumodh _________________________________________________________________ HCL Beanstalk PCs. You could win one. http://server1.msn.co.in/sp03/hclbeanstalktour/index.asp Interested? --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Wed Jul 02 18:46:01 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 96521 invoked from network); 2 Jul 2003 18:45:59 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 18:45:59 -0000 Received: (qmail 6078 invoked by uid 97); 2 Jul 2003 18:48:27 -0000 Delivered-To: carterdavid@example.net Received: (qmail 6069 invoked from network); 2 Jul 2003 18:48:27 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Jul 2003 18:48:27 -0000 Received: (qmail 96209 invoked by uid 500); 2 Jul 2003 18:45:56 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 96186 invoked from network); 2 Jul 2003 18:45:55 -0000 Received: from mail.disney.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 18:45:55 -0000 Received: from Hermes10.corp.disney.com (hermes10.corp.disney.com [83.161.120.155]) by mail.disney.com (Switch-2.2.6/Switch-2.2.4) with ESMTP id h62Ii9928518 for <carterdavid@example.net>; Wed, 2 Jul 2003 11:44:09 -0700 (PDT) Received: from sm-cala-xc03.swna.wdpr.disney.com by hermes.corp.disney.com with ESMTP; Wed, 2 Jul 2003 11:44:47 -0700 Received: from sm-cala-xm11.swna.wdpr.disney.com ([83.161.120.155]) by sm-cala-xc03.swna.wdpr.disney.com with Microsoft SMTPSVC(5.0.2195.5329); Wed, 2 Jul 2003 11:44:36 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Getting an exact field match Date: Wed, 2 Jul 2003 11:44:36 -0700 Message-Id: <carterdavid@example.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Getting an exact field match Thread-Index: AcNAyfytkIlaG/l5SsybNWUuJwAkew== From: "Wilton, Reece" <carterdavid@example.net> To: <carterdavid@example.net> X-OriginalArrivalTime: 02 Jul 2003 18:44:36.0789 (UTC) FILETIME=[FCA6A650:01C340C9] X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Hi, I am indexing XML files. The XML files have a Location element. For example, the Location is /Foo/Bar.html in one of the files. When I update the index, I want to remove the existing document. I search for the Location and delete the existing document like this: Query query =3D QueryParser.parse(location, "LOCATION", new StandardAnalyzer()); Hits hits =3D searcher.search(query); for (int i =3D 0; i < hits.length(); i++) { indexReader.delete(hits.id(i)); } But I never get anything returned from the searcher. I'm passing in the exact value that is in the field. How do I get an exact match of the field? Should I be adding Location as Text or Keyword? I've tried both but can't get it to return what I want. Is the problem because I have slashes ("/") in the field? Does the StandardAnalyzer filter those out or something? Any help is appreciated! Reece --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Wed Jul 02 21:59:31 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 63216 invoked from network); 2 Jul 2003 21:59:31 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 21:59:31 -0000 Received: (qmail 12005 invoked by uid 97); 2 Jul 2003 22:02:00 -0000 Delivered-To: carterdavid@example.net Received: (qmail 11998 invoked from network); 2 Jul 2003 22:02:00 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Jul 2003 22:02:00 -0000 Received: (qmail 62924 invoked by uid 500); 2 Jul 2003 21:59:28 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 62913 invoked from network); 2 Jul 2003 21:59:28 -0000 Received: from fulldegree.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 21:59:28 -0000 Received: from fddell8000 (adsl-64-169-189-2.dsl.snfc21.pacbell.net [83.161.120.155]) by fulldegree.com (8.12.9) id h62LxVjH039968 for <carterdavid@example.net>; Wed, 2 Jul 2003 15:59:31 -0600 (MDT) From: "Lixin Meng" <carterdavid@example.net> To: "Lucene Users List" <carterdavid@example.net> Subject: Any problem with Lucene lucene-1.2-rc4 with JDK 1.4.0_03 Date: Wed, 2 Jul 2003 14:53:47 -0700 Message-ID: <carterdavid@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 In-Reply-To: <carterdavid@example.net> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N I have an indexer (lucene-1.2-rc4) running well under JDK 1.3.1. However, if I switch to JDK 1.4.0_03, after indexing couple hundred documents (~2-3K each), the memory consumption of JVM just shooting up. The whole system basically hangs. Anyone have similar experience with Lucene + JDK 1.4 + Win 2K? Lixin --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Wed Jul 02 22:01:17 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 66141 invoked from network); 2 Jul 2003 22:01:17 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 22:01:17 -0000 Received: (qmail 12051 invoked by uid 97); 2 Jul 2003 22:03:47 -0000 Delivered-To: carterdavid@example.net Received: (qmail 12044 invoked from network); 2 Jul 2003 22:03:46 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Jul 2003 22:03:46 -0000 Received: (qmail 65853 invoked by uid 500); 2 Jul 2003 22:01:15 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 65825 invoked from network); 2 Jul 2003 22:01:14 -0000 Received: from mail.disney.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 22:01:14 -0000 Received: from Hermes10.corp.disney.com (hermes10.corp.disney.com [83.161.120.155]) by mail.disney.com (Switch-2.2.6/Switch-2.2.4) with ESMTP id h62M0g910300 for <carterdavid@example.net>; Wed, 2 Jul 2003 15:00:42 -0700 (PDT) Received: from sm-cala-xc04.swna.wdpr.disney.com by hermes.corp.disney.com with ESMTP; Wed, 2 Jul 2003 15:01:24 -0700 Received: from sm-cala-xm11.swna.wdpr.disney.com ([83.161.120.155]) by sm-cala-xc04.swna.wdpr.disney.com with Microsoft SMTPSVC(5.0.2195.5329); Wed, 2 Jul 2003 15:00:45 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: Results sorted by date instead of score? Date: Wed, 2 Jul 2003 15:00:45 -0700 Message-Id: <carterdavid@example.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Results sorted by date instead of score? Thread-Index: AcNAyfytkIlaG/l5SsybNWUuJwAkewAGx00w From: "Wilton, Reece" <carterdavid@example.net> To: <carterdavid@example.net> X-OriginalArrivalTime: 02 Jul 2003 22:00:46.0077 (UTC) FILETIME=[63B17AD0:01C340E5] X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Search hits come back ordered by score. How do I get my results sorted by the date of the article? I have added the article date as a keyword field. --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Wed Jul 02 23:41:00 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 92020 invoked from network); 2 Jul 2003 23:41:00 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 23:41:00 -0000 Received: (qmail 14540 invoked by uid 97); 2 Jul 2003 23:43:30 -0000 Delivered-To: carterdavid@example.net Received: (qmail 14533 invoked from network); 2 Jul 2003 23:43:30 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Jul 2003 23:43:30 -0000 Received: (qmail 91705 invoked by uid 500); 2 Jul 2003 23:40:57 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 91693 invoked from network); 2 Jul 2003 23:40:57 -0000 Received: from web40907.mail.yahoo.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 23:40:57 -0000 Message-ID: <carterdavid@example.net> Received: from [83.161.120.155] by web40907.mail.yahoo.com via HTTP; Wed, 02 Jul 2003 16:41:05 PDT Date: Wed, 2 Jul 2003 16:41:05 -0700 (PDT) From: Ali Rouhi <carterdavid@example.net> Subject: Query across multiple fields scenario not handled by "MultiFieldQueryParser" To: carterdavid@example.net Cc: carterdavid@example.net, carterdavid@example.net MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Hi I need to perform a search for an expression in multiple fields "as if" they were one field. This is best illustrated by a simple example. Find expression (X AND Y) in fields (F, G). I want this to translate to: ((X in F) OR (X in G)) AND ((Y in F) OR (Y in G)) In other words I want the query to return true *not only* if ((X AND Y) in F) OR ((X AND Y) in G) (1) but *also* the following should give me a true value ((X in F) AND (Y in G)) OR ((X in G) AND (Y in F)) (2) I believe that "MultiFieldQueryParser" just gives me (1) and leaves out the "cross terms" in(2).(If you want a practical example of why one would want to do such a search I would be glad to provide one). Of course I could construct the combination of (1) and (2) manually, but the problem is that I want the "general solution" with a general expression being searched for in multiple fields in the manner of the above example. I have a feeling that this sort of a thing is best done by writing a custom QueryParser.jj. We use Lucene in production code and have great java expertise, but little JavaCC expertise. Suggestions for solving the problem at a higher level than the QueryParser are of course also very welcome. Many Thanks Ali __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Wed Jul 02 23:54:59 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 648 invoked from network); 2 Jul 2003 23:54:59 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 23:54:59 -0000 Received: (qmail 14786 invoked by uid 97); 2 Jul 2003 23:57:30 -0000 Delivered-To: carterdavid@example.net Received: (qmail 14778 invoked from network); 2 Jul 2003 23:57:30 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 2 Jul 2003 23:57:30 -0000 Received: (qmail 353 invoked by uid 500); 2 Jul 2003 23:54:57 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 342 invoked from network); 2 Jul 2003 23:54:57 -0000 Received: from 251.017.dsl.syd.iprimus.net.au (HELO file1.syd.nuix.com.au) (83.161.120.155) by daedalus.apache.org with SMTP; 2 Jul 2003 23:54:57 -0000 Received: from host86.syd.nuix.com.au (host86.syd.nuix.com.au [83.161.120.155]) by file1.syd.nuix.com.au (Postfix) with ESMTP id 33E60B6FEB for <carterdavid@example.net>; Thu, 3 Jul 2003 09:54:48 +1000 (EST) Content-Type: text/plain; charset="iso-8859-1" From: Victor Hadianto <carterdavid@example.net> Organization: NUIX Pty. Ltd. To: "Lucene Users List" <carterdavid@example.net> Subject: Re: Query across multiple fields scenario not handled by "MultiFieldQueryParser" Date: Thu, 3 Jul 2003 09:48:32 +1000 User-Agent: KMail/1.4.3 References: <carterdavid@example.net> In-Reply-To: <carterdavid@example.net> Massage-Id: <carterdavid@example.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-Id: <200307030948.32677@bah> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N > I need to perform a search for an expression in > multiple fields "as if" they were one field. Depending on your time requirment, you can as you have suggested yourself write your own QueryParser. However a simple hack will be to have an "all" field and simple reindex the data in F and G to this "all" field. > little JavaCC expertise. Suggestions for solving the > problem at a higher level than the QueryParser are of > course also very welcome. Well this is definitely not "higher" level than QueryParser :) HTH victor On Thu, 3 Jul 2003 09:41 am, Ali Rouhi wrote: > Hi > This is > best illustrated by a simple example. > > Find expression (X AND Y) in fields (F, G). > > I want this to translate to: > > ((X in F) OR (X in G)) AND ((Y in F) OR (Y in G)) > > In other words I want the query to return true *not > only* if > > ((X AND Y) in F) OR ((X AND Y) in G) > (1) > > but *also* the following should give me a true value > > ((X in F) AND (Y in G)) OR ((X in G) AND (Y in F)) > (2) > > I believe that "MultiFieldQueryParser" just gives me > (1) and leaves out the "cross terms" in(2).(If you > want a practical example of why one would want to do > such a search I would be glad to provide one). > > Of course I could construct the combination of (1) and > (2) manually, but the problem is that I want the > "general solution" with a general expression being > searched for in multiple fields in the manner of the > above example. > > I have a feeling that this sort of a thing is best > done by writing a custom QueryParser.jj. We use Lucene > in production code and have great java expertise, but > > Many Thanks > Ali --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Thu Jul 03 00:22:58 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 24214 invoked from network); 3 Jul 2003 00:22:57 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 00:22:57 -0000 Received: (qmail 15934 invoked by uid 97); 3 Jul 2003 00:25:28 -0000 Delivered-To: carterdavid@example.net Received: (qmail 15927 invoked from network); 3 Jul 2003 00:25:28 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 00:25:28 -0000 Received: (qmail 23848 invoked by uid 500); 3 Jul 2003 00:22:54 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 23833 invoked from network); 3 Jul 2003 00:22:54 -0000 Received: from web40907.mail.yahoo.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 00:22:54 -0000 Message-ID: <carterdavid@example.net> Received: from [83.161.120.155] by web40907.mail.yahoo.com via HTTP; Wed, 02 Jul 2003 17:23:02 PDT Date: Wed, 2 Jul 2003 17:23:02 -0700 (PDT) From: Ali Rouhi <carterdavid@example.net> Subject: Re: Query across multiple fields scenario not handled by "MultiFieldQueryParser" To: Lucene Users List <carterdavid@example.net> In-Reply-To: <200307030948.32677@bah> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N --- Victor Hadianto <carterdavid@example.net> wrote: > > I need to perform a search for an expression in > > multiple fields "as if" they were one field. > > Depending on your time requirment, you can as you > have suggested yourself > write your own QueryParser. However a simple hack > will be to have an "all" > field and simple reindex the data in F and G to this > "all" field. > We have thought of this - but we index an enormous amount of data (Terabytes distributed across many linux boxes) and would prefer to save the disk space. However we may indeed end up doing what you suggest. > > little JavaCC expertise. Suggestions for solving > the > > problem at a higher level than the QueryParser are > of > > course also very welcome. > > Well this is definitely not "higher" level than > QueryParser :) This in itself is useful to know. I can now justify having to learn JavaCC to my boss:). Thanks Ali __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Thu Jul 03 00:27:18 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 31358 invoked from network); 3 Jul 2003 00:27:18 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 00:27:18 -0000 Received: (qmail 16018 invoked by uid 97); 3 Jul 2003 00:29:49 -0000 Delivered-To: carterdavid@example.net Received: (qmail 16011 invoked from network); 3 Jul 2003 00:29:49 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 00:29:49 -0000 Received: (qmail 31056 invoked by uid 500); 3 Jul 2003 00:27:16 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 31044 invoked from network); 3 Jul 2003 00:27:15 -0000 Received: from web40903.mail.yahoo.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 00:27:15 -0000 Message-ID: <carterdavid@example.net> Received: from [83.161.120.155] by web40903.mail.yahoo.com via HTTP; Wed, 02 Jul 2003 17:27:24 PDT Date: Wed, 2 Jul 2003 17:27:24 -0700 (PDT) From: Ali Rouhi <carterdavid@example.net> Subject: Re: Query across multiple fields scenario not handled by "MultiFieldQueryParser" To: Lucene Users List <carterdavid@example.net> In-Reply-To: <carterdavid@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N --- Ali Rouhi <carterdavid@example.net> wrote: > > --- Victor Hadianto <carterdavid@example.net> wrote: > > > I need to perform a search for an expression in > > > multiple fields "as if" they were one field. > > > > Depending on your time requirment, you can as you > > have suggested yourself > > write your own QueryParser. However a simple hack > > will be to have an "all" > > field and simple reindex the data in F and G to > this > > "all" field. > > > > We have thought of this - but we index an enormous > amount of data (Terabytes distributed across many > linux boxes) and would prefer to save the disk > space. > However we may indeed end up doing what you suggest. Just a little clarification to the point I made regarding your suggestion. We do need to preserve the indexes under F and G and be able to search independently on them. This is why having an extra ALL field which contains contents of both F and G would involve extra disc usages. Ali > > > > little JavaCC expertise. Suggestions for solving > > the > > > problem at a higher level than the QueryParser > are > > of > > > course also very welcome. > > > > Well this is definitely not "higher" level than > > QueryParser :) > > This in itself is useful to know. I can now justify > having to learn JavaCC to my boss:). > > Thanks > Ali > > __________________________________ > Do you Yahoo!? > SBC Yahoo! DSL - Now only $29.95 per month! > http://sbc.yahoo.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: > carterdavid@example.net > For additional commands, e-mail: > carterdavid@example.net > __________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Thu Jul 03 08:41:32 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 63846 invoked from network); 3 Jul 2003 08:41:31 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 08:41:31 -0000 Received: (qmail 24523 invoked by uid 97); 3 Jul 2003 08:44:07 -0000 Delivered-To: carterdavid@example.net Received: (qmail 24516 invoked from network); 3 Jul 2003 08:44:06 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 08:44:06 -0000 Received: (qmail 63264 invoked by uid 500); 3 Jul 2003 08:41:25 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 63167 invoked from network); 3 Jul 2003 08:41:25 -0000 Received: from bisoxmsw.blackwell.co.uk (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 08:41:25 -0000 Received: from venus.digimem.net (ns0.digimem.net) by bisoxmsw.blackwell.co.uk (Content Technologies SMTPRS 4.2.5) with SMTP id <carterdavid@example.net>; Thu, 3 Jul 2003 09:41:34 +0100 Received: from venus (venus [83.161.120.155]) by venus.digimem.net (Postfix) with ESMTP id 81F312E27; Thu, 3 Jul 2003 09:41:32 +0100 (BST) Message-ID: <1505915.1057221692398.JavaMail.digimem@venus> Date: Thu, 3 Jul 2003 09:41:32 +0100 (BST) From: Ian Lea <carterdavid@example.net> To: "Wilton, Reece" <carterdavid@example.net> Subject: Re: Getting an exact field match Cc: carterdavid@example.net Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_252_5765853.1057221692392" X-Mailer: digimemmailer v1.0 X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N ------=_Part_252_5765853.1057221692392 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Add Location as Keyword and delete something like this: Term t = new Term("LOCATION", location); indexReader.delete(t); -- Ian. carterdavid@example.net > carterdavid@example.net ("Wilton, Reece") wrote > > Hi, > > I am indexing XML files. The XML files have a Location element. For > example, the Location is /Foo/Bar.html in one of the files. > > When I update the index, I want to remove the existing document. I > search for the Location and delete the existing document like this: > > Query query = QueryParser.parse(location, "LOCATION", new > StandardAnalyzer()); > Hits hits = searcher.search(query); > for (int i = 0; i < hits.length(); i++) { > indexReader.delete(hits.id(i)); > } > > But I never get anything returned from the searcher. I'm passing in the > exact value that is in the field. How do I get an exact match of the > field? Should I be adding Location as Text or Keyword? I've tried both > but can't get it to return what I want. > > Is the problem because I have slashes ("/") in the field? Does the > StandardAnalyzer filter those out or something? > > Any help is appreciated! > Reece > > --------------------------------------------------------------------- > To unsubscribe, e-mail: carterdavid@example.net > For additional commands, e-mail: carterdavid@example.net > > ---------------------------------------------------------------------- Searchable personal storage and archiving from http://www.digimem.net/ ------=_Part_252_5765853.1057221692392 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net ------=_Part_252_5765853.1057221692392-- From lucene-user-carterdavid@example.net Thu Jul 03 12:35:22 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 62600 invoked from network); 3 Jul 2003 12:35:20 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 12:35:20 -0000 Received: (qmail 7238 invoked by uid 97); 3 Jul 2003 12:37:44 -0000 Delivered-To: carterdavid@example.net Received: (qmail 7231 invoked from network); 3 Jul 2003 12:37:43 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 12:37:43 -0000 Received: (qmail 62273 invoked by uid 500); 3 Jul 2003 12:35:17 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 62258 invoked from network); 3 Jul 2003 12:35:17 -0000 Received: from mail.polopoly.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 12:35:17 -0000 Received: from polopoly.com (holmer.intra.polopoly.com [83.161.120.155]) by mail.polopoly.com (8.11.3/8.11.3/SuSE Linux 8.11.1-0.5) with ESMTP id h63CYiJ23077 for <carterdavid@example.net>; Thu, 3 Jul 2003 14:34:45 +0200 Message-ID: <carterdavid@example.net> Date: Thu, 03 Jul 2003 14:36:55 +0200 From: Claes Holmerson <carterdavid@example.net> Organization: Polopoly AB User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0) Gecko/20020529 X-Accept-Language: en-us, en, sv MIME-Version: 1.0 To: carterdavid@example.net Subject: Understanding how indexing works Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Hi, In my job, I have become the new maintainer of a search feature that uses Lucene. I am trying to understand how it works by examining the index it produces. When I list index fields by opening an IndexReader, looping over documents, then looping over fields with Document.fields(), I see a number of fields. However, I don't see the fields that are searchable within this document. If I do IndexReader.terms() directly and then Term.field() on each term, I see fields that no longer should exist since the documents containing them have been deleted. Is this like it should be? How can I easily tell if a term is no longer searchable (or rather successfully found, when searching), apart from actually doing the search? Is there a way to list those terms? Thanks, Claes -- Claes Holmerson Polopoly - Cultivating the information garden Kungsgatan 88, SE-112 27 Stockholm, SWEDEN Direct: 859-312-1121 Mobile: 859-312-1121 Fax: 859-312-1121 carterdavid@example.net, http://www.polopoly.com --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Thu Jul 03 13:27:21 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 87042 invoked from network); 3 Jul 2003 13:27:20 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 13:27:20 -0000 Received: (qmail 8832 invoked by uid 97); 3 Jul 2003 13:29:44 -0000 Delivered-To: carterdavid@example.net Received: (qmail 8825 invoked from network); 3 Jul 2003 13:29:44 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 13:29:44 -0000 Received: (qmail 78860 invoked by uid 500); 3 Jul 2003 13:25:13 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 78804 invoked from network); 3 Jul 2003 13:25:13 -0000 Received: from unknown (HELO isc?mail.infosciences.com) (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 13:25:12 -0000 Received: from Aviran (83.161.120.155 [83.161.120.155]) by isc_mail.infosciences.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2653.13) id N26JPZPM; Thu, 3 Jul 2003 09:28:52 -0400 From: "Aviran Mordo" <carterdavid@example.net> To: "'Lucene Users List'" <carterdavid@example.net> Subject: RE: Results sorted by date instead of score? Date: Thu, 3 Jul 2003 09:25:21 -0400 Message-ID: <004d01c34166$938914d0$6a00a8c0@Aviran> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.4024 Importance: Normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165 In-Reply-To: <carterdavid@example.net> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N You'll need to sort the results after you collected them. There is a project called SortedField in lucene's contribution or sandbox (I don't remember exactly) which will help you sort by any field. -----Original Message----- From: Wilton, Reece [mailto:carterdavid@example.net] Sent: Wednesday, July 02, 2003 6:01 PM To: carterdavid@example.net Subject: Results sorted by date instead of score? Search hits come back ordered by score. How do I get my results sorted by the date of the article? I have added the article date as a keyword field. --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Thu Jul 03 17:02:31 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 24122 invoked from network); 3 Jul 2003 17:02:29 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 17:02:29 -0000 Received: (qmail 16439 invoked by uid 97); 3 Jul 2003 17:04:56 -0000 Delivered-To: carterdavid@example.net Received: (qmail 16432 invoked from network); 3 Jul 2003 17:04:55 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 17:04:55 -0000 Received: (qmail 23857 invoked by uid 500); 3 Jul 2003 17:02:27 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 23846 invoked from network); 3 Jul 2003 17:02:26 -0000 Received: from pony-express.cs.rit.edu (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 17:02:26 -0000 Received: (qmail 16413 invoked from network); 3 Jul 2003 17:02:29 -0000 Received: from unknown (HELO VALUEDDC3E02C5) (83.161.120.155) by pony-express.cs.rit.edu with SMTP; 3 Jul 2003 17:02:29 -0000 From: "Jagdip Singh" <carterdavid@example.net> To: <carterdavid@example.net> Subject: What could be the table structure Date: Thu, 3 Jul 2003 13:03:05 -0400 Message-ID: <000e01c34184$f8438950$f3af1581@VALUEDDC3E02C5> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_000F_01C34163.7131E950" X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2616 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: Normal X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N ------=_NextPart_000_000F_01C34163.7131E950 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Hi, I am new to this mailing list. I am thinking of designing search tool for my web site which has documents (PDF's, DOC, PPT) and HTML, XML pages of course. I want to design basic search engine which can auto index when new item is added and give search results for all including documents and HTML pages. I want to use SQL Server to store index pages and so confused about what should be the table structure to make search engine with Lucene. Please reply if somebody did similar work earlier using SQL Server to store index pages or documents How should I represent output page having search results. Can I get source code of someone's personal search engine using Lucene just to get help to develop for my own personal web page? Sorry if I wrote something wrong as I am new to this list. Thanks, Jagdip ------=_NextPart_000_000F_01C34163.7131E950-- From lucene-user-carterdavid@example.net Thu Jul 03 17:35:36 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 91730 invoked from network); 3 Jul 2003 17:35:34 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 17:35:34 -0000 Received: (qmail 17528 invoked by uid 97); 3 Jul 2003 17:38:01 -0000 Delivered-To: carterdavid@example.net Received: (qmail 17521 invoked from network); 3 Jul 2003 17:38:01 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 17:38:01 -0000 Received: (qmail 91434 invoked by uid 500); 3 Jul 2003 17:35:31 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 91423 invoked from network); 3 Jul 2003 17:35:31 -0000 Received: from unknown (HELO signaturedomains.com) (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 17:35:31 -0000 Received: from anthonyeden.com (berenice.gallery [83.161.120.155]) by signaturedomains.com (8.9.3/8.8.7) with ESMTP id MAA17417 for <carterdavid@example.net>; Thu, 3 Jul 2003 12:32:55 -0400 Message-ID: <carterdavid@example.net> Date: Thu, 03 Jul 2003 13:35:33 -0400 From: Anthony Eden <carterdavid@example.net> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.3.1) Gecko/20030425 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Lucene Users List <carterdavid@example.net> Subject: Re: What could be the table structure References: <000e01c34184$f8438950$f3af1581@VALUEDDC3E02C5> In-Reply-To: <000e01c34184$f8438950$f3af1581@VALUEDDC3E02C5> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N I wrote a Directory implementation which allows you to store indeces in a database. Currently I have only tested it on MySQL but you should be able to get it to work on SQL Server I imagine. If you do then please send me a copy of the DB schema to add in. If you can't get it to work then please send an email and I'll try to help you out. Download: http://www.anthonyeden.com/download/lucene-dbdirectory-1.0.tar.gz http://www.anthonyeden.com/download/lucene-dbdirectory-1.0.zip Sincerely, Anthony Eden Jagdip Singh wrote: > Hi, > I am new to this mailing list. I am thinking of designing search tool > for my web site which has documents (PDF's, DOC, PPT) and HTML, XML > pages of course. > I want to design basic search engine which can auto index when new item > is added and give search results for all including documents and HTML > pages. I want to use SQL Server to store index pages and so confused > about what should be the table structure to make search engine with > Lucene. > Please reply if somebody did similar work earlier using SQL Server to > store index pages or documents > How should I represent output page having search results. Can I get > source code of someone's personal search engine using Lucene just to get > help to develop for my own personal web page? > > Sorry if I wrote something wrong as I am new to this list. > > Thanks, > Jagdip > --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Thu Jul 03 17:42:51 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 11646 invoked from network); 3 Jul 2003 17:42:51 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 17:42:51 -0000 Received: (qmail 17922 invoked by uid 97); 3 Jul 2003 17:45:18 -0000 Delivered-To: carterdavid@example.net Received: (qmail 17915 invoked from network); 3 Jul 2003 17:45:17 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 17:45:17 -0000 Received: (qmail 11325 invoked by uid 500); 3 Jul 2003 17:42:48 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 11312 invoked from network); 3 Jul 2003 17:42:47 -0000 Received: from pony-express.cs.rit.edu (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 17:42:47 -0000 Received: (qmail 21775 invoked from network); 3 Jul 2003 17:42:51 -0000 Received: from unknown (HELO VALUEDDC3E02C5) (83.161.120.155) by pony-express.cs.rit.edu with SMTP; 3 Jul 2003 17:42:51 -0000 From: "Jagdip Singh" <carterdavid@example.net> To: "'Lucene Users List'" <carterdavid@example.net> Subject: RE: What could be the table structure Date: Thu, 3 Jul 2003 13:43:26 -0400 Message-ID: <001601c3418a$9ba526d0$f3af1581@VALUEDDC3E02C5> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook, Build 10.0.2616 X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Importance: Normal In-Reply-To: <carterdavid@example.net> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Thanks Anthony -----Original Message----- From: Anthony Eden [mailto:carterdavid@example.net] Sent: Thursday, July 03, 2003 1:36 PM To: Lucene Users List Subject: Re: What could be the table structure I wrote a Directory implementation which allows you to store indeces in a database. Currently I have only tested it on MySQL but you should be able to get it to work on SQL Server I imagine. If you do then please send me a copy of the DB schema to add in. If you can't get it to work then please send an email and I'll try to help you out. Download: http://www.anthonyeden.com/download/lucene-dbdirectory-1.0.tar.gz http://www.anthonyeden.com/download/lucene-dbdirectory-1.0.zip Sincerely, Anthony Eden Jagdip Singh wrote: > Hi, > I am new to this mailing list. I am thinking of designing search tool > for my web site which has documents (PDF's, DOC, PPT) and HTML, XML > pages of course. > I want to design basic search engine which can auto index when new item > is added and give search results for all including documents and HTML > pages. I want to use SQL Server to store index pages and so confused > about what should be the table structure to make search engine with > Lucene. > Please reply if somebody did similar work earlier using SQL Server to > store index pages or documents > How should I represent output page having search results. Can I get > source code of someone's personal search engine using Lucene just to get > help to develop for my own personal web page? > > Sorry if I wrote something wrong as I am new to this list. > > Thanks, > Jagdip > --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Thu Jul 03 17:49:57 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 26748 invoked from network); 3 Jul 2003 17:49:57 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 17:49:57 -0000 Received: (qmail 19123 invoked by uid 97); 3 Jul 2003 17:52:24 -0000 Delivered-To: carterdavid@example.net Received: (qmail 19111 invoked from network); 3 Jul 2003 17:52:24 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 3 Jul 2003 17:52:24 -0000 Received: (qmail 26400 invoked by uid 500); 3 Jul 2003 17:49:54 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 26328 invoked from network); 3 Jul 2003 17:49:54 -0000 Received: from smtpzilla2.xs4all.nl (83.161.120.155) by daedalus.apache.org with SMTP; 3 Jul 2003 17:49:54 -0000 Received: from as800 (a80-127-237-226.dial.xs4all.nl [83.161.120.155]) by smtpzilla2.xs4all.nl (8.12.9/8.12.9) with ESMTP id h63HnttI086014; Thu, 3 Jul 2003 19:49:56 +0200 (CEST) Content-Type: text/plain; charset="iso-8859-1" From: Ype Kingma <carterdavid@example.net> To: "Lucene Users List" <carterdavid@example.net>, Claes Holmerson <carterdavid@example.net> Subject: Re: Understanding how indexing works Date: Thu, 3 Jul 2003 19:44:22 -0700 User-Agent: KMail/1.4.3 References: <carterdavid@example.net> In-Reply-To: <carterdavid@example.net> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Message-Id: <carterdavid@example.net> X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N Claes, On Thursday 03 July 2003 05:36, Claes Holmerson wrote: > Hi, > > In my job, I have become the new maintainer of a search feature that > uses Lucene. I am trying to understand how it works by examining the > index it produces. > > When I list index fields by opening an IndexReader, looping over > documents, then looping over fields with Document.fields(), I see a > number of fields. However, I don't see the fields that are searchable > within this document. You only see the stored fields that way. > If I do IndexReader.terms() directly and then Term.field() on each term= , > I see fields that no longer should exist since the documents containing > them have been deleted. Is this like it should be? Until you optimize the index, yes. > How can I easily tell if a term is no longer searchable (or rather > successfully found, when searching), apart from actually doing the Optimize before checking the index for the term. > search? Is there a way to list those terms? Yes, but it is slow: for each term, check whether the associated documents are all deleted. Kind regards, Ype --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Fri Jul 04 14:58:44 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 94506 invoked from network); 4 Jul 2003 14:58:44 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 4 Jul 2003 14:58:44 -0000 Received: (qmail 16565 invoked by uid 97); 4 Jul 2003 15:01:11 -0000 Delivered-To: carterdavid@example.net Received: (qmail 16558 invoked from network); 4 Jul 2003 15:01:11 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 4 Jul 2003 15:01:11 -0000 Received: (qmail 94222 invoked by uid 500); 4 Jul 2003 14:58:41 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 94210 invoked from network); 4 Jul 2003 14:58:40 -0000 Received: from unknown (HELO itapoa.datasul.net) (83.161.120.155) by daedalus.apache.org with SMTP; 4 Jul 2003 14:58:40 -0000 Received: from 83.161.120.155 by itapoa.datasul.net (InterScan E-Mail VirusWall NT); Fri, 04 Jul 2003 11:56:21 -0300 Received: by itapoa with Internet Mail Service (5.5.2653.19) id <3GR8XHHL>; Fri, 4 Jul 2003 11:51:21 -0300 Message-ID: <418A5677CF05084F8D50372532E8AF3202770A1A@itapoa> From: Flavio Eduardo de Cordova <carterdavid@example.net> To: Lucene Users List <carterdavid@example.net> Subject: Lucene 1.3rc1 on Java 1.3 Date: Fri, 4 Jul 2003 11:51:20 -0300 Importance: high X-Priority: 1 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N People.. I have a class that searchs for some documents in an existing index (see the code below). It works fine when I run it on a 1.4 JVM but it throws a NullPointerException (or crashes the JVM, in another computer) when I run it on a 1.3 JVM. I'm using Lucene 1.3rc1 and the exception/crash occurs in the "search" method. Is it a bug ? Should I be doing something else ? Have you been using 1.3rc1 on 1.3 JVMs ? public static void searchIndex(String path, String field, String queryStr, String retField) { try { Searcher searcher = new IndexSearcher(path); MyOwnAnalyser analyser = new MyOwnAnalyser(); Query query = QueryParser.parse(queryStr, field, analyser); Hits hits = searcher.search(query); //, filter); for(int i=0; i<hits.length(); i++) { Document doc = hits.doc(i); float score = hits.score(i); System.out.println("Found document \"" + doc.get(retField)); } } catch (Exception e1) { e1.printStackTrace(); } } --------------------------------------------------------------------- To unsubscribe, e-mail: carterdavid@example.net For additional commands, e-mail: carterdavid@example.net From lucene-user-carterdavid@example.net Fri Jul 04 15:30:51 2003 Return-Path: <lucene-user-carterdavid@example.net> Delivered-To: carterdavid@example.net Received: (qmail 35470 invoked from network); 4 Jul 2003 15:30:51 -0000 Received: from exchange.sun.com (83.161.120.155) by daedalus.apache.org with SMTP; 4 Jul 2003 15:30:51 -0000 Received: (qmail 17234 invoked by uid 97); 4 Jul 2003 15:33:19 -0000 Delivered-To: carterdavid@example.net Received: (qmail 17227 invoked from network); 4 Jul 2003 15:33:18 -0000 Received: from daedalus.apache.org (HELO apache.org) (83.161.120.155) by nagoya.betaversion.org with SMTP; 4 Jul 2003 15:33:18 -0000 Received: (qmail 35194 invoked by uid 500); 4 Jul 2003 15:30:48 -0000 Mailing-List: contact carterdavid@example.net; run by ezmlm Precedence: bulk List-Unsubscribe: <mailto:carterdavid@example.net> List-Subscribe: <mailto:carterdavid@example.net> List-Help: <mailto:carterdavid@example.net> List-Post: <mailto:carterdavid@example.net> List-Id: "Lucene Users List" <lucene-user.jakarta.apache.org> Reply-To: "Lucene Users List" <carterdavid@example.net> Delivered-To: mailing list carterdavid@example.net Received: (qmail 35177 invoked from network); 4 Jul 2003 15:30:48 -0000 Received: from s149v112.servlets.net (83.161.120.155) by daedalus.apache.org with SMTP; 4 Jul 2003 15:30:48 -0000 Received: from jivesoftware.com (carterdavid@example.net [83.161.120.155]) (authenticated) by s149v112.servlets.net (8.11.6/8.11.6) with ESMTP id h64FUoE12654 for <carterdavid@example.net>; Fri, 4 Jul 2003 08:30:50 -0700 Message-ID: <carterdavid@example.net> Date: Fri, 04 Jul 2003 11:30:51 -0400 From: Bruce Ritchie <carterdavid@example.net> Organization: Jive Software User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.4) Gecko/20030624 X-Accept-Language: en-ca, en MIME-Version: 1.0 To: Lucene Users List <carterdavid@example.net> Subject: Re: Lucene 1.3rc1 on Java 1.3 References: <418A5677CF05084F8D50372532E8AF3202770A1A@itapoa> In-Reply-To: <418A5677CF05084F8D50372532E8AF3202770A1A@itapoa> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms040400070702010504040001" X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N --------------ms040400070702010504040001 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Flavio Eduardo de Cordova wrote: > I have a class that searchs for some documents in an existing index > (see the code below). It works fine when I run it on a 1.4 JVM but it throws > a NullPointerException (or crashes the JVM, in another computer) when I run > it on a 1.3 JVM. > I'm using Lucene 1.3rc1 and the exception/crash occurs in the > "search" method. > > Is it a bug ? Should I be doing something else ? Have you been using > 1.3rc1 on 1.3 JVMs ? I believe the bug only affects JDK 1.3.0, I've never seen this issue on any JDK newer than and including
{'source': 'Apache Software Foundation Public Mail Archives'}