From generalyharris@example.net Fri Apr 04 02:45:48 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 72572 invoked from network); 4 Apr 2008 02:45:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Apr 2008 02:45:48 -0000 Received: (qmail 83695 invoked by uid 500); 4 Apr 2008 02:45:47 -0000 Delivered-To: yharris@example.net Received: (qmail 83673 invoked by uid 500); 4 Apr 2008 02:45:47 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 83660 invoked by uid 99); 4 Apr 2008 02:45:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Apr 2008 19:45:47 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mslinux.101tec.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2008 02:45:03 +0000 Received: from localhost (localhost [83.161.120.155]) by mslinux.101tec.com (Postfix) with ESMTP id 3D085294012 for <yharris@example.net>; Fri, 4 Apr 2008 04:45:11 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.4.2 (20060627) at 101tec.com Received: from jemez.carrieriq.com (CarrierIQ-FWA002-untru.sjz.navisite.com [83.161.120.155]) by mslinux.101tec.com (Postfix) with ESMTP id A84AD29400B for <yharris@example.net>; Fri, 4 Apr 2008 04:44:37 +0200 (CEST) Message-Id: <yharris@example.net> From: Stefan Groschupf <yharris@example.net> To: yharris@example.net In-Reply-To: <yharris@example.net> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Subject: Re: Lucene-based Distributed Index Leveraging Hadoop Date: Thu, 3 Apr 2008 19:44:34 -0700 References: <yharris@example.net> X-Mailer: Apple Mail (2.919.2) X-Virus-Checked: Checked by ClamAV on apache.org Hi All, we are also very much interested in such a system and actually have to realize such a system for an project within the next 3 month. I would prefer to work on a open source solution instead of doing another one behind closed doors, though we would need to start coding pretty soon. We have 3 fulltime developers we could contribute for this time to such a project. I'm happy to do all the organisational work like setting up the complete infrastructure etc to get it started. I suggest we start with an sourceforge project since this is fast to setup and if we qualify for apache as an lucene or hadoop subproject migrate there later, or is it easy to start a apache incubator project? We might just need a nice name for the project. Doug, any idea? :-) Should we start from scratch or with a code contribution? Someone still want to contribute its implementation? Thanks. Stefan On Feb 6, 2008, at 10:57 AM, Ning Li wrote: > There have been several proposals for a Lucene-based distributed index > architecture. > 1) Doug Cutting's "Index Server Project Proposal" at > http://yharris@example.net/msg00338.html > 2) Solr's "Distributed Search" at > http://wiki.apache.org/solr/DistributedSearch > 3) Mark Butler's "Distributed Lucene" at > http://wiki.apache.org/hadoop/DistributedLucene > > We have also been working on a Lucene-based distributed index > architecture. > Our design differs from the above proposals in the way it leverages > Hadoop > as much as possible. In particular, HDFS is used to reliably store > Lucene > instances, Map/Reduce is used to analyze documents and update Lucene > instances > in parallel, and Hadoop's IPC framework is used. Our design is > geared for > applications that require a highly scalable index and where batch > updates > to each Lucene instance are acceptable (verses finer-grained > document at > a time updates). > > We have a working implementation of our design and are in the process > of evaluating its performance. An overview of our design is provided > below. > We welcome feedback and would like to know if you are interested in > working > on it. If so, we would be happy to make the code publicly available. > At the > same time, we would like to collaborate with people working on > existing > proposals and see if we can consolidate our efforts. > > TERMINOLOGY > A distributed "index" is partitioned into "shards". Each shard > corresponds > to > a Lucene instance and contains a disjoint subset of the documents in > the > index. > Each shard is stored in HDFS and served by one or more "shard > servers". Here > we only talk about a single distributed index, but in practice > multiple > indexes > can be supported. > > A "master" keeps track of the shard servers and the shards being > served by > them. An "application" updates and queries the global index through an > "index client". An index client communicates with the shard servers to > execute a query. > > KEY RPC METHODS > This section lists the key RPC methods in our design. To simplify the > discussion, some of their parameters have been omitted. > > On the Shard Servers > // Execute a query on this shard server's Lucene instance. > // This method is called by an index client. > SearchResults search(Query query); > > On the Master > // Tell the master to update the shards, i.e., Lucene instances. > // This method is called by an index client. > boolean updateShards(Configuration conf); > > // Ask the master where the shards are located. > // This method is called by an index client. > LocatedShards getShardLocations(); > > // Send a heartbeat to the master. This method is called by a > // shard server. In the response, the master informs the > // shard server when to switch to a newer version of the index. > ShardServerCommand sendHeartbeat(); > > QUERYING THE INDEX > To query the index, an application sends a search request to an index > client. > The index client then calls the shard server search() method for > each shard > of the index, merges the results and returns them to the > application. The > index client caches the mapping between shards and shard servers by > periodically calling the master's getShardLocations() method. > > UPDATING THE INDEX USING MAP/REDUCE > To update the index, an application sends an update request to an > index > client. > The index client then calls the master's updateShards() method, which > schedules > a Map/Reduce job to update the index. The Map/Reduce job updates the > shards > in > parallel and copies the new index files of each shard (i.e., Lucene > instance) > to HDFS. > > The updateShards() method includes a "configuration", which provides > information for updating the shards. More specifically, the > configuration > includes the following information: > - Input path. This provides the location of updated documents, > e.g., HDFS > files or directories, or HBase tables. > - Input formatter. This specifies how to format the input documents. > - Analysis. This defines the analyzer to use on the input. The > analyzer > determines whether a document is being inserted, updated, or > deleted. > For > inserts or updates, the analyzer also converts each input > document into > a Lucene document. > > The Map phase of the Map/Reduce job formats and analyzes the input (in > parallel), while the Reduce phase collects and applies the updates > to each > Lucene instance (again in parallel). The updates are applied using > the local > file system where a Reduce task runs and then copied back to HDFS. For > example, > if the updates caused a new Lucene segment to be created, the new > segment > would be created on the local file system first, and then copied > back to > HDFS. > > When the Map/Reduce job completes, a "new version" of the index is > ready to > be > queried. It is important to note that the new version of the index > is not > derived from scratch. By leveraging Lucene's update algorithm, the new > version > of each Lucene instance will share as many files as possible as the > previous > version. > > ENSURING INDEX CONSISTENCY > At any point in time, an index client always has a consistent view > of the > shards in the index. The results of a search query include either > all or > none > of a recent update to the index. The details of the algorithm to > accomplish > this are omitted here, but the basic flow is pretty simple. > > After the Map/Reduce job to update the shards completes, the master > will > tell > each shard server to "prepare" the new version of the index. After > all the > shard servers have responded affirmatively to the "prepare" message, > the new > > index is ready to be queried. An index client will then lazily learn > about > the new index when it makes its next getShardLocations() call to the > master. > > In essence, a lazy two-phase commit protocol is used, with "prepare" > and > "commit" messages piggybacked on heartbeats. After a shard has > switched to > the new index, the Lucene files in the old index that are no longer > needed > can safely be deleted. > > ACHIEVING FAULT-TOLERANCE > We rely on the fault-tolerance of Map/Reduce to guarantee that an > index > update > will eventually succeed. All shards are stored in HDFS and can be > read by > any > shard server in a cluster. For a given shard, if one of its shard > servers > dies, > new search requests are handled by its surviving shard servers. To > ensure > that > there is always enough coverage for a shard, the master will > instruct other > shard servers to take over the shards of a dead shard server. > > PERFORMANCE ISSUES > Currently, each shard server reads a shard directly from HDFS. > Experiments > have shown that this approach does not perform very well, with HDFS > causing > Lucene to slow down fairly dramatically (by well over 5x when data > blocks > are > accessed over the network). Consequently, we are exploring different > ways to > leverage the fault tolerance of HDFS and, at the same time, work > around its > performance problems. One simple alternative is to add a local file > system > cache on each shard server. Another alternative is to modify HDFS so > that an > application has more control over where to store the primary and > replicas of > an HDFS block. This feature may be useful for other HDFS > applications (e.g., > HBase). We would like to collaborate with other people who are > interested in > adding this feature to HDFS. > > > Regards, > Ning Li ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 101tec Inc. Menlo Park, California, USA http://www.101tec.com From generalyharris@example.net Fri Apr 04 02:51:48 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 76562 invoked from network); 4 Apr 2008 02:51:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Apr 2008 02:51:48 -0000 Received: (qmail 87018 invoked by uid 500); 4 Apr 2008 02:51:47 -0000 Delivered-To: yharris@example.net Received: (qmail 86993 invoked by uid 500); 4 Apr 2008 02:51:47 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 86982 invoked by uid 99); 4 Apr 2008 02:51:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Apr 2008 19:51:47 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mslinux.101tec.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2008 02:51:05 +0000 Received: from localhost (localhost [83.161.120.155]) by mslinux.101tec.com (Postfix) with ESMTP id 60A7E29400D; Fri, 4 Apr 2008 04:51:16 +0200 (CEST) X-Virus-Scanned: by amavisd-new-2.4.2 (20060627) at 101tec.com Received: from jemez.carrieriq.com (CarrierIQ-FWA002-untru.sjz.navisite.com [83.161.120.155]) by mslinux.101tec.com (Postfix) with ESMTP id 054C429400B; Fri, 4 Apr 2008 04:51:07 +0200 (CEST) Cc: yharris@example.net Message-Id: <yharris@example.net> From: Stefan Groschupf <yharris@example.net> To: Stefan Groschupf <yharris@example.net> In-Reply-To: <yharris@example.net> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v919.2) Subject: Re: Lucene-based Distributed Index Leveraging Hadoop Date: Thu, 3 Apr 2008 19:51:06 -0700 References: <yharris@example.net> <yharris@example.net> X-Mailer: Apple Mail (2.919.2) X-Virus-Checked: Checked by ClamAV on apache.org > Should we start from scratch or with a code contribution? > Someone still want to contribute its implementation? I just noticed - to late though - Ning already contributed the code to hadoop. So I guess my question should be rephrased what is the idea of moving this into a own project? From generalyharris@example.net Fri Apr 04 13:01:07 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 9108 invoked from network); 4 Apr 2008 13:01:06 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Apr 2008 13:01:06 -0000 Received: (qmail 71129 invoked by uid 500); 4 Apr 2008 13:00:58 -0000 Delivered-To: yharris@example.net Received: (qmail 71054 invoked by uid 500); 4 Apr 2008 13:00:57 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 71001 invoked by uid 99); 4 Apr 2008 13:00:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2008 06:00:57 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Apr 2008 13:00:03 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <yharris@example.net>) id 1JhlWp-00007t-0j for yharris@example.net; Fri, 04 Apr 2008 06:00:23 -0700 Message-ID: <yharris@example.net> Date: Fri, 4 Apr 2008 06:00:23 -0700 (PDT) From: and999 <yharris@example.net> To: yharris@example.net Subject: Problem with Russian Language in Lucene 83.161.120.155 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Nabble-From: yharris@example.net X-Virus-Checked: Checked by ClamAV on apache.org Hi all! I am indexing Russian text with that code. I have a problem, when I try to search for words in different cases. Search is productive, only if the word is in the same form, as in the initial (indexing) text. Help me please. (Whether there can be it because Words in RussianStemmer.cs are stored in the coding KOI8-r?) versions: Lucene.Net 83.161.120.155, Snowball.Net 83.161.120.155=20 using System; using System.Collections.Generic; using System.Text; using Lucene.Net.Documents; using Lucene.Net.Index; using Lucene.Net.Analysis.Snowball; using Lucene.Net.Search; using Lucene.Net.QueryParsers; using System.Text.RegularExpressions; using Lucene.Net.Analysis; class Program { static void Main(string[] args) { string index_path =3D @"D:\index\"; System.IO.StreamReader reader; reader =3D new System.IO.StreamReader(new System.IO.FileStream(@"d:\source\67165.html", System.IO.FileMode.Open, System.IO.FileAccess.Read), System.Text.Encoding.Default); reader =3D new System.IO.StreamReader(reader.BaseStream, reader.CurrentEncoding); String title =3D Regex.Replace(reader.ReadLine(), "<[^>]*>", ""= ); String fullText =3D Regex.Replace(reader.ReadToEnd(), "<[^>]*>"= , ""); Document doc =3D new Document(); doc.Add(new Field("message_title", title, Field.Store.YES, Field.Index.TOKENIZED)); doc.Add(new Field("message_text", fullText, Field.Store.NO, Field.Index.TOKENIZED)); Analyzer a =3D new SnowballAnalyzer("Russian"); bool create_key =3D true; IndexWriter iw =3D new IndexWriter(index_path, a, create_key); iw.AddDocument(doc); iw.Optimize(); iw.Close(); =20 String query_string =3D "=D0=AF=D0=BA=D1=83=D1=82=D0=B8=D1=8F"; Search(query_string, index_path); query_string =3D "=D0=AF=D0=BA=D1=83=D1=82=D0=B8=D0=B8"; Search(query_string, index_path); query_string =3D "=D1=82=D1=83=D1=80=D0=B8=D1=81=D1=82=D1=8B"; Search(query_string, index_path); query_string =3D "=D1=82=D1=83=D1=80=D0=B8=D1=81=D1=82=D0=BE=D0= =B2"; Search(query_string, index_path); Console.ReadKey(); } static void Search(string query_string, string index_path) { Searcher search =3D new IndexSearcher(index_path); Analyzer a =3D new SnowballAnalyzer("Russian"); QueryParser parser =3D new QueryParser("message_text", a); Query q =3D parser.Parse(query_string); Hits hits =3D search.Search(q); string[] result =3D new string[0]; if (hits.Length() > 0) { result =3D new string[hits.Length()]; for (int i =3D 0; i < hits.Length(); i++) { result[i] =3D hits.Doc(i).GetField("message_title").StringValue(); Console.WriteLine("\n{0}. {1}\n", i + 1, result[i]); } } else { Console.WriteLine("\n Missing Search!!!\n"); } } } Russian text example (for indexing): "=D0=93=D1=80=D1=83=D0=BF=D0=BF=D0=B5 =D1=82=D1=83=D1=80=D0=B8=D1=81=D1=82= =D0=BE=D0=B2 =D0=B2 =D0=AF=D0=BA=D1=83=D1=82=D0=B8=D0=B8 =D1=82=D1=80=D0=B5= =D0=B1=D1=83=D0=B5=D1=82=D1=81=D1=8F =D1=81=D1=80=D0=BE=D1=87=D0=BD=D0=B0= =D1=8F =D1=8D=D0=B2=D0=B0=D0=BA=D1=83=D0=B0=D1=86=D0=B8=D1=8F=20 radiomayak.ru=20 =D0=A7=D0=9F =D0=B2 =D0=AF=D0=BA=D1=83=D1=82=D0=B8=D0=B8. =D0=A1=D0=BD=D0= =B5=D0=B6=D0=BD=D0=B0=D1=8F =D0=BB=D0=B0=D0=B2=D0=B8=D0=BD=D0=B0 =D0=BD=D0= =B0=D0=BA=D1=80=D1=8B=D0=BB=D0=B0 =D0=B3=D1=80=D1=83=D0=BF=D0=BF=D1=83 =D1= =82=D1=83=D1=80=D0=B8=D1=81=D1=82=D0=BE=D0=B2. =D0=94=D0=B2=D0=BE=D0=B5 =D0= =BF=D0=BE=D0=B3=D0=B8=D0=B1=D0=BB=D0=B8, =D0=BE=D0=B4=D0=B8=D0=BD =D0=BF=D0=BE=D1=81=D1=82=D1=80=D0=B0=D0=B4=D0=B0=D0=BB. =D0=9E=D0=B1 =D1=8D= =D1=82=D0=BE=D0=BC "=D0=9C=D0=B0=D1=8F=D0=BA=D1=83" =D0=BF=D0=BE =D1=82=D0= =B5=D0=BB=D0=B5=D1=84=D0=BE=D0=BD=D1=83 =D1=81=D0=BE=D0=BE=D0=B1=D1=89=D0= =B8=D0=BB =D0=BE=D1=84=D0=B8=D1=86=D0=B8=D0=B0=D0=BB=D1=8C=D0=BD=D1=8B=D0= =B9 =D0=BF=D1=80=D0=B5=D0=B4=D1=81=D1=82=D0=B0=D0=B2=D0=B8=D1=82=D0=B5=D0= =BB=D1=8C =D1=80=D0=B5=D0=B3=D0=B8=D0=BE=D0=BD=D0=B0=D0=BB=D1=8C=D0=BD=D0=BE=D0=B3=D0= =BE =D0=9C=D0=A7=D0=A1 =D0=90=D0=BB=D0=B5=D0=BA=D1=81=D0=B5=D0=B9 =D0=A5=D0= =BB=D1=8B=D0=B1=D0=BE=D0=B2.=20 =D0=A5=D0=9B=D0=AB=D0=91=D0=9E=D0=92: 25 =D0=BC=D0=B0=D1=80=D1=82=D0=B0 =D0= =B3=D1=80=D1=83=D0=BF=D0=BF=D0=B0 =D1=82=D1=83=D1=80=D0=B8=D1=81=D1=82=D0= =BE=D0=B2 =D0=B8=D0=B7 =D0=B2=D0=BE=D1=81=D1=8C=D0=BC=D0=B8 =D1=87=D0=B5=D0= =BB=D0=BE=D0=B2=D0=B5=D0=BA - =D1=8D=D1=82=D0=BE =D0=B6=D0=B8=D1=82=D0=B5= =D0=BB=D0=B8 =D0=B3=D0=BE=D1=80=D0=BE=D0=B4=D0=B0 =D0=9A=D0=B0=D0=B7=D0=B0=D0=BD=D0=B8 =D0=B8 =D0=B3=D0=BE=D1=80=D0=BE=D0=B4= =D0=B0 =D0=95=D0=BA=D0=B0=D1=82=D0=B5=D1=80=D0=B8=D0=BD=D0=B1=D1=83=D1=80= =D0=B3=D0=B0, =D1=81=D0=BE=D0=B2=D0=B5=D1=80=D1=88=D0=B0=D1=8E=D1=89=D0=B8= =D0=B5 =D0=BB=D1=8B=D0=B6=D0=BD=D1=8B=D0=B9 =D0=BF=D0=B5=D1=80=D0=B5=D1=85= =D0=BE=D0=B4, =D0=BF=D0=BE=D0=BF=D0=B0=D0=BB=D0=B0 =D0=BF=D0=BE=D0=B4 =D0= =BB=D0=B0=D0=B2=D0=B8=D0=BD=D1=83 =D0=BD=D0=B0 =D0=BF=D0=B5=D1=80=D0=B5=D0=B2=D0=B0=D0=BB=D0=B5 =D0=90=D0=B3= =D0=B8=D0=B4=D0=B5=D0=BB=D1=8C =D1=81 =D0=B7=D0=B0=D0=BF=D0=B0=D0=B4=D0=BD= =D0=BE=D0=B9 =D1=81=D1=82=D0=BE=D1=80=D0=BE=D0=BD=D1=8B =D0=B3=D0=BE=D1=80= =D1=8B. =D0=92 11:40 =D0=BE=D1=82 =D0=B3=D1=80=D1=83=D0=BF=D0=BF=D1=8B =D0=BF=D0=BE= =D1=81=D1=82=D1=83=D0=BF=D0=B8=D0=BB=D0=BE =D1=81=D0=BE=D0=BE=D0=B1=D1=89= =D0=B5=D0=BD=D0=B8=D0=B5 =D1=81=D0=BF=D0=B0=D1=81=D0=B0=D1=82=D0=B5=D0=BB= =D1=8F=D0=BC =D0=BE =D1=82=D0=BE=D0=BC, =D1=87=D1=82=D0=BE =D0=BF=D0=BE=D1= =81=D1=82=D1=80=D0=B0=D0=B4=D0=B0=D0=B2=D1=88=D0=B8=D0=B5 =D0=B8 =D0=B4=D0=B2=D0=BE=D0=B5 =D0=BF=D0=BE=D0=B3=D0=B8=D0=B1=D1=88=D0=B8=D1=85 = =D0=B1=D1=8B=D0=BB=D0=B8 =D0=B8=D0=B7=D0=B2=D0=BB=D0=B5=D1=87=D0=B5=D0=BD= =D1=8B =D0=B8=D0=B7-=D0=BF=D0=BE=D0=B4 =D1=81=D0=BD=D0=B5=D0=B3=D0=B0. =D0= =93=D1=80=D1=83=D0=BF=D0=BF=D0=B5 =D1=82=D1=80=D0=B5=D0=B1=D1=83=D0=B5=D1= =82=D1=81=D1=8F =D1=81=D1=80=D0=BE=D1=87=D0=BD=D0=B0=D1=8F =D1=8D=D0=B2=D0=B0=D0=BA=D1=83=D0=B0=D1=86=D0=B8=D1=8F. =D0=92 12:30 =D0=B8= =D0=B7 =D0=B3=D0=BE=D1=80=D0=BE=D0=B4=D0=B0 =D0=AF=D0=BA=D1=83=D1=82=D1=81= =D0=BA=D0=B0 =D0=B7=D0=B0=D0=BF=D0=BB=D0=B0=D0=BD=D0=B8=D1=80=D0=BE=D0=B2= =D0=B0=D0=BD =D0=B2=D1=8B=D0=BB=D0=B5=D1=82 =D0=B3=D1=80=D1=83=D0=BF=D0=BF= =D1=8B =D1=81=D0=BF=D0=B0=D1=81=D0=B0=D1=82=D0=B5=D0=BB=D0=B5=D0=B9 =D0=B4=D0=BB=D1=8F =D1=8D=D0=B2=D0=B0=D0=BA=D1=83=D0=B0=D1=86=D0=B8=D0=B8 = =D0=BF=D0=BE=D1=81=D1=82=D1=80=D0=B0=D0=B4=D0=B0=D0=B2=D1=88=D0=B5=D0=B9 = =D0=B3=D1=80=D1=83=D0=BF=D0=BF=D1=8B =D1=82=D1=83=D1=80=D0=B8=D1=81=D1=82= =D0=BE=D0=B2." --=20 View this message in context: http://www.nabble.com/Problem-with-Russian-La= nguage-in-Lucene-83.161.120.155-tp16491424p16491424.html Sent from the Lucene - General mailing list yharris@example.net. From generalyharris@example.net Wed Apr 09 05:12:59 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 70369 invoked from network); 9 Apr 2008 05:12:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 9 Apr 2008 05:12:59 -0000 Received: (qmail 98109 invoked by uid 500); 9 Apr 2008 05:12:58 -0000 Delivered-To: yharris@example.net Received: (qmail 98091 invoked by uid 500); 9 Apr 2008 05:12:58 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 98079 invoked by uid 99); 9 Apr 2008 05:12:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Apr 2008 22:12:58 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 05:12:14 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <yharris@example.net>) id 1JjSbh-0000M0-Um for yharris@example.net; Tue, 08 Apr 2008 22:12:25 -0700 Message-ID: <yharris@example.net> Date: Tue, 8 Apr 2008 22:12:25 -0700 (PDT) From: oibook13 <yharris@example.net> To: yharris@example.net Subject: a problem about the deprecated method MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: yharris@example.net X-Virus-Checked: Checked by ClamAV on apache.org there is difference between lucene edition 1.4.3 and 2.0.0. e.g class FSDirectory's method getDirectory(File indexDir,boolean XX). i am studying lucene in action which based on lucene 1.4.3, so how can i find its equivalent in new edition? thanks -- View this message in context: http://www.nabble.com/a-problem-about-the-deprecated-method-tp16579552p16579552.html Sent from the Lucene - General mailing list yharris@example.net. From generalyharris@example.net Wed Apr 09 08:19:43 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 72756 invoked from network); 9 Apr 2008 08:19:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 9 Apr 2008 08:19:43 -0000 Received: (qmail 96830 invoked by uid 500); 9 Apr 2008 08:19:43 -0000 Delivered-To: yharris@example.net Received: (qmail 96806 invoked by uid 500); 9 Apr 2008 08:19:43 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 96795 invoked by uid 99); 9 Apr 2008 08:19:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 01:19:43 -0700 X-ASF-Spam-Status: No, hits=1.4 required=10.0 tests=SPF_NEUTRAL,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO an-out-0708.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 08:18:59 +0000 Received: by an-out-0708.google.com with SMTP id c5so603495anc.49 for <yharris@example.net>; Wed, 09 Apr 2008 01:18:56 -0700 (PDT) Received: by 83.161.120.155 with SMTP id 5mr1872429anm.139.1207729136515; Wed, 09 Apr 2008 01:18:56 -0700 (PDT) Received: from ?83.161.120.155? ( [83.161.120.155]) by mx.google.com with ESMTPS id 10sm18356253wrl.11.283.161.120.155.18.55 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 09 Apr 2008 01:18:56 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v753) In-Reply-To: <yharris@example.net> References: <yharris@example.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <yharris@example.net> Content-Transfer-Encoding: 7bit From: Michael McCandless <yharris@example.net> Subject: Re: a problem about the deprecated method Date: Wed, 9 Apr 2008 04:19:15 -0400 To: yharris@example.net X-Mailer: Apple Mail (2.753) X-Virus-Checked: Checked by ClamAV on apache.org Hi, The java-user list is more appropriate for your question (it's focused on using Lucene java, and, more people read it). In this case, you should use the getDirectory(File) method, and then when you instantiate your IndexWriter, pass in the boolean XX as the "create" argument. The javadoc stating the deprecation says this. Mike oibook13 wrote: > > there is difference between lucene edition 1.4.3 and 2.0.0. e.g class > FSDirectory's method > getDirectory(File indexDir,boolean XX). i am studying lucene in > action which > based on lucene 1.4.3, so how can i find its equivalent in new > edition? > > thanks > > -- > View this message in context: http://www.nabble.com/a-problem-about- > the-deprecated-method-tp16579552p16579552.html > Sent from the Lucene - General mailing list yharris@example.net. > From generalyharris@example.net Wed Apr 09 09:34:55 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 99639 invoked from network); 9 Apr 2008 09:34:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 9 Apr 2008 09:34:53 -0000 Received: (qmail 71283 invoked by uid 500); 9 Apr 2008 09:34:52 -0000 Delivered-To: yharris@example.net Received: (qmail 71261 invoked by uid 500); 9 Apr 2008 09:34:52 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 71250 invoked by uid 99); 9 Apr 2008 09:34:52 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 02:34:52 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wa-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 09:34:07 +0000 Received: by wa-out-1112.google.com with SMTP id j40so2137812wah.11 for <yharris@example.net>; Wed, 09 Apr 2008 02:34:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=Spa3kQBq6SmO9m7Yrx1juodvgNNjtCN30SJLhUrtBMI=; b=vK+yQPxkUuZls8QVyMmvHFo3+zpGBavjHZPF7yq8z8ykTFzbKYfFWorX5+MKBgLJP2bsq4Z+VDCDmucqPyxyQ5T65NZm4O9FmImK7E1VUJjdCIncSnVdhXDfJ+I+ZMMo4qju6CVyYeqXZ7MK8TTKKZLJkMcx5HRh0eu24zDdARA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=ZK6SqsUF1hIa7YkdJNnrNtv4qHlk/60ACgKhpGhlLux4UyL4X4veCL6lwv1UTKIxamqfVYVnTs1g1KU1IfTCfeM3pmQcm2co+ZBCIBZBDlNXSM+223dieI0jobXY9E58abqTGmArS2hkd5yXl6WNMROEjz4L+Q7zS+IYQYeATjk= Received: by 83.161.120.155 with SMTP id q1mr2020436waf.75.1207733659277; Wed, 09 Apr 2008 02:34:19 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 9 Apr 2008 02:34:19 -0700 (PDT) Message-ID: <yharris@example.net> Date: Wed, 9 Apr 2008 17:34:19 +0800 From: "Yingfeng Zhang" <yharris@example.net> To: yharris@example.net Subject: Lucene on Hadoop In-Reply-To: <yharris@example.net> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_30439_4487839.1207733659272" References: <yharris@example.net> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_30439_4487839.1207733659272 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi, >From this URL http://yharris@example.net/msg00998.html I see that Hadoop is not suitable for incremental updates if the inverted files is based on it, and what's more, Nutch has adopted Hadoop, that means the incremental updates ability provided by Lucene will not work in Nutch. Also, It is suggested to experiment with HBase, which is a BigTable based on GFS. Since HBase is based Hadoop, then what is a difference if using HBase for incremental indexing? Thanks a lot for attentions. Best Yingfeng ------=_Part_30439_4487839.1207733659272-- From generalyharris@example.net Wed Apr 09 12:37:54 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 65070 invoked from network); 9 Apr 2008 12:37:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 9 Apr 2008 12:37:52 -0000 Received: (qmail 64475 invoked by uid 500); 9 Apr 2008 12:37:52 -0000 Delivered-To: yharris@example.net Received: (qmail 64322 invoked by uid 500); 9 Apr 2008 12:37:51 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 64311 invoked by uid 99); 9 Apr 2008 12:37:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 05:37:51 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 12:37:00 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <yharris@example.net>) id 1JjZYF-0005CX-Rq for yharris@example.net; Wed, 09 Apr 2008 05:37:19 -0700 Message-ID: <yharris@example.net> Date: Wed, 9 Apr 2008 05:37:19 -0700 (PDT) From: oibook13 <yharris@example.net> To: yharris@example.net Subject: Re: a problem about the deprecated method In-Reply-To: <yharris@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: yharris@example.net References: <yharris@example.net> <yharris@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Thank you very much. But where can i find the java-user list.I am reading the lucene API illustration file,there is a Deprecated API list on it,is java-user list? I want ro have it because i want to eliminate other Deprecated method. Michael McCandless-2 wrote: > > > Hi, > > The java-user list is more appropriate for your question (it's > focused on using Lucene java, and, more people read it). > > In this case, you should use the getDirectory(File) method, and then > when you instantiate your IndexWriter, pass in the boolean XX as the > "create" argument. The javadoc stating the deprecation says this. > > Mike > -- View this message in context: http://www.nabble.com/a-problem-about-the-deprecated-method-tp16579552p16585334.html Sent from the Lucene - General mailing list yharris@example.net. From generalyharris@example.net Wed Apr 09 12:42:44 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 66729 invoked from network); 9 Apr 2008 12:42:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 9 Apr 2008 12:42:43 -0000 Received: (qmail 72260 invoked by uid 500); 9 Apr 2008 12:42:42 -0000 Delivered-To: yharris@example.net Received: (qmail 72240 invoked by uid 500); 9 Apr 2008 12:42:42 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 72229 invoked by uid 99); 9 Apr 2008 12:42:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 05:42:42 -0700 X-ASF-Spam-Status: No, hits=1.4 required=10.0 tests=SPF_NEUTRAL,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 12:41:57 +0000 Received: by nf-out-0910.google.com with SMTP id d3so1065410nfc.15 for <yharris@example.net>; Wed, 09 Apr 2008 05:42:06 -0700 (PDT) Received: by 83.161.120.155 with SMTP id p11mr233380ybb.36.1207744924910; Wed, 09 Apr 2008 05:42:04 -0700 (PDT) Received: from ?83.161.120.155? ( [83.161.120.155]) by mx.google.com with ESMTPS id 11sm165385wrl.7.283.161.120.155.42.04 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 09 Apr 2008 05:42:04 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v753) In-Reply-To: <yharris@example.net> References: <yharris@example.net> <yharris@example.net> <yharris@example.net> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <yharris@example.net> Content-Transfer-Encoding: 7bit From: Michael McCandless <yharris@example.net> Subject: Re: a problem about the deprecated method Date: Wed, 9 Apr 2008 08:42:23 -0400 To: yharris@example.net X-Mailer: Apple Mail (2.753) X-Virus-Checked: Checked by ClamAV on apache.org Oh by "java-user list", I mean you should send future emails like this to "yharris@example.net" instead of "yharris@example.net". Mike oibook13 wrote: > > Thank you very much. > But where can i find the java-user list.I am reading the lucene API > illustration file,there is a Deprecated API list on it,is java-user > list? I > want ro have it because i want to eliminate other Deprecated method. > > > Michael McCandless-2 wrote: >> >> >> Hi, >> >> The java-user list is more appropriate for your question (it's >> focused on using Lucene java, and, more people read it). >> >> In this case, you should use the getDirectory(File) method, and then >> when you instantiate your IndexWriter, pass in the boolean XX as the >> "create" argument. The javadoc stating the deprecation says this. >> >> Mike >> > > -- > View this message in context: http://www.nabble.com/a-problem-about- > the-deprecated-method-tp16579552p16585334.html > Sent from the Lucene - General mailing list yharris@example.net. > From generalyharris@example.net Thu Apr 10 02:29:56 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 18960 invoked from network); 10 Apr 2008 02:29:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 10 Apr 2008 02:29:56 -0000 Received: (qmail 86276 invoked by uid 500); 10 Apr 2008 02:29:56 -0000 Delivered-To: yharris@example.net Received: (qmail 86259 invoked by uid 500); 10 Apr 2008 02:29:56 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 86247 invoked by uid 99); 10 Apr 2008 02:29:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2008 19:29:55 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Apr 2008 02:29:13 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <yharris@example.net>) id 1JjmXU-00073k-JN for yharris@example.net; Wed, 09 Apr 2008 19:29:24 -0700 Message-ID: <yharris@example.net> Date: Wed, 9 Apr 2008 19:29:24 -0700 (PDT) From: oibook13 <yharris@example.net> To: yharris@example.net Subject: Re: a problem about the deprecated method In-Reply-To: <yharris@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: yharris@example.net References: <yharris@example.net> <yharris@example.net> <yharris@example.net> <yharris@example.net> X-Virus-Checked: Checked by ClamAV on apache.org look like i have make a mistake,i have considered it as document or something like that.thank you all the time Michael McCandless-2 wrote: > > > Oh by "java-user list", I mean you should send future emails like > this to "yharris@example.net" instead of > "yharris@example.net". > > Mike > > > -- View this message in context: http://www.nabble.com/a-problem-about-the-deprecated-method-tp16579552p16600882.html Sent from the Lucene - General mailing list yharris@example.net. From generalyharris@example.net Sat Apr 12 00:10:26 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 75564 invoked from network); 12 Apr 2008 00:10:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 12 Apr 2008 00:10:25 -0000 Received: (qmail 31926 invoked by uid 500); 12 Apr 2008 00:10:24 -0000 Delivered-To: yharris@example.net Received: (qmail 31914 invoked by uid 500); 12 Apr 2008 00:10:23 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 31903 invoked by uid 99); 12 Apr 2008 00:10:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Apr 2008 17:10:23 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO radix.cryptio.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 12 Apr 2008 00:09:30 +0000 Received: by radix.cryptio.net (Postfix, from userid 1007) id 653F971C2A0; Fri, 11 Apr 2008 17:09:50 -0700 (PDT) Received: from localhost (localhost [83.161.120.155]) by radix.cryptio.net (Postfix) with ESMTP id 5FC8271C1C8 for <yharris@example.net>; Fri, 11 Apr 2008 17:09:50 -0700 (PDT) Date: Fri, 11 Apr 2008 17:09:50 -0700 (PDT) From: Chris Hostetter <yharris@example.net> To: yharris@example.net Subject: Re: Lucene on Hadoop In-Reply-To: <yharris@example.net> Message-ID: <yharris@example.net> References: <yharris@example.net> <yharris@example.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org : >From this URL : http://yharris@example.net/msg00998.html : I see that Hadoop is not suitable for incremental updates if the inverted : files is based on it, and what's more, Nutch has adopted Hadoop, that means : the incremental updates ability provided by Lucene will not work in Nutch. Questions about Nutch's use of Lucene and HDFS are best addressed on a nutch specific list. : Also, It is suggested to experiment with HBase, which is a BigTable based on : GFS. Since HBase is based Hadoop, then what is a difference if using HBase : for incremental indexing? Thanks a lot for attentions. While i'm not an expert on HBase from the single message you linked to i see the comment: "...HBase is designed to be a much more scalable, incrementally updateable DB than BDB or relational DBs..." which suggests to me that while HBase may be built on HDFS, the API abstraction may allow for "Files" which do allow for incremental updates. -Hoss From generalyharris@example.net Mon Apr 14 07:20:10 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 43972 invoked from network); 14 Apr 2008 07:20:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 14 Apr 2008 07:20:09 -0000 Received: (qmail 92192 invoked by uid 500); 14 Apr 2008 07:20:08 -0000 Delivered-To: yharris@example.net Received: (qmail 92176 invoked by uid 500); 14 Apr 2008 07:20:08 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 92164 invoked by uid 99); 14 Apr 2008 07:20:08 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Apr 2008 00:20:08 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 14 Apr 2008 07:19:16 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <yharris@example.net>) id 1JlIyW-00045V-S1 for yharris@example.net; Mon, 14 Apr 2008 00:19:36 -0700 Message-ID: <yharris@example.net> Date: Mon, 14 Apr 2008 00:19:36 -0700 (PDT) From: oibook13 <yharris@example.net> To: yharris@example.net Subject: the question about the example of lucene in action MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Nabble-From: yharris@example.net X-Virus-Checked: Checked by ClamAV on apache.org lucene in action has a instance about explaining indexwriter's MergeFactor,MaxMergeDocs parameter on 2.7.1. Because of lucene's edition, imodified it just like this: public class IndexTuningDemo { =09 =09public static void main(String[] args) throws Exception { =09=09int docsInIndex =3D Integer.parseInt(args[0]); =09=09 =09=09Directory dir =3D FSDirectory.getDirectory( =09=09=09=09System.getProperty("java.io.tmpdir","tmp")+ =09=09=09=09System.getProperty("file.separator")+"index-dir"); =09=09Analyzer analyzer =3D new SimpleAnalyzer(); =09=09IndexWriter writer =3D new IndexWriter(dir,analyzer,true); =09=09 =09=09writer.setMergeFactor(Integer.parseInt(args[1])); =09=09writer.setMaxMergeDocs(Integer.parseInt(args[2])); =09=09writer.setInfoStream(System.out); =09=09 =09=09System.out.println("Merge factor: "+writer.getMergeFactor()); =09=09System.out.println("Max merge docs: "+writer.getMaxMergeDocs()); =09=09 =09=09long start =3D System.currentTimeMillis(); =09=09for (int i =3D 0; i < docsInIndex ; i++){ =09=09=09Document doc =3D new Document(); =09=09=09doc.add(new Field("fieldname","Bibamus",Field.Store.YES,Field.Index.TOKENIZED)); =09=09=09writer.addDocument(doc); =09=09} =09=09writer.close(); =09=09long stop =3D System.currentTimeMillis(); =09=09System.out.println("Time: "+(stop-start)+" ms"); =09} } but the result which came from the program isn't like the books says,but like this(i omit a part of it because it is so much many line): org.apache.lucene.index.IndexFileDeleter@18558d2 main: DecRef "_2r.cfs": pre-decr count is 2 org.apache.lucene.index.IndexFileDeleter@18558d2 main: DecRef "_2s.cfs": pre-decr count is 2 org.apache.lucene.index.IndexFileDeleter@18558d2 main: DecRef "segments_5o": pre-decr count is 1 org.apache.lucene.index.IndexFileDeleter@18558d2 main: delete "segments_5o" org.apache.lucene.index.IndexFileDeleter@18558d2 main: now checkpoint "segments_5q" [isCommit =3D true] org.apache.lucene.index.IndexFileDeleter@18558d2 main: IncRef "_a.cfs": pre-incr count is 1 org.apache.lucene.index.IndexFileDeleter@18558d2 main: IncRef "_l.cfs": pre-incr count is 1 org.apache.lucene.index.IndexFileDeleter@18558d2 main: IncRef "_w.cfs": pre-incr count is 1 =E2=80=A6=E2=80=A6=E2=80=A6=E2=80=A6 org.apache.lucene.index.IndexFileDeleter@18558d2 main: DecRef "segments_67": pre-decr count is 1 org.apache.lucene.index.IndexFileDeleter@18558d2 main: delete "segments_67" Time: 4156 ms i don't know what it is meaning! i would appreciate that if some one could explain it. thanks. --=20 View this message in context: http://www.nabble.com/the-question-about-the-= example-of-lucene-in-action-tp16671486p16671486.html Sent from the Lucene - General mailing list yharris@example.net. From generalyharris@example.net Tue Apr 15 09:17:48 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 75440 invoked from network); 15 Apr 2008 09:17:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 15 Apr 2008 09:17:47 -0000 Received: (qmail 58050 invoked by uid 500); 15 Apr 2008 09:17:48 -0000 Delivered-To: yharris@example.net Received: (qmail 57603 invoked by uid 500); 15 Apr 2008 09:17:47 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 57592 invoked by uid 99); 15 Apr 2008 09:17:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Apr 2008 02:17:47 -0700 X-ASF-Spam-Status: No, hits=1.4 required=10.0 tests=SPF_NEUTRAL,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO yw-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Apr 2008 09:17:01 +0000 Received: by yw-out-2324.google.com with SMTP id 5so1017947ywh.5 for <yharris@example.net>; Tue, 15 Apr 2008 02:17:06 -0700 (PDT) Received: by 83.161.120.155 with SMTP id l1mr7434482ybd.6.1208251026826; Tue, 15 Apr 2008 02:17:06 -0700 (PDT) Received: from ?83.161.120.155? ( [83.161.120.155]) by mx.google.com with ESMTPS id i19sm28783354wxd.33.283.161.120.155.17.05 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 15 Apr 2008 02:17:05 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v753) In-Reply-To: <yharris@example.net> References: <yharris@example.net> Content-Type: text/plain; charset=WINDOWS-1252; delsp=yes; format=flowed Message-Id: <yharris@example.net> Content-Transfer-Encoding: quoted-printable From: Michael McCandless <yharris@example.net> Subject: Re: the question about the example of lucene in action Date: Tue, 15 Apr 2008 05:17:31 -0400 To: yharris@example.net X-Mailer: Apple Mail (2.753) X-Virus-Checked: Checked by ClamAV on apache.org This question actually belongs on the yharris@example.net =20 (instead of general). It will get more attention there. Lucene now logs quite a bit more detail than just segment merges to =20 that infoStream. This particular snippet is the detailing which =20 files are being incref'd & decref'd; this is used to decide what =20 files can safely be deleted. Lines that have "merge" or "merging" =20 will tell you when merges are taking place. Mike oibook13 wrote: > > lucene in action has a instance about explaining indexwriter's > MergeFactor,MaxMergeDocs parameter on 2.7.1. Because of lucene's =20 > edition, > imodified it just like this: > public class IndexTuningDemo { > =09 > public static void main(String[] args) throws Exception { > int docsInIndex =3D Integer.parseInt(args[0]); > =09 > Directory dir =3D FSDirectory.getDirectory( > = System.getProperty("java.io.tmpdir","tmp")+ > = System.getProperty("file.separator")+"index-dir"); > Analyzer analyzer =3D new SimpleAnalyzer(); > IndexWriter writer =3D new = IndexWriter(dir,analyzer,true); > =09 > writer.setMergeFactor(Integer.parseInt(args[1])); > writer.setMaxMergeDocs(Integer.parseInt(args[2])); > writer.setInfoStream(System.out); > =09 > System.out.println("Merge factor: = "+writer.getMergeFactor()); > System.out.println("Max merge docs: = "+writer.getMaxMergeDocs()); > =09 > long start =3D System.currentTimeMillis(); > for (int i =3D 0; i < docsInIndex ; i++){ > Document doc =3D new Document(); > doc.add(new > Field("fieldname","Bibamus",Field.Store.YES,Field.Index.TOKENIZED)); > writer.addDocument(doc); > } > writer.close(); > long stop =3D System.currentTimeMillis(); > System.out.println("Time: "+(stop-start)+" ms"); > } > } > > but the result which came from the program isn't like the books =20 > says,but > like this(i omit a part of it because it is so much many line): > org.apache.lucene.index.IndexFileDeleter@18558d2 main: DecRef =20 > "_2r.cfs": > pre-decr count is 2 > org.apache.lucene.index.IndexFileDeleter@18558d2 main: DecRef =20 > "_2s.cfs": > pre-decr count is 2 > org.apache.lucene.index.IndexFileDeleter@18558d2 main: DecRef > "segments_5o": pre-decr count is 1 > org.apache.lucene.index.IndexFileDeleter@18558d2 main: delete =20 > "segments_5o" > org.apache.lucene.index.IndexFileDeleter@18558d2 main: now checkpoint > "segments_5q" [isCommit =3D true] > org.apache.lucene.index.IndexFileDeleter@18558d2 main: IncRef =20 > "_a.cfs": > pre-incr count is 1 > org.apache.lucene.index.IndexFileDeleter@18558d2 main: IncRef =20 > "_l.cfs": > pre-incr count is 1 > org.apache.lucene.index.IndexFileDeleter@18558d2 main: IncRef =20 > "_w.cfs": > pre-incr count is 1 > =85=85=85=85 > org.apache.lucene.index.IndexFileDeleter@18558d2 main: DecRef > "segments_67": pre-decr count is 1 > org.apache.lucene.index.IndexFileDeleter@18558d2 main: delete =20 > "segments_67" > Time: 4156 ms > > i don't know what it is meaning! i would appreciate that if some =20 > one could > explain it. thanks. > --=20 > View this message in context: http://www.nabble.com/the-question-=20 > about-the-example-of-lucene-in-action-tp16671486p16671486.html > Sent from the Lucene - General mailing list yharris@example.net. > From generalyharris@example.net Tue Apr 15 19:05:03 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 28677 invoked from network); 15 Apr 2008 19:05:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 15 Apr 2008 19:05:03 -0000 Received: (qmail 18459 invoked by uid 500); 15 Apr 2008 19:05:02 -0000 Delivered-To: yharris@example.net Received: (qmail 18443 invoked by uid 500); 15 Apr 2008 19:05:02 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 18432 invoked by uid 99); 15 Apr 2008 19:05:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Apr 2008 12:05:02 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 15 Apr 2008 19:04:19 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <yharris@example.net>) id 1JlqSF-0000ca-H9 for yharris@example.net; Tue, 15 Apr 2008 12:04:31 -0700 Message-ID: <yharris@example.net> Date: Tue, 15 Apr 2008 12:04:31 -0700 (PDT) From: adepue <yharris@example.net> To: yharris@example.net Subject: Looking for duplicate names MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: yharris@example.net X-Virus-Checked: Checked by ClamAV on apache.org I'm new to Lucene, and would like to use it to find duplicate names in a contact list. Is Lucene a good fit? We have a form where a user enters a company or person's name, and we want the system to warn them if there is already a company or person entered with the same or very similar name. Based on the little I know of Lucene, I'm thinking an NGram algorithm (based on characters, not words) would work best... but, I'm not sure if Lucene takes proximity or edit distances into account? For example, say you have these two names: Andrew John John Andrew If a user enters Andy John, without proximity or edit distance, these two names will match about the same, while, obviously, the first name should be ranked higher. Thanks in advance for any help or advice. -- View this message in context: http://www.nabble.com/Looking-for-duplicate-names-tp16705188p16705188.html Sent from the Lucene - General mailing list yharris@example.net. From generalyharris@example.net Wed Apr 16 11:46:09 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 46089 invoked from network); 16 Apr 2008 11:46:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 16 Apr 2008 11:46:08 -0000 Received: (qmail 43988 invoked by uid 500); 16 Apr 2008 11:46:08 -0000 Delivered-To: yharris@example.net Received: (qmail 43958 invoked by uid 500); 16 Apr 2008 11:46:08 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 43946 invoked by uid 99); 16 Apr 2008 11:46:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Apr 2008 04:46:08 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Apr 2008 11:45:24 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <yharris@example.net>) id 1Jm653-00044k-1n for yharris@example.net; Wed, 16 Apr 2008 04:45:37 -0700 Message-ID: <yharris@example.net> Date: Wed, 16 Apr 2008 04:45:37 -0700 (PDT) From: oibook13 <yharris@example.net> To: yharris@example.net Subject: Re: the question about the example of lucene in action In-Reply-To: <yharris@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: yharris@example.net References: <yharris@example.net> <yharris@example.net> X-Virus-Checked: Checked by ClamAV on apache.org that accounts for it. thanks for your detailed explanations. Michael McCandless-2 wrote: > > > This question actually belongs on the yharris@example.net > (instead of general). It will get more attention there. > > Lucene now logs quite a bit more detail than just segment merges to > that infoStream. This particular snippet is the detailing which > files are being incref'd & decref'd; this is used to decide what > files can safely be deleted. Lines that have "merge" or "merging" > will tell you when merges are taking place. > > Mike > -- View this message in context: http://www.nabble.com/the-question-about-the-example-of-lucene-in-action-tp16671486p16721012.html Sent from the Lucene - General mailing list yharris@example.net. From generalyharris@example.net Fri Apr 18 11:43:01 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 88282 invoked from network); 18 Apr 2008 11:43:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 18 Apr 2008 11:43:01 -0000 Received: (qmail 17475 invoked by uid 500); 18 Apr 2008 11:43:00 -0000 Delivered-To: yharris@example.net Received: (qmail 17448 invoked by uid 500); 18 Apr 2008 11:43:00 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 17435 invoked by uid 99); 18 Apr 2008 11:42:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Apr 2008 04:42:59 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO po-out-1718.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Apr 2008 11:42:17 +0000 Received: by po-out-1718.google.com with SMTP id a23so455472poh.0 for <yharris@example.net>; Fri, 18 Apr 2008 04:42:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=VDLaNgtlRzCZ/z4pPlQMVG9W0lxFM0rYVQx1FMiGDHM=; b=hyCU9NRi/U2ukNfGjkRADN/H9tRGMIKwNZ++OOH0wu2bYrvn+gC2LPYjUHuL+htgBEf7j9mlRlEhGcUafJf5Hj+QIBNhBwvynyjxg7GfFIno6NP5jx8kL1jVlY5ftYrsojObMV35dDSToWUHT7oA5X8U4JMqqUMmPoW1nTAK6k0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=pFozUI+uDrkLMlg9PA9Un5TFW+gumiq2XZUZ/IlNdQh/y8Lrq3O/pJxXIYoB/ISAWuaK6N8jR9b0M5lEzaGUSexj6Z/RTCFZQ68jFjWQaksdgp8nBLWm4bbWWV8x3z2sh8AI2/CK352F+fm1uK21HQSZoZhhY8Qq6x0LvWM9ar4= Received: by 83.161.120.155 with SMTP id g14mr1411786rvj.241.1208518950812; Fri, 18 Apr 2008 04:42:30 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 18 Apr 2008 04:42:30 -0700 (PDT) Message-ID: <yharris@example.net> Date: Fri, 18 Apr 2008 19:42:30 +0800 From: "=?GB2312?B?zfW9qNDC?=" <yharris@example.net> To: yharris@example.net Subject: a question about MultiSearcher MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2409_24444479.1208518950830" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_2409_24444479.1208518950830 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline If I search in multiple dirs,I can use MultiSearcher. The idf about a term is log(numDocs/(docFreq+1)+1). In the two kinds of condition: 1.only 1 dir to search 2. 2 dirs to search the idf about a term is not the same? why not lucene consider giving a method to set the numDocs for class IndexWriter? just like: IndexWriter writer=new IndexWrite(...... writer.setNumDoc(num1+num2); Thanks. ------=_Part_2409_24444479.1208518950830-- From generalyharris@example.net Fri Apr 18 12:14:51 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 2993 invoked from network); 18 Apr 2008 12:14:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 18 Apr 2008 12:14:50 -0000 Received: (qmail 62298 invoked by uid 500); 18 Apr 2008 12:14:50 -0000 Delivered-To: yharris@example.net Received: (qmail 62196 invoked by uid 500); 18 Apr 2008 12:14:49 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 62185 invoked by uid 99); 18 Apr 2008 12:14:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Apr 2008 05:14:49 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO po-out-1718.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Apr 2008 12:14:06 +0000 Received: by po-out-1718.google.com with SMTP id a23so470380poh.0 for <yharris@example.net>; Fri, 18 Apr 2008 05:14:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=ILSaDfmypQ9p2scSz673tEcMa3GR8e7hc4bq7Vep56o=; b=xIFWRC8V+2nLUkHEezAm/TMqREX+4h0TUS1+xr1tykyu9ni/baR2a09nMWpkhyMWeAkqS3YxzQipQiducVcDv+cYbY7wNFz25hT1MhM7N50dfIrE7onzsnKiGgfrYQtlOi6G3G1/EcrdT8u7YmfqYTmlkFW4nhiHp3W5PYrwtSY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=iF5Z9W4mqYzCsqzr0SdMRT0J6Cq04r2Rl5V3rEfQgHgEQwdBsWq2lRzO5Uhw4A5znffnfgR+QmGR3KG9V5b8aNdda2NK/iKFLzWUs/mIsuYffHkYKDWhMk9ot71CMzbJxYsf/A6RUGHPWVOdkusiAKH28GR0umuWJHDY8CIMaZM= Received: by 83.161.120.155 with SMTP id a15mr1431567rvg.212.1208520858035; Fri, 18 Apr 2008 05:14:18 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 18 Apr 2008 05:14:17 -0700 (PDT) Message-ID: <yharris@example.net> Date: Fri, 18 Apr 2008 20:14:17 +0800 From: "=?GB2312?B?zfW9qNDC?=" <yharris@example.net> To: "yharris@example.net" <yharris@example.net> Subject: a question about MultiSearcher In-Reply-To: <yharris@example.net> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_2458_15237907.1208520858021" References: <yharris@example.net> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_2458_15237907.1208520858021 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Sorry, I made a mistake. Lucene made it already. ------=_Part_2458_15237907.1208520858021-- From generalyharris@example.net Fri Apr 18 18:20:06 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 49657 invoked from network); 18 Apr 2008 18:20:06 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 18 Apr 2008 18:20:06 -0000 Received: (qmail 41342 invoked by uid 500); 18 Apr 2008 18:20:05 -0000 Delivered-To: yharris@example.net Received: (qmail 41181 invoked by uid 500); 18 Apr 2008 18:20:04 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 40994 invoked by uid 99); 18 Apr 2008 18:20:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Apr 2008 11:20:04 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO smtprelay06.ispgateway.de) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Apr 2008 18:19:10 +0000 Received: from [83.161.120.155] (helo=p5b25d3d2.dip.t-dialin.net) by smtprelay06.ispgateway.de with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.68) (envelope-from <yharris@example.net>) id 1JmvBK-0005So-L1 for yharris@example.net; Fri, 18 Apr 2008 20:19:30 +0200 From: Daniel Naber <yharris@example.net> To: yharris@example.net Subject: Re: a question about MultiSearcher Date: Fri, 18 Apr 2008 20:21:02 +0200 User-Agent: KMail/1.9.6 (enterprise 20070904.708012) References: <yharris@example.net> In-Reply-To: <yharris@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <yharris@example.net> X-Df-Sender: 695637 X-Virus-Checked: Checked by ClamAV on apache.org On Freitag, 18. April 2008, =E7=8E=8B=E5=BB=BA=E6=96=B0 wrote: > If I search in multiple dirs,I can use MultiSearcher. > The idf about a term is log(numDocs/(docFreq+1)+1). > > In the two kinds of condition: > =C2=A0 =C2=A0 1.only 1 dir to search > =C2=A0 =C2=A0 2. 2 dirs to search > the idf about a term is not the same? It should be the same, unless you are using an old version of Lucene (1.4=20 or older). If it doesn't work for you, could you provide a test case? BTW,= =20 there's more traffic and a better chance for an answer on the=20 yharris@example.net mailing list. Regards Daniel =2D-=20 http://www.danielnaber.de From generalyharris@example.net Sat Apr 19 08:37:38 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 88401 invoked from network); 19 Apr 2008 08:37:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 19 Apr 2008 08:37:38 -0000 Received: (qmail 55357 invoked by uid 500); 19 Apr 2008 08:37:38 -0000 Delivered-To: yharris@example.net Received: (qmail 55340 invoked by uid 500); 19 Apr 2008 08:37:37 -0000 Mailing-List: contact yharris@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:yharris@example.net> List-Unsubscribe: <mailto:yharris@example.net> List-Post: <mailto:yharris@example.net> List-Id: <general.lucene.apache.org> Reply-To: yharris@example.net Delivered-To: mailing list yharris@example.net Received: (qmail 55329 invoked by uid 99); 19 Apr 2008 08:37:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Apr 2008 01:37:37 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yharris@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Apr 2008 08:36:54 +0000 Received: by rv-out-0506.google.com with SMTP id k29so547529rvb.5 for <yharris@example.net>; Sat, 19 Apr 2008 01:37:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=dz7LMmFfVc8RYqpBHXkCVMaPNRVRHqmEaFrdx1Y+1p0=; b=XFZGMsmWaJuUGKAWb/2dTdD5rf6oKxF/tq442lon4G3peJve2vykdcQHEBf0PSnLGRqRjq8qXffb8eLjnjLPCbKKSovNe1o+9czM4NW0th/n395TUUrbAr5kPjMRRKXqR/5xar71QZhyNJq5LLdriHPhS6BfoIGIiGJNeU190zM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=SunfB5mumOMef3kr9VMdti3vjW4fuAdmef2qcVI4Se4qDipiGONzDjC3HVo+gyLvcI/fWqAbBcnQzRO9CwZsBKBDvZG+gcsYJcZ9+2H95S+mCM3nk+AXZkPbrgS4qPR1CKdgoMpFhD+u+VSJWT3wdWxc/eMptn1jwdUhWbholfk= Received: by 83.161.120.155 with SMTP id a7mr2123972rvj.58.1208594227600; Sat, 19 Apr 2008 01:37:07 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Sat, 19 Apr 2008 01:37:07 -0700 (PDT) Message-ID: <yharris@example.net> Date: Sat, 19 Apr 2008 16:37:07 +0800 From: "=?GB2312?B?zfW9qNDC?=" <yharris@example.net> To: yharris@example.net Subject: A question about ParalellMultiSearcher and RMI MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_4403_12147794.1208594227599" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_4403_12147794.1208594227599 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I want to use RAMDirectory to raise the peformance of lucene. So I cut the index dir to 3 smaller index dirs(1G one index dir). Then I use RAMDirectory to load the 3 dirs, and rebind the 3 searcher by RMI.(My server has 8G mem) At last, I lookup the 3 remote searcher to new a ParalellMultiSearcher. It works correctorly, but sometimes occur errors like : java.rmi.ServerError: Error occurred in server thread; nested exception is: java.lang.OutOfMemoryError: Java heap yharris@example.net(UnicastServerRef.java:333) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(Unknown Source) at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source) at sun.rmi.server.UnicastRef.invoke(Unknown Source) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Source) at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source) at $Proxy0.search(Unknown Source) at org.apache.lucene.search.MultiSearcherThread.run(ParallelMultiSearcher.java:251) Caused by: java.lang.OutOfMemoryError: Java heap space Thanks. ------=_Part_4403_12147794.1208594227599-- From generalyharris@example.net Sat Apr 19 08:42:06 2008 Return-Path: <generalyharris@example.net> Delivered-To: yharris@example.net Received: (qmail 89015
{'source': 'Apache Software Foundation Public Mail Archives'}