From solr-usergregnicholson@example.org Thu Jan 01 00:29:43 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 23265 invoked from network); 1 Jan 2009 00:29:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 00:29:42 -0000 Received: (qmail 84818 invoked by uid 500); 1 Jan 2009 00:29:39 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 84776 invoked by uid 500); 1 Jan 2009 00:29:38 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 84765 invoked by uid 99); 1 Jan 2009 00:29:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Dec 2008 16:29:38 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 00:29:30 +0000 Received: by rv-out-0506.google.com with SMTP id f6so7705625rvb.5 for <gregnicholson@example.org>; Wed, 31 Dec 2008 16:29:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=Vg7LfznJKXPSvfA0nKdSrJZdOPfVBPfQZx+ayxQD13Q=; b=iaNP65M5PnGVB3Dkby4uFUQwip2mWmpoeg8/FsFXDRbe9O7bd6z8LtgqWauxU7uDeD YG80KRFa0ltUvsf3KqfnHO8rMYPzMf966PJoewwHIV2b0jvq4bOa++HJcFO0WX32BFy5 3gz3I6RUGuJnzoutQJYRg38+uBdJo1E0V5PC4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=iAscsgrzoghTXp7lqJVEIOU7PfCpVI0XBsabTVeF7bdeWTQUOahiwBBQY26Q+bFe4Q enx/4s1XbPp8Enxagr5xG7reOvIB9b6FiIqgrFTAWmWXo+LwpXH3R2e+2upA0wde6Xja +uKDJV86J0U+FPiS98gZ//ahZap6kHRdN0Ni0= Received: by 83.161.120.155 with SMTP id m17mr8045786rvl.252.1230769749463; Wed, 31 Dec 2008 16:29:09 -0800 (PST) Received: by 83.161.120.155 with HTTP; Wed, 31 Dec 2008 16:29:09 -0800 (PST) Message-ID: <gregnicholson@example.org> Date: Wed, 31 Dec 2008 17:29:09 -0700 From: "Mark Ferguson" <gregnicholson@example.org> To: gregnicholson@example.org Subject: Dismax query parser with different field classes MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_107339_19881198.1230769749452" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_107339_19881198.1230769749452 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hello, I have a set of documents in which I have different classes of fields that I would like to search separately. For example, I would like to search the HTML body and title of a webpage using one set of keywords, and the page author using another set. I cannot use the dismax parser for this problem because all keywords will search across all the query fields. However, I like the dismax query parser because it handles matching and scoring very nicely. I read one suggestion on this group which was to make one of the queries a query filter. So for example, I may use the dismax query parser to search the body and title of a webpage, then use a filter query for the author. There are two problems with this approach in regards to what I need: 1. The filter query does not affect scoring, but I need the scoring to be influenced by the results of all fields being searched. 2. A filter query will do a simple AND or OR filter, while I would need the search to be an OR search with higher scoring for multiple matches (related to the first problem). I think what I need is a dismax parser into which the parsed query will not just contain all keywords for all fields, but into which you can specify which fields correspond to which sets of keywords. Has anything like this been tackled before? If not, can someone help point me in the right direction for how I would build this myself? Thanks very much for your time. Regards, Mark Ferguson ------=_Part_107339_19881198.1230769749452-- From solr-usergregnicholson@example.org Thu Jan 01 04:00:52 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 71598 invoked from network); 1 Jan 2009 04:00:51 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 04:00:51 -0000 Received: (qmail 31939 invoked by uid 500); 1 Jan 2009 04:00:48 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 31908 invoked by uid 500); 1 Jan 2009 04:00:48 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 31897 invoked by uid 99); 1 Jan 2009 04:00:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Dec 2008 20:00:48 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO radix.cryptio.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 04:00:42 +0000 Received: by radix.cryptio.net (Postfix, from userid 1007) id 8B0AB71C2DD; Wed, 31 Dec 2008 20:00:21 -0800 (PST) Received: from localhost (localhost [83.161.120.155]) by radix.cryptio.net (Postfix) with ESMTP id 5F94471C2CE for <gregnicholson@example.org>; Wed, 31 Dec 2008 20:00:21 -0800 (PST) Date: Wed, 31 Dec 2008 20:00:21 -0800 (PST) From: Chris Hostetter <gregnicholson@example.org> To: gregnicholson@example.org Subject: Re: Feature Request: Return count for documents which are possible to select In-Reply-To: <gregnicholson@example.org> Message-ID: <gregnicholson@example.org> References: <gregnicholson@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org : The other issue relates to providing (count) feedback for disjoint selections. : When a facet value is selected this constrains the documents and solr returns : the counts for all the other facet values. Thus the user can see all the : possible valid selections (i.e. having a count >0) and the number of documents : which will be returned if that value is selected. However one of the valid : selections is to select another value in the facet, creating a disjoint : selection and increasingly the number of returned documents. However there is : currently no way for the user to know which values are valid to select as the : count only relates to currently selected documents and not documents which are : also still possible to select. I'm not entirely sure if i understand what you're asking for, but it *sounds* like it might be related to SOLR-911 which was recently committed, can you take a look at that functionality and see if that addresses your need? if not a more specific example would be helpful to understand what you're looking for... https://issues.apache.org/jira/browse/SOLR-911 -Hoss From solr-usergregnicholson@example.org Thu Jan 01 04:07:13 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 72336 invoked from network); 1 Jan 2009 04:07:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 04:07:13 -0000 Received: (qmail 37703 invoked by uid 500); 1 Jan 2009 04:07:09 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 37661 invoked by uid 500); 1 Jan 2009 04:07:09 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 37650 invoked by uid 99); 1 Jan 2009 04:07:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Dec 2008 20:07:09 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO radix.cryptio.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 04:07:02 +0000 Received: by radix.cryptio.net (Postfix, from userid 1007) id DA49C71C2EB; Wed, 31 Dec 2008 20:06:40 -0800 (PST) Received: from localhost (localhost [83.161.120.155]) by radix.cryptio.net (Postfix) with ESMTP id D809171C2E1 for <gregnicholson@example.org>; Wed, 31 Dec 2008 20:06:40 -0800 (PST) Date: Wed, 31 Dec 2008 20:06:40 -0800 (PST) From: Chris Hostetter <gregnicholson@example.org> To: gregnicholson@example.org Subject: Re: Flipping data dirs for an (/multiple) SolrCore without affecting search / IndexReaders In-Reply-To: <gregnicholson@example.org> Message-ID: <gregnicholson@example.org> References: <gregnicholson@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org : We have an architecture where we want to flip the solr data.dir (massive : dataset) while running and serving search requests with minimal downtime. ... : 1) What is the fastest / best possible way to get step 1 done ,through a : pluggable architecture. : : Currently - I have written a request handler as follows, that takes care of : creating the core. What is the best way to change dataDir (got as input from : SolrQueryRequest) before creating SolrCore-s. you shouldn't need any custom plugin code to achieve this ... what you describe wounds like exactly what the SWAP command on the CoreAdmin handler was designed for -- CREATE your new core (using the new data dir) warm it however you want (either via the solrconfig.xml or by explicitly hitting it with queries) and once it's warmed up send the SWAP command to replace the existing core with the name you want to use. : 2) When a close() happens on an existing SolrCore - what happens when there is : a long running IndexReader query on that SolrCore . Is that terminated : abruptly / would the close wait until the IndexReaders completes the Query. any existing uses of the Core will continue to finish (i'm not sure of hte timeline of your question, ut i'm guessing this was before the recent jira issue about the close() method and ref counts where this was better explained, correct?) -Hoss From solr-usergregnicholson@example.org Thu Jan 01 04:07:38 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 72381 invoked from network); 1 Jan 2009 04:07:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 04:07:37 -0000 Received: (qmail 38784 invoked by uid 500); 1 Jan 2009 04:07:34 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 38754 invoked by uid 500); 1 Jan 2009 04:07:34 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 38743 invoked by uid 99); 1 Jan 2009 04:07:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Dec 2008 20:07:34 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail.asahi-net.or.jp) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 04:07:25 +0000 Received: from koji-sekiguchi-no-macbook.local (w244069.ppp.asahi-net.or.jp [83.161.120.155]) by mail.asahi-net.or.jp (Postfix) with ESMTP id 34B695C47D for <gregnicholson@example.org>; Thu, 1 Jan 2009 13:07:01 +0900 (JST) Message-ID: <gregnicholson@example.org> Date: Thu, 01 Jan 2009 13:07:00 +0900 From: Koji Sekiguchi <gregnicholson@example.org> User-Agent: Thunderbird 83.161.120.155 (Macintosh/20081209) MIME-Version: 1.0 To: gregnicholson@example.org Subject: Re: http internal error if i enable debugQuery=on References: <gregnicholson@example.org> <gregnicholson@example.org> <gregnicholson@example.org> In-Reply-To: <gregnicholson@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org I can reproduce the error using example of current trunk. I'm seeing what wrong now... Koji vinay kumar kaku wrote: > you mean query term? if so i do have ?q=3Dxyz etc... if not what is tha= t parameter you are talking about?i will send to only one mailing list he= re on.thanks,vinay> Date: Mon, 29 Dec 2008 18:38:33 -0800> From: otis_gos= gregnicholson@example.org> Subject: Re: http internal error if i enable debugQue= ry=3Don> To: gregnicholson@example.org> > Hi,> > Do you have q=3DXXX pa= rameter in your URL as well? Try specifying it.> > > Otis --> Sematext -= - http://sematext.com/ -- Lucene - Solr - Nutch> > > > ----- Original Mes= sage ---->> From: vinay kumar kaku <gregnicholson@example.org>>> To: solr-user@lu= cene.apache.org>> Sent: Monday, December 29, 2008 6:09:24 PM>> Subject: h= ttp internal error if i enable debugQuery=3Don>> >> >> Hi, i got solr on= 23rd dec and using for my application. i have a filter >> &fq=3D-filtern= ame[1+TO+*] all works fine until i say &debugQuery=3Don i get below >> er= ror from my tomcat catalina.out logfile. if i change that filter to say >= > &fq=3D-filtername[1+TO+10]&debugQuery=3Don it works fine.HTTP Status 50= 0 - >> java.lang.NullPointerException: value cannot be null java.lang.Run= timeException: >> java.lang.NullPointerException: value cannot be null at= >> org.apache.solr.search.QueryParsing.toString(QueryParsing.java:469) a= t >> org.apache.solr.handler.component.DebugComponent.process(DebugCompon= ent.java:75) >> at >> org.apache.solr.handler.component.SearchHandler.han= dleRequestBody(SearchHandler.java:179) >> at >> org.apache.solr.handler.R= equestHandlerBase.handleRequest(RequestHandlerBase.java:131) >> at >> org= =2Eapache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleReque= st(RequestHandlers.java:233) >> at org.apache.solr.core.SolrCore.execute(= SolrCore.java:1325) at >> org.apache.solr.servlet.SolrDispatchFilter.exec= ute(SolrDispatchFilter.java:303) >> at >> org.apache.solr.servlet.SolrDis= patchFilter.doFilter(SolrDispatchFilter.java:232) >> at >> org.apache.cat= alina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain= =2Ejava:235) >> at >> org.apache.catalina.core.ApplicationFilterChain.doF= ilter(ApplicationFilterChain.java:206) >> at >> org.apache.catalina.core.= StandardWrapperValve.invoke(StandardWrapperValve.java:233) >> at >> org.a= pache.catalina.core.StandardContextValve.invoke(StandardContextValve.java= :175) >> at org.apache.catalina.core.StandardHostValve.invoke(StandardHos= tValve.java:128) >> at org.apache.catalina.valves.ErrorReportValve.invoke= (ErrorReportValve.java:102) >> at org.apache.catalina.valves.AccessLogVal= ve.invoke(AccessLogValve.java:568) at >> org.apache.catalina.core.Standar= dEngineValve.invoke(StandardEngineValve.java:109) >> at org.apache.catali= na.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) >> at org.apac= he.coyote.http11.Http11Processor.process(Http11Processor.java:844) at >> = org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(H= ttp11Protocol.java:583) >> at org.apache.tomcat.util.net.JIoEndpoint$Work= er.run(JIoEndpoint.java:447) at >> java.lang.Thread.run(Thread.java:619) = Caused by: java.lang.NullPointerException: >> value cannot be null at org= =2Eapache.lucene.document.Field.(Field.java:275) >> at org.apache.lucene.= document.Field.(Field.java:251) at >> org.apache.solr.search.QueryParsing= =2EwriteFieldVal(QueryParsing.java:306) at >> org.apache.solr.search.Quer= yParsing.toString(QueryParsing.java:338) at >> org.apache.solr.search.Que= ryParsing.toString(QueryParsing.java:401) at >> org.apache.solr.search.Qu= eryParsing.toString(QueryParsing.java:466) ... 20 more >> thanks,vinay>> = _________________________________________________________________>> Send = e-mail anywhere. No map, no compass.>> http://windowslive.com/oneline/hot= mail?ocid=3DTXT_TAGLM_WL_hotmail_acq_anywhere_122008>=20 > _________________________________________________________________ > Send e-mail faster without improving your typing skills. > http://windowslive.com/online/hotmail?ocid=3DTXT_TAGLM_WL_hotmail_acq_s= peed_122008 > =20 From solr-usergregnicholson@example.org Thu Jan 01 04:09:01 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 72488 invoked from network); 1 Jan 2009 04:09:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 04:09:01 -0000 Received: (qmail 40004 invoked by uid 500); 1 Jan 2009 04:08:58 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 39968 invoked by uid 500); 1 Jan 2009 04:08:58 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 39957 invoked by uid 99); 1 Jan 2009 04:08:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Dec 2008 20:08:58 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO radix.cryptio.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 04:08:50 +0000 Received: by radix.cryptio.net (Postfix, from userid 1007) id 0A9C371C2E5; Wed, 31 Dec 2008 20:08:30 -0800 (PST) Received: from localhost (localhost [83.161.120.155]) by radix.cryptio.net (Postfix) with ESMTP id 07F8971C2E1 for <gregnicholson@example.org>; Wed, 31 Dec 2008 20:08:30 -0800 (PST) Date: Wed, 31 Dec 2008 20:08:30 -0800 (PST) From: Chris Hostetter <gregnicholson@example.org> To: gregnicholson@example.org Subject: Re: incremental update In-Reply-To: <gregnicholson@example.org> Message-ID: <gregnicholson@example.org> References: <gregnicholson@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org : Is it possible to incrementally index given document? Meaning, I would : like to index filed with large size separate request so that even if it : fails I would have basic document indexed. if i'm understanding your question, you mean you want to be able to send seperate requests to index various pieces of a single document so if later pieces fail, the first pieces are still indexed ... correct? there's nothing like that in Solr .. SOlr will abort if any failure happens when adding a document. -Hoss From solr-usergregnicholson@example.org Thu Jan 01 04:12:19 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 72706 invoked from network); 1 Jan 2009 04:12:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 04:12:19 -0000 Received: (qmail 41884 invoked by uid 500); 1 Jan 2009 04:12:15 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 41856 invoked by uid 500); 1 Jan 2009 04:12:15 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 41845 invoked by uid 99); 1 Jan 2009 04:12:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Dec 2008 20:12:15 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO radix.cryptio.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 04:12:08 +0000 Received: by radix.cryptio.net (Postfix, from userid 1007) id 6EDB371C1FC; Wed, 31 Dec 2008 20:11:47 -0800 (PST) Received: from localhost (localhost [83.161.120.155]) by radix.cryptio.net (Postfix) with ESMTP id 6D2EB71C1FB for <gregnicholson@example.org>; Wed, 31 Dec 2008 20:11:47 -0800 (PST) Date: Wed, 31 Dec 2008 20:11:47 -0800 (PST) From: Chris Hostetter <gregnicholson@example.org> To: gregnicholson@example.org Subject: Re: Full reindex needed if termVectors added to fields in schema? In-Reply-To: <gregnicholson@example.org> Message-ID: <gregnicholson@example.org> References: <gregnicholson@example.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Checked: Checked by ClamAV on apache.org : So, my question is whether adding termVectors="true" to a couple of schema : fields will work similarly to adding new fields, where the updated documents : will get the vectors added and the others won't get them but will continue : to work, allowing me to rebuild "in the background" while not breaking : anything in my existing incremental update/release cycle. I beielve it should work ... if the document you are generating a MLT query for was indexed before the termVectors="true" option was added, the MLT class should use it's fallback behavior of reanalizing the sored text ... if it was reindexed after that, then the termVectors will be used ... i think. -Hoss From solr-usergregnicholson@example.org Thu Jan 01 06:32:23 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 13041 invoked from network); 1 Jan 2009 06:32:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 06:32:22 -0000 Received: (qmail 67813 invoked by uid 500); 1 Jan 2009 06:32:15 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 67782 invoked by uid 500); 1 Jan 2009 06:32:15 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 67771 invoked by uid 99); 1 Jan 2009 06:32:15 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Dec 2008 22:32:15 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail.asahi-net.or.jp) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 06:32:06 +0000 Received: from koji-sekiguchi-no-macbook.local (w244069.ppp.asahi-net.or.jp [83.161.120.155]) by mail.asahi-net.or.jp (Postfix) with ESMTP id 6B36E64D53 for <gregnicholson@example.org>; Thu, 1 Jan 2009 15:31:44 +0900 (JST) Message-ID: <gregnicholson@example.org> Date: Thu, 01 Jan 2009 15:31:44 +0900 From: Koji Sekiguchi <gregnicholson@example.org> User-Agent: Thunderbird 83.161.120.155 (Macintosh/20081209) MIME-Version: 1.0 To: gregnicholson@example.org Subject: Re: http internal error if i enable debugQuery=on References: <gregnicholson@example.org> <gregnicholson@example.org> <gregnicholson@example.org> <gregnicholson@example.org> In-Reply-To: <gregnicholson@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Vinay, This is a bug. I opened SOLR-947. The fix will be committed shortly. So please try the next nightly build. Thank you, Koji From solr-usergregnicholson@example.org Thu Jan 01 23:21:31 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 29029 invoked from network); 1 Jan 2009 23:21:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 23:21:31 -0000 Received: (qmail 99440 invoked by uid 500); 1 Jan 2009 23:21:28 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 99408 invoked by uid 500); 1 Jan 2009 23:21:28 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 99397 invoked by uid 99); 1 Jan 2009 23:21:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 15:21:27 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO bay0-omc3-s2.bay0.hotmail.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 23:21:15 +0000 Received: from BAY114-W38 ([83.161.120.155]) by bay0-omc3-s2.bay0.hotmail.com with Microsoft SMTPSVC501.744.6741); Thu, 1 Jan 2009 15:20:53 -0800 Message-ID: <gregnicholson@example.org> Content-Type: multipart/alternative; boundary="_1baca663-a444-405f-9f33-835357d4c1fd_" X-Originating-IP: [83.161.120.155] From: vinay kumar kaku <gregnicholson@example.org> To: <gregnicholson@example.org> Subject: RE: http internal error if i enable debugQuery=on Date: Thu, 1 Jan 2009 23:20:53 +0000 Importance: Normal In-Reply-To: <gregnicholson@example.org> References: <gregnicholson@example.org> <gregnicholson@example.org> <gregnicholson@example.org> <gregnicholson@example.org> <gregnicholson@example.org> MIME-Version: 1.0 X-OriginalArrivalTime: 01 Jan 2009 23:20:53.0624 (UTC) FILETIME=[97846F80:01C96C67] X-Virus-Checked: Checked by ClamAV on apache.org --_1baca663-a444-405f-9f33-835357d4c1fd_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable thank you. > Date: Thu=2C 1 Jan 2009 15:31:44 +0900 > From: gregnicholson@example.org > To: gregnicholson@example.org > Subject: Re: http internal error if i enable debugQuery=3Don >=20 > Vinay=2C >=20 > This is a bug. I opened SOLR-947. The fix will be committed shortly. > So please try the next nightly build. >=20 > Thank you=2C >=20 > Koji >=20 >=20 _________________________________________________________________ It=92s the same Hotmail=AE. If by =93same=94 you mean up to 70% faster. http://windowslive.com/online/hotmail?ocid=3DTXT_TAGLM_WL_hotmail_acq_broad= 1_122008= --_1baca663-a444-405f-9f33-835357d4c1fd_-- From solr-usergregnicholson@example.org Fri Jan 02 05:39:03 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 18709 invoked from network); 2 Jan 2009 05:39:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 05:39:02 -0000 Received: (qmail 27719 invoked by uid 500); 2 Jan 2009 05:38:59 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 27691 invoked by uid 500); 2 Jan 2009 05:38:59 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 27680 invoked by uid 99); 2 Jan 2009 05:38:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 21:38:59 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 05:38:51 +0000 Received: by rv-out-0506.google.com with SMTP id f6so8306238rvb.5 for <gregnicholson@example.org>; Thu, 01 Jan 2009 21:38:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=28LTdiBNvlu2SW4Hj2ztzzn88BefBuDsWDXJ9NPh7fk=; b=WHn3kZEyGibmDN7SIhyumKDiWCV3G19hVyB3BiMkZj9IF/heZd4rKqy/CuS0ofxbms XL8TEP2UEzUIREAhbP8nAMj71WyZO++utljE/Wggj5bmQecBKivvWGYulR4MP7u5ihjY aqw+4YhEI0Ddw5jJu6fLsUGRyCTi09FtwR8b8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=wuuP8ra1UUjtCgIHPB0Ju+8ZDPuGE/Xiv7pfg2dyQDl1azKBNSR1l8FRyktKMlXqRd qJvw9d5y8XNYNdNs3xR5pe4QOBNktU8XCYJ62YmNngAG4LQQp5D3Lv2j8sGMcv0KLNEY +vUD+fF/Li4qS1BdxCx+L05tpQ1VI8NTfLB/c= Received: by 83.161.120.155 with SMTP id z13mr8701014rvc.30.1230874710443; Thu, 01 Jan 2009 21:38:30 -0800 (PST) Received: by 83.161.120.155 with HTTP; Thu, 1 Jan 2009 21:38:30 -0800 (PST) Message-ID: <gregnicholson@example.org> Date: Thu, 1 Jan 2009 21:38:30 -0800 From: "Lance Norskog" <gregnicholson@example.org> To: gregnicholson@example.org Subject: [JOB] Solr expert available MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_172859_9921745.1230874710437" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_172859_9921745.1230874710437 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Please contact me if you want to hire an expert Java and Solr developer/architect. My recent experience is with a half-billion record gregnicholson@example.org, supplying our video search database to www.videocrawler.com. Thanks, Lance Norskog gregnicholson@example.org ------=_Part_172859_9921745.1230874710437-- From solr-usergregnicholson@example.org Fri Jan 02 08:42:05 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 81964 invoked from network); 2 Jan 2009 08:42:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 08:42:04 -0000 Received: (qmail 78518 invoked by uid 500); 2 Jan 2009 08:42:01 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 78483 invoked by uid 500); 2 Jan 2009 08:42:01 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 78472 invoked by uid 99); 2 Jan 2009 08:42:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 00:42:01 -0800 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 08:41:52 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <gregnicholson@example.org>) id 1LIfb1-0003el-Bs for gregnicholson@example.org; Fri, 02 Jan 2009 00:41:31 -0800 Message-ID: <gregnicholson@example.org> Date: Fri, 2 Jan 2009 00:41:31 -0800 (PST) From: RaghavPrabhu <gregnicholson@example.org> To: gregnicholson@example.org Subject: How can i omit the illegal characters,when indexing the docs? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Nabble-From: gregnicholson@example.org X-Virus-Checked: Checked by ClamAV on apache.org Hi all, I am extracting the word document using Apache POI,then generate the xml doc,which is the document that i want to indexing in the solr. The problem which i faced was,it thrown the error in the browser is shown below. HTTP Status 500 - Illegal character ((CTRL-CHAR, code 8)) at [row,col {unknown-source}]: [1,1592]=20 com.ctc.wstx.exc.WstxUnexpectedCharException: Illegal character ((CTRL-CHAR= , code 8)) at [row,col {unknown-source}]: [1,1592] at com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:675) at com.ctc.wstx.sr.StreamScanner.throwInvalidSpace(StreamScanner.java:660) at com.ctc.wstx.sr.BasicStreamReader.readCDataPrimary(BasicStreamReader.java:4= 240) at com.ctc.wstx.sr.BasicStreamReader.nextFromTreeCommentOrCData(BasicStreamRea= der.java:3280) at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2824) at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1019) at org.apache.solr.handler.XmlUpdateRequestHandler.readDoc(XmlUpdateRequestHan= dler.java:321) at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequ= estHandler.java:195) at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdate= RequestHandler.java:123) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase= .java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:= 303) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java= :232) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio= nFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC= hain.java:206) at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.j= ava:96) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio= nFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC= hain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j= ava:230) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j= ava:175) at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssoc= iationValve.java:179) at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java= :84) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12= 7) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10= 2) at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnect= ionValve.java:157) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav= a:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Htt= p11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446) at java.lang.Thread.run(Thread.java:619)=20 The extracted word document contains the special character (=0Bits like a square box).How can i omit those characters,when i submit the document to the solr. Thanks in advance, Regards Prabhu.K=0B=0B=0B --=20 View this message in context: http://www.nabble.com/How-can-i-omit-the-ille= gal-characters%2Cwhen-indexing-the-docs--tp21249084p21249084.html Sent from the Solr - User mailing list gregnicholson@example.org. From solr-usergregnicholson@example.org Fri Jan 02 12:35:01 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 73692 invoked from network); 2 Jan 2009 12:35:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 12:35:00 -0000 Received: (qmail 10885 invoked by uid 500); 2 Jan 2009 12:34:56 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 10862 invoked by uid 500); 2 Jan 2009 12:34:55 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 10850 invoked by uid 99); 2 Jan 2009 12:34:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 04:34:55 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO yw-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 12:34:49 +0000 Received: by yw-out-2324.google.com with SMTP id 3so1856787ywj.5 for <gregnicholson@example.org>; Fri, 02 Jan 2009 04:34:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=yofY3x9Mnw8HSjFXAgYgoSHsqMyzrkojpAoAlQHhEs4=; b=U/FoY0FOfZUhAMiGLfmiv+yBzdQo2Z0QYy+lqwzI5iAenIJnHPLHj48Qg55rRYBDPW AdlwYyVRhs5VUO95Pp3fcWa9J99XvuOUfCjzuv+LoB8E7iOzPBuyekpZHgiT9EXCxUb4 pWZmRLcf/XUkb5QPVaU6/QBqaOjvi/LOPdb3A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=uYZv3HUJHkwxBbA9sRk9evDn0fSi1Bp/6EYW7Mdvug8DKhhKhoKSdyq6jIBuFJmG32 tiwcqUR00Ok1jVSUNQN8zQNH5tpDDaxCLpJKa9kRuzcug50Q07uazJxHfx4vXbybDQ7R movg66j8G+4Fsa6DDe+BBSli9pL47Hc0hYLtk= Received: by 83.161.120.155 with SMTP id w15mr9569482ybn.52.1230899668079; Fri, 02 Jan 2009 04:34:28 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Jan 2009 04:34:28 -0800 (PST) Message-ID: <gregnicholson@example.org> Date: Fri, 2 Jan 2009 10:34:28 -0200 From: "Alexander Ramos Jardim" <gregnicholson@example.org> To: gregnicholson@example.org Subject: Re: synonyms.txt file updated frequently In-Reply-To: <gregnicholson@example.org> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_173530_7607400.1230899668074" References: <gregnicholson@example.org> <gregnicholson@example.org> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_173530_7607400.1230899668074 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline People, Thanks for all the replies, The business requirement I have is to update the synonyms list every time someone from the sales department establishes a new dictionary (they do that a couple times in a week) I must add the new synonyms to the index. I think I will stick with query time synonyms only for Grant's reason. At least bad is better than worse. 2008/12/31 Grant Ingersoll <gregnicholson@example.org> > > On Dec 30, 2008, at 4:38 PM, Smiley, David W. wrote: > > Grant, the Solr wiki recommends doing expansion at index time and gives >> reasons: >> >> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#head-2c461ac74b4ddd82e453dc68fcfc92da77358d46 >> >> > I personally think "recommends" is too strong of a word, but the points are > valid reasons to do index time synonyms. In Alexandar's case, I think > index-time is a bit more problematic, since he is frequently updating the > synonym list, meaning he would have to reindex every time, otherwise his > stats are going to be even more skewed. > > As for multi-word expansions, the query parser can be fixed or an alternate > one used. > > > > Query-time doesn't work for multi-word expansion. For everyone's >> convenience, I'll quote the remainder of the problems: >> >> >> Even when you aren't worried about multi-word synonyms, idf differences >> still make index time synonyms a good idea. Consider the following scenario: >> >> * An index with a "text" field, which at query time uses the >> SynonymFilter with the synonym TV, Televesion and expand="true" >> * Many thousands of documents containing the term "text:TV" >> * A few hundred documents containing the term "text:Television" >> >> A query for text:TV will expand into (text:TV text:Television) and the >> lower docFreq for text:Television will give the documents that match >> "Television" a much higher score then docs that match "TV" comparably -- >> which may be somewhat counter intuitive to the client. Index time expansion >> (or reduction) will result in the same idf for all documents regardless of >> which term the original text contained. >> >> ~ David Smiley >> >> On 12/30/08 4:33 PM, "Grant Ingersoll" <gregnicholson@example.org> wrote: >> >> >> >> On Dec 30, 2008, at 11:05 AM, Alexander Ramos Jardim wrote: >> >> Hey Grant, >>> >>> Thanks for the info! >>> >>> 2008/12/30 Grant Ingersoll <gregnicholson@example.org> >>> >>> I'd probably write a new TokenFilter that was aware of the reload >>>> policy >>>> (in a generic way) such that I didn't have to go through a whole >>>> core reload >>>> every time. Are you just using them during query time or also during >>>> indexing? >>>> >>>> >>> I am using it at indexing time. >>> >> >> I think that is a bit more problematic. How do you deal with new >> documents having the new synonyms while old docs don't? >> >> Any particular reason you use syns at indexing and not search? Not >> saying there aren't reasons to do it, just query side usually works >> better for this very reason. >> >> > -------------------------- > Grant Ingersoll > > Lucene Helpful Hints: > http://wiki.apache.org/lucene-java/BasicsOfPerformance > http://wiki.apache.org/lucene-java/LuceneFAQ > > > > > > > > > > > -- Alexander Ramos Jardim ------=_Part_173530_7607400.1230899668074-- From solr-usergregnicholson@example.org Fri Jan 02 15:25:36 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 87361 invoked from network); 2 Jan 2009 15:25:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 15:25:36 -0000 Received: (qmail 84769 invoked by uid 500); 2 Jan 2009 15:25:33 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 84731 invoked by uid 500); 2 Jan 2009 15:25:33 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 84720 invoked by uid 99); 2 Jan 2009 15:25:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 07:25:32 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO yx-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 15:25:24 +0000 Received: by yx-out-2324.google.com with SMTP id 3so1869718yxj.5 for <gregnicholson@example.org>; Fri, 02 Jan 2009 07:25:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=G0/qBKQjWextYe592bkPV/LWM+Gl6PHHpR/Z4XRQ+g0=; b=A5LqDXSpxb9vId/P+O4FO/sQwHkf5SZu0/Au4n62GTBztMGOgsPaIHJSEGsGP8aooP elH+rG2i58z43qYbHKxoRzgyeHyjqTRS6VKg2KdPlDBp78bpxy4XRz2QuS9/VlvjcSOG 9PnYyxH+/uzuoiAB5ssnMuZWbALTEYeekoiEQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=uERs4O4Q14TUIpNK0I5DROgZP+eqQFRcAbdtLwF1qm3vecPfQBmav+8B56CSk299DP l3u5KpkzfnNY3uz3MrrAhbJ3PtwlAy6Ub8pqrRN/j+RKea3W292SCjQjcpIw17WkgYKm p/o3AQdVebxGTRRiDUbmWb+IBjtKdgADUEb3A= Received: by 83.161.120.155 with SMTP id r17mr1545777ybd.106.1230909903887; Fri, 02 Jan 2009 07:25:03 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Jan 2009 07:25:03 -0800 (PST) Message-ID: <gregnicholson@example.org> Date: Fri, 2 Jan 2009 13:25:03 -0200 From: "Alexander Ramos Jardim" <gregnicholson@example.org> To: gregnicholson@example.org Subject: Re: synonyms.txt file updated frequently In-Reply-To: <gregnicholson@example.org> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_174148_6908117.1230909903879" References: <gregnicholson@example.org> <gregnicholson@example.org> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_174148_6908117.1230909903879 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Grant, I am following your idea to write a new TokenFilter. As long as I looked in SynonymTokenFilter and Factory code, it is the Factory the reponsible for loading the new Just let me make some stupid questions: 1. I will have to write a custom TokenFilter and TokenFilterFactory, right? 2. SynonymTokenFilterFactory does the "synonyms.txt" parse and creates the SynonymTokenFilter instance. If I want the SynonymTokenFilter to reload synonyms.txt file from time to time, I will need to put the file load and parsing strategies inside my new TokenFilter, right? 2008/12/30 Grant Ingersoll <gregnicholson@example.org> > I'd probably write a new TokenFilter that was aware of the reload policy > (in a generic way) such that I didn't have to go through a whole core reload > every time. Are you just using them during query time or also during > indexing? > > -Grant > > > On Dec 30, 2008, at 8:12 AM, Alexander Ramos Jardim wrote: > > Hello guys, >> >> As the title suggests I must update my synonyms.txt file frequently. What >> is >> the best approach? Should I send a commit after the file is updated? Does >> Solr need to be restarted after the file changes? >> >> -- >> Alexander Ramos Jardim >> > > -------------------------- > Grant Ingersoll > > Lucene Helpful Hints: > http://wiki.apache.org/lucene-java/BasicsOfPerformance > http://wiki.apache.org/lucene-java/LuceneFAQ > > > > > > > > > > > -- Alexander Ramos Jardim ------=_Part_174148_6908117.1230909903879-- From solr-usergregnicholson@example.org Fri Jan 02 15:53:32 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 96793 invoked from network); 2 Jan 2009 15:53:32 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 15:53:32 -0000 Received: (qmail 96446 invoked by uid 500); 2 Jan 2009 15:53:28 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 96416 invoked by uid 500); 2 Jan 2009 15:53:28 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 96405 invoked by uid 99); 2 Jan 2009 15:53:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 07:53:28 -0800 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO yw-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 15:53:21 +0000 Received: by yw-out-2324.google.com with SMTP id 3so1873264ywj.5 for <gregnicholson@example.org>; Fri, 02 Jan 2009 07:52:59 -0800 (PST) Received: by 83.161.120.155 with SMTP id m8mr12119131ybd.230.1230911579198; Fri, 02 Jan 2009 07:52:59 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Jan 2009 07:52:59 -0800 (PST) Message-ID: <gregnicholson@example.org> Date: Fri, 2 Jan 2009 10:52:59 -0500 From: "Brian Whitman" <gregnicholson@example.org> To: gregnicholson@example.org Subject: cannot allocate memory for snapshooter MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_128685_32671240.1230911579190" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_128685_32671240.1230911579190 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I have an indexing machine on a test server (a mid-level EC2 instance, 8GB of RAM) and I run jetty like: java -server -Xms5g -Xmx5g -XX:MaxPermSize=128m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap -Dsolr.solr.home=/vol/solr -Djava.awt.headless=true -jar start.jar The indexing master is set to snapshoot on commit. Sometimes (not always) the snapshot fails with SEVERE: java.io.IOException: Cannot run program "/vol/solr/bin/snapshooter": java.io.IOException: error=12, Cannot allocate gregnicholson@example.org(Unknown Source) Why would snapshooter need more than 2GB ram? /proc/meminfo says (with solr running & nothing else) MemTotal: 7872040 kB MemFree: 2018404 kB Buffers: 67704 kB Cached: 2161880 kB SwapCached: 0 kB Active: 3446348 kB Inactive: 2186964 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 8 kB Writeback: 0 kB AnonPages: 3403728 kB Mapped: 12016 kB Slab: 37804 kB SReclaimable: 20048 kB SUnreclaim: 17756 kB PageTables: 7476 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 3936020 kB Committed_AS: 5383624 kB VmallocTotal: 34359738367 kB VmallocUsed: 340 kB VmallocChunk: 34359738027 kB ------=_Part_128685_32671240.1230911579190-- From solr-usergregnicholson@example.org Fri Jan 02 16:21:09 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 3705 invoked from network); 2 Jan 2009 16:21:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 16:21:09 -0000 Received: (qmail 18610 invoked by uid 500); 2 Jan 2009 16:21:04 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 18584 invoked by uid 500); 2 Jan 2009 16:21:04 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 18573 invoked by uid 99); 2 Jan 2009 16:21:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 08:21:04 -0800 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO yw-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 16:20:58 +0000 Received: by yw-out-2324.google.com with SMTP id 3so1876098ywj.5 for <gregnicholson@example.org>; Fri, 02 Jan 2009 08:20:36 -0800 (PST) Received: by 83.161.120.155 with SMTP id z10mr8356132ybz.12.1230913236655; Fri, 02 Jan 2009 08:20:36 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Jan 2009 08:20:36 -0800 (PST) Message-ID: <gregnicholson@example.org> Date: Fri, 2 Jan 2009 11:20:36 -0500 From: "Brian Whitman" <gregnicholson@example.org> To: gregnicholson@example.org Subject: debugging long commits MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_128792_23679684.1230913236650" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_128792_23679684.1230913236650 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline We have a distributed setup that has been experiencing glacially slow commit times on only some of the shards. (10s on a good shard, 263s on a slow shard.) Each shard for this index has about 10GB of lucene index data and the documents are segregated by an md5 hash, so the distribution of document/data types should be equal across all shards. I've turned off our postcommit hooks to isolate the problem, so it's not a snapshot run amok or anything. I also moved the indexes over to new machines and the same indexes that were slow in production are also slow on the test machines. During the slow commit, the jetty process is 100% CPU / 50% RAM on a 8GB quad core machine. The slow commit happens every time after I add at least one document. (If I don't add any documents the commit is immediate.) What can I do to look into this problem? ------=_Part_128792_23679684.1230913236650-- From solr-usergregnicholson@example.org Fri Jan 02 16:56:08 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 16210 invoked from network); 2 Jan 2009 16:56:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 16:56:08 -0000 Received: (qmail 64765 invoked by uid 500); 2 Jan 2009 16:56:02 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 64484 invoked by uid 500); 2 Jan 2009 16:56:01 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 64473 invoked by uid 99); 2 Jan 2009 16:56:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 08:56:01 -0800 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO yx-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 16:55:50 +0000 Received: by yx-out-2324.google.com with SMTP id 3so1878892yxj.5 for <gregnicholson@example.org>; Fri, 02 Jan 2009 08:55:28 -0800 (PST) Received: by 83.161.120.155 with SMTP id n14mr5717029ybm.121.1230915327867; Fri, 02 Jan 2009 08:55:27 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Jan 2009 08:55:27 -0800 (PST) Message-ID: <gregnicholson@example.org> Date: Fri, 2 Jan 2009 11:55:27 -0500 From: "Brian Whitman" <gregnicholson@example.org> To: gregnicholson@example.org Subject: Re: debugging long commits In-Reply-To: <gregnicholson@example.org> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_129136_32267571.1230915327846" References: <gregnicholson@example.org> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_129136_32267571.1230915327846 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Not sure if these help. Here's the stack trace and jmap -histo output during a long (bad) commit Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode): "Attach Listener" daemon prio=10 tid=0x00002aabf9954400 nid=0x5e1c runnable [0x0000000000000000..0x0000000042048d20] java.lang.Thread.State: RUNNABLE "Lucene Merge Thread #0" daemon prio=10 tid=0x00002aabf9c7f400 nid=0x5e00 runnable [0x0000000041f47000..0x0000000041f47ba0] java.lang.Thread.State: gregnicholson@example.org(RAMOutputStream.java:147) at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:56) at org.apache.lucene.index.MultiLevelSkipListWriter.writeSkip(MultiLevelSkipListWriter.java:146) at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:588) at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:546) at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:499) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4291) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3932) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) "pool-2-thread-1" prio=10 tid=0x00002aabf9b94c00 nid=0x5dc3 waiting on condition [0x0000000041a42000..0x0000000041a42aa0] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaabecd12d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(Unknown Source) at java.util.concurrent.DelayQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) "DestroyJavaVM" prio=10 tid=0x00002aabfc006c00 nid=0x5da3 waiting on condition [0x0000000000000000..0x000000004022ad10] java.lang.Thread.State: RUNNABLE "Timer-2" daemon prio=10 tid=0x00002aabfc007c00 nid=0x5dc1 in Object.wait() [0x0000000041e46000..0x0000000041e46d20] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabef91338> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Unknown Source) - locked <0x00002aaabef91338> (a java.util.TaskQueue) at java.util.TimerThread.run(Unknown Source) "Checkpointer" daemon prio=10 tid=0x00002aabf995b800 nid=0x5dc0 in Object.wait() [0x0000000041d45000..0x0000000041d45da0] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabebf9700> (a java.lang.Object) at java.lang.Object.wait(Object.java:485) at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:152) - locked <0x00002aaabebf9700> (a java.lang.Object) at java.lang.Thread.run(Unknown Source) "Cleaner-1" daemon prio=10 tid=0x00002aabf995b000 nid=0x5dbf in Object.wait() [0x0000000041c44000..0x0000000041c44c20] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabebee158> (a java.lang.Object) at java.lang.Object.wait(Object.java:485) at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:152) - locked <0x00002aaabebee158> (a java.lang.Object) at java.lang.Thread.run(Unknown Source) "INCompressor" daemon prio=10 tid=0x00002aabf9a3cc00 nid=0x5dbe in Object.wait() [0x0000000041b43000..0x0000000041b43ca0] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabeef3500> (a java.lang.Object) at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:154) - locked <0x00002aaabeef3500> (a java.lang.Object) at java.lang.Thread.run(Unknown Source) "pool-1-thread-1" prio=10 tid=0x00002aabfa015800 nid=0x5dbc waiting on condition [0x0000000041941000..0x0000000041941ba0] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00002aaabef91a08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) "Timer-1" daemon prio=10 tid=0x00002aabfa0f1000 nid=0x5dbb in Object.wait() [0x0000000041840000..0x0000000041840a20] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabef19a48> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Unknown Source) - locked <0x00002aaabef19a48> (a java.util.TaskQueue) at java.util.TimerThread.run(Unknown Source) "btpool0-9 - Acceptor0 gregnicholson@example.org:80" prio=10 tid=0x00002aabf9807800 nid=0x5dba runnable [0x000000004173f000..0x000000004173faa0] java.lang.Thread.State: gregnicholson@example.org(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source) at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) - locked <0x00002aaabebcb2a8> (a sun.nio.ch.Util$1) - locked <0x00002aaabebcb290> (a java.util.Collections$UnmodifiableSet) - locked <0x00002aaabebc8b00> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:411) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:167) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488) "btpool0-8 - Acceptor1 gregnicholson@example.org:80" prio=10 tid=0x00002aabf980d000 nid=0x5db9 runnable [0x000000004163e000..0x000000004163ed20] java.lang.Thread.State: gregnicholson@example.org(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(Unknown Source) at sun.nio.ch.EPollSelectorImpl.doSelect(Unknown Source) at sun.nio.ch.SelectorImpl.lockAndDoSelect(Unknown Source) - locked <0x00002aaabebcb5f8> (a sun.nio.ch.Util$1) - locked <0x00002aaabebcb5e0> (a java.util.Collections$UnmodifiableSet) - locked <0x00002aaabebc4d30> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(Unknown Source) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:411) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:167) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488) "btpool0-7" prio=10 tid=0x00002aabf9594800 nid=0x5db8 in Object.wait() [0x000000004153c000..0x000000004153dda0] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaad9d0c3f0> (a org.apache.solr.update.SolrIndexWriter) at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4554) - locked <0x00002aaad9d0c3f0> (a org.apache.solr.update.SolrIndexWriter) at org.apache.lucene.index.IndexWriter.finishMerges(IndexWriter.java:2873) - locked <0x00002aaad9d0c3f0> (a org.apache.solr.update.SolrIndexWriter) at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1718) at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1674) at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1648) at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:153) at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:175) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:359) at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:77) at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate(XmlUpdateRequestHandler.java:226) at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody(XmlUpdateRequestHandler.java:123) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:205) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488) "btpool0-6" prio=10 tid=0x00002aabf98fdc00 nid=0x5db7 in Object.wait() [0x000000004143c000..0x000000004143cc20] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabf15c7b0> (a org.mortbay.thread.QueuedThreadPool$PoolThread) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:541) - locked <0x00002aaabf15c7b0> (a org.mortbay.thread.QueuedThreadPool$PoolThread) "btpool0-5" prio=10 tid=0x00002aabf9b10000 nid=0x5db6 in Object.wait() [0x000000004133b000..0x000000004133bca0] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabf160ab8> (a org.mortbay.thread.QueuedThreadPool$PoolThread) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:541) - locked <0x00002aaabf160ab8> (a org.mortbay.thread.QueuedThreadPool$PoolThread) "btpool0-4" prio=10 tid=0x00002aabf9b97400 nid=0x5db5 in Object.wait() [0x000000004123a000..0x000000004123ab20] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabf160d08> (a org.mortbay.thread.QueuedThreadPool$PoolThread) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:541) - locked <0x00002aaabf160d08> (a org.mortbay.thread.QueuedThreadPool$PoolThread) "btpool0-3" prio=10 tid=0x00002aabf9b96000 nid=0x5db4 in Object.wait() [0x0000000041139000..0x0000000041139ba0] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabf160f58> (a org.mortbay.thread.QueuedThreadPool$PoolThread) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:541) - locked <0x00002aaabf160f58> (a org.mortbay.thread.QueuedThreadPool$PoolThread) "btpool0-2" prio=10 tid=0x00002aabf9a07000 nid=0x5db3 in Object.wait() [0x0000000041038000..0x0000000041038a20] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabf1611a8> (a org.mortbay.thread.QueuedThreadPool$PoolThread) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:541) - locked <0x00002aaabf1611a8> (a org.mortbay.thread.QueuedThreadPool$PoolThread) "btpool0-1" prio=10 tid=0x00002aabf9324400 nid=0x5db2 in Object.wait() [0x0000000040f37000..0x0000000040f37aa0] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabf1613f8> (a org.mortbay.thread.QueuedThreadPool$PoolThread) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:541) - locked <0x00002aaabf1613f8> (a org.mortbay.thread.QueuedThreadPool$PoolThread) "btpool0-0" prio=10 tid=0x00002aabf9a83800 nid=0x5db1 in Object.wait() [0x0000000040e36000..0x0000000040e36d20] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabf161648> (a org.mortbay.thread.QueuedThreadPool$PoolThread) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:541) - locked <0x00002aaabf161648> (a org.mortbay.thread.QueuedThreadPool$PoolThread) "Timer-0" daemon prio=10 tid=0x00002aabf9af6000 nid=0x5db0 in Object.wait() [0x0000000040d35000..0x0000000040d35da0] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabebacf80> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Unknown Source) - locked <0x00002aaabebacf80> (a java.util.TaskQueue) at java.util.TimerThread.run(Unknown Source) "Low Memory Detector" daemon prio=10 tid=0x00002aabf933fc00 nid=0x5dac runnable [0x0000000000000000..0x0000000000000000] java.lang.Thread.State: RUNNABLE "CompilerThread1" daemon prio=10 tid=0x00002aabf933d000 nid=0x5dab waiting on condition [0x0000000000000000..0x0000000040a31360] java.lang.Thread.State: RUNNABLE "CompilerThread0" daemon prio=10 tid=0x00002aabf9339000 nid=0x5daa waiting on condition [0x0000000000000000..0x00000000409303e0] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00002aabf9337400 nid=0x5da9 waiting on condition [0x0000000000000000..0x00000000408306b0] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00002aabf9314400 nid=0x5da8 in Object.wait() [0x000000004072f000..0x000000004072faa0] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabeb86f50> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) - locked <0x00002aaabeb86f50> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(Unknown Source) at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source) "Reference Handler" daemon prio=10 tid=0x00002aabf9312800 nid=0x5da7 in Object.wait() [0x000000004062e000..0x000000004062ed20] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00002aaabeb86ec8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source) - locked <0x00002aaabeb86ec8> (a java.lang.ref.Reference$Lock) "VM Thread" prio=10 tid=0x00002aabf917a000 nid=0x5da6 runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000004011c800 nid=0x5da4 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000004011e000 nid=0x5da5 runnable "VM Periodic Task Thread" prio=10 tid=0x00002aabf9342000 nid=0x5dad waiting on condition JNI global references: 971 Heap PSYoungGen total 1395264K, used 965841K [0x00002aab8d4b0000, 0x00002aabf7f50000, 0x00002aabf7f50000) eden space 1030400K, 93% used [0x00002aab8d4b0000,0x00002aabc83d4788,0x00002aabcc2f0000) from space 364864K, 0% used [0x00002aabe1b00000,0x00002aabe1b10000,0x00002aabf7f50000) to space 352320K, 0% used [0x00002aabcc2f0000,0x00002aabcc2f0000,0x00002aabe1b00000) PSOldGen total 3495296K, used 642758K [0x00002aaab7f50000, 0x00002aab8d4b0000, 0x00002aab8d4b0000) object space 3495296K, 18% used [0x00002aaab7f50000,0x00002aaadf301a78,0x00002aab8d4b0000) PSPermGen total 21248K, used 19258K [0x00002aaaaff50000, 0x00002aaab1410000, 0x00002aaab7f50000) object space 21248K, 90% used [0x00002aaaaff50000,0x00002aaab121e8d8,0x00002aaab1410000) num #instances #bytes class name ---------------------------------------------- 1: 6459678 491568792 [C 2: 6456059 258242360 java.lang.String 3: 6282264 251290560 org.apache.lucene.index.TermInfo 4: 6282189 201030048 org.apache.lucene.index.Term 5: 70220 39109632 [I 6: 6082 25264288 [B 7: 149 20355504 [J 8: 133 20354208 [Lorg.apache.lucene.index.Term; 9: 133 20354208 [Lorg.apache.lucene.index.TermInfo; 10: 160230 8972880 java.nio.HeapByteBuffer 11: 160218 8972208 java.nio.HeapCharBuffer 12: 160210 8971760 org.apache.lucene.index.FieldsReader$FieldForMerge 13: 30440 4095480 <constMethodKlass> 14: 30440 3660128 <methodKlass> 15: 2605 3026184 <constantPoolKlass> 16: 22065 3025120 [Ljava.lang.Object; 17: 1297 2411792 [Ljava.util.HashMap$Entry; 18: 48691 2309696 <symbolKlass> 19: 2604 1981728 <instanceKlassKlass> 20: 2194 1889888 <constantPoolCacheKlass> 21: 27444 1317312 java.util.HashMap$Entry 22: 24954 998160 java.util.AbstractList$Itr 23: 18834 753360 org.apache.lucene.index.FieldInfo 24: 2846 523664 java.lang.Class 25: 13021 520840 java.util.ArrayList 26: 12471 399072 org.apache.lucene.document.Document 27: 3895 372216 [[I 28: 3904 309592 [S 29: 534 249632 <methodDataKlass> 30: 3451 220864 org.apache.lucene.index.SegmentReader$Norm 31: 1547 136136 org.apache.lucene.store.FSDirectory$FSIndexInput 32: 213 120984 <objArrayKlassKlass> 33: 737 112024 java.lang.reflect.Method 34: 1575 100800 java.lang.ref.Finalizer 35: 1345 86080 org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor 36: 1188 76032 java.util.HashMap ... On Fri, Jan 2, 2009 at 11:20 AM, Brian Whitman <gregnicholson@example.org> wrote: > We have a distributed setup that has been experiencing glacially slow > commit times on only some of the shards. (10s on a good shard, 263s on a > slow shard.) Each shard for this index has about 10GB of lucene index data > and the documents are segregated by an md5 hash, so the distribution of > document/data types should be equal across all shards. I've turned off our > postcommit hooks to isolate the problem, so it's not a snapshot run amok or > anything. I also moved the indexes over to new machines and the same indexes > that were slow in production are also slow on the test machines. > During the slow commit, the jetty process is 100% CPU / 50% RAM on a 8GB > quad core machine. The slow commit happens every time after I add at least > one document. (If I don't add any documents the commit is immediate.) > > What can I do to look into this problem? > > > > ------=_Part_129136_32267571.1230915327846-- From solr-usergregnicholson@example.org Fri Jan 02 17:04:21 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 20496 invoked from network); 2 Jan 2009 17:04:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 17:04:21 -0000 Received: (qmail 80055 invoked by uid 500); 2 Jan 2009 17:04:17 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 80023 invoked by uid 500); 2 Jan 2009 17:04:17 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 80012 invoked by uid 99); 2 Jan 2009 17:04:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 09:04:17 -0800 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gregnicholson@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 17:04:10 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <gregnicholson@example.org>) id 1LInR7-0005rr-EO for gregnicholson@example.org; Fri, 02 Jan 2009 09:03:49 -0800 Message-ID: <gregnicholson@example.org> Date: Fri, 2 Jan 2009 09:03:49 -0800 (PST) From: Sushil Vegad <gregnicholson@example.org> To: gregnicholson@example.org Subject: Highlighting not working MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: gregnicholson@example.org X-Virus-Checked: Checked by ClamAV on apache.org Hi, I cant get highlighting to work. I tried everything mentioned about it on the forum. PLEASE HELP... We use solrJ; search a field called content, it is the default search field, indexed and stored. Its type is text, has analyzer associated with it. There is no uniqueKey in the schema SolrQuery query = new SolrQuery(); query.setHighlight(true); query.setQuery(queryStr); When this query is fired, it throws the exception below that says "Message: requires 'name' attribute: lst". What am I doing wrong? I have attached the solrconfig.xml for your reference. THANK YOU. The exception trace is: org.apache.solr.client.solrj.SolrServerException: Error executing gregnicholson@example.org(QueryRequest.java:88) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:126) at test.Client.main(Client.java:143) Caused by: org.apache.solr.common.SolrException: parsing gregnicholson@example.org(XMLResponseParser.java:140) at org.apache.solr.client.solrj.impl.XMLResponseParser.processResponse(XMLResponseParser.java:106) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:427) at org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:206) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:84) ... 2 more Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[3,1440] Message: requires 'name' attribute: gregnicholson@example.org(XMLResponseParser.java:284) at org.apache.solr.client.solrj.impl.XMLResponseParser.readNamedList(XMLResponseParser.java:291) at org.apache.solr.client.solrj.impl.XMLResponseParser.processResponse(XMLResponseParser.java:127) http://www.nabble.com/file/p21254049/solrconfig.xml solrconfig.xml -- View this message in context: http://www.nabble.com/Highlighting-not-working-tp21254049p21254049.html Sent from the Solr - User mailing list gregnicholson@example.org. From solr-usergregnicholson@example.org Fri Jan 02 18:17:44 2009 Return-Path: <solr-usergregnicholson@example.org> Delivered-To: gregnicholson@example.org Received: (qmail 59146 invoked from network); 2 Jan 2009 18:17:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 18:17:43 -0000 Received: (qmail 45287 invoked by uid 500); 2 Jan 2009 18:17:40 -0000 Delivered-To: gregnicholson@example.org Received: (qmail 45051 invoked by uid 500); 2 Jan 2009 18:17:40 -0000 Mailing-List: contact gregnicholson@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:gregnicholson@example.org> List-Unsubscribe: <mailto:gregnicholson@example.org> List-Post: <mailto:gregnicholson@example.org> List-Id: <solr-user.lucene.apache.org> Reply-To: gregnicholson@example.org Delivered-To: mailing list gregnicholson@example.org Received: (qmail 45040 invoked by uid 99); 2 Jan 2009 18:17:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 10:17:39 -0800 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO yx-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 18:17:33 +0000 Received: by yx-out-2324.google.com with SMTP id 3so1887343yxj.5 for <gregnicholson@example.org>; Fri, 02 Jan 2009 10:17:11 -0800 (PST) Received: by 83.161.120.155 with SMTP id f5mr1801108yba.210.1230920231336; Fri, 02 Jan 2009 10:17:11 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Jan 2009 10:17:11 -0800 (PST) Message-ID: <gregnicholson@example.org> Date: Fri, 2 Jan 2009 13:17:11 -0500 From: "Brian Whitman" <gregnicholson@example.org> To: gregnicholson@example.org Subject: Re: debugging long commits In-Reply-To: <gregnicholson@example.org> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_129823_28648805.1230920231337" References: <gregnicholson@example.org> <gregnicholson@example.org> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_129823_28648805.1230920231337 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I think I'm getting close with this (sorry for the self-replies) I tried an optimize (which we never do) and it took 30m and said this a lot: Exception in thread "Lucene Merge Thread #4" org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: gregnicholson@example.org(ConcurrentMergeScheduler.java:314) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291) Caused by: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: gregnicholson@example.org(BitVector.java:91) at org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:125) at org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:98) at org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:633) at org.apache.lucene.index.SegmentMerger.mergeTermInfo(SegmentMerger.java:585) at org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:546) at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:499) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4291) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3932) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) Jan 2, 2009 6:05:49 PM org.apache.solr.common.SolrException log SEVERE: java.io.IOException: background merge hit exception: _ks4:C2504982 _oaw:C514635 _tll:C827949 _tdx:C18372 _te8:C19929 _tej:C22201 _1agw:C1717926 into _1agy [optimize] at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2346) at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2280) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:355) at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcesso ... But then it finished. And now commits are OK again. Anyone
{'source': 'Apache Software Foundation Public Mail Archives'}