From solr-commits-brookschristopher@example.org Sat Nov 01 04:50:18 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 97218 invoked from network); 1 Nov 2008 04:50:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Nov 2008 04:50:18 -0000 Received: (qmail 94864 invoked by uid 500); 1 Nov 2008 04:50:23 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 94827 invoked by uid 500); 1 Nov 2008 04:50:23 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 94818 invoked by uid 99); 1 Nov 2008 04:50:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2008 21:50:23 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Nov 2008 04:49:16 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 0DFA0238889E; Fri, 31 Oct 2008 21:49:57 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r709649 - in /lucene/solr/trunk/contrib/velocity: ./ src/main/java/ src/main/java/org/apache/solr/request/ src/main/solr/conf/velocity/ Date: Sat, 01 Nov 2008 04:49:56 -0000 To: brookschristopher@example.org From: brookschristopher@example.org X-Mailer: svnmailer-1.0.8 Message-Id: <brookschristopher@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: ehatcher Date: Fri Oct 31 21:49:56 2008 New Revision: 709649 URL: http://svn.apache.org/viewvc?rev=709649&view=rev Log: Wire in a SolrResourceLoader bridge for Velocity templates, allowing a JAR of templates to live in conf/lib. Refactored the footer.vm template to use this mechanism as an example. Added: lucene/solr/trunk/contrib/velocity/src/main/java/footer.vm lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrVelocityResourceLoader.java Removed: lucene/solr/trunk/contrib/velocity/src/main/solr/conf/velocity/footer.vm Modified: lucene/solr/trunk/contrib/velocity/build.xml lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrParamResourceLoader.java lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/VelocityResponseWriter.java lucene/solr/trunk/contrib/velocity/src/main/solr/conf/velocity/browse.vm Modified: lucene/solr/trunk/contrib/velocity/build.xml URL: http://svn.apache.org/viewvc/lucene/solr/trunk/contrib/velocity/build.xml?rev=709649&r1=709648&r2=709649&view=diff ============================================================================== --- lucene/solr/trunk/contrib/velocity/build.xml (original) +++ lucene/solr/trunk/contrib/velocity/build.xml Fri Oct 31 21:49:56 2008 @@ -61,7 +61,9 @@ <target name="build" depends="compile"> <solr-jar destfile="src/main/solr/lib/${fullnamever}.jar" basedir="target/classes" - manifest="${common.dir}/${dest}/META-INF/MANIFEST.MF" /> + manifest="${common.dir}/${dest}/META-INF/MANIFEST.MF"> + <fileset dir="src/main/java" excludes="**/*.java"/> + </solr-jar> </target> <target name="compileTests" depends="compile"> Added: lucene/solr/trunk/contrib/velocity/src/main/java/footer.vm URL: http://svn.apache.org/viewvc/lucene/solr/trunk/contrib/velocity/src/main/java/footer.vm?rev=709649&view=auto ============================================================================== --- lucene/solr/trunk/contrib/velocity/src/main/java/footer.vm (added) +++ lucene/solr/trunk/contrib/velocity/src/main/java/footer.vm Fri Oct 31 21:49:56 2008 @@ -0,0 +1,9 @@ +#if($params.getBool("debugQuery",false)) +<pre> + request = $request + + response = $response.values +</pre> +#end + +Generated by <a href="http://wiki.apache.org/solr/VelocityResponseWriter">VelocityResponseWriter</a> \ No newline at end of file Modified: lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrParamResourceLoader.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrParamResourceLoader.java?rev=709649&r1=709648&r2=709649&view=diff ============================================================================== --- lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrParamResourceLoader.java (original) +++ lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrParamResourceLoader.java Fri Oct 31 21:49:56 2008 @@ -30,6 +30,7 @@ public class SolrParamResourceLoader extends ResourceLoader { private HashMap<String,String> templates = new HashMap(); public SolrParamResourceLoader(SolrQueryRequest request) { + super(); // TODO: Consider using content streams, but need a template name associated with each stream // for now, a custom param convention of template.<name>=<template body> is a nice example @@ -44,7 +45,6 @@ templates.put(name.substring(9) + ".vm",params.get(name)); } } - } public void init(ExtendedProperties extendedProperties) { Added: lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrVelocityResourceLoader.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrVelocityResourceLoader.java?rev=709649&view=auto ============================================================================== --- lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrVelocityResourceLoader.java (added) +++ lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/SolrVelocityResourceLoader.java Fri Oct 31 21:49:56 2008 @@ -0,0 +1,34 @@ +package org.apache.solr.request; + +import org.apache.velocity.runtime.resource.loader.ResourceLoader; +import org.apache.velocity.runtime.resource.Resource; +import org.apache.velocity.exception.ResourceNotFoundException; +import org.apache.commons.collections.ExtendedProperties; +import org.apache.solr.core.SolrResourceLoader; + +import java.io.InputStream; + +// TODO: the name of this class seems ridiculous +public class SolrVelocityResourceLoader extends ResourceLoader { + private SolrResourceLoader loader; + + public SolrVelocityResourceLoader(SolrResourceLoader loader) { + super(); + this.loader = loader; + } + + public void init(ExtendedProperties extendedProperties) { + } + + public InputStream getResourceStream(String template_name) throws ResourceNotFoundException { + return loader.openResource(template_name); + } + + public boolean isSourceModified(Resource resource) { + return false; + } + + public long getLastModified(Resource resource) { + return 0; + } +} Modified: lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/VelocityResponseWriter.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/VelocityResponseWriter.java?rev=709649&r1=709648&r2=709649&view=diff ============================================================================== --- lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/VelocityResponseWriter.java (original) +++ lucene/solr/trunk/contrib/velocity/src/main/java/org/apache/solr/request/VelocityResponseWriter.java Fri Oct 31 21:49:56 2008 @@ -49,8 +49,10 @@ baseDir = new File(template_root); } engine.setProperty(VelocityEngine.FILE_RESOURCE_LOADER_PATH, baseDir.getAbsolutePath()); - engine.setProperty("params.resource.loader.instance",new SolrParamResourceLoader(request)); - engine.setProperty(VelocityEngine.RESOURCE_LOADER, "params,file"); + engine.setProperty("params.resource.loader.instance", new SolrParamResourceLoader(request)); + engine.setProperty("solr.resource.loader.instance", + new SolrVelocityResourceLoader(request.getCore().getSolrConfig().getResourceLoader())); + engine.setProperty(VelocityEngine.RESOURCE_LOADER, "params,file,solr"); return engine; } Modified: lucene/solr/trunk/contrib/velocity/src/main/solr/conf/velocity/browse.vm URL: http://svn.apache.org/viewvc/lucene/solr/trunk/contrib/velocity/src/main/solr/conf/velocity/browse.vm?rev=709649&r1=709648&r2=709649&view=diff ============================================================================== --- lucene/solr/trunk/contrib/velocity/src/main/solr/conf/velocity/browse.vm (original) +++ lucene/solr/trunk/contrib/velocity/src/main/solr/conf/velocity/browse.vm Fri Oct 31 21:49:56 2008 @@ -44,19 +44,7 @@ <div class="pagination"> <span><span class="results-found">$page.results_found</span> results found in ${response.responseHeader.QTime} ms</span> </div> -#if (${nlResponse.suggestions.size()} > 0) -<div class="spelling"> - <span>Did you mean?</span> - #foreach ($suggestion in ${nlResponse.suggestions}) - <a href="${request.requestURL}?q=${suggestion}">${suggestion}</a>#if ($velocityCount < $nlResponse.suggestions.size()),#end - #end -</div> -#end -#if (${nlResponse.autoSpell} == true) -<div class="autoSpell">There were no results for your original query, so we automatically searched a few spelling - variants: ${nlResponse.autoSpellQuery} -</div> -#end + <div class="results"> #foreach($id in $doclist.iterator()) #set($doc = $searcher.doc($id,$response.returnFields)) From solr-commits-brookschristopher@example.org Sat Nov 01 04:52:31 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 97866 invoked from network); 1 Nov 2008 04:52:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Nov 2008 04:52:31 -0000 Received: (qmail 96615 invoked by uid 500); 1 Nov 2008 04:52:37 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 96579 invoked by uid 500); 1 Nov 2008 04:52:36 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 96570 invoked by uid 99); 1 Nov 2008 04:52:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Oct 2008 21:52:36 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Nov 2008 04:51:29 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 6AB47238889E; Fri, 31 Oct 2008 21:52:10 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r709650 - in /lucene/solr/trunk/contrib/velocity: ./ src/main/solr/lib/ Date: Sat, 01 Nov 2008 04:52:10 -0000 To: brookschristopher@example.org From: brookschristopher@example.org X-Mailer: svnmailer-1.0.8 Message-Id: <brookschristopher@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: ehatcher Date: Fri Oct 31 21:52:10 2008 New Revision: 709650 URL: http://svn.apache.org/viewvc?rev=709650&view=rev Log: Add ignores Modified: lucene/solr/trunk/contrib/velocity/ (props changed) lucene/solr/trunk/contrib/velocity/src/main/solr/lib/ (props changed) Propchange: lucene/solr/trunk/contrib/velocity/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Fri Oct 31 21:52:10 2008 @@ -0,0 +1 @@ +target Propchange: lucene/solr/trunk/contrib/velocity/src/main/solr/lib/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Fri Oct 31 21:52:10 2008 @@ -0,0 +1 @@ +apache-solr-velocity-*.jar From solr-commits-brookschristopher@example.org Sat Nov 01 10:35:33 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 22359 invoked from network); 1 Nov 2008 10:35:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Nov 2008 10:35:33 -0000 Received: (qmail 72155 invoked by uid 500); 1 Nov 2008 10:35:39 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 72113 invoked by uid 500); 1 Nov 2008 10:35:38 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 72103 invoked by uid 99); 1 Nov 2008 10:35:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Nov 2008 03:35:37 -0700 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Nov 2008 10:34:30 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 0AB0F238896D; Sat, 1 Nov 2008 03:34:41 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r709668 - in /lucene/solr/trunk/contrib/velocity/src/test/java: ./ TODO.txt Date: Sat, 01 Nov 2008 10:34:40 -0000 To: brookschristopher@example.org From: brookschristopher@example.org X-Mailer: svnmailer-1.0.8 Message-Id: <brookschristopher@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: ehatcher Date: Sat Nov 1 03:34:39 2008 New Revision: 709668 URL: http://svn.apache.org/viewvc?rev=709668&view=rev Log: test placeholder - tests will be forthcoming, promise Added: lucene/solr/trunk/contrib/velocity/src/test/java/ lucene/solr/trunk/contrib/velocity/src/test/java/TODO.txt Added: lucene/solr/trunk/contrib/velocity/src/test/java/TODO.txt URL: http://svn.apache.org/viewvc/lucene/solr/trunk/contrib/velocity/src/test/java/TODO.txt?rev=709668&view=auto ============================================================================== --- lucene/solr/trunk/contrib/velocity/src/test/java/TODO.txt (added) +++ lucene/solr/trunk/contrib/velocity/src/test/java/TODO.txt Sat Nov 1 03:34:39 2008 @@ -0,0 +1 @@ +Yes, tests are forthcoming! Sorry Hudson. \ No newline at end of file From solr-commits-brookschristopher@example.org Mon Nov 03 14:33:21 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 96447 invoked from network); 3 Nov 2008 14:33:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Nov 2008 14:33:21 -0000 Received: (qmail 56770 invoked by uid 500); 3 Nov 2008 14:33:27 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 56742 invoked by uid 500); 3 Nov 2008 14:33:26 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 56728 invoked by uid 99); 3 Nov 2008 14:33:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 06:33:26 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 14:32:18 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 609F123888F1; Mon, 3 Nov 2008 06:32:59 -0800 (PST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r710068 - in /lucene/solr/trunk: example/solr/conf/ src/java/org/apache/solr/common/util/ src/java/org/apache/solr/search/ src/test/org/apache/solr/search/ src/test/test-files/solr/conf/ Date: Mon, 03 Nov 2008 14:32:58 -0000 To: brookschristopher@example.org From: brookschristopher@example.org X-Mailer: svnmailer-1.0.8 Message-Id: <brookschristopher@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: yonik Date: Mon Nov 3 06:32:58 2008 New Revision: 710068 URL: http://svn.apache.org/viewvc?rev=710068&view=rev Log: SOLR-667: change markAndSweep algorithm, make FastLRUCache the default for filterCache in example and test Modified: lucene/solr/trunk/example/solr/conf/solrconfig.xml lucene/solr/trunk/src/java/org/apache/solr/common/util/ConcurrentLRUCache.java lucene/solr/trunk/src/java/org/apache/solr/search/FastLRUCache.java lucene/solr/trunk/src/test/org/apache/solr/search/TestFastLRUCache.java lucene/solr/trunk/src/test/test-files/solr/conf/solrconfig.xml Modified: lucene/solr/trunk/example/solr/conf/solrconfig.xml URL: http://svn.apache.org/viewvc/lucene/solr/trunk/example/solr/conf/solrconfig.xml?rev=710068&r1=710067&r2=710068&view=diff ============================================================================== --- lucene/solr/trunk/example/solr/conf/solrconfig.xml (original) +++ lucene/solr/trunk/example/solr/conf/solrconfig.xml Mon Nov 3 06:32:58 2008 @@ -209,6 +209,12 @@ <maxBooleanClauses>1024</maxBooleanClauses> + <!-- There are two implementations of cache available for Solr, + LRUCache, based on a synchronized LinkedHashMap, and + FastLRUCache, based on a ConcurrentHashMap. FastLRUCache has faster gets + and slower puts in single threaded operation and thus is generally faster + than LRUCache when the hit ratio of the cache is high (> 75%), and may be + faster under other scenarios on multi-cpu systems. --> <!-- Cache used by SolrIndexSearcher for filters (DocSets), unordered sets of *all* documents that match a query. When a new searcher is opened, its caches may be prepopulated @@ -216,7 +222,7 @@ autowarmCount is the number of items to prepopulate. For LRUCache, the autowarmed items will be the most recently accessed items. Parameters: - class - the SolrCache implementation (currently only LRUCache) + class - the SolrCache implementation LRUCache or FastLRUCache size - the maximum number of entries in the cache initialSize - the initial capacity (number of entries) of the cache. (seel java.util.HashMap) @@ -224,7 +230,7 @@ and old cache. --> <filterCache - class="solr.LRUCache" + class="solr.FastLRUCache" size="512" initialSize="512" autowarmCount="128"/> Modified: lucene/solr/trunk/src/java/org/apache/solr/common/util/ConcurrentLRUCache.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/src/java/org/apache/solr/common/util/ConcurrentLRUCache.java?rev=710068&r1=710067&r2=710068&view=diff ============================================================================== --- lucene/solr/trunk/src/java/org/apache/solr/common/util/ConcurrentLRUCache.java (original) +++ lucene/solr/trunk/src/java/org/apache/solr/common/util/ConcurrentLRUCache.java Mon Nov 3 06:32:58 2008 @@ -1,5 +1,7 @@ package org.apache.solr.common.util; +import org.apache.lucene.util.PriorityQueue; + import java.util.LinkedHashMap; import java.util.Map; import java.util.TreeSet; @@ -26,11 +28,12 @@ private final int upperWaterMark, lowerWaterMark; private volatile boolean stop = false; private final ReentrantLock markAndSweepLock = new ReentrantLock(true); - private volatile boolean isCleaning = false; + private boolean isCleaning = false; // not volatile... piggybacked on other volatile vars private final boolean newThreadForCleanup; private volatile boolean islive = true; private final Stats stats = new Stats(); private final int acceptableWaterMark; + private long oldestEntry = 0; // not volatile, only accessed in the cleaning method public ConcurrentLRUCache(int upperWaterMark, final int lowerWaterMark, int acceptableWatermark, int initialSize, boolean runCleanupThread, boolean runNewThreadForCleanup, final int delay) { if (upperWaterMark < 1) throw new IllegalArgumentException("upperWaterMark must be > 0"); @@ -99,6 +102,9 @@ // // There is a race between the check and the call to markAndSweep, but // it's unimportant because markAndSweep actually aquires the lock or returns if it can't. + // + // Thread safety note: isCleaning read is piggybacked (comes after) other volatile reads + // in this method. if (stats.size.get() > upperWaterMark && !isCleaning) { if (newThreadForCleanup) { new Thread() { @@ -122,63 +128,232 @@ * config parameter, the second stage takes over. * <p/> * The second stage is more intensive and tries to bring down the cache size - * to the 'minSize' config parameter. + * to the 'lowerWaterMark' config parameter. */ - public void markAndSweep() { + private void markAndSweep() { + // if we want to keep at least 1000 entries, then timestamps of + // current through current-1000 are guaranteed not to be the oldest (but that does + // not mean there are 1000 entries in that group... it's acutally anywhere between + // 1 and 1000). + // Also, if we want to remove 500 entries, then + // oldestEntry through oldestEntry+500 are guaranteed to be + // removed (however many there are there). + if (!markAndSweepLock.tryLock()) return; try { + long oldestEntry = this.oldestEntry; isCleaning = true; - int size = stats.size.get(); - long currentLatestAccessed = stats.accessCounter.get(); - int itemsToBeRemoved = size - lowerWaterMark; - int itemsRemoved = 0; - if (itemsToBeRemoved < 1) return; - // currentLatestAccessed is the counter value of the item accessed most recently - // therefore remove all items whose last accessed counter is less than (currentLatestAccessed - lowerWaterMark) - long removeOlderThan = currentLatestAccessed - lowerWaterMark; - for (Map.Entry<Object, CacheEntry> entry : map.entrySet()) { - if (entry.getValue().lastAccessed <= removeOlderThan && itemsRemoved < itemsToBeRemoved) { - evictEntry(entry.getKey()); + this.oldestEntry = oldestEntry; // volatile write to make isCleaning visible + + long timeCurrent = stats.accessCounter.get(); + int sz = stats.size.get(); + + int numRemoved = 0; + int numKept = 0; + long newestEntry = timeCurrent; + long newNewestEntry = -1; + long newOldestEntry = Integer.MAX_VALUE; + + int wantToKeep = lowerWaterMark; + int wantToRemove = sz - lowerWaterMark; + + CacheEntry[] eset = new CacheEntry[sz]; + int eSize = 0; + + // System.out.println("newestEntry="+newestEntry + " oldestEntry="+oldestEntry); + // System.out.println("items removed:" + numRemoved + " numKept=" + numKept + " esetSz="+ eSize + " sz-numRemoved=" + (sz-numRemoved)); + + for (CacheEntry ce : map.values()) { + // set lastAccessedCopy to avoid more volatile reads + ce.lastAccessedCopy = ce.lastAccessed; + long thisEntry = ce.lastAccessedCopy; + + // since the wantToKeep group is likely to be bigger than wantToRemove, check it first + if (thisEntry > newestEntry - wantToKeep) { + // this entry is guaranteed not to be in the bottom + // group, so do nothing. + numKept++; + newOldestEntry = Math.min(thisEntry, newOldestEntry); + } else if (thisEntry < oldestEntry + wantToRemove) { // entry in bottom group? + // this entry is guaranteed to be in the bottom group + // so immediately remove it from the map. + evictEntry(ce.key); + numRemoved++; + } else { + // This entry *could* be in the bottom group. + // Collect these entries to avoid another full pass... this is wasted + // effort if enough entries are normally removed in this first pass. + // An alternate impl could make a full second pass. + if (eSize < eset.length-1) { + eset[eSize++] = ce; + newNewestEntry = Math.max(thisEntry, newNewestEntry); + newOldestEntry = Math.min(thisEntry, newOldestEntry); + } } } - // Since the removal of items in the above loop depends on the value of the lastAccessed variable, - // between the time we recorded the number of items to be removed and the actual removal process, - // some items may graduate above the removeOlderThan value and escape eviction. - // Therefore, we again check if the size less than acceptableWaterMark, if not we remove items forcefully - // using a method which does not depend on the value of lastAccessed but can be more costly to run - - size = stats.size.get(); - // In the first attempt, try to use a simple algorithm to remove old entries - // If the size of the cache is <= acceptableWatermark then return - if (size <= acceptableWaterMark) return; - // Remove items until size becomes lower than acceptableWaterMark - itemsToBeRemoved = size - acceptableWaterMark; - TreeSet<CacheEntry> tree = new TreeSet<CacheEntry>(); - // This loop may remove a few newer items because we try to forcefully fill a - // bucket of fixed size and remove them even if they have become newer in the meantime - // The caveat is that this may lead to more cache misses because we may have removed - // an item which was used very recently (against the philosophy of LRU) - for (Map.Entry<Object, CacheEntry> entry : map.entrySet()) { - CacheEntry v = entry.getValue(); - v.lastAccessedCopy = v.lastAccessed; - if (tree.size() < itemsToBeRemoved) { - tree.add(v); - } else { - if (v.lastAccessedCopy < tree.first().lastAccessedCopy) { - tree.remove(tree.first()); - tree.add(v); + // System.out.println("items removed:" + numRemoved + " numKept=" + numKept + " esetSz="+ eSize + " sz-numRemoved=" + (sz-numRemoved)); + // TODO: allow this to be customized in the constructor? + int numPasses=1; // maximum number of linear passes over the data + + // if we didn't remove enough entries, then make more passes + // over the values we collected, with updated min and max values. + while (sz - numRemoved > acceptableWaterMark && --numPasses>=0) { + + oldestEntry = newOldestEntry == Integer.MAX_VALUE ? oldestEntry : newOldestEntry; + newOldestEntry = Integer.MAX_VALUE; + newestEntry = newNewestEntry; + newNewestEntry = -1; + wantToKeep = lowerWaterMark - numKept; + wantToRemove = sz - lowerWaterMark - numRemoved; + + // iterate backward to make it easy to remove items. + for (int i=eSize-1; i>=0; i--) { + CacheEntry ce = eset[i]; + long thisEntry = ce.lastAccessedCopy; + + if (thisEntry > newestEntry - wantToKeep) { + // this entry is guaranteed not to be in the bottom + // group, so do nothing but remove it from the eset. + numKept++; + // remove the entry by moving the last element to it's position + eset[i] = eset[eSize-1]; + eSize--; + + newOldestEntry = Math.min(thisEntry, newOldestEntry); + + } else if (thisEntry < oldestEntry + wantToRemove) { // entry in bottom group? + + // this entry is guaranteed to be in the bottom group + // so immediately remove it from the map. + evictEntry(ce.key); + numRemoved++; + + // remove the entry by moving the last element to it's position + eset[i] = eset[eSize-1]; + eSize--; + } else { + // This entry *could* be in the bottom group, so keep it in the eset, + // and update the stats. + newNewestEntry = Math.max(thisEntry, newNewestEntry); + newOldestEntry = Math.min(thisEntry, newOldestEntry); } } + // System.out.println("items removed:" + numRemoved + " numKept=" + numKept + " esetSz="+ eSize + " sz-numRemoved=" + (sz-numRemoved)); } - for (CacheEntry sortCacheEntry : tree) - evictEntry(sortCacheEntry.key); + + + + // if we still didn't remove enough entries, then make another pass while + // inserting into a priority queue + if (sz - numRemoved > acceptableWaterMark) { + + oldestEntry = newOldestEntry == Integer.MAX_VALUE ? oldestEntry : newOldestEntry; + newOldestEntry = Integer.MAX_VALUE; + newestEntry = newNewestEntry; + newNewestEntry = -1; + wantToKeep = lowerWaterMark - numKept; + wantToRemove = sz - lowerWaterMark - numRemoved; + + PQueue queue = new PQueue(wantToRemove); + + for (int i=eSize-1; i>=0; i--) { + CacheEntry ce = eset[i]; + long thisEntry = ce.lastAccessedCopy; + + if (thisEntry > newestEntry - wantToKeep) { + // this entry is guaranteed not to be in the bottom + // group, so do nothing but remove it from the eset. + numKept++; + // removal not necessary on last pass. + // eset[i] = eset[eSize-1]; + // eSize--; + + newOldestEntry = Math.min(thisEntry, newOldestEntry); + + } else if (thisEntry < oldestEntry + wantToRemove) { // entry in bottom group? + // this entry is guaranteed to be in the bottom group + // so immediately remove it. + evictEntry(ce.key); + numRemoved++; + + // removal not necessary on last pass. + // eset[i] = eset[eSize-1]; + // eSize--; + } else { + // This entry *could* be in the bottom group. + // add it to the priority queue + + // everything in the priority queue will be removed, so keep track of + // the lowest value that ever comes back out of the queue. + + // first reduce the size of the priority queue to account for + // the number of items we have already removed while executing + // this loop so far. + queue.myMaxSize = sz - lowerWaterMark - numRemoved; + while (queue.size() > queue.myMaxSize && queue.size() > 0) { + CacheEntry otherEntry = (CacheEntry) queue.pop(); + newOldestEntry = Math.min(otherEntry.lastAccessedCopy, newOldestEntry); + } + if (queue.myMaxSize <= 0) break; + + Object o = queue.myInsertWithOverflow(ce); + if (o != null) { + newOldestEntry = Math.min(((CacheEntry)o).lastAccessedCopy, newOldestEntry); + } + } + } + + // Now delete everything in the priority queue. + // avoid using pop() since order doesn't matter anymore + for (Object o : queue.getValues()) { + if (o==null) continue; + CacheEntry ce = (CacheEntry)o; + evictEntry(ce.key); + numRemoved++; + } + + // System.out.println("items removed:" + numRemoved + " numKept=" + numKept + " initialQueueSize="+ wantToRemove + " finalQueueSize=" + queue.size() + " sz-numRemoved=" + (sz-numRemoved)); + } + + oldestEntry = newOldestEntry == Integer.MAX_VALUE ? oldestEntry : newOldestEntry; + this.oldestEntry = oldestEntry; } finally { - isCleaning = false; + isCleaning = false; // set before markAndSweep.unlock() for visibility markAndSweepLock.unlock(); } } + private static class PQueue extends PriorityQueue { + int myMaxSize; + PQueue(int maxSz) { + super.initialize(maxSz); + myMaxSize = maxSz; + } + + Object[] getValues() { return heap; } + + protected boolean lessThan(Object a, Object b) { + // reverse the parameter order so that the queue keeps the oldest items + return ((CacheEntry)b).lastAccessedCopy < ((CacheEntry)a).lastAccessedCopy; + } + + // necessary because maxSize is private in base class + public Object myInsertWithOverflow(Object element) { + if (size() < myMaxSize) { + put(element); + return null; + } else if (size() > 0 && !lessThan(element, heap[1])) { + Object ret = heap[1]; + heap[1] = element; + adjustTop(); + return ret; + } else { + return element; + } + } + } + private void evictEntry(Object key) { Object o = map.remove(key); @@ -189,6 +364,7 @@ public Map getLatestAccessedItems(long n) { + // we need to grab the lock since we are changing lastAccessedCopy markAndSweepLock.lock(); Map result = new LinkedHashMap(); TreeSet<CacheEntry> tree = new TreeSet<CacheEntry>(); Modified: lucene/solr/trunk/src/java/org/apache/solr/search/FastLRUCache.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/src/java/org/apache/solr/search/FastLRUCache.java?rev=710068&r1=710067&r2=710068&view=diff ============================================================================== --- lucene/solr/trunk/src/java/org/apache/solr/search/FastLRUCache.java (original) +++ lucene/solr/trunk/src/java/org/apache/solr/search/FastLRUCache.java Mon Nov 3 06:32:58 2008 @@ -44,7 +44,7 @@ this.regenerator = regenerator; name = (String) args.get("name"); String str = (String) args.get("size"); - final int limit = str == null ? 1024 : Integer.parseInt(str); + int limit = str == null ? 1024 : Integer.parseInt(str); int minLimit; str = (String) args.get("minSize"); if (str == null) { @@ -52,6 +52,9 @@ } else { minLimit = Integer.parseInt(str); } + if (minLimit==0) minLimit=1; + if (limit <= minLimit) limit=minLimit+1; + int acceptableLimit; str = (String) args.get("acceptableSize"); if (str == null) { @@ -59,12 +62,14 @@ } else { acceptableLimit = Integer.parseInt(str); } + acceptableLimit = Math.max(limit,acceptableLimit); + str = (String) args.get("initialSize"); - final int initialSize = str == null ? 1024 : Integer.parseInt(str); + final int initialSize = str == null ? limit : Integer.parseInt(str); str = (String) args.get("autowarmCount"); autowarmCount = str == null ? 0 : Integer.parseInt(str); - - description = "Concurrent LRU Cache(maxSize=" + limit + ", initialSize=" + initialSize; + + description = "Concurrent LRU Cache(maxSize=" + limit + ", initialSize=" + initialSize + ", minSize="+minLimit + ", acceptableSize="+acceptableLimit; if (autowarmCount > 0) { description += ", autowarmCount=" + autowarmCount + ", regenerator=" + regenerator; Modified: lucene/solr/trunk/src/test/org/apache/solr/search/TestFastLRUCache.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/src/test/org/apache/solr/search/TestFastLRUCache.java?rev=710068&r1=710067&r2=710068&view=diff ============================================================================== --- lucene/solr/trunk/src/test/org/apache/solr/search/TestFastLRUCache.java (original) +++ lucene/solr/trunk/src/test/org/apache/solr/search/TestFastLRUCache.java Mon Nov 3 06:32:58 2008 @@ -2,10 +2,13 @@ import junit.framework.TestCase; import org.apache.solr.common.util.NamedList; +import org.apache.solr.common.util.ConcurrentLRUCache; import java.io.IOException; import java.util.HashMap; import java.util.Map; +import java.util.Random; +import java.util.concurrent.atomic.AtomicInteger; /** @@ -40,7 +43,9 @@ assertEquals(2L, nl.get("lookups")); assertEquals(1L, nl.get("hits")); assertEquals(101L, nl.get("inserts")); - assertEquals(11L, nl.get("evictions")); + + assertEquals(null, sc.get(1)); // first item put in should be the first out + FastLRUCache scNew = new FastLRUCache(); scNew.init(l, o, cr); @@ -55,10 +60,148 @@ assertEquals(1L, nl.get("inserts")); assertEquals(0L, nl.get("evictions")); - assertEquals(4L, nl.get("cumulative_lookups")); + assertEquals(5L, nl.get("cumulative_lookups")); assertEquals(2L, nl.get("cumulative_hits")); assertEquals(102L, nl.get("cumulative_inserts")); - assertEquals(11L, nl.get("cumulative_evictions")); } + void doPerfTest(int iter, int cacheSize, int maxKey) { + long start = System.currentTimeMillis(); + + int lowerWaterMark = cacheSize; + int upperWaterMark = (int)(lowerWaterMark * 1.1); + + Random r = new Random(0); + ConcurrentLRUCache cache = new ConcurrentLRUCache(upperWaterMark, lowerWaterMark, (upperWaterMark+lowerWaterMark)/2, upperWaterMark, false, false, 0); + boolean getSize=false; + int minSize=0,maxSize=0; + for (int i=0; i<iter; i++) { + cache.put(r.nextInt(maxKey),"TheValue"); + int sz = cache.size(); + if (!getSize && sz >= cacheSize) { + getSize = true; + minSize = sz; + } else { + if (sz < minSize) minSize=sz; + else if (sz > maxSize) maxSize=sz; + } + } + + long end = System.currentTimeMillis(); + System.out.println("time=" + (end-start) + ", minSize="+minSize+",maxSize="+maxSize); + } + + /*** + public void testPerf() { + doPerfTest(1000000, 100000, 200000); // big cache, warmup + doPerfTest(2000000, 100000, 200000); // big cache + doPerfTest(2000000, 100000, 120000); // smaller key space increases distance between oldest, newest and makes the first passes less effective. + doPerfTest(6000000, 1000, 2000); // small cache, smaller hit rate + doPerfTest(6000000, 1000, 1200); // small cache, bigger hit rate + } + ***/ + + // returns number of puts + int useCache(SolrCache sc, int numGets, int maxKey, int seed) { + int ret = 0; + Random r = new Random(seed); + + // use like a cache... gets and a put if not found + for (int i=0; i<numGets; i++) { + Integer k = r.nextInt(maxKey); + Integer v = (Integer)sc.get(k); + if (v == null) { + sc.put(k, k); + ret++; + } + } + + return ret; + } + + void fillCache(SolrCache sc, int cacheSize, int maxKey) { + Random r = new Random(0); + for (int i=0; i<cacheSize; i++) { + Integer kv = r.nextInt(maxKey); + sc.put(kv,kv); + } + } + + + void cachePerfTest(final SolrCache sc, final int nThreads, final int numGets, int cacheSize, final int maxKey) { + Map l = new HashMap(); + l.put("size", ""+cacheSize); + l.put("initialSize", ""+cacheSize); + + Object o = sc.init(l, null, null); + sc.setState(SolrCache.State.LIVE); + + fillCache(sc, cacheSize, maxKey); + + long start = System.currentTimeMillis(); + + Thread[] threads = new Thread[nThreads]; + final AtomicInteger puts = new AtomicInteger(0); + for (int i=0; i<threads.length; i++) { + final int seed=i; + threads[i] = new Thread() { + public void run() { + int ret = useCache(sc, numGets/nThreads, maxKey, seed); + puts.addAndGet(ret); + } + }; + } + + for (Thread thread : threads) { + try { + thread.start(); + } catch (Exception e) { + e.printStackTrace(); + } + } + + for (Thread thread : threads) { + try { + thread.join(); + } catch (Exception e) { + e.printStackTrace(); + } + } + + long end = System.currentTimeMillis(); + System.out.println("time=" + (end-start) + " impl=" +sc.getClass().getSimpleName() + +" nThreads= " + nThreads + " size="+cacheSize+" maxKey="+maxKey+" gets="+numGets + +" hitRatio="+(1-(((double)puts.get())/numGets))); + } + + void perfTestBoth(int nThreads, int numGets, int cacheSize, int maxKey) { + cachePerfTest(new LRUCache(), nThreads, numGets, cacheSize, maxKey); + cachePerfTest(new FastLRUCache(), nThreads, numGets, cacheSize, maxKey); + } + + /*** + public void testCachePerf() { + // warmup + perfTestBoth(2, 100000, 100000, 120000); + perfTestBoth(1, 2000000, 100000, 100000); // big cache, 100% hit ratio + perfTestBoth(2, 2000000, 100000, 100000); // big cache, 100% hit ratio + perfTestBoth(1, 2000000, 100000, 120000); // big cache, bigger hit ratio + perfTestBoth(2, 2000000, 100000, 120000); // big cache, bigger hit ratio + perfTestBoth(1, 2000000, 100000, 200000); // big cache, ~50% hit ratio + perfTestBoth(2, 2000000, 100000, 200000); // big cache, ~50% hit ratio + perfTestBoth(1, 2000000, 100000, 1000000); // big cache, ~10% hit ratio + perfTestBoth(2, 2000000, 100000, 1000000); // big cache, ~10% hit ratio + + perfTestBoth(1, 2000000, 1000, 1000); // small cache, ~100% hit ratio + perfTestBoth(2, 2000000, 1000, 1000); // small cache, ~100% hit ratio + perfTestBoth(1, 2000000, 1000, 1200); // small cache, bigger hit ratio + perfTestBoth(2, 2000000, 1000, 1200); // small cache, bigger hit ratio + perfTestBoth(1, 2000000, 1000, 2000); // small cache, ~50% hit ratio + perfTestBoth(2, 2000000, 1000, 2000); // small cache, ~50% hit ratio + perfTestBoth(1, 2000000, 1000, 10000); // small cache, ~10% hit ratio + perfTestBoth(2, 2000000, 1000, 10000); // small cache, ~10% hit ratio + } + ***/ + + } Modified: lucene/solr/trunk/src/test/test-files/solr/conf/solrconfig.xml URL: http://svn.apache.org/viewvc/lucene/solr/trunk/src/test/test-files/solr/conf/solrconfig.xml?rev=710068&r1=710067&r2=710068&view=diff ============================================================================== --- lucene/solr/trunk/src/test/test-files/solr/conf/solrconfig.xml (original) +++ lucene/solr/trunk/src/test/test-files/solr/conf/solrconfig.xml Mon Nov 3 06:32:58 2008 @@ -143,7 +143,7 @@ that match a particular query. --> <filterCache - class="solr.search.LRUCache" + class="solr.search.FastLRUCache" size="512" initialSize="512" autowarmCount="256"/> From solr-commits-brookschristopher@example.org Tue Nov 04 00:34:15 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 67019 invoked from network); 4 Nov 2008 00:34:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Nov 2008 00:34:15 -0000 Received: (qmail 67056 invoked by uid 500); 4 Nov 2008 00:34:22 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 67025 invoked by uid 500); 4 Nov 2008 00:34:22 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 67016 invoked by uid 99); 4 Nov 2008 00:34:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 16:34:21 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 00:33:13 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id 0B38B1114F for <brookschristopher@example.org>; Tue, 4 Nov 2008 00:33:25 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <brookschristopher@example.org> To: brookschristopher@example.org Date: Tue, 04 Nov 2008 00:33:24 -0000 Message-ID: <brookschristopher@example.org> Subject: [Solr Wiki] Update of "SolPython" by IanHolsman X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification. The following page has been changed by IanHolsman: http://wiki.apache.org/solr/SolPython The comment on the change is: As of version 1.3, Solr no longer comes bundled with a Python client ------------------------------------------------------------------------------ - [[TableOfContents]] == Python API == - There is a simple client API as part of the Solr repository: http://svn.apache.org/repos/asf/lucene/solr/trunk/client/python/ + There is a simple client API as part of the Solr repository: http://svn.apache.org/viewvc/lucene/solr/tags/release-1.2.0/client/python/ /!\ :TODO: /!\ it would be good to have a little more info about this ... what it can do, how stable it is, etc... From solr-commits-brookschristopher@example.org Tue Nov 04 04:49:39 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 73363 invoked from network); 4 Nov 2008 04:49:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Nov 2008 04:49:39 -0000 Received: (qmail 4227 invoked by uid 500); 4 Nov 2008 04:49:45 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 4200 invoked by uid 500); 4 Nov 2008 04:49:45 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 4191 invoked by uid 99); 4 Nov 2008 04:49:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 20:49:45 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 04:48:36 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id 75A1111157 for <brookschristopher@example.org>; Tue, 4 Nov 2008 04:48:48 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <brookschristopher@example.org> To: brookschristopher@example.org Date: Tue, 04 Nov 2008 04:48:48 -0000 Message-ID: <brookschristopher@example.org> Subject: [Solr Wiki] Update of "DataImportHandler" by NoblePaul X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification. The following page has been changed by NoblePaul: http://wiki.apache.org/solr/DataImportHandler ------------------------------------------------------------------------------ * '''`deltaQuery`''' : Only used in delta-import * '''`parentDeltaQuery`''' : Only used in delta-import * '''`deletedPkQuery`''' : Only used in delta-import + * '''`deltaImportQuery`''' : (Only used in delta-import) . If this is present this will be used during the import 'phase' of delta import instead of `query` . There is a namespace `${dataimporter.delta.<column-name>}` which can be used in this query. == Commands == From solr-commits-brookschristopher@example.org Tue Nov 04 05:14:11 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 80687 invoked from network); 4 Nov 2008 05:14:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Nov 2008 05:14:10 -0000 Received: (qmail 17354 invoked by uid 500); 4 Nov 2008 05:14:16 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 17324 invoked by uid 500); 4 Nov 2008 05:14:16 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 17315 invoked by uid 99); 4 Nov 2008 05:14:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2008 21:14:16 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 05:13:08 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id 7D05611157 for <brookschristopher@example.org>; Tue, 4 Nov 2008 05:13:19 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <brookschristopher@example.org> To: brookschristopher@example.org Date: Tue, 04 Nov 2008 05:13:19 -0000 Message-ID: <brookschristopher@example.org> Subject: [Solr Wiki] Update of "DataImportHandler" by NoblePaul X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification. The following page has been changed by NoblePaul: http://wiki.apache.org/solr/DataImportHandler ------------------------------------------------------------------------------ * '''`deltaQuery`''' : Only used in delta-import * '''`parentDeltaQuery`''' : Only used in delta-import * '''`deletedPkQuery`''' : Only used in delta-import - * '''`deltaImportQuery`''' : (Only used in delta-import) . If this is present this will be used during the import 'phase' of delta import instead of `query` . There is a namespace `${dataimporter.delta.<column-name>}` which can be used in this query. + * '''`deltaImportQuery`''' : (Only used in delta-import) . If this is present this will be used during the import 'phase' of delta import instead of `query` . There is a namespace `${dataimporter.delta.<column-name>}` which can be used in this query. <!> ["Solr1.4"] + == Commands == From solr-commits-brookschristopher@example.org Tue Nov 04 08:42:54 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 74246 invoked from network); 4 Nov 2008 08:42:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Nov 2008 08:42:52 -0000 Received: (qmail 40028 invoked by uid 500); 4 Nov 2008 08:42:58 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 39989 invoked by uid 500); 4 Nov 2008 08:42:58 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 39980 invoked by uid 99); 4 Nov 2008 08:42:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 00:42:58 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 08:41:50 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id AC3C91114F for <brookschristopher@example.org>; Tue, 4 Nov 2008 08:42:01 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <brookschristopher@example.org> To: brookschristopher@example.org Date: Tue, 04 Nov 2008 08:42:01 -0000 Message-ID: <brookschristopher@example.org> Subject: [Solr Wiki] Trivial Update of "AnalyzersTokenizersTokenFilters" by KojiSekiguchi X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification. The following page has been changed by KojiSekiguchi: http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters ------------------------------------------------------------------------------ Analyzers are components that pre-process input text at index time and/or at search time. It's important to use the same or similar analyzers that process text in a compatible manner at index and query time. For example, if an indexing analyzer lowercases words, then the query analyzer should do the same to enable finding the indexed words. The Analyzer class is an abstract class, but Lucene comes with a few concrete Analyzers that pre-process their input in different ways. If you need to pre-process input text and queries in a way that is not provided by any of Lucene's built-in Analyzers, you will need to specify a custom Analyzer in the Solr schema. + + == Char Filters == + + <!> ["Solr1.4"] == Tokens and Token Filters == @@ -71, +75 @@ } }}} + === CharFilterFactories === + + <!> ["Solr1.4"] + + ==== solr.MappingCharFilterFactory ==== + + Creates `org.apache.solr.analysis.MappingCharFilter`. === TokenizerFactories === From solr-commits-brookschristopher@example.org Tue Nov 04 16:11:36 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 76913 invoked from network); 4 Nov 2008 16:11:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Nov 2008 16:11:35 -0000 Received: (qmail 15058 invoked by uid 500); 4 Nov 2008 16:11:42 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 15032 invoked by uid 500); 4 Nov 2008 16:11:41 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 15023 invoked by uid 99); 4 Nov 2008 16:11:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 08:11:41 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 16:10:21 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id 97A1B1115B for <brookschristopher@example.org>; Tue, 4 Nov 2008 16:11:03 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <brookschristopher@example.org> To: brookschristopher@example.org Date: Tue, 04 Nov 2008 16:11:03 -0000 Message-ID: <brookschristopher@example.org> Subject: [Solr Wiki] Update of "PublicServers" by MelSykes X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification. The following page has been changed by MelSykes: http://wiki.apache.org/solr/PublicServers ------------------------------------------------------------------------------ * [http://www.onmeda.de/ Onmeda] German health portal uses Solr for content search, including faceted searches. (~750.000 articles, ~9000 searches per day, JBOSS) * [http://www.pagenstecher.de] uses Solr do search usergenerated Content, more than 2 Mio documents and 5000 searches a day, using faceted search. [http://www.pagenstecher.de/search.do Try it out] * [http://www.rallyformusic.com] uses Solr for navigation and full site search. + * [http://www.fitnessfootwear.com/ Fitnessfootwear.com] uses Solr for faceted browsing of [http://www.fitnessfootwear.com/c-212-hunter-wellies.aspx Hunter], [http://www.fitnessfootwear.com/c-2-merrell.aspx Merrell], [http://www.fitnessfootwear.com/c-82-caterpillar.aspx Caterpillar]. * [http://www.machdudas.de] uses Solr for location based classified search using geodata. * [http://uaprom.net] and [http://ruprom.net] Ukrainian and Russian B2B sites. Use Solr for search and faceted browsing of products, companies and buying leads. * [http://www.kerevizli.com] uses Solr to search music albums. From solr-commits-brookschristopher@example.org Tue Nov 04 19:09:03 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 87728 invoked from network); 4 Nov 2008 19:09:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Nov 2008 19:09:03 -0000 Received: (qmail 4678 invoked by uid 500); 4 Nov 2008 19:09:09 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 4652 invoked by uid 500); 4 Nov 2008 19:09:09 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 4643 invoked by uid 99); 4 Nov 2008 19:09:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 11:09:09 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 19:08:00 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 34B272388865; Tue, 4 Nov 2008 11:08:12 -0800 (PST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r711360 - /lucene/solr/trunk/client/java/solrj/src/org/apache/solr/client/solrj/SolrQuery.java Date: Tue, 04 Nov 2008 19:08:12 -0000 To: brookschristopher@example.org From: brookschristopher@example.org X-Mailer: svnmailer-1.0.8 Message-Id: <brookschristopher@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: yonik Date: Tue Nov 4 11:08:11 2008 New Revision: 711360 URL: http://svn.apache.org/viewvc?rev=711360&view=rev Log: SOLR-818: fix null pointer exception in SolrQuery.setFields Modified: lucene/solr/trunk/client/java/solrj/src/org/apache/solr/client/solrj/SolrQuery.java Modified: lucene/solr/trunk/client/java/solrj/src/org/apache/solr/client/solrj/SolrQuery.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/client/java/solrj/src/org/apache/solr/client/solrj/SolrQuery.java?rev=711360&r1=711359&r2=711360&view=diff ============================================================================== --- lucene/solr/trunk/client/java/solrj/src/org/apache/solr/client/solrj/SolrQuery.java (original) +++ lucene/solr/trunk/client/java/solrj/src/org/apache/solr/client/solrj/SolrQuery.java Tue Nov 4 11:08:11 2008 @@ -388,6 +388,7 @@ public SolrQuery setFields(String ... fields) { if( fields == null || fields.length == 0 ) { this.remove( CommonParams.FL ); + return this; } StringBuilder sb = new StringBuilder(); sb.append( fields[0] ); From solr-commits-brookschristopher@example.org Tue Nov 04 20:55:34 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 25144 invoked from network); 4 Nov 2008 20:55:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Nov 2008 20:55:34 -0000 Received: (qmail 34947 invoked by uid 500); 4 Nov 2008 20:55:41 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 34917 invoked by uid 500); 4 Nov 2008 20:55:41 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 34908 invoked by uid 99); 4 Nov 2008 20:55:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 12:55:41 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 20:54:21 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id ABB261115B for <brookschristopher@example.org>; Tue, 4 Nov 2008 20:54:32 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <brookschristopher@example.org> To: brookschristopher@example.org Date: Tue, 04 Nov 2008 20:54:32 -0000 Message-ID: <brookschristopher@example.org> Subject: [Solr Wiki] Update of "PublicServers" by MelSykes X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification. The following page has been changed by MelSykes: http://wiki.apache.org/solr/PublicServers ------------------------------------------------------------------------------ * [http://www.onmeda.de/ Onmeda] German health portal uses Solr for content search, including faceted searches. (~750.000 articles, ~9000 searches per day, JBOSS) * [http://www.pagenstecher.de] uses Solr do search usergenerated Content, more than 2 Mio documents and 5000 searches a day, using faceted search. [http://www.pagenstecher.de/search.do Try it out] * [http://www.rallyformusic.com] uses Solr for navigation and full site search. - * [http://www.fitnessfootwear.com/ Fitnessfootwear.com] uses Solr for faceted browsing of [http://www.fitnessfootwear.com/c-212-hunter-wellies.aspx Hunter], [http://www.fitnessfootwear.com/c-2-merrell.aspx Merrell], [http://www.fitnessfootwear.com/c-82-caterpillar.aspx Caterpillar]. + * [http://www.fitnessfootwear.com/ Fitnessfootwear.com] uses Solr for faceted browsing of [http://www.fitnessfootwear.com/c-212-hunter-wellies.aspx Hunter], [http://www.fitnessfootwear.com/c-2-merrell.aspx Merrell], [http://www.fitnessfootwear.com/c-82-caterpillar.aspx Caterpillar]. Dual CPU/2GB RAM. * [http://www.machdudas.de] uses Solr for location based classified search using geodata. * [http://uaprom.net] and [http://ruprom.net] Ukrainian and Russian B2B sites. Use Solr for search and faceted browsing of products, companies and buying leads. * [http://www.kerevizli.com] uses Solr to search music albums. From solr-commits-brookschristopher@example.org Tue Nov 04 22:20:08 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 79511 invoked from network); 4 Nov 2008 22:20:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Nov 2008 22:20:08 -0000 Received: (qmail 39308 invoked by uid 500); 4 Nov 2008 22:20:15 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 39269 invoked by uid 500); 4 Nov 2008 22:20:14 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 39260 invoked by uid 99); 4 Nov 2008 22:20:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 14:20:14 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 04 Nov 2008 22:19:05 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 8DC4323888A2; Tue, 4 Nov 2008 14:19:47 -0800 (PST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: svn commit: r711450 - in /lucene/solr/trunk: CHANGES.txt src/java/org/apache/solr/analysis/WordDelimiterFilter.java src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java Date: Tue, 04 Nov 2008 22:19:47 -0000 To: brookschristopher@example.org From: brookschristopher@example.org X-Mailer: svnmailer-1.0.8 Message-Id: <brookschristopher@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: yonik Date: Tue Nov 4 14:19:46 2008 New Revision: 711450 URL: http://svn.apache.org/viewvc?rev=711450&view=rev Log: SOLR-751: WordDelimiterFilter didn't adjust the start offset of single tokens Modified: lucene/solr/trunk/CHANGES.txt lucene/solr/trunk/src/java/org/apache/solr/analysis/WordDelimiterFilter.java lucene/solr/trunk/src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java Modified: lucene/solr/trunk/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/solr/trunk/CHANGES.txt?rev=711450&r1=711449&r2=711450&view=diff ============================================================================== --- lucene/solr/trunk/CHANGES.txt (original) +++ lucene/solr/trunk/CHANGES.txt Tue Nov 4 14:19:46 2008 @@ -95,6 +95,10 @@ 4. SOLR-805: DisMax queries are not being cached in QueryResultCache (Todd Feak via koji) + 5. SOLR-751: WordDelimiterFilter didn't adjust the start offset of single + tokens that started with delimiters, leading to incorrect highlighting. + (Stefan Oestreicher via yonik) + Other Changes ---------------------- Modified: lucene/solr/trunk/src/java/org/apache/solr/analysis/WordDelimiterFilter.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/src/java/org/apache/solr/analysis/WordDelimiterFilter.java?rev=711450&r1=711449&r2=711450&view=diff ============================================================================== --- lucene/solr/trunk/src/java/org/apache/solr/analysis/WordDelimiterFilter.java (original) +++ lucene/solr/trunk/src/java/org/apache/solr/analysis/WordDelimiterFilter.java Tue Nov 4 14:19:46 2008 @@ -373,6 +373,7 @@ // just adjust the text w/o changing the rest // of the original token t.setTermBuffer(termBuffer, start, len-start); + t.setStartOffset(t.startOffset() + start); return t; } Modified: lucene/solr/trunk/src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java?rev=711450&r1=711449&r2=711450&view=diff ============================================================================== --- lucene/solr/trunk/src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java (original) +++ lucene/solr/trunk/src/test/org/apache/solr/analysis/TestWordDelimiterFilter.java Tue Nov 4 14:19:46 2008 @@ -183,4 +183,99 @@ assertEquals(6, t.endOffset()); } } + + public void testOffsetChange() throws Exception + { + WordDelimiterFilter wdf = new WordDelimiterFilter( + new TokenStream() { + Token t; + public Token next() { + if (t != null) return null; + t = new Token("übelkeit)", 7, 16); + return t; + } + }, + 1,1,0,0,1,1,0 + ); + + Token t = wdf.next(); + + assertNotNull(t); + assertEquals("übelkeit", t.term()); + assertEquals(7, t.startOffset()); + assertEquals(15, t.endOffset()); + } + + public void testOffsetChange2() throws Exception + { + WordDelimiterFilter wdf = new WordDelimiterFilter( + new TokenStream() { + Token t; + public Token next() { + if (t != null) return null; + t = new Token("(übelkeit", 7, 17); + return t; + } + }, + 1,1,0,0,1,1,0 + ); + + Token t = wdf.next(); + + assertNotNull(t); + assertEquals("übelkeit", t.term()); + assertEquals(8, t.startOffset()); + assertEquals(17, t.endOffset()); + } + + public void testOffsetChange3() throws Exception + { + WordDelimiterFilter wdf = new WordDelimiterFilter( + new TokenStream() { + Token t; + public Token next() { + if (t != null) return null; + t = new Token("(übelkeit", 7, 16); + return t; + } + }, + 1,1,0,0,1,1,0 + ); + + Token t = wdf.next(); + + assertNotNull(t); + assertEquals("übelkeit", t.term()); + assertEquals(8, t.startOffset()); + assertEquals(16, t.endOffset()); + } + + public void testOffsetChange4() throws Exception + { + WordDelimiterFilter wdf = new WordDelimiterFilter( + new TokenStream() { + private Token t; + public Token next() { + if (t != null) return null; + t = new Token("(foo,bar)", 7, 16); + return t; + } + }, + 1,1,0,0,1,1,0 + ); + + Token t = wdf.next(); + + assertNotNull(t); + assertEquals("foo", t.term()); + assertEquals(8, t.startOffset()); + assertEquals(11, t.endOffset()); + + t = wdf.next(); + + assertNotNull(t); + assertEquals("bar", t.term()); + assertEquals(12, t.startOffset()); + assertEquals(15, t.endOffset()); + } } From solr-commits-brookschristopher@example.org Wed Nov 05 13:55:58 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 82559 invoked from network); 5 Nov 2008 13:55:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Nov 2008 13:55:58 -0000 Received: (qmail 87060 invoked by uid 500); 5 Nov 2008 13:56:04 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 87035 invoked by uid 500); 5 Nov 2008 13:56:04 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 87026 invoked by uid 99); 5 Nov 2008 13:56:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Nov 2008 05:56:04 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eos.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Nov 2008 13:54:47 +0000 Received: from eos.apache.org (localhost [83.161.120.155]) by eos.apache.org (Postfix) with ESMTP id C2BD111157 for <brookschristopher@example.org>; Wed, 5 Nov 2008 13:55:28 +0000 (GMT) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: Apache Wiki <brookschristopher@example.org> To: brookschristopher@example.org Date: Wed, 05 Nov 2008 13:55:28 -0000 Message-ID: <brookschristopher@example.org> Subject: [Solr Wiki] Update of "SolrResources" by GrantIngersoll X-Virus-Checked: Checked by ClamAV on apache.org Dear Wiki user, You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification. The following page has been changed by GrantIngersoll: http://wiki.apache.org/solr/SolrResources ------------------------------------------------------------------------------ i. [http://www.ibm.com/developerworks/java/library/j-solr2/ Solr for the enterprise] (2007-06-05) * [http://blog.ryaneby.com/archives/solr-in-libraries/ Solr in Libraries] A round-up of experimental library catalog projects using Solr from Ryan Eby, (2007-04-26) * [http://www.ibm.com/developerworks/library/os-php-apachesolr/index.html Enterprise search with PHP and Apache Solr] at [http://www.ibm.com/developerWorks IBM developerWorks] (2008-01-15) + * [http://www.ibm.com/developerworks/java/library/j-solr-update/?S_TACT=105AGX01&S_CMP=HP What's new with Apache Solr] - Coverage of new features and capabilities in Solr 1.3 by Grant Ingersoll. (2008-11-04) == Books == From solr-commits-brookschristopher@example.org Wed Nov 05 14:33:17 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 875 invoked from network); 5 Nov 2008 14:33:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Nov 2008 14:33:17 -0000 Received: (qmail 29865 invoked by uid 500); 5 Nov 2008 14:33:23 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 29833 invoked by uid 500); 5 Nov 2008 14:33:23 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 29824 invoked by uid 99); 5 Nov 2008 14:33:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Nov 2008 06:33:23 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO eris.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Nov 2008 14:32:14 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 398FB238889D; Wed, 5 Nov 2008 06:32:56 -0800 (PST) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r711574 - /lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java Date: Wed, 05 Nov 2008 14:32:56 -0000 To: brookschristopher@example.org From: brookschristopher@example.org X-Mailer: svnmailer-1.0.8 Message-Id: <brookschristopher@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: yonik Date: Wed Nov 5 06:32:55 2008 New Revision: 711574 URL: http://svn.apache.org/viewvc?rev=711574&view=rev Log: remove the now redundant setResponseParser Modified: lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java Modified: lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java URL: http://svn.apache.org/viewvc/lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java?rev=711574&r1=711573&r2=711574&view=diff ============================================================================== --- lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java (original) +++ lucene/solr/trunk/src/java/org/apache/solr/handler/component/SearchHandler.java Wed Nov 5 06:32:55 2008 @@ -379,7 +379,7 @@ // String url = "http://" + shard + "/select"; String url = "http://" + shard; - params.remove(CommonParams.WT); // use default (or should we explicitly set it?) + params.remove(CommonParams.WT); // use default (currently javabin) params.remove(CommonParams.VERSION); SolrServer server = new CommonsHttpSolrServer(url, client); @@ -387,7 +387,9 @@ // use generic request to avoid extra processing of queries QueryRequest req = new QueryRequest(params); req.setMethod(SolrRequest.METHOD.POST); - req.setResponseParser(new BinaryResponseParser()); // this sets the wt param + + // no need to set the response parser as binary is the default + // req.setResponseParser(new BinaryResponseParser()); // srsp.rsp = server.request(req); // srsp.rsp = server.query(sreq.params); From solr-commits-brookschristopher@example.org Wed Nov 05 17:16:05 2008 Return-Path: <solr-commits-brookschristopher@example.org> Delivered-To: brookschristopher@example.org Received: (qmail 81931 invoked from network); 5 Nov 2008 17:16:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Nov 2008 17:16:04 -0000 Received: (qmail 30913 invoked by uid 500); 5 Nov 2008 17:16:11 -0000 Delivered-To: brookschristopher@example.org Received: (qmail 30889 invoked by uid 500); 5 Nov 2008 17:16:11 -0000 Mailing-List: contact brookschristopher@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:brookschristopher@example.org> List-Unsubscribe: <mailto:brookschristopher@example.org> List-Post: <mailto:brookschristopher@example.org> List-Id: <solr-commits.lucene.apache.org> Reply-To: brookschristopher@example.org Delivered-To: mailing list brookschristopher@example.org Received: (qmail 30875 invoked by uid 99); 5 Nov 2008 17:16:11 -0000 Received:
{'source': 'Apache Software Foundation Public Mail Archives'}