From nutch-devjohn93@example.net Thu Jun 01 10:02:44 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 39949 invoked from network); 1 Jun 2006 10:02:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2006 10:02:44 -0000 Received: (qmail 4250 invoked by uid 500); 1 Jun 2006 10:02:42 -0000 Delivered-To: john93@example.net Received: (qmail 4044 invoked by uid 500); 1 Jun 2006 10:02:41 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 4028 invoked by uid 99); 1 Jun 2006 10:02:41 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 03:02:41 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO viefep12-int.chello.at) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 03:02:39 -0700 Received: from [83.161.120.155] (really [83.161.120.155]) by viefep12-int.chello.at (InterMail vM.83.161.120.155 201-2131-118-104-20050224) with ESMTP id <20060601100212.PFV11056.viefep12-int.chello.at@[83.161.120.155]> for <john93@example.net>; Thu, 1 Jun 2006 12:02:12 +0200 Message-ID: <john93@example.net> Date: Thu, 01 Jun 2006 12:02:20 +0200 From: YourSoft <john93@example.net> User-Agent: Thunderbird 83.161.120.155 (Windows/20060308) MIME-Version: 1.0 To: john93@example.net Subject: Re: refetching interval References: <john93@example.net.looksmart_sf.ad.looksmart.com> In-Reply-To: <john93@example.net.looksmart_sf.ad.looksmart.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Antivirus: avast! (VPS 0622-2, 2006.05.31), Outbound message X-Antivirus-Status: Clean X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N It is an interesting 'webgraph project': http://webgraph.dsi.unimi.it/ I don't know it is usable or not for nutch link database for link database. From nutch-devjohn93@example.net Thu Jun 01 10:02:57 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 40181 invoked from network); 1 Jun 2006 10:02:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2006 10:02:57 -0000 Received: (qmail 4735 invoked by uid 500); 1 Jun 2006 10:02:55 -0000 Delivered-To: john93@example.net Received: (qmail 4714 invoked by uid 500); 1 Jun 2006 10:02:54 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 4693 invoked by uid 99); 1 Jun 2006 10:02:54 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 03:02:54 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: neutral (asf.osuosl.org: local policy) Received: from [83.161.120.155] (HELO viefep13-int.chello.at) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 03:02:53 -0700 Received: from [83.161.120.155] (really [83.161.120.155]) by viefep13-int.chello.at (InterMail vM.83.161.120.155 201-2131-118-104-20050224) with ESMTP id <20060601100228.ODIS5789.viefep13-int.chello.at@[83.161.120.155]> for <john93@example.net>; Thu, 1 Jun 2006 12:02:28 +0200 Message-ID: <john93@example.net> Date: Thu, 01 Jun 2006 12:02:37 +0200 From: YourSoft <john93@example.net> User-Agent: Thunderbird 83.161.120.155 (Windows/20060308) MIME-Version: 1.0 To: john93@example.net Subject: webgraph References: <john93@example.net.looksmart_sf.ad.looksmart.com> In-Reply-To: <john93@example.net.looksmart_sf.ad.looksmart.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Antivirus: avast! (VPS 0622-2, 2006.05.31), Outbound message X-Antivirus-Status: Clean X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N It is an interesting 'webgraph project': http://webgraph.dsi.unimi.it/ I don't know it is usable or not for nutch link database for link database. From nutch-devjohn93@example.net Thu Jun 01 17:25:26 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 23235 invoked from network); 1 Jun 2006 17:25:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2006 17:25:26 -0000 Received: (qmail 50370 invoked by uid 500); 1 Jun 2006 17:25:24 -0000 Delivered-To: john93@example.net Received: (qmail 50322 invoked by uid 500); 1 Jun 2006 17:25:23 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 50249 invoked by uid 500); 1 Jun 2006 17:25:23 -0000 Delivered-To: john93@example.net Received: (qmail 50245 invoked by uid 99); 1 Jun 2006 17:25:23 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 10:25:23 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 10:25:22 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id DFC0F7142CC for <john93@example.net>; Thu, 1 Jun 2006 17:24:29 +0000 (GMT) Message-ID: <15946767.1149182669898.JavaMail.jira@brutus> Date: Thu, 1 Jun 2006 17:24:29 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Created: (NUTCH-293) support for Crawl-delay in Robots.txt MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N support for Crawl-delay in Robots.txt ------------------------------------- Key: NUTCH-293 URL: http://issues.apache.org/jira/browse/NUTCH-293 Project: Nutch Type: Improvement Components: fetcher Versions: 0.8-dev Reporter: Stefan Groschupf Priority: Critical Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard. See: http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html Webmasters start blocking nutch since we do not support it. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Thu Jun 01 17:27:25 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 23409 invoked from network); 1 Jun 2006 17:27:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2006 17:27:25 -0000 Received: (qmail 52032 invoked by uid 500); 1 Jun 2006 17:27:24 -0000 Delivered-To: john93@example.net Received: (qmail 51628 invoked by uid 500); 1 Jun 2006 17:27:23 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 51617 invoked by uid 500); 1 Jun 2006 17:27:23 -0000 Delivered-To: john93@example.net Received: (qmail 51614 invoked by uid 99); 1 Jun 2006 17:27:23 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 10:27:23 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 10:27:23 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 2C3647142A2 for <john93@example.net>; Thu, 1 Jun 2006 17:26:30 +0000 (GMT) Message-ID: <20882736.1149182790163.JavaMail.jira@brutus> Date: Thu, 1 Jun 2006 17:26:30 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Updated: (NUTCH-293) support for Crawl-delay in Robots.txt In-Reply-To: <15946767.1149182669898.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-293?page=all ] Stefan Groschupf updated NUTCH-293: ----------------------------------- Attachment: crawlDelayv1.patch A frist darft of a crawl delay support for nutch. The problem I see is that in case ip based delay is configured it can happen that we use the crawl delay of one host for a other host running on the same ip. Feedback is welcome. > support for Crawl-delay in Robots.txt > ------------------------------------- > > Key: NUTCH-293 > URL: http://issues.apache.org/jira/browse/NUTCH-293 > Project: Nutch > Type: Improvement > Components: fetcher > Versions: 0.8-dev > Reporter: Stefan Groschupf > Priority: Critical > Attachments: crawlDelayv1.patch > > Nutch need support for Crawl-delay defined in robots.txt, it is not a standard but a de-facto standard. > See: > http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html > Webmasters start blocking nutch since we do not support it. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Thu Jun 01 18:42:27 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 51687 invoked from network); 1 Jun 2006 18:42:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2006 18:42:26 -0000 Received: (qmail 95600 invoked by uid 500); 1 Jun 2006 18:42:25 -0000 Delivered-To: john93@example.net Received: (qmail 95402 invoked by uid 500); 1 Jun 2006 18:42:24 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 95391 invoked by uid 500); 1 Jun 2006 18:42:24 -0000 Delivered-To: john93@example.net Received: (qmail 95388 invoked by uid 99); 1 Jun 2006 18:42:24 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 11:42:24 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 11:42:23 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 46BC0714022 for <john93@example.net>; Thu, 1 Jun 2006 18:41:30 +0000 (GMT) Message-ID: <30860673.1149187290271.JavaMail.jira@brutus> Date: Thu, 1 Jun 2006 18:41:30 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-289) CrawlDatum should store IP address In-Reply-To: <824887.1148675909801.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-289?page=comments#action_12414273 ] Stefan Groschupf commented on NUTCH-289: ---------------------------------------- Andrzej, I'm afraid I was not able to clearly communicate my ideas and we may be misunderstand each other. Resolve the ip in Parseoutputformat would be only necessary for the new links discovered in the content. Since by default we parse during fetching we would have the chance to use the jvm dns cache, since I guess many new urls point to the same host where we fetched a particular page from. Means if we do not parse separately we would have the best jvm cache usage. We do not lookup IPs of urls we fetch at this time, since these urls already have a ip that was resoved when these urls was first time discovered in a parse process. The only problem we need to handle is what happens in case a ip of a host change. We can simple lookup the ip of a url that throws a protocol error and compare cached and lookup ip. An alternative aproche would be to lookup ip's during crawldb update just for the new urls. Sorry I hope that describe my ideas more clearly. My personal point of view is to store the ip into the crawldatum not into the meta data. > CrawlDatum should store IP address > ---------------------------------- > > Key: NUTCH-289 > URL: http://issues.apache.org/jira/browse/NUTCH-289 > Project: Nutch > Type: Bug > Components: fetcher > Versions: 0.8-dev > Reporter: Doug Cutting > > If the CrawlDatum stored the IP address of the host of it's URL, then one could: > - partition fetch lists on the basis of IP address, for better politeness; > - truncate pages to fetch per IP address, rather than just hostname. This would be a good way to limit the impact of domain spammers. > The IP addresses could be resolved when a CrawlDatum is first created for a new outlink, or perhaps during CrawlDB update. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Thu Jun 01 20:01:36 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 98557 invoked from network); 1 Jun 2006 20:01:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2006 20:01:36 -0000 Received: (qmail 22525 invoked by uid 500); 1 Jun 2006 20:01:33 -0000 Delivered-To: john93@example.net Received: (qmail 22502 invoked by uid 500); 1 Jun 2006 20:01:33 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 22491 invoked by uid 99); 1 Jun 2006 20:01:33 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 13:01:33 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of john93@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail1.basistech.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 13:01:32 -0700 Received: from mail3.basistech.net ([83.161.120.155]) by mail1.basistech.net with Microsoft SMTPSVC+1-308-737-0375); Thu, 1 Jun 2006 16:01:12 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Subject: how to turn on logging, excersize analyzer, tips on debugging plugins? Date: Thu, 1 Jun 2006 16:01:09 -0400 Message-ID: <john93@example.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: how to turn on logging, excersize analyzer, tips on debugging plugins? Thread-Index: AcaFtiAZlwTiuT4ATLCmIH5Vhjn5iQ== From: "Teruhiko Kurosaka" <john93@example.net> To: <john93@example.net>, =?US-ASCII?Q?Jerome_Charron?= <john93@example.net> X-OriginalArrivalTime: 01 Jun 2006 20:01:12.0823 (UTC) FILETIME=[22098870:01C685B6] X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Nutch develpers, I'm writing my a language analyzer and have three questions. Any pointer will be appreciated. 1. How do I turn on the logging facility? 2. Is there an easy way to run just an analyzer plugin, rather than running "nutch crawl"? 3. How do I run debugger (eclipse, in may case) over the plugin code that would be loaded later? I.e. where do I set the break point before the plugin gets loaded? Thank you in advance. -kuro From nutch-devjohn93@example.net Thu Jun 01 21:55:05 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 39445 invoked from network); 1 Jun 2006 21:55:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2006 21:55:05 -0000 Received: (qmail 82518 invoked by uid 500); 1 Jun 2006 21:55:04 -0000 Delivered-To: john93@example.net Received: (qmail 82423 invoked by uid 500); 1 Jun 2006 21:55:03 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 82410 invoked by uid 99); 1 Jun 2006 21:55:03 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 14:55:03 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of john93@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail1.basistech.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 14:55:02 -0700 Received: from mail3.basistech.net ([83.161.120.155]) by mail1.basistech.net with Microsoft SMTPSVC+1-308-737-0375); Thu, 1 Jun 2006 17:54:43 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Subject: i18n in nutch home page is misnomor Date: Thu, 1 Jun 2006 17:54:38 -0400 Message-ID: <john93@example.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: i18n in nutch home page is misnomor Thread-Index: AcaFxfppPCM4tMPMRBqQDmWt7do9qw== From: "Teruhiko Kurosaka" <john93@example.net> To: <john93@example.net> X-OriginalArrivalTime: 01 Jun 2006 21:54:43.0216 (UTC) FILETIME=[FD590900:01C685C5] X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Dear Webmaster of http://lucene.apache.org/nutch/ In the menu bar, under the Documentation heading there is an item called "i18n". The web page linked from "i18n" talks about how to translate=20 (localize) the search GUI. This is not i18n=20 (internationalization) which should mean designing and implementing a program so that works with different character encodings of the world, and it can be localizaed. The localization tasks should not be confused with internationalization tasks. I suggest "i18n" be renamed to "l10n", short for localization. -kuro From nutch-devjohn93@example.net Fri Jun 02 06:57:27 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 95786 invoked from network); 2 Jun 2006 06:57:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 06:57:27 -0000 Received: (qmail 42568 invoked by uid 500); 2 Jun 2006 06:57:25 -0000 Delivered-To: john93@example.net Received: (qmail 42544 invoked by uid 500); 2 Jun 2006 06:57:24 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 42531 invoked by uid 500); 2 Jun 2006 06:57:24 -0000 Delivered-To: john93@example.net Received: (qmail 42528 invoked by uid 99); 2 Jun 2006 06:57:24 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 23:57:24 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jun 2006 23:57:23 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 0AF8A7141E4 for <john93@example.net>; Fri, 2 Jun 2006 06:56:30 +0000 (GMT) Message-ID: <24179038.1149231390014.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 06:56:30 +0000 (GMT+00:00) From: "Stefan Neufeind (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Created: (NUTCH-294) Topic-maps of related searchwords MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Topic-maps of related searchwords --------------------------------- Key: NUTCH-294 URL: http://issues.apache.org/jira/browse/NUTCH-294 Project: Nutch Type: New Feature Components: searcher Reporter: Stefan Neufeind Would it be possible to offer a user "topic-maps"? It's when you search for something and get topic-related words that might also be of interest for you. I wonder if that's somehow possible with the ngram-index for "did you mean" (see separate feature-enhancement-bug for this), but we'd need to have a relation between words (in what context do they occur). For the webfrontend usually trees are used - which for some users offer quite impressive eye-candy :-) E.g. see this advertisement by Novell where I've just seen a similar "topic-map" as well: http://www.novell.com/de-de/company/advertising/defineyouropen.html -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:09:34 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 66339 invoked from network); 2 Jun 2006 15:09:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:09:34 -0000 Received: (qmail 47372 invoked by uid 500); 2 Jun 2006 15:09:29 -0000 Delivered-To: john93@example.net Received: (qmail 47257 invoked by uid 500); 2 Jun 2006 15:09:28 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 47206 invoked by uid 500); 2 Jun 2006 15:09:28 -0000 Delivered-To: john93@example.net Received: (qmail 47176 invoked by uid 99); 2 Jun 2006 15:09:28 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:09:28 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:09:27 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 558EA7141E4 for <john93@example.net>; Fri, 2 Jun 2006 15:08:30 +0000 (GMT) Message-ID: <26934555.1149260910346.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:08:30 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-282) Showing too few results on a page (Paging not correct) In-Reply-To: <32303292.1148433329767.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-282?page=comments#action_12414435 ] Stefan Groschupf commented on NUTCH-282: ---------------------------------------- Is that related to host grouping we discussed? Can we in this case close this bug? > Showing too few results on a page (Paging not correct) > ------------------------------------------------------ > > Key: NUTCH-282 > URL: http://issues.apache.org/jira/browse/NUTCH-282 > Project: Nutch > Type: Bug > Components: web gui > Versions: 0.8-dev > Reporter: Stefan Neufeind > > I did a search and got back the value "itemsPerPage" from opensearch. But the output shows "results 1-8" and I have a total of 46 searchresults. > Same happens for the webinterface. > Why aren't "enough" results fetched? > The problem might be somewhere in the area of where Nutch should only display a certaian number of websites per site. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:14:30 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 68316 invoked from network); 2 Jun 2006 15:14:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:14:30 -0000 Received: (qmail 60127 invoked by uid 500); 2 Jun 2006 15:14:28 -0000 Delivered-To: john93@example.net Received: (qmail 60038 invoked by uid 500); 2 Jun 2006 15:14:27 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 59987 invoked by uid 500); 2 Jun 2006 15:14:27 -0000 Delivered-To: john93@example.net Received: (qmail 59924 invoked by uid 99); 2 Jun 2006 15:14:27 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:14:27 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:14:25 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 8E4497141E9 for <john93@example.net>; Fri, 2 Jun 2006 15:13:31 +0000 (GMT) Message-ID: <33235176.1149261211579.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:13:31 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-286) Handling common error-pages as 404 In-Reply-To: <33244300.1148514090709.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-286?page=comments#action_12414439 ] Stefan Groschupf commented on NUTCH-286: ---------------------------------------- This is difficult to realize since the http error code is readed from response in the fetcher and setted into the protocol status , content analysis can only done during parsing. Also normally such pages do not get a high OPIC score and should be not in the top search results. However this is a wrong configured http server response, so you may should open a bug in the typo3 issue tracking. Should we close this issue? > Handling common error-pages as 404 > ---------------------------------- > > Key: NUTCH-286 > URL: http://issues.apache.org/jira/browse/NUTCH-286 > Project: Nutch > Type: Improvement > Reporter: Stefan Neufeind > > Idea: Some pages from some software-packages/scripts report an "http 200 ok" even though a specific page could not be found. Example I just found is: > http://www.deteimmobilien.de/unternehmen/nbjmup;Uipnbt/IfsctuAefufjnnpcjmjfo/ef > That's a typo3-page explaining in it's standard-layout and wording: "The requested page did not exist or was inaccessible." > So I had the idea if somebody might create a plugin that could find commonly used formulations for "page does not exist" etc. and turn the page into a 404 before feeding them into the nutch-index - although the server responded with status 200 ok. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:21:27 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 73079 invoked from network); 2 Jun 2006 15:21:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:21:27 -0000 Received: (qmail 79637 invoked by uid 500); 2 Jun 2006 15:21:25 -0000 Delivered-To: john93@example.net Received: (qmail 79617 invoked by uid 500); 2 Jun 2006 15:21:25 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 79606 invoked by uid 500); 2 Jun 2006 15:21:25 -0000 Delivered-To: john93@example.net Received: (qmail 79603 invoked by uid 99); 2 Jun 2006 15:21:25 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:21:25 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:21:24 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 3712C7141E4 for <john93@example.net>; Fri, 2 Jun 2006 15:20:30 +0000 (GMT) Message-ID: <25811127.1149261630189.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:20:30 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-288) hitsPerSite-functionality "flawed": problems writing a page-navigation In-Reply-To: <24119929.1148570791759.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-288?page=comments#action_12414441 ] Stefan Groschupf commented on NUTCH-288: ---------------------------------------- HI Stefan, >Also it does go back page by page until you get to the last result-page Isn't it possible to caculate the latest page instead of using a while loop to find the latest page? > hitsPerSite-functionality "flawed": problems writing a page-navigation > ---------------------------------------------------------------------- > > Key: NUTCH-288 > URL: http://issues.apache.org/jira/browse/NUTCH-288 > Project: Nutch > Type: Bug > Components: web gui > Versions: 0.8-dev > Reporter: Stefan Neufeind > Attachments: NUTCH-288-OpenSearch-fix.patch > > The deduplication-functionality on a per-site-basis (hitsPerSite = 3) leads to problems when trying to offer a page-navigation (e.g. allow the user to jump to page 10). This is because dedup is done after fetching. > RSS shows a maximum number of 7763 documents (that is without dedup!), I set it to display 10 items per page. My "naive" approach was to estimate I have 7763/10 = 777 pages. But already when moving to page 3 I got no more searchresults (I guess because of dedup). And when moving to page 10 I got an exception (see below). > 2006-05-25 16:24:43 StandardWrapperValve[OpenSearch]: Servlet.service() for servlet OpenSearch threw exception > java.lang.NegativeArraySizeException > at org.apache.nutch.searcher.Hits.getHits(Hits.java:65) > at org.apache.nutch.searcher.OpenSearchServlet.doGet(OpenSearchServlet.java:149) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) > at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) > at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) > at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) > at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) > at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) > at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) > at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) > at java.lang.Thread.run(Thread.java:595) > Only workaround I see for the moment: Fetching RSS without duplication, dedup myself and cache the RSS-result to improve performance. But a cleaner solution would imho be nice. Is there a performant way of doing deduplication and knowing for sure how many documents are available to view? For sure this would mean to dedup all search-results first ... -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:32:36 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 80231 invoked from network); 2 Jun 2006 15:32:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:32:34 -0000 Received: (qmail 7525 invoked by uid 500); 2 Jun 2006 15:32:29 -0000 Delivered-To: john93@example.net Received: (qmail 7442 invoked by uid 500); 2 Jun 2006 15:32:28 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 7389 invoked by uid 500); 2 Jun 2006 15:32:28 -0000 Delivered-To: john93@example.net Received: (qmail 7377 invoked by uid 99); 2 Jun 2006 15:32:28 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:32:28 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:32:25 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 2A2A47141E6 for <john93@example.net>; Fri, 2 Jun 2006 15:31:31 +0000 (GMT) Message-ID: <13404119.1149262291169.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:31:31 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-292) OpenSearchServlet: OutOfMemoryError: Java heap space In-Reply-To: <8521083.1148837909794.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-292?page=comments#action_12414443 ] Stefan Groschupf commented on NUTCH-292: ---------------------------------------- +1, Can someone create a clean patch file? > OpenSearchServlet: OutOfMemoryError: Java heap space > ---------------------------------------------------- > > Key: NUTCH-292 > URL: http://issues.apache.org/jira/browse/NUTCH-292 > Project: Nutch > Type: Bug > Components: web gui > Versions: 0.8-dev > Reporter: Stefan Neufeind > Priority: Critical > Attachments: summarizer.diff > > java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space > org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:203) > org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:329) > org.apache.nutch.searcher.OpenSearchServlet.doGet(OpenSearchServlet.java:155) > javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > The URL I use is: > [...]something[...]/opensearch?query=mysearch&start=0&hitsPerSite=3&hitsPerPage=20&sort=url > It seems to be a problem specific to the date I'm working with. Moving the start from 0 to 10 or changing the query works fine. > Or maybe it doesn't have to do with sorting but it's just that I hit one "bad search-result" that has a broken summary? > !! The problem is repeatable. So if anybody has an idea where to search / what to fix, I can easily try that out !! -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:40:27 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 85239 invoked from network); 2 Jun 2006 15:40:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:40:27 -0000 Received: (qmail 33455 invoked by uid 500); 2 Jun 2006 15:40:26 -0000 Delivered-To: john93@example.net Received: (qmail 33333 invoked by uid 500); 2 Jun 2006 15:40:25 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 33319 invoked by uid 500); 2 Jun 2006 15:40:25 -0000 Delivered-To: john93@example.net Received: (qmail 33315 invoked by uid 99); 2 Jun 2006 15:40:25 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:40:25 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:40:24 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 29929714022 for <john93@example.net>; Fri, 2 Jun 2006 15:39:30 +0000 (GMT) Message-ID: <24742879.1149262770151.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:39:30 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-291) OpenSearchServlet should return "date" as well as "lastModified" In-Reply-To: <18857758.1148834729796.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-291?page=comments#action_12414445 ] Stefan Groschupf commented on NUTCH-291: ---------------------------------------- lastModified will be only indexed if you switch on the index-more plugin. If you think you should change the way lastmodified and date is stored in the index, please submit a patch for MoreIndexingFilter. > OpenSearchServlet should return "date" as well as "lastModified" > ---------------------------------------------------------------- > > Key: NUTCH-291 > URL: http://issues.apache.org/jira/browse/NUTCH-291 > Project: Nutch > Type: Improvement > Components: web gui > Versions: 0.8-dev > Reporter: Stefan Neufeind > Attachments: NUTCH-291-unfinished.patch > > Currently lastModified is provided by OpenSearchServlet - but only in case the date lastModified-date is known. > Since you can sort by "date" (which is lastModified or if not present the fetchdate), it might be useful if OpenSearchServlet could provide "date" as well. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:46:46 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 87958 invoked from network); 2 Jun 2006 15:46:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:46:46 -0000 Received: (qmail 48704 invoked by uid 500); 2 Jun 2006 15:46:31 -0000 Delivered-To: john93@example.net Received: (qmail 48650 invoked by uid 500); 2 Jun 2006 15:46:31 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 48617 invoked by uid 500); 2 Jun 2006 15:46:31 -0000 Delivered-To: john93@example.net Received: (qmail 48584 invoked by uid 99); 2 Jun 2006 15:46:30 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:46:30 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:46:29 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id ED5BC7141E9 for <john93@example.net>; Fri, 2 Jun 2006 15:45:31 +0000 (GMT) Message-ID: <19820885.1149263131969.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:45:31 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-290) parse-pdf: Garbage indexed when text-extraction not allowed In-Reply-To: <2844741.1148823335433.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-290?page=comments#action_12414448 ] Stefan Groschupf commented on NUTCH-290: ---------------------------------------- If a parser throws an exeption: Fetcher, 261: try { parse = this.parseUtil.parse(content); parseStatus = parse.getData().getStatus(); } catch (Exception e) { parseStatus = new ParseStatus(e); } if (!parseStatus.isSuccess()) { LOG.warning("Error parsing: " + key + ": " + parseStatus); parse = parseStatus.getEmptyParse(getConf()); } than we use the empty parse object: and a empthy parse contans just no text, see getText private static class EmptyParseImpl implements Parse { private ParseData data = null; public EmptyParseImpl(ParseStatus status, Configuration conf) { data = new ParseData(status, "", new Outlink[0], new Metadata(), new Metadata()); data.setConf(conf); } public ParseData getData() { return data; } public String getText() { return ""; } } So the Problem should be somewhere else. > parse-pdf: Garbage indexed when text-extraction not allowed > ----------------------------------------------------------- > > Key: NUTCH-290 > URL: http://issues.apache.org/jira/browse/NUTCH-290 > Project: Nutch > Type: Bug > Components: indexer > Versions: 0.8-dev > Reporter: Stefan Neufeind > Attachments: NUTCH-290-canExtractContent.patch > > It seems that garbage (or undecoded text?) is indexed when text-extraction for a PDF is not allowed. > Example-PDF: > http://www.task-switch.nl/Dutch/articles/Management_en_Architectuur_v3.pdf -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:52:27 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 89333 invoked from network); 2 Jun 2006 15:52:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:52:26 -0000 Received: (qmail 63633 invoked by uid 500); 2 Jun 2006 15:52:25 -0000 Delivered-To: john93@example.net Received: (qmail 63607 invoked by uid 500); 2 Jun 2006 15:52:25 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 63595 invoked by uid 500); 2 Jun 2006 15:52:25 -0000 Delivered-To: john93@example.net Received: (qmail 63591 invoked by uid 99); 2 Jun 2006 15:52:24 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:52:24 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:52:24 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 29988714022 for <john93@example.net>; Fri, 2 Jun 2006 15:51:30 +0000 (GMT) Message-ID: <29883789.1149263490140.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:51:30 +0000 (GMT+00:00) From: "Stefan Neufeind (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Updated: (NUTCH-292) OpenSearchServlet: OutOfMemoryError: Java heap space In-Reply-To: <8521083.1148837909794.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-292?page=all ] Stefan Neufeind updated NUTCH-292: ---------------------------------- Attachment: NUTCH-292-summarizer08.diff As per demand, here is the patch. Please note that it has not throughly been testeed by myself. But the patch looks fine and makes sense :-) Oh, and it compiles clean ... > OpenSearchServlet: OutOfMemoryError: Java heap space > ---------------------------------------------------- > > Key: NUTCH-292 > URL: http://issues.apache.org/jira/browse/NUTCH-292 > Project: Nutch > Type: Bug > Components: web gui > Versions: 0.8-dev > Reporter: Stefan Neufeind > Priority: Critical > Attachments: NUTCH-292-summarizer08.diff, summarizer.diff > > java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space > org.apache.nutch.searcher.FetchedSegments.getSummary(FetchedSegments.java:203) > org.apache.nutch.searcher.NutchBean.getSummary(NutchBean.java:329) > org.apache.nutch.searcher.OpenSearchServlet.doGet(OpenSearchServlet.java:155) > javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > The URL I use is: > [...]something[...]/opensearch?query=mysearch&start=0&hitsPerSite=3&hitsPerPage=20&sort=url > It seems to be a problem specific to the date I'm working with. Moving the start from 0 to 10 or changing the query works fine. > Or maybe it doesn't have to do with sorting but it's just that I hit one "bad search-result" that has a broken summary? > !! The problem is repeatable. So if anybody has an idea where to search / what to fix, I can easily try that out !! -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:56:35 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 90368 invoked from network); 2 Jun 2006 15:56:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:56:35 -0000 Received: (qmail 75628 invoked by uid 500); 2 Jun 2006 15:56:35 -0000 Delivered-To: john93@example.net Received: (qmail 75265 invoked by uid 500); 2 Jun 2006 15:56:33 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 75252 invoked by uid 500); 2 Jun 2006 15:56:33 -0000 Delivered-To: john93@example.net Received: (qmail 75246 invoked by uid 99); 2 Jun 2006 15:56:33 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:56:33 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:56:24 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 2A8E1714022 for <john93@example.net>; Fri, 2 Jun 2006 15:55:30 +0000 (GMT) Message-ID: <10137132.1149263730150.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:55:30 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Closed: (NUTCH-287) Exception when searching with sort In-Reply-To: <20887317.1148559149767.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-287?page=all ] Stefan Groschupf closed NUTCH-287: ---------------------------------- Resolution: Won't Fix http://www.mail-archive.com/nutch-user%40lucene.apache.org/msg04696.html > Exception when searching with sort > ---------------------------------- > > Key: NUTCH-287 > URL: http://issues.apache.org/jira/browse/NUTCH-287 > Project: Nutch > Type: Bug > Components: searcher > Versions: 0.8-dev > Reporter: Stefan Neufeind > Priority: Critical > > Running a search with &sort=url works. > But when using &sort=title I get the following exception. > 2006-05-25 14:04:25 StandardWrapperValve[jsp]: Servlet.service() for servlet jsp threw exception > java.lang.RuntimeException: Unknown sort value type! > at org.apache.nutch.searcher.IndexSearcher.translateHits(IndexSearcher.java:157) > at org.apache.nutch.searcher.IndexSearcher.search(IndexSearcher.java:95) > at org.apache.nutch.searcher.NutchBean.search(NutchBean.java:239) > at org.apache.jsp.search_jsp._jspService(search_jsp.java:257) > at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:94) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:324) > at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292) > at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) > at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) > at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) > at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929) > at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) > at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) > at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) > at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) > at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) > at java.lang.Thread.run(Thread.java:595) > What is in those lines is: > WritableComparable sortValue; // convert value to writable > if (sortField == null) { > sortValue = new FloatWritable(scoreDocs[i].score); > } else { > Object raw = ((FieldDoc)scoreDocs[i]).fields[0]; > if (raw instanceof Integer) { > sortValue = new IntWritable(((Integer)raw).intValue()); > } else if (raw instanceof Float) { > sortValue = new FloatWritable(((Float)raw).floatValue()); > } else if (raw instanceof String) { > sortValue = new UTF8((String)raw); > } else { > throw new RuntimeException("Unknown sort value type!"); > } > } > So I thought that maybe raw is an instance of something "strange" and tried raw.getClass().getName() or also raw.toString() to track the cause down - but that always resulted in a NullPointerException. So it seems I'm having raw being null for some strange reason. > When I try with "title2" (or something none-existing) I get a different error that title2 is unknown / not indexed. So I suspect that title should be fine here ... > If there is any information I can help out with, let me know. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 15:58:27 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 91609 invoked from network); 2 Jun 2006 15:58:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 15:58:27 -0000 Received: (qmail 77909 invoked by uid 500); 2 Jun 2006 15:58:27 -0000 Delivered-To: john93@example.net Received: (qmail 77490 invoked by uid 500); 2 Jun 2006 15:58:25 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 77435 invoked by uid 500); 2 Jun 2006 15:58:25 -0000 Delivered-To: john93@example.net Received: (qmail 77421 invoked by uid 99); 2 Jun 2006 15:58:25 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:58:25 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 08:58:24 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 223EF714022 for <john93@example.net>; Fri, 2 Jun 2006 15:57:30 +0000 (GMT) Message-ID: <33382020.1149263850137.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:57:30 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Closed: (NUTCH-284) NullPointerException during index In-Reply-To: <16209800.1148499089765.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-284?page=all ] Stefan Groschupf closed NUTCH-284: ---------------------------------- Resolution: Won't Fix >Yes, I was missing index-basic. > NullPointerException during index > --------------------------------- > > Key: NUTCH-284 > URL: http://issues.apache.org/jira/browse/NUTCH-284 > Project: Nutch > Type: Bug > Components: indexer > Versions: 0.8-dev > Reporter: Stefan Neufeind > > For quite a few this "reduce > sort" has been going on. Then it fails. What could be wrong with this? > 060524 212613 reduce > sort > 060524 212614 reduce > sort > 060524 212615 reduce > sort > 060524 212615 found resource common-terms.utf8 at file:/home/mm/nutch-nightly-prod/conf/common-terms.utf8 > 060524 212615 found resource common-terms.utf8 at file:/home/mm/nutch-nightly-prod/conf/common-terms.utf8 > 060524 212619 Optimizing index. > 060524 212619 job_jlbhhm > java.lang.NullPointerException > at org.apache.nutch.indexer.Indexer$OutputFormat$1.write(Indexer.java:111) > at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:269) > at org.apache.nutch.indexer.Indexer.reduce(Indexer.java:253) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:282) > at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:114) > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:341) > at org.apache.nutch.indexer.Indexer.index(Indexer.java:287) > at org.apache.nutch.indexer.Indexer.main(Indexer.java:304) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 16:00:35 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 92552 invoked from network); 2 Jun 2006 16:00:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 16:00:34 -0000 Received: (qmail 82528 invoked by uid 500); 2 Jun 2006 16:00:27 -0000 Delivered-To: john93@example.net Received: (qmail 82367 invoked by uid 500); 2 Jun 2006 16:00:27 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 82342 invoked by uid 500); 2 Jun 2006 16:00:27 -0000 Delivered-To: john93@example.net Received: (qmail 82328 invoked by uid 99); 2 Jun 2006 16:00:26 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 09:00:26 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 09:00:26 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 8B9DE7141E9 for <john93@example.net>; Fri, 2 Jun 2006 15:59:31 +0000 (GMT) Message-ID: <13483152.1149263971568.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:59:31 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-284) NullPointerException during index In-Reply-To: <16209800.1148499089765.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-284?page=comments#action_12414453 ] Stefan Groschupf commented on NUTCH-284: ---------------------------------------- Please try discuss such things first in the user mailing list than open a issue. Maintaining the issue tracking is very time consuming. But if there is a bug please continue open bug reports. :) Thanks. > NullPointerException during index > --------------------------------- > > Key: NUTCH-284 > URL: http://issues.apache.org/jira/browse/NUTCH-284 > Project: Nutch > Type: Bug > Components: indexer > Versions: 0.8-dev > Reporter: Stefan Neufeind > > For quite a few this "reduce > sort" has been going on. Then it fails. What could be wrong with this? > 060524 212613 reduce > sort > 060524 212614 reduce > sort > 060524 212615 reduce > sort > 060524 212615 found resource common-terms.utf8 at file:/home/mm/nutch-nightly-prod/conf/common-terms.utf8 > 060524 212615 found resource common-terms.utf8 at file:/home/mm/nutch-nightly-prod/conf/common-terms.utf8 > 060524 212619 Optimizing index. > 060524 212619 job_jlbhhm > java.lang.NullPointerException > at org.apache.nutch.indexer.Indexer$OutputFormat$1.write(Indexer.java:111) > at org.apache.hadoop.mapred.ReduceTask$3.collect(ReduceTask.java:269) > at org.apache.nutch.indexer.Indexer.reduce(Indexer.java:253) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:282) > at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:114) > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:341) > at org.apache.nutch.indexer.Indexer.index(Indexer.java:287) > at org.apache.nutch.indexer.Indexer.main(Indexer.java:304) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 16:00:36 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 92667 invoked from network); 2 Jun 2006 16:00:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 16:00:36 -0000 Received: (qmail 83119 invoked by uid 500); 2 Jun 2006 16:00:30 -0000 Delivered-To: john93@example.net Received: (qmail 83069 invoked by uid 500); 2 Jun 2006 16:00:30 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 82979 invoked by uid 500); 2 Jun 2006 16:00:29 -0000 Delivered-To: john93@example.net Received: (qmail 82693 invoked by uid 99); 2 Jun 2006 16:00:29 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 09:00:28 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 09:00:27 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 88FFC7141F2 for <john93@example.net>; Fri, 2 Jun 2006 15:59:32 +0000 (GMT) Message-ID: <3569442.1149263972558.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 15:59:32 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-281) cached.jsp: base-href needs to be outside comments In-Reply-To: <7939049.1148431949821.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-281?page=comments#action_12414454 ] Stefan Groschupf commented on NUTCH-281: ---------------------------------------- Can you submit a patch file? > cached.jsp: base-href needs to be outside comments > -------------------------------------------------- > > Key: NUTCH-281 > URL: http://issues.apache.org/jira/browse/NUTCH-281 > Project: Nutch > Type: Bug > Components: web gui > Reporter: Stefan Neufeind > Priority: Trivial > > see cached.jsp > <base href="..."> > does not take effect when showing a cached page because of the comments around it -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 16:08:28 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 96833 invoked from network); 2 Jun 2006 16:08:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 16:08:28 -0000 Received: (qmail 2956 invoked by uid 500); 2 Jun 2006 16:08:26 -0000 Delivered-To: john93@example.net Received: (qmail 2931 invoked by uid 500); 2 Jun 2006 16:08:26 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 2920 invoked by uid 500); 2 Jun 2006 16:08:26 -0000 Delivered-To: john93@example.net Received: (qmail 2917 invoked by uid 99); 2 Jun 2006 16:08:26 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 09:08:26 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 09:08:25 -0700 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 3A1F77141E8 for <john93@example.net>; Fri, 2 Jun 2006 16:07:31 +0000 (GMT) Message-ID: <5817682.1149264451234.JavaMail.jira@brutus> Date: Fri, 2 Jun 2006 16:07:31 +0000 (GMT+00:00) From: "Stefan Groschupf (JIRA)" <john93@example.net> To: john93@example.net Subject: [jira] Commented: (NUTCH-275) Fetcher not parsing XHTML-pages at all In-Reply-To: <6226434.1148173291439.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/NUTCH-275?page=comments#action_12414456 ] Stefan Groschupf commented on NUTCH-275: ---------------------------------------- Should we switch off mime.type.magic by default? Some people was reporting the same problems. > Fetcher not parsing XHTML-pages at all > -------------------------------------- > > Key: NUTCH-275 > URL: http://issues.apache.org/jira/browse/NUTCH-275 > Project: Nutch > Type: Bug > Versions: 0.8-dev > Environment: problem with nightly-2006-05-20; worked fine with same website on 0.7.2 > Reporter: Stefan Neufeind > > Server reports page as "text/html" - so I thought it would be processed as html. > But something I guess evaluated the headers of the document and re-labeled it as "text/xml" (why not text/xhtml?). > For some reason there is no plugin to be found for indexing text/xml (why does TextParser not feel responsible?). > Links inside this document are NOT indexed at all - no digging this website actually stops here. > Funny thing: For some magical reasons the dtd-files referenced in the header seem to be valid links for the fetcher and as such are indexed in the next round (if urlfilter allows). > 060521 025018 fetching http://www.secreturl.something/ > 060521 025018 http.proxy.host = null > 060521 025018 http.proxy.port = 8080 > 060521 025018 http.timeout = 10000 > 060521 025018 http.content.limit = 65536 > 060521 025018 http.agent = NutchCVS/0.8-dev (Nutch; http://lucene.apache.org/nutch/bot.html; john93@example.net) > 060521 025018 fetcher.server.delay = 1000 > 060521 025018 http.max.delays = 1000 > 060521 025018 ParserFactory:Plugin: org.apache.nutch.parse.text.TextParser mapped to contentType text/xml via parse-plugins.xml, but > its plugin.xml file does not claim to support contentType: text/xml > 060521 025018 ParserFactory: Plugin: org.apache.nutch.parse.rss.RSSParser mapped to contentType text/xml via parse-plugins.xml, but > not enabled via plugin.includes in nutch-default.xml > 060521 025019 Using Signature impl: org.apache.nutch.crawl.MD5Signature > 060521 025019 map 0% reduce 0% > 060521 025019 1 pages, 0 errors, 1.0 pages/s, 40 kb/s, > 060521 025019 1 pages, 0 errors, 1.0 pages/s, 40 kb/s, -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira From nutch-devjohn93@example.net Fri Jun 02 16:14:28 2006 Return-Path: <nutch-devjohn93@example.net> Delivered-To: john93@example.net Received: (qmail 1845 invoked from network); 2 Jun 2006 16:14:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2006 16:14:28 -0000 Received: (qmail 17682 invoked by uid 500); 2 Jun 2006 16:14:27 -0000 Delivered-To: john93@example.net Received: (qmail 17514 invoked by uid 500); 2 Jun 2006 16:14:26 -0000 Mailing-List: contact john93@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:john93@example.net> List-Unsubscribe: <mailto:john93@example.net> List-Post: <mailto:john93@example.net> List-Id: <nutch-dev.lucene.apache.org> Reply-To: john93@example.net Delivered-To: mailing list john93@example.net Received: (qmail 17503 invoked by uid 500); 2 Jun 2006 16:14:26 -0000 Delivered-To: john93@example.net Received: (qmail 17500 invoked by uid 99); 2 Jun 2006 16:14:26 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jun 2006 09:14:26 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=
{'source': 'Apache Software Foundation Public Mail Archives'}