From nutch-usermartinezgary@example.org Thu Jan 01 03:52:31 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 70895 invoked from network); 1 Jan 2009 03:52:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jan 2009 03:52:31 -0000 Received: (qmail 30604 invoked by uid 500); 1 Jan 2009 03:52:24 -0000 Delivered-To: martinezgary@example.org Received: (qmail 30574 invoked by uid 500); 1 Jan 2009 03:52:24 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 30563 invoked by uid 99); 1 Jan 2009 03:52:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 31 Dec 2008 19:52:24 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO qw-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Jan 2009 03:52:17 +0000 Received: by qw-out-2122.google.com with SMTP id 5so2881926qwi.53 for <martinezgary@example.org>; Wed, 31 Dec 2008 19:51:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=3FNPPwTUkQVa3xw3zK4bcjzP12N5s/ltPVNfN8YOS6Q=; b=K6bVrod4QBB8pA1ptsBxBrMig5MPE5tFKXY2yYudI6oIO8wZuSdqaUx3l4fs/u4qkd G8Zs5my1lXeDarJMJ2VQ3VooMmj2/5ZJSS3PEmJP8hQSoBeKnF54TQmelqPBDAG7Hbgn sLj4i2YsGe3dElUCR2yBGDzeO56Xfo39H1W7k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=TVHMKnzOHoW5oxEnZhfVrbz4HFjZ23Zcsf12Y5k0g1F32JuB9tzAjFuUc9w6mYwHbG Dn1d/BCy5P3IJn91yK9l7LYBZp7DVHfRbgldq4pyqJT3cZ6AZ33m+PtjOKZ4zql7RW4A wb2esiXqNwUMDhq0LSVfexWWPMuuH+pa32Qyg= Received: by 83.161.120.155 with SMTP id u14mr14307244qak.381.1230781915916; Wed, 31 Dec 2008 19:51:55 -0800 (PST) Received: by 83.161.120.155 with HTTP; Wed, 31 Dec 2008 19:51:55 -0800 (PST) Message-ID: <martinezgary@example.org> Date: Wed, 31 Dec 2008 22:51:55 -0500 From: "blackwater dev" <martinezgary@example.org> To: martinezgary@example.org Subject: craw failing MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_145643_15615862.1230781915909" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_145643_15615862.1230781915909 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline I just grabbed nutch and did some crawls and noticed when I did a search, I was getting an error so looked into the log and see this: Indexer: starting Indexer: linkdb: crawl.test/linkdb Indexer: adding segment: crawl.test/segments/20081224211810 Indexer: adding segment: crawl.test/segments/20081224211837 Indexer: adding segment: crawl.test/segments/20081224211858 Optimizing index. Indexer: done Dedup: starting Dedup: adding indexes in: crawl.test/indexes Exception in thread "main" java.io.IOException: Job martinezgary@example.org(JobClient.java:604) at org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java:439) at org.apache.nutch.crawl.Crawl.main(Crawl.java:135) How can I fix this? Thanks! ------=_Part_145643_15615862.1230781915909-- From nutch-usermartinezgary@example.org Fri Jan 02 12:29:48 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 71099 invoked from network); 2 Jan 2009 12:29:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 12:29:46 -0000 Received: (qmail 2318 invoked by uid 500); 2 Jan 2009 12:29:39 -0000 Delivered-To: martinezgary@example.org Received: (qmail 2285 invoked by uid 500); 2 Jan 2009 12:29:39 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 2273 invoked by uid 99); 2 Jan 2009 12:29:38 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 04:29:38 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO n6.bullet.re3.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 02 Jan 2009 12:29:29 +0000 Received: from [83.161.120.155] by n6.bullet.re3.yahoo.com with NNFMP; 02 Jan 2009 12:29:08 -0000 Received: from [83.161.120.155] by t3.bullet.re3.yahoo.com with NNFMP; 02 Jan 2009 12:29:08 -0000 Received: from [83.161.120.155] by omp203.mail.re3.yahoo.com with NNFMP; 02 Jan 2009 12:29:08 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: martinezgary@example.org Received: (qmail 10561 invoked by uid 60001); 2 Jan 2009 12:29:08 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Message-ID; b=mf9RFpnsHQG0TowuCo6YyVI6ks39aSwRPpUYzXwPqNfIZYU6aWokbAA+gaE2t99dqHUoVQptGmEt3M3ZyRHU0AnKrEPS3yGk6Zx6FSTCmwdIjdJJVa8YOft24/vAiKPMAZ+0gs54lRwV22c4ylpg4CaMwd0KQ+s4MnsZTuSv5M8=; X-YMail-OSG: nYLfXggVM1mFCKarUHI24d8Taz9LvPuwGWL8lRzIPngR2z13yYcxezR5KyCcGIji2jFajQo0HeEdtmOusyJnQaL72SD38OhS6ABwEe1s7if4AgzB0fNGI7sLexlZtObgY3cg54cC3R35VTSrDYrbJUY1Nfc- Received: from [83.161.120.155] by web57416.mail.re1.yahoo.com via HTTP; Fri, 02 Jan 2009 04:29:06 PST X-Mailer: YahooMailWebService/0.7.260.1 Date: Fri, 2 Jan 2009 04:29:06 -0800 (PST) From: Neil Rosewarm <martinezgary@example.org> Reply-To: martinezgary@example.org Subject: Crawl Timing_Please help To: martinezgary@example.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-1580565254-1230899346=:7832" Message-ID: <martinezgary@example.org> X-Virus-Checked: Checked by ClamAV on apache.org --0-1580565254-1230899346=:7832 Content-Type: text/plain; charset=us-ascii Dear Friends, Wish you all a happy and prosperous new year 2009! We are trying to crawl 50,0000 pages. we are surprised to see it is taking 3 full days. Though, we have a server with 4GB RAM and Core 2 DUO processor with 250 GB Hard drive, we had numerous issues like power down, internet connectivity etc..... I am really concerned for the time taking for a single crawl. Please, anyone suggest a best way of crawling faster? Also, request you to suggest any idea to configure the crawling from where it stopped due to any reasons like power cut..internet cutoff etc.... Your advises are highly appreciated! Thank you, Neil --0-1580565254-1230899346=:7832-- From nutch-usermartinezgary@example.org Fri Jan 02 12:44:22 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 76968 invoked from network); 2 Jan 2009 12:44:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 12:44:16 -0000 Received: (qmail 19470 invoked by uid 500); 2 Jan 2009 12:44:10 -0000 Delivered-To: martinezgary@example.org Received: (qmail 19438 invoked by uid 500); 2 Jan 2009 12:44:10 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 19367 invoked by uid 99); 2 Jan 2009 12:44:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 04:44:10 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO tm14.itroot.de) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 12:44:02 +0000 From: =?iso-8859-1?Q?H=F6chst=F6tter_Nadine?= <martinezgary@example.org> To: "martinezgary@example.org" <martinezgary@example.org>, "martinezgary@example.org" <martinezgary@example.org> Date: Fri, 2 Jan 2009 13:43:26 +0100 Subject: AW: Crawl Timing_Please help Thread-Topic: Crawl Timing_Please help Thread-Index: Acls1dQqpSZW2J4fTE6Gdcr/v4MyCAAAZXDw Message-ID: <martinezgary@example.org> References: <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> Accept-Language: de-DE Content-Language: de-DE X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: de-DE Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org Hi, Neil, how many unique host do you crawl? Instal jnettop to monitor the bandwidth. Are there any restrictions in resp= ect to bandwidth usage?=20 Cheers, Nadine. -----Urspr=FCngliche Nachricht----- Von: Neil Rosewarm [mailto:martinezgary@example.org]=20 Gesendet: Freitag, 2. Januar 2009 13:29 An: martinezgary@example.org Betreff: Crawl Timing_Please help Dear Friends, Wish you all a happy and prosperous new year 2009! We are trying to crawl 50,0000 pages. we are surprised to see it is taking = 3 full days. Though, we have a server with 4GB RAM and Core 2 DUO processor= with 250 GB Hard drive, we had numerous issues like power down, internet c= onnectivity etc..... I am really concerned for the time taking for a single crawl. Please, anyon= e suggest a best way of crawling faster? Also, request you to suggest any idea to configure the crawling from where = it stopped due to any reasons like power cut..internet cutoff etc.... Your advises are highly appreciated! Thank you, Neil From nutch-usermartinezgary@example.org Fri Jan 02 15:42:06 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 93616 invoked from network); 2 Jan 2009 15:42:06 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 15:42:06 -0000 Received: (qmail 91424 invoked by uid 500); 2 Jan 2009 15:42:03 -0000 Delivered-To: martinezgary@example.org Received: (qmail 91385 invoked by uid 500); 2 Jan 2009 15:42:03 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 91374 invoked by uid 99); 2 Jan 2009 15:42:03 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 07:42:03 -0800 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of martinezgary@example.org does not designate 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wf-out-1314.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 15:41:54 +0000 Received: by wf-out-1314.google.com with SMTP id 28so6376284wfc.20 for <martinezgary@example.org>; Fri, 02 Jan 2009 07:41:32 -0800 (PST) Received: by 83.161.120.155 with SMTP id r4mr7377314wfe.105.1230910892725; Fri, 02 Jan 2009 07:41:32 -0800 (PST) Received: from ?83.161.120.155? (c-71-229-180-231.hsd1.co.comcast.net [83.161.120.155]) by mx.google.com with ESMTPS id 30sm35749976wff.52.283.161.120.155.41.31 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 02 Jan 2009 07:41:31 -0800 (PST) Message-Id: <martinezgary@example.org> From: John Martyniak <martinezgary@example.org> To: "martinezgary@example.org" <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit X-Mailer: iPhone Mail (5G77) Mime-Version: 1.0 (iPhone Mail 5G77) Subject: Re: Crawl Timing_Please help Date: Fri, 2 Jan 2009 08:41:15 -0700 References: <martinezgary@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Neil, That shouldn't take that long, when I do a 50K page crawl it takes a few hours. Have you tried it again? Without the issues that you where talking about. To my knowledge I don't think that you can restart a crawl from where it left off, it would be a cool feature. Also is this hosted or is it running off Dsl/Cable Modem? -John On Jan 2, 2009, at 5:29 AM, Neil Rosewarm <martinezgary@example.org> wrote: > Dear Friends, > > Wish you all a happy and prosperous new year 2009! > > We are trying to crawl 50,0000 pages. we are surprised to see it is > taking 3 full days. Though, we have a server with 4GB RAM and Core 2 > DUO processor with 250 GB Hard drive, we had numerous issues like > power down, internet connectivity etc..... > > I am really concerned for the time taking for a single crawl. > Please, anyone suggest a best way of crawling faster? > > Also, request you to suggest any idea to configure the crawling from > where it stopped due to any reasons like power cut..internet cutoff > etc.... > > Your advises are highly appreciated! > > Thank you, > > Neil From nutch-usermartinezgary@example.org Fri Jan 02 17:48:03 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 40426 invoked from network); 2 Jan 2009 17:48:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jan 2009 17:48:02 -0000 Received: (qmail 15829 invoked by uid 500); 2 Jan 2009 17:47:59 -0000 Delivered-To: martinezgary@example.org Received: (qmail 15664 invoked by uid 500); 2 Jan 2009 17:47:58 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 15653 invoked by uid 99); 2 Jan 2009 17:47:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 09:47:58 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO fg-out-1718.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Jan 2009 17:47:50 +0000 Received: by fg-out-1718.google.com with SMTP id l27so2372822fgb.27 for <martinezgary@example.org>; Fri, 02 Jan 2009 09:47:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=qhVXNmLY//lFriFQrzKgK4q3xSamKTDxO4LfaviFIKE=; b=rOXGwU7m4LjUP/0v/FfMQJbytog8rUdam4dOv3HRdsfhx37u17e9RKUpjGgOjpHgzN snGgLsnLIDeEefhystsVveN36o++peAqvtk3eW7R3Ao1gRbdsaiPvfyT8N3Ay+u9rFuK R15dJgYrTW15iRaFpOeF6pq5qMvysjJ1Jpu1M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=hJT9grJgDkfUuVPuaXObbiAQUqMPhRNiKzsru/VdKO3YtyUfZhx4qj3V5bZe/yknXb 8dgi9xL3eZgjddi0NfxeMT8slu52dx6u7vkMU/ZNdRb0ppkkScFVIYCAR0+lM3c0N2ik bX7D5MtsvAPcCanpI9ME1bIZDNnMtxBwXv/SM= Received: by 83.161.120.155 with SMTP id 17mr10635806fgw.0.1230918450608; Fri, 02 Jan 2009 09:47:30 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Jan 2009 09:47:30 -0800 (PST) Message-ID: <martinezgary@example.org> Date: Fri, 2 Jan 2009 19:47:30 +0200 From: "Boris Shulman" <martinezgary@example.org> To: martinezgary@example.org Subject: next nutch release MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_66877_22279322.1230918450602" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_66877_22279322.1230918450602 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Does anyone knows when 1.0 release is expected? ------=_Part_66877_22279322.1230918450602-- From nutch-usermartinezgary@example.org Sun Jan 04 08:30:56 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 42473 invoked from network); 4 Jan 2009 08:30:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2009 08:30:56 -0000 Received: (qmail 14882 invoked by uid 500); 4 Jan 2009 08:30:50 -0000 Delivered-To: martinezgary@example.org Received: (qmail 14843 invoked by uid 500); 4 Jan 2009 08:30:49 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 14832 invoked by uid 99); 4 Jan 2009 08:30:49 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2009 00:30:49 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO fg-out-1718.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2009 08:30:43 +0000 Received: by fg-out-1718.google.com with SMTP id l27so2559187fgb.27 for <martinezgary@example.org>; Sun, 04 Jan 2009 00:30:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=SlQmkDntxnH5RAsmcZ1i6JVT0qs0SfL6tJgx4pFvkrE=; b=wYKmYh8TIbAfn193s7IzLWUN2QMqrIfyPAxrtLEGvTzeOTCs/CWTS6yvj87zZYmqPv niaUkyi2GNcf/RauWjR8ToI44AIAmzWR+VTTx9Ys0ZdTbAzmgXumdfmP5gq/ArwIjz8y Z7Pr5aaHSfy0WbiTqM5Q/ocA1dj+wu5+5iS0A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=QD373tZRc6ZjJZHCoqZKF9I3Bd7GTRbkp5wHK8pKPA+rVeO2QX12LuHusZldp/qPBd KZN1BtUIwm2PDk6ifdDKgCTGHbTv7zQUGf5fROazgCXYVO3Yfp5LhQ0CD3pVdPN4MMuK rKDTthjwnOUtv59rsoEulqQKXcC2Ck9nTxsIQ= Received: by 83.161.120.155 with SMTP id c8mr11506011fgc.67.1231057820802; Sun, 04 Jan 2009 00:30:20 -0800 (PST) Received: by 83.161.120.155 with HTTP; Sun, 4 Jan 2009 00:30:20 -0800 (PST) Message-ID: <martinezgary@example.org> Date: Sun, 4 Jan 2009 10:30:20 +0200 From: "Boris Shulman" <martinezgary@example.org> To: martinezgary@example.org Subject: next nutch relase MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_78955_29847038.1231057820791" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_78955_29847038.1231057820791 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Does anyone knows when next nutch release is expected? ------=_Part_78955_29847038.1231057820791-- From nutch-usermartinezgary@example.org Sun Jan 04 15:48:58 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 37153 invoked from network); 4 Jan 2009 15:48:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2009 15:48:58 -0000 Received: (qmail 54175 invoked by uid 500); 4 Jan 2009 15:48:52 -0000 Delivered-To: martinezgary@example.org Received: (qmail 53993 invoked by uid 500); 4 Jan 2009 15:48:51 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 53982 invoked by uid 99); 4 Jan 2009 15:48:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2009 07:48:51 -0800 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO s01.igfoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2009 15:48:41 +0000 Received: from localhost (localhost [83.161.120.155]) by s01.igfoo.com (Postfix) with ESMTP id E739FF341C1 for <martinezgary@example.org>; Sun, 4 Jan 2009 09:48:19 -0600 (CST) X-DKIM: Sendmail DKIM Filter v2.5.4 s01.igfoo.com E739FF341C1 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=igfoo.com; s=mail; t=1231084099; bh=hOJzuUWiqB3mFdM90cnn8mF9LJ5Os6SxulHqo9W9ums=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=yQXGZMf8/8Ns BhfiWw99T5pYcG+Dcswt5XsF0S3GIzFQ0fyLshgk+eqA3MPfSsOBd1KmgzFQvBKIrjY T61wLfQDB/jjZn9katWeg/6OSDV2s0u2IMUyMdlVer3cxkF3Irby6sz7K5V6wTLUAYU 1BQSmGmhqvxmD2hdoeCBmwM+k= X-Virus-Scanned: Debian martinezgary@example.org Received: from s01.igfoo.com ([83.161.120.155]) by localhost (s01.igfoo.com [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id 1ZsApTpcOh65 for <martinezgary@example.org>; Sun, 4 Jan 2009 09:48:15 -0600 (CST) Received: from [83.161.120.155] (pool-71-244-0-85.dllstx.fios.verizon.net [83.161.120.155]) by s01.igfoo.com (Postfix) with ESMTPSA id 9A144F34100 for <martinezgary@example.org>; Sun, 4 Jan 2009 09:48:15 -0600 (CST) X-DKIM: Sendmail DKIM Filter v2.5.4 s01.igfoo.com 9A144F34100 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=igfoo.com; s=mail; t=1231084095; bh=hOJzuUWiqB3mFdM90cnn8mF9LJ5Os6SxulHqo9W9ums=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=dV4+MpNIJEcf xAHj/7qQmimAf8NigU78E/QpJltL/uDH7xJeCRW0NfjcqPmxH8KbIu35Mr8haal4tcG TFfM/T/IvajXI2O76SzHni3bpXpboKLlGXSc8vR2AyKDcDiiq3Q0lJcUzbGVg6DI2Da pPoVw8gmVW5bR0rHJKD2G4T28= Message-ID: <martinezgary@example.org> Date: Sun, 04 Jan 2009 09:48:14 -0600 From: Dennis Kubes <martinezgary@example.org> User-Agent: Thunderbird 83.161.120.155 (X11/20081125) MIME-Version: 1.0 To: martinezgary@example.org Subject: Re: next nutch relase References: <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Soon. We are working hard on getting current patches committed and issues resolved. 1-2 weeks. Dennis Boris Shulman wrote: > Does anyone knows when next nutch release is expected? > From nutch-usermartinezgary@example.org Sun Jan 04 16:31:24 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 49815 invoked from network); 4 Jan 2009 16:31:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2009 16:31:23 -0000 Received: (qmail 92966 invoked by uid 500); 4 Jan 2009 16:31:21 -0000 Delivered-To: martinezgary@example.org Received: (qmail 92930 invoked by uid 500); 4 Jan 2009 16:31:21 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 92919 invoked by uid 99); 4 Jan 2009 16:31:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2009 08:31:21 -0800 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=SPF_SOFTFAIL X-Spam-Check-By: apache.org Received-SPF: softfail (nike.apache.org: transitioning domain of martinezgary@example.org does not designate 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wf-out-1314.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2009 16:31:12 +0000 Received: by wf-out-1314.google.com with SMTP id 28so7047656wfc.20 for <martinezgary@example.org>; Sun, 04 Jan 2009 08:30:51 -0800 (PST) Received: by 83.161.120.155 with SMTP id o21mr4838222wfd.150.1231086650913; Sun, 04 Jan 2009 08:30:50 -0800 (PST) Received: from johns-computer.home (c-71-229-180-231.hsd1.co.comcast.net [83.161.120.155]) by mx.google.com with ESMTPS id 22sm34771348wfg.10.283.161.120.155.30.49 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 04 Jan 2009 08:30:50 -0800 (PST) Message-Id: <martinezgary@example.org> From: John Martyniak <martinezgary@example.org> To: martinezgary@example.org In-Reply-To: <martinezgary@example.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Re: next nutch relase Date: Sun, 4 Jan 2009 09:30:46 -0700 References: <martinezgary@example.org> <martinezgary@example.org> X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org Awesome. Thanks for the update. -John On Jan 4, 2009, at 8:48 AM, Dennis Kubes wrote: > Soon. We are working hard on getting current patches committed and > issues resolved. 1-2 weeks. > > Dennis > > Boris Shulman wrote: >> Does anyone knows when next nutch release is expected? From nutch-usermartinezgary@example.org Sun Jan 04 17:45:38 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 68310 invoked from network); 4 Jan 2009 17:45:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 4 Jan 2009 17:45:38 -0000 Received: (qmail 38769 invoked by uid 500); 4 Jan 2009 17:45:31 -0000 Delivered-To: martinezgary@example.org Received: (qmail 38731 invoked by uid 500); 4 Jan 2009 17:45:31 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 38720 invoked by uid 99); 4 Jan 2009 17:45:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2009 09:45:31 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail.cs.utexas.edu) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 04 Jan 2009 17:45:22 +0000 Received: from [83.161.120.155] (cpe-67-9-133-138.austin.res.rr.com [83.161.120.155]) (authenticated bits=0) by mail.cs.utexas.edu (8.14.3/8.14.3) with ESMTP id n04Hixxe024864 for <martinezgary@example.org>; Sun, 4 Jan 2009 11:45:02 -0600 (CST) Message-ID: <martinezgary@example.org> Date: Sun, 04 Jan 2009 11:44:56 -0600 From: Shirley Cohen <martinezgary@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20081209) MIME-Version: 1.0 To: martinezgary@example.org Subject: problem running fetcher using hadoop jar nutch*.job command Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi, I'm new to nutch and am trying to run it on an existing hadoop 0.19.0 install. I'm using the command "hadoop jar nutch-2008-12-02_04-01-57.job", as suggested by Dennis Kubes in an earlier post. I've been able to crawl and generate segments successfully using the following commands: hadoop dfs -put dmoz dmoz bin/hadoop jar nutch-2008-12-02_04-01-57.job org.apache.nutch.crawl.Injector crawl/crawldb dmoz bin/hadoop jar nutch-2008-12-02_04-01-57.job org.apache.nutch.crawl.Generator crawl/crawldb crawl/segments However, when I try to run the fetcher using the command: bin/hadoop jar nutch-2008-12-02_04-01-57.job org.apache.nutch.fetcher.Fetcher crawl/segments/20090104094558 I get the following error: 09/01/04 10:20:31 INFO fetcher.Fetcher: Fetcher: starting 09/01/04 10:20:31 INFO fetcher.Fetcher: Fetcher: segment: crawl/segments/20090104094558 ****calling init JobTracker***** java.lang.NoSuchMethodError: org.apache.nutch.fetcher.Fetcher$InputFormat.listPaths(Lorg/apache/hadoop/mapred/JobConf;)[Lorg/apache/hadoop/fs/Path; at org.apache.nutch.fetcher.Fetcher$InputFormat.getSplits(Fetcher.java:61) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:783) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1128) at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:530) at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:565) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:537) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.util.RunJar.main(RunJar.java:165) at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) Note: The subdirectory "20090104094558" was created by the generator. I'm running the 0.9 release of nutch downloaded from: http://mirrors.24-7-solutions.net/pub/apache/lucene/nutch/ Does anyone know what is going on? Thanks in advance, Shirley From nutch-usermartinezgary@example.org Mon Jan 05 13:55:55 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 71445 invoked from network); 5 Jan 2009 13:55:55 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2009 13:55:55 -0000 Received: (qmail 34932 invoked by uid 500); 5 Jan 2009 13:55:48 -0000 Delivered-To: martinezgary@example.org Received: (qmail 34891 invoked by uid 500); 5 Jan 2009 13:55:48 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 34880 invoked by uid 99); 5 Jan 2009 13:55:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 05:55:48 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO LEERNUG302.Union.lizearle.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 13:55:39 +0000 Received: from le21.Union.lizearle.com ([83.161.120.155]) by LEERNUG302.Union.lizearle.com with Microsoft SMTPSVC406-234-2140); Mon, 5 Jan 2009 13:55:18 +0000 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C96F3D.3E6292C2" Subject: store 'content' field in the index X-MimeOLE: Produced By Microsoft Exchange V6.5 Date: Mon, 5 Jan 2009 13:55:18 -0000 Message-ID: <martinezgary@example.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: store 'content' field in the index Thread-Index: AclvPT5PjIxa1CB0QoKVVWl8XchaEg== From: "Matt Pearson" <martinezgary@example.org> To: <martinezgary@example.org> X-OriginalArrivalTime: 05 Jan 2009 13:55:18.0982 (UTC) FILETIME=[3E8BFE60:01C96F3D] X-Virus-Checked: Checked by ClamAV on apache.org ------_=_NextPart_001_01C96F3D.3E6292C2 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Everyone, =20 Does anyone know of a way I can configure the Nutch crawler to store the contents of a document in the generated index? =20 Thanks =20 =20 Matt =20 =20 Matt Pearson=20 =20 ------_=_NextPart_001_01C96F3D.3E6292C2-- From nutch-usermartinezgary@example.org Mon Jan 05 13:58:47 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 73358 invoked from network); 5 Jan 2009 13:58:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2009 13:58:47 -0000 Received: (qmail 41736 invoked by uid 500); 5 Jan 2009 13:58:43 -0000 Delivered-To: martinezgary@example.org Received: (qmail 41711 invoked by uid 500); 5 Jan 2009 13:58:43 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 41700 invoked by uid 99); 5 Jan 2009 13:58:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 05:58:43 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO bay0-omc2-s38.bay0.hotmail.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 13:58:34 +0000 Received: from BAY136-DS7 ([83.161.120.155]) by bay0-omc2-s38.bay0.hotmail.com with Microsoft SMTPSVC406-234-2140); Mon, 5 Jan 2009 05:58:13 -0800 X-Originating-IP: [83.161.120.155] X-Originating-Email: [martinezgary@example.org] Message-ID: <martinezgary@example.org> From: "Ian.huang" <martinezgary@example.org> To: <martinezgary@example.org> References: <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> Subject: Re: store 'content' field in the index Date: Mon, 5 Jan 2009 13:57:53 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal Importance: Normal X-Mailer: Microsoft Windows Live Mail 14.0.8050.1202 X-MimeOLE: Produced By Microsoft MimeOLE V14.0.8050.1202 X-OriginalArrivalTime: 05 Jan 2009 13:58:13.0632 (UTC) FILETIME=[A6A57800:01C96F3D] X-Virus-Checked: Checked by ClamAV on apache.org I think you need to change the basicindexfilter Field content=new Field("content", parse.getText(), Field.Store.NO, Field.Index.TOKENIZED); change the Field.Store.NO to Field.Store.YES Ian -------------------------------------------------- From: "Matt Pearson" <martinezgary@example.org> Sent: Monday, January 05, 2009 1:55 PM To: <martinezgary@example.org> Subject: store 'content' field in the index > Hi Everyone, > > > > Does anyone know of a way I can configure the Nutch crawler to store the > contents of a document in the generated index? > > > > Thanks > > > > > > Matt > > > > > > Matt Pearson > > > > From nutch-usermartinezgary@example.org Mon Jan 05 17:34:53 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 8395 invoked from network); 5 Jan 2009 17:34:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2009 17:34:53 -0000 Received: (qmail 301 invoked by uid 500); 5 Jan 2009 17:34:51 -0000 Delivered-To: martinezgary@example.org Received: (qmail 280 invoked by uid 500); 5 Jan 2009 17:34:51 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 269 invoked by uid 99); 5 Jan 2009 17:34:50 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 09:34:50 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail.cs.utexas.edu) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 17:34:39 +0000 Received: from [83.161.120.155] (cpe-67-9-133-138.austin.res.rr.com [83.161.120.155]) (authenticated bits=0) by mail.cs.utexas.edu (8.14.3/8.14.3) with ESMTP id n05HYCdg007855 for <martinezgary@example.org>; Mon, 5 Jan 2009 11:34:17 -0600 (CST) Message-ID: <martinezgary@example.org> Date: Mon, 05 Jan 2009 11:34:09 -0600 From: Shirley Cohen <martinezgary@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20081209) MIME-Version: 1.0 To: martinezgary@example.org Subject: Re: problem running fetcher using hadoop jar nutch*.job command References: <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi, I found a workaround to this problem. I was able to run the fetcher with the nutch*.job command using the latest working nighly build from 12-28-2008. Shirley Shirley Cohen wrote: > Hi, > > I'm new to nutch and am trying to run it on an existing hadoop 0.19.0 > install. I'm using the command "hadoop jar > nutch-2008-12-02_04-01-57.job", as suggested by Dennis Kubes in an > earlier post. I've been able to crawl and generate segments > successfully using the following commands: > > hadoop dfs -put dmoz dmoz > bin/hadoop jar nutch-2008-12-02_04-01-57.job > org.apache.nutch.crawl.Injector crawl/crawldb dmoz > bin/hadoop jar nutch-2008-12-02_04-01-57.job > org.apache.nutch.crawl.Generator crawl/crawldb crawl/segments > > However, when I try to run the fetcher using the command: > > bin/hadoop jar nutch-2008-12-02_04-01-57.job > org.apache.nutch.fetcher.Fetcher crawl/segments/20090104094558 > > I get the following error: > > 09/01/04 10:20:31 INFO fetcher.Fetcher: Fetcher: starting > 09/01/04 10:20:31 INFO fetcher.Fetcher: Fetcher: segment: > crawl/segments/20090104094558 > ****calling init JobTracker***** > java.lang.NoSuchMethodError: > org.apache.nutch.fetcher.Fetcher$InputFormat.listPaths(Lorg/apache/hadoop/mapred/JobConf;)[Lorg/apache/hadoop/fs/Path; > > at > org.apache.nutch.fetcher.Fetcher$InputFormat.getSplits(Fetcher.java:61) > at > org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:783) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1128) > at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:530) > at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:565) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:537) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at org.apache.hadoop.util.RunJar.main(RunJar.java:165) > at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) > > Note: The subdirectory "20090104094558" was created by the generator. > > I'm running the 0.9 release of nutch downloaded from: > http://mirrors.24-7-solutions.net/pub/apache/lucene/nutch/ > > Does anyone know what is going on? > > Thanks in advance, > > Shirley From nutch-usermartinezgary@example.org Mon Jan 05 20:33:09 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 95369 invoked from network); 5 Jan 2009 20:33:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2009 20:33:08 -0000 Received: (qmail 28503 invoked by uid 500); 5 Jan 2009 20:33:02 -0000 Delivered-To: martinezgary@example.org Received: (qmail 28227 invoked by uid 500); 5 Jan 2009 20:33:01 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 28216 invoked by uid 99); 5 Jan 2009 20:33:01 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 12:33:01 -0800 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO s01.igfoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 20:32:51 +0000 Received: from localhost (localhost [83.161.120.155]) by s01.igfoo.com (Postfix) with ESMTP id D2418F341C0 for <martinezgary@example.org>; Mon, 5 Jan 2009 14:32:28 -0600 (CST) X-DKIM: Sendmail DKIM Filter v2.5.4 s01.igfoo.com D2418F341C0 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=igfoo.com; s=mail; t=1231187548; bh=Td8du5KRJBtTZ4iW1o49jXhuTFU4jBbxdvxHM3Fubxc=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=n1xj34GYPMj5 FWPhcZqZ4ZQlUqZsuu5TLFSfOnACQEkaSPYMEdogQ8zu7gGzAipUoVQnAtj/+AsBEQn BCFVf3JmOP7f0BpCA2a5GmAHvi2AUtKzQmgABWzd1603jWi/5G/kD/KjYc9rd6AhdEr LdBHrZ66xeOY9dDPcj0tZOVCs= X-Virus-Scanned: Debian martinezgary@example.org Received: from s01.igfoo.com ([83.161.120.155]) by localhost (s01.igfoo.com [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id 1pwRh+AmviFH for <martinezgary@example.org>; Mon, 5 Jan 2009 14:32:27 -0600 (CST) Received: from [83.161.120.155] (pool-71-244-0-85.dllstx.fios.verizon.net [83.161.120.155]) by s01.igfoo.com (Postfix) with ESMTPSA id 14A1CF34174 for <martinezgary@example.org>; Mon, 5 Jan 2009 14:32:27 -0600 (CST) X-DKIM: Sendmail DKIM Filter v2.5.4 s01.igfoo.com 14A1CF34174 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=igfoo.com; s=mail; t=1231187547; bh=Td8du5KRJBtTZ4iW1o49jXhuTFU4jBbxdvxHM3Fubxc=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=rxWs2prDlkK9 BxAv6I1qsrPU2KJ3U9N+4KYQ8BCSTi7mE1yj5Pxe3hRaMbVtabNEYOE1Jc6N9KfMyLt JKN2Z9UC3tdSX+o4JGWrOVrwbL93X4u4XQSy9hymZWxm9UJ0VqEdZ4bkxHAjsW5QE7+ XDIwqIh+uzuAJ7Dj0hzVkGJ54= Message-ID: <martinezgary@example.org> Date: Mon, 05 Jan 2009 14:32:26 -0600 From: Dennis Kubes <martinezgary@example.org> User-Agent: Thunderbird 83.161.120.155 (X11/20081125) MIME-Version: 1.0 To: martinezgary@example.org Subject: Re: problem running fetcher using hadoop jar nutch*.job command References: <martinezgary@example.org> <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org It looks to me like you have a mismatch in the version of hadoop you are using with Nutch. Nutch trunk is on 0.19. You might want to try building from SVN and then retrying. Dennis Shirley Cohen wrote: > Hi, > > I found a workaround to this problem. I was able to run the fetcher with > the nutch*.job command using the latest working nighly build from > 12-28-2008. > > Shirley > > Shirley Cohen wrote: >> Hi, >> >> I'm new to nutch and am trying to run it on an existing hadoop 0.19.0 >> install. I'm using the command "hadoop jar >> nutch-2008-12-02_04-01-57.job", as suggested by Dennis Kubes in an >> earlier post. I've been able to crawl and generate segments >> successfully using the following commands: >> >> hadoop dfs -put dmoz dmoz >> bin/hadoop jar nutch-2008-12-02_04-01-57.job >> org.apache.nutch.crawl.Injector crawl/crawldb dmoz >> bin/hadoop jar nutch-2008-12-02_04-01-57.job >> org.apache.nutch.crawl.Generator crawl/crawldb crawl/segments >> >> However, when I try to run the fetcher using the command: >> >> bin/hadoop jar nutch-2008-12-02_04-01-57.job >> org.apache.nutch.fetcher.Fetcher crawl/segments/20090104094558 >> >> I get the following error: >> >> 09/01/04 10:20:31 INFO fetcher.Fetcher: Fetcher: starting >> 09/01/04 10:20:31 INFO fetcher.Fetcher: Fetcher: segment: >> crawl/segments/20090104094558 >> ****calling init JobTracker***** >> java.lang.NoSuchMethodError: >> org.apache.nutch.fetcher.Fetcher$InputFormat.listPaths(Lorg/apache/hadoop/mapred/JobConf;)[Lorg/apache/hadoop/fs/Path; >> >> at >> org.apache.nutch.fetcher.Fetcher$InputFormat.getSplits(Fetcher.java:61) >> at >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:783) >> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1128) >> at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:530) >> at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:565) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:537) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >> at java.lang.reflect.Method.invoke(Unknown Source) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:165) >> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) >> >> Note: The subdirectory "20090104094558" was created by the generator. >> >> I'm running the 0.9 release of nutch downloaded from: >> http://mirrors.24-7-solutions.net/pub/apache/lucene/nutch/ >> >> Does anyone know what is going on? >> >> Thanks in advance, >> >> Shirley > > From nutch-usermartinezgary@example.org Mon Jan 05 22:36:55 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 68136 invoked from network); 5 Jan 2009 22:36:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 5 Jan 2009 22:36:54 -0000 Received: (qmail 61943 invoked by uid 500); 5 Jan 2009 22:36:52 -0000 Delivered-To: martinezgary@example.org Received: (qmail 61771 invoked by uid 500); 5 Jan 2009 22:36:51 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 61760 invoked by uid 99); 5 Jan 2009 22:36:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 14:36:51 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail.cs.utexas.edu) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Jan 2009 22:36:41 +0000 Received: from [83.161.120.155] (cpe-67-9-133-138.austin.res.rr.com [83.161.120.155]) (authenticated bits=0) by mail.cs.utexas.edu (8.14.3/8.14.3) with ESMTP id n05MaGrX012741 for <martinezgary@example.org>; Mon, 5 Jan 2009 16:36:19 -0600 (CST) Message-ID: <martinezgary@example.org> Date: Mon, 05 Jan 2009 16:36:14 -0600 From: Shirley Cohen <martinezgary@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20081209) MIME-Version: 1.0 To: martinezgary@example.org Subject: Re: problem running fetcher using hadoop jar nutch*.job command References: <martinezgary@example.org> <martinezgary@example.org> <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Yes, that's what I did and it worked :)) Thanks, Shirley Dennis Kubes wrote: > It looks to me like you have a mismatch in the version of hadoop you > are using with Nutch. Nutch trunk is on 0.19. You might want to try > building from SVN and then retrying. > > Dennis > > Shirley Cohen wrote: >> Hi, >> >> I found a workaround to this problem. I was able to run the fetcher >> with the nutch*.job command using the latest working nighly build >> from 12-28-2008. >> >> Shirley >> >> Shirley Cohen wrote: >>> Hi, >>> >>> I'm new to nutch and am trying to run it on an existing hadoop >>> 0.19.0 install. I'm using the command "hadoop jar >>> nutch-2008-12-02_04-01-57.job", as suggested by Dennis Kubes in an >>> earlier post. I've been able to crawl and generate segments >>> successfully using the following commands: >>> >>> hadoop dfs -put dmoz dmoz >>> bin/hadoop jar nutch-2008-12-02_04-01-57.job >>> org.apache.nutch.crawl.Injector crawl/crawldb dmoz >>> bin/hadoop jar nutch-2008-12-02_04-01-57.job >>> org.apache.nutch.crawl.Generator crawl/crawldb crawl/segments >>> >>> However, when I try to run the fetcher using the command: >>> >>> bin/hadoop jar nutch-2008-12-02_04-01-57.job >>> org.apache.nutch.fetcher.Fetcher crawl/segments/20090104094558 >>> >>> I get the following error: >>> >>> 09/01/04 10:20:31 INFO fetcher.Fetcher: Fetcher: starting >>> 09/01/04 10:20:31 INFO fetcher.Fetcher: Fetcher: segment: >>> crawl/segments/20090104094558 >>> ****calling init JobTracker***** >>> java.lang.NoSuchMethodError: >>> org.apache.nutch.fetcher.Fetcher$InputFormat.listPaths(Lorg/apache/hadoop/mapred/JobConf;)[Lorg/apache/hadoop/fs/Path; >>> >>> at >>> org.apache.nutch.fetcher.Fetcher$InputFormat.getSplits(Fetcher.java:61) >>> at >>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:783) >>> at >>> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1128) >>> at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:530) >>> at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:565) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:537) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown >>> Source) >>> at java.lang.reflect.Method.invoke(Unknown Source) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:165) >>> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) >>> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) >>> >>> Note: The subdirectory "20090104094558" was created by the generator. >>> >>> I'm running the 0.9 release of nutch downloaded from: >>> http://mirrors.24-7-solutions.net/pub/apache/lucene/nutch/ >>> >>> Does anyone know what is going on? >>> >>> Thanks in advance, >>> >>> Shirley >> >> From nutch-usermartinezgary@example.org Tue Jan 06 09:30:54 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 89006 invoked from network); 6 Jan 2009 09:30:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 Jan 2009 09:30:54 -0000 Received: (qmail 50735 invoked by uid 500); 6 Jan 2009 09:30:47 -0000 Delivered-To: martinezgary@example.org Received: (qmail 50690 invoked by uid 500); 6 Jan 2009 09:30:47 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 50678 invoked by uid 99); 6 Jan 2009 09:30:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2009 01:30:47 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO LEERNUG302.Union.lizearle.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2009 09:30:38 +0000 Received: from le21.Union.lizearle.com ([83.161.120.155]) by LEERNUG302.Union.lizearle.com with Microsoft SMTPSVC406-234-2140); Tue, 6 Jan 2009 09:30:17 +0000 Content-class: urn:content-classes:message Subject: RE: store 'content' field in the index MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 6 Jan 2009 09:30:17 -0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 Message-ID: <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: store 'content' field in the index Thread-Index: AclvPbueStZkcAHbQAih+swpNEyzAwAou2Eg References: <martinezgary@example.org> <martinezgary@example.org> From: "Matt Pearson" <martinezgary@example.org> To: <martinezgary@example.org> X-OriginalArrivalTime: 06 Jan 2009 09:30:17.0563 (UTC) FILETIME=[62F9C2B0:01C96FE1] X-Virus-Checked: Checked by ClamAV on apache.org Hmmm, the only occurances of Field.Store I can find are Field.Store.YES. The 'content' Field does not appear to be explicitly set in the way that 'boost' and 'digest' are. I guess this question should really be asked on nutch-dev... Thanks Matt -----Original Message----- From: Ian.huang [mailto:martinezgary@example.org]=20 Sent: 05 January 2009 13:58 To: martinezgary@example.org Subject: Re: store 'content' field in the index I think you need to change the basicindexfilter Field content=3Dnew Field("content", parse.getText(), Field.Store.NO,=20 Field.Index.TOKENIZED); change the Field.Store.NO to Field.Store.YES Ian -------------------------------------------------- From: "Matt Pearson" <martinezgary@example.org> Sent: Monday, January 05, 2009 1:55 PM To: <martinezgary@example.org> Subject: store 'content' field in the index > Hi Everyone, > > > > Does anyone know of a way I can configure the Nutch crawler to store the > contents of a document in the generated index? > > > > Thanks > > > > > > Matt > > > > > > Matt Pearson > > > >=20 From nutch-usermartinezgary@example.org Tue Jan 06 17:33:27 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 32821 invoked from network); 6 Jan 2009 17:33:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 Jan 2009 17:33:27 -0000 Received: (qmail 57774 invoked by uid 500); 6 Jan 2009 17:33:21 -0000 Delivered-To: martinezgary@example.org Received: (qmail 57734 invoked by uid 500); 6 Jan 2009 17:33:20 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Delivered-To: moderator for martinezgary@example.org Received: (qmail 65883 invoked by uid 99); 6 Jan 2009 13:42:53 -0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,MISSING_MID,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) From: "VishalS" <martinezgary@example.org> To: <martinezgary@example.org> Subject: Search performance for large indexes (>100M docs) Date: Tue, 6 Jan 2009 19:11:49 +0530 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_010A_01C97032.9FFF1560" X-Mailer: Microsoft Office Outlook, Build 11.0.5510 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3350 Thread-Index: AclwBIXwIZBfDzPbRsGucZAwnHl2Sw== X-Virus-Checked: Checked by ClamAV on apache.org Message-Id: <martinezgary@example.org> ------=_NextPart_000_010A_01C97032.9FFF1560 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Hi, I am experimenting with a system with around 120 million documents. The index is split into sub-indices of ~10M documents - each such index is being searched by a single machine. The results are being aggregated using the DistributedSearcher client. I am seeing a lot of performance issues with the system - most of the times, the response times are >4 seconds, and in some cases it goes upto a minute. It would be wonderful to know if there are ways to optimize what I am doing, or if there is something obvious that I am doing wrong. Here's what I have tried so far, and the issues I see: 1. Each search server is a 64-bit Pentium machine with ~7GB RAM and 4 CPUs running Linux. However, the searcher is not able to use more than 1 GB of RAM even though I have set -Xmx to ~3.5GB. I am guessing this is a Lucene issue. Is there a way we can have the searcher use more RAM to speed things up? 2. The total size of the index directory on each machine is ~70-100 GB. The prx file is 50GB, the fnm and frq files are ~27GB each and the fdt file is around 3GB. Is this too big? 3. I have tried analyzing my documents for commonly occurring terms in various fields, and added these terms to common-terms.utf8. There are ~10K terms in this file for me now. I am hoping this will help me speed up any phrase queries I am doing internally (although there is a cost attached in terms of the number of unique terms in the Lucene index, the total index size has increased by ~10-15%, which I guess is ok.) 4. There are around 8 fields that are searched in for each of the words in the query. Also, a phrase query containing all the words is fired in each of these fields as well. This means that for a 3 word input query, the number of sub-queries in my Lucene query are 24(3*8) term queries and 8(1*8) 3-word phrase queries. Is this too long or too expensive? 5. I have noticed that the slowest running queries (it takes upto a minute sometimes) are many times the ones that have one or more common words. 6. Each individual searcher has a single Lucene indexlet. Would it be faster to have more than 1 indexlet on the machine? 7. I am using a tomcat 6.0 installation out-of-the-box, with some minor changes in the number of threads, the java stack size allocation. If there's anyone else who has had experience working with large indices, I would love to get in touch and exchange notes. Regards, -Vishal. ------=_NextPart_000_010A_01C97032.9FFF1560-- From nutch-usermartinezgary@example.org Tue Jan 06 17:33:43 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 33096 invoked from network); 6 Jan 2009 17:33:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 Jan 2009 17:33:43 -0000 Received: (qmail 59489 invoked by uid 500); 6 Jan 2009 17:33:39 -0000 Delivered-To: martinezgary@example.org Received: (qmail 59461 invoked by uid 500); 6 Jan 2009 17:33:39 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Delivered-To: moderator for martinezgary@example.org Received: (qmail 66392 invoked by uid 99); 6 Jan 2009 13:43:57 -0000 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=MISSING_MID,NORMAL_HTTP_TO_IP,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) From: "VishalS" <martinezgary@example.org> To: <martinezgary@example.org> Subject: RE: error when doing search queries Date: Tue, 6 Jan 2009 19:12:54 +0530 MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook, Build 11.0.5510 In-Reply-To: <martinezgary@example.org> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3350 Thread-Index: Aclm0J/6T65uuVcwTI6AQNq0r3sOFgJM/ggw X-Virus-Checked: Checked by ClamAV on apache.org Message-Id: <martinezgary@example.org> Hi, Try replacing the "/include/header.html" to \"/include/header.html\" in search.jsp. This should do it. -vishal. -----Original Message----- From: Tony Wang [mailto:martinezgary@example.org] Sent: Friday, December 26, 2008 2:00 AM To: martinezgary@example.org Subject: error when doing search queries Finally, I got Nutch installed and working here http://83.161.120.155:8080, but when I submit a search query, the error message is: org.apache.jasper.JasperException: /search.jsp(172,22) Attribute value language + "/include/header.html" is quoted with " which must be escaped when used within the value. I replaced the outer double quote " with ' in search.jsp file, but the error persists. I use Tomcat 6.0.18 and Nutch 0.9 by the way. Thanks! Tony -- Signature: Success is a journey that never ends. From nutch-usermartinezgary@example.org Tue Jan 06 17:34:07 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 33560 invoked from network); 6 Jan 2009 17:34:05 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 Jan 2009 17:34:05 -0000 Received: (qmail 60719 invoked by uid 500); 6 Jan 2009 17:34:01 -0000 Delivered-To: martinezgary@example.org Received: (qmail 60687 invoked by uid 500); 6 Jan 2009 17:34:01 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Delivered-To: moderator for martinezgary@example.org Received: (qmail 78697 invoked by uid 99); 6 Jan 2009 15:23:36 -0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type; bh=fjv3qYik1AbbyDRiURFYHA0fYauMtGGMXJ++bInTWxM=; b=I13SrssdrXPTEVM4CSO0rpYwPUx2uSE7i0ol7C3WqP/Lg0Kljr6UPAtaCKUkSlKyvM 9qX1mCYRaHvSe2tw4zFSraMIblYmYA53IeSE0XvpqWNYB4EUBhxbDUvmxXE5/Tmp8PiL ZOy4Rjpb6gUu0geTStP8sYSP98JjTvKTxRWso= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type; b=FTCVx71tbLrF700pFfabdhwFBA64ObJ704R7KvVJf2PAuR2+TQRPDCxR05kQITcyfl 6udwBzNhPeUZiJkL3oX0VX1RzX1B/nsCPltqa+eO+Qs/t1KxyXJ8uO3e4ziYRSE5ELa7 7m12cH64ow1tfn/dWrFikZDeQac0FCGOFVkXo= Message-ID: <martinezgary@example.org> Date: Tue, 6 Jan 2009 20:53:10 +0530 From: "srinivasarao v" <martinezgary@example.org> To: martinezgary@example.org Subject: Indexing problem MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_43807_17806200.1231255390076" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_43807_17806200.1231255390076 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Hi all, I am trying to crawl a website with depth 5. But while indexing, it is throwing an error "java.io.FileNotFoundException: /tmp/hadoop/mapred/local/index/_-1541990873/_17.tis (Too many open files)". Can anyone reply me with the solution? Thanks Srinivas ------=_Part_43807_17806200.1231255390076-- From nutch-usermartinezgary@example.org Tue Jan 06 17:41:10 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 38919 invoked from network); 6 Jan 2009 17:41:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 6 Jan 2009 17:41:10 -0000 Received: (qmail 68868 invoked by uid 500); 6 Jan 2009 17:41:08 -0000 Delivered-To: martinezgary@example.org Received: (qmail 68835 invoked by uid 500); 6 Jan 2009 17:41:08 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 68824 invoked by uid 99); 6 Jan 2009 17:41:08 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2009 09:41:08 -0800 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO s01.igfoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2009 17:40:58 +0000 Received: from localhost (localhost [83.161.120.155]) by s01.igfoo.com (Postfix) with ESMTP id DC644F341CC for <martinezgary@example.org>; Tue, 6 Jan 2009 11:40:36 -0600 (CST) X-DKIM: Sendmail DKIM Filter v2.5.4 s01.igfoo.com DC644F341CC DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=igfoo.com; s=mail; t=1231263636; bh=y8sWaMrr6bSZ2rPVk+jKaJ5MHlocQwA/JfCacfz9zpM=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=P3dhYQ8wGU79 Nd8G5yWsPxmvGqKn1lUMdXhoF3JglZh7XOa5kxFizOwsh5VfsQA7mYIWA86wCcThuVw eWGA3ngoYj94a1tc7FJ7r+A8DCqxXhcEm27lQ2+exHaKNCr/VdnLiWET7IXQXCdSjn1 A3j/YT+M+2jZRpOk/e5eO9yFY= X-Virus-Scanned: Debian martinezgary@example.org Received: from s01.igfoo.com ([83.161.120.155]) by localhost (s01.igfoo.com [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id u23jmfXtHwrN for <martinezgary@example.org>; Tue, 6 Jan 2009 11:40:35 -0600 (CST) Received: from [83.161.120.155] (unknown [83.161.120.155]) by s01.igfoo.com (Postfix) with ESMTPSA id 5470DF341C0 for <martinezgary@example.org>; Tue, 6 Jan 2009 11:40:35 -0600 (CST) X-DKIM: Sendmail DKIM Filter v2.5.4 s01.igfoo.com 5470DF341C0 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=igfoo.com; s=mail; t=1231263635; bh=y8sWaMrr6bSZ2rPVk+jKaJ5MHlocQwA/JfCacfz9zpM=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=Hb8SR+fPZYoC vsl4Sh3/GKoF6GpPXeHeJEgm3J+idOlslhalHDJ95UhIzM8hktMwHySgSnD4NFNyY87 sib+jrbKxgHgAw8yeROFf6qQzObHVVxyiy1WJYyR1kaK8iR9AWZvc9Os55dXbSkBIwZ megD3uMvG1TZsMGtkuDJKZ5+o= Message-ID: <martinezgary@example.org> Date: Tue, 06 Jan 2009 11:40:34 -0600 From: Dennis Kubes <martinezgary@example.org> User-Agent: Thunderbird 83.161.120.155 (X11/20081125) MIME-Version: 1.0 To: martinezgary@example.org Subject: Re: Search performance for large indexes (>100M docs) References: <martinezgary@example.org> In-Reply-To: <martinezgary@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Take a look on the mailing lists for keeping the indexes in memory. When you get to the sizes you are talking about, the way you get subsecond response times is by: 1) Keeping the indexes in RAM 2) Agressive caching Dennis VishalS wrote: > Hi, > > > > I am experimenting with a system with around 120 million documents. The > index is split into sub-indices of ~10M documents - each such index is being > searched by a single machine. The results are being aggregated using the > DistributedSearcher client. I am seeing a lot of performance issues with the > system - most of the times, the response times are >4 seconds, and in some > cases it goes upto a minute. > > > > It would be wonderful to know if there are ways to optimize what I am > doing, or if there is something obvious that I am doing wrong. Here's what I > have tried so far, and the issues I see: > > > > 1. Each search server is a 64-bit Pentium machine with ~7GB RAM and 4 > CPUs running Linux. However, the searcher is not able to use more than 1 GB > of RAM even though I have set -Xmx to ~3.5GB. I am guessing this is a Lucene > issue. Is there a way we can have the searcher use more RAM to speed things > up? > 2. The total size of the index directory on each machine is ~70-100 GB. > The prx file is 50GB, the fnm and frq files are ~27GB each and the fdt file > is around 3GB. Is this too big? > 3. I have tried analyzing my documents for commonly occurring terms in > various fields, and added these terms to common-terms.utf8. There are ~10K > terms in this file for me now. I am hoping this will help me speed up any > phrase queries I am doing internally (although there is a cost attached in > terms of the number of unique terms in the Lucene index, the total index > size has increased by ~10-15%, which I guess is ok.) > 4. There are around 8 fields that are searched in for each of the words > in the query. Also, a phrase query containing all the words is fired in each > of these fields as well. This means that for a 3 word input query, the > number of sub-queries in my Lucene query are 24(3*8) term queries and 8(1*8) > 3-word phrase queries. Is this too long or too expensive? > 5. I have noticed that the slowest running queries (it takes upto a > minute sometimes) are many times the ones that have one or more common > words. > 6. Each individual searcher has a single Lucene indexlet. Would it be > faster to have more than 1 indexlet on the machine? > 7. I am using a tomcat 6.0 installation out-of-the-box, with some minor > changes in the number of threads, the java stack size allocation. > > > > If there's anyone else who has had experience working with large indices, I > would love to get in touch and exchange notes. > > > > Regards, > > > > -Vishal. > > From nutch-usermartinezgary@example.org Wed Jan 07 00:31:04 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 11290 invoked from network); 7 Jan 2009 00:31:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Jan 2009 00:31:04 -0000 Received: (qmail 31959 invoked by uid 500); 7 Jan 2009 00:30:57 -0000 Delivered-To: martinezgary@example.org Received: (qmail 31920 invoked by uid 500); 7 Jan 2009 00:30:57 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 31909 invoked by uid 99); 7 Jan 2009 00:30:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2009 16:30:57 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ew0-f21.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jan 2009 00:30:49 +0000 Received: by ewy14 with SMTP id 14so8995447ewy.5 for <martinezgary@example.org>; Tue, 06 Jan 2009 16:30:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=eYWFJh4D8wDgD8fjOh/owcB1t8p+azYI0Th5udQ64T0=; b=uoXyz2aH+SYCXxL6le2GFkdwzEaY0z5EFqEWQDHviU+3GuGETxOSPI6AKExsfSRa4Y lYoLCeoAzMlfpmRSkKx25Skg+K0VSmWFQmyjDAUUcTjvhrqsXZa++iMfAgqddYqWp0IC hHvMATH2/gPpwXuDSWlIVcQq9Ul9//P3FxLNE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=tnBP2IWFXwGg2/3IKzkmaTxtO2PD70ccQyBXQK9dBfcKm+y0jhMaH9Am1L5OkRZI2u NlUwmIHOyK1ZrEEfTCz3OrXFVLrO/CIyv63mdLCWcJvQmjL4YuGo5oQnk1IADkT7exUB kidSf5BBObLfvH6yS9pi9NiuDBkkcSup+70qE= Received: by 83.161.120.155 with SMTP id q10mr15582501ebq.179.1231288229055; Tue, 06 Jan 2009 16:30:29 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 6 Jan 2009 16:30:29 -0800 (PST) Message-ID: <martinezgary@example.org> Date: Wed, 7 Jan 2009 01:30:29 +0100 From: "Laurent Laborde" <martinezgary@example.org> To: martinezgary@example.org Subject: Re: Indexing problem In-Reply-To: <martinezgary@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <martinezgary@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Sure... you have too many open file :) you can increase it using ulimit. check the limit with ulimit -a if i remember correctly the command for open file is "ulimit -n" Default is usually 1024. On Tue, Jan 6, 2009 at 4:23 PM, srinivasarao v <martinezgary@example.org> wrote: > Hi all, > I am trying to crawl a website with depth 5. But while indexing, > it is throwing an error "java.io.FileNotFoundException: > /tmp/hadoop/mapred/local/index/_-1541990873/_17.tis (Too many open files)". > Can anyone reply me with the solution? Sure... you have too many open file :) you can increase it using ulimit. check the limit with ulimit -a if i remember correctly the command for max open file is "ulimit -n" Default is usually 1024and not enough for hadoop, depending of the configuration. -- F4FQM Kerunix Flan Laurent Laborde From nutch-usermartinezgary@example.org Wed Jan 07 00:33:11 2009 Return-Path: <nutch-usermartinezgary@example.org> Delivered-To: martinezgary@example.org Received: (qmail 11808 invoked from network); 7 Jan 2009 00:33:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 7 Jan 2009 00:33:11 -0000 Received: (qmail 33918 invoked by uid 500); 7 Jan 2009 00:33:04 -0000 Delivered-To: martinezgary@example.org Received: (qmail 33889 invoked by uid 500); 7 Jan 2009 00:33:04 -0000 Mailing-List: contact martinezgary@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:martinezgary@example.org> List-Unsubscribe: <mailto:martinezgary@example.org> List-Post: <mailto:martinezgary@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: martinezgary@example.org Delivered-To: mailing list martinezgary@example.org Received: (qmail 33877 invoked by uid 99); 7 Jan 2009 00:33:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2009 16:33:04 -0800 X-ASF-Spam-Status: No, hits=3.7 required=10.0 tests=DNS_FROM_OPENWHOIS,FORGED_HOTMAIL_RCVD2,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of martinezgary@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Jan 2009 00:32:55 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <martinezgary@example.org>) id 1LKMLa-0006nn-F8 for martinezgary@example.org; Tue, 06 Jan 2009 16:32:34 -0800 Message-ID: <martinezgary@example.org> Date: Tue, 6 Jan 2009 16:32:34 -0800 (PST) From: ianwong <martinezgary@example.org> To: martinezgary@example.org Subject: Re: Suggestions for
{'source': 'Apache Software Foundation Public Mail Archives'}