From nutch-usersimpsonnathan@example.org Thu Mar 01 17:37:56 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 5127 invoked from network); 1 Mar 2007 17:37:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2007 17:37:54 -0000 Received: (qmail 31437 invoked by uid 500); 1 Mar 2007 17:37:59 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 31408 invoked by uid 500); 1 Mar 2007 17:37:59 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 31397 invoked by uid 99); 1 Mar 2007 17:37:59 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2007 09:37:59 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO talk.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2007 09:37:49 -0800 Received: from [83.161.120.155] (helo=jubjub.nabble.com) by talk.nabble.com with esmtp (Exim 4.50) id 1HMpDc-0005ui-JT for simpsonnathan@example.org; Thu, 01 Mar 2007 09:37:28 -0800 Message-ID: <simpsonnathan@example.org> Date: Thu, 1 Mar 2007 09:37:28 -0800 (PST) From: =?UTF-8?Q?R=C3=BCdiger_Schulz_=28SkyGate=29?= <simpsonnathan@example.org> To: simpsonnathan@example.org Subject: Memory leakduring crawlr? MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Nabble-From: simpsonnathan@example.org X-Virus-Checked: Checked by ClamAV on apache.org Hello, I'm currently using Nutch to crawl a small selection of websites. All together, there are about 60.000 HTML pages and about 10.000 PDFs, which shall all go into the index. I'm using a custom crawl class, which repeats the crawl iterations not based on a given number (depth) but instead repeating until there are no more unfetched pages (quite similar to a Pytho= n script posted some weeks ago). This is ok, as most of these don't change that often, so I could do such a long initial crawl, and because of URL filtering etc. I know it's going to be a definite number of pages. When I run this over the weekend :) I always get a OutOfMemoryException (permgen space) after about 400 iterations, during which about 40.000 pages get indexed. Increasing permgen space for the JVM only increased the time until I get the error. So I ran my Crawl feeding it with only a couple of URLs so that it's done after about 30 minutes, and did some profiling with JDK6' jconsole, jmap an= d jhat. I discovered in jconsole that after each iteration permgen increases about 1MB as more classes are loaded. I'm comparing a jmap dump after about 5 iterations, and another after about 30 iterations, and discover the following: * permgen usage has almost doubled * none of my custom plugins have more than 1 instance * looking for nutch classes, I see that org.apache.nutch.plugin.Extension and org.apache.nutch.plugin.PluginDescriptor have both gone from about 500 to about 3000 instances, org.apache.nutch.plugin.ExtensionPoint from 1500 t= o 1000 instances. * looking further, I see that org.apache.nutch.plugin.PluginRepository and org.apache.hadoop.mapred.JobConf have both increased from 16 to 99 instances. Now I'm wondering if that behavior is intended. It seems there are 3 or 4 instances of JobConf created during each loop (in Generator.generate, in Fetcher.fetch, in ParseSegment.parse and in CrawlDb.update). Is that really necessary? I pass my JobConf in the constructor already. What could I do to run the crawl fully? Increase permgen even more or is there anything else? Thanks for reading, R=C3=BCdiger --=20 View this message in context: http://www.nabble.com/Memory-leakduring-crawl= r--tf3328411.html#a9254396 Sent from the Nutch - User mailing list simpsonnathan@example.org. From nutch-usersimpsonnathan@example.org Thu Mar 01 18:09:37 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 15855 invoked from network); 1 Mar 2007 18:09:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2007 18:09:35 -0000 Received: (qmail 77024 invoked by uid 500); 1 Mar 2007 18:09:41 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 77003 invoked by uid 500); 1 Mar 2007 18:09:41 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 76992 invoked by uid 99); 1 Mar 2007 18:09:41 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2007 10:09:41 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wx-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2007 10:09:29 -0800 Received: by wx-out-0506.google.com with SMTP id i29so515517wxd for <simpsonnathan@example.org>; Thu, 01 Mar 2007 10:09:08 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:subject:content-type:content-transfer-encoding; b=sco8MIKUGH2R/02cnsUtV4FQ0yxBXBSn26v9cpY34GQEw9xo/omweeZlbxFp/L32OlCc0Owkn6T18hZUtu+89sud03e1WlsxnfRwe4LqftaGYhR2ne1MkTAbh9lXCWILU+AKdYWOSNu7/n+Z1ljjUrpVaGt4gdQqcD7GSDhoacU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:user-agent:mime-version:to:subject:content-type:content-transfer-encoding; b=BYcYeY6W0W2AW2iWMNl3QXx3+sWxBWEr4iKn/4DQbNX3Hki8Qz05TaPoKa+gProPjfZwxC8dAsRdPlNffbZc8MvHNOev5tHKdHeqXOnCYw0qhAAnZO4XR1Gt7QSEtDSMAPU2qTyBX3iVePWGkt3+TNS9/XzqEhWzjB8ClttWvrM= Received: by 83.161.120.155 with SMTP id n16mr2064045agb.1172772548194; Thu, 01 Mar 2007 10:09:08 -0800 (PST) Received: from ?83.161.120.155? ( [83.161.120.155]) by mx.google.com with ESMTP id 20sm1950838agd.283.161.120.155.09.05; Thu, 01 Mar 2007 10:09:06 -0800 (PST) Message-ID: <simpsonnathan@example.org> Date: Thu, 01 Mar 2007 12:09:02 -0600 From: =?ISO-8859-1?Q?=22Ricardo_J=2E_M=E9ndez=22?= <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (Macintosh/20061207) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Behavior of nutch-site.xml vs. hadoop-site.xml Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Hi, I'm using nutch-0.9, from the trunk. I've noticed a behavior difference on a plugin unit test if I set the plugin.folders property on nutch-site.xml vs. hadoop-site.xml. If I set it on nutch-site.xml, the unit test works well, but an error is raised if it's on hadoop-site.xml The error is: [junit] WARN [main] (ParserFactory.java:196) - Canno initialize parser parse-html (cause: org.apache.nutch.plugin.PluginRuntimeException: java.lang.ClassNotFoundException: org.apache.nutch.parse.html.HtmlParser Is there a reason why the HtmlParser wouldn't be loaded when the directory is specified on hadoop-site.xml? Thanks in advance, Ricardo J. Méndez http://ricardo.strangevistas.net/ From nutch-usersimpsonnathan@example.org Thu Mar 01 22:26:35 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 14580 invoked from network); 1 Mar 2007 22:26:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2007 22:26:34 -0000 Received: (qmail 1124 invoked by uid 500); 1 Mar 2007 22:26:43 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 360 invoked by uid 500); 1 Mar 2007 22:26:41 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 344 invoked by uid 99); 1 Mar 2007 22:26:41 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2007 14:26:41 -0800 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO an-out-0708.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2007 14:26:29 -0800 Received: by an-out-0708.google.com with SMTP id c3so446021ana for <simpsonnathan@example.org>; Thu, 01 Mar 2007 14:26:09 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=hdWMdrZnHVmYi5PIc8hRqGkCNQMD0goBmjaZLunKVUydvXZ3gij7taQBU82eDNlqalROTICkijf78dab1SgX5iiuj9QKnXCilF97ea3vFlqr5L9X78KAI5sERtCU/T6JntilWDQzkyOp2k1M8L9BJ/wTWn2CnoQdp1jqi0EWfFo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=YHoWqn6pgtVmC8Ypj2glGV6LPpIodAq+W5PkCWwLc56YLNk+eUU71Q2ZxzlEHlWDLRtf1cYRMiF80ob2xad0sr7aU/9ScYdQCambwiznSmIZjDKVdEg2PeiAkNB0Czr7rgrFIiEa5Hc9oUC7g8ljum8pMkRpCj/u/trcYOa4jKM= Received: by 83.161.120.155 with SMTP id e6mr17914and.1172787969080; Thu, 01 Mar 2007 14:26:09 -0800 (PST) Received: by 83.161.120.155 with HTTP; Thu, 1 Mar 2007 14:26:08 -0800 (PST) Message-ID: <simpsonnathan@example.org> Date: Fri, 2 Mar 2007 00:26:08 +0200 From: "Gal Nitzan" <simpsonnathan@example.org> To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml In-Reply-To: <simpsonnathan@example.org> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_17302_9716711.1172787968361" References: <simpsonnathan@example.org> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_17302_9716711.1172787968361 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi, Nutch loads its configuration from nutch-site and nutch-default.xml and not from hadoop conf files so the behavior is correct. HTH, Gal. On 3/1/07, "Ricardo J. M=E9ndez" <simpsonnathan@example.org> wrote: > > Hi, > > I'm using nutch-0.9, from the trunk. I've noticed a behavior > difference on a plugin unit test if I set the plugin.folders property on > nutch-site.xml vs. hadoop-site.xml. If I set it on nutch-site.xml, the > unit test works well, but an error is raised if it's on hadoop-site.xml > > The error is: > > [junit] WARN [main] (ParserFactory.java:196) - Canno initialize > parser parse-html (cause: > org.apache.nutch.plugin.PluginRuntimeException: > java.lang.ClassNotFoundException: org.apache.nutch.parse.html.HtmlParser > > > Is there a reason why the HtmlParser wouldn't be loaded when the > directory is specified on hadoop-site.xml? > > Thanks in advance, > > > > > Ricardo J. M=E9ndez > http://ricardo.strangevistas.net/ > ------=_Part_17302_9716711.1172787968361-- From nutch-usersimpsonnathan@example.org Thu Mar 01 23:57:12 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 53127 invoked from network); 1 Mar 2007 23:57:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2007 23:57:11 -0000 Received: (qmail 31875 invoked by uid 500); 1 Mar 2007 23:57:18 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 31726 invoked by uid 500); 1 Mar 2007 23:57:17 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 31715 invoked by uid 99); 1 Mar 2007 23:57:17 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2007 15:57:17 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wx-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 01 Mar 2007 15:57:04 -0800 Received: by wx-out-0506.google.com with SMTP id i29so596551wxd for <simpsonnathan@example.org>; Thu, 01 Mar 2007 15:56:43 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; b=I/z5LRoSP98QqmGo/jwb2yoPUtD8ET0YANnhyWtEZ1mb7c7rASy1TAXwnXbesxN+TKko2XtcpImNLd40GsOXCoPA74NCgcxcgemJm1iyPvDOXlMTIIRQH0r8WSs5ybvTCJMdlhAsgI+IvX553kXtihJHlhWeZ5Kzy8ncvsbPnh8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; b=KxlX2+Xp0BZ8cm/XwcbRmT5sMJAY09upOAC6F23x3u/gcg/4YqSycn7GK9cWlYbQe+fjgRoklGguPCgyZ9Eze71wyP/aieJmeLeycFMJDkSymU9f9T/rK0e0zmS7HItUm9lBLPb/6zqkz2kgD1eZTRWey117izkZILmhH1Cy+Fs= Received: by 83.161.120.155 with SMTP id b14mr2494199agc.1172793403779; Thu, 01 Mar 2007 15:56:43 -0800 (PST) Received: from ?83.161.120.155? ( [83.161.120.155]) by mx.google.com with ESMTP id 5sm2224931agc.283.161.120.155.56.42; Thu, 01 Mar 2007 15:56:43 -0800 (PST) Message-ID: <simpsonnathan@example.org> Date: Thu, 01 Mar 2007 17:56:40 -0600 From: =?ISO-8859-1?Q?=22Ricardo_J=2E_M=E9ndez=22?= <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (Macintosh/20061207) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml References: <simpsonnathan@example.org> <simpsonnathan@example.org> In-Reply-To: <simpsonnathan@example.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Hi Gal, Thanks for the reply. What has me wondering is that several other plugins _are_ being loaded when I define it on hadoop-site.xml, and actually that defining plugin.folders on that file is the only way I've found so far of getting plugins loaded at all when testing from Eclipse. Moreover, I get this problem even if I define it in both nutch-site and hadoop-site, which would make it seem that the definition in hadoop-site.xml does have an effect. I was assuming they overrode the options from nutch-site.xml - am I mistaken? Ricardo J. Méndez http://ricardo.strangevistas.net/ Gal Nitzan wrote: > Hi, > > Nutch loads its configuration from nutch-site and nutch-default.xml and not > from hadoop conf files so the behavior is correct. > > HTH, > > Gal. > > > On 3/1/07, "Ricardo J. Méndez" <simpsonnathan@example.org> wrote: >> >> Hi, >> >> I'm using nutch-0.9, from the trunk. I've noticed a behavior >> difference on a plugin unit test if I set the plugin.folders property on >> nutch-site.xml vs. hadoop-site.xml. If I set it on nutch-site.xml, the >> unit test works well, but an error is raised if it's on hadoop-site.xml >> >> The error is: >> >> [junit] WARN [main] (ParserFactory.java:196) - Canno initialize >> parser parse-html (cause: >> org.apache.nutch.plugin.PluginRuntimeException: >> java.lang.ClassNotFoundException: org.apache.nutch.parse.html.HtmlParser >> >> >> Is there a reason why the HtmlParser wouldn't be loaded when the >> directory is specified on hadoop-site.xml? >> >> Thanks in advance, >> >> >> >> >> Ricardo J. Méndez >> http://ricardo.strangevistas.net/ >> > From nutch-usersimpsonnathan@example.org Fri Mar 02 13:28:27 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 91535 invoked from network); 2 Mar 2007 13:28:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 13:28:26 -0000 Received: (qmail 56266 invoked by uid 500); 2 Mar 2007 13:28:23 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 56126 invoked by uid 500); 2 Mar 2007 13:28:22 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 56084 invoked by uid 99); 2 Mar 2007 13:28:22 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 05:28:22 -0800 X-ASF-Spam-Status: No, hits=0.9 required=10.0 tests=FORGED_YAHOO_RCVD,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO talk.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 05:28:07 -0800 Received: from [83.161.120.155] (helo=jubjub.nabble.com) by talk.nabble.com with esmtp (Exim 4.50) id 1HN7nW-0002e8-MD for simpsonnathan@example.org; Fri, 02 Mar 2007 05:27:46 -0800 Message-ID: <simpsonnathan@example.org> Date: Fri, 2 Mar 2007 05:27:46 -0800 (PST) From: Munir <simpsonnathan@example.org> To: simpsonnathan@example.org Subject: Arabic language in Nutch MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: simpsonnathan@example.org X-Virus-Checked: Checked by ClamAV on apache.org Can you please tell me if it is possible to use NGramProfile to create arabic profile? if it is ok how? because I tried to run this command but I got error: java org.apache.nutch.analysis.lang.NGramProfile -create <ar> <arabic> <windows-1256> error : syntax error near unexpected token `<' and how I will create ar.ngp file? Please help me to use the Analysis for Arabic. can you tell me the steps one by one? I am using Nutch0.9dev on lunix with tomcat5.5.20. thanks in advance -- View this message in context: http://www.nabble.com/Arabic-language-in-Nutch-tf3333598.html#a9269533 Sent from the Nutch - User mailing list simpsonnathan@example.org. From nutch-usersimpsonnathan@example.org Fri Mar 02 13:43:46 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 99073 invoked from network); 2 Mar 2007 13:43:44 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 13:43:44 -0000 Received: (qmail 82016 invoked by uid 500); 2 Mar 2007 13:43:42 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 81988 invoked by uid 500); 2 Mar 2007 13:43:42 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 81959 invoked by uid 99); 2 Mar 2007 13:43:42 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 05:43:42 -0800 X-ASF-Spam-Status: No, hits=1.4 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (herse.apache.org: 83.161.120.155 is neither permitted nor denied by domain of simpsonnathan@example.org) Received: from [83.161.120.155] (HELO server.handydns.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 05:43:29 -0800 Received: from [83.161.120.155] (helo=[83.161.120.155]) by server.handydns.com with esmtpa (Exim 4.63) (envelope-from <simpsonnathan@example.org>) id 1HN82N-0007qS-6s for simpsonnathan@example.org; Fri, 02 Mar 2007 06:43:07 -0700 Message-ID: <simpsonnathan@example.org> Date: Fri, 02 Mar 2007 15:43:04 +0200 From: Enis Soztutar <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (X11/20070103) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Re: Arabic language in Nutch References: <simpsonnathan@example.org> In-Reply-To: <simpsonnathan@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server.handydns.com X-AntiAbuse: Original Domain - lucene.apache.org X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [47 12] X-AntiAbuse: Sender Address Domain - gmail.com X-Source: X-Source-Args: X-Source-Dir: X-Virus-Checked: Checked by ClamAV on apache.org Munir wrote: > Can you please tell me if it is possible to use NGramProfile to create > arabic profile? if it is ok how? because I tried to run this command > but I got error: > java org.apache.nutch.analysis.lang.NGramProfile -create <ar> <arabic> > <windows-1256> > error : syntax error near unexpected token `<' > > and how I will create ar.ngp file? > > Please help me to use the Analysis for Arabic. can you tell me the > steps one by one? > I am using Nutch0.9dev on lunix with tomcat5.5.20. > thanks in advance > > Yes you can, Since Ngram profile uses java.lang.String and that is UTF-16, you can create ngram profile for arabic (I suppose arabic character set is represented in utf16). You should not use '<' and '>' but instead give the command as : java org.apache.nutch.analysis.lang.NGramProfile -create ar arabic windows-1256 linux uses < and > for redirecting standard input and output. From nutch-usersimpsonnathan@example.org Fri Mar 02 15:07:07 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 37776 invoked from network); 2 Mar 2007 15:07:06 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 15:07:06 -0000 Received: (qmail 61919 invoked by uid 500); 2 Mar 2007 15:07:04 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 61881 invoked by uid 500); 2 Mar 2007 15:07:04 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 61861 invoked by uid 99); 2 Mar 2007 15:07:03 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 07:07:03 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO linuxfly.dragonflymc.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 07:06:52 -0800 Received: from [83.161.120.155] (bigfly.visvo.com [83.161.120.155]) by linuxfly.dragonflymc.com (Postfix) with ESMTP id 10016C20027 for <simpsonnathan@example.org>; Fri, 2 Mar 2007 09:07:47 -0600 (CST) Message-ID: <simpsonnathan@example.org> Date: Fri, 02 Mar 2007 09:06:29 -0600 From: Dennis Kubes <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20061207) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> In-Reply-To: <simpsonnathan@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Here goes with the short answer. :) Configuration has two levels, default and final. It is supplied by the org.apache.hadoop.conf.Configuration class and extended in Nutch by the org.apache.nutch.util.NutchConfiguration class. Although it is configurable, by default hadoop-default.xml and nutch-default.xml are default resources and hadoop-site.xml and nutch-site.xml are final resources. Resources (i.e. resource files) can be added by filename to either the default or final resource set and in fact this is how Nutch extends the Configuration class, by adding nutch-default.xml and nutch-site.xml. Final resource values overwrite default resource values and final resource values added later will overwrite final resource values added earlier. When I say values I am talking about the individual properties not the resource files. Resource files are found by name in the classpath with the HADOOP_CONF_DIR or NUTCH_CONF_DIR being configured in the nutch and hadoop scripts as the first setting in the classpath. You can change the conf dir to pull configuration files from different directories and many tools in nutch and hadoop now provide a -conf options on the command line to set the conf directory. So for your example if you define the property in hadoop-default.xml or nutch-default.xml and it is not defined in either hadoop-site.xml or nutch-site.xml then the property will stand. If you define the property in either nutch-site.xml or hadoop-site.xml then it will override nutch-default.xml and hadoop-default.xml settings. And if you define it in both hadoop-site.xml and nutch-site.xml then the nutch-site.xml will override the hadoop-site.xml settings because nutch-site.xml is added after hadoop-site.xml. And remember only individual properties are overridden not the entire file. Practically you should define properties having to do with Hadoop (i.e. the DFS, Mapreduce, etc) in the hadoop-site.xml and properties having to do with Nutch (i.e. fetcher, url-normalizers, etc) in the nutch-site.xml. Dennis Kubes Ricardo J. Méndez wrote: > Hi Gal, > > Thanks for the reply. > > What has me wondering is that several other plugins _are_ being loaded > when I define it on hadoop-site.xml, and actually that defining > plugin.folders on that file is the only way I've found so far of getting > plugins loaded at all when testing from Eclipse. > > Moreover, I get this problem even if I define it in both nutch-site and > hadoop-site, which would make it seem that the definition in > hadoop-site.xml does have an effect. I was assuming they overrode the > options from nutch-site.xml - am I mistaken? > > > Ricardo J. Méndez > http://ricardo.strangevistas.net/ > > Gal Nitzan wrote: >> Hi, >> >> Nutch loads its configuration from nutch-site and nutch-default.xml and not >> from hadoop conf files so the behavior is correct. >> >> HTH, >> >> Gal. >> >> >> On 3/1/07, "Ricardo J. Méndez" <simpsonnathan@example.org> wrote: >>> Hi, >>> >>> I'm using nutch-0.9, from the trunk. I've noticed a behavior >>> difference on a plugin unit test if I set the plugin.folders property on >>> nutch-site.xml vs. hadoop-site.xml. If I set it on nutch-site.xml, the >>> unit test works well, but an error is raised if it's on hadoop-site.xml >>> >>> The error is: >>> >>> [junit] WARN [main] (ParserFactory.java:196) - Canno initialize >>> parser parse-html (cause: >>> org.apache.nutch.plugin.PluginRuntimeException: >>> java.lang.ClassNotFoundException: org.apache.nutch.parse.html.HtmlParser >>> >>> >>> Is there a reason why the HtmlParser wouldn't be loaded when the >>> directory is specified on hadoop-site.xml? >>> >>> Thanks in advance, >>> >>> >>> >>> >>> Ricardo J. Méndez >>> http://ricardo.strangevistas.net/ >>> From nutch-usersimpsonnathan@example.org Fri Mar 02 15:22:30 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 42187 invoked from network); 2 Mar 2007 15:22:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 15:22:29 -0000 Received: (qmail 1711 invoked by uid 500); 2 Mar 2007 15:22:34 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 1683 invoked by uid 500); 2 Mar 2007 15:22:34 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 1672 invoked by uid 99); 2 Mar 2007 15:22:34 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 07:22:34 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 07:22:23 -0800 Received: by nf-out-0910.google.com with SMTP id i2so1223231nfe for <simpsonnathan@example.org>; Fri, 02 Mar 2007 07:22:02 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=fH8mWSPqV1Jvg2ZnnEyjM8PoYPE+XTC5JbUlq3R615Tdi0R8k/TR9EUIcn8fUiiS+pbDXBZ/2WWMk6yJedPUC4sY5QJ5NJEWDTXLJOoSSg07hnPa5mu1fCUF4w2dmSfy6+ZbTsxeWOm5Q/DAfjOuJtl8MBkPoSSW+GsWQ2SOTxk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=O6CreSrcheOdHR6NFWetiZudINT+dBCfuZgcKsRnl5o3pciKJylzrlF0C/j6J1OnEHTETs47VJdT52FdjHsmGVhSXBMoTD82a87SKkszgI+UNvjQuwNTp4amX54cAHiF1LyMnQio/bjR68XXLYpcOb4ngeZx4Mu1h5PuTKk8iXQ= Received: by 83.161.120.155 with SMTP id e11mr7892956nfj.1172848921169; Fri, 02 Mar 2007 07:22:01 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Mar 2007 07:22:00 -0800 (PST) Message-ID: <simpsonnathan@example.org> Date: Fri, 2 Mar 2007 17:22:00 +0200 From: "=?UTF-8?Q?Do=C4=9Facan_G=C3=BCney?=" <simpsonnathan@example.org> To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml In-Reply-To: <simpsonnathan@example.org> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_56803_28177513.1172848920853" References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_56803_28177513.1172848920853 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 Content-Disposition: inline SGksCgpPbiAzLzIvMDcsICJSaWNhcmRvIEouIE3DqW5kZXoiIDxtZW5kZXpzdGVyQGdtYWlsLmNv bT4gd3JvdGU6Cj4gSGkgR2FsLAo+Cj4gVGhhbmtzIGZvciB0aGUgcmVwbHkuCj4KPiBXaGF0IGhh cyBtZSB3b25kZXJpbmcgaXMgdGhhdCBzZXZlcmFsIG90aGVyIHBsdWdpbnMgX2FyZV8gYmVpbmcg bG9hZGVkCj4gd2hlbiBJIGRlZmluZSBpdCBvbiBoYWRvb3Atc2l0ZS54bWwsIGFuZCBhY3R1YWxs eSB0aGF0IGRlZmluaW5nCj4gcGx1Z2luLmZvbGRlcnMgb24gdGhhdCBmaWxlIGlzIHRoZSBvbmx5 IHdheSBJJ3ZlIGZvdW5kIHNvIGZhciBvZiBnZXR0aW5nCj4gcGx1Z2lucyBsb2FkZWQgYXQgYWxs IHdoZW4gdGVzdGluZyBmcm9tIEVjbGlwc2UuCgpZb3UgY2FuIGxvYWQgdGhlIHBsdWdpbnMgaW4g ZWNsaXBzZSB3aXRoIG51dGNoLXNpdGUueG1sIHRvby4KCkhlcmUgaXMgaG93IEkgZG8gaXQ6CjEp IEFkZCAkTlVUQ0hfSE9NRS9jb25mIGRpcmVjdG9yeSB0byBlY2xpcHNlIHdpdGggTGluayBTb3Vy Y2Ugb3B0aW9uLgoyKSBSdW4gYW50IHRvIGNvbXBpbGUgbnV0Y2gKMykgQWRkIHRoaXMgdG8geW91 ciBudXRjaC1zaXRlLnhtbDoKPHByb3BlcnR5PgogIDxuYW1lPnBsdWdpbi5mb2xkZXJzPC9uYW1l PgogIDx2YWx1ZT5wbHVnaW5zLDxhYnNvbHV0ZV9wYXRoX3RvX251dGNoX3BsdWdpbnM+PC92YWx1 ZT4KPC9wcm9wZXJ0eT4KCjxhYnNvbHV0ZV9wYXRoX3RvX251dGNoX3BsdWdpbnM+IGlzIGdlbmVy YWxseSAkTlVUQ0hfSE9NRS9idWlsZC9wbHVnaW5zLgoKTm90ZSB0aGF0IHRoZXJlIGlzIGEgc21h bGwgYnVnIGluIHRoZSBjb2RlIHRoYXQgcmVhZHMgdGhpcyBvcHRpb24uIElmCmFuIGFic29sdXRl IGRpcmVjdG9yeSBpcyBhZGRlZCB0byB0aGlzIG9wdGlvbiBsaWtlIHRoaXMgYnV0IGl0CmFjdHVh bGx5IGRvZXNuJ3QgZXhpc3QgTnV0Y2ggZmFpbHMgd2l0aCBhbiBOUEUuIEkga2VlcCBmb3JnZXR0 aW5nIHRoYXQKSSBzZXQgdGhpcyBvcHRpb24gd2hlbiBJIGFtIHNlbmRpbmcgYSBqb2IgYW5kIHNp bmNlIEkgZG9uJ3QgaGF2ZSB0aGVzZQpmaWxlcyBvbiBNUiBtYWNoaW5lcyB0aGUgam9iIGZhaWxz LiBZb3UgY2FuIHRyeSB0aGUgYXR0YWNoZWQgcGF0Y2ggdG8KYXZvaWQgdGhhdC4KCltzbmlwXQoK LS0gCkRvxJ9hY2FuIEfDvG5leQo= ------=_Part_56803_28177513.1172848920853 Content-Type: text/x-patch; name=check_if_plugin_folder_is_directory.patch; charset=ANSI_X3.4-1968 Content-Transfer-Encoding: base64 X-Attachment-Id: f_eysrx9df Content-Disposition: attachment; filename="check_if_plugin_folder_is_directory.patch" SW5kZXg6IHNyYy9qYXZhL29yZy9hcGFjaGUvbnV0Y2gvcGx1Z2luL1BsdWdpbk1hbmlmZXN0UGFy c2VyLmphdmEKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PQotLS0gc3JjL2phdmEvb3JnL2FwYWNoZS9udXRjaC9wbHVnaW4v UGx1Z2luTWFuaWZlc3RQYXJzZXIuamF2YQkocmV2aXNpb24gMTMyNikKKysrIHNyYy9qYXZhL29y Zy9hcGFjaGUvbnV0Y2gvcGx1Z2luL1BsdWdpbk1hbmlmZXN0UGFyc2VyLmphdmEJKHdvcmtpbmcg Y29weSkKQEAgLTEzMyw2ICsxMzMsOCBAQAogICAgICAgfSBjYXRjaCAoVW5zdXBwb3J0ZWRFbmNv ZGluZ0V4Y2VwdGlvbiBlKSB7CiAgICAgICB9CiAgICAgICBkaXJlY3RvcnkgPSBuZXcgRmlsZShw YXRoKTsKKyAgICB9IGVsc2UgaWYgKCFkaXJlY3RvcnkuZXhpc3RzKCkgfHwgIWRpcmVjdG9yeS5p c0RpcmVjdG9yeSgpKSB7CisgICAgICByZXR1cm4gbnVsbDsKICAgICB9CiAgICAgcmV0dXJuIGRp cmVjdG9yeTsKICAgfQo= ------=_Part_56803_28177513.1172848920853-- From nutch-usersimpsonnathan@example.org Fri Mar 02 16:00:37 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 59799 invoked from network); 2 Mar 2007 16:00:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 16:00:36 -0000 Received: (qmail 408 invoked by uid 500); 2 Mar 2007 16:00:29 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 386 invoked by uid 500); 2 Mar 2007 16:00:29 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 359 invoked by uid 99); 2 Mar 2007 16:00:29 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 08:00:29 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ik-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 08:00:18 -0800 Received: by ik-out-1112.google.com with SMTP id b35so573184ika for <simpsonnathan@example.org>; Fri, 02 Mar 2007 07:59:57 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=SY4St9X4hPGFuHDoIXvqSxGT7Woq8NSvNbZ+9FRRfYXQEPxzETHHJhXqNSBZsF0ZUGKcJgoiqQnAJHSxSsPeggwoiZ4rUv3oN1hHdLND5DROBHAl8KkwwcOsu0ZUDvVJ4mnwTUxyk9lwlJmc7iiZeAKq4Tzk+k928OuA8Lj4c1o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=ZSqPtmGDksVovEd3ZdFk8I3B2ODjrF2oHge+bAemIg5Uf3h6Paq/e4gUg/y7++g1h+BC335Dk5Sii5QU5FqYBEgHJK+tYeh2FAg0f9p9qhaob2EahPDA74xFHE8XoADNL5DtPfQp4Z5dzxwfnJvpApb2gCNRC3K8NAVvwJv/dr0= Received: by 83.161.120.155 with SMTP id z1mr269366waz.1172851196060; Fri, 02 Mar 2007 07:59:56 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Mar 2007 07:59:56 -0800 (PST) Message-ID: <simpsonnathan@example.org> Date: Fri, 2 Mar 2007 16:59:56 +0100 From: rubdabadub <simpsonnathan@example.org> To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml In-Reply-To: <simpsonnathan@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> X-Virus-Checked: Checked by ClamAV on apache.org Super thanks! Nice explanation. I finally got it :-) I mean how things loads and why! Thank you! I do have one question though, however its a bit different. But if you do have time a lengthy answer is always welcome :-) Question: When the content is parsed by -- let say parse-html or in order to parse meta data for example.. src/java/org/apache/nutch/parseHTMLMetaTags.java Now when I run the ParserChecker.java main method.. I don't see the extracted data parsed the way it shows in parseHTMLMetaTags.. I see only content, outlink, title etc.. no meta tag.. How is that happen .. cos I am trying my best to read the code but I can't go beyond parse.. I started at crawl :-) I don't want to hi jack the thread i just thought you answered the question so clearly.. Regards On 3/2/07, Dennis Kubes <simpsonnathan@example.org> wrote: > Here goes with the short answer. :) > > Configuration has two levels, default and final. It is supplied by the > org.apache.hadoop.conf.Configuration class and extended in Nutch by the > org.apache.nutch.util.NutchConfiguration class. > > Although it is configurable, by default hadoop-default.xml and > nutch-default.xml are default resources and hadoop-site.xml and > nutch-site.xml are final resources. Resources (i.e. resource files) can > be added by filename to either the default or final resource set and in > fact this is how Nutch extends the Configuration class, by adding > nutch-default.xml and nutch-site.xml. > > Final resource values overwrite default resource values and final > resource values added later will overwrite final resource values added > earlier. When I say values I am talking about the individual properties > not the resource files. Resource files are found by name in the > classpath with the HADOOP_CONF_DIR or NUTCH_CONF_DIR being configured in > the nutch and hadoop scripts as the first setting in the classpath. You > can change the conf dir to pull configuration files from different > directories and many tools in nutch and hadoop now provide a -conf > options on the command line to set the conf directory. > > So for your example if you define the property in hadoop-default.xml or > nutch-default.xml and it is not defined in either hadoop-site.xml or > nutch-site.xml then the property will stand. If you define the property > in either nutch-site.xml or hadoop-site.xml then it will override > nutch-default.xml and hadoop-default.xml settings. And if you define it > in both hadoop-site.xml and nutch-site.xml then the nutch-site.xml will > override the hadoop-site.xml settings because nutch-site.xml is added > after hadoop-site.xml. And remember only individual properties are > overridden not the entire file. > > Practically you should define properties having to do with Hadoop (i.e. > the DFS, Mapreduce, etc) in the hadoop-site.xml and properties having to > do with Nutch (i.e. fetcher, url-normalizers, etc) in the nutch-site.xml. > > Dennis Kubes > > Ricardo J. M=E9ndez wrote: > > Hi Gal, > > > > Thanks for the reply. > > > > What has me wondering is that several other plugins _are_ being loaded > > when I define it on hadoop-site.xml, and actually that defining > > plugin.folders on that file is the only way I've found so far of gettin= g > > plugins loaded at all when testing from Eclipse. > > > > Moreover, I get this problem even if I define it in both nutch-site and > > hadoop-site, which would make it seem that the definition in > > hadoop-site.xml does have an effect. I was assuming they overrode the > > options from nutch-site.xml - am I mistaken? > > > > > > Ricardo J. M=E9ndez > > http://ricardo.strangevistas.net/ > > > > Gal Nitzan wrote: > >> Hi, > >> > >> Nutch loads its configuration from nutch-site and nutch-default.xml an= d not > >> from hadoop conf files so the behavior is correct. > >> > >> HTH, > >> > >> Gal. > >> > >> > >> On 3/1/07, "Ricardo J. M=E9ndez" <simpsonnathan@example.org> wrote: > >>> Hi, > >>> > >>> I'm using nutch-0.9, from the trunk. I've noticed a behavior > >>> difference on a plugin unit test if I set the plugin.folders property= on > >>> nutch-site.xml vs. hadoop-site.xml. If I set it on nutch-site.xml, t= he > >>> unit test works well, but an error is raised if it's on hadoop-site.x= ml > >>> > >>> The error is: > >>> > >>> [junit] WARN [main] (ParserFactory.java:196) - Canno initialize > >>> parser parse-html (cause: > >>> org.apache.nutch.plugin.PluginRuntimeException: > >>> java.lang.ClassNotFoundException: org.apache.nutch.parse.html.HtmlPar= ser > >>> > >>> > >>> Is there a reason why the HtmlParser wouldn't be loaded when the > >>> directory is specified on hadoop-site.xml? > >>> > >>> Thanks in advance, > >>> > >>> > >>> > >>> > >>> Ricardo J. M=E9ndez > >>> http://ricardo.strangevistas.net/ > >>> > From nutch-usersimpsonnathan@example.org Fri Mar 02 16:17:32 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 68974 invoked from network); 2 Mar 2007 16:17:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 16:17:31 -0000 Received: (qmail 62559 invoked by uid 500); 2 Mar 2007 16:17:36 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 62540 invoked by uid 500); 2 Mar 2007 16:17:36 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 62529 invoked by uid 99); 2 Mar 2007 16:17:36 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 08:17:36 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO getopt.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 08:17:24 -0800 Received: from [83.161.120.155] (75-mo3-2.acn.waw.pl [83.161.120.155]) (authenticated) by getopt.org (8.11.6/8.11.6) with ESMTP id l22GHCU21430 for <simpsonnathan@example.org>; Fri, 2 Mar 2007 10:17:12 -0600 Message-ID: <simpsonnathan@example.org> Date: Fri, 02 Mar 2007 17:17:04 +0100 From: Andrzej Bialecki <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20070221) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> In-Reply-To: <simpsonnathan@example.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Dennis Kubes wrote: > Practically you should define properties having to do with Hadoop > (i.e. the DFS, Mapreduce, etc) in the hadoop-site.xml and properties > having to do with Nutch (i.e. fetcher, url-normalizers, etc) in the > nutch-site.xml. There are other two important config files: * mapred-default.xml - this is loaded as default resource when a new map-reduce JobConf is created - which means that it is loaded as the last default resource when you prepare the job configuration. Usually you should keep its content to a bare minimum. This is the best place to specify the default number of map and reduce tasks per job. If you feel adventurous you could also put some other stuff there, e.g. set the default compression with mapred.compress.map.output and so on. * job.xml - this file is created dynamically, and represents a serialized JobConf. When map-reduce tasks are started they read this file as their last default resource (note - this is NOT a final resource!). So, if you accidentally distributed mapred-default.xml to all cluster nodes, but in your job you specified a different number of map or reduce tasks, your settings will take precedence. The same with other settings, such as e.g. the compression setting. HOWEVER ... a common error is to put too many properties such as default number of map and reduce tasks in hadoop-site.xml. As Dennis explained, this is a final resource - which means that the values you specify there will ALWAYS override your job settings. This is bad, so don't do it ;) - put them in mapred-default.xml. In other words: use hadoop-site.xml only for things that are always the same for the whole cluster, such as the FS name, jobtracker name:port, temp directories, etc. because values you put there will always override your job settings. And do not put there things that are job-dependent. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: simpsonnathan@example.org From nutch-usersimpsonnathan@example.org Fri Mar 02 16:56:09 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 81893 invoked from network); 2 Mar 2007 16:56:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 16:56:08 -0000 Received: (qmail 43144 invoked by uid 500); 2 Mar 2007 16:56:10 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 43125 invoked by uid 500); 2 Mar 2007 16:56:10 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 43114 invoked by uid 99); 2 Mar 2007 16:56:10 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 08:56:10 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO linuxfly.dragonflymc.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 08:55:59 -0800 Received: from [83.161.120.155] (bigfly.visvo.com [83.161.120.155]) by linuxfly.dragonflymc.com (Postfix) with ESMTP id 61D94C20027 for <simpsonnathan@example.org>; Fri, 2 Mar 2007 10:56:54 -0600 (CST) Message-ID: <simpsonnathan@example.org> Date: Fri, 02 Mar 2007 10:55:37 -0600 From: Dennis Kubes <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20061207) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> In-Reply-To: <simpsonnathan@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Believe it or not I don't think that meta tags are currently stored. I looked through the html parsing code and didn't see anywhere that it could be storing it except in html filters. I see that meta tags are parsed and passed to the html filters but I didn't see any default filter that was storing them. If there isn't a reason why we shouldn't be storing meta tags, if we aren't currently storing them (I could be missing where this is happening :) ), and this is something that people want then I can create an html filter that will store the meta-tags in the Parse MetaData. Dennis Kubes rubdabadub wrote: > Super thanks! Nice explanation. I finally got it :-) I mean how things > loads and why! Thank you! I do have one question though, however its a > bit different. But if you do have time a lengthy answer is always > welcome :-) > > Question: > When the content is parsed by -- let say parse-html or in order to > parse meta data for example.. > src/java/org/apache/nutch/parseHTMLMetaTags.java > > Now when I run the ParserChecker.java main method.. I don't see the > extracted data parsed the way it shows in parseHTMLMetaTags.. I see > only content, outlink, title etc.. no meta tag.. How is that happen .. > cos I am trying my best to read the code but I can't go beyond parse.. > I started at crawl :-) After looking through it > > I don't want to hi jack the thread i just thought you answered the > question so clearly.. > Regards > > On 3/2/07, Dennis Kubes <simpsonnathan@example.org> wrote: >> Here goes with the short answer. :) >> >> Configuration has two levels, default and final. It is supplied by the >> org.apache.hadoop.conf.Configuration class and extended in Nutch by the >> org.apache.nutch.util.NutchConfiguration class. >> >> Although it is configurable, by default hadoop-default.xml and >> nutch-default.xml are default resources and hadoop-site.xml and >> nutch-site.xml are final resources. Resources (i.e. resource files) can >> be added by filename to either the default or final resource set and in >> fact this is how Nutch extends the Configuration class, by adding >> nutch-default.xml and nutch-site.xml. >> >> Final resource values overwrite default resource values and final >> resource values added later will overwrite final resource values added >> earlier. When I say values I am talking about the individual properties >> not the resource files. Resource files are found by name in the >> classpath with the HADOOP_CONF_DIR or NUTCH_CONF_DIR being configured in >> the nutch and hadoop scripts as the first setting in the classpath. You >> can change the conf dir to pull configuration files from different >> directories and many tools in nutch and hadoop now provide a -conf >> options on the command line to set the conf directory. >> >> So for your example if you define the property in hadoop-default.xml or >> nutch-default.xml and it is not defined in either hadoop-site.xml or >> nutch-site.xml then the property will stand. If you define the property >> in either nutch-site.xml or hadoop-site.xml then it will override >> nutch-default.xml and hadoop-default.xml settings. And if you define it >> in both hadoop-site.xml and nutch-site.xml then the nutch-site.xml will >> override the hadoop-site.xml settings because nutch-site.xml is added >> after hadoop-site.xml. And remember only individual properties are >> overridden not the entire file. >> >> Practically you should define properties having to do with Hadoop (i.e. >> the DFS, Mapreduce, etc) in the hadoop-site.xml and properties having to >> do with Nutch (i.e. fetcher, url-normalizers, etc) in the nutch-site.xml. >> >> Dennis Kubes >> >> Ricardo J. Méndez wrote: >> > Hi Gal, >> > >> > Thanks for the reply. >> > >> > What has me wondering is that several other plugins _are_ being loaded >> > when I define it on hadoop-site.xml, and actually that defining >> > plugin.folders on that file is the only way I've found so far of >> getting >> > plugins loaded at all when testing from Eclipse. >> > >> > Moreover, I get this problem even if I define it in both nutch-site and >> > hadoop-site, which would make it seem that the definition in >> > hadoop-site.xml does have an effect. I was assuming they overrode the >> > options from nutch-site.xml - am I mistaken? >> > >> > >> > Ricardo J. Méndez >> > http://ricardo.strangevistas.net/ >> > >> > Gal Nitzan wrote: >> >> Hi, >> >> >> >> Nutch loads its configuration from nutch-site and nutch-default.xml >> and not >> >> from hadoop conf files so the behavior is correct. >> >> >> >> HTH, >> >> >> >> Gal. >> >> >> >> >> >> On 3/1/07, "Ricardo J. Méndez" <simpsonnathan@example.org> wrote: >> >>> Hi, >> >>> >> >>> I'm using nutch-0.9, from the trunk. I've noticed a behavior >> >>> difference on a plugin unit test if I set the plugin.folders >> property on >> >>> nutch-site.xml vs. hadoop-site.xml. If I set it on >> nutch-site.xml, the >> >>> unit test works well, but an error is raised if it's on >> hadoop-site.xml >> >>> >> >>> The error is: >> >>> >> >>> [junit] WARN [main] (ParserFactory.java:196) - Canno initialize >> >>> parser parse-html (cause: >> >>> org.apache.nutch.plugin.PluginRuntimeException: >> >>> java.lang.ClassNotFoundException: >> org.apache.nutch.parse.html.HtmlParser >> >>> >> >>> >> >>> Is there a reason why the HtmlParser wouldn't be loaded when the >> >>> directory is specified on hadoop-site.xml? >> >>> >> >>> Thanks in advance, >> >>> >> >>> >> >>> >> >>> >> >>> Ricardo J. Méndez >> >>> http://ricardo.strangevistas.net/ >> >>> >> From nutch-usersimpsonnathan@example.org Fri Mar 02 17:07:40 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 85826 invoked from network); 2 Mar 2007 17:07:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 17:07:39 -0000 Received: (qmail 59393 invoked by uid 500); 2 Mar 2007 17:07:44 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 59367 invoked by uid 500); 2 Mar 2007 17:07:44 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 59356 invoked by uid 99); 2 Mar 2007 17:07:44 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 09:07:44 -0800 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [83.161.120.155] (HELO getopt.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 09:07:32 -0800 Received: from [83.161.120.155] (75-mo3-2.acn.waw.pl [83.161.120.155]) (authenticated) by getopt.org (8.11.6/8.11.6) with ESMTP id l22H7Mv08712 for <simpsonnathan@example.org>; Fri, 2 Mar 2007 11:07:22 -0600 Message-ID: <simpsonnathan@example.org> Date: Fri, 02 Mar 2007 18:07:13 +0100 From: Andrzej Bialecki <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20070221) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> In-Reply-To: <simpsonnathan@example.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Dennis Kubes wrote: > Believe it or not I don't think that meta tags are currently stored. > I looked through the html parsing code and didn't see anywhere that it > could be storing it except in html filters. I see that meta tags are > parsed and passed to the html filters but I didn't see any default > filter that was storing them. > > If there isn't a reason why we shouldn't be storing meta tags, if we > aren't currently storing them (I could be missing where this is > happening :) ), and this is something that people want then I can > create an html filter that will store the meta-tags in the Parse > MetaData. The reason is simple - space. Storing additional data consumes space, and if someone just occasionally needs this info from one or two pages it's less costly to re-parse the page again. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: simpsonnathan@example.org From nutch-usersimpsonnathan@example.org Fri Mar 02 17:48:24 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 97910 invoked from network); 2 Mar 2007 17:48:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 17:48:23 -0000 Received: (qmail 30899 invoked by uid 500); 2 Mar 2007 17:48:29 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 30868 invoked by uid 500); 2 Mar 2007 17:48:29 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 30857 invoked by uid 99); 2 Mar 2007 17:48:29 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 09:48:29 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ik-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 09:48:17 -0800 Received: by ik-out-1112.google.com with SMTP id b35so602248ika for <simpsonnathan@example.org>; Fri, 02 Mar 2007 09:47:56 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=o0aK0ceaHCzau75A+SVRL7gmqRfOX4tk/WSE56u2cYBKiUGNLZGdypu49IP+PmpM0Qkp2yVOm/zBdToYeKBe68DzXBnMlpSGjAjWYQFDMLLHg1RVHTa1eJav4oA6PgMzaweFEQtpnPqdXThqFL9YBELe/qndLPvcNzVYJGYrg6g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=A1xxK2HvQhrcMd6JNo2/fo0V99P2hjwJKi7zYEdk7SRNTND4hcd5Qr+C5tLfS004XRKlkWorNFwx4eReB78yzE2fmbcXy0XkP2ODlBa0s6Gv9dX3J7edFjkBpBxC488nB5ilJe/cf1k6FvNcFqb9/RO9MqvVag2a/pP8zw4Dkbk= Received: by 83.161.120.155 with SMTP id v15mr336983wae.1172857675778; Fri, 02 Mar 2007 09:47:55 -0800 (PST) Received: by 83.161.120.155 with HTTP; Fri, 2 Mar 2007 09:47:55 -0800 (PST) Message-ID: <simpsonnathan@example.org> Date: Fri, 2 Mar 2007 18:47:55 +0100 From: rubdabadub <simpsonnathan@example.org> To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml In-Reply-To: <simpsonnathan@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> X-Virus-Checked: Checked by ClamAV on apache.org On 3/2/07, Andrzej Bialecki <simpsonnathan@example.org> wrote: > Dennis Kubes wrote: > > Believe it or not I don't think that meta tags are currently stored. > > I looked through the html parsing code and didn't see anywhere that it > > could be storing it except in html filters. I see that meta tags are > > parsed and passed to the html filters but I didn't see any default > > filter that was storing them. > > > > If there isn't a reason why we shouldn't be storing meta tags, if we > > aren't currently storing them (I could be missing where this is > > happening :) ), and this is something that people want then I can > > create an html filter that will store the meta-tags in the Parse > > MetaData. Yes!! Please that would be nice. Maybe we can do metatag-parse, metatag-index metatag-query?? no?? This way those who want this can turn it on as a plugin?? no?? > The reason is simple - space. Storing additional data consumes space, > and if someone just occasionally needs this info from one or two pages > it's less costly to re-parse the page again. Oh I see. Now I understand. But I wonder what is the MetaData parser doing really? is it being used anywhere in the crawl-index life cycle at all? Just wondering... > -- > Best regards, > Andrzej Bialecki <>< > ___. ___ ___ ___ _ _ __________________________________ > [__ || __|__/|__||\/| Information Retrieval, Semantic Web > ___|||__|| \| || | Embedded Unix, System Integration > http://www.sigram.com Contact: simpsonnathan@example.org > > > From nutch-usersimpsonnathan@example.org Fri Mar 02 17:57:26 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 690 invoked from network); 2 Mar 2007 17:57:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 17:57:25 -0000 Received: (qmail 42212 invoked by uid 500); 2 Mar 2007 17:57:33 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 41506 invoked by uid 500); 2 Mar 2007 17:57:31 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 41495 invoked by uid 99); 2 Mar 2007 17:57:31 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 09:57:31 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wx-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 09:57:18 -0800 Received: by wx-out-0506.google.com with SMTP id i29so851453wxd for <simpsonnathan@example.org>; Fri, 02 Mar 2007 09:56:58 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; b=siiXsgeDrwShZxIL2f/RFpExMj4YOPAOoDFNAFBz71SqOgv/BJhKBCSy/hEoIFFB4f75SdIQB0yd8yIKXiqDK0SOOIj60HTbMPccVGkqo/Jfepl6PYGoWgZR+ZsxTwpFb2O+1dlJeA2f142BXsbHYyJGjdvYKBba2ytx4zd8f5Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; b=kJ5xgoPkQ626DVczxqYt5szIDSts9OxTckphSRFiCZQ6JAa4MewdxthkJZ3XDJav+K8mW0pOGsC8ItNbXD9dodMevaiXKH0nC3ZH0qnw9Kh3ASQmqy+819ZWYnb+PRpYt7afiuoWj2ivmlpZdH6DHCk7v8Gsu246jB2dtdxezWw= Received: by 83.161.120.155 with SMTP id f14mr1764895agf.1172858218381; Fri, 02 Mar 2007 09:56:58 -0800 (PST) Received: from ?83.161.120.155? ( [83.161.120.155]) by mx.google.com with ESMTP id 6sm3041826agb.283.161.120.155.56.57; Fri, 02 Mar 2007 09:56:57 -0800 (PST) Message-ID: <simpsonnathan@example.org> Date: Fri, 02 Mar 2007 11:56:56 -0600 From: =?UTF-8?B?IlJpY2FyZG8gSi4gTcOpbmRleiI=?= <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (Macintosh/20061207) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> In-Reply-To: <simpsonnathan@example.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Andrzej Bialecki wrote: > Dennis Kubes wrote: [snip] Dennis, Andrzej, many thanks for the very detailed description. I couldn't find a wiki page detailing this, so I'll probably create one later today with a mix of both your responses, referencing each separately. Please let me know if anyone has a suggestion for a starting point or preexisting page I've missed - the closest I've found is http://wiki.apache.org/nutch/OverviewDeploymentConfigs Ricardo J. Méndez http://ricardo.strangevistas.net/ From nutch-usersimpsonnathan@example.org Fri Mar 02 17:57:50 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 842 invoked from network); 2 Mar 2007 17:57:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 17:57:49 -0000 Received: (qmail 42620 invoked by uid 500); 2 Mar 2007 17:57:55 -0000 Delivered-To: simpsonnathan@example.org Received: (qmail 42593 invoked by uid 500); 2 Mar 2007 17:57:55 -0000 Mailing-List: contact simpsonnathan@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:simpsonnathan@example.org> List-Unsubscribe: <mailto:simpsonnathan@example.org> List-Post: <mailto:simpsonnathan@example.org> List-Id: <nutch-user.lucene.apache.org> Reply-To: simpsonnathan@example.org Delivered-To: mailing list simpsonnathan@example.org Received: (qmail 42582 invoked by uid 99); 2 Mar 2007 17:57:55 -0000 Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 09:57:55 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: domain of simpsonnathan@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wx-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Mar 2007 09:57:42 -0800 Received: by wx-out-0506.google.com with SMTP id i29so851550wxd for <simpsonnathan@example.org>; Fri, 02 Mar 2007 09:57:22 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; b=nKBvb2zS6NlqX0H9EtvhtVF7+n66ssKzu79ccvfOPjU9Qn1BFmKOa5MLuPotYNOP0B+nSVdbVUqs4rnDS/fnwTVKMyh48teCa6s5KcKG9hC2nYCYWkwP0y8N4u9GJkchGmLpUUBLGsD2C16KLXYqgte4fgEa7EIGKewp0oWXRnk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:user-agent:mime-version:to:subject:references:in-reply-to:content-type:content-transfer-encoding; b=J41hl5tWZynIWUcX2uFy+yfPe3wCvSGSVXP+bWJEABhpZxIxhl8lflAEkZ3fiAssypaENW0dfnbvDxhz9kYkzvqB9DN5hsmyS70bPdXeI9cDiKvLnkq9syPbTrVeLx7+6WrYalHHGbYetLrbHf3kozGRHoOfVPaCAgV5bp8wl7c= Received: by 83.161.120.155 with SMTP id x2mr3098111agb.1172858242237; Fri, 02 Mar 2007 09:57:22 -0800 (PST) Received: from ?83.161.120.155? ( [83.161.120.155]) by mx.google.com with ESMTP id 20sm3017275agb.283.161.120.155.57.21; Fri, 02 Mar 2007 09:57:21 -0800 (PST) Message-ID: <simpsonnathan@example.org> Date: Fri, 02 Mar 2007 11:57:20 -0600 From: =?UTF-8?B?IlJpY2FyZG8gSi4gTcOpbmRleiI=?= <simpsonnathan@example.org> User-Agent: Thunderbird 83.161.120.155 (Macintosh/20061207) MIME-Version: 1.0 To: simpsonnathan@example.org Subject: Re: Behavior of nutch-site.xml vs. hadoop-site.xml References: <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> <simpsonnathan@example.org> In-Reply-To: <simpsonnathan@example.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org Doğacan Güney wrote: > Hi, > > On 3/2/07, "Ricardo J. Méndez" <simpsonnathan@example.org> wrote: >> Hi Gal, >> >> Thanks for the reply. >> >> What has me wondering is that several other plugins _are_ being loaded >> when I define it on hadoop-site.xml, and actually that defining >> plugin.folders on that file is the only way I've found so far of getting >> plugins loaded at all when testing from Eclipse. > > You can load the plugins in eclipse with nutch-site.xml too. > > Here is how I do it: > 1) Add $NUTCH_HOME/conf directory to eclipse with Link Source option. > 2) Run ant to compile nutch > 3) Add this to your nutch-site.xml: > <property> > <name>plugin.folders</name> > <value>plugins,<absolute_path_to_nutch_plugins></value> > </property> > > <absolute_path_to_nutch_plugins> is generally $NUTCH_HOME/build/plugins. > > Note that there is a small bug in the code that reads this option. If > an absolute directory is added to this option like this but it > actually doesn't exist Nutch fails with an NPE. I keep forgetting that > I set this option when I am sending a job and since I don't have these > files on MR machines the job fails. You can try the attached patch to > avoid that. > > [snip] Thanks for both the patch and the explanation, Doğacan, I'll add this to the wiki. I also noticed that Eclipse was disregarding conf/nutch-site.xml in favor of src/test/nutch-site.xml, so I've added the configuration item there as well. Regards, Ricardo J. Méndez http://ricardo.strangevistas.net/ From nutch-usersimpsonnathan@example.org Fri Mar 02 18:01:24 2007 Return-Path: <nutch-usersimpsonnathan@example.org> Delivered-To: simpsonnathan@example.org Received: (qmail 1958 invoked from network); 2 Mar 2007 18:01:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2007 18:01:23 -0000
{'source': 'Apache Software Foundation Public Mail Archives'}