From nutch-userjulie68@example.com Mon Jun 01 02:18:46 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 79097 invoked from network); 1 Jun 2009 02:18:46 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 02:18:46 -0000 Received: (qmail 30859 invoked by uid 500); 1 Jun 2009 02:18:57 -0000 Delivered-To: julie68@example.com Received: (qmail 30774 invoked by uid 500); 1 Jun 2009 02:18:57 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 30764 invoked by uid 99); 1 Jun 2009 02:18:57 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 02:18:57 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_HELO_PASS,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mout.perfora.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 02:18:48 +0000 Received: from ca.test.dadi360.com (pool-173-56-128-42.nycmny.east.verizon.net [83.161.120.155]) by mrelay.perfora.net (node=mrus1) with ESMTP (Nemesis) id 0MKpCa-1MAx6W26dw-000cq6; Sun, 31 May 2009 22:18:25 -0400 Subject: Re: Nutch reindex cron From: kevin chen <julie68@example.com> Reply-To: julie68@example.com To: julie68@example.com In-Reply-To: <julie68@example.com> References: <julie68@example.com> Content-Type: text/plain Date: Sun, 31 May 2009 22:18:23 -0400 Message-Id: <1243822703.17403.1.camel@dadih01> Mime-Version: 1.0 X-Mailer: Evolution 2.0.2 (2.0.2-35.el4) Content-Transfer-Encoding: 7bit X-Provags-ID: V01U2FsdGVkX1/bCaYzArofi5S2BYp1czLtu5KKwoxjBLtTdcC fguqfWTScwkGtcrKpJ/LTS4XBPlxmwriJds8fdQ+AqwnHSrKwp cEqAkNlgxkQPqnJ1aXyStHettCJ/qlc X-Virus-Checked: Checked by ClamAV on apache.org You can touch web.xml (under WEB-INF/ ). On Sat, 2009-05-30 at 14:33 -0700, prb wrote: > Hi I have a service to reindex my intranet nightly. > I create a new index and copy the files and reboot tomcat. > This all works great from a terminal session but fails from from unless > I touch index files and reboot manually and then it works but I want to > automate this.. > Is there some trick to force tomcat to reload my new index from a from bash? > Maybe a java -jar one liner or a specific file I can update to trip tomcat ? > Thx > > From nutch-userjulie68@example.com Mon Jun 01 05:08:34 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 27453 invoked from network); 1 Jun 2009 05:08:33 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 05:08:33 -0000 Received: (qmail 806 invoked by uid 500); 1 Jun 2009 05:08:45 -0000 Delivered-To: julie68@example.com Received: (qmail 723 invoked by uid 500); 1 Jun 2009 05:08:44 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 713 invoked by uid 99); 1 Jun 2009 05:08:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 05:08:44 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f227.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 05:08:36 +0000 Received: by bwz27 with SMTP id 27so7761054bwz.5 for <julie68@example.com>; Sun, 31 May 2009 22:08:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type; bh=hQhiQdssBfxSH9sgO2UVbQmye6HGBJIfcRYVteXehGQ=; b=Eni35ImkjMLJz5AKXcRcQ1/bZobEGf6HTMSNKsIeqnEAutaNl+RKVhp4DpPrbH0ehk cih9JERC7wP/DP8dqiPxZPhy2VUw12jEspvfTjc49d2Fs3deVla/NAO77CJuB4/gI2SZ NOg6s4OAELH6fZsa5fmVEi2x53CYrF+fOspBo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=hL0wD3+aE5NM0e7n0K5VglcvDl5Tmsf9zr4SgKch+cD82iFYau0Va8qetVpYYdsiqu kdtHnOcAAlKCGY1+GaMtXGNVFRsSuNOby0YMixJKfXEd74PzhZduGXSoY0paJCKZ9Iee Kd/+DMwTvsT5nLwge3YQlBmr7Y027eRsYa9iw= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 1mr401835hby.160.1243832895060; Sun, 31 May 2009 22:08:15 -0700 (PDT) In-Reply-To: <1243822703.17403.1.camel@dadih01> References: <julie68@example.com> <1243822703.17403.1.camel@dadih01> From: Alexander Aristov <julie68@example.com> Date: Mon, 1 Jun 2009 09:07:54 +0400 Message-ID: <julie68@example.com> Subject: Re: Nutch reindex cron To: julie68@example.com, julie68@example.com Content-Type: multipart/alternative; boundary=001636457e16deba7e046b4269bb X-Virus-Checked: Checked by ClamAV on apache.org --001636457e16deba7e046b4269bb Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit I experienced problems with such approach. Tomcat hang after a few application re-deployments. And it seems it's a problem with Tomcat itself as I saw this issue in different environments and with different applications. Main problem is that Tomcat doesn't release memory and hence it fails with out-of-memory. Best Regards Alexander Aristov 2009/6/1 kevin chen <julie68@example.com> > > You can touch web.xml (under WEB-INF/ ). > > On Sat, 2009-05-30 at 14:33 -0700, prb wrote: > > Hi I have a service to reindex my intranet nightly. > > I create a new index and copy the files and reboot tomcat. > > This all works great from a terminal session but fails from from unless > > I touch index files and reboot manually and then it works but I want to > > automate this.. > > Is there some trick to force tomcat to reload my new index from a from > bash? > > Maybe a java -jar one liner or a specific file I can update to trip > tomcat ? > > Thx > > > > > > --001636457e16deba7e046b4269bb-- From nutch-userjulie68@example.com Mon Jun 01 05:15:40 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 28701 invoked from network); 1 Jun 2009 05:15:40 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 05:15:40 -0000 Received: (qmail 3415 invoked by uid 500); 1 Jun 2009 05:15:51 -0000 Delivered-To: julie68@example.com Received: (qmail 3353 invoked by uid 500); 1 Jun 2009 05:15:50 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 3343 invoked by uid 99); 1 Jun 2009 05:15:50 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 05:15:50 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO polaris.butterflycluster.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 05:15:40 +0000 Received: from localhost ([83.161.120.155]) by polaris.butterflycluster.net with esmtpa (Exim 4.68) (envelope-from <julie68@example.com>) id 1MAzrm-00046Y-4J for julie68@example.com; Mon, 01 Jun 2009 15:15:22 +1000 Received: from 83.161.120.155 ([83.161.120.155]) by www.butterflycluster.com (Horde MIME library) with HTTP; Mon, 01 Jun 2009 15:15:21 +1000 Message-ID: <julie68@example.com> Date: Mon, 01 Jun 2009 15:15:21 +1000 From: julie68@example.com To: julie68@example.com Subject: Re: Nutch reindex cron References: <julie68@example.com> <1243822703.17403.1.camel@dadih01> <julie68@example.com> In-Reply-To: <julie68@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable User-Agent: Internet Messaging Program (IMP) H3 (4.1.6) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - polaris.butterflycluster.net X-AntiAbuse: Original Domain - lucene.apache.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - butterflycluster.net X-Virus-Checked: Checked by ClamAV on apache.org So there is no way of reloading the index with out restarting tomcat? Quoting Alexander Aristov <julie68@example.com>: > I experienced problems with such approach. Tomcat hang after a few > application re-deployments. And it seems it's a problem with Tomcat itself > as I saw this issue in different environments and with different > applications. > > Main problem is that Tomcat doesn't release memory and hence it fails with > out-of-memory. > > Best Regards > Alexander Aristov > > > 2009/6/1 kevin chen <julie68@example.com> > >> >> You can touch web.xml (under WEB-INF/ ). >> >> On Sat, 2009-05-30 at 14:33 -0700, prb wrote: >> > Hi I have a service to reindex my intranet nightly. >> > I create a new index and copy the files and reboot tomcat. >> > This all works great from a terminal session but fails from from unless >> > I touch index files and reboot manually and then it works but I want to >> > automate this.. >> > Is there some trick to force tomcat to reload my new index from a from >> bash? >> > Maybe a java -jar one liner or a specific file I can update to trip >> tomcat ? >> > Thx >> > >> > >> >> > From nutch-userjulie68@example.com Mon Jun 01 05:27:05 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 35836 invoked from network); 1 Jun 2009 05:27:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 05:27:02 -0000 Received: (qmail 12471 invoked by uid 500); 1 Jun 2009 05:27:13 -0000 Delivered-To: julie68@example.com Received: (qmail 12397 invoked by uid 500); 1 Jun 2009 05:27:13 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 12387 invoked by uid 99); 1 Jun 2009 05:27:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 05:27:13 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO polaris.butterflycluster.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 05:27:03 +0000 Received: from localhost ([83.161.120.155]) by polaris.butterflycluster.net with esmtpa (Exim 4.68) (envelope-from <julie68@example.com>) id 1MB02n-0004WP-Qi for julie68@example.com; Mon, 01 Jun 2009 15:26:45 +1000 Received: from 83.161.120.155 ([83.161.120.155]) by www.butterflycluster.com (Horde MIME library) with HTTP; Mon, 01 Jun 2009 15:26:45 +1000 Message-ID: <julie68@example.com> Date: Mon, 01 Jun 2009 15:26:45 +1000 From: julie68@example.com To: julie68@example.com Subject: Re: Eclipse Nutch1.0 IOException References: <julie68@example.com> In-Reply-To: <julie68@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.1.6) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - polaris.butterflycluster.net X-AntiAbuse: Original Domain - lucene.apache.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - butterflycluster.net X-Virus-Checked: Checked by ClamAV on apache.org did you get it fixed? i had something similar a while ago - maybe it was different. but make sure you have the conf folder in your Eclipse project classpath; Quoting Georg Kirschner <julie68@example.com>: > Hi, > > I tried to hook up Eclipse and Nutch but got following error: > > "crawl started in: crawl > rootUrlDir = urls > threads = 10 > depth = 3 > topN = 50 > Injector: starting > Injector: crawlDb: crawl/crawldb > Injector: urlDir: urls > Injector: Converting injected urls to crawl db entries. > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) > at org.apache.nutch.crawl.Injector.inject(Injector.java:160) > at org.apache.nutch.crawl.Crawl.main(Crawl.java:113)" > > Can anyone give me a clue here? > > Nutch 1.0 > Hadoop 0.19.1 > Eclipse Ganymede > JDK 1.6.0_13 > > Thanks > > Georg > From nutch-userjulie68@example.com Mon Jun 01 06:16:05 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 60221 invoked from network); 1 Jun 2009 06:16:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 06:16:04 -0000 Received: (qmail 58289 invoked by uid 500); 1 Jun 2009 06:16:16 -0000 Delivered-To: julie68@example.com Received: (qmail 58258 invoked by uid 500); 1 Jun 2009 06:16:16 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 58248 invoked by uid 99); 1 Jun 2009 06:16:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 06:16:15 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f227.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 06:16:07 +0000 Received: by bwz27 with SMTP id 27so7779313bwz.5 for <julie68@example.com>; Sun, 31 May 2009 23:15:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type; bh=GTgV88evpl813IBrgJHvJZhyyHrvrNwZDZ4AmCwx4e8=; b=F9KbizQAJ8NUNj+KiKPfQIfw+Aqv5uScgSA1lLgokdohHHoQIfBn//9ISUCe3KB3y6 DJumQJO8/LSAZh9PKstSi0lRobAGxXv86S+bGEiIPMtGrnuRe4mofRNLQyD6gsbeqZVO EB9ejWvJtqb7pLgvHMMZs7Oy4Eko5DVMgaLdg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=kjW7Rth0Vd0W7xfBPvze31rTxpEaSVmw+Uz7LbCpSfLaIjRsATPVeynPFJ6cu9JDND Bf+js0vZ/82g/gPB7pB6KR0n7yQQHW29eF3b2FRt1FGBYAHesep18OXT/TiNlL11Mreb HtuzHPolh1T79HhMgG57mY4V3VotFsUzG2nqw= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id o16mr383941hbb.49.1243836946078; Sun, 31 May 2009 23:15:46 -0700 (PDT) In-Reply-To: <julie68@example.com> References: <julie68@example.com> <1243822703.17403.1.camel@dadih01> <julie68@example.com> <julie68@example.com> From: Alexander Aristov <julie68@example.com> Date: Mon, 1 Jun 2009 10:15:26 +0400 Message-ID: <julie68@example.com> Subject: Re: Nutch reindex cron To: julie68@example.com Content-Type: multipart/alternative; boundary=001485f85b0e545cc5046b435b5d X-Virus-Checked: Checked by ClamAV on apache.org --001485f85b0e545cc5046b435b5d Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit you can restart tomcat from bash. I found such issue in jira NUTCH-376 Add methods to control runtime behaviour of NutchBeanNot sure if the case is still valid but I had to implement some such mechanism for my own use. Solr might have something to work better with index. Best Regards Alexander Aristov 2009/6/1 <julie68@example.com> > > So there is no way of reloading the index with out restarting tomcat? > > > Quoting Alexander Aristov <julie68@example.com>: > > I experienced problems with such approach. Tomcat hang after a few >> application re-deployments. And it seems it's a problem with Tomcat itself >> as I saw this issue in different environments and with different >> applications. >> >> Main problem is that Tomcat doesn't release memory and hence it fails with >> out-of-memory. >> >> Best Regards >> Alexander Aristov >> >> >> 2009/6/1 kevin chen <julie68@example.com> >> >> >>> You can touch web.xml (under WEB-INF/ ). >>> >>> On Sat, 2009-05-30 at 14:33 -0700, prb wrote: >>> > Hi I have a service to reindex my intranet nightly. >>> > I create a new index and copy the files and reboot tomcat. >>> > This all works great from a terminal session but fails from from unless >>> > I touch index files and reboot manually and then it works but I want to >>> > automate this.. >>> > Is there some trick to force tomcat to reload my new index from a from >>> bash? >>> > Maybe a java -jar one liner or a specific file I can update to trip >>> tomcat ? >>> > Thx >>> > >>> > >>> >>> >>> >> > > > --001485f85b0e545cc5046b435b5d-- From nutch-userjulie68@example.com Mon Jun 01 08:30:57 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 761 invoked from network); 1 Jun 2009 08:30:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 08:30:57 -0000 Received: (qmail 47987 invoked by uid 500); 1 Jun 2009 08:31:08 -0000 Delivered-To: julie68@example.com Received: (qmail 47920 invoked by uid 500); 1 Jun 2009 08:31:08 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 47908 invoked by uid 99); 1 Jun 2009 08:31:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 08:31:05 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 08:30:55 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <julie68@example.com>) id 1MB2uh-0004xy-2E for julie68@example.com; Mon, 01 Jun 2009 01:30:35 -0700 Message-ID: <julie68@example.com> Date: Mon, 1 Jun 2009 01:30:35 -0700 (PDT) From: Mick Peters <julie68@example.com> To: julie68@example.com Subject: hadoop.log in parallel crawling MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: julie68@example.com X-Virus-Checked: Checked by ClamAV on apache.org Hi dudes, I'm crawling severals sites in parallel and if I crawl with the default nutch_conf I've problems writting at the same hadoop.log. Now I create a nutch_conf_dir per site and set a new hadoop.log.dir per site in log4j.properties, the problem is when i'm running a crawl and export the new NUTCH_CONF_DIR still writting in the default NUTCH_CONF. Thanks in advance! -- View this message in context: http://www.nabble.com/hadoop.log-in-parallel-crawling-tp23811444p23811444.html Sent from the Nutch - User mailing list julie68@example.com. From nutch-userjulie68@example.com Mon Jun 01 11:49:26 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 64131 invoked from network); 1 Jun 2009 11:49:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 11:49:25 -0000 Received: (qmail 97374 invoked by uid 500); 1 Jun 2009 11:49:36 -0000 Delivered-To: julie68@example.com Received: (qmail 97300 invoked by uid 500); 1 Jun 2009 11:49:36 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 97290 invoked by uid 99); 1 Jun 2009 11:49:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 11:49:36 +0000 X-ASF-Spam-Status: No, hits=1.1 required=10.0 tests=FORGED_HOTMAIL_RCVD2,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 11:49:26 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <julie68@example.com>) id 1MB60o-0003xc-8k for julie68@example.com; Mon, 01 Jun 2009 04:49:06 -0700 Message-ID: <julie68@example.com> Date: Mon, 1 Jun 2009 04:49:06 -0700 (PDT) From: Larsson85 <julie68@example.com> To: julie68@example.com Subject: Getting the language-identifier info MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: julie68@example.com X-Virus-Checked: Checked by ClamAV on apache.org Hi I wounder how I can extract the info that the language-identifier plugin produces. If I was allowed to wish I would like the info to come when I dump the data from the segments with the following command bin/nutch readseg -dump crawl/segments/... output -nofetch -nogenerate -noparse -noparsedata -parsetext something like URL888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4 http://... Languag888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4 en Is it possible to get nutch to output it like, if not, is it possible to get the info in some other way? As it is now I cant seem to find the info anywhere. I've done the invert links and index step, but I have no clue on where my language info is stored so I can extract it. -- View this message in context: http://www.nabble.com/Getting-the-language-identifier-info-tp23813763p23813763.html Sent from the Nutch - User mailing list julie68@example.com. From nutch-userjulie68@example.com Mon Jun 01 14:32:12 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 34524 invoked from network); 1 Jun 2009 14:32:12 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 14:32:12 -0000 Received: (qmail 63562 invoked by uid 500); 1 Jun 2009 14:23:51 -0000 Delivered-To: julie68@example.com Received: (qmail 63536 invoked by uid 500); 1 Jun 2009 14:23:51 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 63526 invoked by uid 99); 1 Jun 2009 14:23:51 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 14:23:51 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 14:23:41 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <julie68@example.com>) id 1MB8Q4-0002OR-P0 for julie68@example.com; Mon, 01 Jun 2009 07:23:20 -0700 Message-ID: <julie68@example.com> Date: Mon, 1 Jun 2009 07:23:20 -0700 (PDT) From: Chetan Patel <julie68@example.com> To: julie68@example.com Subject: Re: Arabic language in Nutch In-Reply-To: <julie68@example.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_152550_8044567.1243866200767" X-Nabble-From: julie68@example.com References: <julie68@example.com> <julie68@example.com> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_152550_8044567.1243866200767 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello, I want to crawl Arabic URL (http://www.kuna.net.kw/NewsAgenciesPublicSite/HomePage.aspx?Language=3Dar)= It contains charset windows-1256. I have another URL (http://www.afp.com/afpcom/ar/home) and it contains charset UTF-8. This links work fine(crawling, indexing and searching workin= g properly).=20 When I search in above url it return unreadable characters (=D8=B8=C6=92=D8= =B8=CB=86=D8=B8=E2=80=A0=D8=B7=C2=A7 =D8=B8=E2=80=9E=D8=B8=E2=80=9E=D8=B7=C2=A5=D8=B7=C2=B9=D8=B8=E2=80=9E=D8=B7= =C2=A7=D8=B8=E2=80=A0 =D8=B7=C2=B9=D8=B8=E2=80=9E=D8=B8=E2=80=B0 =D8=B7=C2= =A7=D8=B8=E2=80=9E=D8=B8=E2=80=A6=D8=B8=CB=86=D8=B8=E2=80=9A=D8=B7=C2=B9 = =D8=B7=C2=A5=D8=B7). I want to search properly. =20 Is there any issue with charset? plz help me.=20 Thanks in advance. Regards, Chetan Patel --=20 View this message in context: http://www.nabble.com/Arabic-language-in-Nutc= h-tp9269533p23815696.html Sent from the Nutch - User mailing list julie68@example.com. ------=_Part_152550_8044567.1243866200767-- From nutch-userjulie68@example.com Mon Jun 01 15:59:16 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 81381 invoked from network); 1 Jun 2009 15:59:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 15:59:15 -0000 Received: (qmail 8994 invoked by uid 500); 1 Jun 2009 15:59:26 -0000 Delivered-To: julie68@example.com Received: (qmail 8954 invoked by uid 500); 1 Jun 2009 15:59:26 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 8944 invoked by uid 99); 1 Jun 2009 15:59:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 15:59:26 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-fx0-f159.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 15:59:18 +0000 Received: by fxm3 with SMTP id 3so2513643fxm.5 for <julie68@example.com>; Mon, 01 Jun 2009 08:58:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=WL8QNkZSkxaWxtfWhOsMuM+m/yRTuPEmrbOfFLJauok=; b=c3FLmjIk3GTWT8WSZzmtNEDl+GTbiA4tDEeFabecJ9+XBZ9g9aeAD7JC52DPGkytPE jHKd2fS9TRkdKaxWpSrvPoUIzkcd4lWCrrYDvnchFv2Vwz4nbEgBCqY58nPcY8Y1RsPV r8UrjgifFWalXdMmtk1MZ91+SiwisPU0AWy8A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=cSXjCgnPjo1P0Yl9jLeu1PPa/il9QMm+30bhgFLwIv7Z59KpSANJrKS5ajkGLcH+DO hxF9yZmfP9r2zvUUi029HOefymXYQpx3R4rPeVPN/WB405nsgieDlUf/cwAklS+KPZsE CA8isNcrfF9VbN2YqWKSV0yUTITGHTYpeu95M= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id g12mr3320253mur.87.1243871936472; Mon, 01 Jun 2009 08:58:56 -0700 (PDT) Date: Mon, 1 Jun 2009 17:58:56 +0200 Message-ID: <julie68@example.com> Subject: Problem opening the index From: =?ISO-8859-1?Q?Raymond_Balm=E8s?= <julie68@example.com> To: julie68@example.com Content-Type: multipart/alternative; boundary=00504502de84eb711b046b4b8026 X-Virus-Checked: Checked by ClamAV on apache.org --00504502de84eb711b046b4b8026 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi guys, My webapp does not work anymore suddenly... I get the following in the logs. Where is this coming from ? -Ray- 2009-06-01 17:48:38,688 INFO SearchBean - opening merged index in d:/xxxx/nutch/crawl/index 2009-06-01 17:48:38,715 WARN FileSystem - uri=file:/// javax.security.auth.login.LoginException: Login failed: Expect one token as the result of whoami: autorite nt\julie68@example.com(UnixUserGroupInformation.java:250) at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275) at org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257) at org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1438) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1376) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120) at org.apache.nutch.searcher.IndexSearcher.<init>(IndexSearcher.java:70) at org.apache.nutch.searcher.LuceneSearchBean.init(LuceneSearchBean.java:58) at org.apache.nutch.searcher.LuceneSearchBean.<init>(LuceneSearchBean.java:51) at org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:102) at org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:77) at org.apache.nutch.searcher.NutchBean$NutchBeanConstructor.contextInitialized(NutchBean.java:425) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3764) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4216) at org.apache.catalina.core.StandardContext.reload(StandardContext.java:3025) at org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:904) at org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:495) at org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:99) at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) at java.lang.Thread.run(Thread.java:619) --00504502de84eb711b046b4b8026-- From nutch-userjulie68@example.com Mon Jun 01 16:09:22 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 87231 invoked from network); 1 Jun 2009 16:09:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 16:09:22 -0000 Received: (qmail 25734 invoked by uid 500); 1 Jun 2009 16:09:32 -0000 Delivered-To: julie68@example.com Received: (qmail 25688 invoked by uid 500); 1 Jun 2009 16:09:32 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 25654 invoked by uid 99); 1 Jun 2009 16:09:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 16:09:32 +0000 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=RCVD_IN_BL_SPAMCOP_NET,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO poczta.o2.pl) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 16:09:21 +0000 Received: from poczta.o2.pl (mx12 [83.161.120.155]) by poczta.o2.pl (Postfix) with ESMTP id 352F7104006 for <julie68@example.com>; Mon, 1 Jun 2009 18:09:01 +0200 (CEST) Received: from [83.161.120.155] (ip-52-4.apus-net.pl [83.161.120.155]) by poczta.o2.pl (Postfix) with ESMTP for <julie68@example.com>; Mon, 1 Jun 2009 18:08:58 +0200 (CEST) Message-ID: <julie68@example.com> Date: Mon, 01 Jun 2009 18:08:58 +0200 From: Bartosz Gadzimski <julie68@example.com> User-Agent: Thunderbird 83.161.120.155 (Windows/20090302) MIME-Version: 1.0 To: julie68@example.com Subject: Re: Problem opening the index References: <julie68@example.com> In-Reply-To: <julie68@example.com> Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 8bit X-Antivirus: avast! (VPS 090531-0, 2009-05-31), Outbound message X-Antivirus-Status: Clean X-Virus-Checked: Checked by ClamAV on apache.org I think is the login of user running nutch. It want's "one token" after using whoami but it gets: "autorite nt\system" So change the user for one word like "autorite" and it should help. Thanks, Bartosz Raymond Balmès pisze: > Hi guys, > > My webapp does not work anymore suddenly... I get the following in the logs. > Where is this coming from ? > > > -Ray- > > 2009-06-01 17:48:38,688 INFO SearchBean - opening merged index in > d:/xxxx/nutch/crawl/index > 2009-06-01 17:48:38,715 WARN FileSystem - uri=file:/// > javax.security.auth.login.LoginException: Login failed: Expect one token as > the result of whoami: autorite nt\system > at > org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:250) > at > org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:275) > at > org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupInformation.java:257) > at > org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformation.java:67) > at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1438) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1376) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120) > at org.apache.nutch.searcher.IndexSearcher.<init>(IndexSearcher.java:70) > at > org.apache.nutch.searcher.LuceneSearchBean.init(LuceneSearchBean.java:58) > at > org.apache.nutch.searcher.LuceneSearchBean.<init>(LuceneSearchBean.java:51) > at org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:102) > at org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:77) > at > org.apache.nutch.searcher.NutchBean$NutchBeanConstructor.contextInitialized(NutchBean.java:425) > at > org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3764) > at > org.apache.catalina.core.StandardContext.start(StandardContext.java:4216) > at > org.apache.catalina.core.StandardContext.reload(StandardContext.java:3025) > at > org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:904) > at > org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:495) > at > org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:99) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:875) > at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) > at java.lang.Thread.run(Thread.java:619) > > From nutch-userjulie68@example.com Mon Jun 01 16:49:08 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 819 invoked from network); 1 Jun 2009 16:49:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 16:49:07 -0000 Received: (qmail 74751 invoked by uid 500); 1 Jun 2009 16:49:19 -0000 Delivered-To: julie68@example.com Received: (qmail 74706 invoked by uid 500); 1 Jun 2009 16:49:19 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 74696 invoked by uid 99); 1 Jun 2009 16:49:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 16:49:19 +0000 X-ASF-Spam-Status: No, hits=-1.0 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO qdcxa0587.questdiagnostics.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 16:49:06 +0000 Received: from ([83.161.120.155]) by qdcxa0587.questdiagnostics.com with ESMTP with TLS id 5502699.40069263; Mon, 01 Jun 2009 13:00:40 -0400 Received: from qdcws0934.us.qdx.com (83.161.120.155) by QDCWS0925.us.qdx.com (83.161.120.155) with Microsoft SMTP Server id 8.1.336.0; Mon, 1 Jun 2009 12:48:14 -0400 Received: from qdcws0796.us.qdx.com ([83.161.120.155]) by qdcws0934.us.qdx.com with Microsoft SMTPSVC765.867.9606); Mon, 1 Jun 2009 12:48:13 -0400 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: RE: Nutch reindex cron Date: Mon, 1 Jun 2009 12:48:13 -0400 Message-ID: <julie68@example.com> In-Reply-To: <1243822703.17403.1.camel@dadih01> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Nutch reindex cron thread-index: AcniX2BN+GP2MFLETWiyCVwqGJH0ggAeGZUw References: <julie68@example.com> <1243822703.17403.1.camel@dadih01> From: "Malaviya, Sanjay X" <julie68@example.com> To: <julie68@example.com>, <julie68@example.com> X-OriginalArrivalTime: 01 Jun 2009 16:48:13.0024 (UTC) FILETIME=[C0AE6E00:01C9E2D8] Content-Type: text/plain; charset="US-ASCII" X-Virus-Checked: Checked by ClamAV on apache.org Hi Kevin,=0D=0AI have been trying to create a script for re-indexing (I sup= pose also=0D=0Acalled re-crawling) to run everynight=2E I am having problem= s with the=0D=0Asection I listed below=2E Specially "-adddays 30"=2E It tak= e more than 24=0D=0Ahrs to reindex=2E If I make it smaller then it doesn't = pickup the modified=0D=0Afiles=2E=0D=0A=0D=0AAre you also having similar sc= ript? Or can you tell me where you found=0D=0Athe steps to create re-indexi= ng? Or can I forward my script to you?=0D=0A=0D=0APlease help me=2E I have = been working on it for days=2E=0D=0A=0D=0A# To generate/fetch/update cycle= =0D=0Afor ((i=3D1; i <=3D depth ; i++))=0D=0Ado=0D=0A bin/nutch generate c= rawl/crawldb crawl/segments -topN 1000 -adddays 30=0D=0A segment=3D`ls -d = crawl/segments/* | tail -1`=0D=0A bin/nutch fetch $segment=0D=0A bin/nutc= h updatedb crawl/crawldb $segment=0D=0Adone =0D=0A=0D=0A=0D=0A--Sanjay =0D= =0A=0D=0A=0D=0A-----Original Message-----=0D=0AFrom: kevin chen [mailto:kev= inchen@bdsing=2Ecom] =0D=0ASent: Sunday, May 31, 2009 10:18 PM=0D=0ATo: nut= ch-user@lucene=2Eapache=2Eorg=0D=0ASubject: Re: Nutch reindex cron=0D=0A=0D= =0A=0D=0AYou can touch web=2Exml (under WEB-INF/ )=2E=0D=0A=0D=0AOn Sat, 20= 09-05-30 at 14:33 -0700, prb wrote:=0D=0A> Hi I have a service to reindex m= y intranet nightly=2E=0D=0A> I create a new index and copy the files and re= boot tomcat=2E=0D=0A> This all works great from a terminal session but fail= s from from =0D=0A> unless I touch index files and reboot manually and then= it works but I=0D=0A=0D=0A> want to automate this=2E=2E=0D=0A> Is there so= me trick to force tomcat to reload my new index from a from=0D=0Abash?=0D= =0A> Maybe a java -jar one liner or a specific file I can update to trip=0D= =0Atomcat ?=0D=0A> Thx=0D=0A> =0D=0A> =0D=0A=0D=0A-------------------------= -----------------=0D=0AThe contents of this message, together with any atta= chments, are=0Aintended only for the use of the person(s) to which they are= =0Aaddressed and may contain confidential and/or privileged=0Ainformation= =2E Further, any medical information herein is=0Aconfidential and protected= by law=2E It is unlawful for unauthorized=0Apersons to use, review, copy, = disclose, or disseminate confidential=0Amedical information=2E If you are n= ot the intended recipient,=0Aimmediately advise the sender and delete this = message and any=0Aattachments=2E Any distribution, or copying of this messa= ge, or any=0Aattachment, is prohibited=2E From nutch-userjulie68@example.com Mon Jun 01 17:02:04 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 5504 invoked from network); 1 Jun 2009 17:02:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 17:02:04 -0000 Received: (qmail 93975 invoked by uid 500); 1 Jun 2009 17:02:15 -0000 Delivered-To: julie68@example.com Received: (qmail 93889 invoked by uid 500); 1 Jun 2009 17:02:15 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 93879 invoked by uid 99); 1 Jun 2009 17:02:15 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 17:02:15 +0000 X-ASF-Spam-Status: No, hits=1.8 required=10.0 tests=MIME_QP_LONG_LINE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO prxy.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 17:02:05 +0000 X-Scanned-By: RAE MPP/ClamAV http://raeinternet.com/mpp X-Scanned-By: This message was scanned by MPP Free Edition (www.messagepartners.com)! Received: from [83.161.120.155] (account julie68@example.com HELO [83.161.120.155]) by prxy.net (CommuniGate Pro SMTP 4.2.10) with ESMTP-TLS id 105737045 for julie68@example.com; Mon, 01 Jun 2009 10:02:19 -0700 Mime-Version: 1.0 Message-Id: <p06240800c649b9c65f3c@[83.161.120.155]> Date: Mon, 1 Jun 2009 10:01:29 -0700 To: julie68@example.com From: Ken Krugler <julie68@example.com> Subject: Re: Arabic language in Nutch Content-Type: text/plain; charset="iso-8859-1" ; format="flowed" Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org >Hello, > >I want to crawl Arabic URL >(http://www.kuna.net.kw/NewsAgenciesPublicSite/HomePage.aspx?Language=3Dar)= It >contains charset windows-1256. Are you sure it's really 1256? The charset=20 returned by the server (in the headers) for this=20 page is UTF-*: < Content-Type: text/html; charset=3Dutf-8 And when I look at some of the content with an=20 implicit UTF-8 charset, it seems to make=20 sense...or at least the encoded byte sequences=20 are valid. If there are any native Arabic=20 speakers, then the question is whether this makes=20 sense as the page title: <snip - had to cut out image, as otherwise=20 Apache's mail server rejects this as spam...sorry> If so, then the page encoding really is UTF-8, not 1256. I do see the HTML meta tag that specifies the windows-1256 charset: <META http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dwindows-12= 56"> But I think that might be wrong. -- Ken > >I have another URL (http://www.afp.com/afpcom/ar/home) and it contains >charset UTF-8. This links work fine(crawling, indexing and searching workin= g >properly). > >When I search in above url it return unreadable characters (=FF=9F=FF=96=FF= =DD=D7=A7 >=FF=93=FF=93=D7=A5=D7=A9=96=FF=93=D7=A7=FF=DD =D7=A9=96=FF=93=FF=94= =D7=A7=FF=93=FF=8A=FF=96=FF=92=D7=A9=96 =D7=A5=D7). I want to search= properly. > >Is there any issue with charset? plz help me. > >Thanks in advance. > >Regards, >Chetan Patel -- Ken Krugler 765.867.9606 From nutch-userjulie68@example.com Mon Jun 01 17:32:57 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 21905 invoked from network); 1 Jun 2009 17:32:57 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 17:32:57 -0000 Received: (qmail 38054 invoked by uid 500); 1 Jun 2009 17:33:08 -0000 Delivered-To: julie68@example.com Received: (qmail 37997 invoked by uid 500); 1 Jun 2009 17:33:08 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 37987 invoked by uid 99); 1 Jun 2009 17:33:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 17:33:08 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f227.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 17:33:00 +0000 Received: by bwz27 with SMTP id 27so8159465bwz.5 for <julie68@example.com>; Mon, 01 Jun 2009 10:32:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=MXnKvK0aTAoTCfe/zOBi6PHpUgxzeGLFH4ianrrdHwE=; b=hQ7gKDXaHeu67y0IOnEXsg1QqLbuOdYF4YPhNVlXFgjmKCKH3btpvvIQ88RG/g4Xi7 QwJpvfkBXbKDM8k/SX7usaZU4YC2b2oyXNABUbXWEsA1lmxZiQjiVibFYEIfKh0U/xA+ f0ROAUT6UfWwkXA0iGaWmB9CzfKzdF/flAFcM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=ayQIMWKJtm4jb33XZXsUUcaML4h5topvyiTp+hZgzUjQrr9rKe/p3Noh5ZPkyLac9t zuFbKOlF2Bd3moU5OqoZOrD1RqZ5OnAL+78J0c9Pl6x+v//PIPjg4snGzaQi52w7YVG0 RT74xNLp/ygMAmYUB+faIbH05rYuhrOskhM2w= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id o5mr455902mui.34.1243877558293; Mon, 01 Jun 2009 10:32:38 -0700 (PDT) In-Reply-To: <julie68@example.com> References: <julie68@example.com> <julie68@example.com> Date: Mon, 1 Jun 2009 19:32:38 +0200 Message-ID: <julie68@example.com> Subject: Re: Problem opening the index From: =?ISO-8859-1?Q?Raymond_Balm=E8s?= <julie68@example.com> To: julie68@example.com Content-Type: multipart/alternative; boundary=0016364169dd01976c046b4cd004 X-Virus-Checked: Checked by ClamAV on apache.org --0016364169dd01976c046b4cd004 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable hum where do you change that ? 2009/6/1 Bartosz Gadzimski <julie68@example.com> > I think is the login of user running nutch. > > It want's "one token" after using whoami but it gets: > > "autorite nt\system" > > So change the user for one word like "autorite" and it should help. > > Thanks, > Bartosz > > > Raymond Balm=E8s pisze: > > Hi guys, >> >> My webapp does not work anymore suddenly... I get the following in the >> logs. >> Where is this coming from ? >> >> >> -Ray- >> >> 2009-06-01 17:48:38,688 INFO SearchBean - opening merged index in >> d:/xxxx/nutch/crawl/index >> 2009-06-01 17:48:38,715 WARN FileSystem - uri=3Dfile:/// >> javax.security.auth.login.LoginException: Login failed: Expect one token >> as >> the result of whoami: autorite nt\system >> at >> >> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupI= nformation.java:250) >> at >> >> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupI= nformation.java:275) >> at >> >> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupI= nformation.java:257) >> at >> >> org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformati= on.java:67) >> at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:143= 8) >> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1376) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215) >> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120) >> at org.apache.nutch.searcher.IndexSearcher.<init>(IndexSearcher.java:70= ) >> at >> org.apache.nutch.searcher.LuceneSearchBean.init(LuceneSearchBean.java:58= ) >> at >> >> org.apache.nutch.searcher.LuceneSearchBean.<init>(LuceneSearchBean.java:= 51) >> at org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:102) >> at org.apache.nutch.searcher.NutchBean.<init>(NutchBean.java:77) >> at >> >> org.apache.nutch.searcher.NutchBean$NutchBeanConstructor.contextInitiali= zed(NutchBean.java:425) >> at >> >> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.j= ava:3764) >> at >> org.apache.catalina.core.StandardContext.start(StandardContext.java:4216= ) >> at >> org.apache.catalina.core.StandardContext.reload(StandardContext.java:302= 5) >> at >> org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:90= 4) >> at >> >> org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet= .java:495) >> at >> >> org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.= java:99) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:627) >> at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) >> at >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica= tionFilterChain.java:269) >> at >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt= erChain.java:188) >> at >> >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv= e.java:213) >> at >> >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv= e.java:172) >> at >> >> org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticator= Base.java:525) >> at >> >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java= :127) >> at >> >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java= :117) >> at >> >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.= java:108) >> at >> >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:1= 74) >> at >> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:87= 5) >> at >> >> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.proc= essConnection(Http11BaseProtocol.java:665) >> at >> >> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint= .java:528) >> at >> >> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollow= erWorkerThread.java:81) >> at >> >> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool= .java:689) >> at java.lang.Thread.run(Thread.java:619) >> >> >> > > --0016364169dd01976c046b4cd004-- From nutch-userjulie68@example.com Mon Jun 01 18:24:00 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 60182 invoked from network); 1 Jun 2009 18:24:00 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 18:24:00 -0000 Received: (qmail 18021 invoked by uid 500); 1 Jun 2009 18:24:11 -0000 Delivered-To: julie68@example.com Received: (qmail 17965 invoked by uid 500); 1 Jun 2009 18:24:11 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 17955 invoked by uid 99); 1 Jun 2009 18:24:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 18:24:11 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ew0-f179.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 18:23:59 +0000 Received: by ewy27 with SMTP id 27so8139463ewy.5 for <julie68@example.com>; Mon, 01 Jun 2009 11:23:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=wOrfM7onoKDUr5lZAMhhc3HJ8nXIoKKCr3gG7r4xpK0=; b=dtLbtnZRgg3KgembDXtwSj4aGFaxCEOVfc1GTQQtvikJBLLrhI50Slo1gUmQc1/EaI z2d+GAIC6HjCUyg4EbwDuR1sPQgLMWB/GiLa5WBvcELrmvErfVbhKu6J8SLIC14FW5zF Rz92QMeIogWssvwZxUkXPN1+m9FqoDv8qEI3o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=vzzXuQXASroS+3cFovfahvU7o1p4D9URlFakiQnuCgwy+b+49R+zQA9l9nJMO8EF85 tEsNZhRkdlCwsnFnPKGoX+vb3KOvQ46HhSYKxZKxgk+12F9uUShw8ROkMmVQK6xKBWJu ZJKRv02OKTPN9C+64whsTPcCvNti5lP/oH7Ww= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id p47mr1936291wep.13.1243880616769; Mon, 01 Jun 2009 11:23:36 -0700 (PDT) Date: Mon, 1 Jun 2009 14:23:36 -0400 Message-ID: <julie68@example.com> Subject: Can Nutch crawler Impersonate user-agent? From: Jake Jacobson <julie68@example.com> To: julie68@example.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi, I am testing out Nutch 1.0 and it doesn't seem to be able to crawl my website that has the following robots.txt file: User-agent: imo-robot-intelink Disallow: /App_Themes/ Disallow: /app_themes/ Disallow: /Archive/ Disallow: /archive/ Disallow: /Bin/ Disallow: /bin/ I have the nutch-site.xml defined as: <configuration> <property> <name>http.agent.name</name> <value>imo-robot-intelink</value> <description>ICES Robots Name</description> </property> <property> <name>http.agent.version</name> <value></value> <description></description> </property> <property> <name>http.agent.description</name> <value>ICES Open Source Web Crawler using Nutch 1.0</value> <description></description> </property> <property> <name>http.agent.url</name> <value>http://www.xxx.gov/search/</value> <description></description> </property> <property> <name>http.agent.email</name> <value></value> <description></description> </property> </configuration> When I run the following "./nutch crawl ../urls -dir ../crawl/ -depth 3 -topN 50" from the command line I get: crawl started in: ../crawl rootUrlDir = ../urls threads = 10 depth = 3 topN = 50 Injector: starting Injector: crawlDb: ../crawl/crawldb Injector: urlDir: ../urls Injector: Converting injected urls to crawl db entries. Injector: Merging injected urls into crawl db. Injector: done Generator: Selecting best-scoring urls due for fetch. Generator: starting Generator: segment: ../crawl/segments/20090601180745 Generator: filtering: true Generator: topN: 50 Generator: jobtracker is 'local', generating exactly one partition. Generator: Partitioning selected urls by host, for politeness. Generator: done. Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property. Fetcher: starting Fetcher: segment: ../crawl/segments/20090601180745 Fetcher: threads: 10 QueueFeeder finished: total 3 records. fetching http://www.intelink.gov/ fetching http://www.intelink.gov/blogs/ fetching http://www.intelink.gov/wiki/Main_Page -finishing thread FetcherThread, activeThreads=9 -finishing thread FetcherThread, activeThreads=8 -finishing thread FetcherThread, activeThreads=7 -finishing thread FetcherThread, activeThreads=6 -finishing thread FetcherThread, activeThreads=5 -finishing thread FetcherThread, activeThreads=4 -finishing thread FetcherThread, activeThreads=3 -finishing thread FetcherThread, activeThreads=2 -finishing thread FetcherThread, activeThreads=1 -finishing thread FetcherThread, activeThreads=0 -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0 -activeThreads=0 Fetcher: done CrawlDb update: starting CrawlDb update: db: ../crawl/crawldb CrawlDb update: segments: [../crawl/segments/20090601180745] CrawlDb update: additions allowed: true CrawlDb update: URL normalizing: true CrawlDb update: URL filtering: true CrawlDb update: Merging segment data into db. CrawlDb update: done Generator: Selecting best-scoring urls due for fetch. Generator: starting Generator: segment: ../crawl/segments/20090601180757 Generator: filtering: true Generator: topN: 50 Generator: jobtracker is 'local', generating exactly one partition. Generator: 0 records selected for fetching, exiting ... Stopping at depth=1 - no more URLs to fetch. LinkDb: starting LinkDb: linkdb: ../crawl/linkdb LinkDb: URL normalize: true LinkDb: URL filter: true LinkDb: adding segment: file:/linsearchtools1o/oscrawlers/nutch-1.0/crawl/segments/20090601180745 LinkDb: done Indexer: starting Indexer: done Dedup: starting Dedup: adding indexes in: ../crawl/indexes Dedup: done merging indexes to: ../crawl/index Adding file:/linsearchtools1o/oscrawlers/nutch-1.0/crawl/indexes/part-00000 done merging crawl finished: ../crawl I have a tail on my webserver log files and I see the robots.txt file requested with a 200 but nothing gets into the index. I see the error message "Fetcher: Your 'http.agent.name' value should be listed first in 'http.robots.agents' property." which it is listed first. Any help given would be most appreciated. Jake Jacobson http://www.linkedin.com/in/jakejacobson http://www.facebook.com/people/Jake_Jacobson/622727274 Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter. -- ANONYMOUS From nutch-userjulie68@example.com Mon Jun 01 18:46:52 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 72500 invoked from network); 1 Jun 2009 18:46:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 18:46:52 -0000 Received: (qmail 61382 invoked by uid 500); 1 Jun 2009 18:47:03 -0000 Delivered-To: julie68@example.com Received: (qmail 61337 invoked by uid 500); 1 Jun 2009 18:47:03 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 61327 invoked by uid 99); 1 Jun 2009 18:47:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 18:47:03 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail2.colegroup.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 18:46:54 +0000 Received: from [83.161.120.155] (83.161.120.155) by mail2.colegroup.com with ESMTP (EIMS X 3.3.6) for <julie68@example.com>; Mon, 1 Jun 2009 11:46:32 -0700 Mime-Version: 1.0 Message-Id: <a06210233c649d234b0b9@[83.161.120.155]> In-Reply-To: <julie68@example.com> References: <julie68@example.com> Date: Mon, 1 Jun 2009 11:46:27 -0700 To: julie68@example.com From: "David M. Cole" <julie68@example.com> Subject: Re: Can Nutch crawler Impersonate user-agent? Content-Type: text/plain; charset="us-ascii" ; format="flowed" X-Virus-Checked: Checked by ClamAV on apache.org At 2:23 PM -0400 6/1/09, Jake Jacobson wrote: >User-agent: imo-robot-intelink >Disallow: /App_Themes/ >Disallow: /app_themes/ >Disallow: /Archive/ >Disallow: /archive/ >Disallow: /Bin/ >Disallow: /bin/ Jake: I think you need to add one more line after the last line: Allow: / \dmc -- *+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+ David M. Cole julie68@example.com Editor & Publisher, NewsInc. <http://newsinc.net> V: 765.867.9606 Consultant: The Cole Group <http://colegroup.com/> F: 765.867.9606 *+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+ From nutch-userjulie68@example.com Mon Jun 01 19:19:36 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 83287 invoked from network); 1 Jun 2009 19:19:36 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 19:19:36 -0000 Received: (qmail 17946 invoked by uid 500); 1 Jun 2009 19:19:47 -0000 Delivered-To: julie68@example.com Received: (qmail 17908 invoked by uid 500); 1 Jun 2009 19:19:47 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 17898 invoked by uid 99); 1 Jun 2009 19:19:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 19:19:47 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-ew0-f179.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 19:19:36 +0000 Received: by ewy27 with SMTP id 27so8182150ewy.5 for <julie68@example.com>; Mon, 01 Jun 2009 12:19:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=Zw1smo0GyCB89hU833SMAgV9EmdbfkN630d1w5jCF7Y=; b=H3Xjgs5ME/e1XQnw6XYrAJoksnRer0KvvK683KopiUtRP781L5w69DTWnbmo6VS4Rt ZLmfsq+s52vthjPgA2GXISkfNTtyevlprOazqNFH4sToATkPRiMSLs+T6lHpQkWL8cLS CuxnEy0mcomTnGKPNywSZrhaHWiuy5IUmKfmk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=lEzOc1FFKUvGhT0ergw6Axo3MKhpVmoDPIhEVJ1MZleI5XnigpxM9SYQPVSTkvVKUy P4Fl++p0PRpeWS/316XjtZL2+V63ut71pnzatHBZWNR22UAmDMIRL/eExL7MgZevQDom mRChgycxNROr4lUIYBjvYLn/Q+w5cgaoPghhI= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id 5mr1955712wex.217.1243883954695; Mon, 01 Jun 2009 12:19:14 -0700 (PDT) In-Reply-To: <julie68@example.com> References: <julie68@example.com> <julie68@example.com> Date: Mon, 1 Jun 2009 15:19:14 -0400 Message-ID: <julie68@example.com> Subject: Re: Can Nutch crawler Impersonate user-agent? From: Jake Jacobson <julie68@example.com> To: julie68@example.com Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org The Allow directive in the robots.txt is optional. If you don't have an explicit disallow statement, it means that directory or file is available for indexing. Jake Jacobson http://www.linkedin.com/in/jakejacobson http://www.facebook.com/people/Jake_Jacobson/622727274 Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter. -- ANONYMOUS On Mon, Jun 1, 2009 at 2:46 PM, David M. Cole <julie68@example.com> wrote: > At 2:23 PM -0400 6/1/09, Jake Jacobson wrote: >> >> User-agent: imo-robot-intelink >> Disallow: /App_Themes/ >> Disallow: /app_themes/ >> Disallow: /Archive/ >> Disallow: /archive/ >> Disallow: /Bin/ >> Disallow: /bin/ > > Jake: > > I think you need to add one more line after the last line: > > Allow: / > > \dmc > > -- > *+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*= +*+*+ > =A0 David M. Cole =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =julie68@example.com > =A0 Editor & Publisher, NewsInc. <http://newsinc.net> =A0 =A0 =A0 =A0V: (= 765.867.9606 > =A0 Consultant: The Cole Group <http://colegroup.com/> =A0 =A0 =A0 F: (65= 0) 475-8479 > *+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*+*= +*+*+ > From nutch-userjulie68@example.com Mon Jun 01 22:32:45 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 58960 invoked from network); 1 Jun 2009 22:32:45 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Jun 2009 22:32:45 -0000 Received: (qmail 93386 invoked by uid 500); 1 Jun 2009 22:32:53 -0000 Delivered-To: julie68@example.com Received: (qmail 93333 invoked by uid 500); 1 Jun 2009 22:32:53 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 93313 invoked by uid 99); 1 Jun 2009 22:32:53 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 22:32:53 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO qw-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Jun 2009 22:32:44 +0000 Received: by qw-out-2122.google.com with SMTP id 5so5173112qwd.53 for <multiple recipients>; Mon, 01 Jun 2009 15:32:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=KuiaACGdgEDnVKsUBCZlox/sDKoFY+xUfon5kx0SZcU=; b=n8Awe4QSwtOGPe07n5RraTV/vTnmtXTRycR7cGDVPxLLggJbV1Lt8BGntqsiIh4BRl 4qViMe9+PCcGfzm7jEwCKaFPQ4F0WTQFfWiU1L/HD1hfgqX3sFaQ/d7uQnjMX4L38rUX MNYow28aTxrqdDxrDWjssx45GsDrIDHWQFXa8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=pKWkxC6av3yU5FhRNzf/G744ASgfQM7M4beHj6wcc87sAphy4jdiGJbP9eVmwQCu0x jDY7o47bIeQKxmYqQS7Jomo1rPwThP4YjdnHwoO+TEhM1+VOM+6KVkHK/+aEbjj+bjRh fwSPXrck6v3oukKpYOC1V01oCdY9KUT31+KOE= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id c8mr6165648qam.371.1243895543373; Mon, 01 Jun 2009 15:32:23 -0700 (PDT) In-Reply-To: <julie68@example.com> References: <julie68@example.com> Date: Mon, 1 Jun 2009 15:32:23 -0700 Message-ID: <julie68@example.com> Subject: Question on Efficient field updates in the Lucene index in Nutch From: Vijay <julie68@example.com> To: julie68@example.com, julie68@example.com Content-Type: multipart/alternative; boundary=0015175cb6e200206e046b510092 X-Virus-Checked: Checked by ClamAV on apache.org --0015175cb6e200206e046b510092 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi all, I have a question regarding field updates to the lucene index in nutch. Suppose I am indexing webpages along with tags as an extra field. I want to add an extra tag to a webpage. Is there a clean way for me to do this without having to re-index the page with the updated tags field and deleting duplicates? For example, can I create a new document with the same doc id as the object made from the relevant URL in the past, with the new tag alone as a field, with no content section. Could this enable adding the correct doc id to the new tag postings list, thereby creating the same effect as indexing the webpage afresh with all the given tags? Alternatively is there any other efficient way to do this? Thanks a ton, Vijay --0015175cb6e200206e046b510092-- From nutch-userjulie68@example.com Tue Jun 02 03:13:35 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 69909 invoked from network); 2 Jun 2009 03:13:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2009 03:13:34 -0000 Received: (qmail 97433 invoked by uid 500); 2 Jun 2009 03:13:45 -0000 Delivered-To: julie68@example.com Received: (qmail 97375 invoked by uid 500); 2 Jun 2009 03:13:45 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 97365 invoked by uid 99); 2 Jun 2009 03:13:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Jun 2009 03:13:45 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_HELO_PASS,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mout.perfora.net) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Jun 2009 03:13:36 +0000 Received: from ca.test.dadi360.com (pool-173-52-166-227.nycmny.east.verizon.net [83.161.120.155]) by mrelay.perfora.net (node=mrus1) with ESMTP (Nemesis) id 0MKpCa-1MBKR41kBk-000d8Z; Mon, 01 Jun 2009 23:13:12 -0400 Subject: Re: Nutch reindex cron From: kevin chen <julie68@example.com> Reply-To: julie68@example.com To: julie68@example.com Cc: Alexander Aristov <julie68@example.com> In-Reply-To: <julie68@example.com> References: <julie68@example.com> <1243822703.17403.1.camel@dadih01> <julie68@example.com> Content-Type: text/plain Date: Mon, 01 Jun 2009 23:13:09 -0400 Message-Id: <1243912389.5547.11.camel@dadih01> Mime-Version: 1.0 X-Mailer: Evolution 2.0.2 (2.0.2-35.el4) Content-Transfer-Encoding: 7bit X-Provags-ID: V01U2FsdGVkX19TwQUouyjUEIbioH53GvDRpBCNQInwiQLwz8h xzP4Q1hUCSDcAywiIP6yEyNscJIbCg0pxufR/Iq3rQ/Vp8RbcK XZHozDgaU0IWJfR/gImfbIjI+nrtjOC X-Virus-Checked: Checked by ClamAV on apache.org Yes, I experience problem too if I do it too many times. To cop with it, I restart tomcat server everyday during least traffic period. In my case, I only reload index few times during a day, and it works out fine. On Mon, 2009-06-01 at 09:07 +0400, Alexander Aristov wrote: > I experienced problems with such approach. Tomcat hang after a few > application re-deployments. And it seems it's a problem with Tomcat > itself as I saw this issue in different environments and with > different applications. > > Main problem is that Tomcat doesn't release memory and hence it fails > with out-of-memory. > > Best Regards > Alexander Aristov > > > 2009/6/1 kevin chen <julie68@example.com> > > You can touch web.xml (under WEB-INF/ ). > > > On Sat, 2009-05-30 at 14:33 -0700, prb wrote: > > Hi I have a service to reindex my intranet nightly. > > I create a new index and copy the files and reboot tomcat. > > This all works great from a terminal session but fails from > from unless > > I touch index files and reboot manually and then it works > but I want to > > automate this.. > > Is there some trick to force tomcat to reload my new index > from a from bash? > > Maybe a java -jar one liner or a specific file I can update > to trip tomcat ? > > Thx > > > > > > > From nutch-userjulie68@example.com Tue Jun 02 09:36:41 2009 Return-Path: <nutch-userjulie68@example.com> Delivered-To: julie68@example.com Received: (qmail 82407 invoked from network); 2 Jun 2009 09:36:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Jun 2009 09:36:41 -0000 Received: (qmail 99596 invoked by uid 500); 2 Jun 2009 09:36:52 -0000 Delivered-To: julie68@example.com Received: (qmail 99549 invoked by uid 500); 2 Jun 2009 09:36:52 -0000 Mailing-List: contact julie68@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:julie68@example.com> List-Unsubscribe: <mailto:julie68@example.com> List-Post: <mailto:julie68@example.com> List-Id: <nutch-user.lucene.apache.org> Reply-To: julie68@example.com Delivered-To: mailing list julie68@example.com Received: (qmail 99539 invoked by uid 99); 2 Jun 2009 09:36:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Jun 2009 09:36:52 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of julie68@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Jun 2009 09:36:41 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <julie68@example.com>) id 1MBQPt-0003wB-0A for julie68@example.com; Tue, 02 Jun 2009 02:36:21 -0700 Message-ID: <julie68@example.com> Date: Tue, 2 Jun 2009 02:36:20 -0700 (PDT) From: Chetan Patel <julie68@example.com> To: julie68@example.com Subject: Re: help regarding creating the NGramProfile for Tamil language In-Reply-To: <julie68@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: julie68@example.com References: <julie68@example.com> <julie68@example.com> X-Virus-Checked: Checked by ClamAV on apache.org Hello, I am also looking for a index & searching in Arabic language. If you have got any document, plz give me. Thanks in advance. Regards, Chetan suryas wrote: > > Hi, > I want to index & search Tamil (an Indian language) pages using Nutch. I > have some knowledge of Lucene and just got the "Nutch Basic Tutorial" > working. > > Where do I look for indexing Tamil or any other Indian language pages? > > I'm looking for: > *step-by-step" documentation for indexing and search foreign language > pages, particularly Indian languages (hope that's not too much) > *some examples, samples, tutorials would be nice > > Or if you could just point me in the right direction, that'll be fine too. > > I saw some postings from "saran" & "saravana kumar" talking about this > same thing. Guys, did you figure this out? if yes - could you please help? > > Could someone help? > > thanks, > Surya > > > > saran wrote: >> >> i try to set the classpath >> >> as and i run this command in >> ...../nutch0.9 the nutch directory >> >> java -classpath build/language-identifier/language- >>
{'source': 'Apache Software Foundation Public Mail Archives'}