From nutch-devandrewmullen@example.org Sun Mar 01 06:22:35 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 59408 invoked from network); 1 Mar 2009 06:22:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2009 06:22:35 -0000 Received: (qmail 61422 invoked by uid 500); 1 Mar 2009 06:22:29 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 61382 invoked by uid 500); 1 Mar 2009 06:22:29 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 61373 invoked by uid 99); 1 Mar 2009 06:22:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Feb 2009 22:22:29 -0800 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of andrewmullen@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 06:22:21 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <andrewmullen@example.org>) id 1Ldf3o-000659-H1 for andrewmullen@example.org; Sat, 28 Feb 2009 22:22:00 -0800 Message-ID: <andrewmullen@example.org> Date: Sat, 28 Feb 2009 22:22:00 -0800 (PST) From: dealmaker <andrewmullen@example.org> To: andrewmullen@example.org Subject: Re: Release 1.0? In-Reply-To: <andrewmullen@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Nabble-From: andrewmullen@example.org References: <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> X-Virus-Checked: Checked by ClamAV on apache.org so, what's next? Who's in charge of the release process? Andrzej Bialecki wrote: > > Sami Siren wrote: >> Sami Siren wrote: >>> I think that no one else but me made any guesses about the release >>> date? (since it is virtually impossible due to fact that this is not a >>> paid project). >>>> >>>> Andrzej Bialecki wrote: >>>>> We do exist. ;) We plan to release in February - I can't >>>>> tell you yet when exactly, we need to review the (few) remaining >>>>> issues that we want to resolve before the release. >> oh I now see Andrzej made some also :) . > > Yeah, yeah ... :) let's just push it out sooner than later, that's all > I'm saying. > > > -- > Best regards, > Andrzej Bialecki <>< > ___. ___ ___ ___ _ _ __________________________________ > [__ || __|__/|__||\/| Information Retrieval, Semantic Web > ___|||__|| \| || | Embedded Unix, System Integration > http://www.sigram.com Contact: andrewmullen@example.org > > > -- View this message in context: http://www.nabble.com/Release-1.0--tp21701972p22270310.html Sent from the Nutch - Dev mailing list andrewmullen@example.org. From nutch-devandrewmullen@example.org Sun Mar 01 06:45:49 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 80227 invoked from network); 1 Mar 2009 06:45:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2009 06:45:49 -0000 Received: (qmail 71142 invoked by uid 500); 1 Mar 2009 06:45:48 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 71105 invoked by uid 500); 1 Mar 2009 06:45:48 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 71096 invoked by uid 99); 1 Mar 2009 06:45:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Feb 2009 22:45:48 -0800 X-ASF-Spam-Status: No, hits=2.4 required=10.0 tests=HTML_MESSAGE,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of andrewmullen@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO yw-out-2324.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 06:45:40 +0000 Received: by yw-out-2324.google.com with SMTP id 2so1059928ywt.5 for <andrewmullen@example.org>; Sat, 28 Feb 2009 22:45:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=nmbS38LB5CDOsQ8O2bdmP80dr+BNpwbXQoYP+dy49FM=; b=nqot5Vn9AGwUs7Dm389Yk+V/zA+V+suX2dwXjxGGe+G1PxhDV0M9jbnHbBvW4kumD2 PQ4gNx9pWeCuBrk0AJkK5FN/qby3q6YgdwMXHN4RbrAv1kPEX020iH2oOM0ymaqcrNsy xU2k+SFmsH72eSkcsyibDNtr8ExTrXvkTpMZY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=TwvftVzH/bE5ksX/prLXNkwEk8b9JWssUL/qOnc7okQfGoAZFM+hTHXEEt7uyz7MPt GRNopcKs8IMcw0KywWR3gUfkaPYdIMBw56Tf8cV0iiaJKTCY9k2MosqRy2qWj14RqzZL ehBL6AG0u1zCfNagV03q7NSaw6J30OBcr73hI= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id p4mr7643545ybm.77.1235889919229; Sat, 28 Feb 2009 22:45:19 -0800 (PST) In-Reply-To: <andrewmullen@example.org> References: <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> Date: Sun, 1 Mar 2009 12:15:19 +0530 Message-ID: <andrewmullen@example.org> Subject: Re: Release 1.0? From: Techie <andrewmullen@example.org> To: andrewmullen@example.org Content-Type: multipart/alternative; boundary=001517573dd09e00fb0464090b0c X-Virus-Checked: Checked by ClamAV on apache.org --001517573dd09e00fb0464090b0c Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi All Senior Nutch Developers, Why all this hurry??? There are number of good issues yet to be resolved. Version 1 means, no visible issues. As per my view, we can wait for one more month, instead of getting bad reputation. What do you all think? Thanks, Arun Sharma On Sun, Mar 1, 2009 at 11:52 AM, dealmaker <andrewmullen@example.org> wrote: > > so, what's next? Who's in charge of the release process? > > > Andrzej Bialecki wrote: > > > > Sami Siren wrote: > >> Sami Siren wrote: > >>> I think that no one else but me made any guesses about the release > >>> date? (since it is virtually impossible due to fact that this is not a > >>> paid project). > >>>> > >>>> Andrzej Bialecki wrote: > >>>>> We do exist. ;) We plan to release in February - I can't > >>>>> tell you yet when exactly, we need to review the (few) remaining > >>>>> issues that we want to resolve before the release. > >> oh I now see Andrzej made some also :) . > > > > Yeah, yeah ... :) let's just push it out sooner than later, that's all > > I'm saying. > > > > > > -- > > Best regards, > > Andrzej Bialecki <>< > > ___. ___ ___ ___ _ _ __________________________________ > > [__ || __|__/|__||\/| Information Retrieval, Semantic Web > > ___|||__|| \| || | Embedded Unix, System Integration > > http://www.sigram.com Contact: andrewmullen@example.org > > > > > > > > -- > View this message in context: > http://www.nabble.com/Release-1.0--tp21701972p22270310.html > Sent from the Nutch - Dev mailing list andrewmullen@example.org. > > --001517573dd09e00fb0464090b0c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi All Senior Nutch Developers,<br><br>Why all this hurry??? There are numb= er of good issues yet to be resolved. Version 1 means, no visible issues. <= br><br>As per my view, we can wait for one more month, instead of getting b= ad reputation.=A0 What do you all think?<br> <br><br>Thanks,<br><br>Arun Sharma<br><br><div class=3D"gmail_quote">On Sun= , Mar 1, 2009 at 11:52 AM, dealmaker <span dir=3D"ltr">&lt;<a href=3D"mailt= o:andrewmullen@example.org">andrewmullen@example.org</a>&gt;</span> wrote:<br><blockquote c= lass=3D"gmail_quote" style=3D"border-left: 1px solid rgb(204, 204, 204); ma= rgin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;"> <br> so, what&#39;s next? =A0Who&#39;s in charge of the release process?<br> <div><div></div><div class=3D"Wj3C7c"><br> <br> Andrzej Bialecki wrote:<br> &gt;<br> &gt; Sami Siren wrote:<br> &gt;&gt; Sami Siren wrote:<br> &gt;&gt;&gt; I think that no one else but me made any guesses about the rel= ease<br> &gt;&gt;&gt; date? (since it is virtually impossible due to fact that this = is not a<br> &gt;&gt;&gt; paid project).<br> &gt;&gt;&gt;&gt;<br> &gt;&gt;&gt;&gt; Andrzej Bialecki wrote:<br> &gt;&gt;&gt;&gt;&gt; =A0 =A0 =A0 =A0 We do exist. ;) We plan to release in = February - I can&#39;t<br> &gt;&gt;&gt;&gt;&gt; tell you yet when exactly, we need to review the (few)= remaining<br> &gt;&gt;&gt;&gt;&gt; issues that we want to resolve before the release.<br> &gt;&gt; oh I now see Andrzej made some also :) .<br> &gt;<br> &gt; Yeah, yeah ... :) let&#39;s just push it out sooner than later, that&#= 39;s all<br> &gt; I&#39;m saying.<br> &gt;<br> &gt;<br> &gt; --<br> &gt; Best regards,<br> &gt; Andrzej Bialecki =A0 =A0 &lt;&gt;&lt;<br> &gt; =A0 ___. ___ ___ ___ _ _ =A0 __________________________________<br> &gt; [__ || __|__/|__||\/| =A0Information Retrieval, Semantic Web<br> &gt; ___|||__|| =A0\| =A0|| =A0| =A0Embedded Unix, System Integration<br> &gt; <a href=3D"http://www.sigram.com" target=3D"_blank">http://www.sigram.= com</a> =A0Contact: andrewmullen@example.org<br> &gt;<br> &gt;<br> &gt;<br> <br> </div></div><font color=3D"#888888">--<br> View this message in context: <a href=3D"http://www.nabble.com/Release-1.0-= -tp21701972p22270310.html" target=3D"_blank">http://www.nabble.com/Release-= 1.0--tp21701972p22270310.html</a><br> </font><div><div></div><div class=3D"Wj3C7c">Sent from the Nutch - Dev mail= ing list andrewmullen@example.org.<br> <br> </div></div></blockquote></div><br> --001517573dd09e00fb0464090b0c-- From nutch-devandrewmullen@example.org Sun Mar 01 07:32:47 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 11537 invoked from network); 1 Mar 2009 07:32:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2009 07:32:47 -0000 Received: (qmail 89704 invoked by uid 500); 1 Mar 2009 07:32:41 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 89664 invoked by uid 500); 1 Mar 2009 07:32:41 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 89649 invoked by uid 99); 1 Mar 2009 07:32:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Feb 2009 23:32:41 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 07:32:33 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 46196234C4AC for <andrewmullen@example.org>; Sat, 28 Feb 2009 23:32:13 -0800 (PST) Message-ID: <47914308.1235892733286.JavaMail.jira@brutus> Date: Sat, 28 Feb 2009 23:32:13 -0800 (PST) From: "Andrzej Bialecki (JIRA)" <andrewmullen@example.org> To: andrewmullen@example.org Subject: [jira] Commented: (NUTCH-419) unavailable robots.txt kills fetch In-Reply-To: <17927486.1166964321064.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/NUTCH-419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677785#action_12677785 ] Andrzej Bialecki commented on NUTCH-419: ----------------------------------------- Doug, thank you for your excellent analysis. I'll apply the patch before 1.0 - however, the poor handling of robots.txt needs to be addressed later (in 1.1). > unavailable robots.txt kills fetch > ---------------------------------- > > Key: NUTCH-419 > URL: https://issues.apache.org/jira/browse/NUTCH-419 > Project: Nutch > Issue Type: Bug > Components: fetcher > Affects Versions: 0.8.1 > Environment: Fetcher is behind a squid proxy, but I am pretty sure this is irrelevant. > Nutch in local mode, running on a linux machine with 2GB RAM. > Reporter: Carsten Lehmann > Attachments: diffs, last_robots.txt_requests_squidlog.txt, nutch-log.txt, squid_access_log_tail1000.txt > > > I think there is another robots.txt-related problem which is not > adressed by NUTCH-344, > but also results in an aborted fetch. > I am sure that in my last fetch all 17 fetcher threads died > while they were waiting for a robots.txt-file to be delivered by a not > properly responding web server. > I looked at the squid access log, which is used by all fetch threads. > It ends with many HTTP-504-errors ("gateway timeout") caused by a > certain robots.txt url: > <....> > 1166652253.332 899427 83.161.120.155 TCP_MISS/504 1450 GET > http://gso.gbv.de/robots.txt - DIRECT/83.161.120.155 text/html > 1166652343.350 899664 83.161.120.155 TCP_MISS/504 1450 GET > http://gso.gbv.de/robots.txt - DIRECT/83.161.120.155 text/html > 1166652353.560 899871 83.161.120.155 TCP_MISS/504 1450 GET > http://gso.gbv.de/robots.txt - DIRECT/83.161.120.155 text/html > These entries mean that it takes 15 minutes before the request ends > with a timeout. > This can be calculated from the squid log, the first column is the > request time (in UTC seconds), the second column is the duration of > the request (in ms): > 900000/1000/60=15 minutes. > As far as I understand it, every time a fetch thread tries to get this > robots.txt-file the thread busy waits for the duration of the request > (15 minutes). > If this is right, then all 17 fetcher threads were caught in this trap > at the time when fetching was aborted, as there are 17 requests in > the squid log which did not timeout before the message "aborting with > 17 threads" was written to the nutch-logfile. > Setting fetcher.max.crawl.delay can not help here. > I see 296 access attempts in total concerning this robots.txt-url in > the squid log of this crawl, but fetcher.max.crawl.delay is set to 30. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From nutch-devandrewmullen@example.org Sun Mar 01 08:10:05 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 28428 invoked from network); 1 Mar 2009 08:10:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2009 08:10:03 -0000 Received: (qmail 3782 invoked by uid 500); 1 Mar 2009 08:09:57 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 3740 invoked by uid 500); 1 Mar 2009 08:09:57 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 3731 invoked by uid 99); 1 Mar 2009 08:09:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 00:09:57 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO getopt.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 08:09:48 +0000 Received: from [83.161.120.155] (bvs203.neoplus.adsl.tpnet.pl [83.161.120.155]) (authenticated) by getopt.org (8.11.6/8.11.6) with ESMTP id n2189Qr28501 for <andrewmullen@example.org>; Sun, 1 Mar 2009 02:09:27 -0600 Message-ID: <andrewmullen@example.org> Date: Sun, 01 Mar 2009 09:09:23 +0100 From: Andrzej Bialecki <andrewmullen@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20081209) MIME-Version: 1.0 To: andrewmullen@example.org Subject: Re: Release 1.0? References: <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> In-Reply-To: <andrewmullen@example.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Techie wrote: > Hi All Senior Nutch Developers, > > Why all this hurry??? There are number of good issues yet to be > resolved. Version 1 means, no visible issues. What is a visible issue? The one marked as Bug & Major? We tried to address them, and it seems to me that there's just a couple of them left, those that don't require making broad changes to the code base. As for the other open issues ... if they are not critical they need to be dealt with later. We simply have to make a cut somewhere, otherwise we will continue adding new features and tweaks indefinitely, without ever making a release. > > As per my view, we can wait for one more month, instead of getting bad > reputation. What do you all think? We already waited one month, we even waited 1 year ... Even if we schedule 1.1 to be released two months after 1.0 I would still vote for pushing the 1.0 now. If you think that there are any critical unresolved issues please let us know _now_ - if they can be easily fixed and tested, they will be fixed before this release. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: andrewmullen@example.org From nutch-devandrewmullen@example.org Sun Mar 01 20:13:47 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 39975 invoked from network); 1 Mar 2009 20:13:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Mar 2009 20:13:47 -0000 Received: (qmail 680 invoked by uid 500); 1 Mar 2009 20:13:40 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 661 invoked by uid 500); 1 Mar 2009 20:13:40 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 652 invoked by uid 99); 1 Mar 2009 20:13:40 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 12:13:40 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 20:13:32 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id BCE2E234C4A8 for <andrewmullen@example.org>; Sun, 1 Mar 2009 12:13:12 -0800 (PST) Message-ID: <1241452162.1235938392772.JavaMail.jira@brutus> Date: Sun, 1 Mar 2009 12:13:12 -0800 (PST) From: "Aaron Binns (JIRA)" <andrewmullen@example.org> To: andrewmullen@example.org Subject: [jira] Created: (NUTCH-708) NutchBean: OOM due to searcher.max.hits and dedup. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org NutchBean: OOM due to searcher.max.hits and dedup. -------------------------------------------------- Key: NUTCH-708 URL: https://issues.apache.org/jira/browse/NUTCH-708 Project: Nutch Issue Type: Bug Components: searcher Affects Versions: 1.0.0 Environment: Ubuntu Linux, Java 5. Reporter: Aaron Binns When searching an index we built for the National Archives, this one in particular: http://webharvest.gov/collections/congress110th/ We ran into an interesting situation. We were using searcher.max.hits=1000 in order to get faster searches. Since our index is sorted, the "best" documents are "at the front" and setting searcher.max.hits=1000 would give us a nice trade-off of search quality vs. response time. What I discovered was that with dedup (on site) enabled, we would get into this loop where the searcher.max.hits would limit the raw hits to 1000 and the deduplication code would get to the end of those 1000 results and still need more as it hadn't found enough de-dup'd results to satisfy the query. The first 6 pages of results would be fine, but when we got to page 7, the NutchBean would need more than 1000 raw results in order to get 60 de-duped results. The code: for (int rawHitNum = 0; rawHitNum < hits.getTotal(); rawHitNum++) { // get the next raw hit if (rawHitNum >= hits.getLength()) { // optimize query by prohibiting more matches on some excluded values Query optQuery = (Query)query.clone(); for (int i = 0; i < excludedValues.size(); i++) { if (i == MAX_PROHIBITED_TERMS) break; optQuery.addProhibitedTerm(((String)excludedValues.get(i)), dedupField); } numHitsRaw = (int)(numHitsRaw * rawHitsFactor); if (LOG.isInfoEnabled()) { LOG.info("re-searching for "+numHitsRaw+" raw hits, query: "+optQuery); } hits = searcher.search(optQuery, numHitsRaw, dedupField, sortField, reverse); if (LOG.isInfoEnabled()) { LOG.info("found "+hits.getTotal()+" raw hits"); } rawHitNum = -1; continue; } The loop constraints were never satisfied as rawHitNum and hits.getLength() are capped by searcher.max.hits (1000). The numHitsRaw keeps increasing by a factor of 2 (rawHitsFactor) until it gets to 2^31 or so and deep down in the search library code an array is allocated using that value as the size and you get an OOM. We worked around the problem by abandoning the use of searcher.max.hits. I suppose we could have increased the value, but the index was small enough (~10GB) that disabling searcher.max.hits didn't degrade the response time too much. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From nutch-devandrewmullen@example.org Mon Mar 02 03:05:37 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 3766 invoked from network); 2 Mar 2009 03:05:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 03:05:37 -0000 Received: (qmail 24742 invoked by uid 500); 2 Mar 2009 03:05:36 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 24696 invoked by uid 500); 2 Mar 2009 03:05:36 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 24687 invoked by uid 99); 2 Mar 2009 03:05:36 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 19:05:36 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 03:05:33 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id CA035234C4A8 for <andrewmullen@example.org>; Sun, 1 Mar 2009 19:05:12 -0800 (PST) Message-ID: <292862756.1235963112826.JavaMail.jira@brutus> Date: Sun, 1 Mar 2009 19:05:12 -0800 (PST) From: "Dmitry Lihachev (JIRA)" <andrewmullen@example.org> To: andrewmullen@example.org Subject: [jira] Commented: (NUTCH-705) parse-rtf plugin In-Reply-To: <1557366290.1235708281904.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/NUTCH-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677878#action_12677878 ] Dmitry Lihachev commented on NUTCH-705: --------------------------------------- Yes, it looks a bit like a problem... How can we handle this? > parse-rtf plugin > ---------------- > > Key: NUTCH-705 > URL: https://issues.apache.org/jira/browse/NUTCH-705 > Project: Nutch > Issue Type: New Feature > Components: fetcher > Affects Versions: 1.0.0 > Reporter: Dmitry Lihachev > Priority: Minor > Fix For: 1.1 > > Attachments: NUTCH-705.patch > > > Demoting this issue and moving to 1.1 - current patch is not suitable due to LGPL licensed parts. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From nutch-devandrewmullen@example.org Mon Mar 02 07:23:50 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 6050 invoked from network); 2 Mar 2009 07:23:50 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 07:23:50 -0000 Received: (qmail 13711 invoked by uid 500); 2 Mar 2009 07:23:43 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 13683 invoked by uid 500); 2 Mar 2009 07:23:43 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 13674 invoked by uid 99); 2 Mar 2009 07:23:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 01 Mar 2009 23:23:43 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of andrewmullen@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ti-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 07:23:37 +0000 Received: by ti-out-0910.google.com with SMTP id b8so2727502tic.11 for <andrewmullen@example.org>; Sun, 01 Mar 2009 23:23:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type:content-transfer-encoding; bh=J1nqQVAMsuaNH3+Y65kCgCh0U9UBua6aTYh5RyVlHYQ=; b=kCtM8XcdVQP0+wrQs3D+meS8avibWJXKysjeA6YIzKcLPs9H64leMq/a0aT4fBG+gl CKSiM6IO/9bCby0hj+KqxqLzPsb3saDE6Ff38fntnTb+t+X0I6aQtPhd/CYnloU+zlnc qFsuEiNig52ynomD+xOKvJibddmTO2/xd09K8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type :content-transfer-encoding; b=Ocy+1xKjsMwWjrnnf3c9Ur6d+8nPuHmseB8zXO2WNwq4q0Xns2aqx0s0LWOZU3DOZo Q029qBTQRchfwE7/4JKFuBsHeU4BJjKbF3Cp6kr3DqoUjbhErgRob+s4XEVXcF3pJ1hX TPI8mdjx4oCyZw2IG1WwwqzdrfRpHx2X52EG8= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id b19mr6019338tia.12.1235978595080; Sun, 01 Mar 2009 23:23:15 -0800 (PST) Date: Mon, 2 Mar 2009 12:53:15 +0530 Message-ID: <andrewmullen@example.org> Subject: How to make parse-xml plugin (NUTCH-185) compatible with the latest trunk ? From: Gopikrishnan Kookkal <andrewmullen@example.org> To: andrewmullen@example.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Hi, We used (NUTCH-185) XMLParser configurable xml parser plugin in our project for indexing XML documents. When we moved to NUTCH1.0 source base, we find errors while building the XMLParser plugin due to the changes related to using NutchDocument instead of Document. Has anybody build this XMLParser plugin for the truck. Can anybody suggest me code changes to make Parse-xml code compatible with the latest trunk: Here is the main error i faced, when moving from Document to NutchDocument: nutch\src\plugin\parse-xml\src\java\org\apache\nutch\parse\xml\XMLIndexer.java:79: add(java.lang.String,java.lang.String) in org.apache.nutch.indexer.NutchDocument cannot be applied to (org.apache.lucene.document.Field) [javac] doc.add(field); [javac] Any help would be very much useful. Thanks, From nutch-devandrewmullen@example.org Mon Mar 02 08:28:42 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 38574 invoked from network); 2 Mar 2009 08:28:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 08:28:42 -0000 Received: (qmail 61813 invoked by uid 500); 2 Mar 2009 08:28:42 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 61232 invoked by uid 500); 2 Mar 2009 08:28:41 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 61223 invoked by uid 99); 2 Mar 2009 08:28:41 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 00:28:41 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 08:28:33 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id D7D0F234C4A8 for <andrewmullen@example.org>; Mon, 2 Mar 2009 00:28:12 -0800 (PST) Message-ID: <1506169153.1235982492882.JavaMail.jira@brutus> Date: Mon, 2 Mar 2009 00:28:12 -0800 (PST) From: "Sami Siren (JIRA)" <andrewmullen@example.org> To: andrewmullen@example.org Subject: [jira] Updated: (NUTCH-700) Neko1.9.11 goes into a loop In-Reply-To: <114422886.1235126701877.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/NUTCH-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sami Siren updated NUTCH-700: ----------------------------- Fix Version/s: 1.0.0 Assignee: Sami Siren This one just bit me - the effect is that parsing hangs forever. I am promoting it to be fixed in 1.0. > Neko1.9.11 goes into a loop > --------------------------- > > Key: NUTCH-700 > URL: https://issues.apache.org/jira/browse/NUTCH-700 > Project: Nutch > Issue Type: Bug > Affects Versions: 1.0.0 > Reporter: julien nioche > Assignee: Sami Siren > Priority: Critical > Fix For: 1.0.0 > > > Neko1.9.11 goes into a loop on some documents e.g. > http://mediacet.com/Archive/FourYorkshiremen/bb/post.htm > http://cizel.co.kr/main.php > reverting to 0.9.4 seems to fix the problem > The approach mentioned in https://issues.apache.org/jira/browse/NUTCH-696 could be a way to alleviate similar issues > PS: haven't had time to report to the Neko people yet, will do at some stage -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From nutch-devandrewmullen@example.org Mon Mar 02 08:31:22 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 39587 invoked from network); 2 Mar 2009 08:31:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 08:31:22 -0000 Received: (qmail 64305 invoked by uid 500); 2 Mar 2009 08:31:20 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 64272 invoked by uid 500); 2 Mar 2009 08:31:20 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 64263 invoked by uid 99); 2 Mar 2009 08:31:20 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 00:31:20 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of andrewmullen@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-fx0-f168.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 08:31:11 +0000 Received: by fxm12 with SMTP id 12so2182925fxm.5 for <andrewmullen@example.org>; Mon, 02 Mar 2009 00:30:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:references:in-reply-to :x-enigmail-version:openpgp:content-type:content-transfer-encoding; bh=gz1SECxxsk//cXHXLWaOs+TsjPbFmCX3v25f4Qronvg=; b=XnVo2XOKH0fWdgZ6r1vokN5cAxAUKUV3X2jc41DO867wVILNuNQj2a4vOIsvTq0M85 pbbtr1OgBoxn+Yag/cqTdVlGqJgo+lWAoPH9olFRK7J6qW37KICsypBebeqAwrvr3M2I 8GYAFTVlPN6tbpgWFwAinCMUQlVELxAbnYIDw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:x-enigmail-version:openpgp:content-type :content-transfer-encoding; b=mj5LIywlC7+6dx8JmqjAV5mAq3COU2fs6oYO70GOd+OI22rkS/k51lSl6VTw5GgMby ETbloksaSiEsFyIMKnwdmNdv/D+EqP01es3tgCcHuW9ZvkL2wP1Cs1ID7ibd+emmwIcP ba0RBU4VDekjDS0CHbNh5GpEQ2Zacij1BnAUM= Received: by 83.161.120.155 with SMTP id h16mr5668297faq.94.1235982650901; Mon, 02 Mar 2009 00:30:50 -0800 (PST) Received: from ?83.161.120.155? ([83.161.120.155]) by mx.google.com with ESMTPS id 22sm10797024fkr.14.283.161.120.155.30.49 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 02 Mar 2009 00:30:50 -0800 (PST) Message-ID: <andrewmullen@example.org> Date: Mon, 02 Mar 2009 10:32:52 +0200 From: Sami Siren <andrewmullen@example.org> User-Agent: Thunderbird 83.161.120.155 (X11/20090105) MIME-Version: 1.0 To: andrewmullen@example.org Subject: Re: planning for nutch-1.0-rc1 References: <andrewmullen@example.org> <andrewmullen@example.org> In-Reply-To: <andrewmullen@example.org> X-Enigmail-Version: 0.95.6 OpenPGP: id=9445ED29 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Andrzej Bialecki wrote: > Sami Siren wrote: >> I am planning to build the first rc for nutch 1.0 at Tue 3.3.2009 >> morning (EET). There are still some issues marked as fix for 1.0 in >> Jira. Neither of the two remaining _bugs_ seems too important to me, >> actually I only count the issues assigned to developers as real >> candidates to be included in 1.0: >> >> NUTCH-578 (kubes) >> NUTCH-477 (ab) >> NUTCH-669 (siren) > > There's one Critical issue reported, related to NekoHTML (NUTCH-700). > I'm not sure what are the feature differences (pertinent to Nutch) > between 0.9.4 and 1.9.11 - perhaps downgrading is the safest course of > action. I will take care of that. > > >> I am also volunteering to push all open issues to 1.1 before starting >> the RC build on Tuesday. Any objections on the proposed procedure or >> timing? > > Sounds good. great! -- Sami Siren From nutch-devandrewmullen@example.org Mon Mar 02 09:12:45 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 59126 invoked from network); 2 Mar 2009 09:12:45 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 09:12:45 -0000 Received: (qmail 4025 invoked by uid 500); 2 Mar 2009 09:12:43 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 4000 invoked by uid 500); 2 Mar 2009 09:12:43 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 3991 invoked by uid 99); 2 Mar 2009 09:12:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 01:12:43 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 09:12:34 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id EBFDF234C4AF for <andrewmullen@example.org>; Mon, 2 Mar 2009 01:12:12 -0800 (PST) Message-ID: <21950174.1235985132965.JavaMail.jira@brutus> Date: Mon, 2 Mar 2009 01:12:12 -0800 (PST) From: "Andrzej Bialecki (JIRA)" <andrewmullen@example.org> To: andrewmullen@example.org Subject: [jira] Closed: (NUTCH-419) unavailable robots.txt kills fetch In-Reply-To: <17927486.1166964321064.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/NUTCH-419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrzej Bialecki closed NUTCH-419. ----------------------------------- Resolution: Fixed Fix Version/s: 1.0.0 Assignee: Andrzej Bialecki Fixed in rev. 749247. Thank you! > unavailable robots.txt kills fetch > ---------------------------------- > > Key: NUTCH-419 > URL: https://issues.apache.org/jira/browse/NUTCH-419 > Project: Nutch > Issue Type: Bug > Components: fetcher > Affects Versions: 0.8.1 > Environment: Fetcher is behind a squid proxy, but I am pretty sure this is irrelevant. > Nutch in local mode, running on a linux machine with 2GB RAM. > Reporter: Carsten Lehmann > Assignee: Andrzej Bialecki > Fix For: 1.0.0 > > Attachments: diffs, last_robots.txt_requests_squidlog.txt, nutch-log.txt, squid_access_log_tail1000.txt > > > I think there is another robots.txt-related problem which is not > adressed by NUTCH-344, > but also results in an aborted fetch. > I am sure that in my last fetch all 17 fetcher threads died > while they were waiting for a robots.txt-file to be delivered by a not > properly responding web server. > I looked at the squid access log, which is used by all fetch threads. > It ends with many HTTP-504-errors ("gateway timeout") caused by a > certain robots.txt url: > <....> > 1166652253.332 899427 83.161.120.155 TCP_MISS/504 1450 GET > http://gso.gbv.de/robots.txt - DIRECT/83.161.120.155 text/html > 1166652343.350 899664 83.161.120.155 TCP_MISS/504 1450 GET > http://gso.gbv.de/robots.txt - DIRECT/83.161.120.155 text/html > 1166652353.560 899871 83.161.120.155 TCP_MISS/504 1450 GET > http://gso.gbv.de/robots.txt - DIRECT/83.161.120.155 text/html > These entries mean that it takes 15 minutes before the request ends > with a timeout. > This can be calculated from the squid log, the first column is the > request time (in UTC seconds), the second column is the duration of > the request (in ms): > 900000/1000/60=15 minutes. > As far as I understand it, every time a fetch thread tries to get this > robots.txt-file the thread busy waits for the duration of the request > (15 minutes). > If this is right, then all 17 fetcher threads were caught in this trap > at the time when fetching was aborted, as there are 17 requests in > the squid log which did not timeout before the message "aborting with > 17 threads" was written to the nutch-logfile. > Setting fetcher.max.crawl.delay can not help here. > I see 296 access attempts in total concerning this robots.txt-url in > the squid log of this crawl, but fetcher.max.crawl.delay is set to 30. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From nutch-devandrewmullen@example.org Mon Mar 02 10:22:52 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 12929 invoked from network); 2 Mar 2009 10:22:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 10:22:52 -0000 Received: (qmail 73446 invoked by uid 500); 2 Mar 2009 10:22:49 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 73414 invoked by uid 500); 2 Mar 2009 10:22:49 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 73405 invoked by uid 99); 2 Mar 2009 10:22:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 02:22:49 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 10:22:34 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 12024234C4B0 for <andrewmullen@example.org>; Mon, 2 Mar 2009 02:22:13 -0800 (PST) Message-ID: <1656636931.1235989333072.JavaMail.jira@brutus> Date: Mon, 2 Mar 2009 02:22:13 -0800 (PST) From: "Sami Siren (JIRA)" <andrewmullen@example.org> To: andrewmullen@example.org Subject: [jira] Resolved: (NUTCH-700) Neko1.9.11 goes into a loop In-Reply-To: <114422886.1235126701877.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/NUTCH-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sami Siren resolved NUTCH-700. ------------------------------ Resolution: Fixed reverted to 0.9.4 > Neko1.9.11 goes into a loop > --------------------------- > > Key: NUTCH-700 > URL: https://issues.apache.org/jira/browse/NUTCH-700 > Project: Nutch > Issue Type: Bug > Affects Versions: 1.0.0 > Reporter: julien nioche > Assignee: Sami Siren > Priority: Critical > Fix For: 1.0.0 > > > Neko1.9.11 goes into a loop on some documents e.g. > http://mediacet.com/Archive/FourYorkshiremen/bb/post.htm > http://cizel.co.kr/main.php > reverting to 0.9.4 seems to fix the problem > The approach mentioned in https://issues.apache.org/jira/browse/NUTCH-696 could be a way to alleviate similar issues > PS: haven't had time to report to the Neko people yet, will do at some stage -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From nutch-devandrewmullen@example.org Mon Mar 02 12:32:47 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 61511 invoked from network); 2 Mar 2009 12:32:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 12:32:47 -0000 Received: (qmail 11241 invoked by uid 500); 2 Mar 2009 12:32:41 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 11210 invoked by uid 500); 2 Mar 2009 12:32:40 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 11201 invoked by uid 99); 2 Mar 2009 12:32:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 04:32:40 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 12:32:38 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id A0494234C4AC for <andrewmullen@example.org>; Mon, 2 Mar 2009 04:32:17 -0800 (PST) Message-ID: <484395844.1235997137654.JavaMail.jira@brutus> Date: Mon, 2 Mar 2009 04:32:17 -0800 (PST) From: "Sami Siren (JIRA)" <andrewmullen@example.org> To: andrewmullen@example.org Subject: [jira] Resolved: (NUTCH-669) Consolidate code for Fetcher and Fetcher2 In-Reply-To: <+1-706-389-7387.1228425164331.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/NUTCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sami Siren resolved NUTCH-669. ------------------------------ Resolution: Fixed replaced fetcher with fetcher2 > Consolidate code for Fetcher and Fetcher2 > ----------------------------------------- > > Key: NUTCH-669 > URL: https://issues.apache.org/jira/browse/NUTCH-669 > Project: Nutch > Issue Type: Improvement > Components: fetcher > Affects Versions: 0.9.0 > Reporter: Todd Lipcon > Assignee: Sami Siren > Fix For: 1.0.0 > > > I'd like to consolidate a lot of the common code between Fetcher and Fetcher2.java. > It seems to me like there are the following differences: > - Fetcher relies on the Protocol to obey robots.txt and crawl delay settings whereas Fetcher2 implements them itself > - Fetcher2 uses a different queueing model (queue per crawl host) to accomplish the per-host limiting without making the Protocol do it. > I've begun work on this but want to check with people on the following: > - What reason is there for Fetcher existing at all since Fetcher2 seems to be a superset of functionality? > - Is it on the road map to remove the robots/delay logic from the Http protocol and make Fetcher2's delegation of duties the standard? > - Any other improvements wanted for Fetcher while I am in and around the code? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From nutch-devandrewmullen@example.org Mon Mar 02 14:07:29 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 20951 invoked from network); 2 Mar 2009 14:07:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 14:07:29 -0000 Received: (qmail 7373 invoked by uid 500); 2 Mar 2009 14:07:27 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 7330 invoked by uid 500); 2 Mar 2009 14:07:27 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 7320 invoked by uid 99); 2 Mar 2009 14:07:27 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 06:07:27 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO getopt.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 14:07:17 +0000 Received: from [83.161.120.155] ([83.161.120.155]) (authenticated) by getopt.org (8.11.6/8.11.6) with ESMTP id n22E72h08105 for <andrewmullen@example.org>; Mon, 2 Mar 2009 08:07:03 -0600 Message-ID: <andrewmullen@example.org> Date: Mon, 02 Mar 2009 15:06:54 +0100 From: Andrzej Bialecki <andrewmullen@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20081209) MIME-Version: 1.0 To: andrewmullen@example.org Subject: Re: [jira] Resolved: (NUTCH-669) Consolidate code for Fetcher and Fetcher2 References: <484395844.1235997137654.JavaMail.jira@brutus> In-Reply-To: <484395844.1235997137654.JavaMail.jira@brutus> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Sami Siren (JIRA) wrote: > [ https://issues.apache.org/jira/browse/NUTCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] > > Sami Siren resolved NUTCH-669. > ------------------------------ > > Resolution: Fixed > > replaced fetcher with fetcher2 I'm puzzled .. it seemed the goal was to integrate Todd's patch, which effectively replaces both Fetchers. Does this mean that Todd's version was not ready, or is the current code based on Todd's version? -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: andrewmullen@example.org From nutch-devandrewmullen@example.org Mon Mar 02 14:32:23 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 40213 invoked from network); 2 Mar 2009 14:32:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 14:32:23 -0000 Received: (qmail 48095 invoked by uid 500); 2 Mar 2009 14:32:21 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 48066 invoked by uid 500); 2 Mar 2009 14:32:21 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 48057 invoked by uid 99); 2 Mar 2009 14:32:21 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 06:32:21 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of andrewmullen@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f167.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 14:32:11 +0000 Received: by bwz11 with SMTP id 11so2384094bwz.5 for <andrewmullen@example.org>; Mon, 02 Mar 2009 06:31:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:references:in-reply-to :x-enigmail-version:openpgp:content-type:content-transfer-encoding; bh=lVALSuLmF7ymvJTBE/w/Za0MdACW+plRU0Z3NEPCkOg=; b=n0s7odPwZNI3Rt18rf9jdpuFxtA60f/tvrD3NGHPcq0dK0mHwfnpccliu35OHsxtOu 98e6DeqzDUZkfD76QRTNgO1gtMgcZ51EmMUlYFTOXn/Ss8LYHmOqX25S2DI1ZEcQmevd TEzNdX3T0kvsdkics6+WT2YQqHCMii3hV6E9w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:x-enigmail-version:openpgp:content-type :content-transfer-encoding; b=sDNzkEK9oUAs1CaKJzWkTysJIySmFLMweY72Qxah0+5AbxXGgQquQ7aECcNdCUIWZR yPxScI6klg1fW53mBgIQcFnKbOM2eh/MQe/ZjJBb9a8ZeDFlDSLpyN0Q/S9O79qg/Tel ApaFQsd1w8qtmLyjpHWD0NEqZyC5pF7DKjk6M= Received: by 83.161.120.155 with SMTP id d4mr5981857faq.86.1236004311166; Mon, 02 Mar 2009 06:31:51 -0800 (PST) Received: from ?83.161.120.155? ([83.161.120.155]) by mx.google.com with ESMTPS id b17sm10586907fka.24.283.161.120.155.31.50 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 02 Mar 2009 06:31:50 -0800 (PST) Message-ID: <andrewmullen@example.org> Date: Mon, 02 Mar 2009 16:33:53 +0200 From: Sami Siren <andrewmullen@example.org> User-Agent: Thunderbird 83.161.120.155 (X11/20090105) MIME-Version: 1.0 To: andrewmullen@example.org Subject: Re: [jira] Resolved: (NUTCH-669) Consolidate code for Fetcher and Fetcher2 References: <484395844.1235997137654.JavaMail.jira@brutus> <andrewmullen@example.org> In-Reply-To: <andrewmullen@example.org> X-Enigmail-Version: 0.95.6 OpenPGP: id=9445ED29 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Andrzej Bialecki wrote: > Sami Siren (JIRA) wrote: >> [ >> https://issues.apache.org/jira/browse/NUTCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel >> ] >> >> Sami Siren resolved NUTCH-669. >> ------------------------------ >> >> Resolution: Fixed >> >> replaced fetcher with fetcher2 > > I'm puzzled .. it seemed the goal was to integrate Todd's patch, > which effectively replaces both Fetchers. Does this mean that Todd's > version was not ready, or is the current code based on Todd's version? There was no Todd's path that I could see, he never provided one even after asked multiple times, first by you at dec 2008 then dogacan jan 2009 and finally me last week. My motivation to get this fixed was, as I understood most of the developers thought too, to get rid of the burden of supporting two classes providing roughly the same piece of functionality. I opened a jira for this but closed it soon after as you told me it was a duplicate to this one. So, what I did was: replaced original Fetcher with Fetcher2. The Fetcher is still there to be improved by Todd and andrewmullen@example.org. -- Sami Siren From nutch-devandrewmullen@example.org Mon Mar 02 14:49:09 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 47879 invoked from network); 2 Mar 2009 14:49:09 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 14:49:09 -0000 Received: (qmail 67468 invoked by uid 500); 2 Mar 2009 14:49:05 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 67445 invoked by uid 500); 2 Mar 2009 14:49:05 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 67434 invoked by uid 99); 2 Mar 2009 14:49:05 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 06:49:05 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO getopt.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 14:48:55 +0000 Received: from [83.161.120.155] ([83.161.120.155]) (authenticated) by getopt.org (8.11.6/8.11.6) with ESMTP id n22Emga08769 for <andrewmullen@example.org>; Mon, 2 Mar 2009 08:48:42 -0600 Message-ID: <andrewmullen@example.org> Date: Mon, 02 Mar 2009 15:48:33 +0100 From: Andrzej Bialecki <andrewmullen@example.org> User-Agent: Thunderbird 83.161.120.155 (Windows/20081209) MIME-Version: 1.0 To: andrewmullen@example.org Subject: Re: [jira] Resolved: (NUTCH-669) Consolidate code for Fetcher and Fetcher2 References: <484395844.1235997137654.JavaMail.jira@brutus> <andrewmullen@example.org> <andrewmullen@example.org> In-Reply-To: <andrewmullen@example.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Sami Siren wrote: > Andrzej Bialecki wrote: >> Sami Siren (JIRA) wrote: >>> [ >>> https://issues.apache.org/jira/browse/NUTCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel >>> ] >>> >>> Sami Siren resolved NUTCH-669. >>> ------------------------------ >>> >>> Resolution: Fixed >>> >>> replaced fetcher with fetcher2 >> >> I'm puzzled .. it seemed the goal was to integrate Todd's patch, >> which effectively replaces both Fetchers. Does this mean that Todd's >> version was not ready, or is the current code based on Todd's version? > There was no Todd's path that I could see, he never provided one even > after asked multiple times, first by you at dec 2008 then dogacan jan > 2009 and finally me last week. > > My motivation to get this fixed was, as I understood most of the > developers thought too, to get rid of the burden of supporting two > classes providing roughly the same piece of functionality. I opened a > jira for this but closed it soon after as you told me it was a duplicate > to this one. > > So, what I did was: replaced original Fetcher with Fetcher2. The Fetcher > is still there to be improved by Todd and andrewmullen@example.org. Ok, I understand now - given the circumstances I agree this was the right thing to do. -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: andrewmullen@example.org From nutch-devandrewmullen@example.org Mon Mar 02 16:46:01 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 48289 invoked from network); 2 Mar 2009 16:46:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 16:46:01 -0000 Received: (qmail 716 invoked by uid 500); 2 Mar 2009 16:45:59 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 508 invoked by uid 500); 2 Mar 2009 16:45:58 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 498 invoked by uid 99); 2 Mar 2009 16:45:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 08:45:58 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO mrelay1.uni-hannover.de) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 16:45:49 +0000 Received: from rrzn.uni-hannover.de (rayserv.rrzn.uni-hannover.de [83.161.120.155]) by mrelay1.uni-hannover.de (8.14.2/8.14.2) with ESMTP id n22GjRTS029447 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for <andrewmullen@example.org>; Mon, 2 Mar 2009 17:45:28 +0100 Received: (from wsb@localhost) by rrzn.uni-hannover.de (8.13.8+Sun/8.13.8/Submit) id n22GjQxu029454 for andrewmullen@example.org; Mon, 2 Mar 2009 17:45:26 +0100 (CET) Date: Mon, 2 Mar 2009 17:45:26 +0100 From: Wolfgang Sander-Beuermann <andrewmullen@example.org> To: andrewmullen@example.org Subject: Job offer for Nutch-Lucene Programmer Message-ID: <andrewmullen@example.org> Reply-To: andrewmullen@example.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/83.161.120.155i X-PMX-Version: 83.161.120.1553555, Antispam-Engine: 83.161.120.1550677, Antispam-Data: 283.161.120.155422 X-Virus-Checked: Checked by ClamAV on apache.org Hi, may be I'm totally wrong here with my post. If so: please excuse. I'm posting a job offer for a Nutch-Lucene-Programmer for a job in Germany. Because good knowledge of German language is mandatory for that job, I'm may be doing my second fault - I'll post this job offer here in German: Hallo allerseits, wie ich vom BVL (Bundesamt fuer Verbraucherschutz und Lebensmittelsicherheit) in Berlin erfuhr, wird dort ein Suchmaschinen-Entwickler und Programmierer gesucht. Fuer diesen Job ist es nicht erforderlich, nach Berlin umzuziehen, sondern ein Grossteil der Arbeit kann remote gemacht werden. Der Job ist zunaechst befristet bis Ende 2009 (mit solider Aussicht - aber ohne Garantie - auf Weiterbeschaeftigung). Bezahlung nach 13 TVoeD (frueher BAT2A, je nach Lebenumstaenden ca. 3-4000,-EUR monatlich brutto). Bei Interesse an dem Job: Herrn Dr. Juergen Wallmann anrufen, Tel.-Nr. 030 / 18444 10400. Wenn jmd. im Bekanntenkreis Interesse vermutet: bitte diese Nachricht dorthin weitergeben. Best regards Wolfgang -- Dr. Wolfgang Sander-Beuermann Tel.: 0511-762-4383 Projektleiter Suchmaschinenlabor http://metager.de/suma.html Regionales Rechenzentrum fuer Niedersachsen (RRZN), Univ.Hannover From nutch-devandrewmullen@example.org Mon Mar 02 17:34:28 2009 Return-Path: <nutch-devandrewmullen@example.org> Delivered-To: andrewmullen@example.org Received: (qmail 75300 invoked from network); 2 Mar 2009 17:34:28 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Mar 2009 17:34:28 -0000 Received: (qmail 93299 invoked by uid 500); 2 Mar 2009 17:34:25 -0000 Delivered-To: andrewmullen@example.org Received: (qmail 93238 invoked by uid 500); 2 Mar 2009 17:34:25 -0000 Mailing-List: contact andrewmullen@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:andrewmullen@example.org> List-Unsubscribe: <mailto:andrewmullen@example.org> List-Post: <mailto:andrewmullen@example.org> List-Id: <nutch-dev.lucene.apache.org> Reply-To: andrewmullen@example.org Delivered-To: mailing list andrewmullen@example.org Received: (qmail 93229 invoked by uid 99); 2 Mar 2009 17:34:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 09:34:25 -0800 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of andrewmullen@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-fx0-f168.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Mar 2009 17:34:15 +0000 Received: by fxm12 with SMTP id 12so2431578fxm.5 for <andrewmullen@example.org>; Mon, 02 Mar 2009 09:33:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=ggiMkBmCixc9CLBkO2yyc34zSH5qUVoeMkEftfCSeGo=; b=qcqYwOCP5/25sKyfOM0rEskCTB63HbNHaxy4yJupZVUrNqOu0b8wcx4s7wyw10HtG+ I4T627MDi6xF5DczCS/KaKwDmgKMpECfUlUT8mZ2WZdslqtrRRYAyErGlxSOMHK0iG9m Mz4E36a0ILUXoUaFazdMvHpV1EPo0kkxv4EgE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=oP+hE58Ey02979EiWYllvIeD7NVxeOTENaqu6vllbA0MO7YiI7NnM+BNT0fLZE2sPA XiclPrUWCJohsiIPp7PyNjQk1e4d48cRG60CvNJCbqkPZNv/6LU/z+s/cXJEVQlMTiMb inONaomDaaSfeto98z90SAiRSbNhn9YFURTUo= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id v18mr3059354mul.43.1236015235184; Mon, 02 Mar 2009 09:33:55 -0800 (PST) In-Reply-To: <andrewmullen@example.org> References: <484395844.1235997137654.JavaMail.jira@brutus> <andrewmullen@example.org> <andrewmullen@example.org> <andrewmullen@example.org> Date: Mon, 2 Mar 2009 12:33:55 -0500 Message-ID: <andrewmullen@example.org> Subject: Re: [jira] Resolved: (NUTCH-669) Consolidate code for Fetcher and Fetcher2 From: Todd Lipcon <andrewmullen@example.org> To: andrewmullen@example.org Content-Type: multipart/alternative; boundary=0016367ed5fd07ba0a046426397c X-Virus-Checked: Checked by ClamAV on apache.org --0016367ed5fd07ba0a046426397c Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hey guys, Sorry for the non-responsiveness here. I recently left my old employment and have been packing for a cross-country move. I agree that for 1.0 the best bet is what Sami has done. The code that I was working on is available here: http://github.com/toddlipcon/nutch/tree/nutch-669 But it is not production ready - notably there's a problem whereby it runs out of memory even with a reasonably large heap. I'm not sure if I'll be able to complete working on it, given the cluster (and workload) I was using to test were from my old job, but I'm happy to provide any assistance understanding the work I began if you'd like to try to integrate it for 1.1 -Todd On Mon, Mar 2, 2009 at 9:48 AM, Andrzej Bialecki <andrewmullen@example.org> wrote: > Sami Siren wrote: > >> Andrzej Bialecki wrote: >> >>> Sami Siren (JIRA) wrote: >>> >>>> [ >>>> https://issues.apache.org/jira/browse/NUTCH-669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] >>>> >>>> Sami Siren resolved NUTCH-669. >>>> ------------------------------ >>>> >>>> Resolution: Fixed >>>> >>>> replaced fetcher with fetcher2 >>>> >>> >>> I'm puzzled .. it seemed the goal was to integrate Todd's patch, which >>> effectively replaces both Fetchers. Does this mean that Todd's version was >>> not ready, or is the current code based on Todd's version? >>> >> There was no Todd's path that I could see, he never provided one even >> after asked multiple times, first by you at dec 2008 then dogacan jan 2009 >> and finally me last week. >> >> My motivation to get this fixed was, as I understood most of the >> developers thought too, to get rid of the burden of supporting two classes >> providing roughly the same piece of functionality. I opened a jira for this >> but closed it soon after as you told me it was a duplicate to this one. >> >> So, what I did was: replaced original Fetcher with Fetcher2. The Fetcher >> is still there to be improved by Todd and andrewmullen@example.org. >> > > Ok, I understand now - given the circumstances I agree this was the right > thing to do. > > > -- > Best regards, > Andrzej Bialecki <>< > ___. ___ ___ ___ _ _ __________________________________ > [__ || __|__/|__||\/| Information Retrieval, Semantic Web > ___|||__|| \| || | Embedded Unix, System Integration > http://www.sigram.com Contact: andrewmullen@example.org > > --0016367ed5fd07ba0a046426397c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hey guys,<div><br></div><div>Sorry for the non-responsiveness here. I recen= tly left my old employment and have been packing for a cross-country move.<= /div><div><br></div><div>I agree that for 1.0 the best bet is what Sami has= done. The code that I was working on is available here:</div> <div><br></div><div><a href=3D"http://github.com/toddlipcon/nutch/tree/nutc= h-669">http://github.com/toddlipcon/nutch/tree/nutch-669</a></div><div><br>= </div><div>But it is not production ready - notably there&#39;s a problem w= hereby it runs out of memory even with a reasonably large heap.</div> <div><br></div><div>I&#39;m not sure if I&#39;ll be able to complete workin= g on it, given the cluster (and workload) I was using to test were from my = old job, but I&#39;m happy to provide any assistance understanding the work= I began if you&#39;d like to try to integrate it for 1.1</div> <div><br></div><div>-Todd<br><br><div class=3D"gmail_quote">On Mon, Mar 2, = 2009 at 9:48 AM, Andrzej Bialecki <span dir=3D"ltr">&lt;<a href=3D"mailto:a= andrewmullen@example.org">andrewmullen@example.org</a>&gt;</span> wrote:<br><blockquote class=3D"g= mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l= eft:1ex;"> <div><div></div><div class=3D"Wj3C7c">Sami Siren wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Andrzej Bialecki wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Sami Siren (JIRA) wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> =A0 =A0 [ <a href=3D"https://issues.apache.org/jira/browse/NUTCH-669?page= =3Dcom.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel" target=3D"= _blank">https://issues.apache.org/jira/browse/NUTCH-669?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:all-tabpanel</a> ]<br> <br> Sami Siren resolved NUTCH-669.<br> ------------------------------<br> <br> =A0 =A0Resolution: Fixed<br>
{'source': 'Apache Software Foundation Public Mail Archives'}