From nutch-userhvillarreal@example.net Wed Oct 01 07:29:02 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 80683 invoked from network); 1 Oct 2008 07:29:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 07:29:02 -0000 Received: (qmail 65289 invoked by uid 500); 1 Oct 2008 07:28:53 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 65249 invoked by uid 500); 1 Oct 2008 07:28:53 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 65238 invoked by uid 99); 1 Oct 2008 07:28:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 00:28:53 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rn-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 07:27:52 +0000 Received: by rn-out-0910.google.com with SMTP id j71so188935rne.4 for <hvillarreal@example.net>; Wed, 01 Oct 2008 00:28:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=E30PBxEIdMGZubzsPl/4z18VMAUT9PVI5wo4c3RtFyc=; b=ayWP/QK3p4fkcW0y3RrULL8bSPConl7dtHbCj6h+vRKa4eCkHePxA/JJYjRccCA2dP NUoYyAS9GyzAu6EyG6vzvXOo5tM3qAaAAjQbu61q1xRjeQ53nGbHKJmN9nSYj6QLAdPJ uxBQwckVoAZkTIprTUHIVUMKiPcU9nO/nEVLg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=tikidcUD6DkePmLY6umlGNEfxP63o/RHM8aUK2IQg0z+ZQ1ew1GiPsOarrWUL2p/gB HGqYSrMAYPXJutcgSxhNAKt6RZ2vG8sa34cJCuc1FrGOfECmLSSd3UK0BumOyDgX11Ha nwt4DQyu08Ngh5nx4vXKe8bjhowu0RvqCt/a8= Received: by 83.161.120.155 with SMTP id c20mr11594953ybo.6.1222846106446; Wed, 01 Oct 2008 00:28:26 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 00:28:26 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 10:28:26 +0300 From: "=?UTF-8?Q?Do=C4=9Facan_G=C3=BCney?=" <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: Using S3 with Hadoop/Nutch In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org T24gVHVlLCBTZXAgMzAsIDIwMDggYXQgMTE6NTIgUE0sIEtldmluIE1hY0RvbmFsZCA8a2V2aW5A aGF1dGVzZWN1cmUuY29tPiB3cm90ZToKPiBEb2VzIGFueW9uZSBoYXZlIGV4cGVyaWVuY2UgY29u ZmlndXJpbmcgSGFkb29wIHRvIHVzZSBTMyBmb3IgdXNpbmcgbnV0Y2g/IEkKPiB0cmllZCBtb2Rp ZnlpbmcgbXkgaGFkb29wLXNpdGUueG1sIGNvbmZpZ3VyYXRpb24gZmlsZSBhbmQgaXQgbG9va3Mg bGlrZQo+IEhhZG9vcCBpcyB0cnlpbmcgdG8gdXNlIFMzLiBCdXQgSSB0aGluayB3aGF0J3MgaGFw cGVuaW5nIGlzIHRoYXQsIG9uY2UKPiBjb25maWd1cmVkIHRvIHVzZSBTMywgSGFkb29wIGlzIE9O TFkgbG9va2luZyBhdCBTMyBmb3IgYWxsIGZpbGVzLiBJdCdzCj4gdHJ5aW5nIHRvIGZpbmQgYSAv dG1wIGZvbGRlciB0aGVyZSwgZm9yIGV4YW1wbGUuIEFuZCB3aGVuIHJ1bm5pbmcgYSBjcmF3bAo+ IEhhZG9vcCBpcyBsb29raW5nIHRvIFMzIHRvIGZpbmQgdGhlIHNlZWQgdXJscyBmb2xkZXIuIEFy ZSB0aGVyZSBzdGVwcyB0aGF0Cj4gbmVlZCB0byBoYXBwZW4gdG8gcHJlcGFyZSBhbiBTMyBidWNr ZXQgZm9yIHVzZSBieSBIYWRvb3Agc28gdGhhdCBhIG51dGNoCj4gY3Jhd2wgY2FuIGhhcHBlbj8K CklmIHlvdSB3YW50IHRvIHBhc3MgcGF0aHMgZnJvbSBvdGhlciBmaWxlc3lzdGVtcyBJIHRoaW5r IHlvdSBjYW4gZG8Kc29tZXRoaW5nIGxpa2U6CgpiaW4vbnV0Y2ggaW5qZWN0IGNyYXdsL2NyYXds ZGIgaGRmczovL21hY2hpbmU6MTAwMDAvLi4uLi4KCj4gS2V2aW4KPgoKCgotLSAKRG/En2FjYW4g R8O8bmV5Cg== From nutch-userhvillarreal@example.net Wed Oct 01 07:34:35 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 83035 invoked from network); 1 Oct 2008 07:34:35 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 07:34:35 -0000 Received: (qmail 67694 invoked by uid 500); 1 Oct 2008 07:34:31 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 67665 invoked by uid 500); 1 Oct 2008 07:34:31 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 67654 invoked by uid 99); 1 Oct 2008 07:34:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 00:34:31 -0700 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f13.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 07:33:29 +0000 Received: by gxk6 with SMTP id 6so12877732gxk.5 for <hvillarreal@example.net>; Wed, 01 Oct 2008 00:33:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=vBQ+t+wo3pW+ed1/AA31N00J3SE95HP32TnYUL5NQPI=; b=CBpsWGXTV84bifwLcqeU13SimhzsumWirgEzUmidlj4eqzIz459AuSbzxSZyLhB9OX Tc+CguvYc+etNK3H8t8B1aJxyBrXuR5ZvMbvolRQCMoO/nB7lxDvoFQ71iC4guEJpneR 91SGcP2M6dEbJu2r/NVPPxmPOeVaz/2Fjvsnc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=Q1Jq4nYMTiun9edNSCf4W3NYTAZ20On8Z34DZeBgBktkt1F/+K6EX0jpNeg5n4aYtX 7eDF37D6Cvw8Qj/wuCnIHHfhdQHYTxCVRpn+5Bw+N63bFyQMpN48Z1kUzbcJt/DapPmb CR1IYCJg4lLcDuuLZVk6/pzud9aTYocG5MZ4A= Received: by 83.161.120.155 with SMTP id p1mr11566391ybi.61.1222846383822; Wed, 01 Oct 2008 00:33:03 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 00:33:03 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 10:33:03 +0300 From: "=?UTF-8?Q?Do=C4=9Facan_G=C3=BCney?=" <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: Please help with QueryFilter configuration In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org T24gVHVlLCBTZXAgMzAsIDIwMDggYXQgNDoyNSBQTSwgc3R1ZGVudF90IDxjY2hvdUBjc2NpbmZv LmNvbT4gd3JvdGU6Cj4KPiBIaSEgTnV0Y2ggRXhwZXJ0cywKPgo+IFBsZWFzZSBoZWxwIG1lIHdp dGggdGhlIGZvbGxvd2luZyB0d28gcXVlc3Rpb25zLgo+Cj4gMS4gSXQgc2VlbXMgdG8gbWUgdGhl IEx1Y2VuZSBRdWVyeUZpbHRlciBmdW5jdGlvbnMgZGlmZmVyZW50bHkgZnJvbSBOdXRjaAo+IFF1 ZXJ5RmlsdGVyLiBMdWNlbmUgUXVlcnlGaWx0ZXIgZmlsdGVycyBvdXQgaGl0cyBhbmQgTnV0Y2gg UXVlcnlGaWx0ZXIKPiBtb2RpZmllcyB0aGUgdW5kZXJseWluZyBRdWVyeS4gSXMgbXkgdW5kZXJz dGFuZGluZyBjb3JyZWN0Pwo+CgpZZXMuCgo+IDIuIEhvdyBkbyBJIGNvbmZpZ3VyZSBzbyBOdXRj aCBrbm93cyB0byBzZW5kIHRoZSBxdWVyeSB0byB0aGUgdW5kZXJseWluZwo+IFF1ZXJ5RmlsdGVy IGJlZm9yZSBpc3N1aW5nIGl0cyBzZWFyY2g/IEkgaGF2ZSBhbGwgdGhlIHBsdWdpbnMgaW4gdGhl IHJpZ2h0Cj4gZGlyZWN0b3JpZXMgYW5kIHNlZW4gaW4gbXkgbG9nIGZpbGUgdGhhdCB0aGVzZSBw bHVnaW5zIGFyZSBsb2FkZWQuIEJ1dCB3aGVuCj4gSSBpc3N1ZSBteSBzZWFyY2gsIEkgZGlkbid0 IHNlZSBsb2cgZW50cmllcyAoSU5GTykgaW4gdGhlIFF1ZXJ5RmlsdGVyIGNvZGUKPiBzcGl0dGVk IG91dC4gSSBtdXN0IGJlIG1pc3NpbmcgYSBjb25maWd1cmF0aW9uIGVudHJ5LiBJIHRoaW5rIHRo aXMgZW50cnkKPiBtdXN0IGJlIHB1dCBpbiBudXRjaC1kZWZhdWx0LnhtbC4gQnV0IHdoYXQgYXJl IHRoZSBlbnRyaWVzIGFuZCBjYW4gSSBsb2FkCj4gdGhlIGNvbmZpZ3VyYXRpb24gZmlsZSBhbnkg dGltZSBJIHdhbnQ/IEl0IHNlZW1zIHRvIG1lIHdoZW4gSSBjb25zdHJ1Y3QgYQo+IE51dGNoIFF1 ZXJ5IGl0IGF0dGVtcHRzIHRvIGxvYWQgdGhlIGNvbmZpZ3VyYXRpb24gZHluYW1pY2FsbHkuCj4K CkhhdmUgeW91IG1vZGlmaWVkIHlvdXIgcGx1Z2luLnhtbD8KCkZvciBleGFtcGxlLCBxdWVyeS1i YXNpYydzIGxvb2tzIGxpa2UgdGhpczoKCi4uLi4uCiAgICAgIDxpbXBsZW1lbnRhdGlvbiBpZD0i QmFzaWNRdWVyeUZpbHRlciIKICAgICAgICAgICAgICAgICAgICAgIGNsYXNzPSJvcmcuYXBhY2hl Lm51dGNoLnNlYXJjaGVyLmJhc2ljLkJhc2ljUXVlcnlGaWx0ZXIiPgogICAgICAgIDxwYXJhbWV0 ZXIgbmFtZT0iZmllbGRzIiB2YWx1ZT0iREVGQVVMVCIvPgogICAgICA8L2ltcGxlbWVudGF0aW9u PgouLi4uCgpBbmQgcXVlcnktdXJsIGxpa2UgdGhpczoKCi4uLi4uCjxwYXJhbWV0ZXIgbmFtZT0i ZmllbGRzIiB2YWx1ZT0idXJsIi8+Ci4uLi4uCgpBIHF1ZXJ5IGZpbHRlciBvbmx5IGdldHMgY2Fs bGVkIGlmIE51dGNoJ3MgcXVlcnkgcHJvY2Vzc2luZyBzeXN0ZW0KZGV0ZXJtaW5lcyB0aGF0IGl0 IGhhcyBmaWVsZHMgdG8gcGFyc2UuClNvLCBxdWVyeS1iYXNpYyBpcyBjYWxsZWQgb24gZXZlcnl0 aGluZyB3aXRob3V0IGEgc3BlY2lmaWMgZmllbGQKKHRodXMsIERFRkFVTFQpIGFuZCBxdWVyeS11 cmwgaXMgY2FsbGVkCmlmIHF1ZXJ5IGhhcyAidXJsOi4uLi4uLiIKCgo+IFRoYW5rcyB2ZXJ5IG11 Y2ggaW4gYWR2YW5jZSBmb3IgYW55IGluc2lnaHRzIQo+Cj4gLXN0dWRlbnRfdAo+Cj4KPgo+IC0t Cj4gVmlldyB0aGlzIG1lc3NhZ2UgaW4gY29udGV4dDogaHR0cDovL3d3dy5uYWJibGUuY29tL1Bs ZWFzZS1oZWxwLXdpdGgtUXVlcnlGaWx0ZXItY29uZmlndXJhdGlvbi10cDE5NzQyMDcycDE5NzQy MDcyLmh0bWwKPiBTZW50IGZyb20gdGhlIE51dGNoIC0gVXNlciBtYWlsaW5nIGxpc3QgYXJjaGl2 ZSBhdCBOYWJibGUuY29tLgo+Cj4KCgoKLS0gCkRvxJ9hY2FuIEfDvG5leQo= From nutch-userhvillarreal@example.net Wed Oct 01 07:35:26 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 83331 invoked from network); 1 Oct 2008 07:35:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 07:35:26 -0000 Received: (qmail 69502 invoked by uid 500); 1 Oct 2008 07:35:17 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 69486 invoked by uid 500); 1 Oct 2008 07:35:17 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 69475 invoked by uid 99); 1 Oct 2008 07:35:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 00:35:17 -0700 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wr-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 07:34:16 +0000 Received: by wr-out-0506.google.com with SMTP id c30so178706wra.21 for <hvillarreal@example.net>; Wed, 01 Oct 2008 00:34:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=fq7C5Ns236fulrKBTD/N9T9tPTwLTr1A0WZq/DiG8nY=; b=sl6uZESQK8M8Yt+kRNG4BrnaopHZy+ReTH5cPm03y/LHbf7XPuFHwlRv0zMfSW4C1e pqsiXetelDTiu3JWOa9jSYrseMVbMh0LbVm9GTkFHyUrAhMkja6nqa6ZAfi3OMkXnvAO fYYUcPZ9PRJI0rUgEueCNowHSix50ncD5Ycag= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=RUbw6TSHNvPW7oin15M+Stw2i+UQ0YSyeokufQUnoNEFwj/aGjrKeEiTqRRa24RhXp yeeLJNIb/OtvJmC8uhcJiIXYCY62ABUS2d2Btc5Q5a5pL/9QYc978N/CTnpmAI8CstgV I0Rva4wkf4BQ4Nv8EY2G7vA7TVC3IBrHNdQPM= Received: by 83.161.120.155 with SMTP id s15mr11529076ybb.219.1222846490681; Wed, 01 Oct 2008 00:34:50 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 00:34:50 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 10:34:50 +0300 From: "=?UTF-8?Q?Do=C4=9Facan_G=C3=BCney?=" <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: How to create index using indexes ? In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org T24gVHVlLCBTZXAgMzAsIDIwMDggYXQgNDowMSBBTSwgdXNlcmxpdGUgPGRvcmVtZXNhQHlhaG9v LmNvbT4gd3JvdGU6Cj4KPiBIaSwKPgo+ICBJIHVzZWQgbnV0Y2ggY3Jhd2wgY29tbWFuZCB0byBj cmF3bCBteSBsb2NhbCBmaWxlc3lzdGVtLiBUaGUgY3Jhd2wgY3Jhc2hlZAo+IGF0IHRoZSBlbmQg LSBvdXQgb2Ygc3BhY2UgZXJyb3IuIEN1cnJlbnRseSwgdGhlIHJlYWRkYiAtc3RhdHMgc2hvd3Mg Y29ycmVjdAo+IG51bWJlciBvZiBmaWxlcy4gQWxzbywgZGlyZWN0b3JpZXMgY3Jhd2xkYiwgbGlu a2RiLCBpbmRleGVzIGFuZCBzZWdtZW50cyBhcmUKPiBwcmVzZW50LiBIb3dldmVyIHRoZSAiaW5k ZXgiIGRpcmVjdG9yeSBpcyBub3QgdGhlcmUuCj4KPiBIb3cgY2FuIEkgY3JlYXRlICJpbmRleCIg PyBJIHRyaWVkIHRoZSBudXRjaCBtZXJnZSBjb21tYW5kIHRvIGNyZWF0ZSBpbmRleAo+IHVzaW5n IHRoZSBpbmRleGVzIGRpcmVjdG9yeSAod2hpY2ggY29udGFpbnMgYSBzaW5nbGUgcGFydC0wMDAw MCBkaXJlY3RvcnkpLAo+IGJ1dCBpdCBmYWlscyBnaXZpbmcgIkluZGV4TWVyZ2VyOiBqYXZhLmlv LklPRXhjZXB0aW9uOiByZWFkIHBhc3QgRU9GIiAuCj4KPgoKWW91IGNhbiB0cnkgaW5kZXhpbmcg YWdhaW46CgpiaW4vbnV0Y2ggaW5kZXggPGluZGV4PiA8Y3Jhd2xkYj4gPGxpbmtkYj4gPHNlZ21l bnQxPiA8c2VnbWVudDI+Li4uLi4KCj4KPgo+IC0tCj4gVmlldyB0aGlzIG1lc3NhZ2UgaW4gY29u dGV4dDogaHR0cDovL3d3dy5uYWJibGUuY29tL0hvdy10by1jcmVhdGUtaW5kZXgtdXNpbmctaW5k ZXhlcy0tLXRwMTk3MzQ0OTBwMTk3MzQ0OTAuaHRtbAo+IFNlbnQgZnJvbSB0aGUgTnV0Y2ggLSBV c2VyIG1haWxpbmcgbGlzdCBhcmNoaXZlIGF0IE5hYmJsZS5jb20uCj4KPgoKCgotLSAKRG/En2Fj YW4gR8O8bmV5Cg== From nutch-userhvillarreal@example.net Wed Oct 01 07:38:19 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 84702 invoked from network); 1 Oct 2008 07:38:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 07:38:18 -0000 Received: (qmail 72784 invoked by uid 500); 1 Oct 2008 07:38:13 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 72758 invoked by uid 500); 1 Oct 2008 07:38:13 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 72747 invoked by uid 99); 1 Oct 2008 07:38:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 00:38:13 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f13.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 07:37:12 +0000 Received: by gxk6 with SMTP id 6so12879315gxk.5 for <hvillarreal@example.net>; Wed, 01 Oct 2008 00:36:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=uCUjSvq/gRgBZ3hlspsHG9VuNIHo1dlpsDR/8VJsMy8=; b=jpyvYwQV3N4qviaqnWN/+AnR9Njmt3htrnk9Fi2Iq/1tfbd0NcdsI8YoGotGu1jyuT T4ZXI9KdR8d2FaQbQ2TjT1k4fsc93kywNhyMdYqq0XHtqrXLtQhtbLWR8WTFuBAYSImV 12z/6BAJ9vgMV70fVblBjo1jgSOWeae8dFc6g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=QJRKNWiQbSv4et27x2MxTHu/zwQkrWA8uNa7EGqye4RHXyaCThkoEjx6ciNFgNrHma b4AyWUU1LDL0zbeNNTQBi1mxo5zFrpjTvZzkcyhizzfdxFjPQBf/EV97xmPYw+FPmi9D RubM7tSh6PRFpaDK+ypY/v+98Hz18USL8rq6c= Received: by 83.161.120.155 with SMTP id o17mr11532742ybi.210.1222846606903; Wed, 01 Oct 2008 00:36:46 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 00:36:46 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 10:36:46 +0300 From: "=?UTF-8?Q?Do=C4=9Facan_G=C3=BCney?=" <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: Dumping raw html and javascript In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org T24gTW9uLCBTZXAgMjksIDIwMDggYXQgOToxOSBQTSwgS2V2aW4gTWFjRG9uYWxkIDxrZXZpbkBo YXV0ZXNlY3VyZS5jb20+IHdyb3RlOgo+IE9uY2UgSSBoYXZlIGRvbmUgYSBjcmF3bCBJIGhhdmUg YSBuZWVkIHRvIHBhc3MgYWxsIG9mIHRoZSByYXcgSFRNTCBhbmQKPiBqYXZhc2NyaXB0IHRoYXQg aGFzIGJlZW4gZmV0Y2hlZCB0aHJvdWdoIGEgY3VzdG9tIHBhcnNlci4gRHVyaW5nIGEgZmV0Y2gK PiBkb2VzIG51dGNoIHN0b3JlIGFsbCBvZiB0aGUgcmF3IGNvbnRlbnQgaW5jbHVkaW5nIEhUTUwg dGFncyBvbiBkaXNrPwoKWWVzLCBpZiB5b3UgaGF2ZSBmZXRjaGVyLnN0b3JlLmNvbnRlbnQgc2V0 IHRvIHRydWUgKHdoaWNoIGlzIHRydWUgYnkgZGVmYXVsdCkuCgpSYXcgY29udGVudCBvZiBhIHBh Z2Ugd2lsbCBiZSBzYXZlZCB1bmRlciA8c2VnbWVudD4vY29udGVudCBkaXJlY3RvcnkuClRvIHJl YWNoIGEgcGFydGljdWxhciBjb250ZW50LCB5b3UgbWF5IHRyeSB0aGlzCgpiaW4vbnV0Y2ggcmVh ZHNlZyAtZ2V0IDxzZWdtZW50PiA8dXJsPiAtbm9wYXJzZSAtbm9wYXJzZWRhdGEgLW5vZmV0Y2gK LW5vZ2VuZXJhdGUgLW5vcGFyc2V0ZXh0Cgo+IFRoYW5rcwo+Cj4gS2V2aW4KPgoKCgotLSAKRG/E n2FjYW4gR8O8bmV5Cg== From nutch-userhvillarreal@example.net Wed Oct 01 07:51:07 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 92507 invoked from network); 1 Oct 2008 07:51:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 07:51:07 -0000 Received: (qmail 90094 invoked by uid 500); 1 Oct 2008 07:51:02 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 90059 invoked by uid 500); 1 Oct 2008 07:51:02 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 90048 invoked by uid 99); 1 Oct 2008 07:51:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 00:51:02 -0700 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f13.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 07:49:57 +0000 Received: by gxk6 with SMTP id 6so12884542gxk.5 for <hvillarreal@example.net>; Wed, 01 Oct 2008 00:49:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=J16Mv2tN10Jj/nnxoScWOOdDyYEnRnFJhEIMbcnr5bM=; b=IJJW1ulqLApuLvHgQvbGd5NUFBxNJrPW/xfxmrC4QxWslp0Zb1KTj3wmTUYudfm7gy jE9vwD1nFzb4Cewb5yqBv9FgDVYfZER9ZdFWi6TygkmVlP0pHoJJFxxoEfOSM3ui2OSU kDrYlKhRQTtaN+qdxiuALD9nFivdRyC61Cx7k= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=wfhcTIpjDMMins3GPgQ6ngq+7/Xxmk0ftPq5VFjQRfvEdVOM6/A40hTz2Z20l6V1NE vO8Q/QLrL73ki0QeicwT/PYdksHFYFxZuV/701+tqYyloND0aUs9QS+VOhW3/SSnkbh3 kA864BatieTdFbA+4QuifNqtTHOTdKfAKETj4= Received: by 83.161.120.155 with SMTP id k9mr11610091ybd.44.1222847371263; Wed, 01 Oct 2008 00:49:31 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 00:49:31 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 10:49:31 +0300 From: "=?UTF-8?Q?Do=C4=9Facan_G=C3=BCney?=" <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: Ignoring a url in the crawl In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org T24gTW9uLCBTZXAgMjksIDIwMDggYXQgOToxNyBQTSwgc2FuZ2VldCA8c3Jpbm9va2FsYUBnbWFp bC5jb20+IHdyb3RlOgo+Cj4gSSdtIGhhdmluZyBhIGhhcmQgdGltZSB0cnlpbmcgdG8gYXZvaWQg Y3Jhd2xpbmcgYSBwYXJ0aWN1bGFyIHVybC4KPiBJbiByZWdleC11cmxmaWx0ZXIudHh0IEkgYWRk ZWQgdGhlIGZvbGxvd2luZyB0byBpZ25vcmUgaXQuCj4gLV5odHRwOi8vKFthLXowLTldKlwuKSpi aGVqYWNyeS5jb20vZm9ydW1zLwo+Cj4gVGhpcyB1cmwgaXMgbm90IGluIHRoZSBsaXN0IGluIG15 IHVybHMgZGlyZWN0b3J5LiBJIGFsc28gaGF2ZQo+ICdkYi5pZ25vcmUuZXh0ZXJuYWwubGlua3Mn IHNldCB0byAndHJ1ZScuCj4KPiBIb3dldmVyLCBJIHN0aWxsIHNlZSB0aGUgZm9sbG93aW5nIGR1 cmluZyB0aGUgY3Jhd2wKPgo+IGZldGNoaW5nCj4gaHR0cDovL3d3dy5iaGVqYWNyeS5jb20vZm9y dW1zL21lbWJlcmxpc3QucGhwP21vZGU9dmlld3Byb2ZpbGUmdT0yNzc0Cj4gZmV0Y2hpbmcgaHR0 cDovL3d3dy5iaGVqYWNyeS5jb20vZm9ydW1zL21lbWJlcmxpc3QucGhwP21vZGU9dmlld3Byb2Zp bGUmdT05Ngo+Cj4gSG93IGRvIEkgaWdub3JlIHRoZXNlIHVybHM/CgpUcnkgcnVubmluZwpiaW4v bnV0Y2ggcGx1Z2luIHVybGZpbHRlci1yZWdleCBvcmcuYXBhY2hlLm51dGNoLnVybGZpbHRlci5y ZWdleC5SZWdleFVSTEZpbHRlcgoKVGhlbiBzaW1wbHkgdHlwZSB5b3VyIHVybC4gSWYgYSB1cmwg aXMgZmlsdGVyZWQsIGl0IHdpbGwgYmUgb3V0cHV0CmJhY2sgd2l0aCBhICItIiBhdCB0aGUgYmVn aW5uaW5nLgoKKFlvdSB3aWxsIG5lZWQgdGhlIHBhdGNoIGZyb20gTlVUQ0gtNjU0IC4gT3Igd2Fp dCBhIGNvdXBsZSBvZiBob3VycwphbmQgSSB3aWxsIGNvbW1pdCBpdCkKCj4gLS0KPiBWaWV3IHRo aXMgbWVzc2FnZSBpbiBjb250ZXh0OiBodHRwOi8vd3d3Lm5hYmJsZS5jb20vSWdub3JpbmctYS11 cmwtaW4tdGhlLWNyYXdsLXRwMTk3MjkwMzFwMTk3MjkwMzEuaHRtbAo+IFNlbnQgZnJvbSB0aGUg TnV0Y2ggLSBVc2VyIG1haWxpbmcgbGlzdCBhcmNoaXZlIGF0IE5hYmJsZS5jb20uCj4KPgoKCgot LSAKRG/En2FjYW4gR8O8bmV5Cg== From nutch-userhvillarreal@example.net Wed Oct 01 13:35:38 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 86160 invoked from network); 1 Oct 2008 13:35:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 13:35:38 -0000 Received: (qmail 82999 invoked by uid 500); 1 Oct 2008 13:35:36 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 82069 invoked by uid 500); 1 Oct 2008 13:35:33 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 82058 invoked by uid 99); 1 Oct 2008 13:35:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 06:35:33 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f13.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 13:34:32 +0000 Received: by gxk6 with SMTP id 6so108520gxk.5 for <hvillarreal@example.net>; Wed, 01 Oct 2008 06:35:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:mime-version:content-type:content-transfer-encoding :content-disposition:x-google-sender-auth; bh=e4k/ObBSzcZhZ+RNxWF8D5SpvmwcR/0UDnXM4T2ph/E=; b=bzjHy7Q0gIHR+Bwwo9T8DzJFc1leh0t5dadnHeKNfYXmtiwGgIpR2h7HvRag+nxLA6 AH/lUMOS3p/DFcxcJeJDvZTJ9HcpZbFleP1KZ5b4szHrl4z2XfSDzsCJXaWF7Vvv8vTp 24zxeHthZvwumsTaJNlZ+oP+w8wNLQc/CdC78= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition:x-google-sender-auth; b=m3EzbBQf8TFqLlw7LNUkmLofhSP1ayzxGmqSGJXugxQk/MqaGmN+I6Y39xPg82ZBrc 98oo5KvdkF+8AmksE/vhjhuJ/TZw/jKmn/VVX3aPyErOgob/Y86hPvzdhkTumZURlEYh 2ILptRBG5pBnyezin5Q5QUQJqxWpOYtd/HCFg= Received: by 83.161.120.155 with SMTP id u21mr1144524ybj.23.1222868106697; Wed, 01 Oct 2008 06:35:06 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 06:35:06 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 09:35:06 -0400 From: "Yoav Shapira" <hvillarreal@example.net> Sender: hvillarreal@example.net To: hvillarreal@example.net Subject: How do I crawl a site with a cookie for authentication? MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Google-Sender-Auth: 1489820b2a99d6f3 X-Virus-Checked: Checked by ClamAV on apache.org Hi, I would like to use Nutch to crawl and index an intranet web site for internal use. The site requires authentication, and stores the credentials in a cookie. I've got a valid login and I have the cookie saved, no problem. How do I tell Nutch to use it? I did some research online before asking, but unfortunately I couldn't find a step-by-step answer for a newbie like myself. I see there's an http-client plugin that can support some authentication. Is that what I should use for cookies? If so, how do I configure it? Or is there something else I should be doing? If the documentation / answer exists, sorry for the hassle and please just point me to it ;) -- Thanks, Yoav From nutch-userhvillarreal@example.net Wed Oct 01 14:08:53 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 17105 invoked from network); 1 Oct 2008 14:08:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 14:08:52 -0000 Received: (qmail 75529 invoked by uid 500); 1 Oct 2008 14:08:47 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 75493 invoked by uid 500); 1 Oct 2008 14:08:47 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 75482 invoked by uid 99); 1 Oct 2008 14:08:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 07:08:47 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO el-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 14:07:46 +0000 Received: by el-out-1112.google.com with SMTP id p32so90868elf.14 for <hvillarreal@example.net>; Wed, 01 Oct 2008 07:08:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=yFPHB5R6Qc7HfoePcAWdWw3Bc9koVNeOUlIbJyWH0nM=; b=T3EiYDYjYOyWo0Akb1qQdo0+/BKnWIaDVfYLMQV2BM3OmEsPbVgjBK17ucWDBVZ1Y3 gQ2uSAthE9gbidBWwPbg58VSmXjJRxvGjTMiHFeh/JJM0B57YLazhaQPhm67m5EY6ksR EmsT8X1t0qCwpuETHvh2IwLYu1C1lLJLnqiTw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=fFGtXDWz54ITGNG5BR6uwZwKQ+EAMA4pXINPlXD18z9F1E0otEGX32AUWWoA5bsE03 nCqXGVzTPFoLTi9KE2ofAq6M3k4BMpKZRxs4qRF45xO/9DkvZhdeD5NHlDl/Pm+V3RO5 N7pSh54Tb/7s6awwW2zkHWOmETNWszvFGPrGc= Received: by 83.161.120.155 with SMTP id w18mr4124730ybj.140.1222870100212; Wed, 01 Oct 2008 07:08:20 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 07:08:20 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 17:08:20 +0300 From: "=?UTF-8?Q?Do=C4=9Facan_G=C3=BCney?=" <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: How do I crawl a site with a cookie for authentication? In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org T24gV2VkLCBPY3QgMSwgMjAwOCBhdCA0OjM1IFBNLCBZb2F2IFNoYXBpcmEgPHlvYXZzQHlvYXZz aGFwaXJhLmNvbT4gd3JvdGU6Cj4gSGksCj4KPiBJIHdvdWxkIGxpa2UgdG8gdXNlIE51dGNoIHRv IGNyYXdsIGFuZCBpbmRleCBhbiBpbnRyYW5ldCB3ZWIgc2l0ZSBmb3IKPiBpbnRlcm5hbCB1c2Uu ICBUaGUgc2l0ZSByZXF1aXJlcyBhdXRoZW50aWNhdGlvbiwgYW5kIHN0b3JlcyB0aGUKPiBjcmVk ZW50aWFscyBpbiBhIGNvb2tpZS4gIEkndmUgZ290IGEgdmFsaWQgbG9naW4gYW5kIEkgaGF2ZSB0 aGUgY29va2llCj4gc2F2ZWQsIG5vIHByb2JsZW0uICBIb3cgZG8gSSB0ZWxsIE51dGNoIHRvIHVz ZSBpdD8KPgo+IEkgZGlkIHNvbWUgcmVzZWFyY2ggb25saW5lIGJlZm9yZSBhc2tpbmcsIGJ1dCB1 bmZvcnR1bmF0ZWx5IEkgY291bGRuJ3QKPiBmaW5kIGEgc3RlcC1ieS1zdGVwIGFuc3dlciBmb3Ig YSBuZXdiaWUgbGlrZSBteXNlbGYuICBJIHNlZSB0aGVyZSdzIGFuCj4gaHR0cC1jbGllbnQgcGx1 Z2luIHRoYXQgY2FuIHN1cHBvcnQgc29tZSBhdXRoZW50aWNhdGlvbi4gIElzIHRoYXQgd2hhdAo+ IEkgc2hvdWxkIHVzZSBmb3IgY29va2llcz8gIElmIHNvLCBob3cgZG8gSSBjb25maWd1cmUgaXQ/ Cj4KPiBPciBpcyB0aGVyZSBzb21ldGhpbmcgZWxzZSBJIHNob3VsZCBiZSBkb2luZz8gIElmIHRo ZSBkb2N1bWVudGF0aW9uIC8KPiBhbnN3ZXIgZXhpc3RzLCBzb3JyeSBmb3IgdGhlIGhhc3NsZSBh bmQgcGxlYXNlIGp1c3QgcG9pbnQgbWUgdG8gaXQgOykKPgoKVW5mb3J0dW5hdGVseSwgbnV0Y2gg ZG9lc24ndCBoYXZlIHN1Y2ggYSBmZWF0dXJlIHlldC4gKE9uZSBvZiB0aGUgcHJvYmxlbXMKaXMg dGhhdCB3ZSBkbyBub3QgaGF2ZSBhIHBsYWNlIHRvIHN0b3JlIGNvb2tpZXMgaW4gYSBkaXN0cmli dXRlZCBzZXR1cCkKCj4gLS0KPiBUaGFua3MsCj4KPiBZb2F2Cj4KCgoKLS0gCkRvxJ9hY2FuIEfD vG5leQo= From nutch-userhvillarreal@example.net Wed Oct 01 14:35:30 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 35211 invoked from network); 1 Oct 2008 14:35:30 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 14:35:30 -0000 Received: (qmail 45725 invoked by uid 500); 1 Oct 2008 14:35:24 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 45696 invoked by uid 500); 1 Oct 2008 14:35:24 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 45685 invoked by uid 99); 1 Oct 2008 14:35:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 07:35:24 -0700 X-ASF-Spam-Status: No, hits=2.1 required=10.0 tests=DNS_FROM_RFC_BOGUSMX,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail.sim-gtech.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 14:34:23 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-9" Content-Transfer-Encoding: quoted-printable Subject: RE: How do I crawl a site with a cookie for authentication? Date: Wed, 1 Oct 2008 10:45:12 -0400 Message-ID: <hvillarreal@example.net> In-Reply-To: <hvillarreal@example.net> References: <hvillarreal@example.net> <hvillarreal@example.net> From: "Patrick Markiewicz" <hvillarreal@example.net> To: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Is it possible for you to retrieve a resource by using the url: http://username:password@intranetsite/path/to/resource.htm If that works, you could temporarily give a "nutchuser" an account on = the site (with as little permission as possible), then crawl the = intranet site, and disable the account. Then edit the nutch search page = to strip out the "nutchusername:nutchpassword@" part of each URL when = you present results to the user. That way, only the users who = previously authenticated would have access to that resource. I'm not sure what level of authority you have with the intranet site. = You could do a similar trick by crawling the local filesystem of that = site, and then just having the search page edit each URL to replace the = file system path with a URL path that would work for a logged in user. If you only have your own account, and can't change any other things, = then you might be able to use JMeter to add a cookie and have nutch use = JMeter as a proxy. I have never done this, so I don't actually remember = if JMeter can add a cookie to a request being made by an application = that it proxies. -----Original Message----- From: Do=F0acan G=FCney [mailto:hvillarreal@example.net]=20 Sent: Wednesday, October 01, 2008 10:08 AM To: hvillarreal@example.net Subject: Re: How do I crawl a site with a cookie for authentication? On Wed, Oct 1, 2008 at 4:35 PM, Yoav Shapira <hvillarreal@example.net> = wrote: > Hi, > > I would like to use Nutch to crawl and index an intranet web site for > internal use. The site requires authentication, and stores the > credentials in a cookie. I've got a valid login and I have the cookie > saved, no problem. How do I tell Nutch to use it? > > I did some research online before asking, but unfortunately I couldn't > find a step-by-step answer for a newbie like myself. I see there's an > http-client plugin that can support some authentication. Is that what > I should use for cookies? If so, how do I configure it? > > Or is there something else I should be doing? If the documentation / > answer exists, sorry for the hassle and please just point me to it ;) > Unfortunately, nutch doesn't have such a feature yet. (One of the = problems is that we do not have a place to store cookies in a distributed setup) > -- > Thanks, > > Yoav > --=20 Do=F0acan G=FCney From nutch-userhvillarreal@example.net Wed Oct 01 15:47:42 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 34741 invoked from network); 1 Oct 2008 15:47:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 15:47:42 -0000 Received: (qmail 97346 invoked by uid 500); 1 Oct 2008 15:47:36 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 97307 invoked by uid 500); 1 Oct 2008 15:47:35 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 97266 invoked by uid 99); 1 Oct 2008 15:47:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 08:47:35 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f13.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 15:46:34 +0000 Received: by gxk6 with SMTP id 6so299956gxk.5 for <hvillarreal@example.net>; Wed, 01 Oct 2008 08:47:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=/cmztyL7UK3e9AdhCRB3Ev2nkL7d3OLdN4B8EOM6/OQ=; b=nH8soNALwYu8mhIv2SRBh8UYrApxZQRTxrAbNGluzTMBrhX3b3OIBC95oQD0Xsb8HY 761qSePa7PMsBjtu+8pWhEBvpBBqOF2U8AuDFINjNHh/RD/lvIIwsrmwODS1cauAfd4O S/MlRKO4AADHA+h0ltyca7VXU0TUDhuVZ2vww= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=nqQezUyxzDPP6ov6ffHWRJE+IcpG6z2dNRCoGx05jQK5mSqQ+P3uV3szWTOt4dxjb7 A0G4tgPUD1Q9ESf+FhTVyEzP0iHWIkIVINTFYKRRwFJ9kITGJmRnfujzO8uh07Nshpry LtAGs6nXVec2PhTHzQ8YZEcYNQePEGasVUggY= Received: by 83.161.120.155 with SMTP id z9mr12316504ybg.0.1222876028690; Wed, 01 Oct 2008 08:47:08 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 08:47:08 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 11:47:08 -0400 From: "Yoav Shapira" <hvillarreal@example.net> Sender: hvillarreal@example.net To: hvillarreal@example.net Subject: Re: How do I crawl a site with a cookie for authentication? In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> X-Google-Sender-Auth: a0406513aa1f2478 X-Virus-Checked: Checked by ClamAV on apache.org Patrick, Thank you for the answers. More below: 2008/10/1 Patrick Markiewicz <hvillarreal@example.net>: > Is it possible for you to retrieve a resource by using the url: > http://username:password@intranetsite/path/to/resource.htm The system does not support HTTP Basic authentication at this time, unfortunately. > I'm not sure what level of authority you have with the intranet site. You could do a similar >trick by crawling the local filesystem of that site, and then just having the search page edit The site is dynamically generated. There are no meaningful static files on the file system. > If you only have your own account, and can't change any other things, then you might be >able to use JMeter to add a cookie and have nutch use JMeter as a proxy. I have never This is very intriguing. How would I get started on this? I've used JMeter in the past for simple test plans, but never as an HTTP proxy. Yoav From nutch-userhvillarreal@example.net Wed Oct 01 15:50:53 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 38097 invoked from network); 1 Oct 2008 15:50:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 15:50:53 -0000 Received: (qmail 4557 invoked by uid 500); 1 Oct 2008 15:50:49 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 4233 invoked by uid 500); 1 Oct 2008 15:50:48 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 4222 invoked by uid 99); 1 Oct 2008 15:50:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 08:50:47 -0700 X-ASF-Spam-Status: No, hits=2.6 required=10.0 tests=DNS_FROM_OPENWHOIS,SPF_HELO_PASS,SPF_PASS,WHOIS_MYPRIVREG X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO kuber.nabble.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 15:49:45 +0000 Received: from isper.nabble.com ([83.161.120.155]) by kuber.nabble.com with esmtp (Exim 4.63) (envelope-from <hvillarreal@example.net>) id 1Kl3xz-0001Dp-PL for hvillarreal@example.net; Wed, 01 Oct 2008 08:50:19 -0700 Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 08:50:19 -0700 (PDT) From: student_t <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: Please help with QueryFilter configuration In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Nabble-From: hvillarreal@example.net References: <hvillarreal@example.net> <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Thanks so much Do=C4=9Facan, At which point does the Nutch query processing system calls the query filter? Is it on the server side or client side? On the client side, I see NutchBean.search method is called with a Nutch Query. At that (client side) point, the Query instance looks like content:"+(content:pepsi) +((host:ca^10.0))" I want to filter out the first "conent" field to make it look like a regula= r Lucene BooleanQuery. But I don't know whether I should do it on the client or server side. Would you please elaborate on this? In addition, I am using raw-fields but the filter didn't look like was called. I thought I put in entries in log4j.properties to have it logged bu= t don't know why there are no logging text even after I put in=20 Thanks again! -student_t Do=C4=9Facan G=C3=BCney-3 wrote: >=20 > On Tue, Sep 30, 2008 at 4:25 PM, student_t <hvillarreal@example.net> wrote: >> >> Hi! Nutch Experts, >> >> Please help me with the following two questions. >> >> 1. It seems to me the Lucene QueryFilter functions differently from Nutc= h >> QueryFilter. Lucene QueryFilter filters out hits and Nutch QueryFilter >> modifies the underlying Query. Is my understanding correct? >> >=20 > Yes. >=20 >> 2. How do I configure so Nutch knows to send the query to the underlying >> QueryFilter before issuing its search? I have all the plugins in the >> right >> directories and seen in my log file that these plugins are loaded. But >> when >> I issue my search, I didn't see log entries (INFO) in the QueryFilter >> code >> spitted out. I must be missing a configuration entry. I think this entry >> must be put in nutch-default.xml. But what are the entries and can I loa= d >> the configuration file any time I want? It seems to me when I construct = a >> Nutch Query it attempts to load the configuration dynamically. >> >=20 > Have you modified your plugin.xml? >=20 > For example, query-basic's looks like this: >=20 > ..... > <implementation id=3D"BasicQueryFilter" > =20 > class=3D"org.apache.nutch.searcher.basic.BasicQueryFilter"> > <parameter name=3D"fields" value=3D"DEFAULT"/> > </implementation> > .... >=20 > And query-url like this: >=20 > ..... > <parameter name=3D"fields" value=3D"url"/> > ..... >=20 > A query filter only gets called if Nutch's query processing system > determines that it has fields to parse. > So, query-basic is called on everything without a specific field > (thus, DEFAULT) and query-url is called > if query has "url:......" >=20 >=20 >> Thanks very much in advance for any insights! >> >> -student_t >> >> >> >> -- >> View this message in context: >> http://www.nabble.com/Please-help-with-QueryFilter-configuration-tp19742= 072p19742072.html >> Sent from the Nutch - User mailing list hvillarreal@example.net. >> >> >=20 >=20 >=20 > --=20 > Do=C4=9Facan G=C3=BCney >=20 >=20 --=20 View this message in context: http://www.nabble.com/Please-help-with-QueryF= ilter-configuration-tp19742072p19763425.html Sent from the Nutch - User mailing list hvillarreal@example.net. From nutch-userhvillarreal@example.net Wed Oct 01 17:05:24 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 94273 invoked from network); 1 Oct 2008 17:05:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 17:05:24 -0000 Received: (qmail 47226 invoked by uid 500); 1 Oct 2008 17:05:18 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 47185 invoked by uid 500); 1 Oct 2008 17:05:18 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 47174 invoked by uid 99); 1 Oct 2008 17:05:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 10:05:18 -0700 X-ASF-Spam-Status: No, hits=2.1 required=10.0 tests=DNS_FROM_RFC_BOGUSMX,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail.sim-gtech.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 17:04:18 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: How do I crawl a site with a cookie for authentication? Date: Wed, 1 Oct 2008 13:14:51 -0400 Message-ID: <hvillarreal@example.net> In-Reply-To: <hvillarreal@example.net> References: <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> From: "Patrick Markiewicz" <hvillarreal@example.net> To: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Hi Yoav, If the content is dynamic, presumably it is stored in a database? I was just thinking that it might be easier to use some database utilities to index the information. Do you know how to use JMeter to record the requests that a web browser makes? The browser uses a particular port as a proxy. I know that the JMeter cookie manager can save the cookies that are gathered as part of the request. I'm pretty sure that nutch can use a proxy. http://wiki.apache.org/nutch/SetupProxyForNutch According to this page here: http://jakarta.apache.org/jmeter/usermanual/component_reference.html#HTT P_Cookie_Manager you can manually add a cookie that will be used by all threads. I am guessing that if you set up JMeter to act as a proxy, that this thread would be included as one of those that contains the cookie. If the proxy thread can not have cookies added manually, then this strategy wouldn't work.=20 Patrick -----Original Message----- From: hvillarreal@example.net [mailto:hvillarreal@example.net] On Behalf Of Yoav Shapira Sent: Wednesday, October 01, 2008 11:47 AM To: hvillarreal@example.net Subject: Re: How do I crawl a site with a cookie for authentication? Patrick, Thank you for the answers. More below: 2008/10/1 Patrick Markiewicz <hvillarreal@example.net>: > Is it possible for you to retrieve a resource by using the url: > http://username:password@intranetsite/path/to/resource.htm The system does not support HTTP Basic authentication at this time, unfortunately. > I'm not sure what level of authority you have with the intranet site. You could do a similar >trick by crawling the local filesystem of that site, and then just having the search page edit The site is dynamically generated. There are no meaningful static files on the file system. > If you only have your own account, and can't change any other things, then you might be >able to use JMeter to add a cookie and have nutch use JMeter as a proxy. I have never This is very intriguing. How would I get started on this? I've used JMeter in the past for simple test plans, but never as an HTTP proxy. Yoav From nutch-userhvillarreal@example.net Wed Oct 01 17:37:31 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 15487 invoked from network); 1 Oct 2008 17:37:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 17:37:31 -0000 Received: (qmail 89198 invoked by uid 500); 1 Oct 2008 17:37:23 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 89170 invoked by uid 500); 1 Oct 2008 17:37:23 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 89159 invoked by uid 99); 1 Oct 2008 17:37:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 10:37:23 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ik-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 17:36:21 +0000 Received: by ik-out-1112.google.com with SMTP id c30so449483ika.5 for <hvillarreal@example.net>; Wed, 01 Oct 2008 10:36:37 -0700 (PDT) Received: by 83.161.120.155 with SMTP id 20mr6810164eby.167.1222882596718; Wed, 01 Oct 2008 10:36:36 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 10:36:36 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 10:36:36 -0700 From: "Kevin MacDonald" <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: Using S3 with Hadoop/Nutch In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_9258_21285878.1222882596687" References: <hvillarreal@example.net> <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_9258_21285878.1222882596687 Content-Type: text/plain; charset=ISO-8859-9 Content-Transfer-Encoding: base64 Content-Disposition: inline VGhlIHByb2JsZW0gaXMgbm90IHRoYXQgSSB3YW50IHRvIG1peCBmaWxlIHN5c3RlbXMuIEkgd291 bGQgbGlrZSB0byBzdGFydApmcm9tIHNjcmF0Y2ggd2l0aCBIYWRvb3AgY29uZmlndXJlZCB0byB1 c2UgUzMuIEJ1dCB0byBkbyB0aGF0IEkgdGhpbmsgdGhhdAp2YXJpb3VzIFBhdGhzIHdoaWNoIE51 dGNoIGV4cGVjdHMgdG8gYmUgdGhlcmUgaGF2ZSB0byBiZSByZXByZXNlbnRlZCBzb21laG93Cmlu IHRoZSBTMyBidWNrZXQgdGhhdCBIYWRvb3AgaXMgdXNpbmcuIFNvLCB3aGVuIE51dGNoIHRyaWVz IHRvIHdyaXRlCnNvbWV0aGluZyB0byBuZXcgUGF0aCgiL3RtcC8uLi4iKSBIYWRvb3AgaXMgYWJs ZSB0byBkbyBpdC4gSSB0aGluayBteQpwcm9ibGVtIGNhbiBiZSBib2lsZWQgZG93biB0byB0aGlz OiBIb3cgZG8geW91IHByZXBhcmUgYW4gUzMgYnVja2V0IHNvIHRoYXQKZnJvbSBIYWRvb3AncyBw ZXJzcGVjdGl2ZSBpdCBsb29rcyBsaWtlIGEgaGllcmFyY2hpY2FsIGZpbGUgc3lzdGVtPwpLZXZp bgoKT24gV2VkLCBPY3QgMSwgMjAwOCBhdCAxMjoyOCBBTSwgRG/wYWNhbiBH/G5leSA8ZG9nYWNh bkBnbWFpbC5jb20+IHdyb3RlOgoKPiBPbiBUdWUsIFNlcCAzMCwgMjAwOCBhdCAxMTo1MiBQTSwg S2V2aW4gTWFjRG9uYWxkIDxrZXZpbkBoYXV0ZXNlY3VyZS5jb20+Cj4gd3JvdGU6Cj4gPiBEb2Vz IGFueW9uZSBoYXZlIGV4cGVyaWVuY2UgY29uZmlndXJpbmcgSGFkb29wIHRvIHVzZSBTMyBmb3Ig dXNpbmcgbnV0Y2g/Cj4gSQo+ID4gdHJpZWQgbW9kaWZ5aW5nIG15IGhhZG9vcC1zaXRlLnhtbCBj b25maWd1cmF0aW9uIGZpbGUgYW5kIGl0IGxvb2tzIGxpa2UKPiA+IEhhZG9vcCBpcyB0cnlpbmcg dG8gdXNlIFMzLiBCdXQgSSB0aGluayB3aGF0J3MgaGFwcGVuaW5nIGlzIHRoYXQsIG9uY2UKPiA+ IGNvbmZpZ3VyZWQgdG8gdXNlIFMzLCBIYWRvb3AgaXMgT05MWSBsb29raW5nIGF0IFMzIGZvciBh bGwgZmlsZXMuIEl0J3MKPiA+IHRyeWluZyB0byBmaW5kIGEgL3RtcCBmb2xkZXIgdGhlcmUsIGZv ciBleGFtcGxlLiBBbmQgd2hlbiBydW5uaW5nIGEgY3Jhd2wKPiA+IEhhZG9vcCBpcyBsb29raW5n IHRvIFMzIHRvIGZpbmQgdGhlIHNlZWQgdXJscyBmb2xkZXIuIEFyZSB0aGVyZSBzdGVwcwo+IHRo YXQKPiA+IG5lZWQgdG8gaGFwcGVuIHRvIHByZXBhcmUgYW4gUzMgYnVja2V0IGZvciB1c2UgYnkg SGFkb29wIHNvIHRoYXQgYSBudXRjaAo+ID4gY3Jhd2wgY2FuIGhhcHBlbj8KPgo+IElmIHlvdSB3 YW50IHRvIHBhc3MgcGF0aHMgZnJvbSBvdGhlciBmaWxlc3lzdGVtcyBJIHRoaW5rIHlvdSBjYW4g ZG8KPiBzb21ldGhpbmcgbGlrZToKPgo+IGJpbi9udXRjaCBpbmplY3QgY3Jhd2wvY3Jhd2xkYiBo ZGZzOi8vbWFjaGluZToxMDAwMC8uLi4uLgo+Cj4gPiBLZXZpbgo+ID4KPgo+Cj4KPiAtLQo+IERv 8GFjYW4gR/xuZXkKPgo= ------=_Part_9258_21285878.1222882596687-- From nutch-userhvillarreal@example.net Wed Oct 01 17:51:31 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 24451 invoked from network); 1 Oct 2008 17:51:31 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 17:51:31 -0000 Received: (qmail 10956 invoked by uid 500); 1 Oct 2008 17:51:26 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 10927 invoked by uid 500); 1 Oct 2008 17:51:26 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 10916 invoked by uid 99); 1 Oct 2008 17:51:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 10:51:26 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f13.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 17:50:25 +0000 Received: by gxk6 with SMTP id 6so487038gxk.5 for <hvillarreal@example.net>; Wed, 01 Oct 2008 10:49:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=4jvIfWfETYj4w1Vp/iaNLx4lqBWSc5WKNfLPhvjBato=; b=urhgAJO9jp8CJbQsur4vR4VdWq7F0z14QSyFB7h8vSCKsvXfZKuywSu56tI/l4lEhT a+wdlMwwLIu/NfFShU31SVJpfOZCCTd5jDbxXiC6tLJt273xQexgktiwWHS9QjvUoCSy SuS45jq7VGAqJsnCGebBu6q23BLb4f90i1Ax0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=Q32CnSxV9GpxUODa1KnwjRzfYzSgbLbLUfLtHONU26z8+9AbX199vcl9AMwpLTYGGR cbKML4QGfdYsjGxz1DEqUZ3JpfBcPf9EpDnb1Z73/Dsl4knIAx/3H5gF6p2S7DFpCJCF Z49aYhLjUV6GarPQvIHw6ku87OI2uMqj5PkJE= Received: by 83.161.120.155 with SMTP id p14mr12479250ybg.56.1222883399145; Wed, 01 Oct 2008 10:49:59 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 10:49:59 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 13:49:59 -0400 From: "Yoav Shapira" <hvillarreal@example.net> Sender: hvillarreal@example.net To: hvillarreal@example.net Subject: Re: How do I crawl a site with a cookie for authentication? In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> X-Google-Sender-Auth: 38af81e63a3ca9ce X-Virus-Checked: Checked by ClamAV on apache.org Hi Patrick, Thanks for your help. I'll dig around a bit more, try the proxy thing, maybe try the database approach, and see how it goes. Much appreciated, Yoav On Wed, Oct 1, 2008 at 1:14 PM, Patrick Markiewicz <hvillarreal@example.net> wrote: > Hi Yoav, > If the content is dynamic, presumably it is stored in a > database? I was just thinking that it might be easier to use some > database utilities to index the information. > > Do you know how to use JMeter to record the requests that a web > browser makes? The browser uses a particular port as a proxy. I know > that the JMeter cookie manager can save the cookies that are gathered as > part of the request. > I'm pretty sure that nutch can use a proxy. > http://wiki.apache.org/nutch/SetupProxyForNutch > > According to this page here: > http://jakarta.apache.org/jmeter/usermanual/component_reference.html#HTT > P_Cookie_Manager > you can manually add a cookie that will be used by all threads. I am > guessing that if you set up JMeter to act as a proxy, that this thread > would be included as one of those that contains the cookie. > > If the proxy thread can not have cookies added manually, then this > strategy wouldn't work. > > Patrick > > -----Original Message----- > From: hvillarreal@example.net [mailto:hvillarreal@example.net] On Behalf Of > Yoav Shapira > Sent: Wednesday, October 01, 2008 11:47 AM > To: hvillarreal@example.net > Subject: Re: How do I crawl a site with a cookie for authentication? > > Patrick, > Thank you for the answers. More below: > > 2008/10/1 Patrick Markiewicz <hvillarreal@example.net>: >> Is it possible for you to retrieve a resource by using the url: >> http://username:password@intranetsite/path/to/resource.htm > > The system does not support HTTP Basic authentication at this time, > unfortunately. > >> I'm not sure what level of authority you have with the intranet site. > You could do a similar >trick by crawling the local filesystem of that > site, and then just having the search page edit > > The site is dynamically generated. There are no meaningful static > files on the file system. > >> If you only have your own account, and can't change any other things, > then you might be >able to use JMeter to add a cookie and have nutch use > JMeter as a proxy. I have never > > This is very intriguing. How would I get started on this? I've used > JMeter in the past for simple test plans, but never as an HTTP proxy. > > Yoav > -- Thanks, Yoav From nutch-userhvillarreal@example.net Wed Oct 01 20:07:08 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 14835 invoked from network); 1 Oct 2008 20:07:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Oct 2008 20:07:08 -0000 Received: (qmail 90461 invoked by uid 500); 1 Oct 2008 20:07:04 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 90430 invoked by uid 500); 1 Oct 2008 20:07:04 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 90419 invoked by uid 99); 1 Oct 2008 20:07:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 13:07:04 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ey-out-2122.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 20:06:03 +0000 Received: by ey-out-2122.google.com with SMTP id 6so239636eyi.53 for <hvillarreal@example.net>; Wed, 01 Oct 2008 13:06:19 -0700 (PDT) Received: by 83.161.120.155 with SMTP id i5mr10191527ebi.114.1222891579249; Wed, 01 Oct 2008 13:06:19 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 13:06:19 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Wed, 1 Oct 2008 13:06:19 -0700 From: "Kevin MacDonald" <hvillarreal@example.net> To: nutch-user <hvillarreal@example.net> Subject: urlfilter-suffix not enabled MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_11155_7503152.1222891579245" X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_11155_7503152.1222891579245 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline Is there a reason why the default 'plugin.includes' property does not enabled the 'urlfilter-suffix' plugin? Kevin ------=_Part_11155_7503152.1222891579245-- From nutch-userhvillarreal@example.net Thu Oct 02 04:56:05 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 52682 invoked from network); 2 Oct 2008 04:56:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Oct 2008 04:56:04 -0000 Received: (qmail 33359 invoked by uid 500); 2 Oct 2008 04:55:56 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 33319 invoked by uid 500); 2 Oct 2008 04:55:56 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 33308 invoked by uid 99); 2 Oct 2008 04:55:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Oct 2008 21:55:56 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO wf-out-1314.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 04:54:55 +0000 Received: by wf-out-1314.google.com with SMTP id 28so979403wfc.20 for <hvillarreal@example.net>; Wed, 01 Oct 2008 21:55:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type:references; bh=RApydXMh9/CgLgEz01GRzNYGwiJE2ch00SzN2NRo73g=; b=LYPfAYnJx08Qy3rkayNGX5qyF4RqU6oF8ct3INl0E/6rxfKcME2Ay/wPbfzIvlS3Yv 6OEzBFxFfOpUp86SbbqCZ6sePnfIObln2oudnKG8JZsqCyld4H6B4T0N4J23ltgcIqHz iBv+KUyKNREKLEV5PX0DEFiatc1/gWMxbXJ4s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:references; b=CmeiEqylFjJFnsiZVa/E6+5VS18LMwTBAPvO2FENV3W/OdaYyNPE9d8wDrR0E6docT QeCZlP1imp9n/4N4S6HAFIFutH+DD4/u7DvSfcWPvsvOEwBCyrMbR9kt7ogTa8VsecJ7 YIOsLSCa3LS1IoHUNuXXh223LQhVNZ0N7Z4Tk= Received: by 83.161.120.155 with SMTP id 20mr3687014wfn.149.1222923320572; Wed, 01 Oct 2008 21:55:20 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Wed, 1 Oct 2008 21:55:20 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Thu, 2 Oct 2008 08:55:20 +0400 From: "Alexander Aristov" <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: Using S3 with Hadoop/Nutch In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_97580_25388449.1222923320569" References: <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_97580_25388449.1222923320569 Content-Type: text/plain; charset=ISO-8859-9 Content-Transfer-Encoding: base64 Content-Disposition: inline Tm90aGluZyBzcGVjaWFsIHNob3VsZCBiZSBkb25lIHdpdGggUzMgYXMgSGFkb29wIHdpbGwgdGFr ZSBjYXJlIGFib3V0IGl0Cml0c2VsZi4gWW91IG11c3Qgb25seSBiZSBzdXJlIHRoYXQgdGhlIGJ1 Y2tldCBpcyBhY2Nlc3NpYmxlLiBEb24ndCB3b3JyeQphYm91dCB0aGUgdG1wIGZvbGRlci4KCkkg Y29uZmlndXJlZCBhbmQgcnVuIGhhZG9vcCB3aXRoIFMzIHJlY2VudGx5LCBpdCB3YXMgT2sgdGhv dWdoIEkgY29uc2lkZXJlZAp0byBzd2l0Y2ggYmFjayB0byBoZGZzIGFuZCB1c2UgUzMgb25seSBh cyBhIGZpbmFsIGZpbGUgc3RvcmUoYmFja3VwKSBiZWNhdXNlCkkgc2F3IHNvbWUgcHJvZHVjdGl2 aXR5IGRvd25ncmFkZS4KCkFsZXhhbmRlcgoKMjAwOC8xMC8xIEtldmluIE1hY0RvbmFsZCA8a2V2 aW5AaGF1dGVzZWN1cmUuY29tPgoKPiBUaGUgcHJvYmxlbSBpcyBub3QgdGhhdCBJIHdhbnQgdG8g bWl4IGZpbGUgc3lzdGVtcy4gSSB3b3VsZCBsaWtlIHRvIHN0YXJ0Cj4gZnJvbSBzY3JhdGNoIHdp dGggSGFkb29wIGNvbmZpZ3VyZWQgdG8gdXNlIFMzLiBCdXQgdG8gZG8gdGhhdCBJIHRoaW5rIHRo YXQKPiB2YXJpb3VzIFBhdGhzIHdoaWNoIE51dGNoIGV4cGVjdHMgdG8gYmUgdGhlcmUgaGF2ZSB0 byBiZSByZXByZXNlbnRlZAo+IHNvbWVob3cKPiBpbiB0aGUgUzMgYnVja2V0IHRoYXQgSGFkb29w IGlzIHVzaW5nLiBTbywgd2hlbiBOdXRjaCB0cmllcyB0byB3cml0ZQo+IHNvbWV0aGluZyB0byBu ZXcgUGF0aCgiL3RtcC8uLi4iKSBIYWRvb3AgaXMgYWJsZSB0byBkbyBpdC4gSSB0aGluayBteQo+ IHByb2JsZW0gY2FuIGJlIGJvaWxlZCBkb3duIHRvIHRoaXM6IEhvdyBkbyB5b3UgcHJlcGFyZSBh biBTMyBidWNrZXQgc28gdGhhdAo+IGZyb20gSGFkb29wJ3MgcGVyc3BlY3RpdmUgaXQgbG9va3Mg bGlrZSBhIGhpZXJhcmNoaWNhbCBmaWxlIHN5c3RlbT8KPiBLZXZpbgo+Cj4gT24gV2VkLCBPY3Qg MSwgMjAwOCBhdCAxMjoyOCBBTSwgRG/wYWNhbiBH/G5leSA8ZG9nYWNhbkBnbWFpbC5jb20+IHdy b3RlOgo+Cj4gPiBPbiBUdWUsIFNlcCAzMCwgMjAwOCBhdCAxMTo1MiBQTSwgS2V2aW4gTWFjRG9u YWxkIDxrZXZpbkBoYXV0ZXNlY3VyZS5jb20KPiA+Cj4gPiB3cm90ZToKPiA+ID4gRG9lcyBhbnlv bmUgaGF2ZSBleHBlcmllbmNlIGNvbmZpZ3VyaW5nIEhhZG9vcCB0byB1c2UgUzMgZm9yIHVzaW5n Cj4gbnV0Y2g/Cj4gPiBJCj4gPiA+IHRyaWVkIG1vZGlmeWluZyBteSBoYWRvb3Atc2l0ZS54bWwg Y29uZmlndXJhdGlvbiBmaWxlIGFuZCBpdCBsb29rcyBsaWtlCj4gPiA+IEhhZG9vcCBpcyB0cnlp bmcgdG8gdXNlIFMzLiBCdXQgSSB0aGluayB3aGF0J3MgaGFwcGVuaW5nIGlzIHRoYXQsIG9uY2UK PiA+ID4gY29uZmlndXJlZCB0byB1c2UgUzMsIEhhZG9vcCBpcyBPTkxZIGxvb2tpbmcgYXQgUzMg Zm9yIGFsbCBmaWxlcy4gSXQncwo+ID4gPiB0cnlpbmcgdG8gZmluZCBhIC90bXAgZm9sZGVyIHRo ZXJlLCBmb3IgZXhhbXBsZS4gQW5kIHdoZW4gcnVubmluZyBhCj4gY3Jhd2wKPiA+ID4gSGFkb29w IGlzIGxvb2tpbmcgdG8gUzMgdG8gZmluZCB0aGUgc2VlZCB1cmxzIGZvbGRlci4gQXJlIHRoZXJl IHN0ZXBzCj4gPiB0aGF0Cj4gPiA+IG5lZWQgdG8gaGFwcGVuIHRvIHByZXBhcmUgYW4gUzMgYnVj a2V0IGZvciB1c2UgYnkgSGFkb29wIHNvIHRoYXQgYQo+IG51dGNoCj4gPiA+IGNyYXdsIGNhbiBo YXBwZW4/Cj4gPgo+ID4gSWYgeW91IHdhbnQgdG8gcGFzcyBwYXRocyBmcm9tIG90aGVyIGZpbGVz eXN0ZW1zIEkgdGhpbmsgeW91IGNhbiBkbwo+ID4gc29tZXRoaW5nIGxpa2U6Cj4gPgo+ID4gYmlu L251dGNoIGluamVjdCBjcmF3bC9jcmF3bGRiIGhkZnM6Ly9tYWNoaW5lOjEwMDAwLy4uLi4uCj4g Pgo+ID4gPiBLZXZpbgo+ID4gPgo+ID4KPiA+Cj4gPgo+ID4gLS0KPiA+IERv8GFjYW4gR/xuZXkK PiA+Cj4KCgoKLS0gCkJlc3QgUmVnYXJkcwpBbGV4YW5kZXIgQXJpc3Rvdgo= ------=_Part_97580_25388449.1222923320569-- From nutch-userhvillarreal@example.net Thu Oct 02 09:02:19 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 28097 invoked from network); 2 Oct 2008 09:02:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Oct 2008 09:02:19 -0000 Received: (qmail 46941 invoked by uid 500); 2 Oct 2008 09:02:16 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 46905 invoked by uid 500); 2 Oct 2008 09:02:16 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 46894 invoked by uid 99); 2 Oct 2008 09:02:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 02:02:16 -0700 X-ASF-Spam-Status: No, hits=2.0 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO blu0-omc2-s12.blu0.hotmail.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 09:01:13 +0000 Received: from BLU106-W44 ([83.161.120.155]) by blu0-omc2-s12.blu0.hotmail.com with Microsoft SMTPSVC985.366.1074); Thu, 2 Oct 2008 02:01:26 -0700 Message-ID: <hvillarreal@example.net> Content-Type: multipart/alternative; boundary="_b70b4905-4021-477a-9989-9ec2ab56842a_" X-Originating-IP: [83.161.120.155] From: Edward Quick <hvillarreal@example.net> To: <hvillarreal@example.net> Subject: RE: subcollection Date: Thu, 2 Oct 2008 09:01:25 +0000 Importance: Normal In-Reply-To: <hvillarreal@example.net> References: <hvillarreal@example.net> MIME-Version: 1.0 X-OriginalArrivalTime: 02 Oct 2008 09:01:26.0263 (UTC) FILETIME=[73625870:01C9246D] X-Virus-Checked: Checked by ClamAV on apache.org --_b70b4905-4021-477a-9989-9ec2ab56842a_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I have enabled logging and get the following error with the subcollections = plugin. Any ideas what I do to fix this? Thanks=2C Ed. 2008-10-02 09:37:49=2C401 INFO collection.CollectionManager - Instantiatin= g CollectionManager 2008-10-02 09:37:49=2C401 INFO collection.CollectionManager - initializing= CollectionManager 2008-10-02 09:37:49=2C405 WARN collection.CollectionManager - Error occure= d:java.lang.ClassCastException: org.apache.xerces.dom.DeferredCommentImpl 2008-10-02 09:37:49=2C405 WARN collection.CollectionManager - java.lang.Cl= assCastException: org.apache.xerces.dom.DeferredCommentImpl 2008-10-02 09:37:49=2C405 WARN collection.CollectionManager - at org.apach= e.nutch.util.DomUtil.getDom(DomUtil.java:63) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.collection.CollectionManager.parse(CollectionManager.java:85) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.collection.CollectionManager.init(CollectionManager.java:75) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.collection.CollectionManager.<init>(CollectionManager.java:56) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.collection.CollectionManager.getCollectionManager(CollectionManager= .j ava:115) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.indexer.subcollection.SubcollectionIndexingFilter.addSubCollectionF= ie ld(SubcollectionIndexingFilter.java:66) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.indexer.subcollection.SubcollectionIndexingFilter.filter(Subcollect= io nIndexingFilter.java:72) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.indexer.IndexingFilters.filter(IndexingFilters.java:111) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.indexer.Indexer.reduce(Indexer.java:249) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.nutch.indexer.Indexer.reduce(Indexer.java:52) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.hadoop.mapred.ReduceTask.run(ReduceTask.java:391) 2008-10-02 09:37:49=2C406 WARN collection.CollectionManager - at org.apach= e.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:201) 2008-10-02 09:37:49=2C407 DEBUG collection.CollectionManager - subcollectio= ns: >=20 >=20 > Hi=2C >=20 > I'm trying to get subcollections working in nutch 1.0-dev=2C and have cra= wled our intranet with the subcollection.xml configured as below. However w= hen I submit a query to search.jsp eg=2C >=20 > subcollection:im database >=20 > I don't get any results (as opposed to submitting this without subcollect= ion:im) >=20 > Is this configured wrongly? I realise that subcollection.xml doesn't do r= egex expressions=2C but I wasn't sure if I could just put in part of the ur= l=2C or had to put in the full stem pattern eg=2C http://planet.somdomain.c= om/level1/ >=20 > Thanks=2C > Ed. >=20 > <subcollections> > <subcollection> > <name>default</name> > <id>default</id> > <whitelist> > </whitelist> > <blacklist> > planet.somedomain.com/general/aptrix/bani.nsf/Content/= Weekly+news > /aptprop.nsf/Content/Americas+ > /aptprop.nsf/Content/AB+CityFlyer+ > /aptprop.nsf/Content/CityFlyer+ > /im/barch/ > /im/dms/ > /im/tech/ > </blacklist> > </subcollection> >=20 > <subcollection> > <name>im</name> > <id>im</id> > <whitelist> > planet.somedomain.com/general/aptrix/aptim.nsf/ > planet.somedomain.com/im/barch/ > planet.somedomain.com/im/dms/ > planet.somedomain.com/im/tech/ > </whitelist> > <blacklist /> > </subcollection> >=20 > <subcollection> > <name>news</name> > <id>news</id> > <whitelist> > planet.somedomain.com/general/aptrix/bani.nsf/Content= /Weekly+news > </whitelist> > <blacklist /> > </subcollection> >=20 > </subcollections> >=20 > _________________________________________________________________ > Discover Bird's Eye View now with Multimap from Live Search > http://clk.atdmt.com/UKM/go/111354026/direct/01/ _________________________________________________________________ Get all your favourite content with the slick new MSN Toolbar - FREE http://clk.atdmt.com/UKM/go/111354027/direct/01/= --_b70b4905-4021-477a-9989-9ec2ab56842a_-- From nutch-userhvillarreal@example.net Thu Oct 02 16:37:04 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 36305 invoked from network); 2 Oct 2008 16:37:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Oct 2008 16:37:04 -0000 Received: (qmail 15678 invoked by uid 500); 2 Oct 2008 16:36:59 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 15642 invoked by uid 500); 2 Oct 2008 16:36:58 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 15631 invoked by uid 99); 2 Oct 2008 16:36:58 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 09:36:58 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO gv-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Oct 2008 16:35:57 +0000 Received: by gv-out-0910.google.com with SMTP id p33so303551gvf.23 for <hvillarreal@example.net>; Thu, 02 Oct 2008 09:36:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=mCd1gZBp1+rYtKGStYVTwvBvGC/jLcOsUJsFeOrpWxg=; b=daEzltSQhMEXeo1Fdn6xAN80DVdsJC6pwB/C7cycidfjKwT6BZyNl4MEMNKKtjrMG6 UGfozFGCdkvTrgCgTnxGItpEU0lHyfx2KFCLBERoTop7zHFrHphy2UN2zSqfe+jhw95q PE9RrssVeZvZh5Z1Ma3SKRhezRoGHAAIJm194= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=LpFrchruF7yMXDkYp/mJNRiIwVybT/q+e2bqMX4QtoA4DxE2FhmW+zn4w8lysrSyMm mn5uthKGF0mfp2al6q6FBQcDqnc+I8mPR9po+y4AJ3efVGTmc/gH8jc6FsI+HIB93AtE P9cFZLvubPEKnTXXe4+M9mJ4gkHSC8Kx+us7g= Received: by 83.161.120.155 with SMTP id y1mr6946804mur.53.1222965373837; Thu, 02 Oct 2008 09:36:13 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Thu, 2 Oct 2008 09:36:13 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Thu, 2 Oct 2008 18:36:13 +0200 From: brainstorm <hvillarreal@example.net> To: hvillarreal@example.net Subject: Re: Nutch STOP conditions In-Reply-To: <hvillarreal@example.net> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org Gonna try to reply myself :-S... if you have simpler ways to do it, your advise will be more than welcome. On Fri, Aug 22, 2008 at 1:34 PM, brainstorm <hvillarreal@example.net> wrote: > There's something left I want to ask that I haven't found clearly > explained on FAQ nor mailing list: > > Nutch STOP conditions, meaning: "how to stop a running nutch crawl" > > In other words, how to define crawl: > > 1) "time limit": Crawl for Q hours and stop at-based time limit ? nutch/bin/stop-all.sh... kind of drastic, but I guess it works, but I don't know if the resulting data is consistent :-S > 2) "segments limit": After generating N segments, stop No clue. > 3) "space limit": After M megabytes/space on DFS used, stop. Some hooks on ganglia metrics + pooling to determine if we're out of space ? > 4) "input urls limit": After crawling Z urls from the original (seed) > input set, stop. -depth 1 ? > More "limits" doubts/suggestions are welcome ;) > > I'll put the answer(s) on Nutch wiki (FAQ section) if you don't mind, > I think it could clarify this spot to lots of people on the mailing > list (me included ! :-S). > From nutch-userhvillarreal@example.net Fri Oct 03 12:48:42 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 61538 invoked from network); 3 Oct 2008 12:48:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Oct 2008 12:48:41 -0000 Received: (qmail 4713 invoked by uid 500); 3 Oct 2008 12:48:38 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 4681 invoked by uid 500); 3 Oct 2008 12:48:37 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 4670 invoked by uid 99); 3 Oct 2008 12:48:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Oct 2008 05:48:37 -0700 X-ASF-Spam-Status: No, hits=3.2 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO n5c.bullet.tw1.yahoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 03 Oct 2008 12:47:33 +0000 Received: from [83.161.120.155] by n5.bullet.tw1.yahoo.com with NNFMP; 03 Oct 2008 12:48:05 -0000 Received: from [83.161.120.155] by t1.bullet.tw1.yahoo.com with NNFMP; 03 Oct 2008 12:48:05 -0000 Received: from [83.161.120.155] by t1.bullet.kr1.yahoo.com with NNFMP; 03 Oct 2008 12:48:05 -0000 Received: from [83.161.120.155] by omp111.mail.in2.yahoo.com with NNFMP; 03 Oct 2008 12:48:05 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: {{EMAIL+PHONE}} Received: (qmail 43945 invoked by uid 60001); 3 Oct 2008 12:48:05 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.co.in; h=X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Message-ID; b=KZ4aXg0zLf4lsNkKPzqzKA72YTKs22UIlxtZw+ENXLXhWyMXA0adrcl2RPD+B/QgR7MPLCh8bJLGAhHLBcGsuuysXslCOGoC43h8dm7Vk/g++mMsmVOx+ByRFlKCkMRd2p3rDitc/PM3T3pz/W8A2Qrru1d6jBQlDwKf4xVyTzI=; X-YMail-OSG: z1Ye0zQVM1mdQMjNbRgT9UxU3z_v4HmWsXECi8y1lqCSGWmDH4c727lQxyrWeGkImKcv5XCDBdoGiyW2cM9jEA08LdxEnlt8xaOJPfuhuFGecbWKsdPaHDZwgqthysQAeVugIzKHIM8gH_Ifb2msXiPCpGFA Received: from [83.161.120.155] by web94603.mail.in2.yahoo.com via HTTP; Fri, 03 Oct 2008 18:18:05 IST X-Mailer: YahooMailWebService/83.161.120.155 Date: Fri, 3 Oct 2008 18:18:05 +0530 (IST) From: Nimesh Priyodit <hvillarreal@example.net> Reply-To: hvillarreal@example.net Subject: Re: Crawling XML files and indexing them To: hvillarreal@example.net MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="0-939264808-1223038085=:41616" Message-ID: <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org --0-939264808-1223038085=:41616 Content-Type: text/plain; charset=us-ascii HI everyone, I'm a new user to nutch. Can you please guide me how to crawl and index XML files present in my computer? I would be grateful for the help. Regards, Nimesh --0-939264808-1223038085=:41616-- From nutch-userhvillarreal@example.net Fri Oct 03 15:02:21 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 31702 invoked from network); 3 Oct 2008 15:02:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Oct 2008 15:02:21 -0000 Received: (qmail 6930 invoked by uid 500); 3 Oct 2008 15:02:16 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 6899 invoked by uid 500); 3 Oct 2008 15:02:16 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 6888 invoked by uid 99); 3 Oct 2008 15:02:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Oct 2008 08:02:16 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f13.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Oct 2008 15:01:13 +0000 Received: by gxk6 with SMTP id 6so2771765gxk.5 for <hvillarreal@example.net>; Fri, 03 Oct 2008 08:01:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:mime-version:content-type:content-transfer-encoding :content-disposition; bh=zqeaA65s/3RXnCJWiboR0xhLBCHpflN5YFT7DgqnhT4=; b=u92/JU4cFPrPIBjoc/tY70AxlNQrK7ssjsYkUanrTWcvYRJ+OQ31WHr3mgKTMpAnSN mMzpyJsHfBnG6hqe2RxK1gH5armb8ugUOwPYfT2Q4C3GMNwfkmkonM+ES/FGvW28CGla iktIXR5dG7VeQPVgjpZyD9/skmn9w4kvfi7ps= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:mime-version:content-type :content-transfer-encoding:content-disposition; b=odUEIynWAKa4FOHv+AJG5iS1RoWBggPTmSoOiVsai9/o5JW6AIcZpYyO7l88Njr/6i S8jqkb2rX4gLmCUNyZNT70GmcrjNRMg3C9jZsclDO6TcCUI4oSYKwKlXiTnjGhOdljI7 GH5bdabDvTJC1Mt10G5aZHszCig2tE/dqy5lo= Received: by 83.161.120.155 with SMTP id b7mr774478mus.2.1223046106708; Fri, 03 Oct 2008 08:01:46 -0700 (PDT) Received: by 83.161.120.155 with HTTP; Fri, 3 Oct 2008 08:01:46 -0700 (PDT) Message-ID: <hvillarreal@example.net> Date: Fri, 3 Oct 2008 17:01:46 +0200 From: brainstorm <hvillarreal@example.net> To: hvillarreal@example.net Subject: Uncompressing SEQ files from cmdline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline X-Virus-Checked: Checked by ClamAV on apache.org SG93IGNhbiBJIGVhc2lseSB1bmNvbXByZXNzIGEgZG93bmxvYWRlZCBmaWxlIGZyb20gSERGUyA/ IERvZXMgYW55b25lCmhhdmUgYW55IGphdmEgc25pcHBldCBvbiB0aGlzID8KClNFUV5GXllvcmcu YXBhY2hlLmhhZG9vcC5pby5UZXh0IW9yZy5hcGFjaGUubnV0Y2guY3Jhd2wuQ3Jhd2xEYXR1bV5A XkBeQF5AXkBeQO+/ve+/ve+/vV4/Xk5Hee+/vVx+fkvvv71eXCFeV15AXkBeQDxeQF5AXkBeX15e aHR0cDovLy1qYWNrYWwuZGV2aWFudGFycnQuY29tLwo= From nutch-userhvillarreal@example.net Fri Oct 03 15:43:18 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 99354 invoked from network); 3 Oct 2008 15:43:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Oct 2008 15:43:18 -0000 Received: (qmail 76627 invoked by uid 500); 3 Oct 2008 15:43:16 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 75693 invoked by uid 500); 3 Oct 2008 15:43:14 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 75682 invoked by uid 99); 3 Oct 2008 15:43:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Oct 2008 08:43:14 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO s01.igfoo.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Oct 2008 15:42:11 +0000 Received: from localhost (localhost [83.161.120.155]) by s01.igfoo.com (Postfix) with ESMTP id 3ED26F340E2 for <hvillarreal@example.net>; Fri, 3 Oct 2008 10:42:44 -0500 (CDT) X-DKIM: Sendmail DKIM Filter v2.5.4 s01.igfoo.com 3ED26F340E2 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=igfoo.com; s=mail; t=1223048564; bh=GYx//2Q19hi1g8iMBCcCIyIayAhKf7a874X23idBnWg=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=oWOYY8oxIX5n ke5tol7j+bZPieC1VqyzbFVi54wS208/rK61A2pOac8Ar1LFV5/Qp0uY0YCmiLWhSCZ l+Xd9f6cQGkqgyfsn6Gq0WvIa7vGzeXrINnf0LzqraG6e0EyIck2mNMwnkz0xSisFkr 1lvLMvTdRCo20cV75JSzwqBO8= X-Virus-Scanned: Debian hvillarreal@example.net Received: from s01.igfoo.com ([83.161.120.155]) by localhost (s01.igfoo.com [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id K19tNPk79zx2 for <hvillarreal@example.net>; Fri, 3 Oct 2008 10:42:42 -0500 (CDT) Received: from [83.161.120.155] (unknown [83.161.120.155]) by s01.igfoo.com (Postfix) with ESMTPSA id A4A94F340BE for <hvillarreal@example.net>; Fri, 3 Oct 2008 10:42:42 -0500 (CDT) X-DKIM: Sendmail DKIM Filter v2.5.4 s01.igfoo.com A4A94F340BE DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=igfoo.com; s=mail; t=1223048562; bh=GYx//2Q19hi1g8iMBCcCIyIayAhKf7a874X23idBnWg=; h=Message-ID:Date:From:MIME-Version:To:Subject:References: In-Reply-To:Content-Type:Content-Transfer-Encoding; b=VQShEiRNc+fn y007sqVCKL6ChatTHtRz36FEb47gRhdmN6XPZNvqISwHbuE8fxlNvrFEzZKnvtmoidD RBSwDk60Bv4AYd7jPXNuiryIt1fsQN82aaEVFGhIbQYnJlp6a3sXoESxwZtFypnySAi Ojw7Fn0ZBS/7g1W8GQaX50BGM= Message-ID: <hvillarreal@example.net> Date: Fri, 03 Oct 2008 10:42:42 -0500 From: Dennis Kubes <hvillarreal@example.net> User-Agent: Thunderbird 83.161.120.155 (X11/20080925) MIME-Version: 1.0 To: hvillarreal@example.net Subject: Re: Uncompressing SEQ files from cmdline References: <hvillarreal@example.net> In-Reply-To: <hvillarreal@example.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Checked: Checked by ClamAV on apache.org While some sequence files may be compressed, they are binary not text formats. You would need to use a MR job to output the values to TextOutputFormat. Dennis brainstorm wrote: > How can I easily uncompress a downloaded file from HDFS ? Does anyone > have any java snippet on this ? > > SEQ^F^Yorg.apache.hadoop.io.Text!org.apache.nutch.crawl.CrawlDatum^@^@^@^@^@^@���^?^NGy�\~~K�^\!^W^@^@^@<^@^@^@^_^^http://-jackal.deviantarrt.com/ From nutch-userhvillarreal@example.net Fri Oct 03 16:17:14 2008 Return-Path: <nutch-userhvillarreal@example.net> Delivered-To: hvillarreal@example.net Received: (qmail 8568 invoked from network); 3 Oct 2008 16:17:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Oct 2008 16:17:14 -0000 Received: (qmail 14370 invoked by uid 500); 3 Oct 2008 16:17:10 -0000 Delivered-To: hvillarreal@example.net Received: (qmail 14339 invoked by uid 500); 3 Oct 2008 16:17:10 -0000 Mailing-List: contact hvillarreal@example.net; run by ezmlm Precedence: bulk List-Help: <mailto:hvillarreal@example.net> List-Unsubscribe: <mailto:hvillarreal@example.net> List-Post: <mailto:hvillarreal@example.net> List-Id: <nutch-user.lucene.apache.org> Reply-To: hvillarreal@example.net Delivered-To: mailing list hvillarreal@example.net Received: (qmail 14328 invoked by uid 99); 3 Oct 2008 16:17:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Oct 2008 09:17:10 -0700 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of hvillarreal@example.net designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO rv-out-0506.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Oct 2008 16:16:08 +0000 Received: by rv-out-0506.google.com with SMTP id f6so1596709rvb.5 for <hvillarreal@example.net>; Fri, 03 Oct 2008 09:16:42 -0700 (PDT) Received: by 83.161.120.155 with SMTP id q3mr725924rvl.46.1223050602467; Fri, 03 Oct 2008 09:16:42 -0700 (PDT) Received: from ?83.161.120.155? ([83.161.120.155]) by mx.google.com with ESMTPS id k41sm10123185rvb.9.283.161.120.155.16.37 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 03 Oct 2008 09:16:39 -0700 (PDT) Message-Id: <hvillarreal@example.net> From: Kevin MacDonald <hvillarreal@example.net> To: "hvillarreal@example.net" <hvillarreal@example.net> In-Reply-To: <hvillarreal@example.net> Content-Type: text/plain; charset=utf-8; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable X-Mailer: iPhone Mail (5F136) Subject: Re: Using S3 with Hadoop/Nutch Mime-Version: 1.0 (iPhone Mail 5F136) Date: Fri, 3 Oct 2008 09:16:31 -0700 References: <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> <hvillarreal@example.net> X-Virus-Checked: Checked by ClamAV on apache.org I Sent from my iPhone On Oct 1, 2008, at 9:55 PM, "Alexander Aristov" = <hvillarreal@example.net=20 > wrote: > Nothing special should be done with S3 as Hadoop will take care =20 > about it > itself. You must only be sure that the bucket is accessible. Don't =20 > worry > about the tmp folder. > > I configured and run hadoop with S3 recently, it was Ok though I =20 > considered > to switch back to hdfs and use S3 only as a final file store(backup) =20= > because > I saw some productivity downgrade. > > Alexander > > 2008/10/1 Kevin MacDonald <hvillarreal@example.net> > >> The problem is not that I want to mix file systems. I would like to =20= >> start >> from scratch with Hadoop configured to use S3. But to do that I =20 >> think that >> various Paths which Nutch
{'source': 'Apache Software Foundation Public Mail Archives'}