From nutch-agentsamanthafuller@example.net Sun Mar 11 14:50:07 2007
Return-Path: <nutch-agentsamanthafuller@example.net>
Delivered-To: samanthafuller@example.net
Received: (qmail 29640 invoked from network); 11 Mar 2007 14:50:06 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 11 Mar 2007 14:50:06 -0000
Received: (qmail 81330 invoked by uid 500); 11 Mar 2007 14:50:15 -0000
Delivered-To: samanthafuller@example.net
Received: (qmail 81309 invoked by uid 500); 11 Mar 2007 14:50:15 -0000
Mailing-List: contact samanthafuller@example.net; run by ezmlm
Precedence: bulk
List-Help: <mailto:samanthafuller@example.net>
List-Unsubscribe: <mailto:samanthafuller@example.net>
List-Post: <mailto:samanthafuller@example.net>
List-Id: <nutch-agent.lucene.apache.org>
Reply-To: samanthafuller@example.net
Delivered-To: mailing list samanthafuller@example.net
Received: (qmail 81298 invoked by uid 99); 11 Mar 2007 14:50:14 -0000
Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Mar 2007 07:50:14 -0700
X-ASF-Spam-Status: No, hits=2.9 required=10.0
	tests=HTML_10_20,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (herse.apache.org: domain of samanthafuller@example.net designates 83.161.120.155 as permitted sender)
Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Mar 2007 06:50:03 -0800
Received: by nf-out-0910.google.com with SMTP id g2so1481108nfe
        for <samanthafuller@example.net>; Sun, 11 Mar 2007 07:49:42 -0700 (PDT)
DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed;
        d=gmail.com; s=beta;
        h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type;
        b=peZVxofhomrsAj5GP17uihMhDsxxFQv6pbZCDd0GZb+uoeH9IQd75PsiAaQbwOc7sd+1hyy6+ZM8LFRTT202Dv5kbHHN+CrnNVi76IOihFi9eQaIRg3beD7SJBsFaEh4HPAh13pV91mWVsD57LEnrgZ01QxK+oMsJi81uHABGDI=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:mime-version:content-type;
        b=TWO1pwGLnztQHE3zYKM9GbO+W1/HiGu+hnjdmSwmAk7gOHRwn/W9cMU7ML5DD0JSD56RnusmqKbyppJXH48hCOddfWKUBBIoNABekRGypSNPtOYl4cvAJh8LpWQmkmANxrA8ucDA/ZPRm2yjLnJpmZsUuh0Kfzr0obzngS/E5Ms=
Received: by 83.161.120.155 with SMTP id b7mr491139hug.1173624581943;
        Sun, 11 Mar 2007 07:49:41 -0700 (PDT)
Received: by 83.161.120.155 with HTTP; Sun, 11 Mar 2007 07:49:41 -0700 (PDT)
Message-ID: <samanthafuller@example.net>
Date: Sun, 11 Mar 2007 09:49:41 -0500
From: "d e" <samanthafuller@example.net>
To: samanthafuller@example.net
Subject: Has anyone ever used AmazonEC2 to do lots of spidering concurrently? And what about Amazon S3 (Simple Storage Service) ?
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_22191_3749544.1173624581953"
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_22191_3749544.1173624581953
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

It would seem that one could use
http://wiki.apache.org/lucene-hadoop/AmazonEC2 to run many hours of
spidering in a single hour by having a bunch of xen virtual machine
instances set to do this. If one started up and shut down instances for the
crawl, it would not seem to be expensive to do a lot of work concurrently on
a big crawl. Is this practical? Why or why not?

Having a virtual server farm that apears just when I need it and only costs
me 10 cents / machine per hour and zero when I'm done with my spidering
sounds like something I should explore apart from just the coolness of it.

What about actually running the web front end search site on EC2? Would that
be wise? Will I get the performance that I need to be seen as a responsive
website?

When they talk about Amazon S3 (Simple Storage Service), is this raw disk
space that they are selling or backed up, guaranteed to be there storage?

Does anyone on the list of actual experience with either of these offerings
or any competitive offerings from anyone else?

I assume that Amazon has a huge infrastructure with great connectivity. Do I
get the benefit of all that if I use EC2 really, in practical reality? Will
it actually help me keep my search index up to date?

Thanks!

------=_Part_22191_3749544.1173624581953--

From nutch-agentsamanthafuller@example.net Tue Mar 27 10:30:22 2007
Return-Path: <nutch-agentsamanthafuller@example.net>
Delivered-To: samanthafuller@example.net
Received: (qmail 2251 invoked from network); 27 Mar 2007 10:30:22 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 27 Mar 2007 10:30:22 -0000
Received: (qmail 47251 invoked by uid 500); 27 Mar 2007 10:30:29 -0000
Delivered-To: samanthafuller@example.net
Received: (qmail 47076 invoked by uid 500); 27 Mar 2007 10:30:28 -0000
Mailing-List: contact samanthafuller@example.net; run by ezmlm
Precedence: bulk
List-Help: <mailto:samanthafuller@example.net>
List-Unsubscribe: <mailto:samanthafuller@example.net>
List-Post: <mailto:samanthafuller@example.net>
List-Id: <nutch-agent.lucene.apache.org>
Reply-To: samanthafuller@example.net
Delivered-To: mailing list samanthafuller@example.net
Delivered-To: moderator for samanthafuller@example.net
Received: (qmail 1898 invoked by uid 99); 26 Mar 2007 06:56:27 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: pass (hermes.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
  b=HHhsRRX2SFDIWU5n1ZrkDZn+4PhLZmG7C+6tMjDHopzdjDPYHlfdwZcrRerkEnvfYHvcYeCeffy0EpdPWaDVeMcS7xM7u4wqxnAtHCiTMNhTcYhGvYg6vohOC2GWIRhLLBVekESVXnhrAwG0i4hWYQpYB6BfA997TghGCYFom5E=;
X-YMail-OSG: l1kXvpQVM1lMCBNptgs6YgdZcu_p7OsZAg6uEgs1ihYaexUPAt2TJF86kXh8vsfVQEgksy.UQsaRu.yLDy3OSEBf0TQLi15bJa5g5Oa6v1MaS1VNzItXbxg9U7EbTY1t7cYm2or.UfTS4cKjgcHwpjaXFg--
Date: Sun, 25 Mar 2007 23:54:56 -0700 (PDT)
From: rahul garg <samanthafuller@example.net>
Reply-To: samanthafuller@example.net
Subject: Customizing nutch to be used as a LOCAL SEARCH ENGINE
To: samanthafuller@example.net
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
Message-ID: <samanthafuller@example.net>
X-Virus-Checked: Checked by ClamAV on apache.org

Hi,

I am basically new to nutch.. and i have to build a
local serch engine
for a city..
for example, search engine for city-bangalore, where i
can look for a
restaurant or anything depending on locality...

Any kind of help will be of great use..

Thanks,
Rahul


 
____________________________________________________________________________________
Food fight? Enjoy some healthy debate 
in the Yahoo! Answers Food & Drink Q&A.
http://answers.yahoo.com/dir/?link=list&sid=396545367

From nutch-agentsamanthafuller@example.net Tue Mar 27 23:26:29 2007
Return-Path: <nutch-agentsamanthafuller@example.net>
Delivered-To: samanthafuller@example.net
Received: (qmail 46822 invoked from network); 27 Mar 2007 23:26:28 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 27 Mar 2007 23:26:28 -0000
Received: (qmail 13241 invoked by uid 500); 27 Mar 2007 23:26:36 -0000
Delivered-To: samanthafuller@example.net
Received: (qmail 13069 invoked by uid 500); 27 Mar 2007 23:26:35 -0000
Mailing-List: contact samanthafuller@example.net; run by ezmlm
Precedence: bulk
List-Help: <mailto:samanthafuller@example.net>
List-Unsubscribe: <mailto:samanthafuller@example.net>
List-Post: <mailto:samanthafuller@example.net>
List-Id: <nutch-agent.lucene.apache.org>
Reply-To: samanthafuller@example.net
Delivered-To: mailing list samanthafuller@example.net
Received: (qmail 13055 invoked by uid 99); 27 Mar 2007 23:26:35 -0000
Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Mar 2007 16:26:35 -0700
X-ASF-Spam-Status: No, hits=2.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (herse.apache.org: domain of samanthafuller@example.net designates 83.161.120.155 as permitted sender)
Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Mar 2007 16:26:27 -0700
Received: by nf-out-0910.google.com with SMTP id g2so3162235nfe
        for <samanthafuller@example.net>; Tue, 27 Mar 2007 16:26:06 -0700 (PDT)
DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed;
        d=gmail.com; s=beta;
        h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=MleWTy3EQkW+x1ObyG//BU/lRccCsSEyojgUMn4ptInCef9tVI7lmJKnGRWuobfqUKNU6PKQi6twxw3+6DLVq975yWAC8EOV+pRQOI8MlCkQ42Ylp4z2lmzHhjtcwSz13hdUwAE7QKLbg24hdY6H8LEvbZLC+Kr9YmMxgP8wfSY=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=YpE9cjp2hBc1wUR2+eErtRfqem0sI/REN/1xWU9cmrSnLgsCLv23J2JRbHBvuiwT+sHyiz25j7nK+K5rlDML6W+CCVW37LVuczUnqZEPncwjx2H4gJUnmgiBEOxTSvTUVLEiNASxTJU1LtakwbI+ONRIGawgezfjV48xIsWLLxA=
Received: by 83.161.120.155 with SMTP id l6mr3826611hud.1175037965991;
        Tue, 27 Mar 2007 16:26:05 -0700 (PDT)
Received: by 83.161.120.155 with HTTP; Tue, 27 Mar 2007 16:26:05 -0700 (PDT)
Message-ID: <samanthafuller@example.net>
Date: Wed, 28 Mar 2007 09:26:05 +1000
From: "Paul Liddelow" <samanthafuller@example.net>
To: samanthafuller@example.net, samanthafuller@example.net
Subject: Re: Customizing nutch to be used as a LOCAL SEARCH ENGINE
In-Reply-To: <samanthafuller@example.net>
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_19907_6069412.1175037965940"
References: <samanthafuller@example.net>
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_19907_6069412.1175037965940
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Your best bet is to try the tutorials on the nutch home page.

Cheers

On 3/26/07, rahul garg <samanthafuller@example.net> wrote:
>
> Hi,
>
> I am basically new to nutch.. and i have to build a
> local serch engine
> for a city..
> for example, search engine for city-bangalore, where i
> can look for a
> restaurant or anything depending on locality...
>
> Any kind of help will be of great use..
>
> Thanks,
> Rahul
>
>
>
>
> ____________________________________________________________________________________
> Food fight? Enjoy some healthy debate
> in the Yahoo! Answers Food & Drink Q&A.
> http://answers.yahoo.com/dir/?link=list&sid=396545367
>

------=_Part_19907_6069412.1175037965940--

From nutch-agentsamanthafuller@example.net Wed Mar 28 09:22:54 2007
Return-Path: <nutch-agentsamanthafuller@example.net>
Delivered-To: samanthafuller@example.net
Received: (qmail 11722 invoked from network); 28 Mar 2007 09:22:54 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 28 Mar 2007 09:22:54 -0000
Received: (qmail 74126 invoked by uid 500); 28 Mar 2007 09:23:01 -0000
Delivered-To: samanthafuller@example.net
Received: (qmail 74097 invoked by uid 500); 28 Mar 2007 09:23:01 -0000
Mailing-List: contact samanthafuller@example.net; run by ezmlm
Precedence: bulk
List-Help: <mailto:samanthafuller@example.net>
List-Unsubscribe: <mailto:samanthafuller@example.net>
List-Post: <mailto:samanthafuller@example.net>
List-Id: <nutch-agent.lucene.apache.org>
Reply-To: samanthafuller@example.net
Delivered-To: mailing list samanthafuller@example.net
Received: (qmail 74083 invoked by uid 99); 28 Mar 2007 09:23:00 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received-SPF: pass (hermes.apache.org: local policy)
Received: from [83.161.120.155] (HELO web60623.mail.yahoo.com) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with SMTP; Wed, 28 Mar 2007 02:23:00 -0700
Received: (qmail 78991 invoked by uid 60001); 28 Mar 2007 09:21:31 -0000
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
  b=wkQeA6oGstVKoo+CN8RM5monVcCL+OZpmR+Ve4gFxvEGAGxAuVzoTb+g6YRm5kBLDZcideuvtUJiWMmuUTg4oxlhJuPqtYpq8yoWzXn86AODR/LUG/IYze0MN5xaEuntZrB1OYykkjcERlOXyNDnfZuJpVCebjqCYxNqUjj5GCY=;
X-YMail-OSG: JJ3Qx7oVM1n1MwBYdh__2SPxsSgOK5EVMvCxP9K4uoNmwMRWO7.3aFY4Yily8BhO9RvN7niyVtMqmuitV0bzULbtwRklhaW57eJQp9RxAE0x91lyx6sM
Received: from [83.161.120.155] by web60623.mail.yahoo.com via HTTP; Wed, 28 Mar 2007 02:21:31 PDT
Date: Wed, 28 Mar 2007 02:21:31 -0700 (PDT)
From: rahul garg <samanthafuller@example.net>
Reply-To: samanthafuller@example.net
Subject: Re: Customizing nutch to be used as a LOCAL SEARCH ENGINE
To: samanthafuller@example.net
In-Reply-To: <samanthafuller@example.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit
Message-ID: <samanthafuller@example.net>
X-Virus-Checked: Checked by ClamAV on apache.org


hi,
Thanks for replying.but tutorials i have used, they
are for using nutch in a default mode.
i want it specifically for local search.. basically is
there any good documentation on how nutch indexes
based on fields and how can i add new fields and all.
also docs on query parsing.

Thanks
rahul



--- Paul Liddelow <samanthafuller@example.net> wrote:

> Your best bet is to try the tutorials on the nutch
> home page.
> 
> Cheers
> 
> On 3/26/07, rahul garg <samanthafuller@example.net> wrote:
> >
> > Hi,
> >
> > I am basically new to nutch.. and i have to build
> a
> > local serch engine
> > for a city..
> > for example, search engine for city-bangalore,
> where i
> > can look for a
> > restaurant or anything depending on locality...
> >
> > Any kind of help will be of great use..
> >
> > Thanks,
> > Rahul
> >
> >
> >
> >
> >
>
____________________________________________________________________________________
> > Food fight? Enjoy some healthy debate
> > in the Yahoo! Answers Food & Drink Q&A.
> >
>
http://answers.yahoo.com/dir/?link=list&sid=396545367
> >
> 



 
____________________________________________________________________________________
We won't tell. Get more on shows you hate to love 
(and love to hate): Yahoo! TV's Guilty Pleasures list.
http://tv.yahoo.com/collections/265 

From nutch-agentsamanthafuller@example.net Wed Mar 28 16:10:48 2007
Return-Path: <nutch-agentsamanthafuller@example.net>
Delivered-To: samanthafuller@example.net
Received: (qmail 20784 invoked from network); 28 Mar 2007 16:10:46 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155)
  by minotaur.apache.org with SMTP; 28 Mar 2007 16:10:46 -0000
Received: (qmail 69953 invoked by uid 500); 28 Mar 2007 16:10:53 -0000
Delivered-To: samanthafuller@example.net
Received: (qmail 69927 invoked by uid 500); 28 Mar 2007 16:10:53 -0000
Mailing-List: contact samanthafuller@example.net; run by ezmlm
Precedence: bulk
List-Help: <mailto:samanthafuller@example.net>
List-Unsubscribe: <mailto:samanthafuller@example.net>
List-Post: <mailto:samanthafuller@example.net>
List-Id: <nutch-agent.lucene.apache.org>
Reply-To: samanthafuller@example.net
Delivered-To: mailing list samanthafuller@example.net
Received: (qmail 69914 invoked by uid 99); 28 Mar 2007 16:10:53 -0000
Received: from herse.apache.org (HELO herse.apache.org) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Mar 2007 09:10:53 -0700
X-ASF-Spam-Status: No, hits=2.6 required=10.0
	tests=HTML_00_10,HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (herse.apache.org: domain of samanthafuller@example.net designates 83.161.120.155 as permitted sender)
Received: from [83.161.120.155] (HELO nf-out-0910.google.com) (83.161.120.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Mar 2007 09:10:43 -0700
Received: by nf-out-0910.google.com with SMTP id g2so3386158nfe
        for <samanthafuller@example.net>; Wed, 28 Mar 2007 09:10:22 -0700 (PDT)
DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed;
        d=gmail.com; s=beta;
        h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type;
        b=q+cbQrtOxUsQxlKd0MF3ryA9VFNtILmhC/JmD/XFIXaH5tH0Vi4OKg/0LN0Aox88bYM8sjjOjjFI8n9wSgXm2IFxEysgVhQE8kynFKm1HkcYNwyx2Y4d3qf4kImLmMKV3prB07wOsKntU47+2vnqUwoI5YWRU7bfoeHZ04KIiNk=
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:mime-version:content-type;
        b=CXbITsL5NxWvkbzfbHbeckjIf4Tv33nEyYcXOP9uB593sIAwAiNH1iedTizgqMLOTuZijd9yupbVk7a9tnEn5kpznYjiunn9TL13sj/52I4Pl9kpJuIgQc4fR3XoIm7BL6wDVhX+I9VwhZuH8nfK/UIFFTe0IAZqqjUNHM26irA=
Received: by 83.161.120.155 with SMTP id t14mr19155882bud.1175098221911;
        Wed, 28 Mar 2007 09:10:21 -0700 (PDT)
Received: by 83.161.120.155 with HTTP; Wed, 28 Mar 2007 09:10:21 -0700 (PDT)
Message-ID: <samanthafuller@example.net>
Date: Wed, 28 Mar 2007 17:10:21 +0100
From: "james redden" <samanthafuller@example.net>
To: samanthafuller@example.net
Subject: Help with nutch
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_105581_18758992.1175098221216"
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_105581_18758992.1175098221216
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

When I run nutch as a standard user the crawl barfs at 893M (1021 disk space
used in total). Java jumps up to 100% CPU time at this point. I have no
quotas or limits on this user account. A limitation in java? Has anyone seen
this issue before?

Thanks in advance.

------=_Part_105581_18758992.1175098221216--


{'source': 'Apache Software Foundation Public Mail Archives'}