From melendeztammy@example.org Mon Feb 01 06:44:10 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 16314 invoked from network); 1 Feb 2010 06:44:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 06:44:10 -0000 Received: (qmail 89023 invoked by uid 500); 1 Feb 2010 06:44:09 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 88818 invoked by uid 500); 1 Feb 2010 06:44:08 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 88807 invoked by uid 99); 1 Feb 2010 06:44:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 06:44:08 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f215.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 06:44:00 +0000 Received: by gxk7 with SMTP id 7so3779083gxk.12 for <melendeztammy@example.org>; Sun, 31 Jan 2010 22:43:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:content-type :content-transfer-encoding; bh=FgcDABTfMSPZW/LPRsN2Y/ZDq1//KjnNAe9M+zGiz4E=; b=Nf7ZOsVM+Zu+gPUV6T08BkIJ7CyA8q24vBtCapkZTwHRUH8fGzXVmN1QMlpzIq86l+ cKREwkvYA42/nkC43/IUI71FL/i3GHr/w1sTEYWH5jANZ1OQUhiWuHMaIsW8yHFUPgYl bVjw+K7vxK748DSJGdZWjokiDI4qa62lGvVkY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; b=oCgW8I0QQph4Q4YOfLXNBX4SEONr4ZmJTiI1hIdqSJLFxBffNj+WpLwrOfUDsO2DIO oh/SMnxr7pwUSH7+9qvHuNmGVfAb+UZw8+ISrl+9Zwol4rsiSdHzBIymBSBzOWxeGof8 0JgEvXdCHAKlaxmESQzqpEYxAIcVr0TNCjlOU= Received: by 83.161.120.155 with SMTP id l28mr4666288ann.90.1265006619408; Sun, 31 Jan 2010 22:43:39 -0800 (PST) Received: from ?83.161.120.155? (dhcp-13-121.it.uts.edu.au [83.161.120.155]) by mx.google.com with ESMTPS id 14sm3047094gxk.6.283.161.120.155.43.36 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 31 Jan 2010 22:43:38 -0800 (PST) Message-ID: <melendeztammy@example.org> Date: Mon, 01 Feb 2010 17:43:29 +1100 From: Patrick Barnes <melendeztammy@example.org> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:83.161.120.155) Gecko/20091204 Lightning/1.0b1 Thunderbird/3.0 MIME-Version: 1.0 To: melendeztammy@example.org Subject: Trees and leaf nodes Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit I have a large tree of documents that I want to display in a dynamic treeview, but I'm not sure how best to determine whether a given node has children or not. Each document has a 'path' element giving it's place in the tree, eg this following doc has the parent 'AA', which in turn has the top-level parent 'A'. doc = { '_id':'11203', 'path':['A','AA','11203'] ... }; Fetching and displaying the entire tree is simple... this view: function(doc) { emit(doc.path, doc.display_name); } emits in lexicographical order, and some php code turns it into a nested list. The entire tree is slow to retrieve and display in one go. With jquery treeview-async, I retrieve and display only the top-level nodes on page load, then on expanding a node it will call the server with the ID, asking for children. I have a view that returns just the immediate children of the given key: function(doc) { if (doc.doc_type=='group') { //Emit the path of the parent... doc.path.pop(); emit(doc.path, doc.display_name); } } So things mostly work nicely - I can click on a node, and it will dynamically load children, rinse, repeat. It looks messy if the node has a little 'expand' icon next to it, though - but there are no actual children below it. Is it possible to create a view that will emit as value whether each node has children? Cheers, -Patrick Barnes From melendeztammy@example.org Mon Feb 01 11:05:27 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 84704 invoked from network); 1 Feb 2010 11:05:27 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 11:05:27 -0000 Received: (qmail 27399 invoked by uid 500); 1 Feb 2010 11:05:25 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 27330 invoked by uid 500); 1 Feb 2010 11:05:24 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 27320 invoked by uid 99); 1 Feb 2010 11:05:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 11:05:24 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO sasl.smtp.pobox.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 11:05:13 +0000 Received: from sasl.smtp.pobox.com (unknown [83.161.120.155]) by a-pb-sasl-quonix.pobox.com (Postfix) with ESMTP id C7D939620B for <melendeztammy@example.org>; Mon, 1 Feb 2010 06:04:50 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=date:from:to :subject:message-id:references:mime-version:content-type :in-reply-to; s=sasl; bh=rR2JPqiaboeMMFz204wuoEC4hEU=; b=dU9RlcJ CCi3ffraoLT1YRHslt2n2+kgAO63uAReOBbvyMGYCB+iO24NSe/IH6D/BcWydHCQ YgKlGH0VWw3XsGA4hA8EGMnsTIpvsKYEKf6Cgw65ar+JJiFPnrpJ5HK3BCBRKB3l siw7TVMJXS+qDOuMx/sj4nbyWozKuQcrAv48= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=date:from:to :subject:message-id:references:mime-version:content-type :in-reply-to; q=dns; s=sasl; b=CV+JY7LtJ1L/T7eMHZC6acfxyCJPEjhtl BjtfbVGO1H0z0pwNKr7kB8rNZjCEjnjjIEQw+Qp6fhmUueJwFWoYJEqsr3k8O0H8 GUBLpudVSpzBT1p1RMjUC9gxqqlbpER0qUcvR5X92IKPopw3LbKq3SuS+VJCTir6 5LJ0e/hqkk= Received: from a-pb-sasl-quonix. (unknown [83.161.120.155]) by a-pb-sasl-quonix.pobox.com (Postfix) with ESMTP id C49919620A for <melendeztammy@example.org>; Mon, 1 Feb 2010 06:04:50 -0500 (EST) Received: from zino (unknown [83.161.120.155]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by a-pb-sasl-quonix.pobox.com (Postfix) with ESMTPSA id 8F62A96209 for <melendeztammy@example.org>; Mon, 1 Feb 2010 06:04:50 -0500 (EST) Received: from lists by zino with local (Exim 4.69) (envelope-from <melendeztammy@example.org>) id 1Nbu5I-0001XW-Nj for melendeztammy@example.org; Mon, 01 Feb 2010 11:04:48 +0000 Date: Mon, 1 Feb 2010 11:04:48 +0000 From: Brian Candler <melendeztammy@example.org> To: melendeztammy@example.org Subject: Re: Trees and leaf nodes Message-ID: <melendeztammy@example.org> References: <melendeztammy@example.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <melendeztammy@example.org> User-Agent: Mutt/1.5.20 (2009-06-14) X-Pobox-Relay-ID: 9E3F7726-0F21-11DF-95C8-6AF7ED7EF46B-28021239!a-pb-sasl-quonix.pobox.com X-Virus-Checked: Checked by ClamAV on apache.org On Mon, Feb 01, 2010 at 05:43:29PM +1100, Patrick Barnes wrote: > It looks messy if the node has a little 'expand' icon next to it, > though - but there are no actual children below it. Is it possible > to create a view that will emit as value whether each node has > children? I think the client needs to make two queries. Under a given node: 1. Get the list of the children 2. Determine whether each child also has children You can do (2) using a single multi-key query, e.g. {keys:[c1, c2, c3, ...]} Using your existing view you'll get all the grandchildren, which you probably don't care about but can just check whether there are any or not. To reduce the size of the returned data you could make a grouped query instead. With a suitable reduce function on your view, then you could get back a count of the number of children for each child, which might be useful to display in your UI anyway. HTH, Brian. From melendeztammy@example.org Mon Feb 01 13:16:39 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 21467 invoked from network); 1 Feb 2010 13:16:38 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 13:16:38 -0000 Received: (qmail 93736 invoked by uid 500); 1 Feb 2010 13:16:37 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 93648 invoked by uid 500); 1 Feb 2010 13:16:37 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 93638 invoked by uid 99); 1 Feb 2010 13:16:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 13:16:37 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-px0-f180.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 13:16:29 +0000 Received: by pxi10 with SMTP id 10so2951431pxi.13 for <melendeztammy@example.org>; Mon, 01 Feb 2010 05:16:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:references:in-reply-to :content-type:content-transfer-encoding; bh=dJgJDK7/4RBC/g2mxsNxmx7kJxI0HPge7j1V9S2cWr8=; b=s7LJhVWsy4kCoDG9Ag0S8uAxKTo4VnjYLcW6A+fX7v+CQHAIMUKV4Ndgbxj8KRrPbG Q1vyiv2sKT2+frpWuH9FTTnoekzDN0nJIR6DAICQT6PZa85xa7U9DKOuUHyMiC07y4b5 fGBK1kUMJBskHOBrMitp6gV5uZeR2STjKpxCI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; b=HCWPlfcnuqhpIkKACTrs7jaTDStz/jUoJ2PUKixriMD/HljWnfFDfRUorgDSw/a6b/ XiXd4FFfg3Brg4uxnJCWJGMQXCte1/bi3TYakUFvg6vuIEagv+Ny3Jjncz49gCdgg6Ja iM3uSr7RC4cCyr3vavTvNZvw9m2pxRCz5YMl4= Received: by 83.161.120.155 with SMTP id p10mr2079112waj.179.1265030168908; Mon, 01 Feb 2010 05:16:08 -0800 (PST) Received: from ?83.161.120.155? (124-171-17-24.dyn.iinet.net.au [83.161.120.155]) by mx.google.com with ESMTPS id 22sm644538pzk.14.283.161.120.155.16.05 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 01 Feb 2010 05:16:06 -0800 (PST) Message-ID: <melendeztammy@example.org> Date: Tue, 02 Feb 2010 00:15:58 +1100 From: Patrick Barnes <melendeztammy@example.org> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:83.161.120.155) Gecko/20091204 Lightning/1.0b1 Thunderbird/3.0 MIME-Version: 1.0 To: melendeztammy@example.org Subject: Re: Trees and leaf nodes References: <melendeztammy@example.org> <melendeztammy@example.org> In-Reply-To: <melendeztammy@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Thanks, that sounds like should work. The only boundary case for a query to get the number of children is that if there are no children for that key it will return a blank result, not a '0' record. I thought - hey, if I only cared about presence or absence of children I could just use the get_children view, passing it key=$parent_id & limit=0 ... but the view summary (at least in 0.10.0) only returns total row count and offset, not how many rows would have been returned if limit=0 had not been present. ... okay, having just typed the last paragraph I realise - even if feasible, that would only work using one key at a time. *yawns* But the grouped query makes sense, I'll test it tomorrow. -P On 1/02/2010 10:04 PM, Brian Candler wrote: > On Mon, Feb 01, 2010 at 05:43:29PM +1100, Patrick Barnes wrote: >> It looks messy if the node has a little 'expand' icon next to it, >> though - but there are no actual children below it. Is it possible >> to create a view that will emit as value whether each node has >> children? > > I think the client needs to make two queries. Under a given node: > > 1. Get the list of the children > 2. Determine whether each child also has children > > You can do (2) using a single multi-key query, e.g. > > {keys:[c1, c2, c3, ...]} > > Using your existing view you'll get all the grandchildren, which you > probably don't care about but can just check whether there are any or not. > > To reduce the size of the returned data you could make a grouped query > instead. With a suitable reduce function on your view, then you could get > back a count of the number of children for each child, which might be useful > to display in your UI anyway. > > HTH, > > Brian. > From melendeztammy@example.org Mon Feb 01 14:25:14 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 53940 invoked from network); 1 Feb 2010 14:25:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 14:25:14 -0000 Received: (qmail 87706 invoked by uid 500); 1 Feb 2010 14:25:12 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 87621 invoked by uid 500); 1 Feb 2010 14:25:12 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 87611 invoked by uid 99); 1 Feb 2010 14:25:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 14:25:12 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 83.161.120.155 is neither permitted nor denied by domain of melendeztammy@example.org) Received: from [83.161.120.155] (HELO golias.ruk.cuni.cz) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 01 Feb 2010 14:25:04 +0000 Received: (qmail 29122 invoked from network); 1 Feb 2010 14:24:42 -0000 Received: from nat.uk-simple.cuni.cz (HELO ?83.161.120.155?) (83.161.120.155) by golias.ruk.cuni.cz with SMTP; 1 Feb 2010 14:24:42 -0000 Message-Id: <melendeztammy@example.org> From: =?UTF-8?Q?Karel_Mina=C5=99=C3=ADk?= <melendeztammy@example.org> To: melendeztammy@example.org In-Reply-To: <melendeztammy@example.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: How to import data quickly Date: Mon, 1 Feb 2010 15:24:42 +0100 References: <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> X-Mailer: Apple Mail (2.936) Hi, just for info, on a current project I needed to import 6mil+ of docs, and the sweet spot was 10K docs per on batch upload. Higher values gave worse results. I don't have the numbers handy, but it took couple of hours to convert the docs from CSV and bulk upload them into Couch, I guess like 8hrs (on a rather old IBM Blade machine)... (And the real pain was handling malformed CSV parts, patching FasterCSV to not choke on it, etc.) Karel On 28.Jan, 2010, at 15:02 , Troy Kruthoff wrote: > Just curious, what batch size did you use... I was just getting to > run some test data to see where the sweet spot is for our hardware, > I remember reading somewhere that someone thought it was around 3k > docs. > > Troy > > > On Jan 28, 2010, at 4:21 AM, Sean Clark Hess wrote: > >> Sweet... down to 28 minutes with bulk. Thanks >> >> On Thu, Jan 28, 2010 at 4:25 AM, Sean Clark Hess >> <melendeztammy@example.org> wrote: >> >>> Ah, I forgot about bulk! Thanks! >>> >>> >>> On Thu, Jan 28, 2010 at 4:24 AM, Alex Koshelev >>> <melendeztammy@example.org> wrote: >>> >>>> How do you import data to CouchDB? Do you use _bulk API? >>>> --- >>>> Alex Koshelev >>>> >>>> >>>> On Thu, Jan 28, 2010 at 1:51 PM, Sean Clark Hess <melendeztammy@example.org >>>> > >>>> wrote: >>>> >>>>> I'm trying to import 7 million rows into couch from an xml >>>>> document. If >>>> I >>>>> use a database with a "normal" interface (comparing with Mongo >>>>> here), >>>> the >>>>> process completes in 37 minutes. If I use couch, it takes 10 >>>>> hours. I >>>> think >>>>> it might be due to the overhead of the http interface, but I'm >>>>> not sure. >>>>> >>>>> Is there any way to get data in there faster? >>>>> >>>>> ~sean >>>>> >>>> >>> >>> > From melendeztammy@example.org Mon Feb 01 14:28:15 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 55239 invoked from network); 1 Feb 2010 14:28:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 14:28:15 -0000 Received: (qmail 90362 invoked by uid 500); 1 Feb 2010 14:28:13 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 90297 invoked by uid 500); 1 Feb 2010 14:28:13 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 90204 invoked by uid 99); 1 Feb 2010 14:28:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 14:28:13 +0000 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 83.161.120.155 is neither permitted nor denied by domain of melendeztammy@example.org) Received: from [83.161.120.155] (HELO golias.ruk.cuni.cz) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 01 Feb 2010 14:28:05 +0000 Received: (qmail 30184 invoked from network); 1 Feb 2010 14:27:43 -0000 Received: from nat.uk-simple.cuni.cz (HELO ?83.161.120.155?) (83.161.120.155) by golias.ruk.cuni.cz with SMTP; 1 Feb 2010 14:27:43 -0000 Message-Id: <melendeztammy@example.org> From: =?UTF-8?Q?Karel_Mina=C5=99=C3=ADk?= <melendeztammy@example.org> To: melendeztammy@example.org In-Reply-To: <melendeztammy@example.org> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Subject: Re: Measuring duration of view index building Date: Mon, 1 Feb 2010 15:27:43 +0100 References: <melendeztammy@example.org> <melendeztammy@example.org> X-Mailer: Apple Mail (2.936) Hi Roger, >> Of course, the best thing in my case would be, if couch itself >> would log >> something like "start/end building index for _design/mydoc" -- but as >> far as I know, there's no way to do that? > Look in the log file. apologies, I am severely behind my inbox. I am stupid enough to oversee the log entries for view indices building. But, how'd you relatively easily parse that? If there isn't something a la "Starting building XXX", it's rather cumbersome to work with? I guess `time curl ...` is still my friend. > Do a GET against /_active_tasks which will tell you if view indexing > is > running as well as progress. And I am stupid enough to oversee that, and the nice GUI in Futon, thanks!! Karel From melendeztammy@example.org Mon Feb 01 16:24:34 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 91626 invoked from network); 1 Feb 2010 16:24:34 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 16:24:34 -0000 Received: (qmail 91506 invoked by uid 500); 1 Feb 2010 16:24:32 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 91433 invoked by uid 500); 1 Feb 2010 16:24:32 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 91423 invoked by uid 99); 1 Feb 2010 16:24:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:24:32 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-px0-f180.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:24:24 +0000 Received: by pxi10 with SMTP id 10so3144665pxi.13 for <melendeztammy@example.org>; Mon, 01 Feb 2010 08:24:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=toJWdtUzb+VsjacvCh2moXLnfbyR6OcIjqHgAejMBUo=; b=bRS4qc3yGT6i+2JwOtf1MVszKhRgWH1cyTJJcpGsSl+Kv/H+y96rcMtiifIW1ZBx8Q 0SOySaEIP5j2O9n3KB2EmdV38YglgCnwWOiT72760dYlNOFsOk0T3FfWj5BPXa0vrNk5 Dm7RcGVdJ99KTEzUUKCla6qYybRXYzl/gYnZY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=Ny3hlulqlSJmT2QNQdm458pcpuoHEBQd/FTWFalJiDgvrUBcszw0Npg9nAZ/TUtNp4 77iE86mRjLAlZ/kvTmFe1hUIONS5YolG5mDDOkWtQ0Psmdhe8unhIYdyDBtrnWqpCpO7 rNPPys4p3wBf6CEVFtFZcy9+OoEORRWmmDbeY= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id u16mr1858212rvh.209.1265041443126; Mon, 01 Feb 2010 08:24:03 -0800 (PST) In-Reply-To: <melendeztammy@example.org> References: <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> Date: Mon, 1 Feb 2010 08:24:03 -0800 Message-ID: <melendeztammy@example.org> Subject: Re: Trees and leaf nodes From: Adam Wolff <melendeztammy@example.org> To: melendeztammy@example.org Content-Type: multipart/alternative; boundary=000e0cd10562d8003a047e8c69ec X-Virus-Checked: Checked by ClamAV on apache.org --000e0cd10562d8003a047e8c69ec Content-Type: text/plain; charset=ISO-8859-1 If you only care about whether or not there are children, you could reduce to the count of children for each key. A On Mon, Feb 1, 2010 at 5:15 AM, Patrick Barnes <melendeztammy@example.org> wrote: > Thanks, that sounds like should work. The only boundary case for a query to > get the number of children is that if there are no children for that key it > will return a blank result, not a '0' record. > > I thought - hey, if I only cared about presence or absence of children I > could just use the get_children view, passing it key=$parent_id & limit=0 > ... but the view summary (at least in 0.10.0) only returns total row count > and offset, not how many rows would have been returned if limit=0 had not > been present. > > ... okay, having just typed the last paragraph I realise - even if > feasible, that would only work using one key at a time. *yawns* > > But the grouped query makes sense, I'll test it tomorrow. > > -P > > > > On 1/02/2010 10:04 PM, Brian Candler wrote: > >> On Mon, Feb 01, 2010 at 05:43:29PM +1100, Patrick Barnes wrote: >> >>> It looks messy if the node has a little 'expand' icon next to it, >>> though - but there are no actual children below it. Is it possible >>> to create a view that will emit as value whether each node has >>> children? >>> >> >> I think the client needs to make two queries. Under a given node: >> >> 1. Get the list of the children >> 2. Determine whether each child also has children >> >> You can do (2) using a single multi-key query, e.g. >> >> {keys:[c1, c2, c3, ...]} >> >> Using your existing view you'll get all the grandchildren, which you >> probably don't care about but can just check whether there are any or not. >> >> To reduce the size of the returned data you could make a grouped query >> instead. With a suitable reduce function on your view, then you could get >> back a count of the number of children for each child, which might be >> useful >> to display in your UI anyway. >> >> HTH, >> >> Brian. >> >> --000e0cd10562d8003a047e8c69ec-- From melendeztammy@example.org Mon Feb 01 16:28:08 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 93006 invoked from network); 1 Feb 2010 16:28:08 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 16:28:08 -0000 Received: (qmail 4238 invoked by uid 500); 1 Feb 2010 16:28:07 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 4163 invoked by uid 500); 1 Feb 2010 16:28:07 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 4153 invoked by uid 99); 1 Feb 2010 16:28:07 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:28:07 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO relayout03-q02.dns-servicios.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:27:54 +0000 Received: from relayout03-redir.dns-servicios.com (relayout03-redir.dns-servicios.com [83.161.120.155]) by relayout03.dns-servicios.com (Postfix) with ESMTP id 23F30764E02 for <melendeztammy@example.org>; Mon, 1 Feb 2010 17:27:32 +0100 (CET) Received: from vlc-187.dns-servicios.com (unknown [83.161.120.155]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by relayout03-dsp.dns-servicios.com (Postfix) with ESMTPS id 6A22B61AF6F for <melendeztammy@example.org>; Mon, 1 Feb 2010 17:27:23 +0100 (CET) Received: (qmail 20214 invoked from network); 1 Feb 2010 17:27:24 +0100 Received: from ip85.dominioabsoluto.com (HELO macmini.local) (83.161.120.155) by 82.194.84-53.customers.hostalia.com with SMTP; 1 Feb 2010 17:27:24 +0100 Message-ID: <melendeztammy@example.org> Date: Mon, 01 Feb 2010 17:27:23 +0100 From: Santi Saez <melendeztammy@example.org> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; es-ES; rv:83.161.120.155) Gecko/20100111 Thunderbird/3.0.1 MIME-Version: 1.0 To: melendeztammy@example.org Subject: Best way to store 2^32 IPs in CouchDB Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-RelayOut-Information: AntiSPAM and AntiVIRUS on relayout03 X-RelayOut-MsgID: 6A22B61AF6F.00000 X-RelayOut-SpamCheck: no es spam, SpamAssassin (no almacenado, puntaje=0, requerido 6.5, RDNS_NONE 0.00) X-RelayOut-From: melendeztammy@example.org X-RelayOut-Watermark: 1265646443.5069@tWSmc/VN1liOn9ogjYujlQ X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No Hi, I'm doing some initial tests with CouchDB, trying to store 2^32 IP addresses (approximately 4.3 billions of documents). Documents have only required fields: _id and _rev, but I've noticed that the minimum space occupied by each document is approximately 3.7KB, so I need +14TB disk space only for the basic scheme without any extra field (using IP as unique identifier in integer format). Note that playing with a simple Python script and a binary data file, this data can be stored in 16GB space (each IP 4 = bytes * 2 ^32 addresses). Is it possible to optimize the disk space for what I'm trying to do using CouchDB? Perhaps disabling "something", compressing, or changing _rev field format/size.. thanks!! I haver read the manual for CouchDB perfomance, but I didn't get it: http://wiki.apache.org/couchdb/Performance Regards, -- Santi Saez http://woop.es From melendeztammy@example.org Mon Feb 01 16:31:45 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 94170 invoked from network); 1 Feb 2010 16:31:45 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 16:31:45 -0000 Received: (qmail 12215 invoked by uid 500); 1 Feb 2010 16:31:44 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 12142 invoked by uid 500); 1 Feb 2010 16:31:43 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 12132 invoked by uid 99); 1 Feb 2010 16:31:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:31:43 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO gv-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:31:35 +0000 Received: by gv-out-0910.google.com with SMTP id n8so17848gve.31 for <melendeztammy@example.org>; Mon, 01 Feb 2010 08:31:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=GQODr5a4aaikiwpgBsCxXnVWPevoqoSMymkJefQyLkM=; b=GqpGgNkyjeH63nhzD5KA16ecNvXiTpZssWSr1el/WBMMTyit3QfuxDjXAP7uz4azha HG+ak+W6qy/fhRlBK2oQ7JYSxGNP/hFYq/vx1OAyy9LIlySstRU0B97sVnGfOWlnDAcr 9ShTOEZ1ce2PAI9UstUnAqRdpWnnCJAqFKLc4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=KojfgcNy6Iz+L2FEiUkm+Yy8IRpkT1J9AVqRpUXNVDgBZrEVgpH7u+7DQlhlyewgaR EXFBq+H1dxtAtLDsuToFn4sqs7GcLV8AyNlVHeEvfy0HX2YJ/zMOclNww+e/zXM6JYB4 nJK8eCWGuwlCuDCxLRfVtnetg9kBhQ7iDLfXY= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id e6mr2753627muf.63.1265041874274; Mon, 01 Feb 2010 08:31:14 -0800 (PST) In-Reply-To: <melendeztammy@example.org> References: <melendeztammy@example.org> Date: Mon, 1 Feb 2010 16:31:14 +0000 Message-ID: <melendeztammy@example.org> Subject: Re: Best way to store 2^32 IPs in CouchDB From: Robert Newson <melendeztammy@example.org> To: melendeztammy@example.org Content-Type: text/plain; charset=ISO-8859-1 Try database compaction? B. On Mon, Feb 1, 2010 at 4:27 PM, Santi Saez <melendeztammy@example.org> wrote: > > Hi, > > I'm doing some initial tests with CouchDB, trying to store 2^32 IP addresses > (approximately 4.3 billions of documents). > > Documents have only required fields: _id and _rev, but I've noticed that the > minimum space occupied by each document is approximately 3.7KB, so I need > +14TB disk space only for the basic scheme without any extra field (using IP > as unique identifier in integer format). > > Note that playing with a simple Python script and a binary data file, this > data can be stored in 16GB space (each IP 4 = bytes * 2 ^32 addresses). > > Is it possible to optimize the disk space for what I'm trying to do using > CouchDB? Perhaps disabling "something", compressing, or changing _rev field > format/size.. thanks!! > > I haver read the manual for CouchDB perfomance, but I didn't get it: > > http://wiki.apache.org/couchdb/Performance > > Regards, > > -- > Santi Saez > http://woop.es > From melendeztammy@example.org Mon Feb 01 16:33:21 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 94480 invoked from network); 1 Feb 2010 16:33:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 16:33:21 -0000 Received: (qmail 14706 invoked by uid 500); 1 Feb 2010 16:33:20 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 14651 invoked by uid 500); 1 Feb 2010 16:33:20 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 14641 invoked by uid 99); 1 Feb 2010 16:33:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:33:20 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-bw0-f218.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:33:11 +0000 Received: by bwz10 with SMTP id 10so386193bwz.35 for <melendeztammy@example.org>; Mon, 01 Feb 2010 08:32:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=pnGjJpfXpN6/qGXA+DzNmXJ8XklibHKZSyk8OMHlioU=; b=HpInu48/VGPe73GPIgazQUYatjGo5Dg58fQLskTL6OOQ1ALkrcwcdPT5GM5Kq9gyOX 2JWWB82JJDH8DrBnIspk4iNJLAhr638h8RzGVNQwcmpfz5byPMOyjTX3YRCEmT2r+vpi 9UhtoUsh16u3gse2CqnU6ZJMbAr23ZlS+3TmE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=xJr9cJ/PFlo4TQ7MNxKFX57yAnV6haWDv3eDMLvMXdeZxRaMsx2oMVqqEaprxSwcvq 8i2z6EOpundnDqrN3lTY1oUT0bNQMTTHOmDiddT0zK+FAsTHi0F+B9C586SXrqXmLxsD ihbgQdBxijn2ZuT+EIg8LdNSV7tpxJ50r2e44= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id l14mr3270555bku.50.1265041969761; Mon, 01 Feb 2010 08:32:49 -0800 (PST) In-Reply-To: <melendeztammy@example.org> References: <melendeztammy@example.org> Date: Mon, 1 Feb 2010 18:32:49 +0200 Message-ID: <melendeztammy@example.org> Subject: Re: Best way to store 2^32 IPs in CouchDB From: Elf <melendeztammy@example.org> To: melendeztammy@example.org Content-Type: text/plain; charset=ISO-8859-1 Did you plan to handle IPv6 in future versions of your program? :) 2010/2/1 Santi Saez <melendeztammy@example.org>: > > Hi, > > I'm doing some initial tests with CouchDB, trying to store 2^32 IP addresses > (approximately 4.3 billions of documents). > > Documents have only required fields: _id and _rev, but I've noticed that the > minimum space occupied by each document is approximately 3.7KB, so I need > +14TB disk space only for the basic scheme without any extra field (using IP > as unique identifier in integer format). > > Note that playing with a simple Python script and a binary data file, this > data can be stored in 16GB space (each IP 4 = bytes * 2 ^32 addresses). > > Is it possible to optimize the disk space for what I'm trying to do using > CouchDB? Perhaps disabling "something", compressing, or changing _rev field > format/size.. thanks!! > > I haver read the manual for CouchDB perfomance, but I didn't get it: > > http://wiki.apache.org/couchdb/Performance > > Regards, > > -- > Santi Saez > http://woop.es > -- ---------------- Best regards Elf mailto:melendeztammy@example.org From melendeztammy@example.org Mon Feb 01 16:53:49 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 2185 invoked from network); 1 Feb 2010 16:53:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 16:53:49 -0000 Received: (qmail 53597 invoked by uid 500); 1 Feb 2010 16:53:47 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 53515 invoked by uid 500); 1 Feb 2010 16:53:47 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 53505 invoked by uid 99); 1 Feb 2010 16:53:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:53:47 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO relayout03-q02.dns-servicios.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:53:36 +0000 Received: from relayout03-redir.dns-servicios.com (relayout03-redir.dns-servicios.com [83.161.120.155]) by relayout03.dns-servicios.com (Postfix) with ESMTP id 3EEF2769936 for <melendeztammy@example.org>; Mon, 1 Feb 2010 17:53:12 +0100 (CET) Received: from vlc-187.dns-servicios.com (unknown [83.161.120.155]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by relayout03-dsp.dns-servicios.com (Postfix) with ESMTPS id 27E1E61AF6D for <melendeztammy@example.org>; Mon, 1 Feb 2010 17:52:51 +0100 (CET) Received: (qmail 28064 invoked from network); 1 Feb 2010 17:52:52 +0100 Received: from ip85.dominioabsoluto.com (HELO macmini.local) (83.161.120.155) by 82.194.84-53.customers.hostalia.com with SMTP; 1 Feb 2010 17:52:52 +0100 Message-ID: <melendeztammy@example.org> Date: Mon, 01 Feb 2010 17:52:52 +0100 From: Santi Saez <melendeztammy@example.org> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; es-ES; rv:83.161.120.155) Gecko/20100111 Thunderbird/3.0.1 MIME-Version: 1.0 To: melendeztammy@example.org Subject: Re: Best way to store 2^32 IPs in CouchDB References: <melendeztammy@example.org> <melendeztammy@example.org> In-Reply-To: <melendeztammy@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-RelayOut-Information: AntiSPAM and AntiVIRUS on relayout03 X-RelayOut-MsgID: 27E1E61AF6D.00000 X-RelayOut-SpamCheck: no es spam, SpamAssassin (no almacenado, puntaje=1.499, requerido 6.5, RDNS_NONE 0.00, WEIRD_PORT 1.50) X-RelayOut-SpamScore: s X-RelayOut-From: melendeztammy@example.org X-RelayOut-Watermark: 1265647971.73146@kIXNARo07Uitwg4FPXd61g X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No El 01/02/10 17:31, Robert Newson escribió: > Try database compaction? I have tried database compaction in another testing server (Debian Lenny box) using CouchDB 0.8.0-2, and after database compaction disk size is the same: # curl http://localhost:5984/test {"db_name":"test","doc_count":15999,"doc_del_count":0,"update_seq":15999,"compact_running":false,"disk_size":60330312} # curl -X POST http://localhost:5984/test/_compact {"ok":true} # curl http://localhost:5984/test {"db_name":"test","doc_count":15999,"doc_del_count":0,"update_seq":15999,"compact_running":false,"disk_size":60330312} Acording to the documentation [1]: "Compaction rewrites the database file, removing outdated document revisions and deleted documents". So, it's normal because in my test I have not delete/upadate any document, only inserts. Thanks! [1] http://wiki.apache.org/couchdb/Compaction -- Santi Saez http://woop.es From melendeztammy@example.org Mon Feb 01 16:56:44 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 2862 invoked from network); 1 Feb 2010 16:56:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 16:56:43 -0000 Received: (qmail 61451 invoked by uid 500); 1 Feb 2010 16:56:42 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 61375 invoked by uid 500); 1 Feb 2010 16:56:42 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 61365 invoked by uid 99); 1 Feb 2010 16:56:42 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:56:42 +0000 X-ASF-Spam-Status: No, hits=1.5 required=10.0 tests=SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO gv-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:56:34 +0000 Received: by gv-out-0910.google.com with SMTP id n8so25024gve.31 for <melendeztammy@example.org>; Mon, 01 Feb 2010 08:56:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type :content-transfer-encoding; bh=gM+SBQcJPPe+zN/5PjrQ+opOUAto1CfObvBvAaJTGqw=; b=fMOfF0+M5xwJ93oTaFXtfAvzlDIygRtpoaylSIj3ZjQuMNxK+iXlZUhs1uRAZzPkDj qb+lzqEt2yhLVdhOPYtfVSo4hOUKWwnkuRBqrdKCUlUUXxzyxprjAs8EGdFpChby0TMn RE3gz0L8fm/wDLRcuXjPaul+pJQtVO53OKYUo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=gQq7+RosT+CZaJ5CjzuPT9TZOBqF4CQx6EIivvSwFE9smXiIqp45HnY5dKXDZU/SGL LCib0stzH5CtQStXqiqmpUC28QwN6/zy/LKaOXqUCkY5MT6sHavSros+vyqSTU5+J33C cP1LSsNw1p4MDJVZTq8hbMXXOeZhwdr0R18zA= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id h14mr2871994mug.25.1265043372618; Mon, 01 Feb 2010 08:56:12 -0800 (PST) In-Reply-To: <melendeztammy@example.org> References: <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> Date: Mon, 1 Feb 2010 16:56:12 +0000 Message-ID: <melendeztammy@example.org> Subject: Re: Best way to store 2^32 IPs in CouchDB From: Robert Newson <melendeztammy@example.org> To: melendeztammy@example.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable compaction should reduce disk usage even without updates or deletes, but that is probably not true for 0.8. odd that you get the exact same byte count after compaction... On Mon, Feb 1, 2010 at 4:52 PM, Santi Saez <melendeztammy@example.org> wrote: > El 01/02/10 17:31, Robert Newson escribi=F3: > >> Try database compaction? > > I have tried database compaction in another testing server (Debian Lenny > box) using CouchDB 0.8.0-2, and after database compaction disk size is th= e > same: > > # curl http://localhost:5984/test > {"db_name":"test","doc_count":15999,"doc_del_count":0,"update_seq":15999,= "compact_running":false,"disk_size":60330312} > > # curl -X POST http://localhost:5984/test/_compact > {"ok":true} > > # curl http://localhost:5984/test > {"db_name":"test","doc_count":15999,"doc_del_count":0,"update_seq":15999,= "compact_running":false,"disk_size":60330312} > > Acording to the documentation [1]: "Compaction rewrites the database file= , > removing outdated document revisions and deleted documents". > > So, it's normal because in my test I have not delete/upadate any document= , > only inserts. > > Thanks! > > [1] http://wiki.apache.org/couchdb/Compaction > > -- > Santi Saez > http://woop.es > From melendeztammy@example.org Mon Feb 01 16:57:07 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 2949 invoked from network); 1 Feb 2010 16:57:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 16:57:07 -0000 Received: (qmail 62625 invoked by uid 500); 1 Feb 2010 16:57:05 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 62557 invoked by uid 500); 1 Feb 2010 16:57:05 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 62547 invoked by uid 99); 1 Feb 2010 16:57:05 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:57:05 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-yw0-f204.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:56:57 +0000 Received: by ywh42 with SMTP id 42so4453761ywh.28 for <melendeztammy@example.org>; Mon, 01 Feb 2010 08:56:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type; bh=dSxDeHpIoiotO7M5Kn9k0yvEAN61Bzp41ehFE1eblxU=; b=SNMybExY9diMDQDWF4c4aFzqrBZyvfTjBwOtufqheuXkHiHXfdnYq2Z1FUd3CyzGrj 8LVmkzNXP70ill31EbNhtZ96QgVv3I5S6Rwd96SIoSNiO16DYU09B59vYEpP27VnF+XW DLlL48P6vaLX1pjJwIOmTa7Vh88nUCz3t/kGc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=V0JmisMDyDTDcRgoASNoT02Lp44YzCKDO76iWxpS4urpiahkOWthdgjdVtSIShXKxa QUWzDBBVdgP9HQ/76JGxvtFQxsJG4QjjNVymD/xQ94YSOgF8ezw2O2e1DYAF0et3b8Xw jL6cYMN6EQCxM/9/cvJyaBh+zK8PVlXkrlNX0= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id a5mr5586389ano.190.1265043396606; Mon, 01 Feb 2010 08:56:36 -0800 (PST) In-Reply-To: <melendeztammy@example.org> References: <melendeztammy@example.org> From: Paul Davis <melendeztammy@example.org> Date: Mon, 1 Feb 2010 11:56:16 -0500 Message-ID: <melendeztammy@example.org> Subject: Re: Best way to store 2^32 IPs in CouchDB To: melendeztammy@example.org Content-Type: text/plain; charset=ISO-8859-1 Well, 2^32 of anything is 4GiB per byte stored. So, minimum of four bytes and melendeztammy@example.org. Even with just 1KiB overhead melendeztammy@example.org. I'm left wondering why you would want to store a list of numbers in the first place. HTH, Paul Davis On Mon, Feb 1, 2010 at 11:27 AM, Santi Saez <melendeztammy@example.org> wrote: > > Hi, > > I'm doing some initial tests with CouchDB, trying to store 2^32 IP addresses > (approximately 4.3 billions of documents). > > Documents have only required fields: _id and _rev, but I've noticed that the > minimum space occupied by each document is approximately 3.7KB, so I need > +14TB disk space only for the basic scheme without any extra field (using IP > as unique identifier in integer format). > > Note that playing with a simple Python script and a binary data file, this > data can be stored in 16GB space (each IP 4 = bytes * 2 ^32 addresses). > > Is it possible to optimize the disk space for what I'm trying to do using > CouchDB? Perhaps disabling "something", compressing, or changing _rev field > format/size.. thanks!! > > I haver read the manual for CouchDB perfomance, but I didn't get it: > > http://wiki.apache.org/couchdb/Performance > > Regards, > > -- > Santi Saez > http://woop.es > From melendeztammy@example.org Mon Feb 01 16:59:21 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 3527 invoked from network); 1 Feb 2010 16:59:21 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 16:59:21 -0000 Received: (qmail 67874 invoked by uid 500); 1 Feb 2010 16:59:20 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 67815 invoked by uid 500); 1 Feb 2010 16:59:19 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 67805 invoked by uid 99); 1 Feb 2010 16:59:19 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:59:19 +0000 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO relayout02-q02.dns-servicios.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 16:59:09 +0000 Received: from relayout02-redir.dns-servicios.com (relayout02-redir.dns-servicios.com [83.161.120.155]) by relayout02.dns-servicios.com (Postfix) with ESMTP id 77DDEA709C2 for <melendeztammy@example.org>; Mon, 1 Feb 2010 17:58:47 +0100 (CET) Received: from vlc-187.dns-servicios.com (unknown [83.161.120.155]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by relayout02-dsp.dns-servicios.com (Postfix) with ESMTPS id E8ACB8601FB for <melendeztammy@example.org>; Mon, 1 Feb 2010 17:58:37 +0100 (CET) Received: (qmail 5122 invoked from network); 1 Feb 2010 17:58:37 +0100 Received: from ip85.dominioabsoluto.com (HELO macmini.local) (83.161.120.155) by 82.194.84-53.customers.hostalia.com with SMTP; 1 Feb 2010 17:58:37 +0100 Message-ID: <melendeztammy@example.org> Date: Mon, 01 Feb 2010 17:58:37 +0100 From: Santi Saez <melendeztammy@example.org> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; es-ES; rv:83.161.120.155) Gecko/20100111 Thunderbird/3.0.1 MIME-Version: 1.0 To: melendeztammy@example.org Subject: Re: Best way to store 2^32 IPs in CouchDB References: <melendeztammy@example.org> <melendeztammy@example.org> In-Reply-To: <melendeztammy@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-RelayOut-Information: AntiSPAM and AntiVIRUS on relayout02 X-RelayOut-MsgID: E8ACB8601FB.00000 X-RelayOut-SpamCheck: no es spam, SpamAssassin (no almacenado, puntaje=0, requerido 6.5, RDNS_NONE 0.00) X-RelayOut-From: melendeztammy@example.org X-RelayOut-Watermark: 1265648319.2674@SbqBfpiF/S37jHeWre4Ggg X-Old-Spam-Status: No El 01/02/10 17:32, Elf escribió: > Did you plan to handle IPv6 in future versions of your program? :) It would be another great test.. but using CouchDB, perhaps I will not have enough disk space ;-P Now seriously: any idea to reduce disk space in this test to store 2^32 documents? thanks! Regards, -- Santi Saez http://woop.es From melendeztammy@example.org Mon Feb 01 17:20:53 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 10188 invoked from network); 1 Feb 2010 17:20:53 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 17:20:53 -0000 Received: (qmail 91346 invoked by uid 500); 1 Feb 2010 17:20:52 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 91281 invoked by uid 500); 1 Feb 2010 17:20:51 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 91271 invoked by uid 99); 1 Feb 2010 17:20:51 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 17:20:51 +0000 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [83.161.120.155] (HELO smtp.atwork.nl) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 17:20:40 +0000 Received: from localhost (localhost [83.161.120.155]) by smtp.atwork.nl (Postfix) with ESMTP id C1C8514F597 for <melendeztammy@example.org>; Mon, 1 Feb 2010 18:20:19 +0100 (CET) Received: from waldo.buyways.nl (atwork-154.r-212.178.116.atwork.nl [83.161.120.155]) by smtp.atwork.nl (Postfix) with ESMTP id D528D14F588 for <melendeztammy@example.org>; Mon, 1 Feb 2010 18:20:07 +0100 (CET) Received: from localhost (localhost [83.161.120.155]) by waldo.buyways.nl (Postfix) with ESMTP id 91CD73A1690 for <melendeztammy@example.org>; Mon, 1 Feb 2010 18:20:07 +0100 (CET) Received: from waldo.buyways.nl ([83.161.120.155]) by localhost (waldo.buyways.nl [83.161.120.155]) (amavisd-new, port 10024) with ESMTP id MWoUF+UnCspA for <melendeztammy@example.org>; Mon, 1 Feb 2010 18:20:07 +0100 (CET) Received: from zealand.localnet (beauregard.buyways.nl [83.161.120.155]) by waldo.buyways.nl (Postfix) with ESMTP id 79CC23A168F for <melendeztammy@example.org>; Mon, 1 Feb 2010 18:20:07 +0100 (CET) From: Markus Jelsma <melendeztammy@example.org> Reply-To: melendeztammy@example.org Organization: Buyways To: melendeztammy@example.org Subject: Re: Best way to store 2^32 IPs in CouchDB Date: Mon, 1 Feb 2010 18:19:30 +0100 User-Agent: KMail/1.12.2 (Linux/2.6.31-17-generic; KDE/4.3.2; i686; ; ) References: <melendeztammy@example.org> <melendeztammy@example.org> In-Reply-To: <melendeztammy@example.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <melendeztammy@example.org> X-Virus-Scanned: by @Work X-Virus-Checked: Checked by ClamAV on apache.org Not really, but you could omit about 300 million IP addresses, these are multicast and private network addresses, that'd save you about 1.2GiB already. >Now seriously: any idea to reduce disk space in this test to store 2^32 >documents? thanks! Markus Jelsma - Technisch Architect - Buyways BV http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350 From melendeztammy@example.org Mon Feb 01 17:22:26 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 10541 invoked from network); 1 Feb 2010 17:22:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 17:22:26 -0000 Received: (qmail 92797 invoked by uid 500); 1 Feb 2010 17:22:24 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 92726 invoked by uid 500); 1 Feb 2010 17:22:24 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 92716 invoked by uid 99); 1 Feb 2010 17:22:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 17:22:24 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-gx0-f213.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 17:22:16 +0000 Received: by gxk5 with SMTP id 5so5181196gxk.15 for <melendeztammy@example.org>; Mon, 01 Feb 2010 09:21:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:in-reply-to :references:from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=SKvjaXOsCPDDKLX1+Yn4LCY354XkemO70yeyl9QXC1I=; b=p7ibR07LmDLLgiM4TvYdk/GdKW15Z/GY/Ur5eXe81XgYE8KXO8Hx5EmcU2jwmpL8Rc C1hYuLIm84T9kOmKw/RlWwG21n6DCkmClWiPjdR2BqLedScr28yMC1kc5hyx28l1FQXe XvLQPr+ywzaOjTbXmQojgTpUKvssrvbx96PCo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:content-type:content-transfer-encoding; b=TGn3SNdzqCcbpd2h2wvTueQh5XZ9kfygcIgfsrtYyy8ipwVj5B0FUuNB2jaWd0x69D mnoOvV6Oj9dEPVGo2Wf6rwaKoS/HwZ+7w4jO9D7gwB/h9IOyFTkytKPLQAEpojmDMIf6 SebnHsxn8rFsasE99Yf+SYmGgYjUx3ptNp8IY= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id x41mr3187143wfi.42.1265044915105; Mon, 01 Feb 2010 09:21:55 -0800 (PST) Reply-To: melendeztammy@example.org In-Reply-To: <melendeztammy@example.org> References: <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> From: Phil Rand <melendeztammy@example.org> Date: Mon, 1 Feb 2010 09:21:35 -0800 Message-ID: <melendeztammy@example.org> Subject: Re: Best way to store 2^32 IPs in CouchDB To: melendeztammy@example.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Of course it depends on what you are trying to do, which you haven't told us. The easiest way to reduce your storage needs is to not store data you don't need to. For example, if I wanted to map IP back to hostname, I wouldn't use CouchDB at all, since we already have the DNS. On Mon, Feb 1, 2010 at 8:58 AM, Santi Saez <melendeztammy@example.org> wrote: > El 01/02/10 17:32, Elf escribi=F3: >> >> Did you plan to handle IPv6 in future versions of your program? :) > > It would be another great test.. but using CouchDB, perhaps I will not ha= ve > enough disk space ;-P > > Now seriously: any idea to reduce disk space in this test to store 2^32 > documents? thanks! > > Regards, > > -- > Santi Saez > http://woop.es > --=20 Phil Rand melendeztammy@example.org melendeztammy@example.org From melendeztammy@example.org Mon Feb 01 18:44:18 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 62949 invoked from network); 1 Feb 2010 18:44:18 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 18:44:18 -0000 Received: (qmail 33238 invoked by uid 500); 1 Feb 2010 18:44:16 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 33165 invoked by uid 500); 1 Feb 2010 18:44:16 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 33155 invoked by uid 99); 1 Feb 2010 18:44:16 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 18:44:16 +0000 X-ASF-Spam-Status: No, hits=-2.5 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO relayout02-q02.dns-servicios.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 18:44:06 +0000 Received: from relayout02-redir.dns-servicios.com (relayout02-redir.dns-servicios.com [83.161.120.155]) by relayout02.dns-servicios.com (Postfix) with ESMTP id 2352BD20093 for <melendeztammy@example.org>; Mon, 1 Feb 2010 19:43:44 +0100 (CET) Received: from vlc-187.dns-servicios.com (unknown [83.161.120.155]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by relayout02-dsp.dns-servicios.com (Postfix) with ESMTPS id 622A68603B8 for <melendeztammy@example.org>; Mon, 1 Feb 2010 19:43:33 +0100 (CET) Received: (qmail 7233 invoked from network); 1 Feb 2010 19:43:33 +0100 Received: from ip85.dominioabsoluto.com (HELO macmini.local) (83.161.120.155) by 82.194.84-53.customers.hostalia.com with SMTP; 1 Feb 2010 19:43:33 +0100 Message-ID: <melendeztammy@example.org> Date: Mon, 01 Feb 2010 19:43:33 +0100 From: Santi Saez <melendeztammy@example.org> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; es-ES; rv:83.161.120.155) Gecko/20100111 Thunderbird/3.0.1 MIME-Version: 1.0 To: melendeztammy@example.org Subject: Re: Best way to store 2^32 IPs in CouchDB References: <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> In-Reply-To: <melendeztammy@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-RelayOut-Information: AntiSPAM and AntiVIRUS on relayout02 X-RelayOut-MsgID: 622A68603B8.00000 X-RelayOut-SpamCheck: no es spam, SpamAssassin (no almacenado, puntaje=1.499, requerido 6.5, RDNS_NONE 0.00, WEIRD_PORT 1.50) X-RelayOut-SpamScore: s X-RelayOut-From: melendeztammy@example.org X-RelayOut-Watermark: 1265654613.56269@tOss9rI9eURueonR0iKjhg X-Old-Spam-Status: No El 01/02/10 17:56, Robert Newson escribió: > compaction should reduce disk usage even without updates or deletes, > but that is probably not true for 0.8. odd that you get the exact same > byte count after compaction... In another testing server with CentOS-5 and "couchdb-0.10.0-1.el5", we have: # curl http://localhost:5984/ipv4 {"db_name":"ipv4","doc_count":11836124,"doc_del_count":7,"update_seq":11836141,"purge_seq":0,"compact_running":true,"disk_size":97730438359,"instance_start_time":"1264780216812476","disk_format_version":4} Still is running a compact "Database Compaction" task: # ls -lh /var/lib/couchdb/ipv4.couch* 92G Feb 1 19:59 /var/lib/couchdb/ipv4.couch 34M Feb 1 19:58 /var/lib/couchdb/ipv4.couch.compact So, I have to wait to finish compacting to see if I saved some disk space, thanks!! Regards, -- Santi Saez http://woop.es From melendeztammy@example.org Mon Feb 01 18:44:42 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 63036 invoked from network); 1 Feb 2010 18:44:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 18:44:42 -0000 Received: (qmail 34413 invoked by uid 500); 1 Feb 2010 18:44:41 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 34330 invoked by uid 500); 1 Feb 2010 18:44:41 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 34320 invoked by uid 99); 1 Feb 2010 18:44:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 18:44:41 +0000 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO relayout02-q02.dns-servicios.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 18:44:30 +0000 Received: from relayout02-redir.dns-servicios.com (relayout02-redir.dns-servicios.com [83.161.120.155]) by relayout02.dns-servicios.com (Postfix) with ESMTP id 1FC70D30171 for <melendeztammy@example.org>; Mon, 1 Feb 2010 19:44:09 +0100 (CET) Received: from vlc-187.dns-servicios.com (unknown [83.161.120.155]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by relayout02-dsp.dns-servicios.com (Postfix) with ESMTPS id 02BEF8603E4 for <melendeztammy@example.org>; Mon, 1 Feb 2010 19:44:01 +0100 (CET) Received: (qmail 7368 invoked from network); 1 Feb 2010 19:44:01 +0100 Received: from ip85.dominioabsoluto.com (HELO macmini.local) (83.161.120.155) by 82.194.84-53.customers.hostalia.com with SMTP; 1 Feb 2010 19:44:01 +0100 Message-ID: <melendeztammy@example.org> Date: Mon, 01 Feb 2010 19:44:00 +0100 From: Santi Saez <melendeztammy@example.org> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; es-ES; rv:83.161.120.155) Gecko/20100111 Thunderbird/3.0.1 MIME-Version: 1.0 To: melendeztammy@example.org Subject: Re: Best way to store 2^32 IPs in CouchDB References: <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> In-Reply-To: <melendeztammy@example.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-RelayOut-Information: AntiSPAM and AntiVIRUS on relayout02 X-RelayOut-MsgID: 02BEF8603E4.00000 X-RelayOut-SpamCheck: no es spam, SpamAssassin (no almacenado, puntaje=0, requerido 6.5, RDNS_NONE 0.00) X-RelayOut-From: melendeztammy@example.org X-RelayOut-Watermark: 1265654641.09341@rPIlO6390W78Ank3rqULeQ X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No El 01/02/10 18:19, Markus Jelsma escribió: > Not really, but you could omit about 300 million IP addresses, these are > multicast and private network addresses, that'd save you about 1.2GiB already. Thanks for the tip ;-) Regards, -- Santi Saez http://woop.es From melendeztammy@example.org Mon Feb 01 18:50:42 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 64485 invoked from network); 1 Feb 2010 18:50:42 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 18:50:42 -0000 Received: (qmail 43047 invoked by uid 500); 1 Feb 2010 18:50:40 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 42973 invoked by uid 500); 1 Feb 2010 18:50:40 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 42963 invoked by uid 99); 1 Feb 2010 18:50:40 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 18:50:40 +0000 X-ASF-Spam-Status: No, hits=-4.0 required=10.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO relayout01-q01.dns-servicios.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 18:50:29 +0000 Received: from relayout01-redir.dns-servicios.com (relayout01-redir.dns-servicios.com [83.161.120.155]) by relayout01.dns-servicios.com (Postfix) with ESMTP id 1576F7407D for <melendeztammy@example.org>; Mon, 1 Feb 2010 19:50:08 +0100 (CET) Received: from vlc-187.dns-servicios.com (unknown [83.161.120.155]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by relayout01-dsp.dns-servicios.com (Postfix) with ESMTPS id A57EC9AC370 for <melendeztammy@example.org>; Mon, 1 Feb 2010 19:50:00 +0100 (CET) Received: (qmail 13467 invoked from network); 1 Feb 2010 19:50:01 +0100 Received: from ip85.dominioabsoluto.com (HELO macmini.local) (83.161.120.155) by 82.194.84-53.customers.hostalia.com with SMTP; 1 Feb 2010 19:50:01 +0100 Message-ID: <melendeztammy@example.org> Date: Mon, 01 Feb 2010 19:50:00 +0100 From: Santi Saez <melendeztammy@example.org> User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; es-ES; rv:83.161.120.155) Gecko/20100111 Thunderbird/3.0.1 MIME-Version: 1.0 To: melendeztammy@example.org Subject: Re: Best way to store 2^32 IPs in CouchDB References: <melendeztammy@example.org> <melendeztammy@example.org> In-Reply-To: <melendeztammy@example.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-RelayOut-Information: AntiSPAM and AntiVIRUS on relayout01 X-RelayOut-MsgID: A57EC9AC370.00000 X-RelayOut-SpamCheck: no es spam, SpamAssassin (no almacenado, puntaje=0, requerido 6.5, RDNS_NONE 0.00) X-RelayOut-From: melendeztammy@example.org X-RelayOut-Watermark: 1265655001.18283@kQc1P1HhFjoks2NSjN+iUA X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No El 01/02/10 17:56, Paul Davis escribió: Dear Paul, > Well, 2^32 of anything is 4GiB per byte stored. So, minimum of four > bytes and melendeztammy@example.org. Even with just 1KiB overhead you're at > 4TiB. > > I'm left wondering why you would want to store a list of numbers in > the first place. Imagine a service like Netcraft. I know that there aren't 2^32 active servers, but I wanted to test it with 4.3 billion documents and stress/benchamark CouchDB with other DBs. Regards, -- Santi Saez http://woop.es From melendeztammy@example.org Mon Feb 01 19:08:48 2010 Return-Path: <melendeztammy@example.org> Delivered-To: melendeztammy@example.org Received: (qmail 69094 invoked from network); 1 Feb 2010 19:08:48 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 1 Feb 2010 19:08:48 -0000 Received: (qmail 74816 invoked by uid 500); 1 Feb 2010 19:08:47 -0000 Delivered-To: melendeztammy@example.org Received: (qmail 74745 invoked by uid 500); 1 Feb 2010 19:08:47 -0000 Mailing-List: contact melendeztammy@example.org; run by ezmlm Precedence: bulk List-Help: <mailto:melendeztammy@example.org> List-Unsubscribe: <mailto:melendeztammy@example.org> List-Post: <mailto:melendeztammy@example.org> List-Id: <user.couchdb.apache.org> Reply-To: melendeztammy@example.org Delivered-To: mailing list melendeztammy@example.org Received: (qmail 74734 invoked by uid 99); 1 Feb 2010 19:08:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 19:08:47 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of melendeztammy@example.org designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO mail-yw0-f204.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 01 Feb 2010 19:08:37 +0000 Received: by ywh42 with SMTP id 42so4590462ywh.28 for <melendeztammy@example.org>; Mon, 01 Feb 2010 11:08:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :from:date:message-id:subject:to:content-type :content-transfer-encoding; bh=5W/2WxNwLwRtWQkTd17ilN/uMKP89x/3heJlfck2JGY=; b=VCGjSol3UrirIFzmjANlcnW5OL/TKsi7BO4uE9pjIKUnEND3OFL5L7rrTqnRKxqVZc 3Qa3U3NhuhuESDJOVMCF17wt6EOEux6i4ohWHG4X+RDgNQlToec0FQEY1THGWICRf4C0 +J2/YvHpUvDkGbObW4LHwVgAmsPRsSjp4tBSE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; b=MEfWvZ3/iXBd64q+8XIMOT+Nq3w5JjKHz/SIW/lGOHNpvlaVsO2nRjY79uDOw7jg6F mcgVN8yegRYOtYvuUF+07x+7bIxQBzGsvg0ck0gronIihymF6lnOocGdYHQd6mlXA1NH Woh3/0Twc/OdRASqZsknuhND8JVG5hlxgH1pU= MIME-Version: 1.0 Received: by 83.161.120.155 with SMTP id v25mr5945431ann.152.1265051296657; Mon, 01 Feb 2010 11:08:16 -0800 (PST) In-Reply-To: <melendeztammy@example.org> References: <melendeztammy@example.org> <melendeztammy@example.org> <melendeztammy@example.org> From: Paul Davis <melendeztammy@example.org> Date: Mon, 1 Feb 2010 14:07:56 -0500 Message-ID: <melendeztammy@example.org> Subject: Re: Best way to store 2^32 IPs in CouchDB To: melendeztammy@example.org Content-Type:
{'source': 'Apache Software Foundation Public Mail Archives'}