From couchdb-dev-returndanielmartin@example.com Tue Dec 02 08:55:15 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 908 invoked from network); 2 Dec 2008 08:55:15 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 08:55:15 -0000 Received: (qmail 82543 invoked by uid 500); 2 Dec 2008 08:55:26 -0000 Delivered-To: danielmartin@example.com Received: (qmail 82509 invoked by uid 500); 2 Dec 2008 08:55:25 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 82498 invoked by uid 99); 2 Dec 2008 08:55:25 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 00:55:25 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 08:53:56 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 4CB56234C298 for <danielmartin@example.com>; Tue, 2 Dec 2008 00:54:44 -0800 (PST) Message-ID: <1852591059.1228208084312.JavaMail.jira@brutus> Date: Tue, 2 Dec 2008 00:54:44 -0800 (PST) From: "Justin Walgran (JIRA)" <danielmartin@example.com> To: danielmartin@example.com Subject: [jira] Created: (COUCHDB-167) Files can not be attached to documents with forward slashes in their names to simulate a folder hierarchy MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Files can not be attached to documents with forward slashes in their names to simulate a folder hierarchy --------------------------------------------------------------------------------------------------------- Key: COUCHDB-167 URL: https://issues.apache.org/jira/browse/COUCHDB-167 Project: CouchDB Issue Type: Bug Components: Database Core Affects Versions: 0.9 Environment: Mac OS X 10.5.5, MacBook Pro, CouchDB 0.9.0a721757-incubating Reporter: Justin Walgran Steps to reproduce: 1) Download sample files from http://walgran.com/justin/CouchDB-nested-attachment-sample.zip 2) Unzip CouchDB-nested-attachment-sample.zip 3) Create a new CouchDB database 4) Create a new document called "_design/nested" 5) Add "index.html," "pickle.gif" and "images/pickle.gif" as attachments to the document 6) Browse the index.html attachment Expected Behavior: Browsing index.html shows two copies of pickle.gif on the page Error: The second image fails to load. Notes: Damien suggested using directory separators within file names to simulate folder hierarchies in a November 28th response to the danielmartin@example.com mailing list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From couchdb-dev-returndanielmartin@example.com Tue Dec 02 12:39:45 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 9095 invoked from network); 2 Dec 2008 12:39:45 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 12:39:45 -0000 Received: (qmail 54021 invoked by uid 500); 2 Dec 2008 12:39:56 -0000 Delivered-To: danielmartin@example.com Received: (qmail 53981 invoked by uid 500); 2 Dec 2008 12:39:55 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 53970 invoked by uid 99); 2 Dec 2008 12:39:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 04:39:55 -0800 X-ASF-Spam-Status: No, hits=-1998.5 required=10.0 tests=ALL_TRUSTED,WEIRD_PORT X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 12:38:26 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 6B70F234C29C for <danielmartin@example.com>; Tue, 2 Dec 2008 04:38:44 -0800 (PST) Message-ID: <1048985205.1228221524438.JavaMail.jira@brutus> Date: Tue, 2 Dec 2008 04:38:44 -0800 (PST) From: "Tom Robinson (JIRA)" <danielmartin@example.com> To: danielmartin@example.com Subject: [jira] Commented: (COUCHDB-167) Files can not be attached to documents with forward slashes in their names to simulate a folder hierarchy In-Reply-To: <1852591059.1228208084312.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/COUCHDB-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652350#action_12652350 ] Tom Robinson commented on COUCHDB-167: -------------------------------------- When I used Futon to add the attachment it used a colon instead of a forward slash, so I could access the image at http://localhost:5984/test/_design%2Fnested/images%3Apickle.gif (%3A is a colon) You can use "%2F" in place of a forward slash, in which case it shows up as a forward slash in Futon, but actually requires "%2F" in the URL. If you *actually* try to save an attachment with a forward slash in it the request actually crashes. For example, this: curl -v -H "Content-Type: image/gif" -T ~/pickle.gif http://localhost:5984/test/_design%2Fnested/images/pickle.gif?rev=770-244-2241 causes this error: [error] [<0.290.0>] {error_report,<0.22.0>, {<0.290.0>,crash_report, [[{pid,<0.290.0>}, {registered_name,[]}, {error_info, {error,function_clause, [{couch_httpd_db,db_req, [{httpd, {mochiweb_request,#Port<0.1604>,'PUT', "/test/_design%2Fnested/images/pickle.gif?rev=770-244-2241", {1,1}, {6, {"user-agent", {'User-Agent', "curl/7.19.2 (i386-apple-darwin9.5.0) libcurl/7.19.2 zlib/1.2.3"}, {"host", {'Host',"localhost:5984"}, {"accept", {'Accept',"*/*"}, nil, {"content-type", {'Content-Type',"image/gif"}, {"content-length", {'Content-Length',"806152"}, nil,nil}, {"expect", {"Expect","100-continue"}, nil,nil}}}, nil}, nil}}}, 'PUT', [<<"test">>,<<"_design/nested">>,<<"images">>, <<"pickle.gif">>], {dict,2,16,16,8,80,48, {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, {{[],[],[],[],[],[],[],[],[],[],[], [[<<"_temp_view">>| #Fun<couch_httpd.7.18882544>]], [], [[<<"_view">>|#Fun<couch_httpd.7.18882544>]], [],[]}}}, {user_ctx,nil,[<<"_admin">>]}}, {db,<0.122.0>,<0.123.0>,nil,<0.121.0>, {db_header,0,11, {4833,9263}, {1891797,{4,0}}, {1891616,4}, nil,0,nil,nil}, {stream,<0.124.0>,<0.121.0>}, {btree,<0.121.0>, {1891797,{4,0}}, #Fun<couch_db_updater.770-244-2241>, #Fun<couch_db_updater.7.45786599>, #Fun<couch_db_updater.5.74731249>, #Fun<couch_db_updater.8.118159480>}, {btree,<0.121.0>, {1891616,4}, #Fun<couch_db_updater.9.48828661>, #Fun<couch_db_updater.10.81431945>, #Fun<couch_btree.5.112258129>, #Fun<couch_db_updater.11.102742899>}, {btree,<0.121.0>,nil,#Fun<couch_btree.0.23070627>, #Fun<couch_btree.1.117278773>, #Fun<couch_btree.2.112258129>,nil}, 11,<<"test">>, "/usr/local/var/lib/couchdb/test.couch",[],[],nil, {user_ctx,nil,[<<"_admin">>]}}]}, {couch_httpd_db,do_db_req,2}, {couch_httpd,handle_request,3}, {mochiweb_http,headers,4}, {proc_lib,init_p_do_apply,3}]}}, {initial_call,{mochiweb_socket_server,acceptor_loop,['Argument__1']}}, {ancestors, [couch_httpd,couch_secondary_services,couch_server_sup,<0.1.0>]}, {messages,[]}, {links,[<0.56.0>,#Port<0.1604>]}, {dictionary,[]}, {trap_exit,false}, {status,running}, {heap_size,987}, {stack_size,23}, {reductions,1446}], []]}} [error] [<0.56.0>] {error_report,<0.22.0>, {<0.56.0>,std_error, {mochiweb_socket_server,235,{child_error,function_clause}}}} =ERROR REPORT==== 2-Dec-888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a404:34:56 === {mochiweb_socket_server,235,{child_error,function_clause}} > Files can not be attached to documents with forward slashes in their names to simulate a folder hierarchy > --------------------------------------------------------------------------------------------------------- > > Key: COUCHDB-167 > URL: https://issues.apache.org/jira/browse/COUCHDB-167 > Project: CouchDB > Issue Type: Bug > Components: Database Core > Affects Versions: 0.9 > Environment: Mac OS X 10.5.5, MacBook Pro, CouchDB 0.9.0a721757-incubating > Reporter: Justin Walgran > > Steps to reproduce: > 1) Download sample files from http://walgran.com/justin/CouchDB-nested-attachment-sample.zip > 2) Unzip CouchDB-nested-attachment-sample.zip > 3) Create a new CouchDB database > 4) Create a new document called "_design/nested" > 5) Add "index.html," "pickle.gif" and "images/pickle.gif" as attachments to the document > 6) Browse the index.html attachment > Expected Behavior: > Browsing index.html shows two copies of pickle.gif on the page > Error: > The second image fails to load. > Notes: > Damien suggested using directory separators within file names to simulate folder hierarchies in a November 28th response to the danielmartin@example.com mailing list. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From couchdb-dev-returndanielmartin@example.com Tue Dec 02 19:30:03 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 29605 invoked from network); 2 Dec 2008 19:30:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 19:30:03 -0000 Received: (qmail 42899 invoked by uid 500); 2 Dec 2008 19:30:15 -0000 Delivered-To: danielmartin@example.com Received: (qmail 42620 invoked by uid 500); 2 Dec 2008 19:30:14 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 42609 invoked by uid 99); 2 Dec 2008 19:30:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 11:30:14 -0800 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO relay02.pair.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 Dec 2008 19:28:45 +0000 Received: (qmail 35842 invoked from network); 2 Dec 2008 19:29:31 -0000 Received: from 83.161.120.155 (HELO ?83.161.120.155?) (83.161.120.155) by relay02.pair.com with SMTP; 2 Dec 2008 19:29:31 -0000 X-pair-Authenticated: 83.161.120.155 Message-Id: <danielmartin@example.com> From: Damien Katz <danielmartin@example.com> To: danielmartin@example.com Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: 0.9.0 Release Date: Tue, 2 Dec 2008 14:29:30 -0500 X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org I think we are just about ready for a 0.9.0 release and will be feature complete enough to be considered Beta. Two things I'd like before we release 0.9.0 is some replication testing of the new security stuff and to finally change "count" to "limit" in view query string arguments. Also, some people have complaints about the programmatic interfaces for map/reduce/rereduce functionality. If anyone wants to change these, or any other major aspects of CouchDB interfaces, now is the time. It only gets harder as we move forward. Feedback please. -Damien From couchdb-dev-returndanielmartin@example.com Tue Dec 02 19:36:13 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 31703 invoked from network); 2 Dec 2008 19:36:13 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 19:36:13 -0000 Received: (qmail 51104 invoked by uid 500); 2 Dec 2008 19:36:25 -0000 Delivered-To: danielmartin@example.com Received: (qmail 50819 invoked by uid 500); 2 Dec 2008 19:36:24 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 50808 invoked by uid 99); 2 Dec 2008 19:36:24 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 11:36:24 -0800 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO relay02.pair.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 Dec 2008 19:34:53 +0000 Received: (qmail 39481 invoked from network); 2 Dec 2008 19:34:39 -0000 Received: from 83.161.120.155 (HELO ?83.161.120.155?) (83.161.120.155) by relay02.pair.com with SMTP; 2 Dec 2008 19:34:39 -0000 X-pair-Authenticated: 83.161.120.155 Message-Id: <danielmartin@example.com> From: Damien Katz <danielmartin@example.com> To: danielmartin@example.com Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: 1.0.0 wishlist/roadmap Date: Tue, 2 Dec 2008 14:34:39 -0500 X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org Here is some stuff I'd like to see in a 1.0.0 release. Everything is open for discussion. - Built-in reduce functions to avoid unnecessary JS overhead - Count, Sum, Avg, Min, Max, Std dev. others? - Restrict database read access - Right now any user can read any database, we need to be able to restrict that at least on a whole database level. - Replication performance enhancements - Adam Kocoloski has some replication patches that greatly improve replication performance. - Revision stemming: It should be possible to limit the number of revisions tracked - By default each document edit produces a revision id that is tracked indefinitely. This guarantees conflicts versus subsequent edits can always be distinguished in ad-hoc replication, however the forever growing list of revisions isn't always desirable. THis can be addressed by limiting the number tracked and purging the oldest revisions. The downside is that if the revision tracking limited is N, then anyone who hasn't replicated a document since its last N edits will see a spurious edit conflict. - Lucene/Full-text indexing integration - We have this working to in side patches, this needs to be integrated to trunk and with the view engine - Incremental document replication - We need at the minimum the ability to incrementally replicate only the attachments that have changed in a document. This will save lots of network IO and CouchDB can be version control system with document diffs added as attachments. This can work for document fields too, but the overhead may not be worth it. - Built-in authentication module(s) - The ability to host a CouchDB database used for HTTP authentication schemes. If storing passwords, they would need to be stored encrypted, decrypted on demand by the authentication process. - View server enhancements (stale/partial index option) - Chris Anderson has a side branch for this we need to finish and put into trunk. - View index compaction - Views indexes expand forever, and need to be compacted in a similar way the storage files are compacted. This work will tie into the View Server enhancements. - Document integrity/deterministic revid - For the sake of end to end document integrity, we need a way to hash a document's contents, and since we already have revision ids, I think the revision ids should be the hashes. The hashed document should be a canonical json representation, and it should have the _id and _rev fields in it. The _rev will be the PREVIOUS revision ID/hash the edit is based on, or blank if a new edit. Then the _rev is replaced with the new hash value. - Fully tail append writes - CouchDB uses zero-overwrite storage, but not fully tail append storage. Document json bodies are stored in internal buffers, written consecutively, one after another until the buffers in completely full, then another buffer is created at the end of the file for more documents. File attachments are written to similar buffers as well. Btree updates are always tail append, each update to a btree, even if its a deletion, causes new writes to the end of the file. Once the document, attachments and indexes are commited (fsync), the header is then written and flushed to disk, and that is always stored right at the beginning of the file (requiring another seek). Document updates to CouchDB require 2 fsyncs with ~3 seeks for full committal and index consistency. This is true if you write 1 or 1000 documents in a single transaction (bulk update), you still need ~ 3 seeks. Using conventional transaction journalling, it's possible to get the committal down to a single seek and fsync, and worry about ensuring file and index consistency asynchronously, often in batch mode with other committed updates. This can perform very well, but has downsides like extra complexity and increased memory usage as data is cached waiting to be flushed to disk, and must do special consistency checks and fix-ups on startup if there is a crash. If CouchDB used tail-append storage for everything, then all document updates can be completely flushed with full file consistency with a single seek and, depending on the file system, a single fsync. All the disk updates, documents, file attachments, indexes and file header, occur as appends to the end of the file. The biggest changes will be in how file attachments and the headers are written and read, and the performance characteristics of view indexing as documents will no longer be packed into contiguous buffers. File attachment will be written in chunks with the last chunk being an index to the other chunks. Headers will be specially signed blocks written to the end of the file. Reading the header on database open will require scanning the file from the end, since the file might have partial updates that didn't complete since the last update. The performance of the views will be impacted as the documents are more likely to be fragmented across the storage file. But they will still be in the order they will be accessed for indexing, so the read seeks are always moving forward. Also, the act of compacting the storage file will result in the documents being tightly packed again. - Streaming document updates with attachment writes - Using mime mulitpart encoding, it should be possible to send all parts of a document in a single http request, with the json and binary attachments sent as different mime parts. Attachments can be streamed to disk as bytes are received, keeping total memory overhead to a minimum. Attachments can also be written to disk in compressed format and served over http by default in that compressed format, using 0% CPU for compression at read time, but will require decompression if the client doesn't support the compression format. - Partitioning/Clustering Support - Clustering for failover and load balancing is priority. Large database support via partitioning may not make 1.0 From couchdb-dev-returndanielmartin@example.com Tue Dec 02 19:39:16 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 32828 invoked from network); 2 Dec 2008 19:39:16 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 19:39:16 -0000 Received: (qmail 57496 invoked by uid 500); 2 Dec 2008 19:39:27 -0000 Delivered-To: danielmartin@example.com Received: (qmail 57217 invoked by uid 500); 2 Dec 2008 19:39:27 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 57203 invoked by uid 99); 2 Dec 2008 19:39:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 11:39:27 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of danielmartin@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ik-out-1112.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 19:37:56 +0000 Received: by ik-out-1112.google.com with SMTP id b35so2679161ika.4 for <danielmartin@example.com>; Tue, 02 Dec 2008 11:38:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=/Zh650O+xS33rFdG+qd0uob9L3Nep1L81Ii51LGiIxU=; b=vfCUxfa5nMBI2fnneyl4SuxwXBr7oPdu/dzqXQTBBuRYZhaKweMWQrSuVAIz4YDINj yaAlR+B/mPVMqCDWm9+o7pNLryVh+bqcex9fg/naKBJfI1vigEYHK4XZMLGYz8+nt0rD AeQ5ZqoTlfpTXPXC+btGSTB8G75XLnQdiSNcQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=aEfZ6C5C715dZuWPuakbhBlqphxfCUdIGaTLkVZTAiwf8Nr5F851RabKDgIsOYZ0Qd bgkKdYF2PEd1aUmJTEjMgRaMSVZcWJTrHqUF5Jib8wYwKHH177WXz7GLBw+iFue0ASAN JJl6tuvXXmvaeCgaateHurrkJj+K/EnodvMGw= Received: by 83.161.120.155 with SMTP id o7mr9154114ebb.16.1228246713149; Tue, 02 Dec 2008 11:38:33 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 2 Dec 2008 11:38:33 -0800 (PST) Message-ID: <danielmartin@example.com> Date: Tue, 2 Dec 2008 11:38:33 -0800 From: "Chris Anderson" <danielmartin@example.com> Sender: danielmartin@example.com To: danielmartin@example.com Subject: Re: 0.9.0 Release In-Reply-To: <danielmartin@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <danielmartin@example.com> X-Google-Sender-Auth: 10847897021f697e X-Virus-Checked: Checked by ClamAV on apache.org On Tue, Dec 2, 2008 at 11:29 AM, Damien Katz <danielmartin@example.com> wrote: > I think we are just about ready for a 0.9.0 release and will be feature > complete enough to be considered Beta. Woo hoo! > > Two things I'd like before we release 0.9.0 is some replication testing of > the new security stuff and to finally change "count" to "limit" in view > query string arguments. count -> limit sounds simple enough. Perhaps if you wrote up in English what the replication security requirements are, someone else could add the tests. > > Also, some people have complaints about the programmatic interfaces for > map/reduce/rereduce functionality. If anyone wants to change these, or any > other major aspects of CouchDB interfaces, now is the time. It only gets > harder as we move forward. I'll say that I'm quite happy with the reduce/rereduce API as it currently stands. However, I'd like to make sure that there is room for some stock Erlang reduce functions to be available without a Javascript function definition. For starters reduce=row_count and reduce=sum would both be handy for a lot of applications, and implementing them as Erlang would mean that group=true queries would be fast. I think these features can be added in the future without breaking compatibility with existing libraries, so they don't need to happen now, but I wanted to bring it up in case there's something I'm not seeing. -- Chris Anderson http://jchris.mfdz.com From couchdb-dev-returndanielmartin@example.com Tue Dec 02 19:42:43 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 34038 invoked from network); 2 Dec 2008 19:42:43 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 19:42:43 -0000 Received: (qmail 65678 invoked by uid 500); 2 Dec 2008 19:42:54 -0000 Delivered-To: danielmartin@example.com Received: (qmail 65632 invoked by uid 500); 2 Dec 2008 19:42:54 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 65621 invoked by uid 99); 2 Dec 2008 19:42:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 11:42:54 -0800 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO mail-gx0-f12.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 19:41:26 +0000 Received: by gxk5 with SMTP id 5so698120gxk.12 for <danielmartin@example.com>; Tue, 02 Dec 2008 11:42:12 -0800 (PST) Received: by 83.161.120.155 with SMTP id e16mr1417224aga.72.1228246693934; Tue, 02 Dec 2008 11:38:13 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 2 Dec 2008 11:38:13 -0800 (PST) Message-ID: <danielmartin@example.com> Date: Tue, 2 Dec 2008 14:38:13 -0500 From: "Dean Landolt" <danielmartin@example.com> To: danielmartin@example.com Subject: Re: 0.9.0 Release In-Reply-To: <danielmartin@example.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_94768_6318817.1228246693948" References: <danielmartin@example.com> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_94768_6318817.1228246693948 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline >...and to finally change "count" to "limit" in view query string arguments. I'm a big fan of "limit" -- I know couch ain't sql, but no need to eschew convention without reason. Also, what about changing "skip" to "offset" which is more conventional? And I know this has been asked before, but is it even remotely feasible to make skip/offset/whatever efficient? It seems to me it would make pagination much more logical, rather than having to drop down to startkey/startkey_docid semantics. ------=_Part_94768_6318817.1228246693948-- From couchdb-dev-returndanielmartin@example.com Tue Dec 02 19:56:14 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 37970 invoked from network); 2 Dec 2008 19:56:14 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 19:56:14 -0000 Received: (qmail 92856 invoked by uid 500); 2 Dec 2008 19:56:25 -0000 Delivered-To: danielmartin@example.com Received: (qmail 92827 invoked by uid 500); 2 Dec 2008 19:56:25 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 92816 invoked by uid 99); 2 Dec 2008 19:56:25 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 11:56:25 -0800 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO relay02.pair.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 Dec 2008 19:54:55 +0000 Received: (qmail 50524 invoked from network); 2 Dec 2008 19:54:40 -0000 Received: from 83.161.120.155 (HELO ?83.161.120.155?) (83.161.120.155) by relay02.pair.com with SMTP; 2 Dec 2008 19:54:40 -0000 X-pair-Authenticated: 83.161.120.155 Message-Id: <danielmartin@example.com> From: Damien Katz <danielmartin@example.com> To: danielmartin@example.com In-Reply-To: <danielmartin@example.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Re: 0.9.0 Release Date: Tue, 2 Dec 2008 14:54:40 -0500 References: <danielmartin@example.com> <danielmartin@example.com> X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org On Dec 2, 2008, at 2:38 PM, Dean Landolt wrote: >> ...and to finally change "count" to "limit" in view query string >> arguments. > I'm a big fan of "limit" -- I know couch ain't sql, but no need to > eschew > convention without reason. Also, what about changing "skip" to > "offset" > which is more conventional? And I know this has been asked before, > but is it > even remotely feasible to make skip/offset/whatever efficient? It > seems to > me it would make pagination much more logical, rather than having to > drop > down to startkey/startkey_docid semantics. Sorry, but making skip efficient is hard to do in a partitioned setup, and considering the stateless HTTP api, the startkey semantics actually work quite well. Users don't want view rows 50-60, they want the next or the previous page of results. When using absolute offsets, if the view changes underneath the user between paging requests, they may not even realize the rows have shifted and they are now in a completely different place in the view. -Damien From couchdb-dev-returndanielmartin@example.com Tue Dec 02 20:00:01 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 38892 invoked from network); 2 Dec 2008 20:00:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 20:00:01 -0000 Received: (qmail 97892 invoked by uid 500); 2 Dec 2008 20:00:12 -0000 Delivered-To: danielmartin@example.com Received: (qmail 97847 invoked by uid 500); 2 Dec 2008 20:00:12 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 97836 invoked by uid 99); 2 Dec 2008 20:00:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 12:00:12 -0800 X-ASF-Spam-Status: No, hits=3.4 required=10.0 tests=HTML_MESSAGE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO rn-out-0910.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 19:58:44 +0000 Received: by rn-out-0910.google.com with SMTP id k50so2851066rnd.3 for <danielmartin@example.com>; Tue, 02 Dec 2008 11:59:20 -0800 (PST) Received: by 83.161.120.155 with SMTP id u16mr2167300agb.0.1228247960717; Tue, 02 Dec 2008 11:59:20 -0800 (PST) Received: by 83.161.120.155 with HTTP; Tue, 2 Dec 2008 11:59:20 -0800 (PST) Message-ID: <danielmartin@example.com> Date: Tue, 2 Dec 2008 14:59:20 -0500 From: "Dean Landolt" <danielmartin@example.com> To: danielmartin@example.com Subject: Re: 0.9.0 Release In-Reply-To: <danielmartin@example.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_95142_19620291.1228247960710" References: <danielmartin@example.com> <danielmartin@example.com> <danielmartin@example.com> X-Virus-Checked: Checked by ClamAV on apache.org ------=_Part_95142_19620291.1228247960710 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline On Tue, Dec 2, 2008 at 2:54 PM, Damien Katz <danielmartin@example.com> wrote: > > On Dec 2, 2008, at 2:38 PM, Dean Landolt wrote: > > ...and to finally change "count" to "limit" in view query string >>> arguments. >>> >> I'm a big fan of "limit" -- I know couch ain't sql, but no need to eschew >> convention without reason. Also, what about changing "skip" to "offset" >> which is more conventional? And I know this has been asked before, but is >> it >> even remotely feasible to make skip/offset/whatever efficient? It seems to >> me it would make pagination much more logical, rather than having to drop >> down to startkey/startkey_docid semantics. >> > > Sorry, but making skip efficient is hard to do in a partitioned setup, and > considering the stateless HTTP api, the startkey semantics actually work > quite well. Users don't want view rows 50-60, they want the next or the > previous page of results. When using absolute offsets, if the view changes > underneath the user between paging requests, they may not even realize the > rows have shifted and they are now in a completely different place in the > view. > > -Damien I didn't think of it like that (I rarely do). Makes sense -- thanks for the heads up. ------=_Part_95142_19620291.1228247960710-- From couchdb-dev-returndanielmartin@example.com Tue Dec 02 21:44:17 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 87288 invoked from network); 2 Dec 2008 21:44:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 21:44:17 -0000 Received: (qmail 54247 invoked by uid 500); 2 Dec 2008 21:44:28 -0000 Delivered-To: danielmartin@example.com Received: (qmail 54203 invoked by uid 500); 2 Dec 2008 21:44:28 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 54192 invoked by uid 99); 2 Dec 2008 21:44:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 13:44:28 -0800 X-ASF-Spam-Status: No, hits=0.2 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO relay01.pair.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with SMTP; Tue, 02 Dec 2008 21:42:59 +0000 Received: (qmail 4616 invoked from network); 2 Dec 2008 21:42:45 -0000 Received: from 83.161.120.155 (HELO ?83.161.120.155?) (83.161.120.155) by relay01.pair.com with SMTP; 2 Dec 2008 21:42:45 -0000 X-pair-Authenticated: 83.161.120.155 Message-Id: <danielmartin@example.com> From: Damien Katz <danielmartin@example.com> To: danielmartin@example.com In-Reply-To: <danielmartin@example.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Re: 0.9.0 Release Date: Tue, 2 Dec 2008 16:42:45 -0500 References: <danielmartin@example.com> <danielmartin@example.com> X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org On Dec 2, 2008, at 2:38 PM, Chris Anderson wrote: > On Tue, Dec 2, 2008 at 11:29 AM, Damien Katz <danielmartin@example.com> > wrote: >> I think we are just about ready for a 0.9.0 release and will be >> feature >> complete enough to be considered Beta. > > Woo hoo! > >> >> Two things I'd like before we release 0.9.0 is some replication >> testing of >> the new security stuff and to finally change "count" to "limit" in >> view >> query string arguments. > > count -> limit sounds simple enough. > > Perhaps if you wrote up in English what the replication security > requirements are, someone else could add the tests. So there are two things to test, the first is the ability to send HTTP headers to the replicator, so that you can pass through your http credentials when replicating with a remote host. There is code in the replicator already to do this. This should use the same phoney test authentication scheme already used by the security and validation tests. The second is to test that the validation and authorization stuff work properly with replicated edits, the same as it does with live edits. This should already work, but its not tested. Also there is a good chance validation/authorization failures might not be handled gracefully by the replicator. It should eat failures, keeping statistics about the failures and maybe a record of the last failure, or last N failures. > > >> >> Also, some people have complaints about the programmatic interfaces >> for >> map/reduce/rereduce functionality. If anyone wants to change these, >> or any >> other major aspects of CouchDB interfaces, now is the time. It only >> gets >> harder as we move forward. > > I'll say that I'm quite happy with the reduce/rereduce API as it > currently stands. > > However, I'd like to make sure that there is room for some stock > Erlang reduce functions to be available without a Javascript function > definition. For starters reduce=row_count and reduce=sum would both be > handy for a lot of applications, and implementing them as Erlang would > mean that group=true queries would be fast. I think these features can > be added in the future without breaking compatibility with existing > libraries, so they don't need to happen now, but I wanted to bring it > up in case there's something I'm not seeing. Yes, I added this to the 1.0.0 roadmap email. If you want to have a go at it for 0.9.0, feel free. I think it's worth it. -Damien From couchdb-dev-returndanielmartin@example.com Tue Dec 02 21:59:37 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 97673 invoked from network); 2 Dec 2008 21:59:37 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 21:59:37 -0000 Received: (qmail 78091 invoked by uid 500); 2 Dec 2008 21:59:48 -0000 Delivered-To: danielmartin@example.com Received: (qmail 78045 invoked by uid 500); 2 Dec 2008 21:59:48 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 78034 invoked by uid 99); 2 Dec 2008 21:59:48 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 13:59:48 -0800 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [83.161.120.155] (HELO brutus.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 21:58:26 +0000 Received: from brutus (localhost [83.161.120.155]) by brutus.apache.org (Postfix) with ESMTP id 4EBD9234C29B for <danielmartin@example.com>; Tue, 2 Dec 2008 13:58:44 -0800 (PST) Message-ID: <939324254.1228255124321.JavaMail.jira@brutus> Date: Tue, 2 Dec 2008 13:58:44 -0800 (PST) From: "Adam Kocoloski (JIRA)" <danielmartin@example.com> To: danielmartin@example.com Subject: [jira] Created: (COUCHDB-168) allow user-configurable directory for view indexes MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org allow user-configurable directory for view indexes -------------------------------------------------- Key: COUCHDB-168 URL: https://issues.apache.org/jira/browse/COUCHDB-168 Project: CouchDB Issue Type: Wish Components: Database Core Reporter: Adam Kocoloski Hi, I'd like to be able to store the DBs and the view indexes in separate locations. This patch creates a new config variable called "view_index_dir" that allows for that. Best, Adam diff --git a/trunk/etc/couchdb/default.ini.tpl.in b/trunk/etc/couchdb/default.ini.tpl.in index f4eb49f..c9143cf 100644 --- a/trunk/etc/couchdb/default.ini.tpl.in +++ b/trunk/etc/couchdb/default.ini.tpl.in @@ -4,6 +4,7 @@ [couchdb] database_dir = %localstatelibdir% +view_index_dir = %localstatelibdir% util_driver_dir = %couchprivlibdir% max_document_size = 4294967296 ; 4 GB view_timeout = 5000 ; 5 seconds diff --git a/trunk/src/couchdb/couch_view.erl b/trunk/src/couchdb/couch_view.erl index 4ebbb13..086e39f 100644 --- a/trunk/src/couchdb/couch_view.erl +++ b/trunk/src/couchdb/couch_view.erl @@ -238,10 +238,10 @@ fold(#view{btree=Btree}, StartKey, Dir, Fun, Acc) -> init([]) -> % read configuration settings and register for configuration changes - RootDir = couch_config:get("couchdb", "database_dir"), + RootDir = couch_config:get("couchdb", "view_index_dir"), Self = self(), ok = couch_config:register( - fun("couchdb", "database_dir")-> + fun("couchdb", "view_index_dir")-> exit(Self, config_change) end), -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. From couchdb-dev-returndanielmartin@example.com Tue Dec 02 22:51:22 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 21531 invoked from network); 2 Dec 2008 22:51:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 2 Dec 2008 22:51:22 -0000 Received: (qmail 61855 invoked by uid 500); 2 Dec 2008 22:51:33 -0000 Delivered-To: danielmartin@example.com Received: (qmail 61831 invoked by uid 500); 2 Dec 2008 22:51:33 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 61820 invoked by uid 99); 2 Dec 2008 22:51:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 14:51:32 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO tumbolia.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 22:50:03 +0000 Received: from nslater by tumbolia.org with local (Exim 4.69) (envelope-from <danielmartin@example.com>) id 1L7e1t-00085f-5l for danielmartin@example.com; Tue, 02 Dec 2008 22:47:41 +0000 Date: Tue, 2 Dec 2008 22:47:41 +0000 From: Noah Slater <danielmartin@example.com> To: danielmartin@example.com Subject: Re: 0.9.0 Release Message-ID: <danielmartin@example.com> Mail-Followup-To: danielmartin@example.com References: <danielmartin@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <danielmartin@example.com> X-Noah: Awesome User-Agent: Mutt/1.5.18 (2008-05-17) X-Virus-Checked: Checked by ClamAV on apache.org On Tue, Dec 02, 2008 at 02:29:30PM -0500, Damien Katz wrote: > I think we are just about ready for a 0.9.0 release and will be feature > complete enough to be considered Beta. Me, Chris and Jan have a deadline for the book coming up on the first of January so I would like to leave the release for sometime later that month. This gives me a little time to get in a bunch of fixes that I have been compiling. Thanks, -- Noah Slater, http://tumbolia.org/nslater From couchdb-dev-returndanielmartin@example.com Wed Dec 03 07:10:02 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 79749 invoked from network); 3 Dec 2008 07:10:01 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Dec 2008 07:10:01 -0000 Received: (qmail 50803 invoked by uid 500); 3 Dec 2008 07:10:06 -0000 Delivered-To: danielmartin@example.com Received: (qmail 50742 invoked by uid 500); 3 Dec 2008 07:10:06 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 50721 invoked by uid 99); 3 Dec 2008 07:10:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 23:10:06 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO assmule.apisnetworks.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Dec 2008 07:08:37 +0000 Received: from [83.161.120.155] (206-248-172-247.dsl.teksavvy.com [83.161.120.155]) by assmule.apisnetworks.com (Postfix) with ESMTPSA id 7BB1A21204; Wed, 3 Dec 2008 02:08:53 -0500 (EST) Message-ID: <danielmartin@example.com> Date: Wed, 03 Dec 2008 02:10:48 -0500 From: Jedediah Smith <danielmartin@example.com> User-Agent: Thunderbird 83.161.120.155 (Windows/20080914) MIME-Version: 1.0 To: danielmartin@example.com, danielmartin@example.com Subject: Re: Auto-adding additional fields on PUT/POST? (datetime stamps) References: <danielmartin@example.com> <danielmartin@example.com> <danielmartin@example.com> <danielmartin@example.com> <danielmartin@example.com> <danielmartin@example.com> In-Reply-To: <danielmartin@example.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Chris Anderson wrote: > On Tue, Dec 2, 2008 at 12:01 AM, Jedediah Smith > <danielmartin@example.com> wrote: >> If validations could write, they could serve as a complete data access >> layer. They could handle meta data generation, merging, encapsulation (aka >> implementation hiding/abstraction/conversion) and essentially solve the >> partial update problem. > > I think I've had a change of heart. Damien argues that people would > use validation-writes to generate timestamps. But the timestamps can > be wrong/faked (on remote nodes) and there's no way to validate (on > replication) that they are correct. So putting stamp-capability into > the validation functions would setup false expectations. Apparently it > caused all sorts of trouble in Notes. > > This is an instance of a broader distributed computing axiom: no global clock. If CouchDB can't add timestamps then the client has to do it and the client is sometimes a web browser. Who's more likely to give you bad data? If someone needs timestamps in a multi-master db cluster, then they need timestamps in a multi-master db cluster. Might as well throw them a bone. Anybody in charge of building such a thing is probably somebody who doesn't need their hand held anyway. Notes was a user app.. totally different situation. Oddly, I had to deal with this very problem danielmartin@example.com. Some time deltas were way off and we discovered that changing the clock on the db server affected it. But our problem was that a service was running on the wrong server. >> If merges could be handled by script functions, two-way replication could be >> completely autonomous. Plus, sloppy developers would be more likely to >> actually handle revision conflicts instead of just ignoring them. > > It might be best to save this action for the end of replication, as > replication could apply more than one replicated rev per document. It > would be nice to have a place in design docs for replication conflict > revision, but I don't think validation is that place. > >> Why not just provide hooks to every part of CouchDB? > > I think the important thing to do is design them in such a way as to > give developers the flexibility they need while maintaining the right > expectations about what a distributed database can offer. > > That said, there is a lot of work to add really cool hooks into > CouchDB. One of the most exciting prospects is Comet requests, where > you could do a POST to /db/_since_seq?seq=300 and wait for a response > until the next change after the last change that you knew about > happened. Oh, absolutely.. have many handlers each with very specific semantic requirements. They'll need a way to share code though. Comet is cool and it will be needed in a world of short-stack AJAX apps. But a feed of all updates will be too much data in some cases. How about "live views"? e.g. /db/_view/email/by_date?follow=true&after_seq=300 But then you need a way to represent removed rows.. seems simple enough: {"id":"abc", "key":"123", "deleted":true} Maybe we should move this to couchdb-dev? From couchdb-dev-returndanielmartin@example.com Wed Dec 03 09:11:58 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 21593 invoked from network); 3 Dec 2008 09:11:58 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Dec 2008 09:11:58 -0000 Received: (qmail 38708 invoked by uid 500); 3 Dec 2008 09:12:10 -0000 Delivered-To: danielmartin@example.com Received: (qmail 38673 invoked by uid 500); 3 Dec 2008 09:12:10 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 38662 invoked by uid 99); 3 Dec 2008 09:12:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Dec 2008 01:12:10 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of danielmartin@example.com designates 83.161.120.155 as permitted sender) Received: from [83.161.120.155] (HELO ug-out-1314.google.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Dec 2008 09:10:41 +0000 Received: by ug-out-1314.google.com with SMTP id 36so3668310uga.17 for <danielmartin@example.com>; Wed, 03 Dec 2008 01:11:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=0ESgy5DWATtjtE+Fs3hs+M9wW2KiL+URnA14VWIrPn8=; b=QBBeo2Jf8nxYRZ8KyKIVi4S2GrGenKvcc8TxV5dEE2a4spAAgAaRjjjgjjYf/aI8jb 6xUZMzBO4nRcJ9FN8kZsMuf87VNJriizQ7Kp1PQWcDKLNzjpkfc+aU3EwFe3vLItvTLT Bq7xpgq1stURvNCrPioUGNu01r2nYeFaATqDs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=bJjD24JQm4AtjlsJvzzp72oV7SIdJmKyFJlWoX13cG064TrrSgjzwdIsoK3M855k/P 4DnLKNKrrXn0kboiF75dxpplkOYRZn80M/mzcjpC6Dxczy8IV1pLY3PiLTmMrD+dbqCn LuWsZMJQrJqs6jjqe510sDu0mIPiPlWgjZ1o0= Received: by 83.161.120.155 with SMTP id t4mr1546710ugm.48.1228295478428; Wed, 03 Dec 2008 01:11:18 -0800 (PST) Received: by 83.161.120.155 with HTTP; Wed, 3 Dec 2008 01:11:18 -0800 (PST) Message-ID: <danielmartin@example.com> Date: Wed, 3 Dec 2008 09:11:18 +0000 From: "Paul Carey" <danielmartin@example.com> To: danielmartin@example.com Subject: Re: 0.9.0 Release In-Reply-To: <danielmartin@example.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <danielmartin@example.com> X-Virus-Checked: Checked by ClamAV on apache.org > I think we are just about ready for a 0.9.0 release and will be feature > complete enough to be considered Beta. > This is great news. I'd like to request that COUCHDB-135 gets considered for fixing for 0.9 as it would simplify pagination. Paul From couchdb-dev-returndanielmartin@example.com Wed Dec 03 10:24:11 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 74052 invoked from network); 3 Dec 2008 10:24:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Dec 2008 10:24:11 -0000 Received: (qmail 21457 invoked by uid 500); 3 Dec 2008 10:24:23 -0000 Delivered-To: danielmartin@example.com Received: (qmail 21176 invoked by uid 500); 3 Dec 2008 10:24:22 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 21165 invoked by uid 99); 3 Dec 2008 10:24:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Dec 2008 02:24:22 -0800 X-ASF-Spam-Status: No, hits=3.8 required=10.0 tests=RCVD_NUMERIC_HELO,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO jan.prima.de) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Dec 2008 10:22:53 +0000 Received: from 83.161.120.155.ptr.tiggersystems.info (softdnserr [888a:dbb7:2d49:b3f6:8605:bf4e:caed:e5a4.161.120.155]) (AUTH: LOGIN jan, SSL: TLSv1/SSLv3,128bits,AES128-SHA) by jan.prima.de with esmtp; Wed, 03 Dec 2008 10:17:36 +0000 Message-Id: <danielmartin@example.com> From: Jan Lehnardt <danielmartin@example.com> To: danielmartin@example.com In-Reply-To: <danielmartin@example.com> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v929.2) Subject: Re: 1.0.0 wishlist/roadmap Date: Wed, 3 Dec 2008 11:17:34 +0100 References: <danielmartin@example.com> X-Mailer: Apple Mail (2.929.2) X-Virus-Checked: Checked by ClamAV on apache.org Hi Damien, thanks for taking the time to compile these lists. I'd like to see parallel Map/Reduce execution in 1.0. Right view index creation is run in a single process where it could run in multiple processes. Would it make sense to spawn N `couchjs` instances (N = number of CPUs/cores, or configurable) to run the map and the reduce stages in parallel? This would move (if it isn't already) the bottleneck towards disk I/O rather than JSON serialization. Cheers Jan -- On 2 Dec 2008, at 20:34, Damien Katz wrote: > Here is some stuff I'd like to see in a 1.0.0 release. Everything is > open for discussion. > > - Built-in reduce functions to avoid unnecessary JS overhead - > > Count, Sum, Avg, Min, Max, Std dev. others? > > - Restrict database read access - > > Right now any user can read any database, we need to be able to > restrict that at least on a whole database level. > > - Replication performance enhancements - > > Adam Kocoloski has some replication patches that greatly improve > replication performance. > > - Revision stemming: It should be possible to limit the number of > revisions tracked - > > By default each document edit produces a revision id that is tracked > indefinitely. This guarantees conflicts versus subsequent edits can > always be distinguished in ad-hoc replication, however the forever > growing list of revisions isn't always desirable. THis can be > addressed by limiting the number tracked and purging the oldest > revisions. The downside is that if the revision tracking limited is > N, then anyone who hasn't replicated a document since its last N > edits will see a spurious edit conflict. > > - Lucene/Full-text indexing integration - > > We have this working to in side patches, this needs to be integrated > to trunk and with the view engine > > - Incremental document replication - > > We need at the minimum the ability to incrementally replicate only > the attachments that have changed in a document. This will save lots > of network IO and CouchDB can be version control system with > document diffs added as attachments. > > This can work for document fields too, but the overhead may not be > worth it. > > - Built-in authentication module(s) - > > The ability to host a CouchDB database used for HTTP authentication > schemes. If storing passwords, they would need to be stored > encrypted, decrypted on demand by the authentication process. > > - View server enhancements (stale/partial index option) - > > Chris Anderson has a side branch for this we need to finish and put > into trunk. > > - View index compaction - > > Views indexes expand forever, and need to be compacted in a similar > way the storage files are compacted. This work will tie into the > View Server enhancements. > > - Document integrity/deterministic revid - > > For the sake of end to end document integrity, we need a way to hash > a document's contents, and since we already have revision ids, I > think the revision ids should be the hashes. The hashed document > should be a canonical json representation, and it should have the > _id and _rev fields in it. The _rev will be the PREVIOUS revision ID/ > hash the edit is based on, or blank if a new edit. Then the _rev is > replaced with the new hash value. > > - Fully tail append writes - > > CouchDB uses zero-overwrite storage, but not fully tail append > storage. Document json bodies are stored in internal buffers, > written consecutively, one after another until the buffers in > completely full, then another buffer is created at the end of the > file for more documents. File attachments are written to similar > buffers as well. Btree updates are always tail append, each update > to a btree, even if its a deletion, causes new writes to the end of > the file. Once the document, attachments and indexes are commited > (fsync), the header is then written and flushed to disk, and that is > always stored right at the beginning of the file (requiring another > seek). > > Document updates to CouchDB require 2 fsyncs with ~3 seeks for full > committal and index consistency. This is true if you write 1 or 1000 > documents in a single transaction (bulk update), you still need ~ 3 > seeks. Using conventional transaction journalling, it's possible to > get the committal down to a single seek and fsync, and worry about > ensuring file and index consistency asynchronously, often in batch > mode with other committed updates. This can perform very well, but > has downsides like extra complexity and increased memory usage as > data is cached waiting to be flushed to disk, and must do special > consistency checks and fix-ups on startup if there is a crash. > > If CouchDB used tail-append storage for everything, then all > document updates can be completely flushed with full file > consistency with a single seek and, depending on the file system, a > single fsync. All the disk updates, documents, file attachments, > indexes and file header, occur as appends to the end of the file. > > The biggest changes will be in how file attachments and the headers > are written and read, and the performance characteristics of view > indexing as documents will no longer be packed into contiguous > buffers. > > File attachment will be written in chunks with the last chunk being > an index to the other chunks. > > Headers will be specially signed blocks written to the end of the > file. Reading the header on database open will require scanning the > file from the end, since the file might have partial updates that > didn't complete since the last update. > > The performance of the views will be impacted as the documents are > more likely to be fragmented across the storage file. But they will > still be in the order they will be accessed for indexing, so the > read seeks are always moving forward. Also, the act of compacting > the storage file will result in the documents being tightly packed > again. > > - Streaming document updates with attachment writes - > > Using mime mulitpart encoding, it should be possible to send all > parts of a document in a single http request, with the json and > binary attachments sent as different mime parts. Attachments can be > streamed to disk as bytes are received, keeping total memory > overhead to a minimum. Attachments can also be written to disk in > compressed format and served over http by default in that compressed > format, using 0% CPU for compression at read time, but will require > decompression if the client doesn't support the compression format. > > > - Partitioning/Clustering Support - > > Clustering for failover and load balancing is priority. Large > database support via partitioning may not make 1.0 > > > > > From couchdb-dev-returndanielmartin@example.com Wed Dec 03 10:47:04 2008 Return-Path: <couchdb-dev-returndanielmartin@example.com> Delivered-To: danielmartin@example.com Received: (qmail 85297 invoked from network); 3 Dec 2008 10:47:03 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (83.161.120.155) by minotaur.apache.org with SMTP; 3 Dec 2008 10:47:03 -0000 Received: (qmail 46152 invoked by uid 500); 3 Dec 2008 10:47:13 -0000 Delivered-To: danielmartin@example.com Received: (qmail 46075 invoked by uid 500); 3 Dec 2008 10:47:13 -0000 Mailing-List: contact danielmartin@example.com; run by ezmlm Precedence: bulk List-Help: <mailto:danielmartin@example.com> List-Unsubscribe: <mailto:danielmartin@example.com> List-Post: <mailto:danielmartin@example.com> List-Id: <couchdb-dev.incubator.apache.org> Reply-To: danielmartin@example.com Delivered-To: mailing list danielmartin@example.com Received: (qmail 46063 invoked by uid 99); 3 Dec 2008 10:47:12 -0000 Received: from athena.apache.org (HELO athena.apache.org) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Dec 2008 02:47:12 -0800 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [83.161.120.155] (HELO assmule.apisnetworks.com) (83.161.120.155) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Dec 2008 10:45:38 +0000 Received: from [83.161.120.155] (206-248-172-247.dsl.teksavvy.com [83.161.120.155]) by assmule.apisnetworks.com (Postfix) with ESMTPSA id F36802744E8 for <danielmartin@example.com>; Wed, 3 Dec 2008 05:45:54 -0500 (EST) Message-ID: <danielmartin@example.com> Date: Wed, 03 Dec 2008 05:47:50 -0500 From: Jedediah Smith <danielmartin@example.com> User-Agent: Thunderbird 83.161.120.155 (Windows/20080914) MIME-Version: 1.0 To: danielmartin@example.com Subject: Re: 1.0.0 wishlist/roadmap References: <danielmartin@example.com> In-Reply-To: <danielmartin@example.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Damien Katz wrote: > - Restrict database read access - > > Right now any user can read any database, we need to be able to restrict > that at least on a whole database level. > > - Built-in authentication module(s) - > > The ability to host a CouchDB database used for HTTP authentication > schemes. If storing passwords, they would need to be stored encrypted, > decrypted on demand by the authentication process. I wouldn't launch with anything less than extensible, per-doc read/write
{'source': 'Apache Software Foundation Public Mail Archives'}