My experience with using cp to copy 432 million files (39 TB) - <AUTHOR> http://lists.gnu.org/archive/html/coreutils/2014-08/msg00012.html <comment><USER2> I wrote a little copy program at my last job to copy files in a reasonable time frame on 5PB to 55PB filesystems.

[https://github.com/hpc/dcp](https://github.com/hpc/dcp)

We got an IEEE paper out of it:

[http://conferences.computer.org/sc/2012/papers/1000a015.pdf](http://conferences.computer.org/sc/2012/papers/1000a015.pdf)

A few people are continuing the concept to other tools -- that should be available at [http://fileutils.io/](http://fileutils.io/) relatively soon.

We also had another tool written on top of [https://github.com/hpc/libcircle](https://github.com/hpc/libcircle) that would gather metadata on a few hundred-million files in a few hours (we had to limit the speed so it wouldn't take down the filesystem). For a slimmed down version of that tool, take a look at [https://github.com/hpc/libdftw](https://github.com/hpc/libdftw)

<comment><USER3> And it's interesting and useful for scientific computing where you already have an MPI environment and distributed/parallel filesystems. However, it's not really applicable to this workload, as the paper itself says.

 _There is a provision in most file systems to use links (symlinks, hardlinks, etc.). Links can cause cycles in the file tree, which would result in a traversal algorithm going into an infinite loop. To prevent this from happening, we ignore links in the file tree during traversal. We note that the algorithms we propose in the paper will duplicate effort proportional to the number of hardlinks. However, in real world production systems, such as in LANL (and others), for simplicity, the parallel filesystems are generally not POSIX compliant, that is, they do not use hard links, inodes, and symlinks. So, our assumption holds._

The reason this cp took such large amounts of time was the desire to preserve hardlinks and the resize of the hashtable used to track the device and inode of the source and destination files.

<comment><USER4> Sure, but if you read that article you walk away with a sense of _thats a lot of files to copy_. And the GP built a tool for jobs 2-3 orders of magnitude larger?! Clearly there are tradeoffs forced on you at that size...

</comment></comment></comment><comment><USER5> How about this for a better cp strategy to deal with hardlinks:

1\. Calculate the hash of /sourcedir/some/path/to/file

2\. Copy the file to /tempdir/$hash if it doesn't exist yet

3\. Hard-link /destdir/some/path/to/file to /tempdir/$hash

4\. Repeat until you run out of source files

5\. Recursively delete /tempdir/

This should give you a faithful copy with all the hard-links with constant RAM at the cost of CPU to run all the hashing. If you're smart about doing steps 1 and 2 together it shouldn't require any additional I/O (ignoring the extra file metadata).

Edit: actually this won't recreate the same hardlink structure, it will deduplicate any identical files, which may not be what you want. Replacing the hashing with looking up the inode with stat() would actually do the right thing. And that would basically be an on-disk implementation of the hash table cp is setting up in memory.

<comment><USER6> If you cp your data onto a Plan9 machine, what results is pretty much exactly the process you've outlined.

Plan9's default filesystem is made up of two parts: Fossil, and Venti.

\- Fossil is a content-addressable on-disk object store. Picture a disk "formatted as" an S3 bucket, where the keys are strictly the SHAsums of the values.

\- Venti is a persistent graph database that holds what would today be called "inode metadata." It presents itself as a regular hierarchical filesystem. The "content" property of an inode simply holds a symbolic path, usually to an object in a mounted Fossil "bucket."

When you write to Venti, it writes the object to its configured Fossil bucket, then creates an inode pointing to that key in that bucket. If the key already existed in Fossil, though, Fossil just returns the write as successful immediately, and Venti gets on with creating the inode.

Honestly, I'm terribly confused why all filesystems haven't been broken into these two easily-separable layers. (Microsoft attempted this with WinFS, but mysteriously failed.) Is it just inertia? Why are we still creating new filesystems (e.g. btrfs) that don't follow this design?

<comment><USER5> _> Honestly, I'm terribly confused why all filesystems haven't been broken into these two easily-separable layers. Is it just inertia?_

The penalty for doing content addressed filesystems is of course the CPU usage. btrfs probably has most of the benefits without the CPU cost with its copy-on-write semantics.

Note that what you describe (and my initial process) is a different semantic than hard-links. What you get is shared storage but if you write to one of the files only that one gets changed. Whereas with hardlinks both files change.

<comment><USER6> In effect, hard links (of mutable files) are a declaration that certain files have the same "identity." You can't get this with plain Venti-on-Fossil, but it's a problem with Fossil (objects are immutable), not with Venti.

Venti-on-Venti-on-Fossil would work, though, since Venti just creates imaginary files that inherit their IO semantics from their underlying store, and this should apply recursively:

1\. create two nodes A and B in Venti[1] that refer to one node C in Venti[2], which refers to object[x] with key x in Fossil.

2\. Append to A in Venti[1], causing a write to C in Venti[2], causing a write to object[x] Fossil, creating object[y] with key y.

3\. Fossil returns y to Venti[2]; Venti[2] updates C to point to object[y] and returns C to Venti[1]; Venti[1] sees that C is unchanged and does nothing.

Now A and B both effectively point to object[y].

(Note that you don't actually have to have two Venti servers for this! There's nothing stopping you from having Venti nodes that refer to other Venti nodes within the same projected filesystem--but since you're exposing these nodes to the user, your get the "dangers" of symbolic links, where e.g. moving them breaks the things that point to them. For IO operations they have the semantics of hard links, though, instead of needing to be special-cased by filesystem-operating syscalls.)

<comment><USER7> You seem to be confusing venti and fossil.

<comment><USER8> Can you explain further? I am not a plan9 expert, by any means, but I'm stuck at where GP made the confusion. Thanks!

<comment><USER9> He just swapped the names I think - Venti is the block store, Fossil is the file system layer.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> _Disassembling data structures nicely can take much more time than just tearing them down brutally when the process exits._

A wonderful trend I've noticed in Free/Open Source software lately is proudly claiming that a program is "Valgrind clean." It's a decent indication that the program won't doing anything silly with memory during normal use, like leak it. (There's also a notable upswing in the number of projects using static analyzers on their code and fixing legitimate problems that turn up, which is great, too!)

While you can certainly just let the OS reclaim all of your process's allocated memory at exit time, you're technically (though intentionally) leaking memory. When it becomes too hard to separate the intentional leaks from the unintentional leaks, I'd wager most programmers will just stop looking at the Valgrind reports. (I suppose you could wrap free() calls in "#ifdef DEBUG ... #endif" blocks and only run Valgrind on debug builds, but that seems ugly.)

A more elegant solution is to use an arena/region/zone allocator and place potentially large data structures (like cp's hard link/inode table) entirely in their own arenas. When the time comes to destroy one of these data structures, you can destroy its arena with a single function call instead of walking the data structure and free()ing it piece by piece.

Unfortunately, like a lot of useful plumbing, there isn't a standard API for arena allocators, so actually doing this in a cross-platform way is painful:

• Windows lets you create multiple heaps and allocate/free memory in them (HeapCreate(), HeapDestroy(), HeapAlloc(), HeapFree(), etc.).

• OS X and iOS come with a zone allocator (malloc_create_zone(), malloc_destroy_zone(), malloc_zone_malloc(), malloc_zone_free(), etc.).

• glibc doesn't have a user-facing way to create/destroy arenas (though it uses arenas internally), so you're stuck using a third-party allocator on Linux to get arena support.

• IRIX used to come with an arena allocator (acreate(), adelete(), amalloc(), afree(), etc.), so if you're still developing on an SGI Octane because you can't get enough of that sexy terminal font, you're good to go.

<comment><USER11> Adding some kind of arena-allocation library to both the build & runtime dependencies _solely_ to keep valgrind happy, with no actual improvement in functionality or performance, doesn't seem like a great tradeoff on the software engineering front. I'd rather see work on improving the static analysis. For example if some memory is intended to be freed at program cleanup, Valgrind could have some way of being told, "this is intended to be freed at program cleanup". Inserting an explicit (and redundant) deallocation as the last line of the program just to make the static analyzer happy is a bit perverse.

(That is, assuming that you don't need portability to odd systems that don't actually free memory on process exit.)

<comment><USER12> I don't see why you assume arenas would be added "solely to keep valgrind happy". Arenas have better performance when allocating a high number of small chunks, because an arena can make better performance trade-offs for this use case than the general malloc allocator.

</comment></comment></comment><comment><USER13> This may be a little off topic, but I used to think RAID 5 and RAID 6 were the best RAID configs to use. It seemed to offer the best bang for buck. However, after seeing how long it took to rebuild an array after a drive failed (over 3 days), I'm much more hesitant to use those RAIDS. I much rather prefer RAID 1+0 even though the overall cost is nearly double that of RAID 5. It's much faster, and there is no rebuild process if the RAID controller is smart enough. You just swap failed drives, and the RAID controller automatically utilizes the back up drive and then mirrors onto the new drive. Just much faster and much less prone to multiple drive failures killing the entire RAID.

<comment><USER14> This can not be stressed strongly enough. There is never a case when RAID5 is the best choice, ever [1]. There are cases where RAID0 is mathematically proven more reliable than RAID5 [2]. RAID5 should never be used for anything where you value keeping your data. I am not exaggerating when I say that very often, your data is safer on a single hard drive than it is on a RAID5 array. Please let that sink in.

The problem is that once a drive fails, during the rebuild, if any of the surviving drives experience an unrecoverable read error (URE), the entire array will fail. On consumer-grade SATA drives that have a URE rate of 1 in 10^14, that means if the data on the surviving drives totals 12TB, the probability of the array failing rebuild is close to 100%. Enterprise SAS drives are typically rated 1 URE in 10^15, so you improve your chances ten- fold. Still an avoidable risk.

RAID6 suffers from the same fundamental flaw as RAID5, but the probability of complete array failure is pushed back one level, making RAID6 with enterprise SAS drives possibly acceptable in some cases, for now (until hard drive capacities get larger).

I no longer use parity RAID. Always RAID10 [3]. If a customer insists on RAID5, I tell them they can hire someone else, and I am prepared to walk away.

I haven't even touched on the ridiculous cases where it takes RAID5 arrays weeks or months to rebuild, while an entire company limps inefficiently along. When productivity suffers company-wide, the decision makers wish they had paid the tiny price for a few extra disks to do RAID10.

In the article, he has 12x 4TB drives. Once two drives failed, assuming he is using enterprise drives (Dell calls them "near-line SAS", just an enterprise SATA), there is a 33% chance the entire array fails if he tries to rebuild. If the drives are plain SATA, there is almost no chance the array completes a rebuild.

[1] [http://www.smbitjournal.com/2012/11/choosing-a-raid-level- by...](http://www.smbitjournal.com/2012/11/choosing-a-raid-level-by-drive- count/)

[2] [http://www.smbitjournal.com/2012/05/when-no-redundancy-is- mo...](http://www.smbitjournal.com/2012/05/when-no-redundancy-is-more- reliable/)

[3] [http://www.smbitjournal.com/2012/11/one-big-raid-10-a-new- st...](http://www.smbitjournal.com/2012/11/one-big-raid-10-a-new-standard-in- server-storage/)

<comment><USER15> I think your calculation on failing an array rebuild is wrong. Can you show how you got those numbers?

<comment><USER14> Sure, there were two statements I made.

> _On consumer-grade SATA drives that have a URE rate of 1 in 10^14, that > means if the data on the surviving drives totals 12TB, the probability of > the array failing rebuild is close to 100%._

10^14 bits is 12.5 TB, so on average, the chance of 12TB being read without a single URE is very low, and the probability the array fails to rebuild is close to 100%. I was estimating 10^14 bits to be about 12TB, so the probability is actually 12/12.5 = 96% chance of failure.

> _...he has 12x 4TB drives. Once two drives failed, assuming he is using > enterprise drives...there is a 33% chance the entire array fails if he tries > to rebuild. If the drives are plain SATA, there is almost no chance the > array completes a rebuild._

A RAID6 with two failed drives is effectively the same situation as a RAID5 with one failed drive. In order to rebuild one failed drive, the RAID controller must read all data from every surviving drive to recreate the failed drive. In this case, there are 10x 4TB surviving drives, meaning 40TB of data must be read to rebuild. Because these drives are presumably enterprise quality, I am assuming they are rated to fail reading one sector for every 10^15 bits read (10^15 bits = 125 TB). So it's actually 40/125 = 32% chance of failure if you try to rebuild.

</comment></comment></comment></comment><comment> vhost- These are the types of stories I love. I just learned a boat load in 5 minutes.

<comment><USER16> Is there maybe an archive website dedicated to these kind of stories?

<comment><USER17> At one time there was; it was called the Internet. The archive still exists, but it's been made harder to browse through due to being jumbled up with javascript and cat gifs.

<comment><USER18> It's true. We should have never let the public on the Internet. It has been downhill since then.

<comment><USER19> False choice, isn't it? I mean, the complaint isn't that the public now has sites with massive javascript and related technologies. The complaint is that it has muscled out useful sites that did not use those technologies. And it should be heavily noted that the heavy muscles that have pushed out many of these sites is not necessarily "the public."

<comment><USER20> Kind of funny to say that on a text-only JS-free site that seems to be alive and well, linking to an article on an old-school mailing list archive site. :)

<comment><USER19> Oh, certainly. I just can resonate with the sentiment that these sites aren't the majority.

Even this site, honestly, is less than easy to deal with on a recurring basis. (Consider, hard to remember which was the top story three days ago at noon.) Specifically, sometimes I lose a story because I refresh and something plummeted off the page. Hard to have any idea how far to "scroll back" to see it.

</comment></comment></comment></comment></comment></comment></comment><comment><USER21> I would usually use the tarpipe mentioned already by others for this sort of thing (although I probably wouldn't do 432 million files in one shot):

      (cd $SOURCE && tar cf - .) | (mkdir -p $DEST && cd $DEST && tar xf -)

Another option which I just learned about through reading some links from this thread is pax ([http://en.wikipedia.org/wiki/Pax_%28Unix%29](http://en.wikipedia.org/wiki/Pax_%28Unix%29)), which can do it with just a single process:

      (mkdir -p $DEST && cd $SOURCE && pax -rw . $DEST)

Both will handle hard links fine, but pax may have some advantages in terms of resource usage when processing huge numbers of files and tons of hard links.

<comment><USER22> You know how tar handles hardlinks, right? By creating a giant hash table of every file.

<comment><USER23> How's that going to scale with memory? In-memory hash tables were the downfall of cp here.

<comment><USER22> It's going to scale just like you'd imagine it would. All the people saying "oh, tar was built for this" obviously haven't actually tried replicating the experiment using tar.

<comment><USER23> Pretty much as I'd suspected.

</comment></comment></comment></comment></comment><comment><USER24> I've written a program that attempts to deal with the given situation gracefully: instead of using a hash table, it creates a temporary file with a list of inode/device/path entries, then sorts this according to inode/device, then uses the sorted list to perform the copying/hardlinking. The idea is that sorting should work well with much lower RAM requirements than the size of the file to be sorted (due to data locality, unless the random accesses with the hash, it will be able to work with big chunks, at least when done right (a bit hand-wavy, I know, this is called an "online algorithm" and I remember Knuth having written about those, haven't had the chance to recheck yet); the program is using the system sort command, which is hopefully implementing this well already).

The program stupidly calls "cp" right now for every individual file copy (not the hard linking), just to get the script done quickly, it's easy to replace that with something that saves the fork/exec overhead; even so, it might be faster than the swapping hash table if the swap is on a spinning disk. Also read the notes in the --help text. I.e. this is a work in progress as a basis to test the idea, it will be easy to round off the corners if there's interest.

[https://github.com/pflanze/megacopy](https://github.com/pflanze/megacopy)

PS. the idea of this is to make copying work well with the given situation on a single machine, unless the approach taken by the dcp program mentioned by fintler which seems to rely on a cluster of machines.

There may also be some more discussion about this on the mailing list: [http://lists.gnu.org/archive/html/coreutils/2014-09/msg00013...](http://lists.gnu.org/archive/html/coreutils/2014-09/msg00013.html)

</comment><comment><USER25> So it was all the files in one go, presumably with `cp -r`?

What about doing something with find/xargs/i-dunno to copy all the files, but break em into batches so you aren't asking cp to do it's bookkeeping for so many files in one process? Would that work better? Or worse in other ways?

<comment><USER26> This page may be useful:

[http://unix.stackexchange.com/questions/44247/how-to-copy- di...](http://unix.stackexchange.com/questions/44247/how-to-copy-directories- with-preserving-hardlinks)

The main issue is that there's no api to get the list of files hard linked together: the only way is to check all the existing files and compare inodes. If you're doing a plain copy over 2 fs, you cannot choose which number the target inode will be, so you need to keep a map between inode numbers, or between inodes and file names ("cp" does the later).

<comment><USER27> pedrocr's comment above suggests a good solution:

1\. Copy each file from the source volume to a single directory (e.g. /tmp) on the target volume, named for the source volume inode number.

(edit: I suggest using a hierarchy of dirs to avoid the "too many dentry's" slowdown)

2\. If the file has already been copied, it will already exist in /tmp - looking up the inode is a vanilla directory lookup

3\. Create a hard link from /tmp to the actual path of the file

4\. When all the files have been created on the target volume, delete the inode numbers in /tmp

</comment></comment></comment><comment><USER5> Unix could really use a way to get all the paths that point to a given inode. These days that shouldn't really cost all that much and this issue comes up a lot in copying/sync situations. Here's the git-annex bug report about this:

[https://git-annex.branchable.com/bugs/Hard_links_not_synced_...](https://git- annex.branchable.com/bugs/Hard_links_not_synced_in_direct_mode/)

<comment><USER28> Wow, it's not every day I hear about a filesystem feature that Windows has and Linux doesn't. (On a recent windows system: _fsutil hardlink list <path>_ \-- you can try any random exe or DLL in system32 for an example of a hard link.)

I forget what the api for that looks like if I ever knew. Might be private.

I am surprised, usually Linux is way ahead of Windows on shiny filesystem stuff.

<comment><USER29> Linux just has more filesystems, and sadly a lot of them have various flaws. I'm surprised when people are surprised that Linux isn't some completely superior technical marvel. BSD and Unix systems have been more advanced for decades..

Everyone on Linux still uses _tar_ for god's sake, even though zip can use the same compression algorithms people use on tarballs, and zip actually stores an index of its files rather than 'cat'ing each record on top of the next like an append-only tape archive. (Obviously there are better formats than 'zip' for any platform, but it's just strange that nobody has moved away from tar)

<comment><USER30> tar is good enough for many uses, so people did not move on.

And it doesn't help that tar.gz / tar.bz2 compresses way better than zip in most cases (thanks to using a single compression context, rather than a new one for each file; and also compressing the filenames in the same context), and that it carries ownership and permission information with it - whereas zip doesn't.

The HVSC project, who try to collect every single piece of music ever created on a Commodore C64, distribute their archive as a zip-within-a-zip. The common music file is 1k-4k, goes down to ~500-1000 bytes zipped; The subdirectory+filename are often 100 bytes with a lot of redundancy that zip doesn't use, so they re-zip. Had they used .tar.gz or .tar.bz2, the second stage would not be needed.

</comment></comment></comment></comment><comment><USER31> I found an issue in cp that caused 350% extra mem usage for the original bug reporter, which fixing would have kept his working set at least within RAM.

[http://lists.gnu.org/archive/html/coreutils/2014-09/msg00014...](http://lists.gnu.org/archive/html/coreutils/2014-09/msg00014.html)

</comment><comment><USER32> > Wanting the buffers to be flushed so that I had a complete logfile, I gave > cp more than a day to finish disassembling its hash table, before giving up > and killing the process....Disassembling data structures nicely can take > much more time than just tearing them down brutally when the process exits.

Does anyone know what the 'tear down' part is about? If it's about erasing the hashtable from memory, what takes so long? I would expect that to be very fast: you don't have to write zeros to it all, you just tell your GC or memory manager to mark it as free.

<comment><USER33> Looking at the code, it looks like deallocating a hash table requires traversing the entire table, because there is malloc()'d memory associated with each hash entry, so each entry has to be visited and free()'d. From hash_free() in coreutils hash.c:

        for (bucket = table->bucket; bucket < table->bucket_limit; bucket++)
          {
            for (cursor = bucket->next; cursor; cursor = next)
              {
                next = cursor->next;
                free (cursor);
              }
          }

Whereas if you just don't bother to deallocate the table before the process exits, the OS will reclaim the whole memory block without having to walk a giant data structure. That's a fairly common situation in C programs that do explicit memory management of complex data structures in the traditional malloc()/free() style. Giant linked lists and graph structures are another common culprit, where you have to pointer-chase all over the place to free() them if you allocated them in the traditional way (vs. packing them into an array or using a userspace custom allocator for the bookkeeping).

<comment><USER34> Why exactly is it necessary to to free each hash entry instead of exiting the process?

<comment><USER33> If it's the last thing you do before you exit the process, it isn't necessary, because the OS will reclaim your process's memory in one fell swoop. I believe that's what the linked post is advocating 'cp' should do. (At least on modern systems that's true; maybe there are some exotic old systems where not freeing your data structures before exit causes permanent memory leaks?)

It's seen as good C programming practice to free() your malloc()s, though, and it makes extending programs easier if you have that functionality, since what was previously the end of program can be wrapped in a higher-level loop without leaking memory. But if you really are exiting for sure, you don't have to make the final free-memory call. It can also be faster to not do any intermediate deallocations either: just leave everything for the one big final deallocation, as a kind of poor-man's version of one-generation generational GC. Nonetheless many C programmers see it somehow as a bit unclean not to deallocate properly. Arguably it does make some kind of errors more likely if you don't, e.g. if you have cleanup that needs to be done that the OS _doesn 't_ do automatically, you now have different kinds of cleanup routines for the end-of-process vs. not-end-of-process case.

<comment><USER35> I tend to do this in my C programs because in development usually have malloc() wrapped so that if any block hasn't been free()'ed it's reported at exit() time. This kind of check for lost pointers is usually so cheap that you use it even if you never expect to run on a system without decent memory management.

As an aside, GNU libc keeps ( or at least used to keep, I haven't checked in years ) the pointers used by malloc()/free() next to the blocks themselves, which gives really bad behavior when freeing a large number of blocks that have been pushed out to swap--you wind up bringing in pages in order to free them because the memory manager's working set is the size of all allocated memory. Years ago I wrote a replacement that avoided this just to speed up Netscape's horrible performance when it re-sized the bdb1.85 databases it used to track browser history. The browser would just "go away" thrashing the disk for hours and killing it just returned you to a state where it would decide to resize again an hour or so after a restart. Using LD_PRELOAD to use a malloc that kept it's bookkeeping away from the allocated blocks changed hours to seconds.

</comment></comment></comment></comment></comment><comment><USER36> I appreciate that he had the foresight to install more ram and configure more swap. I would hate to be days into a transfer and have the OOM killer strike.

</comment><comment><USER37> The difficulty is that you are using a filesystem hierarchy to 'copy files' when you actually want to do a volume dump (block copy). Use XFS and xfsdump, or ZFS and zfs send, to achieve this.

Copy with hard link preservation is essentially like running dedupe except that you know ahead of time how many dupes there are. Dedupe is often very memory intensive, and even well thought out implementations don't support keeping book keeping structures on disk.

<comment><USER38> "Normally I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained the bad blocks."

<comment><USER37> I was simplifying... dump backs up inodes not blocks. Some inodes point to file data and some point to directory data. Hard links are references to the same inode in multiple directory entries, so when you run xfsrestore, the link count increments as the FS hierarchy is restored.

xfsdump/zfs send are file system aware, unlike dd, and can detect fs corruption (ZFS especially having extensive checksums). In fact, any info cp sees about corruption comes from the FS code parsing the FS tree.

However, except on zfs/btrfs, data block corruption will pass unnoticed. And in my experience, when you have bad blocks, you have millions of them -- too many to manually fix. As this causes a read hang, it is usually better to dd copy the fs to a clean disk, set to replace bad blocks with zeros, then fsck/xfs_repair when you mount, then xfsdump.

dd conv=noerror,sync,notrunc bs=512 if=/dev/disk of=diskimg

See Also: [http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en- US...](http://xfs.org/docs/xfsdocs-xml-dev/XFS_User_Guide/tmp/en-US/html/xfs- repair.html) [http://xfs.org/index.php/Reliable_Detection_and_Repair_of_Me...](http://xfs.org/index.php/Reliable_Detection_and_Repair_of_Metadata_Corruption)

<comment><USER39> If the risk of keeping the system running while the array rebuilt was deemed to high, I would have just gone with a dd/ddrescue of the remaining disks onto new disks and then moved on from there.

+1 for mentioning ZFS. It's really quite amazing. Almost like futuristic alien technology compared to the other freely available file systems.

</comment></comment></comment></comment><comment><USER40> In light of experience would it perhaps be helpful after all to use a block- level copy (such as Partclone, PartImage, or GNU ddrescue) and analyze later which files have the bad blocks?

I see that the choice of a file-level copy was deliberate: "I'd have copied/moved the files at block-level (eg. using dd or pvmove), but suspecting bad blocks, I went for a file-level copy because then I'd know which files contained the bad blocks."

<comment><USER41> Also there is no mention of unrecoverable file analysis like error handling of cp operations in the article. And with this many files it would not be feasible without using an error log file.

So going with a simple block copy should suffice IMHO.

<comment><USER42> I'm the OP, so I can shred a bit of light on that: Dell's support suggested a file-level copy when I asked them what they recommended (but I'm not entirely sure they understood the implications). Also, time was not a big issue.

I did keep a log file with the output from cp, and it clearly identified the filenames for the inodes with bad blocks. Actually, I'm not sure how dd would handle bad blocks.

<comment><USER41> Thank you for clarification.

I was about to bet on "read fail repeat skip" cycle for dd's behaviour but, looking into coreutil's source code at [https://github.com/goj/coreutils/blob/master/src/dd.c](https://github.com/goj/coreutils/blob/master/src/dd.c) , if I'm not mistaken , dd does not try to be intelligent and just uses a zeroed out buffer so It would return 0's for unreadable blocks.

</comment></comment></comment></comment><comment><USER43> Interesting.

In Windows-land, the default copy is pretty anemic, so probably most people avoid it for serious work.

I'd probably use robocopy from the command line. And if I was being lazy, I'd use the Teracopy GUI.

I think my limit for a single copy command has been around 4TB with robocopy-- and that was a bunch of large media files, instead of smaller more numerous files. Maybe there's a limit I haven't hit.

<comment><USER44> > Teracopy

I've used FastCopy for GUI based larger transfers, it's open source and can handle larger datasets well in my experience. It also doesn't choke on >MAX_PATH paths. Haven't had problems with it. Supposedly it's the fastest tool around...

The only slight issue is that the author is Japanese so the English translations aren't perfect plus the comments in the source are in Japanese.

<comment><USER45> ">MAX_PATH paths"

How does this happen?

<comment><USER46> Technical debt that keeps on giving.

Today there are N applications. "We can't increase MAX_PATH because it will break existing applications!"

Tomorrow there are N+M applications. "We can't increase MAX_PATH because it will break existing applications!"

Repeat forever.

Any time you are faced with a hard technical decision like this, the pain will always be least if you make the change either:

1\. During another transition (e.g. 16-bit to 32-bit, or 32-bit to 64-bit). Microsoft could have required all 64-bit Windows apps to adopt a larger MAX_PATH, among other things.

2\. Right NOW, because there will never be an easier time to make the change. The overall pain to all parties will only increase over time.

</comment></comment></comment></comment><comment><USER47> Another lesson to be learnt is that it's nice to have the source code for the tools we are using.

</comment><comment><USER23> The email states that file-based copy operations were used in favor of dd due to suspected block errors. Two questions come to mind:

1\. I've not used dd on failing media, so I'm not sure of the behavior. Will it plow through a file with block-read failures or halt?

2\. There's the ddrescue utility, which _is_ specifically intended for reading from nonreliable storage. Seems that this could have offered another means for addressing Rasmus's problem. It can also fill in additional data on multiple runs across media, such that more complete restores might be achieved. [https://www.gnu.org/software/ddrescue/ddrescue.html](https://www.gnu.org/software/ddrescue/ddrescue.html)

<comment><USER24> OP said "I went for a file-level copy because then I'd know which files contained the bad blocks". When you copy the block device with ddrescue (dd doesn't have logic to work around the bad sectors and the only sensible action for it is thus to stop, but don't take my word for it), the result will just have zeroes in the places where bad blocks were, and, assuming the filesystem structure is good enough (you should run fsck on it), will give you files with zones of zeroes. But you won't know which files without either comparing them to a backup (which you won't have by definition if you're trying to recover) or with a program that verifies every file's structure (which won't exist for the general case). Whereas cp will issue error messages with the path of the file in question. So the OP's decision makes sense.

<comment><USER23> I've played with ddrescue very lightly. From the GNU webpage linked above, it appears it creates logfiles which can be examined:

 _Ddrescuelog is a tool that manipulates ddrescue logfiles, shows logfile contents, converts logfiles to /from other formats, compares logfiles, tests rescue status, and can delete a logfile if the rescue is done. Ddrescuelog operations can be restricted to one or several parts of the logfile if the domain setting options are used._

That might allow for identification of files with bad sectors.

<comment><USER24> That would need either a hook to the kernel or a file system parser.

Even if you manage to do that, I'm not sure it would be a good idea to continue to use a file system that has lost sectors, even after fsck. Are you sure fsck is fixing any inconsistency? Are there any automatic procedures in place that guarantee that the fsck algorithms are in sync with the actual file system code? (Answer anew for any file system I might be using.) You definitely should do backups by reading the actual files, not the underlying device; perhaps in this case it could be OK (since it was a backup itself already, hence a copy of life data; but then if OP bothered enough to recover the files, maybe he'll bother enough to make sure they stay recovered?)

</comment></comment></comment></comment><comment><USER48> For that many files I probably would've used rsync between local disks. _shrug_

<comment><USER49> And hopefully you would have written up a similar essay on the oddball experiences you had with rsync, which is even more stateful than cp and even more likely to have odd interactions when used outside its comfort zone.

Ditto for tricks like:(cd $src; tar cf - .) | (cd $dst; tar xf -).

Pretty much nothing is going to work in an obvious way in a regime like this. That's sort of the point of the article.

<comment><USER48> Or maybe not. He mentions rsnapshot in the article, which uses rsync under the hood. This implies rsync would have a _very_ good chance of handling a large number of hardlinks... since it created them in the first place.

<comment><USER50> That doesn't follow. If backups are for multiple machines to a big file server, the backup machine will have a much larger set of files than those that come from an individual machine. Further, each backup "image" compares the directory for the previous backup to the current live system. Generally it looks something like this:

1\. Initial backup or "full backup" \- copy the full targeted filesystem to the time indexed directory of the backup machine.

2\. Sequential backups:

a. on the backup machine, create a directory for the new time, create a mirror directory structure of the previous time.

b. hard link the files in the new structure to those in the previous backup (which may be links themselves, back to the last full backup.

c. rsync the files to the new backup directory. Anything that needs to be transfered results in rsync transfering the file to a new directory, the moving it into the proper place. This unlinks the filename from the previous version and replaces it with the full version.

So yeah, the result of this system over a few machines and a long-timeframe backup system is way more links on the backup machine than any iteration of the backup will ever actually use.

<comment><USER48> Yes, it has more links, I realize, but this still doesn't mean it wouldn't work. Give it a shot and report back. (Hah.)

</comment></comment></comment></comment></comment><comment><USER51> _> The number of hard drives flashing red is not the same as the number of hard drives with bad blocks._

This is the real take-away. Monitor your drives. At very least enable SMART, and also regularly run a read on the full underlying drive (SMART won't see and log blocks that are on the way out so need retries for successful reads, unless you actually try to read those blocks).

That won't completely make you safe, but it'll greatly reduce the risk of other drives failing during a rebuild by increasing the chance you get advanced warning that problems are building up.

<comment><USER42> Glad someone noticed it (I'm the OP). Reading the drives systematically is called "Patrol Read" and is often enabled by default, but you can tweak the parameters.

</comment></comment><comment><USER52> The later replies regarding the size of the data structures cp is using are also worth reading. This is a case where pushing the command farther can make you think harder about the computations being done.

</comment><comment><USER53> On Unix, isn't it considered bad practice to use cp in order to copy a large directory tree?

IIRC, the use of tar is recommended.

Something like:

        $ (cd $origin && tar cf - *) | (cd $destination && tar xvf - )

<comment><USER54> Use && there, not ; - consider the result if either of the cd commands fails.

<comment><USER53> fixed

</comment></comment></comment><comment><USER55> > While rebuilding, the replacement disk failed, and in the meantime another > disk had also failed.

I feel the pain. I went thru the same hell a few months ago.

</comment><comment><USER56> Another lesson: routinely scrub your RAID arrays.

<comment><USER57> On debian-based systems, /etc/cron.d/mdadm will already do this on the first Sunday of the month.

</comment></comment><comment><USER58> I wonder how well rsync would have fared here.

<comment><USER36> Rsync can die just from scanning the whole directory tree of files first.

<comment><USER59> The incremental option(enabled by default) introduced in rsync 3.0 greatly reduces the need for scanning the whole directory structure.

</comment></comment></comment><comment><USER60> Maybe this is naive, but wouldn't it have made more sense to do a bunch of smaller cp commands? Like sweep through the directory structure and do one cp per directory? Or find some other way to limit the number of files copied per command?

<comment><USER61> No, because then it wouldn't have replicated the hardlink structure of the original tree. That was the goal, and also the bit that causes the high resource consumption.

</comment></comment><comment><USER62> A problem with cp (and rsync, tar, and linux in general) is there is read- ahead within single files, but no read-ahead for the next file in the directory. So it doesn't make full use of the available IOPS capacity.

</comment><comment><USER63> This is not, not, not how one should be using RAID.

The math is clear that in sufficiently large disk systems, RAID5, RAID6, and friends, are all insufficient.

<comment><USER64> Can you elaborate?

<comment><USER63> [http://www.zdnet.com/blog/storage/why-raid-5-stops- working-i...](http://www.zdnet.com/blog/storage/why-raid-5-stops-working- in-2009/162)

<comment><USER64> Thanks for the link! The article says, that due to the read error rate and the size of today's disks, RAID 5 and RAID 6 have (kind of) lost their purpose.

<comment><USER63> Yep, mathematically no longer safe.

</comment></comment></comment></comment></comment><comment><USER65> >We use XFS

Why?

<comment><USER66> I personally still consider XFS a very mature and reliable filesystem. Both in terms of utility programs and kernel implementation. If I remember correctly, it was ported to linux from SGI/Irix where it was used for decades. It also was the default fs for RedHat/centos for a long time, so it might still have stuck at many shops.

Heres my anecdotal datapoint on which I base my personal believe:

From about 10-6 years ago, when I was doing sysadmin-work at university building storage-systems from commodity parts for experimental bulk data, we first had a load of not-reliably working early raid/sata(?) adapters, and those made ext3 and reiserfs (I think...) oops the kernel when the on-disk structure went bad. Whereas XFS just put a "XFS: remounted FS readonly due to errors" in the kernel logfile. That experience made XFS my default filesystem up to recently when I started to switch to btrfs. (of course, we fixed the hardware-errors, too... :-) )

Also, from that time, I got to use xfsdump/xfsrestore for backups and storage of fs-images which not even once failed on me.

<comment><USER67> As a blithe, new-Linux user (3.5 years), I was bumfuzzled when I saw RHEL/CentOS 7 switched from ext4 to XFS, figuring it to be some young upstart stealing the crown from the king. Then I did some Googling and figured out that XFS is as old as ext _2_! I'm looking forward to discovering how tools like xfs* can make my life easier.

</comment></comment></comment><comment><USER68> Rsync seems a better tool for this. Can be run multiple times and it will just copy missing blocks.

</comment><comment><USER69> it reminds me of crash only software.

</comment><comment><USER70> I would probably have used tar|tar for this, or rsync.

<comment><USER71> You're right to recommend a tarpipe. I've had to copy several very large BackupPC storage pools in the past, and a tarpipe is the most reliable way to do it. (The only downside to BackupPC IMO...)

For future reference for other folks, the command would look something like this:

        cd /old-directory && tar czvflpS - . | tar -C /new-directory -xzvf -

Tarpipes are especially neat because they can work well over ssh (make sure you have ssh configured for passwordless login, any prompt at all will bone the tarpipe):

        cd /old-directory && tar czvflpS - . | ssh -i /path/to/private-key user@host "tar -C /new-directory -xzvf -"

...but tarpipe-over-ssh is not very fast. I have a note that says, "36 hours for 245G over a reasonable network" (probably 100Mb).

Disk-to-disk SATA or SAS without ssh in between would be significantly faster.

<comment><USER72> The prompt goes to stderr, the pipe only pipes stdout, so a prompt should not cause excessive bonage, as long as you're there to respond to it.

Also, don't use -z locally, or even over a moderately fast network. The compression is not that fast and almost always makes things slower.

<comment><USER71> Good to know!

Also, re: bonage, I agree that it "shouldn't", but it definitely did. From my sysadmin notes file:

> The tar operation kicks off before ssh engages; having ssh ask for a > password seems to intermittently cause problems with the tar headers on the > receiving end. (It _shouldn't_, but it seems to.)

</comment></comment></comment></comment><comment><USER73> Is this where a new cp fork comes about called libracp?

</comment><comment><USER74> Feels like a similar situation to this: [http://dis.4chan.org/read/prog/1109211978/21](http://dis.4chan.org/read/prog/1109211978/21)

</comment><comment><USER75> > 20 years experience with various Unix variants

> I browsed the net for other peoples' experience with copying many files and > quickly decided that cp would do the job nicely.

After 20 years you no longer google how to copy files.

Edit: Reading on he talks about strace and even reading cp's source code which makes it even weirder that he had to google how to do this...

Edit2: Comments! Took only ten downvotes before someone bothered to explain what I was doing wrong, but now there are three almost simultaneously. I guess those make a few good points. I'd still think cp ought to handle just about anything especially given its ubiquitousness and age, but I see the point.

And to clarify: I'm not saying the author is stupid or anything. It's just _weird_ to me that someone with that much experience would google something which on the surface sounds so trivial, even at 40TB.

<comment><USER36> Because the man is wise. He also didn't kill a job that appeared to be hung, he started reading the code to figure out why and determined that it would in fact, complete.</comment></comment>


Rent the Backyard (YC S19) wants to build a studio apartment in your yard - <AUTHOR> https://techcrunch.com/2019/07/18/rent-the-backyard/ <comment><USER3> I don't like the idea of taking a free and simple house, which you can sell, and myring it in a contract with another company (no less a startup!) so that when I want to sell, I need to explain the agreement I have with this other party to annex the land. I reckon it would devalue the house.

I think people would be better off borrowing the money and building their own. Where I live (not in US) you might spend say $100k or more building one of these, but without necessarily adding capital value to your house, so you'd break even on your first $100k of rent, after tax. So even building your own may not make sense. The many reason for building on is as a 'granny annex' or a place for your parents to live so you can help them out. Or a place for your kids to live when they move out.

<comment><USER4> Who has time to manage a construction project? Just show me where to sign and send me my monthly checks.

<comment><USER5> Definitely appreciate that many will benefit from farming it out, so I think this company substantially expands the addressable market, but in support of DIY (as someone who was raised in the house my father built), building your own such modest project is very achievable, particularly if you farm out electrical and plumbing concerns. With respect to managing contractors, my understanding is that roughly daily supervision will do, which can be manageable even with full-time employment. You may have better ways of spending your time, but let's not use that as a basis for overly-narrowing our own concerns and capabilities.

<comment><USER3> Why do you have to supervise the work of the trades but not this startup? Why are they more trustworthy? If they are ... then you can probably get someone else who is trustworthy to do the job for straight money rather than owning part of your house.

If this startup did what they did but just arranged finance as a broker and acted as a letting agent I think it'd be a better deal. The issue I have is the splitting ownership and income. I don't think that's in anyone's interest.

<comment><USER5> In one case you're hiring the work yourself, in the other case you're bringing on a partner who is sharing in the rewards. In the latter case the incentives are better aligned.

"Not to oversee workmen is to leave them your purse open." — Poor Richard's Almanac

</comment></comment></comment></comment></comment><comment><USER6> Seattle has an innovative project called the Block project which puts a tiny home into a volunteer's back yard.

Candidates are homeless individuals who have been working with social workers for at least 6 months and are a good fit with the "landlord" which is the homeowner of the hosting site.

The Individual pays minimal rent to cover utilities (hopefully subsidized via the government and facilitated by social workers). The lease is for 5 years with the goal of helping the individual get back on their feet to a traditional housing situation. The project is from the Facing Homelessness organization. They have placed 4 or 5 individuals at this point.

[1] [http://the-block-project.org/home](http://the-block-project.org/home)

[2] [https://www.facinghomelessness.org/index.html](https://www.facinghomelessness.org/index.html)

<comment><USER7> What does the homeowner get out of it?

<comment><USER8> A human being doesn't have to live on the street.

</comment></comment></comment><comment><USER9> Hi everyone, my name is Spencer and I'm one of the co-founders of Rent the Backyard.

Rent the Backyard was founded to help homeowners who want to earn extra income from their underused backyards but don't want to spend a ton of time or money to make it happen. Rent the Backyard provides a turnkey solution — we handle all the logistics of putting a studio apartment in your backyard — helping you earn $10-$20k per year in rental income with as little loss of privacy for you as possible.

We're focused on the Bay Area (particularly San Jose) to start. California has passed a lot of laws in recent years that make these "accessory dwelling units" or ADU easier to build because they believe these units can help play a role in solving the affordable housing crisis. 4,300 ADU were built last year in California and we're hoping to help more homeowners access this new type of housing without needing to borrow money and become a part-time construction manager.

We're building this business not only to help homeowners make more money from their underutilized space but also to help communities increase their supply of housing. We want to be sensitive to the communities we serve and as we build, we're committed to listening to stakeholders and exceeding California's requirements when it's the right thing to do.

Excited to answer your questions here and by email: firstname.lastname@example.com !

<comment> martin-adams I'm from the UK so probably have limited understanding of the US housing regulations. But after reading your FAQ, I was left with the following questions. While I don't need to know the answers, they give an insight in to what went through my mind.

1\. What happens if there are no tenants?

2\. What laws/insurance/responsibility is now applicable as being the landlord?

3\. How do you calculate the price of the house in a sale?

4\. What happens if I go bankrupt?

5\. What if I have a mortgage on my current property?

6\. Who is technically the landlord, me or Rent the Backyard?

7\. What about electricity, plumbing and sewage?

8\. What about breakages to the property and general maintenance and upkeep?

9\. Are they furnished or not, and if so, who furnishes it?

10\. What are your fees to pay for the service and bills to make the difference between earnings and cost to the tennant?

</comment></comment><comment><USER10> I love anything that creates more housing in the bay, but these are 30-year contracts! I can't even imagine getting into a 30 year contract with a company that might not exist in 30 months. What happens when the start-up goes under, or gets acquired, etc? Is the loan made directly to a third party bank?

<comment><USER11> I think of the long-term solar contracts that make it hard to sell a house.

<comment><USER12> this.

anything that restricts or impedes what you can do with a home is a liability, not an asset. Up in SF, even partially tenant-occupied homes typically sell for a sizable discount.

(also doubt that ~10k/year is going to make waves in the target market -- Bay Area homeowners with a back yard big enough to make this viable.)

<comment><USER13> It could be very meaningful to older retired or semi-retired folks who have fully paid for their home and are looking for additional income but don't want to sell their house and leave their neighborhood. AirBnB used folks in that situation here in Vancouver to great effect when pushing for support for short term rentals in the local market.

<comment><USER9> A lot of people we've talked to have been in this exact situation. Older folks who have lived in their home for a long time and watched the cost of living rise while remaining on a fixed income from something like Social Security. Our hope with Rent the Backyard is to help people like this remain in their homes and have a higher quality of life.

</comment></comment></comment></comment></comment><comment><USER14> People who can afford their home will not like watching their property values and quality of life decline because a neighbor "rented" its back yard.

Taking the Uber approach - ignoring laws - won't work here, so this business is entirely at the mercy of permit laws. I find it astounding that some ideas get prominent backing.

Furthermore, home owners will discover that having people travel alongside their homes to reach the rented studios will be unpleasant and unsafe.

Will the tenant be allowed to come and go at any hour? Will the home owner know that the person creeping alongside their bedroom window at night is the tenant rather than a burglar?

Will the tenant be allowed to park in the homeowner's driveway, or will they line the front curbs of the house with their car and any visitors' cars?

<comment><USER9> Hi there - my name is Spencer and I'm one of the co-founders of Rent the Backyard.

We're very much focused not taking the Uber approach. We're working within the context of a set of laws California has passed over the last several years that establish and regulate what the state calls "accessory dwelling units" aka ADU. We're initially focused on building in cities like San Jose that have factored the sustainable and conscious development of ADU into their thoughts on how to combat the housing crisis they face.

4300 ADU were permitted and built last year in California with hundreds of these units being built in the Bay Area ([https://www.sfchronicle.com/bayarea/article/Are-accessory- dw...](https://www.sfchronicle.com/bayarea/article/Are-accessory-dwelling- units-an-answer-to-Bay-13776416.php)).

We're really conscious of the impact we'll have on the neighborhoods we operate in and are committed to working with everyone involved to make the process of installation and ongoing housing have a minimal impact. Our studio apartments are almost completely built when they arrive in your backyard and require very minimal (a few weeks if not days) to bolt together and be ready to go. We're excited to see California have strong laws to protect neighbor's privacy and rights as well and will go beyond what is required by those laws when it's the right thing to do.

<comment><USER14> Hi Spencer. I applaud you for trying to formalize and refine that ADU process (for the goal of helping people find affordable housing).

The California market might be suitable for this, but that market is unique in many ways. I wonder how this can scale beyond CA (perhaps that market is big enough...).

Given the alternatives to unaffordable housing, I see the appeal. But as other commenters have noted, this need exists because of a very warped localized economic situation. The real problem may not be solved, but eventually there will be some shake out (such as vast numbers of people deciding it's not worth living there) that will help to normalize things. When that day comes, your business model will be less relevant.

Cheers and good luck.

</comment></comment></comment><comment><USER15> >In places like the Bay Area

haha was looking for that. My first thought on reading the headline was London. Shed renting is actually a thing there. Next though...SF

But yeah - I totally think this has legs. I live in a smaller more modest space than I could afford. Works for me.

<comment><USER16> yes for slum landlords which is all very well until you get busted by the authorities.

<comment><USER17> I think England is notorious for slum lords and that tenants have very little legal protections.

<comment><USER18> London is notorious for this packing 3 or 4 guys into a ramshackle unsanitary garden shed.

</comment></comment></comment></comment><comment><USER19> I approve this startup.

This is in line with the idea of Missing Middle housing. We used to have more rooms for rent, SROs, small houses etc. Then we largely zoned and regulated them out of existence. The ones we still have are often very old and grandfathered in.

We desperately need to solve the housing supply problem. Government not only can't solve it, it really shouldn't try. It should be looking to deregulate some and get out of the way.

I've studied this problem space for a lot of years. Government interference is part of the problem.

I don't want a nanny state that tries to provide for all your needs. I would like to see single payer healthcare, but for the housing sector, I would like to see government work on being less controlling and less interfering and let the private sector have maneuvering room to handle this.

I wish this company all the best and much success.

</comment><comment><USER20> The price (50% plus an ownership stake in a _structure on my property_ ) seems absolutely insane. Does the ownership stake mean that I can't _stop_ leasing the studio if I want to?

I think most people who really want to do this will work with a local contractor instead.

<comment><USER21> You can stop leasing whenever you want to, buy the company that spent a five digit sum of money on that thing in your backyard will want to get then money back first, which is unsurprising.

<comment><USER20> That doesn't surprise me, but the way a bank would do it is this: 1) loan you the capital to build the structure, 2) create a payment schedule, 3) try to collect on that payment if you miss it.

You could then declare bankruptcy if necessary, which means you'd keep the structure and have some sort of way out from under the debt.

If Rent the Backyard has an ownership stake in the structure, it sounds like they could continue leasing the unit even after you declared bankruptcy.

<comment><USER9> People definitely use traditional home equity loans to build ADUs but this can be quite risky. There are more full-stack providers (like one of our building partners [https://node.eco/](https://node.eco/) who's headquartered in Washington State) which make the process a bit less nerve-wracking, but it's still a lot of time, effort, and risk.

</comment></comment></comment></comment><comment><USER22> I don't have a backyard, and nobody I know in my area that also makes six figures+ a year has one, either. This trend has been documented [0]. Granted, my cohort is all earlier in our careers. But with years of saving, much of it prioritized in 401k's, IRAs, and HSAs, I don't have enough left for a down payment on a single family residence.

[0]: [https://www.washingtonpost.com/business/2019/02/21/are- you-c...](https://www.washingtonpost.com/business/2019/02/21/are-you-city- with-growing-number-high-income-home- renters/?noredirect=on&utm_term=.52d9b83f72bb)

<comment><USER23> Same here, and if I were able to afford a downpayment I would live in daily fear of an earthquake because I wouldn't be able to afford Earthquake insurance, and if I did I wouldn't have the 20% deductible in the bank. I'm convinced that most non-wealthy bay area homeowners are essentially betting their financial future on there being no earthquakes here until the house is greater than 50% paid off. Tenuous is a fitting word I think.

<comment><USER24> California building codes have been pretty strong for many years. Only older residences are likely to suffer catastrophic structural damage when "The Big One" hits.

Earthquake insurance is a terrible deal. Instead of paying insurance premiums, put the money toward earthquake safety retrofitting and saving an emergency fund. There will probably also be some state and federal disaster zone financial assistance.

<comment><USER22> Fortunately, most of the property cost when you purchase in California is for the land, not the structure that sits on it.

</comment></comment></comment></comment><comment><USER25> What prevents a homeowner to go directly to: [https://node.eco/](https://node.eco/) or [https://livemodal.com/](https://livemodal.com/) and do it themselves?

<comment><USER26> Absolutely nothing. There are two things that we provide that going directly to a builder can't get you: 1) No upfront costs. We pay for the entire unit without asking our homeowners to put any money down. 2) Turnkey solution. We do all of the building and rental management for the homeowner, allowing them to sit back and see rental income coming in.

</comment></comment><comment><USER27> This can be a huge success in South Africa. Backrooms are very popular in big townships and provide residence to more than half of townships' population. The good thing is that the laws already allow such kind of dwellings and you won't have to deal with NIMBYs. Best of luck.

</comment><comment><USER28> It seems like the biggest innovation here would be bringing down the price of building a standalone studio apartment. These units probably cost cost $120k+ to build/install?

It would be neat if there was a (high quality) $50k kit homeowners could buy and assemble themselves, almost like IKEA furniture.

<comment><USER29> _It would be neat if there was a (high quality) $50k kit homeowners could buy and assemble themselves, almost like IKEA furniture._

There was a New England company that sold complete kits for small houses with pier and beam foundations, designed to be constructed by two fit adults, including the fasteners! I think they went out of business in the 2000's.

EDIT: I just found another company doing almost the same thing: [https://www.shelterkit.com/gallery.html](https://www.shelterkit.com/gallery.html)

<comment><USER28> All the companies doing this seem to be small/unsuccessful/brand new. Maybe there's some fundamental problem with the business or maybe no one has nailed it yet?

<comment><USER29> What if an Amazon company started putting together such kits? Perhaps there would be a way this could be done for small margins, sort of as a no-loss- leader to generate more Amazon customers?

</comment></comment></comment></comment><comment><USER30> I see a few issues. First, the local zoning commission may not allow such units.

Second, my dog uses the backyard on my dog's schedule, not the renter's. And the renter may dislike walking through a minefield to come and go.

Third, where's the renter going to park?

<comment><USER9> Hi there - my name is Spencer and I'm one of the co-founders of Rent the Backyard.

You're totally right about possible zoning restrictions in most places. The climate in California has really changed in the last two years to allow what the state calls "accessory dwelling units" or ADU ([https://www.sfchronicle.com/bayarea/article/Are-accessory- dw...](https://www.sfchronicle.com/bayarea/article/Are-accessory-dwelling- units-an-answer-to-Bay-13776416.php)). These laws cover everything from parking, to setbacks, to privacy requirements. We'll go beyond what is required by those laws when it's the right thing to do.

These new laws have led to an explosion of building. 4300 of these units were permitted in California last year with 187 in the City of San Jose -- our main focus to start.

We love dogs too but this is definitely an issue. Ultimately, this is a product you'll have to decide works or doesn't work for you :)

<comment><USER30> Well, I think that the issue will be whether the dog works for the _tenant_.

Hypothetical situation: I don't have a dog. I build one of these. A tenant signs a lease. Then I get a dog. But the tenant is terrified of dogs. But they've signed a lease, and (because nobody thought it would be an issue) there's nothing in the lease about whether the owner can have a dog. Now the tenant is really stuck - essentially they are at the owner's mercy.

I don't know if your standard agreement covers this situation, but if not, it's something to think about...

<comment><USER31> In my country (not US) it is by law allowed for either side of the lease to terminate it under extraordinary circumstances.

This situation would certainly fall under that. Obviously, if the other party objects to the termination, you'd need to get some sort of medical verification for your condition.

</comment></comment></comment></comment><comment><USER32> Can someone more familiar with the Bay Area housing issues and this company's approach to increasing housing explain or analyze this for a non-Bay Area audience? I ask because this currently is Bay Area only.

<comment><USER33> I will try.

In the US, there is the concept of Accessory Dwelling Unit - essentially, you can build a small shed in your backyard with a much simpler process, including the paperwork, permit, etc.

If you are a homeowner, you can do that yourself - it means taking care of the permits, finding a construction company, etc, and putting money down, probably in the ballpark of 100-150k to do a good job.

After this, you can rent out your "apartment" / unit in your backyard. Or you can use it for your own purposes.

I would guess it's a fairly good investment, with returns of money invested in the 6-8 year range, at least in the current economic climate of the Bay area.

The "loan" part is tricky: any financial institution would want to put a "lien" on the property (essentially, you can't change ownership or sell the property until you remove the lien, which means until you have repaid the loan).

If you do this with "Rent the Backyard", or by yourself, it means your house can't be sold until you pay the loan.

It also means that potentially not all homeowners will be able to obtain a loan from a financial institution.

I examined this business model in depth in the past (while founder at fabrica.city), and decided it was not interesting. But I hope these guys prove me wrong and succeed.

<comment><USER29> _If you do this with "Rent the Backyard", or by yourself, it means your house can't be sold until you pay the loan._

You probably thought of this already, but I have to ask: What if "Rent the Backyard" is the lender? Couldn't they have a provision in the loan contract allowing the homeowner to move out? What if _Rent the Backyard_ forms an entity with the homeowner, and sells the lot to the entity?

<comment><USER9> Hi there - my name is Spencer and I'm one of the co-founders of Rent the Backyard.

It's tricky (years and $$$) to become a mortgage lender but we have flexibility in our agreement for a homeowner to pass the agreement to a new owner (we'll even move our lien back to the second position so the new homeowner can get a mortgage)

</comment></comment></comment></comment><comment><USER34> In the bay, rent is so high this makes no sense. Why would I give away 50% of rent for something that can pay off my investment in one to two years? Even with the headaches 50% is not worth it. 20% maybe.

<comment><USER9> Unfortunately, a fully-featured unit is really expensive. For a unit like the ones we build (full bathroom with shower, kitchenette, on a full foundation) you'd be hard-pressed to build a unit for less than $100k. This number assumes you're spending a whole lot of your time working on the project. People are building these units and we're excited to see the stock of housing rise in the Bay Area but most people won't make the jump to borrow the money on a home equity line of credit and become a part-time construction manager.

<comment><USER35> Ok.

Can we have a (minimal as you want) financial plan?

I mean, let's say that the building (by a professional contractor, with architect/engineers/whomever professional assistance, with all permits, etc., i.e. "turnkey") is going to cost to a "normal" person US$ 120,000 (or if you prefer 20% more than your $100K (which sounds to me appropriate, probably something in the 50-60 sqm or 500-600 sqm net surface).

Example of such an ADU:

[https://crossconstruction.com/wp- content/uploads/2019/03/Des...](https://crossconstruction.com/wp- content/uploads/2019/03/Design-Path_1bd-1ba.pdf)

How much is the expected income (rent - taxes) per year? US$ 20,000 per year or 1,600 US$/month?

How much is the rate of the mortgage if you "simply" borrow the money from a bank? (let's say a minimal down payment of 10,000 and thus a loan of 110,000)

It should be (roughly) anything between 500 and 600 US$/month for a 30 years mortgage (fixed rate), let's make it 7,000 US$/year.

Besides the downpaynent of 10,000, at fixed prices (which is likely NOT the trend for rents in an area where people resorts to living in ADU's) you get 1,600 US$/month, spending 600, i.e. a net of 1,000 per month, so first year you get nothing (to compensate the down payment) and for the remaining 29 years it makes US$ 377,000 (29x(20,000-7,000).

With your formula, it is 30x10,000=300,000.

But - IF the rents ar going to increase in these 30 years - the difference will soon become _huge_.

</comment></comment></comment><comment><USER36> If there is demand for this, more power to them. However, I feel existential dread considering that demand for this exists (and is potentially growing).

</comment><comment><USER37> Do NOT do this in SF! ANY building, including illegal in-law units and backyard buildings (ANYTHING occupied) is recognized by the city. A single- family home with any other unit is seen by the city as a multiple-unit building, and the backyard renter would have eviction protection. (In SF, "eviction protection" means leases never expire, only shift month to month). Do you want someone to move into a random shack in your yard and then have legal rights to remain there indefinitely?

This company is a nightmare waiting to happen.

</comment><comment><USER38> It's a cool idea, for sure. Both for creating affordable housing, and for increasing the affordability of existing housing.

But as others note, there could be major complications at sale. Especially if the apartment doesn't increase the market value enough to cover the buy-out cost.

Wouldn't it be more straightforward to just build pads and hookups for trailers? Or even for RVs? That'd be much easier to untangle at sale. And it'd probably cost less.

Perhaps the startup could convert shipping containers to apartments. I've read about German ~gypsies that do that.

</comment><comment><USER39> This has been happening lately in my area of Orange County, CA. There is an investor here who has been buying up single family residences with large backyards and then the investor puts these "granny flats" in the backyard to rent them out. The average buyer will simply see the property as a 3bed/2bath with a large backyard for 500k while an investor will see it as 3bed/2bath with a large backyard with opportunity for expansion....well worth more than 500k listed market value.

</comment><comment><USER40> A couple of things to consider for the homeowner, increases in property taxes and insurance but also the taxable revenue from the rental and its impact on personal taxes you will have to pay. And then there is resale value for a property that is locked into multi year agreement.

<comment><USER41> >increases in property taxes

Not an issue in the bay area

>the taxable revenue from the rental and its impact on personal taxes

Progressive income taxation should make this moot. Also many expenses are likely tax-deductible. Combined with FHA terms, mortgage interest deduction, etc. and this is actually rather tax-advantaged over regular rentals

<comment><USER42> _Not an issue in the bay area_

Why not?

Prop 13 protects the homeowners from higher taxes if the property appreciates, but as far as I know that doesn't cover renovations.

Otherwise you could buy a beat up $1M, year it down and still pay the same taxes.

<comment><USER9> The way this works with ADUs is a blended approach. The existing home's tax status is protected by Prop 13 and the ADU as a structure is appraised. While adding an ADU adds considerable resale value to a home, tax assessors don't typically value them that highly. It's also worth noting the ADU also becomes protected under Prop 13 after the initial assessment ([https://maxablespace.com/granny-flat-tax- implications/](https://maxablespace.com/granny-flat-tax-implications/))

</comment></comment></comment></comment><comment><USER43> Yes in my back yard!

<comment><USER9> We thought about calling the company this :)

</comment></comment><comment><USER44> Shipping container offices are all the rage these days. Have you considered using them? I mean, if they're ok to work out of - legally and safely - <USER2> why not live in one? It would certainly cut down on the construction costs.

<comment><USER24> Shipping container offices aren't popular at all. And they aren't suitable for housing either. There is no savings.

<comment><USER9> We did take a pretty deep look into shipping containers. There are a number of sites that will sell you a pretty fully featured container ([https://www.curbed.com/2017/6/21/15839730/shipping- container...](https://www.curbed.com/2017/6/21/15839730/shipping-container- house-for-sale-buy))

While many of them look pretty reasonable, longevity and sense of value are the main issues people have psychologically. Shipping containers often elicit worry about rust and decay which is not unreasonable by my understanding is that they have coatings and paint that can largely prevent this.

The bigger issue is the homeowner's sense of value. While any ADU should increase the value of the property it's on, a shipping container home may make the home a bit harder to sell.

Shipping container homes often come with a very technical installation. Since the entire unit is built, if access into the backyard is limited the unit needs to be lifted over the home (and all the power lines) by a massive and very expensive crane. An easier version of this installation by crane: [https://www.youtube.com/watch?v=KDTLBbfKL3E](https://www.youtube.com/watch?v=KDTLBbfKL3E)

As we've thought about these problems, we've honed in on building beautiful largely prefabricated flat-packed units that are easy to get into the backyard and then bolt together kind of like Ikea furniture.

</comment></comment></comment><comment><USER45> Do it, and innovate real estate investing too. Fix those crappy contracts and low liquidity markets. Make it easier to move capital into these markets. These units could solve so many problems

</comment><comment><USER46> this should do wonders for the housing crisis, outstanding work guys.

edited: its so fitting that on the front page right now there at least two other stories covering real estate and the housing crisis

<comment><USER9> Growing up on the East Coast, I was confused when I first visited the Valley and learned about how expensive housing is. People that don't work for big tech companies can't afford to live here anymore and even workers just starting out at Google are spending >30% of their income on housing. I'm afraid of what that means.

<comment><USER46> I was actually being sarcastic, I think this is a really bad idea. Sorry man, I see that you're one of the cofounders, I just don't believe this is will help the situation at all. Fwiw I lived for several months in SF, in the tenderloin, stacked in a room usually with at least four, sometimes five other people.

</comment></comment></comment><comment><USER47> © Clarus the dogcow

</comment><comment><USER48> If this isn't reflective of an abject failure of local/state government and public policy to plan and manage the needs of housing and population growth, I don't know what is.

Tech companies now employ people, feed people, transport people, educate people, and now they house people. Pretty soon we'll be voting for which tech company governs our society.

All the while gutless local city council members afraid of losing their jobs cower before homeowners, the landed gentry, the rudderless symbolic issues of anti-gentrification and anti-neighborhood change, at the expense of everyone who had the bad luck of being born later and not having gotten theirs yet.

<comment><USER29> _If this isn 't reflective of an abject failure of local/state government and public policy to plan and manage the needs of housing and population growth, I don't know what is._

It's not a failure of local/state government. It's an encompassing failure of society at large. We keep passing NIMBY laws. Society becomes ossified. There's no longer anywhere for the next generation to settle down.

 _Tech companies now feed people, transport people, educate people, house people. Pretty soon we 'll be voting for which tech company governs our society._

Aren't we already voting with our dollars? In a way, it's like our society is becoming the one described in _Snow Crash,_ just with Silicon Valley aesthetics, as opposed to the cyberpunk take on 1990's strip malls. The US government is losing credibility. Everything is privatized and up for sale or lease. Want a better jail cell? Offer the jail franchise your credit card for an upgrade!

 _" What's it gonna be, the Hoosegow or The Clink?" the first MetaCop says. From the way he is talking, he must be talking to the other MetaCop._

 _" The Hoosegow, please," Y.T. says._

...

 _The Hoosegow looks like a nice new one. Y.T. has seen hotels that were worse places to sleep. Its logo sign, a saguaro cactus with a black cowboy hat resting on top of it at a jaunty angle, is brand-new and clean._

                       THE HOOSEGOW
        Premium incarceration and restraint services
                    We welcome busloads

</comment></comment><comment><USER49> When they inevitably go bankrupt who ends up owning the unit? Will it be collateralized and sold off or revert to the original property owner?

<comment><USER26> Hi there -- this is Brian from Rent the Backyard.

We don't see bankruptcy as quite inevitable as you do :) In all seriousness, as written somewhere else on this thread:

"We will need to eventually give custodial ownership to a larger entity (think big bank) in the medium term to mitigate the risk of our company going under for the homeowner, such that, if the worst-case scenario happened to our company, the homeowner continues to make rental income as usual."

<comment><USER50> Sorry, I'm unclear what this means. Will the bank be taking over the relationship in place of Rent the Backyard? Will they buy the stake in return for 50% of rental income?

<comment><USER9> The idea is that there's a third party who formally holds the contract so there's minimal disruption for the homeowner if something happens to our company.

<comment><USER51> I think it is a great idea and I see a lot of potential. If any of you have tried any addition/remodeling in the bay area, you know the difficulties involved. It is hard to find reliable people at reasonable cost. This company is bringing scale advantage and that is the only way to make sense of any construction project in the bay area. A friend of mine checked with a few design-build firm for second floor addition. 400$ per square feet is the minimum price. Even to explore the project there is a huge cost upfront without any assurance of success. I think it is a great way for homeowners to get cashflow with minimal risk.

Question to the founder: Do you see profitability in building or rental or both?

It would be leaving money on the table for the home owner to continue on the contract for 30years. Initially with 0 equity, the owner gets 50% of rent. at year 20, with 50% equity, the owner gets 50% of rent and at year 25 with say 70% equity, the owner still gets 50% rent. From home owners point of view, it is better to buy out sooner rather than later. It is also not hard to get home equity loans. Thanks and all the best

</comment></comment></comment></comment></comment><comment><USER52> So this is Uber but for backyards?

Any moment now someone is going to come up with Uber, but for toilets. I can see it: it'll be called something like Petey Potty, the app will let you connect with Potty Providers and you'll get to use their toilet for a fee. The company will have an ostensible plan of making money by taking a cut off the Potty User fee, and charging a standard fee to Potty Providers. It will be wildly popular in dense urban areas and business districts where it's hard to find a public toilet and around kebab places and it will be hailed as a lifesaver for individuals with Krohn's disease who need to be able to access a toilet with very little warning. There will be an article in Wired and an outraged article in The Guardian (there will be outrage, no questions). The company will lose money for the first ten years of its existence, then it will be sold to Uber, probably, or possibly Yelp.

Oh and- there will be a review system on the app. With five little smiling turds rather than stars.

<comment><USER53> This is the comment you want to make on a board dedicated to startups?

If a startup can gain power and then lobby to change zoning laws, even if they do it in a self serving manner that locks out competition, it would be such a vast improvement it is hard to calculate.

The deadweight loss of zoning is huge.

This is a far more ambitious idea than Uber. I imagine it will fail as the majority of voters own and they vote to immiserate others so their property values remain inflated.

Your comment is gross. The sentiments within it are nothing but regressive, mocking sneers.

</comment></comment><comment><USER54> >builds studio apartments in homeowners' backyards, which are then rented out

That might be one of the most dystopian things I've ever seen on HN.

<comment><USER26> Hi there -- this is Brian from Rent the Backyard. What we are doing here is finding a synergy between homeowners and renters, allowing for people to profit off their unused backyards while increasing the housing stock. Airbnb started the trend of being able to monetize your home in the form of rental income and we see ourselves as an extension of that phenomenon. In our case, homeowners give up zero privacy to make meaningful extra income. We help existing homeowners stay in their homes amidst rising costs in the Bay Area.

<comment><USER54> Oh I don't mean to say it doesn't make sense, both in terms of personal finance and with regards to the housing market. Then again, so does letting people sell their own kidneys and livers for transplant.

>Rent the Backyard will give the homeowner an increasing share of equity in the apartment, until they own it completely after 30 years

What's the life expectancy of these backyard studio apartments? Somewhere around, oh, thirty years?

>Homeowners also can buy out the startup's equity and take full ownership at any time (which they'll need to do if they sell their home and move out).

I'm assuming there's some legal way to transfer the entire contract to another person, so you could sell your house and property along with your stake in the a-yard-ment and the new owner takes over where you left off. Having a renter will lessen the likelihood of having to sell for financial reasons but a good chunk of home sales are prompted by owner becoming unable to keep up the house or live alone in general, due to age or illness.

How's the agreement work with other stuff like use of the yard itself and other stuff that might be concerning like loud music, changes to the fencing setup, planting of flowers the renter is allergic to, etc? Not just for the homeowner, but for the renter. I can imagine many of the owners, having had to put very little work into this, really not giving a crap about their renter or her/his needs.

>What we are doing here is finding a synergy

C'mon man. Your picture doesn't feature nearly enough 80s power tie for you to talk like that.

<comment><USER9> We build units that are just as nice as your existing home. With proper upkeep, they'll last just as long too.

This was something the article didn't quite get right. You're able to pass this agreement on to a next of kin or new buyer. We'll even lower the priority of our lien to allow a mortgage to be issued.

You're right that one of our hopes is to enable people to stay in their homes amidst the rising cost of living. The income from this unit can be used to hire a caretaker or even enable the homeowner to downsize in place -- moving into the ADU and renting out the main home. In this scenario, the homeowner would just need to pay us half the market rent.

Finding a balance between renter and homeowner is a big concern of ours. The scale of the difficulty depends a lot of times on how much space there is on the lot. Most of the time we'll build as far from the main home as possible to help preserve privacy and independence but there's still a balance to find. We have an expectation for homeowners and renters alike to be reasonable and we don't want to need to enforce that with a contract but setting clear expectations and methods of recourse are important.

I don't think I've ever seen Brian wear a tie ;)</comment></comment></comment></comment>


Might Bitcoin's pros also be cons? - <AUTHOR> http://www.bbc.co.uk/news/magazine-22292708

<comment><AUTHOR> If you look at [http://bitcoincharts.com/charts/mtgoxUSD#rg180ztgSzm1g10zm2g...](http://bitcoincharts.com/charts/mtgoxUSD#rg180ztgSzm1g10zm2g25) there certainly seems that a case can be made that the Cyprus fiasco had a huge effect on the value of Bitcoins.</comment>


Later-stage rounds and "setting the bar too high" - <AUTHOR> http://cdixon.org/2011/12/13/later-stage-rounds-and-setting-the-bar-too-high/

<comment><USER2> The key point in the article lies towards the end:

 _An important point to keep in mind is that, in order to maintain flexibility, entrepreneurs shouldn't give new investors the ability to block an exit or new financings. Investors can get this block in one of two ways – explicit blocking rights (under the "control provisions" section of a VC term sheet) or by controlling the board of directors. These are negotiable terms and startups with momentum should be very careful about giving them away._

There's the rub.

Founder control of the board is hard but possible.

I really can't imagine a VC fund being happy about an exit below the valuation at their round, especially when they have deployed a big chunk of their fund via a big round.

Any experienced VCs/founders want to comment on how common it is for startups to be able to negotiate a term sheet where VCs have no explicit blocking rights?

For reference, YC's AA round template documents for _angel investments_ , widely considered to be very founder friendly, include the following:

 _So long as any of the Preferred is outstanding, consent of the holders of at least 50% of the Preferred will be required for any action that: (i) alters any provision of the certificate of incorporation if it would adversely alter the rights, preferences, privileges or powers of the Preferred; (ii) changes the authorized number of shares of Preferred; or (iii) approves any merger, sale of assets or other corporate reorganization or acquisition._

</comment><comment><USER3> Sure, the VCs know how to take care of themselves, but what about the risk of putting your employees underwater?

<comment><USER4> That's every financed startup, regardless of whether the company gets a spectacular valuation or not.

This point feels pedantic (sorry), but it's worth pointing out that underwater employee options aren't some weird corner case. It's the common case.

<comment><USER3> Startups are risky to begin with, but raising a big round shifts the risk from founders to employees.

Based on the emails I get from recruiters bragging about how much money their companies have raised, it's clear that most engineers don't understand this, even if it's obvious to founders.

<comment><USER4> I'm not even disagreeing with you, so much as I'm saying "it is probably a common misunderstanding that less capitalized startups have lower 'employee risk'†".

No; the reality is that the "employee risk" is complex and has as much to do with revenue growth and market conditions as with capitalization. Some firms capitalize on spectacular growth, and some huge capitalizations also create strategic leverage for the whole company that improves outcomes for employees.

But in _every_ case, equity in a financed startup is freighted with enormous risk. The fundamental issue is simple: corporations are managed by boards of directors chartered with looking after the good of the shareholders, but, in _virtually every financed startup in the world_ , the actual board puts employee outcomes low on the list, invariably below investor outcomes.

This is actually as it should be. You could no more demand a more "protective" board for employees as you could a 50% increase in valuations; market conditions and pricing are what controls board makeup.

One of the few gratifying memes emerging on HN over the last two years (just after "use bcrypt") is the notion that startup employee options are very risky. I'm writing noisily to combat any implication on this thread that you can mitigate that risk by picking undercapitalized startups to work for. If you want to avoid the conflicting interest risk with startup boards, avoid financed startups.

Or, don't; financed startups are also the ones that (rarely, very rarely, but still) return huge rewards.

† _Imprecise, but let's have it stand for "risk that employee stock options are worth zero"._

<comment><USER3> The most likely exit for today's startups is a talent acquisition. In that event, would you rather be at an "undercapitalized" startup or an "overcapitalized" startup?

<comment><USER4> What's the difference? The "valuation" of a talent acquisition is about retaining talent. No matter what your shares are worth on paper, your outcome is going to be the same; if the paper itself doesn't say so, the retention grant will.

Why would a company do a "talent acquisition" at all if the best team members were immediately going to leave for a better upside elsewhere?

Similarly, if the outcome for the company is "talent acquisition", the outcome for an employee who leaves before that exit is probably zero.

</comment></comment></comment></comment></comment></comment><comment><USER5> That DropBox example is not typical though. What about the case when investors want to invest a couple of million, and make the valuation 10-20 million. Wouldn't that rule out a potential acquirer who wants to acquire it for $5 million after a couple of years? Because the investors want a bigger return? Investors are looking for home-run. You, as a normal Joe are happy with a base-hit and some fu money.

<comment><USER4> Uh, yes?

As a rule of thumb: you can't take a VC round expecting to exit for $5MM. If you're looking to exit at $5MM ( _which is totally reasonable, in fact more reasonable than shooting the moon_ ), you're looking to bootstrap.

</comment></comment><comment><USER6> Why do founders and VCs get different stocks? I just do not get this logic. I am fine with you getting your money out first, however if that is the case, you shouldn't have any sort of vote on the outcome of the company. You should just be a silent partner.

<comment><USER4> Why is one startup worth $10MM and the other $20MM?

Because prices, like terms, are set by the market.

Is there any fundamental reason a VC gets participating preferred stock? No! By all means, when you fund your next company, refuse that term.

Obviously, every VC you talk to may then walk away. But maybe not.

The question of "why do VCs demand this term" is closely related to your question (and maybe what you intended). The answer appears to be "to protect from the risk of putting $5MM into a company and having an outcome that is lucrative for the founders but loses money for the VCs". The idea is to align the interests of the company operators and the investors, by taking the money the VCs themselves put into the company out of the equation for the outcome.</comment></comment>


Researchers Find Oddities in High-Profile Gender Studies - <AUTHOR> https://arstechnica.com/science/2017/11/researchers-find-oddities-in-high-profile-gender-studies/ <comment><USER2> Another pseudo-scientific sociology authority that fakes papers to keep his fame. No longer surprised.</comment>


Is Google Censoring The Word "Evil"? - <AUTHOR>

A search for "Google is evil" on Google finds an estimated 2,300 results. However, the same search on AltaVista finds 158,000 results. The same search on Ask.Com finds 7,300 results. However, if Google supposedly covers a larger proportion of the web then Google should find the most results. Could the difference be explained by differences in spidering or estimating the size of the results? Or is Google using the code which is used to censor Chinese queries? <comment><USER2> Google now finds 37,000 results for the phrase. So either they're 16 times as evil since January, you caught them on a bad index day, or they read your post and stopped filtering as you caught them out.

As an Adwords user I was sent a survey by Google a few weeks ago and one of the questions asked me to rate how evil I thought they were.</comment>


Untapt Hiring Guide - <AUTHOR> https://github.com/untapt/hiring-guide <comment><AUTHOR> HN has recently seen a surge in conversation about hiring best practices. The company that I work for (untapt) thought it would be helpful to create a document to help hiring managers improve their own hiring process.

We would love feedback from people on each side of the hiring table. What makes a hiring process successful in your eyes?

If there is broad interest, this document may turn into a more substantial GitBook.</comment>


Tools for rapid prototyping: a webapp in 1 week - <AUTHOR> http://durdn.com/blog/2008/11/09/tools-for-rapid-prototyping-a-webapp-in-1-week/

<comment><USER2> Just a list of frameworks. Yeah, I know jQuery is rad, but I was hoping to see some actual advice on the prototyping process, as opposed to the coding process.

<comment><AUTHOR> I am planning to do that. I just posted a quick background story and overview as soon as I had done with the marathon. More to come.

</comment></comment><comment><USER3> For the rails kids, we have Bort (<http://github.com/fudgestudios/bort/tree/master>)

<comment><USER4> for the merb grown-ups, merb 1.0 now has a well factored authentication framework. The docs/howtos are still a little rough, but I've just gone through migrating my existing app which had mature hand-rolled User model, session management, and registration process. With the new merb auth code, I've just dropped quite a bit of code.

also +1 for Blueprint CSS as well. This finally got me out of CSS hell.

</comment></comment><comment><USER5> Next step: use some simple data mining to recommend other movies I might like, and link to a site where I can watch/buy.

<comment><AUTHOR> Yes! Exactly what is at the top of my list.

<comment><USER6> There are a lot items at the top of your list!

</comment></comment></comment><comment><USER7> I really shouldn't nitpick-- particularly on something thrown together in a week by one person-- but I can't resist: I bet you'd double the fraction of people looking at the site who actually build a list if it lets you start building a list before signing up. After I've built one you can remind me that I should sign up if I want to keep it.

Its much better to let people get their feet wet with out jumping through the hoop of a sign-up.

<comment><AUTHOR> Ah totally cool suggestion, thanks. And easily doable with anonymous sessions. Will note it down.

</comment></comment><comment><USER8> I agree that jQuery is a perfect tool for JavaScript programming. And I loved JavaScript after I learned this simple and powerful library.

</comment><comment><USER9> As cool and useful as Pinax is, calling it "well documented" isn't _entirely_ accurate considering the fact that there is barely any documentation to speak of.

<comment><AUTHOR> Yep, that was a bit overenthusiastic of me. But I noticed they are improving the documentation fast. It is a very young project after all.

Maybe I could rephrase "well documented" in: given the clarity and the good structure of the code I did not have problems finding my way around.

<comment><USER10> I agree. It's not that it's extensively documented; it's that the code is easy enough to hack to one's needs ;)

</comment></comment></comment><comment><USER11> What's everyones take on using something like Pinax where a lot of things are done and bundled up for you?

I've never been a huge fan of this approach. I like Django because it removes some of the annoying parts of creating a web app, but I don't like when the user registration, commenting, profiles, avatars, and more site-specific functionality like "groups/message boards" are pre-built for me

<comment><AUTHOR> Recently pinax added a basic_project in addition to a complete_project.

In the basic project fewer things are provided, like registration with openid support and email validation. In addition there's a structure to create your menus and submenus.

So far I found it a very good starting point for my development and not obtrusive at all.

</comment></comment><comment><USER12> For prototyping we just starting using Axure. <http://axure.com/>

Adobe has a Thermo but it's still in labs. <http://labs.adobe.com/wiki/index.php/Thermo>

</comment><comment><USER13> Stories like this suggest that the probability of finding someone in India to do the coding of a complex web app will soon be under $1000.

</comment><comment><USER14> Durdn: just so you know, diffract.me has a typo:

At present we are not ready for public _consuption_ , early early pre-alpha</comment>