The State of ZFS on Linux - <AUTHOR> https://clusterhq.com/blog/state-zfs-on-linux/ <comment><USER2> At a previous job, we built a proof-of-concept Sinatra service (i.e., HTTP/RESTful service) that would, on a certain API call, clone from a specified snapshot, and also create an iscsi target to that new clone. This was on OpenIndiana initially, then some other variant of that OS as a second attempt.

The client making the HTTP request was IPXE; so, every time the machine booted, you'd get yourself a flesh clone + iscsi target and we'd then mount that ISCSI target in IPXE, which would then hand off the ISCSI target to the OS and away you'd go.

The fundamental problem we hit was that there was a linear delay for every new clone; the delay seemed to be 'number of clones * .05 second' or so. This was on extremely fast hardware. It was the ZFS command to clone that was going to slowly.

Around 500 clones, we'd notice these 10/20 second delays. The reason that hurt so bad is that, to our understanding, it wasn't safe to do ZFS commands or ISCSI commands in a parallel manner; the Sinatra service was responsible for serializing all ZFS/ISCSI commands.

So my question to the author:

1) Does this 'delay per clones' ring familiar to you? Does ZFS on Linux have the same issue? It was a killer for us, and I found a thread eventually that implied it would not ever get fixed in Solaris-land.

2) Can you execute concurrent ZFS CLI commands on the OS? Or is that dangerous like we found it to be on Solaris?

<comment><USER3> 1\. I am not aware of this specific issue. However, I am aware of an issue involving slow pool import with large numbers of zvols. Delphix has developed a fix for it that implements prefix. It should be merged into various Open ZFS platforms soon. It could resolve the problem that you describe.

2\. Matthew Ahrens' synctask rewrite fixed this in Open ZFS. It took a while for the fix to propagate to tagged releases, but all Open ZFS platforms should now have it. ZoL gained it with the 0.6.3 release. Here is a link to a page with links to the commits that added this to each platform as well as the months in which the were added:

[http://open-zfs.org/wiki/Features#synctask_rewrite](http://open- zfs.org/wiki/Features#synctask_rewrite)

<comment><USER2> Thanks for the reply.

Regarding #2: On OpenIndiana, we first started with concurrent zfs commands and ruin, I think, the whole pool (maybe it wasn't that drastic, but was still a disaster scenario where key data would be lost). I couldn't believe it.

I was asking anyone who knew anything... 'so if two admins were logged in at the same time and made two zvols, they could basically ruin their filesystem'? No one knew for sure. Crazy stuff.

Anyway, I'm quite glad that's safe now.

</comment></comment></comment><comment><USER3> I am the author. Feel free to respond with questions. I will be watching for questions throughout the day.

<comment><USER4> I don't really have a question, but as a ZoL user I'd just like to say thanks for all the hard work.

It makes management of my disk arrays pretty painless and has some fantastic migration/recovery stuff going on. All of which I'm sure you know!

<comment><USER5> Seconded. ZFS is the only filesystem I trust with my children's baby pictures, as well as to store the git repo's for my personal projects (stuff I don't want on GitHub for a variety of reasons).

<comment><USER3> I am happy to hear that. While I certainly think ZFS is the best filesystem available for storing this kind of data, i would like to add a word of caution that ZFS is not a replacement for backups. I elaborated on this in one of the supplementary blog posts:

[https://clusterhq.com/blog/file-systems-data-loss- zfs/#disk-...](https://clusterhq.com/blog/file-systems-data-loss-zfs/#disk- failure)

<comment><USER5> Absolutely. I have a regular backups strategy that also backs up to a ZFS system :).

</comment></comment></comment></comment></comment><comment><USER6> Tried using ZFS in the earnest and got spooked, felt it was not production ready. Wanted to use ZFS for MongoDB on Amazon Linux (primarily for compression, but also for snapshot functionality for backups). Tried 0.6.2.

Ended up running into a situation where a snapshot delete hung and none of my ZFS commands were returning. The snapshot delete was not killable with kill -9. [https://github.com/zfsonlinux/zfs/issues/1283](https://github.com/zfsonlinux/zfs/issues/1283)

Also, under load encountered a kernel panic or a hang (I forget), turns out it's because the Amazon Linux kernel comes compiled with no preemption. It seems that "voluntary preemption" is the only setting that's reliable. [https://github.com/zfsonlinux/zfs/issues/1620](https://github.com/zfsonlinux/zfs/issues/1620)

That left a bad taste in my mouth. Might be worth trying out 0.6.3 again.

I am still leafing through the issues closed in 0.6.3, but based on what I see, 0.6.2 did not seem production-ready-enough for me:

[https://github.com/zfsonlinux/zfs/issues?page=2&q=is%3Aissue...](https://github.com/zfsonlinux/zfs/issues?page=2&q=is%3Aissue+milestone%3A0.6.3)

<comment><USER3> Your deadlock was likely caused by the sole regression to get by us in the 0.6.2 release:

[https://github.com/zfsonlinux/zfs/commit/a117a6d66e5cf1e9d4f...](https://github.com/zfsonlinux/zfs/commit/a117a6d66e5cf1e9d4f173bccc786a169e9a8e04)

This occurred because it was rare enough that neither us nor the buildbots caught it back in Feburary. George Wilson wrote a fix for it in Illumos rather promptly. However, Illumos and ZoL projects had different formats for the commit titles of regression fixes. In specific, the Illumos developers would reuse the same exact title while the ZoL developers would generally expect a different title, so we missed it when merging work done in Illumos. I caught it in November when I was certain that George had made a mistake and noticed that our code and the Illumos code was different. It is fixed in 0.6.3. The fix was backported to a few distribution repositories, but not to all of them.

The 0.6.3 release was notable for having a very long development cycle. As I described in the blog post, the project will begin doing official bug fix releases when 1.0 is tagged. That should ensure that these fixes become available to all distributions much sooner. In the mean time, future releases are planned to have much shorter development cycles than 0.6.3 had, so fixes like this will become available more quickly.

That being said, I was at the MongoDB office in NYC earlier this year to troubleshoot poor performance on MongoDB. I will refrain from naming the MongoDB developer with whom I worked lest he become flooded with emails, but my general understanding is that 0.6.3 resolved the performance issues that the MongoDB had observed. Future releases should further increase performance.

<comment><USER6> Thank you so much for the information! This is very encouraging. I will definitely give 0.6.3 a whirl!

</comment></comment></comment><comment><USER7> For anyone interested, over the past couple weeks I have heavily researched ZFS, and created a couple screencasts about my findings [1, 2].

[1] [https://sysadmincasts.com/episodes/35-zfs-on-linux- part-1-of...](https://sysadmincasts.com/episodes/35-zfs-on-linux-part-1-of-2)

[2] [https://sysadmincasts.com/episodes/37-zfs-on-linux- part-2-of...](https://sysadmincasts.com/episodes/37-zfs-on-linux-part-2-of-2)

</comment><comment><USER8> Great blog post! Something from personal experience. OpenZFS on FreeBSD feels mostly like a port of illumos ZFS where most of the non-FreeBSD-specific changes happen in illumos and then get ported downstream. On the other hand, OpenZFS on Linux feels like a fork. There is certainly a stream of changes from illumos, but there's a rather non-trivial amount of changes to the core code that happen in ZoL.

<comment><USER3> This is because Martin Matuška of FreeBSD has been focused on upstreaming changes made in FreeBSD's ZFS port into Illumos. At present, the ZFSOnLinux project has had no one dedicated to that task and code changes mostly flow from Illumos to Linux. This is starting to change. A small change went upstream to Illumos earlier this year and more should follow in the future.

That being said, there are commonalities between Illumos and FreeBSD that make it easier for the FreeBSD ZFS developers to collaborate with their Illumos counterparts:

1\. FreeBSD and Illumos have large kernel stacks (4 pages and 6 pages respectively) while Linux's kernel stacks are limited to 2 pages.

2\. In-kernel virtual memory is well supported in FreeBSD and Illumos while Linux's in-kernel virtual memory is crippled for philosophical reasons.

3\. FreeBSD and Illumos have both the kernel and userland in the same tree. FreeBSD even maintained Illumos' directory structure in its import of the code while ZoL's project lead decided to refactor it to be more consistent with Linux.

Difficulties caused by these differences should go away changes made in ZoL to improve code portability are sent back to Illumos.

<comment><USER9> I'm interested in point 2, can you clarify how and why Linux in-kernel virtual memory is crippled or provide a link?

<comment><USER3> There are two issues:

1\. Page table allocations use GFP_KERNEL, even when done for an allocation that used GFP_ATOMIC. This means that allocations that are needed to do pageout and other things to free memory can deadlock. There is a workaround in the SPL that will switch it to kmalloc when this issue occurs. There is also a new mechanism in Linux 3.9 and later that should render this unnecessary.

2\. The kernel deals with kernel virtual address space exhaustion in vmalloc() by spinning until memory becomes available. This is not a problem on current 64-bit systems where the virtual address space used by vmalloc is larger than physical memory, but it is a problem on most 32-bit systems.

</comment></comment></comment></comment><comment><USER10> I may have read the article too fast , but what about cryptography in zol ? is there a way to crypt data on zol ? regards and thks for the article

<comment><USER3> At present, you need to either encrypt the block devices beneath ZFS via LUKS or the filesystem on top of ZFS via ecryptfs. There are some guides on how to do this for each distribution.

There is an open issue for integrating encryption into ZoL itself:

[https://github.com/zfsonlinux/zfs/issues/494](https://github.com/zfsonlinux/zfs/issues/494)

This will likely be added to ZoL in the future, but no one is actively working on it at this time.

<comment><USER10> thks

</comment></comment></comment><comment><USER11> I'm using ZFS right now, because I need something that cares for data integrity, but the fact that it will never be included in Linux is a very big issue for me. Every time you upgrade your kernel, you have to upgrade the separate modules as well - this is the point where bad things can happen. I will definitely be looking into Btrfs once it is more reliable. For now I'm having a bit of a problem with SSD caching and performance, but don't care about it enough for it to be relevant, I just use the filesystem to store data safely and ZFS does an OK job.

<comment><USER12> From what I understand, aside from certain RAID levels, btrfs is production- ready. RAID5 and RAID6 don't have recovery code finished yet, but RAID0, RAID1, "dup" which just keeps 2 copies of each chunk, and "single" mode all work fine.

<comment><USER13> I've setup btrfs on software mdraid (raid 6) as a backup system (not the only backup!). You still get the checksums and snapshotting, but not the flexibility of the btrfs raid system. It has the advantage of being easy to grow, unlike zfs, which can't be resized once created. We've encountered no problems, even though it has been running for around three years of rsyncing and snapshotting.

<comment><USER14> zfs can grow - new vdevs can be added, and existing vdevs can have their disks replaced one at a time with larger disks.

A bigger downside of ZFS, IMO, is lack of defragmentation and similar larger scale pool management. If you ever push a ZFS pool close to its space limit, you can end up with fragmentation that never really goes away, even if you delete lots of files. The recommended solution is to recreate the pool and restore from backup, or create a new pool and stream a snapshot across with zfs send | zfs receive. Not terribly practical for most home users.

<comment><USER13> zfs can sort of grow by adding vdevs, as you say, however it's pretty wasteful due to the new parity drives It was much more efficient to expand the mdraid raid 6 and expand the btrfs onto that. The other backup server does use zfs (albeit the user mode fuse version). I set up that system in a similar way putting zfs-fuse on mdraid 6.

</comment></comment></comment></comment></comment><comment><USER15> I used ZFSonLinux on my laptop and workstation for a couple of years now, with Ubuntu, without any major problems. When I tried to use it in production, I didn't get data loss but I hit problems:

* Upgrading is a crapshoot: Twice, it failed to remount the pool after rebooting, and needed manual intervention.

* Complete pool lockup: in an earlier version, the pool hung and I had to reboot to get access to it again. If you look through the issues on github, you'll see weird lockups or kernel whoopsies are not uncommon.

* Performance problems with NFS: This is partially due to the linux NFS server sucking, but ZFS made it worse. Used alot of CPU compared to solaris or freebsd, and was slow. Its even slow looping back to localhost.

* Slower on SSDs: ZFS does more work than other filesystems, so I found that it used more CPU time and had more latency on pure SSD-backed pools.

* There are alternatives to L2ARC/ZIL on linux, are built-in, and work with any filesystem, such as "flashcache" on ubuntu.

For these reasons, I think ZoL is good for "near line" and backups storage, where you have a large RAID of HDDs and need stable and checksummed data storage, but not mission critical stuff like fileservers or DBs.

<comment><USER3> I mentioned most of these issues in the supplementary blog posts. Here is where each stands:

* There are issues when upgrading because the initramfs can store an old copy of the kernel module and the /dev/zfs interface is not stabilized. This will be addressed in the next 6 months by a combination of two things. The first is /dev/zfs stabilization. The second is bootloader support for dynamic generation of initramfs archives. syslinux does this, but it does not at this time support ZFS. I will be sending Peter Alvin patches to add ZFS support to syslinux later this year. Systems using the patched syslinux will be immune to this problem while systems using GRUB2 will likely need to rely on the /dev/zfs stabilization.

* There are many people who do not have problems, but this is certainly possible. Much of the weirdness should be fixed in 0.6.4. In particular, I seem to have fixed a major cause of rare weirdness in the following pull requests, which had the side benefit of dramatically increasing performance in certain workloads:

[https://github.com/zfsonlinux/spl/pull/369](https://github.com/zfsonlinux/spl/pull/369) [https://github.com/zfsonlinux/zfs/pull/2411](https://github.com/zfsonlinux/zfs/pull/2411)

* The above pull requests have a fairly dramatic impact on NFS performance. Benchmarks shown to me by SoftNAS indicate that all performance metrics have increased anywhere from 1.5 to 3 times. Those patches have not yet been merged as I need to address a few minor concerns from the project lead, but those will be rectified in time for 0.6.4. Additional benchmarks by SoftNAS have shown that the following patch that was recently merged increases performance another 5% to 10% and has a fairly dramatic effect on CPU utilization:

[https://github.com/zfsonlinux/zfs/commit/cd3939c5f06945a3883...](https://github.com/zfsonlinux/zfs/commit/cd3939c5f06945a3883a362379d0c12e57f31a4d)

* There is opportunity for improvement in this area, but it is hard for me to tell what you mean. In particular, I am not certain if you mean minimum latency, maximum latency, average latency or the distribution of latency. In the latter case, the following might be relevant:

[https://twitter.com/lmarsden/status/383938538104184832/photo...](https://twitter.com/lmarsden/status/383938538104184832/photo/1)

That said, I believe that the kmem patches that I linked above will also have a positive impact on SSDs. They reduce contention in critical code paths that affect low latency devices.

Additionally, there is at least one opportunity to improve our latencies. In particular, ZIL could be modified to use Force Unit Access instead of flushes. The problem with this is that not all devices honor Force Unit Access, so making this change could result in data loss. It might be possible to safely make it on SLOG devices as I am not aware of any flash devices that disobey Force Unit Access. However, data integrity takes priority. You can test whether a SLOG device would make a difference in latencies by setting sync=disabled temporarily for the duration of your test. All improvements in the area of SLOG devices will converge toward the performance of sync=disabled. If sync=disabled does not improve things, the bottleneck is somewhere else.

* These alternatives operate on the block device level and add opportunity for bugs to cause cache coherence problems that are damaging to a filesystem on top. They are also unaware of what is being stored, so they cannot attain the same level of performance as a solution that operates on internal objects.

<comment><USER15> Great reply :)

Your last two points are a little weak. Ultimately ZFS does more stuff, so on an SSD its going to be slower than a filesystem which doesn't have all these extra features if you aren't using them. I think there's a trade-off to be had.

All software has bugs, especially ZFS. You could argue it is easier to developer, test, and maintain individual device-mapper building blocks.

</comment></comment></comment><comment><USER3> I have been inundated with feedback from a wide number of channels. If I did not reply to a comment today, I will try to address it tomorrow.

</comment><comment><USER16> ZFS, and most* other file systems are all about _one_ computer system.

While ZFS data integrity features may be useful, they don't prevent the wide variety of things that can go wrong on a _single_ computer. You still need site redundancy, multiple physical copies, recovery from user errors etc.

Large, modern enterprises are better off keeping data on application layer "filesystems" or databases, since they can more easily aggregate the storage of hundreds or thousands of physical nodes. ZFS doesn't help with anything special here.

For the average home user, ZoL modules are a hassle to maintain. You are better of setting up FeeNAS on a 2nd computer if you really want to use ZFS. Otherwise there is nothing much over what XFS, EXT4 or btrfs can offer.

The 'ssm' set of tools to manage LVM, and other built in file systems, is more easier for home users with regular needs.

GlusterFS and others are distributed file systems, but suffers from additional complexity at the OS and management layer.

<comment><USER3> The Lustre filesystem is able to use ZFSOnLinux for its OSDs. This gives it end to end checksum capabilities that I am told enabled the Lustre developers to catch buggy NIC drivers that were silently corrupting data.

Alternatively, there is a commercial Linux distribution called SoftNAS that implements a proprietary feature called snap replicate on top of ZFS send/recv. This is allows it to maintain backups across availability zones and is achieved by its custom management software running the zfs send/recv commands as per user requests.

In the interest of full disclosure, my 2014 income tax filing will include income from consulting fees that SoftNAS paid me to prioritize fixes for bugs that affected them. I received no money for such services in prior tax years.

</comment></comment><comment><USER17> I love ZFS, and I love working with Linux, but I can't help but worry about using ZFS on Linux. Without the needed support from the kernel side, I don't see how it can be useful for production. I can see using it on personal workstations, but for any situation where data loss is critical, you just won't see any uptake. Because of the licensing, ZFS can never be anything more than a second-class citizen on Linux.

That said, I run a FreeBSD ZFS file server just to host NFS that is exported over to a Linux cluster. At least on FreeBSD, there is first-class integration of ZFS into the OS. (I used to also maintain a Sun cluster that had a Solaris ZFS storage server that exported NFS over to Linux nodes, which is where I first got a taste for ZFS).

So, I guess my main question is: In what use cases is ZFS on Linux so useful when native FreeBSD/ZFS support exists?

I'm not saying it can't be done - I just don't understand _why_.

<comment><USER18>
      Without the needed support from the kernel side

Can you clarify what you mean by that?

<comment><USER17> Basically I mean integration with the kernel's code base and all of the testing that entails. After their initial development, file systems all end up migrating to the kernel's code base.

So, I'm not thinking in terms of technical API support, but more development/testing/integration support.

<comment><USER3> That is not a requirement. ZoL has the most sophisticated build system of any Linux kernel module in the world to enable it to live outside of the main tree. ZoL relies on autotools' API checks to do this. In addition, the project has an automated buildbot that helps us to detect regressions in pull requests before they are merged. It is similar in principle to how lustre filesystem development is done. Lustre is also (primarily) outside of the tree and lived entirely outside of the tree for years.

That being said, being inside the kernel source tree is not necessarily a good thing. As I wrote in the blog post, other filesystems on Linux generally do not provide the latest code to older kernels, but ZoL provides the latest code to all supported kernels and distributions. This ranges from Linux 2.6.26.y to the 3.16.y in 0.6.3 and will include 3.17.y in 0.6.4. Something as important as a filesystem should be updated to fix bugs, even if the kernel proper cannot be. The inability of all Linux systems to update to the latest kernel is an issue that Linus Torvalds mentioned at LinuxCon North America 2014 and ZoL is one of the few filesystems that can deal with it on systems where it is deployed.

<comment><USER17> You know... I hadn't given much thought to Lustre. And that is a very good comparison. Lustre is completely out of kernel (in terms of development), so it's not like you're the first to try this.

I guess it really depends on where you want ZoL to be deployed. Lustre is typically deployed on HPC clusters and is managed by people who understand it, how to set it up, and how to manage it. It's not a trivial system to get working, and requires a pretty large budget. It just isn't setup on your typical server. Lustre also has a few big names behind it to provide development resources and support (Whamclound, now Intel).

What is the target market for ZoL? Do you want it to work on servers? Clusters? Personal workstations? They are very different markets.

I'm using ZFS right now for a single storage server that doesn't need to support a large cluster, so ZFS/NFS works great. But I'm using FreeBSD. For my Linux servers, I wouldn't think of using a filesystem that wasn't natively supported by Red Hat. I just don't want the management headache. I'm okay running a single FreeBSD box for my storage needs, but I'm fearful of what happens when I'd need to scale things out to multiple servers.

I wish you luck, I really do. It seems like you're not going into this blind and you're making good decisions. But it will be an uphill battle and you'll always have that specter of Oracle looming over you. It doesn't matter if you're technically/legally right on the licensing front - you'll still have it looming over you until Larry signs off on it (like you said below).

If you can get that sign off, then all bets are off, and you'd be golden!

</comment></comment></comment></comment></comment><comment><USER19> I've used ZFS (FreeNAS) for quite a few years and find it pretty flawless. Trust it's not too dumb a question but what advantage is there to running ZFS on Linux when you can run it on variants of Solaris or BSD just fine?

</comment><comment><USER20> I've used ZoL since it was created, and zfs-fuse before that. I ran it on my workstation for a few years (managing a 4x750gb RAID-Z (= ZFS's RAID-5 impl), with ext3 on mdadm RAID 1 2x400gb root), and then swapped to BTRFS for 2x2TB BTRFS native RAID 1 (which was Oracle's ZFS competitor that seems to be largely abandoned although I see commits in the kernel changelog periodically), and now back to ZFS on a dedicated file server using 2x128GB Crucial M550 SSD + 2x2TB, setup as mdadm RAID 1 + XFS for the first 16GB of the SSDs for root[2], 256MB on each for ZIL[1], and the rest as L2ARC[3], and the 2x2TB as ZFS mirror. I honestly see no reason to use any other FS for a storage pool, and if I could reliably use ZFS as root on Debian, I wouldn't even need that XFS root in there.

All of this said, I get RAID 0'ed SSD-like performance with very high data reliability and without having to shell out the money for 2TB of SSD. And before someone says "what about bcache/flashcache/etc", ZFS had SSD caching before those existed, and ZFS imo does it better due to all the strict data reliability features.

[1]: ZFS treats multiple ZIL devs as round robin (RAID 0 speed without increased device failure taking down all your RAID 0'ed devices). You need to write multiple files concurrently to get the full RAID 0-like performance out of that because it blocks on writing consecutive inodes, allowing no more than one in flight per file at a time. ZIL is only used for O_SYNC writes, and it is concurrently writing to both ZIL and the storage pool, ie, ZIL is not a write-through cache but a true journal.

The failure of a ZIL device is only "fatal" if the machine also dies before ZFS can write to the storage pool, and the mode of the failure cannot leave the filesystem in an inconsistent state. ZFS does not currently support RAID for ZIL devices internally, nor is it recommended to hijack this and use mdadm to force it. It only exists to make O_SYNC work at SSD speeds.

[2]: /tank and /home are on ZFS, the rest of the OS takes up about 2GB of that 16GB. I oversized it a tad, I think. If I ever rebuild the system, I'm going for 4GB.

[3]: L2ARC is a second level storage for ZFS's in memory cache, called ARC. ARC is a highly advanced caching system that is designed to increase performance by caching often used data obsessively instead of being just a blind inode cache like the OS's usual cache is, and is independent of the OS's disk cache. L2ARC is sort of like a write through cache, but is more advanced by making a persistent version of ARC that survive reboots and is much larger than system memory. L2ARC is implicitly round robin (like how I described ZIL above), and survives the loss of any L2ARC dev with zero issues (it just disables the device, no unwritten data is stored here). L2ARC does not suffer from the non-concurrent writing issue that ZIL "suffers" (by design) from.

<comment><USER21> Can you speak more about why ZFS is better than BTRFS?

<comment><USER3> A btrfs versus ZFS comparison probably deserves a blog post of its own, but I will try to address your question. I wrote the following on this topic last year:

[https://groups.google.com/d/msg/funtoo- dev/g9OY_vqVpCM/VTKF8...](https://groups.google.com/d/msg/funtoo- dev/g9OY_vqVpCM/VTKF8Ef9ab4J)

However, significant time has passed and it requires some corrections to be current:

1\. I have not heard of any recent data corruption issues in btrfs, although I have not looked into them lately.

2\. btrfs now has experimental RAID 5/6 support, but is neither production ready nor as refined as ZFS' raidz.

3\. I should have said "inline block-based data deduplication". You can (ab)use reflinks to achieve a file-level data deduplication in btrfs, but it is not quite the same. btrfs now has a bedup tool that makes using reflinks somewhat easier now:

[https://btrfs.wiki.kernel.org/index.php/Deduplication](https://btrfs.wiki.kernel.org/index.php/Deduplication)

4\. btrfs now has some kind of incremental send/recv operation. However, it is not clear to me how it handles consistency issues from having "write-able snapshots":

[https://btrfs.wiki.kernel.org/index.php/Incremental_Backup](https://btrfs.wiki.kernel.org/index.php/Incremental_Backup)

5\. Illumos' ZFS implementation is now able to store small files in the dnode, which is improves its efficiency when storing small files in a manner similar to btrfs' block suballocation. This feature will likely be in ZoL 0.6.4.

Aside from those corrections, what I wrote in that mailing list email should still be relevant today. However, there are a few advantages that ZFS has over btrfs that I recall offhand that I do not see here there or in nisa's reply:

0\. ZFS uses 256-bit checksums with algorithms that are still considered to be good today. btrfs uses checksum algorithms that are known to be weak. In specific, btrfs uses CRC32 on 32-bit processors and CRC64 on 64-bit processors. CRC32 is the same algorithm used by TCP/IP. Its deficiencies are well documented:

[http://noahdavids.org/self_published/CRC_and_checksum.html](http://noahdavids.org/self_published/CRC_and_checksum.html)

I have not examined CRC64, but I am not particularly confident in it. btrfs should have room in its on-disk data structures that would allow it to implement 256-bit checksums in a future disk format extension, but until then, its checksum implementations are vastly inferior.

1\. The ztest utility that I described in the blog allows ZFS developers to catch issues that would have otherwise gone into production and debug them from userland. No other filesystem has something like quite like it.

2\. ZFSOnLinux is the only kernel filesystem driver that is kernel version- independent, so if you are unable to upgrade your kernel, you can still get fixes. The inability of people to always update their kernels is an issue Linus mentioned at LinuxCon North America 2014.

3\. The CDDL gives the ZFSOnLinux a patent grant for the ZFS patent portfolio. This is something that btrfs does not have and will likely never have unless Oracle decides to provide one. Consequently, Oracle is the only company in the world that I know is able to ship products incorporating the btrfs source code without being at risk should btrfs infringe on one of the dozens if not hundreds of patents in the ZFS patent portfolio. A small subset of them can be accessed from the Jeff Bonwick Wikipedia page:

[https://en.wikipedia.org/wiki/Jeff_Bonwick](https://en.wikipedia.org/wiki/Jeff_Bonwick)

<comment><USER22> I got the impression somewhere that btrfs RAID 5/6 support allows, or will allow, new devices to be added to an existing RAID group.

That's an important feature for home and small business users. Having to replace every drive in a RAID group to grow it, as ZFS requires, is painful and expensive.

Fortunately, drives are cheap enough these days that you can just way oversize your pool to begin with. But anyone switching to ZFS should be aware of the need to do this.

<comment><USER23> You can add/delete devices from raid5/6 volumes now. The raid5/6 code is still experimental, in particular while detected problems are fixed on-the-fly to userspace, the fixes aren't written back to drives. That limitation applies to normal usage and scrubbing. A balance detects and fixes these.

Also the determination of a drive being "faulty" (in the md/mdadm sense) and how this gets communicated to userspace isn't in place. If it's in place for ZoL (?) that'd be a considerable difference, a bigger one than checksum algorithms in my opinion.

</comment></comment></comment></comment></comment><comment><USER24> I've created the script below a while (year) ago. It (deb)bootstrap a working Debian Wheezy with ZFS on root (rpool) using only 3 partitions: /boot(128M) swap(calculated automatically) rpool(according to # of your disks, mirrored or raidz'ed).

All commentaries are in Brazilian Portuguese. I didn't have time to translate it to English. Someone could do it and fill a push request.

[https://github.com/turrini/scripts/blob/master/debian- zol.sh](https://github.com/turrini/scripts/blob/master/debian-zol.sh)

Hope you like it.

<comment><USER3> Thanks for sharing. I will let Debian users interested in / on ZFS know that this is available as they ask me about this sort of thing.

</comment></comment><comment><USER25> When swap doesn't work, mmap is unlikely to work correctly either.

Figuring out why that is so is left as an exercise for the poster.

</comment><comment><USER26> Putting production data on a driver maintained outside the mainline Linux kernel is a bad idea.

That isn't a licensing argument - I'm happy to use a proprietary nvidia.ko for gaming tasks, for example, because I won't be screwing up anyone's data if it breaks.

<comment><USER3> You could be "screwing up" someone's data if an in-tree filesystem breaks. If you read the supplementary blog posts, you would have seen the following:

[http://lwn.net/Articles/437284/](http://lwn.net/Articles/437284/)

Nearly all in-tree filesystems can fail in the same way described there. ZFS cannot. That being said, no filesystem is a replacement for backups. This applies whether you use ZFS or not. If you care about your data, you should have backups.

<comment><USER17> Well, to be honest, all filesystems that are currently in the kernel tree started out as being maintained outside the tree. Inclusion into the kernel is normally one of the end goals. It's part of the standard progression - 1) rapid development outside of the tree, 2) once the filesystem is stable, it negotiates for inclusion, 3) inclusion into the kernel, 4) maintenance / updates as part of the main kernel development process.

ZoL isn't even trying to get included into the kernel, so it's a bit of an odd duck here.

</comment></comment></comment><comment><USER27> While I like most parts of ZFS, these days BTRFS is both stable and performs well with a decent feature set. We moved from ZFS and EXT4 to BTRFS for a good portion of our production servers last year - and we haven't looked back.

<comment><USER28> Do you run RAID5/6? I had that running for half year, and it crashed often.

Now on ZFS (raidz) and it works flawless

<comment><USER27> We run it on iSCSI LUNs straight from our SAN so this hasn't been an issue for us.

</comment></comment></comment><comment><USER29> I have a laptop running ubuntu with a single SSD. Does it make sense to run it with ZFS to get compression and snapshots? If I add a hard drive, again does it make sense (perhaps using SSD as cache (arc?) )

<comment><USER30> I've never heard of someone using ZFS with a single disk. You're probably better off with ext4.

The compression and deduplication features of ZFS is terrific on network filers. Compression could possibly improve performance slightly on a single disk system.

With two disks, I'd say you'd probably be better off with running RAID0 (or no RAID at all) and having a great backup plan. Using another SSD to cache writes to another SSD doesn't make a whole lot of sense to me.

<comment><USER20> Honestly, I recommend XFS over ext4. It seems to be a much more mature file system, and Redhat-family distros (RHEL, Centos, Scientific, etc) have switched to XFS as the default filesystem (instead of moving to ext4 from 6.x's default of ext3; 6.x does not support XFS or ext4 for root).

XFS performs better out of the box on a wide range of hardware, while seemingly giving stronger data reliability guarantees (but not anywhere near ZFS's).

ZFS on a single disk, however, will still give you data checksumming, so you can detect silent data corruption. XFS's sole missing feature as a basic filesystem, imo, is data checksumming.

<comment><USER31> RHEL6 uses ext4 as default[1] filesystem. It certainly does support ext4 as root.

[1]: [https://access.redhat.com/documentation/en- US/Red_Hat_Enterp...](https://access.redhat.com/documentation/en- US/Red_Hat_Enterprise_Linux/6/html/6.0_Release_Notes/filesystems.html#idp10337936)

<comment><USER20> Weird, I had to install RHEL6 for a customer, and it defaulted to ext3 and ext4 was not selectable.

</comment></comment></comment></comment></comment><comment><USER32> I've looked into ZFS before for distributions like freenas, is there any solution on the horizon for the massive memory requirements?

For example, needing 8-16gb ram for something like a xTB home nas is high.

<comment><USER3> The "massive memory requirements" only exist if you use data deduplication and care about write performance. Otherwise, ZFS does not require very much memory to run. It has a reputation to the contrary because ARC's memory usage is not shown as cache in the kernel's memory accounting, even though it is cache. This is in integration issue that would need to be addressed in Linus' tree.</comment></comment>


Ask HN: Dream Job or career suicide? - <AUTHOR> Ask HN: Dream Job or career suicide?
For the last two years I have worked remotely for a someone who is trying to convince a financial services company (of about 20 employees) to build a replacement for their existing (dated) software platform. We've been building free prototypes/demos for the last two years. During this time I have been the sole developer.
Overall my job is great. I get along well with my boss, there are no hard deadlines, my hours are 100% flexible, and the work is techincally interesting and low pressure. The pay is a bit below the going rate.
The problem is that I've slowly become convinced that it is very unlikely that we will actually get a contract with the financial services company. Additionally, my boss wants extremely technically complex features (like a time traveling database) in the prototype application without having any indication that the company actually wants those features. The application is his baby so feature X, Y, and Z are critical.  The financial services company also commented in a meeting that his cost estimate was less than 10 percent of what they expected. So basically he is trying to sell the Sun, the Moon, and the Stars to them for dirt cheap.
I have very little experience in custom software sales but it seems to me that eventually the project is going to fizzle (no sale) or turn into a huge mess (a sale but an eventual trainwreck when actual expenses collide with requirements).
In the meantime, my work conditions are great.
My question is: Is it terrible for my resume to work for years on a project that never sees the light of day? Or should I stop worrying and just be happy that right now I have a relaxed work environment? <comment><USER2> >> Is it terrible for my resume to work for years on a project that never sees the light of day?

As a former recruiter turned engineer, I can say that you shouldn't worry too much about it. If there were a surplus of great engineers fighting for table scraps, I might offer different advice. You may find it mildly harder to break into gigs that want more experience working on a fast paced agile team, but the reality is that you'll always have a mildly harder problem breaking into something that asks for experience you haven't been practicing recently.

What I'd be more concerned about is working on tech that is too crusty. If you're programming in MUMPS, using flat files as a relational database, and hosting everything on Windows 3.0 being emulated on OS/2, you may wish to revisit your tech stack. If you can't throw any not-stale tech into the keyword soup on your resume, it'll look like you haven't been motivated enough to keep your skills sharp. Disregard this advice if you have plenty of people trying to recruit you and/or you know people who'd hire you if they could.

</comment><comment><USER3> As an eng manager in SF, I'll form the dissenting opinion here. Get out now.

Products need to ship and go somewhere. You only now what works in terms of your tech stack by dealing with the fires of actual production traffic.

Lots of teams end up focusing on the wrong thing and building a product no one uses. It's totally okay for this to go on for a year. Going on for two years is a huge red flag and it might make you unemployable at a company that has a real product.

Get out as soon as possible.

<comment><USER4> Eh. Plenty of product is private. I worked for a few years on tech inside a military research lab where neither the product nor code will ever be seen by the greater public, but that doesn't change the learning that took place. It's not a concern to me or anyone familiar with an industry like that.

</comment></comment><comment><USER5> Are you getting paid for your work? You mentioned "free prototypes".

You could always just write on your resume that you worked for "X Company", and then give details about the project you worked on, whether or not it sees the light of day is irrelevant. Employers want to see that you can work/code/do your job, not if past projects were a success or not.

<comment><AUTHOR> My boss is paying me. The financial services company hasn't paid my boss for anything. Hence the "free" part.

Thanks for your thoughts.

</comment></comment><comment><USER6> If you are worried about the possibility of losing your job and having to do without a salary, then you should start looking now. But from the way you asked your question, it seems like you are aware of this possibility and you are more worried about how this will affect your long-term career. My 2 cents is that in future interviews, it's very unlikely anyone will ask or care about the success of the project. They care about your technical skills and it sounds like this job is good from that aspect. You should be able to spin this in a positive light.

</comment><comment><USER7> Invariably for questions like this, I think the answer is that you need to go out and see what else is available. Get your resumé up to date, send it to organisations that interest you, attend interviews.

Just going through that process should help you understand where you want/need to be. When you get offers for alternative positions and have met the teams that you could be working with, then you'll know whether you should stay or leave.

</comment><comment><USER8> I've been building software myself and while I haven't officially started to sell anything yet, I must say, what I've gained in experience far exceeds whether the products I created are going to sell or not. I started out knowing almost nothing and just a year later, I can build just about anything with PHP and MySQL. So there is no waste there.

However, "selling dirt cheap" isn't always the way to go. Most companies think: "You get what you pay for" and that may be a mistake on your partner's part. Underpricing a product might seem great in order to beat the competition, but when it comes to companies and profitability: they want results and profits. They don't care how much it costs to get that. They will pay tens of thousands of dollars to have software that is reliable and suits their exact needs. When the client lets you know they expect to be paying more: it usually means they are going to probably have some demands for customization. He wants the clients, but then they will likely ask for custom features or not be happy with something, and.. it may just be a never-ending story, and there is going to be a time where you both have to just say no.. and take the hits and negative reviews.

OR.. say yes, but it is best to charge enough to stay in business and even hire additional employees. I have undercharged clients before... many times and many mistakes.. and after a while, when is it no longer worth it? When will you just bite the bullet and take the hit because.. you didn't ask for enough money to be worth maintaining the project.

Even if your project doesn't see the light of day, but you have the source code, and it can be seen somewhere, I'm sure its not career suicide. What you have probably learned is more amazing and useful. BUT... he may be thinking smaller or just not reaching the right audience, and you may need to be the one to advise him of a different method, whether he wants to hear it or not. Both of you want the same thing: A Saas in which you are both making good money and developing software as a business.

<comment><AUTHOR> Thank you for your thoughts. I certainly have learned a lot - although I had learned how to build full stack applications before I started working for him.

Just to be clear - he's not my partner, he's my client. So I don't have any rights to the source code. Also, we're building a custom software solution for a business and not a Saas product.

<comment><USER8> Well, if he's paying you to do the job, than it doesn't matter if you have something or nothing "to show for it" in the end. Your paycheck and way to make some type of living helps with that. If he can afford to keep paying you, whether he fails or not, is on him.

I thought you were business partners, but if not, than it is his own risk, not really yours.

</comment></comment></comment><comment><USER9> How is your boss funding this project, prior to signing up customers?

Do you have an equity stake, vested or otherwise?

<comment><AUTHOR> I don't know how he is funding it. As far as I know, he has no existing business revenue streams. It seems like he is paying me out of his own pocket.

I don't have any equity.

</comment></comment><comment><USER10> > the work is techincally interesting and low pressure.

What are you learning? What are the big problems in your field? How is this role stretching you professionally? Relative to career suicide vs vitality-- one should be continually adding and building skills that will be in demand.

Richard Hamming offers briliant food for thought on your career > [https://www.youtube.com/watch?v=a1zDuOPkMSw](https://www.youtube.com/watch?v=a1zDuOPkMSw)

</comment><comment><USER11> The OP's comments are dead on this thread. If you want to interact with the OP, turn on showdead on your profile.

<comment><USER12> The entire account is marked dead. I had to "vouch" for the submission to get this thread to show in the first place. I'm not sure why they were hellbanned as this is the only submission and its very much a valid "Ask HN post".

</comment></comment><comment><USER13> If you're working with tech you like and you're earning a comfortable wage, I would say to carry on as long as you like, especially with an unstressed working environment. The trouble, and you seem to be very aware of it, is that everything could change at the drop of a hat - if the deal finally falls through or the costs spiral out of control, you could wind up in a nightmare.

Keep options open. If you're working with good tech (which from the sounds of the software you're writing, you are), then the skills are transferrable and you ought to be able to move to a new job with little warning.

As for the resume, it's great to see a respectable length of time at the same firm (because it shows dedication), and whether or not the project ever came to fruition is irrelevant to a new employer - that's on the people (person) above you. You're the person building it, you know the code and the tech, you're the one who makes things work. That's what to emphasise in your job search.

I can totally understand your desire to remain in a non-stressed work environment, but it would be worth worrying just a little about your future and keeping a close eye on the market. Always keep something available. As much as you may feel loyalty to your current boss, you have to be responsible for your own career and personal state, and if (or when) everything comes crashing down, you can tell him you appreciated the opportunity he gave you, but you must look out for yourself. Don't let him drag you down with him; if his business decisions turn out to be wrong, it's his responsibility, not yours.

At the same time, you can try to have a serious conversation about his approach. Lay out your concerns that you, as the author of this project, are not going to see your efforts bear fruit if the customer bails or he drastically underprices it; it begs the question, if he's priced it so low, what source of income is he using to pay you? Personal funds? Those will run out eventually. Tell him that if his business is to have hope of turning a profit, he needs to charge realistic amounts, especially if the customer can be pursuaded to pay more. After all, some businesses (particularly in the financial sector) will stick to a 'get what you pay for' mindset, and will not be convinced to buy a cheap system no matter how fancy the features are, because they will see it as low quality or unsupportable (the latter is a good possibility when they realise what they actually do need).

In short, try to get your boss to see sense, but carry on as long as you're comfortable, and be ready to change jobs if the wheels start coming off.

</comment><comment><USER14> Until you get paid, all is good. As soon as reality (no customers) bites into your employer and he starts acting the colors of a failed business, politely jump ship and goodbye.</comment>


Why Do We Murder the Beautiful Friendships of Boys? (2017) - <AUTHOR> https://medium.com/s/man-interrupted/why-do-we-murder-the-beautiful-friendships-of-boys-3ad722942755 <comment><USER2> Do we?

America is a very mobile country. One best friend moved a couple of hundred miles when we were in elementary school. My family moved when I was between 8th and 9th grade, which was goodbye to a lot of friends I had known for anything from nine to a couple of years. (There wasn't any Skype in 1969.) I moved after college.

Points off to the author for identifying Alain de Botton as a philosopher. Also for the expression "perform masculinity".

<comment><USER3> I think America used to be very mobile, but if the economic plight of the interior is any indication, it would seem that the country is not as mobile as it once was.

</comment></comment><comment><USER4> > How many times have we heard parents say, "Oh, they'll make new friends," as > if the relationships of children are so shallow and contextual that they can > be swapped out like last year's lunchbox?

This is where the author loses me; for several reasons

1) Parents might have several reasons for severing a child's friendship, whether relocating for a job, disagreements in the adult relationship, or disagreements in moores and values.

2) People will experience loss in many types of relationships. It doesn't mean they were worthless, in many ways this grief means just the opposite. It's similar important for people to be able to create new friendships with new people.

<comment><AUTHOR> The point the author is trying to make IMO is that the _parents_ dismiss the relationships as shallow and contextual, while the children themselves, the ones actually in the relationship, have no choice whatsoever.

Of course relationships will come and go in everyone's life, but in an adult relationship at least there is some choice made by one of the two parties invested in the relationship.

I met my dearest and oldest friend in the fourth grade. If my parents had made a life decision that would have cost me that relationship it would have been a huge, huge (really unimaginable at this point) loss for me.

</comment></comment><comment><USER5> This article starts with several paragraphs of rambling about the author and his wife and whatnot. I don't know if there is an official name for this sort of medium.com writing style, but I have seen it criticized before. Maybe medium.com authors are paid by words?

Anyway, I am sure it's a good article. Sadly, I am now conditioned to close the tab whenever the article starts with the author & friend or partner last Friday...

<comment><USER6> Your loss. It's a great article, with a researcher's results woven in with the author's personal experience of it as a (fairly compelling) "worked example". What you describe as "rambling" is 2 sentences of introduction to the former, followed by 2 sentences of introduction to the latter. If your attention span can't handle 4 perfectly relevant introductory sentences, that's not the article's fault.

</comment></comment><comment><USER7> Cui bono?</comment>


Why Andreessen Horowitz Is Investing in Rap Genius - <AUTHOR> http://news.rapgenius.com/Marc-andreessen-why-andreessen-horowitz-is-investing-in-rap-genius-annotated#note-1103810

<comment><USER2> I was indeed wondering about this. And I listen to hip hop.</comment>


United States Patent number 1 - <AUTHOR> http://www.google.com/patents/about?id=krA-AAAAEBAJ

<comment><USER2> Just to be clear this isn't the first US patent, just the one assigned #1 in the current numbering system (I think?).

More info on the "first" US Patent. <http://www.uspto.gov/news/pr/2001/01-33.jsp>

<comment><USER3> According to wikipedia there were other almost ~10k patents before, but they were destroyed in a fire and are known as X-Patents

<http://en.wikipedia.org/wiki/X-Patents>

<comment><USER4> Google has these too

<http://bit.ly/8X7Ydw>

<comment><USER5> I don't mean to add to the noise floor, but is there a reason you generated a bit.ly link for this? This isn't twitter with particularly restrictive character constraints; all it serves is obfuscation as far as I can tell.

I'm genuinely curious what the reasoning is.

</comment></comment></comment></comment><comment><USER6> I like James Ernest's version better, where a time machine gets patent #1.

<http://www.cheapass.com/products/boardgames/cag034.html>

<comment><USER7> This looks like a ton of fun.

</comment></comment><comment><USER8> How long for OCR to become usable for something like this? Google's OCR transforms the readable "Locomotive Steam-Engine for Rail and Other Roads" to "BAIL AND OTHER ROADS"

<comment><USER9> So far? About 3 decades.

<comment><USER8> According to Wikipedia, it's been much much longer than that, but this is my point.

</comment></comment></comment><comment><USER10> looks way more readable than modern patents. Almost like it wasn't written by a lawyer. Only 4 pages.

</comment><comment><USER4> The number system came later. X series gives the earliest patents, including the first issued: X1

<http://bit.ly/8X7Ydw>

<comment><USER11> Please don't use URL shorteners.

<comment><USER4> Sorry. Here's the link:

<http://www.google.com/patents?id=nrxzAAAAEBAJ>

That's General Washington's signature in the middle, and Edmund Randolph's (first Attorney General) on the bottom.

The cabinet was the PTO in 1790.

</comment></comment></comment><comment><USER12> Man, the beginning of the patent makes this sound like the greatest invention ever. Let trains brave mud,rain,ice and anything. Except I think the US postal service has some prior art to the whole mud,rain and ice thing. [http://en.wikipedia.org/wiki/United_States_Postal_Service_cr...](http://en.wikipedia.org/wiki/United_States_Postal_Service_creed)

<comment><USER13> It appears that Herodotus has prior art on that.

</comment></comment><comment><USER14> I see... patents have always been ridiculous. Unsurprising.

<comment><USER15> What is ridiculous about that patent?

<comment><USER16> That to any engineer worth his/her (but mostly his in that time) salt that must have been blindingly obvious.

After all, the 'rack and pinion' had been around for a while and the rack really is just a gear rolled out.

</comment></comment></comment><comment><USER17> Like the line numbers in the margin.

<comment><USER18> It's that way in almost all patents, I think. It may be a style requirement.

<comment><USER19> It's so when your patent is disputed, you can cite things like "In column 9, lines 34-36 of document E2, it is disclosed that..." (I'm translating just such a dispute claim now. For a tampon applicator. I'd rather be learning about railroads.)</comment></comment></comment>


Why Intelligent People Drink More Alcohol - <AUTHOR> http://www.psychologytoday.com/blog/the-scientific-fundamentalist/201010/why-intelligent-people-drink-more-alcohol <comment><USER2> Half of all people with my IQ or higher have extreme social disorders or schizophrenia. This happens at this end of the bell curve.

Notably, these were not included in the list of potential confounding factors, nor do they appear in the studies published that appear to have the same list of potential confounds as those listed in that paper.

Drinking because you have a non-depression extreme social disorder is fairly unsurprising.

</comment><comment><USER3> For the same reason Homer keeps his crayolla in the nose, DUH!

</comment><comment><USER4> Because we have to put up with morons</comment>


"Innovated in China" to replace "Made in China" - <AUTHOR> http://www.chinnovate.com/innovation-chinese-style-thriving-on-uncertainty/ Innovation Chinese Style:  Thriving on Uncertainty <comment><USER2> Most Americans equate Chinese made goods with cheap, shoddily made goods. But yet they see that Chinese mothers are superior and that these tiger moms are producing A+ test-taking kids. Even the fresh-off-the-boat Chinese kids that arrive in America excel in the public school system within a year or two, though they knew no English upon first arrival. So if the Chinese method of child-rearing produces adults that are super achievers, they should certainly be capable of receiving an A+ in Business Innovation and beating Americans pants down.</comment>


The US Economy Outgrew Money Supply for the First Time Since 2010 - <AUTHOR> https://thesoundingline.com/the-us-economy-outgrew-money-supply-for-the-first-time-since-2010/ <comment><USER2> 1) this website's desktop UI is horrible. Have you seen the side of that left sidebar? it's taking half the real estate screen!

2) I'm happy to hear some good news about our economy (the USA). Lately it seems all you hear is "doom is near", glad to finally hear something positive.</comment>


A video game trailer has been nominated for an Academy Award - <AUTHOR> https://www.engadget.com/2017/06/08/video-game-trailernominated-for-academy-award/ <comment><USER2> The trailer is eligible for being nominated. The actual nominations don't happen until January 2018. Still pretty cool though.

</comment><comment> King-Aaron I don't know what's going on, or what it's trying to tell me the game is about. But I must have it.

</comment><comment><USER3> The movement of all the characters is by somersault?

<comment><USER4> Not all, some models moved by morphological transforms, e.g. flapping wing textures by sinusoidal excitation. I presume it's that it's fairly quick to build a model, but extremely laborious to animate it. For a game that features "everything" I think I can let it slide. :-)</comment></comment>


Non-US Cards Used At Target Fetch Premium - <AUTHOR> http://krebsonsecurity.com/2013/12/non-us-cards-used-at-target-fetch-premium/

<comment><USER2> Damn it i want two way verification on any purchase over X amount! Meaning anytime a charge over X amount (an amount i set) tries to go thru I get a text message asking for me to approve or deny it.

This is getting ridiculous! 1st Adobe who I will never do business with again (had to cancel my accounts & open new ones) and now Target!

<comment><USER3> Although it's not two-factor, some banks will send you push messages or texts for every transaction taking place on the card. I use Simple, they do the former. One nice feature about Simple is you can disable the debit card directly from within the app, and then re-enable it if you need to (for example, if you lose your wallet temporarily).

Simple also sent me a new debit card automatically without prompting, their systems having noticed I'd shopped at Target last week.

The problem with the two factor system as you describe is that while it works for 'cardholder not present' transactions (and there already exist standard two-factor methods for such payments, not that many banks use them), it would make retail POS payments incredibly slow. You'd have to wait for a text/push, you might not have any reception, then you've got to reply, etc. This is one reason why EMV is, for all its failings, more attractive than mag-stripe: it is considerably harder and more expensive to clone EMV cards.

<comment><AUTHOR> You don't need to wait for a text or push notification, and you don't have to be online.

Just use a time-based one-time password.

It's instant, secure, and simple. And already available free in Google Authenticator.

[https://en.wikipedia.org/wiki/Time-based_One- time_Password_A...](https://en.wikipedia.org/wiki/Time-based_One- time_Password_Algorithm)

[http://code.google.com/p/google- authenticator/](http://code.google.com/p/google-authenticator/)

</comment></comment></comment><comment><USER4> Interesting that it's cheaper for the banks to buy back their own cards than reissue them during giftmas.

<comment><USER5> > Interesting that it's cheaper for the banks to buy back their own cards than > reissue them during giftmas.

They are not "buying back" anything. The cards can be resold at will.

<comment><USER4> >They are not "buying back" anything. The cards can be resold at will

I see you have never carded. The banks are searching by BiN to find their own cards, and according to Krebs, buying them back. They said they did this to avoid having to reissue cards during Giftmas which is their biggest shopping holiday thus saving money.

</comment></comment></comment><comment><USER6> If we used chip and PIN in the US would this still have been possible?

<comment><USER7> Can you buy things online with chip and PIN?

<comment><USER8> Yes, you can - [http://en.wikipedia.org/wiki/3-D_Secure](http://en.wikipedia.org/wiki/3-D_Secure)

When I buy something online using my CC, and if the payment processor supports that (mostly local/european shops), I get redirected to a page on my bank for verification. Some banks require the PIN to be entered, some others to enter your login credentials, some coordinates from your code card or even verify with a SMS code.

Not exactly using the chip, but it involves the PIN surely.

<comment><USER9> If the payment gateway decides to not implement this kind of protection, the payment goes through anyway (I'm guessing it'll be more expensive for the merchant, but still).

</comment></comment></comment></comment><comment><USER10> Cool, you can buy CC's there using bitcoin (also Western Union, MoneyGram, and Lesspay). Explains why you can't do the reverse (buy bitcoin with a CC).

<comment><USER11> Bitcoin has always been a popular choice with black marketplaces far before Bitcoin was popular (think SR and the like), and only became more popular as LibertyReserve fell over.

However it isn't really used as a currency, and more as a transport mechanism for cash (seeing how Bitcoin isn't anonymous at all, I'd imagine the strongest reason for these guys to use bitcoin is you can't chargeback with bitcoin)</comment></comment>


Does Joining the S&P 500 Index Hurt Firms? - <AUTHOR> https://www.nber.org/papers/w27593 <comment><USER2> For more on this, do a search for "index inclusion effect".

A 2016, "On the Changes to the Index Inclusion Effect with Increasing Passive Investment Management":

> _This paper provides an empirical analysis of the index inclusion effect for > additions to the S &P 500 index between 1981 and 2015. The analysis finds > that between 1990 and 2015 the average excess return for additions from the > announcement to effective day was 5.64%. The analysis goes further by > exploring the average excess returns due to inclusion in each year._ […]

* [https://repository.upenn.edu/cgi/viewcontent.cgi?article=102...](https://repository.upenn.edu/cgi/viewcontent.cgi?article=1020&context=joseph_wharton_scholars)

NY Fed study (2011), "Is There an S&P 500 Index Effect?":

> _We find that the firms included in the S &P 500 index are characterized by > large increases in earnings, appreciation in market value, and positive > price momentum in the period preceding their index inclusion. This strong > preinclusion performance predicts 1) the permanent increase of market value > and 2) the change in return comovement, reflected in declines of size, > value, and momentum betas, following index inclusion. Nonevent firms with > similar performance experience similar appreciation in value and changes in > comovement coincident with the event firms. Contrary to the consensus in the > literature, our results indicate that – after accounting for the firms' > extraordinary preinclusion performance – index inclusion has_ no permanent > effect _on value and comovement._

* [https://www.newyorkfed.org/medialibrary/media/research/staff...](https://www.newyorkfed.org/medialibrary/media/research/staff_reports/sr484.pdf)</comment>


Economist behind Uber study admits error, adjusts numbers - <AUTHOR> https://twitter.com/StephenZoepf/status/970754550968676352/photo/1?ref_src=twsrc%5Etfw&ref_url=https%3A%2F%2Fwww.recode.net%2F2018%2F3%2F3%2F17074782%2Fuber-mit-study-less-than-4-hour-flawed <comment><USER2> Alex Tabarrok pointed to the well done study, that of course never got anywhere near this level of attention: [https://twitter.com/ATabarrok/status/969651065695227904](https://twitter.com/ATabarrok/status/969651065695227904)

"Dubious. Cited paper uses self-reported data from 1100 drivers. Cook et al. use exactly measured data on 1.8 million drivers and calculate average wage after expenses of $13 per hour. [https://web.stanford.edu/~diamondr/UberPayGap.pdf](https://web.stanford.edu/~diamondr/UberPayGap.pdf) "

<comment><USER3> It definitely does not look like you describe.

The original article is self-reported income, which i agree is crap and proof that it can be interpreted any way one wants, as is show in this thread: this is clearly PR damage-control from uber.

The one you linked is the other end of the spectrum. It is a math formula on values provided by uber and _completely ignores the associated costs_ with the profession. It does not account for down time, it does not account for insurance, car depreciation etc. The $13 is just more meaningless data because all it measured in the paper is "income from uber, taking into account avg on- meter time with pool+surge pricing variations". (they do include the costs, but as "The primary costs drivers face are fuel, maintenance, depreciation, and fines for parking or moving violations. Zoepf et al. (2017) estimate median driver's expenses are 32 cents per mile." which is very far from actual data)

IMHO both are utter garbage.

<comment><USER4> > does not account for down time, it does not account for insurance, car > depreciation etc

Most jobs have those costs. You need a car to get to work; you pay for gas, insurance, tolls, etc. Time is wasted during transit/commute.

Are drivers special in this respect?

<comment><USER5> Yes, insurance rates for ride share cars is higher because it's a commercial vehicle. And yes, most people commute an hour or 2 each day, not rack up hundreds of miles for 8+ hours a day. So yeah, the cost is not comparable.

<comment><USER6> > Yes, insurance rates for ride share cars is higher because it's a commercial > vehicle.

Do Uber/Lyft drivers typically make their activities known to their insurance companies?

<comment><USER7> State Farm had a warning in their recent mailer for policy renewals that you _are not_ covered under your auto policy for any ridesharing activities, although you could purchase a rider to insure you at additional cost.

<comment><USER8> Also, if you say 12 or 24 thousand miles a year and get into an accident with widely unexpected mileage on your car, they will suspect something is amiss and likely investigate.

<comment><USER7> State Farm also obtains your mileage data from "third party providers".

[https://splinternews.com/how-car-insurance-companies-spy- on-...](https://splinternews.com/how-car-insurance-companies-spy-on-your- mileage-1793860442)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> Do any of the naysayers ever actually talk to their uber/lyft drivers? I've never met a single one with bad things to say about the job. Most seem to enjoy the social aspect, and many foreigners take it as an opportunity to practice English while making some spare cash.

Who are you people to presume that these rational agents need to be restricted from earning money in a way that they deem convenient? Especially when their work is at will and unforced? Do you really feel that there is some kind of net good to the loss of work that comes with forcing uber to pay minimum wage, when it means that many content drivers will lose work?

<comment><USER10> Have you considered the incentives that they have when talking to a paying customer? Particularly one that is going to rate them based on the interaction?

Of course they "love their job"; they want to create a positive experience in the mind of someone who has a lot of power over them.

What I'm saying is that your anecdata has an observer effect that you are strongly discounting.

<comment><USER11> If I collected the same anecdata with Uber drivers I know in real life, not in the context of an uber ride, would the observer effect still apply?

<comment><USER12> Yes, lots of people brag about their jobs, life, see Facebook.

</comment></comment></comment></comment><comment><USER13> $8.55 is the profit but it is compared to minimum wage. Minimum wage is not profit as there are costs associated with getting to work (bus/car) and mandatory taxes.

Shouldn't the $8.55 be compared with net income from minimum wage? This seems like a meaningful comparison for people choosing to drive or work minimum wage.

<comment><USER14> This is a very good point; if you drive your car to work then you pay gas and depreciation just like an Uber driver, except you don't get to tax-deduct it.

<comment><USER15> Definitely, but "I commute to/from work" is a lot different than "I drive for 10 hours a day, every day".

<comment><USER13> It's certainly different, but I think it should be in the comparison.

I drive 10 hours a week, not per day. But there's a cost. To say "you make $10/hour" and an Uber driver makes $8.55 after expenses and taxes is not very useful because the Uber driver has more cash in pocket.

I think this is useful for an academic paper to compare apples to oranges. And the author directly compared this to minimum wage which seems like an amateur mistake for a researcher from MIT. Unless they have a policy angle they are promoting and are not a fair broker of info.

<comment><USER14> Agreed - if it was an apples-to-apples comparison we wouldn't need an academic paper.

</comment></comment></comment></comment></comment><comment><USER16> It's too late. Corrections don't really work in the social media age, two different interpretations of the same base facts won't propagate through the same path. I don't know whether the adjustments are valid or invalid, but $3.37 will be an incontrovertible fact for an influential swath of people for a long time.

<comment><USER17> Kind of makes me think of the divide in opinion on modern wage inequality. I'm still not sure whether to think it's purely because of women not being in as well paying positions or whether discrimination within the same job is a significant player, because I see conflicting reports.

<comment><USER18> I find very basic sniff tests tend to score relatively well at figuring out the truth on 'controversial' issues. So for instance this $3/hour earnings just did not make sense as there would rapidly approaching 0 drivers of > 'x months' experience driving for Uber, whereas drivers that have been doing it for years are hardly unicorns.

For the wage gap take as an assumption that companies were genuinely able to pay women less for doing the exact same quality and quantity of work as men. Well what would happen? We already live in a world where big companies relentlessly squeeze every penny they can from minimizing labor costs. The notion that people are intentionally paying more to men just to be around more men defies belief, to put it mildly.

And there's a corollary to this as well. As 'wage gap' entered the zeitgeist, there's no doubt that some companies, ever anxious to provide a bump to their next quarterly, did actively look to see if they could replace their male workers with females to save on labor costs. And the results of their research, in that no companies seem to be doing this, speaks for itself.

<comment><USER19> Controversial issues are controversial precisely because there's no "basic sniff test" that fits all the data. If you find you come to an easy conclusion on a controversial issue, you might not have all the information others do. To take your wage gap example, your explanation only works in the case where hiring managers know men and women are equally qualified, but choose to pay women less because they can get away with it. That's almost certainly not what's going on.

<comment><USER18> You're describing _complex_ issues, not _controversial._ Controversial issues tend to be controversial because people assume a conclusion, and then defend that to the death - facts be damned. When a sufficient number of people on either side of a topic engage in this behavior, suddenly we have a controversial issue.

So for instance on this case, I agree with you. I've no doubt women get paid less than men on average. And tall men get paid more than short men. And fat men get paid less than non-fat men. And I suspect that if we broke it down into hair color, you'd also find discrepancies there. Differences do not mean discrimination.

A problem would be if companies were unlawfully actively discriminating against people. As you yourself said, that is almost certainly not what's going on.

<comment><USER19> Disagreeing with you again. Controversy doesn't happen just because people ignore facts. It happens when the facts don't lead to an obvious conclusion, or when there aren't enough facts to go on, so intelligent people can reach different conclusions. When the only people on one side of a debate are obviously stupid or self-serving, we don't call it a controversy. We call it nonsense and make YouTube videos mocking it. (see also, Flat Earth)

> A problem would be if companies were unlawfully actively discriminating

It's a problem that companies discriminate, period, whether they're doing it intentionally or not. Overlooking skilled short people to hire and promote tall people means they're working against their own best interest and against society's interest, at the same time. (That doesn't mean I'm advocating for any particular solution, or that I think height should be a protected class. I'm just saying it is in fact a problem.)

<comment><USER18> Now you're getting into a touchier issue and making a very large assumption, I suspect without even realizing it. By suggesting that there is a problem you're indicating a belief that you believe all people are roughly identical. As I imagine you may be doing, let's avoid touchier issues and talk about height -- my examples were not off the cuff.

It's extremely difficult to try to objectively measure the competence of an individual in general, but IQ does tend to provide a workable measurement that does tend to be quite predictive and map well to aggregate performance in 'mental' fields. But we're talking about height, so why am I bringing this up? As counter intuitive as it is, height correlates strongly with IQ. We could discuss the possible reasons for this ( _environmental? sexual selection? shared genetic markers? something else?_ ) but the point is that this is not disputed. Here [1][2] are a couple of sources if you were not aware of this. You can find countless more on google scholar searching for 'height iq correlation'.

But this raises the issue that not only do tall people earn more than short people, but that it's highly likely that companies paying more to tall people are indeed working in their, and society's, best interest. The long and short here is that you cannot assume differences are a result of irrational decisions.

So you're left to try to argue that short people bring something to table that's valuable but less visible than their individual competence (at least in so much as IQ is reflective of that). This could be true, but I think that there's scant evidence for such. The first issue you'd need to overcome is the existential case. And I find the way most people try to do this is by getting back to assuming discrimination, yet now you're stuck in circular logic or having to assume your conclusion. _' All people are equal. What about these vast quantities of data showing substantial differences from ostensibly objective assessments? It's caused by environmental issues and discrimination. Why do you think that? Well because it doesn't show all people being equal!'_

And to be clear, everything I'm saying above applies to people considered as groups - not individuals. Einstein, for instance, was about 5'7". It just means there are different distributions among different groups, and so when you consider these groups as a whole you'd expect to see these different distributions reflected.

[1] - [http://journals.plos.org/plosgenetics/article?id=10.1371/jou...](http://journals.plos.org/plosgenetics/article?id=10.1371/journal.pgen.1003451)

[2] - [https://www.sciencedirect.com/science/article/pii/S105381191...](https://www.sciencedirect.com/science/article/pii/S1053811911010251)

</comment></comment></comment></comment></comment></comment></comment><comment><USER20> I feel somewhat vindicated now:

[https://news.ycombinator.com/item?id=16503558](https://news.ycombinator.com/item?id=16503558)

<comment><USER21> Sometimes we are too quick to agree with things that reinforce our beliefs. In this case that ride service platforms are squeezing out the last penny from their captive servants.

Do, hats off to you for doing some due diligence and doing some back of the napkin calculations proving a study incorrect in its assertion.

<comment><USER22> Well, while ~$9/hr is a lot better than ~$4, it's still well in the category of squeezing drivers, IMO.

In fact, if your goal is to squeeze your drivers, you actually don't want to squeeze too hard because soon you'll have no one left to squeeze. $9/hr is perhaps a level a number of people can (barely) subsist at (at least for a while) while $4/hr may be untenable for almost everyone.

(I agree, though, that drivers aren't like captive servants, as some seem to assume.)

<comment><USER20> FWIW I actually agree with you re: squeezing, just something about the numbers either way didn't seem to add up (others were quoting $13/hr, study was saying $3/hr).

</comment></comment></comment></comment><comment><USER23> Even with this revision the hourly wage is well below the $15.00 minimum wage here in Seattle. I wonder if after expenses Instacart and Amazon delivery drivers make any more than the typical Uber driver? I think contractors across all income levels need better protections. Laws should be enacted to guarantee after vehicle expenses that if one drives an Uber for an hour one makes at least a minimum wage.

<comment><USER24> If there are $15/hr jobs, why do you believe so many people are willing to accept half that to drive for Uber? I wonder what their total pay is after tips.

<comment><USER25> Since nobody seems to have any idea how much these things actually pay after costs (see: this correction and the original study), it's plausible that people would expect this to pay more than it does.

<comment><USER26> You think these people making $15/hr don't watch their actual take-home after expenses carefully?

<comment><USER24> The difficult thing to calculate in the short term is wear and tear on the vehicle. Average Joe is potentially only counting fuel and not total costs per mile.

<comment><USER26> Do you talk to Uber drivers?

<comment><USER24> No, but I worked at a pizza place for several years. Everyone was vaguely aware of the fact that they were using up car value delivering pizzas, but only a couple of older full time "career" drivers seemed to actually calculate it out so they knew what they were getting.

</comment></comment></comment></comment></comment></comment></comment><comment><USER27> This value seems to make a bit more "common sense" style sense - but it's still way too low for a "non-tip" job. Hell, it's probably even too low for a job with tips (which bugs me, but separate topic).

Given that Uber is most popular in the bigger cities where the cost of living is correspondingly higher, is it enough?

<comment><USER28> > it's still way too low for a "non-tip" job. Hell, it's probably even too low > for a job with tips

Tangential to your main point, but Uber does have tipping

</comment></comment><comment><USER29> This is still median profit from a sample group where > 80% worked less than 40 hours per week.

The median is someone working well under 40 hours per week.

I wish they would release the overall income distribution based for their model instead of just one number.

</comment><comment><USER30> So they still don't make minimum wage and some lose money.

</comment><comment><USER31> Definitely too low for a primary job. However, it might make sense for people to do as an extra 5-10 hours a week, or during peak festival times when the taxi system is overloaded. If a person needs to turn their extra time into money, but don't have enough extra time to get a second job, it could make sense. I'm not sure either Uber or Lyft could survive on those kind of drivers, though.

<comment><USER32> There may be some ways to make it work more consistently--though not in a way that Uber/Lyft wants. I've met drivers who mostly work 3am to 7am shifts, to catch back-to-back airport runs. The enterprising ones also actively work to get Uber/Lyft out of the process by offering their number directly.

<comment><USER13> "The enterprising ones also actively work to get Uber/Lyft out of the process by offering their number directly."

I do airport runs. A few have given me their card. At first I thought this was cool as they charged $25 instead of $30. But then the driver was unavailable a few times. Managing different cards with a hierarchy of backups is a pain for users.

I switched back to Uber. Interestingly the last one I used charged $35 which was more than Uber. I was already in the car so I paid.

People underestimate the service Uber makes in managing riders and drivers. I don't want to "have a guy" who's not dependable. Perhaps one day, I can afford a full time valet and driver. That day is not near.

<comment><USER33> I don't see this working for Uber/Lyft drivers. For "gigs" like tutoring, it's easy to get a student or parent to split the referral service's fee, because they do more or less nothing after the first session, and a personal relationship is valuable. For car services, you just want to get from A to B at some random time, and Joe Driver may not be near A at that time.

</comment></comment></comment></comment><comment><USER34> Previous threads (on the original story): [https://news.ycombinator.com/item?id=16498551](https://news.ycombinator.com/item?id=16498551), [https://news.ycombinator.com/item?id=16501017](https://news.ycombinator.com/item?id=16501017)

</comment><comment><USER35> Underwhelming correction from the author/economist.

</comment><comment><USER36> Sad that the correction has fewer upvotes than both of the HN threads with the original dubious statistic:

[https://news.ycombinator.com/item?id=16498551](https://news.ycombinator.com/item?id=16498551)

[https://news.ycombinator.com/item?id=16501017](https://news.ycombinator.com/item?id=16501017)

I think this speaks volumes about HN's user base being far less enlightened and fact-oriented than we like to think. Instead it seems that the articles reinforcing our biases are the ones we pay the most attention to.

</comment><comment><USER37> According to the new study the driver make between 8-10 bucks an hour. I guess its better than 3 dollars, but that still isn't a livable wage in America.

<comment><USER38> Would you prefer earning $10 with uber where you set your hours and have freedom or earning $10 at Walmart when you still have to work when your kid is sick

<comment><USER39> No.

</comment></comment></comment><comment><USER40> Uh huh. Does this include commercial insurance for "driving people for money"? Because your personal insurance likely has a clause saying it does not cover commercial activity.

What this means: if you get into an accident while engaging commercial activity, you _aren 't_ insured. And your license is then revoked due to lack of insurance.

Fun times.

<comment><USER41> Uber provides insurance for drivers while they are driving for Uber: [https://www.uber.com/drive/insurance/](https://www.uber.com/drive/insurance/)

</comment></comment><comment><USER42> Either way is more than a grad student makes grading papers and carrying out research plans

</comment><comment><USER43> Raising two billion dollars without having a product in the market is a huge red flag.

Most major break throughs started from a very specific problem, that was addressed, got some market traction, and then refined over time.

Large up front investment without testing the market has been done before, but I can't really think of a single recent example where it went well.

Maybe perhaps SpaceX, but I'm not really familiar with their founding story and costs associated. But also consider that one is literally launching rockets into space, the other is creating a VR/AR headset.

Launching a single rocket costs millions! Buying this or any AR/VR headset will ultimately cost $1,000. Hence I don't see why you need $2B to get this off the ground.

And if they are having a problem shrinking the tech, well then it's not a real product. Because they aren't pitching here is a refrigerator sized computer and a headset, they are pitching here is a headset, so if you haven't shrunk it, then you really haven't built the MVP you were looking far as portability and comfort I'm sure are part of the essential requirements for this product.

<comment><USER44> Commenting on wrong post maybe?

<comment><USER45> Think so.</comment></comment></comment>


Bedrock Linux 1.0beta2 Nyla Released - <AUTHOR> http://bedrocklinux.org/index.html <comment><AUTHOR> I'm pretty excited for this project. Other things that have made this pleased / excited in the past:

* Rust

* Consistent key-bindings in a unix like environment (Mac OSX coming from linux)

* Arch Linux (it was fun to tinker with)

* And many other things I'm sure... they just don't come to mind at the moment.

Does anyone else know of any similar projects? I think between this and GUIX I'd be happy in linux land no matter what breaking changes come along...</comment>


New Ebola outbreak in DRC is 'truly frightening', says Wellcome Trust director - <AUTHOR> https://www.bbc.com/news/world-africa-48615667 <comment><USER2> Humanity is actually doing pretty well this time compared to last time at West Africa. This time we have working vaccine and are using it with abandon.

<comment><USER3> Our only remaining predators are microscopic...</comment></comment>