How I Store My 1's and 0′s: ZFS + Bargain HP Microserver = Joy - <AUTHOR> http://mocko.org.uk/b/2012/06/17/how-i-store-my-1s-and-0s-zfs-bargain-hp-microserver-joy/ <comment><USER3> I have this same hardware. A few notes:

\- For best performance with ZFS, you want a lot of RAM, and this unit will take 8GB of ECC RAM. You want ECC for data integrity in memory, as ZFS does nothing to prevent in-memory data corruption (there's an article on this here [pdf]: [http://research.cs.wisc.edu/wind/Publications/zfs- corruption...](http://research.cs.wisc.edu/wind/Publications/zfs-corruption- fast10.pdf) )

\- You probably want a few mirrors, not RAID-Z, if performance is an issue.

\- You're better off with FreeBSD or Illumos kerneled distros (which have run ZFS for years, and have it in their mainline kernels), rather than Linux (which never will have ZFS in mainline for licensing reasons), for stability alone.

\- You can get an IPMI card for this unit if you want remote manageability.

\- There's an internal USB port if you want to boot off of that. It's kind of handy.

<comment><USER4> Good points - people do need to remember that ZFS was designed for going big. IT does nothing to prevent memory corruption by design.... it was assumed (required?) that your server would use ECC ram.

You also need to ensure you have adequate ram for various cacahing, and CPU power as well for checksum calculation, among other things. It's not a lightweight FS. (in-memory corruption is an issue with all filesysems - <USER2> somewhat more-so with ZFS because it was designed specifically to assume you had reliable ram. You want ram to store the hash cache or whatever too....

One could chuck an SSD in there for cache, if memory is a limit, that should speed things up drastically.

And as with all raid-like systems - you want an appropriate number of hot- spares, cold spares, and a system that monitors it and acts appropriately, especially if you're going with huge drives on slow busses.

You want regularly scheduled scrubs, not too many snapshots, probably disable atime (noatime) to speed up those scrubs, and compression probably off....

dedup I'm still on the fence about - I leave it off, I can only see specific situations where it would be truly useful (dedup+verify)

<comment><USER3> Unless you're using enterprise class SSD's with SLC flash and ultracaps to guarantee that the writes make it in cases of power loss, you're risking your data if using this as ZIL.

Cheap consumer class SSD's are generally fine for L2ARC.

</comment></comment></comment><comment><USER5> I've been hearing awesome things about ZFS for years now. Unfortunately, it can never be part of the Linux kernel due to licensing issues, so we're stuck with "your 1's and 0's are being held by a pre-1.0 version of a filesystem invented by a dead company". How far has btrfs come in its support of ZFS-like features? Still need a few more years? I'll be switching as soon as it's marked stable.

I wonder if the performance might be better if a good 16GB USB stick was used for the OS drive instead of an old laptop drive? The OS needs a lot of random access, but doesn't take up much space.

I also wonder why the author went with Ubuntu 10.04 LTS instead of 12.04 LTS, which would give him two more years of peace of mind. It's been a few weeks since 12.04 came out, so it's pretty stable. It does get kernel updates more often that I'd prefer, though, and GNOME 2 is gone.

<comment><USER6> I find your assessment of the Linux filesystem situation to be inaccurately, and surprisingly, negative. "Stuck with" implies little potential for change, when there are several filesystems which are being developed at a swift pace, solving tough problems with vigor and ingenuity. btrfs is coming along just fine. I can't think of any place with more filesystem development going on than Linux.

I mean, let's just not leave it at idle claims:

btrfs changes in 3.4: [http://kernelnewbies.org/LinuxChanges#head-556161b206bf626d6...](http://kernelnewbies.org/LinuxChanges#head-556161b206bf626d6c84f9973dbdc3c8ef15bd07) 3.3: [http://kernelnewbies.org/Linux_3.3#head-1f03b4babafb1049bea3...](http://kernelnewbies.org/Linux_3.3#head-1f03b4babafb1049bea35793c2c0fb91fae48cd4) 3.2: [http://kernelnewbies.org/Linux_3.2#head-f0a922e9c0ce6f48810d...](http://kernelnewbies.org/Linux_3.2#head-f0a922e9c0ce6f48810dbe204f89a69eab8034eb) 3.0: [http://kernelnewbies.org/Linux_3.0#head-3e596e03408e1d32a7cc...](http://kernelnewbies.org/Linux_3.0#head-3e596e03408e1d32a7cc381d6f54e87feee22ee4)

RAID5/6 barely missed 3.5, but should be in the pull request for 3.6 (other changes for 3.5: <https://lkml.org/lkml/2012/6/1/160>).

What's going on in XFS: <http://lwn.net/Articles/476263/>

Coverage from the 2012 filesystem/storage summit, day 1: <http://lwn.net/Articles/490114/> Day 2: <http://lwn.net/Articles/490501/>

2011 summit, day 1: <http://lwn.net/Articles/436871/> Day 2: <http://lwn.net/Articles/437066/>

Ext4 is still pretty active for a "done" filesystem, too: The last 12 months saw work on online resizing, support for bigger block sizes, a cleanup of mount options, ...

<comment><USER5> Sorry if I came across as negative. The "stuck with" was a reference to ZFS's status as a pre-1.0 PPA, not a reference to Linux filesystems as a whole.

On the other hand, the only in-kernel Linux filesystem that can match ZFS's feature set (for example, resizing a RAID array while the filesystem is online) seems to be btrfs, which probably won't be marked stable for at least another year or two. The latest developments to XFS and ext4, though interesting, aren't particularly relevant if you're looking to build a server like what the article describes.

Still, thanks for the many interesting links!

<comment><USER7> I'll just note that it is the implementation of ZFS that is pre-1.0, not the filesystem itself. The ZFS filesystem itself has been production-ready for several years now. That said, I might use the current Linux implementation for my non-critical data, but for anything important I'd stick to Solaris/OpenIndiana. There's also a decent implementation on FreeBSD, but I'm not a fan of that OS.

<comment><USER8> All the bugs happen in implementation.

<comment><USER7> Agreed. My point was simply that you shouldn't write off the filesystem entirely just because some of the implementations aren't mature.

</comment></comment></comment></comment></comment></comment><comment><USER9> Agreed, ZFS rocks.

I started using OpenSolaris fileservers at home in 2007. I went through multiple upgrades since ten: 5x500GB, 7x750GB, and currently 7x1.5TB (all raidz).

I am just about to upgrade to my 4th server with 6x3TB in raidz2, which will be running FreeBSD this time.

Due to the sheer number of drives and continuous 24/7 operation, I experienced 4 drive failures over the years. Everytime ZFS has handled the failure gracefully, and I was able to swap the drive and rebuild the array without a hitch.

I take daily rotating snapshots - incredibly useful when you accidentally stuff.

I also run weekly scrubs, which have allowed me to witness 2 or 3 silent data corruptions which were automatically self-healed by ZFS (CKSUM column in zpool status).

I mostly use the file server to share videos & photos via NFS, and to store encrypted backups of my laptop. It has become so useful and practical that I started to use it as an iSCSI server as well to boot some diskless Windows test machines for GPU projects.

All in all, ZFS deserves all the praise you hear.

</comment><comment><USER10> I used to build my own file servers but 2-3 years ago I bought a ReadyNAS ProBusiness at home and I love it. Granted it cost me about $1500 at the time, but it has made my life so much easier. I'm at the stage of my life where I'd rather pay extra and save time.

It supports almost everything out of the box, and there's very little configuration. It has 6 hot-swappable bays, and it allows for automatic expansion using their proprietary system, X-RAID 2. I currently have 4 500 GB drives, and 2 1 TB drives, and if I want to expand it, I just buy another 1 TB drive and swap out a 500 GB drive.

It also supports streaming protocols, including ReadyDLNA so I can play movies directly off my PS3. It also seamless supports TimeMachine for my Mac laptops. I really do love this thing.

<comment><USER11> How do you back it up?

<comment><USER12> Not sure about the ReadyNAS ProBusiness, but my NV+ has a USB port that I can plug an external drive in to, push the backup button, and in just a little bit I'll have a drive suitable for taking off-site. The FrontView web interface has a simple way to configure what the backup button does. Very simple.

It also runs Linux under the hood, and I've been able to configure CrashPlan for off-site backups of some key files too.

<comment><USER10> I use the USB port as well and connect a USB drive to it. But I am a bit more paranoid, and I do a bit-by-bit comparison of every file that I copy, after I do the backup.

</comment></comment></comment></comment><comment><USER13> I have about the same setup, but i built my server from scratch have 6 _2TB and I run FreeBSD 9 (plus SSD for os). OS reliability wise and ZFS maturity FreeBSD > Ubuntu. You'll have virtualisation through virtualbox if you'd like to too. But I prefer to separate the storage and virtualisation platforms (do one thing etc). In some ways _BSD is easier and more logical in it's setup and administration, plus you'll have better documentation on the site and higher signal-to-noise ratio in the forums, if you need help.

<comment><USER14> Despite being a heavy computer user I'm positively struggling to fill 1TB even after 2 years of hoarding HD rips, so I'd love to know what you use 62TB for. Also wondering about idle power consumption.

<comment><USER13> bah, formatting error. 6 times 2TB not 62TB. Everything runs of an atom card, with modded cooling for the bridge and cpu (fan less, really awesome, copper heat sinks). The discs are the WD Caviar Green, low power, and they are attached with rubber to silence them. There are two (90 cm) slow spinning fans in front of the disks, one in the quitet zalman psu (120 cm) and one (90 cm) at the back. I built it in an htpc chassis.

</comment></comment></comment><comment><USER15> Serious question.. does no one use SmartOS[1] for this? I wouldn't feel entirely secure running ZFS on Linux when I could just as easily run SmartOS and get the "real" ZFS.

[1] <http://smartos.org/>

<comment><USER3> I run OpenIndiana (and OpenSolaris before that) on mine, and I tend to agree - Linux ZFS is unlikely to ever be as mature as either the Illumos kerneled distros (OpenIndiana, Nexenta, SmartOS), or FreeBSD (and derived systems like FreeNAS), both of which ship ZFS in the kernel as a primary filesystem.

<comment><USER16> How many of the Illumos or FreeBSD developers are working on ZFS? I'm curious if those implementations actually have any more manpower than the ZFS on Linux project.

<comment><USER17> The companies that are betting on ZFS -- Delphix, Nexenta and Joyent (and a bunch more that are less public about their work) -- are overwhelmingly (indeed, exclusively, to the best of my knowledge) on illumos and FreeBSD. Of these, Delphix in particular is of note because of the original ZFS core team members working there: Matt Ahrens (the co-inventor of ZFS), Eric Schrock and George Wilson -- not to mention important ZFS contributors like Adam Leventhal and Chris Siden.[1]

So illumos remains the repository of record for ZFS -- with a close relationship with those working on ZFS on FreeBSD. While the Linux port is certainly a Good Thing, it does not reflect shift in the epicenter of ZFS development...

[1] <http://www.youtube.com/watch?v=-zRN7XLCRhc#t=43m54s>

</comment></comment></comment></comment><comment><USER18> This is almost identical to my backup server except I have 3x2TB drives in a RAID-Z pool. I agree with all the author's "reasons this is awesome" except my #1 reason is data integrity.

ZFS with RAID-Z does block-level checksumming and automatic healing as you access your data. Combine that with a weekly scrub (touches every block so any silent bit flips are healed) and I can do away with my fears that my precious bits are rotting away.

<comment><USER19> ZFS uses block-level checksumming for every block. The auto healing can be used when there is a second copy or another way to rebuild that block, read Raid-Z. So you get the auto healing already when using mirror.

<comment><USER4> as long as you have ECC ram. If not, you're at increased risk than most other filesystems (which are also at risk) - ZFS was designed with the requirement that RAM is reliable.

</comment></comment></comment><comment><USER20> I have been using FreeNAS, which is a slimmed down version of FreeBSD meant to run off of a USB stick. It has a web interface to set up and manage ZFS, and can take regular ZFS snapshots, among other things. It also includes Netatalk and other software to share the disks over the local network.

I use it as a Time Capsule for my MacBook. Hooked up to gigabit ethernet when docked, backups are a breeze.

<comment><USER21> Ditto - FreeNAS is FTW. A new RC of 8.2 was just released in the past week and it's awesome. The web interface was moved over to Django with the 8.x line from 7.x. It's minimal, but feature rich and "just works".

For my home setup I actually run FreeNAS in an ESXi environment. I have 4 different physical disks that I carve up space on and allocate to the FreeNAS VM. This allows me to snapshot upgrades on the base OS and when I'm testing an upgrade I can disconnect the ZFS pool - validate the upgrade went fine, and then reconnect the pool for a ZFS upgrade (if needed). The nice thing about this approach is you can physically move around system very easily if all you need to do is ship out the disk store - and since I have one beefy box for virtualization at home I have my storage system contained within which makes power and space a bit more efficient.

My suggestion to those who are considering this is if you stick about $500-$1000 into a BYOD system you can generally get a high end quad core system with 32GB of RAM and 3-5TB of disk space (with an SSD boot). At that rate I would carve up a few TB for backup and SAN (FreeNAS) and the rest would be for on-box VMs. The FreeNAS VM doesn't need more than 1 proc and about 2-4GB of RAM if you're dealing with a lot of file transfer. You can easily get away with 2GB.

Long story short: ESXi + FreeNAS = 1-box solution for most at home geeks. My motivation was that I was starting to have "box sprawl" and power consumption was getting a bit out of hand. I also run PFSense on my box as well - but in that regard I also have a low power physical system that acts as the primary gateway device in my network. But there's a PFSense running as a VM as well for failover when I do upgrades. Far better than any SOHO gear you can buy for far too much $$$.

</comment></comment><comment><USER22> I have a couple of the HP ProLiant MicroServers, one of which is set up as a NAS server with a RAID-Z array on 5x 2TB drives (running Oracle Solaris 11 Express in order to remain at the cutting edge of ZFS development).

At ~£150 after rebate (was around £120 when I bought mine) the MicroServer is an absolute steal, and ZFS is a dream to administer. Truly a match made in heaven.

</comment><comment><USER23> The ability to add new, differently sized disks with RAID-Z is the killer feature of ZFS. I wonder how Linux ZFS performance & stability compare with the FreeBSD ports? In the past, I've considered using something like FreeNAS for my home storage needs but the ZFS support wasn't ready last time I looked (1+ years ago?).

<comment><USER24> I also wonder the same. I was considering using BTRFS because it's better integrated, but if ZFS is more stable/mature, I'd go with that without even thinking. Has anyone used it for some amount of time?

<comment><USER25> I have used zfs-native on Ubuntu and Fedora a year or so ago for less than a month and found it to be unusable at best - I couldn't even copy my data from source to the backup ZFS disks - just went into a loop with high CPU. That may have changed a bit with later releases but I just don't think getting ZFS to scale and run reliably on different OS is going to happen anytime soon given how much effort and skills it would take.

What I am looking at doing is getting/building Solaris compatible box for my backup needs - that is a daunting task. But if I could do that I can run one of the OSS variants of Solaris - Joyent SmartOS, Nexenta etc..

<comment><USER24> Ah, there go my hopes, dashed. Thanks for the input, I'll try it on a disk I don't need and see if it's unusable, thanks again.

<comment><USER26> I'm currently running a backup box nearly identical to the OP, and haven't had a problem yet (built it over a year ago). ZFS runs like a dream on Ubuntu, even with a slew of oddly sized disks (1TB + 2x2TB + 3TB) at 90% capacity. I've had my SATA card come loose, and ZFS just locks down the FS to r/o so no damage is done. And unlike many other file systems, ZFS's checking utility actually gives you human readable results if there is an error (eg. "/foo/bar is corrupt", not just cryptic messages), and outputs exactly what you should do to repair data.

I recommend at least trying ZFS out in a VM, I guarantee you'll be impressed by the versatility.

<comment><USER24> I'll definitely give it a shot, thanks. Does anyone know what btrfs lacks compared to zfs, other than stability?

<comment><USER27> For a file system that is a big 'other than'.

<comment><USER24> Sure, but I already know ZFS is stable and btrfs is less so.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER28> From the article: "Why not Debian or CentOS? Cool, go that way if you prefer them. But personally I am in luuuuuurve with the Ubuntu ZFS PPA."

With Debian, you can use the PPA as-is. This requires adding that to your /etc/apt/sources.list and manually adding the signing key with apt-key.

Something else the author doesn't directly address is that ZFS on Linux is really only usable on 64-bit systems. Funny things may happen if you use the 32-bit version, such as OOPSing when doing simple things such as ls -a.

I've had nothing but great experiences with running this on my home NAS.

<comment><USER29> Note that the 32bit thing isn't completely a Linux issue. ZFS seems pretty much designed with 64bit in mind.

On the Freebsd side, there is whole section on tuning [0] for i386 users. Some of it might transpose to linux (at least the concepts and things to watch).

[0] <http://wiki.freebsd.org/ZFSTuningGuide>

</comment></comment><comment><USER30> The last non-server HP products I have purchased have died. A desktop for my mother, a laptop for the inlaws, and a small PC for myself.

I am terrified by the quality of their non high-end server gear.

<comment><USER31> Great to know that someone else shares this apprehension. I cannot believe how callous HP is in developing their consumer products.

It first started with our HP laptop that died multiple times over the warranty period. And then some friend's machines as well .. the central issue in almost all the cases : bad thermal management.

<comment><USER32> Mine runs hot, but it's going strong after almost three years, even though it has an AMD CPU with a particularly high TPD compared to others in the same segment (Atoms).

</comment></comment></comment><comment><USER33> Just a quick note on those HP microservers - I have one, and I'm fond of it, but the e-SATA port doesn't support hotplug and the inbuilt ethernet doesn't support jumbo frames (at least, under versions of Linux that I've tried).

The e-SATA thing probably doesn't matter too much, but for anyone looking to run iSCSI or even higher-throughput NFS, the absence of jumbo frames may be a more important consideration.

</comment><comment><USER34> As you scale up your storage, I'd recommend you switch away from RAID-Z to a pool of mirrors (essentially RAID10). It becomes easier to add or upgrade pairs of disks with differing capacities (eg, a pair of 1TB, a pair of 2TB...), and in the event of failure you have more than a snowball's chance of being able to rebuild the array before you have another disk go.

<comment><USER35> I bet a snowball _would_ have difficulty rebuilding the array before another disk went. I like your version better.

</comment></comment><comment><USER36> Love ZFS, hate having to use it in FreeBSD or Illumos, so I like seeing someone successfully using it in Linux even if licensing concerns keep me from doing it. When I saw that he was using "WD Caviar Green" hard drives, though, I cringed.

These "green" hard drives tend to be very aggressive about parking heads and spinning down the platters. While this is fine if a disk is going to sit idle for a long period of time, in cases like an OS partition and memory buffering, these drives start destroying themselves spinning down and cranking up several times a minute.

We had 4 out of 16 fail one week after burn-in in a raidz configuration. We had plenty of hot spares for various reasons of paranoia, and they didn't go all at once, so we recovered and replaced the entire batch with Constellations.

The only positive note was that we are now firmly in love with zpools and zfs.. It made egregious hardware failure a manageable problem.

<comment><USER37> Interesting.

Can you please point me to a more authoritative source for this? I also had a "green" drive fail surprisingly soon. How did you confirm that they spin up and down "several times a minute" by themselves. Are you sure it wasn't your misconfigured OS telling them to do that?

Since green drives are also slower, I just assumed they traded off energy for data transfer speed.

<comment><USER36> You can look for yourself using smartctl -- they still respond after the mechanical failure. Aside from pasting internal emails (not happening :)), you'll have to google for yourself on this one. We didn't do anything novel with these drives at a hardware level to make them fail, and we've got dozens of Constellations in an identical configuration that have had no failures after months of load.

It's not transfer speed that they seem to trade off for, it's access times. That may have been part of the problem -- our reads are extremely random over a very broad range of sectors. This isn't exactly a log or video server usage model. :)

</comment></comment></comment><comment><USER38> We have almost the exact same setup except we use FreeBSD instead of Ubuntu -- mostly because when I setup this server, it was before ZFS support was reliable in Linux.

I can't remember the exact specs on our box, but it was a similar microserver that we've got 4 drives RAIDed in. It started at 1tb per drive but we've since upgraded to 2TB drives.

This replaced a FreeNAS setup that I ran out of the closet in my home office for years when I lived in Atlanta. That server was great but was loud, ran so hot the closet was seriously 20 degrees hotter than the office, and an electricity glutton. When we moved to New York City last year, we decided to consolidate to a small unit for size/heat/power.

Highly reccommended to anyone who needs a media server, general file server and fast-access VM/local enviornment.

ZFS is absolutely the way to go.

</comment><comment><USER39> I've been using an unRAID setup (on an old workstation with plenty of bays) for a couple years now and I've been happy so far. No failures yet though so I haven't really put it to the test. The HP microserver definitely looks nice.

I just use it for storage though. No VMs etc. And I'm not overly concerned about access speeds so long as I can play HD video off it (which I can).

<http://lime-technology.com/>

</comment><comment><USER40> Be extremely cautious when using WD Green drives in a RAID configuration.

They may be quiet/inexpensive, but they also don't ship with Time-Limited Error Recovery enabled (<http://en.wikipedia.org/wiki/Time- Limited_Error_Recovery>). This makes it much more likely they'll drop out of an array. Some models can have this option flipped on; some cannot.

<comment><USER37> Thanks a lot! This is the most useful thing I've read in this thread. I own two green drives.

I installed two Caviar Black (b/c of the 5-yr warranty) drives in a RAID array a few months ago. The RAID edition drives aren't even significantly more expensive. I wish I'd known this earlier.

This is a bummer. [http://wdc.custhelp.com/app/answers/detail/a_id/1397/p/227,2...](http://wdc.custhelp.com/app/answers/detail/a_id/1397/p/227,283/session/L3RpbWUvMTMyMTQzOTc4NS9zaWQvdVhvYmpmSms%3D)

Edit: But on another page, WD says that Caviar drives can be used in consumer RAID configs. Hmm. [http://wdc.custhelp.com/app/answers/detail/a_id/996/related/...](http://wdc.custhelp.com/app/answers/detail/a_id/996/related/1/session/L2F2LzEvdGltZS8xMzQwMDM1MzExL3NpZC9yZFJZaC0taw%3D%3D)

</comment></comment><comment> X-Istence I've got a similar setup, but a custom build. 2 IDE hard drives in zmirror, this is my OS install, I am running OpenIndiana (I absolutely love the stability of the OS) and then 5 hard drives in a raidz with a 6th drive sitting in stand by (when i set this up raidz2 wasn't available yet) to automatically take over in case of a failure.

This machine has now been chugging along for a long time. It stores about 4 TB of personal backups (all my machines back up to it over the network), and various other things such as projects, media files, photos. ZFS is rock solid. I've had drives fail, and the backup drive take over without noticing a single thing.

I've got 4 GB of memory in this machine and I can get write speeds over the network of 80 MB/sec using consumer grade drives, and read speeds over the network of around 120 MB/sec (I easily saturate my Gbit network).

I wouldn't store my backup bits on any other file system, I've had failures with various Linux based raids/file systems that were nonrecoverable, I've used UFS in the past from FreeBSD and had data be silently corrupted, end to end checksumming is absolutely fantastic!

<comment><USER37> What is the network file system to make ZFS available to your other computers? NFS and Samba?

</comment></comment><comment><USER41> I grabbed a HP Microserver a few months ago after my old slow NAS died. One of the best purchases i've made this year.

Initially i tried using Ubuntu Server on it, but there were a few problems with it. I also tried Freenas but beyond the basic "share files and use ZFS" it didn't really offer much in the way of customisation.

So Instead i decided to just put Windows Home Server on it, since all i wanted to do was share files, use basic RAID, and run virtual machines for testing with the minimum of fuss. Windows RDC works fantastically out of the box. For VMs i just used Virtualbox. I stuck Freenas in a VM and left it running for Time Machine - perfect.

I also stuck an SSD i had lying around in the optical bay, so i have 4 drives available in the bays dedicated to storing data.

Despite not using Linux or ZFS, i'm quite happy with this current setup.

</comment><comment><USER42> > Normally we fear expanding a home server because resizing a RAID always > means copying terabytes of data off somewhere else, wiping the lot and > constructing a new one including the original disks.

This is just wrong. Only last week I resized my RAID5 array from 2 drives (basically RAID1) to 4 drives. Reshaping and resizing the ext4 partition was all done online with minimal performance impact. The only bit that took a long time was the reshaping, and even then it managed to add a new 2TB drive in only a few hours.

Even upgrading the entire array to use a larger size of disk is pretty easy. You do need all the disks to be the same size though :(

<comment><USER43> It's also wrong on the opposite front: ZFS cannot change geometries on the fly. I am a fairly big fan of ZFS, but the ability to mix disk sizes and change numbers of disks in a RAIDZ set is not why--it effectively cannot do either.

</comment></comment><comment><USER44> Best takeaway for me from this: RAID-Z on Linux is now ready. Good to know!

</comment><comment><USER45> Cool. I plan on doing something similar in future. One thing though:

Running the FS off a USB stick is a bad idea. No write leveling so its only a question of time before it bombs out.

<comment><USER46> Mount it read-only, put temporary files into ramdisks.

<comment><USER47> Sadly, flash also suffers from read-disturbs. USB sticks can die just from being read a lot.

Plus, sometimes you want logs for troubleshooting... small SSDs are cheap.

</comment></comment></comment><comment><USER48> Well, being able to send incremental backups of ZFS tanks between systems is IMHO one of the best features of the system. Also, snapshots are very handy.

</comment><comment><USER49> Does anyone have recommendations for similar hardware with 5 or 6 drives?

On a different note, the number of possible distributions is a tad confusing. Can someone recommend (say FreeNAS or Illumos) the distribution with the most up to date support of ZFS along with ongoing updates? I'm primarily looking for something that is hassle free to maintain and update. FreeBSD or Solaris based is okay.

<comment><USER50> Not sure if money's an option for you, but I don't think you'll find anything with 5-6 drives at this price. At AUD $280, it's cheaper than a lot of 2-drive NAS' out there.

</comment></comment><comment><USER51> I'm biased (I started Amahi), but try the Amahi server <http://www.amahi.org>

We looked into using ZFS, as there has been some demand for it, but all the licensing and the people around it were hard to deal with (non-responsive, to be precise). I would be cool, though.

<comment><USER9> Expand the _first_ mention of the HDA acronym on the homepage.

You could have based your product on FreeBSD to use ZFS. There would have been no licensing issue or people to deal with since the BSD license allows modification and redistribution.

</comment></comment><comment><USER51> Oh, and BTW, there is also Greyhole, which we integrated in Amahi. It makes a large redundant store out of a JBOD, with replication on multiple selectable spindles, etc. etc. <http://greyhole.net>

</comment><comment><USER52> I am looking to replace my Acer Windows Home Server box with something else, so this article is very timely. Thanks for posting it.

Is the HP Microserver the best computer in its class, or are there other good competitors?

</comment><comment><USER53> I've got 12TB in my unRAID box which has been working perfect for me for a couple of years now.

<http://lime-technology.com/>

</comment><comment><USER54> That's interesting, I've got two QNAP TS-419p boxes running Debian on armel. Had no idea that similar, AND x64-64 based hardware was available for much cheaper!

</comment><comment><USER55> nas4free. FreeBSD 9 based fork.

</comment><comment><USER56> I shall stick to XFS until BTRFS has made enough improvements. That's right everyone XFS has been here for a long time and doing great in production.

</comment><comment><USER57> Am I the only one having trouble reading the font of the post? On a 24 inch monitor running at 1920x1080 W7 Chrome its not exactly easy to read.

</comment><comment><USER58> What surprised me when I first started messing with home file servers was that even with the server plugged into the WiFi router's switch I still couldn't directly play high-resolution video on my laptops. You end up having to try to get some clunky streaming transcoding solution which only works for some file formats and requires a heavy duty CPU.

<comment><USER9> That's because your laptop's wireless connection quality is sub-par. The fileserver might be on the LAN, but the laptop's wireless is going to be your bottleneck.

</comment></comment><comment><USER59> So, just how stable is ZFS on Linux these days? Anyone else with experience care to share?

</comment><comment><USER60> I've heard that the perormance of linux zfs is terrible, but I think it was a phoronix report...

Also, does anyone know how well dtrace was ported to linux? Last thing I remember reading said it was half-assed

<comment><USER61> The old FUSE driver used to perform pretty poorly but I havent used it in a while.

I've been using the native linux kernel driver (obv not in-tree but easy to install) and it's fantastic, I see no slowdowns at all. Whatever performance hit is there if any is worth the benefits :)

</comment></comment><comment><USER62> Wow. This looks like an incredible deal. After planning to buy a new ultrabook soon, I'm thinking of ditching my monsterous desktop and going back to a laptop + server setup. This looks perfect!

... No international shipping? I'm gonna cry.

</comment><comment><USER8> No redundant power supply?

</comment><comment><USER63> Rabiyh

</comment><comment><USER64> Incredibly annoying blue lines...

</comment><comment><USER65> Please stop using fonts with serifs for the web. It's a strain on the eye. Great article, though.</comment>


HyperKitty: Django web interface to access Mailman archives - <AUTHOR> https://gitlab.com/mailman/hyperkitty <comment><USER2> As a guy who has been involved in supporting python.org and the Mailman project since 2003, I am super happy to see someone else post a link here about Mailman! See also [http://www.list.org/](http://www.list.org/)</comment>


Ask HN: Engineering Community to Create Ventilators? - <AUTHOR> Does anyone here know of any COVID-19 or coronavirus related community, forum, Slack channel that has many engineers?
My motivation:
Currently, the virus shows signs of spreading across the US at a speed that will outstrip many medical resources.
One key resource is ventilators, the mechanical breathing systems that you've probably seen before. These ventilators will probably be in critically short supply.
Importantly, these mechanical ventilators actually amount to a face mask and a mechanical bellows. Their construction and performance requirements are relatively simple. With some time, they could be built by anyone with engineering ability, which the US and Canada has a huge pool of talent.
I want to see if there is a community that could consider organizing the production of these systems. <comment><USER2> For a emergency DIY purposes you could buy manual resuscitators and build a device that presses them at right intervals (they are self-inflating).</comment>


What is Turkey's problem with Darwin? - <AUTHOR> http://al-monitor.com/pulse/originals/2017/01/turkey-what-is-turks-problem-with-darwin.html <comment><USER2> Sad but not surprising. Looking at the Turkish society overall it seems to be a regression to the mean. Even Gülen - the new designated enemy, former best friend of the AKP regime - had a book written about how wrong evolution is. And that guy is is a champion of education.

Turkish civil society unfortunately never got strong enough to counter these cranks. Always torn apart by militarists, communists, and now islamists again. edit: oops forgot to mention the ethnic nationalists, they deserve special mention, fuckers.

<comment><USER3> I have to give it to the militarists though: they were certainly secular. When the long-overdue recent military coup failed (or didn't happen, depending on who you ask), it seems to me the flood walls just collapsed irreversibly. Fundamentalist faith is on the rise worldwide, and Turkey has especially been under attack from within and without for all of its history.

The country just feels lost at this point. It wasn't so long ago, Turkey was considered an important ally of the EU, on track to become a member state.

Being German myself, we used to have a special relationship with that country. People forget nowadays that there are so many originally-Turkish people in Germany now because at one point the influx of Turkish workers was a welcome addition to the German economy.

I often wonder about the parents of an ex-girlfriend, who after working their asses off in Germany for their entire lives, moved back to Turkey for retirement - for much the same reasons pensioners in the US move to Florida. They were a friendly, modern, secular couple. I wonder what situation they find themselves in now.

<comment><USER4> I wish secularists would do a little more soul searching on the question of why fundamentalism is on the rise at all.

Personally I think it's pretty simple: fundamentalism offers a sense of community and belonging. In the age of loneliness that's pretty compelling. Secularism offers knowledge and information but nothing human -- no community, no identity, no sense of place or culture. (I can't remember where I first heard "age of loneliness" but I think it's accurate.)

<comment><USER3> _> fundamentalism offers a sense of community and belonging_

That's certainly an interesting point. I must admit I struggle a bit with understanding where these people are coming from, because as an atheist I don't personally experience a lack of community and belonging. I would imagine a forced "sense" of community being prescribed at gunpoint to be quite alienating, but I've heard enough people asserting it to take it seriously.

Maybe fundamentalist islam is just a meme whose time has come. Maybe the mindless, cruel herd is fundamentally more appealing to humans than individuality and free thought. In that case, we're not just seeing the pendulum swing here, we're seeing the transition into a new civilization: devoid of science and insight of any kind, yet wielding the tools of technology for maximum devastation.

<comment><USER4> I agree to an extent but I still see judgement here "mindless, cruel herd is fundamentally more appealing to humans than individuality and free thought".

Huge numbers of people today have _no_ friends and _no_ community. Read a book called "Bowling Alone" for an America-centric view of the problem but it's actually global.

<comment><USER3> _> but I still see judgement here_

That judgement is absolutely intentional. How can I not judge the abject surrender to herd mentality in particular, and fundamentalist belief in general? It's bidirectional, by the way, fundamentalists have no problem judging (and doing worse to) me for refusing to submit.

 _> Huge numbers of people today have no friends and no community._

To be fair, the rational answer might not be to join the Borg.

It's debatable how much say common people even have in the matter when their whole region converts to religious governance. This leaves us with the people who do choose it for themselves, and they're faced with a non-trivial question: "do I agree with these values or do I just join in order to embed myself into a community, any community?"

For those who agree with the fundamentalist message anyway, community is not the driving factor. The message itself is. Those who merely surrender to the message superficially in order to achieve community will probably become disenfranchised quite soon, because the community they chose is a dangerous place, even from the inside. What use is it to be surrounded by people if you still can't connect with them fully? However, by that point, the damage is done: by the rules you can enter, but you cannot leave ever again even if there's still a "you" left. Apostasy costs your life.

</comment></comment></comment></comment></comment></comment><comment><USER5> It is tragic to see millions of people denied a basic understanding of the world and themselves.

<comment><USER6> Those who understand, question. Those who don't, obey.

</comment></comment><comment><USER7> > Islamic creationism exploded in Turkey, often using arguments borrowed from > Christian creationists in the United States.

On that subject, the later are the leading force of misinformation in the whole world.

</comment><comment><USER8> I would expect that Trump will be leant on heavily to push similar changes in the US. Maybe not throwing evolution away in schools, but teaching creationism alongside.</comment>


The Dangers Of Externalizing Knowledge - <AUTHOR> http://techcrunch.com/2010/12/14/the-dangers-of-externalizing-knowledge/ <comment><USER2> Extreme version of counterpoint: By putting all of this knowledge online, we're building an organism bigger than ourselves.

Weak version: We're learning new types of information, and in many cases creating it rather than simply seeking to retain it.

My version is probably somewhere in the middle. But overall this is needlessly alarmist hand-wringing. Hooray for trend pieces!</comment>


Critical Thinking: What Is It Good For? (In Fact, What Is It?) - <AUTHOR> http://www.csicop.org/si/show/critical_thinking_what_is_it_good_for_in_fact_what_is_it/

<comment><USER2> Dave Johnson of Working Software was a direct mail marketing expert. He once explained to me that the reason advertisements coomonly say "BUY NOW! DON'T DELAY!" is that it results in significantly greater sales.

It's known as a "Call To Action". Without such a call to action, many people will not know what to do after reading a direct mail offer letter.</comment>


Ask HN: Is the Android Developer Console not updating for any of you? - <AUTHOR>

It's been 4+ days for me since my Android Developer Console has updated download or sales counts.
Sales are coming through and people are able to download my apps, but the "counts" don't seem to be updating anywhere.
I haven't seen a response from Google.  Maybe this will get their attention, the forum thread below doesn't seem to be getting any internal feedback as of yet either.
http://www.google.com/support/forum/p/Android+Market/thread?tid=62421154b5203024&hl=en <comment><USER2> My download count hasn't updated for a few days. Someone must be on vacation at Google.</comment>


(Yahoo) Pipes & Faucets - <AUTHOR> http://blog.craigslist.org/2009/12/pipes-faucets/#comments

<comment><AUTHOR> Craigslist blocks Yahoo Pipes and there is a Hacker News discussion. <http://news.ycombinator.com/item?id=970934> Then a Techcrunch Post: [http://www.techcrunch.com/2009/12/01/craigslist-yahoo- pipes-...](http://www.techcrunch.com/2009/12/01/craigslist-yahoo-pipes- flippity/)

Jim makes a statement: _"Yesterday afternoon craigslist engineering noticed a disproportionate amount of server/bandwidth resources being consumed by requests referred via Yahoo Pipes, with the lion's share of that activity appearing also to be in violation of CL terms of use. Pipes access has been suspended pending further review."_

If a bunch of people started emailing and tweeting Craig and Jim and then Techcrunch inserted themselves into the story with another story or two about the dramatic situation, they really might open API access.

</comment><comment><USER2> No, they won't. There's no financial incentive for Craigslist to innovate. They own the classifieds market. Jim will spew off some great CEO-speak as to why they're not going to provide an API. Craig will respond with a half- hearted answer about being community driven, and that if the community had wanted an api then they would already have an api.

I've been down this road with Craig before (lets just say he doesn't like being asked tough questions on panels or at parties).

<comment><AUTHOR> I think it should be possible to defeat that argument by calling their bluff and making the community speak.</comment></comment>


When Workers Own Their Companies, Everyone Wins - <AUTHOR> http://www.newrepublic.com/article/118933/economic-benefits-coop <comment><USER2> I believe this is one of the big reasons I chose to go towards trading instead of working in SV. The prop trading companies (and even big finance firms like Goldman) have _far_ more egalitarian profit-sharing structures than places like e.g. Google or Facebook, and you can really see that employees of those trading firms are a lot more motivated in their work than a programmer at a typical SV firm (even at startups that I've seen).

<comment><USER3> I don't think so. Yes, you do get paid more, but you're expected to put in more hours and forced to work within a much more rigid environment with less transparency. Also, throughout finance firms (even among the more tech-saavy ones), the programmers are ostensibly considered second-class to the portfolio managers, and this is reflected in the pay.

So yes, they get paid more, and as a result they may be more a bit more motivated, but there's nothing egalitarian about programmer pay on Wall Street.

<comment><USER2> I'm much more quant than programmer, but even programmers at trading companies get paid at least as well as they do at Google/Facebook (and obviously much better than they'd be paid at a startup) to start with. Not to mention that programmers at SV companies are second class to product managers, so the relative status situation isn't that different.

I don't know where you get the more hours/more rigid attitude from. 9hrs is very typical, weekends are _very_ rare, and the only time you ever exceed that is when you are releasing very new strategies -- strategies which you get a _direct_ cut of the PnL, so you have a very direct incentive to want to get that out there and trading as soon as you can.

Is everyone equal in finance? No. But compared to working in the valley? It's worlds apart. Let's look at a simple example, for Google and Goldman, take a look at average revenue per employee (~1.2m for both) vs. average compensation per employee (600k for GS, ~150-200k for Google). In what way are Google employees not getting screwed over? Prop trading firms have even flatter structures than Goldman.

<comment><USER3> "600k for GS, ~150-200k for Google" is a meaningless comparison. First of all, the 600k is mean, not median, and it's heavily skewed by the relatively few but very high MD/partner salaries. Second, that's across the entire firm, and the GS tech guys are pretty far down the totem pole when it comes to pay. I doubt there's much difference in median programmer pay in GS vs Google, especially recently given the rise in SV salaries.

And if you're a quant, then it's totally not fair to compare yourself against a programmer, because even if you're doing programming, it's a completely different job altogether. Quant jobs (the ones immediately touching the models) are much more statistics focused, and the coding is more a means more than an end.

<comment><USER2> What job is there that coding isn't a means to an end?

</comment></comment></comment></comment></comment><comment><USER4> Here is a list of the largest majority employee-owned companies in the USA: [http://www.nceo.org/articles/employee- ownership-100](http://www.nceo.org/articles/employee-ownership-100).

That is companies that are >50% employee owned.

Companies marked with an asterisk are 100% employee-owned.

The biggest 100% employee-owned company is Lifetouch Photography in MN with 25,000 employees.

But Co-ops and employee owned companies are not the same thing.

I have worked for several small co-ops that used consensus decision making. While it is true that everyone has an equal voice the tendency is to listen more to people with more experience or expertise in the particular area being discussed. Also, there is a lot of delegation that happens where subgroups are formed so not everyone is deciding what type of toilet paper to buy or how exactly we should implement some minor detail. That being said sometimes random people come up with the best ideas, so it was helpful to have them involved.

I found consensus decision making to be a lengthy process but a rewarding one.

<comment><USER5> Consensus decision making is like QA: they can help make sure you're solving the right problem.

The upfront effort often feels inefficient -- especially if the team is lost in the weeds -- but can save a lot of heartache.

I've used various group decisions making processes to great effect. Joint application development (JAD) sessions for project kickoffs (scope setting). Approval voting for triage (issues with most votes get solved first). Roman evaluation (thumbs up/down) for acceptance testing. Story telling for post mortems. Etc, etc.

I've also seen much wasteful decision making. I'm not smart, or yet experienced, enough to identify the qwan that makes certain teams rock and others fail.

</comment></comment><comment><USER6> I think the article does a poor job of explaining why factories are generally not coop. When you have 1 employee = 1 vote, you can't sell (a significant amount of) capital to external entities. This leaves you with borrowing money, for which you can't use a significant amount of capital as a collateral.

That leaves the coops mostly to sectors that are not capital intensive (mostly service and retail).

There is a bit of a trend currently in France around employee-owned companies. One of the most visible is a restaurant in Paris ("le Temps des Cerises", we know we're in dangerous communist territory just by the name).

I know a coop where the employees/owners take personal credits to buy the headquarters.

<comment><USER7> Yes, that's a big problem. The owners of capital want to have a say about what the company does, because their money are at stake.

Perhaps this is just a cultural thing though. Take money lending - although a bank gives capital to someone else in exchange for profit, it doesn't require strict controls over the actual use of the money; it relies on contract law to get the money back.

Similarly, the executives of the publicly traded companies have responsibility to shareholders. Perhaps a similar agreement could be institutionalized with the coops too (yet preserve their democratic nature).

<comment><USER6> For the lending, there will often be a collateral, for a company it will often be shares. That limits the leverage to 50% since a default will make the bank an investor and external investors can't have control.

The collateral will not always be shares, sometimes it could be the acquired physical good, then it's accessible to coops (too bad if you want to borrow for service or immaterial goods). Using real estate or company planes as a collateral suggests that you already got a credit to acquire them in the first place.

</comment></comment></comment><comment><USER8> It sounds like an attractive concept... but I'm not sure about its long term viability.

The larger a business grows, the more people become involved in the cooperative, and the harder it becomes to gain consensus or propose more risky but sometimes critical actions for the business.

Layoffs are most probably off the table, while things like employee benefits will probably become disproportionately popular and hard to cut. Furthermore the whole "own your company" appeal seems to become less of a big deal as more and more join the coop, and you get less of a say over how the company should be run. But conversely it is usually on a large scale that the financial benefits of minority shares ownership can yield decent returns. Seems a bit conflicting.

I've heard some groceries in US converted to co-op model though. Wonder how they are doing now.

<comment><USER4> > I've heard some groceries in US converted to co-op model though. Wonder how > they are doing now.

One example is Woodman's in the upper midwest: [http://en.wikipedia.org/wiki/Woodman's_Markets](http://en.wikipedia.org/wiki/Woodman's_Markets)

They have 15 locations and over 1 billion in annual revenue.

</comment></comment><comment><USER9> I am from Spain and used to work with cooperatives like Mondragon group in the North and lots of small agricultural coops in the south.

It is not paradise, but somewhat it works. It is not for everyone, everybody is the same and you have consensus lock when taking decisions.

About unions relationship, well for me they have nothing to do. Everything changes when the workers own a part of the company, nothing to do with Big Unions on Big Companies or Government.

</comment><comment><USER10> [http://en.wikipedia.org/wiki/Employee_stock_ownership_plan](http://en.wikipedia.org/wiki/Employee_stock_ownership_plan)

My favorite employee owned business is King Arthur flour. I think it's a wonderful thing when a company is owned by people who are part of the day-to- day meat and potatoes of the business.

</comment><comment><USER11> Coops & employee stakes have worked a lot of places, and should be tried more often – but it's clearly not best for everywhere, all the time, or it'd already be a universal practice.

United Airlines became majority employee-owned in 1994 – [https://en.wikipedia.org/wiki/History_of_United_Airlines#Emp...](https://en.wikipedia.org/wiki/History_of_United_Airlines#Employee_Stock_Ownership_Plan) . Was it the best airline by any measure from then, until its 2002 bankruptcy?

Chrysler was majority-owned by a UAW retirement trust for a period starting in 2009... until Fiat finished buying out that stake earlier this year. Was 2009-2014 an "everybody wins" period for Chrysler?

</comment><comment> barry-cotter I'm sure these companies are more humanely managed and have happier employees but I doubt they're economically efficient. If they were they'd outcompete non employees owned companies. We even have an entire sector of employee owned companies where they obviously do win consistently, professional services firms like accountancy (PWC, KPMG, Deloitte, Ernst & Young) or management consulting (McKinsey, Bain, BCG). All the MC firms reserve equity for partners after an up or out process which takes about seven years. Most people never get equity. Accounting firms don't have up or out but there are still partners and non-partners.

What's particularly damning about the efficiency/survival value of employee ownership is that all the (then) Big 5 accounting firms threw off IT project management/procurement firms, all of which were/are normal companies. Accenture os all that's left of Arthur Andersen. It doesn't have partners except the way Goldman Sachs has "partners".

If you want more humanely managed companies mandate it for everyone because employee owned companies lose, consistently.

<comment><USER12> You're presuming a perfect market where the most economically efficient organisation will win. There is no guarantee that our current economy is like that.

<comment> barry-cotter ADBOC. This is like the efficient market hypothesis, the strong version is wrong and the weak version is so weak as to be almost tautological. In a perfect market the most economically efficient organisation would win, end of story. What we've got isn't that but it's an approximation thereof. The closer it gets to fulfilling all the conditions the more likely it is that the outcome is economically efficient.

The only economy I'm aware of that had a lot of employee owned cooperatives was Yugoslavia. It worked better than Soviet bloc communism as far as resource allocation and living standards went but not as well as capitalism.

Co-ops don't expand as much as privately or publicly held firms because they maximise something approximating profit per worker whereas normal firms maximise straight profits. I think I picked up that argument from Paul Krugman. It's been a long time since I read the argument so I may be misremembering but he used Harvard and the UC systems as examples. Harvard doesn't expand its student body, this maximises faculty utility, the UC system does, this maximises student utility. The article even mentions that worker's co-ops have higher profitability than normal firms.

<comment><USER13> _Co-ops don 't expand as much as privately or publicly held firms because they maximise something approximating profit per worker whereas normal firms maximise straight profits. I think I picked up that argument from Paul Krugman._

r- and K-selection. Companies like Google and Microsoft, which grew large at the cost of culture, are r-selective. Valve and Github, which grow more slowly while maintaining open allocation and cultural health, are K-selective.

It seems impossible, in the long run, to be K-selective if you take venture capital. You're now controlled by people with the attention spans of five- year-olds.

</comment></comment></comment></comment><comment><USER14> In the UK, employee or customer owned businesses are a mixed bag. We have some very good ones (John Lewis, Nationwide) and some bad ones (the Co-op group).

I think whether or not a business is employee owned is less important for success than it having a good and capable management team.

<comment><USER15> > We have some very good ones (John Lewis, Nationwide) and some bad ones (the > Co-op group).

The co-op group was a good one for a century and a half, but went astray in the last few years.

Edit: For those unfamiliar, the story of Co-op bank chairman Paul Flowers is good lurid fun.

[https://en.wikipedia.org/wiki/Paul_Flowers_%28banker%29](https://en.wikipedia.org/wiki/Paul_Flowers_%28banker%29)

</comment></comment><comment><USER16> Coop models may work if decision-making structures are defined clearly and 'economically'.

>> _expand the coop model by associating it more closely with unions._

A receipt for disaster. Some European states had this models in the past and completely abandoned them.

<comment><USER17> Care to elaborate? Unions function differently in different EU states: In the UK they were pretty much killed, in France they act as antagonists to the business side, in Germany they act more collaboratively with the business side.

In the latter case in Germany every large enough commpany has to have a democratically elected workers council, that gets a seat on the board. While not the same as co-ownership, this democratic co-determination is arguably one of the more important reasons why labor relations in Germany are much better than in other European countries.

<comment><USER18> France also has democratically elected woker councils. It's no guarantee for succes.

France on the other hand has had a minimum wage for a long time and a 35 hour work week. To me it isn't so clear who is doing a better job.

But clearly French unions are not afraid of confrontation.

<comment><USER19> Germany didn't need a minimum wage because of the strong unions. Now that they lost much power minimum wage is needed. This is also why there's no big push for minimum wage for instance in Denmark or Sweden, because the unions still work there.

</comment></comment></comment></comment><comment><USER20> Profits should be shared with employees along with share-holders.

</comment><comment><USER13> _But about 47 percent of American workers participate in profit-sharing arrangements of some sort. Employee stock ownership plans (ESOPs), for instance, involve around 10 million workers_

Most of those programs are junk, it's worth noting. That's true, even in the Valley, where 0.05% (vesting over 4 years, with nothing gained if the company decides to cliff you) of a 100-person company is typical for a "mere" engineer.

The term I like is "checkbox equity". It's just offered so the company can say, "we offer stock options" on its website. It means absolute jack shit. Far from making you a partner in the company, it has the opposite effect. It's an excuse not to offer raises and to demand unreasonable sacrifice. It's a mediocre equity program that exists just to check off the box. To that I say: fuck you, pay me.

I would rather not have the equity (which is almost always insultingly low) and just get a fair wage. Let's just be honest here. I'm going to be a mercenary unless you give me a legitimate reason not to be, and 0.03% of a 200-person company is 0.06 of a person's labor, which is just _not_ enough to make me give a shit about anything other than my own career. So I won't, and that should be socially acceptable.

The purpose of these Valley startups' equity programs is to encourage a tyranny of the masses, and make the honest mercenary attitude socially unacceptable (allowing the employees to be plundered by the _dis_ honest mercenaries at the top).

</comment><comment><USER21> How hard must they have tried to write that without using the word Marxism!?

<comment><USER22> It isn't Marxism. One of the most well-known advocates of employee-owned companies, Noam Chomsky, is anarchist, not Marxist. Marxism is primarily about controlling political power to achieve economic goals. What the article talks about works without (much) political power.

<comment><USER23> I hate to break the news to you but Chomsky is a Marxist.

<comment><USER24> What's your argument/evidence for that claim? Chomsky has self explicitly said numerous times that he's not a Marxist and prominent Marxists don't consider him to be one of them...</comment></comment></comment></comment>


Coronavirus causes Waffle House to close 365 locations - <AUTHOR> https://www.foxbusiness.com/lifestyle/coronavirus-causes-waffle-house-to-close-365-locations <comment><USER2> For those not aware, Waffle House is so well-known for keeping its stores open in all kinds of disaster situations that there's even an index based on their closures used by FEMA [1]. Closing this many locations for any reason is unprecedented and speaks to the severity of the current situation.

[1] [https://en.wikipedia.org/wiki/Waffle_House_Index](https://en.wikipedia.org/wiki/Waffle_House_Index)</comment>


Dr Dobbs - The Rise And Fall of Languages in 2010 - <AUTHOR> http://www.drdobbs.com/visualstudio/229100358 <comment><USER2> > Over the years, the index has been both praised and vilified. The latter by > language adherents who are unhappy over the decline of their favorite idiom.

I can't say that I trust TIOBE's methodology (looking at search result volume), and that's as an ardent Pythonista.

<comment><USER3> I agree and think this article gives a perfect example of why it isn't accurate. Why would Javascript fall out of the top 10 last year? It makes no sense if you trust the methodology. BUT...if you think about it a system that gives a lot of weight to "mentions" falls apart when frameworks start drawing attention because it divides the number of mentions (how many times have you seen a jquery or Sproutcore article that never mentions the word Javascript?). Hence the drop

<comment><USER4> I almost sent them a critic-gram about this one, then I stopped and thought. I suspect their method is measuring _the fraction of JS activity which is self- perceived as "programming" in the sense of software engineering_. Rightly or wrongly, laying out pulldown menus with JQuery and the like is broadly (not universally) perceived as the practice of "design". Surely all of us know that plenty of echt programming is done in JS, and also plenty of let's-make-this- widget-dynamic work by people whose skills do not include structuring large blocks of code. The distinction is not an XOR, but it is both clear and real, and I believe the Tiobe results reflect it. My further W.A.G. is that their method under-represents the level of software engineering activity done in JS relative to, say, Ruby and Python, but that it does so because developers themselves are lagging in their perceptions.

</comment></comment></comment><comment><USER5> > "I expect this trend will continue, as I strongly doubt that many new green > field projects would today choose Perl as the principal development > language."

Maybe but checking regularly the CPAN recent updates ( <http://search.cpan.org/recent/> ), it seems that Perl is still very active and especially in the activity of providing API or pipe to other technologies/libraries.

At the end, large or medium sized projects are using different programming languages for different tasks and there is no such thing as the "principal development language". Diversity is great...

<comment><USER6> > Perl is still very active and especially in the activity of providing API or > pipe to other technologies/libraries.

If Parrot comes to full fruition, CPAN should be available to (at least) both Ruby and Python about as easily as it's available to Perl.

 _If_ Parrot comes to full fruition.

<comment><USER7> You can already run _some_ CPAN modules in Rakudo (Perl 6) on Parrot. See Blizkost (which comes with Rakudo): <https://github.com/jnthn/blizkost>

Here is an example which works (using CGI module from perl5/CPAN):

        use v6;
        use CGI:from<perl5>;

        my $q = CGI.new;
        print $q.header,
            $q.start_html('Hello World'),
            $q.h1('Hello World'),
            $q.end_html;

PS. re: _some_ CPAN modules - Only "well behaved" Perl5 modules will work. I've tried four and got one to work. However this is a good start methinks!

</comment></comment></comment><comment><USER8> The biggest thing that draws this into question for me is the fact that Visual Basic and Javascript supposedly dropped by the similar amounts.... ??

Among a dozen other languages, I still frequently develop in VB as I have a huge code library for it (crawling back into the 90's) and it still works well as a general Swiss Army knife tool. I have seen the community plummet though and of course it's rapidly fading in it's significance.

Javascript however is on a meteoric rise development wise. It seems it is being adapted to anything possible and I see regular talk of it being used server side now too.

Yet supposedly they both dropped similar amounts? I'm surprised VB didn't drop more, and Javascript "dropped" at all.

<comment><USER9> Maybe interfacing to javascript libraries has replaced actual new javascript development and people search for, for instance, "jquery" instead of javascript.

<comment><USER10> That seems to be the main problems with these metrics. They aren't looking at the popular libraries that make use of the language. If I search "Android OpenGL development" it should count as a query directed at Java. This survey doesn't seem to do that though.

</comment></comment></comment><comment><USER11> An intellectual hackers' site like this one tends to be home to people playing with Scala, Erlang, Haskell, etc. Hacker News in particular was started by Paul Graham, so its seed community had a significant number of Lisp adherents. The startup community in general is biased towards dynamic languages, in particular, Ruby.

All this makes me think that Hacker News readers will tend to be on the leading edge of language popularity. But it is interesting to see the progress of mainstream adoption.

<comment><USER12> I think you're on to something here, at least about the HN community being at the leading edge of what's cool.

Just anecdotal, but I saw that statement in the article of "python surged" and had a WTF moment. Just yesterday I was putzing around with python and Fuse to make a silly filesystem, and found myself thinking "man python should really stop being my go-to language, it's making me seem so old-fashoned". Lately I have had a lot of those thoughts about python. It's my trusty language, and probably my first true love, but this last project and the article was my "python jumped the shark" moment.

So, the question now is:

I have a crapload of useful tools in python, and I don't want to rewrite them in other languages - has anyone made serious progress into calling into one runtime from another? Kind of like the JVM or .Net worlds, but universal?

<comment><USER13> Python "jumped the shark" because more people have started adopting it?

<comment><USER14> That puts things in perspective, doesn't it? There are buzzworthy languages and there are languages the rest of the world uses to get things done.

</comment></comment></comment></comment><comment><USER15> I'm extremely dubious of a metric that claims JavaScript and Delphi have the same market penetration.

</comment><comment><USER16> I guess I'm one of the Tiobe doubters, but I trust the Stack Overflow tag trends more: [http://meta.stackoverflow.com/questions/23554/tag-trends- by-...](http://meta.stackoverflow.com/questions/23554/tag-trends-by-week)

</comment><comment><USER17> The article comments with respect to Python: "What is surprising is that several factors might have argued against a break-out year for Python: Two incompatible versions of the language and the uncertain fate of the Unladen Swallow project."

The two "incompatible" versions of Python, presumably Python 2 and Python 3, is really an indication of the strength and foresight of the Python developers and users. Python 3 has new features, new libraries, and is missing some of the cruft of Python 2. I see this a sign of vibrant growth and long term viability, not of weakness.

The article did not mention the PyPy project, python in python targeting performance on a par or better than CPython. It's Python 2 compatible at the moment, and a lovely piece of work. There will be a webcast Stanford talk by Armin Ringo, one of the developers, on the PyPy system on March 2, 4:15PM Pacific. <http://ee380.stanford.edu>.

And, frankly, I do not see how the "uncertain fate of the Unladen Swallow project" has anything to do with the acceptance of Python.

</comment><comment><USER18> Since a lot of the big technology winners from 2010 were infrastructure plays, I wonder if back-end still plays more of a role than client interface, which, putting node and backbone aside, is where most of the JavaScript is written right now.

</comment><comment><USER19> Could Python be capturing the dissidents of Java?

</comment><comment><USER9> Three things struck me:

1) The persistence of C. Looks like it will remain "the glue" between the machine and "higher level" operations for the next decade and beyond. It's strong niche in the "must be fast as possible and still portable" arena helps, too.

2) The rise of Python.

3) The decline of VB.</comment>


Are we wrong about equality? - <AUTHOR> http://economist.com/briefing/2019/11/28/economists-are-rethinking-the-numbers-on-inequality <comment><USER2> I find that some of the ways economists just fudge data to be downright disturbing. That those are then cheery-picked and offered as input to policymakers is even more concerning.

As a for instance; take the claim that the Reagan years policymaking only caused a _seeming_ boom in numbers due to the fact that people were no longer sheltering earnings in corporations, therefore there had to be a matching decrease elsewhere.

The issue I have with that is that money sheltered in a corporation controlled or heavily influenced by a particular individual is still essentially indistinguishable from being income for that person. It just resides in a nominally transparent legal fiction, that economists are not willing/confident to pierce. If you don't understand those circumstances or take them into account, the map you draw of the territory becomes downright dangerous to the ones using it rather than the help you nominally set out for it to be.

I understand that gaggles of economists are _trying_ to provide alternate slices of data, but from the research point of view, if you are not looking at the real to gather your data, you're playing fantasy football. Which frankly, given the I'll-conditioning of the economic problem scares the shit out me.

<comment><AUTHOR> 'if you are not looking at the real to gather your data, you're playing fantasy football.'

Well put my friend.

Your comment has made me curious about your stance on Hayekian economics v. Keynesian.

<comment><USER2> That is... a lot to unpack, to be frank. I tend to try to integrate the non- economic philosophies of both thinkers into what I believe their understanding of the systems in question were, and by no means do I distill out anything necessarily useful or predictive, merely seductively correlated with what I've seen unfold in my lifetime.

However, so as not to disappoint, I will voice some opinions. I think there may be something to the Austrian school, and I don't necessarily buy Hayek's critics propensity for saying that he's trying to have his cake and eat it to as A) a reason to dismiss him outright and B) not a demonstration of the importance of his points with regard to the folly of trying to centrally plan an economy; something that I've recently come to realize we have been slowly degenerating toward in the United States. It just doesn't seem like it because you have to look at the meta-structure of decision making outside economics in order to realize the staggering shrink in the number of inputs to increasingly large chunks of capital as businesses undergo the process of merging and consolidation.

While the central planning we engage in tends to openly constrain itself to the money supply and interest rates, we still have the problem that there is precious little information truly at hand in order to make meaningful decisions. When decision making is distributed widely, you have a tendency for quick reversion to the mean to the degree that is most appropriate for the conditions in which economic activity is happening (Hayek point). The farther away you are the less fine grained your grasp on the the local aspects of the economic problem, but the greater your potential leverage, as when everyone looks up at you for cues, what you say has a way of going a long way (Keynes); not saying/doing anything, therefore is to actively make a non-interventionist decision, which means that spontaneous organization may happen, but not as quickly(Keynes' attitude toward action projected into Hayek's framework). Keynes seems to air on the side of doing something being better than nothing; as at least taking action in the long run tends to preserve some level of unity or at least preseeve momentum that can be redirected through policymaking as long as it is there, which I don't think is necessarily wrong. Though I think the Austrian's are safe in pointing out that odds are, you will make the problem worse by actively interfering and making decisions on which you have no information to safely make.

I've sat and watched QE deliver exactly what ABCT predicts. Regular interventions necessitated to forestall higher and higher stakes recessions that have spread wider and wider through the tighter and tighter integration between respective economies, and the rapidly centralizing tendency of merger friendly business environments. On the other hand, The prescription the Austrian's tout is unpalatable, or at least useful to me in that it insists the economy will sort itself out if you do nothing, so let the economy do it. I find in Keynes, however, that in the event of a crisis, you can shorten the recovery time by providing the judge required to get people to do the necessary business of transacting through stimulated demand. Economic activity (day to day life) wishes/needs to continue. However, the fact you're in the situation you're in leads me to believe that the Austrian's have a salient point. If you keep mucking with and distorting the market, instead of letting it stabilize on what it can sustain, you steer the economic problem into novel areas undeniably; just as undeniably as you set the stage for an eventual highly disruptive reckoning when the bubble bursts.

So what does it all come to? I think the Austrian school of thinking is overall centered around the idea of the economy tending toward a distributed steady state driven by the non-human environmental factors, and the human responses to those factors. An Austrian economic model may end up leading to economic activity growing to fit it's container so to speak, but only slowly generate an overall increase in the size of the economic pie.

Keynesian economics, however, encourages risk taking and moonshots. Who needs sustainability within the container, when with enough economic oomph thrown at the problem, you can just right out of the container, get a second economic beach head going somewhere else, keep the size of the pie growing for everyone, and when things go south, pump every lever you can to get growth going in some new direction. The biggest issue I see with Keynes' system as a whole is the dismissal of Hayek's implicit assertion of a rational yet incomplete foundation for economics as far as the perspective of a practitioner is concerned.

So it does make sense to me the Austrian school is shunned, yet kept around like some voodoo doll by modern economists. It fundamentally undermines the field in substantial ways, but contains nuggets of how the non- macroeconomically inclined will fundamentally react in the abscence of active macroeconomic pressure.

That is to say, most people uneducated in economics at all tend to assume an Austrian model unconsciously. The Keynesian approach, while predominant, does not seem to line up at all with natural intuition without guided thought. Which ironically comes back to the two major differences in the two schools economic philosophies.

Hayek: Just leave it alone Keynes: Do what you must to keep people transacting. If people feel nervous about where money is going to come from, drop interest rates. Once people are transacting regularly, start pulling the rates back up in order to control inflation.

Where both seem to fail nowadays, is in the consequences of smoothly bridging the transition between a highly interconnected growth optimized economy, and an economy physically constrained by the perception of elevated risk of death in the context of conducting the business of transacting. Neither model has any real oomph without people being willing or able to engage in business actdue to physical environmental constraints trivially.

For what it is worth, the Austrian's are neck to neck with the Keynesians as far as this latest crisis goes from my point of view. By doing nothing, the economy shifted to try to maximize its capability to transact whilst keeping the transactors specially separated. Depending on whether you view the coming stimulus from Congress as an act of monetary policy, or just the economy readjusting itself; you could safely make the argument that some of eitherschool is at work.

This is why I hate economics. It always seems to have had the answer in a post-facto rationalization sense, but it always seems to feel like a coin-flip making the decision of which way to go. Keynesian economics is probably the worst in terms of PR generators for economics though. If there is anything out there that makes people hate banks, it's the natural distaste that comes from someone else mucking about without _knowing_ what they are doing. In that sense, you can't help but appreciate the Austrian's "your idea is as good as mine" approach.

<comment><AUTHOR> Thank you for taking the time to write such an eloquent, in-depth exposition of your POV. Highly enlightening.

This is, in part, why I feel it's flimsy criticism when people attack macro- economists, without clearly articulating where they disagree or preferably offer a better theory.

It's just such an incredibly complex field. And unlike mathematics where you can control all the variables and construct a proof that lasts till the end of time, in economics you're dealing with many hidden, dynamic variables (which are susceptible to butterfly effects). It's hard to run the experiment again sort-a-speak.

Thanks again. Thoroughly enjoyed reading your take.</comment></comment></comment></comment>


Did the Soviets Actually Build a Better Space Shuttle? (2013) - <AUTHOR> http://www.popularmechanics.com/space/rockets/a9763/did-the-soviets-actually-build-a-better-space-shuttle-16176311/ <comment><USER2> "The Kremlin leadership saw the space shuttle primarily as a carrier of space weapons and discounted its advertised scientific and commercial applications as an elaborate smokescreen"

Which, to be fair, somewhat reflects the involvement of the DoD and NRO in the US shuttle design process. After all, the USAF built Vandenberg for a reason, and all the SDI people wanted x-ray killsats, kinetic bombardment satellites, etc.

<comment><USER3> There's a good discussion on the Soviet analysis of the US shuttle program in "The Making of a Soviet Scientist" by Roald Sagdeev. Sagdeev and Keldysh were against the shuttle since they could not find any justification, economic or otherwise, for the space shuttle.

Eventually the argument that won the day was that "The Americans are smart and pragmatic, so there must be some use" for the space shuttle.

[https://en.wikipedia.org/wiki/Roald_Sagdeev](https://en.wikipedia.org/wiki/Roald_Sagdeev)

[https://en.wikipedia.org/wiki/Mstislav_Keldysh](https://en.wikipedia.org/wiki/Mstislav_Keldysh)</comment></comment>


The 2015 Wealth Report - <AUTHOR> http://content.knightfrank.com/research/83/documents/en/wealth-report-2015-2716.pdf <comment><USER2> These guys need to take a lot closer look at Silicon Valley. They report that San Francisco has only 526 people with a net worth over $30 million. That's crazy low.

Just for example, Dustin Moscovitz says that employee number 100 made $200M at Facebook[1]. Imagine the number that made $30M, and that's just one company (see slide 14).

[1] [http://www.businessinsider.com/dustin-moskovitzs-startup- adv...](http://www.businessinsider.com/dustin-moskovitzs-startup-advice- deck-2014-9?op=1)

<comment><USER3> It actually seems low to me too. $30 million is a lot of money, but given the number of companies with $1bn+ valuations recently, you'd think there would be at least 200 from those alone.

It may be they are not counting paper wealth in private companies. It may also be that many of these $1bn+ valuations came in the last year or so (their data is from 2013.)

<comment><USER4> If the company is worth $1BB, then you must own 3% of its equity in order to have $30MM worth of equity of that company [1].

Non-founders rarely, if ever, are given that much equity (commonly cited numbers tend to top out around 2%). The most likely candidates are engineer #1 and an early VP, but engineer #1 has likely been diluted by over 30% since seed stage and even the exec will have been diluted by 10~20%.

At $5BB, a nonfounder needs about 0.6% of the company. By this point though, dilution is usually even more severe, and engineer #1 will likely have had his/her initial share cut by close to 50%. This means that only the earliest of early employees, along with the VP level hires, will have $30MM at a typical $5BB startup (though $5BB startups are hardly typical).

It's the companies that are another order of magnitude higher in valuation that mint wealth of that kind to dozens of employees.

[1] ignoring liquidity preference for the sake of simplicity.

<comment><USER5> MM is often used to denote millions, but I don't think BB is used to denote Billions. Do you have a cite that indicates otherwise, because I don't see it on a quick search.

<comment><USER6> M = 1000

MM = 1000 * 1000

NMM = 1000 * 1000 * 1000

pp = pages

ll = lines

so it's a natural logical extrapolation to go for

bb = billions

[http://www.riskglossary.com/link/mm.htm](http://www.riskglossary.com/link/mm.htm)

<comment><USER7> If the analogy is billions to millions, BB = Billion * Billion

</comment></comment></comment></comment></comment></comment><comment><USER8> Original source is more interesting: [http://content.knightfrank.com/research/83/documents/en/weal...](http://content.knightfrank.com/research/83/documents/en/wealth- report-2015-2716.pdf)

<comment><USER9> Ok, we changed the URL to that from [http://fortune.com/2015/03/09/private- jet-routes/](http://fortune.com/2015/03/09/private-jet-routes/), and the title from "Private Jet: Moscow to Nice Is Most Popular Route for Private Planes".

</comment></comment><comment><USER10> This is purely an anecdotal evidence, but I like to walk in Pacifica, CA which is on the flight path from SFO runway 1R/19L. On the weekends, the number of private jets taking off is equal to the number of commercial jets.

</comment><comment><USER11> In addition to the original info about the Moscow-Nice flight there ([http://content.knightfrank.com/research/83/documents/en/weal...](http://content.knightfrank.com/research/83/documents/en/wealth- report-2015-2716.pdf)) are some more curious stats about Russians: the percentage of Russians of all UHNWIs willing to emigrate (33%), and the percentage of them who send their children to study abroad (61%). Not that the tendency was not known before though.

</comment><comment><USER12> This was a very strange report. Skipping to the middle I thought it was about wealth in general, which seemed like a nice counterpoint to the study of poverty. I'd be very interested in reading about trends in people earning above $60,000 a year, and how the rest of the world can get there. But this was really about "Ultra high net worth individuals" which doesn't strike me as a very interesting topic.

<comment><USER13> I would be pretty interested in something like that too.

</comment></comment><comment><USER14> Beautiful data.</comment>