Tarballs, the ultimate container image format - <AUTHOR> https://www.gnu.org/software/guix/blog/2018/tarballs-the-ultimate-container-image-format/ <comment><USER3> I like simple archives, but can it be not tarballs? For the kinds of application described in this article, tarballs are pretty bad:

Either you extract it from scratch every time you run an app, taking a long time penalty...

... or you extract once to cache, and assume that nothing changes the cache. This is pretty bad from both operational and security perspective:

\- backups have to walk through tens of thousands of files, thus becoming much slower

\- a damaged disk or a malicious actor can change one file in the cache, making damage which is very hard to detect.

There are plenty of mountable container formats -- ISO, squashfs, even zip files -- which all provide much faster initial access, and much better security/reliability guarantees, especially with things like dm-verity.

<comment><USER4> Yes, most tarballs do not support random access (there are some metadata extensions that allow this). This makes large tarballs annoying to use on systems with slow disk I/O (even a hard disk may be too slow (to the degree of being annoying to work with)). This is by far my biggest gripe with the format. Certainly, smaller tarballs are a very handy format as long as you stay inside the Unixy world of computing – and as long as you keep looking out for the various incompatibilities between the different tar implementations.

<comment><USER5> "... there are some metadata extensions that allow this)."

Where to find these extensions? Are they portable between Linux and BSD?

The 1998 dict project included a utility called "dictzip" for random access to the contents of _gzip compressed files_.

Dumb question: Is it possible to create a utility or even a hack that performs "random access" into _tar archives_?

Example use case: the user only wants to untar a small number of selected files from a large tarball such as a source tree. The user has tried both the "-T filelist" option and using memory file systems instead of hard disk drives.

<comment><USER6> A zip file is a concatenation of gzipped files. A .tar.gz is a gzip stream of concatenated files. Anything that could do random access into the contents of a zip file entry could do similar things with a tarball.

<comment><USER7> Not so simple. A bundle of streams is not the same as a stream of bundles.

<comment><USER6> With a transparent random access overlay, the difference mostly disappears, reducing to whether the stream needs to be scanned or whether it's indexed, which is itself orthogonal - zip file directory at the end is redundant.

<comment><USER8> So you mean at each "random access", you actually have to scan the whole .tar.gz file to find the location? For large tarballs, that will definitely hinder performance a lot. The difference does not disappear at all.

<comment><USER6> Apparently you have a comprehension problem.

<comment><USER8> Then enlighten me.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> How about sqlar as a container format? [https://sqlite.org/sqlar.html](https://sqlite.org/sqlar.html) A regular sqlite database file, with anything you like in it. Mountable as a file system with sqlarfs. Written by the sqlite guy.

<comment><USER10> Interesting I didn't know this existed. Is there a way to layer sqlar like docker images? (Besides just tarring them up I guess.)

I wonder if this could be implemented with the WAL/journal system. Make each layer immutably append to the previous layers to make restarting at any layer trivial. I'm not sure if there's such a way to hook into the journal directly like that though.

<comment><USER11> Should be doable with overlayfs (or similar) or alternatively some extensions to sqlar.

sqlar is after all only a table definition, if you don't need FUSE access or are willing to write your own, SQLite3 can go a long way of providing arbitrary neat functionality.

</comment></comment></comment><comment><USER12> I really love the work the guix folk are doing. I'd love to run guixsd on my laptop if it was easy and supported to run plain upstream linux instead of linux-libre. It just seems like such a lovely easy to use project from the little time I've spent playing with it, it's actually a small shame they're part of the "unsexy" GNU project and subject to GNU politics.

<comment><USER13> What GNU politics are you referring to that makes you reconsider using guixsd?

EDIT: Also what's unsexy about GNU? I'm really curious.

<comment><USER14> Its refusal to package firmware binaries, for one, even if that firmware is required to have a useful machine. I'm looking at AMD specifically here, where recent graphics cards (including APU's) don't even do text-mode without the firmware.

(edit: I understand the why of it, and even agree on principle, but it still prevents me from running linux-libre on most of my systems)

<comment><USER15> Next time you build or choose a system, consider one that can run free software.

I did, and it makes most things quite a bit easier.

Edit: I did after struggling with hw requiring nonfree blobs of different shapes and size for a couple of years. Currently I was lucky to get my hands on a system that I can run using linux-libre and the only component I have "extra" is a usb wifi card.

<comment><USER16> > Next time you build or choose a system, consider one that can run free > software.

The only workstation that boots with entirely free software is like, the Talos II PowerPC, with a minimum cost of $5000.

Everyone else requires a binary blob somewhere. Either a UEFI blob, BIOS blob, some kind of driver somewhere, or whatnot. Raspberry Pi, AMD, Intel, everybody.

And before the Talos II, I don't think an "Open PC" devoid of proprietary binary blobs even existed. At least, something that is reasonably modern (ie: 64-bit, decent security, decent support with modern OSes)

<comment><USER17> What about pre-ME thinkpads, after replacing the wifi card with an ath9k/open source firmware one? Does the intel chipset graphic require a blob for simple framebuffer/textmode operation? Because I can't remember including any blobs in the libreboot I use there, and iirc I get output before a linux kernel is able to load device firmware.

It is 64bit, and runs pretty much anything from (from what I can tell, but not sure, due to CHMPXCHG16B) Windows 10, over FreeBSD to Android. Probably even something like QNX.

Yes, you might not call this reasonably modern, but according to the hard facts you listed as qualifiers for being reasonably modern, they tick off.

<comment><USER16> Unless it is running the "coreboot BIOS" (which very few things are), then it has a binary "non-free" blob booting it up.

<comment><USER17> I don't remember whether the video BIOS was extracted from the old binary or if it is the open-source replacement, but I'd tend towards the latter as I don't remember searching for the backup/dump of the original firmware.

And yes, it's running coreboot, and at least CLI/linux-framebuffer arch linux works. I didn't yet get to setting the rest of the system up, but considering I bought it specifically for high-security operation, as the ME can be physically removed without loosing more than the build-in Ethernet port, I'm not pressed to do it anytime soon.

Edit: I'm pretty sure I followed [0], which leads me to the new conclusion that I did use libreboot, a more strict version of coreboot (think coreboot=Archlinux, libreboot=GNU Guix), and had to fiddle with the question whether the open-source video bios would work. This confuses me a little, as I remembered buying an X61s, not an X60s, but from the fact that it booted after flashing, I deduce it had to be an X60.

[0]: [https://libreboot.org/docs/hardware/#list-of-supported- think...](https://libreboot.org/docs/hardware/#list-of-supported- thinkpad-x60s)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> That article made me warm up to guix and its practical side. Are guix app bundles just bare tar archives with /usr/local prefix semantics or do they need special metadata files? How are compiled binaries with hardcoded and/or autoconf'd prefixes handled for relocation (I guess using Linux namespaces somehow)?

<comment><USER19> In Guix every package ends up in its own directory, which may have references to other packages in /gnu/store. An application bundle is really just a package closure, i.e. the directory for the package and all directories it references, recursively. One way to bundle up things is with `tar` (the default of `guix pack`), but Guix also supports other bundling targets, such as Docker. No special metadata files are required.

Relocation currently requires a little C wrapper, which uses Linux namespaces, as the blog post indicates.

If you want something more advanced, such as a bundle that includes an init and services, it's best to use `guix system`, which builds VM images among others.

</comment></comment><comment><USER20> For relocatable ELF binaries, there's also [https://github.com/intoli/exodus](https://github.com/intoli/exodus)

<comment><USER21> The packages that Exodus produces are actually quite similar to those introduced in this announcement. Both tools generate simple tarballs that can be extracted anywhere to relocate programs along with their dependencies, and both tools bootstrap the program execution using small statically compiled launchers written in C. They contrast _guix pack_ against Snap, Flatpak, and Docker, but Exodus would probably make a more apt comparison in many ways.

</comment></comment><comment><USER22> This is remarkably off-beat for the GNU project. Tar files are _far_ from the most ideal tool for container images because they are sequential archives and thus extraction cannot be done using any parallelism (without adding an index and being in a seekable medium, see the rest of this comment). I should really write a blog post about this.

Another problem is that there is no way to just get the latest entry in a multi-layered image without scanning every layer sequentially (this can be made faster with a top-level index but I don't think anyone has implemented this yet -- I am working on it for umoci but nobody else will probably use it even if I implement it). This means you have to extract all of the archives.

Yet another problem is that if you have a layer which just includes a _metadata_ change (like the mode of a file), then you have to include a full copy of the file into the archive (same goes for a single bit change in the file contents -- even if the file is 10GB in size). This balloons up the archive size needlessly due to restrictions in the tar format (no way of representing a metadata entry in a standard-complying way), and increases the effect of the previous problem I mentioned.

And all of the above ignores the fact that tar archives are not actually standardised (you have at least 3 "extension" formats -- GNU, PAX, and libarchive), and different implementations produce vastly different archive outputs and structures (causing problems with making them content- addressable). To be fair, this is a fairly solved problem at this point (though sparse archives are sort of unsolved) but it requires storing the metadata of the archive structure in addition to the archive.

Despite all of this Docker and OCI (and AppC) all use tar archives, so this isn't really a revolutionary blog post (it's sort of what everyone does, but nobody is really happy about it). In the OCI we are working on switching to a format that solves the above problems by having a history for each file (so the layering is implemented in the archiving layer rather than on top) and having an index where we store all of the files in the content-addressable storage layer. I believe we also will implement content-based-chunking for deduplication to allow us to handle minor changes in files without blowing up image sizes. These are things you cannot do in tar archives and are fundamentally limited.

I appreciate that tar is a very good tool (and we shouldn't reinvent good tools), but not wanting to improve the state-of-the-art over literal _tape archives_ seems a bit too nostalgic to me. Especially when there are _clear_ problems with the current format, with obvious ways of improving them.

<comment><USER23> It sounds like you are progressing along the same road that led Rahul Dhesi to invent the ZOO file format.

<comment><USER22> As far as I can tell the only thing ZOO has over tar archives is having a history of each file (using the VMS concepts of file versions) -- meaning that it probably still has some of the problems I outlined above. While that is useful, it is still not as good as it could be. Also, you don't really want file versions with container images, you want to have conceptual "layers" (which would be sort of like having versioned files but it's more like snapshot IDs -- or like ZFS's birth-times).

<comment><USER23> One needs to give it more than a superficial glance. ZOO was designed to be randomly accessible, with the directory headers forming a linked list. It actually _has_ an uncompressed index and _can_ take advantage of seekable files. It also supports both long and short filenames; CRCs of the metadata structures (c.f. the recent kerfuffle about xz); and an extensible, versioned, header mechanism that not only _could_ be extended but actually _already once was_ extended to add the long filename support amongst other things.

<comment><USER22> Is there an actual paper or some high-level summary of the format -- not to mention a modern implementation? The only summary I could find was the one on Wikipedia. I also found the source code of "unzoo" but it's a bit difficult to understand the benefits of a file format if I first have to understand its implementation.

I didn't take a superficial glance out of laziness, it's because I couldn't find any more information about it. But I think you also missed that I mentioned that the style of versioning implemented in ZOO (as far as I can tell based on a Wikipedia page) is not the correct style for snapshot-like versioning.

</comment></comment></comment></comment></comment><comment><USER24> I realize the title is just a hook for the (very cool!) work in the article, but a couple things that tarballs don't/can't specify that Docker containers can:

\- environment variables like locales. If your software expects to run with English sorting rules and UTF-8 character decoding, it shouldn't run with ASCII-value sorting and reject input bytes over 127.

\- Entrypoints. If your application expects all commands to run within a wrapper, you can't enforce that from a tarball.

You can make conventions for both of these like "if /etc/default/locales exists, parse it for environment variables" and "if /entrypoint is executable, prepend it to all command lines", but then you have a convention on top of tarballs. (Which, to be fair, might be easier than OCI—I have no particular love for the OCI format—but the problem is harder than just "here are a bunch of files.")

<comment><USER25> It's not necessarily a good thing for the container to be able to specify locale. Locale should be picked up from the surrounding system; it's just that unfortunately the surrounding system is usually not configured correctly.

And entrypoints/wrappers are definitely possible from a tarball. Just wrap the executables in bin/, replacing them with shell script (or whatever) wrappers pointing to the real executables. That's what Nix/Guix do for languages like Python which require dependencies to be provided by environment variables (as they don't have a way to "close over" the locations of their dependencies).

<comment><USER26> > Locale should be picked up from the surrounding system; it's just that > unfortunately the surrounding system is usually not configured correctly.

And around and around we go

</comment></comment></comment><comment><USER27> Nix has a very similar tool called nix-bundle[1].

[1]: [https://github.com/matthewbauer/nix- bundle](https://github.com/matthewbauer/nix-bundle)

</comment><comment><USER28> Tarballs don't have a TOC and can't easily index into individual entities.

One _could_ create a utility to make tarballs with a TOC and the ability to index while still remaining compatible with tar and gzip. Pigz is one step in the direction.

<comment><USER29> to list what is in a tar ball `tar -vtf tarball.tar`

to extract a particular entity 'tar -vxf tarball.tar path_in tarball_to_entity`

edit: good points on it not being efficient for large archives, just demonstrating it is possible.

<comment><USER28> A tar is a linked list of file paths and contents, it cannot be indexed to a particular file. A compressed tar has to first be decompressed and then the chain of links traversed. Accessing a file in compressed tar is o(n) with where the file is placed within the compressed tar stream.

It isn't that it is possible, it is that is horribly inefficient.

Zips on other hand unify storage and compression such that one has random access to particular file, hence most modern file formats are zips with xml or json inside.

</comment></comment></comment><comment><USER30> A quick FYI, Gobolinux operates much the same way.

1\. Binary packages are simply compressed archives (tarballs) of the relevant branch in the /Programs tree.

2\. branches do not have to actually live inside the /Programs tree. There are tools available to move the branches in and out of /Programs.

All this because Gobolinux leverages symbolic links as much as possible.

<comment><USER27> Gobolinux sort of does this. The main difference is GoboLinux uses "version numbers" while Nix & Guix use hashes. It makes a lot of difference for more complicated stuff.

<comment><USER30> True.

I suspect there are ways to introduce hashes to Gobo, if one were so inclined. But so far nobody has.

</comment></comment></comment><comment><USER31> Does anyone know how this would apply, for example, to sharing a Guile 2.2 application with Debian/Red Hat based distributions? I want to use Guile 2.2 for development, but I am worried because it was only recently was released for major distros (at least with Ubuntu I know it was released with 18.04) and it doesn't seem to support the creation of executables.

<comment><USER32> See this older discussion on statically linking guile [0], one should be able to bake your source into a C program that statically links Guile 2.2 to create a self contained executable. If that is too cumbersome, I would use a container.

[0] [https://lists.gnu.org/archive/html/bug- guile/2013-03/msg0000...](https://lists.gnu.org/archive/html/bug- guile/2013-03/msg00002.html)

</comment></comment><comment><USER33> Please, can we move to an archive format that isn't so sprawlingly massive ?

<comment><USER34> Are you complaining about the complexity of file format itself? My understanding is it's pretty simple: a linked list of headers with the contents of each file after each header. Or are you complaining that it doesn't do compression itself like ZIPs do?

<comment><USER35> One think I dislike about tarballs is the lack of random access support.

<comment><USER36> You can build random access around tarballs, just need to index the header data.

Shameless plug, this is what [https://github.com/cpuguy83/tarfs](https://github.com/cpuguy83/tarfs) does.

Granted, you do have to traverse the entire tarball.

<comment><USER37> > You can build random access around tarballs, just need to index the header > data.

> Granted, you do have to traverse the entire tarball.

So you can't randomly access a tarball, you can cache the linear access you've already done.

<comment><USER38> You can read all the headers without reading the whole file by just seeking over the file data....

<comment><USER32> But you would still have to decompress it. How much support for legacy systems do we need vs just making a slightly better version of a jar?

<comment><USER36> Decompression has nothing to do with tar, though. But I agree, it is painful to deal with tar+gz.

<comment><USER32> > Decompression has nothing to do with tar

Which is exactly the problem. Same issue occurred via volume and filesystem management resulting in ZFS. We need systems that compose and also elide.

So something that both archives as set of files and compresses them w/o losing affordances over the layer below it in the process.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER39> Articles like this are pointless. I get that guix and nix are neat, and I think that every single time something about one of them is posted, but I don't have the slightest clue how to use either one of them.

Do you want to convince people that something like guix is better than docker? Then take something that is currently distributed using docker and actually show how the guix approach is simpler.

i.e. I have a random app I recently worked on where the dockerfile was something like

      FROM python:2.7
      WORKDIR /app
      ADD requirements.txt /app
      RUN pip install -r requirements.txt
      ADD . /app

      RUN groupadd -r notifier && useradd --no-log-init -r -g notifier notifier
      USER notifier
      EXPOSE 8080/tcp
      CMD ./notify.py

How do I actually take a random application like that and build a guix package of it?

Another project I work on is built on top of zeromq, and it would be great to use something like guix to define all the libsodium+zeromq+czmq+zyre dependancies and be able to spit out an 'ultimate container image' of all of that, but all this post shows me how to do is install an existing guile package.

<comment><USER15> With Guix you get full introspection of your entire package dependency graph, you can check and manipulate every aspect - and it is still simple and easy to work with. With GuixSD you get this same introspection and overview, but of your entire system. creating a container, vm or even a docker image is a simple '$ guix system <container|vm> config.scm' away. And your config.scm is as complex as you like it to.

The simplest way would be to package the app for guix and you could just run '$ guix environment <name-of-package>' and you would be dropped into an environment with all your dependencies and whatever else the application requires in your path ready for hacking, get your sources and editor and start working.

If you need a vm or similar though I'd translate your example above into a system config where:

\- packages include python-2.7 and whatever is in requirements.txt (this may mean you have to package a few things, but again this is usually super easy)

\- users and groups are added to the config, as they always are, no extra step necessary.

\- exposing ports and networking is available as options for qemu script guix produces to launch the vm.

\- CMD ./notify.py: create a "simple" service that can be autostarted by the system on boot.

\- filesystem access is also handled by arguments to the qemu script.

As always though there are several paths to Rome, and these are just two of them.

Zeromq and libsodium are already packaged on guix, czmq and zyre looks like they would be simple to package, guix is really quite simple to work with, which I think is the reason so many of the users and devs are running it as our daily drivers, even though it is strictly beta (0.14. I think is the last release).

And pointless, come on - what does that even mean? Does it mean you don't value them? I was quite happy to read about a neat new thing I can use my favorite tool for.

<comment><USER39> > With Guix you get full introspection of your entire package dependency graph

Yes, I know all that. It's neat. I would like to learn more about it.

> The simplest way would be to package the app for guix

I was asking how to package the app for guix, and your response is the simplest way would be to package the app for guix...

> If you need a vm or soimilar though I'd translate your example above into a > system config where: - packages include python-2.7 and whatever is in > requirements.txt (this may mean you have to package a few things, but again > this is usually super easy) - users and groups are added to the config, as > they always are, no extra step necessary. - exposing ports and networking is > available as options for qemu script guix produces to launch the vm. - <USER2> > ./notify.py: create a "simple" service that can be autostarted by the system > on boot. - filesystem access is also handled by arguments to the qemu > script.

Yes, I'm sure it is super easy. How do I do it?

Do you know how to use the dockerfile I posted above? You run

      docker build -t myapp .
      docker run myapp

that's super easy. 9 lines and 2 commands. You can now add docker expert to your resume.

> Zeromq and libsodium are already packaged on guix, czmq and zyre looks like > they would be simple to package,

Well, I was working on a fork of things, so I would have needed to install my forks.

> guix is really quite simple to work with

I'm sure it is!

> And pointless, come on - what does that even mean? Does it mean you don't > value them? I was quite happy to read about a neat new thing I can use my > favorite tool for.

You are correct, I don't really value posts saying how cool and easy something is and how much better it is than other solutions, when they don't actually present a complete solution someone can actually use.

I get that it is not other peoples job to teach me how to use something like guix, but do people not understand why things like Docker won?

<comment><USER15> Right, your dockerfile contains a requirements.txt with unknown complexity and number of packages, your app is without a name and does not have any links to code.

I'd be happy to provide some examples. Say you want your fork of libsodium:

      (define-public my-libsodium
        (package
          (inherit libsodium) ; now anything not defined in this package will be inherited from libsodium
          (source (origin (method url-fetch)
                    (uri "url-to-your-sources")
                    (sha256 (base32 "hash"))))
         ; Add whatever other fields your fork needs.
      ))

Sure it's slightly more verbose. That's a bit of the cost of having something you can actually rely on, with that degree of hackability.

If you actually want help to package these things ask on our mailinglist or IRC, we're happy to help with specifics. But you're basicly complaining that I didn't give you a concrete solution to a problem with several missing details that are important. Docker would not be able to instantiate your python project if it did not know the contents of your requirements.txt.

The thing is docker is huge and bloated; is far from secure, and will probably stay that way for the foreseeable future; has a more or less complete lack of introspection; and is not strictly reproducible (sure, it gets quite far along the way, but it really is not).

Guix on the other hand is rather lightweight, and you have a fair amount of control over how lightweight it should be; builds from source, and has a sort of hotpatching system for security fixes; has introspection and is quite close to bitreproducible.

Sure, docker is _easy_, as long as it works. And I'd argue that because of its complexity and obscurity it is not practically free software.

<comment><USER22> Regarding your concerns about Docker, I agree with that (even though I've been working on Docker and in the wider container community for almost 5 years now). However, there are plenty of tools that are compatible with Docker but provide similar benefits.

For instance, (from the openSUSE community which I'm a part of) we have KIWI that provides builds with full introspection on a package level (similar to what you're doing with Guix). If you build the image inside OBS (our build system) then if a dependency of your image is updated then your image will be rebuilt automatically and published in OBS (where it can be further pushed to any Docker/OCI registry you like). The packages are signed, and the image is also "signed" (though it currently signs the image artifact and doesn't use image signing since that is still not standardised). And most packages in openSUSE are bitreproducible (we build everything in OBS).

The above is far and above much better than the current standard in the "official" world of Docker, but unfortunately because OBS has a UI from the early 2000s (which is when it was written) it doesn't get enough attention outside of the communities that use it (and enjoy using it a lot). Everyone wants Dockerfiles even though they cannot provide these features (and you cannot get package manifests of your images without running a package manager in the image, which means you cannot get vulnerability information from the manifest).

[ Though I'm mostly talking about openSUSE here, I also happen to work for SUSE on the containers team. ]

<comment><USER40> > However, there are plenty of tools that are compatible with Docker but > provide similar benefits.

And Guix is one of them, remember? From the article:

> Add -f docker [to your `guix pack` command] and, instead of a tarball, you > get an image in the Docker format that you can pass to docker load on any > machine where Docker is installed.

:-)

> The above is far and above much better than the current standard in the > "official" world of Docker, but unfortunately because OBS has a UI from the > early 2000s (which is when it was written) it doesn't get enough attention > outside of the communities that use it (and enjoy using it a lot).

This is so true! I've mostly moved on from traditional, imperative package managers and associated distros in favor of the functional package management paradigm exemplified by Guix, but I still recommend openSUSE to my friends who prefer a more traditional/mainstream distro because of the love I have for the Open Build Service and Zypper.

The web interface for OBS does feel clunky these days, but it's a wonderful tool not just for improving the reliability and quality of software packages, but distributing them. Zypper is hands-down the most powerful and complete high-level package management tool I've ever used as part of a binary-based GNU+Linux distro. I love that openSUSE provides an instance of OBS that anyone can use for free to build packages for not just openSUSE but a TON of different distros.

I wish more people would explore, take advantage of, and celebrate OBS just like I wish they'd do the same with Nix and Guix!

</comment></comment></comment></comment></comment></comment><comment><USER41> Do tarballs still have that unfixed/unfixable bug where the extracted files will have the permissions of the person who untarr'd the file?

<comment> master-litty That seems sensible to me, what else would you expect?

<comment><USER41> I expect that they should preserve the ownership and permissions of the original file if I tell it to.

<comment><USER42> How can it have the same owner if it's a different machine, and users aren't the same ?

<comment><USER43> Do tarballs store the user/group names as strings or do they store the uid/gid instead?

It is one of the goofy things about Unix systems is most tools speak uid/gid and woah is you if two machines on the network have "bob" only as different uid's.

Not entirely sure if windows has the same problem as to be honest if you use active directory most of that stuff is auto-magic.

My hunch is that going with the ID vs. the "friendly name" has a bunch of trade offs and whichever you pick will come with serious drawbacks.

<comment><USER44> They can do either - traditional tar formats have uid as a number, but the newer pax format has both numeric and named values.

<comment><USER45> in fact, names are the default. "\--numeric-owner" must be passed to use numeric values.

</comment></comment></comment></comment></comment></comment></comment><comment><USER46> Reinventing Application Bundles only 30 years after NeXTStep, poorly.

<comment><USER27> Why poorly? I don't see anything worse about this.

<comment><USER46> Really? Seems like an awful lot of tooling for what is essentially "Put binary and dependencies in folder. Move folder around at will" in sane environments.

<comment><USER40> The tooling already existed because it's part of a stack that goes from build tool to package manager to operating system configuration manager, with all kinds of features for developers floating around along the periphery. It handles all of these things uniformly, reliably, reproducibly, and in a way that deduplicates shared dependencies.

This article is just showcasing a relatively small bit of tooling on top all that which makes it possible to reuse that work to produce containers out of the very same stuff, in a whole range of formats.

`guix pack` and `nix-bundle` are illustrations of how a novel solution (functional package management) to the very problem to which app bundling constitutes utter capitulation (dependency management) can not only retain the virtues the app bundle approach throws away in the hopes of making deployment simple, but even match it in ease of deployment when _none_ of the infrastructure of the package management system is expected to be present on the deployment target.

From where I stand, that's damn impressive.

All of this was achieved without the kind of 'standardization from above' that Apple gets to do on its platform. It's true that app bundling could have been a lot simpler if the Linux community lived in a locked box at the mercy of a Vampire King bearing the power to upgrade users' kernels in the dead of night without bothering to ask them, who preempted any diversity or choice in operating system components with a uniform common runtime, and gleefully ripped unseemly APIs out from under developers with every OS release. But instead— thank God!— we have such a wide range of environments under the name 'Linux' that I'm ready to agree with you and call it insane. Yet here we see that hackers made it work anyway, without bossing anyone around or compromising on the strengths of proper package management. And that's fucking awesome.

<comment><USER46> Boy, you sure make fragmentation, constant wheel reinventing, and the necessity of complex tooling to perform simple tasks almost sound like a good thing. I suppose it must be for the small percentage of people who value those things over actually being able to do stuff.

Given the near-complete lack of non-oss software support Linux has, it seems like both developers and users rather prefer uniform common runtimes and a lack of diversity in their operating system components. It's almost like a whole lot of things get much easier if there's some kind of standardization.

<comment><USER40> > Boy, you sure make fragmentation, constant wheel reinventing, and the > necessity of complex tooling to perform simple tasks almost sound like a > good thing.

Why, thank you!

Redundancy of efforts in F/OSS is of course a bad thing. It's perhaps even more tragic in free software than in proprietary software, because in free software, developers have fewer formal barriers to drawing upon the work of others. But it's something free software projects can't simply disable by exerting brute control over their users and contributors. The point is that with tech like this, the hackers behind projects like Guix have triumphed in a tougher struggle than NeXT or Apple ever picked. And they've built technology that copes with a wider range of environments, not via ugly hacks on edge cases, but through a thoughtfully designed build system which renders the whole dependency tree of every program it builds transparent, reproducible, and portable. That they had to build a vehicle for such wild and varied terrain is not what I'm celebrating, the cool thing is that they _did_.

> Given the near-complete lack of non-oss software support Linux has, it seems > like both developers and users rather prefer uniform common runtimes and a > lack of diversity in their operating system components.

Alternatively, when you refuse to distribute source code, compatibility for you involves greater demands on your platform, because you can't leave downstream distributors to recompile and you refuse to allow your more capable users to fix your software's incompatibilities. It's almost like a whole lot of things get easier when you distribute source code with your application.

Regardless, I think there are a lot of factors that together explain the predominance of free software on free operating systems. Proprietary software companies aiming to hit as large a market as possible with a single codebase turning away from perceived fragmentation in the 'Linux market' is certainly one of those many factors.

<comment><USER46> > Alternatively, when you refuse to distribute source code, compatibility for > you involves greater demands on your platform, because you can't leave > downstream distributors to recompile and you refuse to allow your more > capable users to fix your software's incompatibilities.

And yet Windows still manages to run software written for a decade+ old version of it, and users often make compatibility patches for now-unsupported software, all without the source or recompilation. I think a big misstep by the OSS community has been its reliance on the crutch of "you have the source, do it yourself", and that includes making their software even work on a system in the first place. It leads to thinking like "it's ok if we break backwards and forwards compatibility, everyone can just recompile!".</comment></comment></comment></comment></comment></comment></comment>


Want to get rid of ads? Pay Google a fee. - <AUTHOR> http://www.cnbc.com/id/102207790

<comment><USER2> reminds me of [http://www.robinsloan.com/epic/](http://www.robinsloan.com/epic/)</comment>


New Actionscript compiler from Adobe - <AUTHOR> http://www.bytearray.org/?p=4789

<comment><USER3> If anyone is still interested in doing Flash development with Flex, do yourself a favor and check out Haxe:

<http://haxe.org/>

The AS3 compiler in Flex is junk. My buglist over the three years I used it was nearly 100 items long. I don't know of a single unfixed issue in Haxe.

Plus, if you write the code properly, you can port it to nearly a dozen platforms (serverside, plus all of the ones on <http://haxenme.org>).

I don't make Flash games any more except for hobby projects, but migrating off of Flex was the best decision I ever made.

<comment><USER4> This is actually an entirely new compiler written from scratch since 2010.

<https://cwiki.apache.org/FLEX/falcon-overview.html>

<comment><USER3> The problem with Flex wasn't its design, it was how buggy all of the edge cases were. Sound files randomly being replaced, weird compile errors that go away once you reset, code generation errors fixed _by adding comments_ , occasional minute long compilation times, out of memory errors (again, reset the computer).

Frankly, complete rewrites under these circumstances are a cop out. It's an easy but ineffective path. What they should be doing is unit tests, isolating bugs and strictly controlling quality. Common bug reports should be triaged, rather than ignored (it's not uncommon to have a bug, and find a set of posts years old discussing workarounds). All it would take to fix Flex would be the will to do it.

The rewrite will have a whole different bunch of issues, and be similarly problematic. It's not like Flex was a bad apple in a bunch of well engineered Adobe software.

<comment><USER5> Did you ever see that movie where that guy suddenly wakes up in a pool of sweat gasping for air because of that nightmare sequence?

That's me when I remind myself of Flex development.

<comment><USER6> Same here!! One of my startups failed because we chose Flex/AIR as the development platform. Nearly tripled our development time, working around bugs in the platform.

<comment><USER7> I've been writing in ActionScript 3/Flex for more than 3 years now and I don't really understand your problems. There never was a compiler bug that completely prevented me from doing my work. There always were easy workarounds to everything. The community is huge. I could always find the answer either by reading the documentation or googling.

Sincerely, compared to C++/Qt/WxWidgets, Python/Qt, Swing and even HTML/jQuery Adobe Flex was the most productive environment for frontend and desktop development for me.

It sounds that you did not have a prior experience in Flex otherwise you would know the kinks of the platform. I would say that it was not the choice of Flex that killed your startup but the uniformed choice of a technology that you never had a prior experience with.

<comment><USER6> 3 years huh? I reckon you have never worked with Flex 2/3 then? The incident I am quoting is over 4 years old and I understand things may have improved with Flex 4. Specifically the Spark library drastically improved UI performance. However for us early adapters its too little too late. A Java backend with a Swing or HTML/JS front end is the safest route to go when programming for multiple platforms.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> Interesting. I should note that most of this stuff (optimization, etc) is available through Haxe. Haxe can also compile to Javascript, iOS & Android; targeting just Flash is risky at this point.

<comment><USER9> Just to add to this. For mobile compilation targets, Android, Blackberry, iOS and webOS, support is added through the Haxe NME framework: (www.haxenme.org/). Haxe doesn't provide those compilation targets on its own.

</comment></comment><comment><USER10> Hmmm, Isn't this late by about 7 to 8 years. We've been asking, requesting and begging them, (while Flash was till with Macromedia) to improve the compiler, make it faster. One of the dreaded thing about Flash was that we have to wait for minutes while it compile. We dreaded the notion of pressing "Ctrl+Enter" on the keyboard.

MTASC by Nicolas Cannasse, came to our rescue and it was a good one - <USER2> fast. Nicolas is also behind haXe.

<http://www.mtasc.org/> <http://haxe.org/>

<comment><USER11> Haxe is actually the "successor" to MTASC

</comment></comment><comment><USER12> It all sounded so AWESOME, up until...

Known issues: Flex compilation is not supported.

So much for multi-threaded compiling. Pfff. I guess we will just wait until the open-source people do it, sometimes next year?

</comment><comment><USER13> Non-linear control flow added to AS3 through a new 'goto' keyword.

<comment><USER14> I don't know what to make of this change. I always assumed goto was something you designed into your language from day one, rather than bolted on in day 4000 or so. Can someone more familiar with language design (or the target use cases--I presume high-performance game programming?) comment on this further?

</comment></comment><comment><USER15> > A function can be inlined when the following constraints are met:

> ...

> * The function does not contain any activations

Does this mean that a function can't be inlined if it calls other functions?

<comment><USER16> Activations are closures, in AS3 compiler parlance.

</comment></comment><comment><USER17> This looks cool, but I'm afraid it's too little, too late. I've been pure Javascript for a long time now.

</comment><comment><USER18> That's all nice, but where are the threads support we have been promised since years back?!

<comment><USER19> Just released last week in Flash Player 11.4: [http://helpx.adobe.com/flash- player/release-note/fp_114_air_...](http://helpx.adobe.com/flash- player/release-note/fp_114_air_34_release_notes.html#overview)

</comment></comment><comment><USER20> Who cares?

The IDE is what needs demolishing and rebuilding properly.</comment>


Ask HN: Which (if any) content feed(s) do you "digest" aside from HN? - <AUTHOR> <comment><USER2> I follow prominent people in web development community on twitter.com and so far it's been the best source ever.

</comment><comment><AUTHOR> I visit reddit out of habit, but I'm looking to cut down since the new experience is so... commercialized I guess.

</comment><comment><USER3> Kottke.org Swiss miss DF</comment>


Show HN: Mastering Modern Payments: Using Stripe with Rails - <AUTHOR> https://www.masteringmodernpayments.com/

<comment><USER2> Hey, author here. This is the second edition, which includes a significant number of edits and a brand new chapter on subscriptions. Here's a link to the preview chapters:

[https://www.masteringmodernpayments.com/read](https://www.masteringmodernpayments.com/read)</comment>


Which startup should I go to? - <AUTHOR>

I'm a student at Hyper Island (http://www.hyperisland.com/) in Sweden and I'm going on an internship after the summer holidays. I really want to go to startup in the San Francisco area but I'm a bit unsure which one I want to go to.
I really love what Nest.com, Stripe.com, SquareUp.com and Pinterest.com are doing. They are solving real problems and seem to have a lot of talented people with them.
Where would you go if you where in my situation?
Thanks in advance to all the amazing HN-people.
Holger HolgerE.dk <comment><USER2> I don't think any one of the startups which you mention currently looking for interns.

<comment><AUTHOR> I'm sure there must be a way to persuade companies to take you as an intern.

Do you have any suggestions on startups you think are doing some really interesting stuff in the San Fran area?

<comment><USER2> I believe that these companies are in early stages of their life which is a more chaotic era of a company. I would suggest startup-like bigger companies such as Twitter and Facebook.

<comment><AUTHOR> That's what I want to get into. We've had several former students from this school who've joined these chaotic startups. I think it sounds fantastic.

</comment></comment></comment></comment><comment><USER3> What real problem is Pinterest solving?

</comment><comment><USER4> Paid or unpaid intern?

<comment><AUTHOR> Paid would be nicest, but if it's a really cool place, where you can learn a lot, unpaid would do too.</comment></comment>


Why Smart People Are Stupid - <AUTHOR> http://www.newyorker.com/online/blogs/frontal-cortex/2012/06/daniel-kahneman-bias-studies.html <comment><USER3> The research is using SAT score as a proxy for general intelligence... I wonder if this sort of heuristic short-cutting actually correlates with test- taking ability more than it correlates with intellgence.

A lot of "test-taking" training basically consists of saving time by training _away from_ full reasoning, in favor of cheap-and-good-enough heuristics. Furthermore, those heuristics are over-fitted to the particular problem types on standardized tests. I wonder how much of this study is actually measuring their ability to trigger test-taking instincts on problem types they're not designed for.

<comment><USER4> It is ridiculous to suppose that one's score on a multiple choice test is an accurate measure of innate ability or real-world intelligence.

The SAT, ACT, IQ tests, and all standardized test like them are socially constructed concepts that ATTEMPT a method of measuring intelligence. Intelligence (in the real world) reaches far beyond one's abilities to answer multiple choice reading comprehension, basic math and writing. Not to mention that problem solving in the real world has no time constraints.

Beethoven would not have gotten a perfect score on his SAT's. However, we all can attest to his innovation, creativity and musical genius. How can a multiple choice test measure the creative abilities of people like Sir Richard Branson, Steve Jobs, or Pablo Picasso?

The idea that "smarter people... were slightly more vulnerable to common mental mistakes" is a nonsensical conclusion. These findings are completely worthless.

<comment><USER5> Don't worry, you're both right. No self-respecting researcher uses SAT scores as proxies for intelligence in the social sciences.

<comment><USER6> The study detailed in the linked article does exactly that...

</comment></comment></comment></comment><comment><USER7> For the types of testing that he's doing, I suspect he's measuring boredom more than anything else, especially since he's testing largely in a university setting. Intelligent people are accustomed to being bored with endless entry- level evaluation exams, and at first glance this looks like it's just one more of them. And because the stakes here are so low (essentially zero), lots of people will just fly through without really reading and analyzing the question.

What he's seeing isn't something new, it's something so old that it's part of popular culture: the absent-minded professor syndrome. It's the stereotype of the brilliant physicist forgets what he's supposed to buy at the supermarket because he's thinking about their quantum properties. Analytic people are horrible at things that don't interest them.

Pay the students $50 for each correct answer, and there's not a doubt in my mind that the results will be the complete opposite of what he's seeing now.

<comment><USER8> I also agree with your comment. I did both questions as fast as I could and only got the first wrong and the second right. It reminded me of a brilliant Civil Engineering professor I once had who was showing us his notes on the projector (the kind with the light bulb and magnifying glass over head) and someone asked him if he could turn off the lights. The student meant the lights, as in the classroom lights so he could see the projection better. The professor turned off the projector instead :)

<comment><USER9> I saw a pound coin and a 10p coin (UK) and got the first one wrong, I saw a bar graph with a lot of doubling bars, and saw the long history of doubling and got the second one right. I saw the money by denomination, and not as a quantity perhaps. Interesting.

<comment><USER10> Just a sec.. where are you seeing that?

<comment><USER9> Sorry, I was describing the visual images I get when solving problems like this, the article is pure text.

<comment><USER8> Hmmm. Interesting that you point that out. I think if we were presented with a math formula (symbols), we'd have aced it.

I think the reason why my wife did it so fast is because it was fed to her as text only - in fact, I just read it to her. She's a lawyer and she's much better at interpreting text than most people.

</comment></comment></comment></comment></comment></comment><comment><USER11> The bat-ball and lily-pad questions are 2 of the 3 questions on a short test called the 'Cognitive Reflection Test' (or 'CRT') meant to measure whether people make the effort to think beyond the obvious (but wrong) answer.

By using those examples, after its headline, this article seems to imply smarter people do worse on these CRT questions. But that is _not_ what I've read elsewhere -- which is that the CRT is positively correlated with other quantitative measures of intelligence (including IQ scores, SATs, and high- school/collegiate grades). 'Smart' people (by those measures) do tend to do better on the CRT.

And if you read this article carefully, you see that while it uses these two CRT questions as examples of tricky questions, when it discusses the results about awareness-of-bias not helping alleviate bias, it isn't necessarily saying smart people do worse on those two CRT questions. It's a bit muddled in what it's saying, and reviewing the linked abstract doesn't help much either. The paper is evaluating some very specific things under the umbrella term 'cognitive sophistication', which might not map to what we usually call 'smart' or even 'test-smart'.

BTW, I personally think the CRT may be especially useful for evaluating software/systems proficiency. The bat-ball question probes understanding of algebra; the lily-pad question probes understanding of geometric growth (and someone accustomed to powers-of-2 will find it easier); the third question probes understanding of parallelism and projected-rates-of-work.

That third question happens to be:

"If it takes 5 machines 5 minutes to make 5 widgets, how long would it take 100 machines to make 100 widgets?"

(A software person might also think of it as: "If it takes 5 cores to compress 5 GB in 5 minutes, how long would it take 100 cores to compress 100 GB?")

<comment><USER12> >>(A software person might also think of it as: "If it takes 5 cores to compress 5 GB in 5 minutes, how long would it take 100 cores to compress 100 GB?")

I don't think that most compression algorithms are Embarrassingly Parallel (<http://en.wikipedia.org/wiki/Amdahl%27s_law>), so it's not clear to me the software scenario is equivalent, or are you saying that the jobs come in 5 gigabyte chunks?

<comment><USER11> You could raise the same objections to the original 'machines'/'widgets' formulation. (There are always economies or diseconomies of scale.)

Typically such questions imply the unstated assumption, "answer to the same level of precision/abstraction as the question itself, and assume you have all the info needed to give an answer". With that assumption both questions should be answerable.

Compression tends to be pretty parallelizable, if by nothing else than choosing to break the input into separate chunks. You might lose a little bit of efficiency in output size – more restarts, each compressor has less global information – but those don't mean slowdowns (and in a few contrived situations might even mean speedups at the cost of size). See for example 'pigz' and 'pbzip2'.

If I were asking this question, I'd accept the 'rough, assuming perfect parallelization' answer as correct-enough in the spirit and level-of-precision implied by the question. If the answerer brought up the difficulties in assuming perfect parallelizability or specific to compression algorithms or choice of inputs, that'd be worth some extra credit, and would trigger followups along the lines of, "how would those factors affect the size?" and "what bottlenecks might you expect?" and "could it ever be faster when split among more machines?"

</comment></comment></comment><comment><USER13> Link to the study linked in the article (PubMed prepublication abstract):

[http://www.ncbi.nlm.nih.gov/pubmed?term=west%20stanovich%20m...](http://www.ncbi.nlm.nih.gov/pubmed?term=west%20stanovich%20meserve)

The psychologist Keith R. Stanovich is quite controversial among other psychologists precisely because he writes about what high-IQ people miss in their thinking, but his studies point to very thought-provoking data and deserve to be grappled with by other psychologists. I have enjoyed his full- length book What Intelligence Tests Miss

[http://yalepress.yale.edu/YupBooks//book.asp?isbn=9780300123...](http://yalepress.yale.edu/YupBooks//book.asp?isbn=9780300123852)

which meticulously cites much of the previous literature on human cognitive biases and other gaps in rationality of human thinking.

And here is the submitted article's link to a description of the Need for Cognition Scale:

<http://www.liberalarts.wabash.edu/ncs/>

<comment><USER14> The book mentioned, "Thinking, Fast and Slow", despite the boring title, is quite good. If you're the sort to hang around lesswrong.com it won't blow your mind, those less exposed to those ideas will find it fascinating (and probably a bit more accessible in book form).

</comment></comment><comment><USER15> "Although we assume that intelligence is a buffer against bias—that's why those with higher S.A.T. scores think they are less prone to these universal thinking mistakes..."

This fallacy is at the heart of the matter. Intelligence and resistance against bias are only loosely correlated. Such resistance comes not from intelligence but from careful study and mental exercise, e.g. looking at various important ethical and philosophical arguments and analyzing them.

This is like saying all large people are strong. There is _some_ dependance but a smaller gym-fly can kick a slacker giant's ass. The sad thing, while it is obvious that you have to exercise your body to be healthy and strong, the fact that the same is quite through fro your brain is often overlooked.

<comment><USER16> Isn't resistance against bias a very requirement for considering someone intelligent? What exactly is intelligence if not the ability to think clearly?

To me, this looks like a definition game. Smart/stupid is a black & white view of looking at it and hence, misleading. As one overcomes his primitive biases, we call him smart, even though he remains susceptible to other biases.

In other words, people aren't smart or stupid. People's actions are smart or stupid in a particular situation.

<comment><USER17> Kahneman divides our thinking into two subsystems: type 1 and type 2. Type 1 thinking is fast, intuitive, unconscious thought. Most everyday activities (like driving, talking, cleaning, etc.) make heavy use of the type 1 system. The type 2 system is slow, calculating, conscious thought. When you're doing a difficult math problem or thinking carefully about a philosophical problem, you're engaging the type 2 system. From Kahneman's perspective, the big difference between type 1 and type 2 thinking is that type 1 is fast and easy but very susceptible to bias, whereas type 2 is slow and requires conscious effort but is much more resistant to cognitive biases.

Traditionally, "intelligence" (as colloquially defined) has correlated with type 2 thinking. So, a reasonable conjecture would be that people who are better at type 2 thinking would use it more and, therefore be less vulnerable to bias. However, this research shows that even those who are very good at type 2 thinking (as measured by their SAT scores and NCS scores) are even more vulnerable to cognitive biases. This is a deeply counter-intuitive result. Why is it that people who have a greater _capacity_ to overcome bias have a greater _vulnerability_ to bias?

<comment><USER18> > Why is it that people who have a greater capacity to overcome bias have a > greater vulnerability to bias?

Overconfidence. If you've become accustomed to thinking of yourself as being better able to avoid cognitive bias, you come to be confident in your abilities, to the point where you (perhaps unconsciously) think of yourself as not susceptible to biases.

<comment><USER17> That's certainly one possible explanation. Another possible explanation is that their brains are just faster _in general_ , so that even though their type 2 systems are faster than others', their type 1 systems are faster yet and manage to override even more consistently than in others. In any case, I don't think it's something that's "obvious" or "expected" by any means, and I do think that it should bear further investigation.

</comment></comment></comment></comment></comment><comment><USER19> Intelligence is overrated as a metric, from the get-go. Being smart doesn't mean anything - accomplishing something, whether that be writing a book, founding a company, making a new scientific discovery, sculpting a masterpiece, etc., is a much better metric.

Unfortunately everyone seems to be hung up on the "idea" of being smart, as if having a high IQ somehow constitutes an accomplishment.

<comment><USER20> I'm on the same page. IQ is property of the genetic dice roll, not something that a person earns. Tangible results based measurements seem more appropriate. Pure intellect is the raw material & needs to be refined/applied to be useful.

edit: grammar

<comment><USER21> The rest of your personality traits that allow you to accomplish great things (diligence, perseverance, focus, empathy, etc), as well as external factors such as being born at the right place and the right time, are also arguably a genetic / environmental dice roll.

<comment><USER20> I'd argue all of those areas you list are far more likely to be improved over the course of a lifetime than sheer intelligence. Pure intellect is pretty much set at birth, or at the least the ability to improve it isn't statistically significant. I'd argue that genetic dice roll and environmental dice roll are quite different things as well. You have FAR more ability to change your environmental situation than your genetic one. Does everyone have an equal chance to alter their environment? No, but who said life is fair?

</comment></comment></comment></comment><comment><USER22> If you'd rather not just accept your current level of cognitive bias, the web site Less Wrong has a bunch of articles by and for people trying to become less wrong about things. Anecdotally, I've noticed that people I know via the Less Wrong community tend to be decidedly less full of crap than average, so it seems to work. For example, here's a series of articles on the subject of avoiding excessive attachment to false beliefs, which I found to be generally entertaining and insightful:

[http://wiki.lesswrong.com/wiki/How_To_Actually_Change_Your_M...](http://wiki.lesswrong.com/wiki/How_To_Actually_Change_Your_Mind)

Any of those articles are a good place to start, so don't be intimidated by the amount of stuff there.

<comment><USER23> _"Logic is the beginning of wisdom, not the end" - <USER2>

Anecdotally I found that the Less Wrong community tends to be decidedly _more_ full of crap than average. In the same vein as spiritual materialism[1], many people that engage in a bias witch-hunt seem to be falling prey to "logical materialism", where the whole exercise turns into people deluding themselves into thinking they're somehow "better" than others because they're less full of crap than average.

It's good to know thyself, but it's no use if your knowledge isn't tempered by wisdom, and you're not going to get that by reading blog posts about cognitive biases online, no matter how good the posts.

[1] <http://en.wikipedia.org/wiki/Spiritual_materialism>

<comment><USER24> It's nice to know I'm not the only one who thinks this. I think it's really telling that a community which is ostensibly concerned with the science of achieving desires spends so much time focused on the "problem" of akrasia.

It's also heartbreaking to see intelligent people getting so excited about ideas like cryonics and personality uploading. I mean, they're interesting things to talk about, but a lot of people on LW seem to actually think they might get to live forever. It's kinda sad.

<comment><USER25> > actually think they might get to live forever

And they aren't horrified at the prospect of that being possible?

Not much thinking going on there, I suppose.

<comment><USER26> Why would you be horrified at the prospect of living?

I suspect it is _you_ that needs to put a bit more thought into this matter.

<comment><USER26> Oh wow! A downvote! So, the message I'm getting is that a) it is appropriate to be horrified at the prospect of living; and furthermore b) asking why is inappropriate...?

This is seriously weirding me out.

<comment><USER27> The message I've taken from this entire subthread is "try to laugh rather than shake my head." The behavior makes a lot of sense if I frame it in status signaling (which helps make sense of so much that I suspect it of being too broad a framework). My own comment on the topic of living is: so it may be physically impossible to live _forever_ , I think shooting for even a "mere" 200 years is doable and would be fucking awesome. At least we're not dogs, they get less than two decades.

</comment></comment></comment></comment></comment></comment></comment><comment><USER28> I'm reading the book Priceless by William Poundstone which discusses the work of Kahneman and Tversky (among others) in great detail as it relates to the psychology of pricing (excellent read, btw).

This is O.T from what the article is saying but mildly O.T (meaning on-topic) and I'd love to hear HN's opinion on this.

One of the problems presented in Priceless is:

Would you rather $3,000 as a sure thing, or an 80% chance of $4,000 and a 20% chance of nothing

versus:

Would you rather a $3,000 loss as a sure thing, or an 80% chance of losing $4,000 and a 20% chance of losing nothing.

The erroneous path that most people take, in the eyes of these researchers, is that they set their base reference point at the sure thing, ie. they say "well the $3,000 is a sure thing so I can assume I have it".

If you do that, then your answers are different:

In the first instance you keep the $3,000 (because it becomes an 80% chance of winning $1,000 versus a 20% chance of losing $3,000).

In the second instance you go to court (because it's an 80% chance of losing $1,000 versus a 20% chance of winning $3,000).

However if you don't "rebase" your reference point, then you would make the same decision in both cases - that is you would take the 80% of $4,000 bet because it's "worth" $3,200.

As much as I realise what they're saying and they say it's statistically incorrect to do this, it really seems to me the most sensible way to make the decisions (which is, I guess, exactly what they're saying right? I'm human, ergo fallible to this kind of illusion).

The thing that kills me is this: if this is a one time thing, I'd rather be sure of the $3,000. If I'm buying and selling these bets all day, then sure I should take the $4,000 at 80% because even if I lose this round, the next time I take the bet will make up for it (ie. law of large numbers).

But what this problem doesn't address is how _often_ I get this opportunity? Depending on my circumstances, $3,000 could be a life changing opportunity, ie. if I "win" $3,000 or $4,000, my circumstances are essentially the same so I should always go for the sure thing. If I lose $3,000 or $4,000 I'm equally screwed, so I should take the risk and try and win in court.

What am I missing?

<comment><USER29> I'm trying to follow.

So you are saying that with the following scenario:

1\. +3000 at P100 or +4000 at P80

2\. -3000 at P100 or -4000 at P80

Depending on the context, the answer is basically universally different if you say "1 then 2" as opposed to "just 1" or "just 2"?

<comment><USER28> The issue is that the answers are (yes basically) universally different regardless of whether both options are provided, but that statistically speaking they are identical.

The foible of humanity that makes the answers different is that we first "rebase" our expectations to the 100% chance, rather than considering our current position to be the baseline.

<comment><USER29> Yes, statistically they are identical. However, the fact that people consider the significance of the objects and the context of the decision isn't a fault of human intellect.

Really, stating otherwise is a fault of human analysis. It's a game theory problem I think (likely a variation of the stag hunt: <http://en.wikipedia.org/wiki/Stag_hunt>). It's a paradox only because Armchair mathematical intuition fails to explain it.

<comment><USER30> No because the stag hunt family of problems are about trust in other people, the situation in the OP is not. It's about risk aversion. Whether or not that is a 'fault of human intellect' is a matter of definition and up for debate, it's not as clear-cut as you make it out to be.

<comment><USER29> Ok thanks. I am no specialist in this by any means. But I do know enough to be cautious when dealing with these types of problems. Glad you clarified.

</comment></comment></comment></comment></comment></comment><comment><USER31> This explains, when you look in your git repository for who created a bug...

When you find it and it's by someone else, it was obviously a stupid, idiotic error that you would never make.

When you find it and it's your own, it was obviously an understandable mistake that anybody could have made.

Particularly if you consider yourself a great coder.

:)

<comment><USER22> You're more likely to go back and notice the stupid broken code you wrote, because it disproportionately comes back to bite you later. The good code just works, so you don't notice it as much. Classic selection bias.

</comment></comment><comment><USER32> Got the ball and bat one right, and the lily pad one. I must not be as smart as I hoped :-(

I think it comes down to having a value system where you'd rather be wrong and corrected (even if you have to do it yourself), as opposed to always projecting yourself as"perfect". Once you accept you aren't perfect, its easier to work towards perfecting what you've got.

<comment><USER33> Ah, that's the author flattering the reader. If the author had picked a genuinely tricky example for the first one it would have turned many readers off.

Of course a question like the old bat and ball one is ridiculously simple, _after_ you've been warned that many people get it wrong and hence that you should probably stop and think for a few seconds before blurting out the first answer that pops into your head. Do it without that warning and it's easier to get it wrong.

<comment><USER34> When I first encountered these, without the "it's going to be tricky" priming, I must admit I got the bat and ball one wrong until I revisited it. The lily pad one I got right with no deep thinking - my explanation is that experience working with bits primed me for treating "doubles" more correctly. That's post hoc and anecdotal, of course, so take it for whatever, but I thought it interesting...

</comment></comment></comment><comment><USER35> Well, at least people are collectively smarter today compared to 100 years ago - the percentage of people who can answer those questions correctly has gone up considerably :-)...

Also, I just _hate_ these kind of questions - they've always been used to prove that I'm stupid by those who knew the answers, and they're not solving anything useful - I need the problem to solve something I care about in order for my brain to fully focus on it and "do the math"...

<comment><USER36> These questions are not designed to demonstrate intelligence if you get them right. They are designed to show how our cognitive biases trip us up, often without us even noticing. You're not stupid if you get them wrong, you're completely normal.

</comment></comment><comment><USER37> FYI the tallest tree in the world is ~ 116 m or 379 feet.

<http://en.wikipedia.org/wiki/Hyperion_(tree)>

<comment><USER38> That question sort of bothered me - I have no idea how big a tree is, and even if I saw one, I have no reference for how many feet it is without doing some fairly exhaustive mathematics (and at the scale of "largest redwood", I'd likely be wrong). Given _some_ information about redwood trees, _of course_ people are going to use that information in the subsequent guess. They're not going to imagine a redwood tree, then imagine a building next to it, then count the floors and estimate the height. Or estimate the girth, then guess a height/girth ratio that makes sense given the composition of a tree, and then estimate the height. They're lazy.

If I asked you, "Will a frooble fit in my pocket/empire states building?", and then asked you to estimate the average size of a frooble, you'd certainly take into account my earlier question.

See <http://lesswrong.com/lw/k3/priming_and_contamination/> for some better examples. IMO, the more insidious form of anchoring is contamination (vs sliding adjustment).

</comment></comment><comment><USER39> For all of the high and mightiness of this article, this bugged me:

In a lake, there is a patch of lily pads. Every day, the patch doubles in size. If it takes 48 days for the patch to cover the entire lake, how long would it take for the patch to cover half of the lake?

If a lilypad is 20 square inches (which is probably conservative), and you started with 1 lilypad, after 48 days of doubling it would cover 1.4MILLION square miles. That is 44 times the surface area of Lake Superior.

I get the point of the question, but if you're trying to play "gotcha" on people, at least ask a reasonable question.

<comment><USER25> It was 20 days when I was a kid.

Inflation, I suppose.

</comment></comment><comment><USER40> My takeaway is that smart people are in fact fairly dumb, in other words even fairly bright specimens of homo sapiens make stupid mistakes and irrational decisions quite often because of this shortcutting.

I also think that on the other hand those types of shortcuts are actually probably very useful aspects of our human intelligence.

I think that within 50 years or so we will see new species/upgraded humans or AIs that actually don't have those problems, because they will have built-in checks and alternative types of intelligence that rely on those shortcuts less.

</comment><comment><USER41> I highly recommend reading "Predictably Irrational" by Dan Ariely if you thought this article was interesting. It covers exactly this subject and makes for fascinating reading. I picked up the book about a week ago via some other post linked here on HN and I'm loving it.

[http://www.amazon.com/Predictably-Irrational-Revised- Expande...](http://www.amazon.com/Predictably-Irrational-Revised-Expanded- Edition/dp/0061353248/ref=sr_1_1?ie=UTF8&qid=1339561700&sr=8-1)

</comment><comment><USER42> Got the lily pad one. I still cannot believe I said 10 cents on the first one with a completely non-sarcastic chuckle.

This article reminds me of pg's reasons to have a co-founder to avoid being delusional. Better be proven wrong on the inside than on the outside.

edit: Although on second thought, I think this bias theory probably extends to organizations as well. Probably that's why big companies sometimes can't see the obvious which a startup does.

</comment><comment><USER43> Full text: <http://dl.dropbox.com/u/5317066/2012-west.pdf>

</comment><comment><USER44> Well I got both those questions right by following the heuristic that the most obvious answer off the top of my head would not be the answer.

To my mind on any test that was supposed to be hard the appearance of any obvious an answer triggers me to check for the proverbial trick question.

On the other hand, most brain puzzler type questions that get discussed on HN (for example interview questions at Google) I find to be damn hard. I can't imagine that "smart" people would do worse than "stupid" people on truly hard problems. I guess that is the area of bias being pointed to in the OP.

</comment><comment><USER45> All of this is covered (much better)in Kahneman's 'Thinking, Fast and Slow'.

</comment><comment><USER46> Here's a good mental test for the author:

When you're done with an article for the "Frontal Cortex" section, read it aloud to yourself and smack yourself in the head with a frozen herring for every time you use the word "we", "us" or "our" in your article. If you have a headache when you're done, burn the draft and rethink the whole thing, b/c your article obviously suffers from a "smug we" bias.

</comment><comment><USER47> I don't see why, when smart people are trained to be lazy, researchers are surprised that smart people are lazy.

<comment><USER48> I am surprised. I thought cognitive traits had high correlation. I did not expect laziness would factor into it so much.

</comment></comment><comment><USER49> Just a couple of weeks ago we had an article about why smart people don't think of others as stupid (<https://news.ycombinator.com/item?id=3984894>), and now they're stupid themselves? I'm puzzled.

<comment><USER48> A careful reading will tell you that the article you refer to is normative not descriptive.

</comment></comment><comment><USER50> _This finding wouldn't surprise Kahneman, who admits in "Thinking, Fast and Slow" that his decades of groundbreaking research have failed to significantly improve his own mental performance. "My intuitive thinking is just as prone to overconfidence, extreme predictions, and the planning fallacy"—a tendency to underestimate how long it will take to complete a task—"as it was before I made a study of these issues," he writes._

Doesn't Kahnman distinguish between intuitive and deliberate thinking? So it could be possible to think better by distrusting our intuitions and deliberating more, right?

</comment><comment><USER51> Well, the ball one I've heard before and the lilypad is obvious if you've been exposed to biology. Is this not more a matter of education not being applied to real-world cases and relying on theoretical teaching?

<comment><USER52> What, exactly, does the lilypad one have to do with biology...?

<comment><USER53> Population growth follows an exponential curve.

</comment></comment></comment><comment><USER54> There is a simple cure for smart people to not be stupid, they can detect those errors and bias easily in other people thinking, but not in their own thinking because introspection doesn't work. So the cure is to play as if you were the actor in a theater, that is pretend that you are not yourself when you are thinking. You should imagine you are thinking as a known stupid person and by miracle you get smart and not so stupid.

<comment><USER55> Easier said than done.

</comment></comment><comment><USER56> This isn't big news for me. It took me about 7 years to understand two courses of high school while finishing my master of science. If you ask me the right question or try to teach me just the right matter in just the right way, a donkey will get it sooner than I do, and I'm talking about possibly years sooner. I might just not see the problem or I might think the wrong way, I don't know. There are things I just don't get.

</comment><comment><USER57> New Yorker articles (that get posted here, anyway) always have some sort of take on things that attempts to bring down the good. Same for The Atlantic. I'm not saying they're always _false_ , but there is a certain kind of thing that these publications are interested in, and it's a kind of thing that makes me feel dirty---or as if they're trying to make me feel dirty. Anyone else noticed this?

</comment><comment><USER58> What did you think of first when you read about the bat and the ball problem? Also, what's your background (e.g. CS, Maths etc.)? As someone who has a relatively strong background in maths, I quickly saw the outlines of a simple, algebraic substituion problem. I'm quite interested in how people analyze problems, so I'd love to see how the HN community approached this.

</comment><comment><USER59> I think this is more an issue of the English language. English is not a good way to speak math or logic. In the bat and ball question I mistakenly (and I'm guessing everyone who got it wrong) ignored the word "more". That word represents an operator and is therefore crucial to the question, but is extremely diminutive in terms of English language.

<comment><USER32> Possibly, I immediately substituted one dollar plus ball for bat, then saw I had 2 balls and a dollar making up $1.10. Or 5 cents a ball. I honestly didn't pull out the algorithm sheet, it just sort of worked itself out in my head before I had time to think: what are they looking for...

Not to say I don't have biases, just not for word-number problems.

Poker on the other hand is another matter, I still chase straights and flushes in games with wild-cards, even though I _know_ those hands are almost worthless.

</comment></comment><comment><USER60> I answered the question correctly, which I'm sure the majority here probably did too. When someone asks a simple mathematical question, I always seem to give it more thought since I always know it must be a dodgy question. 4 years ago, I'd probably of answered the question incorrectly. But the baseball question is an obvious mindfuck.

</comment><comment><USER61> So because I'm suffering from a deep case of the derp today, how are the first guess answers to those questions wrong?

<comment><USER62> Since most people don't get calculus they use simple math to shortcut to answers that sounds right.

<comment><USER63> Calculus? These problems require elementary school arithmetic or, being generous, basic algebra.

<comment><USER62> Think it through.

The first problem is operational and the second problem is change on a slope.

People use what they know to deal with problems, so facing these two they use basic math (subtraction and division) being ignorant of higher-level math concepts such as algebra or calculus. The answers they come up with _appears_ right to their known level of logic.

If you were the type of person that got to learn about high level math concept, and are the studious type to double check answers, then these two problems are condescendingly seen as trivial.

<comment><USER63> I see it differently. The whole point of the article is that these problems affect _smart people_.

>For one thing, self-awareness was not particularly useful: as the scientists note, "people who were aware of their own biases were not better able to overcome them." This finding wouldn't surprise Kahneman, who admits in "Thinking, Fast and Slow" that his decades of groundbreaking research have failed to significantly improve his own mental performance. "My intuitive thinking is just as prone to overconfidence, extreme predictions, and the planning fallacy"—a tendency to underestimate how long it will take to complete a task—"as it was before I made a study of these issues," he writes.

It has nothing to do with knowledge of higher level mathematics because these problems are easily solvable with arithmetic. Lacking calculus doesn't kill you on this problem. An intuitive gut feeling that you've already arrived at the right answer and laziness is the source of confusion.

I've even read about that damn bat and ball problem and it STILL tripped me up this time. I could easily have double checked my answer, but I wanted to read the article. Even a child knowing nothing other than addition could get it right with a little bit of trial and error. I hope after admitting that you see that I _don't_ find the problems condescendingly trivial.

Personally, I found the second problem much easier... probably because programmers have a better intuitive grasp of powers of 2. Bringing in slope is stretching it a bit. Working in reverse from the completely covered lake, it should be obvious that going back one day halves the lily pads. However, I could imagine how someone more familiar with linear processes would get the wrong intuitive result.

>The answers they come up with appears right to their known level of logic.

A studious habit sure, but checking your answer isn't a higher level math skill.

</comment></comment></comment></comment></comment><comment><USER64> Where can I find more of these questions?

</comment><comment><USER65> I studied "Choice & Behavior" at Penn -- the names Kahneman and Tversky were a common refrain. If you're looking to self-teach, my prof Jon Baron has a great course outline online: <http://www.sas.upenn.edu/~baron/p153.html>

</comment><comment><USER66> do we all think about the same thing at the same time or does Jonah Lehrer read HN religiously?

i _just_ watched that talk a couple of days ago because it was posted here: <http://news.ycombinator.com/item?id=4082308>

</comment><comment><USER67> Comments here reinforce the research.

</comment><comment><USER68> These sort of questions always put me into "hold on, think about it" mode and statements like "Your first response is probably to take a shortcut" are simply not true. I'm actually more vulnerable to over-think a problem rather than provide a quick wrong answer.

</comment><comment><USER69> If the lily pad patch was a mere 1 square inch, on the 48th doubling, the pond would have to be (check my math)about 70k square miles - or 10 times the size of Lake Erie. Those lilies would be consuming a serious amount of co2 during that last doubling!

</comment><comment><USER70> Both those questions are trivial and I answered them correctly. This is in line with the article's conclusion: I don't consider myself very smart. I mean I had some moderate successes in my childhood at math competitions, I am a reasonably good programmer, but I am not very smart. I even failed at the on- site Google interview.

But here is the problem with the article: The people who I consider smarter than me (in the mathematical/IQ sense) also answer these kind of questions correctly. This includes my friend working at Google, some researcher mathematicians who I know from math forums who won serious math competitions as a child, etc... These questions are really-really trivial. The researcher mathematician guy who I know do not even make mistakes on 10x more tricky or hard questions, it is scary how he do not make mistakes and thinks incredibly fast. Something seems to be wrong with this study.

</comment><comment><USER71> I remember the SAT as more about checking oneself's first reaction to a problem. They often try to trick you with the obvious answer. The GMAT and GRE were quite similar. I would often have to stop myself from taking shortcuts

</comment><comment><USER72> A bat and ball cost a dollar and ten cents. The bat costs a dollar more than the ball

      bat + ball = 1.1
      bat = ball + 1
      2bat + ball = ball +1.1 +1
      2bat = 2.1
      bat = 1.05
      ball =0.05

<comment><USER73> I parsed that as:

        bat = 1.1
        ball = 1.1
        bat = ball + 1

It confused the hell out of me.

<comment><USER52> I immediately switched to generic variables X and Y, rather than keeping the original names. It helps.

</comment></comment></comment><comment><USER74> I think that smart people are also prone to falling for headlines like this. The reality is much more complex.

</comment><comment><USER75> The last line "The more we attempt to know ourselves, the less we actually understand." is worrying me a bit.

</comment><comment><USER76> Because they spend all day on HN?

</comment><comment><USER77> Taking more shortcuts as I get older too - a constant battle to stop and think...

</comment><comment><USER78> English is a terrible language for formalism: News at 11.

</comment><comment><USER79> Because they don't ask enough questions.

</comment><comment><USER80> No, research did't show that "we do this" or "our approach is that" or "humans aren't rational" -- what the research showed is that the _typical_ person does this or that.

A similar experiment where people draw the wrong conclusions is the Milgram experiment. Yes, most people are obedient to authority figures and do what they are told. But not _everyone_ acts that way.

This research likes to sweep the best human beings under the rug, as if being virtuous is not something to try to emulate, but is something to hide. This explains why the majority of people act the way they do. Perhaps if they were taught that their "we're only human" vices are not the ideal to emulate, perhaps if the best that humanity had to offer were put forth as the ideal instead, then these lesser human beings who make up the majority would become what they might be and ought to be.

<comment><USER81> Animals are less irrational than humans. Children are less irrational than adults. Why do you assume rationality is better, rather than maladaptive?

<comment><USER80> I don't "assume", I induce. See Newton's rules of reasoning. Pay particular attention to Rule IV.

It is clear from many examples that rationality gives us the utmost ability to adapt, prosper, and survive over the long term. And there is _no_ example that truly leads in the other direction. (There are many perverse definitions and applications of "rationality" that seem to trick some people into thinking it does lead in the contrary direction).

<comment><USER10> > I don't "assume", I induce. See Newton's rules of reasoning. Pay particular > attention to Rule IV.

u so smaht

<comment><USER80> Your deranged sarcasm eloquently sums up what's wrong with our educational system: kids are taught that their opinions matter, regardless of how idiotic they are. Well the truth is, your uninformed opinion doesn't matter.

</comment></comment></comment></comment></comment><comment><USER33> Now _that's_ the kind of headline I'd give to my article if I wanted it to reach the top of the HN front page.

</comment><comment><USER36> I notice this all the time, all over the place. It drives me nuts, to the point that I am now extremely skeptical of what we call "intelligence". Taleb's "The Black Swan" really opened my eyes to this. He talks a lot about how we reason in ways that do not correspond to reality.

I don't know what right is, but I know the way we currently think about intelligence is wrong.

<comment><USER59> Agreed. IQ is a terrible method of evaluation also. For instance, if you practice taking IQ tests, I guarantee you will get better at answering those questions. Even if you gauge only the first attempt, how do you know their daily job wasn't very similar to those exercises. In fact, IQ tests are very much like building software to only work in one architecture and then hoping it applies to others. It really only judges speed and word/number pattern recognition. This encourages short cuts and is hardly applicable to any practical situation.

</comment></comment><comment><USER82> I find the abundance of "See? Smart people are actually dumber than I am!" posts amusing.</comment>


A small hotel owner's nightmare with booking.com - <AUTHOR> http://hotelrevu.com/blog/a-small-hotel-owners-nightmare-with-booking-com/

<comment><AUTHOR> The story of how one hotel owner made the mistake of signing up with booking.com and the unfortunate events that followed.</comment>


Sci-fi writers like Arthur C. Clarke imagined we would feed space colonists - <AUTHOR> http://www.slate.com/articles/technology/future_tense/2012/06/arthur_c_clarke_and_other_sci_fi_writers_on_growing_food_to_feed_space_colonists_.html <comment><USER2> The actual title is "Beyond Astronaut Ice Cream: How sci-fi writers like Arthur C. Clarke imagined we would feed space colonists".</comment>