Road to OCIv2 Images: What's Wrong with Tar? - <AUTHOR> https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar <comment><USER2> I think articles like this are important... we need a good way of collecting knoweldge (and especially frustrations) like this in a compact format. Everyone who thinks, "I'll use tar," in response to some problem involving archives, backups, package distribution, container distribution, etc. will at some point run head-first into one of the things that makes tar _weird._ Without good collections of complaints, somebody comes in to solve the problem and ends up only solving the corner of it that they personally care about.

Something I personally find frustrating is that I'd like to be able to create a tar archive without recording UIDs, usernames, or timestamps. The command- line tools aren't really built for that, and yet this seems pretty reasonable, and libraries which create tar files do this by default. (This is covered under the reproducible builds section.)

The next thing that you're likely to want is some kind of random-access archive, for which you'll use either zip (which sticks the directory at the end) or squashfs, maybe.

Finally, the last thing that I often want is just a very binary key-value format which I can distribute and randomly access with reasonable efficiency.

Looking at things from the opposite perspective, I'm personally frustrated by the cross-platform personal backup options. Currently, I'm using Duplicity or some variant, but I find it a bit of and as a result my backup schedule is rather sporadic.

<comment><USER3> I recently learned about [SQLite archives]([https://www.sqlite.org/sqlar.html](https://www.sqlite.org/sqlar.html)) which feel like an interesting option to consider, especially with your comment around sometimes wanting a "binary key-value format" in mind.

Because it's "just sqlite" it's possible to use the archive functionality in archive-like ways, while also using the file as a sqlite db (because that's all it is).

It's not a perfect tool for all scenarios (and I'm thinking it wouldn't be good for backups), but I feel there are definitely some very useful scenarios for it.

</comment></comment><comment><USER4> It may sound esoteric but this stuff is actually Super Damn Important, for at least two big reasons:

1\. Security. Right now it is hard to trust a container image. Determining provenance and/or reproducibility are hampered by the image format and _especially_ hampered by the charming little quirks of tar.

2\. Performance. There are lots of optimisations that would be available if we didn't need to stream out an entire layer before we could do anything use with it. The tar family of formats is, as their name suggests, all about creating linear files intended to be saved _to a tape_. On a tape random access is bonkers. For a container, random access is a frequent operation. On top of that there's the whole hassle of shipping almost-but-not-quite-identical layers over and over and over. There must be petabytes of wasted bandwidth worldwide by now.

</comment><comment><USER5> I'm curious to hear what solution they are proposing for this, especially since there are at least _two_ existing proposals in that field, in the form of OStree and casync: [https://github.com/ostreedev/ostree](https://github.com/ostreedev/ostree) [https://github.com/systemd/casync/](https://github.com/systemd/casync/)

<comment><USER6> There are also a few others, and I've spoken to Lennart about using casync. It's a bit of a mixed bag because while casync does give us mostly what we want (and I've had a very long thread on Twitter with Lennart about it), there are some other concerns I've heard in the past few days from OCI users that make me quite cautious about using a format that cannot be easily made into a runtime format.

For instance, some folks want to have their runtime format be identical to the image format (so that signatures of the image can be used to verify the running containers). This is something that you cannot currently do with stock OCI (though they have worked around it), but should be a consideration for a future format. I only became aware of this concern after writing my blog post, so I will have to include it in the next one. :P

<comment><USER4> My last skim of casync suggested that it's heavily block-oriented. That's fine for backups and FS duplication, but as a distribution format it doesn't really fly.

A lot of folks want two levels of insight into the image's provenance or trustworthiness:

1\. Has the image been tampered with? This is mostly solved by TUF / Notary.

2\. What are the files and where did they come from? This means you need a file-level abstraction that is simple and fast.

Insofar as casync is block-oriented instead of file-oriented, it's a poor fit for the second problem. It doesn't matter how efficient the streaming and storage is if you make people download the entire layer each time you want to check a single file.

</comment></comment></comment><comment><USER7> Using Zip archives on Unix doesn't feel right, but it can solve a bunch of these problems straight away, while remaining highly portable.

<comment><USER8> How about 7z? How do 7z, zip, and tar compare in terms of support for Unixy things like permissions, uid/gid, xattrs, and so on?

<comment><USER9> 7z only supports DOS-style attributes, while Zip supports the full host of Unix metadata like tar does.

<comment><USER10> IIRC the unix metadata is not part of the canonical zip format, but rather an extension introduced by (?) infozip, an implementation of zip for Unix systems (if you use the zip/unzip cli tools on a Unix style system, it's most likely infozip).

</comment></comment></comment></comment><comment><USER11> A better solution might be to just emulate a block device with a sparse file and mount it somewhere with a loopback device.

<comment><USER6> This is what Singularity (and to a lesser extent, LXD) do. The main problem with this is that you don't get de-duplication of transfer (or really of storage) -- any small change in your published rootfs and you'd have to re- download the whole thing. In addition, it requires that the system you're mounting it on supports the filesystem you use (and that the admins are happy using that filesystem).

There is also a potential security risk -- since filesystem drivers are generally not secured against potentially malicious sources (there are plenty of attacks that have been found against the big Linux filesystems when you attack them with un-trusted filesystem data). This is one of the reasons auto- mounting USBs is generally seen as a bad security practice.

Don't get me wrong, there is a _huge_ benefit to using your runtime format as your image distribution format. But there are downsides that are non-trivial to work around. I am thinking about how to bridge that gap though.

<comment><USER12> What about some kind of binary delta diffs such as bsdiff?

<comment><USER6> Yes, and this is what LXD does. I think I mentioned it in the article, but basically the issue is that it requires one of:

1\. A clever server, which asks you which version do you have so it can generate a diff for you. This has quite a few drawbacks (storage and processing costs as well as making it harder to verify that the image you end up with is what was signed by the original developers). But this will guarantee that you will always get de-duplication.

2\. Or you could pre-generate diffs for a specific set of versions, which means it's a lottery whether or not users actually get transfer de- duplication. If you generate a diff for _every_ version you're now back to storage costs (and processing costs on the developer side that increase with each version of the container image you have). You could make it so that the diffs only step you forward one version rather than instantly get you the latest, but then you now have clients having to pull many binary diffs again.

This system has existed for a long time with BSD as well as distributions having delta-RPMs (or the equivalent for debs). It works _okay_ but it's far from ideal, and the other negatives of using loopback filesystems only make it less ideal.

In my view, dumb formats are best.

<comment><USER3> Using [zsync]([http://zsync.moria.org.uk/](http://zsync.moria.org.uk/)) is an alternative option from my understanding?

I could be technically inaccurate, but my understanding is that it's rsync but with the server serving a metadata file which allows the rsync-diffing to happen from the client side rather than the server side - hence no clever server required.

It also doesn't require diffing particular revisions; but only the different blocks will be fetched. It does require server the metadata file, but they're note very large afaik.

</comment></comment></comment></comment></comment><comment><USER13> As someone who was not programming when many of the customizations introduced in the article were introduced: this seems like a cautionary tale that goes against the wisdom about "competing standards": [https://xkcd.com/927/](https://xkcd.com/927/)

The desire to unify formats' competing back-compatibility needs created something that was (because of the standards conflicts/schisms/reunifications) extremely sub-par for most use cases, but (because of the time spent baking common interfaces) just usable enough that it remained the primary basis for storage formats for, if the author is to believed, far longer than it should have.

I wonder how many other tools that are venerated because of their age and ubiquity are similarly decrepit and broken when you peel back the curtain?

</comment><comment><USER14> Coincidentally, I recently submitted an article by Thomas E. Dickey on the subject of tar and portability.

* [https://news.ycombinator.com/item?id=18969290](https://news.ycombinator.com/item?id=18969290)

The hoo-ha over tar and cpio in the 1980s when the subject of standardization came up, and the whole raison d'être of pax, is largely forgotten nowadays. But the problems that were brought to light back then live on.

The thing to bear in mind is that we already have attempts to improve upon this. The problematic formats in the 1980s were _themselves_ products of the second-system effect, adding on various things to the original tape archive format. We are now, in the second decade of the 21st century, well into _umpteenth_ -system effect.

This brings up a point where this article is very wrong. The pax utility did not appear in 2001. pax was in the POSIX draft back in the early 1990s, and had become widespread enough to be in reference books by 1991. The article conflates the PAX extensions with the pax utility. This, and the incorrect dates, are errors promulgated by Wikipedia, which I challenged back in 2018 at [https://en.wikipedia.org/wiki/Talk:Pax_(Unix)#Incomplete_Inf...](https://en.wikipedia.org/wiki/Talk:Pax_\(Unix\)#Incomplete_Information) and which was probably the source of these errors in the article. Always double-check what Wikipedia says on computing topics with a proper reference.

Before being tempted to reinvent the wheel here, I recommend a look at all of the times during the so-called "archiver wars" where this wheel _already was_ reinvented, and learning from them. Of particular note, given this article, is Rahul Dhesi's ZOO file format from 1986. It allowed for multiple generations of a given file, and archive headers marked deleted files with a flag (allowing them to be undeleted), which could be used for "whiteouts". It suffers far less from an extension mess because support for filesystem features from Unix, MS-DOS, and VMS (at least as they were in 1986) was provided in the base data strutures; including 8.3 and long filenames.

But really the basic error here is in using an _off-line archive_ format for an _on-line live filesystem_ mechanism. It's the wrong data structure for the job. Whereas there _are_ right data structures, and have been for years. The history of filesystem formats development includes several cases of addressing the very things mentioned in the article, from deduplication (c.f. ZFS) through generations for deleted files (ODS for Files-11 in VMS, where ZOO got the idea from) to reproducible directory scan orders (c.f. the side-effects of using B-trees in HPFS).

<comment><USER6> > This, and the incorrect dates, are errors promulgated by Wikipedia [...] > which was probably the source of these errors in the article. Always double- > check what Wikipedia says on computing topics with a proper reference.

I didn't use Wikipedia, actually. The primary source was the libarchive documentation, star, bits of GNU tar's docs, and POSIX. The main issue is that it's hard to get a copy of POSIX.1-1988, let alone POSIX _drafts_ from the 1990s.

EDIT: Also, I didn't actually notice there was a rationale section in POSIX.1-2001 which references PAX as existing in earlier standards. I will read through it and update my article accordingly. Thank you!

> Of particular note, given this article, is Rahul Dhesi's ZOO file format > from 1986.

Funnily enough, I have heard of ZOO (not sure where) and looked into it. While it does support file versions (and deletion) and has many improvements over tar, there are many other properties it doesn't have that we'd need (and last I checked there's no real support for it in modern Linux and Unix-likes -- so it makes no difference from a ubiquity perspective).

> But really the basic error here is in using an off-line archive format for > an on-line live filesystem mechanism.

We don't use tar archives as the live filesystem for containers, it's used as a distribution mechanism.

> from deduplication (c.f. ZFS)

While ZFS has de-duplication, it's not really the kind we need and (from memory) zfs-send doesn't include the de-dup tables so they're all generated on the receiving end. Ideally we'd want content-defined de-duplication because that way you can reproducibly generate the blobs.

<comment><USER14> I suggest books. (-:

Fred Zlotnick's 1991 book mentions pax and the (contemporary) POSIX.2 draft.

Mark Horton's _Portable C Software_ from 1990 has a command options listing for pax.

That's just two of the books.

</comment></comment></comment><comment><USER15> Reminds me a bit of "PSD is not my favorite file format"[0]

0: [https://stackoverflow.com/questions/5355708/psd-file- format#...](https://stackoverflow.com/questions/5355708/psd-file- format#5355949)</comment>


Clojure 1.4 released - <AUTHOR> https://groups.google.com/forum/?fromgroups#!topic/clojure/H4f2nbB6gWI

<comment><AUTHOR> Changelog: <https://github.com/clojure/clojure/blob/master/changes.md>

</comment><comment><USER2> mapv and reduce-kv are a great addition; not so much as a feature per se, but rather as a testament of the core team being on the lookout for design patterns that are actually symptoms of lacks in the language [[http://c2.com/cgi/wiki?AreDesignPatternsMissingLanguageFeatu...](http://c2.com/cgi/wiki?AreDesignPatternsMissingLanguageFeatures)].

<comment><USER3> I'm not really sure what you mean... unless I'm missing something mapv is the same as (into [] (map ...)), and based on the source that appears to be exactly what it is. reduce-kv is a bit more complicated but it's not really a huge leap from reduce. Nice additions, to be sure, but are either really design patterns that indicate some lack in Clojure 1.3?

<comment><USER2> Codebases were littered with (vec (map ...)) and (reduce (fn [acc [k v]] ...)) patterns which are nothing but noise. This change removes the need for that, makes the intention clearer and in the case of reduce-kv removes the need for lambdas in some cases (where it was used only for destructuring).

</comment></comment></comment><comment><USER4> Why are they working on _library_ things instead of _language_ things? Half of the features they pushed out belong in user-land, not the core language.

<comment><USER5> Lisps are designed such that the need to work on language things is minimal. Clojure actually has more "language things" than your typical lisp. But anyway you're gonna see more things belong in standard library land, because having a good standard library is part of what makes a good language.

</comment></comment><comment><USER6> Can anyone help me understand the usage of these literals (2.1.1, 2.1.2)? Why and for what are they useful?

<comment><USER7> What I want to know is, what's the benefit of representing your domain- specific datatypes with a literal like #htmlelement [:div {:id "foo"}] rather than just (htmlelement [:div {:id "foo"}])? It seems like all you save is one character. Is there some other benefit I'm missing?

<comment><USER8> Security. You can parse a reader literal without evaluating the resulting data structure. If you're just reading and evaluating, you might find someone injecting some malicious code in there. By using reader literals, you can include custom types like dates or UUIDs or whatever, without needing to trust the entity sending you the data.

<comment><USER7> Oh, I think I get it now: The difference is that the data part of a reader literal has a fixed form, while the argument to a function could be arbitrary code. Is that right?

<comment><USER9> Yes and another thing is that when you want to use Clojure Data together with other languages. You cant calls (mydate "15.11.2000") to a python runtime, to understand it you would need a clojure runtime. When you send #mydate "15.11.2000" you only need a clojure reader. What to do with the date literals can be handled in a python function. The reader is a rather simple peace of code.

</comment></comment></comment></comment></comment><comment><USER10> Did post this earlier today.

<http://news.ycombinator.com/item?id=3857082></comment>


Crystal Structure of the Human Cannabinoid Receptor CB1 - <AUTHOR> http://www.cell.com/cell/fulltext/S0092-8674(16)31385-X <comment><USER2> Now I wanted to dazzle the HN readership with the beauty of molecular structures, but the actual data is apparently not yet available ([http://www.rcsb.org/pdb/results/results.do?tabtoshow=Unrelea...](http://www.rcsb.org/pdb/results/results.do?tabtoshow=Unreleased&qrid=BBB81AC6))

Oh well, look at this beauty instead: [http://www.rcsb.org/pdb/ngl/ngl.do?pdbid=3a12&bionumber=1](http://www.rcsb.org/pdb/ngl/ngl.do?pdbid=3a12&bionumber=1). CB1 would be without work if this juggernaut ever decides to just chill & smoke weed.

<comment><USER3> I love that we live in a time where I can open a web page on a $70 phone, it loads in seconds, and I can smoothly rotate a massive 3D molecule with my thumb.

Seriously that used to be the gold standard for "holy crap that computer is fast".

Kudos to the thousands of people over the years who worked to make that possible.

</comment></comment><comment><USER4> >This site requires Cookies to be enabled to function. Please ensure Cookies are turned on and then re-visit the desired page.

Cookie wall?

At first they said nothing, then they started asking, then warning that you should accept their use of cookies and now they outright block you. I really hope that this abusive practice remains a rare sight for the sake of the little openness remaining in the Web.

</comment><comment><USER5> The poster might want to comment on why he considered this article in particular worthy of posting. Crystal structure studies come out all the time, and at a glance this doesn't look like anything other than routine molecular biology with a droll association.

<comment><AUTHOR> G protein-coupled receptors (GPCR) are the targets for the majority of the best-selling drugs on the market. Approximately 40-50% of pharmaceuticals target them in some way.

There are more cannabinoid receptors than any other GPCR in our bodies. With 113 known cannabinoids, including chemicals that we produce ourselves, the potential for new and novel treatments are huge.

Further understanding the mechanism by which our own endogenous cannabinoids like anandamide, or how other external cannabinoids like THC work is important. It can help us better comprehend the role CB1 and CB2 receptors play in regulating all sorts of functions like pain, mood, appetite, and more.

Additionally, from the paper itself, we're highly interested as to why some synthetic cannabinoids have caused fatal, and extremely dangerous reactions, while chemicals like THC appear to have extremely high thresholds of safety (no recorded overdoses).

I believe cannabis research is incredibly important. This study provides direction, and further understanding for not only what we already know, but what we may enable in the future.</comment></comment>


For Billion-Dollar Companies, Venture Deals Outstrip Going Public - <AUTHOR> http://blogs.wsj.com/digits/2014/08/19/for-billion-dollar-companies-venture-deals-outstrip-going-public/?mod=WSJ_hpp_sections_tech

<comment><USER2> We have the heightened burden of SEC regulations to thank for all these companies staying private. If the regulatory burdens were lower more companies might go to public markets to raise capital, instead of to VCs.

The regulations are meant to protect the everyman, but the perverse result is that the opportunity to participate in these fantastic returns is now restricted to the executive tier at the startups, the rich accredited investors, the VCs, and the private equity firms.

The rest of us are left out ... unless we become founders ourselves.

<comment><USER3> Except in the days before the "heightened burden of SEC regulation" IPO shares were largely only available to wealthy clients of the banks that underwrote the IPOs. The rest of us were just as left out.

<comment><USER4> Not sure I understand. IPO shares are still only available to the wealthy clients of the banks that underwrite them.

<comment><USER3> That was my point - nothing has changed. The OP was suggesting things were better in the past.

<comment><USER5> Yeah, I think that comment conflates three things: 1) The regulatory burden of going public (Sarbanes-Oxley compliance et al.) pushes companies to stay private, which keeps the average Jane from investing in those companies via the public markets.

2) For those companies that are privately held (whether that's because of the regulations in #1 or for more fundamental reasons), _separate_ regulations (accredited investor rules et al.) prevent the average Joe from investing privately.

3) On top of #1 and #2, market forces and norms give more investment opportunities (both public and private) to large, established players than individual laymen. For example:

* Companies may prefer to take $1M each from 5 large investors than $1k each from 5000 people, both to reduce logistical burden and because those large investors statistically have other unique things to offer like expertise, advice, and connections. This is unfortunate for the small-time-but-sophisticated investor, but seems quite rational.

* IPO shares only being offered to large friends and clients of the underwriter. The _justification_ is reduced logistical overhead as in the previous case, but the _true_ motivation is widely suspected to be cronyism.

</comment></comment></comment></comment></comment><comment><USER6> The tech industry is just a vehicle for money laundering by the rich now. At least IPOs gave engineers and workers a chance to win a lottery ticket.

Developers, stop slaving away for the VC's. You are wasting your time and have no chance of getting paid until we reform the economic foundation of our trade.

<comment><USER7> Well you do get secondaries, but it's usually only something like 10% of vested stock, while your still working there. The lack of a public market also makes it far more risky to hold the stock for long term capital gains tax rates. I agree the lack of IPOs make equity earnings less valuable to the typical employee.

<comment><USER8> I wish more people knew this. Even if you beat the odds and the startup you work for goes places, you may only get 10%-15% of the proceeds now, and be hoping for the rest even further out in the future. Almost certainly better to just work for a public company from the start.

<comment><USER9> Are you saying you only get 10-15% of the proceeds of YOUR vested stock? So if I have 1% of a $100m company, my shares would only get me $100k, not $1m? Can you explain or link to a source of more information?

<comment><USER10> Petalpete - the comment above refers to pre-ipo sales of employee stock.

A bunch of private companies that hit billion dollar valuations let employees sell some of their shares. That person is claiming that those sales allow employees to sell 10-15% of their vested stock.

Of course, if they do that, they still keep the other 85-90%.

<comment><USER8> This is exactly what I meant, thanks for filling in the details.

</comment></comment></comment></comment></comment></comment><comment><USER11> Some possible explanations: 1) Inflation - a billion dollars isn't what it used to be. 2) Stock market bubble causing investors to look elsewhere. 3) Founders are younger - they aren't looking to retire yet so let it grow, let it grow, don't need an exit yet.

<comment><AUTHOR> Also some founders are able to trade equity for big cash in a funding round

[http://www.businessinsider.com/snapchats-founders- pocket-10-...](http://www.businessinsider.com/snapchats-founders- pocket-10-million-in-addition-to-the-60-million-round-2013-6)

</comment></comment><comment><USER12> Seems like this would be a natural consequence of the increasing concentration of wealth. The top 1% of the population control 43% of the wealth in the US. But for the 99%, much of that wealth is illiquid and not readily available for investment (particularly speculative investment). So I wouldn't be surprised if the 1% controlled as much as 90% of the liquid wealth available to invest in young companies, making it unnecessary to go to the public markets.

<comment><USER13> By the way, what's the minimum income, to be in top 1% of incomes in the US?

<comment><USER14> Around half a million dollars. Source: [http://www.brookings.edu/research/essays/2014/saving- horatio...](http://www.brookings.edu/research/essays/2014/saving-horatio- alger)

</comment></comment></comment><comment><USER15> The VC model is more conducive to the Greater fool theory: [http://en.wikipedia.org/wiki/Greater_fool_theory](http://en.wikipedia.org/wiki/Greater_fool_theory). While social psychology and public sentiment certainly plays into the valuation of public companies, being a public company typically restricts valuation to more standardized valuation models and scrutiny; it's much more transparent overall.

</comment><comment><USER16> It appears there's only room for 11 or 12 big IPOs per year. If the current trend holds and late-stage investors are creating more private unicorns than the public can bear, we have a problem. The fallout from this mismatch wouldn't look like a tanking stock market. Just a lot of formerly impressive companies quietly being acquired for a relative firesale. I suspect Box and Square are both in this situation.

</comment><comment><USER17> Raising money in public markets nowadays makes the company significantly less agile. Marc Andreessen has been advocating staying private for as long as possible in quite a few places, including here

[http://www.vox.com/2014/6/26/5837638/the-ipo-is-dying- marc-a...](http://www.vox.com/2014/6/26/5837638/the-ipo-is-dying-marc- andreessen-explains-why)

<comment><USER18> Being a company that's large enough to go public makes a company significantly less agile. But of course, I'm totally shocked that a venture investor would argue for companies doing things that benefit venture investors.

<comment><USER17> Well, before he was a venture investor he ran a company. And, generally speaking, VCs tend to be supportive of IPOs.

</comment></comment></comment><comment><USER19> But, if you don't IPO, how do you exit, leaving the muppets with the stock, whilst you run with the cash? Staying private means actually building something? Of value? That is not simply another shitty Rails site, or tries to skirt around regulations and pretends to be 'disrupting' things? Oh dear, this won't go down well at all with the chattering classes on HN!</comment>


How can C Programs be so Reliable? - <AUTHOR> http://tratt.net/laurie/tech_articles/articles/how_can_c_programs_be_so_reliable# <comment><USER2> "C is unreliable" is the wrong way to conceptualize the problem. The argument against C is that it is _inefficient_. It's not that C programs are broken; it's that if you spend six hours writing C code your code will either do less, or be more broken, than what you would have produced had you spent those six hours writing, e.g., Perl or Javascript or Lisp instead.

There's no reason why you can't write correct C code, or correct assembly code for that matter. The challenge is to do so without wasting a lot of time: Any amount of time that you spend consciously thinking about correct memory management or hand-optimizing your opcodes could probably be spent doing something more important, unless you are working on one of the few problems where that kind of optimization is actually the bottleneck.

Of course, the flip side of having to think about every layer is that you get to _see_ and potentially tweak every layer. It's nice to work on something transparent. It's nice to know what is going on down there among the sockets and the buffers. I've been thinking about practicing some C for just that reason, and it seems to be why the OP likes C. But I don't anticipate being very efficient when writing my own web server in C. My website will be better if I just install a big pile of other people's C and get on with designing or writing.

<comment><USER3> I think it is a mistake to think that you can't be efficient just because you're using C. It all depends on the abstractions you need to write your code. When writing code for the web, using Python is faster not just because of the language, but because there is so much ready to be used. Putting together 10 disparate libraries is much easier in Python/Ruby than it will ever be in C and C++. However, there is a lot of code that either doesn't require all these libraries or where the infrastructure is well established in C (think about kernel drivers for example). In that case, programming in C is not fundamentally harder than in Python, if you have enough experience.

<comment><USER4> Regardless of the abstractions you use, buttoning down a C runtime environment so you can be rely on the assumptions behind those abstractions is a chore. There's a pretty famous Mark Dowd Sendmail vulnerability that relies almost entirely on (only- incidentally- sent-) signal timing that I like to use as an example here, but rlimits are another.

The problem is, your code will appear to work fine even if you don't complete the chore. It isn't until your program blows up that it'll even occur to you that there was more work to be done.

<comment><USER5> > There's a pretty famous Mark Dowd Sendmail vulnerability that relies almost > entirely on (only- incidentally- sent-) signal timing that I like to use as > an example here

Would love to see a reference to this.

<comment><USER6> The unwillingness to cut and paste "mark dowd sendmail" into google for something you would "love to see" is amusingly typical of lots of the comments on this thread. It's the first result BTW

<comment><USER4> I like the second result better than the first, for what it's worth. ;)

But come on. While let-me-Google-that-for-you requests are annoying, they are _in toto_ less toxic to threads than comments like yours; at least the lame question generates a factual answer.

</comment></comment></comment></comment></comment></comment><comment><USER7> Great article. My personal take on this is that C programs are so damn reliable because there is nothing under the hood, the building blocks are so simple and transparent that you can follow the thread of execution with minimal mental overhead.

That means that when you lay out your program the most important parts (memory map and failure modes) are clearly visible.

 _IF_ you are a good programmer.

And that's the reason there is an obfuscated C contest, if a C programmer sets his or her mind on being deliberately hard to understand that same power can be used against any future reader of the code. Incompetence goes a long way towards explaining some of C's bad reputation. You can write bad code in any language, but none give you as much rope to hang yourself with as C (and of course, C++).

<comment><USER8>
        the building blocks are so simple and transparent
        that you can follow the thread of execution with
        minimal mental overhead.

I do not agree.

I've seen plenty of code that does weird things with pointers, like passing around a reference to a struct's member, then to retrieve the struct decrementing a value from the pointer + casting. Or XOR-ing pointers in doubly-linked lists for compression. And these are just simple examples.

I've seen code where I was like "WTF was this guy thinking?".

My biggest problem with C is that error handling is non-standard. In case of errors ome functions are returning 0. Some are returning -1. Some are returning > 0\. Some are returning a value in an out parameter. Some functions are putting an error in errno. Some functions are resetting errno on each call. Some functions do not reset errno.

Also, the Glibc documentation is so incomplete on so many important issues that it isn't even funny.

Yes, kernel hackers can surely write good code after years of experience with buggy code that they had to debug.

But for the rest of the code, written by mere mortals, I basically get a headache every time I have to take a peek at code somebody else wrote.

<comment><USER7> > I've seen plenty of code where I was like "WTF was this guy thinking?".

Yes, that happens. But I've seen that in COBOL, Perl, Pascal, Java, PHP and in Ruby as well.

> In case of errors (s)ome functions are returning 0. Some are returning -1.

That's not a feature of the language.

<comment><USER8>
         That's not a feature of the language.

Well, yes, but it's kind of nice when you've got exceptions with stack traces attached.

Some people don't like exceptions, but I do.

<comment><USER9> It is indeed "kind of nice". But the question at hand is whether it's a requirement for writing reliable software. I tend to agree with the posts here that argue that it's not. It saves _time_ for developers, it doesn't meaningfully improve the quality of the end product.

Serious C projects tend to come up with this stuff on their own, often with better adapted implementations than the "plain stack trace" you see in higher level environments. Check out the kernel's use of BUG/WARN for a _great_ example of how runtime stack introspection can work in C.

</comment></comment></comment></comment></comment><comment><USER10> My problem with this article is the use of the word 'flaw' to describe the potential pitfalls of programming in C. Use of that word seems to imply that these things are accidental, and maybe if it had been better designed the problems wouldn't exist.

The original idea of the language (or at least a major part of it) was to be a portable alternative for the many processor specific assembly languages in use - rather than having to write the same functionality for each one, you could write it once in C and then compile it for each platform. If that's your aim, then you will end up directly manipulating memory, and you open yourself up to that whole class of errors - memory leaks, array overruns, pointer arithmetic mistakes. All C gives you is portable access to how processor hardware works, with a few conveniences (y'know - function calls).

If you want to protect against these problems you have to add some extra layers of abstraction between the language and the underlying hardware, and that comes at a cost. That cost is mostly performance, but thanks to Moore's law these days that is a much lower priority hence the abundant use of higher level languages - Java, Python etc.

My point is that C is how it is _on purpose_. This direct access to the hardware comes with some downsides, but they aren't 'flaws', they come hand in hand with the power.

</comment><comment><USER11> I would contend that C does very little in the way of preventing errors and debugging them if they occur. The claim that "[..] only two C-specific errors have thus far caused any real problem in the Converge VM," is completely beside the point. Language specific errors have never been the problem. Java's infamous null-pointer exceptions are not java specific: the C equivalent would be a segfault. And please do note, that Java prints a stack trace by default to help correct the mistake. A huge step forward from C's generic segfault.

The real reason that most C programs in daily use are so robust, is because they are ages old. Many, many man-years have been invested in the production of e.g. BSD, unix tools, POSIX libraries, and even web browsers and word processors.

Why do we use Javascript and even PHP to program web-applications? Because we need fewer lines to get the same result. Moreover, given the correlation between number of lines and number of bugs, shorter programs are better. If we had been limited to C "web 2.0" would have been decades away.

<comment><USER12> Obviously I have no hard data on the reliability of new C programs, but things like git (which I at least find pretty reliable) may serve as counter point to this theory.

<comment><USER10> Don't forget that only the core of Git is C, the rest is big pile of shell scripts, and (I believe) perl.

<comment><USER13> This is not particularly the case. Things with git are initially implemented as shell scripts, as a way of "just getting it done", but are later migrated to C. These days a very large portion of Git is straight C.

</comment></comment></comment></comment><comment><USER14> _when one calls a function like stat in C, the documentation lists all the failure conditions_

Actually, no. When the documentation says

        This function shall fail if:

        [EFOO]   Could not allocate a bar.

it doesn't mean that this is the only possible failure; POSIX states that functions "may generate additional errors unless explicitly disallowed for a particular function".

Except in very rare circumstances, when you make system or library calls you should be prepared to receive an E_NEW_ERROR_NEVER_SEEN_NOR_DOCUMENTED_BEFORE and handle it sanely (which in most cases will involve printing an error message and exiting).

<comment><USER15> That is true, but good man pages still tend to document all or at least a lot of failure conditions. It's quite reasonable to do something like

        if ((fd = open(myfile, O_RDWR | O_NONBLOCK, 0644)) == -1) {
            switch(errno) {
            case ENOENT: case ENOTDIR: case EACCESS:
            case ELOOP: case ENAMETOOLONG: case EPERM:
                warn("Cannot open file");
                goto choose_file_to_open;
            case EISDIR:
                if (chdir(myfile) != 0)
                    warn("Failed to enter %s", myfile);
                goto choose_file_to_open;
            case ENXIO: case EWOULDBLOCK:
                enqueue_open_callback(myfile);
                return;
            default:
                err("Cannot open file");
        }

The above is well-documented in open(2); compare, for instance, <http://docs.python.org/library/os.html#os.open>.

<comment><USER16> `os.open` is a low-level interface whose semantics are platform-dependent. The Python analogue is the builtin `open`, which documents that it raises IOError on a failure to open a file: <http://docs.python.org/library/functions.html#open> .

<comment><USER15> True, but you have to dig a lot to find any details. The above code is easy to write from a good man page (OpenBSD's, in this case.)

(Also, open(2) has all these options for a reason; think symlink races. Python's open() is not sufficient.)

<comment><USER17> Note, open is not part of C. A better comparison would be fopen which, according to the C99 draft I have, just returns NULL on failure.

<comment><USER15> You're technically correct, but that's not the "C" being discussed in this article. Also, fopen(3) on OpenBSD does document values errno after failure (by reference to the malloc(3) and open(2) man pages), and I expect that any reasonable system does likewise.

</comment></comment></comment></comment></comment></comment><comment><USER18> I have a gut feeling that there is some merit to the idea that exception handling isn't all that great. Just so much code out there does not really handle the exceptions, it just exit(1)s. C will teach you to check return values (usually easy enough: if (result==NULL) {fatalerror(1,"result not OK");}) . If you don't, the program will continue to run (derailed). Most 'high-level' programmers will consider an abortion of execution just fine, while C programmers will put more thought into handling an error situation. Few C programs will automatically abort with a core dump on the first occasion of 'record not found'.

</comment><comment><USER19> > compilers were expensive (this being the days > before free UNIX clones were > readily available)

I'm not sure what era the author is referring to, here. In the late 80's, Turbo C broke the price barrier for a decent MS-DOS C compiler at the $79-$99 price range. Shortly after that, Mix began offering their MS-DOS Power C compiler for $20. Tom Swan's book "Type and Learn C++" provided a tiny-model version of Turbo C++ on a disk provided with the book.

The GNU ports djgpp and GCC were available for MS-DOS and Windows in later years.

> the culture was intimidatory;

I'm again wondering what time-period he's talking about. When I started learning C in the late 80's, most of the trade magazines were full of articles that used C as the primary language for whatever programs or techniques were being presented. Dr. Dobbs Journal was full of C code. Before Byte quit publishing source code, one could find a fair amount of C there. Of course, the specialty magazines like The C/C++ User's Journal and the C Gazette contained nothing but C and later C++ code.

> This is a huge difference in mind-set > from exception based languages,

Yes. C is a language that was designed two decades before Java.

At first, I was really taken aback by the author's take on C, but as I tried to digest why he has these perceptions of the language, I ventured to guess that a number of developers who came of age when languages with more modern niceties were available probably also have this view of C. From the perspective of someone who has been able to use more modern languages, C must seem like a rickety bridge that could be dangerous to cross.

A number of points that Mr. Tratt makes, though, pertain to the programmer; not the language. Certainly there are library routines that allow for buffer overflows, like gets(). It's been known for quite a while ( since the Internet worm was unleashed in 1988? ) that fgets() should be favored so that buffer boundaries can be observed. Certainly people writing their own functions may not write them correctly, but this is a matter of becoming conversant with C. It's a matter of attaining the right experience.

<comment><USER20> I would guess he's talking about the early 80s. I was in the same boat in that time period. I didn't have access to C until my senior year in college (1984-85) when we got a new Computer Science department head who was pro-Unix, and we got a Vax 11-750 with BSD Unix on it. I worked on a special project that gave me access to that machine, and there I learned Unix, vi, and C. I thought I had died and gone to heaven!

</comment></comment><comment><USER21> I've recently been curious why Ada isn't more popular in industry and academia beyond its niches in avionics and defense. Seems close in speed & memory usage to C/C++, has good GNU tools, and claims resiliance to the pitfalls of C.

<comment><USER22> ADA is a stereotypical non-hacker language. It was invented by a DOD committee & requires a good deal of 'extra' fluff to write a program.

Also, as I recall, to be a certified ADA-compliant compiler requires having a ton of libraries.

<comment><USER23> Ada ( _not_ ADA) was largely the product of one man, Jean Ichbiah.

The notion that it's some kind of commitee-created monstrosity seems largely to have come from ESR's wildly inaccurate writeup of it in the Hacker's Dictionary. It's actually a fairly small and nice language.

The object oriented extensions didn't follow the "standard" Java "." syntax, though, which probably hurt Ada 95's uptake more than it should have.

</comment></comment></comment><comment><USER24> In the end it all boils down on what you're building. If you've done your fair share of programming (C, C++, Java, PHP, Python, Ruby) then you just go with the tools that are best for the job.

Would I write a complete web service in C? Probably not. Would I write a fast image manipulation/modification library for that specific website if needed in C (or C++)? Probably -- because I like the performance gain when I'm converting 10.000 images.

I love the fact that you can just build components in different languages and then glue them together so you can build awesome products.

<comment><USER7> 10,000 images is not over the break even point yet (unless you do something really hard resulting in a runtime significantly larger than what it would cost to farm out that job to a bunch of EC2 instances, it's all about the question whether your time is more expensive than the cost to rent the hardware to do the job).

It starts to pay off when you write that package either as a service with a large number of users or if you make a general purpose library for inclusion in lots of other programs, especially if they are written in other languages.

</comment></comment><comment><USER25> There could be some filtering going on, both on the type of programs one tends to write in C and the type of people that write C. It could be that problems for which C is chosen tend to be intrinsically more well defined (command line applications, kernels, libraries, etc). It could also be that C intimidates less talented programs so some self-selection could be happening.

<comment><USER26> These days, with new software perhaps. But go back just 10 - 15 years and you'll find pretty much every type of software was being written in C. Even webapps: tons of CGI apps were written in C.

The reality is that doom and gloom about C is overrated. Sure, you can shoot yourself in the foot easily, but most competent programmers will do just fine. That's been my general experience and I don't think I've spent my career surrounded by rockstars :-)

<comment><USER14> I still write CGI programs in C.

<comment><USER27> Same here except I've "upgraded" to FastCGI.

<comment><USER14> Good point. I'd use FastCGI (or equivalent) if I had a significant load; but my CGI programs tend to execute only a few times per hour, so forking is cheaper than keeping a process running.

</comment></comment></comment></comment></comment><comment><USER28> A week ago I would react differently to this article. But I just had my belief system overhauled by reading [http://blog.vivekhaldar.com/post/10126017769/smeeds-law- for-...](http://blog.vivekhaldar.com/post/10126017769/smeeds-law-for- programming).

</comment><comment><USER29> I had posted this a few hours ago (<http://news.ycombinator.com/item?id=3024495>) - I suppose the success of this one has something to do with the submitter? Or the time submitted...

<comment><AUTHOR> As with comedy, timing is everything. I have had items sink without trace, only to see someone submit the same thing and have it get 100, 200 or 300 points and occupy the front page for a day or more.

It happens.

 _Added in edit: Just for reference, I didn't down-vote you. Not least, one can't down-vote replies to one's own submissions or comments._

</comment></comment><comment><USER30> I don't understand the exception argument. You can choose which Exceptions to catch in languages such as Java, just as you would choose which error to deal with in C, but exceptions are so MUCH more powerful because they allow you to check for the error in user code rather than at each function call. In C, errors don't trickle down and you need to deal with them in each level of abstraction, which can be totally useless and time consuming.

<comment><USER31> I'm a noob at languages with built-in structured exception handling. I'm comfortable with the C approach.

 _In C, errors don't trickle down and you need to deal with them in each level of abstraction, which can be totally useless and time consuming._

Time consuming, yes, absolutely. "Useless" I don't understand at all. It's structured exceptions that more often seem useless to me.

The more layers of code an exception bubbles up (or trickles down) through, the less the exception handler can know about where it happened, why it happened, or what the resulting state of the program is. Very often, the only "handling" that can be done is making a report of the exception.

It seems to me that the most useful exception handlers, the ones most likely to actually salvage the situation and allow the program to continue to work, are the ones that immediately follow an exception-throwing call, the ones that don't allow any trickle-down.

But those are degenerates, of course. They're functionally equivalent to C-style error return codes.

<comment><USER32> What you say about error handling is true enough, but it's just not the whole story.

One problem with C-style error handling is that having error handling at all levels makes it impossible to reuse code without tweaking it. Say you handle an error by printing a warning message to stderr, now you can't reuse that code in anything that doesn't want error messages printed there (maybe it doesn't want errors printed, or it needs to localize them, or it's using stderr for something else like in strace). So error handling in C kill reusability.

Another problem with C-style error handling is that the number of error types increases as you pass the buck up levels, but C doesn't have any good way to express more than one type at a time. Say one function returns true or false and another returns an error constant like errno. When one function calls the other, what do you return at the top level? You could map one to the other, but you've lost maybe important information about the error. So error handling in C kills composability.

And a third problem is documentation. With no standard error types that are known to the compiler it is rarely possible to tell the caller they forgot some error handling.

Exceptions address the first two problems, and checked exceptions the third one. A common misunderstanding of Java checked exceptions is that they force the caller to handle the error, when what they really do is force to caller to _document_ the errors it can generate.

</comment></comment></comment><comment><USER33> I don't find C programs to be very reliable at all.

Heavily used ones are as reliable as other heavily used programs, but barely any C programmers even use clang (static analysis) or even the elderly lint and its more modern cousins.

This on top of half of people calling themselves C programmers are really C++ programmers (they really are quite different how you use them in the correct manner), I don't really think he's correctly summarizing the field at all.

edit: I have been a C programmer for most of my career, including embedded linux, cli linux (including research robotics), and C-Servers to communicate to the above

I'm not some guy who just knows python and bitches about "the hard compiled languages" (although I do like python and ruby and objective-C).

</comment><comment><USER34> > if we're being brutally honest, only fairly competent programmers tend to > use C in the first place.

Oh, if only that were true. I've seen some not-so competent programmers churn out lots of C code (and then move on to C++ in order to do some real damage)

</comment><comment><USER35> C programs are reliable because either they're small, or, in the case of the few large reliable ones like the Linux kernel, they have undergone a tremendous number of eyeball-hours of review.

</comment><comment><USER36> I don't see how any language that depends on manual handing of error return codes can ever be considered "reliable". It's far, far too easy to leak memory and other resources. As other posters have noted, the only reason a lot of popular C programs are reliable is that they've been groomed with a fine- toothed comb.

The only low-level language that has any innate claim to reliability is C++ with proper use of the RAII idiom.

<comment><USER37> > The only low-level language that has any innate claim to reliability is C++ > with proper use of the RAII idiom.

Or rather a very carefully chosen subset of C++. See <http://yosefk.com/c++fqa/exceptions.html#fqa-17.3> for some of the problems.

By the way, what about ADA?

<comment><USER38> Ada is typically less well known and not used a lot, but if you want reliable code, it is pretty nice.

Be aware that it is a b&d language.

<comment><USER37> I usually use Haskell at work, which is even more b&d in a sense and it's pretty reliable. But you can not call Haskell low-level.

</comment></comment></comment></comment><comment><USER39> I wrote about some of this on my blog back in Feb. ( [http://www.jayfuerstenberg.com/blog/hot-potato-thoughts- on-j...](http://www.jayfuerstenberg.com/blog/hot-potato-thoughts-on-java- exceptions-and-error-handling) ) Java's exceptions cause Java applications to break often. It's not something that Java engineers want to hear but it's true.

</comment><comment><USER40> Oh, I love C. So simple and so powerful. God help me, I barely write anything in C anymore, but it will always hold a dear place in my heart until the day I die.

Thank you for allowing me this nostalgic indulgence, hackernewers. I know for at least a few of you, it will resonate.

</comment><comment><USER41> C is beautiful because <http://en.wikipedia.org/wiki/Duff%27s_device> is possible.

<comment><USER42> I agree with you.

But C is also ugly because Duff's device is possible. Think about it from an optimizing compiler standpoint. It takes a serious amount of effort to turn Duff's device -type control flow into an intermediate representation that can be somehow optimized. Now compare that to a language that is based on some form of extended lambda calculus.

</comment></comment><comment><USER43> C apps are reliable because C programmers embrace C's direct simplicity. Other languages aspire to be more complex by adding new features and syntax, where C remains stubbornly simple. Still dangerous, but still simple.

</comment><comment><USER44> Previously:

<http://news.ycombinator.org/item?id=3024495>

<comment><USER7> Fancy that, catching Colin at his own game :)

</comment></comment><comment><USER45> This kind of "FUD" surrounding C is definitely exaggerated. There's an awkward knee-jerk glow to the whole article, not least from the fact that the writer admits his inexperience in C. At times it even seems as if he lacks experience in programming, silently admitting his failure to comprehend the computer/software symbiosis altogether. After reading the article I played around with a funny exercise in my mind: I replaced the semantical mentions of C and programming with "tightrope walking", moving his arguments out of the computer programming sphere, and suddenly the general, ridiculous tone of the article stood out even more clear. Tightrope walking can be really, really tricky. Running with scissors can be done in a risky way, I suppose. Practicing pistol marksmanship incurs some risk, too.

</comment><comment><USER46> "pointers... arguably the trickiest concept in low-level languages, having no simple real-world analogy"

Arguably, indeed. The analogy is quite simple - a gigantic roulette wheel with 2^$membusbits slots, except the numbers are sequential. The ball is the pointer and pointer arithmetic involves moving the ball around the wheel.

<comment><USER47> I use Excel as an analogy when explaining pointers. If you imagine the machine's memory as a gigantic, single column excel table, then a pointer -- or address -- to the third slot in the machine's memory would be the value 'A3'.

When you're referencing that row in Excel, you don't copy around its value, but rather, you copy the address of the value. That way, if you change the value, any other cells that reference it will also fetch the new value.

<comment><USER48> An even simpler analogy is a street address. (Or if you want to involve numbers: Postal codes.) An address is a place where people live, but it's not the actual place.

<comment><USER49> That's funny, my CS prof used to use addresses as an example of multi- dimensional array indices: "state" being a major index, down through city, street, to the number on the street being a minor index.

<comment><USER49> Of course, in C, arrays are just funny looking pointers anyway, so I guess an address can be both a pointer and an ordered set of array indices.

</comment></comment></comment></comment></comment><comment><USER50> "Error. Unhandled exception."</comment>


Ask HN: Why Can't AeroFS find engineers? - <AUTHOR>

I've seen the job add for AeroFS on HN a lot of times.  Are they hiring a lot of people, or are they just not finding anybody? <comment><USER2> We're hiring a lot of people :)

From the posting:

      "Our team is small (there are six of us today) and we're looking to double
       it in the near future."

More importantly, we're pretty selective about the type of people we want to hire and the type of team we want to build, so finding the right people is very important to us.

<comment><USER3> I understand you are hiring "a lot of people", but your submissions to HN are too close to each other, imho.

Off topic, from <http://www.aerofs.com/signup>, "early beta?" Your first post to HN happened almost one and a half years ago... ;-)

<comment><USER2> you're probably right regarding the posts to HN being too close to each other. I'll scale them back.

A year and a half ago the product was in early alpha ;) In that time we've revisited and redone a lot of components that have unfortunately kept us in beta longer than we would've liked (we've gotten rid of the virtual drive, rewritten some of our syncing algorithms, and are working on another major re- factor based on user feedback now).</comment></comment></comment>