The death of optimizing compilers [pdf] - <AUTHOR> http://cr.yp.to/talks/2015%2E04%2E16/slides-djb-20150416-a4.pdf <comment><USER2> I wish this was given in a better format instead of just slides, hopefully the talk is actually better. Here's what I think the talk is about:

The pervasive view of software performance is that compilers are better than humans at optimizing code, but the few humans who optimize important bits of code to the maximum extent disagree Similarly, computer programs today are increasingly diverging into a state where there is a tiny amount of extremely performance critical code, and a large amount of code where performance is so good on our hardware today that even horribly unoptimized code has no noticeable effect on performance.

Thus, optimizing compilers are useless on the first type of code (humans are better), and useless on the second (performance doesn't matter). So what good are they at all?

If optimizing compilers aren't useful, what system should we use instead for making performant code? The author and collaborators' experience suggests that the reasons a compiler can't optimize code as well as a human when it matters is that our current programming languages don't give the compiler enough information about the intent of the code to optimize it. Therefore we should design programming languages that on the surface look very unoptimized, but specify enough information that compilers can do a really good job. It sounds like no one knows what such a programming language would look like.

<comment><USER3> Haskell? :)

<comment><USER4> Haskell is a good starting point. It's rather high level.

I'd like to see a variant that banishes bottom _|_ to the same corner that unsafePerformIO already occupies today. I want to see total Haskell. If all programs had to halt by default, the compiler would be much freer in choosing evaluation models---strict vs lazy would make no semantic difference.

<comment><USER5> See the Morte language proposed (not sure if actively developed) by Gabriel Gonzalez, author of many popular Haskell libraries.

<comment><USER6> Found this discussion pretty interesting on the subject : [http://www.reddit.com/r/haskell/comments/2g6wsx/haskell_for_...](http://www.reddit.com/r/haskell/comments/2g6wsx/haskell_for_all_morte_an_intermediate_language/)

</comment></comment></comment></comment></comment><comment><USER7> The talk was given to a packed room at ETAPS, a European computer science conference leaning more on the theoretical side (I suppose everyone was curious how optimizing compilers were dying). All in all, the audience did not bust out the pitchforks, although one might say that "domain-specific compilers" is basically the direction academia has already been heading. I doubt any of the people in the audience who were working on compilers/JITs are planning to stop working on them, though it did make for some fun dinner discussion.

There was one mid-talk exchange was one professor asking djb upfront whether or not he thought, in ten years, Mike Pall (author of LuaJIT) would be out of a job--after all, JITs are basically optimizing compilers. Well, the original question was more diplomatic than that, but eventually he pushed it enough that he got djb to not deny that this would be the case.

The talk was somewhat marred by a very large digression into an undergrad level primer of computer architecture (it probably would have been better served by an extended Q&A session), although the sorting example he finally built up to was pretty cute.

</comment><comment><USER8> I really like the conversation with the compiler approach. I had the good fortune to write some code for one of these: [http://en.m.wikipedia.org/wiki/Cray_XMT](http://en.m.wikipedia.org/wiki/Cray_XMT) which is a multi socket TB-scale shared memory machine with 128 hardware thread contexts per socket. It has an autoparallelizing C compiler that attempts to parallelize mostly for loops where it thinks it can (really quite a clever thing), but you can also tell it where to do things and give it hints through compiler pragmas. The compiler infrastructure will print out annotated copies of the code that tell you where it did and didn't parallelize and why. The effect is that you have this conversation with the compiler in which each of you tries to tell the other how to make the code more parallel (which in the case of the XMT means better and faster). It's very simple but the result can be orders of magnitude improvement.

<comment><USER9> A couple of years back I was working as a programmer, working on an application written in C. The application was always built with all optimizations disabled, because a bug in the compiler's optimizer (is that the correct term?) had bitten the programmers, so they had become really careful.

One day, as an experiment, I built the applications with pretty much all safe optimizations enabled and did a simple benchmark that showed a performance improvement of ... (drum-roll) two percent. This left me scratching my head, wondering if that meant the compiler really sucked at optimizing the code or if the code was so badly written the optimizer could not really do anything about it (or maybe it was so well-written it already was close to optimal without "optimizing" it? Probably not.).

How nice would it be, I thought, if the compiler could give me some kind of feedback on which parts of my source code it did or did optimize, and why.

I am relieved to hear (well, read, technically speaking) that I am not the only person to have had that idea.

Many programmers have all kinds of ideads in their heads about how writing their code in a certain way will make it easier for the compiler to optimize the heck out of it, but I guess few actually go and check if their assumption have any basis in reality.

It would be so incredibly nice to have a compiler that told you about such things, so you could _know_ if your assumptions were true or bogus.

 _Sigh_

<comment><USER10> SGI's compilers provided this feedback back in the 90's. I have been working with domain-specific languages for years but I would have thought what was relatively common back then persisted...

In my experience (computationally-intensive numerical simulations), the optimizers helped. Although we had to use a language that was algebra (Fortran derivatives), write C-tran with #pragmas to help out the optimizer or use hand-coded libraries for things like LAPACK.

<comment><USER11> I'm guessing that the XMT compiler mentioned upthread came from Tera, rather than SGI, but it's still interesting that significant parts of SGI (maybe including those same compilers?) ended up at the same company as the XMT compilers.

</comment></comment></comment></comment><comment><USER12> Some bits of Fran Allen in Coders At Work book: [http://www.codersatwork.com/fran- allen.html](http://www.codersatwork.com/fran-allen.html)

"We were making so much good progress on optimizations and transformations. We were getting rid of just one nice problem after another. When C came out, at one of the SIGPLAN compiler conferences, there was a debate between Steve Johnson from Bell Labs, who was supporting C, and one of our people, Bill Harrison, who was working on a project that I had at that time supporting automatic optimization. The nubbin of the debate was Steve's defense of not having to build optimizers anymore because the programmer would take care of it. That it was really a programmer's issue"

"We have seriously regressed, since C developed. C has destroyed our ability to advance the state of the art in automatic optimization, automatic parallelization, automatic mapping of a high-level language to the machine. This is one of the reasons compilers are . . . basically not taught much anymore in the colleges and universities. "

</comment><comment><USER13> I attended djb's ETAPS talk and am still not true if he was deliberately provocative or genuine. Assuming the latter, I disagree with several of his points. Here I want to bring one to the readers' attention and it's to do with the economics of correctness proofs.

One of his key arguments was that compiler optimisations are difficult to prove correct, and that's one of the reasons why optimising compilers will be replaced by a combination of simple compilers + assembly hand-written by domain experts. It is true that such proofs are (currently) expensive, but misses the point of the economics of correctness proofs: Correctness proofs are difficult, but proving compilers correct amortises that cost over all subsequent uses. In contrast, program specific correctness proofs are typically of comparable difficulty, but don't amortise in this way. Therefore it seems to be cheaper in the long run, to focus on the correctness of optimising compilers. Moreover, as compilers and optimisations are quite a restricted class of algorithms, hence it is more likely that we can reuse (parts of) correctness proofs and prover technology for compilers.

<comment><USER14> As I understand, the point is precisely that optimizations are not a restricted class of algorithms, because you need entirely new optimization techniques for new domains.

<comment><USER13> I'm not sure I understand what you mean. Those "entirely new optimization techniques" cannot be baked into the compiler, they will have to be invented and implemented by a human programmer on an ad-hoc basis. Compilers only use general purpose optimisations.

</comment></comment></comment><comment><USER15> To finish the Don Knuth quote: "There is no doubt that the grail of efficiency leads to abuse. Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. "

"Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified."

<comment><USER16> Sadly at least some of the points of relevance of this quote are outdated.

For example, it would behoove a large company to spend a great deal of time optimizing say, their JSON parsing library. Although it may not identify as a hotspot in any one place in their immense codebase, it's extreme prevalence causes performance degradation subtly but pervasively.

I also measured injected object creation using Guice to be 40x slower than a simple constructor in java (agree or disagree with the 40x, but using reflection to set a variable instead of simple object construction is intuitively far slower).

Guice may not show up on any profiler as problem - but if you slow down object creation by a factor of 40x, something you may do thousands of times per second for the life of your program, you are degrading performance across the board. Rather the same if you simply clock your CPU down a few hundred Mhz.

<comment><USER17> "Guice may not show up on any profiler as problem - but if you slow down object creation by a factor of 40x, something you may do thousands of times per second for the life of your program, you are degrading performance across the board."

If your profiler doesn't point right at the thing that you're doing 40x slower thousands of times, either your profiler is BROKEN or that isn't actually a bottleneck.

<comment><USER18> If you have ten thousand different kinds of objects then the creation of instances of any given class will not rise to the top of the profile.

</comment></comment></comment></comment><comment><USER19> I often boggle at people who claim that compilers are magic and outperform humans-- perhaps thats true for unimportant code that you'd pay no attention to, or with developers who aren't familiar with the underlying micro- architecture at all.

It's pretty usual for me to see a factor of 2 performance for the same algorithm implemented in the same manner when moving from SIMD intrensics (which almost directly map the underlying platform) to hand coded ASM.

Even non-SIMD code can result in some pretty stark changes.

A non-SIMD example from a crypto library I work on which isn't (yet) very well optimized for ARM, benchmarked on my novena (with GCC 4.9.1 -O3):

ecdsa_verify: min 1927us / avg 1928us / max 1929us

And a hand conversion of two hot inner functions (which are straight line multiple and add lattices, no control flow) into arm assembly:

ecdsa_verify: min 809us / avg 810us / max 811us

Again, same algorithm.

(The parallel change for x86_64 is still significant, but somewhat less extreme; in that case converting the same functions is only a 15% speedup overall, partially because the 64bit algorithm is different).

When thats a difference which results in 2x the amount of hardware (or 15% for that matter) for a big deployment; it can justify a LOT of optimization time.

(Or in my case, the performance of this code will have a material effect on the achievable scale and decentralization of the Bitcoin network.)

From a straight development time vs performance perspective I'd use even more hand written code; ... but there is a maintenance/auditing/eview/verification overhead too. And often the same code that you cannot tolerate being slow you also cannot tolerate being wrong.

</comment><comment><USER20> Copied from lobste.rs since I think it'd be interesting to the audience here as well

\---

This "dialogue with the compiler" bit that djb lands on is in some sense obviously the right way to go forward. I've found this to be the case not in optimization—though I'm not in the least bit surprised to see it there—but instead in the world of dependent types. The language that the program writer writes in is often just a skeleton of the genuine information of the program. For instance, in a dependently typed program it's often very difficult for an author to immediately write all of the complex intermediate types required to drive proofs forward, but it's much easier to achieve this in collaboration with the compiler (really the typechecker, and e.g. via a tactics language and interactive proof general-like interface). The ultimate result, the "elaborated" program, contains much, much more information than the skeletal program the programmer originally wrote. It has been annotated by the collaboration of the compiler and the program writer to extract more of the programmer's intention.

The same kind of thing could clearly arise from a "collaboration" over optimization. It's even quite possible that these are the same dialogue as dependent types certainly provide much more information to the compiler about the exact properties the code ought to substantiate—in a nice, machine readable format even.

<comment><USER20> It's worth adding in archive that this link is highly relevant: [http://cs.ru.nl/~freek/courses/tt-2010/tvftl/epigram- notes.p...](http://cs.ru.nl/~freek/courses/tt-2010/tvftl/epigram-notes.pdf)

</comment></comment><comment><USER21> An optimizing compiler may not be better than me at optimizing hot code paths, but my time is a very limited resource. The compiled version may only be 75% as fast as my hand optimized version, but writing that hand optimized version will likely take several times longer. Sometimes it is worth spending the extra time for that performance, but usually it is not.

<comment><USER22> Indeed, plus you need multiple hand-optimized versions. Not only per architecture, but also e.g. pre-AVX and AVX. An optimizing compiler will give you optimizations for all current and future platforms for free.

Another problem is that the number of people who can write good general hand- optimized assembly is small. E.g. I used a numeric Go library (which I will not name, because I should've submitted an issue) that used assembly-'optimized' routines. Replacing those with simple C loops and turning on auto-vectorization beat those hand-written routines handsomely.

</comment></comment><comment><USER23> Very related, but surprisingly not covered, I'll point out what was covered in depth at Mike Acton's CppCon14 keynote "Data-Oriented Design and C++"

[https://www.youtube.com/watch?v=rX0ItVEVjHc](https://www.youtube.com/watch?v=rX0ItVEVjHc)

[http://www.slideshare.net/cellperformance/data-oriented- desi...](http://www.slideshare.net/cellperformance/data-oriented-design-and-c)

And that is: Because of the ever-growing disparity between ALU vs IO speeds, the vast majority of time spent waiting on computers is because of issues that _the compiler can not optimize_. In general, compilers have very few opportunities to rearrange your data structures without your explicit, manual input. They can't help your CPU stall on memory/disc/network IO less by any significant amount. They can only help when your CPU actually has the data it needs to proceed --which often is less than 20% of total execution time.

In that case, no matter how smart GCC gets, it probably can't ever speed up your existing code by over 20% than it does today. It's not allowed to by the spec. I'm not aware of any general-purpose language where this is an option to any significant degree (silent AOS-to-SOA, hot-vs-cold data segregation, tree clustering, etc...)

If your program is too slow, it's almost certainly because you haven't done the hard, still-manual work of optimizing your data access patterns. Not just your Big-O's (N^2) vs (NlogN), but also your Big O's hidden, implicit K. The K that academia actively ignores and that most people rarely think about because it is mostly composed of cache misses that are implicit and invisible in your code. x = sqrt(y) is super cheap compared to x = *y. But, the same people who fret over explicit ALU costs usually think very little of x->y->z.

<comment><USER24> > I'm not aware of any general-purpose language where this is an option to any > significant degree (silent AOS-to-SOA, hot-vs-cold data segregation, tree > clustering, etc...)

Not yet, of course, because it hasn't been released, but Jonathan Blow's Jai language offers transparent switching between AOS and SOA, as well as many other features that make it easy to explore the 'optimization space' of a program manually.

[https://www.youtube.com/watch?v=ZHqFrNyLlpA](https://www.youtube.com/watch?v=ZHqFrNyLlpA)

(for those who don't know, Jonathan Blow was the creator of Braid)

</comment></comment><comment><USER25> If you agree with the author's conclusion that we need better languages that allow us to give the compiler more information about the optimization needs of our program, then I think you have to look in the direction of languages like Haskell, Idris, etc. Fortran can be faster than C because C has aliasing that limits the optimizations that the compiler can perform. Similarly, strongly typed and pure languages like Haskell give you even more. You can do a lot with types, but on top of that Haskell allows you to define rewrite rules that tell the compiler how things can be optimized. This allows the compiler to automatically convert things like this:

        map f (map g items)

...into this:

        map (f . g) items

<comment><USER26> Assuming f and g are not evaluated for their side-effects...

While I get that Haskell allows/demands you to write what you mean, consider an app that adjust reverb and volume up, from 1..10 (here items is the range 1..10). You could argue (rightfully) that the code should be structured differently (in this imaginary case, f and g would be identity functions on integers -- or perhaps Maybe integers (return int if volume/reverb set to the requested number, error/nil otherwise).

I realize this is a strange and contrived example -- the main point is that while functional programming is great -- iff what is wanted is a single multidomain language, it needs to function (heh!) in many different paradigms.

Btw, what would be the appropriate way to define functions for their side effects in Haskell - eg a function like this that takes an integer, performs some kind of i/o, interrupt, writes to a hw address -- to set some external state -- and returns the new value on success? Anyone have some pointers to that at the top of their head? Maybe there's a chapter in "Real World Haskell" that deals with it?

[ed: I just realized that the behaviour of (f . g) and map-map might be similar with the "Maybe identity"-behaviour I sketched above - modulus error handling/recovery (eg: do we want to try f on a given int i, even if g failed?). Oh well :-) ]

<comment><USER25> > Assuming f and g are not evaluated for their side-effects...

And that is the whole point. In Haskell we can know that f and g have no side effects. If they had side effects you would not be able to compose them with the dot operator.

I don't quite understand your example but I assure you that there are elegant ways to accomplish that in Haskell. There was recently a presentation [1] at the NY Haskell meetup that talked about a new Haskell sound synthesis library called vivid.

Making functions that perform side effects is also easy. Here's a simple example:

        intToFile :: Int -> IO ()
        intToFile anInt = do
            writeFile "myfile.txt" (show anInt)

The idea that you have to be multiparadigm to function in real world settings is a fallacy. Haskell is simply more expressive, which equips it to solve just about any problem with more clean and concise code than you're likely to get in other languages. Even if the code size stayed the same the extra safety you get from purity and strong types would be worth it alone.

[1] [https://www.youtube.com/watch?v=xo3zUvPsizo](https://www.youtube.com/watch?v=xo3zUvPsizo)

<comment><USER27> >The idea that you have to be multiparadigm to function in real world settings is a fallacy. Haskell is simply more expressive, which equips it to solve just about any problem with more clean and concise code than you're likely to get in other languages. Even if the code size stayed the same the extra safety you get from purity and strong types would be worth it alone.

Haskell is also not used any major project. It's an academia language.

<comment><USER26> I don't think it's quite right to say it's (still) _just_ an academia language -- but it's absolutely marginal. I use two (three) programs implemented in haskell on a (semi)regular basis: xmonad (and xmobar) and pandoc.

Other than that, I know of git-annex that is both actual programs _and_ is in actual use.

That's still more than the number of programs I use that are implemented in Arc...

Haskell is academic in the sense that it is _very_ opinionated about a few things (as this sub-thread illustrates).

Lets see how Shen, Open Dylan and clojure end up doing ... (and of those, probably only clojure have meaningful _contemporary_ systems implemented in it - so far). Maybe there's even more life left in Common Lisp (sbcl etc) or Scheme (Racket, guile w/guix nix (haskell! yay!) ...).

Then there's OCaml of course...

</comment></comment></comment></comment></comment><comment><USER28> More discussion from a month ago [https://news.ycombinator.com/item?id=9202858](https://news.ycombinator.com/item?id=9202858)

</comment><comment><USER29> There are a few basic optimizations we should routinely have in compilers today, but don't always find there.

\-- Multidimensional array optimizations. Basically, get to the level of subscript calculation overhead seen in FORTRAN compilers of 50 years ago.

\-- Subscript checking optimization. Work on Pascal compilers in the 1980s showed that about 95% of subscript checks could be eliminated or hoisted out of inner loops without loss of safety. This was forgotten during the C era, because C is vague about array sizes. Go optimizes out checks for the simple cases; Rust should and probably will in time. Optimization goal: 2D matrix multiply and matrix inversion should have all subscript checks hoisted out of inner loops or eliminated. Compilers that don't have this feature lead to users demanding a way to turn off subscript checking. That leads to buffer overflows. (Compilers must know that it's OK to detect a subscript check early; it's OK to abort before entering a loop if there will inevitably be an subscript error at iteration 10000.)

\-- Automatic inlining. If the call is expensive relative to the code being called, inline, then optimize. Ideally, this should work across module boundaries.

\-- Atomic operations and locking. The compiler needs to know how to do those efficiently. Calling a subroutine just to set a lock is bad. Making a system call is worse. Atomic operations often require special instructions, so the compiler needs to know about them.

\-- Common subexpression elimination for pure functions. Recognize pure functions (where x=y => f(x) = f(y) and there are no side effects) and routinely optimize. This is essential in code with lots of calls to trig functions.

<comment><USER30> _> Go optimizes out checks for the simple cases; Rust should and probably will in time._

rustc uses the industrial strength LLVM optimiser, which is perfectly capable of elimatinating bounds checks: almost certainly more capable than the main Go compiler in any case.

This has been pointed out to you several times, and so your repeated assertions otherwise are now almost malicious. Maybe you could be more concrete (e.g. with an instance of a subscript check eliminated by the Go compiler but not rustc)?

<comment><USER22> _" Rust essentially never wastes cycles on bounds checking thanks to the design of its iterators. The Servo team tells me that bounds checking has never shown up in any performance profiles."_

Source: [https://news.ycombinator.com/item?id=9392131](https://news.ycombinator.com/item?id=9392131)

<comment><USER30> That's... not exactly relevant. :)

Rust's iterators definitely resolve a lot of problems one might have with bounds checking, but there's still times when one is essentially required to do explicit indexing (e.g. iterating down a column of a multidimensional array) but in such a way that the bounds checks should be removable.

My point was that, modulo bugs, rustc (more specifically, LLVM) _will_ eliminate such bounds checks. With or without iterators.

<comment><USER29> I really need to write a matrix multiply and see what code Rust generates.

</comment></comment></comment></comment></comment><comment><USER31> "For some reason we all (especially me) had a mental block about optimization, namely that we always regarded it as a behind-the-scenes activity, to be done in the machine language, which the programmer isn't supposed to know. This veil was first lifted from my eyes when I ran across a remark by Hoare that, ideally, a language should be designed so that an optimizing compiler can describe its optimizations in the source language. Of course!"

That sounds like he wants some sort of homoiconic assembly or machine language to target. Does such a thing even exist?

<comment><USER32> Some optimisations can be described in any language (eg dead code elimination). But something that maps 1:1 to LLVM IR would work for pretty much everything.

<comment><USER33> "Lisp is... a good assembly language because it is possible to write Lisp in a style that directly reflects the operations available on modern computers."

Peter Norvig, Paradigms of Artificial Intelligence Programming.

</comment></comment></comment><comment><USER34> He actually quotes my rebuttal comment - ""Except, uh, a lot of people have applications whose profiles are mostly flat, because they've spent a lot of time optimizing them."

and his response is "this view is obsolete, and to the degree it isn't, flat profiles are dying".

Oh great, that's nice, i guess i can stop worrying about the thousands of C++ applications google has built that display this property, and ignore the fact that in fact, the profiles _have gotten more flat over time_ , not less flat.

Pack it up boys, time to go home!

Basically, he's just asserting i'm wrong, with little to no data presented, when i'm basing mine on the results of not only thousands of google programs (which i know with incredible accuracy), but _the thousands of others at other companies that have found the same_. I'm not aware of him poring over performance bugs for many many thousands of programs for the past 17 years. I can understand if he's done it for his open source programs (which are wonderful, BTW :P)

He then goes on to rebut other comments with simple bald assertions (like the luajit author's one) with again, no actual data.

So here's some more real data: GCC spent quite a while optimizing interpreter loops, and in fact, did a better job than "the experts" or whatever on every single one it was been handed.

So far, as far i can tell, the record is: If GCC didn't beat an expert at optimizing interpreter loops, it was because they didn't file a bug and give us code to optimize.

There have been entire projects about using compilers/jits to supplant hand- written interpreter loops.

Here's one: [https://code.google.com/p/unladen- swallow/wiki/ProjectPlan](https://code.google.com/p/unladen- swallow/wiki/ProjectPlan)

While the project was abandoned for other reasons, it produced 25+% speedups over the hand written interpreter versions of the same loop by doing nothing but using compilers.

Wake me up when this stops happening ....

He then goes on to make further assertions misundertanding compiler authors and what they do:"A compiler will not change an implementation of bubble sort to use mergesort. ... they only take responsibility for machine-specific optimization".

This is so false i don't know where to begin. Compilers would, if they could, happily change algorithms, and plenty do. They change the time bounds of algorithms. They do in fact, replace sorts. Past that, the problem there is not compilers, but the semantics of languages often do not allow them to safely do it.

But that is usually a programming language limitation, and not a "compilers don't do this" problem.

For example, the user may be able to change the numerical stability of an algorithm, but the programming language may not allow the compiler to do so.

Additionally, it's also generally not friendly to users.

As an example: ICC will happily replace your code with Intel performance primitives where it can. It knows how to do so. These are significant algorithm changes.

But because users by and large don't want the output of ICC to depend on Intel's Math Kernel Library or anything similar, they don't usually turn it on on by default.

GCC doesn't perform quite as much here, because even things like replacing "printf" with "puts" has caused tremendous amounts of annoyed users. Imagine the complaints if it started replacing algorithms.

Past that, i'd simply suggest he hasn't looked far enough into the history of optimizing compilers, because there has been _tons_ of work done on this. There are plenty of high level language optimizers that have been built that will completely rewrite or replace your code with rewritten algorithms, etc.

I stopped reading at page 50.

<comment><USER35> Maybe you should have continued to the end, because as far as I can tell, you are almost completely missing the point.

You are saying "look at all the wonderful things optimizing compilers can do". And he is saying "that's cool, but it doesn't really matter".

Now I am pretty sure he wold concede immediately that there are _some_ examples where it matters, but my experience matches what he is saying very closely.

A little bit on my background: hired in 2003 by the BBC for the express purpose of making one of their systems faster, succeeded at around 100x - 1000x with simpler code[1], hired by Apple to work on their Mac OS X performance team, similar successes with Spotlight, Mail indexing, PDF, etc. Also worked with/on Squeak. Squeak runs a bytecode interpreter, with a bunch of primitives. The interpreter is perfectly adequate for the vast majority of the system. Heck, the central page imposition routine in my BookLightning PDF page imposition program[2] is written in Objective-Smalltalk[3], or more precisely an interpreter for the language that's probably the slowest computer language implementation currently in existence. Yes it's slower than Ruby, by a wide margin. And yet BookLightning runs rings around similar programs that are based on Apple's highly optimized Quartz PDF engine. And by "rings" I mean order(s) of magnitude.

Why is BookLightning so fast? Simple: it knows that it doesn't have to unpack the individual PDF pages to impose them, and is built on top of a PDF engine that allows it to do that[4]. The benefit of an optimizing compiler in this scenario? Zilch.

At the BBC, the key insight was to move from 20+ machine distributed and SQL database backed system to a single jar event logger working in-memory with a filesystem based log[5]. How would a compiler make this transformation? And after the transformation, we probably could have run interpreted byte-code and still be fast enough, though the JIT probably helped us a little bit in not having to worry about performance of the few algorithmic parts.

As to changing a bubblesort to a mergesort, you hand-wave around that, but I think the point is that this is not a safe transformation, because the author may have specifically chosen bubblesort because he likes its characteristics for the data he knows will be encountered (or cares more about that case).

When I worked at Apple, I saw the same pattern: low-level optimizations of the type you discuss generally gained a few percent here and there, and we were happy to get them in the context of machine/system wide optimizations. However, the application-specific optimizations that were possible when you consider the whole stack and opportunities for crossing or collapsing layer boundaries were a few factors x or orders of magnitude.

And here is where the "optimizing compiler" mindset actually starts to get downright harmful for performance: in order for the compiler to be allowed to do a better job, you typically need to nail down the semantics much tighter, giving less leeway for application programmers. So you make the automatic %-optimizations that don't really matter easier by reducing or eliminating opportunities for the order-of-magnitude improvements.

And yes, I totally agree that optimizations should be user-visible and controllable, rather than automagically applied by the compiler. Again, the opportunities are much bigger for the code that matters, and the stuff that is applied automatically doesn't matter for most of the code it is applied to.

[1] [http://link.springer.com/chapter/10.1007%2F978-1-4614-9299-3...](http://link.springer.com/chapter/10.1007%2F978-1-4614-9299-3_11)

[2] [http://www.metaobject.com/Products/](http://www.metaobject.com/Products/)

[3] [http://objective.st](http://objective.st)

[4] [http://www.metaobject.com/Technology/](http://www.metaobject.com/Technology/)

[5] [http://martinfowler.com/bliki/EventPoster.html](http://martinfowler.com/bliki/EventPoster.html)

<comment><USER36> Is it your position that all of OS X could run on Squeak, or gcc -O0, with performance comparable to what it does now?

Because that seems to be the logical conclusion of your position.

Yes, it may be that low-level optimizations are fighting for single-digit percentage improvements. And of course re-architecting big systems at a high level can yield much bigger gains. But you have to remember that the baseline is an entire system that's already using an optimizing compiler.

If you want to argue that optimizing compilers are dead, you'd have to show that you can _remove_ optimizing compilers from your toolchain, and have nobody notice the difference. That's a pretty hard argument to make.

<comment><USER37> This is where my mind keeps coming back too. I keep hearing "it doesn't matter" and yet nearly every compiler I come into contact with is an optimizing compiler. Heck, even with some pretty straight up "performant" code that difference between optimizations on and off in a lot of these can be significant.

C#,C++,C,Go,V8

Just about everything.. I can only imagine what would happen if every process on all the systems I deal with were re-compiled with optimizations off :|

</comment></comment></comment></comment><comment><USER38> His vision:

 _The time is clearly ripe for program-manipulation systems... The programmer using such a system will write his beautifully-structured, but possibly inefficient, program P; then he will interactively specify transformations that make it efficient._

But what if the answers the programmer gives the compiler turn out not to match reality, and some weird bug is introduced that has no representation in the source? The compiler's decisions need to be somehow spelled out in debuggable form.

There is another approach (that can also be complementary). The programmer specifies in advance various specific scenarios, and a JIT compiler guesses which of the scenarios is in effect and optimizes for that (e.g. a certain condition, like input size is always true), but adds a guard (that hopefully adds negligible overhead). If the scenario does not match reality, the JIT deoptimizes and tries another. This process itself, of course, adds overhead, but it's warmup overhead, but it's more robust. This is the approach being tried by Graal, HotSpot's experimental next-gen JIT (and Truffle, Graals complementary a programming language construction toolkit aimed at optimization): [https://wiki.openjdk.java.net/display/Graal/Publications+and...](https://wiki.openjdk.java.net/display/Graal/Publications+and+Presentations)

</comment><comment><USER39> Even if DJB wrote some very effective code, now when he "goes meta" he comes somehow in the strange area of being "not even wrong." Or maybe we miss his ideas when we read the slides instead of hearing him at the talk.

People who make compilers used in the production know: if the naive users claim that "optimizing compilers don't matter" it's because the optimizing compilers are so good in doing what they're doing.

There's the argument which is here buried deep in the discussions and which I think DJB missed to address, nicely stated by haberman:

"If you want to argue that optimizing compilers are dead, you'd have to show that you can remove optimizing compilers from your toolchain, and have nobody notice the difference."

</comment><comment><USER40> So, Mel Kaye got the last word in?

[http://en.wikipedia.org/wiki/The_Story_of_Mel](http://en.wikipedia.org/wiki/The_Story_of_Mel)

</comment><comment><USER41> Interacting with a compiler sounds horrible. Think of the questions it might need to ask: "Hey! I could use AVX2 instructions here, after I inline a bunch of stuff and eliminate some dead code, but it requires doing a bunch of memory copying. Is that a good idea?". How would you answer a question like that?

And then, since optimizations are target dependent, you would need to go through this exercise for each target. Sounds fun.

<comment><USER42> How you'd answer depends very much on what you're doing -- which is the point of asking. You might be able to remove the memory copying by fiddling with alignment somewhere in calling code and adding an annotation saying you did that. Bam, problem solved. Or you might not be able to do that, in which case it would still be nice to know what tradeoffs you're making.

<comment><USER41> But consider, for it to be worth asking, it's got to be something that's very difficult to infer, or the compiler wouldn't need to ask. That means it will probably be difficult for you to infer as well.

In practice, you might have to profile all possible options, which would quickly turn into a major project if you're asked more than a handful of questions.

And, of course, even if you can figure it out, that alignment annotation you just made is only optimal on the one target. Even on x86 it will depend on what instructions are available.

</comment></comment></comment><comment><USER43> I'd be happy if C and C++ had 2,3,and 4 element vectors as built in types, along with cross and dot product operations. There are intrinsics, and GCC has it's own intrinsics that can be used across architectures. But the languages need to have these. They are so fundamental to so many things.

There are many more things to wish for, but I'm starting with one of the simplest.

<comment><USER44> Why would you need this in the language? It seems to me that these things can be provided by a library without any loss.

<comment><USER43> Why would you need this in the language? It seems to me that these things can be provided by a library without any loss.

No loss in function. C++ will certainly let you create classes, but you need to be extremely good to make them low overhead. You also can't pass them by value (the intrinsics can). And lastly, classes don't exist in C.

Vector math is so common it should have direct support and operators.

</comment></comment></comment><comment><USER45> Here's a compiler which uses program synthesis to target a mesh network type architecture: [http://pl.eecs.berkeley.edu/projects/chlorophyll/](http://pl.eecs.berkeley.edu/projects/chlorophyll/) . It uses a guided process like is implied in the last slides.

</comment><comment><USER46> Compiler optimization is obviously a sleeping field currently. There's nothing that's made it into practical compilers to address the bottlenecks shifting from ALU work to data storage and layout considerations.

Consider all the gnashing of teeth and wringing of hands that goes on in C++ circles about inefficient data layout & representation by inferior programmers, and the stories of victorious manual data layout refactorings by performance heroes.

DJB's slides don't address the data side because he only does crypto, and that's one of the fields where the ALU twiddling is still relevant. But crypto is also rarely a bottleneck.

</comment><comment><USER47> The original abstract for the talk (which summarizes the slides) were posted by the author here:

[http://blog.cr.yp.to/20150314-optimizing.html](http://blog.cr.yp.to/20150314-optimizing.html)

</comment><comment><USER48> This is very good. I've been working towards something similar to what DJB is talking about. (Nice to know I'm in good company. :)

In a nutshell, although automated systems will be good (are already and getting better) there will always be aspects that require humans in the loop (unless and until the machines actually become sentient, defined in this context as gaining that _je ne sais quoi_ that humans do seem to have.)

</comment><comment><USER49> The cases where optimizing compilers aren't good enough are where Java Hotspot Compiler and similar techniques really shines. Combined with novel superoptimization techniques, hotspot optimization could far outperform hand- optimization (although AFAIK that hasn't happened in practice yet).

<comment><USER50> I completely agree, although hotspot is really hampered by the fact that objects are allocated willy nilly on the heap instead of together for pipeline efficiency. So while hotspot (and graal) generate lovely assembly the lack of data locality kills a lot of possible performance. Hoping objectlayout.org changes that!

Yet I think that hotspot and intrinsics are a nice case study of showing why optimising compilers are not dead. Even for performance critical code. [https://news.ycombinator.com/item?id=9368137](https://news.ycombinator.com/item?id=9368137) discusses in part how intrinsics (hand optimised) at some point get beaten by the optimiser (simd/superword) and then hand-optimised again to again beat the optimiser. Mostly, because machines change.

A whole problem with static binaries as produced by C and GO compilers is that they assume machines are static. Which leads to lowest common denominator optimiser settings :( When the optimisers are humans this gets even worse. Then you end up with optimisations in your C code that was a good idea 15 years ago, but that make no use of SIMD today (or to short SIMD e.g. SSE2 when AVX-512 is available).

Of course real optimisations happen, not by doing the same thing faster but by doing a faster thing. Take for example the HMMER ([https://news.ycombinator.com/item?id=9368137](https://news.ycombinator.com/item?id=9368137)) family of algorithms. HMMER2 is part of SPECCPU, and the compiler guys doubled the speed of this algorithm, in about 5 years. Then the bioinformaticians redid it as HMMER3 which is quite different in how it globally works and gets 100x speed up in practice.

<comment><USER49> You might find this interesting (it's what I'm talking about when I said superoptimization).

[http://superoptimization.org/wiki/Superoptimizing_Compilers](http://superoptimization.org/wiki/Superoptimizing_Compilers)

</comment></comment></comment><comment><USER51> So just for a start: Which optimizing compiler actually properly solves the optimization problems? I know of none.

When I look at the list of optimization solvers for constrained linear or non- linear programming models (i.e. [https://en.wikipedia.org/wiki/List_of_optimization_software](https://en.wikipedia.org/wiki/List_of_optimization_software)) and the list of compilers the intersection is still zero.

All optimizers are still using their own ad-hoc versions of oversimplified solvers, which never fully explore their problem space. Current optimizing compilers are still just toys without a real-world solver. And it's clear why so. Current optimizable languages are still just toys without a real world solvable optimization goal.

You can think of strictly typed, sound declarative languages where solvers would make sense, or you can think of better languages, like fortress or functional languages, which are not encumbered by not-properly optimizable side-effects and aliasing which harm most modern language designs.

<comment><USER52> > or you can think of better languages, like fortress or functional languages, > which are not encumbered by not-properly optimizable side-effects and > aliasing which harm most modern language designs.

Remember, if you ban aliasing then you can't express any problems requiring aliasing! That kind of sucks for writing programs that are fast in the first place.

I've never had to use a language with really no aliasing (Fortran?), but many languages forbid possibly overlapping arrays, because of the pain interior pointers make for GC.

If you were ever to write an image decoder in Java, you'd see some steps involve reading and writing from different parts of the same image (usually called intra prediction), but now you have to pass all these extra coordinates along with the image array. And the optimizer is not good enough to make up for all the extra work it was just given.

</comment></comment><comment><USER53> Isn't that why people advocates C ? Isn't C just that type of language you can tell the compiler how to optimize ?

C might not give very explicit information on how to optimize, but isn't it simple and bare enough to let the compiler do a better job ?

</comment><comment><USER54> audio of the talk: [http://cr.yp.to/talks/2015.04.16/audio.ogg](http://cr.yp.to/talks/2015.04.16/audio.ogg)

</comment><comment><USER55> What a waste of time.

Yes, specialists can squeeze the last performance improvements in ASM compared to C. Doesn't mean that -O2/-O3, auto-vectorizing can't do a nice job and get to 90% of that

Optimizing DO matter. Just compare -O0 and -O1. Really. It's not because CPUs are fast that people shouldn't do that and compilers shouldn't optimize a bare minimum.

It's _even better_ that the compiler do that because ASM optimizing by hand is very error prone.

And compilers get better every day. Just look at LLVM.

</comment><comment><USER56> Room for optimizing compilers = distance between programming languages and CPU instructions/microarchitecture

</comment><comment><USER57> He is soooo spot on! This "dialogue with the compiler" is going to be really big in the next decades, but it's in no way a death of automatic optimization, it's just the beginning of it.

Here's a simple example how I expect it to work: You write a code that uses a list-like data structure. The compiler then instruments the code and you run some tests. The tests will then be evaluated and the (post?)compiler selects what kind of data structure is to be used (let's make example simple, choices are array vs. list). For instance, if you need to look up elements a lot (based on the evidence from testing), an array will be chosen as the underlying data structure.

And you actually get (if you want to see it, normally this information will be hidden) a little box in the IDE, where the variable is used, that tells you: "Here, an array will be used." You can then with one click say: "I don't want array, make it a list." So for all the possible optimizations, there are two viewpoints presented: The viewpoint of the compiler (based-on evidence from the tests or static analysis) and the viewpoint of the programmer (which allows for confirmation or override in case there are some unknown assumptions).

And if the specifications change (say, we chose list earlier but now we have actually a lot of direct access to elements), you can just recompile the same code with the previously-agreed compiler choices removed! And without changing any line of code, a different and more fitting data structure will be used.

You can easily see this can apply to many things, not just data structures. You can also see that different ways how the dialogue can be implemented are possible, once we syntactically separate the "what" from the "how" in the programming language. In the future, I believe, we will program just with abstract data types, and the concrete type specification will be selected based on the evidence from the running program (or static analysis augmented with that information). So the dialogue will not happen just with the programmer, but the compiler will also observe the real world behavior of the program and facilitate adaptation to that.

In this way, it's even possible to input assumptions that doesn't have to be provably correct. This approach can potentially bridge the static vs dynamic types divide, and others.

Finally, Haskell and functional languages are very nice, but I don't think they are final word in programming. If we wanted the above, they have syntactical problems such as mixing of concrete and abstract types (type classes). Also, there are limits to static analysis in the real world. The future will be lot more interesting.

</comment><comment><USER58> The birth of an optimizing assembler.

<comment><USER52> If you're curious, here is an optimizing assembler.

[https://code.google.com/p/mao/](https://code.google.com/p/mao/)

</comment></comment><comment><USER59> I would prefer death to the font-size: 2^1000px

</comment><comment><USER60> Trying to read this gave me cancer</comment>


Google is doing what? - <AUTHOR> http://www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9104858

<comment><USER2> Google has no doubt been ruling the search engine world. But monopoly is never beneficial for the customers. There should be atleast one strong competitor for google.</comment>


What is the best marketing automation tool for a B2C e-commerce business? - <AUTHOR> <comment><USER2> Which CRM software are you using?

<comment><AUTHOR> Currently we dont have a CRM. I am about to go with Hubspot. What do you think about Hubspot CRM for a B2C Business.

<comment><USER2> The most important consideration with marketing automation is data integrity/accuracy. If you have a CRM, then the biggest risk to data corruption is the integration between marketing automation <> CRM. The main cause of data corruption are the rules governing how data is mapped back/forth between the systems (dupe records, incorrectly overwriting/losing data, etc).

For example, "John Doe -- email@example.com" exists in your CRM, but the customer submits a form on the site using "John L. Doe -- email@example.com". Even though it's the same person, it could create a duplicate record if the rules are based on the email address or name.

Alternatively, the rules could be based on a cookie containing a unique ID. But this can also be problematic when multiple people use the same computer, the customer uses different devices, or the device has high security settings that mess with cookies.

Because of this, the best marketing automation system should integrate seamlessly with CRM, which means you purchase CRM + marketing automation __from the same vendor __. Do not trust third party integrations!! I 've seen firsthand the nightmare of fubar data this creates.

HubSpot marketing automation is excellent for small/midsize business (I've used them before, and it's a solid product). The main downsides to HubSpot:

\- Costs will grow as your database grows. This can get out of hand quickly if you have decent volume.

\- Their unique website visitor <> contact mapping rules is based on a cookie ID, which can be problematic in B2B settings (but not as important in B2C, where devices are shared less often)

I haven't personally used HubSpot CRM, but I would go with them for data integrity reasons. Their CRM is tightly integrated with their marketing automation, which will limit data risks. Plus the CRM is free.

Hope that helps.

<comment><AUTHOR> Thanks for sharing those insights and information. Do you have any recommendations for a good CRM + Marketing Automation tool other than hubspot.

<comment><USER2> If you can afford it, Salesforce is the best. Beyond that, I'm not familiar with any worth using.</comment></comment></comment></comment></comment>


Facebook to remove messaging from core app - <AUTHOR> http://www.nbcnews.com/tech/social-media/want-message-facebook-soon-youll-need-separate-app-n76141 <comment><USER2> Most of the comments here say they only use the FB app for messaging, and they won't install another app. Why is this Messenger app the 'another' one, when the main app clearly doesn't fit your usual use case? I've kept the main app off my phone for well over a year now, and only looked back when Paper came out. Try just having the Messenger app on your phone, no news feed, no ads, no crap. Just messaging, which works incredibly well on a phone that you're already using for lots of messaging.

<comment><USER3> I don't know about the other folks ... but I don't want to get sucked into the Feed. I'm with you on this.

<comment><USER4> My Facebook feed is wholly useless; the few things that might have been of interest there I've already seen on Twitter.

The Facebook Messenger app I've been using pretty consistently recently due to meeting some people and adding them on Facebook, and it's quite good, especially cross-platform (iOS/web).

</comment></comment></comment><comment><USER5> Finally! It's never made much sense to me that Messenger is both part of the main app, and a separate app - it's pointless duplication. Glad to see they're making up their minds.

<comment><USER6> Pointless duplication for sure. So why not get rid of the Messenger app? Why do they need two apps? (Three, if you count Paper) (Four, if you also count the Pages app). Seriously doubt I'll re-install the messenger app even if the main app drops support for messages.

Since iOS doesn't have a back button that works cross app, this sounds bad for usability. Already it pains me to be bounced in and out between the main app and the Pages app, with no working back navigation. This'll be even worse.

<comment><USER7> Because they want users to be able to use Paper instead of the main app, without bundling Messenger into both apps.

</comment></comment></comment><comment><USER8> I don't understand; isn't this how it is now? When I do into messages from the main Facebook it flips to the Messenger app. It's been this way on my phone for several months now.

<comment><USER9> It does this when the Messenger app is installed.

</comment></comment><comment><USER10> As long as they keep Pusheen emoticons I'm fine with it

<comment><USER11> The separate Messaging app currently has stickers, I don't see why they'd remove them.

And yes, the Pusheen emoticons are the best.

</comment></comment><comment><USER12> I only answer my Facebook messages BECAUSE I can answer them on my mobile app. I won't install another app just for Facebook messaging, so I guess those messages will never get answered.

<comment><USER13> I wonder what the platform breakdown is for people who say they wouldn't use the separate messenger app.

The inclusion of chatheads systemwide on Android makes it a great experience for me (of course YMMV). I wish FB Messenger and its chatheads could handle my SMS as well.

<comment><USER14> Yep agreed, Facebook Messenger is one of the best Android apps I use because of chatheads. It was never well integrated but I'm sure Facebook used to work as an SMS client on Android too, not sure why they removed it over integrating with chatheads.

</comment></comment></comment><comment><USER3> About two years ago, I changed the password on my FB account and stashed it somewhere. I deleted most of my photos and things that appear on my timeline. There were just enough folks I know, that I keep it around ... but I don't use it to communicate on a day-by-day basis.

I certainly don't want it on my phone.

But you know, if there's a separate Messenger app, I'd probably install it. Something that allows people on FB to contact me if they need to ask something, without sucking me into the Feed.

<comment><USER15> If you uninstall the Facebook app, their Messenger app works just fine standalone.

Android: [https://play.google.com/store/apps/details?id=com.facebook.o...](https://play.google.com/store/apps/details?id=com.facebook.orca)

iOS: [https://itunes.apple.com/us/app/facebook- messenger/id4546384...](https://itunes.apple.com/us/app/facebook- messenger/id454638411)

Windows Phone: [http://www.windowsphone.com/en- ca/store/app/messenger/3219d3...](http://www.windowsphone.com/en- ca/store/app/messenger/3219d30d-4a23-4f58-a91c-c44b04e6a0c7)

</comment></comment><comment><USER16> Likely this is part of a strategy to get users to install all kinds of FB apps on their phone and slowly take over the phone experience. Next up, Facebook mobile games?

<comment><USER17> Huh? More likely it's part of the strategy to merge Messaging app and Whatsapp.

<comment><USER16> And? One approach doesn't invalidate the other.

</comment></comment></comment><comment><USER18> With this, WhatsApp, and Oculus, it seems like Facebook's initiative in 2014 is to split up into more loosely-connected services rather than a single product.

</comment><comment><USER19> Oh, this is my writeup. Please let me know if you see anything inaccurate or strange.

</comment><comment><USER20> My question is why? Is this because they have WhatsApp under their arms?

</comment><comment><USER21> That is desperate.

</comment><comment><USER22> Oh, what an epic failure. Messaging is plainly said the only reason I'm still using Facebook.

<comment><AUTHOR> Why is that an epic failure? Start using the Facebook Messenger app and you'll have a smaller download when new versions ship. I think it's great.

<comment><USER22> They discontinued Messenger for desktop and now users must do an additional install for accessing Messenger (hint: users are lazy and more likely to ignore the missing functionality at all)... and so they're killing off the only thing that still keeps users accessing Facebook. Yay.

<comment><USER23> Fb was way more engaging when I had to Guess whether someone was available via mobile or web. Since they've done me the favor of confirming that most people are actively browsing on phones I'm already less inclined to open a chat window.

Glad they're making it safer and easier for me to disengage!

<comment><USER24> Remember when everyone moved from AOL to AIM? Now neither are used at all.</comment></comment></comment></comment></comment>


GitHub server troubles - <AUTHOR> https://status.github.com/messages <comment><USER2> > 00:05 PST We are failing over a data service, you may experience exceptions > and delay while receiving notifications. Apologies for the inconvenience.

Not sure what makes this "may". I'm getting 100% failure on my "git push" and their availability dropped to 0%.

edit: It's back again. Lasted about 10 minutes.

<comment><USER3> They seem to have recognized this..5 or so minutes later.

</comment></comment><comment><USER4> This is why you shouldn't move off Redis :D [https://news.ycombinator.com/item?id=13367077](https://news.ycombinator.com/item?id=13367077)

</comment><comment><USER5> Pink Unicorn every time I try to hit the site.

</comment><comment><USER6> I think it is back on somewhat, I could see my main page just now

EDIT: Yep, "8:17 UTC Everything operating normally."

</comment><comment><USER7> Back online.

</comment><comment><USER8> > 17:11 KST Major service outage.

</comment><comment><AUTHOR> Seems to be back, pretty fast. Nice, github!

</comment><comment><USER9> All of github is down for me, not just messages.

</comment><comment><USER9> They just updated it to "Major Service Outage."</comment>


Move Fast without Breaking Things - <AUTHOR> http://blog.pragmaticengineer.com/the-startup-dilemma-move-slow-or-break-things/ <comment> white-flame Racing a car has a great analogy: In order to get faster lap times, you sometimes have to drive slower.

Specifically, you need to slow down for corners in order to maintain your control over the vehicle. Try to drive too fast and you end up wrecked. Even if you don't wreck, correcting from having entered a corner too fast loses far more time than just slowing down properly. Similarly to technical debt, if you start a straightaway from a slower starting speed because you botched the prior corner, that damages your lap time throughout the entire stretch.

In the overall scheme of being able to deliver software you're confident in, it is faster to consider stability inherently in your development process than try to blast out features and bolt on stability later once you're committed to the initial brittle implementation.

<comment><USER2> Racing a car applies in two ways I think: the talent is not evenly distributed, and the masters are defined by their ability to decide when to go fast, and when to go slow.

<comment><USER3> I agree. To be more explicit, the talent is indirectly correlated with how slow your team's "slow" is going to be. I think the degree of slow-ness is what management ultimately feels when they conduct retrospectives on technical debt reduction initiatives.

We recently did a major refactor of our angular code (~1.5 months) and we've been pretty fast, releasing every 2 weeks. We've begun to outpace a lot of the backend development teams, so we now have more time to squash our backlog tickets. We have the prettiest Jira issue reports out of all teams.

</comment></comment></comment><comment><USER4> I think this post misses the point.

Company values are only meaningful when they differentiate that company from others. Everyone wants to "Move Fast and Not Break Things".

"Move Fast and Break Things" was a value judgement. For Facebook, moving quickly was worth the penalty of sometimes breaking things.

This is why most companies have lame values like "Do your best work" or "Be honest". Great values, but nobody disagrees with them—so they're meaningless.

<comment><USER5> I totally agree with this but would add a bit of nuance. I'd love to see all companies do a "X and the cost of Y" value system, but there is another part of stating you have values like "Integrity" which is the same as the Google "Don't be evil" thing. It's a claim that you can be called publicly on, and that's important as well. You'd be surprised how many companies out there aren't willing to make a claim for something like integrity. But yeah, who wouldn't say they value integrity, for instance, when asked?

That being said, I'd also add that most values have those implicitly baked in. For instance, if you do say you value integrity, you're saying you're devaluing growth/money for the value/benefit of integrity.

<comment><USER4> Yep, I agree with your nuance.

FWIW I think integrity is a pretty okay company value. Depending on your industry, valuing integrity may be a trade-off that other companies are unwilling to make

</comment></comment></comment><comment><USER6> I'd add: _think before you code_. Programming is hard because thinking is hard. If you don't use tools to help you think rigorously you're going to ship software with bugs regardless of how good your unit tests are or any other strategy you use to cope with poorly designed software.

I don't believe _not thinking_ is going to help you move faster. You're just going to spend most of your time fixing the problems you hadn't thought of before you started shipping code. And yet this is what we're encouraged to do. We're asked to fix the bugs, do the least amount of work to get it to go, and fix the problems later. We're asked to not think and just code. There are probably a handful of people in the world who can write complex systems in code with consistently correct results. The rest of us need to be more mindful.

We have wonderful tools available to us that can exhaustively check our designs. Using a system like TLA+ allows you to model your system in some pseudo-code and comes with a program that will exhaustively check your code for dead-locks, invariant violations, termination, etc. I've seen it used to find errors in graduate students' _binary search_ algorithms 9 out of 10 times. Imagine what it can do for non-trivial components of your application.

Part of moving fast is knowing how to avoid making errors in the first place. The only way I know how to do that is to _think_. You can't do this properly in code alone. You need higher-level tools to check your assumptions and find holes in your thinking.

You don't see structural engineers slapping together the first thing that works and patching the building later when parts of it fall down. You don't sign waivers before crossing a bridge that claim no responsibility on behalf of the designers if it falls down while you're on it. Nor do you see engineers accepting more money from people to build private bridges that are less likely to fall down. So why do software engineers have no such liability for their creations?

Why not take a few hours to write a proper specification for your distributed locking mechanism and save yourself days or weeks worth of debugging -- or worse -- liability suits when your mistakes harm the interests of your customers or the public at large? Why not use tools that check your thinking so that you can chase more interesting optimization and performance problems?

<comment><USER7> > You don't see structural engineers slapping together the first thing that > works and patching the building later when parts of it fall down.

That's because if they mess up the first time people can die.

I'm guessing the likelihood of death due to a bug in most web applications is probably much less than that of a bridge :)

Software is great because it's possible to iterate quickly. But ultimately, you have a point. Developers should be willing to fix bugs they've created and/or come across in their projects. We should also try to limit the potential of bugs by releasing small chunks of functional work as often as possible. "Releasing" in this sense can mean to production or just to your qa/test/dev/stage environment. It's just important to have many eyes on the product before it's live.

<comment><USER6> Someone doesn't have to potentially die in order for software to be harmful. Neither should it be the deciding factor about whether we need to think critically about the software we create.

Besides it's actually not too hard to write simple specifications for the critical components of your system. It has practical benefits like helping you to write good, rock-solid software. It helps you to clearly and precisely communicate your ideas. You can shake out errors in your design long before you write your code. And there are some errors you will only find by modelling and checking your system.

Web applications have come a long way since simple CGI scripts. They often require sophisticated orchestration mechanisms for managing distributed state and processes. If you get that wrong you end up with corrupted data, deadlocks and race conditions, etc. If you had a tool which helped you sift out those potential problems that exist in your design, would you not use it? Or would you rather risk introducing those errors in your code and hope that your customers never encounter them?

The approach of throwing code into production and "iterating" is precisely the problem I was addressing in my OP. In this approach we're asked to not think and to fix our inevitable mistakes later on after users find them for us. We have software to find a reasonable majority of them for us instead so why not use it? It's already fairly common to use unit and integration tests (to make sure our code operates as expected). Why not have a system for testing our designs?

As I mentioned, it's shockingly easy to write an incorrect implementation of binary search that may appear to work under your unit tests. Writing a specification of the algorithm and checking it with a model checker will show you problems you hadn't thought to consider. This is an algorithm that is taught in the first algorithms class you take and yet even graduate students, who scoff at writing specifications because it's _so boring_ , often get it wrong. That's because programming is hard and it is hard precisely because you have to think. And thinking is hard. We should be using tools to help us think.

</comment></comment></comment><comment><USER8> A lot of the bugs in our code are completely avoidable, but people get in the mindset of, "I need to move fast, bugs don't matter!"

Even if you don't have QA, when you write a bug, at least ask yourself, "what could I have done differently to avoid that bug in the future?" In some cases, it's as simple as, "put repeated code into a function so it doesn't get written twice." That will cut your chances of getting a typo in half.

<comment><USER9> So , really - you actually see typo bugs in the wild? I haven't seen one of those for 20 years.

I do see lots of cases where people didn't test it or used the wrong mechanism.

I have to go slowly because somebody already went fast and broke it.

And a friend once told me "LZW can compress code faster than you can." :) Which does not go to the value of orthogonality of purpose in source code, but I enjoyed the irony of the comment very much.

<comment><USER10> Math-heavy code tends to involve a lot of very similar algorithms, leading to a copy-paste-rename situation that isn't really helped by using a function.

As such it's incredibly common to end up with bugs from assigning to "x, y, y" and not "x, y, z" etc. Sometimes a clever compiler will warn you that you have done something odd, other times you'll be left to discover the runtime error.

<comment><USER9> Yeah, there's that. I've printed out code like that and colored different variables with different color highlighters to deal with it.

</comment></comment></comment></comment><comment><USER11> I think its weird that weve put stability and speed on the same axis. The reality is that you can ship faster with more confidence by focusing on stability from the beginning. Simply slapping some code together and going "it works!" is - in isolation - a recipe for disaster.

<comment><USER12> Only if the requirements are known. If you have firm requirements, you can test for them, you can plan for them, and you move through them much faster when you don't get surprised by random bugs that need to be tracked down.

If you _don 't_ have firm requirements, then each new fact you learn about your market potentially means you may have to invalidate large swaths of work you've already done. The more planning and testing you've done, and the more thoroughly you've covered your edge cases, the more there is to invalidate.

Most of the big business ideas in the last decade were in areas of extreme market uncertainty. Hence, companies that "move fast and break things" have been at a big advantage. This may change in the future - VCs today are enamored with Perez, and IMHO one of the signals that we've crossed over from the installation to deployment phase of the Perez model is when performance, stability, and security become valued more than features & speed of execution. But I'd guess that we have a few more years of moving fast and breaking things first.

<comment><USER13> Having tests can help you ensure that, when you do change one part of your application, unrelated parts don't start breaking. And I don't really buy the argument that, because something might be invalidated later, we shouldn't put in the time now to make sure we get it right based on the info we do have.

<comment><USER12> You should know _which_ part of the software lifecycle you're in and adjust your development practice accordingly.

When I first started my current startup idea, the product design was changing literally multiple times a day. I didn't even bother writing any code, it was all in pencil & paper notes & diagrams. Things have slowed down to about a requirements change every 2-3 days now, and I write code but no tests. If you spend 20% of your time fixing bugs and 80% of your time writing or rewriting features, tests are not the bottleneck; it's not an improvement to spend 50% of your time writing & rewriting tests so you can avoid the 20% spent tracking down bugs.

When I was at Google Search, working on a mature product with billions of users, every change got 100% test coverage, and you were _supposed_ to break a test with every change because that's how you know your test suite is comprehensive. And then it went out through a QA department and full release process. But adding a link to the page then took 2 weeks and 600 lines of code, while adding one to my startup takes 5 minutes and about 3 lines of code.

<comment><USER14> > If you spend 20% of your time fixing bugs and 80% of your time writing or > rewriting features, tests are not the bottleneck; it's not an improvement to > spend 50% of your time writing & rewriting tests so you can avoid the 20% > spent tracking down bugs.

That is a good line of reasoning about when auto-tests are needed.

However you did not take into account time that is required to discover that something is broken. Without tests you do not know if your code change breaks important features. So you should also consider how expensive it is to test your product after every code change. You do not have to test yourself and delegate testing to end users. But turning users into testers could be pretty expensive too (loosing potential customers).

So auto-tests should be introduced much earlier than when you hit "50% maintenance" threshold.

<comment><USER12> You don't need to worry about losing customers if you have none. At the stage I'm talking about, you'll know something is broken when you go to demo it and it doesn't work, and presumably you'll be walking through your demo manually before putting it in front of customers.

</comment></comment></comment></comment></comment></comment><comment><USER15> I'm actually very happy to work somewhere that believes in "Move slow and don't break things".

I believe that churn isn't progress, things with lasting value generally take time, and some problems _can 't_ be solved quickly. I enjoy working on problems that benefit from careful, considered, time-consuming thought.

You might desire something different. That's OK, but nobody is obligated to accept fait accompli the universal necessity or value of moving "fast" (and hopefully not breaking things).

<comment><USER16> Moving slow isn't an option for many businesses.

<comment><USER15> I believe it's an option far more often than people believe.

Customers do follow dependability and lasting value; nobody trying to their solve day-to-day problems actually wants a treadmill.

<comment><USER16> It's one of those cultural decisions you have to make early on in a company's formation and make sacrifices to hold on to because literally everyone else you're connected with will expect you to move at the same speed they're moving at.

</comment></comment></comment></comment><comment><USER17> You ever Notice how the engineering departments at these move fast and break things companies grow exponentially? Thinking of etsy, and Facebook, and a few others.

Part of it is if they survive, they thrived... But I think the other part is that the code starts to become sharded across people's brains. The technical debt piles up so high in some places that it becomes impossible to maintain productivity with the same number of people.

<comment><USER18> I think that's a non sequitur. Every startup engineering department grows exponentially. It's the only way for an engineering department to grow.

</comment></comment><comment><USER19> Move fast and break things for 0.001% of users.

In other words, try bold things, bold refactorings, etc. within bounds.

If a system is designed with good modularity, this can be done with very low risk.

When someone deploys new code at Facebook, I recall reading that it initially goes out to internal users, then to a small percentage of actual users, then eventually to the entire system.

This is simply a systems-based approach. The same applies when thinking about code in a testing environment, staging server, etc. At each level there are additional failure modes or adverse interactions.

When done stupidly, a cowboy approach can lead to downtime, poorly-reasoned quick fixes, and blame within an institution.

Often the same cowboy/girl who acted stupidly and caused the bug is lauded as a hero when he gets things working again a week later after the bug is discovered... when it was his/her bad judgment (or "move fast" mentality) that led the offending code to be shipped in the first place.

We must be honest with ourselves and our teams about bad decisions (even in hindsight) and build processes that let us be bold when we need to.

</comment><comment><USER20> Well, in physics we compute the speed with half life of what we want to keep alive regarding the costs.

Mass transportation is an example of industry where half life is decades/century. Fast for this sector is slower than what the life expectancy of a coder represents.

Games made to support a product ad are ephemeral, 1 month. Faster than a new release of firefox.

Some dangerous radioactive isotopes are 10000 years. So nuclear plant related automation should not be using any CPU.

A citizen data should last for the government as long a citizen live. Government if they aim at remaining stable should consider the half life of their automation to be long, way long. Hence slow changes. Very slow

So do banks. But banks have to adapt with the speed of trades.

Fast and slow can be set by doing something agile hates: careful business analysis.

The problem is how we coupled all the businesses with insane costly rhythm that are forced down on every economical activities by the mean of noncompetitive business practices. Like useless obsolescence driven by HW monopolies, cheap energy, cheap regulated and poor education.

Do we really need 1 Pb hard drive when entropy predicts we will not be able to find any relevant information given we have to much data without increasing the means involves in costly ways?

We don't need fast changing technologies, we need boring, slow changing technologies that are reliable.

</comment><comment><USER21> I have found that using BDD with gherkin syntax can really cut down on many conserns when it comes to software stability and ensuring business worth. Many companies do not put enough focus on automated testing and even when they do they are only testing the surface level with simple funtional tests. If one simply throws the technical requirements into the BDD tests and have your system validate every affected integration prior to shipping then you simply have less debt to pay off as more bugs should be caught prior to shipping. Couple BDD with writing a negative functional test for every bug that is fixed and you should never trigger that specific bug again. In the end these tests will take more time, but if that time is bundled into the time required for the feature, and your code is already layed out in a testable and reusable way, then it is win-win.

</comment><comment><USER22> Possibly related: "Move fast and fix things" by GitHub engineering. [https://news.ycombinator.com/item?id=10739129](https://news.ycombinator.com/item?id=10739129)

</comment><comment><USER23> I think this article misses what is meant by "Move Fast and Break Things". I never took it to mean 'physically (or programmatically) break things apart while moving quickly', I thought it was more about deconstructing things we considered "normal". Like another way of saying "think outside the box", and do that without hesitation or inhibition.

<comment><USER24> _"Most companies mess up by moving too slowly and trying to be too precise. When you're…moving quickly or doing anything like this you want to make mistakes evenly on both sides. We wanted to set up a culture so that we were equally messing up by moving too quickly and by moving too slowly some of the time. So that way, we'd know that we were in the middle."_

[https://www.youtube.com/watch?v=h7_UNu7zEVs](https://www.youtube.com/watch?v=h7_UNu7zEVs)

</comment></comment><comment><USER25> Great article!

> Be Aware of What You Break

I work at Rollbar (www.rollbar.com), and that point is probably the key to our existence. You can't afford _not_ to know what's going wrong. And the best way to know what's going on is to get a full stack trace with all the information you need to reproduce into a platform that will notify the appropriate people of the breakage.

</comment><comment><USER26> Facebook/Zuckerberg have obviously been successful because this particular motto has been dissected countless times in blog posts and HN comments.

Try this search: [https://hn.algolia.com/?q=fast+break](https://hn.algolia.com/?q=fast+break)

</comment><comment><USER27> Von Braun's team launched over 600 V-2 rockets before any of them reached a militarily useful target.

</comment><comment><USER28> I'm not worried about the "Move Fast and Break Things Often" mentality. This will only last until it kills a much of people. Think code in cars, plans, X-ray machines, pill dispensers etc.

</comment><comment><USER29> > In reality more often then not moving fast and breaking things will result > in shipping scrappy software. This is because in the rush to get stuff out > the fastest way time consuming things get skipped. Like user testing, > automation, analytics, monitoring, manual testing - just to name a few.

It sounds like the author could use a dose of his own medicine. The first sentence contains a (common) grammatical error. The third sentence is a fragment. The second is nearly unparseable.</comment>


When Robert Pinsky Wrote a Video Game - <AUTHOR> http://www.newyorker.com/books/page-turner/when-robert-pinsky-wrote-a-video-game <comment><USER2> You can play it online: [http://mindwheelgame.com/](http://mindwheelgame.com/)

Or read a more hackerly post about it: [http://www.filfre.net/2014/03/mindwhell-or-the-poet-and- the-...](http://www.filfre.net/2014/03/mindwhell-or-the-poet-and-the-hackers/)</comment>


Introducing the HP Jet Fusion 3D Printing Solution - <AUTHOR> http://h20435.www2.hp.com/t5/The-Shapes-of-Things-To-Come-The/Introducing-the-HP-Jet-Fusion-3D-Printing-Solution/ba-p/296466#.Vzs7bJMrLUY <comment><USER2> A slightly more informative video about how it works: [https://www.youtube.com/watch?v=XeTdo-w6Qx8](https://www.youtube.com/watch?v=XeTdo-w6Qx8)

Edit: And another much more informative video about the actual printing process: [https://www.youtube.com/watch?v=VXntl3ff5tc](https://www.youtube.com/watch?v=VXntl3ff5tc)

Looks like it uses black ink printing onto plastic beads spread on a printing bed to make some sections melt under light and others not.

Edit2: Probably not actually "black" but "UV Absorbing" for the ink.

<comment> gene-h It almost certainly uses black ink. All parts produced with the machine they have now are to be black. It technically doesn't use beads, it uses plastic powder. It works exactly as you described though.

<comment><USER2> Oh, yeah. I put the UV part in there because of the talk about full color printing. The first time through that video I missed the key phrase "could in the future"[1].

[1] [https://www.youtube.com/watch?v=VXntl3ff5tc&t=3m27s](https://www.youtube.com/watch?v=VXntl3ff5tc&t=3m27s)

<comment> gene-h Why would you need UV absorption for color printing?

<comment><USER2> Well, I'd assume white counts as a color, as well as other light colors. Plus they then go on to talk about printing clear things. You need to absorb energy from somewhere so I figured UV was about all that was left.

</comment></comment></comment></comment></comment><comment><USER3> Fyi: The printer is $130,000.

I'm not saying it isn't great, but it's not exactly for home use. Or if it IS for your home, you have an amazing home, can I please come live with you and use your printer?

<comment><AUTHOR> I've often thought that a new version of the Copy Shop (e.g. Kinkos) would probably be a better start for mass 3D printing because the machine could be much better than the home model and people really don't need to use it that frequently yet. $130,000 might be good enough for that use.

<comment><USER4> Agreed - and I think it will stay that way. Right now most people find it too much expense and trouble to keep a photo-quality color (2D) inkjet at home, and would rather pay per-photo at Kinkos/Walgreens. I don't see any signs of this changing.

I think 3D printers will go the same way. How many little plastic parts does the average consumer ever want to print? Probably not enough, and not rapidly enough, to justify a big purchase, when they can pay per-item at a (physical or online) Kinkos equivalent and get a higher-quality part printed on a machine like this.

</comment></comment></comment><comment><USER5> "Insert the pre-packed HP 3D Materials cartridges into the HP Jet Fusion 3D Processing Station." Will the use of 3rd party materials cartridges void my HP Jet Fusion 3D Processing station warranty? ;)

<comment><USER6> Probably. I doubt you'll be seeing third-party budget materials any time soon.

The printer looks interesting, but the PR is kind of insane. I'd really like to know some basic specs - speed, max object size, and so on - but I can't find them easily under the tsunami of marketing awesome.

<comment><USER7> You're not the target audience :).

Welcome to the XXI century - when companies finally figured that marketing has a much better ROI than producing anything of substance. I wonder if in the limit we'll hit a future when people will be selling each other stories and dreams, not bothering with actual products or services anymore? :).

<comment><USER8> >I wonder if in the limit we'll hit a future when people will be selling each other stories and dreams, not bothering with actual products or services anymore? :).

I think they call that "Kickstarter".

</comment></comment></comment></comment><comment><USER9> In the mean time Carbon ([http://carbon3d.com/](http://carbon3d.com/)) has also 'reinvented' 3D printing but can show a working model: [https://www.youtube.com/watch?v=O2thSsQrZUM](https://www.youtube.com/watch?v=O2thSsQrZUM)

I think Carbon will change 3D printing because they provide better chemical / structural (layer less) properties and better speed so it is more suitable for production models.

<comment><USER10> There are now several startups claiming to have "fast" photopolymer printing technologies. However Carbon is more about the materials R&D than the machine, it was merely a tool. CLIP is the most legitimized with press, funding, customers, and IP. (I just met the Carbon folks today to see their machines)

</comment></comment><comment><USER11> Can someone familiar with the space speak to how much of this is advertising and how much is genuine progress/innovation?

<comment><USER10> I am currently at the Rapid conference in Orlando where the HP announcement was made. When the technology was first announced last year I'm pretty sure they were the only ones using this kind of process. There are a few different powder/binder/inkjet type machines out there now but I don't think anyone is doing fusion in the same way.

The entire discipline of additive manufacturing is rapidly expanding with new processes and applications.

</comment></comment><comment><USER12> Theres a video explaining the print process

[https://www.youtube.com/watch?v=VXntl3ff5tc](https://www.youtube.com/watch?v=VXntl3ff5tc)

</comment><comment><USER13> Is this even real? I didn't even see any printing take place.

<comment><USER14> Sure real. Printing is demoed within the first video where you see laser beams working on a powder bed.

</comment></comment><comment><USER15> Funny, was there footage of actual 3D printing taking place?

<comment><USER16> Judging from the more informative videos linked in other comments, there is a brief timelapse shot of the printing. But I agree that with all the other CGI it is hard to trust it is a real timelapse shot.

</comment></comment><comment><USER17> Boxel by Boxel for Boxel presented by Boxel

</comment><comment><USER18> Who is the target market for this kind of printer?

<comment> gene-h Service bureaus. Companies that 3d print parts for people. Imagine a data center, except instead of servers you have row after row of 3d printers.

</comment></comment><comment><USER19> HP PageWide in 3d. The 2d printers are really nice, and don't suck down power like lasers.

These look like production presses but for 3d parts

</comment><comment><USER20> But will they Carlie Fiorina you on the ink? History says yes.</comment>