No Silver Bullet (1986) [pdf] - <AUTHOR> http://worrydream.com/refs/Brooks-NoSilverBullet.pdf <comment><USER2> The essay is remarkably prescient.

Brooks dared to predict that thirty years later, software would still be created by programmers sitting in front of editors, painstakingly typing up code branches for all of the scenarios a given program is supposed to handle. And their output would still mostly suck, because of the infinity of possible states, mostly due to variables and synchronicity. Complexity that cannot be abstracted away.

Given how starry-eyed we usually are about even the near future, that is bold.

The web, language advances, tooling - these are quality-of-life improvements. They don't 'solve' the inherent complexity of software development, the way it was promised by CASE tools (Brooks' likely target with his essay) and countless other 'business oriented' approaches. In fact those approaches have failed so many times that they have been largely abandoned, so in recent times, Brooks' essay might seem superfluous.

The one advance that might finally challenge the 'no silver bullet' rule is machine learning. Not yet, given that it is still an esoteric tool for a specialized class of problems, as part of traditional software systems. But with increasing computing power, I can imagine a future where machine learning can be set to work on broader tasks and start to look like magic self-directed software development.

<comment><USER3> > Brooks dared to predict

I love this. So rare nowadays to scientists make bold predictions.

Disclosure: I work on Tree Notation ([https://treenotation.org/](https://treenotation.org/)).

Two years ago, in 2017 I predicted Tree Notation will be a silver bullet: by 2027 we will have a 10x improvement in reliability, productivity, and simplicity, thanks to the Tree Notation ecosystem.

2 years later, and thousands of experiments and conversations later, I'm almost positive that will happen.

<comment><USER4> You should make a Long Bet for it.

Every lambda calculus, including yours, corresponds to a Cartesian closed category. Your system is not at all as new as you think; what you are actually inventing are new ways to write down parsers, akin to the many other ways that folks are trying to write down programs in JSON or XML or YAML.

The only advantage that you've got at this point is a 2D addressing scheme, you might think, but E had that in the late 90s: [http://www.erights.org/elang/kernel/LiteralExpr.html](http://www.erights.org/elang/kernel/LiteralExpr.html)

<comment><USER3> > You should make a Long Bet for it.

Second time I heard this suggestion ([https://news.ycombinator.com/item?id=20504193](https://news.ycombinator.com/item?id=20504193)), thanks! Done. [http://longbets.org/793/](http://longbets.org/793/) \- "In 2027, at least 7 out of the top 10 TIOBE languages will be Tree Languages AND/OR 0 out of the top 10 languages on 2017's list will be in the Top 10 in 2027. "

> Your system is not at all as new as you think

I agree. It's mildly new. Tiny tiny little details that make a big difference in practice. To get here I had to stand on the shoulders of giants: built the largest DB of computer languages in the world (over 10k languages with over 1k columns).

> but E had that in the late 90s

Great reference, thanks. E and Kernel-E are great, but the devil is in the details. There are a tremendous amount of tiny little details in Tree Notation and Tree Language that compound to make this something new. You can't take anything out of Tree Notation without losing something important. Everything else you can add via a Tree Language. That makes this closer to binary, imo. There are at least a dozen things that could be taken out of Kernel-E, which itself is a subset of E.

<comment><USER5> What exactly makes Tree Language any different from the context-sensitive encoding of code in binary?

<comment><USER3> Abstraction. Scope. (also, usability?)

Interesting question though. Can you elaborate?

</comment></comment></comment></comment></comment></comment><comment><USER6> I was at a Big Dumb Corp in 2005 or so on a client dedicated team. Our group manager decided to have everyone on the team take turns leading the weekly staff meeting. Mostly because he was lazy and the meeting was a particular waste of time. When it was my turn I printed off copies of this essay for everyone, handed them out, spoke a few sentences suggesting everyone read it. Then ended the meeting. That did not go over well at all with my manager.

<comment><USER7> How did it go over with your coworkers?

</comment></comment><comment><USER8> I wrote a short essay on this four years ago, here: [http://www.fmjlang.co.uk/blog/NoSilverBullet.html](http://www.fmjlang.co.uk/blog/NoSilverBullet.html)

The gist of it is that although Brooks was correct when he wrote No Silver Bullet, since then there's been an enormous increase in accidental complexity and, if that is recognized and removed, an order of magnitude improvement is now possible.

<comment><USER9> That's also the pitch for Dark ([https://darklang.com](https://darklang.com))! Great minds think alike

<comment><USER10> I still can't figure out what Dark is actually about. You seem to focus more on how fast it is to deploy than on what it is, how it works, how you use it, what it looks like or how its different from previous attempts (eg Eve). I'm a bit confused by this since deployment really has never been a particularly large pain point for me (setting up the actual infrastructure has always been a much bigger pain that actually deploying, plus typically I set deployment up once with CI and then don't have to think about it again), certainly not compared to actually creating complex software in the first place. You talk about why you built Dark, but I can't find anything about what it is you actually built. Am I missing an important blog post or page or something?

<comment><USER9> We've been positioning around the deploy a bit recently cause people got really excited by it, but that's not really what Dark is about.

Dark is about removing all accidental complexity from coding, starting with backends. That's pretty vague, but that's honestly how it started: this shit is too hard, what can we do to remove all this shitty stuff that we do.

Setting up infra is one of the three areas that we think is really shit, and that's a large part of Dark. You don't have to deal with infra at all (including DBs, queues, etc); it's all built-in.

Speed of deployment doesn't bother some people, esp those with smaller projects; on big teams it's a massive problem. People got excited about deployless in dark because it seemed like it had a credible vision for how it could be reduced.

We been talking about why we built cause we haven't finished building it yet. We're announcing our move to "private beta" (from "private alpha") on Sep 16th, so that's the kind of stage we're at. We'll be showing how Dark works then.

</comment></comment></comment></comment><comment><USER11> This kind of thing reminds me of a phrase from cycling: It never gets easier, you just go faster.

I feel like that's absolutely how software is. As we have better frameworks and libraries and everything the job doesn't get easier. But what happens is that it's now possible to do something with a team of three that might have taken thirty before.

Just look at Python/django or Ruby/Rails vs doing all that yourself with C and cgi-bin. How big would the C project be before it started to become difficult to work on by virtue of its many lines of code?

I'm not saying these new frameworks solve any kind of difficult theoretical problems by the way. And it's still work to make a django or Rails site. But there are many startups that got going in the last decade with just a few people that during the dotcom bubble might have taken 30, 50, even 100 folks to try and turn into reality.

<comment><USER12> It does get easier too. Better frameworks and libraries have made the job so much easier! The challenge I see is that people don't know how to code. They know the keywords of the language, they know programming constructs, design patterns, algorithms, frameworks, but they don't know how to program. You can know all the ingredients, you might even know the recipe but that doesn't make you a good chef or mean you know how to cook.

Something that I have not seen taught anywhere, no in schools, not at work, not on a thousand online courses is "HOW TO PROGRAM" This is why things have gotten hard for many people, they don't know how to program. But for those that do know how to program, things have gotten actually much more easier.

<comment><USER13> >> This is why things have gotten hard for many people, they don't know how to program

But how does this explain programming over the past 30+ years? You can't blame code camps as to why programming in the 90's has not gotten a magnitude easier, or even harder as you posit.

And what exactly do you mean "how to program"? you throw a lot of shade at the people, institutions and resources involved but don't shine any light on this missing piece...

<comment><USER14> > You can't blame code camps as to why programming in the 90's has not gotten > a magnitude easier

It has. Things that were hard problems then are simple tasks today.

OTOH, if you want to get paid a princely salary, you have to do the things that are still nontrivial. The stuff that has been trivialized has stopped being valuable, the things that used to be intractable but have been reduced to merely challenging have taken their place as worth paying for.

</comment></comment></comment></comment><comment><USER10> > "There is no single development, in either technology or management > technique, which by itself promises even one order-of-magnitude improvement > within a decade in productivity, in reliability, in simplicity."

I think its important to remember that the statement is rather specific:

"no single development" and "which by itself" \-- but there could be many developments, which, together, provide the order-of-magnitude improvement.

"order-of-magnitude improvement" \-- but there could be smaller improvements.

"within a decade" \-- the improvements may take longer.

I'm pointing this out because incremental improvements can and do occur, there's just no _" silver bullet"_ to fast track the process.

<comment><USER15> > but there could be many developments, which, together, provide the order-of- > magnitude improvement.

He says very clearly that this is likely at the remaining of the text.

> the improvements may take longer.

I imagine the point of posting this on HN nowadays is because in 3 decades there wasn't any. Still, something could appear tomorrow, but I imagine it's very unlikely, for the same reasons that are on the article.

</comment></comment><comment><USER16> In _No Silver Bullet Reloaded_ [1], a 20 year retrospective, Brooks said:

"Of the candidates enumerated in "NSB", object-oriented programming has made the biggest change, and it is [unlike almost every other proposed solution] a real attack on the inherent complexity itself."

And then goes on to say that the most promising approach remains reuse, particularly of COTS programs.

[1] [https://www.researchgate.net/publication/221321794_No_silver...](https://www.researchgate.net/publication/221321794_No_silver_bullet_reloaded_retrospective_on_essence_and_accidents_of_software_engineering)

<comment><USER7> > And then goes on to say that the most promising approach remains reuse, > particularly of COTS programs.

Indeed, although I think more beneficial than COTS has been the rise of open source (which could be seen as a variant of COTS). From open source languages with rich standard libraries to application frameworks to whole applications (databases, operating systems, etc).

The benefit from open source hasn't just been the lower TCO of 3rd party software, but also changes in the development process. Perhaps the single biggest process change has been distributed version control. Not only has Git significantly reduced the accidental complexity of collaborating on software projects, it has also become the primary mechanism for distributing open source code.

</comment></comment><comment><USER17> Er, I learned Prolog last summer and realized that about half of my professional career was wasted because I didn't learn it sooner. Dunno if Prolog counts as a Silver Bullet but those folks have slain a lot of werewolves.

</comment><comment><USER18> Thread from 2015: [https://news.ycombinator.com/item?id=10306335](https://news.ycombinator.com/item?id=10306335)

Can it really be that that's the only previous HN discussion of this classic?

<comment><USER19> There are certainly not too many.

Here is one [https://news.ycombinator.com/item?id=10306335](https://news.ycombinator.com/item?id=10306335)

</comment></comment><comment><USER20> The elephant in the room: We have no reliable way of measuring and comparing developer productivity.

</comment><comment><USER21> A great essay. I suggest Out of the Tarpit as a later exploration (20 years after No Silver Bullet) with fantastic and challenging analysis. If reading the whole paper is too daunting, checkout the summary (and subscribe to his regular email summaries of interesting papers) at: [https://blog.acolyer.org/2015/03/20/out-of-the-tar- pit/](https://blog.acolyer.org/2015/03/20/out-of-the-tar-pit/)

<comment><USER22> The Tarpit paper, if taken as a response to Brooks, suffers from a similar problem to the one I discussed here: [https://news.ycombinator.com/item?id=20829129](https://news.ycombinator.com/item?id=20829129)

It tries to build a theory in an Aristotelian manner, i.e. not based on careful observation but mostly on rationalization (and maybe very partial, biased observations). The problem with rationalizations is that they can often be made to support any claim when the empirical picture isn't clear. An additional problem in _this_ particular case is that when _No Silver Bullet_ was published, the same kind of people (PL enthusiasts) made roughly the same arguments, but their predictions proved wrong, whereas Brooks's proved right. It's not the end of the story, but it does mean that their theory needs, at the very least, to be revised.

<comment><USER23> Brooks suppositions on _essential_ tasks are not empirical at all. Otherwise they'd be incidental. And that's what the tarpit paper focuses on.

The tarpit paper is also not a prediction or forecast the way that the No Silver Bullet paper is. It reparameterizes and expands upon essential and accidental tasks and proposes a development framework to minimize accidental tasks.

The company I currently work for uses a code generation framework inspired by the ideas from the tarpit paper. It's very successful in simplifying and speeding up the development process.

<comment><USER22> > Brooks suppositions on essential tasks are not empirical at all.

It is an empirical claim, one that at least has not been refuted by observation.

> Otherwise they'd be incidental.

I don't understand this. It's an empirical claim about software in the wild.

> It reparameterizes and expands upon essential and accidental tasks and > proposes a development framework to minimize accidental tasks.

... and yet, no one has found a silver bullet yet (as per Brooks's definition), nor anything close to it.

<comment><USER23> > It is an empirical claim, one that at least has not been refuted by > observation.

There are no sources or pieces of evidence cited in the section on what the essential tasks are. If it's an empirical claim, then the any claim made in the tarpit paper is certainly equally empirical.

> ... and yet, no one has found a silver bullet yet (as per Brooks's > definition), nor anything close to it.

There was no such claim.

<comment><USER22> > There was no such claim.

Then what does it have to do with No Silver Bullet? Brooks's point isn't that you can't make languages that some people may find more attractive, but that you can't drastically reduce complexity.

> There are no sources or pieces of evidence cited in the section on what the > essential tasks are.

Right, that's why it's a claim. But it comes with an empirical prediction that was later verified by observation.

> If it's an empirical claim, then the any claim made in the tarpit paper is > certainly equally empirical.

Of course it is, but if we take it as a silver-bullet claim (i.e. the ability to drastically cut down complexity), then it just doesn't fit with observation.

<comment><USER23> I'm beginning to get skeptical whether you've actually read the paper. _It is not making a silver bullet claim or denying the forecast originally present in No Silver Bullet._ If so, it would be asserting some way to significantly reduce or completely remove essential complexity. It instead attempts to find a minimal definition of essential complexity, and proposes a method to mitigate the cost of non-essential complexity. _It is supplementary to No Silver Bullet, not a refutation of it._

Not to mention what observations are you talking about? Because I'm not too aware of any tarpit inspired languages/development frameworks, let alone an actual FRP framework.

<comment><USER22> > It is not making a silver bullet claim or denying the forecast originally > present in No Silver Bullet.

The paper's response to Brooks's central assumption, which leads to his prediction is, and I quote the full sentence, "We disagree."

It is an interesting opinion piece but it is entirely "Aristotelian".

> Because I'm not too aware of any tarpit inspired languages/development > frameworks, let alone an actual FRP framework.

Different languages and frameworks have adopted different parts, usually the more practical ones. None proved to be a silver bullet.

<comment><USER23> The actual quote.

>Following Brooks we distinguish accidental from essential diculty, but disagree with his premise that most complexity remaining in contemporary systems is essential

There's no silver bullet claim.

<comment><USER22> It very much is a silver bullet claim (following Brooks's definition). But here's the quote I was referring to, in context:

> Brooks asserts ... that the majority... of the complexity that we find in > contemporary large systems is of the essential type. We disagree.

That means a silver bullet is possible (against Brooks's claim), and then they go on to describe what they think is a particular silver bullet.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER3> If you liked this, years ago Fred Brooks recommended these books to me:

\- DeMarco & Lister Peopleware

\- 2007. Software engineering: Barry Boehm's lifetime contributions to software development, management and research. Ed. by Richard Selby.

\- Hoffman, Daniel M.; Weiss David M. (Eds.): Software Fundamentals – Collected Papers by David L. Parnas, 2001, Addison-Wesley, ISBN 0-201-70369-6.

\- And his: The Design of Design. Start with Part II.

</comment><comment><USER24> Is there a non PDF version of this or is there a way to read PDFs on an iPhone SE without magnifying glass?

<comment><USER25> No abstract but how about [https://www.cgl.ucsf.edu/Outreach/pc204/NoSilverBullet.html](https://www.cgl.ucsf.edu/Outreach/pc204/NoSilverBullet.html) ?

<comment><USER24> Yeah! Thank you.

</comment></comment></comment><comment><USER26> If the conceptual construct is made of concepts, and the power of high-level languages comes from being able to write software in concepts similar to the construct's concepts...

He states that the accidental complexity comes from mapping the conceptual construct to a real implementation.

But does that mean he's saying we already _know_ about the essential complexity of a given problem? That doesn't seem the case to me? The way we express the conceptual construct can also be improved, no?

<comment><USER26> Oh, I hadn't finished reading the article when I posted this comment.

Addressing essential complexity is discussed in the last part of the article ;x

</comment></comment><comment><USER27> > There is no single development, in either technology or management > technique, which by itself promises even one order-of-magnitude improvement > within a decade in productivity, in reliability, in simplicity.

Let's say it's true. Why should we care? Why set the bar at 10x, downplaying smaller (1.5x, 2x, 3x, etc) improvements?

Person 1: Hey, check out this tool that makes creating small web sites 3 times faster.

Person 2: Hah only 3 times faster? Who cares?

<comment><USER15> Because if there were a few techniques that improved our efficiency 10 times either, we should make it our top priority to find and adopt those, because a couple of those are already a gamechanger.

But 1.1x to 3x? There is a huge lot of those. People do say "Only 3 times faster? Why care?" all the time, and go focus on some other development.

<comment><USER27> > Because if there were a few techniques that improved our efficiency 10 times > either, we should make it our top priority to find and adopt those, because > a couple of those are already a gamechanger.

You're not really addressing the question, just talking about an easy hypothetical where no one could disagree.

What decision would one make differently if sold on No Silver Bullet (NSB)? I can't think of one. Would someone not woke on NSB prefer a 3x improvement over a 10x improvement, for example? Of course not.

<comment><USER15> > What decision would one make differently if sold on No Silver Bullet (NSB)?

If silver bullets were available, developers and researchers should let conservative changes aside and focus on studying high risk ideas.

</comment></comment></comment></comment><comment><USER28> The "Silver Bullet Syndrome" video by Hadi Hariri [1] is also worth a watch.

[1] [https://www.youtube.com/watch?v=3wyd6J3yjcs](https://www.youtube.com/watch?v=3wyd6J3yjcs)

</comment><comment><USER29> I love this paper and have used it repeatedly with clients over the years. I especially love the explanation of essential vs. inessential complexity (difficulties). A proto thesis on yak shaving, if you will.

</comment><comment><USER30> Yes Silver Bullet [https://news.ycombinator.com/item?id=20324523](https://news.ycombinator.com/item?id=20324523)

<comment><USER22> The funny thing is that that post doesn't address any of the theoretical arguments made by Brooks. While certainly less rigorous than physics, his argument is analogous to a discussion of the speed-of-light limit, and that "Yes Silver Bullet's" argument is analogous to, "but what if we use a different kind of fuel in our rockets?" and then claiming that that fuel is the silver bullet with neither empirical nor theoretical evidence to support that claim.

Indeed, quite a few PL enthusiasts called Brooks's predictions overly pessimistic based on similar non-arguments back in the '80s (he lists some of them in his followup, _No Silver Bullet, Refired_ ), but reality proved his predictions to be overly _optimistic_. So he was right and they were wrong. The arguments in Yes Silver Bullet had already been put to the test failed. Is that reality a _definitive_ proof that there is little accidental complexity left? Of course not, but it does raise the bar for those who claim there's a lot of it left, certainly well beyond simply _asserting_ that that is the case or considering it a reasonable working hypothesis.

<comment><USER7> So the author does conjecture that statically typed FP would represent a major improvement on accidental complexity. I agree that this is unlikely (though I haven't done enough of that myself to be sure) but most of the article seemed to be making an empirical claim about productivity improvements that have already been achieved.

Fred Brooks' central claim was that there would be no "order of magnitude" improvement by removing additional accidental complexity, and that real improvement would come from tackling essential complexity.

Whether Mark Seeman's empirical claim is true depends somewhat on whether we think the improvements were on the accidental or essential complexity, and whether any of them individually were an "order of magnitude" improvement.

What is accidental vs. essential complexity? According to Brooks, the essential complexity is formulating "complex conceptual structures" that make up the design of the software. In other words, figuring out what the software is supposed to do and how to do it.

Accidental complexity is concerned with "the mapping of [the design] onto machine languages within space and speed constraints". It seems to me that, according to Brooks, the biggest barriers for software development prior to 1986 was that programming languages were pretty horrible for expressing designs and that computers were too slow and limited to be able to effectively execute software (without seriously butchering the design).

His argument seems to be that, as of 1986, most languages were sufficiently high-level and computers (and PL implementations) sufficiently capable that most programmers could easily express their designs and expect them to be able to run efficiently enough. There might still be progress to be made there, but that it wouldn't be a massive improvement. Instead, most of the improvement would come from getting better at designing those "complex conceptual structures".

So, let's take a look at Seemann's list of improvements:

* WWW/Stack Overflow

* Automated Testing

* Git

* Garbage Collection

* Agile (the idea of iterative design/development rather than any process in particular)

Which of these tackle essential vs. accidental complexity (as Brooks defines the terms)? Agile development and automated testing would I think belong in the essential category (as Brooks himself refers to iterative design and tight feedback loops as methods for dealing with essential complexity). Garbage collection, on the other hand, seems like it dealt with accidental complexity.

I don't know about the others, but I would definitely say that garbage collection (especially the fast, low-latency GC we have now) does provide an order of magnitude improvement by reducing _accidental_ complexity. Obviously, GC was invented before 1986, but didn't become widely adopted until (more than) 10 years later.

Will there be other such improvements? Maybe. Statically typed FP doesn't seem like a great candidate to me, at least not by itself. I do however think that Peter Van Roy's formulation of pure FP + deterministic concurrency + message passing + global transactional state may qualify. That's not one single advance, but it does represent (IMO) a order of magnitude improvement on the imperative + "threads and locks" programming everyone was doing 20 years ago.

Still I think Brooks was right in the general sense, being that we should put more emphasis on producing better designs.</comment></comment></comment>


The human brain evolved when CO2 levels were lower - <AUTHOR> https://www.theatlantic.com/science/archive/2019/12/carbon-dioxide-pollution-making-people-dumber-heres-what-we-know/603826/ <comment><USER2> I bought a CO2 monitor, and did the same experiment at my office, few weeks ago. I work in a very crowded room, with only one window.

At first the CO2 measurement was "out of range" (>=3000 ppm), but it was not a surprise because I already suspected something was wrong with the ventilation system. It has been fixed now, but CO2 levels hardly fall under 1500 ppm.

My advice is to open one or more windows whenever possible, at least one hour per day. If possible, one window should be opened the whole day. CO2 tends to accumulate in ambient air and it takes times to dissipate, far more than I imagined.

</comment><comment><USER3> IMHO the Atlantic should wait for peer review before scaring everybody with sensational fears.

EDIT: having read further down I see this article isn't about just the one paper - maybe my comments need peer review.

</comment><comment><USER4> Will turn-of-the-century HVAC systems incorporate CO2 scrubbers?

<comment><USER5> For contemporary home HVAC setups, air exchange systems are already an option. There are also air exchange systems that trigger based on CO2 concentrations. They're demand-driven because concentrations can vary day to day depending on weather conditions, including wind.

</comment></comment><comment><USER6> I'd highly recommend everyone buy a CO2 monitor. You can get an all-in-one monitor on AliExpress for about $80.</comment>


World's most dangerous drug scopolamine removes free will and memory - <AUTHOR> http://www.dailymail.co.uk/news/article-2143584/Scopolamine-Powerful-drug-growing-forests-Colombia-ELIMINATES-free-will.html <comment><USER2> Scopolamine is not the worst drug that plants or animals can create, but is an evil one. Herbivore starts eating a plant, plant first do that the animal forget that was doing, and if persists finally put it at sleep until carnivores arrive. The remains of the corpse feed the plant. A master move if you think about it.

<comment><USER3> Wow, that is insidious. Does it work that way empirically, or is this conjecture?

<comment><USER2> This is the known effect on primates, first amnesia, then unconsciousness that can last for many hours, and finally death (in higher doses).

Herbivores just have learned to avoid it. Even goats that try to eat anything in sight and munch happily cardboard boxes if they can, do not dare to touch this plants. An herbivore unable to wake up and run is quickly spotted by predators.</comment></comment></comment>


Show HN: App to save your [random draw] pool for the 2014 World Cup - <AUTHOR> http://worldcup.nico.io/53a2177ba557d60200e26ee2

<comment><USER2> We use this for our pool and it works great. Switzerland & Russia FTW!</comment>


Show HN: A dummy web server to return arbitrary responses to test your services - <AUTHOR> https://github.com/svenstaro/dummyhttp <comment><AUTHOR> Hi everyone, I realize this is not terribly exciting or inventive but when testing a large company's infrastructure from the inside and outside, I noticed that there are a lot of services that I do not have easy access to that I would still like to test.

So imagine you have a reverse proxy that looks at a certain host and you'd like how that reverse proxy reacts if you send it a 200, 201, 400 or 500. Somehow I couldn't find such a tool and so I built a small dummy web server to do that.

Hopefully someone will find it helpful.</comment>


Emory Bubble v3 Interface Walkthrough - <AUTHOR> http://universitycollaboration.com/emory-bubble-v3-interface Would greatly appreciate your feedback and thoughts as we leverage an exclusive license from the university. <comment><AUTHOR> I would appreciate any feedback. We just finished designing this: www.emorybubble.com/preview</comment>


Ask HN: Can you write great software without writing tests? - <AUTHOR> <comment><USER2> Short answer: No

Great software implies that the software works correctly. Testing can never show the absence of errors but software with the right coverage criterion and sufficient coverage will always have fewer errors than software without tests (Ceteris paribus).</comment>


Show HN: Syncboard. A Universal Clipboard for Android - <AUTHOR> https://syncboard.sellomkantjwa.com/ <comment><AUTHOR> Hi all, I am currently working on a project that aims to create a synchronised clipboard across your devices. So if you copy some text on your computer, that will immediately be available to paste on any devices that you have chosen to synchronise. The first iteration only syncs from your a computer to an Android device. The option to go the other way is next. All feedback is appreciated.</comment>


ZFS High-Availability NAS - <AUTHOR> https://github.com/ewwhite/zfs-ha/wiki <comment><USER2> It's very nice to see someone putting in the work to document this and show others how they were able to get it to work.

But I unfortunately believe, quite sincerely, that redundancy at this level is misguided at best and dangerous at worst.

Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought.

Many people already know this. Even people who build these systems know this, but too many applications already exist that can't be change. Nonetheless, it's important for people building such systems for the first time to be aware that they can make a choice early in the design process and that the status quo is not a good choice. In other words, "Every day, somebody's born who's never seen The Flintstones." ([https://mobile.twitter.com/hotdogsladies/status/760580954532...](https://mobile.twitter.com/hotdogsladies/status/760580954532433920))

<comment><USER3> There are legitimate reasons to have a CP system and not an AP system, it's obviously application specific. Also over-engineered systems come with their own risks and can experience as much down time as a well setup CP system due to human error stemming from the system's complexity (AWS is a good example).

<comment><USER4> NASes are not CP systems, they are noCAP systems. They cannot guarantee neither consistency nor availability in the event of network partition.

<comment><USER5> Very good point. Quite often when talking about distributed system, everyone goes to the CAP theorem and automatically assumes if it is not AP, it must CP then. While in reality (and maybe for good reasons) it might be neither.

<comment><USER3> You sure can but it seems there are not that many practical applications for distributed system that are not tolerant to partitions.

</comment></comment></comment></comment></comment><comment><USER6> FreeBSD has a cool HA ZFS solution you can use.

HAST [https://wiki.freebsd.org/HAST](https://wiki.freebsd.org/HAST)

There are also other solutions which you can hack together like iSCSI or GEOM Gate Network [https://www.freebsd.org/doc/handbook/geom- ggate.html](https://www.freebsd.org/doc/handbook/geom-ggate.html)

I would however strongly recommend a proper distributed storage solution like Ceph or GlusterFS.

<comment><USER7> People will do whatever they want to do. And my comment is not a knock on HAST.

But myself, and I know I am not alone, firmly believe especially regarding HA and ZFS that following anything other than the KISS principle is pain upon pain upon pain.

Wether it's iSCSI, HAST, NFS, HAST or any combination thereof.

The simplest and least painful? Is to simply create two servers with dual nic's and dual HBA's.

Assign half the disks in a mirror to HBA-1 the other half to HBA-2.

Use one nic for public networking, use the other for a direct P2P Gig-E link between server A and server B.

Use zfs replication over the private network link using the granularity you feel most comfortable with. 30 minutes? 15 minutes? 5? It's up to you.

This is the simplest, and IMO most reliable way to go about this. When you start adding in iSCSI, NFS, HAST etc you are simply making things to complex, introducing multiple points of failure, and the pain when something does break? Ill pass. I'd rather have to manually intervene and fail over to server B if server A fails and vice versa than deal with all the pain of additional things that could break or introduce split brained issues etc.

But it's a free world, use whatever solution fits your comfort zone.

<comment><USER4> The biggest issue with your approach is consistency. You just cannot guarantee it unless you are using proper distributed algorithms on every level.

</comment></comment></comment><comment><USER8> Ed White, the guy behind the Github repo, is a very experienced sysadmin (and the guy who knocked me off the top scoring slot on Server Fault). His experience supporting both COTS and custom applications in a variety of environments is top-notch.

</comment><comment><USER9> To add to the downsides, you can't expand RAIDZ vdevs.

If you start with a 6 disk RAIDZ2 and want to add a couple more drives, you can't. The only way to add capacity to the pool is to add an entire new vdev.

Unfortunately I don't think there's an expandable file system that handles parity/erasure coding as well as bitrot that can be used right now. BTRFS might be usable after the RAID5/6 rewrite and Ceph might be more usable after Bluestore comes along (at the moment erasure coding performance seems to be pretty bad, even with an SSD cache) but in the present there's nothing.

I really hope [https://www.patreon.com/bcachefs](https://www.patreon.com/bcachefs) manages to fix this.

<comment><USER10> > Ceph might be more usable after Bluestore comes along (at the moment erasure > coding performance seems to be pretty bad, even with an SSD cache)

I just saturated a 10Gb link in a k=4,m=2 EC configuration on Ceph (Haswell). Too lazy to set up concurrent clients. What do you mean by pretty bad? This is a Hammer cluster without SSD journals or cache.

Bluestore was recently released with Jewel, but shouldn't affect EC performance drastically, if at all.

<comment><USER9> Maybe I'm doing something horribly wrong. My setup was:

\- 6x4TB HDD

\- 2x120GB SSD

\- OSD per disk

\- k=4,m=2 erasure coded HDD pool

\- 1 copy replicated (not) SSD pool

\- SSD pool writeback cache for HDD pool

I was consistently seeing the SSD writeback cache fill up then the disks thrashed at 100% IO utilization, limiting incoming writes to ~40MB/s from memory.

<comment><USER10> Oops, didn't see this reply until now. It's no longer recommended to use a SSD cache pool, much for the reasons you describe.

You will be a lot better off using bcache or similar, on top of the OSD device.

</comment></comment></comment></comment><comment><AUTHOR> I think this is quite amazing, building high-availability storage is a complex endeavour because the entire storage path needs to be 100% redundant.

It's quite an achievement to get this functional using open-source.

</comment><comment><USER11> One day we will hopefully have first class ZFS mirroring using shared nothing hardware. Until then dual-pathed JBODs will have to do.

</comment><comment><USER12> Am I reading the output right, that he is creating three-way stripe over raidz1 volumes? Seems like bit odd configuration for HA solution especially. I mean, doesn't that mean that the array can't handle two disk failures except if it gets lucky?

<comment><USER13> It means that the vdev on top of the three RAIDZ vdevs can lose up to three devices simultaneously, one per RAIDZ each.

Any further failure would result in a complete data loss. Also, if any of the RAIDZ stripes loses more than one vdev simultaneously, this will lead to a complete data loss as well.

<comment><USER14> This is a simple example setup in a 12-drive enclosure. 9 disks, a global hot- spare disk and two cache drives.

Previously, I would have used straight ZFS mirrors for performance and expandability (raidz VDEVs can't be expanded), but at this scale, the 9-disk 3x3 raidz1 setup consistently outperformed a 10-disk RAID mirror setup in random and sequential I/O.

See: [http://i.imgur.com/SeSlmH8.png](http://i.imgur.com/SeSlmH8.png)

<comment><USER15> A set of 5 stripes of mirrors has the same problem that the comment brought up - losing the _wrong_ two disks loses all of your data.

In this case, going to raidz2 across all 12 disks would give an extra disk worth of capacity/performance with the ability to lose any 2 disks without data loss and going to raidz3 should give the same capacity/performance with the ability to lose any 3 disks without data loss.

<comment><USER14> Hot spares and monitoring. RaidZ2 across that number of drives is a performance nightmare and against ZFS best practices.

</comment></comment></comment></comment></comment><comment><USER13> Always on the prowl for cheap just a bunch of disks enclosures for my Solaris / SmartOS systems, from the links in the github readme.md, I learned a whole bunch of stuff, for example, that Sun made the J4400 / J4100 just a bunch of disks external serial attached small computer system interface arrays, and these are dirt cheap on ebay:

[https://docs.oracle.com/cd/E19928-01/820-3223-14/J4200_J4400...](https://docs.oracle.com/cd/E19928-01/820-3223-14/J4200_J4400_Overview.html)

that in turn led me on the search for the actual host bus adapter supporting this, and behold:

[https://docs.oracle.com/cd/E19928-01/820-3222-19/J4200_J4400...](https://docs.oracle.com/cd/E19928-01/820-3222-19/J4200_J4400_ReleaseNote_chapter.html#50401294_59854)

further "digging" on ebay uncovered that these low profile host bus adapters can be had for as cheap as $18.95 USD plus shipping.

Since this stuff is professional grade equipment, the repossession companies are having a really, really hard time reselling this equipment: enterprises will not buy this because it is so specialized that they do not know about it (and they would much rather buy ultra expensive EMC with a support contract), and private persons will not buy this because most people do not even know what it is, or that it exists, never mind that it requires one to have a rack with professionally installed power and possibly cooling.

As a consequence, this equipment goes for peanuts on ebay, and is a way for someone who knows what they are doing to implement highly reliable, enterprise storage solutions for a mere, tiny fraction of the cost.

Not bad for five minutes worth of research, that was time well invested.

<comment><USER14> This is key. I mean, I focus on the HP side and used ProLiant servers and storage enclosures in my examples... But the Sun/Oracle enclosures and components are very inexpensive off-lease and come at a price-point where it's totally possible to self-support.

</comment></comment><comment><USER16> ZFS forces you to choose a drive size a priori and live with it for the life of the system. Thanks but no thanks. I'll stick with OpenStack Swift where I can leverage new larger disks and have my uncoupled highly available storage to boot.

<comment><USER17> This is not true. Read up on mirrored vdevs. [http://jrs-s.net/2015/02/06/zfs- you-should-use-mirror-vdevs-...](http://jrs-s.net/2015/02/06/zfs-you-should- use-mirror-vdevs-not-raidz/)

<comment><USER16> 50% storage efficiency and 87.5% survival is not an option for us. We store medical images by the petabyte and drives fail much too fast at scale.

<comment><USER18> It's great to hear how you are using Swift. I work on that project; feel free to reach out if you have any questions. Contact info is in my profile.</comment></comment></comment></comment>


RIP Passwords: US Government Tackles Security With Online IDs - <AUTHOR> http://www.virtuosimedia.com/business/internet-security/rip-passwords-us-government-tackles-security-with-online-ids

<comment><USER2> Isn't this essentially the same thing as OpenID? Why not move to promote OpenID across more platforms instead of rebuilding the wheel?</comment>


Chrome 59 has cross-platform headless support - <AUTHOR> https://www.chromestatus.com/features/5678767817097216 <comment><USER2> This is fantastic. I'm using a combination of Chrome and PhantomJS for karma testing right now, for [https://github.com/paypal/paypal- checkout](https://github.com/paypal/paypal-checkout) and [https://github.com/krakenjs/xcomponent](https://github.com/krakenjs/xcomponent). There are hundreds of tests opening up hundreds of iframes and popup windows, and sending a lot of cross-window messages, and that ends up being really memory hungry.

Chrome deals pretty well with garbage collection, so long as I'm careful to de-reference closed windows properly¹, and only uses a maximum of 150mbs. PhantomJS eats up almost 6GB of memory before it's done, which makes it almost unusable on machines with less memory or CI boxes. Travis is a no-go.

I'm hoping running Chrome in headless mode should give a nice speedup for our tests.

\-----

¹ Turns out even a closed popup window or iframe keeps a huge amount of memory hanging around. Who knew.

<comment><USER3> We (Chrome) have reached out to PhantomJS to inquire if they're interested in collaborating: [https://groups.google.com/d/msg/phantomjs- dev/S-mEBwuSgKQ/tU...](https://groups.google.com/d/msg/phantomjs- dev/S-mEBwuSgKQ/tU06ZOZcFgAJ)

The DevTools Protocol is the primary API for headless Chrome, but we are excited for higher-level abstractions like PhantomJS & NightmareJS's API to manipulate the browser as well. Plenty of details to work out, but hopefully sometime this year you'll get a drop-in solution for some of your testing to upgrade from Phantom's older QTWebKit to the latest Chromium.

<comment><USER4> A few hours later: [https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuN...](https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE)

<comment><USER5> After his announcement, Vitaly answered to the team: [https://groups.google.com/d/msg/phantomjs- dev/S-mEBwuSgKQ/PQ...](https://groups.google.com/d/msg/phantomjs- dev/S-mEBwuSgKQ/PQdE1A5FAwAJ)

> >Is there interest on your side in adopting Chromium as a runtime? There's > some existing documentation [2] around the API and embedding, but > admittedly, this would be some work.

> We are interested. But I am afraid not in the current state. Currently, > PhantomJS heavily relies on Qt and QtWebKit. It's not that easy to adopt > Chrome as a new runtime.

> But I think we could implement PhantomJS as a completely new (with the same > API) project that will use Chrome - Phantomium!

<comment><USER6> Also see [https://github.com/Vitallium/phantomium](https://github.com/Vitallium/phantomium)

</comment></comment></comment></comment></comment><comment><USER7> I've been testing Chrome headless extensively for the past few months, and while it's a good step, but it's not stable for high-volume or even diverse set of webpages.

Memory usage is pretty high, lot of heavy webpages result in crashes/hangs, there are many inconsistencies between features available in full version and headless, their debugging protocol has different APIs that work on headless/non-headless in Linux or Windows, and so on.

Of the bugs I've submitted, some have been fixed in the upcoming M59, so other critical ones may take longer due to their backlog. I suppose for now (maybe until M61-62), Chrome full with xvfb or even PhantomJS are better options. When you realize that Chrome is about the same size (by LoC) as the Linux kernel [1], you can't help but wish for a leaner & faster headless browser.

There seems to be some work going on building Firefox pure headless as well. Great overall, as long as all the browsers try to follow the RemoteDebug initiative [2].

[1] [https://josephg.com/blog/electron-is-flash-for-the- desktop/](https://josephg.com/blog/electron-is-flash-for-the-desktop/)

[2] [http://remotedebug.org](http://remotedebug.org)

<comment><USER8> I've been successfully using Chrome headless in a 500MB Docker container for dumping the DOM for [https://www.prerender.cloud/](https://www.prerender.cloud/) for months (rendering a large variety of sites without restarts for weeks at a time)

Run it with:

      --js-flags="--max_old_space_size=500"

to force the VM to keep it GC'd below 500

Chrome v55 was a 30% memory savings, before that I used 1GB containers.

It's not perfect, but I am definitely pushing high volume (multiple tabs, concurrent activity) and I am not having any significant stability issues and I am pushing diverse sets of webpages.

<comment><AUTHOR> What the best strategy that worked for you on high volumes, multiple tabs or multiple docker instances? I am wondering if multiple tabs is as efficient as multiple windows/instances.

Thanks.

<comment><USER9> I'm using more than one Chrome process so I can kill the processes every so often (e.g. after timeout or when they get stuck). Inside each Chrome instance I use 16 tabs, there might be a number of factors at play:

\- Are you worried about same-origin pollution if you run multiple tabs from the same origin in the same process? If so -> Extra process \- Do you have to take screenshots? You can only take screenshots of the tab that's in the foreground, so you have to activate it first to take the screenshot. This might fail if you have lots of tabs which roughly trigger at once.

You can see what I've built at [https://urlscan.io](https://urlscan.io) btw.

</comment></comment></comment></comment><comment><USER10> [https://bitbucket.org/lindenlab/dullahan](https://bitbucket.org/lindenlab/dullahan)

I've been working on a fully open source Windows/macOS library (via Chromium Embedded Framework) that allows you to render pages to memory (and then of course to bitmaps, textures etc.) as well as inject synthesized mouse/keyboard/JavaScript events. It currently uses (what amounts to) Chrome 57.

Looks like this might make my project obsolete.

<comment><USER11> Thank you anyway for your efforts; I hope you interpret this as a validation of your having correctly identified a sorely-needed tool.

<comment><USER10> Thanks keville - I've poured my heart into this on nights and weekends after I'm done with my day job for a long time now but what you say is so true - hopefully we'll get access to a more robust solution compared to my modest hacks.

</comment></comment></comment><comment><USER12> A lot of phantom js talk here makes me want to recommend [http://ghostinspector.com](http://ghostinspector.com)

It's a phantom js (and other headless browser) web service. Using the site, you can quickly create different tests, scheduled tests, chained tests, keep screen shots, create videos of multi step tests, and have historical information of it all.

Can't say enough good things about the site.

Edit: also there's a great chrome extension that will record your mouse clicks and keyboard commands to make creating a test that much simpler.

<comment><USER13> +1 to GhostInspector; I used them at a previous company a few years ago, and it was very useful.

They were just starting, but service was rather reliable, and their tech support was excellent (maybe because we were early customers). We used to run a bunch of automated tests for monitoring and compliance, archiving hourly screenshots over different builds for later comparison.

</comment></comment><comment><AUTHOR> Also checkout [https://github.com/cyrus-and/chrome-remote- interface](https://github.com/cyrus-and/chrome-remote-interface) for an easy way to fully control those headless instances

<comment><USER14> If you're using Go, [https://github.com/knq/chromedp](https://github.com/knq/chromedp)

It's also worth noting that the 57+ series has a nice embedded viewer you can use to view the actual viewport via devtools.

<comment><USER15> WOW. That's fantastic. As I noted in a comment above, I wrote a tool called crabby that uses Selenium and Chrome headless to do automated page testing and to report results back to metrics engines like Graphite, Datadog, Prometheus, and Riemann. The biggest problem I have is the unreliability of chromedriver and the extreme resource consumption of Chrome + Selenium. It's really too much for your average public cloud instance if you want to test any more than, say, one or two pages a minute.

Do you know if chromedp can access any of the timing measurements?

<comment><USER14> Yes, you can access everything via the underlying APIs. chromedp is a relatively new project (only about 4 months old), so there isn't much yet in the way of high level timing / profiling, but we hope to add that to the code base when we have some bandwidth to do so.

</comment></comment></comment></comment><comment><USER16> The feature I was most interested in when they announced this last year was virtual time. The Developer Resources link has it listed ([https://chromium.googlesource.com/chromium/src/+/lkgr/headle...](https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md)) but it's a broken link, unfortunately.

Mostly, I'd like to know how the control of the virtual time system would be exposed. Would it be through the C++ API, or could it be made available through the debugging protocol?

</comment><comment><USER17> Any way of scripting this to automate button clicks etc? I use PhantomJS for this now but found it to be incredibly unstable for complex pages.

<comment><USER18> There is an excellent library called nightmare (based on electron). If you need print-screens you can use xvfb

or wait for this issue: [https://github.com/segmentio/nightmare/issues/224](https://github.com/segmentio/nightmare/issues/224)

<comment><USER19> it should be noted that nightmare isn't safe for untrusted websites: [https://github.com/segmentio/nightmare/issues/1060](https://github.com/segmentio/nightmare/issues/1060)

</comment></comment></comment><comment> fake-name Shamelessly bumping my project to produce a nice python API for the Chromium/Chrome-remote-debugger-protocol: [https://github.com/fake- name/ChromeController](https://github.com/fake-name/ChromeController)

I'm trying to replace PhantomJS in my infrastructure with chromium. Not having to build my own chromium will be a very nice thing.

</comment><comment><USER20> I've been trying to test audio and video with headless browsers (namely PhantomJS) but have experienced extreme difficulty, I wonder if headless Chrome is able to support/supports already HTMLAudioElement or HTMLVideoElement or any media interface that would make, for example, testing YouTube or SoundCloud embeds easier.

</comment><comment><USER21> Related: I want to take screenshots of a few news websites for a little fake news project of mine, and most approaches return something completely different than what I'm seeing when I open Chrome.

Limited height would be better/ok (something like the first 3000 pixels).

Low volume / can be slow (30 seconds would be ok).

Those news websites many times have infinite scrolling.

I've tried:

\- phantomJS (rendering sucked, tried every technique I could find to wait for JS to load)

\- wkhtmltopdf (almost ok, generates a huge 30M image with all the height, no antialiasing it seems)

\- [https://github.com/gen2brain/url2img](https://github.com/gen2brain/url2img) (this was the best so far, uses Qt bindings but not the latest version)

\- actually run a headless browser in DigitalOcean with xvfb-run and take a screenshot: I failed at this

What I didn't tried was Selenium, because it seemed even harder.

How would you guys do it?

<comment><USER22> I use xvfb for selenium testing. It is really easy to set up and can take screenshots or videos while automatically browsing websites.

<comment><USER21> Thanks - I tried Selenium (on the desktop) with geckodriver now and it rendered well. The only thing is that long screenshots didn't work but there is probably a workaround for that.

</comment></comment></comment><comment><USER23> Any chance for webgl here? Would be nice for automatic screenshots and webgl tests.

<comment><USER24> It is being discussed.

[https://docs.google.com/document/d/1VTcYz4q_x0f1O5IVrvRX4u1D...](https://docs.google.com/document/d/1VTcYz4q_x0f1O5IVrvRX4u1DVd_K34IVUl1VULLTCWw/edit?disco=AAAAAxbGHos)

</comment></comment><comment><USER25> Oh my goodness I have been waiting for this day for a while - we ran into PhantomJS problems with keyboard/mouse eventing and the HTMLVideoElement for testing, this sounds like it should be the cure for our woes of having to hack around PhantomJS's deficiencies.

</comment><comment><USER26> Nice!! Would creating WebRTC data channel be possible?

server side SCTP to client(p2p over SCTP/data channel) would be cool.

<comment><USER27> It's already possible. I'm using webrtc and node-electron to connect a golang server for my MMO project. I have a farm of 4 nodejs processes running under tmux acting as proxies for unreliable communications.

[https://www.emergencevector.com](https://www.emergencevector.com)

<comment><USER28> Which WebRTC library are you using on the golang server? Last time I looked, I couldn't find anything stable.

<comment><USER27> I am using electron-webrtc. You need xvfb installed. I was able to get this running under node, but only running the processes under tmux. You will also need to install some random shared libraries Chromium needs, but which node- electron doesn't install, but these are obvious from error messages.

Then I'm using simple-peer on top of that. There's also a library for UDP communications from the node process to the golang process.

</comment></comment></comment></comment><comment><USER29> What does this mean for Electron and other apps that depend on Electron?

<comment><USER30> I am looking forward to cli tools powered by electron. Imagine the beauty of a grep writting in js, executed by a chrome binary. :D

<comment><USER31> Imagine the terrible performance of grep written in js, executed by a chrome binary.

</comment></comment></comment><comment><USER32> Would this run as a chrome driver for Selenium? What is needed to make this work with Selenium?

<comment><USER33> It should be possible, but as a browser rather than a driver so you'd still need Chromedriver to glue Chrome and Selenium together. In Karma with karma- chrome-launcher[1] you can pass options to the browser using the flags option, eg;

customLaunchers: { chrome_headless: { base: 'Chrome', flags: ['\--headless'] } }

I've not tested that yet but I can't really see a reason why it wouldn't work.

[1] [https://github.com/karma-runner/karma-chrome- launcher](https://github.com/karma-runner/karma-chrome-launcher)

</comment></comment><comment><AUTHOR> Almost one year old but there was a talk on headless chrome at the Blink conference (BlinkOn 6):

Video: [https://youtu.be/GivjumRiZ8c](https://youtu.be/GivjumRiZ8c)

Slides: [https://docs.google.com/presentation/d/1gqK9F4lGAY3TZudAtdcx...](https://docs.google.com/presentation/d/1gqK9F4lGAY3TZudAtdcxzMQNEE7PcuQrGu83No3l0lw)

More links: Headless Chrome architecture: [https://docs.google.com/document/d/11zIkKkLBocofGgoTeeyibB2T...](https://docs.google.com/document/d/11zIkKkLBocofGgoTeeyibB2TZ_k7nR78v7kNelCatUE/)

Mailing list: [https://groups.google.com/a/chromium.org/forum/#!forum/headl...](https://groups.google.com/a/chromium.org/forum/#!forum/headless- dev)

All of those links are on [https://chromium.googlesource.com/chromium/src.git/+/master/...](https://chromium.googlesource.com/chromium/src.git/+/master/headless/README.md)

</comment><comment><USER34> Fantastic news, not a minute too soon! Can't wait to get rid of PhantomJS. Now if only this was a standard feature of all browsers...

</comment><comment><USER35> Is it possible to install Chromium on a server without X environment? Last time I checked it requires a shit ton of dependencies.

</comment><comment><USER36> PhantomJS has plenty unsolved issues(up to 1.7k+), a replacement instead of combination might be a better choice.

</comment><comment><USER37> How fast is headless vs. normal? According to

[https://developers.google.com/web/fundamentals/performance/c...](https://developers.google.com/web/fundamentals/performance/critical- rendering-path/render-tree-construction)

the chrome browser spends a decent amount of time on other steps such as parsing HTML. I wonder how much time could be saved by not rendering pages into pixels.

</comment><comment><USER38> This page doesn't load for me (IE behind corp firewall). How does one drive/automate a headless browser? What kind of API is there?

<comment><USER39> Docs and a Youtube presentation...

[https://chromium.googlesource.com/chromium/src/+/lkgr/headle...](https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md)

[https://www.youtube.com/watch?v=n6biclFh0i0&feature=youtu.be](https://www.youtube.com/watch?v=n6biclFh0i0&feature=youtu.be)

</comment></comment><comment><USER40> I put together an example of how to run chromium with --headless driven by cucumber/capybara: [https://dkastner.github.io/2017/04/21/headless- chrome.html](https://dkastner.github.io/2017/04/21/headless-chrome.html)

</comment><comment><USER41> For those interested, Firefox is also going to support a headless mode. The current nightly supports headless SlimerJS on Linux and more platforms will come soon.

[https://bugzilla.mozilla.org/show_bug.cgi?id=1338004](https://bugzilla.mozilla.org/show_bug.cgi?id=1338004)

</comment><comment><USER42> Looks like the launch bug is private? [https://bugs.chromium.org/p/chromium/issues/detail?id=705916](https://bugs.chromium.org/p/chromium/issues/detail?id=705916)

</comment><comment><USER43> Have people found many issues that come up in Chrome but aren't found in PhantomJS? We used to use a headless browser but switched to PhantomJS and haven't had any real issues.

(We should probably run under the real IE but jut haven't been bothered.)

<comment><USER44> Phantom does not support more recent JS syntax/tweaks. We have an app that is aimed at more recent browsers only so can use latest ES6 features, and had to move from Phantom to in-browser tests (the alternative would be to use babel for transpilation, but then we wouldn't be testing the code that is actually released to users)

</comment></comment><comment><USER45> Can anyone confirm - would this work with a Flash/SWF application? i.e. could I use the headless mode to interact with the Flash Application to run some commands and retrieve the output?

I tried googling around but didnt find much to say either way...

</comment><comment><USER46> It seems that Chromium 59 still can not be installed on Raspberry Pi, or anyone has done it?

It will be great to use this headless Chromium on Raspberry Pi to execute some routine web browser jobs.

Does it support the extensions installed on Chromium? Curious.

</comment><comment><USER47> Does this mean I no longer need to use phantomjs for my tests?

</comment><comment><USER48> How fast is the debugging mode? I tried the first debugging protocol when Chrome added it and it was very difficult to use. I assume this time is different?

</comment><comment><USER49> Doesn't seem to work on OSX. Connecting to debug port from a different chrome brings up an empty page.

Running Version 59.0.3069.0 (Official Build) canary (64-bit)

<comment><AUTHOR> Works for me when using [https://github.com/cyrus-and/chrome-remote- interface](https://github.com/cyrus-and/chrome-remote-interface)

<comment><USER50> Looking at [https://developer.chrome.com/devtools/docs/debugging- clients](https://developer.chrome.com/devtools/docs/debugging-clients), it seems we are missing a Ruby client. Last time I tried it gave me some headaches trying to talk to the websockets, but hopefully someone smarter than me can pick it up.

</comment></comment></comment><comment><USER51> This is great if the headless mode supports the web extension API because it means that we can run our security tools almost as command line tools.

</comment><comment><USER52> i use when they announced headless mode on linux, and built generating thumbnails from captured screenshots of websites and uncovering the technologies used on websites

and the api is available for free, [https://github.com/letsvalidate/api](https://github.com/letsvalidate/api)

</comment><comment><USER53> Can somebody explain to me what is this good for?

Thank you.

<comment><USER54> Headless browsers are frequently used in buildserver testing of web apps.

</comment></comment><comment><USER55> Is this confirmed to work with Flash? I know selenium did not support Flash so it caused some dev issues.

</comment><comment><USER56> Given the rough comments on the Electron story earlier, this should be welcome by all.

</comment><comment><USER57> Not sure if this can enable SSR (server side rendering) for any client side lib?

</comment><comment><USER58> What is the use case for headless Chrome?

<comment><USER59> > Headless mode allows running Chromium in a headless/server environment. > Expected use cases include loading web pages, extracting metadata (e.g., the > DOM) and generating bitmaps from page contents -- using all the modern web > platform features provided by Chromium and Blink.

Practically speaking, software developers will use headless Chrome to automate testing of product functionality. Today, developers use systems like Selenium or PhantomJS to accomplish this feat, but it's a painful process to maintain these headless browser execution engines. Adding headless support into Chrome means that developers can count on the presentation of their application on a given version of the Blink engine run within Chrome.

</comment></comment><comment><USER60> is this is chromium, too?

</comment><comment><USER61> Please let this be capable of generating PDFs from HTML from the command line.

<comment><USER62> FWIW, you can do that now, using phantomjs (which is chrome) [http://phantomjs.org/screen-capture.html](http://phantomjs.org/screen- capture.html)

<comment><USER63> Also with [https://wkhtmltopdf.org/](https://wkhtmltopdf.org/) and [http://pandoc.org/](http://pandoc.org/)

<comment><USER64> And, if you need much higher fidelity and control of HTML/CSS -> PDF, there's the fantastic Prince library, [http://princexml.com/](http://princexml.com/) (nonfree)

(I've been using Prince for over a decade, rendering everything from prescription labels, packing slips, receipts, resumes, books, and more. It's great.)

<comment><USER65> That's also an API service, [http://docraptor.com](http://docraptor.com), with a different pricing model that uses the Prince library for PDF rendering.</comment></comment></comment></comment></comment>


Apple: March 9th 'Spring Forward' Event - <AUTHOR> http://www.apple.com/live/ <comment><USER2> Well, considering we change the clocks to "spring forward" 2 a.m on March 8, all signs point to some kind of watch being released. The Swiss must be shaking in their boots while frantically clicking through the Android SDK. ;)

</comment><comment><USER3> I don't even care about new hardware or products. I just want it to all be incredibly stable and well-supported.

<comment><USER4> From your lips to god's ears. But, I don't think "stability and support" are gonna make the agenda for March 9.

</comment></comment><comment><USER5> I'm a programmer, and know very little about art (sadly).

The picture in the background here is very beautiful. Someone who makes that kind of thing, how do they do it? How do they chose colours, and how would they go about making the shapes actually appear on a screen?

<comment><USER6> It looks like a small handful of semi-transparent shapes (they look like leaves to me) with different colors that are overlayed on each other. Since they're different semi-transparent colors, the colors blend to form a new shade where they have overlapped. Repeat this a few times with logic that rotates the leaf around the axis a little below the bottom center of the screen, and there you have it.

<comment><USER7> Yep. It's just a more gradated version of the iOS Photos app icon. It's pretty easy to see what makes the simpler version look the way it does, when you blow it up[1]: there are just eight translucent roundrects with gradients, and everything else emerges from how they overlap.

(And from the fact that the gradients form a complete hue continuum. Look at the small triangles in the middle to see the destination colors, and see how they match the source colors of their neighbours.)

[1] [http://www.icreatemagazine.com/wp- content/uploads/2013/08/Pr...](http://www.icreatemagazine.com/wp- content/uploads/2013/08/Preview.png)

</comment></comment></comment><comment><USER8> Presumably announcing the Apple Watch is available to the huddled masses...

<comment><USER9> I am looking forward to the outrage when the Apple Watch gold edition is priced at $5-10k.

[http://daringfireball.net/linked/2015/02/24/caldwell- edition](http://daringfireball.net/linked/2015/02/24/caldwell-edition)

<comment><USER10> ...while lasting about 3 hours on a single charge [http://9to5mac.com/2015/01/22/apple-targets-for-apple- watch-...](http://9to5mac.com/2015/01/22/apple-targets-for-apple-watch- battery-life-revealed-a5-caliber-cpu-inside/)

Really, if they will get away with that, the world has surely gone crazy.

<comment><USER11> I love how they reason it away in the article:

"Considered separately, the active use app, clock, and fitness numbers sound very low, but the reality is that people will passively wear the Apple Watch for most of the day, actively interacting with it only for short periods of time."

Essentially they're saying, "People will buy an expensive watch that does all this extra fancy stuff, but really they won't bother much with the extra fancy stuff - because who actually wants to fiddle around with something on their wrist for any protracted length of time?"

I think Apple has enough die-hard fans, who will buy almost anything they make, for it to be hard for them to do something that will register as a genuine complete flop... but I think the Apple Watch really could struggle to get much traction amongst the masses, and I don't just mean the expensive gold model.

<comment><USER6> If you used your iPhone at full power all day, it would be dead in three hours. If you left it in standby, it would last for days. Most people do a combination of these things, leading to the battery life averaging out to be around one day of normal use.

Is that essentially saying people bought an expensive phone that does all this fancy stuff but don't bother with it because they don't want to fiddle with something in their hand for any length of time?

Try flooring the accelerator on your car and see how long the tank of gas lasts. Compare it to idling away a full tank. Do people shy away from hot- rodding their engines just because they don't want to fiddle with the throttle, or because doing the speed limit and accelerating slowly tends to suit their needs just fine?

<comment><USER11> The aforementioned article about the Apple Watch talks of "heavy application use" and suggests it might only last 2.5 hours. If you read, for example, [http://www.trustedreviews.com/iphone-6-review-battery- life-a...](http://www.trustedreviews.com/iphone-6-review-battery-life-and- verdict-page-6), it suggests "heavy use" of the iPhone 6 (at 50% screen brightness) yields about 14 hours of battery life. That's a pretty significant difference.

While it's easy to say "it's more compelling to do stuff on a phone than it is to do it on a watch, so you'll naturally use the phone more", that rather makes me wonder why I would bother buying the watch, especially when you actually need the phone as well.

Time will tell how people use these devices and whether the battery really holds up to a reasonable person's usage. And, of course, it seems likely (though not certain) that later generations will have improved battery life.

<comment><USER6> Because of the big screen, it's easier to browse the web on a tablet than on a phone. Easier than that is a laptop. A PC has an even bigger screen than that! But people still use phones, mostly for the minor convenience of having it in your pocket instead of in a bag, but also for the unique interactions it provides.

I use my PC for eight hours a day straight. I use my phone for maybe 10 minutes here, 10 minutes there, maybe an hour long call once or twice a day. Is there anything wrong with that? Should I be using my phone more? I mean, why would I spend all the money on having a powerful phone when I'm only using it occasionally?

That's the logic you seem to be putting towards this watch. I'm not going to defend the battery life, it's totally crap. I'm just arguing against the logic. You seem to be saying if you're not using the device constantly, it's not worth having. I don't use my car constantly. I don't use my phone constantly. I don't use my Pebble constantly. But they're nice to have for their own reasons. The point of the watch is not to use it 100%, all the time. It's to use it when it makes sense to use it. Heavy application use isn't what it's designed for.

Still, I agree the battery makes it useless even for the intended functionality.

<comment><USER11> I agree with a lot of what you're saying. It just seems to me that it's a lot of money to spend on something that you aren't intending to use all that much (regardless of the battery life problem). You have a phone in your pocket that works quite well for many tasks; do you really need to spend a bunch more money on a fancy watch that does some of the same things as your phone, via your phone, in possibly a less useful way?

I really don't know, perhaps Joe Public will see it very differently to me; certainly those of my friends who don't work in tech, but overwhelmingly do own iPhones, I have spoken to about the Apple Watch seem fairly uncertain as to its general usefulness.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> I was wondering why they were painting the Yerba Buena center yesterday. The railing and awning were yellow and I saw a bunch of painters out there painting everything gray. When I walked by today, almost everything was gray. I'm hoping for some retina macbook air, new macbook pro and of course the watch.

</comment><comment><USER13> My early 2011 MBP finally gave up the ghost this past weekend and I bought a new 15" on Monday. I _knew_ this was going to happen.

<comment><USER14> The event's probably going to be all about the watch and none about the rest.

<comment><USER13> Yeah, I read that after the fact. I'd still give it an outside chance that they announce Broadwell MBP's but if Apple Watch is the star of the show, then that's somewhat unlikely.

</comment></comment></comment><comment><USER15> I've purchased a Pebble and, more recently, a Moto 360. I love the idea of smart watches but I feel like they need to be as passive as possible for the best user experience. So far the Apple Watch looks like it needs a lot of interaction to do different things. I'm excited to see a real demo of the interface to see if that's really the case.

</comment><comment><USER16> Quad core i7 on the mini's please. It would be nice to have parity with 2012 in 2015.

</comment><comment><USER17> I get that a retina macbook air would be great, but the big selling point of the macbook air is its battery life, and I don't anticipate them sacrificing that.

<comment><USER18> The refurbished Macbook Airs on Apple's web store are a pretty good indicator if there's a refresh coming.

Usually the 15-18% refurb discount jumps to around 25-30% if those models are going to get discontinued. Not seeing that on the store (yet).

</comment></comment><comment><USER19> If they're going to sell a watch, does this mean our iPhone alarms will forevermore work correctly during DST switchovers and calendar year increments?

</comment><comment><USER20> I hope this is the new Macbook Pro

</comment><comment><USER21> just give me a retina mbair pleeeease!

</comment><comment><USER22> Yes, tangent, but please support the movements in several states to abolish daylight savings time. Having time move out from under us has long been a pain for programming robust systems. Perhaps the 5th or 6th generation of Apple watch will not need to do such an archaic adjustment.

<comment><USER23> Daylight Savings Time has some ill effects, true, but if you think that making life easier for programmers is a point that should convince the general public, you're sorely out of touch. For one, it's not all that difficult in the scheme of things. For another, programmers dealing with dates in the past will always have to consider it. Lastly, the general perception is that programmers get paid quite well enough and can suck it up and deal with this minor annoyance.

<comment><USER24> Just wait till we travel to other planets' timezones. That's when programmers will really earn their paychecks.

<comment><USER25> Or when timezones depend on how fast you are traveling.</comment></comment></comment></comment>


Go to www.owl.com and you get AOL.com with an Owl background - <AUTHOR> http://www.owl.com

<comment><USER2>
        Set-Cookie: adCanvas=Paper_Owl; Path=/

That would do it.</comment>


Comodo CA rebrands as Sectigo - <AUTHOR> https://sectigo.com/newsroom/comodo-ca-rebrands-as-sectigo <comment><USER2> Ironically this webpage's certificate is verified by "COMODO CA Limited". I suppose it takes time to inform browsers that "SECTIGO" certificates are safe.

<comment><AUTHOR> Yeah it'll probably take years before we start seeing Sectigo root certs. Maybe finally they can also stop capitalizing their name.</comment></comment>


Cloudflare has been down 16 times and counting in 16 days - <AUTHOR> https://www.cloudflarestatus.com/history <comment><USER2> Once you go big, the probability of getting errors and downtime increase too. Not even K8S HA will ever be safe. Just get on with it.

</comment><comment><USER3> Misleading title</comment>


$5,000 Security Breach - <AUTHOR> http://blog.joemoreno.com/2014/04/5000-security-breach.html

<comment><USER2> >'They' spin up spot instances which isn't subject to Billing Alerts. You'll need to cancel those spot instances, revoke your AWS credentials, and change your account password," he said.

This doesn't make sense at all. Amazon should let us if monthly bill > X send me a priority email and phone call. Why do they hide behind these dark patterns? I thought they were better than that.

<comment><USER3> The fact of the matter is amazon is a giant company and cannot thoroughly think through each piece of logic in their system.

The reward for focusing on this before-hand is much lower than just writing a check for $5k to this person and then fixing later (lot of $5k checks from amazon today. Wheres mine?)

</comment></comment><comment><USER4> >"'They' spin up spot instances which isn't subject to Billing Alerts.

Spot Instances aren't subject to AWS Billing Alerts? Is this common knowledge?

<comment><USER5> I'm not sure it's _true_.

</comment></comment><comment><USER6> Why the cliffhanger? Why not just finish your thought? As is, this story provides nothing other than showing the Amazon provided some helpful customer service.

</comment><comment><USER7> We had an almost identical event last Sunday in one of our dev accounts: multiple high end spot instances in multiple regions with a newly created security group pointing to a suspect IP.

We caught and corrected it quickly, but we still don't know how the keys leaked out - we have chalked it up to lower security practices since it's not a production account and is shared by more people (e.g. no 2-factor on it). We started to investigate, but then Heartbleed happened.

I wish there were more mechanism in AWS to prevent bills from mounting up, but the basic billing alarms worked in this case. I can't imagine how or why spot instances would be excluded from alerts, their cost certainly is included in the estimates that alerts are based on.

</comment><comment><USER8> Another reminder to be extra careful about checking in AWS credentials into version control. You never know when you might open source that repository and someone can easily yank it from Github.

<comment><USER9> Actually this is what happened to us... our AWS credentials had accidentally been put into Github for a hackathon project several months ago.

Coincidentally, the incident also occurred around the same time (April 1-2). We were hit with $13,000 worth of EC2 usage before we shut them down and changed our AWS key... We reported to Amazon, and they are working on a refund.

</comment></comment><comment><USER10> The instance was spun up on April 2, but Heartbleed wasn't disclosed for almost a week later. I highly doubt anybody used the Heartbleed 0-day to access your account.

<comment><USER11> According to Cloudflare ([http://blog.cloudflare.com/answering-the-critical- question-c...](http://blog.cloudflare.com/answering-the-critical-question-can- you-get-private-ssl-keys-using-heartbleed)), exploiting heartbleed may actually be very difficult. So yeah, it's very unlikely for that to have happened.

<comment><USER12> Well, getting an SSL private key is difficult as they don't often get into memory and are quite long (difficult to get from 64k at a time). Whereas AWS credential keys are something that get into your servers RAM much more frequently and are shorter strings. So it could easily be remote memory exploitation. But more likely social engineering or some other easy path in.

<comment><USER13> Heartbleed only exposed SSL memory (like incoming or outcoming connections), but not other memory (particularly not program memory), containing AWS keys.

</comment></comment></comment></comment><comment><USER14> A similar thing happened to me when I made public a previously private repo on Amazon and forgot to scrub it for AWS key. Luckily I got an email from Amazon within 24 hours of the instances being started, and as such my bill was just $360, which in any case they waived off. You might want to check your Github repos.

</comment><comment><USER15> I have 2-factor auth enabled on my AWS login - but am I right in thinking that if someone has my API keys that they don't need the 2nd factor?

<comment><USER16> Yes, someone with your access and secret keys can spin up instances, create buckets, and do everything else that the stolen keys are authorized for.

<comment><USER5> Which is why most things should be done with IAM keys specifically locked down to minimal privileges.

<comment><USER15> Apparently I have access keys that predate the release of IAM ! Fortunately there's a convenient "disable" link on security keys page.

</comment></comment></comment></comment><comment><USER17> The fact that the consultant - unprompted - knows exactly what is going on suggests to me that Amazon needs to fix this. If one customer gets burned then the customer is an idiot. If lots of customers get burned then maybe its not the customers that are the problem.

</comment><comment><USER18> Is there anything you have done that might have compromised your account. Moreover bill could as well been much larger than $5000 i guess.

</comment><comment><USER19> Is it too pedantic to want the tech support to just say they were probably mining cryptocurrency instead of bitcoin? It's most likely the intruder was mining either litecoin or whatever coin is most profitable for the month. I know it's all very much the same but they were almost certainly not mining bitcoin.

A $5,000 AWS instance would mine about $1 worth of bitcoin and would not be worth the time logging into someones account.</comment>


Creating FastAPI (8k stars) from a collage of ideas and experience - <AUTHOR> https://sourcesort.com/interview/sebastian-ramirez-fastapi <comment><AUTHOR> Editor here. This is a great in-depth interview that goes through the full process behind creating a popular open source framework from Sebastián Ramírez of FastAPI.

Any feedback on the style/format is welcome.</comment>