How to Compromise the Enterprise Endpoint - <AUTHOR> https://googleprojectzero.blogspot.com/2016/06/how-to-compromise-enterprise-endpoint.html <comment><USER2> "Because Symantec uses a filter driver to intercept all system I/O, just emailing a file to a victim or sending them a link to an exploit is enough to trigger it - the victim does not need to open the file or interact with it in anyway."

That seems big. Is there any precedent on AV software vulnerabilities of this scope?

<comment><USER3> Worse than that. Apparently,Tavis emailed the exploit to Symantec in a password protected zip file. He included the password in the body of the email. The email server, running Symantec, grabbed the password out of the email, decrypted the zip file, and upon reading the exploit code, crashed itself.

<comment><USER2> I very much started reading your post as a comedic sci-fi ending to the story.. but now I am actually not sure. It would be quite creative to think of that scenario! Did this happen?

<comment><USER4> I'm afraid this did actually happen[1].

[1]: [https://bugs.chromium.org/p/project- zero/issues/detail?id=82...](https://bugs.chromium.org/p/project- zero/issues/detail?id=820#c1)

<comment><USER5> I do not see any indication in the link you posted. Am I missing something?

<comment><USER4> > I think Symantec's mail server guessed the password "infected" and crashed > (this password is commonly used among antivirus vendors to exchange > samples), because they asked if they had missed a report I sent.

</comment></comment></comment></comment></comment></comment><comment><USER6> So, what do people run on their servers / macbooks for AV? Anything?

I was in a meeting just last week with our new "head of Security" who exclaimed when i stated our Macbooks nor or Ubuntu severs run any AV software (We run firewalls and things like fail-2-ban, but no traditional AV).

I know i'm going to get into a debate with them over this, so, what would be a good 'win-win' type position for me to fall back on to satisfy this point and not clutter my machines up with junk, if there is such a thing?

<comment><USER4> The first tool I run on new Macs is osxlockdown[1] (use [2] if you want a UI). It disables a bunch of features and enables things like the firewall. Make sure you don't disable things you're actually using, though. I don't run any AV, but I use OpenDNS Umbrella, a DNS-level malware blocking service with the capability to switch to "active" traffic filtering (basically a MitM proxy, though that part is completely optional). It's a neat tool with a nice dashboard, and the pricing is okay with $20/year.

OSSEC is a great tool for servers, but not the kind of "Let's just throw some AV at it so we can tick that compliance box" tool many are looking for.

[1]: [https://github.com/SummitRoute/osxlockdown](https://github.com/SummitRoute/osxlockdown)

[2]: [https://objective-see.com/products/lockdown.html](https://objective- see.com/products/lockdown.html)

<comment><USER7> >[PASSED] Disable IPv6

I'd like to see the rationale behind disabling IPv6 outright.

Edit: [https://github.com/SummitRoute/osxlockdown/issues/4](https://github.com/SummitRoute/osxlockdown/issues/4)

</comment></comment></comment><comment><USER8> From the perspective of a person who thankfully no longer has to support any Windows based platforms:

"Symantec considered harmful"

full stop.

Let's not forget this: [http://arstechnica.com/security/2015/10/still-fuming- over-ht...](http://arstechnica.com/security/2015/10/still-fuming-over-https- mishap-google-gives-symantec-an-offer-it-cant-refuse/)

Symantec should have suffered the CA "death penalty" and had its trust removed from the browsers that hold most of the global market share.

<comment><USER9> What if there were a middle ground between browsers trusting and rejecting a CA? What if there were a yellow "proceed with caution" warning on sites using certificates issued by CAs that have very occasionally behaved improperly?

<comment><USER4> If this gets implemented in a way that's clearly visible to the user, the effects on the CA are almost indistinguishable from the death sentence. The vast majority of customers would switch to some competitor that's not affected. Even a handful of saved customer support cases due to this warning is going to make up for the cost of switching. Most importantly, if users are confronted with warnings all the time, they become increasingly oblivious to them and might very well click through more important warnings down the line.

</comment></comment></comment><comment><USER10> It's hard for me to believe that anyone uses this crap software. A few years ago I spent hours uninstalling it for a friend. It has slowed his laptop to a crawl and he was about to buy a new one. After the uninstall, it was snappy enough to use for a few more years. Really, that software is some of the worst I've ever witnessed, and I've seen some shit.

<comment><USER11> Before windows 10, windows antivirus were the best advertising for Apple computers

<comment><AUTHOR> What has Windows 10 changed on this front?

<comment><USER12> Presumably now Windows 10 is the best advertisement for Apple

</comment></comment></comment></comment><comment><USER13> A note for everybody asking "why on earth does anybody run this software": When my company had to get corporate liability insurance in 2007/2008, the actual insurance contract stipulated "having AV installed on all machines". We did solve it by having an unused folder with ClamAV on every box, but I was impressed by the fact that AV is pretty much legally mandated for enterprises.

<comment><USER14> This is very common in boilerplate enterprise contracts. They will often have provisions about compliance with certain security and disaster recovery standards.

</comment></comment><comment><USER15> I always wonder why, despite all these flaws and vulnerabilities, big enterprises still use them.

Is there some kind of "compliance" or "regulation" that mandates companies to install them on every workstation?

<comment><USER16> Well would you want to be the guy (or gal) responsible for explaining why all your computers don't have A/V installed after an infection?

<comment><USER15> I would make that argument a few years back when 3rd party A/V is the only choice. But now you have the built-in Windows Defender, which is a valid A/V, why not upgrade your OS and get it for free?

<comment><USER17> Some endpoint vpn solutions scans the pc to try to make sure it is secure. Some of these might not recognise or accept all antivirus solutions.

(Source: doing remote support)

</comment></comment></comment></comment><comment><USER18> A bug in their software would be forgivable. This article pointed out both an extremely poor design decision (lots of unnecessary code in the kernel) as well as a serious organizational problem (not doing vulnerability management). These are especially bad considering that they supposed to be a security company.

In both cases, one bad example means it's likely there are many more still undiscovered.

</comment><comment><USER19> Win32k before Win10 used to do TrueType/Type 1 parsing in the kernel, with an entire bytecode virtual machine!

<comment><USER20> So what? Linux, today, has a full bytecode interpreter

<comment><USER21> Bigger attack surface in the kernel, for something that doesn't need to be there, and that is historically very difficult to code securely.

</comment></comment></comment><comment><USER22> Anti virus is like a compromised immune system: it joins the other side and will help to kill the host in short order. It's a miracle these companies are still in business and it is very sad to see Peter Norton's name dragged through the mud like this over and over again.

</comment><comment><USER23> Many years ago, installing Malwarebytes Anti-Malware dramatically reduced the amount of on-site technical support calls for my well-meaning but too trusting ("I just clicked on it") parents. This was before I was able, with the help of my brother-in-law, to convert them to Apple/Mac.

Is Malwarebytes Anti-Malware still the gold standard for Windows Malware protection? What is the gold standard for Windows virus protection now?

<comment><USER24> The gold standard is now Apple MacOs or Linux. I've converted my parents over to Apple systems and no longer have to do any support for them, Apple products "just work". Worth the premium in saved support, in my book.

I use Ubuntu as my main OS, personally.

<comment><USER19> To be honest, Mac OS X do have things like Gatekeeper.

<comment><USER25> Do you mean MacKeeper? GateKeeper is Apple's software that prevents you from installing non-verifed apps, MacKeeper is (from what I can tell, haven't installed it or looked at the site) spamware.

</comment></comment></comment></comment><comment><USER26> IMHO, the security industry has been guilty of adding complexity to existing systems rather than doing its duty of stripping it away.

<comment><USER27> I'm not really sure I consider McAfee, Norton and the like the security industry. They're definitely a _part_ of it, but in the same way car dealerships are part of the auto industry. They provide a service, but there's a real debate to be had about whether they are more beneficial or harmful.

<comment><USER26> Fair point, though at times it seems that the markets and regulators in this space tend to favor turnkey products, appliances, et cetera. To ride on your car analogy (pun intended), one finds it is not easy buying directly from a manufacturer, the heavy bias is towards the dealerships.

</comment></comment></comment><comment><USER28> The software you buy to keep you safe actually exposes you to more risk than if you didn't buy it. How ironic.

<comment><USER29> How much more? I always imagined that it's more likely for a standard user to open every email attachment and execute it than it is to get targeted by a malicious attacker who knows what software your users are running and writes exploits tailored for them, but I could be wrong.

<comment><USER30> No tailoring required with something like this since it's worm-ready; blast out a ton of emails to seed the worm, then post exploit the worm emails itself to everyone in your address book. It won't take too long before a significant portion of the vulnerable systems win the world are infected.

See Blaster, Slammer, CodeRed for historical examples....

</comment></comment></comment><comment><USER31> Isn't Norton antivirus itself malware? And McAfee too, for that matter? I finally convinced my mom and my wife to stop downloading it everytime they update Adobe Flash. (Yes, they still do that. On Windows of course. Sigh. One thing at a time.)

<comment><USER19> At least it is only McAfee Security Scan which don't seem to have the kernel stuff.

<comment><USER32> I thought even Security Scan had the {network, FS} filter drivers?

</comment></comment></comment><comment><USER25> Here's a question I have every time I see "RCE" type issues, and I'm completely serious when I ask: what is the use case for allowing remote execution in your software? Why would you want to allow arbitrary code to be executed? Or am I perhaps misunderstanding this, is it some sort of break out of the program bounds which allows execution?

<comment><USER33> It's a vulnerability. No one wants it (except attackers).

Of course there are some situations that some people actually do want (controlled) remote code execution, such as ssh.

</comment></comment><comment><USER34> What are the reasons one would want to use an antivirus? Can someone share some insight on how does antivirus actually work?

<comment><USER35> Traditionally they scan files, either in a batch job or on the fly, to see if the file matches a signature, basically a hash, of known viruses. This worked when there were only a couple thousands viruses in the wild. Now this approach is pointless because there are so many viruses, odds of the antivirus having seen it before are effectively zero. Not to mention countermeasures virus writers take make this even harder.

<comment><USER36> Signatures are definitely not hashes. They usually match families of malware or methods of obfuscation. Nor do any respectable AVs scan after something has started running...

AVs usually upload suspicious files from user PCs for further analysis to help aid in discovering new variants/types. The odds your AV has seen it is proportional to the odds of you running into it.

The industry is moving away from signatures regardless. Now AVs use runtime heuristics to spot bad behavior of executables and block them even if they've never been seen; Cylance literally only does this and it's pretty effective.

<comment><USER37> Sounds like the same old snake oil to me.

> if being "part of a community" means we need to share our algorithmic, > unique conviction engine with Big AV so they can steal our convictions, then > yes we will not be able to meet that criteria.

I have no love for Symantec but this new breed of security software can go screw itself for all I'm concerned.

</comment></comment></comment></comment><comment><USER38> Quite a few major enterprises use SEP/SEPM in combination with other IPS/IDS. Time to make sure everything is updated I suppose. Good work project zero.

</comment><comment><USER39> I've been saying this for years, but when are people going to realize that running Norton on your PC is actually worse than not running AV software at all?

</comment><comment><USER40> In the fast-moving world of IT security it's refreshing to see that Symantec's web site makes no mention of these profoundly important vulnerabilities on their landing page

They don't seem to have any Status / Current Alerts style pages -- but on their somewhat hard to find blog page we find the most recent update from the guys is from two days ago:

"Malicious app found on Google Play, steals Viber photos and videos"

[http://www.symantec.com/connect/symantec-blogs/symantec- secu...](http://www.symantec.com/connect/symantec-blogs/symantec-security- response)

EDIT: Oh, they have a Vulnerabilities page - [https://www.symantec.com/security_response/landing/vulnerabi...](https://www.symantec.com/security_response/landing/vulnerabilities.jsp) \- with the most recent entries listed as 13 days ago (blimey that US mm/dd/yyyy date format is uncomfortable).

<comment><USER40> Symantec have described some vulnerabilities that sound like these ones, dated 2016-06-28 (no time) here:

[https://www.symantec.com/security_response/securityupdates/d...](https://www.symantec.com/security_response/securityupdates/detail.jsp?fid=security_advisory&pvid=security_advisory&year=&suid=20160628_01)

Detailed description and credits sections don't seem to be in complete alignment with details of OP, but I may be misreading.

<comment><USER41> I don't think that they have much to do with the OP. The OP's post is easy to understand, Symantec's seems to be corporate double-speak.

The more important question is what should Symantec/Norton users do to prevent being exploited right now?

</comment></comment></comment><comment><USER42> Windows 10 has a built-in antivirus that's very effective, safe, and doesn't impact system usability. There's little reason for anyone on Windows 10 to run Symantec/Norton.

<comment><USER43> > and doesn't impact system usability.

I have to disagree with this statement. It's kind of a resource pig, actually. When it got to the point it was affecting my day-to-day productivity, I deleted MsMpEng.exe from my hard drive, and now my machine is snappy and responsive again.

<comment><USER44> I used to have this problem, but then I configured it to run scans overnight.

</comment></comment></comment><comment><USER45> I am not surprised in the least. Norton Antivirus is one of the worst of its kind. I've used it for many years. Every single virus/trojan/adware infection I got went straight through Norton Antivirus without it doing _anything_. Back as a kid I opened a lot of downloaded executables, like games, and some of them were infected. Later, I got more cautious with executables but got rid of all antivirus software - best software decision ever. My computers have never been faster.

</comment><comment><USER46> > googleprojectzero.blogspot.my

Why is this linked to on a .my domain? Is this an official mirror, or is there something sketchy going on here?

<comment><USER4> Blogspot, for reasons that defy logic, redirects visitors to "localized" domains (I assume via Geo-IP). OP is probably from Malaysia.

Not only is it incredibly annoying and breaks things like URL deduplication, it also leaks your origin when you share links. What a great feature!

<comment> tushar-r >Blogspot, for reasons that defy logic, redirects visitors to "localized" domains (I assume via Geo-IP). OP is probably from Malaysia.

Because a whole bunch of countries keep wanting to block stuff. So:

[https://support.google.com/blogger/answer/2402711?hl=en](https://support.google.com/blogger/answer/2402711?hl=en)

If we receive a request to remove content that violates local law, that content may no longer be available to readers on local domains where those laws apply. Note: Country-specific domains is not a different blog address, but a domain redirect based on the country where you're currently located.

<comment><USER4> I'm having difficulties understanding this. If Google receives a request to remove content from some specific country, I would assume they are doing this based on Geo-IP rather than based on the domain users enter. It wouldn't really make sense otherwise, as the link to the blocked content would not necessarily need to use the localized domain, so this would be ineffective. It seems like an unimportant implementation detail - one that seems to do more harm than good.

<comment><USER47> It means countries that want to block certain content can block all the other blogspot.* domains. i.e. blogspot.us is a site that they promise will only ever contain content that's legal in the US, so any country-level firewall in the US should leave that site unblocked, and so on.

<comment><USER4> That still seems like an odd trade-off. The same logic should apply to things like YouTube and Google+, but they don't have a ccTLD redirect for those sites.

<comment><USER47> Blogspot is older (and/or maybe put more thought into it early on - a blogging platform is naturally a speech medium whereas YouTube may not have thought of itself in the same terms), country-level firewalls are a lot more sophisticated these days.</comment></comment></comment></comment></comment></comment></comment>


Choose Boring Technology - <AUTHOR> http://boringtechnology.club/ <comment><USER2> Always glad to see this making the rounds, as it has been influential for me, especially the perspective of "Happiness comes from shipping stuff".

I often think about this part, in particular:

>Then we walked away. We didn't do anything related to activity feeds for years after that. We barely even thought about it.

>Then one day I said, "hey, I wonder how activity feeds is doing." And I looked at it and was surprised to discover that in the time since we'd launched it, the usage had increased by a factor of 20. And as far as I could tell it was totally fine.

>The fact that 2,000% more people could be using activity feeds and we didn't even have any idea that it was happening is certainly the greatest purely technical achievement of my career.

I get a huge kick from making something, and then forgetting about it. For me, implementing something that provides value with near-zero maintenance for years is the ultimate sign that I've done well.

<comment><USER3> _For me, implementing something that provides value with near-zero maintenance for years is the ultimate sign that I 've done well._

Such a good point it's worth repeating. We have a bunch of small tools of which the code might not exactly be brilliant or adhering all possible good practices, but after 10+ years they still _just work_ without any unsurprising behavior nor bugs and still also just build/deploy with a click of the button so to speak. That's just nice.

Does make me wonder sometimes what exactly all other knowledge I gathered since then is good for. Some of it is definitely wasted on shiny stuff. But most of it is software architecture and with repect to the subject I'd translate that as 'how do I make this large scale application a combination of all those small and nice tools, and make that combination itself also work as those small nice tools'.

<comment><USER4> I've found that systems usually get robust over time - most issues in commonly run code paths get ironed out in production, and if we don't keep modifying the code, and the system's relative external world remains stable, things chug along. It is the greatest feeling though!

We have a DOS application (inventory, accounts etc.) that has been in operation at a couple of retail stores for about 15 years now. No major change after the second year.

This transiency of our work as programmers is something I am yet to come to terms with. I can't recollect where, but there was an interesting consolation in that while the code might be transient, the value created was real, and that's what matters. But the code is an artifact of our waking hours; it matters, like memories matter.

<comment><USER5> You were still deploying a DOS application in 2004 ? Do you mind telling us more about the reasons why ?

<comment><USER4> That was the only programming environment (CA-Clipper) I knew at the time. VB6 was a thing, but I had some GUI-antipathy, coming from the older, "purer?" world of DOS. Also POS terminals needed first-class keyboard accessibility. SQL wasn't welcome either - you had to give up the fine grained control of row-level cursors in the flat-file xBase database and use a rather unwieldy looking language.

I would still love to go back to the ergonomics of building user interfaces in DOS. No mouse, no events, no callbacks - just straight imperative programming where everything, including user input, blocks. Nothing async - even network communication was thru Novell Netware file writes, using shared locks. And a single screen to design: 25 rows and 80 columns, and just 16 colors.

After doing GUI work for many years after DOS, thru VB6, jQuery DOM manipulation, Angular etc., ReasonReact and Hooks are the closest I've come to recapturing the ease of building UIs again. I'm also looking forward to trying out concur one of these days ([https://github.com/ajnsit/concur- documentation/blob/master/R...](https://github.com/ajnsit/concur- documentation/blob/master/README.md#event-handling)) -- it is a Haskell GUI library that lets you do evented handling in an imperative-looking manner.

<comment><USER6> Well, if you would be using Turbo Vision, then there was mouse, callbacks, events, the whole stuff. And async would just be a couple of chained interrupts.

I also used CA-Clipper, Summer '87 and the OOP variants 5.x.

Most of the stuff did use a mouse and some TUI navigation, with menus and stuff, we had a couple of libraries for it.

Now doing those forms it was pretty neat.

<comment><USER7> I had a pascal turbo vision accounts app. The client decided to share the data between two machines over a 10baseT network. With the exception of an occasional, but fully recoverable write issue (the user just had to retry their last save operation), I didn't have to lift a finger to get it past y2k. Mouse, printer (serial and parallel) were no problems. Good times.

<comment><USER4> Good times indeed!

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> A huge part of our VC bubble in OSS infrastructure (NoSQL, clouds, middleware, automation) is fueled by a generation of technologists not choosing boring technology.

This is a great presentation, and great advice.

What this piece misses are the marketing, hiring practices and incentives in that capital pool that is fueling FOMO (fear of missing out) as the main driver of our technology trends.

Most people orbiting IT wants to be a part of something that changes the world: it's their ticket to higher paying job elsewhere, bragging rights at a conference, internet fame/notoriety via blog posts, etc. This is how we wind up with Service Mesh proxied MongoDB on Kubernetes as THE ANSWER, and all the ensuing political battles / turf wars to fight boring alternatives as "yesterday's news".

It would be nice to see a greater focus on business outcomes, but I find few have the patience or bandwidth to truly track the success of a technology choice, unless you're a larger company making a periodic cash bet with a business partner (eg software vendor). And even then, failure can be swept under the rug sometimes.

It requires strong leadership and management (or a great culture) to encourage your teams to communicate tradeoffs and make these sorts of "boring tech" decisions without it being more about individuals playing a turf game.

<comment><USER9> Not sure I agree with this perspective. Many of these companies are started by engineers or product managers who have seen the boring solutions not work for them. In some cases they either see it not work many times or see a future where other organizations get to that point and need a better solution. In digging into what these companies do it becomes clear that they aren't doing it just to push new tech, but maybe I'm the one who hasn't seen enough.

You mention k8s and related stuff isn't always the answer. Seems you think it's somewhat valuable? I'd like to hear what sort of tech you think is creating a bubble by being so useless! Certainly k8s can't be it, because for large companies the automation abilities of such a system are quite clear. Does it intro new problems, of course, and if they aren't well solved yet you'll have to think hard about the benefits of adopting The New Way, but to call this a bubble is hard to accept.

<comment><USER10> " _Many of these companies are started by engineers or product managers who have seen the boring solutions not work for them._ "

Most of the time (not all, but almost all of the time), if a "boring solution" doesn't work for you, the problem is _you_ , not the solution. There's a reason it is boring.

<comment><USER11> >if a "boring solution" doesn't work for you, the problem is you, not the solution

Were the people who hated their pagers and wanted cell phones idiots? All the people who were writing GUIs in C and debugging memory leaks...was the problem that they were shitty programmers, or is there something to be gained by spending more time on the business logic and less time on memory management?

It's healthy to see the shortcomings of current solutions, and to wonder if there is a better way. It's not healthy to chase shiny objects simply because they are shiny, or to massively complicate your tech stack because it helps marginally with a short-term problem.

<comment><USER12> > Were the people who hated their pagers and wanted cell phones idiots?

Tangential comment here. No they are not idiots, but I can tell you from experience (my team and I build and manage a wide area paging network supporting over 40000+ emergency services first responders over an area of 230,000 km²+) that there are still advantages to paging protocols over 3G/4G in certain circumstances.

Coverage is one of them. Sometimes the best tool for a particular job is the older one.

</comment></comment></comment></comment></comment><comment><USER13> There's a certain amount of ladder kicking involved in telling people to choose boring and beige technologies after you started up your career chasing after new and exciting shiny things.

Every developer should spend some time working at the bleeding edge, so they know how it feels to get cut. The best time is absolutely at the beginning, when you're a fresh grad and have the energy. You have the rest of your life to work on boring stuff that pays the bills, and who knows, if you are successful with a new technology early on, you might just be working on it for a very long time, continually breaking new grounds.

Choose bold technology early, then boring.

<comment><USER14> The problem is that those people who chose shiny new technology only see the benefit, while all the others in the company will rot in hell because of the stupid choice of a junior engineer (who jumped ships 3 times meanwhile). I don't see how does that benefit the company? The whole point of the article is that you should make wise technology decisions which benefits the whole organization.

<comment><USER15> > The problem is that those people who chose shiny new technology only see the > benefit, while all the others in the company will rot in hell because of the > stupid choice of a junior engineer (who jumped ships 3 times meanwhile). I > don't see how does that benefit the company? The whole point of the article > is that you should make wise technology decisions which benefits the whole > organization.

Sure, if your interests are closely aligned with the company. But as you noted, the junior engineer jumped ships 3 times already, probably with significant salary increase each time, and in part due to a CV that includes the use of exciting modern tech, which shows he's eager to learn and is passionate etc. Why would he care how the old company is doing?

People often don't stay at the same place long enough to feel the long-term consequences.

People will do whatever the (job) market rewards and is fun.

It's a bit different when you wear multiple hats. Then it may be best to focus on one specialization and keep things boring for the rest. For example at a small company, as a single-person data scientist/engineer/developer you may want to focus less on the IT fads and more on the modeling, because your CV is best padded by fancy new ML models, not by fancy new database engines. Or vice versa.

<comment><USER16> True, this puts the onus on the CTO not the junior developer who will make lots of dumb choices, not just shiny-chasing.

But there's no reason you can't work a day-job doing boring technology and do fun stuff on your side projects.

Companies should actively support your side-projects, especially if they involve R&D experimentation that could be useful in the long run to the company. Whether that's paying for you to attend educational events or any books/online video courses/supporting SaaS services/etc or giving you 10-20% of your week to do what you want.

<comment><USER17> There are too many CTOs and senior developers out there who are too busy working on their pet projects to care.

The foremost role of senior devs (at least in a company with juniors and mid- levels) should be technical oversight. Set up the architecture, set standards, and do the code reviews.

Not sit in the corner and learn React.js.

<comment><USER16> Why would anyone pay a senior developer $100-200k+ to sit in a corner and learn React.js?

I was talking about junior devs making (predictable) bad choices not senior devs. Your senior developer should know better than to shiny chase and value delivery over cute technology, otherwise he's not a senior developer.

Some startups can't afford a senior developer or simply don't know any better... but even junior can pump out working apps. Then it's just up to the next evolution round when you build a professional team instead of some cowboys.

</comment></comment></comment></comment></comment></comment><comment><USER18> Another case of someone discovering, after 10+ years in tech, that code is a liability and you're supposed to solve problems instead of chasing trends and padding the resume.

Great that he's spreading the word!

<comment><USER19> Code is not "a liability".

It is a tool that can be used masterfully or utterly abused.

Every craftsman has to invest in their tools. And for someone to truly master their craft they must sometimes hone their tool skills speculatively, without short-term gain, and by sacrificing the time and attention used for other things, like actual projects.

If everyone just obeyed their project manager and always focused 100% on the problem at hand, we would be producing sad, boring things. If you want to see what that's like look at enterprise applications.

<comment><USER20> Code absolutely _is_ a liability. It is not a tool, it's a means to an end.

A simple tool, like a hammer, is not volatile over time. The house that you build with your tools is. It will weaken and rot, and given enough time it will crumble. It takes care and maintenance to keep it safe to live in.

It is the same with code in production; at some point in time in the future, through no fault of your own, the code can become "wrong" as reality changes around it.

Code is also costly to maintain, sometimes even for small code bases. As the world changes, so must the code.

One of the definitions of "liability" in the Merriam-Webster dictionary [0] is _" a feature of someone or something that creates difficulty for achieving success"._ This definition is often met by code.

[0] - [https://www.merriam- webster.com/thesaurus/liability](https://www.merriam- webster.com/thesaurus/liability)

<comment><USER21> > It is not a tool, it's a means to an end.

You could almost call it "something (such as an instrument or apparatus) used in performing an operation."[0]

[0][https://www.merriam-webster.com/dictionary/tool](https://www.merriam- webster.com/dictionary/tool)

<comment><USER20> If you are going to use my own trick and use the same dictionary against me, at least have the common decency to also use my own words against me!

Definition 1C in the link you provided lists _" a means to an end"_ as a definition of a tool, which I explicitly mentioned code was.

So I must concur, code is a tool. But I'm adamant that it is also a liability.

</comment></comment></comment></comment></comment><comment><USER22> There's a longer term issue that appears to be missing here. At what point do you change? There must be a point otherwise we'd all be here writing COBOL/CICS with some whizzy Javascript interface library.

Over time it becomes harder and harder and more and more expensive to maintain old technology. Because frankly maintaining old technology is pretty boring and career destroying so you need to be paid more and more to do it.

The marginal benefit of the new shiny is indeed an overhead you should try and avoid, but also you have to dodge the trap of being stuck in a technology when everybody else has moved on.

Anyway back to the Adabas support...

<comment><USER23> Recipe for getting things done:

1\. Place every new & cool technology into mental quarantine for 3-5 years. 2\. If, after that: a) the tech is still widely used, and doesn't seem to be getting overtaken by something else b) you're about to start a NEW project where the tech would help c) you're not in a rush and feel like trying something new

...then go for it.

Learning complex tech that just arrived is a waste of your life, if you want to accomplish things. It's only useful if your aim is to appear knowledgeable and "in the loop" to your developer peers.

<comment><USER24> This works, but it does force you to ignore new enabling technologies that make new use cases economical, where most innovation and value creation is.

You'll be very efficient at accomplishing old use cases, which is just as well, because you'll need it - the market for them is most probably commodified, very competitive, and with low margins. Dominated by big actors with economies of scale. Not a comfortable place to be.

You'll probably get into very few new use cases, because by the time your 3-5 years quarantine passes on their enabling technology, they're already explored, if not conquered, by several competitors. The exception to this is new use cases without a new enabling technology, but those tend to ressemble fads, you'll have no tech moat, so again they'll be a very competitive, low margin market.

New techs create value only when they solve some use case more efficiently for someone. This creates value. Not all new techs do this. God knows that especially in software engineering, people deliver new techs for fun and name recognition all the time. Managers allow it as a perk, because of the tight labor market. But it's a mistake to consider all new techs to be in this later category.

New techs are also tech moats, giving you negotiating power to capture some of the created value. Without the tech moat, you better have a non-tech one (regulatory, institutional, market structure, economy of scale) because otherwise the value you capture will quickly be pushed to your marginal cost - if that - by competition.

<comment><USER25> You're talking about a tech stack, which is not the same as, for example, a modern web application. You can build a perfectly modern web app with 'old' Java/JEE stack, backed by an unsexy SQL-based database. You don't need Node.js with MongoDB.

Techstacks very very very rarely enable new use-cases. They are the equivalent of fashion statements by young developers who haven't learned what it means to support software for 10-20 years.

<comment><USER26> Do you similarly feel like frontend stacks have seen no meaningful innovation?

I think your argument works fine-ish for backends but it's bananas to suggest that jQuery is the same thing as React or Svelte. I do security for a living and maybe 100% of all jQuery sites have XSS. If I find a React page I can just grep for dangerouslySetInnerHTML and I'm 80% of the way there. (I am exaggerating, but hopefully my point is clear: from both a development perspective and a safety perspective, React is not just a shiny new version of jQuery.)

<comment><USER27> The procedural nature of jQuery just makes for buggy as hell websites as well. Manual DOM updates etc. etc.

React being 'declarative' tends to end up with more stability in regards to UX (e.g. complex search forms). Makes the integration of third-party components smoother too.

<comment><USER26> Sure! I've written large apps in it and am familiar with the technology. My point is that whatever the reason, frontend clearly has made strides. So, is it:

1\. Frontend was less mature to begin with

2\. Frontend has a unique, definitional state management problem in the DOM

3\. Actually, we can make real progress sometimes

4\. Really, frontend hasn't made strides, you're just ignoring $x

5\. Several/none of the above?

(I think real progress is possible and disillusionment with some new technologies should not prevent us from trying. But also that the DOMs unique state management problem is so overt that functional approaches almost couldn't help but dominate.)

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> Author mentions Clojure a few times as an example of shiny-thing. Ironically, it's one of the few languages that I can reliably do in the browser, on the backend (on real servers or say, Lambdas), and build native binaries for -- which addresses the later "many tools" problem better than a "boring" programming language like Ruby. Unless you don't care about frontend code I guess :-)

(Overall this talk is fine! I think the problem/technology set illustration is particularly useful. I run Latacora's Cloud practice and we own all production infrastructure and "no, more boring, boringer than that, keep going" is a good way to describe what that looks like. We deploy half a dozen projects with exactly the same zero config storage and in ECS Fargate because I'm not managing servers.)

<comment><USER28> But: \- good luck finding a Clojure programmer if your current one quits. \- good luck finding answers for your exotic bug/performance issue etc. Code is a liability, it's much more than language/VM/compiler features

<comment><USER26> I've ran multiple org that are largely or almost exclusively working in Clojure(Script) and trained over a dozen folks to be proficient Clojure programmers, and have not found that any of those to be a problem. For example: you mentioned debugging serious performance problems, but the JVM has some of the most advanced instrumentation on any platform.

This is actually a point subtly made in the talk: your real production problems are probably going to be a lot more subtle than "oh, it's Python's fault". It's "this table ends up getting VACUUMd a lot and a minor version change made the disk size required 3% bigger and that combined with natural growth and a logging bug suddenly caused a catastrophic failure meaning everything started swapping". Yes, one point is what tool you use (a shiny new graph database is likely to be fundamentally less operationally mature than Postgres) but more important is your collective expertise in your toolbox, because a production P0 incident is a bad time to learn what autovacuum_naptime is.

For example: those fancy performance debuggers probably don't work on your Lambda runtime. I don't see that as a Clojure problem, because you would've had the same thing if you wrote it in Java+Dropwizard or Flask+Python or Go (which are presumably in the boring category).

Is the flip side of that argument that you should only write things in PHP, because that is what the market has decided where programmers are the most fungible?

<comment><USER29> As a counterpoint, I have had a developer who had wrote production ClojureScript tell me it was the worst of all worlds - it doesn't abstract away issues of the DOM and yet you still have to debug what was happening in JS and translate it over to Clojure.

Another thing I noticed is that most developers who had to touch Clojure in my org all pretty much didn't like it at all.

<comment><USER30> Clojurescript on its own doesn't give you much for web programming and, not worth it over Javascript. It shines when combined with a react wrapper.

<comment><USER31> > not worth it over Javascript

It absolutely is. As someone who's dealt with JS/TS/Coffeescript and a bunch of other "scripts" that compile, transpile to Javascript for very long time, I can honestly say: Clojurescript today is the only truly production-ready viable alt-js PL.

and you don't have to use it with React. React with immutable data structures just makes sense. Once it stops making sense for any reasons, Clojurescript community will move on to using something else.

</comment></comment></comment></comment></comment></comment><comment><USER32> I'm only part way through the presentation, but I loved this line.

"And when people succumb to this instinct they tell themselves that they're giving developers freedom. And sure, it is freedom, but it's a very narrow definition of what freedom is."

Yup. You get the freedom to choose a language and/or database with unknowns but you lose the freedom to leave work at a reasonable hour to see your family, pursue a hobby, or just veg out. Experimenting with new (to your organization) languages and infrastructure pieces is something you do between projects, not during.

It reminds me of something a jazz musician said in an interview, "The audience doesn't pay to see you rehearse. Rehearsal is done on your own time away from audiences." Rehearsals are where you try new techniques, harmonies, instruments, etc.

<comment><USER33> I'm uncomfortable with this rehearsal analogy. I'd rather have a company pay me to learn new things than spend my free time coding when I could be seeing my family, pursuing a hobby, or vegging out. From my employer's perspective, I agree that boring is usually better, but as an individual, I'd rather learn on the job than having to feel the need to "rehearse" for my job.

<comment><USER32> I didn't mean no learning on the job. The time to learn is between projects. If your employer keeps people on projects constantly, then learning happens between tasks.

</comment></comment></comment><comment><USER34> If you work on a Node backend with Javascript, Where does the idea of switching to Typescript fall into this discussion? Is it a boring technology, or a shiny new technology? It is still using the same tech in Node, which you already know the benefits and pitfalls for, but it isn't like there is no overhead to start consuming Typescript if you haven't used it before.

My hope is for those who agree with the author's premise, that it is still using the same "boring" technology and more people switch to it, because over the last two years when we switched to use Typescript, it has gotten significantly more valuable as more and more people have used it because now a majority of the dependencies I take on have type definitions provided via DefinitelyTyped, or are included with the package.

<comment><USER35> It fits in the ruby part here: [http://boringtechnology.club/#33](http://boringtechnology.club/#33)

It would be something that you are adding to the stack. Yes, you are intending to replace something, but in practice there will still be legacy nodejs hanging around.

But crucially this one: [http://boringtechnology.club/#43](http://boringtechnology.club/#43)

If you are spending time (and therefore money) changing from one language to another, you are not making features for the business. Yes, you might get more speed re-implementing features, but its almost never going to make up for the hit you took in porting everything.

<comment><USER36> It ain't just about speed, though. If TypeScript is able to prevent entire classes of bugs, then that lowers the ongoing maintenance costs relative to the costs of maintaining the non-TypeScript version. It'll also make implementing new features easier and faster (and therefore cheaper) in the long run specifically because of that avoidance of bugs interfering with delivery of those features. Both of those factors can (and often do) easily outweigh the upfront cost of porting everything.

Even speed alone, though, can help substantially reduce other more tangible costs; if your TypeScript backend is faster than your non-TypeScript backend, then that translates to needing less server resources to achieve the same result (and/or being able to handle heavier loads without needing to upgrade or expand your server resources).

<comment><USER35> I see those arguments but:

o the new feature speed is only gained after a complete re-write

o in 80% of businesses people cost >> than hosting

o re-writing always introduces more bugs

The only time that I would agree to something like this, is to get all my teams onto the same platform. Again, thats not a technical decision though.

</comment></comment></comment></comment><comment><USER37> Like many developers I have a strong desire to work with more exciting technology, even when there's no business case for it. Strangely enough, I've found the best outlet for that energy (other than personal projects when I get free time) is configuring Emacs.

It's less disruptive than putting a new language or database into production, and makes me significantly more content to work with a boring stack. Plus I get a more comfortable computing environment out of it.

Maybe someday I'll get Emacs just the way I like it, and need to find something else to distract me from chasing the new and shiny, but I kind of doubt it.

<comment><USER38> I am the same way. I can tell when my main job gets boring because I start to tinker with emacs more and it provides that intellectual stimulus I need. Not to mention helping optimize away some of the more rote parts of my job.

</comment></comment><comment><USER14> > "You can't do poetry if you're worried about what you're going to eat today"

Well let me just add this:

> "Shortly after his release from the prison at Meung-sur-Loire he was > arrested, in 1462, for robbery and detained at the Châtelet in Paris. He was > freed on November 7 but was in prison the following year for his part in a > brawl in the rue de la Parcheminerie. This time he was condemned to be pendu > et etranglé ("hanged and strangled"). While under the sentence of death he > wrote his superb "Ballade des pendus," or "L'Épitaphe Villon", in which he > imagines himself hanging on the scaffold, his body rotting, and he makes a > plea to God against the "justice" of men." \- I wouldn't call that the > highest step on the pyramid

</comment><comment><USER39> I am more and more resonating with these feelings, for the good and bad. In fact, my own writing reflects that over the years: [https://joaodlf.com/](https://joaodlf.com/) , I started my blog with information regarding technologies like Spark, Cassandra, Go, etc, and I am now writing about Postgres and Python web dev and Celery.

Boring is boring but it simplifies so much. I still love new technology, but I am now much more likely to try it out extensively and privately before even dreaming of bringing it to my company. An example: A few years ago I got very excited about Go, I started introducing it at work and solved some difficult issues with it. I could not get the rest of the developers to gain an interest in the language, though. Effectively, no one else but me could touch any of that code, this is not good for business. I have now revamped that same code, using old technology, and learned a lot along the way - so much so that I now feel like this old tech behaves as nicely as the more complicated, flashy, new language implementation.

<comment><USER40> I literally had this conversation yesterday as a "why I'm not sure web development is for me, long-term."

The level of familiarity that can be achieved with a tool in the timeframe allotted before the "oh no, we're behind-the-times!" fervor strikes just doesn't seem sufficient to me. I'll have co-workers coming to me with the tool-specific roadblocks they're hitting, and have reached the point where I can easily say "yeah, I've been there, you'd never guess but the problem is with [X], just do [Y]." And just as I'm getting really efficient, I've got to throw it all out because that tool isn't cool anymore, and nobody wants to work with not-cool tools, and we've all got our resumes to worry about.

I wonder if there are some cultural changes that could help mitigate this. If there really is an endorphin rush when working with a fancy new tool, why is that there and what can we do to replace that? Is it resume-building, is it enjoyment of that stage of knowing-nothing, is it happiness when you easily do that one thing that was annoying the shit out of you with the old tool?

Can you pick apart why you were excited about and wanted to adopt Go?

</comment></comment><comment><USER10> I am in complete agreement with this. (Sing it, brother!)

But,

" _My friend Andrew wears the same brand of black shirt every day. He thinks that if he conserves the brainpower it would take to pick something to wear, he'll bank it and be able to use it later for something else. [...] I like to think about it like this. Let's say that we all get a limited number of innovation tokens to spend. [...] These represent our limited capacity to do something creative, or weird, or hard. We really don't have that many of these to allocate. Early on in a company's life, we get like maybe three. Not too many more than that._ "

I don't buy this. If you try to save up your brainpower, innovation tokens, intellectual capital, or whatever, what you'll find when you come to try to use it is that you don't have it. Sort of like if you try to save up your upper body strength for a couple of years before you try to climb El Capitan. But I am just saying that is a poor example.

<comment><USER41> The meme that you can 'save up' or 'train' your mental reserves has seen a bit of a come-back lately. However, the scientific evidence of such a thing is scant to none. In general, people that claim you can train your focus and eureka-moments are not looking at the evidence. Brett at AoM has a good intro article on it here: [https://www.artofmanliness.com/articles/motivation-over- disc...](https://www.artofmanliness.com/articles/motivation-over-discipline/)

Tl;DR: Be disciplined, not motivated. Don't worry too much as well.

<comment><USER42> It is not just about saving "brain power". It is also about saving time.

</comment></comment></comment><comment><USER43> Somewhat playing devil's advocate, but if everyone joins that club, we keep the status quo and there's no more progress. If everyone joined this club in the 50's/60's we'd still be writing assembly. It's the guys pushing new shiny things that allows our domain to go forward, we just need to accept that 90% of the shiny new things eventually turn out to be crap. It's about the other 10%.

<comment><USER44> You can argue there has been a lot of progress in web development since 2000, and you can argue it hasn't. I think we did a lot of circles around ourselves, creating new frameworks that do the same stuff as the ones before them. The real progress was in browsers and CSS which allowed things like access to device cameras/inputs, rtc technologies and adaptible layouts.

</comment></comment><comment><USER45> Discussed at the time: [https://news.ycombinator.com/item?id=9291215](https://news.ycombinator.com/item?id=9291215)

</comment><comment><USER46> The phrase "Bleeding edge" was coined for a reason...

The HN audience typically lives at the bleeding edge and assumes the world does too.

</comment><comment><USER47> That's why I keep choosing Rails for my new projects. Boring as hell. Gets the job done.

</comment><comment><USER48> I choose boring technology because it less likely to waste my time. If I choose exciting technology, I will inevitably run into a cryptic error, even if I follow "Getting Started" perfectly, and it will be either difficult or impractical to resolve properly. That happens to me all the time, and I imagine it's due to insufficient testing and developers relying too heavily on their local environment.

Better to work on something boring, but was carefully constructed over time and doesn't require 1/10 the dependencies of today's newfangled thing.

</comment><comment><USER49> I proudly used "boring" to describe a massive simplification of my work's architecture that I have been working on for the last few years. It has been a successful culling of tech down to the bare minimum (as always- still a work in progress).

Anyway my boss got so so offended and angry that I would use the word "boring". I spent the rest of the day trying to explain it and calm him down.

Anyway use caution with the word "boring". It's more toxic than nuance would suggest.

<comment><USER50> I don't think the word is the toxic one in your story.

</comment></comment><comment><USER51> It's fun to dive in and "modernize all the things" \- and certainly you can learn a lot in the process (perhaps at the expense of the business).

I don't believe that the only reasonable alternative to that is to "choose boring tech" though. Like so many things in life, it's very contextual, and this is where you need someone with a lot of background and war wounds who can discern valuable improvements from yet-another-CADT-library-rewrite. Sometimes they're hard to differentiate from an old crusty cynic though.

I'm resistant to either pole - I think there's often a 'middle way', where you can leverage proven, high-quality software (think PostgreSQL) while still benefiting from modern approaches (pragmatic functional idioms, etc). I'm old enough to remember when the industry was generally more entrenched in ossified approaches - it definitely has its downsides too!

The hard part is when a (supposedly) proven technology turns out to be a turd. Much of the job of choosing good tech appears to be using heuristics to avoid low quality tech with good marketing. I'd love to have better tools to guide that problem!

</comment><comment><USER52> Great points. I was reminded about a period of close to ten years when "my stack" was Apache Tomcat with servlets and JSP. I would handle background tasks in threads initialized in servlet init methods. For me it was a universal platform for anything I was required to develop and deploy.

I was in Chicago last month and had breakfast with an old customer (I had never met him in person even though I did a ton of work for him between 2000 and 2006). One of my tasks for him was writing a Sharepoint clone that ran for five years totally unattended by his ops team. After five years they ran out of disk space and did a quick migration to a larger server. My customer thought that in five years they had never restarted to system or rebooted the server (yikes!, no security updates?).

</comment><comment><USER53> Whenever somebody brings up this "unknown unknowns" thing, I'm reminded again why we ended up in such a shitstorm in Iraq:

Rumsfeld listed three out of the four possible combinations of "known" and "unknown" that he had his minions thinking about for him.

But, evidently, neither he nor his minions thought of or spent any time on the fourth one: unknown knowns. Those are the ones you think you know, but you're _wrong_. And that's what knawed 5 trillion dollars out of our guts (and killed hundreds of thousands of men, women, and children) over the next N years since.

It is not a new observation. Usually Mark Twain gets credit for "It ain't what you don't know that gets you, it's what you _think_ you know but ain't so."

</comment><comment><USER54> (1) Discussion from 2015: [https://news.ycombinator.com/item?id=9291215](https://news.ycombinator.com/item?id=9291215)

(2) «You can't pay people enough to carefully debug boring boilerplate code. I've tried.» (Yaron Minsky; via [1]).

"Boring" is a wrong word, I think. Choose _hassle-free_ technology that is known to work without surprises. Don't chose simplistic technology that produces busywork, even if it's rock-solid and bullet-proof. Choose a reliable technology that operates on the right level of abstraction, or closest to it.

[1]: [https://news.ycombinator.com/item?id=9161366#9161917](https://news.ycombinator.com/item?id=9161366#9161917)

</comment><comment><USER55> One thing I don't see mentioned is: why are people not honest with themselves?

We're using hyper-scalable databases, "serverless", layers upon layers of caching, microservices with dozens upon dozens of interconnects, all built out from the start for the odd chance that the thing we build will be the next unicorn startup.

But what is the reality? Many fail before the tech stack ever becomes a bottleneck and those who do not fail rarely reach the scale that requires the absurdities we use today.

A good old LAMP stack on a decent dedicated server is, honestly, way more than enough for 99.9% of startups and doesn't need highly expensive dedicated AWS and other specialists to get up and running in nearly no time.

</comment><comment><USER56> Such strongly opinionated statements should be taken with caution. In our profession context is key and this theory does not always apply in every context.

Imagine if the Netflix guys were told to stay boring and not invent new shiny things? Ofcourse not everyone is in that scale.

I do not want to use the latest shiny tech but I do not want it to be boring as well. We are confusing KISS and boring here. A company I used to work at was running a job server at 90% utilization all the time and it even worked as a jump box, this is simple and boring but its also plain stupid. Invest some time and make it proper.

Maybe when I am 50 years old I will want a boring setup, but until then no. I want to be engaged and active in what I do.

<comment><USER57> One thing is to "use boring technology" and a different thing is to "build boring products"... I can build shiny new things with boring technology.

<comment><USER56> by new shiny things in the netflix scope i meant like sidecars not the service itself

</comment></comment></comment><comment><USER58> I wanted to disagree with this article, but it's all perfectly reasonable. It makes a lot of sense and it's well written. It's also not an absolutist viewpoint, which makes it even more reasonable.

The danger I see here is that when the places I worked in chose "boring, well- tested" technologies they picked COBOL or ColdFusion. It's not true that I would be happy to ship software that worked if done using these tools. If I had to work with them again I'd shoot myself in the... face. I don't think I ever learned anything valuable from them, except maybe "sometimes the tasks you're given will be boring, and that's how life goes".

</comment><comment><USER59> This is great and I agree 100%.

The problem we faced is hiring people. We found that in order to attract talent, we had to let them use the shiny new technology. Otherwise it would be hard to attract anyone.

<comment><USER26> I think the real punchline there is just "hiring is hard", because elsewhere in the comments someone has made the argument that you shouldn't use shiny technology because it's too hard to find people (specifically, you'd be screwed if that one person left). Maybe that's a company maturity thing: fancy tech is good for attracting talent early, boring tech is good for attracting a sufficiently large ops team later.

</comment></comment><comment><USER60> Interesting… most of the motivation to use new tech is that it enables me to solve problems more efficiently. When you use older, but more established technology, its limitations still hold back the problem solving a bit. This is especially hard to take when I know there are better solutions out there. And it also costs more time & money.

There probably is a pain point somewhere, where it makes sense to switch. I did like the point about reducing the cost of new technology – that may be quite often easier said than done though.

<comment><USER61> But the pain points of older technology are known whereas those of newer tech aren't yet.

</comment></comment><comment> mkr-hn >> _" My friend Andrew wears the same brand of black shirt every day. He thinks that if he conserves the brainpower it would take to pick something to wear, he'll bank it and be able to use it later for something else."_

I do something similar, but in a way that keeps it interesting. I always pull a shirt off the right and move the hanger to the left for clean ones out of the dryer. Pants are the same. Stacked stuff always comes off the top. Clean ones go on the bottom.

I also went with Blogger in 2019 rather than fuss over blog engines. It's exportable, free, runs under your own domain, has few restrictions on front- end customization, and is unlikely to go out without warning. At worst, I defer thinking about it until later when I have a better sense of my needs.

Related to that, I started on a PHP-based static blog engine. I reasoned that PHP was made for smushing data and HTML together, so it was perfect!

[https://github.com/CondensedBrain/blogengine](https://github.com/CondensedBrain/blogengine)

But I also saw a future where I had to maintain all the bits and pieces. I really just wanted a blog, not a blogging engine. I hadn't read _Choose Boring Technology_ yet, but it affirms the thinking that led me to pick Blogger over a custom engine and other pre-made options.

</comment><comment><USER62> Dan's "re-use the technologies you already have deployed instead of adding redundant technologies" point is what resonates most with me.

> The interesting thing here is that none of the tools that you pick may be > the "right tool" for any given job. But they can still be the right choice > for the total set of jobs.

Even if that "one choice" isn't totally boring, at least you only have one thing to figure out all the kinks of and support.

</comment><comment> kyle-rb >The grim paradox of this law of software is that you should probably be using the tool that you hate the most. You hate it because you know the most about it.

I think I'm the opposite way. I tend to like technologies when I know all the quirks, e.g. CSS:

"Well of course you have to make the parent element positioned in order to get the child's absolute positioning to work, how else would you do it?"

</comment><comment><USER63> While I see the point, I also see that all "boring" technologies were new kids on the block at some moment, and often succeded at replacing previous "boring" thing. So maybe the proper advice should sound less absolute. Something like: Be cautious, and don't discard less exciting technologies, just because they are older.

<comment><USER64> That's entirely the text of the article. But your headline is a lot less punchy than his.

</comment></comment><comment><USER65> This is a fantastic piece! I've been advocating for "complexity points" for years now. MBSA (make boring sexy again!)

</comment><comment><USER66> > The failure modes of boring technology are well-understood

Well-understood failure modes can't get you to resilience and are not all that well-understood to begin with. I guess it's just a false justification people use to stick to familiar broken old technology and broken ways or maybe to avoid relying on people with rare expertise, I don't know. If things were as simple as this we would just handle all the failure modes of "boring technology" in software and get perfectly working systems. But we can't, all the unknown unknowns and not understood failure modes are still there in boring technology, and reliability is a "do backups" afterthought. Systems really have to be designed for reliability if reliability is important and currently software reliability is too cutting edge to be boring.

<comment><USER67> > I guess it's just a false justification people use to stick to familiar > broken old technology and broken ways or maybe to avoid relying on people > with rare expertise, I don't know.

This sounds a bit nose in the air to me.

> Systems really have to be designed for reliability if reliability is > important and currently software reliability is too cutting edge to be > boring.

And yet, people have posted examples of "boring" software running for over a decade.

</comment></comment><comment><USER36> I agree pretty wholeheartedly with using tried-and-true tools instead of chasing the new shiny.

I somewhat disagree, however, with the idea that a diversity of tools in use always necessarily leads to higher costs. Sure, using the right tool for each job increases the number of tools you have to maintain, but using the _wrong_ tool for a job increases the maintenance costs of that tool while also introducing an opportunity cost from the resulting inefficiencies of using that tool for something it's not designed or optimized to do.

As an analogy, you could use a screwdriver to turn screws or to pry something apart. Sure, this is cheaper than buying an actual prybar to use for prying, and you only have to have one tool in your toolbox instead of two, but this is a surefire way to break your screwdriver in a warranty-violating way.

</comment><comment><USER68> It seems like a lot of this is just the tension between what management wants to be focusing on (big picture product or business-level questions) vs what it makes sense for individual engineers to be focusing their efforts on.

As an individual contributor, it's vitally important to keep up with new technologies in order to remain employable. Maybe that isn't in fact the best strategy for your current job, but it's crucial to getting your next job.

Put another way, there are probably a lot of businesses that sensibly standardized on ColdFusion in the early 2000s. Engineers who buckled down and focused on "So what's your company trying to do?" using CFML could easily find themselves in a really bad spot--unless they were also sacrificing their home life to teach themselves something else.

<comment><USER69> If it's vitally important to keep up with new technologies in order to remain employable why are the 7 most employable programming languages (C,C++,Java,PHP,JS,Ruby,Python) at least 25 years old? The job market says the opposite - that new programming languages hardly get a look-in.

<comment><USER68> Programming languages alone aren't the only things you have to know in order to be a professional developer. It's also important to keep up with the ecosystem around that language, and industry-wide trends in application design, testing and deployment/ops.

Good luck getting a Javascript job if all you know is jQuery, and you've never touched Webpack/React/whatever else JS devs need these days. Maybe JS is an extreme case, but some version of that same phenomenon exists for all the languages you mentioned.

</comment></comment></comment><comment><USER70> I get so excited when I see a page that renders immediately with server side markup, it's a rare old treat.

</comment><comment><USER71> An older piece that I loved is Dan's point on just allowing a certain number of technology tokens in any project:

[https://mcfunley.com/choose-boring-technology](https://mcfunley.com/choose- boring-technology)

I think one big part of the mistake is using Hacker News, Reddit, conferences as a sign of what companies are actually using. Instead, I liken it to a bazaar:

[https://www.nemil.com/on-software-engineering/beware- enginee...](https://www.nemil.com/on-software-engineering/beware-engineering- media.html)

Oh, and remember marketing is not reality, even though marketers try awfully hard:

[https://www.nemil.com/mongo/3.html](https://www.nemil.com/mongo/3.html)

</comment><comment><USER72> > But anyway, despite that, I'm mostly not a tool-obsessed engineer.

I didn't think there was many of us out there ;)

I like cool new tools, but my obsession is solving the problem in the most effective manner. Too often I see people starting with the tool and then trying to shape the problem to fit the tool.

</comment><comment><USER73> Working on new hardware technology, it's really interesting for me to think about OP's call to only use "proven technology," but how do you get a technology to the proven state if everyone is only using previously proven technology? You can force it if you control a platform (e.g. stuff wireless hotspots everywhere and have wifi capability attached to new laptops), but that's pretty rare in hardware. [https://www.siliconvalleywatcher.com/intels- centrino-and-how...](https://www.siliconvalleywatcher.com/intels-centrino-and- how-it-sparked-the-wifi-hotspot-revolution/)

</comment><comment><USER74> Related: Lateral Thinking with Withered Technology [https://en.wikipedia.org/wiki/Gunpei_Yokoi](https://en.wikipedia.org/wiki/Gunpei_Yokoi)

</comment><comment><USER75> As a meta observation, I noticed this was submitted by user luu and his website also has a blog post with a similar theme ("boring languages"[1]). Therefore, I wonder if the "boring vs exciting" advice somewhat depends on the personality. I.e. if person has a tendency to prefer conservative technology, it means external advice that advocates "boring tech" will resonate with that person.

I think choosing boring technology makes sense but I'll give a contrarian view anyway.

A lot of people like new and shiny things because it's _interesting_ and _spurs the imagination_ for future work.

For example, in 1974 when the new Altair 8800 made the cover of Popular Electronics[2], an excited Paul Allen showed the magazine to a 19-year old Bill Gates. The "boring tech" advocates would tell them they're wasting their time with the "new fad of microcomputers" and they should look at boring IBM 360 mainframes instead. Those IBM machines were around since the 1960s and were more proven. The problem is that Bill and Paul _weren 't interested_ in those mainframes. The choice isn't proven/unproven. The choice was really interested/uninterested.

Similar situation with WhatsApp in 2009. Apple just released the iPhone SDK in 2008. The older tech was PalmOS, Windows Mobile CE, and Symbian. Those legacy mobile operating systems were around since the 1990s. It didn't matter to Jan Kuom that the new iPhone didn't have a track record. It was the more interesting platform to work on and he made a bet to write an app for it.

From what I read, Google's first AdWords server in 2000 was built on MySQL. MySQL was released in 1995 so using a relatively new 5-year old technology may have been more risky than picking traditional Oracle RDBMS which had been around since the late 1970s.

It's ok to choose new and unproven tech if you have a coherent thesis of why it makes sense to try it. You also have to be willing to pay the price if your bet turns out to be wrong. You can also deliberately choose boring tech in as many areas as possible so it frees up brainpower to aggressively make a bet on new unproven tech in a particular area where you think it will make a difference.

Also, your role dictates the freedom to use new and unproven tech. If you're an employee at a mature and conservative company, you'll be constrained to choose boring technology to minimize risk. (This reinforces the saying, _" Nobody ever got fired for buying IBM."_) On the other hand, if you're an entrepreneur, there's a good chance you'll need to make a calculated risk with new exciting technology that is (1) unproven, (2) has non-existent documentation, (3) does not have much tooling to make implementation easy.

At one point, even all the "boring technology" was new and interesting. In 2019, what's some new and unproven tech that we should look at?

[1] [https://danluu.com/boring-languages/](https://danluu.com/boring- languages/)

[2] [https://www.google.com/search?q=altair+8800+magazine+cover&s...](https://www.google.com/search?q=altair+8800+magazine+cover&source=lnms&tbm=isch)

<comment><USER76> > From what I read, Google's first AdWords server in 2000 was built on MySQL. > MySQL was released in 1995 so using a relatively new 5-year old technology > may have been more risky than picking traditional Oracle RDBMS which had > been around since the late 1970s.

well they could've also used postgres/ingress. but at the time it was way less used than mysql and way more conservative. mysql was probably used because in the 2000's mysql was already a huge community. way bigger than most shiny graph databases.

<comment><USER75> An ex-googler mentioned somewhere else that Oracle was the "real commercial db" in this story about the early AdWords server:

[https://web.archive.org/web/20120305152003/http://eldapo.blo...](https://web.archive.org/web/20120305152003/http://eldapo.blogspot.com/2007/05/lets- get-real-database.html)

</comment></comment></comment><comment><USER77> I'm not sure I agree with some of the examples... For instance, since Redis can replace Memcached in addition to other chores, what would it have taken for a migration strategy to eliminate an older tech in favor of the newer one?

In the end, I understand how each tech adds costs and values... however, one shouldn't be afraid to update/change things. It's too easy to be stuck on a decade and a half old tech if you are and don't at least try to keep up a little. Then the upgrade is exponentially more costly.

<comment><USER78> I don't think the point is not to migrate. The point is to discuss and make the decisions globally not at the whim of "this makes me happy" or "we have a new problem".

</comment></comment><comment><USER79> There's a big movement in the Web community to move everything to JavaScript. Js on the server side with Node, html generated by Javascript, and the latest trend CSS-in-js.

On one hand, this can be argued as centralizing on one technology that will reduce long term cost because the operational stack around it can be reused: tests, build tools, deployment. However it very much can be seen as adding another technology to manage since it's adding a shiny new library to maintain with all the bugs and kinks along the way.

</comment><comment><USER80> This is one of my favourite tech talks ever. So very, very true. I agree with everything he says. On the other hand, writing software the same way without changing is hard to do and there is the temptation to try something new for the sake of trying something new. It's very strong with me and I've learned to indulge my urge to play with shiny new toys in a way that doesn't blow the main project to kingdom come. I need to write something that will go nowhere just to get it out of my system.

</comment><comment><USER81> "The long and short of it is that you have to talk to each other.

Technology has global effects on your company, it isn't something that should be left to individual engineers. You have to figure out a way to make adding technology a conversation.

This might be underwhelming if you were expecting a big insightful secret here, but trust me, this is beyond many technology organizations."

Well said. Also, clutch french revolution references.

</comment><comment><USER82> > The grim paradox of this law of software is that you should probably be > using the tool that you hate the most. You hate it because you know the most > about it.

The presentation talks about "innovation" tokens. You only get so many to spend.

There are also pain tokens. And you must spend a threshold of them to understand the quoted statement. It's not fun, but you will be enlightened.

</comment><comment><USER6> Pretty much this, and if shinny technology is actually worthwhile it will stick around, and then it is time to bother learning it.

</comment><comment><USER83> Probably too late to contribute to the discussion, but I chose Windows Batch for the Tron project ([https://old.reddit.com/r/TronScript](https://old.reddit.com/r/TronScript)) because it's super reliable and works on every version of Windows.

</comment><comment><USER84> During the 20th century, the Mig-21 was tried and true. It was rugged and dependable. One US test pilot described it as, "Fuel it up and go!" It would fit all the qualities of "boring" tech in this article. However, as a spectator, there's really nothing boring about it.

</comment><comment><USER85> I wrote a MySQL/python sandwich ordering app 20 years ago. It's still in use- when they moved to a new site, they had to do a data update. I offered to rewrite using modern web tech but they said they were happy with it as it because it was boring and simple and got the job done.

</comment><comment><USER86> This is why I stick with Django, DRF, Postgres, Reactjs, linux vms and REST over HTTP. Once the business problems are solved we can try out new tech. With this stack I've built many web apps and there are fewer surprises.

</comment><comment><USER87> I think it's much more important to choose a minimal set of technologies, than to choose ones that are boring. If one 'shiny new' tech can replace two boring ones, I am first aboard that hype train.

</comment><comment> dynamite-ready This is an excellent article, but it's almost advocating a doctrine, when perhaps the best approach is to have no doctrine at all.

I appreciate the idea of mastering a language or technology. It is often much easier than adopting something new.

But the main reason for adopting something new, is to make doing something much easier. That 'something' is often a call to solve a new problem.

I feel you generally come to understand why you should adopt a technology, after mastering your current set of tools.

If we were to really take to this idea of maximising the tools that are already available, then we probably all still deploy applications over FTP.

I think the two most valuable take-aways for me, are the importance of discussion, and the desire to have as small a number of dependencies as possible.

<comment><USER17> > when perhaps the best approach is to have no doctrine at all

A general rule of thumb in life is you should follow rules until you understand when to break them, and despite the ageism in this industry I feel a decade under your belt of high quality experience really helps.

KISS is a great general guideline .

</comment></comment><comment><USER88> 1 million times yes. We used to have a late adopter group in Amazon for people who do not get excited about new and shiny and keep on rolling with stable and battle tested tech.

</comment><comment><USER89> McFunley is awesome! Been an inspiration since "Choose Boring Technology" was released! It's a shame not many ppl follow his advice.

</comment><comment><USER44> Yeah but its sometimes hard to figure out what are those boring technologies. What is a boring webrtc or websockets server for example?

</comment><comment><USER17> One step ahead of you! I'm a C# developer.

</comment><comment><USER90> Solutions are like referees, good ones goes unnoticed, bad ones inspire hate.

</comment><comment><USER91> ... wait, is MongoDB considered a boring technology?

... but Cassandra is "shiny"?

</comment><comment><USER92> This will be largely lost on most practitioners of software delivery...

Unfortunately.

</comment><comment><USER93> I was hoping for an article about choosing drilling equipment

</comment><comment><USER94> Nice, but too long. I'd hate to sit through all of it.

</comment><comment><USER95> This so depends on the talent of your team. If you are normal then play it safe as most of us are. But if you want to make something great then you must resort to exploration, discovery and invention.

<comment><USER26> Part of the point of the article is that you probably don't need new technology to create a new _product_.

<comment><USER95> Right, for an incremental product for sure. Slack is an incremental product but now is worth billions. Point taken but Slack is not SpaceX.

<comment><USER26> Your example of a sector where we can't use conservative technology to make progress is space travel? ;-)

But look, I see your point, but the "counterpoint" doesn't actually disagree: the point is that a lot of incremental improvements in infrastructure feel transformative, but you get to pay the full cost regardless.

</comment></comment></comment></comment><comment><USER96> Wow so serious the comments in this thread. I immediate thought of the Boring Company and how they want to un-fuck traffic in ways that governments can't. Then I got let down a lot. Boring.

</comment><comment><USER97> is there a video of this being presented somewhere?

<comment><USER2> [https://www.youtube.com/watch?v=wUmx- dlYS34](https://www.youtube.com/watch?v=wUmx-dlYS34)

</comment></comment><comment><USER98> One of the best articles have seen recently!

</comment><comment><USER99> what if the shiny new technology is optimized for lower maintenance cost relative to older technology?

</comment><comment><USER100> When reading the headline, I felt as if this was a logan/advertisement for Elon Musk's company.

<comment><USER101> Same. I blame it on the capitalization. "Choose boring technology" reads like a statement. "Choose Boring Technology" does not.

</comment></comment><comment><USER102> So many words. So few ideas.

</comment><comment><USER103> I agree with this, but from a different angle, and possibly different reasons.

I don't think we should use boring tech for sake of it. I think we should use boring ideas, and sometimes boring ideas come in new packaging that brings better tooling and community support to the table.

Single server architecture, relational databases, are an idea. Postgres is an implementation of such idea. What we understand about the failure modes of something like Postgres, are not actually specific to Postgres, but to the ideas behind it dating back to Codd himself, and most other databases that came along the way.

Same thing can be said from DBs that derived from the BigTable paper (Cassandra, Scylla, etc). Of course they have specific operational differences, but they have at least twice as much common ground. The way we understand the failure modes of these projects, is by looking at the ideas behind it (Gossip, Paxos, AP, SSTables, LSM-Trees). We understand things by stripping away their specifics, and seeing how their core ideas interact with each other. The specifics of each solution are merely tie breakers.

This is why I would never call something like Clojure or Elixir "shiny new hype technology".

Clojure is nothing but a modern implementation, backed by an excellent community, of mature ideas that have stood the test of time. Lisp itself is as old as the dinosaurs, and functional programming is not as new as people think. It has just been "rediscovered" recently, once the hype/aversion passed and we were able to look at it with sober eyes.

Elixir is also a modern implementation, with better tooling/community, of things that have been in BEAM for 20+ years. Apache's Samza/Storm for stateful stream processing are reimplementations of things the Erlang community has been able to do with Mnesia since at least 1999. It's also something we sometimes circumvent by using things like Redis to deal with stateful computations, and Google recently seems to have realized this and published a recent paper, trying to go back to that old idea[1].

In short, I think it's a really good thing that we have new tech that tries to make the old ideas better, and I generally feel pretty safe experimenting with those. You just have to learn to recognize these values, and adequately investigate solution candidates before jumping into what could very well be just hyped up crap, because sometimes, it isn't.

[1] [https://ai.google/research/pubs/pub48030](https://ai.google/research/pubs/pub48030) discussion [https://news.ycombinator.com/item?id=19823022](https://news.ycombinator.com/item?id=19823022)

</comment><comment><USER104> Not having the site https-enabled is also a tenet of boring technology? /s

</comment><comment><USER105> Thanks for the advice dad! It all makes sense now!

</comment><comment><USER106> If you want your business to succeed, you need to have multiple edges against the competition. Simply using what other people are using isn't going to give you a technological edge.

Look at Paul Graham with Viaweb. He used lisp because it gave him a significant advantage against other players stuck with C++. Or Jane Street, started by three ex-Susquehanna guys using OCaml.

There's a lot of technology that isn't boring that also isn't unstable either. These are technologies that probably should be more boring at this point (Common Lisp, Haskell, OCaml, Scala) but are rock solid and stable. These are the kind of technologies you should be using.

On the otherhand, if all your "tech" start-up is doing is running a CRUD website, it probably doesn't matter what you're using.

</comment><comment><USER107> Everything you think is boring was new at some point.

Ultimately, shiny is what other people like, boring is what you like. What other people like is complicated, what you like is simple. Et cetera

</comment><comment><USER108> (2018)

</comment><comment><USER109> Is this about BoringSSL or The Boring Company? There's so many new and exciting boring technologies to follow that it's hard to keep track.</comment>