Dear Client, Here's Why That Change Took So Long - <AUTHOR> https://www.simplethread.com/dear-client-heres-why-that-change-took-so-long/ <comment><USER3> What gets me is that software has been a mainstay of modern business for _at least_ 30 years. And this whole time, every single professional software developer has been telling every single non-software developer the exact same thing, over and over: this takes longer to do than you think. If, say, 80% of developers were knocking things out, problem free, in an hour or two and the other 20% were hanging back like a 50's union boss saying, "yeah, that's going to be an all-day job easy", then maybe I could understand why they STILL think we're lying. Even if 20% of the devs could get things done in the time they seem to think it takes and the other 80% were hemming and hawing I could still understand this perspective. But that's not the ratio. 0% of devs can reliably complete tasks in the time that MBA's seem to think it should take and 100% of devs take longer than they "wish" it would take and THEY STILL AREN'T PAYING ATTENTION.

<comment><USER4> On the flip side, I've seen far too many teams move at the speed of molasses, for reasons unrelated to the intrinsic complexity of the problem. Bureaucracy, analysis paralysis, no automated testing, accidental complexity, tech debt, poor retention of experienced developers, poor compensation resulting in sub- par hires, insufficient training and mentoring for new hires etc etc. I wouldn't be so quick to assume that every single development team is operating at their most ideal.

<comment><USER5> I'm dealing with this right now. A team that is downright arrogant about reducing their well-padded schedule, taking _months_ to do a job that should take weeks, at most. Getting them to do _anything_ is like pulling teeth, and accompanied with lots of arrogant lectures about how hard it is to estimate, how you can't rush quality, and so on.

It's mostly bullshit. This team has historical reasons for their bloated schedules, but at root, they've simply been coddled, and never forced to justify their behavior.

I've been on both sides of the table now. Developers like to gripe and moan about "unrealistic" schedules, but without aggressive pushback from management, a huge number of programmers will simply never ship.

<comment><USER6> Sounds like a hiring problem. Start hiring some new devs and keep them separate so as to not contaminate them. As they come up to speed, let one of the other devs go. Wash, rinse and repeat.

Note: I'm a dev and I can tell you I very much dislike working with prima donnas.

<comment><USER7> Oh yes, keep them separate from the people who know the application, how it works, what it integrated with, all the codes, all the issues. Brilliant!

<comment><USER6> Right!?

</comment></comment></comment></comment></comment></comment><comment><USER8> The idea is sound, but this letter is truly surprising to me. Instead of a succinct summary of what goes into the change, we get a long-winded narrative that reads to me as full of excuses. I think a more useful email would be a breakdown of the time actually spent, and a proposal for improvement, something closer to this (obviously sent after the feature has been deployed, and obviously including additional items for documentation, code review, whatever else takes time in your development process):

      Summary
       - 0.5 hours investigation and planning
       - 0.5 hours feature development
       - 4.5 hours manually testing
       - 0.5 hours deployment

      Proposal
       - 8 additional hours now adding automated testing

Obviously, depending on your client, they may prefer a more verbose format or need some more explanation, but probably 3-6 sentences at most.

<comment><USER9> Do you think a housing contractor is "making excuses" when they apply the rationale requesting to take down a wall in the house by saying: 1) we need to investigate the condition 2) check if its load bearing 3) check if it runs electrical or plumbing 4) check if it is to code

No, they're telling you the rationale of what goes into making a change. I agree that this isn't something you should send to clients. But, even sending a summary to someone who invokes this question will ask the same thing for any item you summarize. The next question will be, "Why does it take 4.5 hours to manually test?"

So, explaining the rationale, at least once, will help the customer understand the process and could contest specific elements rather than trying to hide it in vagueness.

<comment><USER8> Really, I don't think that the developer is "making excuses" here. Which makes it all the more unfortunate that the letter as written _sounds_ like excuses.

> The next question will be, "Why does it take 4.5 hours to manually test?"

I'm not sure it will. In my experience, clients are usually surprised about how long something takes not because they think specific tasks are taking too long, but because they aren't really aware of what the tasks are nor of their tradeoffs.

"You're right, it took longer than expected. I chose to spent some time setting up automated tests. That took extra time now, but will save much more time later."

</comment></comment></comment><comment><USER10> No client would ever read this, and much of the language in it wouldn't make sense to the kind of client that most needs the explanation.

You're better off avoiding getting to this point in the first place. Maintain a good relationship with cooperative clients and they won't (usually) complain, because they value your work. You should fire uncooperative clients and let them be someone else's headache, when possible.

Adding a little more detail to line items in invoices helps a lot too. "Fix bug report #493" should usually be, "Investigate report of incorrect discount calculation (#493), modify 1 file, review code, deploy to test, test for regressions (all tests passed), deploy to production."

It seems dumb and repetitive to us, but one of those descriptions looks like 4 hours' work to the person approving the invoice, and the other doesn't.

</comment><comment><USER11> I worked in software development within companies for 20+ years. The "why does it take so long" conversation has come up a lot.

So on one hand, I see the argument. Simply opening unknown code, and making a change no matter how small, is a risky game. You need to research the impacts, test, and walk slowly through a deployment you haven'y done in ages. I totally get that.

But I also see the other side. Why DOES it take 8 hours to do a simple one- line code change? That's ridiculous. Somehow, we've developed these fragile systems. We've trapped ourselves in processes that add 7 hours to any change we make, no matter how small.

The status quo does not need to remain. It should be easier to make small changes. It should be cheaper to respond to simple requests. The client is actually right to question us when they just want an email to appear a day earlier and it costs them $1500.

<comment><USER12> I'm not sure what companies you've worked with, but the answers to your questions are very easy and come with experience. There's a universe of testable things. You'll never have 100% coverage (this is a software development law). Even if you were to do everything right, and build incredible automated deployment and testing systems, at best you'll end up reducing but never entirely eliminating fragility. To even get to that level of perfection would require a tremendous amount of overhead and time that most enterprises don't want to invest in the systems they build.

It's akin to trying to constantly remodel/add additions to a building. You may decide that you want new floors, but when you tear up the carpet you realize there's tons of water damage that was being covered.

</comment></comment><comment><USER13> It seems like most of the comments here either didn't read or missed the point of the article. Yes, this is a simple change. Yes, pushing a fix out in a day is actually pretty fast. But the author using a simple example to illustrate a point. Better than taking a complex example that takes 5 paragraphs to explain. Moreover you can easily extrapolate all the parameters in the article for complex examples.

The primary point of the article is that there are very common inefficiencies in our industry that, if we tackled responsibly, would greatly reduce the turn around time of producing changes to the code base.

The point I took away the most from is how much having a single point of change for a single concept and periodically cleaning the code base to keep it this way can dramatically increase productivity.

<comment><USER14> I read the whole thing, and if the author was trying to use this as a simple example to illustrator a more general point, they did not communicate that effectively.

<comment><USER15> I agree, I thought it both overly long-winded and unclear. If I were the guy he was addressing I'd suspect he was flannelling me.

<comment> munk-a There were a bunch of corollary questions about user scenarios that came up when I was trying to read it. I think it was a poor way to write it out - <USER2> through an actual real world issue, touch on the problems you foresaw and those you missed and the costs all of those totaled to, it's far more interesting.

</comment></comment></comment></comment><comment><USER16> When the customer demands a written explanation of exactly why a small change took 4 hours, an engineer has to investigate that completely and write up an analysis in language and sufficient detail for the customer to understand. The given letter would take me a couple hours to revise and get the wording just right and diplomatic. Since the customer is challenging the billing and there is the subtext of fraud, the letter needs to be approved by management and reviewed by legal. The engineer and others are also taken off other tasks to work on this letter project. Providing a legally sound and technically accurate letter of this nature likely costs around $1000 to provide, plus lost opportunity cost.

<comment><USER17> Exactly! When requests like this start popping in, it signals some ongoing distrust or latent disagreement between devs and the client.

If I were that client, a response of that kind would rather infuriate me as mudding the "clear" picture I see.

I believe, in this case, there's a bigger issue to address - the issue of trust and responsibilities. If client's expectations about how things should be done overpower his understanding of what the devs do, then projects/contract assessments probably missed the target audience. To realign such expectations the team needs more than just a bark back email.

</comment></comment><comment><USER18> I do not think that is a good email for discussing a single, small feature. No client is going to want to read it. It's too long and detailed (1855 words!) and will probably only _increase_ their frustration.

It is a good idea to discuss with stakeholders, at a high level, why things _seem_ to take long, but I find that works best as a conversation. No one wants a tome like this in their inbox.

<comment><USER19> To speak popular manager jargon.

There are some 2D conversations that are necessary. Dates, times, places. That sort of thing.

This is solidly 3D communication. It needs to absolutely be face to face. Otherwise the receiving person on that e-mail will (a) not read it, and (b) take it as a passive aggressive way to get out of work.

Oh, and (c) look at the length and complete unnecessary detail of that email that should be a meeting as another reason to accuse you of wasting time.

</comment></comment><comment><USER14> I'm not a dev, but this feels like arguing with a strawman. There are really customers who complain that a change to their codebase took _a single day_ to implement? I would be utterly thrilled if I could get a vendor to turn around anything that quickly

<comment><USER20> I've worked in a web agency that billed by the hour.

Clients would ask for justification on why something would take 5h of support instead of 2.5h because the previous similar request was done in half the time during the development of the app.

What they don't understand[1] is that if they ask for an _urgent_ fix after deployment, they're going to pay for a developer that had free time this very moment to pull and install the project. That dev then need to read and understand the codebase, make the fix, test it, make a pull request, ask for someone else's billable time to approve the pull requests, merge any conflicts, schedule a release, push the release, test again and document the new feature. And that's if the request is in fact as simple as it look like.

Often, requests look very similar but are very different to each other. "Why does it takes 3 hours to add a button on the sidebar while it takes 1 hour to do it in the content of the FAQ page?" Well, one is done by the content team and require no code or release. The other needs to be done in the code. Etc.

[1] They are told in very simple words that this is how it works.

<comment><USER21> > Often, requests look very similar but are very different to each other. "Why > does it takes 3 hours to add a button on the sidebar while it takes 1 hour > to do it in the content of the FAQ page?"

There is a good XKCD on this topic: [https://xkcd.com/1425/](https://xkcd.com/1425/)

</comment></comment></comment><comment> munk-a This simple change and the recommended fix has another issue that this post missed or dismissed as a non-issue. What if the deadline is pushed back after the email reminder is sent? Do some users rely entirely on the email reminders and might they need a followup email saying "Just kidding, actually you've got more time" \- assuming you resolve that question, when the due date comes up again should you send another email? What should happen (with these daily emails) if a task is pushed back an hour without pushing it out of the 24 hr bound? What should happen if it does push it out of the 24 hr bound but just by a little bit? Should we encode some sort of grace period of ignoring the change? If a due date is scheduled and due to go out immediately should we allow a grace period for canceling it?

The technical questions behind why a change takes a long time are legion, but so are the UX changes that all need to be accounted for. In software there is no intelligent actor that can solve edge cases you missed when they come up, instead you're building a system that will handle all of those cases (maybe by occasionally crashing, granted) so user stories need to be vetted and resolved.

<comment><USER22> The post covered that, tho not the specific details you mention.

<comment> munk-a That's why I mentioned it, actually taking the time to see what a change can effect itself takes time, and if changes are pushed through in an hour then no one is stopping to think about what systems you might be breaking.

</comment></comment></comment><comment><USER23> Too much protest here. I feel you'd be eventually judged on those numbers as well. Are you a time & materials shop? The fact is that you either start a clock when you start working on something and stop it when it's done or you have standard times that things should take (like an auto shop does.) If you're working efficiently, you can tell your client that you work efficiently and this is simply how long the change took. If you're procrastinating and charging the client for the time, you should fix that.

<comment><USER24> I don't think four hours is unreasonable for any type of shop (with 4-8 being CYA estimating). One or more resources likely have to do most of the steps below.

* Document the customer request

* Investigate the customer request

* Create and document the requirement

* Assign the task

* Switch current context

* Deploy a dev environment

* Make the change

* Submit the change to QC

* Test the change

* Deploy the change

* Clean up dev environment

* Document the change

* Bill the customer

Ideally, you would encourage the customer to bundle multiple requests, so that most of the tasks could be shared among them. However, if they insist that they need it now, then they need to pay accordingly.

This story seems like it could be solved with improved up-front communication.

<comment><USER23> Totally agree it's a reasonable time. I think that it's a waste of time to argue why it takes time to do your work. You'd only write that letter because you feel bad about charging your client for your time.

A better letter if you wanted to write one would be about the benefits you provide and the completeness of your work. I wouldn't even mention the time it takes.

<comment><USER25> If you're charging T&M money, and the client comes back with "why does it cost that much?", what would you suggest? Assuming burning the bridge isn't an option.

<comment> munk-a Itemize the bill, break it down relentlessly into tiny components and show how long different facets tied to the feature build took, ideally pin some of the time on them if it's appropriate ("Remember when you guys were a bit wobbly on the background color, well we had quite a few meetings and dev was tied up while we were talking, it ended up taking us about three hours between those two hour meetings and then an additional thirty minutes to redo the change three times - thankfully we held off on testing until we'd all gotten on the same page.")

The more explanation you get out there the better, if you're honest (or a good faker) people will be reassured by your explanations - but accept that not all clients are profitable ventures, a customer who can't pay for the time it'll take isn't one you should take on, just try and leave things in an amiable state.

<comment><USER25> > Totally agree it's a reasonable time. I think that it's a waste of time to > argue why it takes time to do your work. You'd only write that letter > because you feel bad about charging your client for your time. > A better > letter if you wanted to write one would be about the benefits you provide > and the completeness of your work. I wouldn't even mention the time it > takes.

I understand your response, but the (grand?) parent seems to be indicating you don't do that, and just explain that you do good work.

</comment></comment></comment></comment></comment></comment><comment><USER26> My favorite idea here is how it's a developer's job to articulate the issues with a client's code base to the clients themselves. Some clients just want the problem to go away, but smart ones recognize the importance of a high- level understanding of the state of their code base.

If you work for a cheapskate who's just nickel and diming about billable hours, the "Why does this take so long" question is nothing more than the rhetorical whining of a bad client. But in a healthy client-developer relationship, the question is important. It's a less-articulated, unfamiliar version of the following, which any good developer would find amiable:

"I, as someone invested in the success of my company, am seeking insight into a part of that success that only you have. I trust that you're not playing Minesweeper on our dime, and yet our current process isn't delivering the returns we expect. As someone who knows better than I do, are my expectations unreasonable and in need of adjustment, or are there investments that can be made to get us there?

</comment><comment><USER27> I believe I understand what the author of this post is trying to accomplish, but for virtually every "client" I have ever worked for, their eyes would have glazed over by the time they'd read 25% of this response (if they could even get that far).

It is worth it to try to help your client understand why seemingly simple changes take a long time to implement. I'd just be surprised if the written response in this post would be of much help to most clients.

The face-to-face has a much better shot at being helpful, but, of course, that depends a great deal on the client also.

</comment><comment><USER28> Part 4: learn to bill by the day and never again have to worry about justifying how long a simple-sounding code change took to make.

<comment><USER25> So would you bundle up a series of these into a day? Or just assume the client is cool with this taking a day?

I believe you're also assuming you're servicing a client whose got a lot of budget to blow on a lot of small changes.

<comment><USER29> Servicing a client with a very tight budget is a colossal pain. It might be inevitable early in your career but try to get past it as soon as you can. If your client has more time than money they will waste a lot of your time quibbling over price.

The best clients have more money than time and will pay you well to make their problems disappear. They pay you to think about the software for them, so they can spend their valuable time thinking about the business instead.

<comment><USER25> Yeah, I need to develop comfort enough to become a contractor - at the moment as an employee it means you go wherever they tell you.

</comment></comment></comment></comment><comment><USER30> "Of course, you can't just deploy a change to production without, at least, running it locally or on a test server to make sure the code executes correctly"

Dear client of course we can, just prepare for commits in history: "That simple fix" 6h ago "Fix the fix" 3h ago "Really fix" 1,5h ago "Really really fix the fix" 10min ago

If all goes well enough, we won't have to craft "Customer data fix" SQL query commit the next day, just because someone forgot about that totally hidden stored procedure that was not included in the "Really fix". Which takes another 6 hours to prepare on next business day and invoice instead of being 3-4h of proper fix for "one liner" turns into bill for 12 hours and 2 days of lost production time.

Yours, humble developer.

</comment><comment><USER31> To me the main concern is that this shouldn't even be a _code_ change: this stuff should be a matter of configuration. In a well-written system the dev time for it should be zero.

But in general, the whole letter feels wrong: either the complications of such a trivial change depend entirely on legacy code that isn't responsibility of the current maintainer, and therefore the client knows well how hard and expensive even simple changes are (and this should have been made clear when the project started); or the maintainer is asking the client to pay for developments (refactorings, tests, or a different, more general implementation of the feature) that weren't agreed beforehand.

<comment><USER22> I've actually come around to the opposite conclusion over the course of my career: everything should be hard-coded by default (tho using a well-named variable/constant/identifier). Making data-driven (e.g. configuration data driven) code is always more work and, unless you have strong evidence otherwise, [You Aren't Gonna Need It]([https://martinfowler.com/bliki/Yagni.html](https://martinfowler.com/bliki/Yagni.html)).

</comment></comment><comment><USER32> > Changing the tasks due email to be delivered a day earlier should be a one- > line change. How could that take 4-8 hours?

This is what's in dispute? Something else in the relationship is wrong.

</comment><comment><USER33> On the flip side, when a change _doesn 't_ take long to implement because of a successful campaign waged against technical debt, it's important to communicate that to the stakeholder too, even though they won't ask. I implemented such a change on Friday afternoon and made sure to talk about why it went well at sprint review and at sprint retro.

</comment><comment><USER34> Our internal model of the world is always simpler than the world itself (otherwise it wouldn't be a model).

This whole article is just saying "Dear client, here's why your model is insufficient".

Which is a much nicer email to write than "Dear client, here's why my model was insufficient". Which is what happens when the estimates go south

<comment><USER8> I'd almost argue that, in the example, the clients model wasn't insufficient. If you want to include "implement automated testing" in a small feature request without telling your clien, then your client is probably right to be surprised!

<comment><USER35> Perhaps I misread it, but I think the only testing being included was for the specific change. The bit at the end was "if we had the time to do a lot of general refactoring and testing not tied to the particular feature being done right now it would save time in the future".

</comment></comment></comment><comment><USER36> Software development is magic to most people. You have the same laptop as them, and somehow you make stuff that works and makes them money by flexing your fingers.

They don't know how to internalize that - they are dealing with a magician that turns dirt into gold and gets paid by the hour.

It's totally irrational. If you were a plumber then there would be a physical manifestation of the work.

Software managers who have never written code are also often suspect, in my opinion. They can be professionals of magic management without understanding, liking, or even appreciating their field.

</comment><comment><USER37> I'm a non-technical founder who generally doesn't bug my team about why a change took so long but that's because they communicate with me.

Thing is, working with a developer as a non-technical team member can be a frustrating, opaque experience. Communicating progress is eye-opening for non- technical colleagues but when a programmer does not communicate, then obviously the non-technical members have no idea what's going on.

Developers can forget too, that the one small change might be holding up marketing, sales and customer support, all of whom themselves are getting flack from above about why X customer is still angry, or why the press release isn't being sent out yet etc. "Waiting on a dev" isn't an answer that reflects well on anyone.

The "Dear Client" letter wouldn't be necessary if there was more communication. It can even be automated. Here's what I see in a Slack channel with my colleagues:

github APP [8:52 PM] New branch "fix-password-recovery" was pushed by xxxx [yyyyyy] Pull request submitted by xxxx #466 Improve password recovery • Fix styling • Ensure the visitor is signed out of all sessions • Redirect to sign in instead of 404 when an old recovery link is visited

semaphore APP [9:05 PM] xxxx's build passed — d61157d Improve password recovery on fix-password-recovery

I never need to doubt xxxx when I can see the myriad small tasks, failed builds, the commits etc.

<comment><USER7> > Thing is, working with a developer as a non-technical team member can be a > frustrating, opaque experience.

And vice-versa. Imagine knowing something very well, something quite complicated. Then not only knowing how to fix that issue, but explain it to a child. Now, constantly having to handhold that child over every step, even when they don't even need to know, and it is slowing you down having to do that. And maybe not even knowing the solution, but trying different things, and having to explain each to that child.

What you have setup sounds awful. Learn the technical side, or let them get on with the job. The updates you need should be at a daily standup.

> then obviously the non-technical members have no idea what's going on.

You still don't know what is going on, you just pretend you do.

> Developers can forget too, that the one small change might be holding up > marketing, sales and customer support

Then make this clear during standup.

</comment></comment><comment><USER38> > we always include time for writing automated tests in our estimates. It can > slow down initial development, but it greatly improves the efficiency of > operating and maintaining a software system. It isn't until a system grows > that you truly start to feel the pain of not having tests, and by that point > it can be a monumental task to work tests back into the system

Not only this, but also: I've recently come to understand the idea that allocating enough time for writing unit tests and integration tests, tends to uncover design issues. If a test is taking longer to write than it seems like it should, or if a particular method or bit of functionality seems harder to test than it was to write, that is a signal that you may have a design issue!

Writing the automated tests will help you find out when you have written some code that stinks, if you know what to look for and have the time to step back and think about it. It's called "Red/Green/Refactor" for a reason, it's not "Red/Green/Red/Green/Red/Green" – if your estimates only allocate enough time to write the feature and nominally prove that it works, you're missing a critical part of the TDD pattern and you're not getting nearly as much long- term value out of your tests as you could.

If you don't write automated tests at all, it's even worse, because those design issues will only show themselves when it's time to make a change, and your bad design is now standing in the way, preventing you from iterating quickly when you really need it.

</comment><comment><USER39> > but it could provide a benefit that greatly reduces the effort to make a > similar change in the future.

Even worse, if we _don 't_ invest that time, the effort to make a similar change in the future will keep _increasing_. We need to invest that time just to keep it from getting _worse_.

</comment><comment><USER40> Software programming and maintenance is mainly a mind game. The change requestor won't see you taking out a bunch of tools to get ready and they won't see you make the changes. It's hard for them to see why changes take the time they take. Also the time it takes to modify code is relative to the codebase, your understanding of it and the changes needed. I've had situation's when I've made changes in minutes but other changes have taken days. There's no way to standardize code change. Other professions can set a time on changes because there's been plenty of past data that can be referenced.

So given this, it is reasonable for requestors to feel like a change should be immediate because it seems simple relative to the way they themselves make changes when they need to. They are trying to use their past experience to come up with the time it should take to make a small change. So many things seem easy when you don't have the right experience.

The only way to counter this is to define very specifically why it will take a day vs whatever time they think it should take to make the changes. Even then it's a hard sell since the requestor will feel that a lot of what you are doing is a waste of time which in their mind equals a waste of money since time is money in business. This is how things have been, are, and will be.

I think part of a programmers training should include how to deal with customers since we all have had to deal with this situation and will continue to do so. It's part of being a programmer for hire.

</comment><comment><USER41> I work on a project consists of tens of devs that continuously delivered applications for almost a decade, and I joined this team 3 yrs ago. At some peaks, there were almost a hundred people. We still continuously deliver new features to the project every day.

We bill by hours as a lot of people suggest in this thread. Which works pretty well. We also have flexible iteration plans, the clients can prioritize any feature that is important to them. If a feature does not worth it, it will likely stay in backlog forever.

Most things are really smooth IMO, though explaining why technical stuff is costly/mandatory is really hard to deal with. Because both of us want the project to be an ever-green project, we need technical advancement, architectural extensibilities. It's so hard to even convince myself if I considered myself a non-technical person. Why do you need to split into services? Why things are not immediately synchronized after the splitting? Why do you need a job here, and what's a job exactly?

At the end of the day, it seems when you are convincing people about things they've no idea, you really need to be trusted -- just like you kind of trust doctor, and being suspicious about witch doctors. To achieve this, it seems better first to be business-focused and solving problems to build trust and reputation, as some kind of credits.

</comment><comment><USER42> > I know that investments like this can be hard to make, precisely because > there aren't any new visible rewards

Taken individually yes, there's no new 'visible' reward.

Looking at the larger picture, say, 3 months from now, changes that used to take 3 days now take 3 hours, and there are fewer outages, reduced downtime, higher uptime, and fewer (or no) hair-on-fire-we-lost-customers issues any longer.

Establishing both short term metrics (request turnaround time) and long term metrics (uptime, data loss, security breaches, customer satisfaction, team satisfaction, employee retention, etc) will help understand justification of effort.

</comment><comment><USER43> That's an excellent set of explanations. One way to forestall most of them is the magic phrase "change order".

Anytime you get into a conversation where the customers starts off with "Can't you just..." then it's your responsibility to let them know a change order needs to be filed so you can estimate the new impacts it will have, including cost.

Some of the excellent techniques in OP's article will be employed, but "change order" terminology and workflow minimize these requirements after you've stepped them through the change order workflow once or twice.

</comment><comment><USER44> A good software system should be understandable and modified by junior developers.

That's why most of "advanced patterns" doesn't work in real world.

Simple, clean, boring code is what a productive programmers should follow.

</comment><comment><USER45> Even with 6 to 8 hours for the change, it's far too short. This is not a code change. This is a business process change. The ramafication is far greater than one line change.

</comment><comment><USER46> Nice. I also like this similar version from Microsoft:

How many Microsoft employees does it take to change a lightbulb? [https://blogs.msdn.microsoft.com/ericlippert/2003/10/28/how-...](https://blogs.msdn.microsoft.com/ericlippert/2003/10/28/how- many-microsoft-employees-does-it-take-to-change-a-lightbulb/)

</comment><comment><USER47> My last two projects have been for the military and a medical device. The example here seems so wonderfully simple (even as an example of how much has to happen on a simple change). Add in all the process/design/QA/validation documents that need to be created -> passed off -> executed -> recorded and you are at much longer than a day.

</comment><comment><USER48> Well, despite all the protestations that the letter tries to surface, some of which are legitimate -- sometimes it really is:

"We did not design our system to make simple changes simple, and are now paying the price for it in the gap of we can deliver versus what our customers intuitively believe and should expect as response time for simple changes."

</comment><comment><USER49> Mostly during my career, giving the technical explanation to the PM, they are appropriately able to pushback and get the client calm and reasonable, only when the one thing that took long that's now taking longer, that's when it gets tedious...

</comment><comment><USER50> I call the the "one little button syndrome", where a client thinks that just adding one little button to a web page should take a few minutes. They don't realize that that one little button may require a whole bunch of code to support on the backend.

</comment><comment><USER51> > Actively clean a codebase

I've worked for a lot of customers as a developer, I can count on 0 hands the number of times "actively cleaning the code base" would have approved by a stakeholder, especially if there is a budget assigned to the project.

</comment><comment><USER9> This post is amazing. I try to explain this all the time. Reminds me of a previous post on HN where people were defending bosses condescending questions of "why does X take so long?"

</comment><comment><USER52> Part of the issue is just the siloed nature of many businesses. The business team has no idea what engineers actually do. Marketing just thinks it's like editing a complex word document. The clients you're working with have never interacted deeply with their own internal dev team. Their interface is always a customer facing position.

Getting engineering to understand the business side and the business side to understand how engineering works would go far in clearing up many misconceptions and allowing things such as tech debt paydown, cleanups and refactors, etc.

</comment><comment><USER53> While some of that may be true, it feels ridiculous/inappropriate as a response. The customer doesn't want a disertation about programming/support practices. The customer wants something made cheaper.

That can be satisfied to a large degree. You can just as well tell the customer: "OK, I'll do the obvious chnage, if something else breaks, we'll fix it later. It seems likely all will be fine. Ok?"

Some customers will be ok with that. And you'll bill for 30mins or whatever and everyone's happy.

<comment><USER54> This doesn't make sense. If you're selling a product, the quality of your product shouldn't be customized/determined by your customer. Your customer should pay you to get the product with the described quality. If you never test your code and do 4-8 our changes in 30mins, in a year your codebase will be write-only mess.

<comment><USER53> I'm not selling a product. I'm selling a service. And cost of that is determined by the service level.

At the "I change it you test it" level, it can be pretty cheap and fast.

</comment></comment></comment><comment><USER55> Good lord, tl,dr. And I'm a dev. Client sure as hell wouldn't read all that. And I know this because at the start of my career I wrote those kind of long- winded explanations before I realized that nobody was reading them.

Brevity is a virtue.

"Dear client, the change itself took very little time but making sure that this change did not have unintended ripple effects took a lot more. We are careful and deliberate when deploying changes to your production application because in this mission critical application, downtime and errors are not acceptable. If you'd like more detail I can provide it during our next regularly scheduled catch-up."

</comment><comment><USER56> > "In other words, spending 1/2 day to a full day (4-8hrs) on this task is > roughly 2x-4x the effort..."

My personal rule of thumb is: It either takes 1 minute, 1 hour, 2 hours, 4 hours, or 8 hours. Or obviously longer, but for the back of the envelope type of things these tiers work.

There's no 3 hours or 5 hours. Nothing ever take 3 or 5 hours. Perhaps sometimes 6 but at that point - due to distractions and side fires - it's a full day. Call it 8 and stop kidding yourself.

</comment><comment><USER57> Dear developers and other readers (megous, jmilloy, thaumaturgy),

This is probably not a real email to a client or meant to sound like one. Stop responding to it like that's what this is (and perhaps also, remember that satire and social criticism exist). It IS meant to highlight one of the common problems of software development for interested readers.

</comment><comment><USER58> How many Microsoft employees does it take to change a lightbulb?

[https://blogs.msdn.microsoft.com/ericlippert/2003/10/28/how-...](https://blogs.msdn.microsoft.com/ericlippert/2003/10/28/how- many-microsoft-employees-does-it-take-to-change-a-lightbulb/)

</comment><comment><USER59> Just send them a git diff ...

</comment><comment><USER60> Chances are, it would had taken me 4 hours to compile that response alone.

</comment><comment><USER61> It takes at least 2-3 hours just to write that letter to the client ...

</comment><comment><USER62> Quick survey for folks:

How long does it take to push a spelling fix out to customers?</comment>


Inside Google Spanner, the Largest Single Database on Earth - <AUTHOR> http://www.wired.com/wiredenterprise/2012/11/google-spanner-time/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Top+Stories%29 <comment><USER2> What kind of accuracy exists between servers inside the same data center? I assume there are some internal delays (OS stacks, switches, etc) when synchronizing time inside a server group.

I mean, even if you had a picosecond accurate clock available for use inside a server farm, you would still need a way to query it with a known (not necessarily zero; just known) latency to synchronize several machines. Servers are not known latency machines (unless specialized hardware is involved).

How is that accomplished?

And what happens when two transactions happen below the system accuracy limit? (like two transactions pertaining to the same data, 20ns apart, in different servers; impossible to order).

Surely they have solved this, I just wonder how.

<comment><USER3> I'd be worried about much more than 20ns. Someone email the ntp list not to long ago asking about getting nano-second resolution time, and one of the responses pointed out that light travels about a foot in a second. So do you want the time on this side of the room, or that side?

<comment><USER4> A foot a second is off by several orders of magnitude. It's closer to 300 thousand km per second in vacuum, and around 200 thousand km per second in fiber.

<comment><USER5> He obviously means a foot in a nanosecond.

</comment></comment></comment></comment><comment><USER6> It's interesting to see that the creators of BigTable and the early proponents of eventual consistency have invested the last 4.5 years building a system that adds back strong consistency guarantees.

If the Spanner paper is as important as BigTable, ACID may become the new goal for those building distributed systems.

Full disclosure: I'm with FoundationDB, which is a distributed NoSQL database with high performance cross-node ACID transactions. <http://www.foundationdb.com>

<comment><USER7> From what I understand of this specific project, they "solved" the consistency problem (which is quite a feat, granted).

They don't say they have an ACID NoSQL database, which is, to me, an oxymoron: ACIDity is useful if you have a powerful query language.

In the end don't you fear that you might simply reinvent SQL? Or, am I missing something?

<comment><USER8> It seems to me more like they have "solved" (or broken) the CAP theorem ... <http://en.wikipedia.org/wiki/CAP_theorem>

* Consistency (all nodes see the same data at the same time)

* Availability (a guarantee that every request receives a response about whether it was successful or failed)

* Partition tolerance (the system continues to operate despite arbitrary message loss or failure of part of the system)

According to the theorem, a distributed system can satisfy any two of these guarantees at the same time, but not all three.

<comment><USER7> The CAP theorem states that _at one point in time_ you can only have two in three.

It doesn't say much more than the obvious. Obviously, if one node parts, you're either inconsistent or available.

<comment><USER9> Just to clarify what the CAP theorem says: If one node parts, then _that node_ is either inconsistent or not available. A fault-tolerant database could potentially stay up with a single unavailable node.

</comment></comment></comment></comment></comment><comment><USER10> I had to chuckle when I read this:

"As Fikes points out, Google had to install GPS antennas on the roofs of its data centers and connect them to the hardware below."

This is usually one of the first things an enterprising sysadmin does at companies when they first start thinking about time - drop a GPS receiver on the roof (and they usually come up with a bunch of cool graphs showing where all the satellites are over time).

Soon thereafter, and a bit of reading about the NTP protocol, they realize that just adding:

      server 0.pool.ntp.org
      server 1.pool.ntp.org
      server 2.pool.ntp.org
      server 3.pool.ntp.org

to their ntp.conf is sufficient for 99.99% of all endeavors which require accurate time, outside of big physics, and, apparently Google's Spanner Database.

This part was a bit incomplete:

"Typically, data-center operators keep their servers in sync using what's called the Network Time Protocol, or NTP. This is essentially an online service that connects machines to the official atomic clocks that keep time for organizations across the world. But because it takes time to move information across a network, this method is never completely accurate,"

Much of the purpose (and math) behind the NTP protocol is to deal with network lag. And it does a pretty good job doing so.

Reading about the True Time Api at: [http://static.googleusercontent.com/external_content/untrust...](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/ko//archive/spanner- osdi2012.pdf)

"This implementation keeps uncertainty small (generally less than 10ms) by using multiple modern clock references (GPS and atomic clocks)"

So - apparently 10ms is their breakpoint - 10ms is about the limit of what you can expect out of NTP, so I guess it makes sense that if Google needs to do 10ms or better, something of their own invention would be required. Cool graph on the paper showing that 99.9% of variance across data centers thousands of kilometers apart are < 10ms deviation.

<comment><USER11> Using NTP is ok until your network connection is broken for hours. Think backhoe into the fibre.

<comment><USER10> Data Centers don't lose network connectivity - designed to have ingress and egress far enough apart that a local disaster can't take down the entire ring. Also, as I noted, one of the first things that enterprising sysadmins do is toss a Stratum-0 receiver (GPS receiver) on the roof, which feeds into the NTP protocol.

Finally - the entire purpose behind TrueTimeAPI is to sync up spanner's commits so they are consistent across the replicated/sharded nodes. Without Network, that database capability would come to a halt way before network time became a problem.

The bigger issue is the <10ms requirements. NTP over the Internet does not get you that consistently, and certainly not at the 99.9% success that Google was able to achieve with TrueTime API.

</comment></comment></comment><comment><USER12> View all pages: [http://www.wired.com/wiredenterprise/2012/11/google- spanner-...](http://www.wired.com/wiredenterprise/2012/11/google-spanner- time/all/)

Also: > "We can commit data at two different locations — say the West Coast [of the United States] and Europe — and still have some agreed upon ordering between them," Fikes says, "So, if the West Coast write happens first and then the one in Europe happens, the whole system knows that — and there's no possibility of then being viewed in a different order."

That's a large enough scale that you have to deal with relativity (light takes almost precisely 0.03 seconds to go from Palo Alto to Paris, eg). So in some sense there __is __no correct ordering. Anyone know how they deal with this? Have they just chosen some arbitrary point to make their reference frame, for purposes of ordering commits?

<comment><USER13> The entire underpinnings of GPS relies on the speed of light to calculate positions as the GPS satellites send their position, velocity and a current time stamp. These can be used to generate a distance from the satellite based on the velocity of the light. Through multiple distance readings of multiple satellites a position can be determined. Relativity does in fact play a role since the satellites are not geosynchronous they are moving in comparison to Earths reference frame. This creates non negligible errors in GPS which have been accounted for in the GPS equations. If they hadn't been we would have much more error in GPS

<comment><USER14> there are at least two relativistic effects involved in gps timings - <http://www.aapt.org/doorway/tgru/articles/ashbyarticle.pdf> \- but they would not be solved if the satellites were in geosynchronous orbit as the earth itself does not have a single _non-accelerating_ reference frame.

</comment></comment></comment><comment><USER15> If you want more detail, this article links a research paper that describes the system in detail. Very clever focussing on the timebase as a way to improve distributed consistency; I'd always assumed NTP was sufficient. [http://static.googleusercontent.com/external_content/untrust...](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/archive/spanner- osdi2012.pdf)

The part of the article that stood out to me is that Spanner is used in F1, the new backend datastore for AdWords. That's a significant vote of confidence.

</comment><comment><USER16> Interesting article. Mainly on the timing aspect though. tl;dr version here <http://tldr.io/tldrs/50b375dd52b89ec3440000df>

</comment><comment><USER17> Past and current Googlers that frequent HN are notoriously absent from this thread. Come on, guys! Surely your NDA must allow some vague commentary...

<comment><USER18> Well, I don't know of the details of Spanner, but even if I did I wouldn't dare trying the limits of that NDA. Job's too nice to lose over that :)

</comment></comment><comment><USER19> "[. . .] the company's online ad system — the system that makes its millions [. . . ]"

That is an understatement.

<comment><USER20> Is there a breakdown of their income sources? I would have guessed this is the prime income earner by a large margin.

<comment><USER19> From their 2010 10-K:

"Advertising revenues made up 97% of our revenues in 2008 and 2009, and 96% of our revenues in 2010. We derive most of our additional revenues from offering display advertising management services to advertisers, ad agencies, and publishers, as well as licensing our enterprise products, search solutions, and web search technology."

2010 Revenue: 29.321 Billion

Therefore, advertising brought in >$28 Billion in 2010 (Therefore they average $1 Million in advertising revenue ever ~19 minutes, hence the laughable understatement of the article)

Source: [http://investor.google.com/documents/20101231_google_10K.htm...](http://investor.google.com/documents/20101231_google_10K.html)

<comment><USER21> It's an expression, not an understatement.

</comment></comment></comment></comment><comment><USER22> Does this mean that Google datacenters are vulnerable to GPS jamming and/or spoofing now?

<comment><USER23> If you read the Spanner paper, there is a hierarchy of timers involved. GPS is one level, but every datacenter also has machines equipped with atomic clocks. I suspect even with GPS failure they could run entirely on atomic clocks, it would just increase their uncertainty, which would increase the commit times (Spanner is based on the idea that a transaction is committed when the time at which it was committed is guaranteed to have passed). As far as I know, it could actually run really slowly without special timing equipment.

<comment><USER24> Please someone tell me where you can buy an atomic clock :-?

<comment><USER25> Here you go: <http://www.thinksrs.com/products/PRS10.htm>

</comment></comment></comment></comment><comment><USER26> Also see: <http://news.ycombinator.com/item?id=4526710>

</comment><comment><USER27> "We can commit data at two different locations — say the West Coast [of the United States] and Europe — and still have some agreed upon ordering between them,"

I'm a bit confused by this. How will this solve the situation when the first transaction renders a second transaction forbidden. To keep it simple, say an account with only $10 and two transactions trying to withdraw $10 each.

<comment><USER28> It's been a while since I last read the paper, but I think the second transaction would fail (optimistic locking) and must be re-tried (and as such would see the updated balance).

</comment></comment><comment><USER29> Argh, "And, yes, you do need two separate types of time keepers" - No, you must establish a quorum of time keepers. Almost everyone's advice when setting up high reliability time keeping systems is to use 1 clock, or >3\. 2 is no better than 1.

</comment><comment><USER30> > VC is Google shorthand for video conference

That is the case nearly everywhere, I think :-)

</comment><comment><USER31> m( "Spanner" is the colloquial German word for voyeur. Not the best name for a database. :)

<comment><USER32> And the British use "spanner" where the Americans use "wrench." lucian1900 has it right. And because of that, you're either going to spend ages finding (or creating) words to market by, or you're going to just name the thing, get over it, and get back to work.

<comment><USER33> "Spanner" means 'foolish' or 'stupid' in the UK

<comment><USER34> Are there any words in the UK that don't mean "foolish" or "stupid"?

<comment><USER35> Yes. "fag" means cigarette.</comment></comment></comment></comment></comment>


How Hollywood's addiction to franchises changed movies in 2014 - <AUTHOR> http://grantland.com/features/2014-hollywood-blockbusters-franchises-box-office

<comment><USER2> Really, what he's lamenting is the end of big-budget original movies. At the end of the day, it's hard to argue Hollywood should invest $300m into something completely untested when Avengers will easily make $1bn.

Fortunately, while the big blockbusters cost astronomical amounts of money, the actual cost of making a movie is only going down. There are independent studios popping up, and while they aren't usually considered part of the Big 6, they do tend to invest in them(While not grossing, a ton, they still have decent margins esp. if it becomes a sleeper hit). As long as you don't demand that Hollywood spoon-feed you taste from their marketing budget, there is plenty at your local independent theater to watch.

</comment><comment><USER3> Anything with a sequence-number in its title is, by definition, not worth wasting hard-earned money on.

And don't get me started on low-quality CGI effects which continually take you 'out of narrative'.

<comment><USER4> Terminator 2?!

But its reasonable not to spent money on cliched crap for adolescent if you are not adolescent.

</comment></comment><comment><USER5> Previous discussion: [https://news.ycombinator.com/item?id=8828155](https://news.ycombinator.com/item?id=8828155)</comment>


Share and Discuss Useful Growth Resources/Exchange Ethical Growth Tips - <AUTHOR> http://growthhackers.com/

<comment><USER2> Great execution, thanks!

Is there a way to see all submissions and/or comments I've up-voted? Kind of favorites/stared itens...

Also, what's exactly the purpose of the "discussions" menu item?</comment>


How to lose $172k per second for 45 minutes (2013) - <AUTHOR> https://sweetness.hmmz.org/2013-10-22-how-to-lose-172222-a-second-for-45-minutes.html <comment><USER3> I'm amused by the tone. It's like the author doesn't realize that 99% of software development and deployment is done like this, or much much worse. Welcome to the real world.

We work in an incredibly immature industry. And trying to enforce better practices rarely works out as intended. To give one example: we rolled out mandatory code reviews on all changes. Now we have thousands of rubber-stamped "looks good to me" code reviews without any remarks.

Managers care about speed of implementation, not quality. At retrospectives, I hear unironic boasts about how many bugs were solved last sprint, instead of reflection on how those bugs were introduced in the first place.

<comment><USER4> > I'm amused by the tone. It's like the author doesn't realize that 99% of > software development and deployment is done like this, or much much worse. > Welcome to the real world.

Agree with this, a lot of developers are in a filter bubble where they stick to communities that advocate modern practices like automated testing, continuous integration, containers, gitflow, staging environments etc.

As a contractor, I get to see the internals of lots of different companies - forget practices even as basic as doing code reviews, I've seen companies not using source control with no staging environments and no local development environments where all changes are being made directly on the production server via SFTP on a basic VPS. A lot of the time there's no internal experts there that are even aware there's better ways to do things instead of it being the case they're lacking resources to make improvements.

<comment><USER5> > I've seen companies not using source control

> ...all changes are being made directly on the production servers via SFTP

I know this used to be common, but recently? Curious how often this is still the case.

<comment> frosted-flakes I have seen it recently. I did my best to change the practice before I left the company, but was mostly unsuccessful. Given that they were still running some of the spaghetti-code PHP scripts that were written in 1999 and still used PHP4 in _new_ development they were stuck in the stone ages. To give a little perspective, support for PHP4 ended in 2008, so they had almost a decade to update, but didn't.

<comment><USER6> "If it ain't broken, don't fix it". And then one day the server goes boom, the backup was incomplete, and everyone is trying to find the usb flash disk with Spinrite in it.

Meanwhile the CEO who was rejecting the €¥$£ in yh budget since 2000 is angry at everyone!

Oh the times I have seen this!!!

<comment> frosted-flakes Oh, now that you mention backups, that was a nightmare too. Thankfully, the production database was backed up daily on magnetic tape and stored offsite, but the code was generally edited live on the server, and backups consisted of adding ".bak20190402" to the end of the file. Needless to say, losing code wasn't uncommon.

This was for a 100+ year old company with millions of dollars in annual revenue that was owned by the government. So, yeah. 100% the IT director's fault, who'd been there since the early 90s.

</comment></comment></comment></comment></comment></comment><comment><USER7> discussed previously at:

[https://news.ycombinator.com/item?id=6589508](https://news.ycombinator.com/item?id=6589508)

I remember the week after this. Everyone I knew who worked at a fund was going over their code and also updating their Compliance documents covering testing and deployment of automated code.

 __As a side note __one of hte biggest ways funds tend to get in trouble from their regulators is to not follow the steps outlined in their compliance manual. Its been my experience that regulators care more that you follow the steps in your manual than those steps necessary being the best way to do something.

I came away from this thinking the worst part of this was that their system did send them errors, its just that when you deal with billions of events emailing errors just tend to get ignored as at that scale you generate so many false positives with logging.

I still don't know the best way to monitor and alert users for large distributed systems.

The other take away was that this wasn't just a software issue but a deployment issue as well. It wasn't just one root cause but a number of issues that built up to cause the issue.

1) New exchange feature going live so this is the first day you are actually running live with this feature

2) old code left in the system long after it was done being used

3) re-purposed command flag that used to call the old code, but now is used in the new code

4) only a partial deployment leaving both old and new code working together.

5) inability to quickly diagnose where the problem was

6) you are also managing client orders and have the equivalent of an SLA with them so you don't want to go nuclear and shut down everything

<comment><USER8> > I came away from this thinking the worst part of this was that their system > did send them errors, its just that when you deal with billions of events > emailing errors just tend to get ignored as at that scale you generate so > many false positives with logging.

I write apps that generate lots of logs too...I think an improvement lies in some form of automated algorithmic/machine learning (to incorporate a buzzword in your pitch) log analysis.

When I page through the log in a text editor, or watch `tail` if it's live, there's a lot of stuff that looks like

        TRACE: 2019-04-01 09:45:03 ID A1D65F19: Request 1234 initiated
        ERROR: 2019-04-01 09:45:04 ID A1D65F19: NumberFormatException: '' is not a valid number in ProfileParser, line 127
        WARN : 2019-04-01 09:45:04 ID A1D65F19: Profile incomplete, default values used
        WARN : 2019-04-01 09:45:14 ID A1D65F19: Timeout: Service did not respond within 10 seconds
        TRACE: 2019-04-01 09:45:14 ID A1D65F19: Request 1234 completed. A = 187263, B = 1.8423, C = $-85.12, T = 11.15s

Visually (or through regex), you can filter out all the "Request initiated" noise. Maybe the default value warning occurs 10% of the time, and is usually accompanied by that number format exception (which somebody should address, but it still functions, and there's other stuff to fix). But maybe the "Timeout error" hasn't been seen in weeks, and the value of C has always been positive - that is useful information!

Don't email me when there's a profile incomplete warning. Don't email me any time there's an "ERROR" entry, because that just makes people reluctant to use error level logging. Definitely don't email me when there's a unique request complete string, that's trivially different every time. But do let me know when something weird is going on!

<comment><USER9> Don't mean to sound snarky, but there are tools that do this and have been for years. If you've been grepping through logs for the last 3 years, you're doing it wrong for the cloud era.

Often times the answer is writing better alert triggers that take historical activity into account to cut down on false positives. Other times it's simply to reduce the number of alerts. In every case you need an alerting strategy that takes balances stakeholder needs, and you need to realign on that strategy quarterly. It's ultimately an operational problem, not a technical one.

Alas, back in the real world, logging is always the last thing teams have time to think about...

<comment><USER10> Care to share what types of tools do this? I'm genuinely interested. I haven't come across a log management tool that uses AI to detect abnormal conditions based on the log message contents like the OP describes. I stick to Papertrail for the most part though so I'm likely out of the loop.

<comment><USER9> I've used and really liked DataDog in the past. It has some rudimentary ML functionality for anomaly detection of certain fields, but it's only getting better.

I've also had clients in the past use Splunk with ML forecasting models that inject fields as part of the ingest pipeline. I don't know the details of that implementation; I just know how the dev teams were using it.

</comment></comment></comment></comment></comment><comment><USER11> Deployment is where the really scary bugs can happen the easiest.

I've been working on a warehouse management software (that was running on mobile barcode scanners each warehouse worker had, as he moved stuff around the warehouse and confirmed each step with the system by reading barcodes on shelves and products).

We had a test mode, running on a test database, and production mode, running on the production database, and you could switch between them in a menu during the startup.

During testing/training users were running on the test database, then we intended to switch the devices to production mode permanently, so that the startup menu wouldn't show.

A few devices weren't switched for some reason (I suspect they were lost when we did the switch and found later), and on these devices the startup menu remained active.

Users were randomly taking devices each day in the morning, and most of them knew to choose "production" when the menu was showing. Some didn't, and were choosing the first option instead.

We started getting some small inaccuracies on the production database. People were directed by the system to take 100 units of X from the shelf Y, but there was only 90 units there. We looked at the logs on the (production) database, and on the application server, but everything looked fine.

We were suspecting someone might just be stealing, but later we found examples where there was more stuff in reality on some shelves than in the system.

At that time we introduced a big change to pathfinding, and we thought the system was directing users to put products in the wrong places. Mostly we were trying to confirm that this was the cause of the bugs.

Finally we found the reason by deploying a change to the thin client software running on the mobile devices to gather log files from all the mobile devices and send to server.

<comment><USER12> I bet you had one engineer who claimed that the real problem was that the users were stupid and not that the deployment process was error prone.

I've heard about this case many times before but somehow in the other renditions they downplayed or neglected to mention that the __deployments were manual __. As this story was first explained to me, one of the servers was not getting updated code, but I was convinced by the wording that it was a configuration problem with the deployment logic.

Performing the same action X times doesn't scale. Somewhere north of 5 you start losing count. Especially if something happens and you restart the process (did I do #4 again? or am I recalling when I did it an hour ago?)

<comment><USER13> Was the deployment process of your parent post actually error-prone? From what I gathered, the developers were unaware of the lost handheld scanners. I imagine if they did they could've proactively put them out-of-service until found.

<comment><USER11> We had automatic updates in the thin clients (that's how we were able to add "logging to server" on all of them at once).

The problem was - the startup menu with testing/production choice was enabled independently of the autoupdate mechanism (separate configuration file ignored by autoupdates) for some technical reason (I think to allow a few people to test new processes while most of the warehouse works on the old version on production database).

</comment></comment></comment></comment><comment><USER14> My company's legacy system (which still does most revenue producing work) has deployment problems like this. The deployment is fully automated, but if it fails on a server it fails silently.

I rarely work on this system, but had to make an emergency change last summer. We deployed the change at around 10 pm. A number of our tests failed in a really strange way. It took several hours to determine that one of the 48 servers had the old version still. It's disk was full, so the new version rollout failed. The deployment pipeline happily reported all was well.

We got lucky in that our tests happened to land on the affected server. The results of this making past the validation process would be catastrophic. Not as catastrophic as this case I hope, but it'd be bad.

We made a couple human process changes, like telling the sysadmins to not ignore full disk warnings anymore (sigh). We also fixed the rollout script toactually report failures, but I don't actually trust it still.

<comment><USER15> Ignoring disk full condition ! really.

Handling an out of space condition should be part of your test suite - <USER2> certainly was back when I looked after a Map reduce based Billing system at BT and that was back in the day when a cluster of 17 systems was a really big thing.

<comment><USER16> I think the parent did well openly and honestly raising a personal example where missing a "basic" check caused near career changing problems - I applaud them for sharing a difficult situation.

I was concerned that it's possible to read your comment as if it was critical of the parent - was that your intention?

<comment><USER14> To clarify a little, I was neither responsible for monitoring the underlying hardware or the deployment systems in this case. I also didn't have access to fix it myself. It took me a couple hours to go from "random weird test results" to "full disk broke the deploy".

</comment></comment></comment></comment><comment><USER6> > Knight did not design these types of messages to be system alerts, and > Knight personnel generally did not review them when they were received

So they received these 90mins before they were executed, and as it so happens in many organizations, automated emails fly back and forth without anyone paying attention.

Also.. running a new trading code, and NOT have someone looking at it LIVE on the kick-off, that is simply irresponsible and reckless.

</comment><comment><USER12> I bring up this story every time someone talks about trying to do something dumb with feature toggles.

(Except I had remembered them losing $250M, not $465M, yeow)

The sad thing about this is if the engineering team had insisted on removing the old feature toggle first, deploying that code and letting it settle, and only _then_ started work on the new toggle, they may well have noticed the problem prior to turning on the flag, and it certainly would have been the case that rolling back would not have caused the catastrophic failure they saw.

Basically they were running with scissors. When I say 'no' in this sort of situation I almost always get pushback, but I also can find at least a couple people who are as insistent as I am. It's okay for your boss to be disappointed sometimes. That's always going to happen (they're always going to test boundaries to see if the team is really producing as much as they can). It's better to have disappointed bosses than ones that don't trust you.

</comment><comment><USER17> I had a chance to get familiar with deployment procedures at Knight two years after the incident. And let me tell you, they were still atrocious. It's no surprise this thing happened. In fact, what's more surprising is that it didn't happen again and again (or perhaps it did, but not on a such large scale).

Anyway, this is what the deployment looked like two years after:

* All configuration files for all production deployments were located in a single directory on an NFS mount. Literally, countless of _.ini files for hundreds of production systems in a single directory without any subdirectories (or any other structure) allowed. The_.ini files themselves were huge as it typically happens in a complex system.

* The deployment config directory was called 'today'. Yesterday's deployment snapshot was called 'yesterday'. This is as much of a revision control as they had.

* In order to change your system configuration, you'd be given write access to the 'today' directory. So naturally, you could end up wiping out all other configuration files with a single erroneous command. Stressful enough? This is not all.

* Reviewing config changes were hardly possible. You had to write a description of what you changed, but I've never seen anybody attach an actual diff of changes. Say you changed 10 files, in the absence of a VCS, manually diff'ing 10 files wasn't anybody wanted to do.

* The deployment of binaries was also manual. Binaries were on the NFS mount as well. So theoretically, you could replace your single binary and all production servers would pick it up the next day. In practice though, you'd have multiple versions of your binary, and production servers would use different versions for one reason or another. In order to update all production servers, you'd need to check which version each of the server uses and update that version of the binary.

* There wasn't anything to ensure that changes to configs and binaries are done at the same time in an atomic manner. Nothing to check if the binary uses the correct config. No config or binary version checks, no hash checks, nothing.

Now, count how many ways you can screw up. This is clearly an engineering failure. You cannot put more people or more process over this broken system to make it more reliable. On the upside, I learned more about reliable deployment and configuration by analyzing shortcomings of this system than I ever wanted to know.

</comment><comment><USER18> I realize that the consensus is that lots of companies do this kinda thing. I don't know if it's 99% - but the percentage is pretty high.

However what's neglected to mention is the risk associated with a catastrophic software error. If you are say instagram and you lose your uploaded image of what you ate for lunch, that is undesirable and inconvenient. The consequences of that risk should it come to fruition is relatively low.

On the other hand if you employee software developers that are literally the lifeblood of your business for automatic trading, you'd think that a company like that would understand the consequences of treating this "cost-center" as a critical asset rather than just a commodity.

Unfortunately you would be wrong. Nearly every developer I have ever met that has worked for a trading firm has told me that the general attitude towards nearly all it's employees that are not generating revenue as a disposable commodity. It's not just developers but also research, governance, secretarial, customer service, etc. This is a bit of a broad brush but generally the principles and traders of those aforementioned firms are arrogant and greedy and cut corners whenever possible.

In this case you'd think these people would be rational enough to know that cutting corners on your IT staff could be catastrophic. This is where you would be wrong. Most of the small/mid sized financial firms that I have had friends who worked there have told me they generally treat their staff like garbage and routinely push people out who want decent raises/bonuses, etc. These people are generally greedy and also egocentric and egomaniacal, and they believe all their employees are leaching from their yearly bonus directly.

This story is not a surprise to me in the least. What's shocking is no one in the finance industry has learned anything. Instead of looking at this story as a warning, most of the finance people hear this story and laugh at how stupid everyone else is and that this would never happen to them personally because they're so much smarter than everyone else.

<comment><USER19> >>> Instead of looking at this story as a warning, most of the finance people hear this story and laugh at how stupid everyone else is and that this would never happen to them personally because they're so much smarter than everyone else.

What if we're smarter than everyone else? When I was in big bank, we had mandatory source control, lint, unit tests, code coverage, code review, automated deployment, etc... pretty good tools actually. Not everybody is stuck in the stone age.

Even in a small trading company before that, we had most of the tooling although not as polished. Very small company with a billion dollars a month in executed trade. One could say amateur scale.

<comment><USER18> Big bank is not the same as a small/mid sized trading firm. Banks have regulations they need to meet, and typically do things by the book.

I'm not an expert here. Part of what I said is based on the 6 different people I've met who have worked in the industry. I'm just saying if you have $400+ million to lose and you rely on the IT infrastructure allows you to make that money then you can spend a few million on top notch people and processes to prevent this kind of thing. I worked at a relatively large media company, and every deployment has a go/no-go meeting where knowledgable professionals asked probing questions, you defended your decisions. I've love to know what they did in Knight Capital. The idea of re-using an existing variable for code that was out of use strikes me as a terrible idea.

</comment></comment></comment><comment><USER20> What baffles me is how they got his far into operations with such dreadful practices, 100-200k could have got them a really solid CI pipeline with rollbacks, monitoring, testing etc,

But spend 200,000 on managing 460,000,000? No way!

<comment><USER21> How would CI help in this case? It isn't even software bug, it's a process issue - they had old code running on one of out of 8 servers. The monitoring was triggered, but no action was taken.

<comment><USER22> I disagree that it isn't a software bug.

"The new RLP code also repurposed a flag" \- this is the moment when terrible software development idea was executed that resulted in all of the mess.

Of course I don't know the full context and maybe, just maybe there was a really solid reason to reuse a flag on anything.

What I observe more often is something like this though:

      1. We need a flag to change behaviour of X for use case A, let's introduce enable_a flag.
      2. We want similar behaviour change of X also for use case B, let's use the enable_a flag despite the fact the name is not a good fit now.
      3. Turns out use case B needs to be a bit different so let's introduce enable_b flag but not change the previous code so basically we need them both true to handle use case B.
      4. Turns out for use case A we need to do something more but things should stay the same for B.
      5. At this point no one really knows what enable_a and enable_b really do. Hopefully at least someone noticed that enable_a affects use case B.

If you have an use case A, create a handle_a flag. If you have a use case B create handle_b flag _even if they do exactly the same thing_ as more than likely they do exactly the same thing only for now.

What would probably be even better is separate, properly named config flags for each little behaviour change and just use all 5 of those to handle different use cases.

edit: formatting

<comment><USER23> > _If you have an use case A, create a handle_a flag. If you have a use case B > create handle_b flag even if they do exactly the same thing as more than > likely they do exactly the same thing only for now._

A hard lesson to learn, and a hard rule to push for with others who have not yet learned.

Imagine what our species could do if experience were directly and easily transferable...

<comment><USER22> Hah exactly :)

Same goes for functions, classes, React components, DB tables and everything else.

Just model it as close as possible to the real world. The world doesn't really change that often. What does is how we interpret and behave within it (logic/behaviour/appearance on top).

If you have a Label and Subheader in your app, create separate components for them. It doesn't matter that they look exactly the same now. Those are two separate things and I guarantee you more likely than not at some point they will differ.

My rule of thumb is: If it's something I can somehow name as an entity (especially in product and not tech talk) it deserves to be its own entity.

<comment><USER23> It's funny though, because my experience has led me to the exact opposite approach. Modeling based on real world understanding has been very fragile and error prone, and instead modeling as data and systems that operate on that data has been very fruitful.

</comment></comment></comment></comment></comment></comment><comment><USER24> Makes me feel less bad about rm -rf 'ing a product database and losing 1 hour of client data, the other week. Maybe I should show them this...

<comment><USER25> I would argue you shouldn't have been able to do that in your organization without bypassing (several) significant safeguards.

Did you forget a where clause while deleting data on a table, or were you actually on the production server hosting the database?

Any code you write that interacts with a database (or really any production code at all...) should be reviewed before being merged. And developers shouldn't be writing raw SQL commands on a production server. It's hard for me to see this as anything other than an organizational failure rather than your own.

EDIT: Based on the number of downvotes this has received, I can only imagine we have a lot of devs on HN who cowboy SQL in production...holy hell how can any of what I said be controversial.

<comment><USER26> While I mostly agree, many companies have a tech department of half a dozen people, and implementing and enforcing every good practice devops isn't always realistic.

That said, I'd expect at least a backup of production, then again he said he lost 1 hr of data so it was likely between backups.

<comment><USER27> If you haven't been able to invest the time to do database maintenance tasks in a safe way, at the very least enforce a 4-eyes principle and write up a checklist / script before hacking away in the production database.

I mean I get it, I've made mistakes like this as well knowing I shouldn't have (we had test and prod running on the same server, about 40K people received a test push notification). But the bigger your product gets, the less you can afford to risk losing data.

<comment><USER26> I totally agree, if feasible those steps should be done!

I was just trying to explain that many business like the one I'm at don't do business in tech (mine sells wholesale clothing), with 6 people in the tech dep, so understandably there's certain limitations on how far best practices can go. While I would usually consider it a mistake, if you thought you were just making a quick, what should be read only query, and it happens to hit some random edgecase-bug and crash a db... _Edit_ , continuing - Sure you should have tested that on the test DB first, but I'd be kinda understanding of how that happened.

Depends on the business too, if you're a startup-tech company then yea, get your -stuff- together! It's just a lot of business only need their website and some order management, their focus isn't on the tech side of things.

</comment></comment></comment></comment></comment><comment><USER28> Loosely related - this is what terrifies me about deploying to cloud services like Google which have no hard limit on monthly spend - if background jobs get stuck in an infinite loop using 100% CPU while I'm away camping, my fledgling business could be bankrupt by the time I get phone signal back.

<comment><USER29> Woah, how does Google Cloud _still_ not support budget capping?

It has budget alerting, so the capabilities are obviously there, but it's never been added. Instead, there's just a vaguely insulting guide on writing a script to catch the alert and trigger a shutdown...

<comment><USER30> Pretty sure Google cloud does support it

Pretty sure aws still doesn't

<comment><USER29> Google App Engine has spending cutoffs. Cloud allows API call cutoffs, but for actual spend it only has alerts. Their 'controlling budget' page sends you to a guide on writing your own triggers to respond to those alerts:

[https://cloud.google.com/billing/docs/how- to/budgets](https://cloud.google.com/billing/docs/how-to/budgets)

</comment></comment></comment></comment><comment><USER31> This is one of the classics of the genre. If you're interested in software reliability/failure, you should read some of COMP.RISKS .. and then stop before you get too depressed to continue.

</comment><comment><USER32> > This is probably the most painful bug report I've ever read

I suggest further reading, starting with Therac-25.

<comment><USER33> An honest bug report for the recent Boeing fuckup would be even worse. They deployed unconscionably shitty software (MCAS system) that killed a total of 346 people in two perfectly airworthy planes.

</comment></comment><comment><USER34> Totally unrelated, but the title made me think back to one of my previous roles in the broadcast industry. If you're using a satellite as part of your platform, every second that you aren't transmitting to your birds (satellites), you're losing a massive amount of money. There are always a lot of buffers and redundant circuits in those situations, but things can always go wrong.

Funny tangent- the breakroom at that job was somewhat near the base stations. Some days around lunchtime we'd have transmission interruptions. The root cause ended up being an old noisy microwave.

</comment><comment><USER35> Needs (2013) tag. As usual, human negligence is to blame.

<comment><USER36> Is there any case where human negligence is not to blame?

<comment><USER37> Pompei?

<comment><USER38> Building a city on a known lava field?

<comment><USER39> It had been "silent" for about 300 years when Pompeii was destroyed. And before that, it had mainly had small series of low-level eruptions. Basically, for much of known history at that time, it was a safe place to live.

</comment></comment></comment></comment></comment><comment><USER40> Just popping in to say i believe the Equifax hack was also do to a 'bad manual deployment' similar to this. They had a number of servers but they didnt patch one of the servers in their system. Hackers were able to find this one server with outdated and vulnerable software and took advantage of it.

I think deploys get better with time, but that initial blast of software development at a startup is insane. You literally need to do whatever it takes to get your shit running. Some of these details dont matter because initially you have no users. But if your company survives for a couple years and builds a user base, you still have the same shitty code and practices from early times.

</comment><comment><USER41> I have no sympathy for high frequency traders losing everything.

So many more interesting and meaningful uses of computing than trying to build a system to-out cheat other systems in manipulating the global market for the express interest of amplifying wealth.

<comment><USER42> A trading bot is a money-making machine and so is Facebook. What's worse, a "headless" machine that is directly manipulating buy/sell orders to feed off market inefficiencies, or a machine that lures humans in, then converts their attention and time into money?

<comment><USER43> I've thought before that transforming our soft grey matter into gold is basically what the Rosicrucian alchemists were on about.

</comment></comment></comment><comment><USER44> I watched the market go haywire on this day. Attentive people made a cool buck or two as dislocations arose.

What's crazy is that there were already rules in place to prevent stuff like this from happening - namely the Market Access Rule [https://www.sec.gov/news/press/2010/2010-210.htm](https://www.sec.gov/news/press/2010/2010-210.htm) which was in place in 2010.

When the dust settled, Knight sold the entire portfolio over via a blind portfolio bid to GS. Was a couple $bn gross portfolio. I think they made a pretty penny on this as well.

</comment><comment><USER45> >Knight relied primarily on its technology team to attempt to identify and address the SMARS problem in a live trading environment.

Ah the good old "fk it we'll do it live" approach to managing billions.

</comment><comment><USER46> Could use a [2013] tag, but this story is fascinating and horrifying and I re- read it every time it pops up. It's a textbook case of why a solid CI/CD pipeline is vital.

<comment> java-man ... and a requirements tracker linked to the code.

</comment></comment><comment><USER47> Who did they hire to develop this software?

<comment><USER48> Incidents such as these are rarely a people failure and nearly always a process failure. People will always make mistakes — perhaps seniors will make fewer mistakes than juniors, but no-one makes no mistakes.

<comment><USER49> People are responsible for making the processe, no? It's still a people problem.

<comment><USER50> Right, it's usually something like this:

A: I'd like to hire some people to improve our processes. It will take time and money and prevent future problems, but you will never notice.

B: Time and money and no new features? No way, I won't approve that.

A: _tries to sell it some more even though they are technical and not a salesperson_

B: No.</comment></comment></comment></comment>


Ask HN: How did you learn a new language as an adult? - <AUTHOR>

How did you come about learning a new language as an adult? <comment><USER2> Daily immersion in the language (articles/books/wikipedia/podcasts/newscasts/films) in addition to a decent grammar work book and a very good dictionary.

Don't bother keeping a good record of vocabulary.

You're brain will remember what you need to know, just keep doing it everyday (read/listen/watch then speak/write).

It's like exercising; a bit every day is (usually) better than a lot once a week.

Grammar drills, looking up dictionary definitions (use cases) and general exposure over a sustained period of time is by far the most realistic way to learn a language as an adult.

<comment><AUTHOR> _It's like exercising; a bit every day is (usually) better than a lot once a week._

+1.

<comment><USER3> That upwards pointing arrow is like giving the comments you think help the discussion a +1.

<comment><USER4> He was emphasizing a particular point, and I agree that point is the single most important in the entire thread.

</comment></comment></comment></comment><comment><USER5> Hey! It just so happens my start-up is dedicated to creating fun language learning apps (free in app store), if you're learning Chinese or Spanish and have an iOS device, give it a whirl: <http://321speak.com/>

I pretty much comment on every hn language learning thread hoping for feedback...

<comment><USER6> Wow, nice app! I tried the Chinese version on my 3g, I like how it has male and female voices, and the images and design was top notch. I tend to like the immersion style, it's how children learn basically. I can see this really being successful for travelers and kids, especially as software like rosetta stone is so expensive, and this seems at least on par if not better (from what I've seen so far). Bon chance! Are you planning other languages?

</comment></comment><comment><USER7> Same question was posted a few days ago:

<http://news.ycombinator.com/item?id=1603562>

</comment><comment><USER8> I found this to be an interesting perspective on language learning as an adult: [http://askakorean.blogspot.com/2010/01/koreans-english- acqui...](http://askakorean.blogspot.com/2010/01/koreans-english-acquisition- and-best.html)

</comment><comment><USER9> Had 5 years of german. I'm from Norway, and most people have 3+ years of learning a 3rd language. But I only learned enough german to make fun of it :-/

Halli hallo wie geht es so? Ich bin eine kleine HN-er.

</comment><comment><USER10> Just six days ago, this topic got a lot of discussion: <http://news.ycombinator.com/item?id=1603562>

edit: ramchip beat me to it.

</comment><comment><USER11> i am working on a browser plugin i call babelfin to help build vocabulary.</comment>


World's "Fastest" Small Web Server Released, Based On Lisp - <AUTHOR> http://tech.slashdot.org/article.pl?sid=09/05/25/1553220&from=rss

<comment><USER2> Discussion over at <http://news.ycombinator.com/item?id=625746>

<comment><AUTHOR> Ah I missed that.. Especially with that heading

<comment><USER2> No harm - no foul.</comment></comment></comment>


Minimum-wage offensive could speed arrival of robot-powered restaurants - <AUTHOR> http://www.washingtonpost.com/business/capitalbusiness/minimum-wage-offensive-could-speed-arrival-of-robot-powered-restaurants/2015/08/16/35f284ea-3f6f-11e5-8d45-d815146f81fa_story.html

<comment><USER2> If machines do take over, maybe we should have raised the minimum wage sooner. If robots can do the job better than humans, why did the restaurant owners drag their feet? Maybe they thought they could continue to exploit low wage workers, and not invest in more efficient plant and equipment?</comment>


Coming soon: day-long MongoDB training sessions in SF and NYC - <AUTHOR> http://blog.mongodb.org/post/378210035/practical-mongodb-training-with-kyle-banker

<comment><USER2> I reviewed MongoDB and really loved the feature set but the following seem like big problems to me:

Keys in the document values are stored in the document. Big keys = more space required.

No async client drivers (still true? The last time I surveyed was in November)

"It doesn't support rollback or anything like that, so you'd have to do that yourself."

Smaller problems or non-problems:

It is not durable (writes every N seconds). Granted, most "NOSQL" experiments are not _strictly_ durable (e.g. Redis, Tokyo Cabinet, etc.)

4MB limit / document (I don't have values this large anyway)

32-bit host limits MongoDB DBs to 2GB total size (so use 64-bit)

Auto-sharding is in alpha, and I'm glad to see it coming. But, things like this scare me: "If a shard is down, the query will return an error." Compare that feature to something like Riak wherein multiple copies of values are stored (Riak's W, DW parameters). MongoDB with auto-sharding does not seem fault tolerant. Am I missing something on this?

Please let me know your thoughts as I do like MongoDB for its speed and query operators but the above concerns me.

<comment><USER3> Sharding is use with replication to provide failover. This will be more automated in the upcoming sharding beta.

Key storage in the document objects make it more 'schemaless' which is actually incredibly useful; the document format (BSON) is efficient. In practice that works just fine. Try it out sometime on a test project?

<comment><USER2> Well I'm pretty much compelled to try at it this point. The reviews around here are pretty stellar.

Do you think that MongoDB would perform well on a shared environment like a Linode VPS? Does MongoDB have any sort of resource gotchas like "oh, you need at least 512MB RAM for the thing to even start up" or the like?

<comment><USER3> it can run in a very small footprint. the one gotcha is that as it uses memory-mapped files, you need 64 bit OS for multi-GB databases.</comment></comment></comment></comment>


Show HN: Simple creative game: make your friend guess movies using only 3 emojis - <AUTHOR> https://renga.party/p/ub7m43z4l <comment><USER2> This is quite fun! One problem is that it's easy to accidentally spoil the answers by guessing a previous movie in a series or the original/remake of the movie in question - but that might be hard to fix automatically?

<comment><AUTHOR> Indeed! We think to build a "Hint" system where user earn them by creating Renga and spend them to have access to the movie director or => the timeline (with only wrong answers like as it now)

</comment></comment><comment><AUTHOR> Hi HN, Made this simple game for film lovers. No login required. Let's have some fun and be creative :) Hope you're going to like. Open to feedback.</comment>