Ask HN: What is your company's on-call rotation like? - <AUTHOR> Also looking for readings on best practices.<p>Curious about your company's on-call for details about:<p>* expected duties (only answer on-call, do other work, etc)<p>* how deep does your on-call dive into code to fix, vs triaging, patching, and creating follow up work to fix permanently?<p>* priority order of work (on-call tickts vs regular work vs existing on-call tickets vs special queue of simple tasks)<p>* what happens when the current on-call can't complete in time?<p>* how do you manage for other teams' risk? (ie their api goes down, you can't satisfy your customers)<p>* any other tidbits <comment><USER5> To software developers in this thread who are on-call; I'd like to share some thoughts with you. I've worked at places that do have on-call rotations and others that have none. I will no longer work at a company that requires me to be on-call.

Why? It says a lot when a company doesn't put the effort into various forms of testing and QA to ensure that production software does not have critical issues that warrant at 2am call. Unit, functional, integration, load, and simulation tests should be written for every single piece of critical infrastructure. You should be hammering these things in staging environments with 10-100x of your normal peak load.Use something like Gore to replay live traffic against a version in staging or QA environments. Yes, that takes work, but to me it's better upfront than to wake me up in the middle of the night or to know that when I go home I have to have my phone around me at all times. The business should care about these things too; it's their product and they should care enough about you to make sure good processes are in place to ensure quality production software.

That said, when I was at non-on-call companies there are definitely times when something does happen that warrants immediate attention. Generally someone in operations would get the first call, they check logs, diagnose the issue, and call a developer familiar with the app that's causing the issues if that's the case. I don't mind waking up because I know it has to be something serious that slipped past our processes.

<comment><USER6> … whereas I'd conversely argue that volumes are also spoken when a company does not require its developers to be on-call. Often, those volumes are written with negative undertones and a narrative that speaks to developers not owning or understanding their software stack. Worse, in an exercise reminiscent of companies offering unlimited PTO, "not on-call," when read between the lines, could really mean "always on-call." I currently work with teams that operate this way, and the amount of burnout is staggering.

As someone who has leapt between development and operations time and time again, I can anecdotally say that the systems where developers felt like they had ownership stake (including a desire to know "what went wrong," often only accurately conveyed via an OC shift) were better designed, better maintained, had longer MTBF and had dramatically shorter MTBR.

That doesn't mean that the development team that engineered the system in the first place should always be the first line of defense; rather, it means that there should still be some sort of escalation tier in case the operations team to which the service has been handed off needs some further assistance. I can recount stories for days wherein I was frustrated as someone in operations that I couldn't reach someone from a development team in the middle of the night for a critical service--where VP- and SVP-levels were barking at me to fix it--only to hijack repository permissions and write patches myself for their software. It should go without saying, naturally, that reprimands for such heroic acts (heaven forbid I actually fix the damn thing) were definitely forthcoming the next morning.

That whole mess is wrong and anti-collaborative. Without the guardrails of a well-defined on-call shift, this is what no-on-call organizations devolve to.

This all comes with a caveat though: Again, anecdotally, I've found that tossing the development team to the fire for the first six or so months of a service's production lifetime, before even allowing them to ask for operations handoff, pays dividends in terms of meeting (or exceeding) business goals, keeping operations' frustrations low, and delivering a quality service that other teams can rely on. This goes in concert with automated unit/functional/whatever testing, knocking parts over in production in a controlled manner, continuous reviews of documentation, commit-gating code reviews, monitoring that makes sense, and so on and so forth.

(As an aside, folks on the operations side have enough to worry about in terms of integrating all of the infrastructure to make everything appear to Just Work™. Adding the burden of having to reverse engineer tossed-over-the-wall "It's an ops problem now" garbage is akin to the trend in the initial days of the devops movement for operations folks to toss systems automation over the wall to developers. It's disrespectful. Work together. Show trust and solidarity by carrying a pager alongside the ops guys to say, "Yes, I'm right here with you in case you need me.")

Tom Limoncelli somewhat recently put out "The Practice of Cloud System Administration"[0], and I implore that you give it a cover-to-cover read. Even folks who align with the development side of the house will get some benefit from it.

Finally, to your point about testing in a non-production environment: Even with a barrage of testing in staging or QA, you will still find problems in your software that only exist in production, and it won't be for lack of trying to unify fiddly things like configuration parameters or runtime versions.

0: [http://the-cloud-book.com/](http://the-cloud-book.com/)

<comment><USER7> > Worse, in an exercise reminiscent of companies offering unlimited PTO, "not > on-call," when read between the lines, could really mean "always on-call." I > currently work with teams that operate this way, and the amount of burnout > is staggering.

I've found the best way to screen for those sorts of problems is to ask everyone in the interview about their personal habits, instead of about process. Don't ask about vacation policy; ask when's the last time your interviewer went on vacation. Don't ask about the on-call rotation, ask about what issues came up outside work hours in the last month.

</comment></comment></comment><comment><USER8> For those doing on-call and required to be responsive to alerts at all times during your shift, one thing to think about is your pay for this burden. Some cardiologists get over $3K/day to just be on-call (whether or not anything comes up) which means keeping within range of the hospital, being sober, not being more than a few minutes away from their phone, and so on. While generally nobody's life is on the line should a server go down, if you're expected to go beyond "if I hear my phone alert and I'm sober and am near my laptop" level of effort, ensure you are compensated appropriately.

<comment><USER9> 3k a day? do you have a source for that?

<comment><USER10> Appears to be less than 3k/day on average. Source: [http://www.acep.org/Clinical---Practice-Management/Survey-- M...](http://www.acep.org/Clinical---Practice-Management/Survey--Most-Docs- Get-Extra-Pay-for-On-Call-Duties/)

<comment><USER8> Thanks for looking this up. I did some research many years ago when I was in an unpleasant on-call situation (unreliable third-party dependency) and negotiated pay for it as a result. I recall seeing a $3500 daily rate figure at the time for a cardio surgeon, but that may have been an outlier.

While the numbers may be off, one point I didn't make in my initial post but should have is that anybody doing on-call is providing a valuable service. It's no different than paying a babysitter to watch TV while your kids sleep. Both babysitters and devops/IT/whatever are compensated for their ability to act in case the unexpected happens. If someone's not getting paid for that, then it must not be valuable to the company to have that coverage.

<comment><USER11> Yeah, the article linked there is nicely specific. From the article (and these are median):

Neurological surgeons had the highest median daily rate for providing on-call coverage, about $2,000 a day. Near the top of the pay scale were neurologists ($1,500), cardiovascular surgeons ($1,600), internists ($1,050), and anesthesiologists ($800).

Among the specialists earning lower median daily rates for on-call compensation were: psychiatry ($500), general surgery ($500) gastroenterology ($500), ophthalmology ($300), and family medicine without obstetrics ($300), according to the MGMA survey data.

</comment></comment></comment></comment></comment><comment><USER12> Current rotation for me is: 1 out of 3 weeks from 7 am till 12 midnight as a systems administrator. Our SLA to customers is non-existent, so on-call duties are best-effort (if you are in the shower and it takes 30 minutes to respond, no real problem). Furthermore, this also means that it is up to the person on call to determine what needs to be done.

Outside of the regular work hours you are only expected to work when an alert / call comes in. If at all possible you should fix it, if it would take > 1 hour you start looking at temporary workarounds that will hold until you are back at the office. During regular work hours on-call tickets always get priority. When problems arise with external dependencies we open tickets in their support system and wait for their response. Most important in our case is to keep the support desk up to date so customers don't open too many tickets and to let people know we are working on it.

Depth of on call work depends. It usually consists of rebooting / restarting services and checking if they come up as expected and trying to find the root cause of the problems. If the root cause can be easily identified and fixed, fix it immediately. Otherwise, open a ticket to fix it asap.

</comment><comment><USER13> I started being on-call a few months ago. For us it works like this:

\- 1 week in 4 Monday 00:00 to Sunday 23:59, with a fixed pay adjustment (same fixed amount added to our salary every month, not a % increment).

\- If you get a call, you're on the clock and get paid regular over-time rate. Theoretically this only applies if you get called and are doing something for more than 30 minutes though.

\- I know most of the on-call managers personally so don't mind picking up the phone for a quick opinion for free, if it's at a reasonable time.

\- Online within an hour once we've picked up the phone.

\- Senior on-call manager gets woken up by Ops first and then will decide which parts of the system are affected. After going through known issues and work arounds will then work out which part of the system needs to be looked at and makes the call to the on-call developer that is appropriate (ie. which team).

\- Duty is primarily to diagnose and provide knowledge to the on-call manager to make a decision so service can come back up again.

\- Don't do code fixes (J2EE on my team, mix of Java and C++ on others) - <USER2> have far too much to go wrong for hacking around like that, but may do config changes and some scripting. The line can get a bit blurred.

\- In our team we should raise a ticket/Jira for every time we get a call so that it's visible to everyone what happened and see where our pain points are.

\- During office hours, Production issues are distributed in the team in the same way as bugs, according to domain knowledge and individual capacity.

\- We tend not to get called out except for when we do installation in to Production. We have a fairly heavy weight process for releases - which we are working on.

\- I am at least 4 levels of communication away from any customers, including 1st and 2nd line teams.

Edit: formatting

</comment><comment><USER14> On-call is paid at a weekly rate for being the on-call person. There's an additional bump if you're actually called out, pro-rata on an hourly basis to your salary.

You are required to respond to calls within the SLA, and work on problems until they're fixed (definition of fixed being service restoration - <USER3> cause canalysis for complex problems and hence fixes will happen during normal hours) or until you've been working long enough you need to be spelled.

If you're on-call you're still expected to be doing your day job, but production takes priority, and projects are expected to accomodate that.

On-call rotation is generally rotated around teams, so one week in 4 - <USER4> depending on team size.

<comment><USER15> We use a very similar system where I work, except that we have one "extra" person on-call in case the first one gets sick, has a hard disk failure or likewise, or just needs extra help for some reason.

Both people are paid a fixed amount per week + a fixed hourly rate if called out. Weeks with holidays give you double pay. Simple but it seems to work well.

</comment></comment><comment><USER16> Funny that I decided to finally create an account to answer this.

Our team is heavily silo'ed, and each developer is responsible for a specific set of code. Since we have gone through a bunch of reductions and reorganizations, this ends up meaning that often one person is responsible. Our customer is a 24/7/365 operational unit, and it is our only customer.

So, the end result is any given developer is on call all the time, with no pay compensation for that at all. Not only that, but you should expect to be called about not only a defect, but in cases where the customer is trying to use our software in a way it was never designed to be used. So, get a call at 2 AM on the weekend and design and implement a new feature, release to production to debug and test.

I really dislike the way we operate, and recently the entire QA team was let go so I have a feeling things are only going to get worse.

<comment><USER17> Holy crap, I thought places like that only existed as straw mens in articles about bad practices. It'd unlikely that any other place would be worse, you have little to lose by changing jobs.

A company that behaves like that is actively trying to find ways to cut your job too and they will just as soon as they think they can get away with it... Or maybe they'll cut someone else and have you do their job too, for efficiency.

<comment><USER11> The really good and really bad places tend to exist and not get talked about. That is, a surprising number of "strawman" workplaces, good and bad, tend to actually exist.

</comment></comment></comment><comment><USER18> In Brazil the labor relations are heavily regulated. Each hour you are on-call you make 1/3 of a regular hour of work, and while you answer a call and work on weekends or at night you are paid twice the regular hourly wage.

You are expected to answer the phone promptly, be sober and do anything to remedy the problem including reaching for other people that can fix it (does not have to be a permanent solution).

</comment><comment><USER19> Work at a well funded Series B startup

On-call is unpaid and no additional bump if called out or public holidays. No additional vacation in lieu either :(

\- Expected duties are to fix the issue and bring production systems back up, all engineers have context on production systems and can escalate if any additional context is needed. We have an established workflow of steps to take to determine root cause. We can also directly ping all engineers to provide additional help should there be a critical issue with a system you are unfamiliar with.

\- If you are on-call, the on-call duties come first and foremost above everything else. Expected response time is 10-15 minutes from issue raised at all times.

\- Mitigating risk is via communication and getting enough context to know the bad outcomes and how to mitigate any issues.

\- On-call rotation is very arbitrary and undefined (we have five engineers who can do on-call duty), we try to share the duties equally but some get assigned way more on-call than others. (I was on-call most weekends last couple of months and all throughout the Xmas period for example)

<comment><USER20> I would not consider working like that; 10-15 min response time? If on-call is only during office time then there is something about it although 10-15 minutes is mental. But outside office hours you need to get paid and paid a lot. Why would you do that unpaid? During the xmas period? You are getting screwed here.

<comment><USER21> > 10-15 min response time?

Unfortunately, yes. I'm (devops/sysadmin) in the same boat. <10 minute response time expected when on-call, regardless of time of day.

Below market pay, but I work from home. It works for now.

</comment></comment></comment><comment><USER22> Everyone takes a turn being on-call in one week cycles. We have an on-call day and an on-call night. Any infrastructure alarms go to the on-call person and he/she is expected to address the issue, which can include escalating it to the person who knows the most about it if the severity warrants that, or deciding it's not important and silencing for some time period.

On-call fixes until the crisis is mitigated. If something can be addressed the next day during normal hours, it is.

On-call tickets are the most important. In our company, "on-call" just means you get the infrastructure alarms for that week and time period, and gotta deal with them when you get them. There is no alteration in pay for on-call weeks.

If the current on-call can't respond within 10 minutes (or forgets to ack in PagerDuty), the next person on the escalation schedule will be notified and expected to deal with the issue.

Other teams' risk is mitigated because if it's breaking production, we just call them and make them fix it. The whole company is "always on-call" in some sense, because if your thing is breaking production, you're going to get a call and get asked to fix the problem.

We feel like the tooling on this, even with PagerDuty whose whole job is managing on-calls, is subpar and are constantly talking about creating an in- house replacement.

</comment><comment><USER23> A few years ago I worked at a company that paid £30/day for being on-call and £30/hour (or part thereof) while investigating / fixing incidents. We were usually on call for a week at a time with a rotation of around 6 developers.

We got SMS or phone call (we had a non-technical ops team that worked early/late monitoring systems and tried to do preliminary fixes) and were expected to be available and investigate 24/7\. If you didn't acknowledge the SMS alert within 30 minutes everyone else on the team got it, so it was in your best interest to do so to avoid waking up everyone else :)

Some of the services we had were pretty critical, so the fix in that case was 'as long as it takes to get it working', and then putting in a permanent fix during business hours. For lower priority services, we'd just leave it broken and fix it during business hours. If you were up fixing something in the night you weren't expected to be in at 8am the next day.

In terms of escalation if there was a major problem that couldn't be handled we'd usually get in touch with a few other people on the team to get their input. This hardly ever happened though, the only time I can remember is when the ventilation system failed in a data centre which was !FUN!

After that I worked at a startup that didn't pay for any overtime or on call, which is one of the reasons why I left. Since then I've been contracting, so haven't had to worry about on call but I'd be happy to do it again for the right price.

</comment><comment><USER24> 1) We don't keep a formal on call but 3 of us are tied to an automated alert system and whoever has a chance to take care of it, does. We are all full stack devs so generally we can fix it at the time. If its complicated, we can help it hobble and fix it later.

2) We get 3-4 alerts a year that have to be handled before the next business day.

3) As such, there is no real work priority, triage, etc. You resolve it immediately. There is no other priority. [ Any on-call event == lost money ]

> * how do you manage for other teams' risk? (ie their api goes down, you > can't satisfy your customers)

Asynchronous processing. I buffer until their API is back up. I do this for literally dozens of companies from small manufacturers to Amazon.

There really isn't any other good way to handle it and if you need to do it otherwise that really is a fundamental architectural problem that should have been resolved at the design phase.

<comment><USER17> Has your buffering got any back-pressure at all? Or do you buffer until the disk is full?

<comment><USER24> It would need to be down for ~96 hours under our heaviest 4 day period in our history for it to fill the disk. All of them would have to also go down simultaneously.

There is backpressure at 85% disk fill (this also is an on-call trigger event since it shouldn't ever happen in practice). Suffice to say, this never happens in the real world without hardware failures.

Disks are cheaper than fatiguing engineers with on-call events and this is accomplished by basically having a nearly empty 3 VM cluster with 512 GB SSD (Raid 1 pair, so 2 disks) each. The load on the rest of the VM host is negligible given its being asynchronously processed from this cluster already, so its just really filling the extra disks on the VM host we dedicated to this purpose.

Just realize we build to 4 9s.

YTD failures 3662/75631129 = 0.00004841921

This doesn't trigger an on-call event because it recovered automatically but the cluster does fail every so often for ~3k events. This is for a single API call.

</comment></comment></comment><comment><USER25> Overview: On call one week in six; pay £300/week for being available (extra for national holidays), and time-and-a-half for any time spent dealing with calls, rounded up to the next 15 minutes.

Expected duties: Respond to major problems outside of office hours, in response to a phone call. Be online fixing the problem within 15 minutes of being called. Typical workload 2 hours per week. Be somewhere with a reliable internet connection, and where you can stay even if an issue takes a few hours to resolve (no mobile internet from campsites). Be sober.

Fixes: Because of the need for code review and second testing, emergency code releases never happen out of hours. Instead, all code deployments include a backout plan, which reverts to a known-good version of the code (not a bug- free version, there's no such thing, but a version that hasn't caused major problems when run for several weeks). Data in the database may need to be manually fixed.

If you're patching an issue by deleting some bad data, and in your judgement deleting the data might delete evidence needed to identify the root cause of the problem, try to identify the root cause if time allows.

Priority: On call is only called for major problems - either their software has raised an alert saying they should be called, or there is a problem costing thousands of pounds a minute such as "website down", "customers cannot place orders" or "customers will not receive orders".

There isn't a formal SLA, but as it costs thousands of pounds a minute the expectation is to fix as fast as you safely can, without mistakes that make the problem worse.

Escalation: To their team leader, or to some other senior software engineer on their team (our team leaders have all been senior software engineers on the team they lead)

APIs down: In a rather old-fashioned design move, all critical components are maintained in house. Contact the out-of-hours on call for whatever team maintains the broken system. If it's a non-critical component.

</comment><comment><USER26> Programmer at a Hospital -- Only have to do stuff if I get paged (rotation is 1 every 4 weeks for 7 days, 5PM-8AM). If the issue(s) affects a clinical caregiver then the issue needs to be resolved asap no matter what, priority is only regarding the issue you were paged for, if the on-call person can't resolve they will usually page additional people or escalate to a manager who can find the right people. If another team's issues are causing us problems we just need to inform the customer about what's going on and then keep them updated.

</comment><comment><USER27> I work at a big famous company that strongly believes in devops.

I'm on call about one week in 6. (About 12 people in the rotation, with two on-call at a time.)

When on call, the ticket queue is supposed to have priority, but managers have an unfortunate tendency to say "that guy's on call so he's not doing anything so let's have him do it." So what ends up happening is that high-severity tickets get priority, then whatever random stuff the manager wants done, then low-severity tickets. So low-severity tickets tend to remain open for way too long because the oncalls aren't making them a priority.

For high-severity tickets, we escalate if we can't fix things quickly. First bring in the other oncall, then bring in the manager, then bring in anyone else on the team who can help.

When a dependency breaks, we can ticket its owners to fix it. If it gives us a high-severity ticket, then they get a high-severity ticket too. Ultimately there's a tradeoff between using fewer dependencies in the name of controlling your own fate, versus using more dependencies to avoid reinventing the wheel. Each team has to find that balance for itself.

I don't enjoy being on call, but I think it's the best solution we have. Stuff breaks and someone has to fix it. If you don't pick someone, then everyone has to fix it. If the same people who build something also have to fix it, they have strong incentives to do things correctly.

</comment><comment><USER28> Previous role:

* On-call is a week long (Tuesday to Tuesday 10am) rotation

* Primary duty on a page is to triage and be a point person during the incident

* Proper procedures during day-to-day coding should make the on-call shift be uneventful 99.9% of the time

* Managers are escalation point, then Directors, then VP of Eng

* Revert to previous state and call it a night is an acceptable fix - the primary issue can be resolved the next day

* Blameless post-mortems

</comment><comment><USER29> Hi,

My company does 24/7 devops for some clientele. We are a team of 6 for reference. We have been doing this for many years since before "devops" was the term for it. Some of the platforms we have also built for these clients, and some we simply manage, or we have only built+manage the automation.

In terms of answering your question:

\- expected duties. In terms of what we are selling the client there is a long specific list of what exactly we can provide the client. In practice however the list is exhausting, and intended 1/2 for CYA, and we will fix whatever issues arise. Some of that is seeing things that need work/adjustment/tuning/optimizing before problems arise, and then as you can imagine the whole being-on-call thing means that you must be an expert at resolving issues that no one saw coming. In terms of SLA stuff, we are supposed to be on-scene (digitally) in 15 minutes, but we are never that slow.

\- how deep. Think of it in phases. When something needs fixing, you must quantify the urgency. If there is an urgent fix needed this is ground zero. You do whatever you can, if that involves waking people up, or checking out code bases that are new to you and patching some developers code in the middle of the night, its all on the table. In ground zero mode, you do whatever it takes. Once the bandaid/fix/solution is in place to "stop the bleeding" as I like to say, then there is follow ups, that may include working with the client to have them implement the elegant permanent fix, or if they are in over their head, sometimes that lands on us too. But it serves no-one to have a bandaid that is going to just get ripped off every night. So we see the problem from A-Z, even if our offramp is on B, or M. If it's not urgent, we will just wait for tomorrow to discuss with the client, it falls out of the scope of on-call.

\- priority. Well, on-call work is billed and agreed different to ticket work since we are primarily a consulting company. So these are different buckets. But we also have our own products that we own 100%, and the priority is quite easy. If something is broken/down/about-to-break, it trumps everything else. Regular tickets are great, but they are meaningless if the platform for which your developing them against can't stay functional. On-call work rarely has a "queue" or even really a ticket system.

\- there is no "complete in time" it's either done or you failed. I say this, having failed too, and it sucks, but it is what it is. If something breaks, and you can't fix it, you don't just go home. But.. sometimes you do, but you walk away from the scene with your tail in-between your legs, no one plans for that.

\- managing other teams risk. Communication. Putting energy in ahead of time, and bringing things up before they break is huge. Also if you say "Hey you should turn left, there is a cliff!", and the client is insistent on turning right, this can do two things. A - they know and hopefully its recorded in an email or in a meeting that you wanted to turn left. B - if your absolutely certain they are going to run over a cliff, but your still on the line / have to support the darn thing anyway, you can quietly put a bunch of pillows at the bottom of the ravine, and prepare for the inevitable. When the car goes over the cliff, and everything almost grinds to a halt, and you manage to have been correct about it, but also managed to bail them out, you score a lot of gratitude from the client, and ongoing future relations.

This is all from the point of "consulting" mostly, but I have done this same type of work for many large companies directly on their payroll too, it all applies, and the bigger the company the more it's like consulting in some way also, because bigger companies are more compartmentalized. I have also done this in many small startups where your all just one small team.

Holidays and vacations are important, but they will never truly be the same after years of this. We are pretty good at it now, and we really do try to keep everyone up to speed with where "the bodies are buried" with all of our clients infrastructure. That is the hardest part. Everyone can look at a perfectly groomed wiki or set of 100% refined chef/puppet cookbooks/modules, but the world isn't perfect. So the hard part is learning how to take the punches with elegance, and people need a break. It really does take at least 3-4 people to have a not-insane 24/7 schedule.

We generally plan about 1-3 weeks ahead depending on the time of year and put it into our scheduling system, which is also the system that does our paging. We rotate weekends, and work it out amongst ourselves. Some people have things like date nights that we know about and try hard to not screw up.

Don't build your pager system yourself, you have enough stuff to do. I won't plug them because I don't want this to sound like an advert, but do yourself a favor and pay the money to a company that specializes in bubbling up alerts to notifications that can do phone/sms/email/ios/android/push/etc. These have really helped us manage the insanity & schedule.

If there is any advice, simply don't be frightened to say when you don't know something, even if your tasked with fixing it. There is little room for ego when your the one that ultimately has to be responsible. Being on call means you can't pass the buck, and the sooner you know what you don't know, the sooner you're able to learn it!

Edit: Typos

</comment><comment><USER30> Last time I was in a role which involved on-call rotation:

> expected duties (only answer on-call, do other work, etc)

Only answer pages. My employer did shifts a bit differently from most companies - only six hours per shift, no fixed schedule (decided a week in advance) and only outside of work hours (pages during work hours were handled by whichever sysadmins were on duty), which worked quite well to avoid burning out sysadmins. On-call shifts were paid, and shortages of volunteers were rare.

I'd expect to spend maybe fifteen minutes per shift fixing things, on average (this is in managed hosting, so a page could be any of our customers' services).

> how deep does your on-call dive into code to fix, vs triaging, patching, and > creating follow up work to fix permanently?

In my case (sysadmin for a managed hosting company) the code involved was often not under our control; the standard practice was to escalate to the customer if the cause of the outage was a bug in the application. The usual process when suspecting a bug was to track it down if possible (the codebases were usually unfamiliar, so this wasn't always the case), work around it as best we could (e.g., temporarily disable a buggy search indexer which was leaking memory, et cetera), and then get in touch with the customer (by email if the workaround was expected to last until work hours, by phone if not). Occasionally I'd fix the bug in-place and send the customer the patch, but this was technically outside scope.

> priority order of work (on-call tickts vs regular work vs existing on-call > tickets vs special queue of simple tasks)

The only priorities were resolving the pages at hand and arranging followup where needed (usually raising a ticket to be followed up during work hours).

> what happens when the current on-call can't complete in time?

Generally the on-call sysadmin would resolve whichever pages they had acknowledged; in the event of an extended outage the acking sysadmin was expected to brief and hand over to the person on the next shift.

> how do you manage for other teams' risk? (ie their api goes down, you can't > satisfy your customers)

In practice, we could escalate to anyone in the company for a serious outage we were unable to handle ourselves. This was pretty rare, as a small ops-heavy company, but everyone had access to everyone else's cell phone number and an outage-inducing bug was usually sufficient cause to wake someone up if it couldn't be worked around.

</comment><comment><USER31> Developer on-call every 8 weeks or so. 90% of the job is checking logs and saying "Yup it's working" or "Nope $VENDOR is down again." Get paged on average 3 times a cycle. One time I actually got to fix some code. That was awesome. Just feels really weird that no one in the tier 1 center can check logs at 6AM. But then again there is a lot of turnover.

Don't get paid for On-Call, but the job is super flexible with hours so we get flex time which is well worth it. I like being able to take a nice 2 hour lunch every once in a while.

</comment><comment><USER32> I work for a major games developer/publisher. Our submission packages are being built this week and will be submitted next Monday. We are supposed to be on-call this weekend just in case any problems in any of the systems we were responsible arise. No, I am not paid for being on-call, just like I am not being paid overtime. From what I can tell people are going to come in this weekend just to play the game and make sure everything works even though they are not getting paid for it.

<comment><USER33> Given that this weekend work is scheduled into the process, not paying you is intentional and well, grossly deviant. Grow a backbone or find another job. For all of us, stop letting this shit slide.

</comment></comment><comment><USER34> I'm basically expected to be 24/7\. Which gets a bit much some weeks.

<comment><USER21> What stops you from looking for another job?

<comment><USER34> Nothing. :)

But for all the long hours and high stress levels, I do still quite enjoy my job. Which makes it all the more harder sometimes, because as bad as my current place might get on occasions, I know there's very few positions out there that are sufficiently better to encourage me to leave.

But the plus side is that it puts me in the fortunate position where I can turn down jobs. A luxury many people don't have.

<comment><USER21> Completely understand. I'm in the same position. Thanks for replying!

</comment></comment></comment></comment><comment><USER35> In a past life covering out of hours (typically 9pm through 6am) for a core application for an insurance company the set-up we had was that we had a ladder of 4-6 staff providing on-call support. Whom ever was top of the list was first paged and if they didn't acknowledge within x minutes we kept going down the list until somebody responded. Once you got called you owned all incidents for that night and the next morning you would be moved to the bottom of the list.

When we got called out we were well recompensed at minimum of two hours of 3x hourly rate. If you resolved incident in 10 minutes you still got 6 hours of pay. If you got another call within those first two hours then you didn't get anything extra until you were working for 2 hours and 1 minute , but if the 2nd call came in after 3 hours you started the clock again. During times of instability (new code release) we often had management agree to you working from first call-out until the day shift came in to order to minimise downtime. When we were working during the night the on-call always had priority, but if you were sat around waiting for the system to do something then we all did something else to keep us awake. But there was no expectation that it was other tickets or project work. We made personal choices of doing day work or it could be crossword puzzle etc.

We all had areas of expertise and if you caught something you didn't know how to resolve then we had the authority to ring anybody. And that person would also receive the same compensation. In 5 years doing this I never had a complaint from calling somebody else out. Did we get tired yes? Most of us had project roles or team lead roles during the day but we all knew that the production system had priority and we and our own line management would deal with the project work accordingly. Did we get burned out on this? No because the rota meant that you still had a life. If you were top of the list and had other commitments that night you'd negotiate coverage with somebody else further down the list to see who could go top for the night.

My employer did this because the cost of not doing and having downtime during the day was a lot higher. If the system failed out of hours then it could impact our business and we could lose c.8,000 hours of day shift working time that would still require salary payment. Plus reputation damage etc. Most I ever accumulated over a weekend was 32 hours at 3x (situation so bad required to restore database to last known good and then catch-up log files).

One night I also got it wrong and in trying to correct the incident forced us into a situation when we had to do a database restore. That caused about 4,000 man/hours of downtime. When my bosses boss came in at 6:30am (he got called out by my boss) I got sent home being told I was too tired to continue support, day shift staff would take over and the missive to ring him when I had slept. I made the phone call expecting a significant dressing down only to be told, everybody else who looked at the diagnostics said they would have done exactly the same, we've all learned, and you stay top of the list tonight so that we know you're not scared to get back on the horse. Truth be told it did knock my confidence and anything that was a little out of the ordinary after that I always ended up calling somebody else for a consult. After a while I engineered my self off the on-call list.

</comment><comment><USER36> \- Every team is free to choose the on-call practices (escalations, ramp-up for new members, etc) that work for them, although many practices are shared. Tooling is the same across the board. I'll speak for my team.

\- Rotations are mixed, consisting of weekdays and weekends. In other words, if you're on-call M-F this week, in a few weeks you'll go on-call over the weekend.

\- If the on-call engineer does not pick up, the incident is escalated to the manager, then their manager, all the way up to the CTO. Some teams have secondaries before managers are involved.

\- The on-call engineer would normally work off the regular backlog. We have talked about pulling them out to work on on-call + tech debt exclusively, but there hasn't been enough on-call churn to justify this.

\- The on-call is on the hook for resolving the immediate issue. In many cases this does not mean actually fixing the underlying problem. Those get written up as tickets to be prioritized as part of the backlog.

\- The priority is determined as a team. As a manager, I encourage the team to aggressively tackle on-call issues, don't want to be a blocker for that. If something consistently pages us, it is guaranteed to make it to the top of the backlog fast. We have also chosen to prioritize product work over on-call tickets when we felt our pace was good and the on-call churn not too terrible. Kanban makes priority changes pretty easy.

\- Not sure what you mean by "on-call can't complete in time". On-call-related tickets end up on the backlog so anyone can tackle them, not just the person that filed it.

\- We have a pretty good on-call culture and the teams are very sensitive to other teams' pain. If we're being paged about an issue with another team's code (shouldn't happen too often), there's always the option to page their on- call and triage together.

\- We track operational churn, send a weekly on-call hand-off email to the team with notes about the pages and steps taken, and have operational review meetings where these emails are reviewed (+ any other operational matters) and next steps are determined. Maintaining transparency around operational pain and building structure around the follow-up process has been really helpful in reducing our weekly churn.

\- I think the ideal number of team members that should be on-call is around 5 to 8. Anything less than that and rotations become very frequent and a burden on the individuals. Anything more and the rotations are so rare that every on- call feels like the first time.

\- Last piece of advice is to FIX THE PROBLEMS (or change your alerts). Build whatever process you need to make sure that you have plenty of leeway to address anything that pages you constantly. Don't get overly attached to the alerts - you might find that changing sensitivity thresholds or deleting the alert altogether might actually be the right answer (please don't do that for the wrong reasons though :)). If you're paged for a bunch of things that you can't fix, you're probably doing it wrong. Just like technical debt, if you don't tend to on-call issues they WILL get out of hand. Picking away at it slow and steady will almost certainly help reduce it from a torrent to a trickle.

N.B. I work at PagerDuty. A bunch more suggestions here: [https://www.pagerduty.com/blog/tag/best- practices/](https://www.pagerduty.com/blog/tag/best-practices/).

</comment><comment><USER37> We have no developers on call

</comment><comment><USER38> I've worked two different on-call systems. The first was an enterprise company with a very defined process. 1 hour of pay for every 10 on-call (even if you're salaried, they figure out your hourly pay for this). Must be able to respond and/or get into the office in 1 hour, otherwise they call your manager to get someone working on the issue. Our 24/7 support would triage all issues before escalating on the on-call engineer--they were able to resolve most issues. Rotation period was roughly 1 week in 3 or 4 on a volunteer basis. We did have a group of software engineers who had an on-call rotation for the internal applications that powered the business. If I needed additional support for the application I could tell our 24/7 support folks to page/call them and they would hop online, which was needed on rare occasion when I couldn't fix an application error or a bug slipped the manual and automated QAs.

My current on-call system is considerably different as a startup. Most weeks there will be no pager alerts, some weeks will be particularly bad because something fell out of stability due to some other changes. There is a primary and secondary on-call. Each level has 20 minutes to respond and get online before it escalates (no office requirement since we're cloud based). The rotation goes through devops and all the software engineers, so you're on call for 2 weeks in 8-10 with no extra pay. I wouldn't recommend this method of scheduling because it's mandatory for all and some people don't take the duties seriously because nothing bad will happen if you let your pager slip to the next line, other than irking your coworker. There's no incentive to learn how to do the repairs or do them. I've seen a lot of "Oh my phone was off, I didn't realize I was on-call" that never happened at my previous job with the volunteers for extra pay. Having the secondary is nice because it is a guaranteed person you can escalate to for help on a complex issue, and they are available if you need to be indisposed for a short period of time.

About your specific questions: Your on-call duties are to communicate, fix what's broken, coordinate any additional escalations that need to happen, and most likely host the postmortem. Nothing more. Update your status page, send out an alert internally that X is broken, etc. When it's fixed and normal service has resumed, communicate that as well. You're the point of contact for anyone with questions about the issue, not anyone you had to bring online for a fix because if they're interrupted they're not fixing.

Don't start non on-call work. Fix to the extent that you know how and will make it stable until business hours. Not everyone has coded every system and knows the "permanent fix" for every issue. Your priority is based on what is broken and the business criticality of it. If multiple systems have failed, you fix the most critical ones first, which are normally the customer facing ones. There should be no "existing on-call tickets" because on-call bandaids and makes a high priority issue in the normal work queue.

If on-call can't complete the work in a reasonable amount of time, it may make sense to raise other people who can help. If it's going to take you 4 hours to get the $CRITICAL_FUNCTIONALITY back online, but getting Joe to help you will only make it take 1, then by all means try to get Joe if he's available. Again, based on your rotation he may not intend to be available, may not have his phone on, and may not want to take the call.

If you're dependent on 3rd parties, your application needs to take that into account. If you wake up because the external API is down and your functionality is down because of it, all you're going to be doing is losing sleep waiting for it to come back up to send out the all-clear. This changes somewhat if your external API comes with a SLA and a telephone number -- by all means call and start their triage chain.

A good thing to have is a sync up meeting with the on-call folks so you can establish patterns in the alerts that may not be evident by any one person.

</comment><comment><USER39> * expected duties (only answer on-call, do other work, etc)

Expected duty is to solve the problem but there are often escalation paths to take. If a problem is not solved within a required SLA period then the company can be forced to pay the client penalty fees.

* how deep does your on-call dive into code to fix, vs triaging, patching, and creating follow up work to fix permanently?

They go as deep as necessary, or as deep as is dictated by an operational manual pertaining to that particular client/environment.

* priority order of work (on-call tickts vs regular work vs existing on-call tickets vs special queue of simple tasks)

On-call is always higher priority since it's an add-on service that clients pay for.

* what happens when the current on-call can't complete in time?

See above, penalty fees mostly.

* how do you manage for other teams' risk? (ie their api goes down, you can't satisfy your customers)

Not sure I understand the question, if an API goes down and affects our services then that API needs to be monitored and handled by our on-call team.

* any other tidbits

I'm not in the on-call team but I stay available for specialized expertize if the 1st line can't solve an issue.

I know how they work though so here's one example. It all depends on the clients SLA but let's say the client has 99% uptime, 24/7 on-call duties in their contract.

In that case one person out of 5 rotates an on-call device (phone) every 5 weeks. In the strictest of SLA they're required to respond within 15 minutes and sometimes have a solution within 4 hours. This varies wildly from contract to contract.

Of course an incident manager is available, redundantly, and is tasked with coordinating skills between departments to solve an issue within the designated SLA.

Alerts come into the device and the tech can respond to alerts directly via the device to acknowledge, force re-check or simply disable. There is also a more featured web interface for the alerts to access via a browser.

Alerts are sent with SMS through a self-hosted gateway. Directly attached to the monitoring server, not using any e-mail translation API.

Alerts are logged, and in some cases a ticket is created for an alert.

Preferably a manager should work out the on-call schedule, but techs often trade weeks and are more than capable of handling it themselves.

They receive an added monthly compensation including overtime. So any work must be reported in a time reporting utility to eventually lead to payed overtime depending on the contract it pertained to and the time when it happened.</comment>


GHC 7.6 is now live: poly kinds, dynamic use of cores, numeric type literals... - <AUTHOR> http://www.haskell.org/ghc/docs/7.6.1/html/users_guide/release-7-6-1.html?7.6.1

<comment><AUTHOR> Much discussion here: [http://www.reddit.com/r/haskell/comments/zgeen/ghc_761_offic...](http://www.reddit.com/r/haskell/comments/zgeen/ghc_761_officially_released/)

Highlights:

* RTS now supports changing the number of capabilities at runtime

* Dataflow based code gen is on

* Unboxed tuples (register allocated structs) are now first class

* Multi-way if syntax

* Lambda case syntax

* Type level naturals and strings

<comment><USER3> I've been away from Haskell for a while, it's great to see interesting things still being done. Wondering what the use of type-level naturals in Haskell is though? I dabbled with Agda a bit where they're used to satisfy the termination checker, is that sort of thing being done in Haskell now?

Also one thing that always gave me pause about Haskell was the record system, and the inability to have fields with the same name in the same module (especially when the fields are generated from a database). Has there been any progress on that front?

<comment><USER4> _Also one thing that always gave me pause about Haskell was the record system, and the inability to have fields with the same name in the same module (especially when the fields are generated from a database). Has there been any progress on that front?_

There have been a zillion proposals, but as far as I know, not much actual consensus or implementation work.

Anecdotally, I've never run into a situation where this was anything more than a minor annoyance. I don't doubt that it can be a problem under the right set of circumstances, but I think the issue gets more attention than its magnitude truly merits. People theorize a lot about the difficulties it can hypothetically cause, but in practice, it seldom actually manifests itself.

<comment><USER5> In my experience it was a perpetual pain point. Do you namespace all your field names with a common prefix based on the outer type name? Or do you just get lucky and not have the names collide?

<comment><USER6> In the past most people prefixed their field names with the type name. Nowadays, the most common thing is to put types in their own module, and use the module system to sort out any conflict. Qualified imports can add a prefix, but you can also import parts unqualified, or hide one problematic field.

<comment><USER7> While I prefer the module solution, it's not always possible when you have mutually recursive record definitions. Well, technically it is possible - <USER2> it's very unpleasant [1].

[1] [http://www.haskell.org/ghc/docs/latest/html/users_guide/sepa...](http://www.haskell.org/ghc/docs/latest/html/users_guide/separate- compilation.html#mutual-recursion)

</comment></comment></comment></comment></comment></comment><comment><USER4> Every new GHC release is like a little Christmas morning! The pace of new feature development impresses me.

I'm particularly interested in the new type-level literals (which seem like they will make it much less cumbersome to express certain static constraints) and the convenient new syntax for multi-way if expressions (I've long wanted Lisp's `cond` in Haskell) and case analysis on lambda arguments.

<comment><USER8> `if` is not a function. [http://www.haskell.org/haskellwiki/If-then- else#Replace_synt...](http://www.haskell.org/haskellwiki/If-then- else#Replace_syntactic_sugar_by_a_function)

Functional `if` and `cond` are here: [http://hackage.haskell.org/packages/archive/cond/0.4.0.1/doc...](http://hackage.haskell.org/packages/archive/cond/0.4.0.1/doc/html/Control- Conditional.html)

<comment><USER4> Did I suggest that it was a function? If so, I didn't intend to.

<comment><USER9> 'If' is not quite analogous to cond, and 'if' is a bit of a hack, and multiway is more hack on top, is all I wanted to highlight.

And the cond you wanted has already been here for you.

</comment></comment></comment></comment><comment><USER10> Does anyone know how multiple arguments work with lambda-case[1]? Is the solution just using a tuple and manually currying like so (suggested here[2])?

       curry (\case (Nothing,_) -> ...)

(It seems like this would defeat the purpose of lambda-case for lambdas with more than 2 arguments, and it is awkward even for those with exactly 2 arguments.)

[1] [http://www.haskell.org/ghc/docs/7.6.1/html/users_guide/synta...](http://www.haskell.org/ghc/docs/7.6.1/html/users_guide/syntax- extns.html#lambda-case) [2] [http://hackage.haskell.org/trac/ghc/wiki/LambdasVsPatternMat...](http://hackage.haskell.org/trac/ghc/wiki/LambdasVsPatternMatching)

<comment><USER11> You can still do \a b -> case (a,b) of ...

<comment><USER12> And, perhaps, use uncurry $ \case ...</comment></comment></comment>


Elon Musk email to Tesla employee about missing an event to witness the birth - <AUTHOR> http://www.businessinsider.com/elon-musk-email-tesla-employee-2015-5 <comment><USER5> My first day at SpaceX I ran into a friend. I asked him how he was doing and he said he just had a kid. "How old is she now?" "Oh no, like, my wife just gave birth an hour ago."

The guy was working through the birth of his first born. The story continued for months afterwards and it's not a good one.

That and many other stories are what led me to quit. I took the summer off and didn't work. I hung out with my dog and got my pilot's license. A few months later my dog unexpectedly passed away one night. I was so, so happy I spent the last months of his life with him instead of at SpaceX.

I get working on big goals, I get working hard. I don't get throwing away everything else. It's not just your life it's the people who need a father and a friend.

<comment> at-fates-hands >> I get working on big goals, I get working hard. I don't get throwing away everything else. It's not just your life it's the people who need a father and a friend.

Agreed.

The quote which finally got me to realize my family was more important than my job:

 _" When you're gone would you rather have your gravestone say, 'He never missed a meeting.' Or one that said, 'He was a great father'."_

<comment><USER6> I agree that family is important and it's especially important to be there for other people, but regarding your quote, to the two gravestone inscriptions above I'd much prefer: "He helped mankind make its first steps on Mars.", or "He helped ensure the world is a good place to live for generations to come.".

<comment><USER7> The question is about whether you'd have it say that _or_ that you were a great father. It's rare to be able to do both, particularly if the company's CEO says having a child is "no excuse" to miss an event. Not a deadline or a board presentation or an earnings report. An _event_.

<comment><USER8> Pretty sure the ppl of Nasa/JPL who actually put a man on the moon, worked reasonable hours and almost never missed dinner with family. So, I don't think they're mutually exclusive

EDIT: So, everybody was screwed then? I must have mixed anecdotes :)

<comment><USER9> "The room, the hours that we worked were incredible. I don't think anyone ever worked anything less than 10 to 12 hours each day. Saturday was a normal day of work; in fact, that's the way we felt it should be. We were given this impossible dream by President Kennedy, and we were living it. We were doing the kinds of things that engineers would kill for. And as part of this process, we'd go and open up our pay—we were surprised we were getting paid by this thing here. As long as we had enough money to make things meet, that's all we needed. The job was our life, and we lived this literally every day."

\- NASA Flight Controller Gene Kranz

[http://www.pbs.org/wgbh/nova/tothemoon/kranz.html](http://www.pbs.org/wgbh/nova/tothemoon/kranz.html)

<comment><USER10> 10 - 12 hours a day is pretty typical in the entertainment business. Most tv shows do 12s for an entire year, movies about the same for less, and VFX workers during crunch time (which is near permanent for some) 12 is usually the minimum. Plus your weekends are gone.

At least the guys in the Space Program were doing for a far more valuable reason.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> The wording the article uses is offensive. A dad isn't in the delivery room just to "witness" the birth, or at least he shouldn't be. He's there to support the mother, advocate for her, help her make decisions regarding how the birth goes, support the newborn baby, advocate for him or her, and (maybe) make very difficult medical decisions if either is compromised.

Most of my children were born without a lot of complications, and I didn't have to do much decision-making or advocacy, just support my wife, talk to her, hold her hand, rub her back, cut the cord, take pictures, hold the baby, and all that. My baby girl responded immediately to my voice in a way that she didn't respond to other voices, because she had heard my voice a lot while she was in the womb.

My most recent turned into a difficult birth, with both mother and baby very sick with a rare infection, and an emergency C-section, and then with my newborn son in the hospital for ten days of IV antibiotics. Because my wife was in recovery I had to immediately provide "kangaroo care" \-- that is, skin to skin contact with my newborn son, to help him adapt. Also, to hold him as much as possible in the neonatal ICU, while he had IVs stuck in and medication pumped in. My wife was not very mobile at that point.

Once she was a little more mobile the friends we had tapped to watch our other kids were all tapped out and so I had to be a stay-at-home dad for them.

Parenting responsibilities and spouse responsibilities are not like work responsibilities. They just aren't. Anyone who says or behaves otherwise is a sociopath.

<comment><USER12> > A dad isn't in the delivery room just to "witness" the birth, or at least he > shouldn't be.

But I mean, surely the "witnessing" is important on its own, no?

<comment><USER13> We went through the Lamaze classes where they tell you all the things you'll be doing to help your wife birth your first child. Then we went to the delivery room where the nurses pointed next to my wife's head and said, "you stand here" while they did all the things I thought I was supposed to do.

And, honestly, I didn't mind. It was nice letting the professionals do their jobs and I could sort of be a cheerleader (bad term but its the best I can come up with at the moment) and witness the birth of my child.

<comment><USER14> Hospitals and experiences differ, neither of my kids' births were like that.

</comment></comment></comment></comment><comment><USER15> _Musk even scolded a Tesla employee who missed an event to witness the birth of his child_

If true, that diminishes my respect for Elon just a touch. I mean, OK, I get it... when you're trying to change the world, you want passion, you want commitment, etc. Fine. But here's the thing: in the end, you're just as dead either way. Change the world, you're still dead. Don't change the world: You're dead. The only thing that _really_ counts in many ways (if such a thing can be said to even exist) is what happens "between the goalposts" so to speak.

If I were working for Elon and he said something like that to me, I'm pretty sure my answer would be "bugger off". You only get to witness the birth of a child once, ya know?

It's also important to remember that, as a founder, you'll _never_ find people who are exactly as committed to your project and your ideals as you are. You just can't. It will always be _your_ baby, and it will always hold significance to you that it will never hold for anyone else in the world. You may find people who are committed and passionate, but they still won't be you. To expect otherwise is folly.

<comment><USER16> If this statement is true and in the proper context (who knows), this all but kills my respect for Musk. It basically puts him at the same level as Jobs, Ellison and others who are just egotistical, sociopathic assholes.

Look, Musk /is/ out to change the world. His employees might not be, at least to the degree and level as he is. And, if they are, they'll likely not be remembered along with Musk. There is often only one, maybe two names attached to great achievements. If I'm an employee at a company like that (and I have been, and I have killed myself with long hours making someone else rich when I was young and dumb, before I realized what it was all about), I would do my job very, very well and make sure I have the balance in life /I/ want.

You don't want me to see the birth of my child? Sorry, fuck off. It's my time, my life, my family, my kid. And we aren't talking about a tee-ball game. This is the BIRTH. We aren't talking about kindergarten parent-teacher conferences. This is the birth, where you support your wife and see your child for the first time.

And, sadly, Musk probably doesn't realize because he is so wrapped up in his own world, but having a child IS changing the world for a few folks. But, ya know, sociopaths don't care about other people, just themselves.

I'm continually saddened that we put people on pedestals who so clearly aren't deserving of it. Musk is just another example of someone who /really/ is out for theirs and want to get it any way possible. I think we all thought, or maybe hoped, he was different. I don't know that he is.

<comment><USER17> I'm mostly just happy that we've largely figured out how to channel driven sociopaths into relatively productive pursuits like building rockets (and, yes, less productive things like finance) instead of conquering, pillaging, and salting the earth.

<comment><USER16> This is easily the best 'glass half full' statement on corporate sociopaths I've ever seen.

Thank you!

<comment><USER17> Heh, you're welcome!

I truly do think that one of the keys of modern civilization is that we've figured out a system where the best way to get ahead is to give people something they want, rather than by killing and stealing.

These people might be the ones tossing books into the Tigris until it runs black with ink, but instead they're "just" abusing people because of their family obligations, bending labor laws, etc.

Not to defend that sort of behavior in any way, but it's better!

</comment></comment></comment></comment></comment><comment><USER17> I'm a fan of Musk and his companies. I own one of his cars (as far as they can be called "his," obviously there are tons of people involved) and it's even better than all the "best car in the world!" people say. I follow every SpaceX launch and am eager to see them pull off their stage recovery attempts soon.

That said, this doesn't surprise me at all. He doesn't seem like the kind of guy I'd want to work for. My admiration is about vision and execution, not personal relationships. He may well be a dick, and it would be no shock if he is.

Maybe this is what it takes to do big things. Maybe it's entirely unnecessary. Either way it reminds me a bit of Steve Jobs, who was undeniably a massive jerk, but who got things done.

I forget exactly what my point was here. I guess it's just that this is a bit of a "water is wet" story mixed in with celebrity gossip. It's highly important for the people involved but not terribly relevant for the rest of us.

<comment><USER18> Jobs made fancy white electronic devices - he just sold a lot of them - <USER2> justify rude behavior with something that trivial simply doesn't seem right to me.

<comment><USER17> I'm not saying that anything justifies rude behavior. I'm just commenting on a case where rude behavior and success occurred together.

</comment></comment></comment><comment><USER19> That's over the line if it's true in my opinion. But again, all we have here is some anecdotal quotes that might be really out of context. It could just be just on a status update meeting on Monday 10am and the guy who's responsible for a critical component did not show up and no one knows why.

</comment><comment><USER20> From Justine Musk, Elon Musk's wife during those times:

"Extreme success results from an extreme personality and comes at the cost of many other things. Extreme success is different from what I suppose you could just consider 'success', so know that you don't have to be Richard or Elon to be affluent and accomplished and maintain a great lifestyle. Your odds of happiness are better that way. But if you're extreme, you must be what you are, which means that happiness is more or less beside the point."

...continued at: [http://www.quora.com/How-can-I-be-as-great-as-Bill-Gates- Ste...](http://www.quora.com/How-can-I-be-as-great-as-Bill-Gates-Steve-Jobs- Elon-Musk-and-Richard-Branson) (answered by Justine herself)

<comment><USER21> This is a thought I've been having myself a lot lately. None of the successes I have actually leave me happier at the end of the day, they just kind of exist. Trying to balance success with happiness is a curious dilemma since to so many people the two are intertwined.

Very interesting read though- I appreciate you linking it here.

<comment><USER20> Following Justine's thesis: Do you mean that you feel you're an extreme person, like Musk, you must be what you are, which means that happiness is more or less beside the point? Or the opposite: that you're a non-extreme person, and feel that you're focusing too much on success and not enough on happiness?

</comment></comment></comment><comment><USER22> If true -- that is, Musk was criticizing an employee _specifically_ for being at their child's birth instead of work -- then he's an ass, and he's _wrong_. I would expect somebody with 5 children would know better.

 _Every_ man or woman put on a pedestal will inevitably disappoint their admirers in some regard. Steve Jobs, despite his skill at pushing Apple into a highly profitable and desirable direction, appears from most accounts to be a horrid, miserable human being. Same with every politician and celebrity.

I doubt this will change much; Musk's goals and reputation hinge on technical delivery, not interpersonal skills.

</comment><comment><USER23> I don't care about Musk either way, so I doubt this is too biased in his favor, but I don't think many people would knowingly say something like the phrase in the article to someone who went to his child's birth. I consider it more likely that Musk didn't know that the person was missing because of the birth, or something similar, than that someone would say "yeah this company event is more important than your child being born".

<comment><USER24> I've heard him talking before, and this doesn't surprise me at all. He has a definite mean streak, and he's almost gleeful about being nasty.

</comment></comment><comment><USER25> A good number of "world changers" prioritize their work over everything else in their lives, including themselves and the people around them. They expect their team to do nothing less, so if he did say that, I'm not exactly surprised.

This wouldn't work for me, and I'd imagine it wouldn't work for a lot of people who prioritize their families over whatever project they may be working on. I still like Musk and the things he does, but I have no desire to work for him if that means working 24/7 like he does.

<comment><USER26> History repeats itself and people who are trying to "change the world" don't care about any casualties on their way.

Ironically, world could change so much by everyone being kind and grateful to others. No invention or conquest can bring this kind of change.

<comment><USER6> You're saying it as if Elon was Genghis Khan. He's not conquering anything, he's just trying to solve two of the big problems of humanity, and he invites volunteers to help him. You don't believe in his mission? Don't join, and don't take places because you want to "build a resume" (it seems to be a recent trend now). It's not like he doesn't keep saying that Tesla and SpaceX are not ordinary jobs.

<comment><USER26> No, I didn't say this. I'm saying that he's making the same mistake as thousands of leaders before him: placing achievements first and people second.

<comment><USER27> How is that a mistake? His companies both had periods where they barely survived, and now they are driving forces in solar power, electric cars, and private space operations. Had the people there not been as dedicated as they were, some or all of them might not still be here.

<comment><USER26> It would not be a tragedy for me. Well-being of these people is more important than an imaginable entity named "company".

</comment></comment></comment></comment></comment></comment><comment><USER28> To everybody who is talking about this being out of context: in what context is the alleged behavior okay?

<comment> sjm-lbm The article makes it sound like "That is no excuse" is referring to taking time off work to see your child born - in other words, there is nothing that could excuse someone taking off time for that. It's possible that he's merely saying that taking time off for something so important is no excuse for failing to request the time off with HR, failing to ensure that a different team member could be present in a meeting, or something along those lines - <USER3> other words, not that taking time off for important family reasons isn't OK, but rather that things still need to be handled properly even for things as important as taking time off to be with your partner while she was giving birth. Something like that might not be the kindest thing to do, but I can at least understand the logic behind it.

<comment><USER28> The alleged e-mail seems to mitigate against that reading:

> That is no excuse. I am extremely disappointed. You need to figure out where > your priorities are. We're changing the world and changing history, and you > either commit or you don't.

"Figure out where your priorities are," not follow procedure. "Commit or you don't." There is not a terrible lot of ambiguity here.

</comment></comment></comment><comment><USER29> ...just when you start think there might be someone in Silicon Valley who is worth working for and learning from... it turns out he is also a flawed human being who can and will wreak havoc on this employees' personal lives.

If this is true, and not taken out of context, Musk has some serious soul searching to do.

</comment><comment><USER30> While it's not unbiased, his ex-wife's article on their marriage and this comment makes me think Musk has NPD. I was going to say he's a psychopath, but he has exhibited no signs of antisocial disorder. Rather, he's exhibited a lack of empathy and illusions of grandeur.

Seriously, it's more than possible to achieve great things while having an enriching family life. It's about involving your family in your dreams while working on having healthy relationships. It's possible, it takes a lot of work, but most worthy things are.

</comment><comment><USER31> Before anybody jumps to conclusions about Musk, we need to first know the source is accurate and the email is real, and also we need to know the context of the email.

It could be very well that the employee failed to communicate the fact his wife may go into labor prior to the event, and with no heads up of any kind, he gave the excuse after missing the event. In which case, I completely agree with Musk, if any employee is expecting to be out soon, they need to make it known. And let's face it...you have quite a few months notice on when the baby is due.

<comment><USER16> Yes, you have a rough idea when babies are due. My middle child was born 5 weeks early, however. My last was born 3 weeks early. I've had employees send me a note a day after saying 'sorry, baby came early. more later'.

If you have any reaction other than "oh, ok, hope everyone is healthy", you are probably a terrible human being.

<comment><USER32> Huh, my coworkers who have had pregnant spouses generally gave us a heads-up around week 24... unlikely to be that premature, but it helps everyone plan for that possibility.

<comment><USER33> Even if you give people at work a heads up at week 24, that's still a lot of weeks left to go in the ideal case. I caused my mother to get an emergency C-section 4 weeks early and it still wreaked havoc with everything - <USER4> importantly, my father missed my birth because he had to go pick up my grandmother flying in from halfway across the world, who thought 4 weeks was plenty early.

Besides, this kerfuffle says volumes about how some people would still react to an unexpected emergency. Death or severe illness in the family? Immediate medical emergencies? Taking care of myself and my immediate family comes before any corporate bullshit, personally. I wouldn't have it any other way.

</comment></comment></comment></comment><comment><USER34> If this is true it's a little bit sad. If a person isn't present in the most important moments of a family is "changing the world" of the family for worse and not for better.

<comment><USER35> > is "changing the world" of the family for worse and not for better

Hypothetical counterexample: What if he were busy curing cancer and thus making sure that the future world of that very family would be a nicer place, one where the children would never have to experience that disease?

<comment><USER16> That is a classic false dichotomy. You never, ever need to choose between those two things. You can have some semblance of both in nearly ALL cases.

<comment><USER35> I was thinking of a generalization. Choosing between working on something important in the future for many people vs. spending time in the now with a few people you value.

I.e. if you don't single out the event "birth of a child" but look at the more generalized category "spending time with family and friends" then the response may be more understandable.

If Elon truly convinced himself that he's basically trying to do his own contribution to saving the world (climate change and such) then in his mind he may be working to save countless future lives at the expense of some personal time of his own and his employees. And thus "birth of a child" and "going bowling with friends" don't seem all that different.

It's basically a side-effect of convincing himself absolutely of his own mission.

Also note that scolding is not necessarily the same thing as punishment. It's just a harsh way of stating a preference. I.e. that he would prefer if they'd spend more time on saving the world.

<comment><USER36> That's just a rationalization. I really doubt Elon cares about anything but his own driving ambition. People who can't prioritize their own loved ones over their jobs, surely don't care for "humanity". They are the very definition of selfish. They simply do what they want first and then rationalize it afterwards.

On top of that, these selfish people expect/demand that their employees work infinitely harder than they do on a ROI basis. Example: A start up founder may worked twice as many hours as the founding engineer but the founder stands to make a _lot_ more than twice as much when the company has it's liquidation event.

<comment><USER6> > _I really doubt Elon cares about anything but his own driving ambition. > People who can 't prioritize their own loved ones over their jobs, surely > don't care for "humanity". They are the very definition of selfish. They > simply do what they want first and then rationalize it afterwards._

No, I think what you wrote is selfish and is a typical rationalization of having a botched moral compass. I should prioritize my own over the rest of the world, so it's ok to fuck everyone over as long as I and my family are happy. This is how a lot of world seems to operate. You wouldn't have companies polluting rivers, or shoppers cheating you on every occasion possible, if those people were not prioritizing themselves and their families over other humans. It's easy to rationalize the harm by saying to yourself, "I have kids to feed, so fuck those other people, I need that money".

> _On top of that, these selfish people expect /demand that their employees > work infinitely harder than they do on a ROI basis. Example: A start up > founder may worked twice as many hours as the founding engineer but the > founder stands to make a lot more than twice as much when the company has > it's liquidation event._

While this may apply to many startups, it doesn't apply to SpaceX and Tesla, and it's disingenuous to discuss them in such terms. The ROI on electrification of transport, switching everyone away from fossil fuels and enabling life on another planet is _fuckton lot for every single human being_. It dwarfs most of the personal sacrifices you could make. And still, no one is telling you to do it; people who join those companies know up-front what's the goal, and how much dedication is required.

<comment><USER37> " I should prioritize my own over the rest of the world"

It feels like you're suggesting that a job such as SpaceX is automatically beneficial to the rest of the world.

I suggest that you are assuming too much. Could SpaceX usher in a golden age of transport and colonisation? Sure. It might also fail.

I suggest that you are also assuming too much by suggesting that parenting your children is selfish and (implicitly) doesn't benefit the world. It's possibly that SpaceX takes us all to Mars and none of the kids/descendants of its employees were ever going to achieve world changing things without their mum/dad being around more. It's also possible that some of those kids would have achieved greatness if only they had a more present mum/dad in their life.

You can't bail the future down, it's made of unrealised possibilities that we make real with our choices. Those choices are either implicit or explicit and we have no way of knowing what will actually happen.

</comment></comment></comment></comment></comment></comment></comment><comment><USER38> -it true-

I don't know how people here are deffending this kind of attitude. If you decide to bring another person to this planet, you have a huge responsibility. You can balance things out and work a lot and have a family. But if you really want to work 23 hours a day without paying proper attention to your family, for the sake of your children and spouse, just don't form a family. The damage you'll do, is irreversible.

You can't have it all in life... And one has to make hard choices.

<comment><USER13> That "23 hours a day" part is what strikes me as suspect. Unless they're being hyperbolic--and it doesn't seem like that's what they're trying to do--then its not sustainable for anyone.

</comment></comment><comment><USER39> "Changing the world." The world is bigger than the technology in it. The world consists of relationships too. I didn't learn this early enough. While you're out building rocketships don't forget to build relationships.

What a lonely world it would be if we had everything but nobody to care about us.

<comment><USER6> Not everyone has to build rocketships. Let the few who want to take one for the team, and the rest can enjoy the fruits of their sacrifice.

</comment></comment><comment><USER40> Not excusing the comment, it's pretty harsh.

It seems that everyone praises the prioritization of family over work. The quote that's brought up goes like:

"When you're gone would you rather have your gravestone say, 'He never missed a meeting.' Or one that said, 'He was a great father'."

This supposes that the acceptable thing to do for any human being is to desire having a family and caring for your own kind. It also supposes that work can only be a succession a meaningless events.

I see another take on this. I see that some projects can be life projects. Some of those life projects can happen to be 'work'. Desiring the completion of a life project, as your mean to live your humanity, is as valid in my opinion, as the desire to fulfill your humanity through the experience of family.

When you look at someone like Elon Musk and criticize him for not caring about family, you criticize him for having a different set of priorities than yours. The argument is an appeal to emotion and some assumed higher truth. It's not valid. There's more than one way to focus your life. You don't _have_ to focus on having and raising children. You might want to do that. Or not.

       ---

That said, it's pretty rude to expect other people to focus their life on a project instead of their families. However, I can understand that a leader desire that their company be formed of people likely focused and minded.

</comment><comment><USER41> Whatever the truth of the matter I think this highlights a specific thing people should pay attention to when they choose their employer: different companies have different values and when considering joining, it's good to make sure those values are aligned with ones personal values. Having a strongly disaligned value base can lead to burnout and depression.

Tesla values seem to be in the extreme end of the spectrum. I am not saying that this is good or bad, just that they are not necessarily aligned with the values of all.

But all in all, having a car is pretty pointless if there is no-one to drive them. Hiring clever people and then hindering their pro-creation is not really helping the society as talented parents usually get talented children, and for children the most important time of their development is their first years. Screw those, and the person is considerably worse off. A healthy child might get well of and by a fancy Tesla in 30 years. I'm not saying slave laboring the father would make the infant a mental patient and a destitute, but discouraging healthy parenthood is hedging the bets towards worse outcomes.

The same argumentation can be said of space travel. What's the point of traveling to Mars unless the needs of future of the species are taken care of as well. This includes resources for nurture and care.

</comment><comment><USER42> Imagine a world where that level of insensitivity was inextricably coupled with the sort of accomplishments he produces for humanity.

I'm not saying they are, but if asshole-ness and makes-awesomethings-ness were linked, would it be preferable for him to be "a nice guy" and not "a guy who drove plausible solutions to global online payments, solar power, electric cars, and space travel"?

Or would we rather have another nice guy that didn't do those things?

It's hard to imagine that anyone who continues to work for Elon Musk's companies right now doesn't have alternatives. So they're _CHOOSING_ that, for whatever reason. Maybe they tolerate his behavior to be a part of what he does. But nobody is really trapped there without options.

Maybe (probably) it's possible to be nice and be awesome. But it's not important to me that Elon Musk is nice. If I have to pick between his creation and his compassion, I pick creation.

</comment><comment><USER43> Father of two kids I will not trade the moment of my kids birth for anything in the world. As for Mr Elon Musk, go home... you are another psycho CEO from america crime corporate

</comment><comment><USER44> For possible context, Elon Musk's first wife wrote about their relationship during the midst of her divorce[1].

[1] [http://www.marieclaire.com/sex- love/advice/a5380/millionaire...](http://www.marieclaire.com/sex- love/advice/a5380/millionaire-starter-wife/)

</comment><comment><USER45> If true that would be hard to not shape my perspective of the man. I can't respect the Daniel Plainviews of the world.

</comment><comment><USER46> People will justify this sort of abuse because they see it as a necessary cost for a bright future.

Do you really think we'll create a bright future by continually setting the precedent it's ok to abuse people to get there? Won't tomorrow's leaders simply use their own visions of the future to justify their own abuse?

</comment><comment><USER47> Well, isn't Elon Musk one of those guys who thinks he's going to live forever? He probably figures he can spend time with family 1000 years from now after things settle down a bit.

<comment><USER6> He said he plans to die on Mars ("just not on landing"). I guess he plans to do it being happy that the civilization is no longer facing imminent collapse because of energy and climate problems, and that humans are multiplanetary species.

</comment></comment><comment><USER48> I'm changing the world. I'm changing history. I'm fully committed. His name is Theodore and he turns three in July.

</comment><comment><USER49> There's not enough information for a fair opinion on the matter, so I flagged the article. It's not the first time that _ambiguous_ sources try to smear Elon Musk. The ex-wife rant was enough.

There are so many variables we are not aware of. Dismissing Elon as a "the bad guy" is giving too much credit to an unknown source.

</comment><comment><USER50> There's a constructive dismissal lawsuit if I've ever seen one.

</comment><comment><USER51> Is it me or is this post disappearing off the front page rather quickly?

<comment><USER6> It's probably being flagged, because with its lack of context, it's more of a smearing job than an actual revelation.

</comment></comment><comment><USER52> John Chambers seems nice, and Cisco did okay.

</comment><comment><USER53> Changing the world through government grants, subsidies, and prison labor...

</comment><comment><USER18> > That is no excuse. I am extremely disappointed. You need to figure out where > your priorities are. We're changing the world and changing history, and you > either commit or you don't.

What an idiot - he became rich - so what ...

</comment><comment><USER54> 1) If true, NONE of us here know under what circumstances this happened. Maybe Elon hadn't slept for days at the time and later apologized, or maybe the employee was really at fault. NONE OF US KNOW.

2) Some of the quotes from the book make me think the author merely went through other people's interviews/articles and compiled them into a sensationalist book.

<comment><USER55> While this is true, it's also worth nothing that HN places Elon Musk on a pedestal and becomes incredibly defensive when someone criticizes him.

The reality is probably somewhere in the middle: Elon Musk is no doubt an amazing man, but he is not without flaws. I think we do need people pointing out those flaws to balance out all the hype and hero-worship.

<comment><USER54> Agreed. I just get angry when people think it's a good idea to chime in on "gossip". This isn't Gawker.

</comment></comment></comment><comment><USER56> All the negative press and comments about this miss the point. The guy has very specific, very ambitious goals that he has a track record of achieving. He prioritizes those literally above everything and expects the same from key employees.

Working at Tesla, which has already done a ton to change the entire car industry is a huge privilege, and no one's making you do it. If you can get a job there, you can get a job elsewhere, but if you do want to have your job there and help reshape the world behind someone with a track record like Elon Musk's, yeah, you're going to have to play by his rules. If other stuff is more important to you, go work at like 95% of other companies.

<comment><USER57> I am reminded of this famous Stallman incident, where he basically scorns at another developer for having a baby:

[http://edward.oconnor.cx/2005/04/rms](http://edward.oconnor.cx/2005/04/rms)

<comment><USER6> Which reminds me of this RMS classic:

[http://www.art.net/Studios/Hackers/Hopkins/Don/text/rms- vs-d...](http://www.art.net/Studios/Hackers/Hopkins/Don/text/rms-vs- doctor.html)

I can't say I disagree with RMS. Having a child is really nothing special. Every other person on the planet does that. And while I respect that for many parents, having their own kid is the best moment of their lives, I think RMS is right to point out that except said parents, no one else cares. It's a huge problem on social media now; one of the most annoying things is people continuously spamming everyone with photos of their newborns.

<comment><USER37> _You_ might not care, RMS might not care, but please don't assume that "no one else cares". I don't know you, and I don't know RMS, but he doesn't seem like the most emotionally intelligent person, perhaps you aren't either. Nothing wrong with that, you are who you are.

Some people are capable of responding very strongly to the emotions of others. I'm not saying they're better than you, just that they're different to you, and that they exist :)

<comment><USER6> Ok, I might overstated it for dramatics, but what I'm complaining about is that being considerate should go both ways. I'll be happy for you that you have a kid and congratulate you, but please don't assume that it's in any way as important for me as it is for you.

It reminds me a thing my friend told me once, that she sometimes wonder whether if she'll have a kid, will she be as obnoxious about it on Facebook as many of her friends.</comment></comment></comment></comment></comment>