Post-Mortem for Google Compute Engine's Global Outage on April 11 - <AUTHOR> https://status.cloud.google.com/incident/compute/16007?post-mortem <comment><USER2> This is a very good Post-Mortem.

As I assumed it was kind of a corner case bug meet corner case bug met corner case bug.

This is also why I am of afraid of a self driving cars and other such life critical software. There are going to be weird edge cases, what prevents you from reaching them?

Making software is hard....

<comment><USER3> Self driving cars don't have to be perfect. They just have to be safer then driving is today [1].

The real question is if society can handle the unfairness that is death by random software error vs. death by negligent driving. It's easy to blame negligent driving on the driver, we're clearly not negligent so it really doesn't effect us right? But a software error might as well be an act of god, it's something that might actually happen to me!

[1]: [https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_i...](https://en.wikipedia.org/wiki/List_of_motor_vehicle_deaths_in_U.S._by_year)

<comment><USER4> Well No, There is an upper limit on the damage a bad driver can do by say crushing his car with a bus or something like that. Imagine a bug or malware triggered at the same moment world-wide. It could kill millions. So it not as simple as 'It just has to be better than a human'

<comment><USER5> I've been itching to release this terror movie plot into the wild:

 _It 's 2025 and more than 10% of the cars on the road in the US are self- driving. It's rush hour on a busy Friday afternoon in Washington, DC. Earlier that day, there'd been a handful of odd reports of self-driving Edsels (so as not to impugn an actual model) going haywire, and the NTSB has started its investigation._

 _But then, at 430pm, highway patrol units around the DC beltway notice three separate multi-Edsel phalanxes, drivers obviously trapped inside, each phalanx moving towards the Clara Barton Parkway, which enters DC from the west. Other units notice four more phalanxes, one comprising 20 Edsels, driving into DC from the east side, on Pennsylvania Avenue._

 _At this point, traffic helicopters see similar car clusters, more than two dozen, all over DC, all converging on a spot that looks to be between the Washington Monument and the White House._

 _We zoom in on the headquarters of the White House Secret Service. A woman is arguing vociferously that these cars have to be stopped before they get any closer to the White House. A colleague yells back that his wife is one of those commandeered cars and she, like the rest of the "hackjacked" drivers and passengers is innocent._

<comment><USER6> I'm afraid Daemon (novel) beat you to the punch. It's an excellent novel, about fairly similar situations.

[http://www.goodreads.com/book/show/6665847-daemon](http://www.goodreads.com/book/show/6665847-daemon)

<comment><USER7> And a really fun read. If someone here decides to get this book, get the sequel as well - Daemon ends on kind of a cliff-hanger.

<comment><USER8> Freedom (TM) is the sequel, and the author (Daniel Suarez) has a few other near-term what-if-this-all-goes-skynet books which are equally good.

<comment><USER9> Thank you! I didn't know there was a sequel. I really enjoyed the first book.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> > There are a number of lessons to be learned from this event -- for example, > that the safeguard of a progressive rollout can be undone by a system > designed to mask partial failures -- ...

This is a really important point that should be more generally known. To quote Google's own "Paxos Made Live" paper, from 2007:

> In closing we point out a challenge that we faced in testing our system for > which we have no systematic solution. By their very nature, fault-tolerant > systems try to mask problems. Thus they can mask bugs or configuration > problems while insidiously lowering their own fault-tolerance.

As developers we can try to bear this principle in mind, but as Monday's incident demonstrated, mistakes can still happen. So, has anyone managed to make progress toward a "systematic solution" in the last 9 years?

<comment><USER11> Google actually does have a systematic solution: fault injection. Google's systems are designed so that you can (manually, if you have the right privileges) tell an RPC to fail regardless of whether it would otherwise have succeeded, and then test the response of the system as a whole.

The problem is that these failure cases are exercised much less frequently than the "normal execution" code paths are. For example, every year Google does DiRT [1] exercises which test system responses to a large calamity, eg. a California earthquake that kills everyone in Mountain View and SF including the senior leadership, and also knocks out all west coast datacenters. The half-life of code at Google (in my observation) is roughly 1 year, which means that half of all code has never gone through a DiRT exercise. The same applies to other, less serious fault injection mechanisms: they may get executed once every year or two, and serious bugs can crop up in the meantime. Automated testing of fault injection isn't really feasible, because the number of potential faults grows combinatorially with the number of independent RPCs in the system.

I'd be willing to bet that the two bugs that caused this outage were less than 6 months old. In my tenure at Google, the vast majority of bugs that showed up in postmortems were introduced < 3 months before the outage.

[1] [http://everythingsysadmin.com/2012/09/devops-google- reveals-...](http://everythingsysadmin.com/2012/09/devops-google-reveals- their-di.html)

<comment><USER12> Testing doesn't detect failure, it only detects the failure of a test. Real failures happen more often than test failures, for the same test on the same code with the same input and output. The best systematic solution would detect real failures, not see what happens when you fail a test.

<comment><USER11> That's monitoring, then. As Steve Yegge's Platforms Rant [1] mentioned, testing and monitoring are two sides of the same coin. Google does both, but the original thread-starter here was asking about how to detect failures when the system itself is designed to mask & recover from failures. (FWIW, most such systems do log when they've encountered a failure condition and recovered from it, and this stat is available to the monitoring system.)

[1] [https://plus.google.com/+RipRowan/posts/eVeouesvaVX](https://plus.google.com/+RipRowan/posts/eVeouesvaVX)

<comment><USER12> Basically, yes. But we don't have to make a traditional monitor, or have it be an extra component. Monitoring all the facets of, say, a code deployment, or a software build, or performance testing, is a dynamic thing. It may fail, or it may succeed, or it might be _suspicious_.

Normally we design systems for humans to determine that 3rd part; in this case, there should have been a system where humans could see the one or two pieces of unusual activity and investigated. But there wasn't, or it didn't work right. So a "fix" would be to develop software that adapts to nondeterministic behavior the way a human does. I wouldn't exactly call that monitoring, though.

</comment></comment></comment></comment></comment><comment><USER13> Great writeup! PS: "To make error is human. To propagate error to all server in automatic way is devops." -DevOps Borat

</comment><comment><USER14> It looks like there were at least three catastrophic bugs present:

1\. Evaluated a configuration change before the change had finished syncing across all configuration files, resulting in rejecting the change.

2\. So it tried to reject the change, but actually just deleted everything instead.

3\. Something was supposed to catch changes that break everything, and it detected that everything was broken but its attempt to do anything to fix it failed.

It is hard to imagine that this system has good test coverage.

<comment><USER15> I'm attempting to even imagine how one would build a useful way to test this. Would they have to have a secondary, world-wide datacenter network with all their various services behind it?

<comment><USER16> While testing would have been quite difficult, any simple canary release or timed release mechanism would have prevented this / limited the damage. At such mission critical systems, applying any global change in a such manner is asking for it, Devops can also be SPOF, this seems one such case.

<comment><USER17> They had a canary release mechanism in place. This is described in the post mortem.

> These safeguards include a canary step where the configuration is deployed > at a single site and that site is verified to still be working correctly, > and a progressive rollout which makes changes to only a fraction of sites at > a time, so that a novel failure can be caught at an early stage before it > becomes widespread. In this event, the canary step correctly identified that > the new configuration was unsafe. Crucially however, a second software bug > in the management software did not propagate the canary step's conclusion > back to the push process, and thus the push system concluded that the new > configuration was valid and began its progressive rollout.

Taking no cofirmation of the canary testing process as a signal to go ahead though is not just a bug but a design flaw IMO.

</comment></comment></comment></comment><comment><USER18> _In this event, the canary step correctly identified that the new configuration was unsafe. Crucially however, a second software bug in the management software did not propagate the canary step's conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout._

Classic Two Generals. "No news is good news," generally isn't a good design philosophy for systems designed to detect trouble. How do we know that stealthy ninjas haven't assassinated our sentries? Well, we haven't heard anything wrong...

<comment><USER19> It may not be good design, but it might be necessary / practical design. If you have enough machines that some percentage of them are down or unreachable at any given time, you can't wait for full go-ahead before proceeding; you'll never get full go-ahead. So you're left with probabilistic solutions, and as T approaches infinity the expectation of more than zero false-positives approaches 1.

<comment><USER18> The whole point of the canary sub-population, though is that 1) It's not your whole population. 2) You want to find out empirically if something's wrong.

</comment></comment></comment><comment><USER20> I'm waiting for the time when they push over the air updates to airplanes in flight.

"You can fly safely, we have canaries and staged deployment"

A year forward:

"Unfortunately because the canary verification as well as the staged deployment code was broken, instead of one crash and 300 dead, an update was pushed to all aircraft, which subsequently caused them to crash, killing 70,000 people."

I'm not 100% sure why they don't do the staged deployment for google scale server networking over a few days (or even weeks in some cases) instead of a few hours, but I don't know the details here...

It's good that they had manually triggerable configuration rollback possibility and a pre-set policy so it was solved so quickly.

<comment><USER21> Yeah ,the part with canary code rub me the wrong way too.

 _These safeguards include a canary step where the configuration is deployed at a single site and that site is verified to still be working correctly_

This sounds very unprofessional imho. "Touch this cable to see if there is electricity running" sort of thing.

Is that really how its should be done?

<comment><USER22> If you're doing electrical work, _eventually_ you're going to have to touch the cable!

</comment></comment></comment><comment><USER23> At Google, they do these really awesome post-mortems when there's a major failure. It provides a point of reflection, and are usually well written entertaining reads. Didn't know they made (some?) public.

They're a good learning exercise writing one, and is more of a learning exercise than a punishment.

<comment><USER24> It's worth noting that the publicly posted postmortem is not the same as the internal postmortems (which include much more detail, specific action items, timelines etc). The SRE book ([https://landing.google.com/sre/book.html](https://landing.google.com/sre/book.html)) has a whole chapter on our internal postmortems, which is probably a better learning exercise in how to write one.

Source: I work on the team that writes these external postmortems.

</comment></comment><comment><USER25> Completely off topic, but this thread is an example of why I (and a lot of people) want collapsible comments native to HN. I'm on my phone, in Safari, and I had to scroll for over 20 seconds just to reach the second comment. The first comment was a tangent about self-driving cars, which while relevant, I didn't want to read about.

<comment><USER26> Especially considering that nearly every post on HN features an often tangential first comment that goes on and on and on...

</comment></comment><comment><USER27> >However, in this instance a previously-unseen software bug was triggered, and instead of retaining the previous known good configuration, the management software instead removed all GCE IP blocks from the new configuration and began to push this new, incomplete configuration to the network.

>Crucially however, a second software bug in the management software did not propagate the canary step's conclusion back to the push process, and thus the push system concluded that the new configuration was valid and began its progressive rollout.

I assume the software was originally tested to make sure it works in case of failure. It would be interesting to know exactly what the bug was and why it didn't show in tests.

<comment><USER28> Network management software complexity is supposed to be one of things that SDN was built to solve (by introducing more modularity and defined interfaces). But in this case the fault was at the edge with BGP route updates, which the internet has been doing for decades. I share your curiosity in the specific bug.

However, this is a great detailed post-mortem from a service provider. Your Telco or ISP will never provide this much detail...

</comment></comment><comment><USER29> Attention startups: _this_ is what incident post-mortems should look like.

<comment><USER30> Many startups probably don't have anywhere near the same level of SLA nor revenue of GCE.

<comment><USER29> Of course. You should still have a goal to strive towards.

</comment></comment></comment><comment><USER31> This is very interesting. From the little I understand (sorry for using AWS terms as I am more versed with AWS than GCE) this can happen to AWS as well right? even if your software is deployed to multiple AZs / multiple regions, if bad routing / network configuration makes it through the various protection mechanisms then basically no amount of redundancy can help if your service is part of the non functional IP block. I mean it seems no matter how redundant you are, there will always be somewhere along the line a single point of failure, even if it has multiple mechanism to prevent it from happening, if all of these mechanisms fail, then it's still a single point. What prevents this from happening at Azure / AWS? Is there anything that general internet routing protocols need to change to prevent it from happening?

e.g. I'm sure that we will never hear that Bank of X has transferred a billion dollar to an account but because of propagation errors it published only the credit but didn't finish the debit and now we have two billionaires. This two or more phase commit is pretty much bulletproof in banking as far as I know, and banks are not known to be technologically more advanced than Google, how come internet routing is so prone to errors that can an entire cloud service unavailable for even a small period of time? I'm far from knowing much about networking (although I took some graduate networking courses, I still feel I know practically nothing about it...) So I would appreciate if someone versed in this ELI5 whether it can happen in AWS and Azure regardless of how redundant you are, (which leads to a notion of cross cloud provider redundancy which I'm sure is used in some places) and whether the banking analogy is fair and relevant, and if there are any RFCs to make world-blackout routing nightmares less likely to happen.

<comment><USER32> I'm not sure the AWS network follows the same setup, AWS has very distinct blocks between the US/EU/APAC compared to GCP where you can inherit the same IP if you quickly delete/recreate instances in different regions?

<comment><USER33> I was going to post the same comment too.

My understanding, from the odd bits and bobs of information I have, is that AWS regions are typically managed somewhat independently.

</comment></comment></comment><comment><USER30> > . Internal monitors generated dozens of alerts in the seconds after the > traffic loss became visible at 19:08 ... revert the most recent > configuration changes ... the time from detection to decision to revert to > the end of the outage was thus just 18 minutes.

It's certainly good that they detected it as fast as they did. But I wonder if the fix time could be improved upon? Was the majority of that time spent discussing the corrective action to be taken? Or does it take that much time to replicate the fix?

<comment><USER34> Having worked in ISP operations on BGP stuff (admittedly more than 10 years ago), it was both too slow and too fast.

If the rollout took 12 hours instead of 4 or the VPN failure to total failure was multiple hours instead of minutes, they'd have had enough time to noodle it out. Eventually at a slow enough deploy rate they'd have figured it out. It only took 18 hours to make the final report after all, so an even slower 24 hour deploy would have been slow enough, if enough resources were allocated.

On the opposite side, most of the time when you screw up routing the punishment is extremely brutal and fast. If the whole thing croaked in five minutes, "OK who hit enter within the last ten minutes..." and five minutes later its all undone. What happened instead was dude hit enter, all is well hours later although average latency was increasing very slowly as anycast sites shut down. Maybe there's even shift change in the middle. Finally hours later it finally all hit the fan meanwhile the guy who hit enter is thinking "it can't be me, I hit enter over four hours ago followed by three hours of normal operation... must be someone else's change or a memory leak or novel cyberattack or ..."

Theoretically if you're going to deploy anycast you could deploy a monitoring tool to traceroute to see that each site is up, however you deploy anycast precisely so that it never drops... Its the titanic effect, why this is unsinkable, why would you bother checking to see if its sinking? And just like the titanic if you break em all in the same accident, that sucker is eventually going down, even if it takes hours to sink.

<comment><USER28> Hmm. Seems like this begs for a different way to solve the problem, like alarming on major changes to configuration files or better recognition of invalid configs, i.e. google should be able to make a rule that says "if I ever blackhole x% of my network then alarm"...

<comment><USER34> The first one is alarm fatigue. Like the "Terror Thermometer" or whatever its called where we're in eternal mauve alert meaning nothing to anyone. All our changes are color coded as magenta now. Or its turned down such that one boring little ip block isn't a major change. After all, it isn't. Of course you (us) developers could run crazy important multinational systems on what to us networking guys was one boring little IP block who cares about such as small block of space.

The second one is covered in the article, their system for that purpose crashed and then the system that babysits that crashed and then whatever they use to monitor the monitors monitor system didn't notice. Probably showed up in some dude's nightly syslog file dump the next day. Oh well. If your monitor tool breaks due to complexity (as they often do) it needs to simplicate and add lightness not slather more complexity on. Usually monitoring is more complicated and less reliable than operating, its harder computationally and procedurally to decide right from wrong than to just do it.

The odds of cascaded failure happening are very low. Given fancy enough backup systems that means all problems will be weird cascaded failure modes. That might be useful in training.

When I was doing this kind of stuff I was doing higher level support so see above at least some of my stories are weird cascaded impossible etc. A slower rollout would have saved them, working all by myself I like to think I could have figured it out by comparing BGP looking glass results and traceroute outputs from multiple very slowly arriving latency reports to router configs with papers all over my desk and multiple monitors in at most maybe two days. Huh, its almost like anycast isn't working at more sites every couple hours, huh. Of course their automated deployment is complete in only 4 hours, which means all problems that take "your average dude" more than 4 hours of BAU time to fix are going to totally explode the system and result in headlines instead of a weird bump on a graph somewhere. Given that computers are infinitely patient, slowing down the rollout of automated deployments from 4 hours to 4 days would have saved them for sure. Don't forget that normal troubleshooting places will blow the first couple hours on written procedures and scripts because honestly most of the time those DO work. So my ability to figure it out all by myself in 24 hours is useless if the time from escalation to hit the fan was only an hour because they roll out so fast. Once it hit the fan a total company effort fixed it a lot faster than I could have fixed it as an individual.

Or the strategy I proposed where computers are also infinitely fast, roll out in five minutes, one minutes to say WTF, five minutes to roll back, 11 minute outage is better than what they actually got. Its not like google is hurting for computational power. Or money.

I'm sure there are valid justifications for the awkward four hour rollout thats both too fast and too slow. I have no idea what they are, but the google guys probably put some time into thinking about it.

</comment></comment></comment></comment><comment><USER35> > Finally, to underscore how seriously we are taking this event, we are > offering GCE and VPN service credits to all impacted GCP applications equal > to (respectively) 10% and 25% of their monthly charges for GCE and VPN.

These credits exceed what is promised by Google Cloud in their SLA's for Compute Engine and VPN service!

<comment><USER36> ... which is precisely (almost word-for-word) what the post-mortem goes on to say. Is there something specific you're trying to call attention to here?

<comment><USER35> Nop. Probably did too much copy-pasting :( Mearly wanted to highlight the point.

</comment></comment></comment><comment><USER37> Nice post mortem.

That outtage gives GCE at best a four 9's reliability for 2016.

<comment><USER38> Based on the higher level status page:

[https://status.cloud.google.com/summary](https://status.cloud.google.com/summary)

It looks like GCE uptime is _well_ below four 9's reliability for a sliding 1 year timeframe.

<comment><USER39> Traynor was quoted in a networkworld article last year saying they aim for three and a half nines (99.95%). But you need to read into the incidents more carefully -- figuring out actual "uptime" is quite hard. Consider the longest- lasting incident:

      "On Tuesday 23 February 2016, for a duration of
       10 hours and 6 minutes, 7.8% of Google Compute Engine
       projects had reduced quotas.  ...  Any resources that
       were already created were unaffected by this issue."

I'm not sure off the top of my head how I'd try to compute the overall availability #s from that one. One can possibly try to determine and sum the effects on the individual customers, but we can't from the information provided. But it's certainly less overall downtime than just counting it as a 7 hour failure.

<comment><USER38> Agreed. It is difficult to tell. But if the bug is preventing you from processing (because you can't save the existing results) then it's essentially down time for new processing. There are also connectivity issues by region and DNS issues. It is difficult to get exact downtime considering partial failures.

That said, this is the second major asia-east1 downtime in 90 days:

[https://status.cloud.google.com/incident/compute/16002](https://status.cloud.google.com/incident/compute/16002)

</comment></comment></comment></comment><comment><USER40> I always like Google's serious attitude towards engineering, even after they have made some mistakes, they never try to hide anything.

</comment><comment><USER41> > However, in this instance a previously-unseen software bug was triggered, > and instead of retaining the previous known good configuration, the > management software instead removed all GCE IP blocks from the new > configuration

> Crucially however, a second software bug in the management software did not > propagate the canary step's conclusion back to the push process

I'm sure the devil is in the details, but generally speaking, these are 2 instances of critical code that gets exercised infrequently, which is a good place for bugs to hide.

</comment><comment><USER42> Do SLAs even matter in the slightest? Or are they just sort of "feel-good" things or ways for negotiators to demonstrate their worth?

<comment><USER36> SLAs aren't about guaranteeing uptime. They're about setting consequences for downtime.

<comment><USER43> But once there are strong consequences for downtime the service provider is going to set up training, monitoring, oncall, etc to make sure things stay within the SLA limits. So you are effectively negotiating uptime.

</comment></comment></comment><comment><USER44> "Lessons learned from reading post-mortems" [http://danluu.com/postmortem- lessons/](http://danluu.com/postmortem-lessons/) is a good place to dig deeper

The first graph quoted from a survey paper is a classic fitting the GCE outage well:

Initial error --92%--> Incorrect handling of errors explicitly signaled in software

<comment><USER45> [https://mitpress.mit.edu/books/engineering-safer- world](https://mitpress.mit.edu/books/engineering-safer-world) is also an excellent resource that more people who care about post-mortems should read.

(As background, the author, MIT Prof. Nancy Leveson, summarizes decades of work in the field, offers groundbreaking new theoretical tools that scale up to some of the world's most complex accidents, and has the experience and evidence to back up their relevance e.g. via work on Therac-25, the Columbia Space Shuttle, and Deepwater Horizon to name just a few...)

</comment></comment><comment><USER46> I love his signature: "Benjamin Treynor Sloss | VP 24x7".

</comment><comment><USER47> > However, in this instance a previously-unseen software bug was triggered, > and instead of retaining the previous known good configuration, the > management software instead removed all GCE IP blocks from the new > configuration and began to push this new, incomplete configuration to the > network.

Always test your crash / exception handling / special case termination+recovery code in production.

I have seen this too often. Most often in in "every day" cases when service has a "nice" catch way of stopping and recovering. Then has a separate "if killed by SIGKILL/immediate power failure" crash and recovery. This last bit never gets tests and run in production.

One day power failure happens, service restart and tries to recover. Code that almost never runs, now runs and the whole thing goes into an unknown broken state.

<comment><USER48> See [https://en.wikipedia.org/wiki/Crash- only_software](https://en.wikipedia.org/wiki/Crash-only_software)

</comment></comment><comment><USER49> This isn't the first time a config system at Google causes a major outage.

[https://googleblog.blogspot.com/2014/01/todays-outage-for- se...](https://googleblog.blogspot.com/2014/01/todays-outage-for-several- google.html)

<comment><USER50> That's entirely unsurprising. The recent major Facebook outage was also caused by bad configuration IIRC.

See: [http://danluu.com/postmortem-lessons/](http://danluu.com/postmortem- lessons/)

> Configuration > > Configuration bugs, not code bugs, are the most common > cause > I've seen of really bad outages. When I looked at publicly available > > postmortems, searching for "global outage postmortem" returned > about 50% > outages caused by configuration changes. Publicly > available postmortems > aren't a representative sample of all > outages, but a random sampling of > postmortem databases also > reveals that config changes are responsible for > a disproportionate > fraction of extremely bad outages. As with error > handling, I'm > often told that it's obvious that config changes are scary, > but > it's not so obvious that most companies test and stage config > > changes like they do code changes.

<comment><USER51> Great link there! Also check out his list of public postmortems at [https://github.com/danluu/post-mortems](https://github.com/danluu/post- mortems)

PS. On HN you should use asterisks to _italicize_ instead of > for quoting.

</comment></comment></comment><comment><USER52> My post yesterday seems even more relevant today: [https://news.ycombinator.com/item?id=11477552](https://news.ycombinator.com/item?id=11477552)

It's a shame it's not easier or more common for people to create clones of (most|all) of their infrastructure for testing purposes.

Something like half of outages are caused by configuration oopsies.

If you accept that configuration _is_ code, then you also come to the following disturbing conclusion: the usual test environment for critical network-related code in most environments is the production environment.

<comment><USER53> The main issue there is that "environments" are defined by configuration, so if you try to set up a configuration test environment, you run into a direct logical impass: either your configs are production configs, and thus not a separate environment, or they're different from production configs, and thus may provide different test results from production.

<comment><USER52> While I agree with you, I think we could get closer to "production" than is common right now.

In an AWS environment, imagine a setup where all that differs is the API keys used (the API keys of the production vs test environment). What gets tricky is dealing with external dependencies, user data, and simulating traffic.

For an example more relevant to today's issue: imagine a second simulated "internet" in a globally distributed lab environment. With BGP configs, fake external BGP sessions, etc, servers receiving production traffic, etc.

I get that it's a lot of work to setup and would require ongoing work to maintain - and that it's hard/impossible to have it correctly simulate the many nuances of real world traffic - and yet I also think in many cases it would be sufficient to prevent issues from making it into production.

</comment></comment></comment><comment><USER54> For the amount this cost them, they should have bought CloudFlare. If you play with [global BGP anycast] you are bound to get burned. This is not the first time that BGP took out your entire routing. This is probably not the last time that BPG will take out your entire routing. Whoever's job it was to watch the routing, I am sorry.

Pulling your own worldwide routes because you have too much automation; it will make a good story once it's filtered down a bit! Icarus was barely up in the air, too early for a fall.

</comment><comment><USER55> The thing that stood out for me was:

"...team...worked in shifts overnight..."

<comment><USER56> (Usual disclaimer: I speak for myself, not for my employer, etc.)

The team in charge of solving this particular problem is located in two sites in two different timezones. This is true of most critical SRE teams at Google, and it is precisely to be able to have 24h coverage in these time sensitive situations.

In the 2+ years I have spent in SRE I have never heard of a single instance of an SRE being asked or even encouraged to stay after hours (let alone overnight) for incident remediation. There is quite a lot of emphasis being put on work/life balance.

<comment><USER48> Wow, that's amazing to read, having served as a de-facto SRE (like every other SDE) at an unnamed competitor to GCE, where I was expected to stay up all night if necessary to resolve an issue (relatively few teams had follow-the- sun coverage). I swore I would never carry a pager again after that, but maybe Google really is different.

</comment></comment></comment><comment><USER57> How important for redundancy/quality of service is the feature of advertising each region's IP blocks from multiple points in Google's network? It seems like region isolation is the most important quality that Google's network could provide, and their current design is what made something like this possible, not just the bugs in the configuration propagation. They mention the ability of the internet to route around failures, so why not rely on that instead?

</comment><comment><USER58> as devops Borat was saying all along, automated propagation of a error as the main root cause here. A error (new configuration) should be rolled out site by site - ok us-east1, move onto us-west1 ... ok, move onto ... . A canary site may be the first in sequence, yet success ("no failure reported") can't be a big "ok" for automated push to all sites at the same time.

</comment><comment><USER59> I hope that one of their solutions is the obvious one; make change control testing a closed loop instead of an open loop. (Watch for /success/ reported instead of failure notification.)

</comment><comment><USER60> > configuration file

configuration files strike again - remember knight capital?

</comment><comment><USER61> What does Google use for BGP? Quagga, OpenBGPD, BIRD, their own?

Also, does anyone have a link to statistics on global BGP software usage? I'm curious what the marketshare looks like.

<comment><USER62> Google has contributed ISIS and BGP code to Quagga in the past, as well as funding some testing at the OSRF. Presumably they use it in at least some parts of their operations.

</comment></comment><comment><USER63> The postmortem used the word "quirk." They might consider drilling down on the specifics there. Especially if that is the heart of the bug/accident.

</comment><comment><USER64> Just a thought. Maybe change the name from 'post-mortem' to, anything else before the event actually is a post-mortem.

</comment><comment><USER65> Networking issues in either the storage or communication subsystems of any platform normally result in wide-spread disruptions.

</comment><comment><USER66> What is the reason different GCE regions use the same IP blocks?

</comment><comment><USER67> What is defense in depth? It is mentioned as a core principle.

<comment><USER68> [https://en.wikipedia.org/wiki/Defense_in_depth_(computing)](https://en.wikipedia.org/wiki/Defense_in_depth_\(computing\))

</comment></comment><comment> awinter-py chaos monkey?

</comment><comment><USER69> > Crucially however, a second software bug in the management software did not > propagate the canary step's conclusion back to the push process, and thus > the push system concluded that the new configuration was valid and began its > progressive rollout.

Perhaps the progressive rollout should wait for an affirmative conclusion instead of assuming no news is good news? I'm not being snarky, there may be some reason they don't do this.

<comment><USER70> Presumably it received a false positive (or it was interpreted as such). This really seems like the root cause, and I suspect a case of happy path engineering striking again.

</comment></comment><comment><USER51> TLDR; they simply didn't test their (global!) custom route announcement management software. An edge case was triggered in production, and they gee- whiz-automatically went offline. Epic fail.

PS. To the downvoters, truth hurts.

<comment><USER71> I think you're getting downvoted due to the snarky tone more than any "truth" you are stating.

<comment><USER51> Well, how to phrase the same thing briefly without sounding snarky?

<comment><USER72> You only need to change a few words:

"In other words, they simply didn't test their (global!) custom route announcement management software. An edge case was triggered in production, and unsurprisingly they automatically went offline."

<comment><USER51> There is no accounting for taste.

</comment></comment></comment></comment></comment><comment> herrvogel- A bit of topic, but it really bugs me, that the banner on the top so pixilated.

</comment><comment><USER73> DRY "The inconsistency was triggered by a timing quirk in the IP block removal - the IP block had been removed from one configuration file, but this change had not yet propagated to a second configuration file also used in network configuration management."

<comment><USER48> Yes, DNS was clearly designed by idiots who had never heard of DRY.

</comment></comment><comment><USER74> I think most people are missing the main failure point: Why does one change propagate automatically to all regions?

All this could have been contained if they deployed changes on different regions at different times. That would also help with screwing less your overseas users by running a maintenance at 10am their local time :-)

<comment><USER53> > These safeguards include a canary step where the configuration is deployed > at a single site and that site is verified to still be working correctly, > and a progressive rollout which makes changes to only a fraction of sites at > a time, so that a novel failure can be caught at an early stage before it > becomes widespread. In this event, the canary step correctly identified that > the new configuration was unsafe. Crucially however, a second software bug > in the management software did not propagate the canary step's conclusion > back to the push process, and thus the push system concluded that the new > configuration was valid and began its progressive rollout.

The system _does_ do progressive rollouts, which are essentially what you are referring to (albeit perhaps at a different pace). The number of changes being rolled out means that it's not really feasible to hand roll out configurations to different regions, so the checks are automated. In this case, the automated checks failed as well.

<comment><USER48> Waiting a longer time between regional rollouts (so monitoring systems would have time to detect serious failures) would sacrifice deployment latency, but not deployment throughput (assuming deployments can be made in parallel). For continuous deployment, throughput really matters more than latency.</comment></comment></comment>


Rackspace Goes Down. Again. Takes The Internet With It. Again. - <AUTHOR> http://www.techcrunch.com/2009/12/18/rackspace-down/ <comment><USER3> I can't believe how TechCrunch is turning into a tech-tabloid.

Yes, there are issues (perhaps sometimes serious) that needs reporting - but - the way it is done by TC isn't quite the quality I'm excepting.

<comment><USER4> I agree with you but I'd bet those are TechCrunch's most popular posts. In fact, the author (MG Siegler) was essentially hired because he's good at writing those types of post.

People like trashy journalism. It's like the Tiger Woods thing. I don't mind the networks reporting on it but the fact that it's been the top item on "legitimate" news sites should tell you how modern day journalism works.

</comment></comment><comment><USER5> Amazon just started looking a whole lot more attractive. They may not have such fanatical support, but they do know how to fix their stuff.

<comment><USER6> Well, Rackspace knows how to fix their stuff too, which is why it's up again after the problem appeared. ;-)

</comment></comment><comment><USER7> Attention Internet start-ups: operations is your core competency. You can't just expect to push your application code to "the cloud" and have somebody else make it scalable and fault tolerant.

It's fine to use a managed hosting service provider if you're _just getting started_ and paying month-to-month based on credit cards and can't afford networking gear (hardware itself can be leased). However, it shocks me to see venture funded, post-series A start-ups _exclusively_ relying on others to do their operations (including systems administration).

Problem with outsourcing your operations to somebody else is that you're outsourcing it to somebody who has _zero_ knowledge of your application and is also responsible for at the very least _dozens_ of other customers. Essentially, rather than developing _your own_ vertical technology team, you're relying on a on horizontal technology team whose resources multiple other companies (including your competitors) are fighting for.

That's _exactly_ how poorly run big companies function (multiple engineering teams competing for attention of monothlic operations, SCM, release, QA etc... teams). Well run Internet giants, however, function much differently. If you look at Google's job openings, you can note that they don't hire IT/Systems Administrators (aside from data-center technicians and corporate IT). Instead, they have Site Reliability Engineers (SREs) assigned to specific properties. The SREs are actually engineers who are able to write and debug code and deeply understand the application-specific stack they operate rather than treat it as a black box.

Sure, there are great business and technical reasons (edit: it said financial, which I felt was an unfair strawman) reasons to use a managed/"cloud" provider (EC2 and Rackspace cloud are very attractive due to the ability to add/remove nodes as traffic goes up and down, as well as provision machines ad-hoc for analytics tasks/MapReduce) -- but even then, you're not off the hook for operations. It's still _your_ responsibility to ensure that at the very least, there's a "hot standby" disaster recovery site. Yes, it's not easy - <USER2> running a successful business isn't supposed to be.

<comment><USER8> Comparing Techcrunch to Google is a bit lopsided, don't you think? Operations for most websites is like vehicle maintenance for a pizza delivery company: essential for success, but hardly their "core competency". This is especially true if the website is something relatively standard like a blog or news site. It takes quite a bit of scale and customization before it makes sense to hire full-time dedicated sysadmins.

<comment><USER9> > Operations for most websites is like vehicle maintenance for a pizza > delivery company: essential for success, but hardly their "core competency".

Very interesting analogy.

<comment><USER7> Would you fly on an airline that treated their aircraft maintenance the same way? So why would you expect advertisers to run their ads on sites that don't have a _person who understands the application_ carrying a pager?

</comment></comment></comment></comment><comment><USER10> TechCrunch ought to get a little green light image in their sidebar that goes red and displays the text "Rackspace is down" every time it is.

<comment><USER11> Yeah, and followed by a GoogleWave of Erlang threads on HN. Please.

</comment></comment><comment><USER12> Linode.

<comment><USER13> Linode had several hours of downtime back in October: [http://www.linode.com/forums/archive/o_t/t_4778/host_reboots...](http://www.linode.com/forums/archive/o_t/t_4778/host_reboots_october_27th_2009.html)

Not to knock on Linode - I use them - but as earlier commenters have stated, downtime is a fact of life, and no host is immune.

</comment></comment><comment><USER14> It's amusing that Techcrunch considers themselves and a handful of other startups "the internet".

This is why we architect provider independent solutions. Keep your TTLs low. Replicate your site to another provider, and be prepared to hit the switch when your primary provider inevitably fails. All datacenters lose power, all networks get borked, and all storage systems fail. It's a fact of life.

<comment><USER15> Yeah the snotty attitude advanced in the article is pretty offensive. If you run something on the internet and rely on it for your dinner, you don't really have an excuse for single points of failure. Do your job.

It's exceptionally easy to engineer around these issues if you don't just outsource all of your thinking to a provider.

<comment><USER16> I don't know, I sort of side with TechCrunch on this one. Yes, you can replicate a site (some easier than others, and a blog should be relatively easy), but that's still no excuse for datacenter downtime, and Rackspace has had a prolific amount of downtime.

 _Any_ datacenter downtime is serious business -- at a good datacenter, it's a small-ish incident every few years. Rackspace, on the other hand, seems to have a large incident every few months, meaning it's probably best to take your servers elsewhere.

<comment><USER17> I used to think that Data Center Downtime was serious business, but I've come around to believing that it shouldn't be. If your data center going down is a problem, then you don't really have a very robust failover plan.

If, instead, you _plan_ for the Data Center to go down, and treat a data center failure as a trivial issue, then your DR plans start to become significantly more robust.

In particular, the companies that I admire are ones that routinely swap, on a routine basis, their DR and production facilities - and when a production data center goes down they don't even bother to wait - they just light up the DR center and are back in business.

Most of the SaaS Financial Hosting companies (Oracle Financials) that I've talked with will provide you with that feature.

</comment></comment></comment></comment><comment><USER18> Holiday party. Someone spilled egg nog on a surge protector.

</comment><comment><USER19> This seems like a good candidate for <http://sadtrombone.com></comment>


Ask HN: Are Royalty Investments attractive options to investors? - master-litty A royalty investment is where an investor exchanges capital for royalties on one or more products, sometimes capped to a mutually agreed amount.
I believe these aren't pursued often because it's a complicated and expensive model for most business structures. As a result I'm having a hard time researching the general consensus about it.
Suppose I had a startup where it perfectly fit the model. Our company would sell products that are sold once to a single customer. Investors could perform a royalty investment by granting us capital once, then receive royalties on those kinds of sales until certain conditions are met.
Compared to traditional equity-based investments, I feel this arrangement can benefit both parties if arranged correctly. Companies can retain equity and investors could see larger returns quickly, especially if no exit strategy was being considered at the time.
How does HN feel about these types of investments? <comment><AUTHOR> Typically investment structures take the form that they take because investors understand them and are familiar with them and the forms have risks profiles that are somewhat understood. This means the difficulty with a company seeking a different kind of investment for a new product has to both:

1\. Sell investors on the future of the company's product.

2\. Sell investors on the company's risk analysis of the investment form.

The effort on the second point is a distraction (grifting aside) because it takes energy away from creating a product and it has a lower expected return. In some ways, cooking up a new investment scheme might be easier than creating a product.

Anyway, there are industries that operate on a similar principle. Notably real-estate has investment syndicates with limited partners. A key difference is that real-estate investment is typically asset backed.

Good luck.

</comment><comment><USER2> Somewhat related: [https://en.wikipedia.org/wiki/Hollywood_accounting](https://en.wikipedia.org/wiki/Hollywood_accounting)

</comment><comment><USER3> These are a big feature of the current season of "Shark Tank". Kevin O'Leary seems to offer one to many if not most of the contestants.</comment>


Those who can, code - <AUTHOR> http://sdtimes.com/code-watch-those-who-can-code/ <comment><USER2> "Being a programmer is as close as a human can come to being a magician." I have this quoted in my twitter. But I have to point out that without teamwork, a developer alone can only accomplish up to some point.</comment>


(Python) Generator Tricks For Systems Programmer (pdf) - <AUTHOR> http://www.dabeaz.com/generators/Generators.pdf? <comment><USER2> I love generators! They produce attractive code that is understandable (once one really understands what a generator is)

I remember using one when I was building a Mail DB search appliance (on the backburner..) with the mail retrieval code being this:

<https://gist.github.com/1014045>

Basically, after importing, retrieving emails is as simple as this:

        get_messages, close_server  = __pop3()
        all_messages = get_messages()
        for message in all_messages:
                fn(message)
        close_server()

__pop3 returning a simple generator, while close_server deals with shutting off the server / cleaning up the closure itself. No need to retrieve/initialize all the emails at once (with the memory / other overhead involved); just one at a time (with the connection being kept open in the background). Normally if you'd want to do that, you'd have to stick the "message handling code" into the "message retrieval code", or at least pass along a "handling" function - neither of which really seems that kosher to me. Generators solved that problem handily.

<comment><USER3> Looks like you could benefit from python's context managers (aka the "with" statement <http://www.python.org/dev/peps/pep-0343/> ). It allows you to do something at the start, do stuff in the middle, then guantee that something is done at the end, even if an exception is thrown. things that need to be closed afterwards are a great example of where with statements help.

<comment><USER2> Thanks, I'll take a look.

</comment></comment></comment><comment><USER4> An alternate (and, arguably, more elegant) approach to "Parsing with Regex" and "Tuples to Dictionaries" is to use named regex groups and the MatchObject's "groupdict" method like so:

      logpats = r'(?P<host>\S+) (?P<referrer>\S+) (?P<user>\S+) \[(?P<datetime>.*?)\] '\
                r'"(?P<method>\S+) (?P<request>\S+) (?P<proto>\S+)" (?P<status>\S+) '\
                r'(?P<bytes>\S+)'
      logpat  = re.compile(logpats)
      groups  = (logpat.match(line) for line in loglines)
      log     = (g.groupdict() for g in groups if g)

</comment><comment><USER5> If you really like the "pipelined" approach to describing data processing, postfix languages _sing_ at it. Factor (<http://factorcode.org/>) is well worth investigating.

edit: I built a little toy system for working with generators in Forth- some aspects are a bit messy, but you can still write code like

      100 upto sum dup * 100 upto { dup * } map sum -

([https://github.com/JohnEarnest/Mako/blob/master/examples/Alg...](https://github.com/JohnEarnest/Mako/blob/master/examples/Algorithms/Feasible.fs))

<comment><USER6> Apart from chaining generators, you can write a simple helper to pipe general expressions.

        def pipe(cur_val, *fns):
            """
            Pipes `cur_val` through `fns`.
            ::
                def sqr(x): return x * x

                def negate(x): return -x

            `pipe(5, sqr, negate)`

            is the same as
            ::
                negate(sqr(5))
            """
            for fn in fns:
                cur_val = fn(cur_val)
            return cur_val

I find `pipe(5, sqr, negate)` neater than `negate(sqr(5))`, especially when the nesting is greater than 2.

This is inspired by clojure's threading macro -> and a re-factoring of @fogus's code snippet on stack overflow. He used reduce to implement the loop - though loops can be implemented with reduce, and that's how you generally do it in clojure et al., it isn't idiomatic Python.

</comment></comment><comment><USER7> Python's iterators and generators are a perfect example of doing practical lazy evaluation. Streams in Scheme and lazy sequences in Clojure are similar in many ways.

A good programming language should try to find a sweet spot that has the best of both worlds, using lazy and eager evaluation in different situations. It's also a lot harder to convert code from eager to lazy evaluation than it is to do the other way, so I'm thinking that lazy evaluation should be the "default", as it is in Python 3 (range vs. irange in Python 2) and in Clojure for iteration and sequences.

</comment><comment><USER8> Inspired by this very presentation, I recently knocked together a library for munging text streams in Python, rather than calling through to sed and grep.

If anyone's interested, the code's here: <https://github.com/inglesp/munger.py>

<comment><AUTHOR> Is there any special reason for why you didn't use imap here [https://github.com/inglesp/munger.py/blob/master/munger.py#L...](https://github.com/inglesp/munger.py/blob/master/munger.py#L116) ?

<comment><USER8> Habit, I think. Will change it when I get the chance.

</comment></comment></comment><comment><USER9> I've really enjoyed this presentation. Thanks for sharing

Implementation of "tail -f filename" in Python given as example. (slightly altered)

      def follow(filename):
           f = open(filename)
           f.seek(0,2)
           while True:
               line = f.readline()
               if not line:
                   continue
               yield line

      for new_line in follow('/tmp/test.file'):
           print new_line

This is so cool.

</comment><comment><USER10> I love this pdf. The one thing I've been wanting is for the original author, or someone else with another ~1gig dataset, to do the test again with pypy and give us the speed figures.

<comment><USER11> Running the same scripts (mostly, except my Apache log lines were a bit longer) on a 1.1GB file resulted in:

python 2.7.1: list - 15.7s / generator - 14.1s

pypy 1.7: list - 7.5s / generator - 7.7s

On a MBP 2.2ghz i7 w/8gb. Fun!

<comment><USER10> Thanks, the future looks bright. Although RAM usage would be higher for pypy the improvement is massive.

<comment><USER11> I checked, PyPy used about 30mb, while vanilla Python was right around 3-4mb. So an order of magnitude.

</comment></comment></comment></comment><comment><AUTHOR> Reading "Hacking Scrabble (part 1)" <[http://news.ycombinator.com/item?id=3347720>](http://news.ycombinator.com/item?id=3347720>); source, I remembered about this presentation. I learned a lot about it and thought you guys might like too.

<comment><USER12> The follow-up presentation here is just as good, in my opinion: <http://www.dabeaz.com/coroutines/Coroutines.pdf>

<comment><AUTHOR> Thank you! I'll read it ASAP.

</comment></comment></comment><comment><USER13> I've got an implementation of tee in Python. Python itself comes with a reverse tee (reading from multiple files/file-like objects), but there was nothing out there for writing once to multiple file (or file-like objects).

It wasn't as easy as one would think, because trying to use it in conjunction with something like subprocess.Popen() presents a problem since it requires a file object that has an actual file handle (i.e. file.fileno()) rather than just some mock-object that implements the right instance methods.

I need to get it up on github and/or pypi.

<comment><USER14> <http://docs.python.org/library/itertools.html>

has an implementation of tee

<comment><USER13> That's not the tee that I'm talking about.

This takes one iterator and returns n multiple iterators:

<http://docs.python.org/library/itertools.html#itertools.tee>

This is what I'm talking about when I'm talking about tee:

<http://linux.die.net/man/1/tee>

Python has a reverse tee:

<http://docs.python.org/library/fileinput.html>

This iterates over input from multiple sources, but my implementation takes input from a single source and writes it to multiple destinations.

</comment></comment></comment><comment><USER15> Is there an equivalent in Ruby?

<comment><USER6> Not as high level as Python's generator expressions, but you can use Ruby's fibers to implement generators.

<comment><USER16> Enumerator solves typical generator scenarios reasonably well, and are easy to implement for your own classes. They use Fibers internally, and include Enumerable. Enumerator supports both internal and external iteration.</comment></comment></comment>


KeystoneML – Machine Learning Pipelines with Apache Spark - <AUTHOR> http://keystone-ml.org/

<comment><USER2> I'm one of the authors of KeystoneML. Happy to answer any questions about it here.

<comment><USER3> Is KeystoneML a replacement on Mllib?

<comment><USER2> Nope! KeystoneML is a research project exploring how to best support end-to- end pipelines for large scale machine learning. It's complementary to MLlib and some functionality from KeystoneML may find its way into MLlib in the future.</comment></comment></comment>


Ask HN: Paying for University as an undocumented student - <AUTHOR>

Hey HN,
I've been living in the United States since the age of 2, but I was actually born Saudi Arabia. During my sophomore year in high school my parents clued me in on the fact that they never actually got me my citizenship, and that I only had an overstayed tourist visa.
But things became really problematic when I graduated and realized that my illegal status makes me ineligible for all of the scholarships I earned (one in particular is a full ride Floridian scholarship) and would force me to pay out of state tuition.
So with no way to pay for school, I took a gap year which I'm four months into. I've been supplementing this off time with OCW computer science lecture videos and a ton of self study; however, everything's much more arduous and lonely than studying in a college environment and I don't want to fall behind.
I've come up with two solutions: apply to universities in Canada and Europe and hopefully be admitted to one of those countries on a student visa or stick it out here and find a way to pay for school. So I'm coming to you, HN, for some advice on how I could possibly pay for school. Are there private, merit based scholarships I could earn? Could I work a job with only a ITIN and no SSN? Is there a way I can petition for in-state tuition?
Please, any advice would help me out.
edit: I realize, on paper, I'm a criminal. I really don't see it that way, and I'd like to try everything I can to stay in the United States. I also cannot apply for citizenship with an overstayed visa and leaving the country would prevent my reentry for 10 years; something I'm trying to avoid.
edit 2: I'm here to talk about how to pay out of state tuition, not my status. For those who are curious, here are the only options I have when it comes to dealing with my status: get married, leave the country and be barred from reentry for 10 years, or wait for reform. So my stopping being a "criminal" is not a matter of choice or lack of will. <comment><USER2> Talk to an immigration lawyer, ASAP. Drop me an email if you need one. You're in a tough spot, without a US passport even Europe/Canada will be hard.

You could just go to a cheaper college but why risk it?

<comment><AUTHOR> I have talked to a lawyer and read USCIS's documentation, and fully understand my options (marriage, barred from reentry, or wait for reform.) However, if you know a good lawyer, I'm not resistant to a second or third opinion.

That's worrying though -- would my history in the US negatively affect me overseas?

<comment><USER2> Yeah, those options seem right to me, but IANAL.

If you get deported it will most likely affect your visa situation for Canada/Europe, if you leave voluntarily, maybe not. Honestly, The Internet is a bad place to ask for legal advice.

<comment><AUTHOR> You're right, but I was looking for some hidden scholarships or less obvious methods for paying out of state tuition. I'm also pretty open about my situation, since I know ICE doesn't really deport (almost)law abiding people.

<comment><USER3> Exactly, ICE won't deport you unless you get in the news or f__k up big time and get booked (such as, for example, driving over X mph in certain states) and on top of that if someone in the po-lice really hates you (they'll play dumb otherwise). I have my own opinion on why this is, something to do with actually needing illegals, but don't want to start a flamewar.

</comment></comment></comment></comment></comment><comment><USER4> Sorry to hear that. Even not being born here, you are as american as anyone else and since you came when you were only 2, you didn't do anything wrong as well.

However, the law is not on your side... I wish they had done part of the immigration reform to deal with people on your situation. I suggest talking to a lawyer asap to see what you can do. You could even apply as a refugee, since you don't speak arabic and coming back to your home country could put you in risk...

thanks,

</comment><comment><USER5> I have a friend of mine who found himself in a similar situation. He is working on his PHD now. He approached the school staff and set up something where he would pay everything out of pocket even given his meager means this guy worked hard to pay off every penny.

This is nothing that is impossible, i hope it works out.

</comment><comment><USER6> Canada's not going to be a whole lot cheaper. It seems my school is 3-6 times more expensive for non-Canadians. <http://findoutmore.uwaterloo.ca/financing/expenses.php>

I'm guessing the others are going to be fairly similar.

</comment><comment><USER3> I read your post thoroughly and all the replies so far.

Advice: 1) There are plenty of merit-based scholarships. Forget about Florida and think about out-of-state universities. Don't aim very high and get into a Tier 2 University where you will be their super-fab superstar.

2) Don't plan on leaving the country or even leaving to do work any time soon. There are full-ride merit-based scholarships and your family just needs to provide you the money for some clothes, some books and an occasional ticket home.

You didn't mention options that are breaking the law, but that are likely to have little repercussions for you if you get caught (second hand experiences):

1) Do not on any forms circle anything that would indicate you are a non-US citizen.

2) If there is a form asking if you are a U.S. resident, circle yes. If this ever becomes an issue you can lie that you thought it was for tax purposes (for which you are a resident).

3) Imagine your parents never told you your real story. What are the real world situations in which someone will ask for your birth certificate? There are just a few and one of them is for getting an SSN.

4) Play dumb. With a merit-based scholarship you'll buy yourself years more of time. U.S. degrees in Comp. Sci are highly valued overseas and require little to no paperwork to convert into whatever country's accreditation system.

5) Talk to immigration lawyers, they usually have good ideas.

6) Try to gather information from other illegals. With all the security paranoia it might be harder nowadays, but I bet there is still a somewhat accessible underground that will, for around $15000, be able to procure you an SSN with your name on it. That is all you need, once you have an SSN, you are gold my friend and no one ever needs to know. Don't go for it, but find out if such networks still exist or if everything got so computerized that an SSN is impossible to obtain.

7) Talk to more lawyers.

8) If there is ever a deportation procedure against you, this can be stalled for a loooong time I think with a case like yours. Be prepared to write lots of correspondence with powers that be. Lawyers are good.

9) Accept only under-the-table cash in hands for occasional contracting/coding work. Accept a car or some other good. Sell it later. For cash. You can make enough like this to support yourself while in college.

10) Wait until some legislation gets passed admitting the X million of illegals into the country. Number X will grow over years and eventually a generation will be quietly legalized. This will happen sooner or later. A big one happened in the early 1990s. You might spend the next 20 years waiting for this, but if anything like the DREAM act eventually passes, you're gold and welcome to America.

Oh and yes, if things are getting desperate, consider forging a Florida birth certificate if your in-state scholarship documentation requires it. My guess is they have nowhere near enough resources to verify these. If the scholarship requires an SSN, you are fux0red.

If there are people upset about me posting this stuff, I'd just like to remind that I hope this still adds signal and that the name of this website is Hacker News.

<comment><USER7> Do not buy an SSN/fake birth cert or falsely claim citizenship. Overstaying a visa is an administrative violation but not a crime as such. False claims /documents are a felony and will torpedo your case, even decades later. Just forget about those ideas.

<comment><USER3> I agree, the SSN/fake birth cert is the riskiest most dangerous proposition .. I have interacted with people who have done this some 15 years ago and they are still around, they have a family, a US passport and a steady decent paying job. What I don't know is how many got busted and landed in deep stuff.

</comment></comment></comment><comment><USER8> I have worked in higher education in various U.S. states and several other countries as both a professor and an academic adviser. In the latter case, my job was to help students "navigate the system." Getting through college and paying for it can be complicated if you are trying to do it alone. It's worth learning how to pay the game and how to find help getting through.

A few thoughts:

Finding merit-based scholarships as a non-citizen is not easy. The key is to go local. Funding from your department, school, or local community groups and businesses is much less likely to require any proof of legal status. Most scholarship committees frankly don't care. I have worked as a scholarship judge. The last thing you want is extra info from a thousand applicants that won't help you choose the best people. However, such scholarships are usually small, in the $500 - $2,000 range, and are usually awarded after you have already been enrolled in a certain school. The ones that pay out more have more applicants and are a long shot. These will be a bonus but not a way to pay most of your tuition. Sometimes it's easier just to work than take the time to apply to the smallest awards.

Besides scholarships, another thing to think about is the price tag of the university. Costs vary widely among states. Community colleges are much cheaper than four-year schools and I would strongly recommend starting out at one. It's easy to find a good community college. Pick an upscale suburban neighborhood, a top-tier university area, or liberal arts college town and find the nearest community college. Ivy-type schools usually overshadow a really good community college. The bonus is that you can hang out with the top students from the four-year school while paying cheap tuition. You might even be able to build a support network that helps you transfer into the four-year school. Since you're undocumented, lean toward areas with large numbers of undocumented students, or at least places that are not hostile to immigrants. To state the obvious, Arizona and Texas would not be first on my list. Places that tend to spend more on public infrastructure and have rich people hanging around that give money away are probably a good bet, too. I'm guessing you won't find as much financial largess in the South.

Among U.S. states, California seems to have the most tolerant university system for undocumented students I've seen, probably partially because there are so many in the system and because everyone is related to someone, or at least has a friend, who is undocumented. They have their own category in the system: AB 540.* Even though you would not be legally considered AB 540, I think you could just tell people you are when interacting with college staff and faculty. If "undocumented" is a nice way to say "illegal," "AB 540" is a nice way to say "undocumented." Plus, it protects you because the sort of person who would turn you in doesn't know what it means. Unlike other states I've been in, students openly discuss their status with college staff, professors, and classmates in CA. I have talked with admission and registration staff at several urban colleges about this issue. They are adamant that they do not and will not share information on legal status with any branch of the U.S. government. Always be careful, but you're better off in this kind of environment. For example, I overheard a student approach an engineering department chair about the fact that he couldn't apply for most big corporate internships because of his status. The professor advised the student to continue studying as long as he could (i.e. to the PhD level) and get top grades. He said that becoming indispensable in some technical field will earn you a visa in the U.S. or a good job in some other country. Not sure if that's accurate or good advice, but it was interesting that the student felt comfortable approaching a professor he didn't know well about this problem, and the professor addressed it completely matter-of-fact-ly.

Undocumented students, even those from middle class families, tend to be impoverished because they have to pay more tuition and work very low paying jobs. In general, I have noticed that California college administrative staff tend to be more understanding of very low income students than staff in other states. For example, they will make arrangements for installment payments of smaller sums of money. The community college system allows for paying bills later in the term as well. In some states, the computers automatically drop your classes if you haven't paid after the first two weeks, but I believe this is illegal in CA. In the CA State University (CSU) system, annual in-state tuition is now around $6,000 a year, a bargain compared to many other states, but considered expensive by locals. Out-of-state tuition is considered extremely expensive, which means you will get more help and sympathy trying to pay your tuition. Increases in tuition are happening regularly due to the budget fiasco, but having sympathetic people to help you through is worth a lot. Often, staff control scholarship dollars locally and have extra money to allocate. Most students don't know this, but when not enough people apply for money dedicated to African-American and Latino students, it can often go to you if you ask for it. Getting to know these staff people is worth a lot. They often have titles such as academic advisor, MESA coordinator, TRIO--there are lots of acronyms that just refer to different pots of federal money that pay their salaries, but they are usually housed in "Student Affairs" or "Student Services." The key web search terms related to these people are "students at risk," "disadvantaged" or "first-generation" students. "Leaving college" is the PC term they use for dropping out. When I did this job, my university had a row of five or six offices that each did the same thing, but our salaries all came from different funds. Any of us would have helped you if you had asked. Many officially support a certain ethnicity--according to their job description--but if you ask for their help, they will help you regardless. If you are familiar with the concept of a "case manager," they do something similar. Their job is to know about where money and resources are, and which people in the rest of the university (financial aid/registration/etc.) are helpful and which should be avoided. Pick a region where you want to study and start visiting these people. Just cold call them and walk into their offices. You will eventually find someone who will help you out, and that's a good place to go to school.

If this doesn't work out and you do end up deciding to leave the country, you have a few things going for you.

To increase their international rankings and sometimes to meet local political initiatives, universities in many countries are trying to ratchet up the number of foreign students they accept. Most of them can't find enough people from "rich" countries. Some of them will give you a free ride so that their "international program" is slightly more international and not all Chinese and Pakistani students. Search online for "international studies" or "international business" programs.

Top universities in some countries would love to have you just because you are "American" to them (the irony) and speak English. You are guaranteed to get into most of the top schools in South Korea, for example, and then be guaranteed lab research opportunities with whomever you ask, if you are interested in science or engineering. Your degree will have much more value inside the country than outside it, but if you are actually interested in the place where you study, it might be worth it.

Since you're from Saudi, you might consider the Middle East. There's loads of money in new universities in places like the UAE and they want international students. The money still exceeds the quality, but they are intent on spending to get the quality up, and money is what you're looking for. Go there for undergrad, then get a free graduate (or "post-graduate" as they call it) degree in Europe.

Some countries still have cheap or free tuition. Look at Scandinavian schools. Look at the top schools in Latin America, such as UNAM in Mexico City. Many of these schools will have extensive course offerings in English, although I recommend learning the local language on your way through school. Heck, maybe you could even stay in the U.S. and figure out how to study online at a foreign university, although you'd probably have to pay cash. Check out Australia for that. They have been rapidly expanding their higher education system internationally.

I guess I could go on, but this should get you started thinking about these things. A couple of websites that will give you a few more search terms:

* <http://ab540.com/WhatIsAb540.html>

[http://www.californiacolleges.edu/finance/financial-aid- faqs...](http://www.californiacolleges.edu/finance/financial-aid- faqs/undocumented-students.asp)

<comment><AUTHOR> Thank you righto, this is incredibly useful. Monday morning I'm going to talk to as many advisors as I can.

</comment></comment><comment><USER9> If you're not legally entitled to be in the US, then get the fuck out of the US.

<comment><USER10> How broken is our system that someone can manage to stay 16+ years without getting caught. I mean really, the laws and policies need to be adjusted to make that impossible.

<comment><AUTHOR> I think it's worse that I can't earn my citizenship. The immigration system has neglected undocumented students: a very specific niche of illegals.

I'm completely American and you'd never guess that I'm illegal unless I told you (the stigma is all illegals speak broken English and are some sort of Latino, which I'm not.)

<comment><USER10> Your parents didn't do a good job taking care of you. Sucks for you.

<comment><USER5> By that same judgement your parents didn't teach you the virtue "if you don't have something good to say, you should keep your mouth shut"</comment></comment></comment></comment></comment>


FreeBSD Security Advisory: TCP reassembly vulnerability - <AUTHOR> http://lists.freebsd.org/pipermail/freebsd-announce/2014-April/001546.html

<comment><USER2> This advisory is written to create the impression that the impact of the vulnerability is limited. But this is a memory corruption flaw. I've heard rumors that there are bigger problems here than simply crashing hosts or leaking memory, but it's possible that the code execution scenarios people are talking about rely on some second bug, coupled with kernel information leaks.

Either way, patch this quickly.

<comment><USER3> I'd like to hear any details of the rumours you're talking about. We looked carefully at this bug and couldn't find any potential for more severe impacts -- even the kernel memory leakage is hard for remote attackers, since the memory is leaked to the local side of the connection, not the remote side.

<comment><USER4> It's probably not too hard to get a web server to effectively echo the 'corrupt' / reassembled packet it eventually receives? I guess it depends on how much control you have over the content that directly precedes the disclosed kernel memory.

But there are probably ways to get a system to echo the data it receives that are less likely to be logged.

<comment><USER3> Web servers are why we decided that extracting data would be possible in some circumstances; the problems of needing to get the stack fragment to appear in the right place following the right data was why I said it would be difficult.

</comment></comment></comment></comment><comment><USER5> The patch shows the nature of the error:

      Index: sys/netinet/tcp_reass.c
      </comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>=
      --- sys/netinet/tcp_reass.c    (revision 264836)
      +++ sys/netinet/tcp_reass.c    (working copy)
      @@ -211,7 +211,7 @@ tcp_reass(struct tcpcb *tp, struct tcphdr *th, int
            * Investigate why and re-evaluate the below limit after the behaviour
            * is understood.
            */
      -    if (th->th_seq != tp->rcv_nxt &&
      +    if ((th->th_seq != tp->rcv_nxt || !TCPS_HAVEESTABLISHED(tp->t_state)) &&
               tp->t_segqlen >= (so->so_rcv.sb_hiwat / tp->t_maxseg) + 1) {
               V_tcp_reass_overflows++;
               TCPSTAT_INC(tcps_rcvmemdrop);
      @@ -234,7 +234,7 @@ tcp_reass(struct tcpcb *tp, struct tcphdr *th, int
            */
           te = uma_zalloc(V_tcp_reass_zone, M_NOWAIT);
           if (te == NULL) {
      -        if (th->th_seq != tp->rcv_nxt) {
      +        if (th->th_seq != tp->rcv_nxt || !TCPS_HAVEESTABLISHED(tp->t_state)) {
                   TCPSTAT_INC(tcps_rcvmemdrop);
                   m_freem(m);
                   *tlenp = 0;
      @@ -282,7 +282,8 @@ tcp_reass(struct tcpcb *tp, struct tcphdr *th, int
                       TCPSTAT_INC(tcps_rcvduppack);
                       TCPSTAT_ADD(tcps_rcvdupbyte, *tlenp);
                       m_freem(m);
      -                uma_zfree(V_tcp_reass_zone, te);
      +                if (te != &tqs)
      +                    uma_zfree(V_tcp_reass_zone, te);
                       tp->t_segqlen--;
                       /*
                        * Try to present any queued data

</comment><comment><USER4> Almost sounds like Heartbleed in the TCP stack...

      Additionally, because the undefined on stack memory may be overwritten by
      other kernel threads, while extremely difficult, it may be possible for
      an attacker to construct a carefully crafted attack to obtain portion of
      kernel memory via a connected socket.  This may result in the disclosure of
      sensitive information such as login credentials, etc. before or even
      without crashing the system.

I'm sure there's some great branding opportunities around 'tcp_reass.c'...

<comment><USER2> Heartbleed-style vulnerabilities aren't rare. At a security conference several years ago, someone dumped a full image of a running core router with one of them.

</comment></comment><comment><USER6> Is there any evidence that this has already been exploited in the wild?</comment>


Bletchley's wartime codebreakers meet again - <AUTHOR> https://www.thetimes.co.uk/edition/news/bletchleys-wartime-codebreakers-meet-again-zh6grpgrb?CMP=Sprkr-_-Editorial-_-thetimes-_-News-_-Imageandlink-_-Statement-_-Unspecified-_-TWITTER&linkId=41784700 <comment><USER2> This article is mostly hidden behind a membership sign-in.

Notably absent, and getting no mention in the viewable part of the article, Alan Turing.</comment>


Ask HN: How do you display code examples on your site? - <AUTHOR> I'm looking for a nice way of presenting HTML code snippets with syntax highlighting (doesn't have to be editable).
I've previously used highlightjs but haven't been enamored with it.
I particularly like:
- https://statickit.com/ - https://css-tricks.com
Anyone have any tips? I'd love to hear what other people are using. <comment><USER2> [https://prismjs.com/](https://prismjs.com/)

I like it a lot. Simple, works great. Examples (these snippets are highlighted for bash and clojure): [https://jacobobryant.com/post/2019/learn- clojure/](https://jacobobryant.com/post/2019/learn-clojure/)

</comment><comment><USER3> statickit.com and css-tricks.com both seem to use [https://prismjs.com/](https://prismjs.com/) for highlighting</comment>


Unlike Babies, the Best Managers Come with Instructions - <AUTHOR> https://torch.io/unlike-babies-the-best-managers-come-with-instructions/?hss_channel=tw-983379969773420544 <comment><USER2> Manager READMEs start out with good intentions, but every example I've seen is flawed for multiple reasons:

1) It's manager-centric. Why would you give employees a guide to their manager when you could give them an "Employee README" that is focused on the new employee? When I join a company, I want to know what's expected of me so I can succeed. If the company delivers this in a form of a manager README then I'll still be missing the big picture goals of the company. Managers - Focus on your employees, not yourself.

2) Manager READMEs become an aspirational target. Describing yourself in text is challenging for the ego. I've seen too many manager READMEs that read like a picture-perfect version of what the manager aspires to be, rather than a description of how the person actually operates.

3) The self-promotion aspect. When managers post their README all over internal company chat, public Twitter, their LinkedIn profiles, Medium think pieces, and the company blog (like this article), the document becomes a brand-building exercise. This puts even more pressure on managers to only write down their most perfect, idealized vision for what they want to be. Or they spin their flaws as some sort of superpower with creative wording. I once read a public manager README that was nothing like that person's actual management style, but it read as if that person was a textbook-perfect manager.

My recommendation: Skip the manager README fad and focus on your employees. Write an "Employee README" onboarding guide for each of your employees, that you customize with individual expectations and information for each employee before they join the company. Keep it in a shared document that you can update over time to include an up-to-date job description, list of responsibilities, and key expectations for that employee. Don't try to make management all about the manager's personal brand.

<comment><USER3> > Why would you give employees a guide to their manager when you could give > them an "Employee README" that is focused on the new employee?

Why not do both? "Hey folks, here's my README, I'd encourage you to write something similar and share it with everybody."

<comment><USER2> > Why not do both? "Hey folks, here's my README, I'd encourage you to write > something similar and share it with everybody."

When it comes to team building, I think it's much more valuable to say "this is how we operate" instead of having everyone write down "this is how _I_ operate". The former is a mutual guidebook for everyone to work together, while the latter feels like an edict that everyone else needs to work around you.

<comment><USER4> How do you handle the case where two people operate differently but need to interoperate?

<comment><USER2> De-escalate any tension between the two people

Check for any overtly abusive behavior that requires correction. Usually rare, but worth checking.

Provide basic coaching and suggestions for working together in a healthy manner.

And most importantly: Let them know that they're expected to figure out how to operate together on their own like professionals. Managers can't be mediators for every impedance mismatch between employees.

</comment></comment></comment></comment></comment><comment><USER5> IMO the best managers learn to change their behaviors based on each report's needs and doesn't request that their reports conform to the manager's needs.

<comment><USER6> I had to read this sentence twice before understanding that a "report" is a person. Maybe that has become accepted terminology in some circles. I hope not. It's terribly dehumanizing.

<comment><USER5> What word would you prefer to use for "person whom a manager manages"? If one just uses "person" it is unclear if it's any arbitrary person or a person under a manager's umbrella

<comment><USER7> "Directs" is another common word I hear used by managers.

<comment><USER5> Which makes sense as the formal phrase is "direct report".

While it was unintentional in my original post, "report" seems to work better than "direct" here as IMO my statement holds for a manager's indirect reports as well and thus "report" covers both direct and indirect. But now I'm just amusing myself with word games :)

</comment></comment></comment></comment></comment><comment><USER8> Unpopular opinion:

In Silicon Valley, people managers can easily be a waste of time and money, and can cause unnecessary drama on your team. Think about the kind of so- called developer who would rather manage people than write code.

We still need founders, executives, directors, advisors, etc. don't get me wrong. But has anyone ever come across an engineering manager worth their salt? Why did we start deploying people managers en masse to engineering teams? I miss the holy trinity setup: Dev, designer, PM. It worked well IMO.

Our industry is shooting itself in the foot with Engineering Managers. The ones I've come across are self-obsessed, as evidenced in these cringeworthy READMEs and this article that was written.

<comment><USER9> Without (competent) engineering leadership.

\- Whos going to push back and sales and marketing for harassing you directly or taking direct blame for failures?

\- Who will represent the engineering team in leadship meetings to make sure policies/decisions are made with their needs in mind?

\- Who's going to fight for your raises and promotions?

\- Who's going to make sure that there is budget for what engineering needs to do and keeps a proper record to keep finance happy?

Please thing about it... any company that grows to 30-50 people (even startups) need to have management and processes that eventually lead to (even unintentional) politics. The other departments will have leaders/managers defending/supporting them... its foolish to not understand or respect competent leaders.

<comment><USER10> The examples that you've given are political (represent, push back, take blame, fight), except for the example of keeping records (which is management).

Managements role, technical leadership role and politician role are different roles. The required skill sets and experience are different. The outcome of grabbing a wrong role is usually poor. Not confusing politician, technical leader and manager roles is probably a good idea to improve outcomes. As checking for required experience, to take the role.

<comment><USER9> Management is full of politics... sorry but its true. A true leader must have all the skills needed to properly benefit and support the team.

You look at my examples as negatives... its the wrong way to look at it... thoes other managers of other departments? Are the bad people? No... but they have a job and responsibility to represent their departments and do from their point of view whats best for their team and the company.

I feel like there is a segment of workers in engineering has this weird self damaging hatred for leaders/managers other departments do not have... then thoes same people get pissed when they dont have a leader and their department is glossed over because no one represents them when/where some of the most important decisions are made.

<comment><USER10> Notice, there was nothing said about roles being bad. There was nothing said about impossibility of combining multiple roles.

</comment></comment></comment></comment></comment><comment><USER11> To build on what another commenter said, the idea of manager READMEs made perfect sense when they started out a few years ago - here are my quirks, here's how I operate, etc.

The problem is they've turned into something that justifies bad behavior more often than not. So many of them have this undercurrent of "I'm an asshole - don't take it personally!" which is just totally not OK. The solution isn't to proudly broadcast and justify that you're an asshole and expect others to adapt — the solution is to not be an asshole.

That being said, things like "I tend to work late so if I email you on the weekend, I don't expect you to reply until Monday" are totally reasonable and I think, on the whole, a good thing.

<comment><USER12> Very much agree with your analysis here. It's like many, many good ideas that have come up in SV (agile/scrum, OKRs, etc), when looked at as a useful tool for the individual that should be utilized on an individualized basis, they are incredibly useful. When made standard and forced into use-cases and personality types they don't work for, they can be mediocre or even counter- productive. They will also be used in ways not entirely desired, such as for self-promotion, self-justification, gaming things, etc. Managers also tend to get overly prescriptive about how to use the tool (usually a sentence that starts with, "here's how we do $TOOL at $COMPANY" is often heading there).

I've used OKRs, Kanban/agile and many other tools for me at an individual level and loved them. I've had the exact same tools at companies become a waste of time, a point of contention, and only be a "value add" for management.

</comment></comment><comment><USER9> Ugh as someone with 12 years of experience and about 8 ish in a teach lead/management role. This seems super weird to me.

An Engineering Manger/Director/VP/CTO's main job is 2 things.

\- To be the over arching intermediate between engineering and other business units (note: this does not mean walling off engineering, engineers SHOULD understand and work with business units when it makes sense).

\- To support, mentor, shield, and basically do what ever is necessary for their team/department/org to be as successful as possible.

This sounds VERY self centered and I honestly believe product and engineering managers need to be to most selfless people possible to actually be effective. I would call it backwards even... a leader/manager needs to tailor himself and his approach to both his people and his cross-department equals in order to be the most successful.

</comment><comment><USER13> This is the silliest, most ridiculous, and self absorbed thing I've read in a while. So much so that I suspect it's actually satire. If it's not then silicon valley sounds like another planet, in my 2+ decades of employment I've never had any sort of issues with management. If I have any questions, I ask, if I have any feedback, I give it. That clears up any ambiguity.

<comment><USER14> > in my 2+ decades of employment I've never had any sort of issues with > management

Then you're incredibly fortunate. I've had to deal with management issues at multiple companies in a shorter span of time.

<comment><USER13> Sure, but a README doesn't magically fix shitty management, if anything it just a justification for it.

Not to mention people are just horrifying bad at seeing themselves as others see them. I think of the scene in A Christmas Story when the kid expects to get a glowing review of his paper but instead got a D grade. It actually reminded me of a family member of mine who got called into his boss's office. He was bragging that he was going to get a bonus or promotion and instead he got a demotion due to poor performance and bad work ethic.

I've had five software jobs and five unskilled jobs.

<comment><USER14> Oh, certainly. I think a README could've been helpful from good managers, but it could just as easily make bad managers worse.

</comment></comment></comment></comment><comment><USER15> See also _The Manager FAQ_ by Peter Seebach:

[http://www.seebs.net/faqs/manager.html](http://www.seebs.net/faqs/manager.html)

<comment><USER16> Thanks for this one! Reading his Hacker Faq already...

</comment></comment><comment><USER17> I wrote one for myself that is yet to be published. Will give it a 2nd thought.

The reason I thought it would be helpful is that I am not a high feedback person. I seldom praise (need to work on that), I am not expressive, I've been told I 'play my cards close to the vest.' by someone who directly reported to me. That really got me thinking that maybe the circumstances where I am not getting what I want is because my staff really cannot figure out what I'm looking for in general. So I wrote up a 'read me', before I knew there was this trend.

I included self-deprecating statements about things like assigning a task to multiple people because I forgot it had already been assigned 'Don't take it personally, just let me know and I'll fix it.' And, if they need me to attend a meeting, make sure it's on my calendar, or, no chance I'll remember.

In the end, it's only going to be as helpful as it is 'correct' and clearly described. That depends on each manager's level of self-awareness, combined with their sense of self confidence. You'd hope managers would have both qualities, but if not, this can really be a red herring, or even be used as a dark pattern.

<comment><USER18> > The reason I thought it would be helpful is that I am not a high feedback > person. I seldom praise (need to work on that)

The reason I would be afraid to standardize manager READMEs is specifically that I do not trust the average manager to publicly acknowledge and privately work on flaws like that. For managers who do that, I'm not that concerned about the exercise because it's more likely to produce something useful for the employees. For the other type of manager, I'd be afraid of accidentally creating the expectation that the employee needs to adopt the manager's style, which I would privately recommend to any employee, but which I would vehemently oppose as anything resembling an official expectation.

</comment></comment><comment><USER19> Tangent here - saw Torch and nostalgia of [http://www.computinghistory.org.uk/det/1039/Torch- Computers-...](http://www.computinghistory.org.uk/det/1039/Torch-Computers- Ltd/) came a flooding.

</comment><comment><USER20> I'm wondering if anyone has been in a manager-less team. I would like to think that's possible and if so what was the experience like?

<comment><USER21> it works fine as long as everyone is extremely reasonable, and everyone's primary motivation is the success of the project.

unfortunately, this is very unlikely to be true. and increasingly less so as the team size increases.

one confounding factor here is the heavy use of religion to guide technical decisions. the decisions we make now about the direction of the product and the attributes of the codebase are largely subjective. there are arguments on all sides of a position.

a good team, properly enculturated, understands the process. when the issue is important enough to justify digging in and when its just a personal bias. a good team helps all of its members stay on the path and not end up in giant ratholes. a good team uses its experience to inform future decisions rather than create artificial strictures. a good team has enough time and maturity to interact constructively with the rest of the business and the customers.

I shouldn't say 'good' team, I should say 'perfect'

good management greases all of these points of potential friction. but I think at the end of the day management is there to address the last point. its natural and expected for engineers to keep a narrow focus on the technical aspects of the project itself. someone's got to make sure its evolving in a direction that's meaningful in an external context.

<comment><USER4> Yes, teams don't need managers unless they have people that need managing. Also, teams don't need developers if they don't have any code that needs writing. Ignorant dismissal of the value of other kinds of work besides their own, combined with a lashing out against suggestion that they might not be perfect, is one of (a vocal portion of? HN's nasty vices.

</comment></comment></comment><comment><USER22> This started out well.

</comment><comment><USER23> Perhaps it's cultural (I'm from the UK), but I can't imagine writing something so painfully self-absorbed - and then expecting people to read it.

<comment><USER24> When I read the hn title I thought the direct reports wrote a readme about the manager. That made a whole lot more sense to me. How many coffees does (he/she) need to drink before it's ok to approach their desk. How much detail do they require when explaining an issue. These are the features I'd like documented for new hires.

</comment></comment><comment><USER25> IT people really love these kind of forced parallels. Can't really think of another sector that does this. Maybe it's all that "Uber/Airbnb but for X" thinking.

Unless your surname is .exe I don't think you need a readme file. You could do the human thing though and discuss expectations with your team though...</comment>