Google Kubernetes Engine's third consecutive day of service disruption - <AUTHOR> https://status.cloud.google.com/incident/container-engine/18005 <comment><USER2> I am currently evaluating GCP for two separate projects. I want to see if I understand this correctly:

1) For three whole days, it was questionable whether or not a user would be able to launch a node pool (according to the official blog statement). It was also questionable whether a user would be able to launch a simple compute instance (according to statements here on HN).

2) This issue was global in scope, affecting all of Google's regions. Therefore, in consideration of item 1 above, it was questionable/unpredictable whether or not a user could launch a node pool or even a simple node anywhere in GCP at all.

3) The sum total of information about this incident can be found as a few one or two sentence blurbs on Google's blog. No explanation nor outline of scope for affected regions and services has been provided.

4) Some users here are reporting that other GCP services not mentioned by Google's blog are experiencing problems.

5) Some users here are reporting that they have received no response from GCP support, even over a time span of 40+ hours since the support request was submitted.

6) Google says they'll provide some information when the next business day rolls around, roughly 4 days after the start of the problem.

I really do want to make sure I'm understanding this situation. Please do correct me if I got something wrong in this summary.

<comment><USER3> When everything works, GCP is the best. Stable, fast, simple, reliable.

When things stop working, GCP is the worst. Slow communications and they require way too much work before escalating issues or attempting to find a solution.

They already have the tools and access so most issues should take minutes for them to gather diagnostics, but instead they keep sending tickets back for "more info", inevitably followed by a hand-off to another team in a different time zone. We have spent days trying to convince them there was an issue before, which just seems unacceptable.

I can understand support costs but there should be a test (with all vendors) where I can officially certify that I know what I'm talking about and don't need to go through the "prove its actually a problem" phase every time.

<comment><USER4> As someone who works for Government and Enterprise - all I care about sometimes is how a company behaves when everything goes wrong.

The issue with outages for the Government organizations I have dealt with is rarely the outage itself - but strong communication about what is occurring and realistic approximate ETAs, or options around mitigation.

Being able to tell the Directors/Senior managers that issues have been "escalated" and providing regular updates are critical.

If all I could say was a "support ticket" was logged, and we are waiting on a reply (hours later) - I guarantee the conversation after the outage is going to be about moving to another solution provider with strong SLAs.

<comment><USER5> Very similar thing at our office. Considering the scale of which we run things, any outage could be a potential loss of millions _every minute_.

Sure, we use support tickets with vendors for small things. Console button bugging out, etc. But for large incidents, every vendor has a representative within an hour driving distance and will be called into a room with our engineers to fix the problem. This kind of outage, with zero communication, means the dropping of a contract.

Communication is critical for trust, especially if we're running a business off it.

<comment><USER6> Going single cloud on that scale is simply irresponsible though.

You need failovers to different providers and hopefully also have your hardware for general workloads

And suddenly the CEO doesn't care anymore if one of your potential failovers is behaving flaky in specific circumstances

Not saying it's good as it is.. communication as a saas provider is - as you said- one is the most important things... But this specific issue was not as bad as some people insinuate in this thread

<comment><USER7> Agree, if we are really talking about millions per minute (woah), then you can afford to failover to AWS.

</comment></comment></comment></comment></comment></comment><comment><USER8> We had an issue a few weeks ago where the google front-end servers were mangling responses from Pub/Sub and returning 502 responses, making the service completely unusable and knocking over a number of things we have running in production. Despite paying for enterprise support and having in a P1 ticket, we had to spend Friday to Sunday gathering evidence to prove to the support staff that there was indeed a problem, because their monitoring wasn't detecting it. Right now I'm doing something similar (and since Friday!) but for TLS issues they're having. Again, because their support reps don't believe there's a problem. There are so many more problems than they ever show on their status page...

<comment><USER9> They work for Google so obviously they are much smarter than you. If theres a problem its probably the customers fault. /sarcasm

<comment><USER10> I was so mad to read that until you said /sarcasm :p

That being said I really do think there is a difference between who is working at google today and the google we all fell in love with pre-2008.

I am sure there are a amazing people still working at google, but nowhere near like it was.

The way I like to think about google is that some amazing people mad ea awesome train that builds tracks in front of it -- you can call them gods maybe -- but those people are gone -- or a least the critical mass required to build such a train has dwindled to just dust. What we have left is a awesome train full of people pulling the many levers left behind.

To make things even worse my last interview as a SRE left me wondering if even the people who are there know this as well, and they are actually working hard to keep out those who might expose light on to this. I don't say that because I did not get the job -- I am actually happy I did not get extended a offer.

I say this with one exception, the old-timer who was my last interview. I could tell he was dripping in knowledge and eager to share it with any that would listen. I came out of his 45 min session learning many things -- I wold actually pay to work with a guy like that.

I would also like to point out that the work ethic was not what I expected. I was told that when on call, my duty was to figure out the root cause was in the segment I was responsible for. I don't know about you, but if my phone rings at night I am going to see through to a resolution and understand the problem in full -- even if it is not on the segment that I was assigned.

/end rant

<comment><USER11> You've managed to roll the myth of 10x engineers, start-up geniuses, nostalgia and gut feelings into one message of very dubious veracity.

At the same time you ignored the massive complexity and size of Google compared to what they were at the beginning.

This is voodoo organisational analysis.

<comment><USER12> 10x engineers are real. Start-up geniuses are real. The large majority of people have their heads up their asses. Wake up and smell the coffee

<comment><USER13> You can't find and hire geniuses for every component. Systems should scale with the average engineer in mind (so should code). We have all smelt the coffee and it smells even better when your team is efficient and well rested.

<comment><USER12> I don't disagree. Building around 10x'ers in this way creates god complexes and unhappy "senior" engineers (since they don't do any of the cool work). Having one very early still juices your productivity

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> "The data says engagement is down 46%, I think its time we drop the product."

\- Someone at Google right now, probably.

<comment><USER14> I can assure you that's not the case! Also, while people like to repeat this meme, Google Cloud does have a formal deprecation policy ([https://cloud.google.com/terms/](https://cloud.google.com/terms/)), whose intent is to give you some assurances.

(I work at Google, on GKE, though I am not a lawyer and thus don't work on the deprecation policy)

<comment><USER15> > Google may discontinue any Services or any portion or feature for any reason > at any time without liability to Customer

for any reason

at any time

<comment><USER16> Nice job cherry picking text.

> 7.1 Discontinuance of Services. Subject to Section 7.2, Google may > discontinue > any Services or any portion or feature for any reason at any > time without > liability to Customer.

Let's take a look at Section 7.2:

> 7.2 Deprecation Policy. Google will announce if it intends to discontinue or > > make backwards incompatible changes to the Services specified at the URL > in > the next sentence. Google will use commercially reasonable efforts to > continue > to operate those Services versions and features identified at > > [https://cloud.google.com/terms/deprecation](https://cloud.google.com/terms/deprecation) > without these changes for at least > one year after that announcement, > unless (as Google determines in its > reasonable good faith judgment): > > > (i) required by law or third party relationship (including if there is a > change > in applicable law or relationship), or > > (ii) doing so could > create a security risk or substantial economic or material > technical > burden. > > The above policy is the "Deprecation Policy."

To me that looks like a reasonable deprecation policy.

<comment><USER17> > To me that looks like a reasonable deprecation policy.

It might be, until they jack up the prices 15X with limited notice (looking at you, Google maps [1]). No deprecation needed, just force users off the platform unless they're willing to pay a massive premium.

[1] [https://www.google.com/search?q=google+maps+price+increase](https://www.google.com/search?q=google+maps+price+increase)

<comment><USER18> Google Maps has never been subjected to that policy, unlike GCP services. These org chart divisions are real but only clear to Googlers, Xooglers (I'm in this category), and people who pay extremely close attention.

The fact that they're all Google makes reputation damage bleed across meaningfully different parts of what's in truth now a conglomerate under the umbrella name Google.

<comment><USER19> Except all the Google maps setup and API keys are generated from the gcp UI and the billing happens on the cloud platform as well. While maps didn't start as a gcp product, they seem to have rolled it in to gcp fully.

<comment><USER18> Not fully. Really what happened is they did a re-org that gave them Google Cloud as an umbrella brand including GCP, Google Maps Platform (this new version of Google Maps as a commercial service), Chrome, Android, G Suite...

The bit of Maps Platform integration for management of the billing and API layer was called out in the announcement blog as an integration with the console specifically, and the docs and other branding around Maps Platform remain distinct from GCP still in excessively subtle ways that Googlers pay more attention to than everyone else, like hosting the docs on developers.google.com instead of cloud.google.com and having Platform in its name separately from Cloud Platform.

This stuff makes sense to Googlers not only because of the org chart but also because Google has a pretty unified API layer technology and because Google put in a lot of work to unify billing tech & management. Reusing that is efficient but not always clear.

But you're right to be confused. Their branding is a mess and always has been. This is the same company that thought Google Play Books makes sense as a product name.

Google's product / PR / comms / exec people are very bad at understanding how external people who don't know Google's org chart and internal tech will perceive these things, or at least bad at prioritizing those concerns.

They live and breathe their corporate internals too much to realize this. Some Google engineers and tech writers realize the confusion but pick other battles to fight instead (like making good quality products).

They do at least document which services are subjected to the GCP Deprecation Policy (Maps is not there): [https://cloud.google.com/terms/deprecation](https://cloud.google.com/terms/deprecation)

As for what products are actually part of GCP, it's the parts of this page that aren't an external partner's brand name, aren't called out separately like G Suite or Cloud Identity or Cloud Search, and aren't purely open source projects like Knative and Istio (as opposed to the productized versions within GCP), with the caveat that the level so far of integration into GCP of Google acquisitions like Apigee, Firebase, and Stackdriver varies depending on per- company specifics: [https://cloud.google.com/products/](https://cloud.google.com/products/)

G Suite and Cloud Identity accounts can be used with GCP, just like any other Google accounts. They are part of Google Cloud but not Google Cloud Platform.

Hope I waded through the mess correctly for you. :)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> Hi - I work at Google on GKE - sorry about the problems you're experiencing. There's a lot of people inside Google looking into this right now!

It looks like the UI issue was actually fixed, and that we just didn't update the status dashboard correctly. But we're double checking that and looking into some of the additional things you all have reported here.

<comment><USER20> The status dashboard is inaccurate and/or a lie. It only tells about the GKE incident, while in fact the problem also impacts Google Compute Engine users. I was unable to create any google compute instance today, not even a basic 1vcpu, on NA and Europe-west.

As another comment pointed out, what's the point of having so many zones and redundancy around the globe if such global failure can still happen? I thought the "cloud" was supposed to make this kind of failure impossible

<comment><USER21> > I was unable to create any google compute instance today, not even a basic > 1vcpu, on NA and Europe-west.

I've been creating GCP instances in us-central1-a and us-central1-c today without issue. Which zone were you using in NA?

I have been noticing unusual restarts, but I haven't been able to pin down the cause yet (may be my software and not GCP itself).

<comment><USER20> Tried on us-east, us-north, europe-west, also tried asia, with different instance sizes and with both UI and CLI. None worked for me.

<comment><USER22> Same here.

</comment></comment></comment></comment></comment><comment><USER23> Question to Google employees:

Why do you guys suffer global outages? This is your 2nd major global outage in less than 5 years. I'm sorry to say this, but it is the equivalent of going bankrupt from a trust perspective. I need to see some blog posts about how you guys are rethinking whatever design can lead to this - twice - or you are never getting a cent of money under my control. You have the most feature rich cloud (particularly your networking products), but down time like this is unacceptable.

<comment><USER24> > I'm sorry to say this, but it is the equivalent of going bankrupt from a > trust perspective.

It's the opposite really: the expectation that service providers have no unexpected downtime is unrealistic, and it's strange this idea persists.

<comment><USER25> (disclaimer: I work for another cloud provider)

I agree, in general, outages are almost inevitable, but global outages shouldn't occur. It suggests at least a couple of things:

1) Bad software deployments, without proper validation. A message elsewhere in this post on HN suggest that problems have been occurring for at least 5 days, which makes me think this is the most likely situation. If this is the case, presumably given this is multiple days in to the issue, rolling back isn't an option. That doesn't say good things about their testing or deployment stories, and possibly their monitoring of the product? Even if the deployment validation processes failed to catch it, you'd really hope alarming would have caught it.

or:

2) Regions aren't isolated from each other. Cross-region dependencies are bad, for all sorts of obvious reasons.

<comment><USER24> That shouldn't, but they do. S3 goes down [1]. The AWS global console goes down, right after Prime Day outages [2]. Lots of Google Cloud services go down [3, current thread]. Tens of Azure services go down hard [4].

Are software development and release processes improving to mitigate these outages? We don't know. You have to trust the marketing. Will regions ever be fully isolated? We don't know. Will AWS IAM and console ever not be global services? We don't know.

Blah blah blah "We'll do better in the future". Right. Sure. Some service credits will get handed out and everyone will forget until the next outage.

Disclaimer: Not a software engineer, but have worked in ops most of my career. You will have downtime, I assure you. It is unavoidable, even at global scale. You will never abstract and silo everything per region.

[1] [https://www.theregister.co.uk/2017/03/01/aws_s3_outage/](https://www.theregister.co.uk/2017/03/01/aws_s3_outage/)

[2] [https://www.cnbc.com/2018/07/16/aws-hits-snag-after- amazon-p...](https://www.cnbc.com/2018/07/16/aws-hits-snag-after-amazon-prime- day-downtime.html)

[3] [https://www.cnet.com/news/google-cloud-issues-causes- outages...](https://www.cnet.com/news/google-cloud-issues-causes-outages-for- snapchat-spotify-and-others/)

[4] [https://www.datacenterknowledge.com/uptime/microsoft- blames-...](https://www.datacenterknowledge.com/uptime/microsoft-blames- severe-weather-azure-cloud-outage)

<comment><USER26> Can't speak for Google, but Facebook and Salesforce chose Cells for HA.

[http://highscalability.com/blog/2012/5/9/cell- architectures....](http://highscalability.com/blog/2012/5/9/cell- architectures.html)

<comment><USER24> Doesn't look like it was all that helpful to Facebook (as of 1542038976). Facebook.com errors out currently.

> Facebook Platform Appears to be down

> A check of > [https://developers.facebook.com/status/dashboard/](https://developers.facebook.com/status/dashboard/) > returns an error and I'm unable to login with facebook to some of my mobile > apps.

[https://news.ycombinator.com/item?id=18434262](https://news.ycombinator.com/item?id=18434262)

</comment></comment></comment></comment></comment></comment><comment><USER27> Say I were a CTO (I'm nowhere near it), why would I choose GCP over AWS or Azure? Even if after doing a technical assessment and I thought that GCP was technically slightly better, if something happened, the first question I would be asked is "why did you choose GCP over AWS?"

No one would ever ask why you chose AWS. The old "no one ever got fired for buying IBM".

Even if you chose Azure because you're a Microsoft shop, no one would question your choice of MS. Besides, MS is known for thier enterprise support.

From a developer/architect standpoint, I've been focused the last year on learning everything I could about AWS and chose a company that fully embraced it. AWS experience is much more marketable than GCP. It's more popular than Azure too, but there are plenty of MS shops around that are using Azure.

<comment><USER28> \- Native integration with G-Suite as an identity provider. Unified permissions modeling from the IDP, to work apps like email/Drive, to cloud resources, all the way into Kubernetes IAM.

\- Security posture. Project Zero is class leading, and there's absolutely a "fear-based" component there, with the open question of when Project Zero discovers a new exploit, who will they share it with before going public? The upcoming Security Command Center product looks miles ahead of the disparate and poorly integrated solutions AWS or Azure offers.

\- Cost. Apples to apples, GCP is cheaper than any other cloud platform. Combine that with easy-to-use models like preemptible instances which can reduce costs further; deploying a similar strategy to AWS takes substantially more engineering effort.

\- Class leading software talent. Google is proven to be on the forefront of new CS research, then pivoting that into products that software companies depend on; you can look all the way back to BigQuery, their AI work, or more recently in Spanner or Kubernetes.

\- GKE. Its miles ahead of the competition. If you're on Kubernetes and its not on GKE, then you've got legacy reasons for being where you're at.

Plenty of great reasons. Reliability is just one factor in the equation, and GCP definitely isn't that far behind AWS. We have really short memories as humans, but too soon we seem to forget Azure's global outage just a couple months ago due to a weather issue at one datacenter, or AWS's massive us- east-1 S3 outage caused by a human incorrectly entering a command. Shit happens, and it's alright. As humans, we're all learning, and as long as we learn from this and we get better then that's what matters.

<comment><USER27> Your response is from a geek's viewpoint. No insult, intended, I'm first and foremost a 30 year computer geek myself - started programming in 65C02 assembly in 6th grade and still mostly hands on.

But, whether it is right or not, as an architect/manager, etc, you have to think about what's not just best technically. You also have to manage your reputational risks if things go south and less selfishly, how quickly can you find someone with the relevant experience.

From a reputation standpoint, even if AWS and GCP have the same reliability, no one will blame you if AWS goes down if you followed best practices. If a global outage of an AWS resource went down, you're in the same boat as a ton of other people. If everyone else was up and running fine but you weren't because you were on the distant third cloud provider, you don't have as much coverage.

I went out on a limb and chose Hashicorp's Nomad as the basis of a make or break my job project I was the Dev lead/architect for hoping like hell things didn't go south and the first thing people were going to ask me is why I chose it. No one had heard of Nomad but I needed a "distributed cron" type system that could run anything and it was on prem. It was the right decision but I took a chance.

From a staffing standpoint, you can throw a brick and hit someone who at least thinks they know something about AWS or Azure GCP, not so much.

It's not about which company is technically better, but I didn't want to ignore your technical arguments...

 _Native integration with G-Suite as an identity provider. Unified permissions modeling from the IDP, to work apps like email /Drive, to cloud resources, all the way into Kubernetes IAM._

You can also do this with AWS - use a third party identity provider and map them to native IAM user and roles.

[https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_cr...](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for- idp.html)

 _Cost. Apples to apples, GCP is cheaper than any other cloud platform. Combine that with easy-to-use models like preemptible instances which can reduce costs further; deploying a similar strategy to AWS takes substantially more engineering effort._

The equivalent would be spot instances on AWS.

From what (little) I know about preemptible instances, it seems kind of random when they get reassigned but Google tries to be fair about it. The analagous thing on AWS would be spot instances where you set the amount you want to pay.

 _Class leading software talent. Google is proven to be on the forefront of new CS research, then pivoting that into products that software companies depend on; you can look all the way back to BigQuery, their AI work, or more recently in Spanner or Kubernetes._

All of the cloud providers have managed Kubernetes.

As far as BigQuery. The equivalent would be Redshift.

[https://blog.panoply.io/a-full-comparison-of-redshift-and- bi...](https://blog.panoply.io/a-full-comparison-of-redshift-and-bigquery)

 _Reliability is just one factor in the equation, and GCP definitely isn 't that far behind AWS_

Things happen. I never made an argument about reliability.

<comment><USER29> > The equivalent would be spot instances on AWS.

They're equivalent in the sense that you have nodes that can die anytime, but it's much more complicated. You could technically have a much lower cost on AWS by aggressively bidding low but we've had a few instances where the node only lived a few minutes.

Preemptibles nodes are max 24h, and from our stats, they really live around that amount of time. I think the lowest we've had was a node dying after 22h.

You also save out of the box because they apply discount when your instance is running for a certain number of hours.

You can even have more discount by agreeing to a committed use which you pay per month instead of one-shot unlike AWS.

I'm going to add a few more reasons to the above reply:

\- UI and CLI is so much better in GCP

I don't have to switch between 20 regions to see my instances/resources. From one screen, I can see them all and filter however I like.

\- GCP encourage creating different projects and apply same billing.

It's doable in AWS too, of course, but coupled with the fact that you have different projects and regions, and you can't see all instances of a project at once, this makes a super bad experience

\- Networks are so much better in GCP

Out of the box, your regions are connected and have their own CIDR. Doing that in AWS is complicated.

\- BigQuery integration is really good

A lot of logs and analytics can be exported to BigQuery, such as billings, or storage access. Coupled with Data Studio and you have non technical people doing dashboards.

\- Kubernetes inside GCP is a lot better than AWS'

[https://blog.hasura.io/gke-vs-aks-vs- eks-411f080640dc](https://blog.hasura.io/gke-vs-aks-vs-eks-411f080640dc)

\- Firewall rules > EC2 Security Group

\- A lot of small quality of life that makes the experience a lot better overall

... like automatically managing SSH keys for instances, instead of having a master ssh key and sharing that.

Here's the thing though, a lot of GCP can be replicated, just like what you linked for the identity provider. With GCP, there's a lot of stuff out of the box -- so dev and ops can focus on the important stuff.

Overall, AWS is just a confusing mess and offers a very bad UX. Moving to GCP was the best move we've made.

<comment><USER30> Not much to add here other than this reflects my experience as well.

Moved for bizdev reasons, and really appreciated the improved quality of life.

</comment></comment></comment></comment></comment><comment><USER31> This has been going on longer than three days. We have been dealing with this exact issue since at least Monday (11/5) morning in us-central1.

<comment><USER32> same here. using gcloud, not web console

</comment></comment><comment><USER33> >Nov 09, 2018 05:59

>We will provide more information by Monday, 2018-11-12 11:00 US/Pacific.

Wait, did the people tasked with fixing this just take the weekend off?

<comment><USER34> The people tasked with fixing this aren't the ones providing the updates.

<comment><USER33> Fair point but still seems odd that the people providing updates took the weekend off during a large scale customer impacting issue. I'm sure all the people spending the weekend trying to mitigate the impact of this on their infrastructure would love to have timely updates.

<comment><USER35> It's weekend, why wouldn't you take it off? It's just silly software.

<comment><USER36> More to the point, why would you depend on Google for any critical infrastructure after this?

<comment><USER37> Because you tried to run your own infrastructure and it was so much worse.

</comment></comment></comment></comment></comment></comment><comment><AUTHOR> Status page is inaccurate as issues doesn't only affect the web UI, the same operations are not functioning via the CLI.

<comment><USER38> Its kinda strange that HN seems to be the most effective way to give feedback to Google Cloud :/

<comment><USER39> I also find it weird that on HN where normally people are very skeptical of any argument without data backing it, when it comes to this outage, people are assuming everything written here affects everyone.

Perhaps some of the issues are localized? Perhaps it's even user error (it happens, you know?). But because a small amount of HN users say "it's everywhere!" then suddenly people reach for their pitchforks.

Sometimes we just don't have all the information.

</comment></comment></comment><comment><USER27> A generic question: Our company is completely dependent on AWS. Sure we have taken all of the standard precautions for redundancy, but what happened here could just as easily happen with AWS - a needed resource is down globally.

What would a small business do as a contingency plan?

<comment><USER40> Infrastructure as code.

Terraform using AMIs plus chef recipes that work in the cloud and bare metal. Dont use AWS specific services.

This would allow you to spin over to another cloud provider , vsphere or bare metal with minimal work

<comment><USER41> I think you are downplaying minimal

<comment><USER25> I don't think OP was intending "minimal" to mean it would be easy to get to the stage where it's possible, just that once you've got all your infrastructure-as-code stuff set up correctly, you ought to be able to just be pressing buttons / running scripts and have your infrastructure up and running in another cloud provider.

Even when working in small companies with small infrastructure, I've kept recreation of infrastructure as one of my high priorities (one reason it really bugged me in one job to have to depend on Oracle Databases that I couldn't automate to the same degree.)

In my mind, it's not different from the importance of having, and testing restoration of, backups. If your infrastructure gets compromised somehow, or you find yourself up the creek with your provider, you've _got_ to be able to rebuild everything from scratch.

<comment><USER37> The infrastructure has always been the easy part, as long as the company is willing to pay for multiple datacenters.

Then you realize a lot of software and databases can only run from a single instance, zero support for multi regions, and you're not gonna to rewrite everything and resiliency just can't happen.

</comment></comment></comment></comment></comment><comment><AUTHOR> UPDATE: Got some clarity, these issues are caused by "resource exhaustion" meaning there are no resources left to be allocated.

<comment><USER30> I'm curious to see if this is true.

I faced some pretty serious resource allocation issues earlier in the year. The us-west1-a region was oversubscribed. I was unable to get any real information from support with regard to capacity. Eventually my rep gave me some qualitative information that I was able to act on.

</comment></comment><comment><USER42> I honestly don't mind if providers have outages - we can't expect 100.00% accuracy, I know the systems I manage certainly don't achieve that.

One thing I _do_ care about though, is root cause analysis. I love reading a good RCA, it restores my faith in the company and makes me trust them more.

(I'm not affect by the GKE outage so opinions may differ right now!)

</comment><comment><USER43> Do not use GCP without paying for support. We have had resource allocation errors for weeks, as have a lot of other people. Check out the posts in their forum where folk on basic support get zero love. [https://groups.google.com/forum/?utm_medium=email&utm_source...](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!forum/gce- discussion)

</comment><comment><USER44> Been trying to spin up vm instances all day, had to try every single zone just to get one up. Not only is this incredibly harmful to a technology business dependent on this infra, it wasnt obvious to me what the issue was until I tried creating instances. Nothing says, hey resources are constrained here, try this one. Just about ready to bite the bullet and move to aws.

<comment><USER22> Same here. We have spent 2 days trying to create instances and migrate images just to figure out later they can't start.

Right when I convinced our project to get migrated from AWS...

<comment><USER45> Same question... why would you do that? AWS is super stable most of the time. I have been running k8s over EC2 (not eks) for a year and works like a charm. I've even run experiments using spot instances and it's pretty good (no guarantee there of course).

<comment><USER22> We did. Our customer had a bucket shared to a role they created, we spent a week back and forth trying to mount such bucket in our instance using fuse. Mounting with gcsfuse took 5 minutes (although no role used, so perhaps an unfair comparison). In general, I found Gcloud a lot easier to work with.

</comment></comment></comment></comment><comment><USER46> Seems to be some weird underlying issue going on at GCP at the moment. Had cloud build webhooks returning a 500 error. Noticed we were at 255 images and deleting some fixed the issue. Created a P2 ticket about the issue before we managed to solve it and haven't had a response in 40+ hours.

The timeline of this disruption matches when we started experiencing cloud build errors.

<comment><USER47> Outsider here, but I believe Cloud Build runs on GKE Jobs, so if they're having trouble, it does indeed sound related.

</comment></comment><comment><USER48> "third consecutive day of service disruption" is not an accurate statement? Latest update was Nov 11 saying things resolved on Nov 9.

[https://status.cloud.google.com/incident/container- engine/18...](https://status.cloud.google.com/incident/container-engine/18005)

<comment><USER48> If all nodes in GKE clusters were down for 3 days, I would consider this newsworthy and shocking. This... is not. Come on, people.

</comment></comment><comment><USER28> Cloud providers have all of the potential in the world to make each region truly isolated. I shouldn't have to architect my application to be multi- cloud, at least for stability reasons.

Yet, somehow every major cloud provider experiences global outages.

That old AWS S3 outage in us-east-1 was an interesting one; when it went down, many services which rely on S3 also went down, in other regions beside us- east-1 because they were using us-east-1 buckets. I have a feeling this is more common than you'd think; globally-redundant services which rely on some single point of geographical failure for some small part.

<comment><USER49> AWS regions are very much isolated from each other.

We know because we are still waiting here in ap-southeast-2 for services such as EKS to be made available. Pretty sure that any reliance within their backend services on us-east-1 was just a temporary bug and nothing systemic.

</comment></comment><comment><USER50> Our company is dependent on this as well and the way customer service has been handling this has been abysmal thus far.

</comment><comment><USER51> There is no magic public clouds have incredibly complex control planes and marketing fluff aside you would very likely experience much better uptime at singe top tier DC than @ a cloud provider.

</comment><comment><USER52> The is not only GKE. But for GCE as well. I cannot create instance is almost all zones. I tried both preemptible and normal as well.

Always saying resource not available. My account is a pretty new account.

In contrast, one of my friend is having a pretty old account which is very active. He has no such issue.

So I think due to this issue, Google has enabled some resource limitation for new accounts.

But they should properly communicate this issue.

</comment><comment><USER53> Oh man must be a tough time to be an SRE at google cloud. But... they're Google. They have been doing internal cloud for years and years. Borg — which K8s is a reimplementation if — has been the heart of Google for so long now you'd think they'd be able to architect their systems to have no outages whatsoever. I mean nobody is perfect but this looks bad.

<comment><USER9> Goes to show outsourcing infrastructure is more about blame shifting so that when things go wrong its "not our fault" than reducing actual downtime.

</comment></comment><comment><USER54> Doesn't GKE "just" run an independent Kubernetes cluster on customer VMs? How is a widespread outage like this possible?

<comment><USER55> GKE does the creation of the VMs and setup of them, joining them to the cluster and applying labels for example.

The specific issue appears to be about creating new "node pools". Creating standard VMs in GCP works fine however, so this is specific to GKE and their internal tooling that integrates with the rest of GCP.

GKE doesn't (at least to my knowledge) allow you to create VMs separately and join them to the cluster in any kind of easy fashion.

<comment><USER56> It's actually not just GKE, there have been issues creating normal VMs since late Friday night. It seems anything that required creating VMs gave back resource exhaustion errors. I finally got a cluster for us-east1 setup last night so it looks like the resource issues are clearing up though.

</comment></comment></comment><comment><USER57> As of this morning, I am _still_ unable to reliably start my docker+machine autoscaling instances. In all cases the error is "Error: The zone <my project> does not have enough resources available to fulfill the request"

An instance in us-central1-a has refused to start since last Thursday or Friday.

I created a new instance in us-west2-c, which worked briefly but began to fail midday Friday, and kept failing through the weekend.

On Saturday I created yet another clone in northamerica-northeast1-b. That worked Saturday and Sunday, but this morning, it is failing to start. Fortunately my us-west2-c instance has begun to work again, but I'm having doubts about continuing to use GCE as we scale up.

And yet, the status page says all services are available.

Is the typical of others' experiences?

</comment><comment><USER58> Right now we're experiencing an issue where a small percentage of end users on our GKE site are getting super slow speeds. The issue is ISP related as they can switch to a 4G hot spot in the same location and get normal speeds... and inside our system the timing looks normal. So there's a slowdown either TO the load balancer or FROM the load balancer. Took a week to convince Google's support contractor to even believe it wasn't an issue with our site and their advice is generally along the lines of Turn it off and Turn it back on again (which might actually fix the problem) though that's easier said than done in GCP.

</comment><comment><USER59> I use preemptible machines in autodialing and for first time did not have any machines available for multiple hours yesterday. I am wondering whether this falls under the normal preemptible behaviour or this service degradation.

</comment><comment><USER60> If anyone is interested, here is my documented experience with this issue. I freaking love GCP and GKE, although I have not production environment as it was a HA cluster in us-central1. Working federation now.

[https://stackoverflow.com/questions/53244471/gke-cluster- won...](https://stackoverflow.com/questions/53244471/gke-cluster-wont- provision-in-any-region)

</comment><comment><USER55> Is this just about creating new pools? I haven't noticed an issue with our existing pools scaling.

<comment><AUTHOR> You were able to add more Nodes to you're pool? Are you using any auto scaling?

</comment></comment><comment><USER61> When guerilla marketing backfires

</comment><comment><USER62> As someone currently trying to decide between GCP and AWS for a project, is this a regular occurrence?

And for those who have used both, which would you go with today?

</comment><comment><USER63> Had it affected all regions or just some?

Is there another status page Google? Coz the last update I'm looking at...is dated on the 9th..

<comment><USER14> The general page is at [https://status.cloud.google.com/;](https://status.cloud.google.com/;) you can scroll down to see GKE, and my (unofficial) belief is that [https://status.cloud.google.com/incident/container- engine/18...](https://status.cloud.google.com/incident/container-engine/18006) should have closed out [https://status.cloud.google.com/incident/container- engine/18...](https://status.cloud.google.com/incident/container-engine/18005)

_If_ that's the case, something else is causing the error messages other people are seeing

</comment></comment><comment><USER64> Offtopic but are there some documented exceptions to the "keep the original title" rule?

</comment><comment><USER65> Why do cloud providers have more global outages than major flagship websites like google.com?

<comment><USER66> Whey don't run on the same infra. Amazon.com doesn't run on AWS.

<comment><USER67> On the contrary, it does. They made the transition gradually.

</comment></comment></comment><comment><USER68> Things break after everybody has gone home on a Friday? 3 day disruption.

</comment><comment><USER69> I'd like to upvote, but 666 points seemed relevant.

</comment><comment><USER70> Time to use Mesos.

</comment><comment><USER71> I have a question. At what point does k8s make sense?

I have a feeling that a microservice architecture is overkill for 99% of businesses. You can serve a lot of customers on a single node with the hardware available today. Often times, sharding on customers is rather trivial as well.

Monolith for the win! Opinions?

<comment><USER72> As someone whose daily work happens on k8s, I'd say you better be paining a lot before you move to k8s. I take great care to avoid this, but if you aren't careful, you can end up "feeling" productive on k8s without actually being productive. K8s gives a lot of room for one to tweak workflows, discuss deployment strategies, security, "best practices", etc. And you can get things done reasonably fast. But that's like a developer working all day on fine tuning their editor and comparing and writing plugins and claiming that they are getting productive.

The key issue here is that k8s was written with very large goals in mind. That a small business can easily spin it up quickly and run a few microservices or even a monolith + some workers is just coincidental. It is NOT the design goal. And the result of that is that a lot of the tooling and writing around k8s reflects that. A lot of the advice around practices like observability and service meshes comes from people who've worked in the top 1% (or less) of companies in terms of computing complexity. What I'm personally seeing is that this advice is starting to trickle down into the mainstream as gospel. Which strangely makes sense. No one else has the ability to preach with such assurance because not many people in small companies have actually been in the scenarios of the big guns. The only problem is that it's gospel without considering context.

So at what point does k8s make sense? Only when you have answers to the following:

* Getting started is easy, maintaining and keeping up with the going ons is a full time job - Do you have at least 1 engineer at least that you can spare to work on maintaining k8s as their primary job? It doesn't mean full time. But if they have to drop everything else to go work on k8s and investigate strange I/O performance issues, are you ready to allow that?

* The k8s eco system is like the JS framework ecosystem right now - There are no set ways of doing anything. You want to do CICD? Should you use helm charts? Helm charts inherited from a chart folder? Or are you fine using the PATCH API/kubectl patch commands to upgrade deployments. Who's going to maintain the pipeline? Who's going to write the custom code for your github deployments or your brigade scripts or your custom in house tool? Who's going to think about securing this stuff and the UX around it. That's just CICD mind you. We aren't anywhere close to the weeds about deciding if you want to use ingresses vs Load balancers and how you are going to run into service provider limits on certain resources. Are you ready to have at minimum 1 developer working on this stuff and taking time to talk to the team about it?

* Speaking about the team, k8s and Docker in general is a shift in thinking - This might sound surprising but the fact that Jessie Frazelle (y'all should all follow her btw) is occasionally seen reiterating the point that containers are NOT VM's is a decent indicator that people don't understand k8s or Docker at a conceptual level. When you adopt k8s, you are going to pass that complexity to your developers at some point. Either that or your dev ops team takes on that full complexity and that's a fair amount to abstract away from the developers which will likely increase the work load of devops and/or their team size. Are you prepared for either path?

* Oh also, what do your development environments start to look like? This is partly related to microservices but are you dockerizing your applications to work on the local dev environment? Who's responsible for that transition? As much as one tries to resist it, once you are on k8s you'll want to take advantage of it. Someone will build a small thing as a microservice or a worker that the monolith or other services depend on. How are you going to set that up locally? And again, who's going to help the devs accumulate that knowledge while they are busy trying to build the product. (Please don't put your hopes on devs wanting to learn that after hours. That's just cruel).

I can't write everything else I have in mind on this topic. It'd go on for a long long time. But the common theme here is that the choice around adopting k8s is generally put on a table of technical pros and cons. I'd argue that there's a significant hidden cost of human impact as well. Not all these decisions are upfront but it is the pain that you will adopt and have to decide on at some point.

Again, at what point does k8s make sense? Like I said, you ideally should be paining before you start to consider k8s because for nearly every feature of k8s, there is a well documented, well established, well secured parallel that already exists in the myriad of service providers. It's a matter of taking careful stock of how much upfront pain you are trading away for pain that you WILL accumulate later.

PS - If anyone claims that adopting a newer technology is going to make things outright less painful , that's a good sign of immaturity. I've been there and I picture myself smashing my head into a table every now and then when I think of how immature I used to be. Apologies to people I've worked with at past jobs.

PPS - From the k8s site, "Designed on the same principles that allows Google to run billions of containers a week, Kubernetes can scale without increasing your ops team." <\-- is the kind of claim that we need to take flamethrowers to. On paper, 1 dev with the kubectl+kops CLI can scale services to run with 1000's of nodes and millions of containers. But realistically, you don't get there without having incurred significantly more complex use cases. So no, nothing scales independently.

<comment><USER73> I fully agree with you, and personally have taken the path of using Docker Swarm as a step-up to k8s, as it was so much easier to get along with. I would certainly recommend this to smaller businesses.

</comment></comment></comment><comment><USER74> Daily reminder that there's no "cloud", just other people's computers. ( ͡° ͜ʖ ͡°)

</comment><comment><USER75> If a hosting service is down and nobody uses it, is there really any disruption?</comment>


Tsū is a free network that gives the social revenues back to you - <AUTHOR> https://www.tsu.co/

<comment><USER2> Nice idea, so why don't you just let people ,like, join right now instead of creating needless artificial scarcity. Bigger, more connected and wealthier companies could be doing the same thing next week and you'd be left there with your short codes and not enough people.</comment>


Decoded: GNU Coreutils - <AUTHOR> https://www.maizure.org/projects/decoded-gnu-coreutils/index.html <comment><USER2> Interesting project, this would make a good teaching tool for people interested in more operating system interaction.

On the website it would be nice if you had a way to flag which ones were fully annotated. But I like the main page design. The block graphics on the flow is very helpful.

You've done some very nice work here.

<comment><USER3> Agreed, this would be very beneficial to students and people learning C.

<comment><USER4> I have used the source code for Plan 9's core utilities as a teaching aid for students learning the basics of C and operating systems to great effect. I have found GNU's tools to contain a lot of "cruft" to make them run fast, which is not necessarily conducive for those who are not familiar to why they work.

<comment><USER5> Busybox and Toybox also contain pretty simple and easy to follow implementations.

<comment><USER2> Busybox is a fun read on how they got all that functionality into such a small space. It always amazes me that my home router has all those functions because of Busybox being loaded.

OTOH, I shouldn't because I think my current router has more CPU / Memory than my first mainframe.

</comment></comment></comment></comment></comment><comment><USER6> Huh, I'd never heard of FTS before, the functions used by Coreutils to traverse the filesystem:

[http://man7.org/linux/man-pages/man3/fts.3.html](http://man7.org/linux/man- pages/man3/fts.3.html)

<comment><USER7> FWIW, the POSIX equivalents are `ftw` and `nftw`[1]. POSIX 2008 deprecates `ftw`.

[1]: [https://pubs.opengroup.org/onlinepubs/9699919799//functions/...](https://pubs.opengroup.org/onlinepubs/9699919799//functions/nftw.html)

<comment><USER8> ftw/nftw is a crap interface as it relies on C callbacks and isn't re- entrant.[1] Android/Bionic, Darwin/macOS, DragonflyBSD, FreeBSD, glibc, musl libc, OpenBSD, NetBSD, and even Solaris all support the FTS API.[2]

Of the extant Unix environments only AIX and QNX seems to lack it. HP-UX doesn't seem to support it, either, but I don't count HP-UX as extant. In any event it's trivial to copy an FTS implementation from a BSD.

I'm a strong advocate for adherence to POSIX, but in this case there's significant benefit from using FTS and very little, if any, cost.

[1] FTS uses a C callback for its comparator, which can be a headache but not nearly to the same extent as it is with nftw.

[2] It's originally from BSD Reno. It was adopted by glibc, which in turn has forced commercial vendors like Solaris to support it as they're chasing Linux/glibc API compatibility. I would expect AIX to add it eventually.

<comment><USER7> Ah, linux-man bites again: the Linux versions of `nftw` and `ftw` claim to be re-entrant, so I assumed that POSIX specified them as such. Looks like I was wrong about that.

<comment> Anthony-G Red Hat-based distributions include POSIX man pages out of the box.

        $ whatis ftw
        ftw (3)              - file tree walk
        ftw (3p)             - traverse (walk) a file tree

Run `man 3p ftw` to read the POSIX version.

On Ubuntu systems, the following packages can be installed to provide the POSIX man pages:

        manpages-posix
        manpages-posix-dev

</comment></comment></comment></comment></comment><comment><USER9> Super interesting. I had a brief look at "head" and "df" and the flows seem to be complete.

The flow for "tr" is not completely right given that the second parameter ("string2") is optional with certain options.

The text for "yes" mentions how it fills a buffer with multiple copies of the string to achieve its very high performance. Nice!

<comment><USER10> Reminds me of a time at a large animation studio where I was working on performance issues with the in-house asset management system.

The asset management tool was fully commandline based, no GUI. An artist showed me what was involved in committing their scene: launch the command, type yes, hit return, wait about 30 seconds, type yes, hit return, repeat. I piped yes to stdin and it ran for 47 hours.

If you need a performance-optimized version of "yes" there are probably other aspects of your architecture you should look at first.

<comment><USER11> yes(1) takes a custom "expletive", so in the performance context it's usually used for flood filling in situations where e.g. /dev/zero or /dev/urandom are not appropriate sources for some reason.

<comment><USER4> yes outputs newlines, though, which makes it somewhat annoying to use for flood filling in practice.

</comment></comment></comment></comment><comment><USER12> Nice project. Just looked for the source code of "yes" ([https://github.com/coreutils/coreutils/blob/master/src/yes.c](https://github.com/coreutils/coreutils/blob/master/src/yes.c)), cause it seems to be the easiest code to understand.

I didn't get it. Damn. Looks still complicated.

<comment> yjftsjthsd-h Yeah, `yes` ended up complex in order to get better performance; if you want something simpler, maybe try [https://www.maizure.org/projects/decoded-gnu- coreutils/env.h...](https://www.maizure.org/projects/decoded-gnu- coreutils/env.html) ?

</comment></comment><comment><USER13> Oh, this is great! And it's one piece of something I've been looking for (so hopefully a good comment section to ask for more): a book on how (GNU/)Linux 'works'.

i.e. I'm not interested in a book of commands or cheat sheets, I can use `man` and SO, Arch wiki, etc. for that when needed.

I think part of the problem is that I don't know what I don't know - but I discovered namespaces (`/proc/<pid>/ns/<type>`) recently and don't know much about it but thought it was interesting, and managed to use it to do what I needed to overcome a problem I was having with `ip netns`. A similar 'aha' was with inodes a while ago.

So, any recommendations for a book on 'how Linux works'? I think it's a gap in my understanding (academically EE/CS - hardware, up to OS but not Linux- specific, theoretical CS; professionally software, 'using' Linux). Best candidate I've found is Brian Ward's 'How Linux Works: what every superuser should know'.

</comment><comment><USER14> I'm not particularly interested in coreutils per se but this is a super- interesting way to look at any piece of software. I just _love_ the block diagrams... why can't source code look like this?

Great work!

<comment><USER15> There are languages that model code like this. Using one for a while will likely teach you why. If you think "spaghetti" is an apt description for text code, wait until you see flow based code that's evolved in place for a while.

<comment><USER14> Presumably all the detail is shown at once? I'm picturing something more like Google Maps navigation where you zoom into the source diagram and the code appears.

Perhaps I'm imagining an alternative way to manage source trees, or modules, rather than the code in individual source files. A hierarchy like a traditional file system but with extra kinds of relationships. Perhaps it'd have to be curated like reference documentation, perhaps it could be automatically generated from source directives. I dunno, just idle thoughts...

</comment></comment></comment><comment><USER16> This is probably going to be an unpopular opinion, I dont agree with the criticism of the use of goto.

I know, I know, the classic "goto Considered Harmful". And, yes, it can be abused just like every other programming construct.

I see 2 generally good use cases for goto: 1) error handling to cleanup in failure in C (in C++ RIAA constructs ahould be used instead) 2) exitting nested loops, instead of a flag and a break.

I've also seen C++ code use do while loops with while(false) so that they could use "break" to exit the non loop. Its very confusing and non obvious, because you see the do, and thus expect a loop. Meanwhile (the absurdly long function) has the while(false) on a separate screen, not making it clear that the code doesn't actually loop. Just use the goto. It's clearer and omits the abuse of a looping primitive (that doesn't actually loop).

Yes, goto can be abused, so can many other statements.

Sorry, /rant

</comment><comment><USER17> that's fairly cool. Wish such sites were available for more open source packages.

Always wanted to know the algorithms used in sort.

So ls has more source lines than sort? Funny, but a bit depressing too.

<comment><USER18> ls is kind of complicated, has a billion flags, and has to print pretty so it has to do some ioctls to get the terminal width and such, then column things out nicely so they don't overflow. Glad I'm not implementing ls from scratch.

<comment><USER19> exa[0] dubs itself a "modern" ls written in Rust. Interesting to compare. Still over 6,000 lines not including comments.

[0] [https://github.com/ogham/exa](https://github.com/ogham/exa)

</comment></comment></comment><comment><USER20> This guy has some really nice articles, but I'm unavailable to find any link to his RSS feed. Any help?

</comment><comment><USER21> I've been trying to contribute additional flags for some coreutils commands, but I've been having trouble building on macOS. Should I use a certain VM or container instead? Does anyone have a link to instructions for building on a mac?

<comment><USER22> Do you have a modern GCC installed?

<comment><USER21>
      $ gcc -v
      Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/c++/4.2.1
      Apple LLVM version 10.0.1 (clang-1001.0.46.4)
      Target: x86_64-apple-darwin18.6.0
      Thread model: posix
      InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

<comment><USER23> I would not at all be suprised to learn that GNU coreutils does not build with clang/llvm. Try installing a real GNU C compiler with homebrew (brew install gcc)

</comment></comment></comment></comment><comment><AUTHOR> If you like this, the author posts a number of other 'Decoded' episodes: [https://www.maizure.org/projects/](https://www.maizure.org/projects/)

</comment><comment><USER24> I never noticed `runcon` before, it and `chcon` seem decidedly out of place being in core-utils, being that they're a) linux specific, b) SElinux specific.

</comment><comment><USER25> This is amazing. The author needs a donate button!

</comment><comment><USER26> Are you doing this by manual inspection or in some semi-automated or automated way? In any case, very nice work.

</comment><comment><USER27> this is very cool. Yet GNU utils have so many options that the interesting bits are difficult to find. The same display but for openbsd utils would be still more enlightening.

</comment><comment><USER28> the diagram is really nice,is it done by dia?

I use dia once a while, and wish freeplane could draw arrows between its nodes as freeplane is easier to add child nodes.

</comment><comment><USER29> This is fantastic work!

</comment><comment><USER30> It is true archeology.

</comment><comment><USER31> Or, just browse the originals:

[https://www.tuhs.org/cgi- bin/utree.pl?file=V7/usr/src](https://www.tuhs.org/cgi- bin/utree.pl?file=V7/usr/src) [https://www.tuhs.org/cgi- bin/utree.pl?file=4.3BSD/usr/src](https://www.tuhs.org/cgi- bin/utree.pl?file=4.3BSD/usr/src)

and some of their current decedents:

[https://svnweb.freebsd.org/](https://svnweb.freebsd.org/) [http://cvsweb.openbsd.org/cgi-bin/cvsweb/](http://cvsweb.openbsd.org/cgi- bin/cvsweb/) [http://cvsweb.netbsd.org/bsdweb.cgi/src/?only_with_tag=MAIN](http://cvsweb.netbsd.org/bsdweb.cgi/src/?only_with_tag=MAIN) [https://gitweb.dragonflybsd.org/dragonfly.git](https://gitweb.dragonflybsd.org/dragonfly.git)

conveniently all in one source tree, self hosting, and buildable with a single command.

it is almost as if they were designed as part of a single coherent operating system along with the very C language itself..

as discussed previously, typically with more intelligible sources:

[https://news.ycombinator.com/item?id=14542938](https://news.ycombinator.com/item?id=14542938)

and usually with better man pages ([https://www.freebsd.org/cgi/man.cgi](https://www.freebsd.org/cgi/man.cgi))

unix came first. gnu and posix came later.

the unix way is the standard, not posix, and not gnu.

from stallman himself ([https://stallman.org/articles/posix.html](https://stallman.org/articles/posix.html)):

" It seemed to me that nobody would ever say "IEEEIX", since the pronunciation would sound like a shriek of terror; rather, everyone would call it "Unix". That would have boosted AT&T, the GNU Project's rival, an outcome I did not want. So I looked for another name, but nothing natural suggested itself to me.

So I put the initials of "Portable Operating System" together with the same suffix "ix", and came up with "POSIX". It sounded good and I saw no reason not to use it, so I suggested it. Although it was just barely in time, the committee adopted it. "

so, as can be seen, posix is RMS's way of keeping people from calling unix unix, and as a result today, now that AT&T Unix is nearly dead, people lose track of the fact that BSD _is_ Unix, and don't know what Unix actually is, and are forced to 'decode' the sometimes deliberately less clear cloned sources not knowing to look at the simpler original.

<comment><USER6> I think you missed the point of the website. It's not about the source code itself, but providing a higher-level overview of the utilities' architecture.

<comment><USER31> Did I?

"This resource is for novice programmers exploring the design of command-line utilities. It is best used as an accompaniment providing useful background while reading the source code of the utility you may be interested in. "

^ "best used as an accompaniment"

I posit that GNUs intentionally obfuscated code design[1]^ requires the use of such a site, whereas, reading the originals and their decedents, provides a similar level of understanding program functionality _in addition_ to the actual historical context in which most of the tools were developed _and, also,_ typically occurs in the same source tree as the kernel, c library, and build toolchain implementing the OS side of the equation and so is a better and more productive thing to do with one's time that will still apply to reading and understanding coreutils sources.

apologies if unclear. but kudos to the author for trying to do something positive to spread knowledge of system internals, lest I be misconstrued.

.. [1] [https://www.gnu.org/prep/standards/standards.html#Reading- No...](https://www.gnu.org/prep/standards/standards.html#Reading-Non_002dFree- Code)

.. ^: the design of a utlity like 'cat'\+ has generally obvious direct implementation. coming up with something else to be different necessitates doing goofy stuff. see also other hn thread referenced.

.. +: see also: [https://github.com/coreutils/coreutils/blob/master/src/cat.c](https://github.com/coreutils/coreutils/blob/master/src/cat.c) vs [https://www.tuhs.org/cgi- bin/utree.pl?file=V7/usr/src/cmd/ca...](https://www.tuhs.org/cgi- bin/utree.pl?file=V7/usr/src/cmd/cat.c)

<comment><USER32> I'm pretty curious on how you conclude that GNU is "intentionally obfuscating code" from [1]. I read [1] as good advice to avoid inadvertently getting code into GNU that could be claimed by copyright. Focus on speed instead of memory; simplicity instead of speed. I don't see "make it different for difference sake."

I also conclude the opposite from the cat implementations. I really don't see how the BSD cat is a more "pure" or straight forward implementation. I find the GNU cat to have way more options and easier to read source code. My guess is that novice programmers would find the GNU version easier to grok, which also probably aligns more to the goal of GNU. Or I just like to read obfuscated code.

<comment><USER33> _I really don 't see how the BSD cat is a more "pure" or straight forward implementation. I find the GNU cat to have way more options_

Way more options is "cat -v considered harmful" not thought any more?

<comment><USER32> If you aren't trying to be Unix I would imagine you aren't thinking that. GNU probably doesn't subscribe to that notion.

</comment></comment></comment></comment></comment></comment><comment><USER34> Rather than a flow chart, etc. It would better to convert the utilities to simple C with no error handling or optimization. So `cat` would be about 5 lines. No mmap() or fancy stuff.

<comment><USER35> You might find source of Xv6 - educational version of unix - useful. Here is their implementation of cat, all 44 lines of it: [https://github.com/mit- pdos/xv6-public/blob/master/cat.c](https://github.com/mit- pdos/xv6-public/blob/master/cat.c)</comment></comment>