Ask HN: How Long Does it usually take to become profitable? - <AUTHOR>

Supposing that I start up today, how long do you think would it take to become profitable. I mean to exclude here start ups that might go viral, or, on the other hand, start ups that might take much longer than usual. I mean to include on average, only, and without funding. Would you say one or two years?<p>If so, how would you support yourself during this time? <comment><USER2> I don't think you can give a fixed timeframe. Many startups are not "profitable" but are capable of paying their founders salaries. Depends on what your profit goals are.

</comment><comment><USER3> I would use:

Y = λf.(λx.f (x x)) (λx.f (x x))

Where y is months to profitability, λ is # of cofounders, and x is funding.

Serious answer: Who knows. Some businesses models just can't achieve profitability no matter how much time, money and brainpower you throw at it. So without some idea of the idea, no way to really answer the question.

</comment><comment><USER4> The average is 31 days per co founder, plus 10 days for each 100K in funding.</comment>


OUC coal plants linked to east Orange County cancer spike, lawsuit claims - <AUTHOR> https://www.orlandosentinel.com/news/os-ne-ouc-coal-class-action-suit-20181218-story.html <comment><USER2> OUC exists for the benefit the City of Orlando's residents. Stanton was built where it was built to externalize it's pollution keeping utility rates for its citizen customers low and avoiding property value impacts to the city. It was deemed acceptable to externalize the impact on the unincorporated community of Bithlo. Since Stanton was built, the area within city limits around the old OUC Ivanhoe Plant has significantly gentrified.

Orange County began promoting development in the area in the 1980's. It widened Lake Underhill Road and extended Alafaya Trail south of SR50 toward FL528 (the Bee Line). Stanton was part of the development plan because OUC/Orlando chipped in on the infrastructure costs.

The lawsuit's 30,000 potentially affected people are not among OUC's 200k power customers. They live outside Orlando's city limits in Orange County. Their power is at market rate from Duke Energy. All those people live out near Stanton by design. The first of those developments, Huckleberry, was affordable and convenient because the road network was planned for further development, e.g. the 408.

Coal firing Stanton was always a bit of boondoggle. Orlando is a long way from any mineable coal face. People knew its location would exhaust into both of Florida's major water sheds. The site sits in the south end of the St. John's watershed and just north of the Florida Everglades headwaters in Deseret Ranch south of the Bee Line.

</comment><comment><USER3> I live in a former east german neighborhood in Berlin. When I moved here 10 years ago, lots of apartments were still coal heated. Buildings had lots of dirt on them and you could smell it in the winter. Burning coal creates lots of smelly smog. That's also a huge problem in places like China.

Smog kills people by the millions. It's a slow killer that's why nobody gives a shit. Average lifespan decreases measurably by many years in areas with heavy smog. Living next to a busy street or coal plant is not good for your health.

That's why diesel, coal, etc. need to get banned in a hurry. Suing coal plants for killing their neighborhoods is just as logical going after tobacco companies. It will basically kill that industry in no time. Good riddance; the sooner the better. Meanwhile, make them pay for the damage they do. Incentivize them to clean up their act or pull the plug on their operations.

<comment><USER4> What do you propose as replacement for coal? Both for electricity and heating up homes?

<comment><USER3> Clean energy + batteries seem to be eating that up rapidly in places where they invest in that kind of thing. Simply ramp up investments there and this problem can be solved.

Even the US is doing this. Meanwhile, force remaining plants to install filters and fine them appropriately for killing their neighbors every month they don't do this. Of course, there's no such thing as cheap coal when you factor that in. Most of this stuff is already unsustainable from an economic point of view. That's why plants are shutting down, plans for new ones are getting shelved, etc.

Coal without subsidies and government protection, is not going to last long.

<comment><AUTHOR> I agree with you, but interesting part about this suit is that it does not claim that contaminants are being emitted via the smoke. There are essentially large hills of radioactive coal ash just sitting on the site, the residue from the burned coal. They are just sitting in the open and the wind is blowing it into the surrounding area.

<comment> village-idiot Ironically, these piles are usually produced by the electrostatic filters that prevent smog and smoke. IIRC most plants keep them in (really toxic) ponds specifically to prevent the dust issue.

<comment><USER5> Can't we process that into something useful? There has to be some useful thing that can be done with that.

<comment><AUTHOR> They use some of it to make cinder blocks.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> In Ontario we closed every single one of our coal plants over a decade ago and it is a night and day improvement in air quality. Our power generation is mostly nuclear and hydroelectric, with about 5-10% filled in by a combination of natural gas and wind generation. I drive around in an electric car which is fueled almost entirely by non-CO2 emitting energy. That feels good.

A global ban on all coal energy would be a modest but important improvement on many fronts, even if it means a switch to natural gas.

<comment><USER7> Too late. To get this natural gas would require fracking, and much of the world now believes their environment will be destroyed if they frak. So we end up with developing nations burning dirty coal when switching to nat-gas would be cheaper and 99% cleaner.

<comment><USER8> >believes their environment will be destroyed if they frak.

Because there are valid reasons to believe so based on the way some fracking operations have been run. Contaminated ground water (look at Wyoming) and potentially even seismic related risk (a 5.6M earthquake in Oklahoma may have been directly caused by fracking).

Hydraulic fracturing can also result in its own air pollution, noise pollution etc.

Sure, it can probably be done safely but a single mistake could have wide- reaching and long-lasting implications. Instead of replacing one bad thing with another risky thing we need to get less spooked by modern nuclear reactors,and renewables, until (if ever) we figure out fusion.

<comment><USER9> "directly caused by fracking"

Those big earthquakes in Oklahoma were caused by waste water injection wells so "indirectly caused by hydrocarbon extraction" (fracking and oil well waste water being injected) would be closer to the truth.

</comment></comment></comment></comment><comment><USER10> Good. It's outrageous that people are allowed to dump waste on other people and we just act like it's perfectly normal. It's even more outrageous that the people who claim to be the most concerned with individual rights and basic freedoms don't see pollution for the massive violation of property and personal rights that it obviously is.

<comment><USER9> I would say that almost all of "them" are concerned with pollution, but people with that view are not the ones you hear about. They would just sound like reasonable people then. It wouldn't set off peoples' outrage (like you just stated) button and get them to click on the article taking about "them".

</comment></comment><comment><USER11> I lived in Avalon Park and Stoney Brook for 3 years, got diagnosed with Non- Seminoma Testicular Cancer (one of the rarer ones) 3 years ago.

I'm conflicted on whether this is responsible or not, no family history of Testicular Cancer. I'm not sure if I should join this class action lawsuit, it seems unlikely to go anywhere since proving that cancer was caused by _XYZ_ is difficult.

<comment><USER2> I hope you have a quick and full recovery.

<comment><USER11> It was an arduous process, 12 weeks of chemotherapy due to it spreading to 2 lymphnodes.

But 3 years later I am still cancer free!

</comment></comment></comment><comment><USER12> Holy shit. I lived in Eastwood for 3 years. My brother was born and raised in that neighborhood and still lives there. My father died of pancreatic cancer. He was one of the first homes bought and built on his street when Eastwood was first being built. His wife suffered from ovarian cancer at one point. I dont know what to say.

Edit: My wife went to high school in Avalon Park for 4 years and lived in that neighborhood for most of her life as well. She also had teenage osteo sarcoma.

</comment><comment><USER13> I'm fairly sure EU powerplants have been required to do ash capture (usually electrostatic) in addition to acid rain mitigation measures for some time, to avoid this kind of thing.

</comment><comment><USER14> It is incredibly ironic that people protesting nuclear powerplants increase their exposure to radiactivity :)

<comment><USER15> How many coal power plants have been built _because of_ the closure of nuclear power plants?

<comment><USER16> A while back there was a story on HN that link to a page that listed every country and how much of their energy consumption get created from different sources.

One very noticeable aspect is that almost every country create their energy through either coal or nuclear power, with all other usually representing a singular digit. If it has 20% coal, then around 70% will be nuclear. If its 70% coal, 20% nuclear. Nothing is more clear cut when comparing France and Germany next to each other.

<comment><USER10> How recent were those numbers? The US is currently around 30% coal, 20% nuclear (so only about 50% combined), 32% gas, and almost all of the rest is renewables of various sorts. Germany is 35% coal, 12% nuclear, 13% gas, 35% renewables. France is 72% nuclear, 12% hydro, 8% coal and gas. I'm having trouble finding the exact breakdown of coal and gas for France, but it looks like gas dominates.

<comment><USER17> Is probably talking about [https://www.electricitymap.org/?wind=false&solar=false&page=...](https://www.electricitymap.org/?wind=false&solar=false&page=map&remote=true). Is live, or close to live data

<comment><USER16> It looks like it but either I remember very incorrectly, or the numbers for Germany has changed significant since the summer. Found this which might explain why: [https://www.energy-charts.de/power.htm?source=all- sources&we...](https://www.energy-charts.de/power.htm?source=all- sources&week=51&year=2017)

One thing through, did forget that gas hold a large portion of energy production and had clumped it together with coal.

</comment></comment></comment></comment></comment></comment><comment><USER18> For those in the EU: [http://archive.is/GfC2X](http://archive.is/GfC2X)

</comment><comment><USER19> Forgive me if I missed it, but what's the evidence the plant's operations _actually_ caused cancer cases? I work for a state cancer registry, and in my experience, reports of potential cancer clusters are rarely shown true. Going by a presentation and the audience's reactions at this year's NAACCR conference, most registries have similar experiences.

That's not to say the claims are disproven, but a cancer diagnosis is a rare event, and childhood cancer is an order of magnitude rarer.

This is one of those times where "small and biased sample" is a valid criticism.

<comment><AUTHOR> The article is not great. The actual lawsuit is more informative. At its heart its a property damage lawsuit rather than personal injury / death. It claims to have physical proof that the area is contaminated with polonium and other carcinogens far in excess of allowable levels. Specifics in regards to cancer levels are also in the lawsuit. Link to the legal filing is in this thread.

</comment></comment><comment><USER20> I can't read this story because of geoblocking.

How can a _coal_ company possibly expose anyone to Polonium, half-life of 138 days and not produced by any chemical process?

<comment><USER21> Po-210 is in the decay chain of U-238:

[https://nucleonica.com/polonium210.aspx](https://nucleonica.com/polonium210.aspx)

 _it is constantly being produced in nature from the decay of uranium-238 – the main component of uranium in the earth. Due to this "equilibrium", the concentration of Po-210 in nature is approximately 75 µg of Po-210 for every ton of uranium-238_

It's a commonly known byproduct from burning coal, even though it's very rare, the sheer volume of coal burnt means that a lot of it accumulates.

[https://www.sciencedirect.com/science/article/pii/S001623611...](https://www.sciencedirect.com/science/article/pii/S0016236118314467)

[http://www.shl.uiowa.edu/news/polonium.xml](http://www.shl.uiowa.edu/news/polonium.xml)

[https://www.chromatographytoday.com/news/gc- mdgc/32/breaking...](https://www.chromatographytoday.com/news/gc- mdgc/32/breaking-news/coal-burning-power-plants-affect-po210-in-marine- life/15363)

<comment><USER20> Ok, those PPMs are surprisingly high.

1.5 million tonnes per year, 100ppm U-238 -> 150 tonnes U-238/year, 75μg Po/tonne U -> 11,500μg Po/plant-year, LD-50 human adult < 1μg

</comment></comment></comment><comment><AUTHOR> The actual legal complaint:

[https://myeclerk.myorangeclerk.com/CaseDetails?caseId=108169...](https://myeclerk.myorangeclerk.com/CaseDetails?caseId=10816988&caseIdEnc=cN7eLZFEyFcmazNah%2BT9pVndLKRaPwmr13rBLDwPxjJuFfboWcif1co2veb%2B%2FBq6MK6Rj%2FhqtJQteqGw0lbyUlPDugWIwj%2FSsFPqszeaYfs%3D&fbclid=IwAR0c2SXeIu55gPSrPTQGiNdQXYrked1enDKT9RB9Vkw8Kj2MSeyE3qj_gco)

</comment><comment><USER22> I wonder why they need those coal plants. There's an enormous looking nuclear plant on the outskirts of Orlando.

<comment><AUTHOR> That's actually this plant, a coal and natural gas plant.

<comment><USER10> It should be noted that the iconic hyperboloid nuclear cooling towers are actually unrelated to nuclear power. Many nuclear plants use them, but many do not, and many non-nuclear plants also use them.

<comment><USER23> The cooling towers are used to cool waste water before discharging back to rivers/lakes, if I am not mistaken. Too bad there's not a practical way to put that heat to better use.

<comment><USER10> Right. Nuclear plants turn heat into electricity the same way coal plants do, they just generate the heat differently, so the second stage of the process looks similar for both. Some plants are built where there's enough water flow (or enough lack of concern over the environmental problems) that the towers aren't needed.

<comment> village-idiot For a while the Russians had a designed plant that produced electricity, steam heat for nearby homes, and desalinated water with what energy was left. Apparently the design was abandoned, which is kind of a shame.</comment></comment></comment></comment></comment></comment>


Dear Google Cloud: Your Deprecation Policy Is Killing You - <AUTHOR> https://firstname.lastname@example.com/dear-google-cloud-your-deprecation-policy-is-killing-you-ee7525dc05dc <comment><USER2> I think Google couldn't do better, even if they wanted to.

The reason is the engineering culture and the promotion system. People expect to switch to new projects, and work towards to a promotion within 2-3 years. This shows in all products, and presumalby also the reason for the deprecation. New people come on the project, want and need to do a major overhaul, and do not have resource or incentives to support the old stuff.

The other half is that most projects are done efficiently by relatively small groups of engineers. I remember that Android Market was still a few people in MTV, with no payment possibilities outside of the US, when Apple already had significant revenue from apps. And that shows everywhere. I'd argue Google is incapable of tackling projects that require more than a few hundred people for more than 2-3 years.

But as a billion dollar company, that does not work. Throwing money at obvious new billion dollar opportunities like Microsoft did with gaming and cloud, and following through over 5 years is not possible for Google.

<comment><USER3> My armchair thought: what if the CEO said "wow, Yegge is right, we've gotten really off trail here. you need to show leadership in maintaining a 3+ year old system at Google for promotions?" Could that happen and would it work?

<comment><USER4> The promotion system isn't the issue, the promotion system was the issue 10 years ago. For the last 10 years Google has been hiring and promoting and keeping people who prioritize shiny new things more than maintaining old things.

If you start rewarding maintaining old things you aren't going to suddenly get what you want, you're just going to lose a tonne of people because they don't want to do what you're telling them and you won't be able to hire good people with your priorities because no one is going to believe they can have a good career at Google by prioritizing stability and long term support. It's all downstream of culture, and culture change is very difficult.

Of course if you do buy into this idea that this is the issue, then GCP has already lost. It would be years before they could make a dent in that culture, and more years after that for external customers to know and trust Google's new culture. By that time there's going to be a dominant player anyway. So maybe actually it's best to stick with the current culture - try and win on your strengths than try to fix your defficiencies.

<comment><USER5> It might be even deeper than that. I don't know if it's representative, the few Googlers I talked to were there only for the money. Their view of the company was actually quite negative.

Keeping backward-compatibility requires people that care. That have enough pride in their work to counter-balance the grind.

<comment><USER6> Obviously having people that care about the work they do is a positive thing, but there are plenty of people in plenty of industries managing to do a good job despite only being in it for the money.

It might not be a good business decision for Google, but surely if they create enough positions that are solely focussed on the boring stuff but are paid better to make up for it, there is a price point at which they'd get enough interested people.

<comment><USER7> That would be a career suicide to become a support engineer like this. 4 years later, when his stocks dry out, he'd have to switch the company and he'd need to tell a convincing story why the new company should pay him the new market rate. And "I supported legacy code" is not such a story. There's always an option to go to Microsoft and support legacy stuff for life, but beware that MS pays peanuts (relatively speaking) and with the MS pay you'd be priced out of the housing market.

<comment><USER8> To show that you deserve whatever pay you need to show impact and complexity. I don't think you would have a problem showing either when talking about maintaining Google scale products...

<comment><USER7> Now we are talking. The thing is, only few work on those big projects. The rest are doing god knows what and have to jump ships often. If your resume says you spent 5 years maintaining a smallish noname project, your career is at risk. I don't believe that Google keeps all its 100k employees busy with complex and high impact projects.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> > Let's say hypothetically that Apple was dumb enough to pull a Guido van > Rossum, and declare that Swift 6.0 is backwards-incompatible with Swift 5.0, > much in the way that Python 3 is incompatible with Python 2.

Damn, I was waiting for this. The whole essay was a setup for this paragraph. Reframes the argument using a shared traumatic experience for everyone. This humanizes the effect, we all know someone that lost someone to the Python2/3 transition. I can't count the number of Python friends that turned into Gophers, joining some cult of middle class bearded tech dad with a side business distilling pear brandy.

Best Yegge Essay Yet! Welcome back Steve.

 __edit, finished the essay. He _smoked_ GCP in this one, damn damn hard. Hickory.

<comment><USER10> I don't get the Python3 bw-incompatibility hate. Getting rid of cruft is literally the only way to get rid of cruft. It's literally the Pythonic to not have cruft.

I dread the day Python will be as bloated as C++ or Java, neither that dare to remove things.

<comment><USER11> Probably less to do with backwards compat then how it was handled. The first few versions of python 3 didn't really provide any compelling benefit, (indeed things like performance were worse, not better), and most people didn't _care_ about the cruft it removed. The "print" statement being a great example. Was it kind of weird? Sure. Did that weirdness have any consequences at all? Not really. Was it worth it to force people to replace thousands of lines of code for something that was essentially a minor aesthetic issue, if that? No way.

Also the python 2 string type was more in line with how other unix utilities work, so while I do think the 3 string type is generally better for the web, it wasn't an obvious improvement for the other large usage of python. While it's a very versatile language, I would say the major niches of python are: web dev, shell scripting, and scientific computing. The changes in python 3 were (somewhat) helpful for web dev, but really were inconsequential and/or actively harmful for the other uses.

Mixed with how condescending the python dev team acted about it, and how it really wasn't until about python 3.6 that there was any compelling reason to move to python 3,you can see where most of the hate comes from.

And by the way, even if Python is very popular today, I do think that rift really hurt the language. I worked at least at one company where Python was considered for a project, but eventually shelved because of management's confusion over the 2/3 issue.

<comment><USER12> Agree with everything you said here, but let me riff on this

> The "print" statement being a great example. Was it kind of weird? Sure.

I am reminded of the Emerson quip "A foolish consistency is the hobgoblin of little minds". The Py2 print statement is special/weird because it is a very important and special behavior, _especially_ for newcomers. It's not randomly special, it's special because it models something that deserves special treatment.

Py3 is more consistent, but less humane. I use it over Py2 only because the market has moved on, but every time I type print with parentheses I curse Guido.

<comment><USER13> The funny thing is, if you type print with space it _recognizes_ what you meant, and still wouldn't do it, raising an exception instead. It would've cost them literally nothing to support both ways to write print, since the parser has to detect statement print anyway.

<comment><USER10> Yes, but it would be a different way to do the same thing. It'd be anti- Pythonic.

</comment></comment></comment></comment></comment></comment><comment><USER14> It's odd to see an article emphasising backwards compatibility, but not a single mention of Microsoft (I even ctrl+F'd the page source to check!) They've regressed a bit in the recent years but I'd still consider MS the gold standard for back-compat. I have Win32 binaries I last modified over 15 years ago, small (or perhaps _tiny_ \- size measured in KBs) utilities that I use daily. They worked perfectly on Win95, and still do on Win10.

I wonder when/if the software industry will ever stabilise. It's possibly the only industry where frequent and disruptive change continues to happen, and is even welcomed by many in it (not me).

<comment><USER15> The absolute worst in my small experience with it is anything touching the node ecosystem. Try to follow any guide or use any library not updated in the last week and there will be breaking API changes and deprecation warnings everywhere.

<comment><USER16> The sad part about the js ecosystem is that even the biggest corps like Google can't keep their own js ecosystem stable on the last version. I have a small Angular app with Firebase backend and every time I come back to it to update, the libs are out of sync

<comment><USER17> Google is notoriously bad at maintaining (or even designing) their open-source stuff. I wouldn't say "even the biggest corps like Google" \-- Google does considerably worse at this than many hobbyist maintainers! I just don't use Google libraries anymore, if I can at all avoid it.

See also my other comment[1] - Angular is an example of such a magical does- everything framework.

[1] [https://news.ycombinator.com/item?id=24168279](https://news.ycombinator.com/item?id=24168279)

<comment><USER18> We've abandoned an app because of the inability to keep Firebase working on it (JS libraries). Between out of sync Typescript definitions, broken APIs, broken interfaces, "this version of some other dependency works with this version of Firebase thanks to some random transient dependency", it just became too much. We won't use the JS libraries at all anymore.

Angular is the only one that has remained even remotely usable, but certainly not stable. It's never as simple as running the upgrade utilities, changing the version, and being done. It ALWAYS takes at least a day to find all the "little" things they didn't feel fit to mention in the upgrade docs.

It's aggravating. I genuinely don't like using Google software, at all.

I've felt for a long time that Google is coasting on the momentum of the early web and the awe people felt for what they built early on. That hasn't been Google for a long, long time.

</comment></comment></comment></comment></comment><comment><USER19> The answer to why this is, I'm beginning to realize after talking to dozens of Googlers working at Google as acquaintances, friends, former coworkers, HN comments and twitter personalities is - _Google has an active disrespect if you're not part of their in group_. If you haven't passed their tests, you're basically an object of derision - a mark, an object just used for incrementing a CPM metric, or an incompetent, low-class buffoon who doesn't deserve to have their jet setting luxury lifestyles. And unfortunately, everything trickles down from that.

<comment> wh-uws (warning rambling rant)

I feel like this was super true in the pre Facebook days but they were humbled A LOT by the massive flop of Google plus.

I honestly don't feel like I get the my s __* doesn 't stink vibe from them anymore.

Especially because everyone knows their Achilles heel now.

But for their ad revenue what would they be?

Everything else material from came from an acquisition or is infra stuff.

It's cool but they also had to learn the hard lesson of Hadoop too.

Yeah they built big table... But the rest of the industry standardized on Hadoop (at the time) because they never released an implementation and so they started losing out on good hires because they didn't want to get locked into proprietary Google infra.

I think you see them recognizing this with intiatives like all of the stuff around kubernetes.

But hey I wouldn't build anything on their (non open) stuff and get more scared by the day over my dependence on El Goog because they've shown time an again how little they care for us common plebs.

But I feel like they get better when they lose but they've only started losing more recently

(Rant over)

<comment><USER20> > Yeah they built big table... But the rest of the industry standardized on > Hadoop (at the time) because they never released an implementation and so > they started losing out on good hires because they didn't want to get locked > into proprietary Google infra.

Hadoop and Bigtable are not the same thing. Bigtable is a noSQL database. Hadoop is a big data processing framework. Hadoop is actually an open source implementation of MapReduce, which was developed at Google and which has since been replaced by Flume

<comment><USER21> Hadoop is a collection of things: HDFS (filesystem), Hbase (keyvalue database), MapReduce (data processing), and YARN (resource scheduling and job management).

Hbase was open-source implementation of Bigtable while HDFS was a copy of Google File System. The divergence appeared with other processing frameworks like Spark but now there's Apache Beam to unify it all.

</comment></comment></comment></comment><comment><USER22> This very long article doesn't describe even a _single_ instance of anything GCP has deprecated.

It's a long rant with a bunch of f-words, and the claim that he gets deprecation e-mails "about once a month".

Can someone who is informed point to actual GCP services that have been, or are being, deprecated?

From the article, I have no idea whatsoever if this is some huge actual problem with core services... or pulling support for ancient versions of Linux... or sunsetting experimental/beta services that never had guarantees to begin with.

I'd find it pretty surprising for a platform as relatively new as GCP to already be deprecating anything meaningful, so I'd really appreciate if anyone has actual facts here.

(Also I know a lot of people who use GCP, and I've never heard anyone complaining about services being deprecated, so to hear someone complaining about it happening monthly is pretty surprising.)

<comment><USER23> Python2 on AppEngine is a huge, painful Google GCP deprecation.

I have a friend who built an AppEngine app within the first year of the platform's release, and he's spent the better part of the past 9 months doing almost nothing but managing his company's migration to Python3 AppEngine.

It wasn't just migrating from Python2 to Python3 syntax. Google used it as an opportunity to also turn down tons of APIs and services they no longer felt like maintaining.

One big difference was the Users API. Python2 AppEngine had a pretty simple way to authenticate users that you could set up in an hour.[0] In Python3, they've dumped that entirely and tell you that you have to move to a totally different solution, and they offer no help with that migration.[1]

In Python2 AppEngine, you essentially got a memcache for free. The ndb library[2] allowed applications to read/write to Cloud Datastore, but it also automagically managed the cache for you so that you didn't have to set up a separate cache service. In Python3, they've dumped that and force you to maintain your own cache service and handle all the caching logic yourself.[3]

It looks like the ndb library is pretty complete at this point,[4] but when I was looking at a migration earlier this year, the ndb library wasn't even out of beta. Meanwhile, they were breathing down their customers' necks about moving off the Python2 version.

I had a fairly small (~8 KLOC) Python2 AppEngine app that I wrote in 2018, but when I realized how much of a hassle it would be to migrate to Python3, I just rewrote it using Gridsome and moved off of AppEngine entirely.[5]

[0] [https://cloud.google.com/appengine/docs/standard/python/user...](https://cloud.google.com/appengine/docs/standard/python/users)

[1] [https://cloud.google.com/appengine/docs/standard/python/migr...](https://cloud.google.com/appengine/docs/standard/python/migrate- to-python3/migrating-services#user_authentication)

[2] [https://cloud.google.com/appengine/docs/standard/python/ndb](https://cloud.google.com/appengine/docs/standard/python/ndb)

[3] [https://cloud.google.com/appengine/docs/standard/python3/mig...](https://cloud.google.com/appengine/docs/standard/python3/migrating- to-cloud-ndb#caching)

[4] [https://googleapis.dev/python/python- ndb/latest/index.html](https://googleapis.dev/python/python- ndb/latest/index.html)

[5] [https://whatgotdone.com/michael/2020-04-17](https://whatgotdone.com/michael/2020-04-17)

<comment><USER24> But Python 2 on App Engine is not officially deprecated yet (e.g. not listed here: [https://cloud.google.com/appengine/docs/deprecations](https://cloud.google.com/appengine/docs/deprecations)) and they have not sent the "[Action Required]" email (of which OP talks about) for that yet, so I'm not sure it counts. They are "just" recommending migration.

I have a ~60 KLOC AppEngine Python 2 app (started in 2011), but I'll probably migrate that over only when the deprecation actually occurs, as:

\- that gives the python-ndb library more time to mature (performance and bugs),

\- easier migration paths for some of the non-py3 services I use (e.g. Images, Users, Memcache) might appear when deprecation actually occurs, either from Google or 3rd parties,

\- I'm in no hurry,

\- I will be surprised if Google gives less than 1 year of deprecation notice (Python 2.5 got 4 years from deprecation to termination).

(If the APIs I use were available for py3, I would've probably already migrated.)

</comment></comment></comment><comment> john-shaffer > [Google] contracted with a company called Bitnami to create a bunch of > "click to deploy" solutions, or perhaps I should write "solutions", because > they don't solve a fucking thing. They're just there as checkboxes, as > marketing filler, and Google never gave a shit whether any of them worked > from Day One.

I've had similar experiences with broken Bitnami images that makes it seem like they aren't even tested before being released [1][2]. Their WordPress images do work, but the way file permissions are configured doesn't play well with the way they have WordPress set up [3]. Have I just had bad luck, or is this a common experience?

[1] [https://community.bitnami.com/t/cant-authenticate-to- fauxton...](https://community.bitnami.com/t/cant-authenticate-to- fauxton/71354) [2] [https://community.bitnami.com/t/couchdb-server-log-gives- wro...](https://community.bitnami.com/t/couchdb-server-log-gives-wrong- username/71595) [3] [https://github.com/WP2Static/wp2static/issues/598](https://github.com/WP2Static/wp2static/issues/598)

<comment><USER25> Bitnami broke their MongoDB Helm chart. We're careful to pin versions, and hadn't anticipated someone would re-write history and break published code.

Can't get too upset, we don't pay them anything. Not in a rush to use them again though.

<comment><USER26> Have you reached out to (community) support? My colleagues there tend to do their best to help the community.

EDIT Disclaimer: I work for Bitnami; not in that team though. I reached out internally to see if this is a known issue, but it's Saturday...

EDIT 2: does the downvote mean bitnami community support is so bad or did my wording offend anybody?

<comment> john-shaffer There are lots of random downvotes for no real reason. Sometimes perfectly ordinary posts are dead. I try to upvote gray posts if they're not worthless.

That said, my take (and that of siblings) is that I'm much better off building my own containers than serving as unpaid Q&A for Bitnami. It would be smart to use containers built by people more expert than I am. Containers that are broken by default do not meet that criteria. So, "reach out to community support" is not helpful to people that have already decided to ditch Bitnami.

> We're careful to pin versions, and hadn't anticipated someone would re-write > history and break published code.

This is exactly the kind of thing that will make me ditch a company forever and never look back.

<comment><USER26> It's not clear what happened though.

We use immutable tags like "4.2.8-debian-10-r50" which we never overwrite.

Then we have "semantic versioned" moving tags, like "4.2.8", "4.2", ... which resolve to the latest image matching that prefix.

Moving tags are what they are; I'm personally not a fan; all major popular images have that ([https://hub.docker.com/_/golang](https://hub.docker.com/_/golang), [https://hub.docker.com/_/python](https://hub.docker.com/_/python), ...)

You can read more about Bitnami tagging scheme at: [https://docs.bitnami.com/tutorials/understand-rolling- tags-c...](https://docs.bitnami.com/tutorials/understand-rolling-tags- containers/)

If you want to pin your image you should either use the most specific tag or just use the digest:

        $ crane digest bitnami/mongodb:4.2
        sha256:8650c2d92eea97732eae359a140ee86ee3923a2a19b19443e1dc01ec20d5387d
        $ docker run bitnami/mongodb:4.2@sha256:8650c2d92eea97732eae359a140ee86ee3923a2a19b19443e1dc01ec20d5387d

Now, we might have introduced a regression between some version of a container and the next minor version; shit happens; it's hard to tell without more specific information though.

> > We're careful to pin versions, and hadn't anticipated someone would re- > write history and break published code.

> This is exactly the kind of thing that will make me ditch a company forever > and never look back.

On the other hand, assuming that instead of a misunderstanding, what you saw is actually the image behind an immutable tag such as 4.2.8-debian-10-r50 being replaced, this is a serious security issue; somebody could have hacked docker hub, or crafted a valid certificate for docker hub and MitM'd you,

I'd also ditch a company forever and never look back if they honestly don't care about that problem, which I assure you it's not the case.

We'd greatly appreciate reporting such cases to firstname.lastname@example.com / firstname.lastname@example.com .

</comment></comment></comment></comment></comment><comment><USER27> Google is an absolutely fantastic company from which to get free stuff. Gmail, Maps, Docs, Youtube? Mmm-mm good!

Buy things from Google, though? You mean, give Google money and in exchange expect them to adhere to some kind of standard of behavior, support, and customer service? Go get a coffee because you're clearly not awake yet.

<comment><USER28> This is the exact opposite of the usual complaint: that if you don't pay for support you can expect Google to arbitrarily shut your account and scoop up all your data - however paying them actually allows you some control.

<comment><USER29> Many of the frequent horror stories on here about account suspensions and completely unresponsive support are coming from people or companies with paid accounts.

I do believe that there is a level of spending where you can actually get a hold of someone competent who can look into things. But that level appears to be rather high.

<comment><USER30> About 5 years ago, Hewlett-Packard tried being a paid GSuite customer, paying for tens of thousands of seats, with a path to onboarding the entire company. We could not get support, or even a TAM who would answer their phone.

After a year, HP dumped GSuite, and went to O365 instead. We got a responsive TAM, full multi-tier support, and white glove handholding for migration. And, I think it was cheaper, too.

There is NO level of spending as a customer at which Google actually gives a shit.

</comment></comment></comment></comment><comment><USER31> It is a total hassle to keep up with Googlers changing everything constantly. It's not just GCP it's every platform they control. Try keeping a website on the right side of Chrome policies, a G Suite app up, a Chrome extension running.

Thousands of engineers chasing promotions by dropping support for live code. If it was their code they wouldn't do it. The org is broken. If you want to see what mature software support looks like, check out Microsoft. Win32 binaries I wrote in college still run on Win 10. Google looks unimpressive by comparison. But they all got promoted!

<comment><USER32> I'm reminded of a blog post I once read, which annoyingly I now cannot find, on how the job of a library maintainer is to not break the API. If you break the API, you've failed at your job. This mindset makes more sense than treating API breakages as a routine case-by-case decision. As the article puts it, breaking changes are breaking the library's commitments. (The library/service distinction doesn't matter much here.)

I remember that same old blog post also lamented overhearing a conversation in which a developer bragged about how their idea for a change was so good that everyone agreed to break the library API to implement it. This reminds me of how a startup being 'highly disruptive' is sometimes fetishised as an end in itself, but of course it's even worse than that: they're celebrating undermining the library's dependability.

<comment><USER33> sounds kind of like this famous linus rant: [https://lkml.org/lkml/2012/12/23/75](https://lkml.org/lkml/2012/12/23/75)

<comment><USER34> Is he always like that? That's toxic af

<comment><USER30> Complaints about "toxic" are a screen to hide the fact that you can't say "incorrect".

</comment></comment></comment></comment></comment><comment><USER35> > I have been inexorably pushed away from GCP, towards cloud agnosticism.

On a side note I think this is what all of us should be aiming for. Every time someone on a team I'm working for starts trying to use some very complex tool that a SaaS/cloud has built for us I always urged them to look for a simpler solution.

Do you really need AWS Lambda/kinesis/etc or can you write your code and deploy it onto our cluster as a normal service/api/control loop?

If you make it really easy to write code, deploy it, and monitor it, you'll often find it's way less devops time to write your software as apis/control loops than it is to use a large amalgamation of cloud services. I also haven't found a good way to unit test cloud services and their interlinking.

<comment><USER36> On the contrary, every time someone comes to me with a suggestion to build something in-house or use an open source solution when a aws service exists I say the hell with that. I don't want to maintain it, build features into it or operate it.

I want full in vendor lock-in, it saves an immense amount of time.

You just need to pick a vendor that gives a shit about it's customers to be locked in to - and that's simply not google.

</comment></comment><comment><USER36> > I'm not actively developing on AWS, so I don't have as much of a sense for > how often they sunset APIs that they have previously dangled alluringly > before unwitting developers.

SimpleDB still works, it's not even deprecated or grandfathered to not accept new users. There is really nothing else you need to know about how AWS treats deprecation than that single example.

Deprecation is practically unheard of in AWS. If it does happen you can be damn sure it takes years to actually remove obsolete functionality and there is a fairly easy migration path to something of equivalent functionality.

<comment><USER37> Another example: There was a change to how objects within S3 could be addressed by http(s): [https://aws.amazon.com/blogs/aws/amazon-s3-path- deprecation-...](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation- plan-the-rest-of-the-story/)

Initially the (justified) change gave you ~18 months to implement necessary changes. Seems feedback was negative regardless, so they modified it in a way so current usage remains unaffected.

<comment><USER38> They didn't have a choice, their internal teams weren't ready for S3 SigV4. For example, EMR was not ready for SigV4. That means ton of customers can't use EMR.

</comment></comment></comment><comment><USER39> Ten years ago it would've been inconceivable to me, but I have to admit that my default assumption is that Google cannot follow through on any of its projects. It is no longer a "live player"[1] and lumbers along purely on its draining intellectual endowment and incumbency.

Network effects and increasing anti-competitive practices will keep Google at the top for years to come, but without a cultural reinvention it cannot possibly regain the prestige it once held as an institution.

[1] [https://firstname.lastname@example.com/live-versus-dead- players-2b24...](https://firstname.lastname@example.com/live-versus-dead- players-2b24f6e9eae2)

</comment><comment><USER40> Can't help wondering if some of the fall-off in Rails usage is due to a similar attitude toward deprecation of formerly supported APIs, which has made upgrading large Rails apps to a new major release a pretty significant project.

Things they've removed have included the entire original ActiveRecord query API, and the RJS templating system for Javascript. Those two required rewrites of significant portions of any app that tried to bridge the transition using only supported APIs. And every new point release (say, 5.1 to 5.2) brings a enough more minor deprecations to make an upgrade not a task, but a project. That's forcing a lot of significant rework, and every one of those gives people a chance to just take their project elsewhere.

(There's actually a third alternative. The release after these two major components were removed from core Rails, they were still semi-supported as add-on libraries. In both cases, core support for those was ultimately cut off -- but people with old apps could try to keep at least portions of them working to minimize the damage. I've done that to help avoid rewrites in components of an older Rails app. I didn't support all of either old API -- for example, rewriting uses of deprecated options to `belongs_to` was easier than keeping them on life support. But on balance, it was easier than the rewrites.)

</comment><comment><USER41> I have personally used a large number of cloud compute services: AWS, packet, Alibaba, Digital Ocean, nimbix, Azure, Oracle, ... I still use all of these services from time to time. If you carefully look at my list of cloud vendors you realize that there is only one of the big cloud vendors missing: GCP. Why?

When GCP came out I wanted to switch from AWS to GCP because AWS was costing me a lot of money and GCP was marketed as being cheaper. So I went to their website and signed up. Or at least I tried to sign up, because GCP did not care about individuals, you had to be a company! No other cloud vendor I tested had this requirement! So up to this date I have never tried to sign up with GCP again.

<comment><USER42> Let me guess, you're based somewhere in Europe? I don't remember the specifics, but the "business only in Europe" thing was 100% the fault of the EU's (tax?) laws. Not sure if that ever got resolved, actually.

<comment><USER41> Yes, I am based in Europe. If this was EU's fault then why didn't any of the other cloud vendors have this requirement?

</comment></comment></comment><comment><USER43> The mishandled Python 3 transition sure created a lot of pain (I experienced some of it, even though I only started with Python in 2016), but I never realized how much it hurt the language's momentum. This must be one of the reasons why JavaScript is so popular today. JS benefits here from being run in browsers, which the author points out are fantastic at back compat, even as the dev culture surrounding the language doesn't value it at all.

<comment><USER44> _I never realized how much it hurt the language's momentum._

Yes. The combination of a botched transition with a high arrogance level was hell. Here's something I wrote in 2015 after porting a medium-sized system from Python 2 to Python 3.[1] I listed a number of package problems encountered during the conversion. Ones that indicated those libraries weren't being used much in production. I discovered that the SQL connector broke if you loaded a database of tens of thousands of records, for example. And oh, did I get hate mail. You can see the comments below mine.

That's what led me to Go. Go is a mediocre language. That's a strength when you just need to get something done. It comes with a set of libraries which are heavily used internally within Google. So, if you're doing something that is server-side for some web-related task, the libraries for that are probably both present and well-debugged. And they don't seem to be deprecated rapidly. I'm not using them for any Google-specific services, so I can't speak to that.

[1] [https://lists.archive.carbon60.com/python/python/1187081/?pa...](https://lists.archive.carbon60.com/python/python/1187081/?page=1;mh=-1;)

<comment><USER43> Yeah I have the same thoughts about go, which I've been using every day for the last four years. It's super rare for an upgrade to break anything (has happened though). When it comes to running stuff in prod, boring is good.

</comment></comment></comment><comment><USER45> I've been hiring dev ops engineers lately, and the consensus seems to be in the candidates I've talked to that _all_ of the cloud providers are somewhat painful to work with.

I've worked the most with AWS so I'll use it as an example, but it seems like they sell on "new services", so they are incentivized to release quickly, which leads to services with a lot of holes and poor documentation, making it almost essential to get paid support to work around these issues.

It makes me wonder what it would look like to build a minimal cloud product which focused on a few core services with an emphasis on reliability, performance and developer ergonomics.

<comment><USER46> The only option I can see as an alternative "minimal" cloud is one that runs Kubernetes on one of the "mid-tier" hosting platforms.

DigitalOcean, Linode, heck even OVH now have K8s offerings with ready-to-go (and free) control node/panel instances.

Of course K8s isn't shallow but at least when you master their way of doing things you're free to go and do whatever you want.

It's cheaper too.

<comment><USER47> One small note of caution there is that using managed k8s solutions has some consequences for flexibility of what you can do.

For one example, in most managed k8s setups, you can't change the admission controllers used on a cluster unless the cloud provider exposes this via an interface. Some cloud providers do more of this than others.

So with a minimal k8s managed option you might get stuck if your cloud provider doesn't support an admission controller or other API server option you need.

</comment></comment></comment><comment><USER48> I have heard that at Google, you get promoted/rewarded for launches. To me, that seems like an obvious perverse incentive to continuously deprecate things, so new things can be "launched" to replace them. There are no "launches" if you're just maintaining backwards compatibility!

Maybe GCP should take a hard look at their internal incentives?

<comment><USER49> That's one of the points of the essay. Steve also argues that it's not a GCP problem, but a Google-wide problem (excluding Android and Chrome).

</comment></comment><comment><USER50> "Fuck yooooouuuuuuuu. Fuck you, fuck you, Fuck You. Drop whatever you are doing because it's not important. What is important is OUR time."

Most NPM packages.

<comment><USER51> > "[...] Drop whatever you are doing because it's not important. What is > important is OUR time"

> Most NPM packages

NPM packages tend to be free and open source so morally they don't owe us anything. So yes what is important is the time of the authors of the packages. I am grateful someone took the trouble to release them and many popular ones are actually well maintained.

But cloud services that you are paying for are a totally different ball game. The company does need to maintain backwards compatibility for a good amount time and they need to value your time.

<comment><USER52> I don't care about moral debts or obligations. I care about whether stuff works and I can get shit done.

Other ecosystems (like Python) don't have the same culture as NPM. There are clean, well-documented, maintained packages.

Google doesn't have a moral obligation to support GCE customers either. It's just that everyone I know who has used GCE would never use it in production, and loudly advises everyone they know not to do so either. They'll be bleeding money at some point, not to mention they're already bleeding reputation. Google went from smartest-people-in-the-room to smug-incompetent-arrogant- douchebags. That's hard to get over. That's about good business, not some kind of moral obligation.

<comment><USER50> Same for Java.

</comment></comment></comment></comment><comment><USER53> "GNU Emacs, which is a sort of hybrid between Windows Notepad, a monolithic- kernel operating system, and the International Space Station."

The article was worth a read for that quote alone.

</comment><comment><USER54> I'd be interested in some concrete examples of things GCP have depreciated. I'm an (admittedly fairly light) user of GCP and haven't come across any myself.

<comment><USER55> I saw that he mentioned a few of them in the article. He writes:

 _I can tell you that virtually everything I've used, from networking (legacy to VPC) to storage (Cloud SQL v1 to v2) to Firebase (now Firestore with a totally different API) to App Engine (don't even get me started) to Cloud Endpoints to... I dunno, _everything_, has forced me to rewrite it all after at most 2–3 years._

<comment><USER56> I work on Firebase. His characterization of Firestore is incorrect and lazy. It did not replace Realtime Database (which is the product he's calling Firebase) they are two products which live side by side and both are being actively improved.

People assumed that because we made a new database we were killing the older one, but that was never true.

<comment><USER57> Well, that's a novel take on the customer is always right

<comment><USER58> "the customer is always right" doesn't mean every individual customer is always correct and you should never disagree with any of them, it means the customer, in aggregate, is correct and you need to react to customer opinions.

But also idk what else you'd want them to say "the thing you're taking shit isn't deprecated" seems like an important response to "you mishandled this deprecation".

</comment></comment></comment></comment></comment><comment><USER59> Google thinks it can operate the same in paid user bases as it does in free user bases.

You aren't a customer, you're a user who paid to bypass a rate limit. There is a difference.

</comment><comment><USER60> I really want to like google cloud. Every time I use it things seem better set up than in AWS. I feel like I can't even bring it up at work though because of the issues with deprecation.

<comment><USER61> I hear you. At the end of the day though a little more effort on front end setup is infinitely better than the time wasted later on when you have to rebuilt everything every time Google decides to get rid of another product / service / platform / setting. Burned by that one too many times and moved on.

</comment></comment><comment><USER62> So much ranting, yet not a single example of a GCP product that Google has actually killed. Got examples?

I mean, there's stuff like Python 2 support being sunset on managed services, but as Yegge himself points out that's on Guido, not GCP.

<comment><USER17> The old Firebase API. No longer possible to create new databases that can be used from the old API.

Which in practice meant that "migrating to a new database under a different account" translated into "rewriting half a codebase due to a cascade of breaking changes in the surrounding tooling".

<comment><USER63> Are you talking about the Firebase RTDB? You can't create new ones anymore? That's news to me.

</comment></comment></comment><comment><USER64> Scathing. Well written. Extremely hard to argue with.

GCP will fail, I think everyone knows it, and even if they don't, they _fear it_.

</comment><comment><USER65> I had Google cloud break twice this week. First their "AI Notebooks" images broke some dependency, then some IAM change broke the cloud console interface.

Though our engg. team is on it, Pointless hours of productivity lost :-(

</comment><comment><USER66> Has AWS ever sunsetted a service?

I know they deprecated SimpleDB but I don't think they ever actually shut it down for existing customers, that might have changed in the last few years though.

<comment><USER61> As you mention my understanding is that that there's one service no longer offered to new customers but they are still supporting it for those that were using it. This is a big difference between AWS and Google's approach to customers. A number of friends' companies got hit really hard when Google suddenly decided they no longer wanted to do something anymore (regardless of what their customer thought). That list is big and growing: killedbygoogle.com

</comment></comment><comment> em-bee i think this is the key point of the article:

 _it actually winds up being less DevOps work, on average, to support open- source systems running on bare VMs, than to try to keep up with Google's deprecation treadmill._

<comment><USER67> I honestly feel like this is a better idea in general, at least in data science stuff.

For context, I tend to get dumped with undocumented poorly written untested DS code bases every time I move to a new job.

The difference between setups where there's a bare VM/server and it runs using standard tools (command line scripts) or at least some open-source orchestration and any kind of proprietary technology tied to infrastructure is massive.

Additionally, the server stuff can be maintained and debugged by a _far_ larger number of people, thus reducing bus factor.

Like, maybe in the web dev world there are things that everyone knows that are AWS services (maybe S3 counts, I guess) but in the DS world, I just know that I'm in for a whole lot more pain if they haven't just used bare servers.

<comment> em-bee this is the classic advantage of Free Software and Open Source over non-free tools. you often get a mix of commercial services and inhouse custom development that is no better documented or supported than FOSS alternatives.

</comment></comment></comment><comment><USER68> There are always trade-offs though.

1) If you're busy maintaining backward compat, you're not busy building innovative new things -- Microsoft. 'nuf said.

2) You have a built-in excuse to not make your new stuff equal in functionality or greater to the old stuff, because hey, customers can just use the old stuff right? I built some software that talks to O365 Sharepoint. Ok, I have two choices of API: the older Sharepoint API or the newer Graph API. They recommend Graph, ok. Build the product and put it into production. Get a new customer requirement for fine-grained auth, and then find out that Graph doesn't handle fine-grained auth -- only the Sharepoint API does that. Oops. Ask Microsoft when that capability is coming to Graph? No timeline, because the Sharepoint API isn't deprecated.

3) Keeping old stuff working may be easier, but building _new_ stuff is harder because the landscape is muddier -- how many times have you looked at Microsoft docs and found multiple ways to do things with no idea if the docs you're looking at actually apply to the approach you're using now?

I think there are ways to have (mostly) the best of both worlds. Linux does it by bring as much stuff in-tree as they can -- they break compat all the time for out-of-tree stuff. Ever try to keep a proprietary VMWare module building without errors? In-tree KVM never breaks because they're super careful with the ABI. Some languages have explicit mechanisms by which they attempt to keep things current, but make it easier on users -- Kotlin is an example of this [1] but its a young language so time will tell whether they can actually thread this needle. My experience so far is yes, I think they can -- I've updated code from Kotlin 1.2 to 1.3 to 1.4 with relatively little pain.

[1] [https://kotlinlang.org/docs/reference/evolution/kotlin- evolu...](https://kotlinlang.org/docs/reference/evolution/kotlin- evolution.html)

</comment><comment><USER69> I can hear cries from tenderfoot developers whining about supporting multiple options and backwards compatibility...

"But that's sooooo muuuuch too teeessstt!! Whaaaaaaaaaaaaaaaa!!!!!!!!!!"

But that's your job dude. So what if it costs more money and time. Fucking pay for it anyway.

I feel like startup software dev needs a swift kick in the ass or three with that phrase.... "do it anyway!"

Who fucking cares if these founder assholes only make an 800% return instead of 900%? Do it right.

<comment><USER70> You didn't read the article, did you? Startup developers don't mind supporting multiple options and backwards compatibility. Often they have to because they don't have the time and the money to rewrite everything, so it's common in startups to run and support multiple versions of APIs, clients etc.

It's _Google_ who doesn't provide backwards compatibility, upgrade paths and migration options.

> But that's your job dude. So what if it costs more money and time. Fucking > pay for it anyway.

So why doesn't Google pay for it anyway?

> I feel like startup software dev needs a swift kick in the ass or three with > that phrase.... "do it anyway!"

You mean Google devs.

<comment><USER69> I read the whole article. My reaction is such because I feel that much of software dev is plagued by the same bullshit that the article describes about Google

I've been on the web since the mid 90s. The last 15 years have had more cool things taken away from me because of sunset bullshit than the first half of that time period. Software (even web apps) should be eternal, there should never be "features removed" or "EOL" announcements. If they don't want to run something they should open source it and let the community take over. What the hell is the point of hoarding all this wonderful IP if it serves noone?

</comment></comment></comment><comment><USER71> And then there is AWS, which has depreciated exactly one service, SDB, but only after they created a much better replacement (DynamoDB) and offered to help migrate all of their large customers, so they could get the usage percent so low that almost no one was affected by the shutdown.

<comment><USER72> Disclaimer: AWS SDE

The focus on consistent behavior and once launch, never deprecate practice has caused many headaches, I personally know of control planes specifically written to accommodate old behaviors that a few customers had years ago even when the entire service has been updated. But IMO this is still much better than deprecating production services and behaviors on the fly.

<comment> yjftsjthsd-h But by making in Amazon's problem, the customer never experiences breakage, and that's a _huge_ win.

</comment></comment></comment><comment><USER73> This puts into words what I have suspected for a long time. When we were considering which cloud platform to move onto, we considered Google Cloud for about five seconds before someone said "How long until they kill it?". And all of us knew exactly what they meant. It feels like Google's main mode of operation is killing projects. We ended up going with AWS.

</comment><comment><USER74> Besides AWS and Azure what's the third major cloud provider he's referring to? (I can think of others, just haven't really thought any were close to Amazon/Microsoft/Google.)

<comment><USER75> Was wondering the same thing. Alibaba? IBM?

<comment><USER19> Last I checked Alibaba was 3rd in the international market and Oracle cloud was growing. Steve might be referring to Oracle getting 3rd place.

</comment></comment></comment><comment><USER76> " it's well-known that they refuse to host (as a managed service) any third- party software until after AWS has already done the same thing and built a successful business around it, at which point their customers hold them at gunpoint. But that's the bar, to get Google to support something."

This isn't true, of course. They supported managed Kafka before AWS did: [https://cloud.google.com/confluent](https://cloud.google.com/confluent)

<comment><USER77> Gcp doesn't have a 1st party Kafka offering. Confluent cloud is still 3rd party just with some nice integrations with GCP.

If you have an issue it's not GCP support or GCP's eng job to fix the cluster. There's real service uptime implications of this.

</comment></comment><comment><USER78> People are confusing new versions of things, or new APIs, or new products, or not recommenced anymore, with _actually removing products_. From this thread people are complaining about:

\- Datastore -> Firestore in Datastore Mode: This was a seamless transition with no API change, Google changed the underlying tech.

\- Firebase -> Firestore: Never happened. They are different products. The Firebase Realtime Database still exists.

\- gcloud CLI updates: It is a CLI tool, it gets new versions.

\- App Engine with Python 2.7: It still exists, you can still use it. There is a new App Engine v2 that supports Python 3.

\- Cloud SQL v1 to v2: They had a tool to automatically upgrade, it set up replication and switched over.

AWS has new versions too. RDS Aurora v1 (MySQL 5.6) can't be upgraded in-place to RDS Aurora v2 (MySQL 5.7)[1]. AWS Lambda runtimes get deprecated[2].

[1] [https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Updates.20180206.html)

[2] [https://docs.aws.amazon.com/lambda/latest/dg/runtime- support...](https://docs.aws.amazon.com/lambda/latest/dg/runtime-support- policy.html)

</comment><comment><USER79> I had a few laughs with this, but (maybe because I have been using Python since 1.2) I just can't relate to the Python bits.

I migrated all my "working set" stuff to Python 3 piecemeal over a couple of weeks and never looked back (except when I needed to pick up some older project and take an hour or so to refactor it).

The "There Will Come Soft Rains" reference, however, was brilliant. I hadn't thought of that story for _decades_ and it hit home.

<comment><USER80> > _I just can 't relate to the Python bits._

I work in visual effects/animation, and 2020 is literally the first year anyone is even _trying_ to use Python 3 (and Python 2.x) is still supported everywhere. Prior to 2020 Python 3 couldn't even be used at all (and I mean that literally: could not be used outside of isolated toy examples on alpha software that wasn't actually running in production).

We (as an industry) have written thousands and thousands and thousands of business-necessary logic/scripts/libraries in Python 2.x that have to be updated to Python 3 _that work across commercial tools_ so literally every tool has to be updated at the same time in tandem to Python 3, so much so that the VFX Platform[0] project was created in 2014 _specifically to address how badly Python 3 had fucked everyone in my industry_.

It's all make-work too: literally none of us actually _need_ Python 3's new features.

Javascript's "use strict"; is the way to do backwards compatibility if you really really really need to change the semantics and keep existing code working as-is.

[0] [https://vfxplatform.com/](https://vfxplatform.com/)

</comment></comment><comment><USER81> >Businesses are taking a mercenary's accounting of their dual mobile teams (iOS and Android), and starting to realize that those phony-sounding dog-and- pony-show cross-platform development systems like Flutter and React Native have real teeth, and using them could cut their mobile team sizes in half, or alternately, make them twice as productive.

Is there much Flutter and React Native dev going on at big companies outside of Google and Facebook?

I know plenty of smaller companies are using them because they don't have what sometimes feels like near-infinite money to throw at mobile engineers, but I can't think of a single Big Name On Blind with iOS or Android position listings that mention Flutter or RN. They're all native.

Even at Google and Facebook, usage of either of their respective frameworks seems to be an exception and not the norm, and they're still just components in a larger native project. I wonder if the goal is to eventually re-write everything? But that seems impossible given the size of these applications' source.

</comment><comment><USER82> I do hope the intermediate solution he mentions, tooling that upgrades your code for you, starts to be used more. Go did this in the early days before 1.0, but it is fairly rare still outside the internal Google tools. It is a big chunk of work obviously but it saves a huge amount of downstream work, and lets everyone move forward.

<comment><USER83> Right? This is a strategy/trade-off that depends on migration tooling that exists and is user friendly.

</comment></comment><comment><USER84> While I agree that GCP's deprecation policy is painful, I don't think he's painting a compelling picture that the deprecation policy, specifically, is the reason GCP isn't ranked higher.

Taking two of the examples of "bad platform moves": Python 3 and Apple.

Where I work, we are simultaneously dealing with Python 3 and GCP deprecations[1], and it sucks. But Python is more popular than ever![2] Yes, some people (including us, at least for our website code) left Python, but lots and lots of people have moved to Python 3.

I've seen lots of people say "oh, the move from 2 to 3 wasn't that bad because of the tooling." That's probably true, and we probably wouldn't be moving to Go were it not for the fact that we had API changes _and_ Python's changes to deal with (and Go gives us other benefits).

And then there's Apple. Apple has a long history of maintaining compatibility for a time, and then getting rid of the bits that are holding them back. There are tradeoffs.

Maybe Android doesn't break things for developers, but Apple makes it so that iPhone users can keep updating their phones to the latest OS (and apps) for much longer than the typical Android phone. Because of deprecations and a general desire to make apps that fit the platform, developers of popular apps on iOS put the time into updating them, and these apps are really nice to use as a result.

I think fit and finish on non-first-party native Mac apps tends to be better than those on average on Windows, though that's just my opinion.

Carrying backwards compatibility comes with tradeoffs, and it's possible to be successful by making different tradeoffs than maximizing backward compatibility.

[1]: [https://blog.khanacademy.org/go-services-one-goliath- project...](https://blog.khanacademy.org/go-services-one-goliath-project/) [2]: [https://redmonk.com/sogrady/2020/07/27/language- rankings-6-2...](https://redmonk.com/sogrady/2020/07/27/language- rankings-6-20/)

</comment><comment><USER85> One thing that I don't understand about all this deprecation talk, as pointed out here:

> So when it comes to shouldering the burden of compatibility, you need to pay > for it. Not us.

Surely if there are shoppers/customers affected by a deprecated service on GCP (or any cloud provider), then that would indicate a paying customer behind it. That payment should at least cover costs of running that service (infra, development, software and maintenance). In that case, why deprecate at all? - version the API and continue running it as long as it economically makes sense?

</comment><comment><USER86> TLDR of the article:

> In the Emacs world [...], when they make an API obsolete, they are basically > saying: "You really shouldn't use this approach, because even though it > works, it suffers from various deficiencies which we enumerate here. But in > the end it's your call."

> Whereas in the Google world, deprecation means: "We are breaking our > commitments to you." [...] It means they are going to force you to do some > work, possibly a large amount of rework, on a regular basis

It's fine if they want to go hunting for their Perfect API v10™, but just leave the old APIs alone - build abstractions under them if needed. Don't push that work onto your customers.

</comment><comment><USER87> Two things this essay made me think about.

First: Why don't more programs or services go the `make-obsolete` route and allow people to use deprecated APIs forever? If maintaining compatibility is the goal then I can't see how removing a function in a new API version will ever work. Maybe it's because companies are expecting people to be dumb and call support when they use this deprecated API that was specifically designated as unsupported. "But I can use it," they'd say, and still complain that it breaks or something. Maybe that workload is what they're trying to avoid. But maybe they'd silently move away instead if continuing to use the platform meant rewriting everything.

Second: Why don't people usually have automated API refactoring tools? When I was trying to port a Minecraft mod to a later version I found a bunch of things had changed in the engine. Okay, go to MCP and look at what changed. Guess what? It's all a bunch of Markdown. They painstakingly went through the process of labeling every single API change and method renaming and so on, and they put it all in _Markdown._

There was so much missed potential there. Imagine if you had a policy encouraging your developers to put down every API change you make in your program in a machine-readable, schematized format that could be printed to a webpage or Markdown or something else, but could _also_ be combined with a static analysis tool to at least print out the parts that need changing, if not refactor them, just like Google's internal tooling? If developers don't notate those changes by hand then that information is going to be permanently confined to unparseable changelogs and obscure IRC conversations. That's the kind of thing that a semver increment can never capture the nuance of. Why guess what broke or manually parse changelogs based on a major version increment when you can have a computer do all the work for you?

Maybe it's because a changelog is "information to throw away," that you'd have the painful time trying to read through the moment things break and then can forget about a couple of days later when the refactoring is done. But the changelogs and migration steps are essentially a bridge closing a chasm, with your entire userbase on one end trying to get to the other, and if it's not made easy enough for them to cross they will either give up and leave or stay on the other side forever.

</comment><comment><USER88> > One fun bit of trivia about Bigtable is that they had these internal > control-plane entities (as part of the implementation) called tablet > servers, which had large indexes, and at some point they became a scaling > bottleneck.

Anyone understand the technical jargon here? I'm a bit lost.

<comment><USER89> Check out the original bigtable paper and it'll make more sense. I think the author maybe misremembering some technical details here though.

<comment><USER90> Maybe he's talking about the way that bigtable metadata1 is itself hosted on bigtable? The statements don't make much sense.

</comment></comment></comment><comment><USER89> Clearly author is making things up - there were no gummy sharks in micro kitchens. Sergey spoke at lengths about this.

> I considered using Google Cloud Bigtable for my online game, but it costs an > estimated $16,000/year for an empty Bigtable on GCP. I'm not saying they're > gouging you

Ok, but they are

<comment><USER91> [https://techcrunch.com/2020/04/07/google-cloud-makes-it- chea...](https://techcrunch.com/2020/04/07/google-cloud-makes-it-cheaper-to- run-smaller-workloads-on-bigtable/)

<comment><USER89> Cool so i can build a three node cockroach/tidb/tikv cluster that will have much higher reliability with the same performance characteristics for less money. Sounds like they still gouging you.

</comment></comment></comment><comment><USER92> There are other tradeoffs between cloud providers that are important to keep in mind here. In my experience, GCP has been easier to get started with, and has offerings that are by-and-large more comprehensible. One thing I really appreciate is that a lot of their client libs have automatic auth within GCP (last time I used AWS this wasn't the case or I didn't know about it).

As someone who hasn't been burned by it personally, it is hard to quantify the actual risk and maintenance cost of GCP's deprecation policy, but I know decision makers at larger organizations that are rightfully afraid of it.

<comment><USER93> Easier to get started ? Their UI , IAM architecture and firebase being kind of separate is so confusing .

Aws used to be simpler , while they are screwing it up now, the recent route53 upgrade was terrible , a single click action has now become 4-5 clicks they are still better than GCP.

Azure has the really the best interface (outside of DO) , it is easy to go to related and nested resource in one freaking app. Both google and aws end up forcing you to open three four tabs to setup interconnected apps .

DO interface is really really good ,however they don't have the complexity of features that the big three do , so not a fair comparison

<comment><USER92> That's just been my experience – I remember having much more pains trying to get AWS IAMs and permissions to work, and I really like that they give you suggestions like "this machine could be smaller" or "this role never actually uses all these permissions."

Also I haven't worked _that_ much with firebase, but it seems like a great example of the benefit of using GCP. Firebase is a cohesive and accessible solution to a lot of what can be fairly nightmarish technical problems. This kind of thing will always depend on the project/team/team size, but I'm just trying to say that there are significant benefits to GCP that should be considered.

<comment><USER93> Google are not competing with DO for developer mindspace, for large enterprises stability matters a lot more than new features and enterprise support matters too, for startups serving enterprises they don't do much either, what's left is consumer focused companies like SnapChat who can thrive on just innovative new tech.

I can get Microsoft on a call anytime, I have account managers responsible who talk to me atleast once a month, reach their product teams, get preview access, get the MS account manager of my customer to help with a deal, put them in front of my customer, help with compliance, even get their sales guys to recommend my product. To a lesser extent I can do a lot of that with AWS too, with even sub $100k/year spends they will still put an account manager for you. I am not sure any of this was possible with GCP for most customers.

It is _extremely hard_ to get a human from Google to talk to you even at $100k+/year GCP spends. Google has contracted a lot of partners to do all the heavy lifting in support for them so they don't have to do the hard work. It does not work, the partners can not do much beyond what is available on the portal or clarify beyond the documentation.

Azure serves enterprises really well, and has really made effort with developers and their support for startups in fantastic, AWS is not as good yet, however it feels like they are really trying and their tech popularity works in their favour and they care about backward compatibility _a lot_ S3 API _from 2006_ still works. With Google and GCP it does not even look as though they are trying at all.

<comment><USER94> >S3 API from 2006 still works

An interesting note is that Amazon actually planned to EOL part of this vis-a- vis what path to address objects at, and then walked it back (for buckets created before a certain date): [https://aws.amazon.com/blogs/aws/amazon-s3-path- deprecation-...](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation- plan-the-rest-of-the-story/) I think the behavior is still considered "old style" if not explicitly "deprecated", but is "supported".

<comment><USER93> Yes. That decision to continue to support is costing them a ton of money every day. The older API is _very_ expensive, This is also why backblaze originally did not add a S3 compatibility layer.[1]

This is exactly the kind of commitment I expect from Microsoft or Amazon, it is why enterprises pay premium for a product.

In a similar situation I imagine Google would have just sent a note and with a window of few months and shutdown the old API. Great for innovation and keeping your tech cutting edge, not so much for the customers who are not as agile as they and can only move slowly if at all.

[1] [https://www.backblaze.com/blog/design-thinking-b2-apis- the-h...](https://www.backblaze.com/blog/design-thinking-b2-apis-the-hidden- costs-of-s3-compatibility/)

</comment></comment></comment></comment></comment></comment><comment><USER95> I recently migrated off of digital ocean into aws.

I was originally planning to go to go for the native k8s experience.

Then I read somewhere online that was doesn't sunset services where as gcp does.

I was half way through the migration process to gcp and bailed on a dime.

<comment><USER96> What I really want is a cloud provider that has AWS's "we don't kill anything anyone is still using" and Google's probability of the documentation being both where you expect, having what you need, and being current and accurate (and, yes, I know Google is far from perfect on that, but dear God is AWS utterly _horrible_ and that's just for the services that they are actively pushing.)

</comment></comment><comment><USER97> I would like to offer some kind words for the author of this post but in all honesty I can't.

Why is that guy still giving money to GCP if it's so bad?

Go back to AWS. There's a reason if AWS is still the market leader.

</comment><comment><USER98> Google messed up when it sold boston dynamics (where's my cooking, dishwashing, and laundry (folding) robot?), didn't buy github, and sold motorola (i just bought my wife a motorola edge)

</comment><comment><USER99> So what is the 4th best cloud platform that he thinks might overtake GCP?

<comment><USER100> Oracle, or globally, Alibaba.

</comment></comment><comment><USER101> Well, hmmm, if only there was one or two more FUs _then_ I would have agreed with the Op.(sarcasm).

A phd quit my employer and went to google. On last week we were discussing relative differences. He pointed out searching for _X_ really had no wrong results and even if you didn't like it who you gonna call? Google probably is a little siloed from customers in a way many other business are not. Our IT systems deal in money. There's nowhere to run and hide from customers.

</comment><comment><USER102> This is why Microsoft, for all its faults, has huge mindshare in the enterprise: they don't give up on anything, and when they do, they give you years to make the transition.

</comment><comment><USER103> What is the fourth cloud offering that may be number 3 soon.

I've used AWS and Azure gig.current uses GCP (don't get me started on that) but what is the fourth?

</comment><comment><USER104> Weren't GCP users running around telling everyone _" Google only shuts down consumer products and not its cloud services!!"_

<comment><USER105> We've been in GCP for 5 years and this really hasn't been an issue for us. Good price, solid performance and reliability, some fancy stuff you can't get elsewhere.

</comment></comment><comment><USER106> One thing I was left curious about after reading this is, what is that third cloud that Google could fall behind of? I know they're after AWS and Azure, but there are several candidates for that fourth place.

<comment><USER107> Alibaba, I think. Probably the 'natural' choice for Chinese-facing developers, but I think Azure has a region there as well.

</comment></comment><comment><USER108> The main problem of Google Cloud is simply that that no amount of money can pay off a 10 year late start in a race against the #1 incumbent.

Microsoft still wins on the desktop. The only way to beat them was to compete in new emerging markets.

</comment><comment><USER109> Android is the top example of this by far, everything is half finished & deprecated.

</comment><comment><USER110> Am I in the minority being a GCP fan? Articles like these leave me conflicted because on the one hand he's right but on the other hand I actually really like GCP and Firebase is a godsend. It would really suck if GCP goes under.

</comment><comment><USER111> I've been saying this about Google for a decade... everyone thinks I'm nuts.

</comment><comment><USER112> A perfect, spot on analysis.

The only thing that's weird about this is that it's about 3 years late.

If this had been written 3 years ago it would have been topical.

But really it's old news that google deprecates everything and leaves it's developers in the lurch and that as a result no one wants to use their stuff.

It's not surprising to hear this from Steve Yegge cause he's a super switched on guy, but it's surprising to hear this from him NOW.

Google's cloud ship sailed years ago, they just haven't yet got to shutting the whole thing down, but the writing is on the wall.

Google seems totally oblivious to how much developers distrust them, which is probably because dump trucks turn up every hour or so and dump cash onto everything which probably makes everyone feel that everything they do is awesome, even if it isn't.

<comment><USER113> I mean maybe I'm not their dream customer or whatever, but we're happy with google cloud for our purposes - GKE, CloudSQL, Redis.

I really enjoyed the essay and surely is some truth, but these products have all been pretty good for us.

<comment><USER112> No doubt what is there works and works well.

The issue is that developer confidence is very low due to google being in a never ending relentless drive to cancel stuff.

</comment></comment></comment><comment><USER114> I used to work with Steve. He's a good story teller. But a story teller tells good stories by having a fairly casual relationship with facts and the truth.

For example, emacs lisp suffers from bitrot. Some core things haven't changed in forever but also you can't pick up elisp config from 15 years ago and it's fine today.

All live systems evolve and change. In this case emacs is just so slow that Steve thinks that it doesn't.

<comment><USER115> Nobody pays EMACS a few million a year and runs their core business through it.

</comment></comment><comment><USER116> That's the price you pay for being lazy and using closed, proprietary technologies and acting on their marketing. Like GCP and AWS were the only clouds there. Like you had to have everything hosted by them under "planet- scale" buzzword.

</comment><comment><USER94> Honestly I just use GCP for everything only because AWS console logs me out all the time and 2faing with TOTP is a pain.</comment>