Ask HN: If I Close My Data Centers, What About the People/Jobs Lost? - <AUTHOR> (throwaway account)
I have a chance to basically migrate 90% of my F50's data centers to commercial providers. The cost savings are awesome, but what about the people currently doing legacy stuff? theyre doing stuff like manual config, and future state is everythign automated as much as possible in the cloud and shutter the data centers.
I think the harsh truth is that many wont have jobs post move, but maybe I'm being cynical. maybe theres a chance to retrain the employees, but even doing that automation generally reduces the workforce.
whats your take? what do I tell current employees? how many can I realistically be able to save from being jobless? <comment><USER4> When Netflix moved from the datacenter to the cloud, there were a whole bunch of really good people with skills we didn't need anymore -- sysadmins, DBAs, datacenter techs, etc.

Netflix went to them and said, "do you want to learn some new skills that apply to our new cloud based infra?" Some said no, and were given big severance checks and glowing recommendations. Others said yes, and were retrained.

And some were retrained but it didn't work out, it turned out they weren't interested in the new work, or just weren't needed, and so they too were given big severance checks and glowing recommendations, and went to look for new jobs using both their old and new skills.

It's nice that you're worried about them, but the best thing you can do for them is offer them the chance to learn new skills, either through new jobs or paying for them to learn new skills.

<comment><USER5> This seems best - offer them the choice.

Don't go "learn to code cloud formation in 30 days or lose your job with no severance".

</comment></comment><comment><USER6> Disclosure: I work for Google Cloud (and I assume/hope someone at Google is competing for your business).

Some of those folks happily do updated things, but others (like people that rack servers) won't have an obvious new role in a cloud focused world.

My (now) colleague at Nic Harteau gave a talk at our Leaders' Circle event talking about precisely this in Spotify's move to Cloud: how could he keep the infrastructure team as they moved to cloud? As it turned out, most of them happily adjusted what they were doing because the mission was the same: serve the product teams / groups that need infrastructure from them. Sure, now you don't configure a physical firewall, but there are still firewalls in cloud. In a sense, they're even more important!

I _believe_ Nic said only one person from the group left. The engineer was focused on building their own SDN stack on-prem. That job, and other "things the provider completely handles" do go away. But that person likely would have been massively useful in helping guide how Spotify should use Google Cloud's own SDN setup. There are lots of ways to architect how services interact in clouds, and a lot of it comes down to funny networking.

So I'd say more than you might think, but please don't mislead them.

</comment><comment><USER7> I will assume that at the current state you are able to pay both the datacenter costs and people costs since it seems that is what you are currently doing.

With that assumption in mind, if you do migrate and realize that you now don't need 80% of the people you needed before, it could mean one of two things:

\- you are now able to pocket whatever savings you were able get from the migration and either save it or invest in something to potentially keep that money in the economy, just in a different way

\- you now have 80% of your people free to do whatever other things you didn't do because they had to maintain datacenters. That is huge in terms of efficiency gains and allows you to redirect that money to increase the value provided by your company either by repurposing the current workforce or getting rid of them and bringing people you could previously not bring on board due to existing costs

Either way I believe that everybody is better off After the changes.

Efficiency gains such as the one you are describing is what makes it possible for economies to grow and keep moving forward.

Can you imagine if you had to handle cows, pigs and farms to have a BLT? It's very good that such things are outsourced because that allows for improvement that would otherwise not be possible.

<comment><USER8> Odds are high that the anticipated cost savings sits largely on the folks they intend to sack, rather than on any power/real estate/bandwidth/hardware savings.

<comment><AUTHOR> Actually the cost savings hinge on the massive cost savings from moving to a commercial cloud, laying off workforce would save more, but would prefer to save a bunch by moving to cloud and keep the workforce for other things.

<comment><USER9> That's extremely surprising.

I see two possible explanations (both of which may apply):

(1) You're getting the cloud servers for a remarkable discount, which, as other commenters mentioned, may not last if you don't put in the engineering effort to be vendor-agnostic (though I'm not sure I agree it's as huge a cost as otherwise implied).

(2) You're grossly over-paying for your current environment. Given what I know about the purchasing habits over merely large (nowhere near the size of F50, mind you) companies and, more importantly, the pricing of "enterprise" hardware and _support_ , this seems likely.

If the second is true, then I expect you could gain the same savings, if not more, by implementing your own "private cloud" by emulating the purchasing and ops habits of larger-scale startups or established tech companies (but not yet so huge as to be too custom) who run their own hardware.

You'd still have to retrain and put an end to manual configuration, but I expect that would ultimately be welcomed. Some might be able to keep their jobs almost as-is.

What likely would _not_ be welcomed, and therefore might make it impossible to do, is the political upheaval, since you'd have to put the kibosh on buying expensive, brand-name gear, all but a few bare-bones service contracts, and the level of enterprise sales perks that the decision makers have come to expect. You'd probably have to hire someone who knows how to be frugal like that from an engineering/ops standpoint for architecture/purchasing and is willing to work for such a large company and isn't already employed by FAAMG.

<comment><AUTHOR> Maybe so, I haven't made this move before, which is part of the reason I'm asking here, as unlikely as it may be that "someone who is in this position would be asking on HN." Sorry for being vague on all of this. Regardless:

1.) No specific discount, this is standard, public pricing on major commercial clouds, with some tricks applied like reserved instances, which is part of the reason I am not worried about low prices first and then ballooning pricing later,

2.) Our operations and maintenance and sustaining are in fact more expensive that anyone would reasonably expect, I believe partially due to vendors getting their hooks in decades ago and expanding scope and increasing prices, I've walked the data center floors and coded with the ops teams.

It is #2 that is really hitting us. I have personally moved several apps as a PoC and you wouldn't believe the savings, or maybe you would, which is even more when not ported 1:1 but used in conjunction with smaller VMs and autoscaling.

I would like to retrain to to end all manual configuration and have the current workforce take over all of this, which is a big deal for me personally. Thank you for the response.

<comment><USER10> I'd been wondering about the possibility of #2 when I made my other original reply a few hours ago. Thanks for clarifying.

The first thing I'm reminded of is [https://www.backblaze.com/blog/petabytes- on-a-budget-how-to-...](https://www.backblaze.com/blog/petabytes-on-a-budget- how-to-build-cheap-cloud-storage/). It's from 2009 but this image jumped out at me: [https://www.backblaze.com/blog/wp- content/uploads/2009/08/co...](https://www.backblaze.com/blog/wp- content/uploads/2009/08/cost-of-a-petabyte-chart.jpg)

Let me guess... you're using kit from the vendor at the bottom of that list? :)

No need to Y/N; I'll assume Y because the sentiment's probably representatively/loosely correct.

The commentator your/this reply is to points out an interesting aspect:

> _What likely would not be welcomed, and therefore might make it impossible > to do, is the political upheaval, since you 'd have to put the kibosh on > buying expensive, brand-name gear, all but a few bare-bones service > contracts, and the level of enterprise sales perks that the decision makers > have come to expect._

This sort of jumped out at me a bit, it's a very reasonable assumption to make, particularly as you say you're an old-school operation, which suggests to me that the tech components are heavily politicized due to their management by non-technical types.

Well... the landscape is indeed changing, as clarified by the 9-year old article above.

So major shifting-around of vendor selection is going to ruffle (and maybe squash) a few feathers, but these "old world" tech enterprises cannot reasonably expect everything to keep going indefinitely because commodity- hardware based solutions have become so accessible that it's harder than ever to sugarcoat proprietary systems to even mildly-technical management, particularly with compute and storage, so the writing is on the wall, and you can use that in your favor.

(Well, except clouds. That seems to be where the old-world's shifted to. Heh.)

And the thing is, for a $B operation (for really any value of $B..$BBB), the cost-benefit absolutely goes in your favor if you build everything out and run it yourself. Plus, if you really are a $BBB operation, _you can build your own servers and network kit_. As in, you get to start answering questions like what features you want on the motherboards (no Ethernet since you're supplying your own 10G kit? sure thing... oh you don't want any onboard SATA either? okay), how you want the systems to boot (build your own secure boot implementation!), what BIOS/EFI features you want (you want the half-hour RAM test disableable for your staging boxes? fine), how long you'd like your enclosures to be (36" is really long but your racks can handle the length you'll be able to fit a remarkable number disks in 4U), what firmware optimization suggestions you have for your HDDs (a writeable vendor area in SMART config for how many seconds you want each disk to pause for before spinning up so systems come up slowly and don't kill their breakers? done)... and all of these things will directly translate to cost savings.

If there's one thing that differentiates "old-world" datacenters from shots of FAANG, Backblaze, etc, it's that all the big players' servers look really, realy _boring_ , particularly the enclosures. (Well, okay, the blue LEDs make up for it. :P) And this is because there's almost nothing in those machines that isn't needed. Can you get away with removing VGA? (You almost certainly can with virtualization hosts, even if you're just doing KVM or Xen on top of Linux.) Would doing short 24V or 48V runs help? (Complex, but could lower power consumption.) All of these things serve to save money. How much exactly probably depends on what sorts of overheads you're currently dealing with; just switching to barebones systems that do have a few bells and whistles but are still very minimalist may get you an appreciable saving that all the obscure things I've listed here may not be necessary (ie may have negative/negligible/not-worth-it cost-benefit analyses).

I guess my question is at this point, is the decision scoped down to "what vendor are we selecting" or is building your own DC still on the table? I so hope it is, because IMO that's where the cost savings really is.

Hm, maybe you could go "I have one more idea I want to try before we commit to this, it'll require $$$,$$$" and then go and replace your most expensive storage array or something :P that could be pretty convincing...

> _I would like to retrain to to end all manual configuration and have the > current workforce take over all of this, which is a big deal for me > personally._

I think it's cool you're really getting behind this and making your job description part of your identity :P hopefully this is respected and you don't get too heavily stepped on.

There are some really interesting comments in this thread, and a fair bit of reiteration that clouds are generally expensive solutions. I hold this sentiment myself, FWIW, despite admittedly having a LOT less experience with the practical side of things than everyone else here.

This was what I was getting at with the "the offer will be what the buying team wants to hear" \- at this point it's almost like the vendor sales are actors in a real-time movie that's exactly what the buying team wants to watch. On top of that one vendor is really going to be as good as another (I've been mentioning GCE and AWS, I forgot Azure.)

Cannot keep writing as I must go, may continue later

<comment><USER9> > Plus, if you really are a $BBB operation, you can build your own servers and > network kit. As in, you get to start answering questions like what features > you want on the motherboards

I suspect it's this kind of over-reach that ends up scaring decision-makers off from _any_ in-sourcing (for lack of a better term) solution. To me, the key is in sticking with commodity solutions.

Even putting aside that the computer tech part of the business isn't their core competency, custom hardware is _hard_. In the worst case, you create interdepedent vendor lock-in and indefinite delays. Is there an example of any large company, other than Google, who has mastered this?

I found it interesting that you would mention Backblaze since, even back then, I questioned (I think even here on HN) their decision to go with a custom chassis and the non-standard, low-performance technology of SATA port multipliers. Well over a year before that blog post, there was enough competition in the SAS-expander backplane-in-a-chassis market (e.g. Chenbro, SuperMicro) that I was using them at a cash-strapped startup instead of used Dell MD1000s.

What we _don 't_ know, however, is if those backplanes were actually a _commodity_ yet, at their scale. Maybe Backblaze couldn't actually get enough cheap SuperMicro chasses soon enough. I do recall some resellers charging what seemed like comically high markups, but maybe they were banking on a scarcity I was blissfully unaware of, at my low volume. Still, the fact that the option is conspiciously absent from their bar chart tends to make me suspect their summaries and conclusions, even if I always appreciate their raw data (e.g. disk stats).

To tie this all back to the original topic, sticking with commodity solutions, be they hardware or cloud, and re-training your staff on them will help them avoid the potential career dead-end of over-specialization.

<comment><USER10> > _Even putting aside that the computer tech part of the business isn 't their > core competency, custom hardware is_ hard.

Hmmm. Good point. (Maybe I scared OP away, heh.)

> _Is there an example of any large company, other than Google, who has > mastered this?_

Well, I certainly don't know. I've heard Cloudflare gets their systems made, for some definition of "from scratch", by Quanta. Presumably there's a bit of a decision-making process in there, and presumably Quanta handles everything related to the "why is the devkit sample broken again #51383" side of things and ships a more or less finished product (maybe even with finished enclosures). My reference point for this knowledge is [https://www.youtube.com/watch?v=LA- gNoxSLCE&t=19m31s](https://www.youtube.com/watch?v=LA-gNoxSLCE&t=19m31s) (2015).

Regarding SAS, maybe past a certain ordering quantity (or anything similar that would reveal a massive operation) the market would expect certain behavior like requesting value-adds, ordering with only one supplier, or other things that would generate lock-in. I of course have absolutely no idea [if this sort of thinking is reasonable].

Your points about commodity at scale sound quite reasonable. (You clearly have more experience than I do, which is kind of zero, heh.)

As for suspicion about summaries and conclusions, well, (tangential example) the radial graph on [https://about.gitlab.com/](https://about.gitlab.com/) (the one from Forrester) doesn't show GitHub :D

Yeah, building own hardware is sadly still within the realm of over- specialization at this point. I guess this is because 4+-layer PCB printing is still eye-watering so the amount of cheap iteration that can be done is restricted. There are probably similar cost-benefit disparities as well.

I guess, for the sake of discussion, then... what's the best vector into the middle of "commodity solutions" that scale massively (which this operation sounds like it needs a bit of)? I mean in terms of rough vendor selection etc, what to look for, etc. Admittedly this question is based on my own complete naivete and OP probably has at least some info on this sort of thing available or accessible.

<comment><USER9> > I've heard Cloudflare gets their systems made, for some definition of "from > scratch", by Quanta

I looked just glanced at that video, it looked familiar. What I'm reasonably confident of is that no meaningful definition of "from scratch" applies. AFAICT, it's a direct competitor of SuperMicro's Twin2 line. It's just slightly tweaked (if at all).

Now, it's certainly possible that Quanta offers customization that SM doesn't, like deleting extra on-board parts, but I'd be surprised if they allow even something like a choice in BMC/IPMI platform. Certainly nothing as extreme as exotic power options.

My point being that, if Quanta suddenly decided to jack up the price, CloudFlare wouldn't be over a barrel due to a custom solution. Worst case, they'd just have to pay (presumably higher) SuperMicro prices. Best case, they find a different white-label vendor, because what they're looking for is a commodity, nothing special, readily available.

> Regarding SAS, maybe past a certain ordering quantity (or anything similar > that would reveal a massive operation) the market would expect certain > behavior

That seemed true 2 years prior (2006) when high-port-count SAS expander chips were mainly available super-expensive standalone devices, presumably targeted at the "SAN" market. It took a while for manufacturers and then customers (other than me) to twig to the fact that one could attach a cheap SATA disk to each SAS port on an expander, thereby making it relevant to the lower- end/frugal market.

> I guess, for the sake of discussion, then... what's the best vector into the > middle of "commodity solutions" that scale massively

Before I answer, I'll pick a nit.. I'm not actually convinced that this business needs truly _massive_ scale.

Especially given the OP's mentioning somewhere in the thread that the greatest savings in a POC cloud migration were from a _smaller_ VM with auto-scaling. That suggest that even the current environment is over-sized, and that can easily get worse as computers get more powerful for the same size/cost. Remember, it's a traditional F50 business, not a web-scale tech company.

> (which this operation sounds like it needs a bit of)? I mean in terms of > rough vendor selection etc, what to look for, etc

That said, the job of vendor selection is pretty easy. Since it's commodity, it's pretty much about price.

At times, that has meant that, with a steep enough discount, you could consider Dell (and I've heard HP, but I never had such a discount) for servers, just not for storage. Supermicro seems to have been low-markup all along, and there are always white-labels, though that may not make sense at low scale.

Networking gear isn't _quite_ commoditized, but we're close, with open- source/SDN type platforms.

One thing I've learned is to be open to different vendors for different commodity parts. Sometimes this means considering actually doing some of ones own assembly work, such as mounting disks in arrays. The modern knee-jerk reaction is to reject anything involving that much of the customer's labor [1] on the grounds that it's not scalable or somehow categorically too expensive, but that just leads to an insidious form of lock-in (which is, at least, temporary and limited to the transaction at hand). The integrator may be giving you a great deal on the server but soaking you on the disks (or maybe they're getting soaked on the disks and merely passing it along).

[1] Which can be contracted out, though that's fraught with its own problems, not the least of which are cost inflation and, perhaps more importantly, less "invemstment" in the quality of the final product. Really, though, I think the amount of labor is grossly over-estimated, especially working in anything approaching assembly line fashion.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> Since you're thinking towards the future, think also to the time when you will probably have to move an awful lot of your stuff back in-house, which might be much sooner than you'd expect. Plenty of companies which outsourced in a big way some years back have already been through a similar cycle.

A former Fortune 500 employer of mine went through the outsourcing boomerang. Now I understand that they have big plans to move everything to the cloud within the next two years. Knowing their systems like I do, and knowing that the CEO who is driving this is fairly noobish (note that I said CEO rather than CIO), I don't think this plan is going to work out too well for them. But given that since I left they've had to file bankruptcy, and in doing so involuntarily dropped to about half of what they were before (employees, assets, etc.), I guess this is just par for the course for them now.

</comment><comment><USER12> You obviously have a good heart for asking the question!

I have a feeling that whatever decisions you finally take would take into consideration the wellbeing of your employees.

I would assume that the folks who are doing the legacy stuff have the necessary aptitude to pick up the skills that are needed to automate for the future. Can their existing data center skills be translated into learning about AWS/Azure, VMs, Containerization and the necessary scripting and DSLs around those toolsets?

Obviously there will be reductions in headcount but give them the option to retool for the future. Even if they are not all retained but if you retrain them then they will still have options elsewhere with relevant skillsets.

You are finally running a business after all where revenue and profit are the drivers.

Good luck and I would say that your employees are luckier than those who consider them as mere resources and not people with lives and loved ones to take care of.

</comment><comment><USER13> Are you sure the savings will be real? I've read that real savings results from using the cloud for bursts while keeping the base load on-prem. (Never done this personally so can't speak from experience.)

Once you let those people to you will be losing their knowledge in this area. Not sure if it matters.

</comment><comment><USER14> I really think the worst thing you could do is continue to run the business in an inefficient manner. Improving technology and as a result efficiency is the reason we have advanced so much as a species.

However, you don't need to be callous in your pursuit of efficiency. You should try to give these people an opportunity to update their skills and get a new position. You might not be in a position to do that however.

</comment><comment><USER15> There are a lot of benefits to migrating to the cloud, in my experience, cost isn't one of them. Unless of course you are mucking around with accounting tricks by cutting employees and moving that cost to services or cost of goods sold, I'm not a expert in this, but it works and will save you money because you can control these expenses more easily and because of how payroll vs services are accounted for.

You should aim to keep 50% of your workforce. 25% won't want to change 10-25% won't be able to. I would invest in training on automation and multicloud / hybrid cloud management. I recently worked on a large scale migration and you will definitely discover that the cloud wont work for a lot of your high performance computing, or it will be cost prohibitive. You also need to take into consideration intellectual property, does the company really want all of its IP on another conglomerates infrastructure? But most importantly you need to avoid lock-in to one provider at all cost, training your trusted resources who understand the business requirements on hybrid cloud or multi cloud implementation will really enable cost savings because you will have leverage during price negotiations. If you get locked into one provider you are going to get royally screwed on long term cost, seen it happen.

</comment><comment><USER16> We're in the process of moving our infrastructure to Azure. Being the public sector we take things relatively slow, but we've already fully embraced Azure ad as well as office 365, one drive for business, planner and all those other things.

The next step is our servers, they are already virtual and hosted in our own cloud, but moving our database cluster to a commercial cloud is still a big step politically.

Anyway, you retrain your employees. You may need fewer in the long run, but finding people who are actually professional at stuff like setting up on- premise security inside Azure hasn't been easy for us. We've had 3 very highly regarded consultant agencies to help us, and they quite honestly didn't know anything but tech-book Azure. More than once we've had to send consultants home because it became obvious they knew less about this stuff than our people.

Azure has also opened up for a range of automation through services, but that's not really automatic either. We've had to retrain our people in powershell, and now we're running a lot of stuff through the orchestra service. So right now, we've actually not needed fewer people, just better trained ones.

I image you could replace Azure with Aws or any such service and it be the same story.

<comment><USER17> If you don't mind me asking - why Azure specifically?

<comment><USER16> We're the public sector, we operate more than 500 it systems, some of which won't be leaving the windows platform.

We run office 365, Skype for business and handle our Security through af and adfs for 10.000 employees.

So we're already heavily invested in Microsoft as far as what our IT staff is trained in, and they prefer Azure to say AWS often for non-technical reasons, but that sort of motivation is still important from a management point of view, even if you can run windows in AWS.

That's one reason. The other factors is that we're in the EU and Microsoft were the first to implement our laws, without fighting them. Unlike AWS I can physically visit our Azure servers, though this requires you to throw a lot of money at Microsoft. Last but not least, Microsoft is a trusted partner that we've worked with for 25 years, Amazon is not.

There weren't any other options than AWS and Microsoft, because we're in the eu and in the public sector, and AWS wasn't under the privacy laws at the time we started.

A lot of our neighbors are choosing AWS though, so it's certainly viable today.

</comment></comment></comment><comment><USER18> If you are in a position to make decisions for that many people, is it a safe assumption that you have a fairly robust knowledge of how to enact organizational change? Because based on my limited knowledge, major change like this has processes to be worked through which will help answers some key questions -- what is the current and future state, who will gain/lose, what will your communication plan and transition plan be, etc.

You have much more knowledge to give those answers than we do. What to tell employees, and who can/not be saved from losing their job depends on those answers.

I will say that open communication, with complete answers of what people should expect, will help the change to happen without losing the trust of those who do get to stick around. You will have some harsh truths to share. If I worked for you, I would want to hear them as soon as there are enough answers to not leave me in limbo on how it affects me, personally. Not sooner, not later.

<comment><USER19> I would second this comment and add that proper change management is _needed_ to ensure a smooth transition. As soon as the word is out, if you don't have a plan and/or don't communicate in the right way, your people will start sending out resumes and that can wreck havoc in your business.

As you might expect, replacing top notch techies is quite hard. So managing the transition and making sure those that won't be kept are better off is not only a good heartened approach but important to the success of the project.

Management consultants usually get a bed rep but that's something they're pretty good at. If you're an F50, you have the budget to hire a top one.

</comment></comment><comment><USER20> so, uh, I'm not weighing in on the job stuff... but I suggest you check your numbers on how hosting in the public cloud is cheaper than your own data centers.

I mean, sure, automating things is great, and running your own hardware cloud- style is great for a lot of workloads... but that doesn't have a lot to do with the decision to rent rather than own your servers.

Cloud providers charge a _lot_ compared to owning your own hardware, unless your load is extremely bursty.

The bandwidth prices are particularly eregrious. They are like what you would pay for bandwidth 10+ years ago.

I personally ran a small VPS company and managed to turn a (small) profit while selling compute for rather less than amazon, even though I had to rent space and power from a fancy pants datacenter by the rack. I was essentially paying retail, and I only had 20-30Kw of capacity, max, and even at that scale, owning was cheaper than renting.

I can only imagine that at F50 scale, the economics tilt even further towards own, as you can much better amortize the labor than I could

(and it's not like the datacenter loses money, either. Renting datacenter space is like renting commercial realestate. When your lease is up, the renewal rate goes up to whatever competing space costs + what it costs you to move. Owning is so much better in that respect.)

<comment><USER21> Amazon etc negotiate prices for large accounts so the price you pay will not be the same price that a Netflix pays.

<comment><USER20> Sure, and it's possible OP really _is_ saving money by renting their hardware instead of owning. I don't know the details of their needs or of their costs or what cloud provider they are moving to or what deal they cut with that provider.

I'm just saying, there's currently a huge push to rent your essential services from service providers such as amazon, and from what I've seen, it's pretty easy to "move to the cloud" and then find that your total cost goes up pretty dramatically vs. owning hardware.

I mean, yea, "cloud" does make sense sometimes. But... it is sold as being cheaper, and quite often it is not.

Even if the rates look good up front, you've got the same problem that you have when leasing commercial realestate from professionals; they are professionals, which means that when it comes time to renew the lease, the rate will go up to match what renting something similar would cost from a competitor, plus slightly less than whatever they think it will cost you to move. If you can't switch cloud providers and your cloud provider knows it, you can count on your negotiated rates going up next contract renewal.

<comment><USER9> > it's possible OP really is saving money by renting their hardware instead of > owning.

Is it, though? Other than the special case of Netflix, it strains credibility that any cloud provider would be willing to give enough of a discount to match, let alone beat, ones own hardware. How would they make money?

This can be especially true if one "unfairly" compares a cloud provider's relatively expensive hardware choices to ones own cheaper choices. For example, AWS with dual 10GE motherboards and CPUs that weren't the best performance-per-dollar when they bought them, if all you need is single gigabit (and don't need particularly high single-core CPU speed).

> the rate will go up to match what renting something similar would cost from > a competitor, plus slightly less than whatever they think it will cost you > to move.

Fortunately, there's an upper bound, which is the public price, which has, at least historically, been coming down. Also, AWS has the spot instance market, which can provide downward price pressure from even that ceiling.

<comment><USER20> >Is it, though? Other than the special case of Netflix, it strains credibility that any cloud provider would be willing to give enough of a discount to match, let alone beat, ones own hardware. How would they make money?

Right now, as I understand it, Uber is losing money driving _me_ to work every day, and I'm not really sure how they will ever make a profit off of that. if they double their prices? I'm gonna buy a car, drive my own self to work, and use uber once a week when I go to the bar. I don't really see how buying fake market share (rather,market share that only exists when you are dumping) helps them when it comes time to not dump anymore.

Point being, there are plenty of people who are willing to sell two dollar bills for a dollar when the economy is hot. (I'm not suggesting amazon does this... but they certainly could. It's possible. )

Even in more rational markets, loss leaders are a big thing, and in leasing situations, where it's hard to move once you are in? loss leaders are a really common sales tactic. How often have you gotten "first month free" rent deals on one year leases, that work out to below market for the first year (because of the free first month) but above market after that, even if they don't actually raise the rent?

Datacenter leases often have a similar 'ramp up' period, where you pay less at the beginning of a contract than towards the end, even not counting the fact that they jack up your bill by however much they think it will cost you to leave.

I mean, these all can be really good deals if you know that it's easy for you to move later,and it's certainly possible (though maybe not easy) to setup your infrastructure so you can switch cloud providers with little cost.

>Fortunately, there's an upper bound, which is the public price

This is true...but that's a pretty high upper bound, when you compare to owning hardware/leasing pipe and power at all but the smallest scale.

>which has, at least historically, been coming down.

Historically, the cost of compute has come down, though slower than the cost of hardware (at least until lately) the cost of bandwidth has come down... much more slowly.

>Also, AWS has the spot instance market, which can provide downward price pressure from even that ceiling.

I think spot instances are a difference in kind (and actually a really great deal, if you have a workload that can be interrupted like that) - but most workloads I have worked with, especially in corp, don't really work that way.

<comment><USER9> I think most of us here understand the concept of loss leaders and of sacrificing profits (even operating at a loss) for market share.

My point isn't that it doesn't happen, nor that it _couldn 't_ happen in this space. Rather, my point is, why would they bother (be willing)?

> there are plenty of people who are willing to sell two dollar bills for a > dollar when the economy is hot. (I'm not suggesting amazon does this... but > they certainly could. It's possible. )

In fact, we pretty much know, because Amazon has revealed it, that AWS is making money hand-over-fist, and they do this while having sizeable (existing) market share.

Would the other providers compete on (non-public) price just to gain (new) market share? It's _possible_ , but I maintain it's an extraordinary proposition, considering Microsoft's overall software ecosystem integration advantage with Azure (much better lock-in) and Google's various attempts at features, including AI hardware.

Of course, this is all fairly academic, because..

> This is true...but that's a pretty high upper bound, when you compare to > owning hardware/leasing pipe and power at all but the smallest scale.

The OP admitted that the cost savings is based on public pricing, so the decision makers don't actually care about the hugely inflated cost of cloud infrastructure! (Or, as in my personal experience, they refuse to believe it, which makes job hunting particularly surreal). That further supports my belief that the providers have no motivation to negotiate price.

> Historically, the cost of compute has come down, though slower than the cost > of hardware (at least until lately)

(Did you mean the parenthetical to apply to this or to the statement about bandwidth?)

Is there anywhere that tracks this? I've only ever done that-point-in-time pricing analysis, never historical tracking. I tend to wonder how much of the difference is due to Amazon's specific choice of hardware for each generation. That is, would the trend look different if you limited the comparison to the cost of hardware with only the same specs as theirs (which are, of course, not necessarily, or even ever, the best bang-for-the-buck)?

> I think spot instances are a difference in kind (and actually a really great > deal, if you have a workload that can be interrupted like that) - but most > workloads I have worked with, especially in corp, don't really work that > way.

Agreed, and they require additional engineering work (and forethought) to take advantage of properly. I just like to mention them to avoid any cries of "but cloud doesn't _have_ to cost full price" when venturing into the territory of how expensive it is. (Also, with AWS, previous generation, and reserved instances, though knowled of the latter is implied with the spot market).

<comment><USER20> >Would the other providers compete on (non-public) price just to gain (new) market share? It's possible, but I maintain it's an extraordinary proposition, considering Microsoft's overall software ecosystem integration advantage with Azure (much better lock-in) and Google's various attempts at features, including AI hardware.

The better their lock-in, the more sense it makes for them to lose money at first, if that's what it takes to get you into bondage. (assuming they have a way of raising the prices later, or of dropping their costs later) I'm just saying that it would totally make sense to do this if it got you more business. Maybe they only do it for especially juicy customers? I mean, yeah, I'm reaching for some way that it might conceivably be cheaper to rent servers when you have already done the work of setting up your own datacenter, and it's hard.

(now, I do understand that most in house datacenters have ridiculous policies that managers would be happy to pay 5x to get around. But if you are at f50 scale, you can setup your own "internal cloud" with amazon like policy affordances.)

In the aughts, the standard thing to do when starting a VPS provider was to come in as cheap as you possibly could, get as many customers as you could, and then just leave your prices the same as the hardware (and thus power) costs fell out from under you. I mean, I don't know anyone who lost money on their marginal customer, but plenty started out with margins that wouldn't have been any good long-term.

That said, I don't know of any cloud infrastructure providers who seem to be losing money on their marginal customer, and as you point out, there's very good evidence that AWS makes a killing. My impression is that it's an industry with pretty good margins, once you scale enough to amortize your labor costs, which is harder than it looks. I mean, in those days, hardware(and thus power) costs were dropping dramatically, labor costs were not, and even if you have a scalable system where one person can handle a lot of servers, you need a minimum size team, which gets expensive fast.

>(Did you mean the parenthetical to apply to this or to the statement about bandwidth?)

the recent change is that hardware has gotten... more expensive in the last little bit. not just ddr4 (but mostly ddr4... and that's most of your cost, a lot of the time, as that's the thing you least want to oversubscribe) To me? this is shocking. Honestly, I haven't really been keeping up with bandwidth prices since I sold my thing, so I wouldn't know as much about that side of things.

>Is there anywhere that tracks this? I've only ever done that-point-in-time pricing analysis, never historical tracking.

pcpartpicker has nice historical graphs for hardware pricing that are publicly accessible (but that don't go back so far) One of my super early projects was a web price scraper for books... I wish I had kept up with that project and expanded it to computer parts, because there's a lot of data that I don't know how to get anymore.

[https://pcpartpicker.com/trends/price/memory/](https://pcpartpicker.com/trends/price/memory/)

>I tend to wonder how much of the difference is due to Amazon's specific choice of hardware for each generation. That is, would the trend look different if you limited the comparison to the cost of hardware with only the same specs as theirs (which are, of course, not necessarily, or even ever, the best bang-for-the-buck)?

From what I've seen, the x86 server market is pretty efficient. Yes, yes, the professionals will take you if you let them, but it's pretty easy to get to within 10% of the tray price even on small purchases, and it's hard to get much below tray prices, even on large purchases - (dramxchange is a pretty good resource if you want to know about 'tray price' for ram) I mean, x86 hardware is... x86 hardware. Dell vs. HP vs. supermicro- it mostly even uses the same chipsets, and most of the cost is the cpu/ram, so there's really not a lot of difference in who you buy from, as long as you get quotes from more than one vendor.

The times I've seen people get really ripped off, they got sold different types of hardware; like one place I worked was getting sold blade servers using FBDIMMs. Blade servers for "efficency" \- of course, there was only enough power for one blade chassis per rack. And FBDIMMs used like 3x the power of low voltage reg. ecc ddr2. I pointed out the existence of the low power xeons that took reg. ecc ddr2 rather than FBDIMMs, and ran enough benchmarks to show that they were about as good as the regular xeons of similar speed... and used a lot less power. I'm not aware of any similar traps in the market right now, but I am a little out of the loop.

My personal theory as to the lack of cost concern in "cloud" is that times are hot. When times are hot? you jump. make hay while the sun is shining. If spinning up cloud servers is faster than developing your datacenter infrastructure? who cares if it costs 5x as much. Setting up openstack is a huge pain in the ass, and you're going to have growing pains for the next six months, if your team is really good.

My personal theory is that this trend will reverse with the next downturn. Meanwhile, I'm working for other people, which is pretty nice in a lot of ways, even though it does feel sad that all the time I spent in managing x86 hardware is something people just don't care about anymore.

<comment><USER9> > nice historical graphs for hardware pricing that are publicly accessible

Even then, that's only half the comparison, though. Do you know of anyone keeping historical AWS prices? That would seem more likely, actually, than a reliable pc-parts database.

> I mean, x86 hardware is... x86 hardware. Dell vs. HP vs. supermicro- it > mostly even uses the same chipsets, and most of the cost is the cpu/ram, so > there's really not a lot of difference in who you buy from, as long as you > get quotes from more than one vendor.

Noooo.. no you too! :)

I find that this is true, but only if the workload is primarily CPU- _and_ RAM-bound. My sense is that somewhere between the dot-com boom and now (500x increase in speed?) CPUs got fast enough, and software got parallel enough, that the challenge in scaling shifted CPU capacity to I/O. (More RAM always seems to help, but I agree there's not much differentiation there).

As such, at any given point in time, each vendor's offering at a particular (otherwise competitive) price point might not make sense. The mobo might not have enough PCIe slots with enough lanes (or too many on too few slots). The onboard 10GE might use a chipset that performs poorly under OpenBSD. The chassis might hold comically few disks (ahem, Dell) or have no SAS expander backplane option.

The other differentiator, is at the high end, for >2S systems. It's arguably no longer "commodity" hardware, but the parts are more similar than different, and there's no lock-in if one can wait at least the full release cycle. Still, it's rare that anyone is brave enough to buck the trend and pay the 2x-4x price premium to scale "up" before embarking on the engineering nightmare that is scaling "out" (which I'm not convinced has any less of a hardware price premium, just that it can grow without bound).

> Blade servers for "efficency"

I've never understood that mass delusion, either, though at least not there do exist high-power-density datacenters that can house them. I assume someone managed to spread the myth that it's _space_ that's expensive at a datacenter, rather than power and cooling.

> I'm not aware of any similar traps in the market right now

I'm not, either, at this very moment, but every time I've looked, out of curiosity, there's something like that. All the vendors will have at least one product that seems objectively worse in every way to the older products, including being more proprietary, that I wonder if it isn't some custom design that flopped that they're trying to dump.

>My personal theory as to the lack of cost concern in "cloud" is that times are hot.

Agreed. Easy money, be it from VCs, low interest rates, and/or a booming economy (perhaps fueled by the first two) helps explain it.

> Setting up openstack is a huge pain in the ass,

Is that really the go-to option? I'd think there are much lighter-weight alternatives. I'd have thought provisioning the base infrastructure were much less a PITA than writing the "infrastructure as code" which one has to do even for public cloud.

> growing pains for the next six months,

Is it really that long? Or is that just the perception?

> if your team is really good.

Maybe that's the problem. Good help is hard to find?

Still, what about 1-2 years down the line of making hay at those 5x costs? There are plenty of companies like that and, at some point, it has to at least be slightly tempting to go through the effort, including hiring. Maybe the money really is too easy.

> My personal theory is that this trend will reverse with the next downturn.

I've been thinking the same thing, but I'm wondering what will happen with all that excess capacity that cloud providers are suddenly left holding.

> Meanwhile, I'm working for other people, which is pretty nice in a lot of > ways, even though it does feel sad that all the time I spent in managing x86 > hardware is something people just don't care about anymore.

I've always worked for other people (which also makes me worry about too strong a downturn), and the hardware management has only been a _part_ of what I do.

I'm not so much sad at the apathy toward one of my skills. It's more that I'm so limited in how much performance I can squeeze out of the infrastructure or how low I can keep the cost. It makes for weird tension.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> Just do it. Most old school sysadmins want to write their Perl and sh scripts not terraform and cloud formation scripts.

I got my devs to embrace GCP, AWS, ansible, terraform, docker, k8s, devops and SRE before most of the admins. Most of them left to go safe companies where they can stick to their old ways.

<comment><USER23> _I got my devs to embrace GCP, AWS, ansible, terraform, docker, k8s, devops and SRE before most of the admins._

Yes, devs are often enthusiastic because they think they can ditch those boring old operations people who always stop them having fun.

Next thing they know they're on call 24/7 and don't get to write application code anymore, just operations stuff...

The truth is the need for ops people doesn't evaporate when you go cloud, be wary of anyone who says it does. This is the same dynamic that played out with MongoDB.

<comment><USER24> The new reality is that the barriers are gone. Ops must dev, dev must ops.

Who can get away with a job that just requires them to open a Java IDE? Those jobs are just as extinct as a pure ops role.

<comment><USER23> With MongoDB the devs said "we don't need a DBA anymore". But what they failed to understand was that the DBA was _not_ the person who knew all the database command syntax, but rather the person who was responsible for the data - <USER2> availability, security, and so on. And getting rid of that person didn't get rid of the reason the job existed. I dare say that came as a bit of a shock. Well, serves them right for stabbing a fellow Worker in the back! And it is playing out exactly the same way in Cloud. All the responsibilities still exist, apart from the most basic ones such as "sweep the floor of the datacentre". Now who's going to take them?

</comment></comment></comment></comment><comment><USER25> Cloud/Comerical <> automation.

I have been burned in the past by "commercial", thankfully I wasn't the guy who had to go to the board hat in hand and say "this isn't in our control"...

Every comercial provider is going to do everything in their power to "lock you in" they are going to put up features that are "better enough" that you migrate to them and then they own you.

Automate, reduce your workforce as needed (it will be less than your plan) and then look at the commercial provider as a second step, look at them in relation to a real DR plan. You might not like what you see for day to day operations but having them on stand by in the even of an emergency could make you a hero and present a massive cost savings already.

</comment><comment><USER26> I have some experience in this area. My thoughts:

Retraining can be done as part of the project, but you need to figure out what the project and sub projects looks like first. Then you an know roughly how you'll move people around at various phases. It takes time to build new systems and time to train. Line those up and you can take advantage of them.

Be careful with costs. They will blow out if you aren't mindful.

It's going to be messy for a LONG time, which causes a huge amount of pain in the org. Be ready for this.

Moving to cloud brings a whole new set of problems you haven't thought of yet. For example, cleanup becomes more critical (old resources cost money, but can't be "seen" because they're not physical and just a hidden line item somewhere).

There's more. My email is in my profile if you wish to reach out.

<comment><USER27> Cleanup is even a problem in smallish startups with some dozens off devs. I cannot imagine the amount of security groups a F50 must have after some years

</comment></comment><comment><USER28> This happened to my friend in his 50s. He was a storage specialist working for a bank. Since they outsourced to IBM a few years ago he's been stacking shelves in a supermarket.

<comment><USER29> If you let yourself get that specialized then you're always going to be one or two innovations away from a layoff. I kind of feel that anyone who inhabits a niche like that for a long time is going to be in danger of becoming a victim of their own comfort seeking. Our industry is not kind to those who don't continually challenge themselves and learn new things. As a counter example to your friend's situation, I'll be turning 58 this november, and I'm an SRE/devops engineer working on kubernetes-based infrastructure in google cloud. Five or six years ago I was a back end .NET guy, a few years before that I was doing ASP web apps. Before that C++ and COM/DCOM stuff, and before that Orbix/CORBA, and before that... you get the picture. The old saw is that nothing stays the same, so if you're not moving with the world you can be sure it's moving past you.

</comment></comment><comment><USER30> You're taking the marketing approach as opposed to coming up with real solutions -- think critically about the business need, investigate the cost savings and make a transition plan for those who actually add value to the organization.

1) Consider if your organization has a future strategic value for having the org capability in house (is this datacenter simply a necessary evil or does your organization deploy new technologies to new locations for any part of its business)?

If so, you may be outsourcing a core part of your organization's capability (consider it). If not (e.g. these are all simply HR and other general business systems [ or services which do not depend on anything physical], you should proceed to evaluating costs

2) By the thought that you would outsource the majority of jobs means that you probably shouldn't have had the employees in the first place (in the Fortune 50 I know, we mostly used contractors with a quarter dozen employees). Hence I would evaluate if there are other business units which might benefit from their skill (power related, analyst related et al) and offer training per the previous notes.

A good decision to be a part of and my hope is that your answers here inspire the right questions (to allow for the right transition plan for your organization and company as a whole).

</comment><comment> ezequiel-garzon Out of curiosity, what does F50 refer to here? Google doesn't help, nor does [https://en.m.wikipedia.org/wiki/F50](https://en.m.wikipedia.org/wiki/F50) .

<comment><USER31> It refers to the 50 biggest companies in the [https://en.wikipedia.org/wiki/Fortune_500](https://en.wikipedia.org/wiki/Fortune_500)

</comment></comment><comment><USER32> Good luck being at someone else's mercy when the problems that are being held together by duct tape come to a head.

<comment><USER33> At the same time, I rather negotiate with a service provider than employees who have implemented their position into the system.

</comment></comment><comment><USER34> Since it sounds like you're going to save a bunch of money, I suggest letting them go and using that money to provide very generous severance packages (months of salary at least). Then they can choose to retrain themselves, take some time off, whatever. It works for Netflix.

</comment><comment><USER35> Kudos for keeping your sights on helping your team! I'm a big fan of holding on to a group of people that understands your business (I'd rather retrain a technology than instill passion and domain experience).

Not 100% relevant (and not my better writing) but here are some of the things I've seen companies do in terms of moving people around from NOCs that might give you some ideas [https://www.pagerduty.com/blog/future-of- noc/](https://www.pagerduty.com/blog/future-of-noc/)

If you'd like a soundingboard to brainstorm with feel free to ping me at email@example.com

</comment><comment><USER36> Hate to do this, but be mindful people bearish on the benefits of moving in the first place will be downvoted crazily, potentially by people who have an economic incentive to convince you this is a good idea.

</comment><comment><USER37> They need cloud certifications. They should see this as a fabulous opportunity to get paid to do cloud training, hands-on work. If you become a cloud architect or a cloud security expert, whether it is at your company or somewhere else, there will always be great opportunity. If they refuse to learn to serve a changing market, that is where they will run into problems. You can only ride a single skill for so long. If you stayed in the horse-shoe business after about 1904, life was not great.

<comment><USER31> I'd recommend CNCF's training for Kubernetes: [https://www.cncf.io/certification/training/](https://www.cncf.io/certification/training/)

We offer a free edX course and then a paid one that prepares you for the Certified Kubernetes Administrator exam.

Disclosure: I am executive director of the CNCF and helped create these courses.

P.S. No job will be around forever. You are not doing you or them any favors by trying to maintain them in inefficient roles. But you can invest the resources to bring them with you to the new world.

</comment></comment><comment><USER38> Your employees need skills that are useful and in demand. They need those going forward, whether you close your data centers or not.

I'd suggest you be upfront with them. Use this as a pro-active way to show you want them around, but it's not business-as-usual. The good ones will see this as an opportunity for the long term, while others may view this as a time to move on.

To reinforce plans, show them what skills you need after the move. This will give a north star to motivate those who want career mobility.

</comment><comment><USER39> The implementation of hybrid cloud support is the key thing to consider pursuing both from a strategic and technical perspective. I have provided complex hosting for large corporations for more years than I like to remember and it is the only thing of lasting and permanent value for a few reasons:

Having a completely migratable cloud setup, between any cloud provider, or back to your own environment (OpenStack, etc), is the single most important long term investment to consider for your stack.

Your situation sounds very advantageous in having the expertise ready to containerize machines, and dockerize applications that may not be. This too, is a very marketable skill for people who may be leaving.

Historically, connectivity has driven location of data. Popularity has oscillated between on-premise vs client-server setups (cloud). Currently, the cloud is in vogue, for many great reasons. The pendulum though seems to have historically swung both ways.

While network speed will continue to increase, and computing power will continue to increase, other factors may play into why a corporation wishes to keep it's data in-house, be it political, security, industry practices, or a sudden wish to change directions due to a failure too many of a cloud provider.

The cloud providers do a very good job of making it easy to get sucked into their world.

It would be interesting if you followed up here with the decision you made and why. Not enough thought seems to be out in the open about these types of decisions, let alone at your decisions.

</comment><comment><USER40> This sounds like a very exciting time, the work is changing, the very economics of the business are changing. Remember to lead throughout this transition. Communicate well, frame the transition in a positive way and take care of people. Do not allow your exuberance at the potential cost savings blind you to the pitfalls in a project of this magnitude. The transition will be harder than you think, take longer and cost more. That being said, it's probably worth doing. Along the way you must remember that your employees are a super important part of your business so treat them accordingly.

Offer training. The automation work is more gratifying and valuable anyway. You'll be offering them a step forward. Some will take it, some will move on. Even those who move on may find that the change catapults them forward as well.

I trained a colleague in cloud automation in two months. Admittedly he is very capable and committed, but you'll find that in some of your people too. He's now worth more than anyone I could have hired. You have the opportunity to elevate your existing employees significantly. Embrace that opportunity.

PM me if you'd like to see the training materials I've put together.

<comment><AUTHOR> I would love to see those training materials! I don't see any contact info, but maybe you could shoot me an email to the email in my profile?

</comment></comment><comment><USER41> If your business has competitors, and they move to the cloud, conceivably they will be able to lower prices, and/or provide more service/features because of cloud scale. In most instances, the less your competitors charge, the more customers flock to your competitors. It is not personal, but economic in my opinion.

<comment><AUTHOR> Our business is an old, traditional business that happens to have lots of data centers for legacy reasons. Even though we spend $BBB on data centers, they're not the core of the business and a very small part of overall revenue. Realistically, even if we didn't move from data centers we would be good as long as the core non-tech business functioned.

<comment><USER41> As an economic principal what you suggest is not the best use of inventor assets and compounds when you factor in employee efficiency/return per employee. If you invested your hard earned money into this business you would want cost efficiencies and high productivity. Company executives have a fiduciary responsibility to do so in the US. If you're running a business and are not seeking cost efficiency but instead are more interested in employee satisfaction, likely you should be in HR and let the CEO handle day to day operations and the bottom line, so you can contribute to the company by driving employee retention, satisfaction or some metric that reflects positively on the bottom line. It's OK to the a CEO. It's OK to be in HR. It's not OK to be a CEO in HR. It's not OK to be a HR as CEO.

<comment><AUTHOR> Point well taken. But as many have mentioned in this thread, it is very hard to hire good people and the people I have already have deep knowledge of the systems and business and are good at what they do. Just from that efficiency perspective it makes sense to retrain per fiduciary responsibility. The fact that I can keep lots and lots of people working instead of terminating is an awesome bonus that I feel great about.

</comment></comment></comment></comment><comment><USER42> I imagine it is going to take you a LONG time to do this. Some apps will migrate easily, others not so much. If you want to look after your people build a program of work in place that helps you realise immediate benefit while giving you the change to reskill people as you migrate to newer tech.

</comment><comment><USER43> Hire for attitude, Train for skills - If you've good dedicated people that you can trust, I'm sure they can be trained for other positions. Good people are always hard to come by, and they will be even more loyal if they will see that the company doesn't ditch them.

</comment><comment><USER44> Your competency is eventually going to do the same and save money and have advantages over you if you don't take advantage of the awesome cost savings. Then you may cease being competitive and would have to close the data centers anyway.

As others said, you could retrain them.

</comment><comment><USER45> Retrain them in AWS, devops, or something that has similar need, versus hiring from outside.

<comment><AUTHOR> Best way to retrain without having to pay AWS a huge fee for all the certs?

<comment><USER46> Negotiate a discount for the certs as part of the large contract you'll be signing with AWS

</comment></comment></comment><comment><USER47> While automation is removing some jobs, we are far from a fully automated society. I obviously don't know about the specific job markets where your data centers are, but in general, the ones having the hardest time finding a job aren't the ones with years of work experience in running data centers.

They might have to retrain and if they live in a small town they might have to move/commute to find something new however. This can be hard enough for some people that they won't be happy about it.

In the end, I think it's hard to argue that we should keep being inefficient just to save jobs. History haven't been nice to that kind of companies.

</comment><comment><USER48> First path is to retrain them. Getting more cost efficient can coincide with overall corporate growth.

Some people prefer to remain in their specialty or struggle to make the leap. In that case if they have tech skills, someone will take them. It may require a move, but they will land. It's one thing if you're firing people in the dark days of a deep recession. (Folks let go in 2001 had to leave California) It's another thing if you're letting people go into a market of 4% unemployment. If they can't get jobs with similar pay, then you were overpaying them.

</comment><comment><USER49> Tech jobs exist because of automation and they eventually go away because automation. But every dollar saved in salary is available to spend on hiring people to do other jobs. It works out in the long run, but that doesn't make it easy on those who have to transition.

I think it would be useful if employers eliminating jobs could have sort of a reverse auction where other employers offer comparable jobs in exchange for a portion of the savings. And the workers could either take their severance or accept one of those offers with no interruption in their benefits or work.

</comment><comment><USER50> I would build your own cloud. You have customers. Don't give them away.

</comment><comment><USER51> My rule of thumb: invest 30% of the cost savings into training your people on skills you need now but also with a view to future plans. Invest in preparing your people for new ways like DevOps, microservices. With their experience and knowledge they can save you even more looking forward.

And even if you don't need them all, by training them you give them a better chance when finding new jobs. A small investment can go a long way. Also helps your companies reputation when people don't simply get removed.

</comment><comment><USER52> You should do the move. Sysadmin is now a niche discipline, people work in devops now. The tech economy is good and their skills are getting more obsolete.

Just start slowly migrating things, and train them in devops skills. Then, even if you have a reduction in a year, they will have a year worth of modern skills in EC2 devops and the like.

It's going to happen, so better it happens now than in the future.

</comment><comment><USER53> I've never managed more people than myself, BUT don't underestimate how difficult it is to find people in IT. It's really, really hard, and even harder to find people willing to stay longer than 2 years.

Loyal tech workers are invaluable right now. But then again I'm not the one who has to make that choice.

</comment><comment><USER54> Maybe this is unrealistic somehow, but what about paying the difference in cost directly to the affected employees?

</comment><comment><USER55> You can reach out privately to other companies and leaders you know if they may have a position for them. HR can also help find positions for people, that's part of their job.

What you can't do make a decision that's good for a very small number of people at the expense of everyone else in the company if you're a leader.

</comment><comment><USER56> Your people shouldn't be doing manual config regardless of if you're on-prem or cloud. If they are, you're going to have a tough transition to cloud, and you're still going to need a lot more of those people than you think you will.

</comment><comment><USER57> Can you last long term if you don't move? If there are huge savings to be made others in the same space might jump on the opportunity.

It's a horrible thing to have to let people go but the alternative may be far worse.

</comment><comment><USER10> Saying first what I want you to see the most: please add a throwaway email to your bio profile (and maybe add a new comment with it so more see it). I guarantee people will want to say hi privately - not [just] vendors but random people who have on-the-ground advice.

\--

Hopefully you still see this even though it's been a few hours.

Glad I went through the thread: there was quite a bit of extra narrative and detail in your comment replies.

> _Our business is an old, traditional business that happens to have lots of > data centers for legacy reasons. Even though we spend $BBB on data centers, > they 're not the core of the business and a very small part of overall > revenue. Realistically, even if we didn't move from data centers we would be > good as long as the core non-tech business functioned._

> _Thousands [of engineers], many close to retirement age. Think a company > like UPS /Fedex that isn't at its core a tech company but with tech > datacenters for logistics and billions in revenue._

> _[T]he cost savings hinge on the massive cost savings from moving to a > commercial cloud, laying off workforce would save more, but would prefer to > save a bunch by moving to cloud and keep the workforce for other things._

\--

Ok, here's my take.

I wonder what your [job] position is, heh. Maybe someone with a CTO's ear, or maybe you're an unusually empathetic CTO.

In any case you're clearly at the point where you've had the internal discussions, the trigger has been assembled and is ready to pull, and you and/or the team (committee?) has probably already been groomed by all the gigantic providers' sales teams, which of course pull out all the stops in your case. If I understand these things correctly, if you haven't yet had an amazing lunch or two on one or more vendors' dimes, you will :) (or at least the high-level people will, would be nice if you get included).

Sales is important. But always remember, particularly for large potential accounts like yours, the sales numbers will always be what the high-level execs want them to be, and the sales teams will be spinning the story the execs want to hear. The on-the-ground experience is always going to be rough. You've probably experienced similar transitions, and that's what's informing your skepticism some of your team won't have jobs. Nice you're giving your people some thought.

On the one hand, it doesn't really matter who you sign up with, as vendor ceases to matter somewhat short of the $B mark.

On the other hand, try to mitigate the risk of discovering that a vendor can't provide a given feature at some eleventh hour, or you may end up in a confused tangle of AWS and GCE deployments (because the other major vendor will of course have the missing feature, and the fragmentation will begin).

How I'd achieve this: every time you go "ok, so this is our implementation blueprint", start the rounds of internal technical enumeration over again. Spend time with the engineers to understand pain points. Talk to as many people on the ground as possible, especially the forgotten people in the corners. One-on-ones, or small groups of people who are known to gel well with each other (no impedance mismatches, work well together), could be good for extricating annoying gotchas. This could look like devs aimlessly paging through source code and saying what they think (this could catch tiny things well), or it might be more structured with general architectural discussion.

Do NOT have the cloud provider vendors do this work (of talking to your existing team); I wouldn't be surprised if they offer to do it, but if they do, all their feedback ultimately goes to the sales guys (via an internal engineer digestion process), and not directly to you.

Well, the analysis team will probably generate a high-level report that contains all the information they've discovered (this is probably legally mandated for full disclosure about what they now know), but all the on-the- ground details about inefficiencies, vendor-specific pain points and so forth will only be derived by vendor-internal engineering debriefs, and you'll naturally never get any of this particularly valuable information - unfortunately this approach is within the vendors' business focus, as inefficiency means more compute resources used ($), and pain points mean more vendor-infrastructure maintenance ($). You could respin this statement to say that the vendor who makes the most attractive offer is the one who knows their system is the worst fit for your system, but I don't know :)

Instead, if you don't have the internal resources (or freshness/objectivity) to put together a bird's-eye view, get an outside consultancy to help, directing them to investigate your current architecture top-to-bottom-and- back-to-top-again, and advise on ways the system can be rearchitected for efficiency and optimization within modern cloud infrastructure. (They'd then be the ones doing the one-on-ones and so forth I mentioned above.)

Another thing that may be extremely useful would be to take the "we're moving to the cloud traction" and carefully smudge out (some of) the dividing line between "we're overhauling everything to migrate it and make it work most efficiently within $vendor" and "we're rearchitecting major components so they work more efficiently overall" so the line between these two sentiments is blurred, and both things wind up happening in lockstep. Obviously the two ideas are intrinsically distinct and this can only be taken so far - "rewrite everything from scratch to get it running in the cloud" wouldn't pass _any_ accounting costing or time-boxing :) - but you can probably push this far enough to extract some usefulness from it (eg, you probably have an idea how long upper level will be willing to wait for this transition to finish - <USER3> years? 1 year? 6 months? - and you might be able to squeeze some interesting and useful tidy-up operations into that window).

Blurring the lines between these two things will play to your existing devs' strengths, as rewriting/restructuring is easier on the brain than the "replace engine in flight" of picking up all the existing state, punting it into to the cloud ad-hoc, and running after all the resulting fires that get created and scrambling to get it all working perfectly.

It won't be a perfect transition if you're really a $BBB operation, and restructuring will be the easiest way to cohesively get the existing devs into a flexible frame of mind and ready to deal with whatever curveballs get thrown their way.

This is one way you could efficiently achieve keeping a lot of your existing team.

Plus, restructuring means everything can slowly be built up onto the new system; picking everything up at once and expecting it to land on its feet probably won't go very well.

Depending on how things go, if you bring in new techs to assist with understanding modern best practice/hitting the ground running, I cannot recommend enough pitting them against your current team when doing onboarding/interviewing, as they'd be an EXCELLENT catalyst for judging candidates' character.

These are a couple of lateral ways the current workforce could be creatively employed during this time. I reckon careful management would see everyone kept on until retirement.

The current team knows all the systems inside out. Letting them go would make everything a thousand times worse.

Also, another reason I thought it would be cool to put an email is because your post makes for a perhaps-unwitting but nonetheless effective and interesting recruitment drive; I'm aware of very few (okay, zero) high-level people at equally high-level companies who really care about their employees. In my case this is because I don't have any connections, but the same probably applies to many others.

I don't know why [https://news.ycombinator.com/item?id=16283252](https://news.ycombinator.com/item?id=16283252) stood out to me a few months ago, but your post resonates similarly. Probably because it incorporates a similar human response too. (Note that I have no association with the potential companies listed and have never talked to any of the users in that thread, I just thought it was interesting)

<comment><AUTHOR> These are really helpful insights. Thank you. Position wise yes, high level newly-minted CTO-ish-type office sounds about right, sorry to be vague. Decision is made, implementation and vendor selection is the hard part. I have already moved many applications as proof of concept and the savings seen is one driver of this change, and me being the one that moved them is one reason I get to be involved/doing/advising/engineering this. We are not FAANG and not located in cool locales so harder to attract great talent.

The cloud vendors are in fact already offering to take on a lot of this work of transitioning, which I am avoiding to avoid lock in at all costs.

And for others reading it's not UPS/Fedex just to throw that out there, I saw another thread on UPS yesterday trying to move huge organization's ops to 21st century and was trying to think of examples of what I assumed were large legacy companies that are not core tech and those popped to mind. That thread also served as a catalyst as I'm also wrestling with moving a huge orgs ops to the 21st century.

I have taken your advice on email: firstname.lastname@example.com

<comment><USER9> > Decision is made, implementation and vendor selection is the hard part. >The > cloud vendors are in fact already offering to take on a lot of this work of > transitioning, which I am avoiding to avoid lock in at all costs.

This gives you a distinct advantage, in that you could, subsequently, move _back_ and be your own "private cloud" vendor for even further cost savings.

> We are not FAANG and not located in cool locales so harder to attract great > talent.

You may find that becomes less true as housing prices in cool locales continue to rise. Alternatively, you could open offices in those locales, since, if your infrastructure is going to be cloud-based (public, private, or hybrid), it's less important that the staff be co-located with it.

There's also the notion of a remote-first engineering culture that I'm sure you can find plenty of evangelism for, just by searching.

</comment></comment></comment><comment><USER58> If you don't do the move someone else will eventually given the monetary incentive. This isn't something that can be stopped.

Better that it happens under the watch of someone that clearly cares

</comment><comment><USER21> You're not going to be able to just flip a switch and do that migration instantly. You have time to retrain people and some people will adapt better than others.

</comment><comment><USER59> Why not automate your stuff at your current datacenter?

</comment><comment><USER60> Embracing automation and moving to cloud systems means layoffs for a dying/stagnating company and an opportunity engine for an innovating one. Our startup does GPU visual graph analytics for boosting visibility, and we see a world of a difference in velocity etc when we engage with a company that has cloud data lakes etc than one that doesn't. And, that includes F50s and federal agencies: super normal!

Technology employees should embrace new tools; that's part of the job that ensures compounding effects over time.

</comment><comment><USER61> How many are them? (10? 100?)

Are some of them close to retirement age?

<comment><AUTHOR> Thousands, many close to retirement age. Think a company like UPS/Fedex that isn't at its core a tech company but with tech datacenters for logistics and billions in revenue.

<comment><USER62> Thousands? I really find that hard to believe.

<comment><USER62> Im not sure why I was down-voted. Even if you close data centers and move to the cloud, you still need personnel to manage the infrastructure.

</comment></comment></comment></comment><comment><USER63> What type of commercial providers are you saving money by moving to?

</comment><comment><USER64> Progress

</comment><comment><USER65> Just retrain them.

They're going to have tangential skills to the new platform.

Get your network team to learn all about VPCs and networks and security groups and peering and VPNs, train them to be your in-house experts to deal with changes and expansions and to help trouble-shoot issues.

Chances are you're still going to have systems, your systems engineers can be repurposed to better tie in your monitoring tools, your security tools, etc. You're going to need authentication and identity management, and probably have internal people handling it.

If they're motivated to keep their jobs, all it should cost you is a $5-20K/head in training.

That's peanuts compared to the turmoil of everyone thinking they're going to lose their jobs, or the costs of hiring new people w/ pre-existing cloud skills. It is cheaper than the stop-gap of paying outside consultants till you can bring in the interior staff. Also, these people _know_ your existing systems. They know your business logic, they know what needs to be up and they know what can't break on Thanksgiving weekend. That is valuable knowledge that is a hidden cost in rehiring the work force.

You're going to find new challenges in the new system. If your employees are willing to transition, then transition them. It is your best bet.

If you have a subset of employees that can't/refuse to transition, put them on performance improvement plans. No rumor mill - if you're on a PiP you need to shape up or get out; if you're not on a PiP keep doing well, your job is safe.

<comment><USER66> But is OP really going to have the same cost savings if they retain everyone? I am assuming labour is a huge part in that figure.

<comment><USER67> The ideal situation is to be in a growing company, and to move employees from shrinking areas into growing areas.

For example, if the company needs 8 fewer people to install and set up servers, but 10 more people to work on robust automated code deployments to support business growth, and both jobs need Unix skills.

Hence, the setting-up-servers department budget shows cost savings, the overall technology budget grows by 2 employees instead of by 10, and business growth means overall profit increases.

</comment></comment></comment><comment><USER68> > theyre doing stuff like manual config, and future state is everythign > automated as much as possible in the cloud and shutter the data centers

Ok, who let the C'level out of his cage? Quick, convene a meeting about meetings and distract him with new synergistic products that mesh the work output of the teams into a new daily reporting product with shiny graphs for him to make informed decisions about, preferably by proposing an "cloud only" data scientist.

But seriously. This comment alone shows this person doesn't have a clue wtf they are talking about. If your guys were so focused on manual config that was a management problem, and isn't an inherent fact about the current state of data centers. So you decided to move data centers, you just moved the jobs somewhere else, to the new provider (Let me guess, amazon, microsoft, or google...)

Say it with me: "There is no such thing as "the cloud"."; Say it again: "There is no such thing as "the cloud"."

Your shit still runs on servers, that at the end of the day someone manages the systems of. So they manage the systems at a lower level that is transparent to you as the modern datacenter is able to hotload blocks, devices, vm's, containers, etc across a swath of machines, clusters, etc.

At the end of the day, you made a business decision that decided managing your own on prem wasn't for you. For some companies, that can be the right strategic move, but as the salty ol bofh who has seen it happen, it's not all puppy dogs and rainbows and there are many pros and many cons to such a move... with "the boss" usually completely underestimating the cons.

Next up: the boss asks how to ease the transition of the low level programmers because he's hiring a new wave of nodejs/meanstack devs because "there's so many of them he can pay them less than the greybeards" and "they are young and know all the cool new devops agile cloud cycle methodologies" and "c++ is dying".

Whatever f50 this is I'd longshort the fuck out of it. I bet it's in a downslide for other reasons and IT is the easy dept to squeeze politically while ignoring the root causal issues that are probably management related.

tldr; Data centers aren't going anywhere, your shit still runs on servers, people still design build and rack boxen, design, fine tune, and diagnose networks, munge and wrangle data, etc, etc, and the skills they have are still in high demand the world over. Disagree? Tell me how many data centers as a percentage that have fully automated robotized racking?

<comment><AUTHOR> You really have this completely wrong. I am a systems/network engineer and am newer to the business side. Asking here to get some upsight into the people side. Thank you for pointing out that things run on servers, that was news to me.

<comment><USER9> Sarcasm is rarely well-received here. Also, I think anyone reading any of the rest of the thread would see that you do have an understand of the issues, so no need to defend yourself.

Conversely, the parent commenter uses hyperbole to the point of appearing _not_ to have that understanding.

For example, even without "fully robiticized" datacenters, there are remarkable economies of scale before that, including ordering servers by the rack (or maybe even row at a time), pre-assembled by a vendor.

Things really _can_ be different, even though they _can_ be the same.. even both at the same time!

</comment></comment></comment><comment><USER69> Jobs in the private sector do not come with lifetime guarantees. If you have new jobs you could train them to do, great. Otherwise, give the people you will lay off notice and severance, and let them figure out what to do.

<comment><USER70> To any companies wondering why their employees jump ship at the drop of a hat, or why there's just no loyalty anymore, this attitude is it.

<comment><USER71> Sure, but it's a cycle that is easy to fall into and hard to break now that we're here. Distrust begets distrust.

<comment><USER70> Then those with the power, the employers, can work to earn that trust.

<comment><USER9> This can't be over-emphasized, that, especially with no union, the power dynamic is _immensely_ unbalanced. Even just the information assymetry is huge, since, despite recent innovations like Glassdoor, workers know very little about even each other's circumstances.

The parent's comment use of the word "cycle" implies, by association with phrases like "cycle of violence", that there might be some equality or that both parties may be victims to some degree.

In the face of such assymmetry, such a claim would be extraordinary, requiring a strong argument, more than just an implication.</comment></comment></comment></comment></comment>


Dropbox In Space - <AUTHOR> https://blog.dropbox.com/index.php/dropbox-in-space/

<comment><USER2> Consumer GPS devices do not work above 60,000ft for regulatory reasons[1]. They are required to disable themselves so that they could not be used to build intercontinental ballistic missiles. This likely contributed to the failure to receive GPS data.

[1]: <http://en.wikipedia.org/wiki/CoCom>

<comment><USER3> If you shop around you can find units that work. The regulation only requires disabling if the device is both high and moving fast, although most manufacturers disable if it is high or fast.

The manufacturers often don't say in their specs which way they implemented this, so best practice is to go with a model that others have had success with in high altitude ballooning.

</comment></comment><comment><USER4> I know this comment isn't really 'on-topic', but ugh--seeing URLs with 'index.php' in them just drives me crazy.

<https://blog.dropbox.com/index.php/dropbox-in-space/></comment>


Expected Soon: First-Ever Photo of a Black Hole - ajay-d https://www.nytimes.com/2019/04/08/science/black-hole-photo.html <comment> ajay-d "The data were too voluminous to transmit over the internet, and so had to be placed on hard disks and flown back to M.I.T." One can only wonder how 'large' they're talking about.</comment>


How to sell your company to Microsoft - <AUTHOR> http://www.techvibes.com/blog/jon-gelsey-director-of-acquisitions-and-investments-at-microsoft-talks-tactical-at-banff-venture-forum

<comment><USER2> This appears to be the guy that makes acq decisitons, or works for the acq department. I know I read this intently.

</comment><comment><USER3> > have your investors deck be 100% complete, be prepared and be quick

Anybody know what this means?

<comment><USER4> He refers to the Powerpoint presentation that you would show to VC's or other potential investors. It includes the high level objectives of your company, why you're different, market size, plans, etc.

Refer to Guy Kawasaki's famous blog post on the 10/20/30 rule for a good intro: [http://blog.guykawasaki.com/2005/12/the_102030_rule.html#axz...](http://blog.guykawasaki.com/2005/12/the_102030_rule.html#axzz0SqMyZEG9)

I also prefer to have about 20 backup slides at the end that address most common questions. Usually these will be deeper drill downs into market sizing, competitors, financials, short/medium/long term plans.

The successful entrepreneurs who I've worked with are almost fanatical about the investor deck. They obsess over every word on every slide. It's both incredibly inspiring and utterly painful.</comment></comment>


Share your thoughts, not your life - <AUTHOR> https://www.indiegogo.com/projects/xahive-expansion-project

<comment><AUTHOR> Share your thoughts, not your life.

Is your privacy secure in this constantly evolving global information economy?

Between companies, hackers, popular social media and unsecured communication tools such as text messaging, "free" email providers, faxing and file sharing - your personal data is unfortunately not just your own. Your intended private conversations are in fact, not private, rather they are subject to data mining and potential privacy breaches.

XAHIVE is a tool that helps address these gaps found within contemporary communication tools such as texting, email, file sharing and faxing. XAHIVE is on a mission to protect your privacy and ensure your information remains secure while sending messages – meaning your messages will actually remain private due to our encryption options. Our free social media platform uses Advanced Encryption Standard (AES), which is an encryption algorithm for securing sensitive material, and is currently touted as the most convenient and secure form of encryption available. In addition, XAHIVE uses two forms of messaging: private messaging – which is secure messaging between yourself and other users, and public messaging, where you can "buzz" to any other XAHIVE users that are situated near you, all while remaining anonymous.

Our service facilitates communication between people, small businesses, professionals and not-for-profit organizations. XAHIVE allows you to connect with friends and network with new clients/businesses, while ensuring your sensitive information remains private and secure all while complying with Canadian and US privacy laws. What sets XAHIVE apart from other communication technologies is the fact that not even XAHIVE has access to user's messages – meaning we won't and cannot do any data mining and won't and cannot share your personal information with invasive third parties. XAHIVE has designed our apps in order to ensure that unscrupulous third parties are hindered from trying to breach your account(s).

We believe that you should be free to communicate with other people using text, voice or video - without having to pay the price of your privacy and security, hence we believe that you should share your thoughts, not your life.Share your thoughts, not your life.

Is your privacy secure in this constantly evolving global information economy?

Between companies, hackers, popular social media and unsecured communication tools such as text messaging, "free" email providers, faxing and file sharing - your personal data is unfortunately not just your own. Your intended private conversations are in fact, not private, rather they are subject to data mining and potential privacy breaches.

XAHIVE is a tool that helps address these gaps found within contemporary communication tools such as texting, email, file sharing and faxing. XAHIVE is on a mission to protect your privacy and ensure your information remains secure while sending messages – meaning your messages will actually remain private due to our encryption options. Our free social media platform uses Advanced Encryption Standard (AES), which is an encryption algorithm for securing sensitive material, and is currently touted as the most convenient and secure form of encryption available. In addition, XAHIVE uses two forms of messaging: private messaging – which is secure messaging between yourself and other users, and public messaging, where you can "buzz" to any other XAHIVE users that are situated near you, all while remaining anonymous.

Our service facilitates communication between people, small businesses, professionals and not-for-profit organizations. XAHIVE allows you to connect with friends and network with new clients/businesses, while ensuring your sensitive information remains private and secure all while complying with Canadian and US privacy laws. What sets XAHIVE apart from other communication technologies is the fact that not even XAHIVE has access to user's messages – meaning we won't and cannot do any data mining and won't and cannot share your personal information with invasive third parties. XAHIVE has designed our apps in order to ensure that unscrupulous third parties are hindered from trying to breach your account(s).

We believe that you should be free to communicate with other people using text, voice or video - without having to pay the price of your privacy and security, hence we believe that you should share your thoughts, not your life.</comment>


Barobo launches the Mobot – a low cost modular robot - <AUTHOR> http://www.barobo.com/

<comment><AUTHOR> My first take on this, more to come.

[http://flexibilityenvelope.com/barobo-launches-the-mobot- a-l...](http://flexibilityenvelope.com/barobo-launches-the-mobot-a-low-cost- modular-robot)</comment>


"What's new with Firefox": Release notes or an ad popup for Firefox mobile? - <AUTHOR> https://www.mozilla.org/en-GB/firefox/80.0/whatsnew/all/?oldversion=79.0 <comment><USER2> I believe this is a better link: [https://www.mozilla.org/en- US/firefox/80.0/releasenotes/](https://www.mozilla.org/en- US/firefox/80.0/releasenotes/)

<comment><USER3> There's also this link with more details about the migration from the old Firefox to the new one: [https://support.mozilla.org/en-US/kb/firefox-android- upgrade...](https://support.mozilla.org/en-US/kb/firefox-android-upgrade-faqs)

</comment></comment><comment><USER4> It seems to have broken, substantially - it keeps closing (and asking to send a report to Mozilla) pages that worked fine the other day. Hopefully this isn't a new trend.</comment>


Glibc change exposing bugs (or how memcpy can bite you back) - <AUTHOR> http://lwn.net/Articles/414467/

<comment><USER2> The response of the developer (Andreas Schwab) is one of the most unprofessional I've seen in a very long time, but then I have a particular loathing for bugs that manifest as subtle runtime peculiarities. Technically he's correct in that it's not a bug if you were to follow the letter of the (API) law but to break existing functionality at such a fundamental layer and then say "hey, not my problem!" blows me away.

I'm not quite sure how I would handle it in a project as large as glibc but if this was my project I'd probably provide a fast_nonoverlapping_memcpy that did all the neat platform specific tricks for speed in addition to the stupid simple naive memcpy and then let the developer select which one he wanted to use. That or add a trick to memcpy that checks to see if they do overlap and delegate to memmove if they do.

<comment><AUTHOR> Such a runtime check would be too costy, I believe.

I do agree, though, that it's not glibc's problem, however, I also agree with Linus (and other commenters) who believe that knowingly shipping a Fedora release with broken flash is stupid, especially when the problem and the fix is already known, and is fairly trivial.

(Those programs that desperately need the faster memcpy, if it is faster to begin with, can LD_PRELOAD it)</comment></comment>