Ask HN: Is your company sticking to on-premise servers? Why? - <AUTHOR> I've been managing servers for quite some time now. At home, on prem, in the cloud...
The more I learn, the more I believe cloud is the only competitive solution today, even for sensitive industries like banking or medical.
I honestly fail to see any good reason not to use the cloud anymore, at least for business. Cost-wise, security-wise, whatever-wise.
What's a good reason to stick to on-prem today for new projects? To be clear, this is not some troll question. I'm curious: am I missing something? <comment><USER7> Like many others have pointed out: Cost.

I'm the CTO of a moderately sized gaming community, Hypixel Minecraft, who operates about 700 rented dedicated machines to service 70k-100k concurrent players. We push about 4PB/mo in egress bandwidth, something along the lines of 32gbps 95th-percentile. The big cloud providers have repeatedly quoted us _an order of magnitude_ more than our entire fleet's cost....JUST in bandwidth costs. Even if we bring our own ISPs and cross-connect to just use cloud's compute capacity, they still charge stupid high costs to egress to our carriers.

Even if bandwidth were completely free, at any timescale above 1-2 years purchasing your own hardware, LTO-ing, or even just renting will be cheaper.

Cloud is great if your workload is variable and erratic and you're unable to reasonably commit to year+ terms, or if your team is so small that you don't have the resources to manage infrastructure yourself, but at a team size of >10 your sysadmins running on bare metal will pay their own salaries in cloud savings.

<comment><USER8> A few years ago I was trying to start a company and get it off the ground. We had to make decisions on our tech stack and whether we were going to use AWS and build around their infra. Our business was very data heavy and required transferring large datasets from outside to our databases. Even in our early prototypes, we realized that we couldn't scale cost-effectively on AWS. I figured out that we could colocate and rent racks, install HW, hire people to maintain, etc... for way less than we could use the cloud for. I was shocked at the difference. I remember saying to my cofounder why does anyone use AWS, you can do this on your own way cheaper.

Later I worked at a FAANG and remember when Snap filed their S1 when they were going public they disclosed that they were paying Google $5B and we were totally shocked at the cost compared to our own spend on significantly larger infra.

I think people don't realize this is doable and it's great to hear stories like yours showing the possibilities.

<comment><USER9> Dropbox did the same thing a few years back - moved everything from Amazon S3 to their own storage.

My guess is they did it for cost reasons.

<comment><USER10> That S3 is eventually consistent with object updates (HTTP PUT) might also screw up things for a company whose core value is synchronized storage.

<comment><USER11> I don't mean to sound daft, just clarifying my own understanding, but isn't Dropbox eventually consistent (as a system)?

<comment><USER10> Oh, sure, but when they think they have written something to S3 and got a successful HTTP response back from the API, perhaps they want to be able to tell clients to go fetch the new data from the bucket. But those clients may not get the new data then, due to eventual consistency.

<comment><USER12> S3 is immediately consistent for new objects unless the service received a GET on the object before it was created. It's easy to use this to make an immediately consistent system.

<comment><USER13> S3 ListObjects calls are eventually consistent (i.e. list-after-put). EMRFS [1] and S3Guard [2] mitigate this for data processing use cases.

[1] - [https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-f...](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr- fs.html) [2] - [https://blog.cloudera.com/introducing-s3guard-s3-consistency...](https://blog.cloudera.com/introducing-s3guard-s3-consistency- for-apache-hadoop/)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> I work in Livermore Computing at LLNL.

We manage upwards of 30 different compute clusters (many listed here: [https://hpc.llnl.gov/hardware/platforms](https://hpc.llnl.gov/hardware/platforms)). You can read about the machine slated to hit the floor in 2022/2023 here: [https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el- capita...](https://www.llnl.gov/news/llnl-and-hpe-partner-amd-el-capitan- projected-worlds-fastest-supercomputer).

All the machines are highly utilized, and they have fast Infiniband/OmniPath networks that you simply cannot get in the cloud. For our workloads on "commodity" x86_64/no-GPU clusters, we pay 1/3 or less the cost of what you'd pay for equivalent cloud nodes, and for the really high end systems like Sierra, with NVIDIA GPUs and Power9's, we pay far less than that over the life of the machine.

The way machines are procured here is different from what smaller shops might be used to. For example, the El Capitan machine mentioned above was procured via the CORAL-2 collaboration with 2 other national labs (ANL and ORNL). We write a 100+ page statement of work describing what the machine must do, and we release a set of benchmarks characterizing our workload. Vendors submit proposals for how they could meet our requirements, along with performance numbers and test results for the benchmarks. Then we pick the best proposal. We do something similar with LANL and SNL for the so-called commodity clusters (see [https://hpc.llnl.gov/cts-2-rfi](https://hpc.llnl.gov/cts-2-rfi) for the latest one). As part of these processes, we learn a lot about what vendors are planning to offer 5 years out, so we're not picking off the shelf stuff -- we're getting large volumes of the latest hardware.

In addition to the cost savings from running on-prem, it's our job to stay on the bleeding edge, and I'm not sure how we would do that without working with vendors through these procurements and running our own systems.

<comment><USER15> > All the machines are highly utilized, and they have fast Infiniband/OmniPath > networks that you simply cannot get in the cloud.

It's weird that these networking technologies are not used more in "plain" datacentre settings, since networking latency and throughput has to be a significant challenge to scaling up non-trivial workloads and achieving true datacentre-scale computing. We hear a lot about how to "scale out", but that's only really feasible for relatively simple workloads where you just seek to do away with the whole issue of keeping different nodes in sync on a real-time basis, and accept the resulting compromises. In many cases, that's just not going to be enough.

<comment><USER16> There are a lot of people from the National Lab super computer world who end up in High Frequency Trading for just the reason you describe.

Specifically, how do you optimize a large cluster of computers to operate at the lowest possible latency. For the National Labs, those computers could be in the lab or with other labs around the world. For the HFT folks, the machines could be in an exchange or spread across multiple exchanges around the world.

Source: I used to be head of Global Latency Monitoring for a HFT.

<comment><USER17> I'm curious why you moved to LLNL from HFT?

<comment><USER18> Money is a safe guess. Research pay scales aren't even close to private sector, especially not finance.

<comment><USER19> It sounds like the opposite direction happened here.

</comment></comment></comment></comment></comment></comment><comment><USER20> We are a 1000-2000 person company and we have probably on the order of $100M of servers and data centers and whatnot, and I think we spend about 2/3rds of that every year on power/maintenance/rent/upgrades/etc.

We don't generally trust cloud providers to meet our requirements for:

* uptime (network and machine - both because we are good at reliability [and we're willing to spend extra on it] and because we have lots of fancy redundant infrastructure that we can't rely on from cloud companies)

* latency (this is a big one)

* security, to some degree

* if something crazy is happening, that's when we need hardware, and that's when hardware is hard to get. Consider how Azure was running out of space during the last few months. It would have cost us an insane amount of money if we couldn't grow our data centers during Corona! We probably have at least 20-30% free hot capacity in our datacenters, so we can grow quickly.

We also have a number of machines with specs that would be hard to get e.g. on AWS.

We have some machines on external cloud services, but probably less than 1% of our deployed boxes.

We move a _lot_ of bandwidth internally (tens of terabytes a day at least, hundreds some days), and I'm not sure we could do that cheaply on AWS (maybe you could).

We do use <insert big cloud provider> for backup, but that's the only thing we've thought it was economical to really use them for.

<comment><USER21> Hundreds of terabytes a day is really not that much, depends on what latency can you accept. I often run computations over datasets that are petabytes in size, just for my own needs. A big data move would be at least tens of petabytes or more like hundreds, or thousands.

Also surprised about latency, latency from what to what? Big cloud providers have excellent globally spanning networks. Long distance networking is crazy expensive, though, compared to the peanuts it costs to transfer data within a data center.

Reliability - again, not sure I buy it. Reliability is "solved" at low levels (such as data storage), most failures occur directly at service levels, regardless of whether you have the service in house or in the cloud.

The rest of your points make sense.

<comment><USER20> > Hundreds of terabytes a day is really not that much

How much would it cost to move this across boxes in EC2? I actually don't know, that's not a rhetorical question. A lot of our servers have 10-40gbit links that we saturate for minutes/hours at a time, which I suspect would be expensive without the kind of topology optimization we do in our datacenters.

> Also surprised about latency

We've spent a surprising amount of money reducing latency :) We're not a high frequency trading firm or anything, but an extra 1ms (say) between datacenters is generally bad for us and measurably reduces performance of some systems.

> Reliability is "solved" at low levels

To whatever extent this may be true, it's certainly not true for cloud providers. One obvious example is that EC2 has "scheduled maintenance events" where they force you to reboot your box. This would cost us a lot of money (mostly in dev time, to work around it).

Also, multi-second network dropouts in big cloud datacenters are not uncommon (in my limited experience), but that would be really bad for us. We have millisecond-scale failover with 2x or 3x redundancy on important systems.

<comment><USER22> > How much would it cost to move this across boxes in EC2?

Nothing. You generally only pay for data going out of cloud providers. Not data going in or data being transferred within the same region.

> One obvious example is that EC2 has "scheduled maintenance events" where > they force you to reboot your box. This would cost us a lot of money (mostly > in dev time, to work around it).

You're not going to have a successful cloud experience unless you build your applications in a cloud suitable way. This means not all legacy applications are a good fit for the public cloud. Most companies really embracing the cloud are mitigating those risks by distributing workloads across multiple instances so you don't care if any one needs to be restarted, especially within a planned window.

> Also, multi-second network dropouts in big cloud datacenters are not > uncommon (in my limited experience), but that would be really bad for us. We > have millisecond-scale failover with 2x or 3x redundancy on important > systems.

Are these inter-region network dropouts or between the internet and the cloud data center? You're not going to be relying on a public internet connection to the cloud for critical workloads.

All that being said, there are plenty of workloads which I don't think fit well in the cloud operating model. You may very well have one of them.

<comment><USER23> You pay for cross-AZ Traffic in AWS, and that adds up really fast.

<comment><USER24> Yep. Got bitten HARD by this recently, $1.5k inter-az transfer charges that we never saw coming.

Our fault, I suppose -- but multi-az is prohibitively expensive if you need to run anything data heavy distributed.

<comment><USER25> I'm working on reducing a $50K per month bill for Inter-AZ traffic at the moment.

> but multi-az is prohibitively expensive if you need to run anything data > heavy distributed.

If you communicate between your AZs via ALBs, multi-az is effectively free. Our bill is so high because within our Kubernetes cluster, our mesh isn't locality aware; it randomly routes to any available pod. 2/3rds of our traffic crosses AZs.

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> I'm slowly coming to the complete opposite opinion you seem to have.

I've worked almost entirely for companies that run services in various cloud infrastructures - Azure/Heroku/Aws/GCP/Other.

I recently started a tiny 1 man dev shop in my spare time. Given my experience with cloud services it seemed like a no brainer to throw something up in the cloud and run with it.

Except after a few months I realized I'm in an industry that's not going to see drastic and unplanned demand (I'm not selling ads, and I don't need to drive eyeballs to my site to generate revenue).

So while in theory the scaling aspect of the cloud sounds nice, the reality was simple - I was overpaying for EVERYTHING.

I reduced costs by nearly 90% by throwing several of my old personal machines at the problem and hosting things myself.

So long story short - Cost. I'm happy to exchange some scaling and some uptime in favor of cutting costs. Backups are still offsite, so if my place burns I'm just out on uptime. The product supports offline, so while no one is thrilled if I lose power, my customers can still use the product.

Basically - cost, Cost, COST. I have sunk costs in old hardware, it's dumb to rent an asset I already own.

There might well be a point when I scale into a point where the cloud makes sense. That day is not today.

<comment><USER27> What's the time trade-off?

I've been drawing out my plans lately for a hobby project, all 100% on AWS.

Being able to spin up my entire infrastructure with Terraform, build out images with Packer, setup rules for off-site backups, ensure everything is secure to the level I want it, etc. - It takes me next to no time at all.

I can't imagine buying hardware, ensuring my home is setup with proper Internet, configuring everything here, and then still needing off-site backups anyway.

Now, keep in mind - I'm definitely coming in from a Millennial point of view. My entire career was built on cloud. I've never touched hardware apart from building a computer back when I was 15 or something. I understand virtual.

But being able to build up and tear down an entire setup, having it completely self-restore in minutes. Can't beat that.

Napkin math has me at ~$50/mo: Full VPC, private/public isolated subnets, secure NACLs and security groups, infinitely extendable block storage and flat-file storage, near-instant backups with syncing to a different continent, 5 servers, DNS configurations, etc.

All depends what you're doing too - of course. But for me, just the trade-off of working with what I know and not needing to leave my cafe of choice, still not breaking the bank - and if I do, having instant tear down and restore. Bam.

<comment><USER28> What kind of setup did you have for 5 servers at $50/mo on AWS? Interested to know - our EC2 instances that are about 1/4 as powerful as a laptop cost $60+/mo

<comment><USER27> Certainly nothing powerful :-) I can get away with t3.nano and t3.micro for what I'm doing at the moment. But the beauty of cloud, is that I can scale up when I eventually need it.

5x t3.nano will be ~$25/mo 5x t3.micro will be ~$50/mo

All of my AMIs are EBS optimized and require a minimum of 8GB for the root drive (Although they only use ~1.6GB. Not bothering to hack around this to save a buck.) So that'll be 40GB EBS block storage. Plus I want ~20GB spread across 3 of the machines.

So EBS should be ~$6/mo.

I only need the volumes of those last 3, the others are good to go with their base AMI or user-data init script. So I only need snapshot backups of ~20GB. Being priced incrementally and having minimal changes, I'll only be charged ~$1/mo for that + off-site another $1/mo

So, currently experimenting with the t3.nano - Cost is ~$36/mo. One of these servers will be used as a personal VPN, and I expect ~75GB/mo coming from my laptop. So bandwidth charges at $9/mo.

Total $45/mo - For what I have planned now, at least.

<comment><USER29> That's exactly the reason I gave up on AWS, I need an accountant to do the math every month :D

Now I rent a 4GB Linux box for 5$/m with no Dockers or whatsoever and happy that it just works

<comment><USER30> I also hate this complexity with a passion. I love cloud, but pricing can be a real nightmare.

I don't use AWS specifically but when I needed to know the price of some cloud service or group of services I spin up the service (or services) in a brand new project and let it run for 24 hours under similar working environment to see the impact, then after checking the results (the breakdown of each service's price in that day) I just close the project entirely, no left overs.

So I tend to successfully avoid these strange, terribly organized, cloud- specific, service-specific calculator where I can easily forget one aspect of the service that might cost a lot of money absolutely randomly.

Obviously it is a bad strategy if things are expected to reach $200/month and/or you do 'price evaluation' frequently, but otherwise it is stupid easy. I barely spent $50 each year doing this (small company and sporadic system changes)

But the best part is that the final daily price of your system is as precise as it can possibly be and that is worth something.

</comment></comment></comment></comment></comment></comment><comment><USER31> University research group here.

Simply, _cost_

Our compute servers crunch numbers and data at > 80% util.

Our servers are optimized for the work we have.

They run 24/7 picking jobs from queue. Cloud burst is often irrelevant here.

They deal with Terabytes or even Petabytes of moving data. I'd cry paying for bandwidth costs if charged €/GB.

Sysadmin(yours truly) would be needed even if it were to be run in the cloud.

We run our machines beyond 4 years if they are still good at purpose.

We control the infra and data. So, a little more peace and self-reliance.

No surprise bills because some bot pounded on a S3 dataset.

Our heavy users are connected to the machines at a single hop :-) No need to go across WAN for work.

<comment><USER32> In germany it's a pretty common think for universities to have some servers for themself.

1\. Their use case is kinda different. The servers mostly run heavy CS research related stuff. E.g. they might have heavy CPU load and heavy traffic between they servers but they have less often heavy traffic to the "normal internet" (if they have heavy traffic to the outside it's normally to other research institutes which not seldom have dedicated wire connections).

2\. They might run target specific optimized CPU or GPU heavy compute tasks going on for weeks at a time. This is really expansive in the cloud which is mostly focused in thinks like web services.

3\. When they don't run such tasks in the research groups they want to allow their juniors to run their research tasks "for free". Which wouldn't work with a payment model as done in the cloud.

4\. They don't want to relay on some external company.

Also I'm not sure are there even (affordable) cloud systems with compatible spec? (like with 4+TB of _RAM_ , I'm not kidding this is a requirement for some kind of tasks or they will take way to long and requires additional complexity by using special data structures which support partial offline data _in the right way_ , which can be very costly in dev time)??

<comment><USER33> It's not just CS. The computational chemistry and materials science crystallography folks can have jobs that run for days or weeks too.

<comment><USER34> I'm at a center for computational biology - our genomics guys have been known to use 90% of our university's HPC capacity ;-) My own work (ecological modelling) is not as heavy, but when I run a full experiment, that takes a 32 core machine about two weeks to complete.

</comment></comment></comment></comment><comment><USER35> Meta-comment:

cost: [https://news.ycombinator.com/item?id=23098576](https://news.ycombinator.com/item?id=23098576)

cost: [https://news.ycombinator.com/item?id=23097812](https://news.ycombinator.com/item?id=23097812)

cost: [https://news.ycombinator.com/item?id=23098658](https://news.ycombinator.com/item?id=23098658)

abilities / guarantees: [https://news.ycombinator.com/item?id=23097213](https://news.ycombinator.com/item?id=23097213)

cost: [https://news.ycombinator.com/item?id=23090325](https://news.ycombinator.com/item?id=23090325)

cost: [https://news.ycombinator.com/item?id=23097737](https://news.ycombinator.com/item?id=23097737)

threat model: [https://news.ycombinator.com/item?id=23098612](https://news.ycombinator.com/item?id=23098612)

cost: [https://news.ycombinator.com/item?id=23097896](https://news.ycombinator.com/item?id=23097896)

cost: [https://news.ycombinator.com/item?id=23098297](https://news.ycombinator.com/item?id=23098297)

cost: [https://news.ycombinator.com/item?id=23097215](https://news.ycombinator.com/item?id=23097215)

That's just the in-order top comments I'm seeing right now. (please do read and upvote them / others too, they're widely varying in their details and are interesting)

The answer's the same as it has always been. Cloud is more expensive, unless you're small enough to not pay for a sysadmin, or need to swing between truly extreme scale differences. And a few exceptions for other reasons.

<comment><USER36> There is also another answer... I work for a CDN, so we can't really use the cloud when in many ways we ARE the cloud.

Although we do often make jokes about "what if we just move the CDN to AWS?"

<comment><USER37> It is a pity East Dakota won't make jokes like these. Can you imagine cloudflare running on aws? What happens when someone tries to denial of service them while on aws?

On a different note, Netflix still runs "on the cloud", right? I mean what does it really mean? Dropbox can still have most of its stuff on aws and do the expensive part on premises if cost is a concern?

The truly bizarre stuff happens at hybrid cloud.

<comment><USER38> Netflix runs it's own bandwidth/cdn. Sometimes it actually has a pop/box INSIDE your ISP [https://openconnect.netflix.com/](https://openconnect.netflix.com/).

<comment><USER37> My understanding is the website, the user services such as authentication, heart beat (not sure what is the proper technical term but the thing that says where I am in a particular episode). That and internal apps like project tracker not to mention dev/test.

At least in my imagination. At my work, I'm not even worth throwing an SSD at my work computer. My manager is powerless to help as the company has some kind of deal to only buy from HP? No idea what kind of glue procurement is sniffing at this company...

</comment></comment></comment></comment></comment><comment><USER39> We have around 20 servers in a colo center down the street.

At this number of servers we can still host websites that have millions of users (but not tens of millions). They are not exotic servers either. In fact by now they are, on average, around 11 years old. And costed anywhere from 2k to 8k at the time of purchase. Some are as old as 19 years. Hell, when we bought some of them - with 32GB of memory each - AWS had no concept of "high memory" instances and you had to completely pay out your ass for a 32GB server, despite ram being fairly cheap at the time.

We have no dedicated hardware person. Between myself and the CTO, we average maybe a day per month thinking about or managing the hardware. If we need something special setup that we have no experience in, we have a person we know that we contract, and he walks us through how and why he set it up as he did. We've used him twice in the last 13 years.

The last time one of us had to visit the colocation center was months ago. The last time one of us had to go there in an emergency was years ago. It's a 5 minute drive from each of our homes.

So, why exactly should we use the cloud? We have servers we already paid for. We rent 3 cabinets - I don't recall the exact cost, but I think its around $1k per month. We spend practically no time managing them. In our time being hosted in a colo center - the past 19 years - we've had a total of 3 outages that were the fault of our colo center. They all lasted on the order of minutes.

<comment><USER40> I think people who have no experience managing servers dramatically overestimate how much time it takes to manage servers. Depending on your team, it can definitely be easier to manage your own hardware than to manage your cloud infrastructure.

<comment><USER41> In my experience, it's not the time required but that a lot of development teams don't have a sysadmin or ops skillset.

<comment><USER42> I live in a software engineering world professionally but my background is in traditional "neckbeard" Linux system administration. This ends up making me "DevOps" but honestly a lot of what I've ended up doing in my career is basic sysadmin for organizations that get a remarkably long ways before realizing they need it - things like telephony and video surveillance become really unreasonably expensive when you end up relying on a cloud service because you don't have the skillset to manage them in-house.

This is purely my opinion, but I think that 1) there is a strange shortage of IT professionals (people who are _not_ software engineers but instead understand _systems_ ) in much of the industry today, and 2) a lot of tech companies, even those that are currently well functioning, might be able to save a lot of money if they hired someone with a conventional IT background. This is a little self-serving of course, but it really does astound me when I see the bills that some companies are paying cloud services to do something that is traditionally done in-house by an IT department. And not everything can readily be outsourced to some "aaS" provider, so on top of that you end up with things like software companies with multi-million budgets running an office network that consists of a consumer WiFi router someone picked up at Fry's - not realizing that they are losing a lot of time to dealing with how poorly that ends up working.

I think part of the problem rests in academia - at least in my area a lot of universities seem to have really backed off on IT programs in favor of CS. I went through an undergraduate program that involved project management, decision analysis, and finance courses because these were considered by the college (I would say accurately) critical skills for the IT field. But that program had an incredible two students and was widely considered inferior to the CS program with hundreds.

Another part of the problem though seems to rest in industry. The salary differential between "DevOps Engineer" and "IT Analyst" is incredible when in practice they end up doing mostly the same thing in a lot of small orgs. So I end up walking sort of an odd line of "I have a long background in IaC but I also know about conference room equipment." And I'm not saying that everything with a Cisco/Tandberg badge isn't overpriced, but Zoom rooms can end up costing just as much and seem to be less reliable - not surprising for a platform which, by practical necessity of the lack of IT support in many orgs, is built on the Silicon Valley time-tested architecture of "five apple consumer products taped together."

<comment><USER43> From my experience, large enterprises sabotage the effectiveness of internal IT with bureaucracy and politics in a misguided attempt to eliminate all possibility of mistakes being made.

It's usually done with the "let's pretend it is ITIL" process.

Let me give two examples where if I had been the client then I would absolutely have sprinted for the cloud if I could, or at the very least start talking it up as much better.

1) System outage, time to fix 5 hours and 3 minutes. The 5 hours was me sitting in front of my computer with screens open showing the problem and waiting for various managers/decision-makers to fly by and take a look as they were ping-ponging around the office panicking about what would be impacted by the fix. Everything that was going to get impacted was already impacted by the system not working, and I had to explain that to them multiple times. Towards the end of the day, I eventually got the go-ahead to do the 3 minutes of work to fix the system. This system being down had prevented another team from doing any work for the entire afternoon.

2) Two full days of politics and paperwork to get approval to do 30 minutes of work, all while the client was impatiently asking "is it done yet" every few hours.

</comment></comment></comment></comment></comment><comment><USER44> Yes. Why? Cost, availability, flexibility, bandwidth. For a lot of companies, on-prem servers are the best solution for efficiency and cost.

One great example. We were paying $45k/yr for a hosted MS Dynamics GP solution. For $26k we brought it in house with only a $4k/yr maintenance fee. We bought a rackmount Dell, put on VMWare, have an app VM and a DB VM. My team can handle basic maintenance. In the past 11 months we haven't had to touch that server once. We have an automated backup, pulls VMs out daily and sends them off to Backblaze. Even if we need to call our GP partner for some specialized problem, it's not $45k/yr in consulting costs.

We had a bunch of Azure servers for Active Directory and a few other things. When I came in 2 years ago I set up new on-prem DC VMs, and killed out absurd Azure monthly bill, we were saving money by month three. A meteor could take out Atlanta and the DCs are our satellite offices would handle the load just fine until we restored from backups and we'd STILL save money. We've had MORE uptime and reliability since then too.

If I have a server go down, we have staff to get on it immediately, no toll free number to dial, no web chat to a level 1 person in India, etc.

Our EMR is hosted, because that's big enough that I want to pay someone to be in control over it, and someone to blame. However, there have been many times where I'm frustrated with how they handle problems, and jumping from one EMR to another is not easy. And in the end they're all bad anyway. Sometimes I DO wish we were self hosted.

The Cloud is just someone else's computer. If they're running those machines more cheaply than you are, they're cutting out some cost. The question is, do you need what they're cutting?

<comment><USER45> > The Cloud is just someone else's computer. If they're running those machines > more cheaply than you are, they're cutting out some cost. The question is, > do you need what they're cutting?

They're cutting overhead and getting better deals on hardware than you could ever get.

Their efficiency is their profit margin.

<comment><USER46> > Their efficiency is their profit margin.

Last time I checked, AWS had a profit margin in the 40%-50% ballpark.

Sorry, but the semiconductor industry doesn't operate with any kind of markup that would allow such profit margins from "getting better deals on hardware". The only one able to make that kind of profit used to be Intel on high-end server CPUs, and even they are now pressured by AMD and custom ARM silicon options. Anything else needed for a server, RAM or flash chips or whatever, is usually selling on thin single-digit margins.

Cloud provider profit margin is perfectly logical and explainable through lock-in effects keeping their customers paying big markups to stay in AWS infrastructure. Be it software that was built against AWS proprietary services, be it having the necessary engineering skills to manage AWS infrastructure in the team but lacking the skills to manage on-prem hardware, be it the enterprise sales teams of cloud operators schmoozing CTOs of big corporations and making them jump on a "going into cloud" strategy as some kind of magic bullet to future-proof their corporations' IT, be it the psychological effect that makes "using the cloud" apparently a mandatory thing to be "cool" in todays' silicon valley culture, and therefore by extension the whole worlds' IT engineering culture.

The most ironical of them all is this weird effect that drives people to rationalize these things, writing comments like yours, because nobody likes to admit they've painted themselves into a corner of lock-in effects. And of course there's the irony of this all being history repeating itself: anyone still remembering when IBM dominated the IT industry?

<comment><USER47> > that would allow such profit margins

The percentages don't _quite_ hit that amount of discount, but they are much much higher than (I at least) expected.

</comment></comment></comment></comment><comment><USER48> Mostly, headspace. If I run my own server, I just need to apply my existing Ubuntu sysadmin knowledge. If I use AWS, I have to learn a whole load of AWS- specific domain knowledge, starting with their utterly baffling product names. My time is more valuable than that.

Also, sheer cost. Literally everyone I know in my particular part of the industry uses Hetzner boxes. For what I do, it's orders of magnitude cheaper than AWS.

<comment><USER49> That's how you get old, when your time is more valuable than a massive shift in technology.

<comment><USER50> Nah, we already did mainframes in the 1970s. Renting CPU time only makes sense if you don't need CPU time or you like wasting money.

</comment></comment></comment><comment><USER51> Try running any service with an average egress exceeding 10 Mbit/s then tell me cloud still makes sense. By the time you reach 1 Gbit/s the very idea of it is enough to elicit a primitive biological defensive response.

We don't do on-prem but we do make heavy use of colo. The thought of cloud growth and DC space consolidation some day pushing out traditional flat rate providers absolutely terrifies me.

At some point those cloud premiums will trickle down through the supply chain, and eventually it could become hard to find reasonably priced colo space because the big guys with huge cash-flush pockets are buying up any available space with a significant premium attached. I don't know if this is ever likely, but growth of cloud could conceivably put pressure on available physical colo space.

Similar deal with Internet peering. There may be a critical point after which cloud vendors, through their sheer size will be able to change how these agreements are structured for everyone.

<comment><USER45> Netflix runs on the cloud and does 30% of all internet traffic.

That being said, 99% of that traffic is served from servers in colos now, but 10 years ago it was all served from CDN providers like Akamai, which is just a specialized cloud.

<comment><USER52> This is kind of a big caveat ("Netflix is in the cloud but almost none of the work is done there"), and something I have to mention to non tech decision makers when they say "but Netflix!". I even have a slide for presentations just for this ("You Are Not Netflix").

<comment><USER45> 99% of the work is done on the cloud. What comes off of those colo servers is literally just bits streaming from disk to network. There is no transformation or anything. No authentication, no user accounts, no database. Nothing.

Just static files served efficiently.

<comment><USER53> > What comes off of those colo servers is literally just bits streaming from > disk to network

So... the core of their business?

<comment><USER45> Not at all. It was so "not core" that it was outsourced.

The core of their business is recommendations, encoding, and authentication. All of those are done 100% on the cloud.

<comment><USER54> Sure but probably 99% of the

> 30% of all internet traffic.

is outside of the cloud.

<comment><USER55> Anyone know the bill Netflix has for running on the cloud ?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER56> Why stay on premise?

Cost. On-prem is roughly on-par in an average case, in my experience, but we've got many cases where we've optimized against hardware configurations that are significantly cheaper to create on-prem. And sunk costs are real. It's much easier to get approval for instances that don't add to the bottom line. But for that matter, we try to get our on-prem at close to 100% utilization, which keeps costs well below cloud. If I've got bursty loads, those can go to the cloud.

Lock-in. I don't trust any of the big cloud providers not to jack my rates up. I don't trust my engineers not to make use of proprietary APIs that get me stuck there.

Related to cost, but also its own issue, data transfer. Both latency and throughput. Yeah, it's buzzwordy, but the edge is a thing. I have many clients where getting processing in the same location where the data is being generated saves ungodly amounts of money in bandwidth, or where it wouldn't even been feasible to transfer the data off-site. Financial sector clients also tend to appreciate shaving off milliseconds.

Also, regulatory compliance. And, let's be honest, corporate and actual politics.

Inertia.

Trust.

Risk.

Interoperability with existing systems.

Few decisions about where to stick your compute and storage are trivial; few times is one answer always right. But there are many, many factors to consider, and they may not be the obvious ones that make the decision for you.

</comment><comment><USER57> Cost and Latency.

My team and I run the servers for a number of very big videogames. For a high- cpu workload, if you look around at static on-prem hosting and actually do some real performance bencharking, you will find that cloud machines - <USER2> convenient - generally cost at least 2x as much per unit performance. Not only that, but cloud will absolutely gouge you on egress bandwidth - leading to a cost multiplier that's closer to 4x, depending on the balance between compute and outbound bandwidth.

That's not to say we don't use the cloud - in fact we use it extensively.

Since you have to pay for static capacity 24/7 - even when your regional players are asleep and the machines are idle, there are some gains to be had by using the right blend of static/elastic - don't plan to cover peaks with 100% static - and spin up the elastic machines when your static capacity is fully consumed. This holds true for anything that results in more usage - <USER3> busy weekend, an in-game event, a new piece of downloadable content, etc... It's also a great way to deal with not knowing exactly how many players are going to show up on day 1.

Regarding latency, we have machines in many smaller datacenters around the world. We can generally get players far closer to one of our machines than to AWS/GCP/Azure, resulting in better in-game ping, which is super important to us. This will change over time as more and more cloud DCs spring up, but for now we're pretty happy with the blend.

</comment><comment><USER58> AI compute is so much cheaper on-prem that it's not even in question.

And there are clients that demand it.

And researchers, in general, like to do totally wacky things, and it's often easier/cheaper to let us if you have physical access.

<comment><USER59> +1 on this. Get a nice server with some GPUs and you'd save a lot more than paying the super expensive costs on cloud.

</comment></comment><comment><USER60> I'm in rural iowa and you really can't bank on a solid internet connection. One of my clients decided to iaas-ify all their servers and it works great except when it is windy out. They're on fixed wireless and the remote mast has some sway to it. 3-4 times a year they get struck by lightning and have a total work stoppage until all their outside gear can get replaced. Even their VDI is remote so all the thin clients just disconnect and they are done for the day.

Also, my clients aren't software development firms. They are banks and factories. They buy a software based on features and we figure out how to make it work, and most of the vendors in this space are doing on-prem non-saas products. A few do all their stuff in IAAS or colo but a lot of these places are single-rack operations and they really don't care as long as it all works.

A lot of people in small/midsize banks feel like they are being left out. They go to conferences and hear about all the cool stuff in the industry but the established players are not bringing that to them. If you can stomach the regulatory overhead, someone with drive could replace finastra/fiserv/jackhenry. Or get purchased by them and get turned into yet another forever-maintenancemode graveyard app.

</comment><comment><USER59> Founder of a growing startup:

Started with a cluster of Raspberry Pis and expanded onto an old desktop. Primarily did this for cost (raspberry pis alone were more powerful than a GCP $35/mo instance). Everything was fine until I needed GPUs/handling more traffic than those Raspberrys could handle. So I expanded by including cloud instances in my Docker Swarm cluster (tidbit: Using Traefik and WireGuard)

So half on-prem half in the cloud. Honestly just scared GCP might one day cancel my account and I'll lose all my data unless I meet their demands (has happened in the past) so that half on-prem stores most of the data.

<comment><USER61> At $35/month though GCP would only have to save you a half an hour of maintenance for it to be worth it though.

<comment><USER59> Well, given that I am using Docker it doesn't really matter much... but the bigger issue is: GCP in the past has completely blocked access from accounts when they detect random things.

Unless I meet their demands, my entire infra is gone/down for days, which I can't deal with.

</comment></comment></comment><comment><USER62> I use a $5/mo DigitalOcean VPS droplet instead of AWS or other "cloud" service. I only have to host an analytics dashboard ( [https://usertrack.net/](https://usertrack.net/) ), I don't need scaling and this way I know exactly how much I will pay. The resources are more than enough for my needs and I don't think it could be much cheaper even on the most optimized pay-per-minute of use cloud platforms.

I also have some other APIs hosted in the same way (eg. website thumbnail generation API), for the very low traffic I have and no chance of getting burst traffic I think the use case of a VPS or dedicated server is perfect.

<comment><USER61> Whenever I need to host something small and I'm trying to decide between DO and AWS I always ask myself. Would I rather be surprised by the bill or my website crashing from too much traffic? I almost always pick DO because I don't want to mess something up and lose a few hundred dollars.

<comment><USER63> Wholeheartedly agree. I think AWS is moving in the right direction with Lightsail[0], which is a service very similar to DO droplets and includes transfer. Nice if you want to use AWS for like one or two other services, but I tend to still go with DO for small things.

[0]: [https://aws.amazon.com/lightsail/](https://aws.amazon.com/lightsail/)

<comment><USER62> That sounds interesting. By "moving in the right direction" do you mean that it's still in beta or not released yet? Or that it's just the first step of many to come?

<comment><USER64> Lightsail works well now. It's a little over three years old. In the first year or so after it was released, they were notorious for being slow in most regards compared to their peers (it launched using rebranded instances from AWS, and used spinning disks, going up against SSDs their competitors were all using). They've largely caught up on performance with DigitalOcean, Linode, Vultr and similar.

That said, I've stuck with DigitalOcean even though Lightsail tests fine. I've had a great experience over the years with DO and see no reason to leave.

</comment></comment></comment></comment></comment><comment><USER65> On-prem makes your cost control proactive, rather than reactive. Nobody gets a new server without first having a purchase order approved - and the burden to get that approval falls on the person who wants the server.

In the cloud, at least the way it's generally used, cost control is reactive: You get a bill from AWS every month, and _if you 're lucky_ you'll be able to attribute the costs to different projects.

This is both a strength and a weakness: on-premise assets will end up at much higher utilisation, because people will be keen to share servers and dodge the bureaucracy and costs of adding more. But if you consider isolation a virtue, you might prefer having 100 CPUs spread across 100 SQL DBs instead of 50 CPUs across two mega-databases.

</comment><comment><USER66> Lots of great insights here, which fully accord with my experience, even in the small end of town.

About a year ago, I was in a meeting with my new CEO (who had acquired my company). My side of the business had kept hardware in-house, his was in AWS. We had broadly similar businesses in the same industry and with the same kind of customers.

My side of the business needed to upgrade our 5+ year old hardware. The quote came to $100K; the CEO freaked out. I asked him how much he spent on AWS?

The answer was that they spent $30K __per month __on AWS.

The kicker is that we managed 10x as many customers as they did, our devops team was half the size, and we were rolling out continuous deployment while they were still struggling to automated upgrades. Our deployment environment is also far less complicated than theirs because there isn't a complex infrastructure stack sitting in front of our deployment stack.

There was literally no dimension on which AWS was better than our on-prem deployment, and as far as I was able to tell before I quit, the only reason they used AWS was because everyone else was doing it.

<comment> pickle-wizard With all the job hopping that goes on in tech, there is a lot of Resume Driven Development. People want to use AWS because it will help them get their next job.

I'm finally in a job that I'm happy with and can see myself staying here until retirement. I have noticed that has changed my technology recommendations. For example we recently started looking at configuration management tools. Ansible is the obvious choice from a resume perspective as it is very popular. I ended recommending Powershell DSC. Why, because our environment is mostly windows, the team is familiar with Powershell, and for our use case is much faster. Powershell DSC is not as popular so it won't help me get another job. When it comes time to expand the team, I can hire someone who understands configuration management tools or powershell, and get them up to speed in a day or two.

</comment></comment><comment><USER67> Personally, I'd rather have capex than opex.

My observations from working with, and in the "cloud":

The "cloud" does benefit from it's scale in many ways. It has more engineers to improve, fix, watch, and page. It has more resources to handle spikes, whales, and demand. Almost everything is scale tested and the actual physical limits are known. It is damn right impressive to see what kind of traffic the cloud can handle.

Everything in the "cloud" is abstracted which increases complexity. Knowledgeable engineers are few and far between. As an engineer you assume something will break, and with every deployment you hope that you have the right metrics in place and alarms on the right metrics.

The "cloud" is best suited for whales. From special pricing to resource provisioning, they get the best. The rest is trickled down.

Most services are cost-centers. Very few can actually pay for the team and the cost of its dependencies.

It's insane how much VC money is spent building whatever the latest trend of application architecture is. Very few actually hit their utilization projections.

</comment><comment><USER68> We hear from our customers mostly what has been said here: cost and mental overhead. There is a bit of a paradox - companies that plan to grow aggressively are wary of AWS bills chopping their runway in half - they're very aware of _why_ cloud providers give out a year for free to most startups - they recoup that loss very fast once the cash faucet opens up.

What really gets me is that most cloud providers promise scalability, but offer no guard-rails - for example diagnosing performance issues in RDS - <USER4> goal for most cloud providers is to ride the line between your time cost and their service charges. Sure you can reduce RDS spend, but you'll have to spend a week to do it - so bust out the calculator or just sign the checks. No one will stop you from creating a single point of failure - but they'd happily charge for consulting fees to fix it. There is a conflict on interest - <USER5> profit from poor design.

In my opinion, the internet is missing a platform that encourages developers to build things in a reproducible way. Develop and host at home until you get your first customers, then move to a hosting provider down the line. Today, this most appeals to AI/ML startups - they're painfully aware of their idle GPUs in their gaming desktops and their insane bill from Major Cloud Provider. It also appeals to engineers who just want to host a blog or a wedding website, etc.

This is a tooling problem that I'm convinced can be solved. We need a ubiquitous, open-source, cloud-like platform that developers can use to get started on day 1, hosting from home if desired. That software platform should not have to change when the company needs increased reliability or better air conditioning for their servers. If its a Wordpress blog or a minecraft server or a petabyte SQL database - the Vendor should be a secondary choice to making things.

<comment><USER69> I've found that Kubernetes mostly solves this problem. I say mostly because for AI/ML workloads that require GPUs, we still rely on running things on bare metal locally, and deploying with GKE's magic annotations and Deep Learning images. But for anything else, I haven't had an issue going all in on k8s at the beginning, even with very small teams.

<comment><USER68> Yep! My startup is [https://kubesail.com](https://kubesail.com), so I agree :)

As for ML on Kube, I agree, there have been and still are some rough edges. The kernel drivers alone make a lot of out-of-the-box Kubernetes solutions unusable. That said, we've had a lot of success helping people move entirely onto kube - the mental gain alone from ditching the bash scripts or ansible playbooks (etC) alone is pretty freeing.

</comment></comment></comment><comment><USER70> Yes. Three major reasons:

\- Cost. It's vastly cheaper to run your own infra (like, 10-100x -- really!). The reason to run in cloud is not to save money, it's to shift from capex to opex and artificially couple client acquisition to expenditure in a way that juices your sheets for VCs.

\- Principle. You can't do business in the cloud without paying people who also work to assemble lists of citizens to hand over to fascist governments.

\- Control. Cloud providers will happily turn your systems off if asked by the government, a higher-up VP, or a sufficiently large partner.

EDIT: I should add. Cloud is great for something -- moving very fast with minimal staffing. That said, unless you get large enough to renegotiate you will get wedged into a cost deadend where your costs would be vastly reduced by going in-house, but you cannot afford to do so in the short term. Particularly for the HN audience, take care to notice who your accelerator is directing you to use for cloud services -- they are typically co-invested.

<comment><USER71> Regarding the shift in CapEx to OpEx, on-prem servers can also be leased, keeping their costs in OpEx.

</comment></comment><comment><USER72> I've been studying like a fiend to get AWS certs and thoroughly understand the cloud value proposition, especially for ephemeral workloads and compliance needs. I'm all for cloud solutions that make sense and love when serverless/usage-only systems can't be deployed. That said, I recently started work on a friend's system that he has had running in colo for a long time. It's absolutely insane how long his systems have been up. There are processes that have been alive since 2015, with some hosts having uptime linger than that. He's got a nice HA configuration but hasn't had any incidents that have triggered failover. He recently built a rack for his home with 384gb ram and gobs of cpu across 3 nodes, with rack, nice switch and UPS for just shy of $2500 ( he is quite the bargain hunter... ). I did some quick math and found a similarly equipped cluster (of just VMs, not dedicated hosts) has a 1.1 month break-even with on-demand costs, no bandwidth considered. Sure, maybe a 1 year reservation could make it a 2-3 month break even instead, but why? Those machines can easily give him 3-5 years of performance without paying another dime.

If you can feasibly run workloads onpremise or colo and have a warm failover to AWS you could probably have the best of all worlds.

<comment><USER73> If he has processes that have been up since 2015 how is he patching? That's one of my biggest gripes with on-prem, it's easy to leave something that works alone... until it gets popped by a 5 year old vuln.

In cloud I'm constantly looking at what we have because I have good billing tools in place to see what we're paying for.

</comment></comment><comment><USER74> I always find it important to separate "cloud" into 2 categories:

1\. IaaS - Which I mainly define as the raw programmable resources provided by "hypercloud" providers (AWS, GCP, Azure). Yes, it seems that using an IaaS provider with a VPC can provide many benefits over traditional on-prem data centers (racking & stacking, dual power supply, physical security, elasticity, programmability, locations etc).

2\. SaaS - I lump all of the other applications by the hundreds of thousands of vendors into this category. I find it hard to trust these vendors the same way that I trust IaaS providers and am much more cautious of using these applications (vs OSS or "on-prem software" versions of these apps). They just don't have the same level of security controls in place as the largest IaaS providers can & do (plus the data is structured in a way that is more easily analyzed, consumed by prying eyes).

<comment><USER75> What about first-party SaaS? Those can also be big features that bring people to some cloud providers. Not all SaaS requires you to trust your data/availability to some random vendor. Of course those first-party SaaS aren't typically suitable for lift-and-shift by their very nature, and they can still have some rough edges, but IMO you can expect them to be almost as reliable as IaaS

<comment><USER74> First-party SaaS meaning things like RDS, DBaaS, queues, LBs etc? Most of that I would sort of put into a IaaS controlled PaaS, rather than true IaaS SaaS. Yes, these are generally higher on the trust spectrum as they don't involve additional vendors accessing/managing/storing data.

<comment><USER75> A major one I'm thinking of is BigQuery, also of course all the various db/queue solutions outside of your typical S3 clone as you mentioned. That would make sense viewing them as platforms though

</comment></comment></comment></comment><comment><USER76> I work for a large video games publisher, as you might expect we use a lot of windows.

Windows server licenses on AWS and GCP are hundreds of times more expensive at our scale. Incidentally we actually do have some cloud infra and we like it, but the licensing cost is half the total price of the instance itself.

In fact, you might not know this but games are relatively low margin, and we have accidentally risked the companies financial safety by moving into the cloud.

<comment><USER73> TBF windows licensing in general is a shit show, to the point where just handling that is a specialized ability potentially warranting a full time position.

</comment></comment><comment><USER77> I'm a solo founder who's bootstrapped a saas that's in one state. I'd started out with the cloud then moved to a private cloud in a colocated data center. Saved more than 80% in monthly costs. Got faster speeds, better reliability and a ton of extra compute and network capacity. I just bought used servers from eBay that are retired from big corps. Nothing significant has really changed in the last five years on compute so I'll happily take their depreciated assets :)

__Modern__ servers are really awesome and I totally recommend them. You can do a ton remotely.

</comment><comment><USER78> Many of the stories here are from large companies, where the costs are quite a different beast. I want to offer an opposite view - from a small company (20-30 people), which is usually the kind of company best suited for the cloud.

We ran a number of modelling jobs, basically CPU intensive tasks that would run for minutes to hours. Investing in on-prem computers (mostly workstations, some servers), we got very solid performance, very predictable costs and no ops issues. Renting beefy machines in the cloud is very expensive and unless you get crafty (spot and/or intelligent deployment), it will be prohibitive for many. Looking at AMD's offering these days, you can get sustained on-prem perf for a few dollars.

Three details of note: 1) We didn't need bursty perf (very infrequently) - <USER6> this been a need, the cloud would make a lot more sense, at least in a hybrid deployment. 2) we didn't do much networking (I'm in a different company now and we work with a lot of storage on S3 and on-prem wouldn't be feasible for us), 3) we didn't need to work remotely much, it was all at the office.

Obviously, it was a very specific scenario, but given how small the company was, we couldn't afford people to manage the whole cloud deployment/security/scaling etc. and beefy workstations was a much simpler and more affordable endeavour.

</comment><comment><USER79> The idea of the cloud is to only pay for what you use. Your on-premise server is idle 99% of the time so why are you paying for a full server?

If that's not true, it turns out it's quite expensive to run things in the cloud. If your workload is crunching numbers 24/7 at 100% cpu, it's better to buy the cpu than to rent it.

<comment><USER80> Cloud servers tend to be more reliable as well if you don't run your own datacenters. We have lost our internet connection or power 3 times in the last year in the office. Its not the end of the world since we can go to 4g for our own usage but if our servers were hosted locally this would be a huge issue.

<comment><USER32> Don't forget that between cloud and servers in the company there are still VPS and rented dedicated hardware in a data center.

So you:

1\. Don't manage hardware.

2\. But manage a server (OS+software stack).

3\. Have reliable internet, power and physical security from the data center you are renting your hardware from (if you trust them fully!).

4\. Have fixed cost but also fixed resources. Tends to be cheaper for many tasks. Especially CPU/GPU heavy ones.

<comment><USER80> I consider VPSs to be cloud servers. Is this not common?

<comment><USER62> I mentioned in another comment that I use VPS and not cloud services. I think of cloud as the auto-scaling infrastructure with dynamic pricing. I think of VPS as just sharing a dedicated machine with others, so each one gets a few cores and shares other resources. The implementation of VPSs nowdays is probably more similar to cloud services, where your own space might be moved around to another physical machine without any downtime.

<comment><USER80> So you consider cloud servers to be what most people call serverless (S3/serverless functions/etc)?

<comment><USER62> I do hate the term "serverless" as it makes no sense, but I think of cloud as a system that automatically spins-up/down VPSs based on your current usage. This means the infrastructure/software also allows for automatically load- balancing between those VPSs. So I think of cloud as the VPS servers that are used to host the actual data + the layer on top that does all the scaling, provisioning, load-balancing, etc.

</comment></comment></comment></comment></comment></comment></comment><comment><USER81> We spend ~$50k/mo on serverless infrastructure on AWS.

It hurts sometimes, given we were fully colocated about 4 years back, and I know how much hardware that could buy us every month.

However, with serverless infra we can pivot quickly.

Since we're still in the beta stage, with a few large, early access partnerships, and an unfinished roadmap, we don't know where the bottlenecks will be.

For example, we depended heavily on CloudSearch, until it sucked for our use case, so we shifted to Elasticsearch, and ran both clusters simultaneously until we were fully off of CS. If we were to do that on-prem, we'd have to order a lot more hardware (or squeeze in new ES cluster VMs across heavy utilization nodes).

With AWS, a few minutes to launch a new ES cluster, dev time to migrate the data, followed by a few clicks to kill the CloudSearch cluster.

Cloud = lower upfront, higher long term, but no ceiling. On-prem = higher upfront, lower long term, but ceiling.

<comment><USER82> If "Cloud = lower upfront, higher long term, but no ceiling. On-prem = higher upfront, lower long term, but ceiling" is true, then how come the revenue of cloud companies keeps going up?

That would mean the incoming rate of users who are just starting off and find Cloud worthwhile is more exit rate of mature users who are finding on-prem more worthwhile than cloud

<comment><USER83> If you're spending more than 50k/month on AWS where is the money to move to on-prem? When they got you, they got you.

</comment></comment></comment><comment><USER84> The (startup) Oxide podcast has good history/stories about on-prem servers, from veterans of pioneering companies. They are fans of open-source firmware and Rust, and are working to make OCP-based servers usable for on-prem. In one podcast, they observed that cloud is initially cheaper, but can quickly become expensive with growth. There is a time window where you can still switch from cloud to on-prem, but if that window is missed, you're left with high switching costs and high cloud fees.

[https://oxide.computer/podcast/](https://oxide.computer/podcast/)

<comment><USER85> Their co founder Bryan Cantrill gave a talk at Stanford on what they are trying to do, essentially offer on prem servers comparable to what "hyperscalers" like Google and Facebook put in their data centers — highly efficient and customizable (in low level software) iirc.

[https://youtu.be/vvZA9n3e5pc](https://youtu.be/vvZA9n3e5pc)

</comment></comment><comment><USER71> Manufacturing. The cost to a factory if the internet is down is too great. Each facility has its own highly redundant virtualization infrastructure hosting 50 to 100 VMs.

<comment><USER86> I was in manufacturing IT before moving to big tech. Our big campuses in the US & Europe had 40-80mbps internet circuits. The remote facilities in developing countries often only had 10mbps MPLS connections to a regional hub. To be 100% honest, we had 10x the outages caused by crappy local infrastructure than anything having to do with a SaaS service or IaaS/PaaS provider. Seriously, things like bad storms, a snake (cobra!) sneaking into the server room and frying itself and a machine it was snuggling against, utility workers accidentally severing cables, generators failing during power outages, labor strikes, and so much more. Moving to the cloud -- or even just hosting everything centrally -- was much more stable than maintaining a fleet of distributed machines.

</comment></comment><comment><USER87> I work in gamedev. Build servers, version control, etc. are almost always on- premise, even if a lot of other stuff has been transitioned to the cloud. There's a few reasons:

1) Bandwidth. I routinely saturate my plebian developer gigabit NIC links for half an hour, an hour, longer - and the servers slurp down even worse. In an AAA studio I am but one of hundreds of such workers. Getting a general purpouse internet connection that handles that kind of bandwidth to your heavily customized office is often just not really possible. If you're lucky your office is at least in the same metro area as a relevant datacenter. If you're really lucky you can maybe build a custom fiber or microwave link without prohibative cost. But with those kinds of geographical limitations, you're not so much relying on the general internet, so much as you're expanding your LAN to include a specific datacenter / zone of "the cloud" at that point.

2) Security. These servers are often completely disconnected from the internet, on a completely separate network, to help isolate them and reducing data exfiltration when some idiot installs malware-laden warez, despite clear corporate policy threatening to fire you if you so much as even _think_ about installing bootleg software. Exceptions - where the servers _do_ have internet access - are often recent, regrettable, and being reconsidered - because of, or perhaps despite, draconian whitelisting policies and other attempts at implementing defense in depth.

3) Customizability. Gamedev means devkits with strict NDAs and physical security requirements, and a motley assortment of phone hardware, that you want accessible to your build servers for automatic unit/integration testing. Oddball OS/driver/hardware may also be useful for such testing. Sure, if you can track down the right parties, you might be able to have your lawyers convince their lawyers to let you move said hardware into a datacenter, expand the IP whitelists, etc... but at that point all you've really done is made it harder to borrow a specific popular-but-discontinued phone model from the build farm for local debugging when it's the only one reproducing a specific crash when you want to debug and lack proper remote debug tooling.

...there are some inroads on the phone farms (AWS Device Farm, Xamarin Test Cloud) but I'm unaware of farms varied desktop hardware or devkits. Maybe they exist and just need better marketing?

I have some surplus "old" server hardware from one such gamedev job. Multiple 8gbit links on all of them. The "new" replacement hardware often still noticably bottlenecked for many operations.

</comment><comment><USER88> There's a renewed interest in on-prem bare metal recently with a lot of different offerings helping to make various parts of the stack easier to manage.

Awesome bare metal is a new repo created by Alex Ellis that tracks a lot of the projects: [https://github.com/alexellis/awesome- baremetal](https://github.com/alexellis/awesome-baremetal)

Also we (Packet) just open sourced Tinkerbell, our bare metal provisioning engine: [https://www.packet.com/blog/open-sourcing- tinkerbell/](https://www.packet.com/blog/open-sourcing-tinkerbell/)

</comment><comment><USER89> 1\. 500TB of storage for 3-6mo of CCTV footage.

2\. Bought a hanful of $700 24 core Xeons on eBay 2 years ago for 24/7 data crunching. Equivalent cloud cost was over $3000/mo. On-Prem paid off within a month!

3\. Nutanix is nice. Awesome performance for the price and almost no maintenance. Got 300+ VDI desktops and 50+ VMs with 1ms latency.

<comment><USER90> > $700 24 core Xeons on eBay 2 years ago

Can you clarify? I don't think such a product exists as a single chip at that price point. The Threadripper 3960X costs $1400, and that released less than a year ago.

Edit: Looking up Intel chips on Wikipedia, I think you might be using 12-core/24-thread chips...

[https://en.wikipedia.org/wiki/Skylake_(microarchitecture)#Xe...](https://en.wikipedia.org/wiki/Skylake_\(microarchitecture\)#Xeon_Bronze_and_Silver_\(dual_processor\))

<comment><USER91> I picked up three Dell R900's for an average of $200 each, with 4x Xeon E7450 and 128GB of ECC RAM. No hyperthreading (2011!), it's 24 real cores.

They're noisy and use lots of power, but you can't argue with the value for money.

<comment><USER92> I have one similar to this. I don't think it is an R900, but it's an older 1U rack mount. I forget if it's a 12 or 24 core xeon, but it was dirt cheap, came with 72 gigs of RAM, and sounds like a jet engine turning it on. I recently built a Ryzen box with 128 gigs of RAM and it's much quieter...

</comment></comment></comment></comment><comment><USER93> CEO of an AI startup & former AWS employee here.

The cloud sucks for training AI models. It's just insanely overpriced in a way that no "Total Cost of Ownership" analysis is going make look good.

Every decent AI startup––including OpenAI––has made significant investments in on-premise GPU clusters for training models. You can buy consumer-grade NVIDIA hardware for a fraction of the price that AWS pays for data center-grade GPUs.

For us in particular, the payback on a $36k on-prem GPU cluster is about 3-4 months. Everything after that point saves us ~$10k / month. It's not even close.

When I was AWS, I tried to point this fact out to the leadership––to no avail. It simply seemed like a problem they didn't care about.

My only question is why isn't there a p2p virtualization layer that lets people with this on-prem GPU hardware rent out their spare capacity?

<comment><USER94> Are TPUs too application specific to replace GPUs? It seems cloud TPUs could be competitive with GPUs in terms of $ per number of target epochs, provided you can do data parallelism for your workloads.

Also, IBM offers bare metal pricing which is _somewhat_ cheaper than virtualized instances attached to GPUs (and faster too).

I think GPU virtualization is not quite there yet because Nvidia does not give access to core GPU functionality needed for efficient virtualization - you're stuck with using their closed-source libraries and drivers.

</comment></comment><comment><USER95> Bandwidth is the big reason to stay on-prem. Good peering contracts can more than make up for any cloud advantages for bandwidth intensive uses.

Now the hard part is turning those cost advantages into operational improvements instead of deficiencies.

</comment><comment><USER96> Becuse we don't trust a foreign cloud provider with our client's data. Why is that so hard to understand?

All the best cloud providers are from the US and as a european company with clients in european government and healthcare we are often not morally or legally allowed to use a foreign provider.

The sad thing is that this is an ongoing battle between people on a municipal level who believe they can save money in clouds, and morally wiser people who are constantly having to put the brakes on those migration projects.

<comment><USER97> What about the existing/upcoming technologies of let you use the cloud without trusting it?

<comment><USER96> Doesn't matter because all US companies are subject to US laws and agencies. Even if the data is in Ireland, they are obliged to cooperate and before you know it all our patient records are leaked in the states.

And speaking of Ireland, I have a memory of a Microsoft EULA saying that even if the data is stored in Ireland they can't guarantee that it won't be transferred to the US.

</comment></comment></comment><comment><USER98> Small company here (30 total people, 8 in IT/software).

\- unwillingness to seed control of the critical parts of our software infrastructure to a third party.

\- given our small team size and our technical debt load we are not currently able to re-architect to make our software cloud-ready/resilient.

\- true cost estimates feel daunting to calculate, whereas on-prem costs are fairly easy to calculate.

<comment><AUTHOR> I agree with the last 2 points. Estimates are hard to get right, and rearchitecting an existing app is probably not worth it.

What about your first point though? Do you not trust a 3rd party to maintain infrastructure properly? In what way?

</comment></comment><comment><USER99> I'm application architect at enterprise-type org. We have a few SaaS applications, but all the big dogs, including custom dev, run in-house on a dual data centre vmware environment. It's cheaper for us to spin up more VMs in-house, so there's no real cloud driver for things that just live on VMs. On the other hand, our ops team are still working on network segmentation and self-service, but I regularly get a standard VM provisioned in less than 20 mins. If we had to buy new data centres it might be different.

But the real reason we're not deeper in the cloud is that our business-types insist on turn of the century, server-based software from the usual big vendors, and all the things that integrate with them need to use 20th century integration patterns, so for us migrating to the cloud (in stages at least) would have drawbacks from all options without the benefits. It's only where we have cloud-native stuff that we can sneak in under the radar for stand-alone greenfields projects, or convince the business types that they can replace the Oracles and Peoplesofts with cloud-first alternatives will things really change.

<comment><USER100> Last company I was at was more or less as you describe.

Now.. at a company in a different industry, there's a 5+ year plan to move 100% to cloud. Nascent efforts are about 18 mos old already, no apps are live yet.

Fortunately, they've been using a container approach for their on-prem stuff for a while, so some stuff can move over pretty easily, a lot of things will get a touch-up or more interesting upgrade along the path to the cloud environment.

Not even talking about de-commissioning the DCs yet, but those will get defunded as things go on.

</comment></comment><comment><USER101> Security... not against hacker but against provider's government.

I worked for some french or european companies, with IPs and sensitive informations, and US business competitors. By the US law, US companies may have to let the US gov spy on their customers (even non US, even on non US location), so this may be a problem for strategic sectors, like defense for example.

In that case, sensitive informations is required to be hosted in country by a company of the country, under the country law.

Of course, it's not against "cloud" in general... only against US cloud providers (and chinese, and...)

</comment><comment><USER102> I work on two projects, neither of which use cloud.

For my day job, it is privacy and legal constraints. I work for the government and all manner of things need to be signed off on to move to cloud. We could probably make it work, but in government, the hassle of doing so is so large that it is not going to happen for a long time.

In my contract project, it is a massive competitive advantage. I won't go into too many details, but customers in this particular area are very pleased that we do not use a cloud provider and instead host it somewhat on-premise. I don't see a large privacy advantage over using the cloud, but the people buying the service do simply because they are paranoid about the data and every single one of them could personally get in a lot of trouble for losing the data.

Not my project, but intensive computing requirements can be much more cheaply filled by on-premise equipment (especially if you don't pay for electricity), so my university does most of its AI and crypto research on-premise.

</comment><comment><USER103> My company moved from all on prem, to all in AWS. Having used both, I'd much rather go back to on prem. I did architecture, capacity planning, system tuning, deployments, etc. I knew everything about all of them, and treated them as sacred. The next generation came in, deciding not to learn anything about systems and brought in the 'systems as cattle' attitude and everything that comes with it.

I try to remain objective, there are some pros to AWS, but I still much prefer my on prem setup. It was way cheaper, and deployments were way faster.

</comment><comment><USER104> \- Uptime

The number and frequency of outages in Azure are crazy. They happen non-stop all year around. You get meaningless RCAs but it never seems to get better, and if it did, you'd have no way of knowing.

Compare this with doing stuff internally - you can hire staff, or train staff, and get better. In the long run outsourcing and trusting other companies to invest in "getting better" doesn't end very well. Just because they moved their overall metrics overall from 99.9 to 99.91 may not help your use case.

\- Reliability

Their UIs change every day, there's no end to end documentation on how things work. There's no way to keep up.

\- Support

Azure's own support staff are atrocious. You have to repeatedly bang your head against the wall for days to get anyone who even knows the basic stuff from their own documentation.

But it's also difficult to find your own people to do the setup too. Sure, lots of people can do it, but because it's new they have little experience and end up not knowing much, unable to answer questions, and building garbage on the cloud platform. Because there's no cloud seniority, it hasn't been around for long enough.

\- Security

Cloud providers have or can get access and sometimes use it.

\- Management

I've seen too many last minute "we made a change and now everything will be broken unless you immediately do something" notifications to be happy about.

\- Cost

It's ridiculously expensive above a certain scale, and that scale is not very big. I don't know if it's because people overbuild, or because you're being nickel-and-dimed, or if you're just paying so many times above normal for enterprise hardware and redundancy. It's still expensive.

Yes, owning (and licensing) your own is expensive too.

For smaller projects and tiny companies, totally fine! It's even great!

\- Maturity

People can't manage cloud tools properly. This doesn't help with costs above.

PS: I don't think any other cloud service is better.

</comment><comment><USER105> Not exactly on-premise but we rent 2 big dedicated server (ovh) + install VMWare ESXi on them. Going to the cloud would cost more, the price would be unpredictable, only to solve a scale problem we won't have. And customers love to know their data are hosted in France by a French company, not by Google or Amazon :)

</comment><comment><USER106> Network storage. MacBooks don't have a lot of space and artists like to make HUGE psd files.

Bonus points for small stuff like RADIUS for wifi and stuff. Groups charging $5/user for junk like that is absolutely awful with a high number of staff.

With a staff of 100, a single box with a bunch of hard drives is two months worth of cloud and SaaS.

TCO needs to come down by like at least 100x before I consider going server- less.

</comment><comment><USER107> We own infra because we need to own and control it and also because it's just vastly cheaper at the scale we use.

Besides, we do have things like our own S3, k8s and other cloud-ish utilities running so we do not miss out that much, I guess.

</comment><comment><USER108> This conversation sounds a lot like the debate around globalization and outsourcing manufacturing to me. It might be a stretch but there is something here.

There is room for both Cloud and On-Prem to exist. This endless drive by industry to push everyone to cloud infrastructure and SaaS, in my humble opinion, will look exactly like the whole supply chain coming from the east during a pandemic.

The economics of it look great in a lot of use cases, but putting our whole company at the mercy of a few providers sounds terrible to me. Even more so when I see posts on HN about folks getting locked out of their accounts with little notice.

It does not take much to bring our modern cloud to a grinding halt. For example, a mistake by an mostly unheard of ISP lead to a massive outage not less than a year ago(1).

It was amazing to see the interconnections turn to cascading issues. 1 ISP goofs. 1-2 major providers have issues and the trickle down effect was such that even services that thought they were immune from cloud issues were realizing that they rely on a 3rd party that relies on a different 3th party that uses cloudflare or AWS.

So, even though I think the cloud is (usually) secure, stable, resilient, etc... I still advocate for its use in moderation and for 2 main use cases.

1 - elastic demands. Those non-critical systems that add some value or make work easier. Things we could do without for several days and not hurt the business much.

2 - DR / Backup / redundancy. We have a robust 2 data center / DR fail over system. Adding cloud components to that seems reasonable to me.

(1)[https://slate.com/technology/2019/06/verizon-dqe-outage- inte...](https://slate.com/technology/2019/06/verizon-dqe-outage-internet- cloudflare-reddit-aws.html)

Edit: Spelling and clarity

Edit2: New reasons to stay on prem are happening all the time. [https://www.bleepingcomputer.com/news/security/microsofts- gi...](https://www.bleepingcomputer.com/news/security/microsofts-github- account-allegedly-hacked-500gb-stolen/)

</comment><comment><USER109> I worked for a famous large tech company that makes both hardware and software. On stuff that runs on customer datacenters.

There are plenty of companies that run their infrastructure to keep their data secure and accessible.

It's not the type of companies that blog about their infra or are popular on HN. Banks and financial institutions, telcos, airlines, energy production, civil infrastructure.

Critical infrastructure need to survive events larger than a datacenter outage. FAANGs don't protect customers from legal threats, large political changes, terrorist attacks, war.

</comment><comment><USER110> The way I think about it this: not using the cloud is like building your own code editor or IDE after assembling your own laptops and desktops. It may make you happier and it's a great hobby, but if you're trying to run a business you need to do a cost benefit analysis.

We currently have double digit petabytes of data stored in our own data centres, but we're moving it to S3 because we have far better things to do with our engineers that replace drives all day, and engineering plus hardware is more expensive than S3 Deep Archive - but it wasn't until Deep Archive came out.

We put out hundreds of petabytes of bandwidth, and AWS is horribly expensive at first glance, but if you're at that scale you negotiate private pricing that brings it to spitting distance of using a COLO or Linode/Hetzner/OVH - the distance is small enough that the advantages of AWS outweigh it, and it allows us to run our business at known and predictable margins.

Besides variability (most of our servers are shut nights, weekends and when not required), op ex vs cap ex, spikes in load (100X to 1000X baseline when tickets open), there's also the advantage of not needing ops engineers and being to handle infrastructure with code. If you have a lot of ops people and don't need any of the advantages, and you have lots of money lying around that you can use on cap ex, and you have a predictable load pattern, and you've done a clear cost benefit analysis to determine that building your own is cheaper, you should totally do that. Doesn't matter what others are doing.

<comment><USER111> You're not building everything from scratch on-prem. There are excellent tools for deploying and managing infrastructure like Terraform, Ansible, Puppet that funny enough are used to deploy to cloud as well. Add a self-hosted Kubernetes cluster to that and your on-prem is not that different than a cloud.

As for ops people - you might not need an engineer to replace failed hard- drives, but you'll need a DevOps person to manage Cloud Formation templates and such, and they cost more.

</comment></comment><comment><USER112> One of the reason is a legacy systems. Some companies are too tied up to old custom made systems built on old software and hardware virtually not convertible to cloud. You will be surprised but there are big corporation still using AS400 and not planning to switch anytime soon. If you heard in recent news US unemployment system was still built on COBOL... In 2020... Another reason is a cost. I love AWS! Its fantastic to be able to create and launch servers or the whole farm of servers in the matter of minutes! And ability to convert physical servers to virtual and upload them to the cloud is breathtaking! But my monthly bill started at $300 and grew to $18K per month in less than 3 years. And that was for just a few virtual servers with Windows OS and SQL. My company realized that we can have a state of the art datacenter with WmWare and SAN on premises for the fraction of that price. Put second one on the other coast (one of our other offices) and you have your own cloud with better ping and six digits figure saving a year. For the last I would name vendor lock. With vSphere its very easy to move your virtual servers between AWS, Azure and Google (assuming you can afford all 3 and licensing cost of WmWare) but have you ever tried to "download" your server back to premise? It's virtually impossible or made so hard by cloud players trying to keep you up they're in the clouds. With all said I read that Netflix (I believe its Netflix) saving hundreds of millions dollars per year by using Amazon instead of its own servers. I also read somewhere that Dropbox moved away from AWS...

</comment><comment><USER113> A job or two ago:

Everything on-site for a couple reasons (50 servers). Mainly because as a manufacturing company, machines on the shop floor need to talk to the servers. This brings up issues of security (do you really wan to put a 15 year old CNC machine 'on the internet'?). Also, if our internet connection has issues, we still need to build parts.

The other big part of it is the mindset of management and the existing system, which was built to run locally, does Amazon offer cloud hosted Access and Visual Basic workers?

</comment><comment><USER114> We're looking at moving from AWS to having some machine space rented in two data centers. The reason is purely cost.

There are still some computers on site due to equipment being tied to it, telephony stuff, etc.

My last company was looking at "moving to the cloud", with the idea that its data centers were too expensive, but found out that the cloud solutions would be even more expensive, despite possible discounts due to the size. They still invested in it due to some Australia customers wanting data to be located there.

</comment><comment><USER115> I haven't personally done detailed cost analysis lately, but if you have systems that regularly operate at 80+% of capacity, I can't see how the operating costs of any cloud operator can be cheaper than operating it yourself. Their whole pricing model is based on being able to over-provision compute and move workloads around within a datacenter. As much as people talk about failing hard drives and other components at scale, failure rates are still low enough that you could operate dozens of systems at full capacity for 3+ years with maybe a handful of legit hardware failures. To rent that same compute from any cloud provider would cost significantly more. The cheapest "Dedicated Host" on AWS will cost you almost $12k over 3 years if you pay for it on-demand, and it's equivalent in specs to something you can buy for ~$2k.

> am I missing something?

I'd want more background behind what you mean by "at least for business". What kind of business? Obviously IaaS providers like Digital Ocean and Linode are are type of business that would not use other clouds. Dropbox and Backblaze as well would probably never use something like S3. And there are legitimate use cases outside of tech that have needs in specific teams for low latency compute, or its otherwise cost and time prohibitive to shuttle terabytes of data to the cloud and back (3D rendering, TV news rooms, etc). If you're talking about general business systems that can be represented by a website or app with a CRUD API, then most of that probably doesn't require on-prem. But that's not the only reason businesses buy servers.

</comment><comment><USER116> As most have mentioned already: cost

We started out with Emvi [1] on Kubernetes at Google Cloud as it was the "fancy thing to use". I like Kubernetes, but we paid about 250€/month just to run some web servers and two REST APIs. Which is way too much considering that we're still working on the product and pivoting right now, so we don't have a lot of traffic.

We then moved on to use a different cloud provider (Hetzner) and host Kubernetes on VMs. Our costs went down to about 50€ just because of that. And after I got tired managing Kubernetes and all the complexity that comes along with it, we now just use a docker-compose on a single (more powerful) VM, which reduced our cost even futher to about 20€/month and _increased_ performance, as we have less networking overhead.

My recommendation is to start out as simple as possible. Probably just a single server, but keep scaling in mind while developing the system. We can still easily scale Emvi on different hardware and move it around as we like. We still use Google Cloud for backups (together with Hetzners own backup system).

[1] [https://emvi.com/](https://emvi.com/)

</comment><comment><USER117> Anything with massive storage and massive compute that doesn't need low latency is a great fit. I still host ~300TiB and ~250 cores at home because the cloud cost would be astronomical. Edit: This is for personal stuff related to internet-wide scan data and domain tracking. See [https://github.com/hdm/inetdata](https://github.com/hdm/inetdata)

</comment><comment><USER118> 1\. Our customers run our software on their own machines for security and data-control reasons. As soon as something's running on someone else's hardware, the data is out of your control. Unless you're going to accept the (often massive) cost of homomorphic encryption, AND have a workload amenable to that, it's a simple fact.

2\. Everything we do in house is small enough that the costs of running it on our own machines is far less than the costs of working out how to manage it on a cloud service AND deal with the possibility of that cloud service being unavailable. Simply running a program on a hosted or local server is far far simpler than anything I've seen in the cloud domain, and can easily achieve three nines with next to no effort.

Most things which 'really need' cloud hosting seem to be irrelevant bullshit like Facebook (who run their own infrastructure) or vendor-run workflows layered over distributed systems which don't really need a vendor to function (like GitHub/Git or GMail/email).

I'm trying to think of a counterexample which I'd actually miss if it were to collapse, but failing.

</comment><comment><USER119> I actually am moving my startups servers from AWS to a home server.

Reasoning:

* We know how much compute is need.

* We know how much the new servers can compute.

* We have the ability to load balance to AWS or Digital Ocean or another service as needed.

* This move provides a 10x speed improvement to our services AND reduces costs by 70%.

For reference, had to call the ISP (AT&T) and they agreed to let me host my current service. It's relatively low bandwidth, but has high compute requirements.

</comment><comment><USER120> We operate an X509 PKI with a Trust anchor and CA. Its not impossible to run the Hardware Security Module (HSM) in the cloud but its off the main path. Its more plausible to run it in a D.C. but it invites security threats you don't have, if you run it inside your own perimiter. Of course you also then invite other risks, but its a balancing act and it has to be formally declared in your Certification Practice Statement (CPS)

We also run some registry data which we consider mission critical as a repository. We could run the live state off-prem, but we'd always have to be multi-site to ensure data integrity. We're not a bank, but like a bank or a land and titles administration office, registry implies stewardship in trust. That imposes constraints on "where" and "why".

Take central registry and the HSM/related out of the equation, if I was building from scratch I'd build to pub/sub, event-sourcing, async and in-the- cloud for everything I could.

private cloud. If you don't control your own data and logic, why are you in the loop?

</comment><comment><USER121> We are a research group at a company with 2000 or so employees. We have a few GPU machines to train models, which are utilized nearly around the clock.

AWS and co's GPU-enabled servers are exceedingly expensive. Most of the GPU models on those machines are also very old. We pay maybe 1/3 or less to maintain these machines and train models in-house vs paying AWS.

Mind you, we use AWS for plenty of stuff...

</comment><comment><USER122> I work in R&D for a telecomms/optics company.

All servers are on premises. Not allowed to have a laptop. No access to emails/data outside of the office. No USB drives, printing documents, etc.

Reason? Protect IP. From who? Mostly Huawei.

Good and bad: When I walk out the door... I switch off. The bad is that working from home isn't realy an option. Although they have accommodated somewhat for this pandemic.

</comment><comment><USER123> We sell hosting to large number of different customers who for whatever reason, mostly legal, are required to keep data within the borders of Denmark. There is no Google, Amazon, Azure or DigitalOcean data centers in Denmark, so cloud isn't an option for them.

Regarding cost, well it depends. We try to help customers to move to cloud hosting if it's cheaper for them. It almost always will be if they take advantage of the features provided by the cloud providers. If you just view for instance AWS as a VMware in the cloud, then we can normally host the virtual machines for you cheaper and provide better service.

You have to realize that many companies aren't developing software that's ready for cloud deployment. You can move it to EC2 instance, but that's not taking advantage of the feature set Amazon provides, and it will be expensive and support may not be what you expect. You can't just call up Amazon and demand that they fix your specific issue.

</comment><comment><USER124> > The more I learn, the more I believe cloud is the only competitive solution > today, even for sensitive industries like banking or medical.

Then learn A LOT more and start with mainframes and their reliability.

</comment><comment><USER125> Our business has had greatly increased load due to COVID-19. It would have been very nice to buy a 128 core EPYC bare metal server to run our SQL Server on at this time, to buy us time to rearchitect to handle the load. Instead we are stuck with 96vCPUs because that is the most Amazon can do.

Its also very very expensive to have a 96vcpu VM on amazon!

</comment><comment><USER126> I'm one of two IT persons at a food processor, in a small town. Despite living in an area where a majority of IT & Programmers work at the hospital or an insurance company, my boss has run our own networks and servers since the days of Novell, and we continue to run Windows servers on-premise, instead of the cloud. It does lead to interesting situations, like finding out a Dell server shrieking next to you is running max fan speed because the idrac is not connected.

Neither of us have any experience with the cloud, whereas we have a lot of Microsoft experience. We still rely on OEM licenses of Office, because Office 365 would be 3x or more expensive. We have a range of Office 2019, 2016, 2013 OEM, and we get audited by them nearly every year.

We use LastPass, Dropbox and Github, but only the basic features, and LastPass was an addition last year after someone got into our network through a weak username/password.

In our main location, we have three ESX boxes, running several virtual servers, and then we have a physical server for our domain controller, file sharing and DHCP, DNS in other locations. We also switched to a physical server for our new ERP application server, which hasn't yet been rolled out.

Projects like upgrading our ERP version can take months, but we have a local consulting team, with a specialist in our particular ERP solution, as well as a Server and Network specialist, and we also have a very close relationship with our ISP, who provides WAN troubleshooting.

Our IT budget is small relative to our company revenue, so most cloud proposals would raise our costs manyfold. We continue to use more services like Github and Lastpass, and we both have multiple hats.

I'm a developer, in-house app support, Email support, HR systems support, ERP support, PC setup, and I run our Data Synchronization operation and my boss runs EDI. I do a lot of PowerShell and Task Scheduler, but I've got familiar with bash through git bash.

</comment><comment><USER127> Here is a stupid example: Excel vlookups work on a network drive, but not on a cloud service like Dropbox or OneDrive. The absolute path can't resolve if it's used across multiple Excel users. If the users store the file locally, each will have a different path on their computers. Excel stores actual paths. [1]

There is one way around it: Mounting the cloud server as a network drive (some providers do this by default, but OneDrive is not one of them, neither is Dropbox).

I don't know of a way of mounting OneDrive as a virtual drive; I would be interested to know.

It sounds stupid, but the above was a real life scenario.

[1] Only if the files are closed. Excel can change the path if you have the file open, but it can't change it to multiple option across different PCs. But as I have mentioned before, Excel doesn't seem to document all of their more subtle features.

</comment><comment><USER128> Unreliable internet.

A retail company may decide that the best place to put up a new branch is coincidentally (though there might be a correlation) at the edge of what the available ISPs currently cover. They might have to make a deal to get an ISP to extend their area to where the store is going to be. However, because of lack of competing ISP options on the part of the retailer, and the lack of clients in the retailer's area on the part of the ISP, that service is probably not going to be all that reliable.

Also, that retail company may experience a big rise in sales after a natural disaster occurs, when communications (phone/cell/internet) are mostly down for the area. One tends not to think about stuff like that until it happens at least once.

It's very important for the ERP/POS systems to be as operational as possible even when the internet is down.

</comment><comment><USER129> On the smaller end of the scale, I have a $12K/mo spend with Azure. I decided to go back to Coloc.

For under $50K, I have 4 machines with an aggregate 1TB RAM, 48 cores, 1 pricy GPU, 16TB of fast SSD, 40 TB HHD, and infiniband @ 56GB/sec. Rent on the cabinet is less than $1K/mo. It's going to cost me about $20K in labor to migrate.

So the nominal break-even point is six months but the real kicker is that this is effectively x10-30 the raw power of what I was getting on the cloud. I can offer a quantitatively different set of calculations.

It also simplifies a bunch of stuff: 1\. No APIs to read blob data -- just good old files on an ZFS share. 2\. No need to optimize memory consumption. 3\. No need for docker/k8s/etc to spin up under load. Just have a cluster sitting there.

There are downsides but Coloc beats the cloud for certain problems.

</comment><comment><USER130> I'm going to buck the trend and say cloud is great. We do cloud, on-prem and colo (16 racks in two different DCs).

Procurement is a nightmare especially when your vendor is having problems with yields (thanks Intel!) and the ability to scale up and scale down without going through hardware procurement process saves us millions of dollars a year.

We avoid the lock-in by running on basic services on multiple cloud providers and building on top of those agnostically.

Spend is in the millions per month between the cloud providers, but the discounts are steep. We're essentially had to build our own global CDN and the costs are better than paying the CDN services and better than running our own hardware & staffing all those locales.

It's a no brainer. We'll continue to operate mixed infrastructure for quite some time as certain things make sense in certain places.

</comment><comment><USER131> PhD student/researcher. Most of the compute I do are HPC/scientific computing style and run on University or US National Lab machines. We thought about cloud, but the interconnect (MPI-style code) is very important for us and it's not very clear to me what's available there in the cloud.

</comment><comment><USER132> Self-driving company here. We're doing on-prem because we have a clear projection on the amount of data we'll need to injest, store and process using GPUs.

The advantages of running things in a cloud are clear -- and as an infrastructure team we have challenges around managing physical assets at scale, however it's clear with the cost of cloud providers that eventually we would have to pull data into a datacenter to survive at some point.

Co-location costs are fixed, and it's actually easy to make a phenomenal deal now-a-days given the pressure these companies are under.

The real trick of it all is that regardless of running on-prem or in the cloud, we need to run as if everything is cloud native. We run Kubernetes, Docker, and as much as possible automate things to the point that running one of something is the same as running a million of it.

</comment><comment><USER133> There is not one clear cut answer on this. It depends on what your company values, i.e. cost vs. agility. If you are using the cloud for what it was meant for availability, scalability, elasticity--and those are the things that your org values--its the right fit for you. If on the other hand you value cost then it clearly isn't the right fit.

One other point I'll make, the true value of cloud isn't in IaaS, renting VMs from anyone is relatively expensive compared to the costs of buying a server and maintaining it yourself for a number of years. The true value of the cloud is when can architect your solution to utilize the various services the cloud providers offer, RDS/DynamoDB, CDN, Lambda, API Gateway, etc. so that you can scale quickly when you need to.

</comment><comment><USER42> For hobby projects, I own a moderately outdated 1U "pizzabox" installed in a downmarket colocation facility in a nearby major city. Considering the monthly colocation rate and the hardware cost amortized over two years (I will probably not replace it that soon but it's what I've used for planning), this works out to appreciably less than it would cost to run a similar workload on a cloud provider. It costs about the same or possibly a bit less than running the same workload on a downmarket dedi or VPS provider, but it feels more reliable (at least downtime is usually my fault and so under my control) and the specs on the machine are higher than it's easy to get from downmarket VPS operations.

Because my application involves live video transcoding I'm fairly demanding on CPU time, which is something that's hard to get (reliably) from a downmarket VPS operation (even DO or what have you) and costly from a cloud provider. On the other hand, dual 8 core Xeons don't cost very much when they're almost a decade old and they more than handle the job.

There are a few fairly reputable vendors for used servers out there, e.g. Unix Surplus, and they're probably cheaper than you think. I wouldn't trust used equipment with a business-critical workload but honestly it's more reliable than an EC2 instance in terms of lifetime-before-unscheduled-termination, and since I spend my workday doing "cloud-scale" or whatever I have minimal interest in doing it in my off-time, where I prefer to stick to an "old fashioned" approach of keeping my pets fed and groomed.

And, honestly, new equipment is probably cheaper than you think. Dealing with a Dell account rep is a monumental pain but the prices actually aren't that crazy. Last time I purchased over $100k in equipment (in a professional context, my hobbies haven't gotten that far yet) I was able to get a lot for it - and that's well less than burdened cost for one engineer.

</comment><comment><USER134> One of my favorite things (at least for personal projects) about using the cloud is so-called "platform as a service" systems like Heroku, where I don't have to get down in the weeds, I just push code and the process starts (or restarts).

Is there something like that I could use on my own hardware? I just want to do a fresh Linux install, install this one package, and start pushing code from elsewhere, no other configuration or setup necessary. If it can accept multiple repos, one server process each, all the better. I know things like Docker and Kubernetes exist but what I want is absolute minimal setup and maintenance.

Does such a thing exist?

<comment><USER135> You're looking for Dokku

Same git push deploys, heroku-compatible buildpacks or Dockerfiles, all on your own hardware, MIT license.

<comment><USER134> This looks perfect, thank you! I knew there was no way I could've been the first person to think of this

</comment></comment></comment><comment><USER136> After a big cloud-first initiative, several managers left; leaving implantation to Linux sysadmins now in charge of cloud. Treated cloud as some colo facility, dump all apps in one big project/account, cloud costs spun quickly out-of-control and lots of problems with apps not being segregated from each other. Cloud declared 'too expensive' and 'too insecure', things migrated back on-prem, team now actively seeks to build and staff colo facilities with less than 10ms latency somewhere outside coastal California (Reno, Vegas, PHX) which just isn't gonna happen because physics.

</comment><comment><USER137> International Traffic in Arms Regulations (ITAR) Compliance - much easier to keep it on site, off-site compliance is costly. Also, cost over time. Better control of performance requirements for certain applications.

</comment><comment><USER138> We're in the process of moving a greenfield project from AWS to a more traditionally hosted solution.

AWS turned out to be 5-10 times more expensive; what's worse, our developers are spending more then half their time working around braindead AWS design decisions and bugs.

A disaster any way you look at it.

There are good reasons to chose AWS, but they're never technical. (Maybe you don't want to deal with cross-departmental communications, or you can't hire people into a sysadmin role for some reason, maybe you want to hide hosting in operational expenses instead of capital, etc.)

</comment><comment><USER139> Bandwidth costs.

Most dedicated servers come with unmetered bandwidth so not only is it cheap to serve large files but your bandwidth costs won't suddenly explode because of heavy usage or a DDoS attack.

</comment><comment><USER140> Here is how we went about w/ CSPs (AWS, Azure, GC, Oracle). Thoughts welcome

Getting Started --> Definitely go w/ CSPs. No need to worry about infra.

Pre Product Market Fit + Steady Growth --> On Premise, because CSPs might be expensive until you find a consistently profitable business.

Pre Product Market Fit + HyperGrowth --> CSPs since you wont be able to keep up [we never got to this stage]

Product Market Fit w/ Sustainable Good Margins --> CSPs, pay to remove the headache [we never got to this stage]

Side Note: w/ GPUs, CSPs rarely make sense

</comment><comment><USER141> I agree with you OP.

Our company provides on-premise ERP systems to small (we're talking at most 20 person companies) wholesale distributors.

Pre-COVID, I was pushing for a cloud solution to our product and pivoting our company towards that model. We're at a hybrid approach when COVID hit.

What ends up happening with an on-premise/hybrid cloud model is we end up doing a lot of the sysadmin/IT support work for our customers just to get our ERP working. This includes getting ahold of static IP addresses (and absolving responsibility), configuring servers/OSes, and several other things along the same vein that's wholly irrelevant to the actual ERP like inventory management and accounting.

Long story short, these customers of ours end up expecting us to maintain their on-premise server without actually paying for help or being knowledgable about how it all works. We keep pitching them the cloud but they're not willing to pay us a recurring fee even though it actually saves the headaches of answering the question "who's responsibility is it to make sure this server keeps running?"

I think a lot of these answers here are dealing with large-scale products and services where the amount of data and capital costs is so massive it makes sense to start hiring your own admins solely to maintain servers. For these small mom-and-pop shops who are looking for automation, the cloud is still the way to go.

<comment><USER128> Deja vu. I think you totally hit the nail on the head with that last paragraph. On-premise ERP systems probably only make real sense for (non- small) companies that wish to avoid relying on the internet (because e.g. their business strategy requires that freedom) and can hire long-term sysadmins/programmers that can provide support to those systems.

<comment><USER141> Have you had experience selling on-premise systems? I'm really curious how other companies handle the sysadmin and IT support issue.

</comment></comment></comment><comment><USER142> Finance space, under 100 people. Most servers are either latency-sensitive or 24/7 fully-loaded HPC. Neither case fits the cloud model. We do use cloud for build/test, though.

</comment><comment><USER143> Many organizations do have private clouds.

If by cloud you mean a public cloud like Google, Amazon, or Microsoft, then forget about it; not with these companies piping data directly to U.S intelligence.

<comment><USER144> What's the difference between private cloud and on prem?

Does it just mean you let everyone spin up VMs etc rather than requiring them to go through IT?

<comment><USER52> Mostly. Cloud is just bin packing compute and data storage. There are many use cases where it's cheaper for you to host the hardware you cloud on instead of a public cloud provider.

</comment></comment></comment><comment><USER47> You've been managing servers for quite some time but you've never considered the security implications of hosting all of your sensitive data on someone else's computer? You say you're not trolling but I genuinely don't see how those two facts are compatible, except if you work in an industry where the data just doesn't matter. If that were the case though, you shouldn't feel compelled to judge what banking or medical institutions are doing.

<comment><AUTHOR> We use a public cloud, and believe me: we do consider the security implications. We veeery much do.

</comment></comment><comment><USER145> Quebec power and internet pricing is really competitive. For residential services I pay $0.06/kw + $80/mo for 1Gbps fiber with 1ms ping to 126.96.36.199 (USD).

As a result, I run a power-hungry Dell r610 with 24 cores and 48GB of ram with 20+ services on it for many different aspects of my company. All the critical stuff runs on DigitalOcean / Vultr, but the 20+ non-critical services like demo apps, CI/CD, cron workers, archiving, etc. run for <$200/yr in my closet.

<comment> blaser-waffle This is also why there are a lot of data centers in the Greater Montreal area, FWIW.

</comment></comment><comment><USER146> We are small team startup and i am personally annoyed about pricing of a clouds.

I have a 3 smallish VM for build server + managed SQL. It cost 500$/mo. It doesn't make sense. Having my own VMs on ESXi makes everything very different - most of the time this VMs do nothing, but you want to make them performant from time to time, so there are a plenty of resources because all other vms are too mostly IDLE.

In cloud they are billed as if they are 100% loaded all the time.

I am not really satisfied with latencies and insane price for egress traffic. I just can't do backups daily since it could cost whooping 500$/mo just for the traffic. This is just insane, i can't see how it could scale anywhere for B2C market. For B2B it might work really well though since revenue per customer is much higher.

We are not moving to our own DC, but just keep realtime stuff in the cloud, but anything that is not essential is being moved somewhere else. Bonus is that you need off-site backups anyway in the case if cloud vendor will just ban you and delete all your data.

Startups might move fast and iterate, but if you don't have your own servers you always reduce your usage because it could grow fast effectively reducing your delivery capacity.

</comment><comment><USER147> Background: I consult extensively in the SaaS space and ask people this question all the time in the course of strategy reassessments, transactional diligence, etc.

1\. Physical control over data is still a premium for many professional investors. As a hedge fund CIO told me recently when I asked her why she was so anti-cloud migration, "I want our data to live on our servers in our building behind our security guard and I want our lawyer, not AWS's, to read the subpoena if the SEC ever comes for us."

2\. There are a lot of niche ERP- and CRM-adjacent platforms out there -- e.g., medical imaging software -- where the best providers are still on-prem focused, so customers in that space are waiting for the software to catch up before they switch.

3\. A lot of people still fundamentally don't trust the security of the cloud. And I'd say this distrust isn't of the tinfoil hat, "I don't believe SSL really works" variety that existed a decade ago. Instead it's, "we'd have to transition to a completely different SysAdmin environment and we'd probably fuck up an integration and inadvertently cause some kind of horrendous breach".

</comment><comment><USER148> For our case, the need is very specific as we are working with mobile apps.

Building iOS apps require macOS and even though there are some well-known "Mac hosting" services, none of them are actual cloud services similar DigitalOcean, Azure, AWS, etc.

So it is much less expensive and actually easier to scale and configure to host the Macs onprem.

(Off the record: If it is for internal use only, you can even stick in a few hackintoshes for high performance.)

</comment><comment><USER149> Quite frankly, "cloud" is a convenience and elasticity service at a steep premium, with downsides.

Contrary to popular belief, it does not in the slightest save you a sysadmin (most just end up unknowingly giving the task to their developers). And contrary to popular belief, the perf/price ratio is _atrocious_ compared to just buying servers.

For some of the loads I had been doing the math for, I could rent a colo _and_ buy a new beefy server every year with money to spare for the yearly cost of something _approximating_ the performance in AWS...

</comment><comment><USER150> It's the people cost, not the hardware.

Hardware is super cheap:

-A 40 slots rack, with gigabit fiber, dual power and a handful of public IP addresses, costs on average 10000€/y.

-A reconditioned server on eBay with 16 cores and 96GB of RAM costs 500€ (never seen them break in 3 years).

-A brand new Dell Poweredge with AMD EPYC 32 core and 64GB of RAM will cost 3000€.

-Storage is super cheap: 500GB of SSD costs 80€ (consumer stuff is super fine as long as you wisely plan between redundancy and careful load) and rotational disks are even cheaper. Never seen a rotational disk break.

Once bought, all of this is yours forever, not for a single month. You can pack very remarkable densities in a rack and have MUCH more infrastructure estate at disposal than you would ever afford on AWS.

The flip side of the coin is that you need operation expertise. If it's always you, then ok (although you won't be doing much more than babysitting the datacenter). Otherwise, if you need to hire a dedicated person, people is the most expensive resource yet and that should definitely be added to the cost of operations.

</comment><comment><USER32> A company which:

1\. Does "security" critical stuff (like affecting the security of people not data if breached)

2\. Which besides certain kinds of breaches has lowish requirements to performance and reliability (short outages of a view minutes are not a big problems, even outages of half a day or so can be coped with)

3\. Slightly paranoid founders, with a good amount of mistrust into any cloud company.

4\. Founders and tech-lead which have experience in some areas but toughly underestimate the (time)cost of managing servers them self, and that _kinda_ hard to do secure by yourself (wrt. long term DDOS and similar).

So was it a good reason? Probably not. But we still went with it.

As a site note while we did not use the could _we didn't physically manage server either_. Instead we had some dedicated hardware in some compute center in Germany which they did trust. So no "physical" managements securing etc. needed and some DDOS and network protection by default. Still we probably could have it easier without losing anything.

On the other side if you have a dedicated server hardware in some trusted "localish" compute center it's not _that_ bad either to manage.

</comment><comment><USER151> I did a startup with a co-founder in 1998, before cloud was a thing. We hosted at above.net first, then he.net following above.net's bankruptcy. Both were very good and we never had colo-related problems with either, though he.net was significantly cheaper.

We started with 2 white-box PC's as servers, 2 mirrored RAID1 drives in each. We added a 3rd PC we built ourselves: total nightmare. The motherboard had a bug where, when using both IDE channels, it overwrote the wrong drive. We nearly lost our entire business. Putting both drives on the same IDE channel fixed it, but that's dangerous for RAID1.

A few years in, we needed an upgrade and bought 5 identical SuperMicro 2U servers with hardware RAID1 for around $10K. Those things were beasts: rock solid, fast, and gave us plenty of capacity. We split our services across machines with DNS and the 5 machines were in a local LAN to talk to each other for access to the database server. The machines' serial ports were wired together in a daisy-chain so we could get direct console access to any machine that failed, and we had watchdog cards installed on each so that if one ever locked up, it automagically rebooted itself. When I left in 2005, we were handling 100's of request/s, every page dynamically generated from a search engine or database.

Of course it took effort to set all this up. But the nice thing is, you control and understand _everything_. Some big company doesn't just do things to you, you have no idea what is happening, and they're not talking. And if things do go south, you can very quickly figure out why, because you built it.

The biggest mistake we made was in the first few years, where we used those crappy white-box PCs. Sure, we saved a couple thousand dollars, but we had the money and it was a terrible trade. Night and day difference between those and real servers.

</comment><comment><USER152> > The more I learn, the more I believe cloud is the only competitive solution > today, even for sensitive industries like banking or medical. I honestly > fail to see any good reason not to use the cloud anymore, at least for > business. Cost-wise, _security-wise_ , whatever-wise. [emphasis mine]

Most people here seem to point out cost and utilization. I would like to offer another perspective: security.

I worked in both of these industries: finance ("banking", not crypto or PG) and medical (within a major hospital network). The security requirement, both from practical and legal perspectives, cannot be understated. In many situations, the data cannot leave an on-prem air-gapped server network, let alone use a cloud service.

It costed us more to have on-prem servers as we need a dedicated real estate and an engineering team to maintain. Moreover, the initial capital expenditure is high -- designing and implementing a proper server room/data center with climate control, power wiring, and compliant fire extinguishing system are not trivial.

</comment><comment><USER153> For business logic at a fairly large school the free and open tools that make the cloud so productive get used here a lot. We get to leverage commodity hardware and network infra very effectively for on-premises[1].

You have to have a good recovery plan for when equipment X's power supply fails but when deploy is all automated, it's very easy to overcome swapping bare metal, and easy to drill (practice) during off hours.

This makes it _much_ easier to meet regulatory compliance: either statutory or regulations your org has created internally (e.g. financial controls in-org, working with vulnerable people or children, working with controlled substances, working with sensitive intellectual property.)

Simply being able to say you can pull the plug on something and do forensic analysis of the storage on a device is an important thing to say to stakeholders (carers, carers families, pupil parents.)

I'm so grateful to be living in the modern age when "cloud" software exists[2], but I don't have to be in the cloud to use it.

The downside: you need trained staff and it's completely inappropriate if you need any kind of bandwidth, power consumption, or to support round the clock business (which we do not because, out here on the long tail, we work in single city so still have things like evenings weekends and holidays for maintenance!)

— [1] Premise vs premises is one of those "oh isn't the English language awful" distinctions. Premise is the logical foundation for some other theory ("the premises for my laziness is that because the sky is grey it will probably rain so I'm not going to paint the house") where as the premise_s_ means physical real estate property ("this is my freshly painted house: I welcome you onto the premises".)

[2] Ansible, Ubiquiti, arm SBCs like raspberry pi, Docker, LXC, IPv6 (making global routing for more tractable, IPv4 for the public and as an endpoint to get on the VPN.)

</comment><comment><USER154> If you interested, here is an article about how Bank of America chose to build out its own cloud to save about 2 billion a year.

[https://www.businessinsider.com/bank-of- americas-350-million...](https://www.businessinsider.com/bank-of- americas-350-million-internal-cloud-bet-striking-payoff-2019-10?op=1)

</comment><comment><USER155> Cost.

We recently moved one rack to the different DC in the same city and used Digitalocean droplets to avoid downtime. Services running on Linux were migrated without high-availability (e.g. no pgsql replication, no redis cluster, single elasticsearch node...) and we just turned off Windows VMs completely due to licensing issues and no need to have them running at night.

The price of this setup was almost 4x higher than what we pay for colo. Our servers are usualy <5 years old Supermicro, we run on Openstack and Ceph (S3, rbd) and provide VPNaaS to our clients.

AWS/GCP/Azure was out of question due to cost. We considered moving Windows servers to Azure with the same result - the cost of running Windows Server (AD, Terminal, Tomcat) + MS SQL was many times higher than the price of colo per month. It is bizarre that you can buy the server running those VMs approximately every 3 months for the Azure expenses (Xeon Platinum, 512GB RAM).

</comment><comment><USER156> There is no cloud solution that lets you ship and forget a system.

Come in 15 years ? It still works. Is that possible with cloud even in short periods, like 2 years ? No.

Will it ever be possible? No.

Thats primary reason for me. I can use cloud only for stuff that are nice but not mandatory to have for service to work, like status page.

Plus, work is more enjoyable then using somebodies else stuff.

</comment><comment><USER73> Cloud falls over if you have a lot of data egress. I don't work on those types of workloads (mainly in PCI and a little bit of HIPAA) so I stick to cloud for the sheer convenience factor (it's easier to fix a security group than having to drive to the office and plug in somewhere like I had to earlier today). Dealing with hardware has become a very specific skill set. I have it, but I don't enjoy it, so I don't look for it.

I still have to build physical networks occasionally (ex: we are building a small manufacturing facility in a very specific niche that's required to be onsite for compliance reasons) but the scale is so small that I can get away with a lot of open source components (pfsense netgates are great) and not have to use things that are obnoxious to deal with (if I never have to deploy cisco anything ever again I won't be upset).

</comment><comment><USER157> I have a friend who is an IT manager for a large chain hotel in the Caribbean.

I keep asking him about why they still use on premises equipment and it boils down to:

* Cost for training / transitioning + sunk cost fallacy * Perceived security risk (right or wrong) * IT is mostly invisible and currently "works" with the current arrangement, why change?

</comment><comment><USER158> We're (Kentik) a SaaS company and one key to having a great public margin is buying and hosting servers. In our case, we use Equinix in the US and EU to host, with Juniper gear for routing/switching, and the customary transit, peering at the edge.

One secondary factor is that we've only monotonically increased, and it's way cheaper to keep 10%-15% overprovisioned than to be on burst price with 50%+ constant load.

But the simplest math is - we have > 100 storage servers that are 2u 26x2tb flash, 256gb RAM, 36 cores. They cost $18k once, which we finance at pretty low interest over 36 months (and really last longer than that). Factor in $200-400/mo to host each depending (I think it's more like $200, but it doesn't matter for the cloud math).

That same server would be many $thousands/month on any cloud we've seen. Probably $4-6k/mo, depending on the type of EBS-ish attached. Or with the dedicated server 'alternate' they are moving to offer (and Oracle sorta launched with).

It'd be cheaper but still > 2x as expensive on Packet, IBM dedicated, OVH, Hetzner, Leaseweb (OVH and Hetzner the cheapest probably).

Three other factors for us:

1) Bandwidth would be outrageous on cloud but probably not as outrageously high as just the servers, given that our outbound is just our SaaS portal/API usage

2) We'd still need a cabinet with router/switch infra to peer with a couple dozen customers that run networks (other SaaS/digital natives and SPs that want to send infrastructure telemetry via direct network interconnect).

3) We've had 5-6 ops folks for 3 of the 6 years, 3-4 for the couple years before that. As we go forward, as we double we'll probably +1. It is my belief that we'd need more people in ops, or at least eng+ops mix, if we used public cloud. But in any case, the amount of time we spend adding and debugging our infra is really really really small, and the benefit of knowing how servers and switching stuff fails is huge to debugging (or, not having to debug).

All that said - we do run private SaaS clusters, and 100% of them are on bare metal, even though we _could_ run on cloud. Once we do the TCO, no one yet has wanted to go cloud for an always-on data-intensive footprint like ours.

Good luck with your journey, whichever way you go!

And happy to discuss more, email in profile

</comment><comment><USER159> I think when the economy is hit hard a lot of companies are going to have to look at what they can do to make sure they remain profitable since investor appetite has changed. This means some companies will have to look at what is being spent on cloud providers. Renting RAM by the hour makes sense if you are optimistic about future revenue growth, but if the market changes and you have to worry about how you can sustain profits while just keeping your current customer base then this makes a lot less sense. The cloud vs on prem argument also really includes all the things in between, e.g. colo, managed servers, VPSes and also better tooling to manage your own VM and container clusters on these, which I think will now get increased attention and competition when people are considering alternatives to the big cloud providers in order to bring down costs.

</comment><comment><USER160> I picked up a few refurbished dell servers off Ebay for super cheap a while back - and usually use these with Cloudflare Argo tunnels to host various servers. However, since these are just sitting on the floor next to my desk, usually rely on cloud for any applications with high uptime requirements.

Recently though, I've been working on some distributed systems type projects which would allow these servers to be put in different physical locations (and power grids), and still continue to operate as a cohesive whole. This type of technology definitely increases my confidence in them being able to reliably host servers. I wouldn't want to be reliant on the cloud for large scale services though, from my understanding you can get some crazy cost savings by colocating some physical servers (especially for large data storage requirements).

</comment><comment><USER161> AWS is one of most expensive options, and far not perfect, I can't understand why people consider it as default choice... To compare - dedicated server on hetzner (core i7, 32gb ram, 20tb network) is cheaper than medium VM on AWS. If the product is growing, cloud cost can quickly become the biggest expense for company. Than it will make sense to spend some time make things run in more cost effective way

I think if you choose cloud hosting that costs about same as renting dedicated server plus settings virtualization by yourself - than it's a fair choice (can check on [https://www.vpsbenchmarks.com/](https://www.vpsbenchmarks.com/) or similar)

Another sweet configuration is dedicated servers with kubernetes: good user experience for developers, easy to setup and maintain, easy to scale up/down

</comment><comment><USER162> My employer is a mid-sized university. Cost is the main issue.

Our environment is a mixture of in-house developed apps and COTS. Until recently, our major COTS vendors didn't have cloud solutions. Now they have cloud solutions but they're far too costly for us to afford. So we need to keep them in-house and continue to employ the staff to support it.

Our in-house apps integrate the COTS systems. Our newer apps are mostly in the cloud. But the older ones are in technologies that need to stay where the database server is, which is in our server room for the reason stated in the last paragraph. Rewriting the apps isn't on our radar due to new work coming in.

Historically, outsource vs. in-source seems to ebb and flow. The clear path is usually muddied when new technologies come out to reduce cost on one side or the other.

</comment><comment><USER163> My SO's brother works in the studio video recording industry, and is a very IT-savvy guy. We had a long discussion last holiday season about the state of cloud adoption in that industry. He told me (this is obviously secondhand) that most of the movie industry is not only off the cloud, but exclusively working in the realm of colocating _humans_ and data (footage).

This is for many reasons. The one that comes back to me now is that the file sizes are HUGE, because resolution is very high, so bandwidth is a major concern. Editors and colorists need rapid feedback on their work, which demands beefy workstations connected directly with high bandwidth connections to the source files. Doing something like this over a long distance network (even if the storage was free) would be prohibitively expensive, and sometimes literally impossible.

So the write loads are basically the antithesis of what cloud optimized for: "random sequential reads of typical short length, big append only writes". The big production houses (lucasarts famously) are also incredibly secretive about their source material, and like to use physical access as a proxy for digital access.

It leads to some seemingly strange (to me as a cloud SWE guy) decisions. He pretty much exclusively purchases top of the line equipment ( _hard drives /ssds_), and keeps minimal if any backups for most projects because there simply isn't any room. It's a recipe for disastrous data loss, and apparently it's something that happens quite often to this day. It's just extremely prohibitively expensive to do version control for movie development.

I don't know to what extent cloud technologies can solve for this domain. I asked him if Netflix was innovating in this area, since they're so famously invested in AWS, but he said that they mostly contracted out the production stuff, and only managed the distribution, which makes sense. The contractors don't touch the cloud at all, for the most part.

Again most of this is secondhand, I'd be curious to hear more details or reports from other people in the movie industry.

</comment><comment><USER164> We moved all AWS servers to a colo. Saving 80% of the cost.

</comment><comment><USER165> There are very few niches where the cloud makes sense: Namely, where you are either too small to benefit from a single server and a single entry-level IT guy (think three or four person companies with low need for technical competency), or where you are expecting rapid growth and can't really rationally size out your own hardware for the job (in this case, the cloud is useful initially, leave it later once your scale is more stable).

In every other case, you are paying for the same hardware you could buy yourself, plus the cloud provider's IT staff, plus your own IT staff which you likely need anyways to figure out how to deal with the cloud provider, _and_ then the cloud provider's profit margin, which is sizeable.

</comment><comment><USER166> Surprised no one mentioned a third option: cheap vps providers like digitalocean or vultr. They've become real contenders to big clouds recently, providing managed databases, storage and k8s. And their bandwidth costs are close to what you'd get in colo.

</comment><comment><USER167> My company cannot run our infrastructure in the cloud because we do performance/availability monitoring from ~700 PoP's around the world.

Not running our infrastructure in the cloud is part of our value proposition.

Our customers depend on us to detect and alert them when their services go down. We _have_ to be operational when the cloud providers are not, otherwise we aren't providing our customer with a valuable service.

Another reason we don't run in the cloud is because we store a substantial amount of data that is ever increasing. It's cheaper to run our own SAN in a data center than to store and query it in the cloud.

The final reason is our workloads aren't elastic. Our CPU's are never idle. In that type of use case, it's cheaper to own the hardware.

</comment><comment><USER168> Many of the customers of my previous employer had hardware on premises, including the customer that I was handling.

It had both compute and storage (netapp). It had two twin sites in two different datacenters. The infra in each site consisted basically in six compute servers (24c/48t, 128gb ram) and netapp storage (two netapp heads per site + disk shelves).

Such hardware has basically paid itself across its seven or eight years of life, and having one of the sites basically in the building meant basically negligible latency.

The workload was mostly fixed, and the user base was relatively small (~1000 concurrent users, using the services almost 24/7).

It really checks all of the boxes, does all it is supposed to do and in a cheap manner.

</comment><comment><USER169> What do you mean by "servers"? Anything that isn't a client machine, or just customer-facing infra?

We have a pool of 15 build servers for our CI. They run basically 100% cpu during office hours and tranffer terabytes of data every day. They have no real requirements for backup, reliability etc, but they need to be fast. If I run a pricing calculator for hosting those in the cloud it's ridiculous. We are moving source and CI to cloud, but we'll probably keep all the build machines on-prem for the foreseeable future.

For customer facing servers the calculation is completely different. More traffic means more business. Reliability, Scalability and backup is important and so on.

</comment><comment><USER170> I would like to add, that even for small teams/projects the cost is the reason. We have small business project, with only one server, few hundred customers, with varying traffic (database sync between thick clients, web portal,..). We were considering cloud, but with the features we needed (few different databases, few APIs,..) it would be cca $1000/month (with reasonable response times - could be cheaper but terribly slow). Having our own on premise server, the price is back just after few months and then just the minimal cost of connectivity, energy and occasional maintenance.. it just didn't make any sense for us to choose cloud.

</comment><comment><USER171> I worked in Switzerland and a reason to use on premise here is _Security_

Many detail banks, asset management company or high security company refuse to use any public Cloud.

They want to have a strict and traceable list of people who have physical access to their hardware.

This in order to control any risk of dataleak [1].

In practice they use generally on-premise installation. They ren space in a computer center and own there a private cage monitored with multiple cameras. Meaning they know exactly anyone touching their hardware and enforce security clearance for them.

[1]: [https://en.m.wikipedia.org/wiki/Swiss_Leaks](https://en.m.wikipedia.org/wiki/Swiss_Leaks)

</comment><comment><USER172> My last three employments have had me and my team build three different platforms with three different "providers". Many lessons learned!

Chronological order:

1\. E-commerce, low volume (1000-5000RPM), very high value conversions, highly localized trade.

We built an on-prem stack using hasicorp here. This place had on-prem stuff already in place, the usual vendor driven crap - expensive hypervisor, expensive spoffy SAN, unreliable network. Anyway, my platform team (4-5 guys) built a silo on commodity hardware to run the new version of this site. This is a few years back, but the power you get from cheap hardware these days is astounding. With 6 basic servers, in two DCs, stuffed with off the shelf SSDs we could run the site and dev teams no problem. Much less downtime compared to the expensive hyperconverged blade crap we started on at basically no cost. There's a simplicity that wins out using actual network cables and 1u boxes... LXC is awesome btw! Using "legacy" vmware, emc, hp etc for non-essential on- prem? Cloud is tempting!

2\. Very high volume (Billions of requests per day), global network. AWS. Team tasked with improving on-demand scalability. We implemented kubernetes on AWS and it really showed what it's about! After 6-7 months of struggle with k8s < 1.12 things turned around when it hit 1.12-1.13-ish we got it to act how we wanted. Sort of, at least. Cloud just a no brainer for this type of round-the- clock, elastic workload. You'd need many millions up-front to even begin building something matching "the cloud" here. Lot of work spent tweaking cost though. At this scale cloud cost management is what you do.

3\. Upstart dev-shop. No rpm outside dev (yet). Azure. About 30 devs building cool stuff. Azure sucks as IaaS, they want you to PaaS that's for sure. Cloud decision had been made already when I joined. Do you need cloud for this? No. Are there benefits? Some. Do they outweigh the cost? Hardly. In the end it will depend on how and where your product drives revenue. We pay for a small local dev datacenter quarterly, which i find annoying.

Just some quick thoughts off the top of my head (on the phone so excuse everything).

Happy to discuss further!

</comment><comment><USER173> Our software runs in core (mobile) network systems, but that's at our customers. We ourselves have a rack in our office that runs things like Git repos, project management, virtual machines for development / testing, build servers, and instances for trainings.

We're concerned about corporate espionage and infiltration, so we can't trust our servers being out of our sight. Most people don't have the code on their physical machines either; I'm a cocky breath of fresh air in that regard in that I prefer my stuff to run locally instead of the (slow, underpowered) VMs, I trust Apple's encryption a lot.

</comment><comment><USER174> I work for a US Stock Exchange, and some of the technologies that we rely on are not permitted in the cloud. The performance metrics we need are usually only achievable on highly tuned bare metal deployments as well so cloud is usually not an option. I guess it really depends on your workload, but I think there is a very healthy amount of production being deployed and worked on businesses own datacenter/private clouds.

</comment><comment><USER175> We have mostly switched to, or are in the midst of switching to, the cloud.

Services that we will continue to run on-premises (as an exception to that rule) are some machine learning _training_ clusters (where we need a constant, high-level amount of GPU and cloud provider pricing for GPU machines is very far off the mark of what you can build/run yourself) and some file shares for our production facilities where very large raster files are created, manipulated, sent to production equipment, and then shortly afterwards deleted.

Most everything else is going to the cloud (including most of our ML _deployed model_ use cases).

</comment><comment><USER176> this assumption doesn't consider threat models where the vendor could be part of your problem. E.g. if you're based in country A and work on super secret new Tech for an emerging industry, then hosting in country B may not be an option.

Imagine a company in Europe that decides to host it's files on Alibaba Cloud in the US.

Imagine the US Department of State hosting it's files with Google.

Imagine an energy company working on new reactor Tech, ...

Imagine a Certificate Authority which has an offline store of root certificates which need to come online to sync twice a day.

Imagine cases where you need a hardware HSM.

Then there is also Cost as others have pointed out. AWS cost structure is so complex that whole business models[1] have sprung up to help you optimize the price tags or reduce the risks of huge bills. that's right: you need to have a commercial agreement with another partner that has nothing to do with your cloud just to work around aggressive pricing. The guy who started this ~2 years ago has grown to 40+ people (organically), is based in Switzerland and is still hiring even in this recession. It should give you an idea of how broken the cloud is.

Lastly there is also the lock-in. All the hours that you have to sit down and learn how the AWS IAM works is wasted once you decide to move to another cloud. The cost for learning how to use the 3rd party API is incurred by you not the cloud vendor. For people who think lock-in isn't much of a problem remember your whole hiring strategy will be aligned to whatever cloud vendor you're using (look at job description that already filter out based on AWS or GCP experience). Lock-in is so bad that for a business it is close to the rule of real-estate (location, location, location), only that it's to the advantage of the cloud vendor not you as the customer.

[1] optimyze.cloud

[2] _" I have just tried to pull the official EC2 price list via JSON API, and the JSON file is 1.3gb"_ [https://twitter.com/halvarflake/status/1258161778770542594](https://twitter.com/halvarflake/status/1258161778770542594)

</comment><comment><USER177> If you have a lot of data but not a lot of users, it's prohibitively expensive to pay monthly hosting and network egress fees when you can buy cheap hard drives, use ZFS, and whatever server protocol you desire.

</comment><comment><USER178> I notice all the banks I'm working for are moving to the cloud. A few years ago they all had their own data centers, sometimes really big, well-designed custom data centers. But they're all moving to the cloud now.

I've personally been wondering whether that's wise, because financial data and the handling of many banking processes are a bank's core business. It makes sense that a bank should be in control of that. And it needs to obey tons of strict banking data regulations. But apparently modern cloud services are able to provide all of that.

</comment><comment><USER179> Cloud costs way way more compared to on-prem solution for my company.

We need random access to about 50TB of files, and quite a decent number of VMs.

For storage on-perm vs cloud: buy was cheaper to have after 3(!) months.

For VMs(some of them could be containerized though): 1 year

It was cheaper to buy second-hand decent server, slap SSDs and just install a decent hypervisor. Those costs also include: server room, power usage, admins etc.

We do use cloud backups for the most important stuff.

Cloud is cheaper if your business is a something that is user-based - as in you might need to scale it, hard.

If you aren't doing anything like that it is absurdly expensive.

</comment><comment><USER180> Can you recommend dedicated hosting provider that: 1\. Has US, EU and Asia regions. 2\. Let you rent servers per month. 3\. Has decent pricing. Not premium, doesn't have to be low cost. I expect excellent egress pricing and 1/2-1/4 cost for CPU compared to the cloud. 4\. Reliable network. GCP premium network is very good. How does dedicated providers and VPS providers (Linode and DO) compare? 5\. Easy to use management and dashboard. Experienced really bad dashboards and hard to use Java tools to install and manage dedicated servers.

</comment><comment><USER181> \- cost, as well evidenced in other comments here. The hyperscalers are orders of magnitude more expensive than dedicated hosting or using collocation providers.

\- lock in, all the hyper scalers want to sell you value add services that make it hard or impossible to move away.

\- concentration risk, hyper scale providers are a well understood target for malign actors. It's true they are better protected than most.

\- complexity, if you think about how little time the hyperscalers have been operating in comparison with corporate IT they have created huge technical debt in the race to match features.

</comment><comment><USER182> We do a hybrid approach which I think makes sense for a lot of companies. Our mission critical stuff runs in the cloud, but anything that has to do with staging environments and development we do on-premises. It's pretty easy to host yourself, if you have a couple of decent engineers looking after it (depending on scope, of course!).

Redundant power, redundant internet connections, and a few racks of Dell servers and gigabit switches. Why did I mention Dell? They just don't seem to die. We used HP for a few years but had a few bad experiences.

</comment><comment><USER183> Security (we're a bank)

<comment><AUTHOR> How is the cloud less secure than your on-prem servers? I would argue that it's easier to keep track of all the threats with the tools available from big cloud providers.

<comment><USER184> The big question is - how much do you trust these providers.

<comment><USER185> I would guess that approximately zero banks own data centers that are operated by their own employees. There might be a few exceptions to this, but the reality is that most banks don't view technology as part of their core business. So this largely gets outsourced to IT consulting firms like Infosys, IBM, Wipro, etc.

The big question is - how much do you trust _these_ providers, and do you think they are more competent at security than Amazon/Google/Microsoft?

<comment><USER32> Or formulate it differently:

1\. Trust a provider which whole existence relies on trust and which you can audit or at last cross-check the audit and security processes (as a Bank you are normally not a small customer).

2\. Trust a provider which might 1st be a potential competitions in some business fields. 2nd is so big that it can easily afford losing you. 3nd for the same reason doesn't allow you any insights into there internal processes. etc.

Plus many of the banks having their own hardware also have their own IT team. So it's often about trusting your own people. I mean either you keep your it or you outsource _and_ go into the cloud. Keeping local servers but outsourcing IT at the same time seems kind not very clever tbh.

</comment></comment></comment></comment></comment><comment><USER186> Cost is the sole reason for us as well. We have ~600-700 dedicated servers around the globe, and generate a lot of egress traffic (~20PB/mo). We last ran the figures a year or so ago, and it'd cost us around 13-15x in network costs alone.

A common thread of a lot of the replies to this post is network traffic costs. If one of the cloud providers can figure out a way to dramatically (and I mean at least 10x) reduce their network transfer pricing, then I think we'll see a second wave of companies adopting their services.

</comment><comment><USER187> It is so so so much cheaper. We moved to AWS and tried setting up our servers with specs that were comparable to what our physical servers had been. We just about died after the first month. Our bill was higher for one month than for multiple years running our physical servers. We had to way dial them back to get the run rate down to a reasonable number. Now we have serious buyers remorse. Performance is terrible. The cost is still more per month than we ever had with our physical servers by a large amount.

</comment><comment><USER188> We do not use the cloud. We operate (24/7) facilities in remote locations where we do not have super reliable internet connection (we do have redundant links including three different fibres on distinct paths plus radio links but still). For this reason alone nothing critical can be in the cloud. In our experience, however, cloud offerings are not that cheap compared to purchasing and operating your own machines. Besides, one still needs sysadmins even when operating infrastructure in the cloud.

</comment><comment><USER189> On-prem can make sense where your computing needs are constant and predictable. You can arrange to have exactly what you need, and the total cost of buying and operating it may be less than it would be to get comparable resources in the cloud.

If your computing needs vary over time, then provisioning on-prem for peak load will mean that some of your resources will be idle at non-peak times. It may be cheaper to use cloud resources in cases like these, since you only need to pay for the extra capacity when you need it.

</comment><comment><USER190> Am the CTO of a small company making payroll software. We don't have on premise servers but we currently are moving away from azure to renting VPSs from a local provider.

The cost benefits are huge and since our app is mostly a normal web app we dont need that many fancy cloud things. And i dont see us needing it in the future.

I really dont understand why a company doing similar things would wanna go the could route. Its so damn expensive and its not always easy to use and setup.

</comment><comment><USER191> We are software development company. Most of our compute/storage needs are for build/test cycles. We recently bought ~100K worth of additional hardware to migrate some of that work off AWS. The storage / virtualization is done using Ceph & OpenNebula. Including colocation/electricity/networking costs, the investment will pay for itself in ~9 months. If I would include deployment costs & work to migrate the jobs off the AWS -- it will pay for itself in 11 months.

</comment><comment><USER192> PII and draconian security policies. We are not a tech company, so we can't fine-tune or have nuanced policies, we just have to build a wall around everything . In our web password recovery process, we can't tell people if their login was correct or not, because that might help a brute force attacker infer they got that right. Even though we have that rate limited anyway. I don't know why we can't just tell people the login was found or not found like banks etc.

<comment><USER193> What industry? It's good practice to not share information like that in any context, since attackers that have a bank if email addresses that are trying to figure out which if a few reused passwords might be used for a given website would have a harder time if they dont even know if the email has an account with a given website, or if an alternate email is used instead etc.

</comment></comment><comment><USER194> Not a company but a personal project:

[https://github.com/krisk84/retinanet_for_redaction_with_deep...](https://github.com/krisk84/retinanet_for_redaction_with_deepstream/wiki)

I haven't analyzed the TCO yet but the bandwidth costs alone from hosting my curated model of 100GB in the cloud (Azure blob) have greatly exceeded my total operation spend from downloading the entire dataset and running training. By an order of magnitude.

</comment><comment><USER195> The smartest approach is to be able to run anywhere, which is increasingly practical due to VMs, Docker etc.

(At least funded) startups should start with the cloud as speed to completion is key, but can later optimize for cost.

Elasticity of the cloud is also great, dealing with peak demands dynamically without having to purchase hardware.

I'd suggest larger companies to use at leas two cloud vendors to add resilience (when MS Teams went down, so did Slack - I was told they both use MSFT's cloud).

</comment><comment><USER196> Because Elon doesn't believe in the cloud. We were one of the few teams that got AWS access, and we were told not to rely on it too much because it's temporary...

</comment><comment><USER197> Just talking about my side project here: a local server + ngrok is easier to use and cheaper than anything in the cloud.

In general, I would say any noncritical system I would host on-prem.

</comment><comment><USER198> Cost. We're a small company, five person team and we need a development environment. All our stuff is built around VM and Docker. So scores of little test nodes that get run, or beta environment in the cloud was costly (>$300/mo base, sometimes 3x). For $1000 we put a box in the office we all VPN to that runs all the necessary VM and Docker. The variable, sometimes expensive cost for Test/Beta in cloud was replaced with low fixed cost.

</comment><comment><USER199> Sure, for tiny intel servers it makes sense to rent vms. It won't start to hurt until 1.5 years later at which point the project needs to become profitable anyways.

I run a couple of on-premise xeon gold machines with 96 gb ram and 40+ cores on each. Their total purchase cost was the monthly cost of renting them on the cloud. Also, you will never get the full benefit of the servers you use unless they are dedicated instances with no virtualization layer.

</comment><comment><USER200> It depends on what you have to do. If your stack includes a series of microservices/monoliths connected to the typical OLTP DB then you might very well sit entirely on cloud. Things change when you need heavy lifting like having big ElasticSearch or ClickHouse clusters, or any other operation that requires heavy CPU and high RAM capacity. In that case using providers like Hetzner can cost you 1/10 of the bill compared to AWS.

</comment><comment><USER201> Manufacturing plants across the world controlling production lines: no way to go in the cloud for that. We put the reporting data in the cloud, no problem there.

</comment><comment><USER202> "Server" is such a broad term. In our case, aside from cost as others already mentioned, the distance and latency is very important. The servers must be located as close to client devices as possible and reasonable, and they are synced with clients using PTP to microseconds (depends on the actual distance and topology). Cloud is a no go, and we a using bare metal K8S for graceful handling of failures and updates.

</comment><comment><USER203> Most of my equipment has physical interfaces, video and audio in and out

Some equipment is very latency sensitive -- I'm talking microseconds, not milliseconds.

More generic tasks need easy access to that specialist equipment (much of which doesn't quite grasp the concept of security)

Given that we therefore have to run a secure network across hundreds of sites on multiple continents, adding a couple of machines running xen adds very little to the overhead.

</comment><comment><USER204> Well, for one thing: jobs. It takes a lot of IT man-power to manage an on- premise solution, especially when you run everything on-premise. Just imagine if the CTO were to switch the company to a cloud based solutions, it would save the company millions of dollars but also it would mean cutting a lot of jobs. Gov departments that use on-premise do so for security reasons and to maintain existing jobs.

</comment><comment><USER205> Amazon/Azure/GCP - they're _businesses_. They charge you 50-70 points of margin in order to run your computers. If you're R&D-limited, that's not important to you, but if you're a more mature company that's cost-limited then it matters a lot. If it's a core part of your business, it'll never be cheaper to pay another company to profit from you. Period.

</comment><comment><USER206> Yes, some of the leadership thinks that they can build a better cloud than MS or AWS. It is pretty hilarious to watch how spectacularly they fail.

[https://forrestbrazeal.com/2020/01/05/code-wise-cloud- foolis...](https://forrestbrazeal.com/2020/01/05/code-wise-cloud-foolish- avoiding-bad-technology-choices/)

</comment><comment><USER207> We are going to the cloud, but you have to be careful. With on-prem the limit of the cost is the server. Someone writes inefficient code and it just doesn't work. In the cloud there are 1000 ways to overspend and the vendors purposefully don't make it easy to track or keep things under control.

It's kind of like outsourcing. If you don't know what you are doing, cost goes up and quality goes down.

</comment><comment><USER208> Cloud is only good if you don't care about costs and plan to scale without looking back.

For example, building a Ceph based Software Defined Storage with croit.io for S3 comes for 1/10 to 1/5 of the AWS price in TCO. Same goes for any other product in the cloud.

If you only need it a short time up to 6 months, go to the cloud. If you plan to have the resources longer than 6 months go to Colocation.

</comment><comment><USER209> Cloud will continue to dominate, even if it's more expensive. Why? Because the best companies are able to focus on what makes them special, and outsource the rest.

Cost and security are important, but they may not be most important. In a business, the scarcest commodity is FOCUS. By outsourcing anything that isn't core to your product, you can excel at what differentiates you.

</comment><comment><USER210> On top of what others have said, when outside the US the Cloud Act has been a big one for most previous companies I worked for.

Using AWS / Azure / Google Cloud (even using datacenters from your own country) implies that the US government can access your data at will.

As soon as you treat sensitive information, especially related to non-US governments, this becomes a blocking factor.

</comment><comment><USER211> In some industries, cloud is not an option. For example, certain privacy laws like HIPAA preclude uploading data to third parties, in which case, you need things to be on-prem. There are also a lot of places in the world where internet access is limited. Sometimes you need to solve problems beyond the simple "web saas in cloud" use case

</comment><comment><USER212> Own servers cheaper than cloud. We calculated two years ago, it was up to 4x for HPC. Best cost-wise option is used servers.

</comment><comment><USER213> Legacy seems to be missing from the comments. Before the advent of Cloud IaaS (Infrastructure as a Service) a very large ecosystem of On-Premise hardware and software flourished. The question needs to be considered in the context of greenfield vs brownfield systems as the trade-offs involved differ drastically.

</comment><comment><USER214> It all depends. Outside of SaaS, If you have a mature data center operating model and truly understand your costs, there won't be a strong cost savings story for many types of workflows.

If you suck and don't understand costs, or don't automate, or spend a lot of time eating steak with your Cisco team, you'll save money... at first.

</comment><comment><USER215> There's a fake believe in my company that all the data must be on premise because of privacy concerns.

We've never used cloud services and we do not want to use it.

Some are saying it's a matter of costs, but you know? For a dual node server (hot standby) we were asked 120K € + 50K € only for configuration fees.

</comment><comment><USER216> Cloud servers tend to hide their limitations behind payment tiers which makes it hard to really know how far you can push things. Also there are various turns, conditions, cache rules, change management strategies that are hidden when dealing with someone else's constantly changing box of magic.

</comment><comment><USER217> We install in to remote locations - you can't access the cloud if your connectivity is down so local resources is a hard must.

Though we have adopted something close to an "Edge Computing" solution... I guess it comes down to "Why not both?" :)

I think it also depends on your definition of "server"

</comment><comment><USER218> Just want to say I love threads like these. I hope one day where I work we can have two or three obscenely beefy servers and be done with it. I'm planning for something similar probably Q2 2021 as our expenses grow too large on a manged hosting platform like Heroku/Render/Aptible.

</comment><comment><USER40> There are some things you just can't do in a VM.

The company I work for actually develops and hosts an AWS clone for the Linux Foundation, but with very specific requirements. They have special needs that requires baremetal machines and "real" networking between them across 6+ NICs per server.

</comment><comment><USER219> Not many services are available in gov cloud regions, so we're stuck with on- prem nearly everything.

</comment><comment><USER220> my 2cents: the 'big' clouds (AWS, GCP, Azure) and the big-brand clouds (Oracle, IBM, etc.) are attractive for BigCustomerCo because:

1\. Replace capital expenditure of in-house infrastructure + staff with OpEx that can be dialled down

2\. Get to benefit from the economies of scale that the cloud vendors get (those Intel CPUs get a lot cheaper when purchased in the 1000s)

3\. Get to leverage big shiny new tech like Big Data and AI that's 'advertised' as 'out-of-the-box'

My only concern really is that the big cloud players are all fighting for dominance and market share. What happens in the next 5-10 years time when they start raising prices? Different kind of lock-in - customers won't have the expertise in-house to migrate stuff back.

</comment><comment><USER221> Zseries are a huge deal to move to the cloud; it's just not worth the risk for most organizations.

</comment><comment><USER222> Yes. Cost of GPUs if you want them on regularly and reliably, and you don't need 2000 of them. We run small/mid-sized operations on-demand and the latency of spinning up instances is not competitive and the cost is outrageous to have them on standby.

</comment><comment><USER223> There is extra complexity with managing cloud based solutions. Logging in, setting up ssh keys. Ok it's all automatable but if I want a basic server set up for doing a simple task quickly it's often a lot easier to run it on an internal server.

</comment><comment><USER224> If your product doesn't run connected to the internet, it is difficult to make the case for cloud development. You need developers who understand hardware, and abstraction layer cloud provides is a handicap instead of an enabler.

</comment><comment><USER225> I think it depends on what is the alternative and what hardware you are buying. If you go for top of the line HPE or dell or Cisco hyperconverged stuff that allow you to be, sort of, your own cloud, you will end up with the same large bill.

</comment><comment><USER226> I work for a German company. They run their own DC (unfortunately, I'm not privy to precise numbers I could share, but we must be in the 1000s of hardware servers).

Why? Because our (German) clients don't _thrust US cloud providers_.

</comment><comment><USER227> Hybrid cloud is the moneymaker solution, but there are no out of box solutions for it.

</comment><comment><USER228> Gigantic Elastic Search cluster - according to Elastic the largest in Europe (as of 2 years ago) - used in production. Broke again and again on AWS. We needed more shards than AWS supports. Moved to bare metal again.

</comment><comment><USER229> When cloud computing started, alternative software deployment was complicated, after a decade, much has been improved. So ease of management is also one of the factor. Not all need state of the art data and AI.

</comment><comment><USER230> We use on-prem GPU nodes for training deep learning models. Our group estimated the cost vs. cloud and it was significantly cheaper to go on-prem. I can't speak to security-wise and whatever-wise though :)

</comment><comment><USER231> We colocate and as a small tech business, I can't imagine doing anything else. We don't spend more on payroll due to colocating and AWS/etc would easily double our annual non-payroll expenses.

</comment><comment><USER232> One good reason for your list: diversification. You don't want all the banking systems of a country all running on AWS. It's an unacceptable single point of failure risk for a country.

</comment><comment><USER233> Like many said, cost.

But also - legalities. Most cloud providers have very unclear rules and what exactly happen should you be in breach. For this reason, our business prefers to have most of the control.

</comment><comment><USER234> There are several levels, not just on-prem vs cloud. You can for example co- locate, rent dedicated servers, rent a single VPS, or put your website up on a web hosting provider...

</comment><comment><USER180> How do you manage installation and upgrades of dedicated servers? What do you use for block storage and object storage? Kubernetes and Ceph seems to be hard to setup and maintain.

</comment><comment><USER235> to control the costs. after the original purchase, the MRC is a fixed cost for hosting and for the network access. also, for the total control - the network, io, cpu performance does not change randomly. with better predictability our IT team can give more precise SLA.

we're not 100% on-prem, but aws, gcloud and azure are the worst examples of 3rd party hosting - unpredictable and with complicated billing. we're considering the alternatives to big 3 for the "cloud hosting"

</comment><comment><USER236> I work in a very large financial institution in small tech team. I would give an arm and a leg to not deal with soul sucking bureaucracy that is our IT department.

</comment><comment><USER237> This thread has been illuminating, a lot more non-cloud people than I thought! Drinking the cloud kool-aid had me thinking cloud was the only realistic way to go.

</comment><comment><USER238> My client's customers are geolocated (continental US) and their personal data is sensitive. So their server is in their own firewalled server closet.

</comment><comment><USER239> Client data confidentiality. I know it's a weak argument, but if the contract requires that we store data in-house, there's no choice.

</comment><comment><USER240> It's not secure if you have air gap requirements or issues with employees from another company technically having access to all of your data.

</comment><comment><USER241> I normally host on prem and also rent dedicated servers elsewhere as standby. Way cheaper than the cloud and full control the way I want.

</comment><comment><USER242> We run one of the largest Hadoop clusters on the planet. On-prem is very cost efficient if you're running jobs flat out 24/7

</comment><comment><USER243> Confidentiality. Protecting sensible data (avoiding letting some hostile obtain or even modify them) seems impossible on the cloud.

</comment><comment><USER244> We are staying on-premise for security and privacy reasons.

We can't store our data outside of the company (or even worse: outside of the EU)

</comment><comment><USER245> TCO.

<comment><AUTHOR> Which, all things considered, seems lower on the cloud. Could you give more details on this answer?

<comment><USER246> A 2u asus with 2 xeon silver, 64gb ram and 4 x 2080ti is maybe us$15k?

We'll use it for as long as its producing useful output. Lets say 5 years? Probably a little longer?

A 60bay 4u western digital with 14tb drives is under us$50k?

definitely got a dell md1280+1u server with 70x10tb 2yrs ago for under us$50k. Fully populated the following year..

A 2u dell with maybe 20 cores n 128gb ram each should cost less than us$10k.

And we just got 4 or 5 dell switches with 48x10g ports for 50-70k? I'm not sure.

What's the equivalent in the cloud?

<comment><AUTHOR> Are you keeping tabs on the cost of the building and associated security? Video cameras, locks... Also the manpower needed to install and maintain that equipment.

Similarly, your 70*10tb tells nothing. What's the redundancy on that, how much of that space did you lose to it, where do you store your backups and at what cost?

As for networking, having those switches is nice, but you still need your internet connection if you're serving anything online from this.

<comment><USER246> The manpower to run that stuff is already doing desktop support.

Storage is usually a stripe of 10 disks in raidz2 + 2 spares. Then we do a nightly zfs send..

The internet connection for the hosting is 100mbps. The users share 1gbps of internet.

We pay only for hardware. Rent and utilities. And the internet connections. Everything else we can do it ourselves.

I mean. Seriously. Whats the cost of running 100 x 4 x 2080ti in the cloud for a year? Or storing 500tb(1 instance only,no redundancy) ?

</comment></comment></comment></comment></comment><comment><USER247> If your customers still want the system to run air-gapped from the internet, cloud is basically off the table.

</comment><comment><USER248> Cost and data security, while cost actually weights more. It is simply not true that the cloud is cheaper.

</comment><comment><USER249> I work in a data science company with 30+ engineers. We've spent 80k dollars on GKE last month only..

</comment><comment><USER250> Yes just moved from 4 servers in AWS costing 700 usd/month to a single dedicated one that costs 40.

<comment><USER251> Just out of curiosity, what downtime is in acceptable range for that single server?

<comment><USER250> At night noone cares

</comment></comment></comment><comment><USER252> Cloud is even more muddled as a term in this space than usual (Hybrid/private added to the mix).

</comment><comment><USER253> >I honestly fail to see any good reason not to use the cloud anymore, at least for business. Cost-wise, security-wise, whatever-wise.

Problem is point of failure, many businesses need to be independent and having data stored in the cloud is a bad idea overall. Because it produces point of failure issues. Consider if we ever got a real nasty solar wind and the electric grid goes down, the more we rely on the internet and centralize infrastructure into electric devices, the more it becomes a costly point of failure.

While many see redundancy as "waste" in terms of dollars, notice that our bodies have many billions of redundant cells and that's what makes us resilient as a species, we can take a licking and keep on ticking.

Trusting your data to out-side sources generally is a bad idea any day of the week. You always want to have backups and data available in case of diaster, mishap, etc.

Like no one has learned from this epidemic yet. Notice that our economic philosophy didn't plan for viral infections and has forced our capitalist society to make serious adjustments. Helping people is an anathema to liberals and conservatives / republicans / democrats, so for void to come along and actually force co-operation was a bit tragically humorous.

As a general rule you need redundancy if you want to survive, behaving as if the cloud is almighty is a bad idea, I'm not sold on "software as a service" or any of that nonsense. It's just there to lull you into a false sense of security.

You always need to plan for the worst case scenario for surviveability reasons.

</comment><comment><USER254> At least a few years ago, tons of company were afraid of the 'cloud'.

It does change right now.

</comment><comment> blaser-waffle Big public clouds have a genuine purpose but there is a shit-ton of marketing and FUD being thrown around about them -- I'd bet my hat that's what this post is, given the phrasing up top.

I'm not a fan. In short:

Cost. CapEx and depreciation vs. OpEx. The numbers look amazing for ~3 years until the credits and discounts wear off. Then it's just high OpEx costs forever. Meanwhile I can depreciate my $10k server over time and get some cash back in taxes; plus it's paid for after a couple years -- $0 OpEx outside of licenses, and CentOS has no license cost.

Once you have significant presence in someone's cloud, they're not going to just lower costs either -- they've got you now. What in American Capitalism circa 2020 makes you think they won't find a way to nickle and dime you to death?

It's not going to reduce headcount, either. Instead of 14 devops/sysadmins, now I have 14 cloud admins, sitting pretty with their Azure or GCP certs. Automation is what's going to reduce those headcounts and costs, and Ansible+Jenkins+Kubernetes works fine just with VMware, Docker, and Cisco on- prem.

Trust. The Google Cloud just had a 12-hour outage -- I first read about it here on HN. AWS and Azure have had plenty of outages too... usually they're just not as open as Google is about it. You also have to trust that they won't get back-doored like what happened to NordVPN's providers, and that they're not secretly MITM'ing everything or dup-ing your data. We (and some of our clients) compete with some of the cloud providers companies and their subsidiaries, and we know for a fact that they will investigate and siphon any data that could give them an advantage.

Purpose. _We just don 't need hyper-scalable architecture._ We've got a (mostly) fixed number of users in a fixed number of locations, with needs that are fairly easy to estimate / build for. Outside of a handful of sales & financial processing purposes, we will never scale up or down in any dramatic fashion. And for the one-off cases, we can either make it work with VMware, or outsource it to the software provider's SaaS cloud offering.

If we were doing e-commerce -- absolutely. Some sort of android app? Sure, AWS or Azure would be great. But it's a lot of risk and cost with no benefit for the Enterprise orgs than can afford their own stuff.

</comment><comment><USER255> Bandwidth is the problem. You can't run a Youtube on any cloud provider.

</comment><comment><USER256> Security. And Independency.

</comment><comment><USER257> I work for AWS, so (in the most pedantic way possible) technically yes

</comment><comment><USER258> Yes, but we work in financial information so very weird requirements.

</comment><comment><USER259> We use servers on the cloud (IaaS) but still self-host all our apps.

</comment><comment><USER260> Local caching recursive DNS servers work best close to the clients.

</comment><comment><USER261> Yes online.net and the like companies save costs

</comment><comment><USER262> In physics, a gallon of water weights 8.34 lbs. (For this analogy, a gallon of water is a unit of work.) And the gallon of water weighs 8.34 lbs irregardless if it is sitting on my desk in a physical building, or on your desk, in the cloud. Same weight, same unit of work. same effort. For a brand new, greenfield application, the cloud is a no brainer. I agree 100%. But for legacy applications, and there are so, soooo many, the cloud is just some one else's computer. Yes, the cloud is more scaleable, yes, the cloud is more manageable, and yes, you can control the cpu/storage/memory/network in much finer amounts. But legacy applications are very complicated. They have long tails, interconnections to other applications that cannot immediately be migrated to the cloud. I have migrated clients off of the cloud, back to on premise or to (co-lo) local hosting, because without rewriting the legacy application, the cloud costs are simply too great.

The essence of IT is to apply technology to solve a business problem. Otherwise, why would the business spend the money? The IT solution might be crazy/stupid/complex but if it works, many business simply adopt it and move on. Now, move that crazy/stupid/complex process to the cloud and surprise, it is very, very expensive. So, yes, the cloud is better, but only for some things. And until legacy applications are rewritten on-premise will exist.

One final insight. The cloud costs more. It has been engineered to be so, both from a profitability standpoint(Amazon _is_ a for profit company) but also because the cloud has decomposed the infrastructure of IT into functional subcomponents, each of which cost money. When I was younger, the challenge for IT was explaining to management, the ROI of new servers, expanded networking, additional technology. We never quite got it right and often had it completely wrong. That was because we lacked the ability to account for/track and manage the actual costs of an on-premise operation. Accounting had one view, operations had another view and management had no idea really, why they were spending millions a year and could not get their business goals accomplished. The cloud changed all of that. You can do almost anything in the cloud, for a price. And I will humbly submit, that the cost of the cloud - minus the aforementioned profitability, is what on-premise organizations should have been spending all along. Anyone reading this and who has spent time in a legacy environment, knows that it is basically a futile exercise of keeping the plates spinning. On-premise failed because it could not get management to understand the value on in-house IT.

As I said, the costs are the same. A gallon of water weighs what it weighs regardless of location. It will be interesting to see, I predict the pendulum will swing back.

</comment><comment><USER263> Because we don't trust either amazon or google.

</comment><comment><USER264> Cost-wise is still a pretty compelling argument.

</comment><comment><USER265> Annual costs, backups, security and latency.

</comment><comment><USER266> on-premises not on-premise

</comment><comment><USER267> we are not zoomers</comment>