An update on last week's customer shutdown incident - <AUTHOR> https://blog.digitalocean.com/an-update-on-last-weeks-customer-shutdown-incident/ <comment><USER2> That's a really fair and reasonable response. Not sure what else people really expect here.

> The template used for response in account denial will be removed entirely. > If account access is denied during an appeal, which often is the case as > most appeals are true bad actors, the agent must create a reasoned response.

Glad this is seen as an issue and corrected.

IMO, this probably would have made this whole thing never escalate if a better response was previously in place for everyone.

Accidents, shotty support, whatever — all expected these days unless you have big cash money agreements in place.

But to kill an account of a responsive person with a gigantic middle finger email without reasoning was a pretty dumb process in place. You can see the email on the Twitter thread somewhere.

Glad it's fixed! Still a DO fan here

Edit: TALKING ABOUT THIS: [https://pbs.twimg.com/media/D76ocofXoAY_xB5.png](https://pbs.twimg.com/media/D76ocofXoAY_xB5.png)

<comment><USER3> > That's a really fair and reasonable response. Not sure what else people > really expect here.

The root cause of suspension is incomprehensible to me. They were suspended because they launched a set of instances and these were using 100% CPU. How is that unreasonable and cause for suspension?

I'm not a Digital Ocean customer, but if I were, I'd expect to be able to use the resources I bought without risk of being suspended. This is the root cause. It was compounded by incompetent customer support, but I really do not understand the suspension cause.

The response tackles all secondary factors, but does not talk about the root cause. I'd expect it to.

<comment><USER4> Having your instances run at 100% CPU pretty much raises a red flag at any cloud provider. Depending on your plan it either gets shut off (like in this case) or you get a notice about "suspicious" behavior and a bit of time to fix the "issue".

<comment><USER3> What's next? Having your disks use too much I/O causes the same response? Or actually using the RAM you pay for?

I run my own iron, with cloud only for elastic loads. Every time I launch a cloud instance, it will be using 100% CPU, otherwise I wouldn't launch it. It's unacceptable to label that profile as "suspicious". It never happened to me on AWS or Azure.

<comment><USER5> > ...you pay for

The major indicator here was the lack of payment history, so they hadn't paid for it but were working off of credit. I think it's a nuance that's very important.

<comment><USER3> I'm sorry to dig heels, but that's no excuse. If the credit they were given allowed them to use the resources, it follows that using the resources is not a breach of contract.

From the description I imagine Digital Ocean offers a free period or tier, to reduce friction in customer acquisition. This is a marketing tool, and must not, in any way, cause situations like the one described.

If a marketing tool induces service failure, it has no place in a professional setting.

<comment><USER5> Credit and promo codes are also used extensively for fraud. If a business had been in operation for a while solely on credit, it may well generate a false positive in a fraud detection algorithm if it scaled dramatically.

But it is important to disconnect monetary spending from coupons or vouchers as they are not equivalent.

You mention free tier but that's not what was at issue here. Also, 10 additional instances isn't in the free tier of any cloud service I've used.

I'm not saying that DO is correct, but I believe the parent argument was a simplification if the events in question. Also, DOs handling of it via support was far worse than the initial algorithm, imo.

<comment><USER3> > But it is important to disconnect monetary spending from coupons or vouchers > as they are not equivalent.

They must be. If they are not, then you've entered the territory I referred, where marketing actions are impacting service availability. This impact is not acceptable in professional services.

In this specific case, if voucher giveaways produce ingress of resource leeches (cryptominers that will never result in real customers), and if it is impossible to prevent this undesired ingress without impacting existing customers (which it is), then that marketing action must stop. This is the conclusion I expected from the post-mortem.

<comment><USER5> Money is fungible and fiat while vouchers are vendor-locked and not fiat, that's why they can't be evaluated the same.

I won't try to argue whether they should be removed in their entirety, that's not even an option I had even considered until now.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER6> Their apparent conclusion that high CPU% for a few hours or half day or whatever means "cryptocurrency miner - ban ASAP!" is naive and flawed.

Compute offload is an ancient and fairly common use case for the public cloud; my VPS (or ten..) should be able to burn 100% CPU for many hours compiling a large project, even if it means they make less profit than they would have had I instead run a static web server that sleeps on IO, imposing nearly no CPU load.

At the very least they should provide some objective, quantitative guidance on exactly how many CPU-seconds-per-hour they consider acceptable/not-abuse (or, if not CPU-seconds, then increased host power consumption, or whatever they are ultimately trying to limit to ensure they can pack a few hundred near- zero-load servers onto the same host to make glorious truly massive profits all the time).

Don't make customers guess at whether their workload will trigger some opaque but hyper-aggressive abuse automation or not.

<comment><USER7> I think the heuristic they use is spike of high cpu + non-established billing history, not just CPU. That seems to me much more indicative of potential fraud, though by no means foolproof.

<comment><USER6> Indeed, but AFAICT there are apparently still some opaque, undefined CPU% limits for people paying with CC instead of free credits. They also mentioned elsewhere that customers paying via PO are exempt from the automated miner murderer, but that was was news to me and I guess just furthers my point: we shouldn't have to trawl HN threads to understand your CPU% abuse limits; they should be spelled out specifically and quantitatively in the main TOS, for each type of payment method, and any other factor(s) that effect them.

</comment></comment></comment><comment><USER8> Reading this response it seems that crypto-mining is not allowed on digital ocean as they have checks against it. The TOS doesn't say so explicitly but does note that:

>violation of any of these Terms of Service or any law, or if you misuse system resources, such as, by employing programs that consume excessive network capacity, CPU cycles, or disk IO

By my reading that seems to mean that you're not allowed to use your VMs to their full capacity due to them being over-provisioned. This is in contrast to AWS who are more explicit on which instances (T instances) are over- provisioned and exactly how they're throttled.

<comment><USER9> If you want to do cryptocurrency mining on DO that is actually okay with us. Some of the other respondents are correct the behavior we were looking for was really around fraudulent accounts being created and performing cryptocurrency mining. This is why the trigger that flagged this account was using payment history as a key factor in the triggering.

<comment><USER10> Your post-mortem implies this is not allowed at all.

<comment><USER11> _Your post-mortem implies this is not allowed at all._

Not sure why you were downvoted, I had the same impression, after reading:

 _...an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers_

This sounds like they don't permit extended high CPU loads due to the impact it can have on other customers.

<comment><USER12> The keyword here is 'fraudulent'. High-cpu-loads is allowed, but an automated service monitors for fraudulent activity.

<comment><USER13> What would "fraud" mean in this context? Are they talking about customers who don't pay their bill to DO? (If so, seems like the account should just be temporarily suspended until the bill is paid.) Or are they talking about fraud to other parties, like phishing sites? (If so, I don't see the connection to crypto mining.)

<comment><USER14> My understanding is that they're trying to prevent users from creating new accounts, running 100%CPU until it's time to pay the bill and then just not paying, moving on to another new account.

edit: from elsewhere ITT it seems they're doing this with stolen credit cards.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> I have no relation to DO, but I'm surprised by the negative responses in this thread. I can't think of any other major company conducting a public postmortem for a customer service failure (as opposed to networking/ops failure). Not only are they changing their policies across the board, taking on more risk to improve customer experience, but they are hiring extra people so it does not happen again. Kudos for that!

And of course DO will still retain the ability to suspend your account for suspected fraud - that is the case with any cloud services company, and any online business in general (check your ToS). Again, I can't think of any business that will en masse promise to never react to any fraudulent users. It's how this process is performed that matters and that's what they are improving.

<comment><USER16> > I can't think of any other major company conducting a public postmortem for > a customer service failure (as opposed to networking/ops failure).

There are many companies that have done this in the past. They are not doing this out of the goodness of their hearts, this is lip service for the fact that their mishap blew up in their faces on twitter. Do you really think they would have gone at length to highlight to the public this incident had it not gone viral?

> Not only are they changing their policies across the board, taking on more > risk to improve customer experience, but they are hiring extra people so it > does not happen again. Kudos for that!

There is no telling that they are actually going to follow through with anything. Mere lip service.

The bottom line is, people host their businesses and livelihoods on cloud providers and they (the cloud providers) should take the necessary care and precautions when taking destructive actions. Maybe err on the side of the customer instead of shutting down someone's entire business because of some automated heuristic. Maybe have a better response time than __29 hours __. Maybe teach basic communication and develop processes so that care agents can see and react appropriately to recent activity on the account. These are not revolutionary concepts, they are simple things that demonstrate customer care, something DigitalOcean is sorely lacking.

<comment><USER17> > precautions when taking destructive actions.

No data was lost, it is not destructive in anyway.

> because of some automated heuristic.

If the customer had "payment history" none of this would have happened. Probably it was being used under "startup credits"

> people host their businesses and livelihoods on cloud providers

people shouldn't run entire operation on credits and blame DO in twitter.

Only issue is that DO took 29 hours, apart from that i see no problem with DO.

<comment><USER18> It should be pretty hard to shit down a legit biz. Seemed automatic in this case.

<comment><USER19> I mean it is. Unless your business is wholly dependent on a service from my business.

</comment></comment></comment></comment></comment><comment><USER20> Key takeaway for me: DigitalOcean might still kill your account at any time, revoking your access to your data.

Refusing service is fine, but holding my data hostage and refusing access to it is not, so I am making a note to not consider DO for any kind of hosting.

<comment><USER21> Because I reinstalled my droplet with a different filesystem manually, the snapshot restore doesnt work. Support tells me they cant do anything, so my 2~ of chat logs are sitting in a disk image that they cant restore bc they need to mount it for some reason...

<comment><USER22> Hey there! I would love to follow up on the issue you're describing here. It looks like you tried to bring a disk image over from a provider in a format that we don't support, and unfortunately there's nothing trivial we could do to get a working Droplet out of it (which is a requirement for us to expose the volume within the systems we have).

I can't promise a super fast resolution - but I'd be happy to work internally to see if there's any outside-the-ordinary workarounds we can supply here if you're willing to follow back up on the ticket.

<comment><USER21> I replied to my ticket (#2710287). Thank you so much for giving it a shot by the way.

</comment></comment></comment></comment><comment><USER23> I appreciate that they did this.

It's sad to me that your only chance in hell of getting huge companies to listen to you is by shamespamming across social media.

That, coupled with the clear issues following procedure from support, paint a clear picture: customer service is an area to skimp on for big tech.

<comment><USER24> In what way is DigitalOcean a "huge company"? At ~300 employees, it's closer to the SMB's definition of a small business (<250 employees) than mid-size (<500 employees).

In all fairness, having worked at a few tech startups, it can be hard to scale customer service to keep up with demand—you don't control how many support tickets come in, and it takes a lot more time to hire and train new customer service agents than it does to spin up new servers, and if you _over_ -hire, it's a lot more costly than shutting down some servers.

<comment><USER25> DO is claimed to be "third-largest hosting company in the world in terms of web-facing computers", so that should give you an idea of how many customers they have.

<comment><USER26> How do they manage so many boxes with so few people? Do they rent metal and resell it with value add software?

<comment><USER27> By skimping out on support, clearly.

</comment></comment></comment></comment></comment><comment><USER28> Digital Ocean: allow me to do some extended verification so you know exactly who I am and reduce your risk. In exchange, there is no automated locking, rather we are contacted and have 24 hours to mitigate the issue.

Requirements:

* 1 year continuous ontime payments at $250+/mo usage

* automatic billing is set up

* billing limits are set up and have been reviewed within the last year

* copy of our business insurance and license

* u2f on all accounts

Fair?

<comment><USER9> The first few items on your list are actually a part of what we meant by "having billing history with us". There are a number of things we look at in that bucket. We use these items as a part of validating users before taking any action (yes, we clearly failed on this account due to the credits which is a clear bug). As far as offering things like a copy of your business license or other means of verification that isn't a bad idea. As an example people paying with POs today are excluded from the algorithm already.

<comment><USER29> Please make it official, so that people can have peace of mind knowing that they've got that "verified" badge. People hate having to wonder whether they're at risk of crossing an invisible, inscrutable, and constantly changing threshold. See: PayPal and AdSense account forfeitures. You could do so much better than that.

</comment></comment></comment><comment><USER30> As happy as I am to see this post itself, the mistakes made here are pretty appalling.

Killing customer accounts by automated action without any human check just seems like a recipe for disaster. Even if you can respond faster to crypto issues, the effects of a false positive are just unacceptable.

Though apparently the human checks at Digital Ocean don't work either.

<comment><USER24> According to the post, that's not what happened? The customer account wasn't terminated by the automated system, but rather by the second Abuse Ops agent.

 _Upon a second review by a different Abuse Operations agent [...] the agent fully denied access back into the account. This action triggered the final "access denied" communication to the customer._

<comment><USER31> That was after the automated process locked access to the account and _powered off_ all associated machines.

</comment></comment></comment><comment><AUTHOR> Digital Ocean's follow-up to "DigitalOcean Killed Our Company" [https://news.ycombinator.com/item?id=20064169](https://news.ycombinator.com/item?id=20064169)

</comment><comment><USER32> We dropped DO from our company usage after similar issues, though honestly DO probably wasn't the right place for our product at that stage of development. What was meant as a POC became technical debt and an outage forced is to come to terms with the fact that by the time the issue happened we had more than enough of our own capacity to run on our own metal.

Kudos to DO for the open incident management. As someone who does this myself, these are often really painful and hard to get right.

</comment><comment><USER33> Good response from DO, but this line jumps out at me.

> Responses to account locks were not prioritized differently from a ticket > management standpoint to be above less severe tickets.

That's arguably the biggest failure, IMO. The fact that an action which locks/terminates an account is not prioritized any different than a general ticket is pretty jaw-dropping, and I'm glad they're going to change that.

<comment><USER9> Yeah... that one was painful and we are fixing it. At least if the priority placed this at the top of the queue we could have acted faster. Probably the same outcome due to the other issues involved in this incident though.

<comment><USER33> I appreciate all your transparency and engagement on this issue. It probably would have had the same outcome, yes, but potentially resolved much more quickly. Regardless, the fact that you're fixing it is music to my ears.

</comment></comment></comment><comment><USER34> The startup claimed they had all their backups on digitalocean, which contained data of Fortune 500 companies.

A startup who has fortune 500 clients must have history. I don't get then why digitalocean says they do not have payment history. Either the startup moved a few weeks ago - but then why don't they have offsite backups if they just moved. Or because they're french they did have payment history but did not have an American credit card to similar.. not sure what's up

<comment><USER35> Their payment history was with DO credits according to the post.

<comment><USER36> Maybe one of the problems is DO's views of credits. Maybe things would work out better if they would treat credits like real money instead of phony money and tighten up how the credits are handed out.

<comment><USER37> Yes, DO have stated that how they viewed credits was a mistake and that they will be addressing it.

</comment></comment></comment></comment><comment><USER38> I really cannot blame DO for this incident. These kinds of things must be handled in a learn as you go way and when I consider what one can do with a DO VPS (or 10) it's astonishing. I would expect them to automatically flag some uses.

A business "relationship" is a two way thing. You call and talk to people, and tell them what you want to do, and ask if it's ok.

When I've called and talked to DO reps about what I've wanted to do they have been very accommodating.

</comment><comment><USER39> Here's what's really wrong. This is a B2B service with B2C-grade terms of service. You don't want to base your business on one of those. Not one with a "sole discretion" termination clause. Those are for low-value consumer facing services only.

Compare, say, these terms of service from a major dedicated server hosting company.[1]

 _Either of the parties may terminate this Agreement (including all existing Orders) if: The other party breaches any material obligation under it (other than our obligations covered by an SLA), and fails to begin to cure such a breach within ten days of written notice of such a breach from the non- breaching party, or fails to completely cure such a breach within thirty days of the original written notice; OR a force majeure event continues for more than thirty days._

Now that's what a reasonable B2B contract looks like. That seems to be fairly standard for dedicated server hosting.

[1] [https://info.codero.com/hubfs/Linked%20Assets/Legal%20Docume...](https://info.codero.com/hubfs/Linked%20Assets/Legal%20Documents/Codero_TOS.pdf)

</comment><comment><USER40> Nothing in the statement from Digital Ocean indicates that they won't kill your account or shutdown your systems - that's not the sort of cloud host any company can afford to use.

Cloud providers that kill accounts - or SAY they kill accounts, must be dropped and not used.

The worst thing that should be possible is for your account to be suspended.

AWS, if there is a billing issue, prevents you making changes to your infrastructure via the console until the billing issue is sorted out - this is good and reasonable.

""Peer review of account terminations. For any account appealing a lock, two agents will be required to review the submission prior to issuing a final deny.""

\- I can imagine how this plays out:

(service agent 1 turns to next service agent along) 'This looks like a bad account - I think I should shut it down, what do you think buddy?'

(service agent 2) - 'Yep I trust you, shut it down.'

<comment><USER41> The article discusses violating TOS, not a billing issue. I don't think it's unreasonable to disable an account in that circumstance but I agree that deleting images/resources without allowing customers to defend themselves and backup the systems would be unfair.

<comment><USER42> > The article discusses violating TOS, not a billing issue.

That's not the impression I got. It sounds like the issue was that a account with misinterpreted payment history was showing bitcoin-mining-like usage patterns. Mining is not against the terms of use, they were just erroneously convinced themselves that the customer was not going to pay for it.

<comment><USER42> I'd like to apologize for the typos in my previous comment that I neglected to notice before the edit window expired.

</comment></comment></comment></comment><comment><USER43> I have read and written similar RCA's in the past, this one is very good IMHO.

</comment><comment><USER44> Barry Cooks did a phenomenal job with this after-action. He not only publicly accepted fault on DO's behalf (+1), not only stated the incident timeline clearly and without bias (+2), but also showed mitigation steps and procedural changes to avoid this in the future and prioritize customer business interests (+3). Many medium and larger sized companies should take note of this handling style (looking at you, Google and Facebook). I love that there was no generic PR "we're very sorry". Succinct, accurate, and without spin (+4).

<comment><USER45> I agree, the incident report was well done. The combination of factors that led to the issue was described in clear detail, and I was glad to see a concrete plan to improve various aspects to avoid future cases like this. It certainly helped to regain trust.

</comment></comment><comment><USER46> I had been expecting a short blip of an update denying anything of consequence (a twitter post promised a status update, but well, you know...) but this transparency significantly exceeds expectations. Nicely done DO.

You may want to explain service credits in some light detail though, for those that are unfamiliar with them.

</comment><comment><USER47> That is everything I'd hoped to see as a developer and digital ocean customer. Good response.

<comment><USER48> Totally. I'm fairly new to DO and after seeing what happened was re-thinking my decision. But this is a solid followup, "we made a mistake" post so I think I can rest easy.

<comment><USER49> They hoped so.

One wonders how many others didn't get enough Twitter cred, before. That some low-level ticket stamper (even a high-level ticket-stamper) had authority to deep-six a customer on no more say-so than high CPU usage tells us more about the company than an incident report massaged by marketing communication specialists. Simply, the latter sounds good because it has been made to sound good by sounds-good experts, and could say anything; but the event itself is ground truth.

They will need a lot more time and good behavior to live this down.

<comment><USER9> I agree on the twitter cred point. The fact that this happened in the end, personally I think it is a good thing as it highlighted a weakness we must fix.

We trust our people high-level, low-level whatever to make important decisions everyday. thats why they are here.

The "marketing communications specialists" are getting slammed a lot here, so I will just point out that they spend most of their time rolling their eyes at my crappy grammar, spelling and ludicrous number of comma splices. I don't think our goal was to sound like anything. We just wanted to lay out our investigation and the follow on work we are undertaking.

Totally agree with your point that trust is earned and we lost many peoples in the last few days. That will take time and as you say good behavior to earn back, but that is what we are committed to doing.

<comment><USER49> I talk about mktg comms because I have worked at places where angry customers got earnest letters promising changes, but the manager expected to implement the changes said "No, we're not doing that!" Or "OK" but nothing happened. So I don't give much credit for promises, even when it was the right thing to promise.

Giving your ticket punchers authority is good when they are authorized to do what customers need to get or keep going. Giving them authority to eliminate customers, not so much.

I have to agree with the commenters who say it was an exemplary postmortem.

Hospitals have been doing formal postmortems for many years, but the number of them didn't start down until they instituted checklists.

</comment></comment></comment></comment></comment><comment><USER50> I think this response is really good.

Now that we're all not picking it, however, I think they should remove the "People" section. They did a good job of adjusting process instead of blaming people. The people section, however, might lean toward blaming people. They didn't, in this case, but it could.

<comment><USER9> Hey there. Thanks for this feedback. I think it is important to be open honest but not blame-oriented in our review of the situation. People make mistakes and that is okay, so long as they aren't willful or due to incompetence. Neither of which was the case here. The key thing is not to create a situation where a mistake is an individuals fault. My general view is if people are making mistakes then we have done something wrong as a company and need to understand and fix the tools/training/process that led to the mistake.

<comment><USER51> I'm involved in work around reviewing medical care.

Generally, a "People" section that mentions processes not being followed is an incomplete root cause analysis.

Why was it was possible for the process not to be followed?

There's obviously a limit to how far it makes sense to drill down with why why why, but stopping at "someone didn't follow guidance" is too early.

</comment></comment></comment><comment><USER52> Huh. Well, that's how you handle a post mortem! You outline what you did wrong, and then you outline how you're going to fix it. And it looks like the proposed fixes are appropriate, so...

DO, like (nearly?) all companies (not to mention most people), is obviously greedy and self-interested, and yes, I'm sure a major driver of the quality of response was the twitter storm that erupted, and I don't want to excuse the underlying mistake which was significant, but...

...at least the responded well eventually!

<comment><USER53> Agreed, and it was more or less the response I was looking for a couple of days ago.

We'll be staying with DO.

We already use other VPS services as backup, and will probably add one or two more. But because of their well-documented response (and at least being able to identify what went wrong, and hopefully to fix it), we aren't going to drop DO.

Thanks for the response, and congrats on standing out in a very small crowd of companies who can own up to their customer service problems.

A very, very small crowd indeed.

</comment></comment><comment><USER54> Digital Ocean notoriously doesn't invest well in data science or machine learning, even having some key data science people leave recently.

I interviewed for a data science job there & the team of engineers seemed really unhappy. They reported into the director of operations, which is a weird place for data science to report, and the managers I met definitely viewed it as a paranoid cost center kind of thing.

Also in the interview process I recall that Digital Ocean made a very low offer and refused to discuss negotiating it. Seemed clear that cheap hires were mandatory for data science / machine learning.

I wouldn't be surprised if this lack if investment meant that some data science intern or bootcamp grad is designing this automatic fraud shut down system, and that there's a glaring lack of investment in professional usability for a system like that.

<comment><USER9> Sorry to hear that you had a bad experience and left with a bad impression of that team. We have a number of data sciences efforts including in the core R&D group where we are growing and working to improve models in support of a number of fleet monitoring tasks

</comment></comment><comment><USER55> I don't quite get the "running on credits w/ no payment history" and "ruined our business" combo. How can they run a business and never pay?

<comment><USER56> Many (all?) of the cloud providers offer credits to startups (credits as in free $ to spend on their services). So if they hadn't burned through that, there would be no payment yet. (The startup I work at got $20k in credits and didn't pay a dime for the first year)

<comment><USER55> I knew they gave credits, but I didn't realize it was to the level of $20k worth of credits. I think I got a couple hundred from DO.

</comment></comment></comment><comment><USER57> Did anyone notice DO leaked customer financials in this post? If I were a startup running on credit, I definitely wouldn't want to advertise that. WTF?

<comment><USER58> I don't understand what you think the issue here is?

My interpretation of this is that the customer had pre-paid credit on their account. Meaning they had not been through the typical bill cycle yet (hitting an external payment method).

How are you interpreting that they are running on credit? As in their account is in debt and they haven't paid yet?

<comment><USER59> Perception is everything. Many companies, especially Fortune 500, will do deep research before doing business with anyone. I've been through them where have been disqualified specifically due to our infancy and lack of proof of long standing. If someone read/mis-read someone's post that gave them the idea that the company didn't have enough runway, they might move to the next potential suitor.

</comment></comment></comment><comment><USER60> IIRC it had to blow up on Twitter before DO paid any attention. At the end of the day that's why it's an issue, because they didn't sort it out until it went public.

I suppose the moral of the story is - have offsite backups, so you can switch VPS providers in an emergency.

</comment><comment><USER61> When I clicked on this I had assumed DO had gone down last week. I was surprised when I finally realized what they were talking about. I think it is cool and commendable to offer this level of transparency on an issue like this.

Anecdotally, I use Digital Ocean for a few miscellaneous services, on an account I've had forever. I have never had any issues with it. I used to use lesser known low-end VPSes, but stopped when I lost a bunch of data on an incident involving a provider's failed RAID controller. (It was my fault for not backing up, but I was young and foolish; they mostly served me well, but I do prefer the assurances of bigger providers nowadays.)

</comment><comment><USER62> Depending on the severity and length, it could still have a long-term impact on that business. Also a bit unsettling that seemingly basic safety controls failed. But it is good to see DO being open and thorough about this incident.

<comment><USER43> That's the only way to get confidence back. I especially like the two peer review policy.

</comment></comment><comment><USER39> If it were not for public shaming on Twitter, the guy would still be turned off.

</comment><comment><USER63> Dear DO, From your RCA it appears this is a type of fraud where stolen credit card is used to create a new cloud account and run up a huge charge in a short amount of time. Nowadays it could be for cryptocurrency mining (a few years ago and maybe still today, it could have been to run spambots or botnets or whatever).

I suggest your trust and safety team handle the payment fraud as a separate issue (using payment network intelligence) and resource abuse (spam or botnet) as a separate issue (by monitoring abuse reports, external underground intelligence; NOT resource monitoring or traffic monitoring).

It seems like in this case, weak muddied signals were combined to draw false- positive conclusions.

Also, it is equally important to build reputation score for good users and use that as a backstop to prevent them from getting shot by misbehaving fraud detection algorithms.

Since your business might be a lot of small customers, it is important you find a good way to easily trust a small customer with little usage and little spend. One way you could do this is by having a reasonable default cap on the resources for a new or small account. You could ratchet up this cap after verifying the payment instrument trustworthiness (through automated checks or manual verification process).

Hope this helps.

</comment><comment><USER36> One of the problems is the credits initiative for startups (I'm assuming that's how this customer ended up running on credits.)

Companies have to grow to quite a big size before they consider offering various discounts and programs. By that time, the systems and processes are plentiful in number, complexity, and interaction. Management decides to implement a startup credits program and because it's not an instant money maker, it doesn't get treated carefully enough and causes various edge cases for the program's users (and hopefully none for the standard type of user).

In DO's case, startups should be vetted before being gifted credits and therefore excluded from the crypto checks and shutdown potential.

Sometimes it ends up in the customer's favor:

For an example of how poorly one-off programs can end up being implemented, my company is receiving special consideration from Stripe. No fees are being charged at the moment. Well, a customer asked for a refund. I issued the refund. Stripe paid out $25 to the bank account but took back only $23 for the refund because the code that does refunds doesn't know about the fee exemption. Good guy me emailed them about their bug but not much came of it.

</comment><comment><USER64> Automated systems that are infallible are great. Most are not infallible, and they should just provide notices to humans.

Humans should (be adequately trained to) review and handle considerations where termination of service is involved.

From reading the DO response, it does sound like humans were involved (eventually). However, unless a customer's use of the service is posing an immediate and severe threat (security, DoS, whatever), service should not be stopped until AFTER a human has adequately reviewed the situation.

Stories like this remind me why sometimes it's better to use smaller providers who are less automated...

</comment><comment><USER65> I like it when a company conducts a full failure analysis and takes responsibility. Doesn't happen often. Hope DO meaningfully improves its service as a result.

Its catastrophic to get locked out like that.

</comment><comment><USER66> _The communication regarding denial of access to the account creates a sense of helplessness; the finality without explanation requires correcting._

Lots of big companies who deal with small partners (developers, sellers, etc.) could learn from this, including Apple, Amazon, and Google. Lack of explanations, vague explanations, or confusing explanations for account shutdowns or other penalizations are the norm. And for some of these companies it's nearly impossible to talk with someone who can clarify what's wrong.

</comment><comment><USER67> So you cannot use your VPS to do whatever you want with it? I am having trouble to understand what's wrong with crypto mining so you get access to your VPS you paid for denied.. Or was that some sort of free plan he was running on? Still... why not introducing CPU quotas rather than blocking?

<comment><USER68> As mentioned elsewhere in the thread, accounts with high cpu usage _and no billing history_ were locked - symptomatic of cryptominers creating accounts, using free credit or stolen CCNs, and ditching.

</comment></comment><comment><USER69> I think its great they wrote the post, but the tone still leaves a sour taste.

I think they could have worded it better.

</comment><comment><USER70> "The communication regarding denial of access to the account creates a sense of helplessness; the finality without explanation requires correcting."

Right there, in one sentence, DO has figured out the one thing that all those millions of CPU cores at Google have failed to grasp.

</comment><comment><USER71> I think this is a great response.

While from my own experience I don't see myself using DO again (see my previous posts, I had a similar experience except I didn't complain externally), the points in future measures look like they'll go a long way.

Best of luck to all future customers.

</comment><comment><USER72> > The account owner leveraged Twitter as an avenue to call attention to the > mistake

What an arrogance! Because official channel was silent, you forgot to mention.

> Shortly thereafter, DigitalOcean investigated the issue and the Raisup > account was unlocked

2 days is "shortly thereafter" for you?

</comment><comment><USER73> Its a good response, clearly outlines causes and future effects. But I'd be very cautious to deal with a company which found such lazy detection mechanism to be adequate, considering the potential cost to real clients.

</comment><comment><USER10> _Additional hiring has been approved for both Support and AbuseOps to reduce ticket queue wait times._

So this is the way they determine their support department is underresourced? By twitter shaming?

</comment><comment><USER74> Something tells me making their abuse department 24/7 won't help I love DO but i had to make a support ticket recently and it took them about 2-3 days to respond.

</comment><comment><USER75> Barry Cooks for President, or any political role where explaining things in a balanced way seems impossible to the current individuals.

</comment><comment><USER76> It's good to see a company pay attention and take action.

That said I would never use them, Amazon AWS is just a smarter solution all round.

</comment><comment><USER77> How are the customer damages compensated? With just sorry, we ruined years of your work with couple of bad clicks?

<comment><USER6> I don't think there is a single cloud provider that accepts unlimited liability and wholly compensates customers for lost data, lost sales during downtime that was the provider's fault, etc. Their liability is generally strictly limited to the cost of service... so something like a $1 credit for the 6 days that your $5/mo VPS was down. Unless of course you are a very large customer that credibly threatens to switch, at which point you may get some special treatment above and beyond the TOS, though even then, rarely enough to fully recover your actual loss..

Ultimately if there is a lot of money on the line you need to do the work and pay the money to be multi-vendor, automatically failing over to AMZN or MS or DO or whatever when there is some massive screwup that takes down GOOG for half a day.

</comment></comment><comment><USER78> It's better than Amazon or Google. I have had accounts shutdown on both with absolutely no recourse.

<comment><USER79> You have exactly the same recourse with those that this person had with DigitalOcean: Be very loud and as public as possible with the problem and it'll get escalated to someone who can make a rational, reasonable choice (and we've seen the same sort of things happen to other big companies on here and twitter), or simply override prior choices purely for PR purposes. This is how many businesses operate now, with zero mechanisms to escalate outside of getting a torch-wielding mob riled up. It seems horribly counterproductive (I mean, my end impression of this whole incident is certainly not _more_ positive about DO -- it's something that should never have happened), but it's how it's done now.

<comment><USER78> This was a few years back. I don't really care about those accounts anymore. Support was just automated responses and when I tried to call them directly, I was directed to the email accounts with automated responses.

My Amazon account was banned because A buyer (I believe was a competitor) purchased an item and claimed it was a fake. I had proof it wasn't, but it didn't really matter. Other than this I had nearly 100% positive feedback. What's funny is that I now buy thousands of dollars per month for my business through Amazon and I keep getting pestered to sign up with a business account.

Google banned an Adsense account when they somehow thought I was faking clicks. I still have no idea where they got this from. My site was't even live yet and I wasn't clicking on anything or even displaying ads beyond a simple test page with no traffic.

</comment></comment></comment><comment><USER80> Excellent postmortem. Great work DO team, and hope these problems get resolved quickly.

</comment><comment><USER81> Great response and ownership.

</comment><comment><USER82> Kudos for this clear summary and planned improvements. Really good job folks.

</comment><comment><USER83> I do a lot with cloud providers for my customer's products and have worked with Digital Ocean's products once before. I didn't have a particular opinion on them[0], and there's some things that seem off about the twitter thread when placed against the incident update report that this thread is linked to. So, all of that to say, I'm giving Digital Ocean the benefit of the doubt.

There are, however, a few things that could be improved about this process:

> Peer review of account terminations. For any account appealing a lock, two > agents will be required to review the submission prior to issuing a final > deny.

The devil is in the details. Do this in a manner that the person confirming that the account is committing fraud is unaware that they are confirming another's denial; otherwise "dude, can you approve that termination I just did? I want it out of my queue/that guy was a dick." is a risk.

Off the top of my head, I'd probably generate two support tickets (linked, but without that link presented to the account termination CSR team member), assigned directly to this person, hidden from others ("hiding", along with training/process improvements is likely enough). If one person disagrees with the termination, close out the other person's ticket. If the CSR sub-org for this is global, place them with staff in opposing time-zones to optimize unnecessary confirmations (though you use a valuable measurement on how consistent your staff is)

> Services that result in the power down of resources will no longer > automatically take action on any account, regardless of lack of payment > history, for accounts that were engaged more than 90 days prior. These cases > will be escalated for manual review.

I can't count how many services I've deployed that started under 90-days ago where the customer failed to add their account information to the service. I can't count them because I don't know. Usually our customer creates their account with instructions from us, and creates an account for us to use which doesn't have permissions over payment details. I wouldn't be surprised if I've had an app go past production that the customer simply forgot to do that important step on Day-1, or if the customer procrastinated until production, etc. We ask, but I've been lied to about stupider things (thankfully rare, but surprising from people who otherwise look like "grown-ups").

Minimally, it sounds like the whole process here is missing a "Hey, WTF _is_ that thing you're running? Call us or we'll need to turn it all off" alert at least a little while before it ... turns it all off. At login, put a clear notice "We want you to love our services, so we let you try them without asking you for payment information. Unfortunately, we have to have monitoring in place to prevent hostile actors from loving us, too. Because of this, accounts newer than 90-days might have services shut off in error. If you want a notification an hour before action will be taken, provide your mobile phone number and we'll send you a text. Or you can enter credit card information/confirm your identity (not sure what options are available here) and we'll keep the bots from bothering you"

Of course, all of this costs money. And based on the incident response times, an explanation other than "failure to prioritize correctly" might very well be "failure to staff properly/have the tooling in place to handle the volume". Considering the competition in this market, I wouldn't be terribly surprised if "we can't afford it" plays into some of that.

[0] A little less awful than AWS in a lot of ways for the task I had to do.

</comment><comment><USER84> I don't like this.

"[...] cites the link to an older account, connected through a shared SSH key, as additional justification for making the decision to deny access."

</comment><comment><USER85> > fraudulent high-cpu-loads

That's enough for me to never use DO again. One pays for a server but then using it is fraudulent?!

<comment><USER86> They were using credits so DO feared that this was a fake for credits account created to mine crypto...

<comment><USER87> that's not really relevant -- no customers are really allowed to use a VPS as if it were bare-metal.

This is a pretty routine issue for people (myself included) who have mistakenly considered using a few temporarily spun droplets for a few hours of intense number crunching.

The 'nicest' complaint I've received was from Linode, which was less of a complaint a more of a warning like 'Do you know what your VPS is doing?' rather than 'Don't do this with your VPS.'. They never really told me to stop -- just wanted to make sure that it was intentional.

<comment><USER85> > no customers are really allowed to use a VPS as if it were bare-metal

I don't agree. What I'm paying for are vCPUs not actual CPUs. So I know I'm not getting bare metal and that the compute power is already managed by VPS provider.

Are you saying that on top of that I also have to monitor and be responsible my own CPU usage or risk getting banned?

What are the rules?

That's why I wouldn't do anything serious on DO, just use it for low-cost side projects.

AWS has explicit CPU credits on their cheaper instances. They built their system to allow bursts of activity. If Digital Ocean doesn't have that their offering is simply weak.

Also I recall few years ago Digital Ocean send me a message that referenced specific processes running on my VM. I know that they have access to that if they want to but looking into specific processes was stepping over the line in my book.

I moved my servers from Digital Ocean shortly after that.

</comment></comment></comment></comment><comment><USER88> Rule 1: Never shutdown customer services at the first machine/AI detected potential issue. Always contact customer to verify, always.

DO never suitable for business use case.

</comment><comment><USER89> At every turn of this story, there is a resounding "we fucked up". Shameful behavior on the part of an "infrastructure provider."

It doesn't matter that they create a post-mortem. All it means is that I can't trust them to get things done correctly.

</comment><comment><USER90> Like I commented in the last post they've entirely missed the point. You shouldn't be allowed to deny a client access to their data unless there's some law being broken. Data is property and you shouldn't be able to suddenly deny a client access their property.

</comment><comment><USER91> I don't like the weaselly passive voice. And it's still not clear if a person who doesn't have the clout on social media can get attention.

<comment><USER92> [http://www.lel.ed.ac.uk/~gpullum/passive_loathing.pdf](http://www.lel.ed.ac.uk/~gpullum/passive_loathing.pdf)

<comment><USER9> Thanks for the pointer. I'm going to blame my dad/up bringing for my over use of passive voice. He will be deeply amused by this. I will read the paper and attempt to improve.

</comment></comment></comment><comment><USER93> And at the end of the day: post-mortem no post-mortem no one really cares. The technical details are always irrelevant to be useful and in 6 months no one remembers anyways.

Hell you could write a post mortem: "shit hit the fan" and it wouldn't be any more or less insightful.

<comment> grey-area I'm a customer of Digital Ocean and I care. I'm pleased with this response and it is exactly what I want to see when a company makes a mistake.</comment></comment>


How Two Guys Raised $100K To Make A Magical Keyboard - <AUTHOR> http://techcrunch.com/2011/12/05/the-touchfire-chronicles-how-two-guys-raised-100k-to-make-a-magical-keyboard/

<comment><USER2> There are two issues with extended text entry on an iPad: the lack of tactile feedback, and the tradeoff of typing and viewing ergonomics. They've added tactile feedback, but you still need to keep the iPad near horizontal in order to type on the screen.

I've got a bluetooth keyboard/case/stand made by Zagg (branded by Logitech), and although I haven't tried the TouchFire, I would prefer my current setup. A side bonus is easy accessibility to all characters—I don't do any programming on this, but I write LaTeX and Markdown regularly, so making curly brackets or asterisks easier is a nice plus. I thought I would benefit from the increased screen real estate (when using a bluetooth keyboard, the on-screen keyboard is hidden), but in practice I don't think it actually helps.

Here's a non-affiliate link to the product I've got: <http://www.amazon.com/gp/product/B0054JE706>

And here's an affiliate link, if you're into that sort of thing: <http://tinyurl.com/dy4skxe>

_edit:_ You can't really see how the product is also a case on the Amazon page. Here's the product page, in another non-affiliate link: [http://www.zagg.com/accessories/logitech-ipad-2-keyboard- cas...](http://www.zagg.com/accessories/logitech-ipad-2-keyboard-case)

<comment><USER3> iPad 2 weight: 21.2 ounces

Zagg keyboard weight: 13.6 ounces (64% of an iPad 2)

TouchFire weight: < 1 ounce (4% of an iPad 2)

Zagg keyboard: $92

TouchFire: $45

<comment><USER2> It's about $10 cheaper than that from Amazon or in stores, but sure. It adds weight. Neither are a large enough weight to impact your carrying of an iPad (the smart cover is nearly 5 oz! :), and if you lift the iPad up to look at something, you aren't also lifting the case.

I'd say a better dimensional concern to have is that when you're using it as a case, it more than doubles the device thickness.

One of these products is a case, stand, and keyboard, the other is a keyboard accessory. I tried to explain why one is better than the other... this makes my iPad feel like an awesome laptop when I bring it along. Apples and oranges.

</comment></comment></comment><comment><USER4> I still fail to see how this improves over a thin Bluetooth keyboard, that wirelessly communicates with the iPad..

It seems like it takes up just as much space as phyiscal, but requires covering the screen, like the software KB does.

<comment><USER3> TouchFire actually takes up much less space than a Bluetooth keyboard. The wall thickness of the silicone keys is half the thickness of a credit card, and it is flexible so it can fold with the iPad's cover.

And you can quickly retract it off the screen when you want to do non-typing tasks. There are magnets in all four corners which snap it into the retracted position. Take a look at the iPad 1 video on the Kickstarter site. That shows retraction in use...

</comment></comment><comment><USER5> I think for programmers a keyboard which projects a screen should be a better idea than a screen which projects a keyboard.

</comment><comment><USER6> Looks interesting - I didn't realize the first tablets were developed 20 years ago.

<comment><USER7> Apple likes to take a revisionist history approach to their new products. They explicitly label them as innovative devices (which they may be) but implicitly attempt to create the perception that they've invented the device. See: iPod, iPhone, iPad. There's a huge number of people who didn't know MP3 players, smartphones, and tablets existed before the day Apple showcased them.

<comment><USER8> Is there any tangible proof for this? Is it just that they don't label it a tablet, smart phone or MP3 player? It might just as well be people who are angry that their products don't receive the same media attention. I'm not saying the media are fair, but I don't buy that Apple's marketing is actively revisionist.

<comment><USER7> Take a look at the announcement of the iPad [1]. "You can browse the web on it. It's the best browsing experience you've ever had. To see a full web page you can manipulate with your fingers." "You can turn the iPad any way you want, it automatically adjusts." "You can see your photos and flip through them."

Note that I said they explicitly state that it's a tablet, a smartphone, an MP3 player, but they implicitly imply that every feature they have or device they've created has just been invented. The above quotes go a long way toward showing that. Touchscreens, even larger than the iPad, had been around for a long time. Gyroscopic screens had been around for a long time. I had one on my Lenovo tablet before the iPad was even announced. Job didn't announce it as a new tablet, he announced it as _the_ tablet, and described the features as if they were new. Looking back on the featureset he took pride in, no one would ever present something as mundane as a touchscreen picture slideshow unless they were, with a least some part of their conscious mind, trying to imply that it was a new thing.

And it was easy to believe, at every launch. Tablets, consumer smartphones, and MP3 players were not hugely popular, there was a huge consumer market that had not heard of them. Did Apple enlarge the market? Sure. But Jobs implied, with a straight face, that everything they were doing was new and original. It simply wasn't. It's akin to driving a truck into the jungle and showing the natives your four-wheel-drive invention.

Remember the classic Slashdot post about the announcement of the iPod in 2001. Even then, gadget-minded folks knew that the iPod was a flashy but ultimately unoriginal product.

[1] <http://www.youtube.com/watch?v=1PpeSbDoeAE>

</comment></comment></comment></comment><comment><USER3> Yeah, a lot of people aren't aware of the history.

</comment><comment><USER9> Hey, they're even old like me, so cool

</comment><comment><USER10> off-topic : was kinda cool that in the video there was a guy with a Punta del Este T-shirt, that's where I'm from.</comment>


Show HN: Moderator dang - <AUTHOR> http://cosmonautdreams.com/images/dang.jpg <comment><AUTHOR> This probably doesn't meet the criteria for an allowed HN post but I laughed when I thought of it. The moderators here do a good job and this is just meant as a funny tribute. Please don't shadowban me for life, it was meant with purely good intentions.

</comment><comment><USER2> Perhaps make a comic on the many events when the mod saved the day here at HN.

<comment><AUTHOR> I honestly dont know how they stay on top of things with the throughput of the site (24/7) and such a small moderation staff, and how they are still remotely friendly after dealing with internet comments all day everyday.</comment></comment>


What Would You Do with a $40 Computer? - <AUTHOR> http://lifehacker.com/5266621/what-would-you-do-with-a-40-computer <comment><USER2> I really wish some people who have commented on that post would never come on HN.

Example comment: "doesnt seem like it has much use... though it is cool to see computers being made so small. id probably use it for a usb cup warmer"

</comment><comment><USER3> I think there's "something" here. To summarize the answers to the question I've seen so far:

1\. File/print server. Very real use-case.

2\. Control motors, device controllers around the home that can be connected via USB.

What if a 40$ computer had wi-fi connectivity? What if one couples it with wi- fi sensors such as <http://www.aginova.com> sells and wi-fi enabled device controllers. What kind of apps one can build given such as combination?

Now, first one is obvious and clearly valuable use-case. What about the second one?

<comment><USER4> You can get an Asus router with a USB port for $45. Don't think I'd pay $40 for the USB port minus the router.

<comment><USER5> Well, the Sheevaplug probably has a much higher-end CPU. And an SD card slot. And a bunch more ram. And so on.

</comment></comment></comment><comment><USER6> Some discussion already at <http://news.ycombinator.com/item?id=623561>

</comment><comment><USER7> If it has a number of digital or analog I/O ports, robot, most certainly.

</comment><comment><USER8> How come all the promotional pictures of this exclude the external storage you have to add for this to actually provide meaningful sharing? I think a little larger device with room for a 2.5" drive would be more useful.

</comment><comment><USER9> Beowulf cluster.

</comment><comment><USER10> hmm... i wouldn't buy a $40 computer.

<comment><USER11> Why?

<comment><USER10> What would I do with it? It would have a computing power of a calculator...

I'd better buy a MacBook that I have never tried. A lot more expensive, but also a lot more positive comments about and reasonable screen/computing power...</comment></comment></comment>


EBay to Ditch PayPal for Dutch Payment Processor Adyen - <AUTHOR> https://www.bloomberg.com/news/articles/2018-01-31/ebay-to-ditch-paypal-for-dutch-processor-adyen-lowering-costs <comment><USER2> Here's what puzzles me. I have looked into Stripe, and just browsed the Adyen webpage. Both of those services seem to require you to maintain your own "active" server that can run server side code.

PayPal seems to be unique in being able to take payments from a passive web page, because the customer conducts their transaction at PP's website.

This is why I continue to use PP for my tiny little business (without eBay). Even though I consider myself reasonably tech savvy, I don't trust myself to maintain a website that is compatible with everybody's browser, phone, etc., and that guarantees the security of their personal data. Moving to another payment processor requires a quantum leap in technology that I'd rather not keep up with. I'd rather design another gizmo.

From time to time I look around for an alternative to PP, and haven't found one yet. I suspect that many small-time eBay sellers may be in the same boat.

<comment><USER3> The reason for processing on your own server is so you own the entire checkout experience. If you redirect he user to another site for payment, you run the risk of losing track of that user. Also, the third-party's branding may clash with your own.

<comment><USER4> Going to a well known third party is a guarantee that my payment data isn't sitting in a poorly secured vendor database. That's more important to me than a seamless transaction. PayPal is most useful when you don't fully trust the site you are buying from to have their shit together.

<comment><USER5> This. Entirely.

I used to work for a small ecommerce webdev shop. I've worked on sooo many shitty insecure shopping carts over the years I simply know not to trust basically any small website asking for my card. Completely unencrypted, storing CVVs, sending CC details as GET parameters, I've seen it all. It's painfully common!

If a company is not a huge name, and is handling your credit card info themselves, they are mishandling your credit card information in one way or another. I guarantee it.

Having been through PCI compliance, it's no joke. It's really not worth doing yourself in my strong opinion.

<comment><USER6> This. I wish there was a fundamentally different way of handling payments. Something like a one time hash you give the vendor that serves like a digital check(cheque). This way, if they lose the hash they lost their money instead of the customer's.

Handing a merchant credit card details "feels" like handing them a blank Check and trusting them to not take more than they should because of the risk of them losing the data.

Edit: The digital hash should say where money comes from, to who, for what purpose, time, auth code and so on.

I live in Denmark, and they have a system of phone banking here called "Mobile Pay" it works kind of like that. Where you transfer money to the merchant public phone number and show them proof of sending.. However it only works for in-person payments where you can swing your phone to show.

<comment><USER7> > I wish there was a fundamentally different way of handling payments

Seconded. At the risk of simply rephrasing everything you just said:

It's a pity the banks and credit card companies just refuse to innovate.

When I buy something from an online vendor, they should never get my full card details. PayPal ameliorates things (you can _generally_ trust PayPal), but really there should be no need for PayPal. The banks/credit-card companies should provide a convenient way to authorise the payment, in a way that doesn't trust the vendor.

Here in the UK, we already have chip-and-pin, and card readers (the kind that show unique numbers - they're used for online banking), but we're still stuck with the trust-the-vendor-or-use-PayPal model for online payments.

Using card-readers for online payments would also help with credit-card theft.

<comment><USER8> > (you can generally trust PayPal)

I agree, but I also have a counterexample:

"Oh, you bought SOFTWARE?? All those pretty marketing pages about our amazing safety and protection system do not apply to virtual products. We agree this vendor totally screwed you over, but it's not our problem."

(this was a few years ago, may have changed)

edit: It eroded my trust in the company completely. I don't really have an opinion of their technology.

<comment><USER7> PayPal have certainly had a few screw-ups. My personal favourite: when they locked $750,000 of MineCraft sales money. [http://www.escapistmagazine.com/news/view/103385-PayPal- Free...](http://www.escapistmagazine.com/news/view/103385-PayPal- Freezes-750K-in-MineCraft-Devs-Account)

At a glance it seems software products _are_ now covered - [https://www.paypal.com/il/webapps/mpp/ua/useragreement- full?...](https://www.paypal.com/il/webapps/mpp/ua/useragreement- full?locale.x=en_IL#7)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> Potentially a great move for eBay to reign in processing fees and to consolidate the dispute resolution process within their own platform. A large pain point for many eBay users has been Paypal's opaque dispute process. (I admit to bias: I lost ~$5,000 in Paypal balance while in college due to Paypal siding with a dishonest international buyer)

For those of you contemplating Adyen vs. Stripe: Adyen is much more "bare metal." Think more like a modern Authorize.net. Nobody comes close to Stripe's turnkey developer-friendliness.

<comment><USER10> (I work at Stripe.)

Glad to hear on the developer friendliness! If there are ways we can continue to improve on that front, please shoot me a note: firstname.lastname@example.com

I'm curious though what you mean when you say Adyen is much more "bare metal" than Stripe. We don't typically talk too much publicly about our underlying infrastructure (our goal is to abstract away that [hopefully] unnecessary complexity), but we do strive to be as close to the bare metal as possible. (We're directly connected to all of the major card brands, and have "acquiring licenses" in numerous markets.)

<comment> master-litty I interpreted that as a positive nod towards Stripe. In other words, Adyen and Stripe provide equivalent functionality but Adyen requires more work to adopt.

<comment><USER10> I wasn't exactly sure which direction to read it :-) We often get questions around our underlying infrastructure and how we process payments (this is what I work on at Stripe!), so that bias is probably affecting my read...

</comment></comment></comment></comment><comment><USER11> One of the bigger drawbacks of Adyen vs Stripe, although we wanted to work with them a lot, is, they require a crazy reserve (in the millions) if your model is subscription based. The logic behind it from them is, that they must be able to refund all your customers in case you go bankrupt, and you have subscribers left hanging without full-filled service they paid in advance for.

Although I get the logic behind it, not one other PSP requires such a huge reserve, therefore we decided not to work with them.

Todays payments world is v competitive and players like checkout.com and many others are v aggresive trying to disrupt stripe's dominance in this area

<comment><USER12> Are you saying that to process recurring fees on Stripe (as so many startups do) that you need this reserve?

<comment><USER13> Definitely not, from experience. I think the opening poster means that Adyen needs such a large reserve, since I definitely know subscriptions-based startups that don't have any reserve with Stripe

<comment><USER12> _phew_ :)

<comment><USER11> yes, I meant adyen, not stripe, sorry for confusion

</comment></comment></comment></comment></comment><comment><USER14> I've never heard of Adyen here in Australia before this. I assume they are far more Europe/US oriented? I assume they will be rolling out the new integration world wide, so that it will become more ubiquitous? I believe Paypal has a local office in Aus, so I presume Adyen will be setting up local offices in most countries?

(I also noticed on the video that it is pronounced "Adi-an" where I first thought "Ad-yen" which makes them sound more like an ad wholesaler than a payment processor.)

<comment><USER15> I doubt Adyen is well known among North Americans either (the first market being rolled out for ebay). Ayden is used by Uber and Netflix and some other big names but that's mostly hidden in the background as they are integrated directly into the platforms. Unlike Paypal which has an obvious branded layer for all transactional layers.

I'm curious what the integration with Ebay will look like. Will users be redirected to their Ayden accounts ala Paypal or will it be branded via Ebay?

<comment><USER16> I think the term eBay is using is intermediated payments. I think as a buyer, you would pay to eBay and eBay would pay to seller. So should be frictionless.

</comment></comment></comment><comment><USER17> If anyone else, like me, thought eBay owned PayPal:

> On October 3, 2002, PayPal became a wholly owned subsidiary of eBay. On > September 30, 2014, eBay Inc. announced the divestiture of PayPal as an > independent company, which was completed on July 20, 2015.

<comment><USER18> Yes. The Masters of the Universe decided that there was "value to be unlocked" by spinning off PayPal. Or at least...someone's spreadsheet said there was value to be unlocked..

<comment><USER19> And there actually was... PayPal itself is still (after this news) worth more than both eBay and PayPal were as a single company. Add both together and it was an obvious win.

<comment><USER20> Strange that since eBay probably owns a lot of PayPal stock that they would try to tank Paypal. Maybe eBay sold their Paypal stock holdings ahead of this change.

<comment><USER19> eBay has no stake in PayPal, that was the whole purpose of splitting it out. Shareholders of eBay before the split were given PYPL shares on a 1:1 basis.

</comment></comment></comment></comment></comment><comment><USER21> I'm all for a competitor in this space. PayPal is very cocky and stagnant.

<comment><USER22> Heh, I felt PayPal has been playing catch up for years now.

Cocky maybe, certainly not stagnant.

<comment><USER21> >certainly not stagnant

Hmm. Maybe I'm misinformed? What kind of innovative things have they done recently? As a PayPal merchant, the UI is still a clunky mix of old and new, and weird session errors and logging in twice, etc. We just barely got past SHA1 certs.

</comment></comment></comment><comment><USER23> (Autoplay video)

I wonder when eBay will change their rules that currently state that you _must_ offer Paypal and cannot _mention_ other payment options (including _cash_ ) in a listing.

<comment><USER24> I quit eBay (and failed to get back to a comparable experience, unfortunately) the day (years ago) that they stopped allowing people to sell and specify payment options that didn't have the extra Paypal fee. Forcing everyone to use Paypal and pay the fee was obviously corrupt.

</comment></comment><comment><USER25> Does anyone here use Adyen? We use Stripe but are starting to consider options that may increase conversions.

<comment><USER26> Adyen will be better if you have more consumer cards vs. business cards being charged. Nearly 100% chance of this, otherwise Stripe may win.

<comment><USER27> Is there middleware that can route transactions based on criteria to various payment processors?

<comment><USER26> There definitely is not. Good idea though, however the margins would be razor thin. Possibly too thin.

<comment><USER28> Google "payment routing". There definitely _is_ middleware you can buy to route payments to different processors. I've built my own for my business on top of Spreedly. I'm not sure what you think the issue would be with margins; the routing service doesn't have any direct costs other than servers.

</comment></comment></comment></comment></comment><comment> mark-r When Ebay first started requiring Paypal, it made sense - Paypal was clearly the frontrunner for online payments and Ebay's homegrown effort was a pale comparison. Also Ebay was a real powerhouse that could move the needle with a decision like that.

Today I feel like Paypal has a lot more traction than Ebay, and this is going to be a big flop for them. As a consumer I don't want to sign up for yet another payment service.

<comment><USER29> I think the advantage of Adyen is that you don't need a new account. It supports payment processors that are used all over the world.

<comment><USER30> This has its own disadvantages. My biggest reason for preferring Paypal to Stripe or other systems is that my Paypal account is linked to my bank account, which means I don't have to recharge my debit card (and pay the charge fee) for each purchase. Account-less systems can't run the proper verification to allow for that

</comment></comment></comment><comment><USER31> Never heard of Adyen, but it's great to see more Euro companies compete against SV.

</comment><comment><USER32> Previous discussion: [https://news.ycombinator.com/item?id=16280553](https://news.ycombinator.com/item?id=16280553)

</comment><comment><USER33> Wonder why they didn't go with Stripe, but with an European competitor? This is great for the European tech scene.

<comment><USER34> Stripe "is"/"was" also European :)

<comment><USER35> Stripe has never been European, it was always headquartered and operated out of the US.

It'd be like claiming that SpaceX is South African.

[https://www.startupgrind.com/blog/the-collison-brothers- and-...](https://www.startupgrind.com/blog/the-collison-brothers-and-story- behind-the-founding-of-stripe/)

<comment><USER36> And that Amazon is Portuguese...

</comment></comment></comment></comment><comment><USER37> They didn't "ditch" PayPal, at least not according to the e-mail I saw. They merely made it a non-default option. It's not like you can't still use PayPal.

</comment><comment><USER38> Nooooo, oh good, PayPal is available till 2023.

</comment><comment><USER39> Absolutely 100% did not see that coming.

</comment><comment><USER40> Stripe/Braintree are a lot better than PayPal. PayPal will never take your processing history into consideration if their algorithms decide that your account is connected to some account with past violations. This happened when my developer used his API keys on our production resulting in our account with 1M+ revenue/2 years (very few chargebacks if any) of operation banned. As a small startup, it was a death sentence for our business, finding another processor at high volume is difficult when you've no history to show!

<comment><USER41> I've avoided paypal for ages because of their heavy handedness with freezes and bans.

They just can't be trusted not to fuck over your business. Not intentionally... but algorithmically. And once that happens their customer support rarely seems to be able to fix the issue.

They just don't seem to care about false positive vendor issues.

<comment><USER42> I had an issue with my paypal account in Australia where they thought it was ok to pull money out of my credit card when it expired. This failed of course and so they decided to just put me into a negative balance. They never emailed me until about a year later when they threatened to send debt collectors... so I added my singapore credit card. However I couldn't. It would fail to accept the card. Support told me that my card was stolen because I'm using a singapore card on an Australian account. In the end not having an Aussie bank account or card I was forced to open a new singapore account, send money to the old account. Then close both accounts...

</comment></comment></comment><comment><USER43> Na na na na Na na na na Hey hey hey Goodbye

<comment><USER44> Could you please not post unsubstantive comments here?

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)</comment></comment>


A primer on the cruel, tacit laws of type-level programming in Haskell - <AUTHOR> https://www.aymannadeem.com/haskell/2020/05/15/Kindness-for-Mean-Girls.html <comment><USER2> Kinds in Haskell seem to be evolving in the direction of controlling the representation of datatypes in memory. * (also called Type) is the kind of types whose values are "lifted": they can be lazy computations which might cause an infinite loop, throw an exception, and so on.

There are also kinds for types whose values are "unlifted", meaning they really correspond to actual values in memory, not lazy thunks.

Right now the language for expressing unlifted datatypes is quite limited. Hopefully it will be expanded so that we can write "strict Haskell" more easily.

"Levity polymorphism" paper [https://www.microsoft.com/en-us/research/wp- content/uploads/...](https://www.microsoft.com/en-us/research/wp- content/uploads/2016/11/levity-pldi17.pdf)

"Kinds are calling conventions" paper [https://www.microsoft.com/en- us/research/uploads/prod/2019/0...](https://www.microsoft.com/en- us/research/uploads/prod/2019/03/eta.pdf)

The -XUnliftedNewtypes extension [https://downloads.haskell.org/ghc/latest/docs/html/users_gui...](https://downloads.haskell.org/ghc/latest/docs/html/users_guide/glasgow_exts.html?highlight=unlifted%20newtypes#extension- UnliftedNewtypes)

[https://stackoverflow.com/questions/35318562/what-is- levity-...](https://stackoverflow.com/questions/35318562/what-is-levity- polymorphism)

[https://stackoverflow.com/questions/56311510/what-does- type-...](https://stackoverflow.com/questions/56311510/what-does-type- signature-for-undefined-mean-in-haskell/56317845#56317845)

<comment><USER3> I'm not sure that "evolving" is the right phrase - unlifted types have always been a part of low-level (generally compiler-specific) Haskell.

</comment></comment><comment><USER4> There's a "Stop trying to make Haskell happen, Gretchen - it's not going to happen!" joke here, somewhere.

<comment><USER5> He's ... just not your type?

<comment><USER6> What Kind of joke is that?

</comment></comment></comment><comment><USER7> There seems to be a nigh-irresistible urge for people, when explaining type systems, to pick examples that are absolutely useless.

None of the types in the theme here make much sense. They are arbitrary. The contribution to writing a robust program is questionable and they don't do much to explain concepts.

May as well stick with classic "Goats are mammals; that should be enough to get you going with OOP" examples, at least they are less distracting.

I'm not a fan of this article.

<comment><USER8> That's a problem with the domain itself. Type system examples can be two of: short, concrete, well-motivated. Short concrete examples don't have time for motivation; concrete well-motivated examples require lots of text (or else are unreadable); and short well-motivated examples are forced to be highly abstract.

There's no substitute for actually encountering a real-world problem which makes you want/need type-level programming. The article makes the sensible choice of not trying to invent one for you. I don't think it should be bemoaned for that.

</comment></comment><comment><USER9> Holy 6MB 2,046px × 1,416px image, Batman!

<comment><USER10> Yep. Multiples of them totalling 35.4MB of pointless, bandwidth-sucking stupidity.

<comment><USER11> a testimony on the need to train people to think about memory with lazy fp ?

<comment><USER10> If by that you mean lazily as in functional programming, I don't see that would help.

<comment><USER10> OK, I get it finally. Bit thick today! Upvoted.

<comment><USER11> no worries, humorous comments are not the first expectation on HN

have a nice day

</comment></comment></comment></comment></comment></comment><comment><USER12> A discussion of the Haskell type system is not what I expected when clicking that link.

<comment><USER13> Be honest, what was your very first thought?

<comment><USER14> I thought it was going to be a piece in favour of showing kindness to people who themselves are mean.

(I never even considered "this is HN so it's probably tech-related". Don't know what that says about me or the site!)

<comment><USER15> Yeah, me too. If it wasn't explained in the discussion, I wouldn't click on it. (Maybe HN needs some tags.)

But I am actually glad it's about Haskell, I am a gonna read it now (and it's refreshing to see it being written by woman).

I don't mind the movie references, I don't think it's sexist. (Quite the opposite, I think standard mathematical writing is too dry and benefits from having an unrelated theme to add a bit of spice.)

Current reality is hard for everyone. When men behave like men, some women complain, when women behave like women, some men complain, it's really hard to please everyone. Take it easy, everyone.

</comment></comment></comment></comment><comment><USER16> Does anyone else think the title is sexist by today's standards?

<comment><USER17> I have strictly no idea why OP is using women nor in the title or illustrations

<comment><USER16> It reinforces the stereotype that only women are mean. I would dislike a title reinforcing stereotypes about me: "Detox for Alcoholic East Europeans: Haskell type enforcement".

<comment><USER18> No such stereotype says only women are mean.</comment></comment></comment></comment>


Checksumming in filesystems, and why ZFS is doing it right - <AUTHOR> https://oshogbo.vexillium.org/blog/73/ <comment><USER2> I've been wondering for a while why blocks are used as the checksummed unit. Why not files (since zfs/zpool breaks traditional layering anyway)?

<comment><USER3> Because blocks, not files are duplicated across raid sets. If doing a 512kb read of a 10tb file you dont want to have to check the entire thing.

</comment></comment><comment><USER4> Cool, I'd never heard of the filesystem feature 'salf-healling'. Is using a spell checker a forgotten skill?</comment>


Show HN: Productivity / Goal Management Tool - <AUTHOR> https://complice.co/ <comment><AUTHOR> Not mine, just think it looks neat.

<comment><USER2> If it's not yours please remove the "Show HN" from the title. From [https://news.ycombinator.com/showhn.html](https://news.ycombinator.com/showhn.html)

> _What to Submit_

> _Show HN is for something you 've made that other people can play with. HN > users can try it out, give you feedback, and ask questions in the thread._

If it's yours, don't try to hide it, because it's usually easy to spot. It's ok to submit your stuff, but don't submit too much.

<comment><AUTHOR> > If it's yours, don't try to hide it, because it's usually easy to spot. It's > ok to submit your stuff, but don't submit too much.

Not mine.</comment></comment></comment>


Clearview Says Its Software Identified a Terrorism Suspect. Cops Say Otherwise - <AUTHOR> https://www.buzzfeednews.com/article/ryanmac/clearview-ai-nypd-facial-recognition <comment><USER2> A recent related thread: [https://news.ycombinator.com/item?id=22083775](https://news.ycombinator.com/item?id=22083775)

</comment><comment><USER3> I guess the question is..

What's more problematic: false positives or false negatives?

<comment><USER4> [https://en.wikipedia.org/wiki/Blackstone%27s_ratio](https://en.wikipedia.org/wiki/Blackstone%27s_ratio)</comment></comment>


Nothing in Biology Makes Sense Except in the Light of Evolution - <AUTHOR> http://www.pbs.org/wgbh/evolution/library/10/2/text_pop/l_102_01.html <comment><USER2> _A theory can be verified by a mass of facts, but it becomes a proven theory, not a fact._

This is a ridiculously bad description of how science and fallibility work.

<comment><USER3> Well, yes, theories are never proven true in any ultimate sense. But it explains how people work rather well.

If a theory works time and again to explain something then, eventually, it's considered proven. For example, Newtonian physics/gravity isn't strictly correct true, but it still useful. It works to explain things well enough that it's pretty much the only model 99% of the world will ever need, and, in that sense, it's considered true.

<comment><USER2> The distinction between "theories" which aren't proven true, and facts which are, is false. Nothing is proven true in an ultimate, certain sense.

That isn't important. But not because if something seems to work a bunch of times (how many?) then it's true. That isn't a way forward. The way forward is to focus on what the best available explanation is, and to improve it through criticism.

<comment><USER2> So the pattern over many threads is:

1) article says something dumb

2) i say so briefly. i can't very well always explain in detail when no one has even expressed interest

3) someone espouses some dumb and imprecise theory in reply, that vaguely sounds right

4) i point out a reason it's wrong (in this case, there was the "how many?" question which is a full refutation)

and then 5:

the first comment is downmodded for lack of detail

and the further detail is too nested, and too technical/precise/doesn't have the "sounds nice" feel to it. and still i keep it brief because the person didn't show interest in learning.

for another example of the kind of followup i mean, something like "arguments from the authority of experience are invalid" or "correlation is not causation". these get downmodded. presumably b/c they are unhelpful or everyone knows them. except, they are true, and people violate them all the time.

anyway, does anyone know the solution?

<comment><USER4> Perhaps the solution is a bit of humility. Half the time you think you're "correcting" somebody it seems that you're actually completely misunderstanding their point, and half the other half you're just misunderstanding the issue. The condescending tone doesn't help, either.

<comment><USER2> Interestingly, your comment lacks humility. How about quoting to back it up?</comment></comment></comment></comment></comment></comment>


Ask HN: What are some of the ways to learn codebases - <AUTHOR> Last time https://news.ycombinator.com/item?id=8263402 article appeared in 2014, I wanted to see if there are methodologies that people have adopted not listed in this article? I would like to hear your experiences. Feel free to contribute here or at https://gist.github.com/hhimanshu/714962f4205d7dddc7e5991a86412805. I would make notes and update the page. Thanks <comment><USER2> Learning a codebase requires fast navigation. ctags or a good IDE help with that.

</comment><comment><USER3> _" Follow the money!"_ \- but instead of money, follow the data;

\- sort source code by filesize

\- look at the large methods / code blocks

\- browse through the /examples/ and /tests/ dirs to familiarize yourself with typical usage

</comment><comment><USER4> Runtime call tracing is the missing methodology. I haven't found anything better over the years and always implement them if I have to learn things quickly. Made one for nginx even.

<comment><AUTHOR> thank you. Could you elaborate on this method please?

</comment></comment><comment><USER5> Biggest thing for me is to talk to whomever wrote it, or if they're not available, whomever has the most knowledge of the codebase.</comment>


"What May Happen in the Next Hundred Years", from c.1900 - <AUTHOR> http://www.howtobearetronaut.com/wp-content/uploads/2011/07/Ladies-Home-Journal.jpg

<comment><USER2> Remarkable how much of that has come true now ... and how unthinkable most of it was then.

Even those predictions which haven't happened are more by choice than by ability.

<comment><USER3> I find it interesting to understand where the blindspots are.

For example, mosquitos, flys, rats and mice were predicted to be exterminated. So did the scourge of these pests simply get mitigated to the point where further investment in their extermination was unwarranted?

Completely missed airplanes (to be expected, the Wright Brothers had yet to fly) but air-ships as flying fortresses? Not connecting their train imagery with their military history. (if you have high mobility slow moving 'forts' are not effective)

The population number was over and the life expectancy under (330M US pop in 2000, vs 350 - 500M predicted), life expectancy prediction of 50 yrs vs 72 years. That is an interesting combination to get wrong but the blind spot was that people would voluntarily have fewer children.

Amusing that everyone was supposed to vote themselves into the US. Can you imagine the chaos if Mexico elected to join the US and applied for statehood? Wow.

So its clearly harder to predict political change, easier to extrapolate technological curves, and somewhere between the two lies emergent technolgy/attitudes.

</comment></comment><comment><USER4> Pretty remarkable how accurate those predictions were --some were off, but the hit rate was pretty good. Man[kind] will see around the world... That English would establish its own grammar rather then try to foist Latin on it (such that one could not end a sentence with a preposition because it ran against Latin grammar but was natural in English). The rise of UPS and home delivery of goods...

Good stuff.

</comment><comment><USER5> The most influetial 20th century improvement that they missed was the rise of the airplane, in particular the jet airplane. It obviated the need for "fast electric boats" from New York to London, as well as a transcontinental high- speed railroad. Without planes, we probably would have a 150mph train connecting New York to San Francisco.

</comment><comment><USER6> Whenever I read these types of speculative predictions from the past, I always get the urge to chuckle at the things they inevitably missed, or the sillier predictions - but tempered by the ways we have failed to live up to their dreams.

(As an aside: 'strawberries as large as apples' - but how would they taste?)

</comment><comment><USER7> The one plausible idea that has not been introduced, at least how they propose, would be pneumatic tubes for delivering packages. I have to think that such a system would be more efficient and cheap at some point than delivery by freight.

<comment><USER8> Sort of related is this plan to use quadrotors for small item delivery in remote areas. It could work well in developed areas too, I think.

<http://matternet.net/>

</comment></comment><comment><USER9> "There will be No C, X or Q in our every-day alphabet. They will be abandoned because unnecessary. Spelling by sound will have been adopted, first by the newspapers." Seems this will never happen.

</comment><comment><USER10> Interestingly enough, the only big thing they really missed was the rise of the personal computer and the Internet.

<comment><USER11> "Photographs will be telegraphed from any distance", "Man will see around the world", "Grand opera will be telephoned" and "Telephones around the world" are all quite close approximations, all based on universal access to communication technology.

The "University education will be free to all ... all will receive free eyeglasses and medical care" part was a depressing read - in the US we're probably further from that now than when the article was written.

<comment><USER12> Large parts of Europe have free university education and free medical care.

</comment></comment></comment><comment><USER13> Interesting, but almost impossible to read because the scanner resolution is too low.

<comment><USER14> The full res image: [http://www.howtobearetronaut.com/wp- content/uploads/2011/07/...](http://www.howtobearetronaut.com/wp- content/uploads/2011/07/Ladies-Home-Journal.jpg)

</comment></comment><comment><USER15> Pretty interesting how many he got right or close....</comment>


What's the best privacy focused DNS host? - <AUTHOR> I've heard of 1984's, but I'm not sure if it's the best. <comment><USER2> your own, seconded.

</comment><comment><USER3> Hosting your own DoH server</comment>


MythBusters Co-Host Adam Savage on Obsession  - <AUTHOR> http://www.wired.com/video/foratv/foratv/13312836001/mythbusters-cohost-adam-savage-on-obsession/16494275001 <comment><USER2> Old submission:<http://news.ycombinator.com/item?id=404504></comment>


IPad Success not the Fruit of Apple's Labor - <AUTHOR> http://pcper.com/article.php?aid=917

<comment><USER2> He's sort of right but also sort of wrong. For an example of why he's wrong, try using a Moto Droid and notice how mediocre the experience is. This is not just because of the hardware.

Creating a high quality "computer appliance" experience is non-trivial, and Apple has done a very good job of it.

Even watching the android on iPhone 3G demo video in another thread, one realizes that extremely few people would ever prefer Android to iPhone OS, at least in the currently available versions.

Sure the differences are subtle and small, and maybe the overall edge Apple has is small... But the question is, as consumers get more demanding, what is Apple's slight edge worth?

<comment><AUTHOR> One of my friend just put his week old Google Nexus on ebay because he found Nexus's UI vastly inferior to that of iPhone. He already owns iPhone btw.

The author in article raises some valid points and I do think that Apple's applications on iPad or iPhone are nowhere near that of apps made by independent developers. But initially it was Apple that laid down the groundwork and developers are just picking up from that.

</comment></comment><comment><USER3> Sorry, but this is all wet.

"Any of these applications could have been written for other tablets."

Sigh.</comment>