SMS is not 2FA-secure - <AUTHOR> https://www.issms2fasecure.com/ <comment><USER2> This is great; it's a Princeton research project from Arvind Narayanan's (@random_walker) group, in which their team made 10 attempts to SIM-swap each of 5 different carriers, including T-Mobile, AT&T, and Verizon (all three of which were, weirdly, less secure in some ways than the 2 MVNOs they tested).

Most notably: AT&T and Verizon both use call logs to authenticate SIM swaps from people who don't know the account PIN; requestors are asked to list recently made outbound calls, or in some cases _inbound_ calls. A targeted attacker can trick a customer into making a known call (or, obviously, can simply call the customer to make inbound call records), and then authenticate with them.

AT&T uses billing statement data as a factor. But the research team was able to "spoof" billing statement data by purchasing prepaid refill cards and applying them to a target's account.

The report also identified a bunch of online services for which SMS was used not just as a second factor but, through account recovery, as a sole factor, meaning you're substantially worse off with SMS authentication than you are without it at those services. The reality is probably worse than the report highlights, since a lot of account recovery processes are informal and ad-hoc, and can be socially engineered into relying on SMS.

<comment><USER3> So how SHOULD this problem be solved? How should account recovery work?

<comment><USER4> Walk into a store and provide a government ID and the original SIM card. If customer doesn't have the sim/phone, send a recovery code to the billing address on file in lieu of the SIM card.

<comment><USER5> > Walk into a store and provide a government ID and the original SIM card.

This is how it works in Poland since September 2019, after some recent SIM- swap attacks. You can swap SIM or get a replacement if stolen only at store showing government ID. It is free of charge with Orange and not always free with T-mobile.

But this has some downsides in real life.

1) I had to walk my 88 yo Mom to the store to swap SIM card.

2) Every clerk at every shop can do that so for a determined criminal it is possible to bribe or threaten one.

3) Virtual operators (MVNOs) usually do not have physical locations and there is a dozen of them.

<comment><USER6> The problem is that the ID is still checked by the clerk. They could be bribed or tricked by a fake ID.

A recovery code snail-mailed/e-mailed to the account holder when they first open the account is the correct way to go, and if they can't provide it they need to go through a lengthy process where many factors are used to authenticate them (verify their physical address, verify their ID, ask to confirm last call records, billing details, etc).

<comment><USER7> You can require the clerk to note the document ID to avoid bribery.

<comment><USER8> How would this work exactly?

<comment><USER7> The clerk has to use some kind of online system to connect the new sim to the customers phone number. The system would obviously require the clerk to authenticate himself and could require him to enter the passport number or other document ID he checked to verify the customers identity.

If later it turns out this was a sim swapping attack you can verify if the clerk entered a valid document ID. He can't do that without having been presented a proper document, so you can tell if he checked.

<comment><USER9> Its just convenience over security. Lot of things can be done but then the extra burden that companies have to go through. Think about that people don't use app based authentication because it's inconvenient even though it matters to them. How can you expect carriers to do it

<comment><USER7> That's easy, just make the carrier financially liable for the damages caused by sim swapping attacks.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> My understanding is that you don't even need to do a SIM swap, because the SS7 signaling system is insecure. SIM Swap is likely the easiest way as wage-slave employees are quite pliable to bribes[0]. But if you want to be even more anonymous, you can apparently re-route texts remotely [1].

0: [https://www.nbcbayarea.com/news/local/mans-1m-life- savings-s...](https://www.nbcbayarea.com/news/local/mans-1m-life-savings- stolen-in-cell-phone-scam/192416/)

1: [https://www.kaspersky.com/blog/ss7-hacked/25529/](https://www.kaspersky.com/blog/ss7-hacked/25529/)

I thought both these vectors were already common knowledge to HN readers.

<comment><USER11> Yep, the security problems with the mobile system are ghastly.

\- Stingrays...

\- Operator app pushes to SIM cards...

\- Secret GSM processors and software internals

\- Voice / text / data "ciphering"

\- Protocol-level "emergency" tracking features

\- Silent SMS (sounds like its from a bad cop show but its actually a real thing it turns out.) "They do not show up on a display, nor trigger any acoustical signal when received. Their primary purpose was to deliver special services of the network operator to any cell phone." \-- sounds like it has a completely legit use...

The list goes on. It's enough to make anyone want to get the tin foil out. But at least in this case there's a simple and clear recommendation: --not to use 2-factor auth by SIM--.

<comment><USER12> > sounds like it has a completely legit use...

The original purpose of silent SMS was to send voicemail or missed call notifications to handsets, which would trigger an icon to be displayed on the device. Sending a regular SMS would be annoying as the user would have to delete it - after you've listened to your voicemail, another silent SMS can be sent to turn off the notification. Also originally SMS was stored in the SIM itself which had limited memory, so it would be not be very convenient if you didn't receive a voicemail message as your SIM was full. Remember this is a 28 year old feature of GSM.

The tracking argument seems somewhat mute, maybe when this first came to light 10 years ago it wasn't the case, but nowadays I would be very surprised if operators do not keep detailed logs of all the IMEI (unique identifier for a given device) and IMSI (same, but for the SIM) that connect to their towers.

</comment></comment></comment><comment><USER13> Not in Russia. Numerous examples exist when victim's number was linked to attacker's sim card to obtain 2FA code, then linked back to victim's sim so he does not notice anything.

This happened both by government-linked parties, where they are able to coerce providers to do it, mostly targeting prominent political opposition members. It also happened without government involvement, done by provider's personnel with sufficient access and some entrepreneur attitude.

The rule of thumb to protect against it:

\- do not use SMS 2FA

\- if you do, use a foreign SIP number with SMS capabilities

\- if you HAVE to use local sim, use SIM that belongs to someone else and noone knows you use it

<comment><USER14> > if you do, use a foreign SIP number with SMS capabilities

Any good providers? I've tried Twilio SMS forwarding, but different services (e.g. Steam) reject it for 2FA since they're pretty much considered throwaway numbers, I suppose there's some sort of blacklist

<comment><USER6> For UK numbers lookup Andrews & Arnold. They have mobile numbers from a national carrier's numbers block so no way for them to be flagged as VoIP. They work fine with both calls & SMS.

</comment></comment></comment><comment><USER15> I _want_ my things protected by a human with a process to unlock/reset/.. given some kind of proof of identity.

Because with 99.99% certainty the person that needs to unlock the account is me, and not an attacker.

Even with a dozen backup yubikeys and spare codes written down I'd _still_ be much more likely to lock _myself_ out than be attacked.

If it's one thing I have learned the hard way it's that the most dangerous person in the equation is myself. I won't trust myself with any kind of security.

<comment><USER16> My ideal solution for an ultimate reset/unlock solution would be to show up and have my DNA sampled. Impossible for me to lose the reset key there, and with appropriate DNA extraction procedures, it is nearly impossible to spoof.

<comment><USER17> The issue with using permanent characteristics for auth is that you lose the ability to revoke one credential in favor of another.

<comment><USER18> That's not a problem if you have to physically show up though, since no one can spoof that.

<comment><USER19> As another person said, you're literally leaving it everywhere you go.

If you need a blood sample, then would donating blood be considered compromising security?

Identity is what your DNA is. Password is a secret. Your DNA is not a secret.

<comment><USER20> I think requiring you to be physically present and having a human take the sample in a prescribed manner serves as an effective 'password' \- unless it's a live sample, the DNA is useless.

<comment><USER19> I think there's a misunderstanding of what is possible with DNA[0]. We take DNA from dead stuff all the time.

I will agree with "you have to be physically present" is good enough password. This is Yubikey, which works fantastic. The problem with DNA is when it is compromised - you can't throw it away/change it without exorbant effort (bone marrow transplant? and then you're simply taking on someone else's identity? is that identity theft?).

[0] [https://www.quora.com/Do-we-require-live-cells-when- extracti...](https://www.quora.com/Do-we-require-live-cells-when-extracting- DNA)

<comment><USER16> I think people are misunderstanding what is being suggested here. The idea is that, for example, to unlock your bank account, you have to go to the bank where trusted bank employees will extract your DNA and have it sequenced, resulting in you being given access again. Others cannot spoof being you in this scenario because they cannot implant your DNA in themselves.

<comment><USER19> Ah you're right. I've re-read it and it is physically present someone verifying you using your DNA.

Which I agree, that works great, but quite narrow in the the use cases at that point.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> Worth noting that this is just for US and for prepaid SIMs, from their paper "We examined the types of authentication mechanisms in place for such requests at 5 U.S. prepaid carriers—–AT&T, T-Mobile, Tracfone, US Mobile, and Verizon Wireless".

It doesn't mean that for the rest of the world SMS 2FA is completely secure, it's just a lot more difficult (or impractical/impossible) to do a SIM swap so easily. As mentioned in another comment below, SS7 vulnerabilities are another attack vector, globally available and without requiring a SIM swap.

<comment><USER22> These 5 carriers were studied, but where's the evidence that any other carrier is any better (or that you're any better off as a post paid customer of AT&T, T-Mobile or Verizon)?

<comment><USER23> MetroPCS (prepaid MVNO now something like a subsidiary of TMo) required the 8-digit PIN on the account in order to change IMEIs. A bot would take down all the info, then if/when it was to a phone you'd never used on their network before, you got put on hold to wait to talk to a human and provide your PIN and new IMEI all over again. Then you'd hang up, power off, and move your SIM. But that was ~18 months ago, before it became "Metro by T-Mobile", so I don't know.

</comment></comment></comment><comment><USER24> I thought this was going to be one of the otherwise-plaintext black and white web pages with <h1>NO.</> centered in the middle, but interestingly it's actual research, and a nice read (even if nothing new) at that.

<comment><USER25> If it's nothing new then why do people keep saying it's better to have SMS 2FA then to not have it. The research says "websites should eliminate SMS based MFA altogether".

</comment></comment><comment><USER26> The answer is no, but is it more secure than no 2FA?

Of course there are much better 2FA options, but for the general public, they are probably too complicated to use.

Everyone understands SMS.

<comment><USER27> Have you seen the prompt system, as used by Google, Micosoft, Okta, et al.?

In my strictly personal opinion, responding to a notification that asks if a login attempt is you is clear enough that people need minimal training to make use of it. This might just be me, though.

In my career, I've definitely seen people actively choose SMS over other factors on offer. It was easier for them, and in many cases shouldn't have been offered. Your point about SMS being better than nothing is wise and true and insightful, but it's perhaps not always the question as faced in practice.

<comment><USER28> They (and similar corporate 2FA solutions like PingID and similar systems used by banks) basically assume uninterrupted access to the internet which is generally a poor assumption. It often breaks down when you're traveling either due to network or roaming issues just when you desperately need access.

In all these situations, I've found companies which offer a back up SMS option very valuable since it usually gets delivered.

<comment><USER29> HOTP-based 2FA systems (like Google Authenticator) do not require internet connections.

<comment><USER30> Don't most folks uses TOTP-based schemes with Google Authenticator?

Unless you're using HOTP to mean HOTP and all extending schemes.

</comment></comment></comment></comment></comment><comment><USER31> In Switzerland, we have Mobile ID: [https://www.mobileid.ch/en](https://www.mobileid.ch/en)

It uses the SIM to implement a challenge-response mechanism where a PIN is prompted by your phone. While not perfect, it's vastly better than using SMS, without being less convenient.

I don't know if other places leverage the fact that SIMs are smart cards which are perfectly able to perform this kind of stuff given the proper infrastructure.

<comment><USER32> How does this work?

<comment><USER6> Presumably there's an applet in the SIM card that holds a key pair and allows you to sign stuff by providing the SIM PIN. You interact with it via STK which is an old standard allowing SIMs to tell the phone to draw rudimentary UIs and ask the user for input.

</comment></comment></comment><comment><USER33> Protip: security is not black and white

The word "secure" is not binary.

sms as a 2fa is secure.

Just not as secure as a authy totp account

...which is not as secure as a unclonable totp system

...which is not as secure as a hardware token based otp system

...which is not as secure as a hardware token that also requires you enter a pin and a fingerprint to activate it and only communicates using hard coded encrypted messages with the legit service that issued it.

<comment><USER34> To defeat the Authy account recovery process, you need to perform an active SMS attack (SIM swap, etc) and then prevent the target from seeing the recovery warning emails for 24 hours. Therefore, Authy customers should only tell trusted people that they are going on a weekend off-the-grid camping trip.

</comment></comment><comment><USER35> The big benefit of SMS for the website is that it outsources the problem of lost 2FA tokens. What happens if the user loses a yubikey. Or changes phones and did not back up their TOTP. With SMS authentication, even if the user loses a phone, they can go down to the local cell phone store and get a new phone on their number and be back in business without the website having to get involved.

<comment><USER36> > What happens if the user loses a yubikey.

Always buy two. ;-)

Joking aside, I've moved almost every 2FA to hard token, soft-token, or google voice. But the root of trust is still LastPass & Google. I don't see an easy way out of dependency other than power of attorney. Even worse: I worry what happens to my protected assets as I age and possibly face memory loss.

<comment><USER37> Bad idea: google will disable your google voice after some time of not logging in.

I got bitten in a bad way!

Hopefully twilio will start creating "recognized" numbers someday, as my twilio number is unusable for TOTP. There seems to be a blacklist of all twilio voip numbers.

<comment><USER36> Interesting! I generally check it once a week. Any idea what their timeout is?

</comment></comment></comment></comment><comment><USER38> The title is mangled, because someone misparsed the question.

The question is "Is SMS 2FA secure?", not "Is SMS 2FA-secure?" There is no such property as 2FA-secure.

Title should read: "SMS 2FA is not secure".

<comment><USER39> I agree. "SMS is not 2FA-secure" implies SMS is not suitable for 2FA at all. In reality, SMS 2FA is still very valuable to most people, even though it's not secure enough.

</comment></comment><comment><USER40> But how else are you supposed to encourage users to give you their phone numbers so you can track them better?

<comment><USER41> No company would ever do that, right? Especially a social media company. Clearly there would be public outrage and their stock would plummet.

<comment><USER42> For anyone who doesn't detect the sarcasm or is unfamiliar: [https://techcrunch.com/2018/09/27/yes-facebook-is-using- your...](https://techcrunch.com/2018/09/27/yes-facebook-is-using-your-2fa- phone-number-to-target-you-with-ads/)

</comment></comment></comment><comment><USER43> Just use a token like yubikey. I have a small fleet and am very happy with the decision.

The only problem is there are very few services that get it right. Get it right means support multiple tokens and allow to truly disable any other means of logging in or recovering the password.

Most services seem bent on allowing many ways of logging in without giving a choice. For example, they will advertise they use 2fa tokens but then if you can't produce one they will still allow you to log in with SMS or mail (ie. password recovery by mail). Facebook will not even let you set up tokens without having SMS set up as a factor and the phone number verified.

I hope slowly developers will get more aware and they will be better tooling (and stack exchange answers to ctrl+c ctrl+v...) to do it correctly.

<comment><USER44> What about Authenticator? Maintaining a small fleet of yubikeys costs as much as a whole phone.

[https://play.google.com/store/apps/details?id=com.google.and...](https://play.google.com/store/apps/details?id=com.google.android.apps.authenticator2&hl=en_US)

<comment><USER45> Know that if your single phone dies with all your totp credentials, you're sunk.

<comment><USER46> Only if they neglected to offer backup codes (which anyone who does TOTP should).

Otherwise, you can just grab a few backup codes out of your fireproof safe and register your new totp code, or go to the bank and get them out of your bank vault.

Sure, the fireproof safe costs as much as a few yubikeys, but if you go the yubikey route you both need the yubikeys and a fireproof safe and bank vault for your spare yubikeys too.

<comment><USER47> Fireproof safe? Just use a text file and encrypt it.

Or a password manager file only for TOTP backups.

<comment><USER43> Don't use text file. Use regular piece of paper, put it in tamper-evident envelope and keep register with events regarding envelopes.

When you take something from envelope, note the date, the why, the number of envelope you opened and the number of envelope you then put the piece of paper in. Every time you open envelope check with the register that the numbers agree.

</comment></comment></comment></comment></comment></comment><comment><USER48> And yet my bank (Chase) only supports email and sms 2fa with no option for OTP/TOTP. Is this just a institution dragging their feet or are there more regulatory reasons why they won't allow more secure authentication?

<comment><USER49> At least they offer codes via email. I can (and do) secure access to my email account and domain registration with a very long password and a Yubikey. That's "good enough" for my purposes.

<comment><USER50> Most email is unencrypted during transit, so a state level adversary can still easily intercept it. For most people this is however sufficient.

<comment><USER49> Absolutely, a state-level opponent could get up to some shenanigans though I would argue that a state has much easier methods to go cracking into my Visa card. My threat model doesn't include nation states targeting me specifically because, simply, if one comes after me I am screwed anyway.

As for email being unencrypted, I think most of it now is encrypted during transit (thanks to the Big Two providers knocking points off a spam score if a message does come via TLS) and even if it weren't the password is also not known so the second factor is not useful. For example, I just tried to log in to chase.com and the code they emailed me at 1752 MST is 067315.

If I've been phished so hard that posting this is useful, again I'm screwed.

</comment></comment></comment></comment><comment><USER51> Is SMS 2FA Secure? No, I agree.

Is SMS 2FA enough for most of the people today? Yes

Is SMS a cost-benefit solution for most uses? Yes

<comment><USER52> Is offering or forcing SMS 2FA and not offering an option for only TOTP asinine? Yes.

It's free, and requires a tiny bit of additional configuration to enable. No reason not to offer it.

<comment><USER53> In a previous company, one of the employees enabled 2FA for their staff account (it was mandatory), stored the backup codes on his phone (presumably as a photo) and it fall in the ocean the next day.

With large enough numbers, you'll see everything, but you don't even need large numbers to get people whose lives are made more difficult by technology.

<comment><USER52> Yes, that is exactly what I want. Life should be much more difficult without the TOTP and backup codes, so much that it takes a great deal of resources to get around it, if at all possible. Maybe even providing heavy documentation such as a Facetime call with various proof so that fraudulent actors are sufficiently deterred.

<comment><USER54> Dude. If somebody wants into your account specifically, they'll get into it. 2FA, specifically SMS based 2FA, is really about the provider getting mass compromised because people recycle their password across all their sites.

It great for keeping people using scripted attacks against a huge list of accounts. It isn't really to keep people specifically after your account out.

If somebody wants _your shit_ and _specifically your shit_.... they'll get it...

<comment><USER55> > If somebody wants your shit and specifically your shit.... they'll get it...

How? I don't think Brian Krebs has been hacked, even though he's extremely targeted by hackers (his site is literally the benchmark for performing DDOS attacks on).

</comment></comment></comment></comment></comment></comment><comment><USER9> DontPort.Com - I built this to fix this. I've been a victim of this 4 times and was too much frustrated. Unfortunately Sim swap is only one way to get your 2FA but the risks are much higher which I am working to solve one by one

<comment><USER56> Which common cases does the insurance cover? What uncommon cases does it not?

How do you protect against an insider attack?

</comment></comment><comment><USER57> You know what's funny? LinkedIn is supposed to be a 'professional' social network (Microsoft owned) and a friend of mine was asked to add a phone number 'For security purposes'. I knew this was suspiciously involving 2FA SMS + a bonus of spam callers and I told him to press "Not Now". Whilst the world is moving to U2F and time-sensitive codes, a security system using SMS 2FA is now equivalent to a single PC running Windows XP in a bank.

But its not just LinkedIn. Its a huge list of major companies including some FAANG ones too. Oh dear.

<comment><USER51> Not true. Not true by far. That's an over statement. 2FA is only one of two factors, you need the the password, you need the mobile number and you need to obtain a duplicate or being close to your victim.

You should be worried if you are a POI or you are being targeted personally. And if it is so, SIM Swapping it's just one option and if it doesn't work there are other methods (breaking in, stealing yubikeys, mobiles...)

<comment><USER58> You don't always know if you are a target.

</comment></comment></comment><comment><USER59> All of this happens because we've outsourced digital identity to the telecom companies. Telecom companies are not competent at establishing identity. It's not their job. There is only one entity that is the real root provider of identity and that is the government.

We are never going to get the benefits of digital identity until the government wakes up and brings its services into the digital age.

</comment><comment><USER60> No, it isn't at all. Moreover, sim swap is not necessary at all.

Anybody with direct access to SS7 can send a fake roaming request for your sim card.

</comment><comment><USER61> Definitely not. In my home country the banks use SMS 2FA.

It's a complete shitshow. Occasionally syndicates manage to get both sides of 2FA lined up (insiders) and clean out someone's account.

Then the bank says not my problem - you didn't keep your password safe. And the cell provider says not my problem - not intended as security mechanism. Leaving the customer poor and sht out of luck.

</comment><comment><USER62> Want to point to the Google blog article about the effectiveness of different 2FA techniques. SMS is between 76% targeted -100% autoamted bot effective.

[https://security.googleblog.com/2019/05/new-research-how- eff...](https://security.googleblog.com/2019/05/new-research-how-effective-is- basic.html)

</comment><comment><USER63> I don't really trust Authy being used as much as I think anyone else would. For the sites that I have used it with, an Authy account, or just the app is not required. Therefore codes are just sent over plain SMS.

Consider this scenario. Twitch now enforces the use of all accounts that want to stream all require the use of 2FA, after the whole artifact fiasco. Anyone over the age of 13 is able to do this. I don't expect everyone at this age to have a phone number, and I assume these people would rely on their parents phone to pass this.

As authy is completely optional, people may choose to not require another app for their account, in effort of just quickly jumping through another hoop blocking them from going live, or the device owner not wanting to have an app installed.

It just feels weird knowing that this can be a point of failure for a service that solely relies on a single 2FA method that could from an attack like this down to the individual and how they operate.

</comment><comment><USER64> I'm surprised more websites aren't taking advantage of TouchId. It's so easy for people to use.

Even login.gov supports it!

[https://www.slashgear.com/chrome-is-adding-touch-id-and- fing...](https://www.slashgear.com/chrome-is-adding-touch-id-and-fingerprint- sensor-support-for-web-apps-14545704/)

<comment><USER6> Touch ID and other biometrics are enforced locally.

The device is first enrolled, the website gives the device a secret value which the device can put in its secure element. When needing to authenticate again the device checks biometrics _locally_ and if correct then the secure element releases the secret value which is then either passed onto the website or used as part of a challenge-response authentication.

This means if you lose or reset your device you can't get back in despite having the right biometrics.

<comment><USER64> Yes, that's a security feature. It's also true for Google Authenticator, by design. You cannot officially back up/share codes because of the potential vulnerabilities that a backup would open up.

Yubikey has the same problem you describe. If your key stops working, you'd also be locked out. Yubikeys can spontaneously stop working in my experience.

To mitigate this, sites like login.gov allow you to add multiple devices, so you can have it on e.g. your laptop and your phone, and yubikeys if you'd like. I generally do all three for important sites (or multiple Yubikey when touchID is not offered).

Anyway, my point is that offering TouchID makes a more secure 2FA very, very convenient for the average person. I'm just surprised more developers haven't offered it even though it's been in Chrome for a couple years.

</comment></comment></comment><comment><USER65> And then there was this... [https://www.vice.com/en_us/article/5dmbjx/how- hackers-are-br...](https://www.vice.com/en_us/article/5dmbjx/how-hackers-are- breaking-into-att-tmobile-sprint-to-sim-swap-yeh)

</comment><comment><USER66> This post [1] makes a very good point...

Using a few old Google accounts, I experimented with Google's account recovery options and discovered that if a Google account does not have a backup phone number associated with it, Google requires you to have access to the recovery email account OR know the security questions in order to take over an account. However, if a backup phone number is on the account, Google allows you to type in a code from an SMS to the device in lieu of any other information.

[1] [https://tech.vijayp.ca/adding-a-phone-number-to-your- google-...](https://tech.vijayp.ca/adding-a-phone-number-to-your-google- account-can-make-it-less-secure-f1cc7280ff6a)

</comment><comment><USER67> In many European countries SIM cards are actually locked to a number.

If you change your SIM card you'll have to reset your 2FA setup on the banks site. It's done on the carrier side.

I still hate it tho, as you are locked to a phone number and it sucks when you move countries a lot.

<comment><USER6> This relies on carrier cooperation. Given they are the ones that caused this shit-show in the first place I wouldn't trust them to make it right. An insider capable of SIM-swaps would also be able to override this mechanism.

</comment></comment><comment><USER68> The sad part about all of this is the complexity of the new solutions. Webauthn is strong but fiendishly complex. How did we get here?

Websites used weak hashes (md5 and sha1... efficient to compute and attack) to store passwords and allowed users to set short, weak passwords (12345, letmein).

Long passwords randomly generated by password managers and stored as strong hashes (Argon2id) by websites are secure, not guessable and even difficult/expensive to attack offline when the database is dumped.

This approach is simple and easily understood by everyone involved (users and site admins) and would be suitable for the security of 99% of websites. Leaving 'Account Recovery' as the only remaining challenge.

</comment><comment><USER69> It's really too bad nobody tests Google Fi. I wonder if they would hold up better?

</comment><comment><USER70> This research doesn't seem complete; particularly the use of a phone number for authentication, that isn't SMS.

Facebook Messenger uses a phone number either tied to your Facebook Account, or identified as the phone number on your mobile device, to immediately log you into a Messenger account, with zero authentication. They literally ask you "Is this your account?" and you just click "Yes" and you are in that account. Even if it's not yours.

If you use prepaid phone numbers, or link one to your account, you can often get into Facebook Messenger accounts that aren't yours.

</comment><comment><USER71> The odd thing about SMS 2FA is the amount of critical services that rely on it as the only method of extra protection.

You want your: bank, utility provider, entity-that-has-lots-of-personal-data- on-you to offer other secure options.

This might be a costing issue though. When your customers number in the millions, your call center is probably handling thousands of "im locked out" issues per day and these need to be handled in x-minutes. Other security options might cause the time meant to handle these scenarios to increase and SMS is generally 'simple' compared to them.

</comment><comment><USER72> The only good solution at this point is to legislate cell carriers to make SMS more secure. Everyone perceives it as secure, everything uses it for auth, and it aught to be secure for its own sake.

<comment><USER73> The problem here is not SMS itself, but that it's trivial to socially engineer most operators into transferring somebody else's number to you.

(FWIW, SMS is also insecure in that it's fairly easy to passively snoop on SMS comms, but that's a separate problem.)

</comment></comment><comment><USER74> In Turkey if you change your SIM card your bank is notified and you cannot login to your bank account. You have to re-validate yourself with a long phone call and re-create a new password.

</comment><comment><USER75> Word of warning if you know somebody who uses Bank of America: their customer support has a mechanism to push you a 2nd factor code over SMS and then they actually ask you for the code over the phone. The text message looks 99% identical to the 2nd factor code you get when normally logging in to your bank account.

Support does this to 'verify your identity' and authorize doing arbitrary things like even moving $100,000 out of your bank account.

And no, their security team won't fix this.

</comment><comment><USER76> Apple needs to help move the needle on 2FA/U2F/webauthn. Unfortunately you can even create an apple Id without a dreaded SMS "authentication".

</comment><comment><USER77> This may explain why I get called every week from all over Europe by people that think they missed a call from me.

I knew it had to be a scam, but hadn't put the pieces together.

</comment><comment><USER78> uh.. No, and NIST has been saying this for about 5 years.

</comment><comment><USER79> Apple is forcing users who upgrade to Catalina to use 2FA and if you do not have multiple devices this defaults to SMS.

There is no option to turn it off.

</comment><comment><USER80> Ran Bar-Zik, from Israel, created a technique to hack most voice 2FA by using a weak voicemail password. It was largely used in 2019 to hack Brazilian politicians, including state ministers. The hacked telegram messages were passed to Glenn Greenwald, linked to Assange.

<comment><USER81> Do you have any sources on that (the specific technique that was used)? Google returned nothing.

AFAIK, they were hacked using plain simple SIM swap/cloning.

<comment><USER80> [https://politica.estadao.com.br/blogs/fausto-macedo/wp- conte...](https://politica.estadao.com.br/blogs/fausto-macedo/wp- content/uploads/sites/41/2019/07/DECIS%C3%83Ospoofing.pdf)

(Portuguese)

</comment></comment></comment><comment><USER82> It would be nice if the carriers allowed you to specify you wanted to restrict SIM swapping. When I lost my 3 SIM to get the number transferred to a new one I went to a 3 store with my passport. I'd be fine with that being the only method they'd allow.

</comment><comment><USER83> I have been saying this for years.

The amount of knowledge one needs to port a phone number is unbelievably little, and peoples very nature to be helpful works against us. Up until maybe just very recently you needed the account number of the phone number and the last four of a social security number... sometimes, just the account number. Also, the last four of one's social security number is perhaps the shittiest way to authenticate _ANYTHING_. For many years, a lot of sites online would show you the last four of an account holder's SSN (and some places still probably do) if you have an email address, correct name, and phone number or physical address.

Getting the account number would likely be even easier thanks to helpful store reps... Just go in and make up an excuse why you need it or forgot it, it's like "social engineering 101" because it seems so benign to most people. You already know the name, address, and phone number-- you just "forgot" your id at home... Or one could just listen to them call each-other write down their info and then call another store.

With those two things in hand, the phone number is pretty much the attacker's, and getting it back would take more than enough time for extensive amounts of damage... ESPECIALLY if that is the only line on the account (or they took all the lines)... I'd guess a bare minimum with near immediate recognition of the real problem (your number heisted) and police involvement, probably a minimum of ~12 hours.

So, if phone numbers are so bad why are they ever used? IMHO, that's because they aren't to provide security, they're to provide easy tracking between your virtual life and your physical one. You're only securing the businesses data pipeline, not your personal data.

If you want 2FA (and everyone should) use Google Authenticator or a Yubikey... or whatever I'm not trying to shill brands just ideas that work.

</comment><comment><USER84> Unrelated -- I love how the domain name is literally the average Google query for when this becomes breaking news. Clever to make your domain name a literal Google query if you want to spread an idea...

</comment><comment><USER85> Now I only need to find out on which ones of my 200+ accounts this feature is enabled… Honestly, it would be easier for me if the EU just made it illegal, forcing services to disable it for me.

<comment><USER42> Why? It's not "secure" but it's more secure than nothing.

The paper mentions some websites that claim to use SMS 2FA, but actually use SMS as a single factor for password resey. While that's really bad I think the solution is to fix those broken implementations not to stop using SMS 2FA everywhere in favor of using nothing.

<comment><USER25> The paper also said, "websites should eliminate SMS based MFA altogether".

</comment></comment></comment><comment><USER86> I know a few people who have been hacked with this method via t-mobile in order to control chat rooms on telegram and steal crypto.

According to this paper, t-mobile has the smallest surface area, which is sad.

<comment><USER87> Why sad? _someone_ has to have the smallest

Would it be less sad if it was Verizon?

<comment><USER88> To him it's sad because "I know X is vulnerable and X is best" means he's at risk to the same thing as his friend no matter what he does.

</comment></comment></comment><comment><USER89> Worth having a private number on a low or PAYG plan and use that for your security separate from your main mobile number.

After all, most have a spare phone and great use for those Nokia's.

<comment><USER37> This is my conclusion too. A payg that you recharge with cash, so it is unliked to your name in the telecom operator databases is the best protection against rogue employees.

</comment></comment><comment><USER90> What's with all the redacted entries? Without some context, I assume that these are companies that threatened some sort of legal action if their name was published?

<comment><USER91> Later in the paper they mention that they're temporarily redacted due to responsible disclosure rules.

<comment><USER90> They say there are 361 sites pulled from TwoFactorAuth.org's list of sites, and they were able to access 145 of them.

In describing the set they initially drew from, it seems like they've described the 17 redacted sites simply by describing their complementary set (the 128 sites that are secure).

</comment></comment></comment><comment><USER92> [http://isThisYourPaperOnSingleServingSites.com/](http://isThisYourPaperOnSingleServingSites.com/)

</comment><comment><USER93> I wish more websites (and other application protocols) would support client- side certificates in addition to the username and password for authentication.

</comment><comment><USER94> Where were these five providers based? Was it just in the US? I wonder if the controls are more stringent; my suspicion would be that they are in Europe.

</comment><comment><USER95> What about google voice? I've noticed a lot of sites block it, but it seems as safe as email for 2fa.

</comment><comment><USER96> Haven't we known this for quite some time? I'm just amazing many places still use it!

</comment><comment><USER97> Why don't they just do two verification charges between .01 and .99 to the card on file.

</comment><comment><USER98> Could using an im service like Matrix to send the authentication token be more secure?

</comment><comment><USER99> Does anyone know the reason for all the Redacted entries on the interactive datasets?

</comment><comment><USER25> So can we finally stop saying it's better to have SMS 2FA then to not have it?!

</comment><comment><USER100> Short answer: no.

Long answer: nooooooooooo.

</comment><comment><USER101> Betteridge's law of headlines is an adage that states: "Any headline that ends in a question mark can be answered by the word no".

[https://en.m.wikipedia.org/wiki/Betteridge's_law_of_headline...](https://en.m.wikipedia.org/wiki/Betteridge's_law_of_headlines)

<comment><USER102> The key part here is to only post this comment under headlines for which the answer in "no".

<comment><USER101> What I normally do is remember the law, and generally I don't click through headlines which are questions.

But this one felt more like it would be something a little deeper.

So then I clicked through - and the page fills up with the word "no".

So I feel pretty duped.

</comment></comment></comment><comment><USER103> Isn't perfect the enemy of the good, especially in security?

</comment><comment><USER104> SMS is worse than 1FA.

</comment><comment><USER105> Would be freaking awesome if Wells Fargo understood this...

</comment><comment><USER106> People always focus on SIM swaps and signal security, but neither of those apply to Google voice numbers. So in the context of Google voice, is there still any reason to not use SMS 2FA?

<comment><USER52> Bank of America doesn't send SMS to google voice phone numbers.

<comment><USER107> What happens if you port to Google voice, can you no longer access your account?

<comment><USER52> I don't know, all I know is I can't receive BoA 2FA SMS on my Google Voice number. This is what BoA says:

>You are consenting to be contacted at the phone number selected for the purpose of receiving an authorization code. If you selected text message, Wireless and text message fees may apply from your carrier. Supported carriers include: Alltel, AT&T, Cellular One, T-Mobile, Virgin Mobile, U.S Cellular and Verizon Wireless.

Although I just remembered this person on Hacker News replied to me 1+ year ago that their Google Voice number does work with BoA 2FA SMS, so maybe it's just my specific GV phone number?

[https://news.ycombinator.com/item?id=18196014](https://news.ycombinator.com/item?id=18196014)

</comment></comment></comment></comment><comment><USER108> When my gf lived in Malaysia, she added her phone number to FB and forgot about it. Years later, after having moved back to Vietnam, the number was recycled and someone was able to use that number to gain access to her FB account and reset the password.

Getting access back to her account took a bunch of steps, including adding her current number.

The interface for FB really makes it seem like you might lose access to your account if you don't provide them with your number. Even better is that FB exposes a small list of your friends (and the total count) of everyone who has given them their phone number.

tl;dr: Don't add your phone number to FB.

</comment><comment><USER109> Compared to what?

Better than not having it? Yes.

Better than committing a 4,096Kb PK to memory and confirming all interactions with mental arithmetic? No.

<comment><USER110> The article says "We found 17 websites on which user accounts can be compromised based on a SIM swap alone", that seems like a pretty clear indication that it can be worse than nothing.

I happen to think the benefits of SMS 2FA, even when working as intended, are negligible. It seems like a bad idea to waste the finite amount of developer good will we have asking services to implement it.

Literally the only attack that SMS 2FA has any impact on is credential stuffing, and even then it's debatable. Credential stuffing is using the credentials stolen from one service to compromise another. If you don't reuse passwords, then you don't need SMS 2FA.

If you do reuse passwords - then it seems impossible you're not also vulnerable to phishing. After all, you're already willing to hand over your credentials to anyone who asks. SMS 2FA is not a solution to phishing, as the tokens themselves can be phished.

<comment><USER109> If you can compromise the account, based on a SIM swap alone, then that site has 1FA (The phone number).

2FA requires you to have 2 factors at the same time. e.g. When I log onto amazon from a new browser with valid username+password it additionally requires me to confirm via my phone number.

1or1FA (e.g. reset your password via SMS if you forget your password) is just increasing the attack area on 1FA (would be more secure without it).

Problem it's trying to solve, is that it's conventionally unacceptable to lock people out of their accounts.

<comment><USER110> Let's say that it wasn't you who logged in with a valid username and password, it was an attacker.

Under what circumstances does the phone number prompt prevent the attacker from accessing your account?

Perhaps they used phishing? Then they can just phish the SMS code as well.

Perhaps they're a MITM over an insecure channel? Then they can just wait for you to enter the SMS code.

Perhaps you installed their malware? Then they can just inject some code into the browser.

</comment></comment></comment></comment><comment><USER111> From the past few months I'm collecting all resources on the use of SMS as a 2FA: [https://github.com/kdeldycke/awesome-iam#sms-based- authentic...](https://github.com/kdeldycke/awesome-iam#sms-based- authentication)

TL;DR: don't.

</comment><comment><USER112> The TLD is the question, the title is the answer.

</comment><comment><USER113> No

</comment><comment><USER114> No Its not

</comment><comment><USER115> Hhhhhh yes you can check more details [http://www.jobsfinderuae.com/2020/01/accountant- jobs/](http://www.jobsfinderuae.com/2020/01/accountant-jobs/)

</comment><comment><USER116> Did we ever seriously doubt that? 2FA is just a made-up reason to have your phone number anyway, all the services that require that don't really want anything other than that.

</comment><comment><USER117> HN seems to be getting a lot of these submissions lately where the question asked in the title is the same as the domain name. Sometimes the content of the page doesn't even answer the question.

Feels like some kind of spammy PageRank manipulation going on. I'm happy to be wrong about this, but I wanted to see if anyone else has noticed. Maybe I'm just smoking crack waffles again.

<comment><USER44> This is clearly a PR effort associated with the research paper hosted on the site.

It's a single-purpose domain name, so the Name of the domain is the same as the Name of the content on the domain.</comment></comment>


Check out the pizza-related search spikes on NYE - <AUTHOR> http://correlate.googlelabs.com/search?e=pizza&t=weekly

<comment><AUTHOR> Also, apparently, the midwest really likes pizza: [http://correlate.googlelabs.com/search?e=pizza&t=all](http://correlate.googlelabs.com/search?e=pizza&t=all)</comment>


The state has lost control: tech firms now run western politics - <AUTHOR> http://www.theguardian.com/commentisfree/2016/mar/27/tech-firms-run-western-politics-evgeny-morozov <comment><USER2> I had a hard time reading this. It felt like a long way to try to make a point that never seemed to be made in a convincing way.

The only thing clear is the author's lament for tech firms, silicon valley and capitalism. There are really few if any points made that are anchored in reality.

I'm not quite certain how we can pin disfunctional government on Facebook and Google. If anyone else figures it out please do post here.</comment>


The Startup Myth - <AUTHOR> http://www.sixmonthmba.com/2009/03/the-startup-myth.html

<comment><USER3> I thought it was a great article.

The author is basically telling people not to create bubble startups. If you want to create a business, create one that has value. Don't just create a startup for some romanticized ideal.

It is sound advice, even if it cuts a little too close to the bone for some.

<comment><USER4> i feel like it would be a great article if it weren't playing totally on semantics to write a sensationalist article.

he's basically labeling startups and startup culture as the "bad business" subset of entrepreneurs. it is not the case that all startups are bad businesses, nor is it the case that all entrepreneurs are running good businesses.

he could totally skip all of that labeling and write an article on running good business.

</comment></comment><comment><USER5> I took an eighteen month detour from my own business to go work for another startup. I was broke from trying to get my company to break-even, and they were boomer-aged Microsoft millionaires who told great stories about their product and the funding that was right around the corner. That was in 2006, and they're still telling the same story today. The difference now is that they no longer have any employees, and the CEO is essentially living out of his car.

They obviously didn't eat Ramen (although it's possible they might now), and they weren't 60 days late on the power bill, but they were just as absorbed with the idea of the startup. For them it was about telling the story of their imminent success, and more importantly, getting other people to believe it.

You obviously have to believe in what you're doing, and of course the startup atmosphere can be incredibly thrilling, but the author's note that "we need more people wanting to build companies, not startups" rings particularly true for me. I don't think he's saying that every startup should plan to become a staid, monolithic corporate giant. I think he's saying that you should have a credible plan for organic growth. Even if your end goal is to get funded so you can get out, this will be a thousand times more likely if you can demonstrate multiple consecutive quarters of growth. Then even if you don't get funded, you've at least created a viable business.

</comment><comment><USER6> Is there really a huge faction in our community who isn't in business to make money? When I left my middle class desk jockey job to work for trailbehind.com the chance (however small) to create and own a valuable business was the main motivator.

I think that the author has a fundamental misconception about what a "badge of honor" is. Soldiers don't spend months or years in the trenches with the purpose of earning "badges of honor" they go into the trenches because they have to. The badges are awarded after the fact to recognize the accomplishment of putting everything on the line for some greater purpose.

While we are not soldiers, we do risk big chunks of our life because we believe in our products and businesses. We eat Ramen Noodles and scrimp by because the pursuit of our goals require us to, not because we like it.

I also think that the author has misconstrued the "badges of honor" concept. People get badges of honor as a reward for being in the trenches, they don't go down into the trenches with the main intent of earning a badge of honor. You are in the trenches because you have to be. While

<comment><USER7> There is a difference between being in it to make money and having no plan to make that money. Those that have a plan find it very easy, even in todays climate to make money. Those who don't won't. The future is clear, make money or perish.

I often have to point out to entrepreneurs the difference between an idea, project and a business, regardless of how many "eyeballs" they have.

</comment></comment><comment><USER8> _Entrepreneurship is about turning every dollar of resources spent into something worth more than a dollar._

Including your time. Even if you can live on $100 per week, your time is still worth $100 per hour, so that's how much value you need to create with every hour.

But it doesn't need to be $100 cash, but of value. Warren Buffett doesn't evaluate companies by their current price, but by their likely future worth. A key factor in evaluating worth is their economic moat - their enduring advantage over competing companies. Therefore, creating more than $100 of economic moat in an hour is a good use of your time.

Of course, Buffett doesn't invest in startups or even tech companies like IBM. Tech is too unpredictable: a product that is good + a market that actually wants it. Your success or failure is not predictable from data points - <USER2> pretty much _is_ the first data point.

</comment><comment><USER9> Umm, considering the hard attitude the article presents, the actual content is really rather fluffy.

I can think of a word in response to this article: straw-man.

<comment><USER4> i think it might be hasty generalization or slippery slope instead of straw man.

there are plenty of people who do live the life as described. however, there are plenty more who don't. he's attacking a fraction like its the whole.

<comment><USER9> You may well be right.

Regardless, the article is clearly less than great.

</comment></comment></comment><comment><USER10> ... get a billion or die trying!

I don't buy the "create big companies to create many jobs" mantra. A one man company creates one job, and leaves one more job available in the market for someone else. That's a contribution in my book.

<comment><USER11> The job market is hardly a zero-sum game, though. By growing from a startup to a major player in an industry you helped found, you can do a lot more economic good than you would by creating one job and leaving one available in the existing market.

</comment></comment><comment><USER12> > _"The world would be a much better place if less people tried to build > startups."_

fewer people

I really don't want to be a grammar nazi, but starting off with an error in the very first sentence made me less interested in reading the rest of the article.

Yes, I realize that English grammar is changing, that countable/uncountable distinctions are less important than they used to be, and that "less + a countable noun" doesn't sound bad to everyone anymore. But to some of us, it looks like it was written by a struggling high school student. There are so, so many things to read on the internet that readers can be driven off by minor blemishes at the beginning of a piece.

</comment><comment><USER13> Well done. I love what I do... I hope that is what it means to run a start-up.</comment>


Analysis: Why politicians don't just say what they mean - <AUTHOR> http://www.cbc.ca/news/world/why-politicians-and-academics-don-t-just-say-what-they-mean-1.2618268

<comment><AUTHOR> It's a problem not limited to politicians. Startups often use a lot of jargon to sound further along than they currently are. So, I thought this would be a helpful reminder to focus on clarity in our writing.</comment>


Ask HN: How to sell an App Store company with 15m installs? - <AUTHOR>

(I'm a long time user of HN, just posting under a new account for this)
I've been developing on the App Store for about 9 months and have built a set of apps that collectively have over 15m installs. Among these have been several top 100 apps.
I'm considering selling them all, and I'm wondering how to go about it. Obviously if a buyer knows you want to be bought, it instantly lowers the price.
I'd really appreciate any thoughts/suggestions anyone has? <comment><USER2> First off, I'm going to disagree with the assumption that knowing you want to sell instantly lowers the price. Businesses get bought and sold every day. There's no shame in wanting to sell a company you've built. As long as you don't come across as desperate to sell (which it doesn't sound like you are), then you'll be able to get a fair price. In fact, you'd face the possibility of getting a better price if there are multiple bidders.

Secondly, you might want to consider selling off the individual applications piecemeal. There's the possibility that you'd get a larger amount of money by splitting them up then by selling the whole company.

There are a couple of other things that will affect the price you receive. Are the apps still in the top 100? Are they paid apps, and if so, what kind of cashflow are you seeing from them. If they're free apps, are they monetized in any way (in-app purchase, advertising, etc.)?

I guess the other question is how much you're looking to get for everything. Hell, depending on your answers to the questions above, _I_ might be willing to buy you.

In any case, I'm curious why you're considering selling everything off. Would you be willing to share with us the reasons?</comment>


Amazon Web Services are down - <AUTHOR> http://status.aws.amazon.com/?a <comment><USER2> Some quotes regarding how Netflix handled this without interruptions:

"Netflix showed some increased latency, internal alarms went off but hasn't had a service outage." [1]

"Netflix is deployed in three zones, sized to lose one and keep going. Cheaper than cost of being down." [2]

[1] <https://twitter.com/adrianco/status/61075904847282177>

[2] <https://twitter.com/adrianco/status/61076362680745984>

<comment><USER3> "Cheaper than cost of being down." This is very insightful. Many of us look at the cost of multi zone deployments and cringe, but its a mathematics exercise. (.05 * hours in a year)*(cost of being down per hour) = (expected cost of single zone availability). Now just compare to 2-3x your single zone deployment cost. Don't forget the cost of being down per hour should include lost customers as well.

<comment><USER4> I'm actually surprised if incurring 50% extra hardware costs really is cheaper than the cost of being down. If Netflix is down for a few hours, then it costs them some goodwill, and maybe a few new signups, but is the immediate revenue impact really that great? Most of Netflix's revenue comes from monthly subscriptions, and it's not like their customers have an SLA.

<comment><USER5> Actually, they do. and Netflix proactively refund customers for downtime. Usually it's pennies on the dollar, but i've had more than refund for sub 30 minute outages which have prohibited me from using the service.

Netflix are very very sensitive to this problem because it's much harder for them to sell against their biggest competitor (local cable) since they rely on the cable to deliver their service. If the service goes down, then the cable company can jump in and say, "You'll never lose the signal on our network" -- blatantly untrue, but it doesn't matter.

When you're disrupting a market, remember that what seem trivial is in fact hugely important when you're fighting huge well-established competition :)

</comment></comment></comment></comment><comment><USER6> Amazon's EC2 SLA is extremely clear - a given region has an availability of 99.95%. If you're running a website and you haven't deployed across across more than one region then, by definition, your website will have 99.95% availailbility. If you want a higher level of availability use more than one region.

Amazon's EBS SLA is less clear, but they state that they expect an annual failure rate of 0.1-0.5%, compared to commodity hard-drive failure rates of 4%. Hence, if you wanted a higher level of data availability you'd use more than one EBS volume in different regions.

These outages are affecting North America, and not Europe and Asia Pacific. That's it. Why is this even news? Were you expecting 100% availability?

<comment><USER7>
        Amazon's EC2 SLA is extremely clear -
        a given region has an availability of 99.95%.
        If you're running a website and you haven't
        deployed across across more than one region then,
        by definition, your website will have 99.95%
        availailbility. If you want a higher level of
        availability use more than one region.

Good point.

let P(region fails) = 0.05% and let's assume (and hope) that the probability of failure of one region is independent of the state of the other regions.

P(two regions fail) = P(one region fails and another region fails) = P(region fails) * P(region fails) = 0.05% * 0.05% = 0.0025%

Making your availability = 100% - 0.0025% = 99.9975%

Ultimately it's more of a business decision if you want to pay for the extra 0.0475% of availability. I would think (or hope) that most engineers would want it anyway.

The numbers at this size appear insignificant. How would one (say an engineer) convince "the management" that the extra 0.0475% of availability is worth the investment/expense?

<comment><USER8> "let's assume (and hope) the probability of failure of one region is independent of the state of the other regions."

In practice, that's not true, and it's not true enough to ruin the entire rest of your calculations. For Amazon regions to function independently, they'd have to be actually, factually _independent_ ; there is no interaction between them. The reaction to one node going down is never to increase the load on other nodes as people migrate services, etc. There's fundamentally nothing you can do about the fact that if enough of your capacity goes out then you will experience demand in excess of supply.

If you want true redundancy you will at the very least need to go to another entirely separate service that is not Amazon... and if enough people do that, they'll break the effective independence of that arrangement, too.

(This is a special case of a more general rule, which is that computers are generally so reliable that the ways in which their probabilities deviate from Gaussian or independence tends to dominate your worst-case calculations.)

<comment><USER7> I agree with you 100% that they're not independent, but I don't know enough about the data to model the probabilities of failure and availability in a HN comment :-)

After today's event, it would certainly be interesting to see how resource consumption changed in other availability zones and at other providers during this outage.

I wonder if that could be measured passively? What I mean is, by monitoring response times of various services that are known to be in specific regions and seeing how that metric changes (as opposed to waiting on a party that has little-to-no economic benefit to release that information.)

</comment></comment></comment></comment><comment><AUTHOR> Current status: bad things are happening in the North Virginia datacenter.

EC2, EBS and RDS are all down on US-east-1.

Edit: Heroku, Foursquare, Quora and Reddit are all experiencing subsequent issues.

<comment><USER9> Not all EC2 & EBS instances are down. I have several in US-east-1a and 1 is down, while all of the others are working.

<comment><USER10> Same with us. About 10% of our 700+ volumes are having problems right now.

It's hard to tell for sure since there isn't any load.

</comment></comment></comment><comment><USER11> 4/21/2011 is "Judgement Day" when Skynet becomes self aware and tries to kill us all. <http://terminator.wikia.com/wiki/2011/04/21>

I am just a little freaked out right now.

<comment><USER10> Don't worry. If skynet is in EC2, we'll be fine.

<comment><USER12> You don't understand, Skynet is using all Amazon resources, hence the outages ;-)

Amazon have stated many times that amazon.com itself runs mostly on AWS platform, but it works fine now ...

<comment><USER13> AWS platform on a private cloud, it is not the same as AWS platform for us commoners.

</comment></comment></comment></comment><comment><USER14> A couple of hours into the failure, and no sign of coverage on Techcrunch (they're posting "business" stories though). It shows how detached Techcrunch has become from the startup world.

Edit: I tweeted their European editor about it and he's posted a story up now.

<comment><USER15> Perhaps this isn't really news. These days it's normal.

<comment><USER15> It's ugly, but true enough. You don't have to like it to acknowledge it. It's just another cloud outage bringing down one or more high profile sites. It's a "dog bites man" story.

</comment></comment></comment><comment><USER16> This feels the same way as hearing that the whole Internet just got shut down.

</comment><comment><USER17> I guess this is one Reddit outage that can't be blamed on poor scaling

<comment><USER10> Thankfully, no. :)

But yeah, right now we're shutting everything down to try and avoid possible data corruption. Once they restore service, hopefully we'll be able to come back quickly.

<comment><USER18> SHUT. DOWN. EVERYTHING.

Good plan though. :p

<comment><USER10> My original comment was "We're going Madagascar on the servers." Then I remembered I was on HN, not reddit. :)

<comment><USER19> _> Then I remembered I was on HN, not reddit._

Right now we may as well be on reddit.

</comment></comment></comment></comment></comment><comment><USER20> Why is ELB not mentioned at all on the Service Health Dashboard?

We're experiencing problems with two of our ELBs, one indicating instance health as out of service, reporting "a transient error occurred". Another, new LB (what we hoped would replace the first problematic LB), reports: "instance registration is still in progress".

A support issue with Amazon indicated that it was related to the ongoing issues and to monitor the Service Health Dashboard. But, as I mentioned before, ELB isn't mentioned at all.

<comment><USER21> We've got a single non-responsive load balancer IP in one of our primary ELBs (others have been fine for several hours now), so while everything else for us is up & running, still have transient errors for folks that get shunted to through that one system.

The interesting thing about the ELB in a situation like this is that I believe it may, in many instances, be better to hobble along and deal with an elevated error rate if at least some of your ELB hosts are working than to re-create the entire ELB somewhere else, especially if you're a high-traffic site where you may hit scaling issues going from 0 to 60 in milliseconds (OMMV, but we've been spooked enough in the past not to try anything hasty until things get back to normal).

<comment><USER20> We have an identical load balancer to one that is causing problems so we're lucky enough to reroute traffic through that one instead to get to the same boxes. (The boxes serve two different APIs through two different DNS CNAMEs so we split the ELBs for future and sanity). In this case, it's helped us out. Alternatively, we would've just routed all traffic to our west coast ELBs.

</comment></comment></comment><comment><USER22> Quora says: "We'd point fingers, but we wouldn't be where we are today without EC2."

<comment><USER23> Nice way to point fingers while saying you're not.

</comment></comment><comment><USER24> I just launched a site on Heroku yesterday and cranked up the dynos up in anticipation of some "launch" traffic. Now, I can't log in to switch them off. Thanks EC2, you owe me $$$s

<comment> mike-cardwell Isn't that a design flaw in Heroku? Shouldn't you be able to log into Heroku and change stuff like that even if the entire of Amazons cloud service is down?

<comment><USER25> > Nothing special-case here: we deploy with git push, just like any other > Heroku user. Dogfooding is good for you. > <http://blog.heroku.com/archives/2009/4/1/fork_our_docs/>

<comment> mike-cardwell That's all well and good, but it's no use for their customers if/when Amazon goes down.

</comment></comment></comment></comment><comment><USER26> I think this is a good example of how the "cloud" is not a silver bullet to making your site always up. AWS provides a way to keep it up, but it is up to each developer to ensure that they are using AWS in a way to make sure their site can handle problems in one availability zone.

I think we will see more of a focus from big users of AWS about focusing on how to create a redundant service using AWS. Or at least I hope we will!

<comment><USER10> This outage is affecting all AZ's in the East. So even a multizone setup wouldn't help for this one. Only a multiregion setup.

This outage is a lot like having your entire datacenter lose power.

<comment><USER27> If "multi-region" means North America, Europe and Asia Pacific, doing so would also improve world-wide latency (e.g. here in Australia...).

Could you use this outage to justify switching to multi-region?

<comment><USER25> A blog post last month touched on this: "Q: Why is reddit tied so tightly to the affected availability zone?

A: When we started with Amazon, our code was written with the assumption that there would be one data center. We have been working towards fixing this since we moved two years ago. Unfortunately, progress has been slow in this area. Luckily, we are currently in a hiring round which will increase the technical staff by 200% :) These new programmers will help us address this issue."

Not sure if the costs of data transfer between regions (charged at full internet price) would justify the added reliability/lower latency though.

</comment></comment></comment></comment><comment><USER28> Instead of enumerating who's down, I'd be more interested to hear about those that survived the AWS failure. We could learn something from them.

</comment><comment><USER29> Quora is down, and evidently "They're not pointing fingers at EC2" -- <http://news.ycombinator.com/item?id=2470119> \-- I was going to post a screen shot, but evidently my Dropbox is down too.

</comment><comment><USER30> Holy crap. An Amazon rep actually just _posted_ that SkyNet had nothing to do with the outage:

[https://forums.aws.amazon.com/message.jspa?messageID=238872#...](https://forums.aws.amazon.com/message.jspa?messageID=238872#238872)

</comment><comment><USER31> I'm seeing 1 EBS server out of 9 having issues (5 in one availability zone, 4 in another). CPU wait time on the instance is stuck at 100% on all cores since the disk isn't responding. Sounds like others are having much more trouble.

</comment><comment><USER32> Silver lining: Hopefully I can test my "aws is failing" fallback code. (my GAE based site keeps a state log on S3 for the day when GAE falls in a hole.)

<comment><USER33> This code should be well tested by now. Amazon is doing you a failure by being rubbish.

</comment></comment><comment><USER34> AWS/S3 has become the new Windows - great SPOF to go for if you want to attack. This space needs more competition.

<comment><USER35> Two years ago TechCrunch was publishing an article every time Rackspace went down listing all the hot startups down along with it. AWS is no more a SPOF than any other major hosting provider.

</comment></comment><comment><USER36> <http://venuetastic.com/> \- feel bad for these guys. They launched yesterday and down today because of AWS. Murphy's law in practice.

<comment><USER37> Wow. I can only imagine the intense frustration the site owner must be feeling right about now. Makes you really stop and question the whole "cloud" based service. Or at least should make you realize you need fall-backs other than the cloud service itself.

</comment></comment><comment><USER38> So when big sites deal use Amazon Web Services for major traffic, do they get a serious customer relationship? Or is it just generic email/web support and a status page?

<comment><USER39> There are subscriptions for various levels of support, from $50/mo for 12 hour response time to $15k/mo for 15 minute response time.

<http://aws.amazon.com/premiumsupport/>

<comment><USER38> Interesting pricing. The Platinum seems priced to have no one use it, considering how much of a jump it is over Gold.

<comment><USER40> I would say, rather, that it is priced to have very specific sorts of customer using it.

The relationship between the pricing tiers changes fairly drastically, depending on how much you are already spending on Amazon Web Services. Gold, for instance, starts out at 4x the price of Silver support, but by the time you're spending 80K/month on services, it's only a $900 premium (and stays there no matter how much bigger your bill is). At the $150K/month level, it's a 2x jump from Gold to Platinum, which may or may not be a huge jump, considering the extra level of service you get.

</comment></comment></comment></comment><comment><USER41> It's a bit ironic that Amazon WS has become a SPoF for half the internet.

</comment><comment><USER10> Yes, they are. :(

<comment><USER42> My four-day weekend is already off to a bad start(UK here).

<comment><USER43> It could be worse/better, you could have an Australian 5-day weekend.

<comment><USER44> To clarify, this Monday is ANZAC Day, a commemorative holiday for troops who fought for Australia. Because that's also Easter Monday, the ANZAC Day holiday is moved to Tuesday, despite commemorative services being held on Monday.

<comment><USER45> Actually the official line is that Easter Monday got moved to Tuesday.

<comment><USER44> Interesting. Thanks. What did the Catholic Church have to say about that? Is it that Easter Monday is still on Monday, but the holiday is on Tuesday?

<comment><USER46> Easter is the holiday, and it's on Sunday. I'm pretty sure the Pope doesn't care much what people do the day after Easter (or the day after that).

<comment><USER44> Ah, thanks. I misread the wikipage, and didn't realise.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER23> Assuming the problem is indeed with EBS, I would say this should be a warning sign to anyone considering going with a PaaS provider, which Amazon is quickly becoming, instead of an IaaS provider like Slicehost or Linode.

The increased complexity of their offering makes it more likely that things will break, leaving you locked in.

I did a 15 minute talk on the subject, which you can check out here: <http://iforum.com.ua/video-2011-tech-podsechin>

EDIT: here are the slides if you can't bother watching the video <http://bit.ly/eqDNei>

<comment><USER47> Every time someone makes the claim that downtime should be a warning sign about going with a PaaS provider (or, indeed, an IaaS provider, or in some cases, people even make this claim about going with someone else's Data Center) - I always respond: "And why do you believe that you would do any better?"

Every environment I've been involved in as an operations professional for the last 15 years has experienced downtime regardless of how much we invested in staging, testing, change control procedures, redundancy and ITIL methodology.

While it is true that there are some environments in the world that experience no, or close to no downtime (The NYSE, google.com, the Space Shuttle) - it's usually not worth the investments those organizations make in ensuring 100% uptime.

I'm more interested in seeing what the monitoring methodology is, what the response protocols are, and what the reported downtime has been over the last year has been, than I am about getting too concerned about the occasional outage.

<comment><USER23> The problem with Amazon is that despite touting an open API, their infrastructure internals and practices are a trade secret, so the likes of Eucalyptus are having to play catchup. In other words, I cannot replicate their infrastructure in my own data center, even I had the money to pay them. I suspect this is the main reason that Heroku didn't move off Amazon, and not the fact that Amazon was providing them great value for money.

There is definitely value in platforms, but those platforms should be built on completely open standards with no vendor lock in or influence, as is the case with OpenStack for example: [http://www.theregister.co.uk/2011/02/10/rackspace_buys_opens...](http://www.theregister.co.uk/2011/02/10/rackspace_buys_openstack_partner/)

If you didn't watch my talk, I should point out that I'm working on Akshell (<http://www.akshell.com>), which is itself a platform provider, so I am in fact agreeing with you.

<comment><USER48> Compared to PaaS provides, Amazon is easy to migrate off of because they are simply giving you virtual hardware.

The basics of their setup are well known: Xen and Linux. EBS is some sort of block-based network storage. True, we do not know the backend storage setup, but it does not really matter. The Xen VM simply sees a disk device. NetApp, EMC, Dell, HP and many others all have products that offer similar functionality, including snapshots.

The only part that is potentially hard to migrate off is the security groups, and then only if you are using named groups. If you are just using IP based rules, most any firewall would work.

</comment></comment></comment></comment><comment><USER49> From EngineYard: "It looks like EBS IO in the us-east-1 region is not working ideally at this point. That means all /data and /db Volumes which use EBS have bad IO performance, which can cause your sites to go down."

</comment><comment><USER50> They better start writing their explanation now. Multiple AZ's affected?

<comment><USER10> They better be cutting me a check too.

<comment><USER14> As an advertiser on Reddit, you guys better be cutting me a check too :)

(although my app is also dead right now so not as if the advertising would do me any good...)

<comment><USER10> Send me the link to your ad (when we come back up) and I'll comp you a day.

<comment><USER51> Isn't it paid on impressions?

<comment><USER52> I don't know how the sidebar ads work, but the featured links at the top of the page are run on a sort of auction system. Everybody that wants a piece pays however much they want (minimum of $20) for the day, then all the ads are totaled up and each ad gets a percentage of pageviews corresponding to the their percentage paid on that days revenues.

If that wasn't clear (and I'm not sure it was), assume you and I were both the only advertisers on reddit for a day. If I pay $20 and you pay $20, we would both have our ads displayed on 50% of pageviews for that day. If instead I paid $80 and you still only paid $20, then I would get 80% of pageviews to your 20%, regardless of the total pageviews for the day.

</comment></comment></comment></comment></comment></comment><comment><USER53> Had our blog go down. Didn't realize it was AWS wide..did a reboot. Now I am in reboot limbo. Put an urgent ticket into Amazon. They just said they are working urgently to fix the issues. Let's see how long this goes.

</comment><comment><USER54> In case anyone is late to the party and missed the non-green lights on the AWS status dashboard, here is the page as of about 9:30 EDT...

<http://screencast.com/t/p69xAoDJRSer>

</comment><comment><USER14> Given that Heroku's parent company (Salesforce) owns a cloud platform, it seems kinda inevitable now that Herkou will perhaps sooner-than-later switch back-ends (or at least use both)

</comment><comment><USER55> Everyone talks about SLAs but I believe it doesn't consider the fact that the EBS vols are still up (not on fire, and available) and are phantom writing or that the network is queued up the wazoo so writes don't even happen in a timely manner as you'd expect.

<comment><USER56> I'm not sure that just because they are up, yet unusable, would negate an SLA.

You could have a dedicated server in a datacenter - if the network goes out, your machine is still up and happily waiting to serve requests - but it's still unusable and not actually in service.

</comment></comment><comment><USER26> So do we get some credit on our AWS accounts? I haven't really read their SLA for EC2.

<comment><USER57> I did the math on it, with the downtime so far it's almost approaching the magic 99.95% barrier where everyone gets a 10% bill credit. Wouldn't that be something to light a fire under the collective arses of those in charge of keeping EBS stable.

<comment><USER58> > Wouldn't that be something to light a fire under the collective arses of > those in charge of keeping EBS stable.

Yeah, the only possible explanation is that Amazon's put a bunch of lazy guys in charge of EBS. How hard a job could it be?

<comment><USER57> I'm certain they're plenty talented and hard-working, but there's got to be something in their way. Just like the guy said as he was leaving Reddit, management seems to be pointing fingers in different directions as to why this is going on and how it can be resolved. It's sad, though, because I don't want to think of Amazon as a company with a potentially toxic corporate culture. I enjoy my free Prime membership as a student :D

<comment><USER39> Amazon's corporate culture works pretty well. The problem is that these large- scale multi-tenant services are new technology and very difficult to create and run.

</comment></comment></comment></comment></comment><comment><USER29> [http://www.information-age.com/channels/the-cloud-and- virtua...](http://www.information-age.com/channels/the-cloud-and- virtualization/news/1619753/amazon-web-services-struck-by-performance- issues.thtml)

</comment><comment><USER30> Being unable to get much done here, my co-workers have found other things to do in the office: <http://www.youtube.com/watch?v=u1-oGxDHQbI> :-P

<comment><USER10> What did I just watch and why?

</comment></comment><comment><USER59> All hosting services go down occasionally. If you want to stay up you need to build a fault-tolerant distributed system that spans multiple regions and potentially multiple providers.

Also, Amazon should fix EBS.

</comment><comment><USER60> Ruh Roh. The service I'm using to acquire accommodation seems to be dependent on AWS. Guess I'm going to be homeless tomorrow if it doesn't get fixed. :X

Note to self. Don't ever build a service reliant on AWS.

<comment><USER61> If its us (crashpadder.com) - sorry for the inconvenience. Email us hello [at] crashpadder.com and we'll do all we can when we're back up.

We'd moved to AWS from an even worse host a few months ago, and until this morning had been pretty impressed...

</comment></comment><comment><USER62> So what percentage of the top 1000 sites are now crippled by this?

</comment><comment><USER63> It's definitely a limited outage. My three instances seem to have operated all night with no problem. Two of them are EBS instances.

</comment><comment><USER64> Wish were able to download our ebs snapshots, which are supposedly hosted on S3. What does everyone else do?

<comment><USER35> I take the snapshots. I also have servers send backups to each other each night. I also have a nightly cron job run and rotate backups of the most critical databases to an external drive on my home network. A Tonido Plug does that job (Ubuntu on a tiny ARM server in a plug that costs virtually nothing to run).

Now, some of the databases are simply too large or under too much load to take a live backup while the sites are running. Those I run on Amazon RDS with the MultiAZ feature enabled. There should be two copies of the database running at all times, both servers keeping a 3 day binlog for point-in-time backups, and making a nightly snapshot to S3. I have to rely on Amazon for that.

But I still take daily home backups of the most valuable individual tables off those servers, like user registrations and payment records. Even if I can't have off-site backups of the whole database, I'll have off-site copies of the part I'd need most in case of an Amazon-entirely-offline catastrophe.

</comment></comment><comment><USER65> It seems like availability zone us-east-1c it's working, i can launch a EBS backed instance right now.

</comment><comment><USER66> Today, April 21st 2011, according to the "Terminator", Skynet was launched... No wonder AWS is down

</comment><comment><USER53> 1:23EST and Reddit is back up. Quora/4SQ still down. My site still down.

</comment><comment><USER67> Thankfully, my major clients are using the Asia EC2 Instances!

</comment><comment><USER68> reddit.com is down, but luckily <http://radioreddit.com> is not.

</comment><comment><USER69> It's really mostly EBS failures, so the title is overly dramatic. And EBS has been known to have issues.

<comment><USER70> This is incorrect. EBS is part of the issue, as well as intermittent connectivity failures for EC2 instances.

We're also seeing connectivity issues with our elastic load balancers.

</comment></comment><comment><USER71> Yay for relying on the cloud \o/

<comment><USER72> Yes! It's like those dumb people that rent out 5 floors of a 100 story building, instead of building their own SkyScraper. Then, horror of horrors, 2 of the 8 lifts breaks down for 12 hours.

Such idiots for paying rent, even though building SkyScrapers isn't their core business.

<comment><USER71> Poor analogy; if the lifts break, just use the stairs.

Note that I've written "relying", as in "depending on the cloud without any alternative backup". _Depending_ on the cloud means depending on something you cannot influence.

<comment><USER72> Ok, it's a bomb scare and the building's shutdown for 12 hours.

How I was supposed to get ' "relying", as in "depending on the cloud without any alternative backup" ' from what your originally wrote, I don't know.

<comment><USER71> I'm sorry if you didn't understand "relying" as I meant it. I'm not a native speaker, so my assumption of the meaning of "relying" may have been wrong.

I'll try to elaborate: yes, there are of course certain things that you cannot avoid. That's why I meant you should not "rely"/"depend on the cloud without any alternative backup" on the cloud being available. Redundancy, if your service is crucial!

If you're a small company, and cannot go to your building for a day, well then I hope you're not that important so no big damage (financial or otherwise) is caused. Just enjoy your free day ;-) But if there's a bomb threat in, for example, a hospital, I surely hope they have some kind of backup plan!

Cloud as an alternative mechanism, or maybe even the primary resource for your site -- great, why not! But please don't just throw everything into "the cloud" trusting on the belief that the cloud's operator is impossible to fail.

<comment><USER72> You said 6 words in your original post. And now you're trying to suggest I was to get the above 166 words from the original 6?

You raise perfectly valid and interesting points. It's just a shame they're not the parent comment because then they would have invoked some thoughtful discussion.

<comment><USER73> It's a shame your sarcastic reply derailed the conversation.

<comment><USER72> Fair call.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER74> Amazon down - just one more reason to try out BO.LT and their amazing CDN and page sharing services...

Just launched: [http://techcrunch.com/2011/04/21/page-sharing-service-bo- lt-...](http://techcrunch.com/2011/04/21/page-sharing-service-bo-lt-lets-you- copy-edit-and-share-almost-any-webpage/)

</comment><comment><USER75> This is like witnessing your parents having sex while a kid: you sort of knew this is a possibility but it is a devastating blow to your belief system nevertheless.

The amount of services I use that depends on Amazon is amazing. They have really become the utility company of the Web.</comment>


Rome Is the Fulcrum of AMD's Datacenter Pivot - <AUTHOR> https://www.nextplatform.com/2019/07/31/rome-is-the-fulcrum-of-amds-datacenter-pivot/ <comment><USER2> I'd tend to expect Radeon for compute to do best at the very largest companies where the cost of optimizing everything for the workload is spread over running on a whole lot of hardware as opposed to using NVidia where there's already going to be a library optimized for what you're doing.

</comment><comment><USER3> I know for the first time I started using AMD based instances on AWS the other day. They are better bang for the buck. I suspect this will even become more the case, especially since they didn't suffer from many of the major security issues of late.</comment>


Raft Consensus Algorithm - <AUTHOR> http://raftconsensus.github.io/ <comment><USER2> I use etcd[1] which is an implementation of Raft for cluster configuration. I have a hard time seeing how Raft or consensus in general scales to a general solution for reliable consistency services.

One can not scale by adding servers because all potential members must be known, and consistency is only guaranteed if you are willing to require confirmations forcing full round trip communication with all members for every write. Anything more than 7 servers or so, and things start getting dodgy. Try to improve reliability by adding servers in geographically diverse areas is untenable altogether.

I don't see how simply having multiple servers with failover, and a version clock isn't equally or even more reliable due to it's scalability. The byzantine generals problem[2] is more about reliable voting and trust rather that high performance and reliability.

I'd very much appreciate more insight on this.

1\. [https://github.com/coreos/etcd](https://github.com/coreos/etcd)

2\. [http://en.wikipedia.org/wiki/Byzantine_fault_tolerance](http://en.wikipedia.org/wiki/Byzantine_fault_tolerance)

<comment><USER3> A few things:

1\. You don't have to round trip to every member before acking a write request, only to a majority of them.

2\. If you use the right data structures for your state machine (like persistent data structures), you don't have to fully commit a write before you start the next write. That is, as long as you can quickly revert to a previously committed state (which you can with persistent data structures, it just becomes a CAS on a pointer) you can pipeline requests and batch commits. That lets you start on write n+1 using the state machine results of write n, even though write n hasn't been committed yet. You can then commit a bunch of results (n, n-1, n-2, ...) as a single batch. That does create a dependency that some write j may be rolled-back because a write i < j failed to commit, but you would have had that dependency anyways (you wouldn't have even started write i unless write j had committed). So, although latency of a single request is bounded by the time to replicate a log entry to n/2 + 1 hosts, the throughput is not bounded by the latency of a single write.

The raft paper doesn't actually say you can do this (it says things like "don't apply state machines until a transaction has been committed), but that's because it assumes arbitrary side effecting state machines. However, with some restrictions (like using persistent data structures), you can relax that requirement. Relaxing that requirement can give quite good throughput.

3\. You can horizontally scale raft the same way you horizontally scale any other distributed data store: using consistent hashing. You would use a hash ring of raft clusters.

4\. You can loose up to n/2 hosts from a raft cluster and it will still work.

5\. Raft is "multiple servers with failover and a version clock". In fact it's "multiple servers with failover, a version clock, and strong sequential consistency".

<comment><USER2> Thanks, what confuses me though is why you must retain n/2 hosts, and not be able to recover at least in many cases from a loss down to a single host. I understand that a known majority insures that a write will always persist/propagate even after a network segmentation and later re-join. But what if the network isn't segmented but the servers are simply down, or there wouldn't be a conflict regardless because the relevant clients couldn't have written to the segmented network anyway? When they rejoin they would all know they are behind the one that remained online (as one does with version clocks), so they could just get all missing events and be consistent again.

The majority requirement would, seems to me, reduce reliability over the ability to failover down to a single server.

Does anyone know of a list or test suite of all failures that a consistency algorithm is resistant too, then at least I could understand the reasoning. I get the impression that there are a lot of failure modes that are handled at some cost, but are not relevant to my use cases.

<comment><USER3> In a leader election, a node won't vote for a candidate that doesn't have a record it believes to be committed. Thus, writing to a majority ensures that when the leader dies, the new leader will have all the committed dats. Thus, up to n/2 - 1 nodes can be lost with a guarantee that no committed data is lost.

If you don't need strong consistency (after a write commits, all future reads will see the write), you can use simpler replication strategies.

Redis, for example, using async replication, that is not guaranteed to succeed. A redis master may ack a write, and a subsequent read may see it, but a loss of the master before replication occurs can result in data loss. The failover is not guaranteed to contain all writes. Some times weak consistency is ok. Sometimes it's not, but eventual consistency is ok. Other times strong consistency is needed.

Raft is useful when you need strong consistency.

Raft does support membership changes (adding and removing nodes from the cluster), so it can support losing more than n / 2 - 1 nodes, just not simultaneously.

<comment><USER2> (click) Now I finally get it! Thanks so much, very helpful.

</comment></comment></comment></comment></comment><comment><USER4> Having implemented raft a couple of times now, I'm convinced it's too high level and not powerful enough at the same time. I think I'm going back to two phase commit for the data and s.d.p for providing fault tolerance to the coordinator. Treode has a great implementation of something like that.

<comment><USER5> Rafts been around for only about 2 years, and you've implemented it multiple times already?

<comment><USER6> Confirmed, Alex Newman writes distributed databases in his spare time.

[https://github.com/posix4e](https://github.com/posix4e)

[https://github.com/cloud-software-foundation/c5](https://github.com/cloud- software-foundation/c5)

[https://www.linkedin.com/in/posix4e](https://www.linkedin.com/in/posix4e)

<comment><USER5> Well, that makes pretty good sense then. Alex, I'd love to hear your thoughts on fleshing out the comment you made above. Do you have a blog?

<comment><USER4> I do not right blog but you can bug me on irc! delamancha on efnet

</comment></comment></comment></comment></comment><comment><USER7> "In my experience, all distributed consensus algorithms are either:

"1: Paxos,

"2: Paxos with extra unnecessary cruft, or

"3: broken. "

— Mike Burrows (possibly)

Edit: I think the reason this exists is best explained in this blog post ([http://kellabyte.com/2013/05/09/an-alternative-to-paxos- the-...](http://kellabyte.com/2013/05/09/an-alternative-to-paxos-the-raft- consensus-algorithm/)). It's not that this provides more than Paxos, the goal is to write something that's simpler and easier to implement. The article here doesn't actually explain this, and doesn't compare it meaningfully with Paxos, making it difficult to reason about.

<comment><USER8> Paxos was published after the first randomized consensus algorithms, and Paxos has been shown to be a variation on those principles, so not sure the Paxos worship is all that warranted..

(Please read my bio before responding..)

<comment><USER7> Certainly Paxos is not the be-all-end-all of randomized consensus algorithms. However, what algorithm do you think we should worship?

<comment><USER9> Hmmm... Ben Ors randomized consensus algorithm? Can't beat the simplicity of that one...

<comment><USER8> Yes! In fact published before Paxos, and studiously ignored by Lamport + shown to be equivalent.

</comment></comment></comment></comment></comment><comment><USER10> Relevant previous discussions: [https://hn.algolia.com/?query=raft%20consensus&sort=byPopula...](https://hn.algolia.com/?query=raft%20consensus&sort=byPopularity&prefix&page=0&dateRange=all&type=story)

</comment><comment><USER11> Lecture here: [https://www.youtube.com/watch?v=YbZ3zDzDnrw](https://www.youtube.com/watch?v=YbZ3zDzDnrw)</comment>