Gitlab Support is no longer processing MFA resets for free users - <AUTHOR> https://about.gitlab.com/blog/2020/08/04/gitlab-support-no-longer-processing-mfa-resets-for-free-users/ <comment><USER2> Good - 2FA is the responsibility of the user and resetting it kind of invalidates the security it helps bring.

I think the bigger issue is that people's 2fa codes are still tied to their phone. You can lose your phone at any moment, which is why i've always disliked apps like Google Authenticator which don't let you export 2fa keys (for good reason).

I personally use 1password, but there's definitely room for a cloud storage solution that safely holds 2fa credentials

<comment><USER3> Google Authenticator now allows you to export your keys to another phone.

I keep my keys in analog form - I print QR code for every service. We know how to handle valuables stored on paper.

<comment><USER4> Would you be willing to describe the process you use to do this?

<comment><USER5> Not op, but my usual process is:

* when setting up 2FA, a website shows a QR code

* I screenshot the QR code, and print it out on an A4 sheet, with an annotation of what service it is for

* I scan the QR code from the A4 sheet on two different phones.

* Back on the website, I continue 2FA setup process only after the A4 sheet is printed, and both phones show the same codes

* The A4 sheet goes in a folder for safe keeping

* One phone goes in my desk drawer for daily use

* The other phone goes in my "go" bag that I take with me on short trips etc.

Both phones are used exclusively for Google Authenticator:

* they have no extra apps

* they have a screen lock

* they are always in flight mode

Started doing this when I got burned by having to extract GA's sqlite file from mostly-dead Nexus One over adb.

<comment><USER6> interesting idea. I wonder what the minimum you need for a 2FA-only device would be.

<comment><USER7> The Passport by Foundation Devices is a device that's pretty close to this, it just happens to do more than only 2FA: [https://foundationdevices.com/](https://foundationdevices.com/)

</comment></comment></comment></comment></comment></comment><comment><USER8> Is it just me, or does this make my MFA-protected account safer?

I wish conpanies offered this as a feature, in the sense I'm much more worried about someone SEing their way into my account rather than me losing access to all my MFA methods and backup codes or whatever.

<comment><USER9> It makes your MFA-protected account safer from takeover, but also creates a new risk of completely losing access to your account _and username_ forever.

Github has the same policy, and it deterred me from using MFA for a long time, and when I finally did, I added a large number of alternatives, including the not-so-secure SMS.

I think the fear of being _temporarily_ unable to access your account is already a major reason why people don't use MFA. Turning the threat into a _permanent_ loss will make people even more reluctant to do it.

Dealing with some of the backup methods (e.g. off-site copies of backup codes) is tedious, and I wouldn't be willing to do it for less important sites. If I had 2FA on Gitlab, I'd probably turn it off now. (Actually, just saw they don't have an option to add a phone number - I'd definitely turn it off.)

<comment><USER10> > ... but also creates a new risk of completely losing access to your account > and username forever.

I may have misread the announcement, but it sounds like upgrading to a paid account recovers (npi) the ability to recover your account.

<comment><USER11> How does one upgrade their account if they cannot access it at all?

<comment><USER10> I suspect if one can a) exhibit genuine interest in becoming a customer waving their credit card around, and b) demonstrate ownership of 1FA for the account, then this would be relatively straightforward.

Or rather, at least as straightforward as the process is meant to be for existing paid-up customers, as per TFA.

</comment></comment></comment></comment></comment><comment><USER12> Although framed as a security improvement, I'm sure it's also a massive support burden. When you have hundreds of thousands or millions of users, at some point you probably have support staff who do nothing but helping users reset their MFAs all day every day. It seems fair not to do this for free users. Some services gate MFA to paid accounts which seems like a worse trade- off.

With free users you also have less information available to you as a provider in determining if the reset request is legit or not.

<comment><USER13> I'm sure it'd be abused but it'd be cool if I could put a support payment in escrow should I need to escalate an issue for special attention.

</comment></comment><comment><USER14> For comparison, here's GH Policy:

[https://docs.github.com/en/github/authenticating-to- github/r...](https://docs.github.com/en/github/authenticating-to- github/recovering-your-account-if-you-lose-your-2fa-credentials)

> Warning: For security reasons, GitHub Support may not be able to restore > access to accounts with two-factor authentication enabled if you lose your > two-factor authentication credentials or lose access to your account > recovery methods.

I think it's hard to securely restore an account that is using MFA without being vulnerable to social engineering, SMS takeover, etc. If it's on a corporate account it's easy -- just talk to IT. But for semi-anonymous free accounts? I'm not sure what the expectation here is. What are some good strategies you've seen other providers take?

<comment><USER15> Make the reset take three days, during which time emails and SMS are sent to the addresses on file alerting them that they may be being attacked and should cancel the recovery if so.

<comment><USER16> Though note this will still work for a _targeted_ attack. You wait until your target will be out of the loop and then begin your attack run. But it would definitely help.

<comment><USER15> I'm not sure there are many attractive targets left who are uncontactable by email and SMS for three continuous days.

<comment><USER17> Does no one else take real vacations these days?

<comment><USER18> Even on vacations people do connect atleast once a day. Even if to just check on possible family emergencies.

<comment><USER19> That depends on the country. In Europe 4 weeks of summer holidays are common, in some countries even the law. An increasing number of people logs in to work accounts during holidays, but generally it's considered poor work-life balance. I don't take pressure either way, I might or might not log in once or twice during my 3 weeks (twice a year) if I am curious about something. My private mail I scan occasionally, but it has happened many times that I haven't done so for 2 weeks. There is enough stress in my life, I don't need to be connected every day of the year. There is a phone for personal emergencies. And if someone is afraid someone is dying before they get there they probably can never move out of their parents' home.

</comment></comment></comment></comment></comment></comment></comment><comment><USER20> This seems to create an interesting security loophole. If someone figures out our GitLab password (i.e. by looking over our shoulder), they can just log in, enable MFA and we are locked out, forever.

Think about this. Any criminal who gets access to your Gitlab account can make it impossible for you to access it ever again.

If I used GitLab, I would seriously consider moving somewhere else.

<comment><USER21> Or you can just enable MFA yourself, and prevent that from happening? Am I missing something?

<comment><USER22> In your world no one ever makes mistakes or errors, it must be nice.

<comment><USER21> Not sure how you get that conclusion from my comment, of course people make mistakes - my suggestion is one way to prevent such a mistake? i.e. if you have MFA enabled, it "doesn't matter" as much if you _do_ make a mistake and accidentally reveal your password to someone

</comment></comment></comment></comment><comment><USER23> There was a discussion about the topic of MFA resets on the Risky Business podcast[0] in which the host, Patrick Gray, suggested companies require a one time fee for MFA resets. I think the suggested amount in the show was $50 which seems reasonable enough for western markets. It creates a deterrent for attackers and in the case of free products like GitLab allows the support costs to be covered. Additionally the act of payment itself can help prove identity.

0: [https://risky.biz/soapbox43/](https://risky.biz/soapbox43/)

<comment><USER24> So, not only are developers working on open source projects in their free time expected to maintain and provide support for free, they're now also expected to pay in order to provide baseline security for their users?

If NPM or any other package repository introduced this, do you think maintainers of commonly used open source projects wouldn't feel obliged to pay up? Generally immediately after a traumatic event such as having their phone stolen.

Even if you never plan to update your package, you can't fix a security vulnerability without spending money.

<comment><USER25> Don't lose your recovery codes and you should be fine, right?

<comment><USER24> Yeah, sure, set up a recovery process I don't need anywhere else in life and hope I never mess that up. Then have several thousand people (or far more if I'm lucky) rely on it, I'm sure that'll go well!

I don't need recovery codes for anything in my professional nor personal life outside open source programming.

<comment><USER25> You don't have any other MFAs you need to manage in your digital life? That seems ... odd. It seems like you're deciding to make an issue of, well, how MFA is _supposed_ to work.

<comment><USER24> I have other MFAs, but they all have real-world recovery routes.

</comment></comment></comment></comment></comment></comment><comment><USER26> I don't care how high up you are on your infosec high horse, but the likelihood and potential damage caused by a developer losing access to their 2F device is _far higher_ in _nearly every scenario_ than someone being hacked.

The only correct response to this is for companies to make it against internal policy for developers to enable 2FA. Which is sad.

<comment><USER27> Gitlab allows you to have multiple U2F tokens. From what I can tell it is is at least 10. I myself probably own eight U2F tokens. Anyone of the U2F tokens I have registered can be used to log me into Gitlab, assuming I remember my username/password ;)

Besides the somewhat minimal cost of $20 for a U2F token I don't see any reason people should not have multiple U2F tokens and register them to their accounts.

<comment><USER26> this is pretty much the high horse I was talking about. If you think a majority of people do this, or are likely to do this given the right process, you're living in a very small bubble.

<comment><USER27> > you're living in a very small bubble.

Why did you try to make this personal by attempting to insult me?

</comment></comment></comment></comment><comment><USER18> This looks like a page that people would find _after_ they lose access to their account permanently. There's a lot of CYA language here. Maybe they should have this at signup for MFA or force people to read next time they login.

<comment><USER28> Support Manager for GitLab here.

I appreciate this feedback, and you're right. We don't want folks to get themselves in a position where they lose access.

Our current language when you enable MFA is here: [https://gitlab.com/gitlab- org/gitlab/-/blob/adc7dbeb387adc69...](https://gitlab.com/gitlab- org/gitlab/-/blob/adc7dbeb387adc690cbe1fff650846bc48970348/app/views/profiles/two_factor_auths/_codes.html.haml#L2)

> Should you ever lose your phone or access to your one time password secret, > each of these recovery codes can be used one time each to regain access to > your account. Please save them in a safe place, or you __will __lose access > to your account.

We're directly emailing our most at risk users and are still processing resets in the mean time. Additionally, many users will see a CTA banner reminding them to regenerate their recovery codes if they haven't recently.

If there's anything else we can do - I'm happy to hear it! I've had to rely on recovery services in the past because of pure bad luck and a move to a new country, so we didn't take this decision lightly.

<comment><USER24> I made this post on the forums: [https://forum.gitlab.com/t/gitlab-support-is- no-longer-proce...](https://forum.gitlab.com/t/gitlab-support-is-no-longer- processing-mfa-resets-for-free-users/40905/15?u=tboby)

In summary:

Please consider some kind of exemption for non-commercial open source projects over a certain size.

This change would force me to choose between unacceptable risk to my users, or severe impact on my hobby/life balance and mental health due to the extreme personal responsibility I would have to take to mitigate it.

It's already terrifying enough to publish applications that users run on their systems. If I make an error I can cause all sorts of harm. But at least I only have to worry about that when developing.

Now, if I enable MFA, I can never relax. If I lose my work MFA, there's a perfectly safe process to recover. If i lose my personal MFA it's a few hours of calling banks. If I lose my GitLab MFA I harm hundreds of people. So, I have to permanently vigilant for something I already give so much to for free.

<comment><USER29> Thanks for your feedback, I'm a community advocate at GitLab and just wanted to point out that our team has responded to your forum post here: [https://forum.gitlab.com/t/gitlab-support-is-no-longer- proce...](https://forum.gitlab.com/t/gitlab-support-is-no-longer-processing- mfa-resets-for-free-users/40905/17?u=)

</comment></comment></comment></comment><comment><USER30> I generally support not resetting MFA credentials, and understand where Gitlab is coming from, but wish there were an easier way for the average user. I think that easier way is getting two FIDO2 keys (they're pretty cheap and will get cheaper), and have one on your keychain and one at home, as a backup.

<comment><USER31> For a casual user (unlikely to be specifically targetted for attack) I think SMS is a good option. If you lose your phone then you can just order a replacement sim card and you have your second facto back.

<comment><USER32> As someone who travels frequently and has moved to different countries, SMS is the absolute worst.

If a service asks me to verify my number after crossing a border, the chances of me ever being able to log into that account ever again are basically zero.

Then there's Google. Google doesn't have a phone number on file for me, but they sometimes demand that I input a phone number and enter a verification code to access my accounts. This has led to me needing to ask a total stranger to help me login to my accounts, potentially allowing them full access to all of my data in the name of "security." But hey, nobody ever said Google hires smart people.

<comment><USER33> This is a very niche case. Why can't you recieve sms abroad? Do you leave your phone behind when you travel? Why can't you give Google your current phone number?

<comment><USER34> Not OP but I face this problem:

\- Why can't you recieve sms abroad?

Because I use e-sims and pay $10 for 30 days of data only vs. the $30 my provider would add to my bill for "global data".

> Do you leave your phone behind when you travel?

No, but I do leave my US number "dormant" in that i can't use it.

> Why can't you give Google your current phone number?

Because they already have a clear enough data picture of who i am, and I don't want them to have any more data that I'm willingly giving them.

<comment><USER30> Can you give some more info on eSIMs, where you get them from, how much data they have, etc?

</comment></comment></comment></comment></comment></comment><comment><USER35> > If you are caught where you are not able to provide your MFA token and > without these backup methods, your account will be irrecoverable.

This seems absurd. I vaguely remember another SaaS tool I used that had this policy, but I don't understand it. Even crypto exchanges allow recovery if you lose all traditional recovery methods by submitting documentation like your scanned driver's license among a couple other pieces of info proving you are who you are.

If you offer MFA, I love the idea of following best security practices for users recovering their accounts when losing MFA devices/tokens/etc., but there has to be a "best practice" that includes recovering an account after a due diligence process has been followed proving who a user is who she says if she lost all normal recovery methods.

Edit: It does look like they have a different, less "fatal" avenue to recover work related accounts ("What if this is a work account?" FAQ), but I'd still be scared to host my FOSS repos there and make a mistake only to lose my account forever.

<comment><USER28> Support Manager at GitLab here.

We did use to do identity card verification. The issue that we had was that we often didn't have a lot of information about the folks who opened free accounts.

Often names would be pseudonyms or match only partially with their ID. Not to mention, of course, the difficulty of verifying the authenticity of IDs from all over the world.

We wrote about this back in 2018: [https://about.gitlab.com/blog/2018/10/08/enforcing- managing-...](https://about.gitlab.com/blog/2018/10/08/enforcing-managing-2fa- support-security/)

<comment><USER9> I suspect that support cost also played into the decision.

Consider requiring payment: It covers your support costs, and provides _some_ ties to a real-world identity as well as rate limiting/imposing a real cost on attackers.

For credit cards, AFAIK you can set which security level to apply (i.e. whether you'd rather have a higher fraud risk or more shopping cart abandonement because the customer didn't have/want to deal with whatever auth the bank requires at the highest security level). I imagine cranking this to the max would reduce the risk of stolen cards significantly.

I'd imagine requiring

\- identity verification (even if you can't link it to the account, it will deter attackers) through a third party provider \- payment (both to cover the cost and as a second form of verification/audit trail generation) \- inactivity of the account \- contacting and warning the user for a week \- requiring the user to confirm a confirmation link at the beginning and end (i.e. an attacker would need control of the user's e-mail)

would strike a good balance. If you don't want human judgement in the loop on your side, it can also be completely automated if the ID check is done by a third party.

If there is no way to recover, it creates a perverse incentive to not use 2FA in the first place.

<comment><USER35> > I suspect that support cost also played into the decision.

I respect this if it's the case, but just say it, don't hide behind a "best practice" security blanket when your true motive includes other factors.

> If there is no way to recover, it creates a perverse incentive to not use > 2FA in the first place.

Agree. The user has to now perform a cost benefit analysis in her head to determine if she'll use MFA with the most punitive risk being she loses access to her account forever.

</comment></comment></comment></comment><comment><USER36> Instead of just saying they won't do it for free accounts, they could charge a fee for the service.

<comment><USER37> Good God, imagine the bad publicity that comes out of that. "This is advertised as a security measure, but it's clear it's just a way to extort the user at his most vulnerable!"

<comment><USER36> I don't know. Free service that _isn 't_ even paid for with ads or privacy- invasion, like legitimately free. If something happens on _your_ end, you can pay them for service. What's wrong?

Which is worse publicity:

A) if you use this security feature, and something goes wrong on your side, you will lose your account forever

B) if you use this security feature, and something goes wrong on your side, helping you get your account back will require a lot of work and due-diligence which we can't afford to do for free

It's not _SO_ different from:

A) if your device fails outside the warranty, it cannot be repaired

vs B) if your device fails outside the warranty, it will be expensive to repair

<comment><USER37> > Which is worse publicity:

I think it's not so clear. If someone tells me "I can't authenticate, and so I'm locked out of my account", that sounds like his fault. If someone tells me "I can't authenticate, and they'll let me into my account, but only if I pay them money", then that sounds like their fault. I do carefully say _sounds_ —I recognise the underlying tech is the same in both cases; but the nature of publicity is to attach much more about appearance than to technical facts.

</comment></comment></comment></comment><comment><USER38> > I don't like this and I want to tell someone.

I'm sure this is meant to come across as maybe slightly tongue-in-cheek, and is also meant to provide users an outlet they feel they can vent in productively, but... it reads as dismissive, and leaves me with the impression that they don't actually care to hear any feedback.

<comment><USER28> Support Manager (and person who wrote that line) - it's certainly not meant to be dismissive or tongue in cheek. If you've got some suggested wording to help take away that feeling, I'm happy to submit (or merge) and MR to the blog post to make it feel less that way.

We actually do want (and care) about your feedback and wanted to provide a clear way to give that. In the past we've gotten support requests, tweets, comments on GitLab issues and a myriad of other creative ways of voicing thoughts, opinions and ideas.

My hope was to streamline feedback into a single place that GitLab support and our community teams are actively monitoring. There's been some helpful discussion there (and here) already.

<comment><USER38> Sorry, wrote that and then went offline for a couple of days. Perhaps it's my own cynicism about corporate engagement coming through, and that may be unfair given how open GitLab is as a company -- but the sentence beneath that heading just says "we're accepting community feedback, write your thoughts over here".

I guess the core of my (admittedly petty) complaint about the language is that it doesn't really indicate a willingness to listen, more than it steers people towards a box in the corner they can vent into. A bit of language along the lines of "we think this is the right decision, but we're willing to reconsider" would maybe come off as more considerate, I guess?

</comment></comment></comment><comment><USER39> Seems like a good way to discourage the use of mfa...

<comment><USER40> Seems to me like it actually makes not having MFA even riskier, as if you fail to have it, someone pwns you, and then adds MFA to your account, it's literally a dead end to you.

</comment></comment><comment><USER41> Isn't this why almost every site with 2FA support asks you to print out backup codes and keep them somewhere safe, and warns you that you won't be able to recover your account if you lose them? I thought this was standard practice. I'm surprised it seems to be so controversial.

<comment><USER42> That does somewhat presuppose access to a printer.

<comment><USER41> Slightly less convenient, but if you don't have access to a printer you could always just write the codes down.

</comment></comment></comment><comment><USER43> Why not introduce a model where one who has lost their MFA keys pays something like, say, 50 $ to make up for the time the support team spends on the ticket?

Such a decision is something that would make me either leave the service, not trust it with anything important or not enable MFA.

(Side note, this is also valid for Google, Twitter, Facebook, AWS and other services that take pride in letting AI manage everything with no avenue of contacting a human with authority to override the AI)

</comment><comment> jeppesen-io As someone who had two phones break and loose my 2FA for github, this makes me sad

They were willing to help me - took a week but I got my account back

<comment><USER44> Not sure about gitlab, but at least on github you get recovery codes that you can use if you don't have access to your phone.

<comment><USER29> GitLab does have this, here is a link to our docs with more info: [https://docs.gitlab.com/ee/user/profile/account/two_factor_a...](https://docs.gitlab.com/ee/user/profile/account/two_factor_authentication.html#recovery- codes) and how you can regenerate them if you lose them: [https://docs.gitlab.com/ee/user/profile/account/two_factor_a...](https://docs.gitlab.com/ee/user/profile/account/two_factor_authentication.html#generate- new-recovery-codes-using-ssh)

</comment></comment></comment><comment><USER45> This makes sense. If it bugs you, just upgrade to a paid account. Being able to manually verify your identity in case of emergency is a real service and worth paying money for.

</comment><comment><USER46> What are the current best options for hardware tokens then?

<comment><USER16> If you mostly use a desktop or laptop, purchase two different FIDO authenticators that match the form factor you need.

If purchasing a new laptop (or having one purchased for you) and you run Windows or Mac OS consider fingerprint devices that can turn it into a "Platform Authenticator" able to prove that the person with the authorised fingerprint and the machine authorised are together and wish to sign in.

If you mostly use a phone, look for a newer iPhone or for select Androids (e.g. Google's "Pixel" series) with fingerprint readers, again these function as a platform authenticator.

 _Unfortunately_ for the latter case U2F isn't an option. Sites should be migrating to WebAuthn (and please people do not _implement_ U2F instead of WebAuthn in 2020, for the same reason you wouldn't build a new Flash video site, nothing new supports that technology any more, stop it)

Today GitLab is U2F only, there is a logical upgrade path to WebAuthn, and they seem to be working through a bunch of patches to land it, but for now it's less compatible with a site with WebAuthn (e.g. GitHub). So that means if you use the site largely or exclusively from a phone hardware tokens do not buy you much yet.

<comment><USER47> _Sites should be migrating to WebAuthn (and please people do not implement U2F instead of WebAuthn in 2020, for the same reason you wouldn 't build a new Flash video site, nothing new supports that technology any more, stop it)_

I think this is terrible advice because WebAuthn isn't supported by Linux browsers and I still want to be able to login to services on the internet.

<comment><USER16> Which "Linux browsers" are you thinking of?

Firefox only implements WebAuthn, including on Linux, the U2F support in Firefox is essentially a "reverse polyfill" in which the browser pretends it can do U2F but actually is just wiring some commonly used parts of U2F to the WebAuthn implementation to tide you over until your site gets WebAuthn.

Chrome on Linux certainly supports WebAuthn although it does have U2F support.

I'm sure Lynx can't do WebAuthn, but I'm also 100% sure I don't care.

<comment><USER47> This site does not work for me with Firefox 79.0 on Linux with a Yubikey 5:

[https://demo.yubico.com/webauthn](https://demo.yubico.com/webauthn)

Is this a problem with the backend rather the browser?

Edit: This site works [https://webauthn.io/](https://webauthn.io/) My mistake!

<comment><USER16> The anti-fingerprinting in Firefox will flag that first site you linked.

The site is asking your FIDO Authenticator to provide "attestation". In theory this _could_ be useful to require users to pick a high quality authenticator. It might make sense, for example for Big Bank to require customers use an authenticator Big Bank bought in bulk with Big Bank branding, because they trust it.

But, although countermeasures against tracking were incorporated in the FIDO design for attestation, it is unavoidably a potential privacy risk and so Firefox decided to flag it. Sites which try to ask for attestation during registration get a "Fingerprint" icon in newer Firefox versions. Click the fingerprint and opt either to press on anyway or to Anonymise the authentication (essentially pretend your device hasn't got any attestation) on this site.

For a site owner: Do _not_ do attestation just because it seems cool and worked in Edge or whatever. If you don't _need_ this sensitive information do not request it. You need to work out a policy, which authenticators are OK and which aren't - if that seems like too much then _don 't do attestation_. If the answer is "actually any authenticator seems good" then _don 't do attestation_. If you allow users to just not have an authenticator (100% of general purpose web sites today) then _don 't do attestation_ for those who want one. If you are throwing the data away because you have no idea how to process it _don 't do attestation_.

</comment></comment></comment></comment></comment></comment><comment><USER48> This is the right move. Get two u2f keys. Keep one on your person on your keys, keep the other in your car, at your parent's, or in a safe deposit box depending on your personal threat surface.

</comment><comment><USER49> Side note - the SSH-based recovery mechanism is great and delightful. I happened to need a reset for salsa.debian.org and it saved me from bugging Debian's GitLab folks for support.

</comment><comment><USER50> I wonder if services like Git..b would ever enable MFA based on ssh. That is, configure their endpoint to require both the key and the password to log in, push or fetch.

</comment><comment><USER51> Question:

How secure would it be to use a QR code as a password, and scan it with a phone camera?

<comment><USER49> It'd be about as secure as writing down a password or using a password manager.

Note that it would be a password, not multi-factor authentication. These days, really, the important part isn't so much "something you know" vs. "something you have," it's the nature of how the credentials work.

A password is sort of a one-way credential, something that serves by itself to authenticate you (or whoever is in possession of it). Anyone who holds the password can authenticate until it's changed. You should pick a long password, something that it hard for others to guess.

A phone number (SMS or phone call) requires an active cell network connection. This is spoofable/interceptable/SIM-jackable, but for the most part, it retains the important property that _at the time you log in_ , the site makes a _request to your device_ to confirm the login. Things like Duo Push is a less attackable version of this mechanism. In both cases that makes it harder to use stolen credentials if "stolen" simply means 'digitally copied" \- they'd need some way of disabling the actual physical device in your possession, too.

A code generator doesn't require an network connection, but it's still active unlike passwords: it changes every 30-60 seconds. That means a code that logs you in now can't actually log you in forever, so someone who happens to intercept a particular login request (say, someone who watches you type) can't come back and log in later. They'd need the long-term secret which isn't ordinarily revealed by the app except during backups. Things like RSA hardware tokens are less attackable versions of this mechanism. An SSH key also falls in this category - logging in with the SSH key doesn't reveal the entire key, the way logging in with a password does.

A security key (U2F/FIDO/WebAuthn) is active like a code generator, but it also _verifies the website that 's requesting the credential_. If someone tricks you into visiting gitIab.com (with a capital I), your browser will send that domain name to the security key, which will not respond with your actual gitlab.com credentials. A password manager with a browser extension is a _weaker_ form of this mechanism - there are a lot more avenues to attack, but fundamentally none of the above mechanisms attempt to be resistant to phishing / social-engineering attacks.

So, sure, you could use a large QR code as a password, but that doesn't get you the protections against attacks that the other MFA mechanisms do. It's well worth setting up MFA on important accounts even if you are using a long and unique password, because it protects you from attacks other than password- guessing.

<comment><USER16> One thing you missed here:

In most of these designs the Relying Party (a web site in these cases) has to store _secrets_ and we know bad guys steal secrets and the site won't always find about it immediately or sometimes at all.

For a password we somewhat mitigate this using password hashing. _If_ the site uses a halfway decent password hash and you use a halfway decent truly random password then this is enough to protect both of you. But if you use a bad password or the site does a less than great job protecting it then game over.

For SMS the site stores your phone number (not terrible but not great for bad guys to learn) and maybe a temporary code (bad guys may be able to use this by learning it before you do but that's typically a small time window)

But for TOTP and similar "One time" codes the site stores the generator value. Bad guys who have this value can make any number of fresh codes at any time.

What jumps out here for WebAuthn is that the site stores nothing of value. A random-looking identifier that's useless for anyone other than the Relying Party, and a Public Key that's useless for anything at all other than confirming signatures made with the corresponding Private Key.

I've published the actual database contents for some of my own WebAuthn credentials on HN before, because they don't matter, unlike even an Argon2id password hash they are designed to be utterly useless to bad guys.

That's a huge benefit because there are fewer people trying to break in because what you have ain't worth stealing any more.

</comment></comment></comment><comment><USER52> I think this is interesting in that the correct response as a user is to literally maintain a backup mirror gitlab account in case you lose access to your original one. What a bizarre change from a SaaS company who's value proposition is to make things easier than doing them yourself.

</comment><comment><USER53> Why the distinction between paid and free accounts though?

</comment><comment><USER54> This is a terrible idea which encourages people to use weak security: it's effectively telling people that if they enable MFA, GitLab will ensure that they suffer irrecoverable damages — but if they don't enable MFA, everything is recoverable. That is the opposite of what we want from a security perspective and it risks causing users to be less secure everywhere else because having seen that message will make them question whether they'll regret enabling MFA even on sites which have good security policies.

GitLab should respect the trust that users place in it and account account for the kind of life events which happen to people every day: houses get flooded or burned, safes are burgled, people have to leave abusive situations in a hurry and may be escaping household members who actively try to sabotage them, people die without having made perfect handover plans (especially relevant this year), etc. Of particular interest, consider why the browser manufacturers reversed course on HTTPS key pinning after seeing attackers successfully use it to prevent legitimate users from regaining control — if implemented, this would make it extremely risky to use GitLab because everyone would be one compromise away from an attacker having permanent control of your account — and that would make GitLab look extremely bad for having put themselves in the position of supporting the attacker _unless_ you pay GitLab money (I am certain this is not the _intention_ but it is definitely how it would look to say that FOSS developers are second-class).

There are a number of fallback techniques which can make most of those situations recoverable, and it's okay if they're slow or inconvenient — nobody complains that their airbags are messy. Some ideas:

1\. Have a process where someone can use a trusted third-party to verify identity: for example, register identifiers (drivers license, passport, etc.) and have a form which can be notarized after checking ID and mailed in. It's slow but that works even in the "I lost everything I wasn't wearing" scenario — and if you did just lose your house to a natural disaster or flee an abusive ex, you probably have bigger concerns than getting control of your GitLab identity back in less than a week.

2\. Configure next-of-kin / trusted friends who can approve a reset request, perhaps requiring more than one.

3\. Allow the user to configure some number of IdPs to approve an unlock, increasing the level of compromise that an attacker would need to hit before getting the ability to perform a reset. There are drawbacks to this approach but most people do not have threat models which are meaningfully resistant against someone who can compromise multiple of {Apple, Google, Facebook, Microsoft, login.gov, etc.} and could be pretty effective combined with a time-delay (e.g. send notifications for n days before actually resetting MFA.

<comment><USER29> I'm on the community advocates team at GitLab and we really appreciate your feedback. I wanted to point out that our team responded to your post on the GitLab forum, you can see the response here: [https://forum.gitlab.com/t/gitlab-support-is-no-longer- proce...](https://forum.gitlab.com/t/gitlab-support-is-no-longer-processing- mfa-resets-for-free-users/40905/16?u=)

<comment><USER54> Thanks — I've been replying there but figured I should comment here since this is where I first saw it. I definitely hope this leads to some UX work, especially through the lens of thinking about how badly it will fail for vulnerable users.

</comment></comment></comment><comment><USER55> I agree, but having worked in SaaS, and done a lot of partnerships, Microsoft has infinite leverage to turn something like MFA services into a co-branding exercise that pays for itself.

<comment><USER56> Github is owned by Microsoft, not Gitlab (which is whom this link refers to).

<comment><USER57> > _Github is owned by Microsoft, not Gitlab_

Yet.

<comment><USER55> I love this optimistic take on my failed comment :)

<comment><USER57> As I'm a Symbian user, I could not be so optimistic :/

<comment><USER55> I am sorry for your lots :(

</comment></comment></comment></comment></comment></comment><comment><USER19> The free internet services model is fundamentally broken. It's just not sustainable that good service can be offered for free.

Where do you get free groceries, fee petrol/gas or anything else?

</comment><comment><USER58> Exactly just use a password no one is trying to hack your stuff, and the biggest risk is you.

So sick of services that constantly tell me to use MFA.

No I don't want to and if I thought the service was that important I would use it.

<comment><USER59> no... and im not even going to explain why

<comment><USER58> Is it cause you think China or some hacker gives a shit about you? Newsflash you are no one to most people. Your actions matter to no one but yourself and the people close to you.</comment></comment></comment>


Macro Shots Taken Inside Instruments (2012) - <AUTHOR> https://fstoppers.com/commercial/pictures-unbelievable-macro-shots-taken-inside-instruments-6129 <comment><USER2> The text on the page doesn't mention it, but surely the instrument with the round soundhole and 5 visible strings is a guitar.</comment>


The safe career path of the future: skip college and start a company - <AUTHOR> http://www.curiousjuice.com/blog-0/bid/119988/The-safe-career-path-in-the-future-skip-college-and-start-a-company

<comment><USER2> You know what; education is changing.

Khan Academy is only getting stronger, MIT, Stanford and others are offering free courses online.

There seems to be an opportunity here for a "startup business college" -- a service which you attend, and maybe pay a MUCH smaller fee which would cover the curation, organization and guidance/tutorship/mentorship through the amazing resources available.

I am not sure I agree 100% with "skip college, start a company" -- but "seek out and devour all the amazing free teachings from stellar sources".</comment>


Git Source Code Review: Diff Algorithms (2013) - <AUTHOR> http://fabiensanglard.net/git_code_review/diff.php <comment><USER2> `git diff` is a beautiful thing; I often use it instead of the usual diff (with --no-index --color --color-words)

</comment><comment><USER3> Immediately recognized the name from the guy who does the DOOM source code reviews and am assuming that it is likewise a must read [http://fabiensanglard.net/doom3/index.php](http://fabiensanglard.net/doom3/index.php)

</comment><comment><USER4>
        > Diff Algorithms

I got my hopes (dreams?) up for a second - I thought this was a magical tool to actually perform `diff` _on_ algorithms, to compare the equivalence of different implementations...</comment>


Elon.io – Learn Japanese or Turkish - <AUTHOR> https://elon.io/ <comment><AUTHOR> Thanks for the heads up. I currently run Elon.io from the Netherlands. I hope not to run into legal trouble, but a name change now would be quite a hassle. But then again, it would be even more of a hassle later. Even so, I think it might be wisest not to use my limited resources now on legal matters.

In case you think differently, please advice :)

</comment><comment><USER2> Just a heads up: there's a university in the US called Elon. If you plan to operate in the US, you may run into legal issues related to that.</comment>


Can I reserve "Dave"? - Early adapters win! Reserve your Bmark username now - <AUTHOR> http://www.bmarkcentral.com/

<comment><USER2> How is B*Mark any different from the other social bookmarking sites out there? Seems to me like the UVP is: you can reserve your name.</comment>


Comparison of opensource time series db's - <AUTHOR> https://blog.dataloop.io/top11-open-source-time-series-databases <comment><USER2> Great comparison, and I hope the state of the world keeps getting better for TSDBs so we don't need to build our own at some point - but I disagree re:

\</comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>- Performing queries across billions of metrics looking for labels that only match a few of them (a common scenario with time series data at scale) is really slow in Cassandra. This is because of the way it stores data in columns. This extends to any columnar database including Google's BigQuery which all have a natural disadvantage with time series data. \</comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>-

There's nothing inherently limiting in columnar databases that makes it slow to match only a few elements that match only a few out of billions or trillions of records.

... but a classic columnar store might not be as efficient for storage, or might take 5-10x the nodes to return with the same speed with that kind of filtering, depending on storage and clustering mechanisms used.

<comment><AUTHOR> Hi, the wording could probably use some tidying up around that part and I'm open to suggestions. However, I do think it's a big problem with columnar based time series databases.

When somebody wants to query for a few points matching certain dimensions in Cassandra there's no getting around the fact that you have to do a scan across potentially billions of data points.

Whereas if the index lives outside in something relational like Postgres the lookup becomes insanely cheap and you're not having to scan over a bunch of data.

There are quite a few databases that don't have an efficient external index. For those, running 10 times the number of nodes would certainly speed things up, but it's probably just a good idea to avoid databases like that if you want fast queries.

<comment><USER2> Sorry to keep being pedantic, but I think it's important to thinking about approaches to scalable and performant TSDBs, and I still disagree :)

Your example re: Cassandra is a problem with a particular example of columnar based time series database, not inherently with using columnar-store based backends for time series data.

At Kentik, our in-house backend deals with 80+ columns wide (what would be tags in TSDB) for primarily network data, and querying across tens of billions of records (tens of devices of data for 90 days) usually takes .5-2 seconds.

That's deployed on ~7 backend data nodes, running heavily multi-tenant with 300k-2m records/second ingested and averaging 450 queries/minute across a week (don't have a peak query # handy).

But there's also nothing that says that a columnar store database can't have indexes per column built-in (vs. external).

</comment></comment></comment><comment><USER3> > I'm only interested in time series databases for use by developers and > operations to store and retrieve data that pertains to the health and > performance of the services they build and operate. Everything in this blog > will judge the entries based on their suitability for that task.

That is a very particular problem, in which the data storage is a minimal [yet important] aspect of the full system.

You're probably going the wrong route if you're trying to redesign your own and you'll only realize that way too late when you'll have to design your own metrics collection, own graphing, own alerting, own...

The standard proven open-source stack:

collectd/statd (metrics collection + whipser/graphite (storage) + grafana (cute graphs and dashboards).

The latest fad is to replace graphite with prometheus (which is better in some aspects but has it own fault).

Both these open source tools will satisfy your purpose.

HARDCORE LIMITATIONS: Both these open source tools are entirely single node. There is no form of sharding nor high availability nor horizontal scaling.

(Rules of thumb: Should be fine up to 100 hosts and applications. Then get ready to throw big hardware and tune retention aggressively.)

\---

Some quick maths:

8 bytes per metrics * every 5 second = 967 kB per metrics over the week

967 kB per metric * 100 metrics per host * 100 hosts = ~10 GB per week for high precision

Any of the parameter can spiral by tenfold (depending on the setup, retention, hosts, metrics per app...). That means going straight into TB range and scaling issues where one node is simply out of the question.

\---

It's pretty clear that the open source solutions don't scale and are hard to maintain... so what's next when we outgrow them?

Switch to the latest generation of monitoring tools. The two best solutions are datadog and signalfx. They both accept custom metrics from your app.

And... oh wait I just noticed that dataloop.io is a new SaaS solution trying to compete with them. Oops :D

<comment><AUTHOR> The blog discusses storage sizes with very similar maths examples. DalmatinerDB uses 1 byte of storage per point compared to at the top end Elasticsearch using 22 bytes per point.

Build vs buy is an age old discussion. You won't convince anyone to switch from one side to the other. There will continue to be people like you and me who would prefer to buy, and others who want to build and run it themselves. As you have found out I don't need to be convinced as I started a company to address the issue of there being no good options to buy at the time. In most cases, for monitoring micro services, I'd buy a SaaS solution. I founded Dataloop 3 years ago so not really a new startup any more. We're past Series A and starting to grow.

It's true that we compete with Datadog and SignalFX in that area although our real competition is open source with 90% of the addressable market using older tools like Nagios etc. As the shift to the cloud and micro services happens I'm sure it won't be a winner takes all market. Dataloop tends to focus on the enterprise end of the scale whereas Signalfx is more developer focussed and Datadog is more operations and SME.

When you say best I'd argue that's subjective. Signalfx charges by the metric and that gets very expensive. Datadog limits you to 100 metrics per node with an agent based pricing model. Dataloop uses per node pricing that's much cheaper with unlimited metric volume. We're aiming to keep the costs extremely low by investing in highly efficient backend storage.

The reason people are moving away from Graphite to InfluxDB and Prometheus is the dimensional data model. Graphite simply isn't as powerful. Similarly, StatsD aggregates down to the service and doesn't help pinpoint the outlier. Prometheus collects all metrics in their raw format far more efficiently and will let you instantly drill down into what is causing the issue.

To answer your question about what's next after you outgrow open source solutions that don't scale.. well that was kind of the point of the blog! DalmatinerDB scales to millions of metrics per second on a single node and linearly as you add additional nodes. It isn't exactly hard to maintain either as it's based on Riak Core.

I guess the final thing to say was that this wasn't really an advert for Dataloop. Our business model doesn't depend on selling database features. Unlike other SaaS companies we're happy to release the work done on our time series database for free and available as open source.

Why would we do that? Mostly because it's fun to do open source stuff. Also because hiring Erlang developers is pretty hard and this gives me an excuse to talk at conferences where they hang out.

We've had a team of people working on this stuff now for over a year and as you've mentioned no open source time series databases really scale. It's a problem we've solved and are giving away for free. I must be really bad at conveying that message in the blog.

<comment><USER3> A full metrics/monitoring/alerting solution and a metrics storage engine are two different purposes. (The first one being solved by the latest SaaS tools, including dataloop.io).

I limited the previous message to the monitoring use case because it is already quite long and a topic of it's own but I'd like to address the storage as well.

There are many reasons one would need a time series database for an application. In which case he'd need that kind of comparison.

\---

There are a few things which I'd like to see about storage systems:

\- What kind of features does it have to compress and/or aggregate data? Does it have any?

Some systems can take 4 bytes per int, other can take 50. Some can store diff, some do not... That makes a huge difference.

\- Can it cluster horizontally? Also, does it scale write horizontally?

We can have 50 CPU systems with 10TB of SSD array noawadays, but we probably won't. It's actually rather challenging to scale vertically on AWS/GCE (not so much on softlayer), not to mention the nightmare of having a single point of failure for maintenance and issues.

I suppose we get that with the read/write number per 1 node and per 5 nodes systems, which brings me to the next point.

\- The performance numbers are somewhat misleading IMO.

You say yourself that you didn't do benchmarking. You're just taking some random facts you found on the internet and showing that as data.

\- You should include the versions of the database in the table. Features change over time.

\- Are you backing and contributing to DalmatinerDB? For some reasons, the link between dataloop and dalmatinerDB wasn't clear to me on the first read. (Not to mention, you're not even advertising your product or your company).

\- How much of DalmatinerDB magic is based on ZFS? Does it actually need ZFS to run?

As far as I remember, ZFS is still a BSD/Solaris citizen only. (And don't tell my that it's coming into the next ubuntu release, it's just an hypothetical future until actually done ;) )

Anyway. It's a welcomed comparison. Good work =)

\---

> our real competition is open source with 90% of the addressable market using > older tools like Nagios etc.

An interesting point of view. I personally consider 90% of the nagios market to not-be-a-market at all. It belongs to people who only uses it because it's free (as in no-money) and can be downloaded easily.

Free automatically brings the students, the amateurs trying things in their garage/homelab, micro deployment where it's enough, many companies and people who simply don't value their time or the quality of what they deliver, and finally all who have no money whatsoever or can't go through the hassle of the buying-stuff department.

<comment><AUTHOR> The raw data in the spreadsheet, which I have continued to update and is now up to 15 databases with 30 characteristics each, is indeed applicable to both people who want to pick up a database and use it in their monitoring stack, or who want to use a time series database for another purpose.

Some of the newer additions like Warp10 and Akumuli for example have comments about being great choices for sensor applications and those who need highly performant local time series storage written in C++.

That's a good point about the compression. I bundled that all in the row 'bytes per point after compression' which gives the end result for each database but I haven't noted which ones actually compress. However, there is 1 database that uses lossy compression so that is noted. That's all in row 11 currently.

Someone else also raised the clustering question and I added a row for 'dynamic cluster management' to address the question of whether you could just horizontally scale by adding a node without bringing the system down.

The performance numbers are a big problem regardless of how they are calculated. Firstly, I'd like to address the issue of whether having them or not is important. I believe it's incredibly important to have some kind of idea up front before investing time trying out a database. If we agree that performance numbers are essential, especially considering the database list is up to 15 and growing and we need some kind of method to reduce the number to trial, then we're onto the next problem. Benchmarking 15 databases in a uniform way as a science experiment is about 7 master thesis worth of work (there are actually several on the topic of benchmarking a few of those databases together that are interesting reads). To put it plainly such a benchmark won't ever happen and if if ever does someone would find a way to undermine it for their use case and setup. Therefore the usefulness of the numbers in the list is more a practical ballpark estimate as outlined in the blog.

We did however benchmark DalmatinerDB and released easy to re-create results, the exact box hardware, test method, code and a little graph. I used the same mechanism to benchmark InfluxDB and got reasonably close to the figures they released.

The version of each database is in row 29. It probably wasn't when you first saw the sheet but I added it soon after along with a maturity field.

Dataloop is a SaaS company of which I'm a co-founder and we needed a database a few years ago to build our SaaS monitoring product. None of the things that existed at the time were all that appealing so we picked up an already open sourced project (DalmatinerDB) and have put about 5 people working for a year improving it under the direction of Heinz (the original author) and contributed it all back.

The reason to contribute it back wasn't completely altruistic. I'm personally interested in time series databases, as a co-founder I need to go on the road and talk at conferences, and chatting about DalmatinerDB seemed like a good way to get my slides up and a big 'we are hiring' one at the end. Also, hiring Erlang devs is hard. The more community contributors we have the bigger the potential hiring pool of known good people.

For the ZFS question there's the choice of SmartOS (which is what Heinz built it on) or Linux. Ubuntu 16.04 has been out for several months and supports native ZFS. Dataloop runs DalmatinerDB on ZFS on Ubuntu. Heinz and several of his customers running Project Fifo (which DalmatinerDB came out of) usually run it on SmartOS. I'm guessing far more people would be interested in running it on Linux than SmartOS. The database is intrinsically linked to the filesystem for proper running. It relies on the way the compression works to achieve 1 byte per data point storage volume. As well as the way the filesystem makes atomic writes. You can run DalmatinerDB without ZFS but you'd quickly run out of disk space and your data integrity wouldn't be as well known.

The bit I do agree with fully is the marketing sizing questions.. it's all a gamble based on opinion. We can probably agree it's a huge area for growth right now with more people moving to cloud and spinning up far more infrastructure than ever before. All I personally know, being a SysAdmin most of my career, is that things are getting more complex and open source isn't scaling very well, or when it does you still need a team to manage it. If you had asked big companies 10 years ago who would want to outsource their email hosting I think you'd get a much different answer to today. The trend long term is towards SaaS and I believe the 90% of companies who today are using open source will see the same shift. Will I get the amateurs? Probably never, but then the 90% market numbers come from our own research:

[https://blog.dataloop.io/2014/01/30/what-we-learnt- talking-t...](https://blog.dataloop.io/2014/01/30/what-we-learnt-talking- to-60-companies-about-monitoring/)

And then validated time and time again by others:

[https://kartar.net/2015/08/monitoring-survey-2015--- tools/](https://kartar.net/2015/08/monitoring-survey-2015---tools/)

I honestly don't think that even if the mega combo of Prometheus / DalmatinerDB / Grafana was polished today it would eat into either Dataloops or Datadogs business. Some people want to run their own, others want to buy SaaS. Over time we're going to see the shift to SaaS that has happened with most other products and by having a foot in the door with DalmatinerDB, Dataloop should hopefully create some good-will and credibility from the people who want to run a monitoring stack themselves. At least then if they move on into an environment where they do want to buy SaaS hosted monitoring we're going in warm with a good reputation.

</comment></comment></comment></comment><comment><USER4> Is there a publication date for this? That's a really important attribute to include with comparisons like this.

<comment><AUTHOR> Hi, that is a good point. It was all written yesterday so is pretty up to date :) I've added some dates to the top of the blog post. Thanks!</comment></comment>


Some Mac APIs Available Only To Apps Sold Through Online Mac App Store - <AUTHOR> http://mjtsai.com/blog/2012/01/25/pdfpen-and-icloud/

<comment><USER2> I'm looking at this as an iOS developer and not a Mac App developer, but I think this is overblown a bit.

In this case, the APIs are the iCloud related features. There are many reasons to allow only approved apps access. First, to some extent Apple are subsidizing the cost of iCloud and they want to ensure that they get their piece of any app using it. Also, it gives Apple the opportunity to make sure that an app that uses iCloud doesn't abuse it. In fact, I've had an iOS app rejected because it was putting to much data in iCloud. (Which wasn't unexpected, the customer for the app was warned it would likely happen)

I'll be worried if it applies to an API not related to a service but rather a core piece of tech.

</comment><comment><USER3> It's not that this API is available to App Store only apps, it is that it is available only if you have a signing key from Apple. That is because these APIs make connections to Apple's server. It would make sense that Apple would, you know, not let just everyone into their server with every user's information.

So, this post is blatantly misleading, and whoever wrote it should be ashamed.</comment>


Humble GameMaker Rebundle includes games and their source code - <AUTHOR> https://www.humblebundle.com/gamemaker-rebundle <comment><AUTHOR> It's uncommon for commercially successful games to also release their source code. But I wonder: when a game is made by a small team or solo developer, without any intent during development to release the source code, how likely is it that the source code will be useful to others?

There's also no mention of code licenses or use of game assets, both of which seem like important details.</comment>


Texas has enough sun and wind to quit coal, Rice researchers say - <AUTHOR> https://www.houstonchronicle.com/business/energy/article/Texas-has-enough-sun-and-wind-to-quit-coal-Rice-13501700.php <comment><USER2> There's a huge "wind belt" running north from the Texas panhandle up to Canada. At one time, T. Boone Pickens was looking into exploiting this, but he went for oil instead. There's no big electrical load in that area, so high- voltage DC lines to the West Coast, east to the Midwest, and south to the major Texas cities would be needed.

2,000 km transmission lines are now routine.[1] They've taken 2-4 years to build in Brazil and China. So this is quite feasible.

Politically, building long-haul transmission lines in the US is a huge hassle. Efforts to build a 730-mile line between Wyoming and Nevada (where it can attach to existing lines to California) have been underway since 2013, one jurisdiction and landowner at a time.[2] And this for something which is completely non-polluting. It's DC; you don't even get induced stray voltages.

[1] [https://www.power-technology.com/features/featurethe- worlds-...](https://www.power-technology.com/features/featurethe-worlds- longest-power-transmission-lines-4167964/)

[2] [http://www.transwestexpress.net/news/index.shtml](http://www.transwestexpress.net/news/index.shtml)

<comment><USER3> Yeah, and areas of that "wind belt" can get pretty violent at the extremes. It includes Tornado Alley [1], which we got to experience in full force in 2017. The house got hit by a mile-wide EF4 tornado. We were fortunate -- it only took half the house and one of the vehicles -- but the three houses around the corner were left to slabs, swept clean as the day the foundations were laid -- and hundreds of heads of cattle from the surrounding ranches were just gone -- no one ever figured out where they landed; however, other items were discovered 50 miles away. That was a crazy day.

[1] [https://en.wikipedia.org/wiki/Tornado_Alley](https://en.wikipedia.org/wiki/Tornado_Alley)

</comment></comment><comment><USER4> The New York Times recently published some charts showing how each state generates its electricity: [https://www.nytimes.com/interactive/2018/12/24/climate/how-e...](https://www.nytimes.com/interactive/2018/12/24/climate/how- electricity-generation-changed-in-your-state.html)

<comment><USER5> Great link - thanks! I was curious to see how total energy consumption changed over the same time span. According to a super-useful EIA tool[0], it was apparently (and surprisingly) almost flat, despite a 15% population increase.

[0] [https://www.eia.gov/totalenergy/data/browser/?tbl=T01.01#/?f...](https://www.eia.gov/totalenergy/data/browser/?tbl=T01.01#/?f=A&start=1949&end=2017&charted=4-6-7-14)

</comment></comment><comment><USER6> For wind and solar, the following sites are eye-opening

[https://globalwindatlas.info/](https://globalwindatlas.info/)

[https://globalsolaratlas.info/](https://globalsolaratlas.info/)

Its very likely the first metro area in US to live off Renewables is going to be Dallas-Ft.Worth. Of course, Austin is close too.. but all those wind farms are closer to Big D than Austin.

<comment><USER7> These charts miss Hydroelectricity, which is currently 70% of renewable energy production worldwide.

Seattle is more or less entirely powered by renewables, with 97% of Seattle City Light's energy coming from Hydro, Nuclear or Wind: [http://www.seattle.gov/light/FuelMix/](http://www.seattle.gov/light/FuelMix/)

<comment><USER8> There are multiple definitions of "renewable energy" out there. Palo Alto doesn't count "large hydro" as part of their renewables goal.

<comment><USER9> The problem with hydro is mostly fish. Anyways, from an economic perspective, hydro is definitely renewable, and works much the same way a wind turbine does (substitute blowing wind with flowing water).

</comment></comment></comment></comment><comment><USER10> There are already technologies that can create oil from energy+water+air. The key component that can make it economically viable is an excess of energy. And that happens to be the main issue with solar and wind, we need some kind of energy storage solution to store the excess produced energy. Common solutions explored are batteries and hydro dams.

However, producing oil from the excess energy is also a form of energy storage. And Texas already has the infrastructure to refine and distribute oil. Since the produced oil would pull CO2 from the air, it would be effectively carbon neutral.

What if Texas blankets the state in solar and wind? They could continue to be a oil producer, but with the additional benefit of producing carbon neutral fuel. With a carbon tax, this fuel could become quickly price competitive with traditional "fossil" fuels (pulled from the ground). And even without a carbon tax, it's possible a significant percentage of the population would be willing to pay extra for carbon neutral fuel. Or another way would be for government to require a certain percentage (such as the current 10% ethanol, they could say 10% must be carbon neutral fuel source). There's a lot of possibilities and potential to much more quickly address climate change than waiting for EV cars to become the majority.

<comment><USER11> syngas definitely seems like one of those holy-grail technologies that could save us from ourselves if we can make it economical. Plug-in replacement for transportation, one of the most difficult places to decarbonize. As a bonus, syngas is pure by nature, so engines burn cleaner without any need to worry about things like "low sulfur diesel".

</comment></comment><comment><USER12> Texas will be powered by wind and solar while supplying the rest of the country with oil and natural gas.

<comment><USER13> That's probably not a terribly stupid idea; transporting energy as gas in a pipeline might be significantly more efficient than electricity through wires.

The oil and gas are going to be extracted as long as the global market prices make it worthwhile to do so. The only way that changes is if renewables come down in price enough to make fossil extraction uneconomical. So the more we build out, the better, even if it happens at the same time as fossil extraction.

</comment></comment><comment><USER14> We're still really only partially (fractionally even) serious about pursuing renewable energy.

We need to increase capacity of renewable energy production, of course, but that is the easiest part of the whole equation. We also need serious investment into energy storage, by orders of magnitude more than what we have today. That will be at least as large an investment as building out the base capacity. And we need massive improvements to the power grid to make it possible to transmit power from where it's generated to where it's used, which will be at least as big an investment as either the capacity or storage problem.

Fortunately, all of these problems will benefit from technology advancements, and all of them can represent capital investments which provide lasting long term value (not just in CO2 emissions reductions but also in improving the resiliency of the power grid). There are tons of people working on the generation problem. The storage problem is about in the same state that generation was maybe 20+ years ago. And the grid problem? You could fit everyone working on it into a conference room, a small conference room. We desperately need to amp up investment in every aspect of this field ASAP.

</comment><comment><USER15> potholer54 (who runs a _truly great_ bad anti-global-warming junk science debunking channel -- totally check it out) also mentioned something about a recent discovery for an economical way to covert wind and solar power into ammonia, as a medium for storing hydrogen as a liquid. I'm not entirely sure about that, because while potholer54 is _truly great_ in his usual wheelhouse, he's perhaps a bit bad at evaluating new technologies. (Like, he counts solar roadways as as promising future development.)

However, if that's true, we could see pipelines and tanks of ammonia criss- crossing Texas and the rest of the US, allowing us to fungibly transfer renewable energy. Basically, we could use the systems and techniques we have already developed for doing this with Natural Gas, including the kinds of interfaces those systems have with the current power grid.

Unfortunately, a cursory search for this technology seems to contradict this.

<comment><USER16> Storing environmentally friendly generated electricity in reactive chemicals would solve lots of problems (storage; reuse of existing energy industry infrastructure; energy-dense fuel for cars, trucks, and planes; lowering CO2; etc.) However, ammonia is quite corrosive.

Because the steel pipes transporting natural gas throughout the US are under pressure, they are subject to a peculiar form of degradation, _stress corrosion_. Normally, steel can rust or corrode when exposed outdoors. Simple coatings of paint, tars, polymers, etc. can protect steel, and natural gas transport pipes are coated on the outside to protect them from moisture, air, and the natural PH variations found in different environments. These pipelines and their coatings are examined periodically to prevent catastrophic failures. The pipes are perhaps one-half to two meters in diameter and have a wall thickness of one to three centimeters if I recall.

Stress corrosion is not like normal rust on the surface of steel. It happens _inside_ the steel, along the inter-granular crystalline boundaries within the steel itself. Somehow, the stress inside the steel produces higher rates of corrosion (there is a large literature on this subject, google "stress corrosion"). This type of corrosion can be dangerous because pipes can burst without visible warning of their condition. Detection of stress corrosion requires inspection systems that are much slower (ultrasound, electromagnetic induction measurements, etc.) and less practical than simply looking for corrosion on the outside of the pipes.

Unfortunately, ammonia is corrosive to steel and the insides of existing pipelines and the valves and other fittings would not be prepared to handle ammonia. So ammonia wouldn't be able to utilize the existing natural gas pipeline infrastructure.

I'm not a metallurgist nor even someone that remembers much of my university chemistry classes so I welcome comments from a more informed HN reader.

<comment><USER15> _So ammonia wouldn 't be able to utilize the existing natural gas pipeline infrastructure._

That's not what I'm advocating. I was talking about the precedent of companies that know how to run pipelines, do transactions with storage pools, and scheduling. Of course the physical pipes and other equipment would have to be different. But all of the commercial contractual, financial, and legal frameworks are all pretty much worked out.

</comment></comment></comment><comment><USER17> Texas is unique in another way. It's electrical system is not interconnected with the rest of the nation. [https://www.nerc.com/AboutNERC/keyplayers/Documents/NERC_Reg...](https://www.nerc.com/AboutNERC/keyplayers/Documents/NERC_Regions_Color.jpg)

<comment><USER18> There are still DC Ties connecting to the Western Interconnect and the Eastern Interconnect.

<comment><USER19> ERCOT (which manages the grid in most, but not all, of Texas) has 5 DC ties to other grids. Two are to the Eastern Interconnect and 3 are to Mexico.

El Paso isn't in ERCOT, but is connected to the Western Interconnection

</comment></comment></comment><comment><USER20> Very cool read - hopefully my fellow Texans advocate for a homegrown energy mix with plenty of co-benefits.

The Solutions Project[1] is developing roadmaps[2] for transitioning the entire planet to 100% renewable energy. My college advisor[3] is the lead scientist and one of the smartest people I've ever encountered.

[1] [https://thesolutionsproject.org](https://thesolutionsproject.org)

[2] [https://thesolutionsproject.org/why-clean- energy/#/map/state...](https://thesolutionsproject.org/why-clean- energy/#/map/states/)

[3] [https://web.stanford.edu/group/efmh/jacobson/](https://web.stanford.edu/group/efmh/jacobson/)

</comment><comment><USER21> A thought occurred to me while driving the other day: Texas seems to be more of an "oil" state than an "energy" state, but it could be devoting a percentage of its largess from exporting oil / oil tech to developing safe nuclear power. For all I know this is already happening, I have no idea, but it seems like if they don't get off "oil" and get onto "energy", the state could be the Detroit of the 2040s

<comment><USER22> Texas is very much THE energy state. They are the number one producer of oil, natural gas, and wind power in the US. They are among the highest producers of total renewable power in the US as well. If it wasn't for hydroelectric (which Texas is not very suitable for) it would be the number 1 producer of renewable energy in the States. There are also significant levels of Nuclear power produced in Texas at 10% of it's total electrical generation capacity. Texas comprises roughly 1/3 of the US power grid and operates the only State independent stand alone power grid in the US. Texas produces 20% of the US's total energy output and produces twice as much electricity as the second highest producing state (Florida). The power grid is Texas is also extremely reliable and resilient even by American standards.

The state is also a huge technology sector (aerospace, chemical, semiconductor, and software) does a lot of manufacturing, and operates 2 trade borders, the Gulf ports and Mexico.

</comment></comment><comment><USER23> I am here in San Antonio, I wanted to get Solar on the house, the cheapest quote was $14,000. A lot of new build houses here have solar panels on the top. The issue with that is some fire departments will refuse to do anything but contain the fire if you have solar panels (i guess they need training).

CPS energy (Our big power company) keeps sending fliers that i can be on wind power, the catch is it costs more. Seriously, i can pay more to ensure my power comes from (or is offset by) wind production.

I hope Texas does do more Solar and Wind power. Solar water heaters would be nice here too.

<comment><USER9> Be careful with solar power hot water. After living in China, I will never ever live in an apartment again whose only hot water option is solar. Way too many warm/sort of cold showers have traumatized me.

</comment></comment><comment><USER24> > between wind energy from West Texas and the Gulf Coast, and solar energy > across the state, Texas could meet a significant portion of its electricity > demand from renewable power without extensive battery storage. The reason: > These sources generate power at different times of day, meaning that > coordinating them could replace production from coal-fired plants.

Under ideal conditions, or even on cloudy, calm days?

If it's only on certain days, the headline and this quote are misleading.

</comment><comment><USER25> Why quit when you can use both?

<comment><USER26> There is an abundant body of evidence showing that coal is a highly polluting source of energy.

<comment><USER25> Can coal be a net zero emission fuel?

<comment><USER27> Technically, yes. But coal's main advantage over other electricity sources is that it's cheap. If you capture all the emissions it will no longer be cheap.

Here's one disastrously-over-budget attempt at a coal plant with CO2 capture:

[https://en.wikipedia.org/wiki/Kemper_Project](https://en.wikipedia.org/wiki/Kemper_Project)

Had they forced the plan to completion, it would have generated more expensive electricity than a new nuclear plant. More expensive than battery-backed solar. More expensive than _any other_ utility scale electricity source in the United States.

</comment></comment></comment></comment><comment><USER28> This article is shortsighted garbage.

Natural gas peaking plants make up the difference when demand modulates.

In the last 30 years, mankind has gone from 90% fossil fuels to 89% fossil fuels usage.

To make solar, wind, and other renewables, you need lots of liquid hydrocarbons and their counterparts.

<comment><USER29> > _Natural gas peaking plants make up the difference when demand modulates._

They explicitly say this in the 'garbage' article that's talking about coal, not liquid hyrdocarbons.

FTA:

> _Weather, however, remains unpredictable. Texas would still need battery > storage and natural gas-fired power plants to fill in gaps when, for > example, winds might slacken earlier than expected._

</comment></comment><comment><USER30> I wonder what the fall out is going to be when some weather system arrives that provides cloud and no wind for several days and there isn't enough natural gas, coal, nuclear, grid-scale battery storage or hydro capacity left to meet all of the demand.

I am sure it will be a rare event and probably preferable to continued c02 emissions. Probably will encourage more investment in energy storage.

Exciting time to be a power engineer!

<comment><USER31> No wind for several days would take some serious geoengineering to accomplish given there is a day/night cycle and temperature differentials create wind. Let alone also somehow obtaining cloud cover at the same time which won't create a wind from the borders not cloud covered.

<comment><USER32> That isn't correct. According to a paper from the Fraunhofer Institute:

>...Long phases of no or little wind power are a potential thread to future energy systems with a high share of renewable energies. A frequently cited example observed in Germany was a whole week with very little wind power due to temperature inversion in January 2009.

>...We find the average duration of low wind power feed-in phase to grow linearly with the threshold: Phases with a wind power feed-in of less than two percent of installed power are typically four hours long and phases with less than five percent feed-in are on average seven hours long. However, a period of wind power feed-in below eight percent of installed power that lasts one week occurs every two years and a period of more than ten days occurs every ten years.

[https://tu-dresden.de/bu/wirtschaft/ee2/ressourcen/dateien/l...](https://tu- dresden.de/bu/wirtschaft/ee2/ressourcen/dateien/lehrstuhlseiten/ordner_enerday/ordner_archiv/ordner_enerday2014/ordner_pacp/ordner_fpap/folder-2014-04-22-7870620856/pltz_fullpaper_2014.pdf?lang=en)

<comment><USER31> I was speaking of wind in absolutes - even then it produces 3% which while not adequate unless massively overbuilt is still something. Although it doesn't mention solar during that time period.

<comment><USER32> >I was speaking of wind in absolutes -

Well... the original poster was referring to the potential problem where you don't get any electricity from wind for several days, so it might have been better if you dealt with that issue rather than speaking of wind in absolutes since utility grade windmills need at least roughly 13 MPH winds to generate power.

>...even then it produces 3% which while not adequate unless massively overbuilt is still something.

No, not really.</comment></comment></comment></comment></comment>


Static Site Generators - <AUTHOR> https://www.contentstack.com/blog/all-about-headless/static-site-generator-gatsby-and-contentstack <comment><USER2> Faster load time, easier scalability, and the overall affordability that these two elements introduce are probably the strongest argument in favor of looking into static site generators.</comment>