Larry Elison and Trump Pushing Hydroxychloroquine Health Info Collection App - <AUTHOR> https://www.washingtonpost.com/politics/oracle-to-partner-with-trump-administration-to-collect-data-on-use-of-antimalarial-drugs-to-treat-covid-19/2020/03/24/ecbb8b76-6de2-11ea-b148-e4ce3fbd85b5_story.html <comment><AUTHOR> Katherine Eban just published this extensive article about Larry Ellison's hydroxychloroquine app on Vanity Fair, based on leaked internal documents. And in this MSNBC interview, she talked about how Larry Elison convinced Trump to push hydroxychloroquine to anyone who wanted to take it, and also distribute his cloud-based Protected Health Information sucking "information collector" app to all those human guinea pigs:

Vanity Fair looks at Trump's plan to battle virus with unproven drug. Vanity Fair contributor Katherine Eban discusses new reporting on the president's plan to beat COVID-19, which involved distributing millions of doses of an unproven drug.

[https://www.msnbc.com/morning-joe/watch/vanity-fair-looks- at...](https://www.msnbc.com/morning-joe/watch/vanity-fair-looks-at-trump-s- plan-to-battle-virus-with-unproven-drug-82512965690)

>Q: Hey Katherine, it's Willie Geist. Just want to ask you quickly about where this idea came from to President Trump. He doesn't typically peruse the New England Journal of Medicine I think in his free time looking for treatments to problems. So we know Rudy Giuliani was in his ear pushing this. We know certain Fox News hosts either visited the White House or called him to talk about this as they pushed it on their air. Where did the idea even come from for Donald Trump to raise this?

>A: You know, there's been some reporting around this, you know, a couple of bloggers and online things that sort of filtered up to the president. We also know that Larry Ellison, who is the Oracle founder, was talking about this to him. Ellison offered to donate an app which could track patient symptoms. Patients who were using these experimental drugs. So somehow, and I think it's not exactly clear yet, but will become clear, the hydroxychloroquine idea coalesced with the idea of using this app to monitor symptoms, and that the hydroxychloroquine was going to be pushed out very widely, and I think that is the piece that our story in Vanity Fair really adds to, was the idea for this very wide and largely uncontrolled distribution of this unproven and potentially dangerous drug.

"Really Want to Flood NY and NJ": Internal Documents Reveal Team Trump's Chloroquine Master Plan

[https://www.vanityfair.com/news/2020/04/internal- documents-r...](https://www.vanityfair.com/news/2020/04/internal-documents- reveal-team-trumps-chloroquine-master-plan)

>Inside the administration, as the White House cobbled together a plan to make chloroquine drugs widely available to the American public, Trump's political appointees began exerting tremendous and unwelcome pressure upon career health officials. As part of the plan, Oracle, the technology company co-founded by billionaire Trump fundraiser Larry Ellison, designed and built an app to collect data from physicians and patients tracking the response to various experimental treatments for COVID-19. (A source familiar with Oracle's app called it "an information collector; it does not recommend therapies or treatment plans.") [...]

>On the evening of March 23, the FDA's chief counsel, Stacy Amin, emailed lawyers and other officials within HHS, the National Institutes of Health, and the FDA, urging action. The proposal that she relayed, as she spelled it out, was to have BARDA sponsor what is called an investigational new drug study. An IND permits a new drug in preclinical development to be shipped across state lines to be studied. In this case the IND would have covered an old drug with a potential new use. Amin, who served as a special assistant to President Trump before assuming her current role at the FDA in September 2018, wrote, "The President is announcing this tonight and I believe the WH would like it set up by tomorrow with data to flow into the Oracle platform." She then asked, "What needs to be done and what requirements do we think can be waived or use enforcement discretion?" [...]

>As health officials navigated a minefield of long-standing regulations that were impeding the White House campaign, the message from the presidential podium was exultant: Trump had zeroed in on a potential cure and had slashed red tape to speed it to patients in need. On April 4, the president declared that the tech giant Oracle had donated a "very sophisticated" web portal to gather real-time data on how patients were responding to the new treatments. [...]</comment>


How to Get Beyond Tribal Politics - <AUTHOR> http://www.wsj.com/articles/how-to-get-beyond-our-tribal-politics-1478271810 <comment><USER2> I wonder to what degree the recent bifurcation in American politics may call for a slight return to more local governance. Much of the mistrust seems to originate in the idea that the [coastals|hicks] will get an outsized say in their inverse communities. If Alabama wants to teach that the world is 10kyo then is that really the worst thing? I think it's pretty bad, I think that it dooms a generation of Alabamans to not work in STEM jobs (statistically), but the cost might be worth it to depressurize the political environment across the country. The other likely option seems continued mutual alienation and eventual political violence. I don't like the idea of leaving more liberally inclined pockets of Americans to the whims of their more conservative and populous neighbors but presumably the inverse is also true so it might be a worthy compromise. The final point I'll make is that the virtue of different approaches will be born out in the variable results of states approaches and vindicate or indict them, ultimately leading to a gradual consensus.

Caveat: yes this would be super dangerous for a lot of liberal priorities (topically trans rights comes to mind). There is however the overriding reality that the legitimacy of government needs to be overwhelmingly accepted in a healthy democracy and telling 40% of the country to pound sand may be ultimately more counterproductive.

</comment><comment><USER3> What would be nice is for people to start checking members of their own tribe or party. Saying Trump is Hitler and things of that nature is ridiculous. Republicans are not the KKK. Tell your friends to tone it down a notch.

On the other side when some can only describe liberals as libtards or some such juvenile insult there is a problem. Or when a tiny group on campus does something ridiculous and then extremists say all liberals are like them.

Stop puting up FB posts on something political that don't cite sources. If it doesn't have a easily linkable source then it is almost always a half truth. Most situations are shades of gray.

<comment><USER4> The whole Hitler thing is strangely universal. I've previously read about Abe being compared to Hitler.

A stranger incident was a picture of Modi next to a Swastika - the (Indian) authors found no irony in using a Hindu symbol, and embedding in it the European epistemology of "Aryan pure blood", in order to appeal to people who they were appealing to.

Admittedly the elites of the latter are indistinguishable from those in the US, and this is imaginably true of Japan too.

Is this true outside the Anglosphere and their peripheries too ?

Edit: I think Assange has the situation right. Trump is symbolic of "White trash" \- much as Modi is symbolic of the un-anglicized mass in India. Their opposition, dub themselves with various epithets like "progressive" "liberal" "secular" etc.. until the cows come home.

[https://www.youtube.com/watch?v=_sbT3_9dJY4](https://www.youtube.com/watch?v=_sbT3_9dJY4)

That said, it should be noted that this doesn't necessarily mean the contenders are going to do anything drastically different from the status-quo. Certainly has been true in India.

<comment><USER5> > A stranger incident was a picture of Modi next to a Swastika - the (Indian) > authors found no irony in using a Hindu symbol, and embedding in it the > European epistemology of "Aryan pure blood", in order to appeal to people > who they were appealing to.

You're missing some context -- that most Indian-locals have. Swatika is a common religious symbol used on everything from new stationary to inaugurations/events. The "Nazi"-ness of it is likely unknown to most.

So it would only be strange to those for whom the symbol is only taught/learnt in the Hitler-WW2 context.

</comment></comment></comment><comment><USER6> The polar sentiments and social divide in America that Trump took advantage of will not go away after the election. I really don't want them to be swept under the rug, but I worry about the civil unrest if we continue to debate each other in such barbaric ways.

<comment><USER7> > that Trump took advantage

You mean the division that was caused by the current administration?

The divide in this country was here before the election... it's why the "alternative" candidates (Trump and Bern) were on track to win (if not for DNC shenanigans). People are sick of politics as normal.

Those politics are driving the wedge between groups, causing cops to be ambushed and are destroying this country/world from the inside out.

<comment><USER8> It's clear you have strong opinions about this, which indicates you care. That's great.

I agree with you that the Trump comment was unnecessary, but agree with the sentiment of the rest. I chose to ignore that small part of the statement to focus on the part we agree on.

The manner in which you're expressing those opinions sound like you're trying to call the gp out rather than try to figure a constructive way forward. Do you disagree? Do you think there is a way forward? If so, what is your method? Does your comment support that method?

One thing you agree with the gp on is that a wedge is being driven, and it sounds like you also believe that's a bad thing. How can we work together to mend that?

<comment><USER7> It's hard to say how we get past the polarization that is dividing everyone. Past the partisan politics.

It's hard to arrive at a constructive conversation for complex topics within the span of a text message - or a few paragraphs.

Both sides are pulled further left/right and people spend more time in echo chambers where they unfriend/get unfriended by those with different opinions.

I'm not sure how you get past these hurdles in a "text messaging"/facebook type environment.

"Trump takes advantage" of the divide caused by the "current administration". I try not to be vitriolic but its hard not to be angry at those in power for letting it get this bad. And by "those in power", I do include BOTH sides.

<comment><USER8> I completely agree. I'm amazed at how infuriated I can find myself sometimes.

One advantage of the text messaging/Facebook-type environment is that it's async. We have the opportunity to step back, take a breath, re-read what we wrote before we press send. I think I end up deleting at least a third of the comments I start to write because I realize I haven't figured out ways to express something properly. Or I'll blanket it in so many qualifiers to try to make sure I'm not misunderstood that the actual content is hard to parse out. And just text is a difficult, low-bandwidth medium because we don't get to see any of the cues associated with body language or tone.

But hopefully I'll get better with practice. And I don't want to stop trying, because I think it's really important for us to figure this out.

Thanks for taking the time to respond. It gives me hope that we'll be able to get through this.</comment></comment></comment></comment></comment>


Show HN: Open-Source Alternative to Netlify - <AUTHOR> https://staticdeploy.io/ <comment><USER2> Neat, i was looking for an alternative to Netlify. My hugo site does need some automation!

<comment><USER3> I've been using render.com which isnt open source but works quite well.</comment></comment>


Lua code: security overview and practical approaches to static analysis [pdf] - <AUTHOR> http://spw17.langsec.org/papers/costin-lua-static-analysis.pdf <comment><USER2> > Security Analyzer ... walks the AST and performs the taint propagation and > analysis on the AST. This way it is able to detect potential vulnerable > blocks that allow dangerous tainted inputs from the user into the sensitive > sinks.

This is basically the only thing about their detection method, which is pretty disappointing, even more so comparing to exciting things like 'inter- and intra-procedural data flow analysis' in related work section.

> Alternatively, one could use the parsers from the existing tools, such as > LuaCheck [17] or lua-checker [19]. Unfortunately the parsers in those tools > are tailored to particular versions of Lua specifications and adapting them > to all or latest Lua specifications may be untrivial or may introduce bugs > and unnecessary complexity.

Wrong, at least for Luacheck parser: it supports all Lua syntax starting from 5.1.

</comment><comment><USER3> I just realized that Lua is open source but does not accept outside code contributions. Meh.

<comment><USER4> They have an active mailing list: [https://www.lua.org/lua-l.html](https://www.lua.org/lua-l.html)

And they are welcoming towards patches, though they don't copy them all in verbatim: [https://www.lua.org/faq.html#1.9](https://www.lua.org/faq.html#1.9)

> _1.9 – Do you accept patches?_

> _We encourage discussions based on tested code solutions for problems and > enhancements, but we never incorporate third-party code verbatim. We always > try to understand the issue and the proposed solution and then, if we choose > to address the issue, we provide our own code. All code in Lua is written by > us. See also the previous question [On the lack of a public, official VCS]._

Few large projects accept all outside contribution. And Lua would not be as versatile, as portable, or as small as it is if it did.

<comment><USER5> I was about to say, accepting all outside contributions would promote bloat and kill portability. I don't think they are doing this in an elitist way, just as a matter of practicality.</comment></comment></comment>


Moving on from SpiderOak Hive to BTSync - <AUTHOR> http://dougbelshaw.com/blog/2013/09/26/moving-on-from-spideroak-hive-to-btsync/

<comment><AUTHOR> As requested, providing an update to this post which hit the front page of HN last month!

[https://news.ycombinator.com/item?id=6289664](https://news.ycombinator.com/item?id=6289664)</comment>


Why some smart people don't take action? - <AUTHOR> http://www.lifebeyondcode.com/2010/03/14/why-some-smart-people-dont-take-action/

<comment><USER2> Very insightful. It's like standing on a trampolin. The higher you climb, and the longer you wait, the harder it is. And if you start off too high, it gets even worse.

Great blog, this other article on why smart people are reluctant to share their thoughts and knowledge is also a great read: [http://www.lifebeyondcode.com/2010/03/2009/12/26/why-some- sm...](http://www.lifebeyondcode.com/2010/03/2009/12/26/why-some-smart-people- are-reluctant-to-share/)

</comment><comment><USER3> Thanks a ton KVS for featuring this here. Much appreciated.</comment>


All extensions disabled due to expiration of intermediate signing cert - <AUTHOR> https://bugzilla.mozilla.org/show_bug.cgi?id=1548973 <comment><USER2> There's a workaround that involves going to about:config and setting xpinstall.signatures.required to false.

However, if you're running the Stable or Beta version, it will only work under Linux. On Windows and MacOS you'll need to download Nightly or the Developer Edition.

To fix this on MacOS I did the following:

1\. Downloaded and installed Firefox Nightly

2\. Ran /Applications/Firefox\ Nightly.app/Contents/MacOS/firefox-bin --profilemanager

3\. Changed the profile to "default" so my normal Firefox profile would be used

4\. Started up Firefox Nightly, opened about:config, then set xpinstall.signatures.required to false

Not sure if it's a good idea to use my default profile in Nightly. It might be a wiser idea to copy it instead.

<comment><USER3> Upgrading your profile from Release to Nightly, which occurs automatically when you open it with Nightly, is a one-way irreversible step. This could prevent your profile from being used with Release without crashes, or lose profile data such as bookmarks or saved passwords when later used with Release, depending on what work is underway in Nightly and if it happens to be backwards-compatible. Be sure to backup your profile if you choose to switch channels.

Note: I am told that Developer channel uses a separate profile, but there are instructions below showing people how to override that, at which point this warning becomes relevant once again.

<comment><USER4> Oof. Would you happen to know if it's the same with the developer edition as well?

<comment><USER5> The developer edition has its own user profile.

<comment><USER4> And I told the developer edition to use my regular profile because that's the one that has all my settings and add-ons and I didn't realize the risk was there. Guess at this point all I can really do is hope and cross the bridge when I get there.

<comment><USER6> If you're on Mac, you should be able to recover the old profile with time machine. Or if you are on windows and have another backup setup.

</comment></comment></comment></comment></comment></comment><comment><USER7> Update: We have rolled out a partial fix for this issue. We generated a new intermediate certificate with the same name/key but an updated validity window and pushed it out to users via Normandy (this should be most users). Users who have Normandy on should see their add-ons start working over the next few hours. We are continuing to work on packaging up the new certificate for users who have Normandy disabled.

<comment><USER8> I've been through all of Firefox `about:config` a few times in the past, fixing preferences to, e.g., try to disable umpteen different services that leak info or create potential vulnerabilities gratuitously, but this is the first I recall hearing of Normandy.

Apparently I missed `app.normandy.enabled`, because I think I would've remembered a name with connotations of a bloody massive surprise attack.

Incidentally, `app.normandy.enabled` defaults to `true` in the `firefox-esr` Debian Stable package. Which seems wrong for an ESR.

For personal use (not development), I run 3 browsers (for features/configurations and an extra bit of compartmentalization): Tor Browser for most things, Firefox ESR with privacy tweaks for the small number of things that require login, and Chromium without much privacy tweaks for the rare occasion that a crucial site refuses to work with my TB or FF setup.

Today's crucial cert administration oops, plus learning of yet another very questionable remote capability/vector, plus the questionable preferences- changing being enabled even for ESR... is making me even less comfortable with the Web browser standards "big moat" barrier to entry situation.

I know Mozilla has some very forthright people, but I'd really like to see a conspicuous and _pervasive_ focus on privacy&security, throughout the organization, which, at this point, would shake up a lot of things. Then, with the high ground established unambiguously, I'd like to see actively reversing some of the past surveillance&brochure tendencies in some standards. And also see some more creative approaches to what a browser can be, despite a hostile and exploitive environment. Or maybe Brave turns out to be a better vehicle for that, but I still want to believe in Mozilla.

<comment><USER9> I too use Debian's Firefox ESR. I noticed the "Allow Firefox to install and run studies" option in Privacy & Security Preferences a long time ago. It was unchecked and greyed out (i.e., unclickable), and a label below it says "Data reporting is disabled for this build configuration", so I gave it no further thought. This morning I woke up and launched Firefox, noticed this headline, and then noticed my extensions were still running. I looked in about:config and lo and behold, app.normandy.enabled=default [true]. I'll be filing a bug with debian to disable this in the build configuration.

Edit: There are some questions about whether Normandy is really enabled in Debian Firefox ESR even if the about:config setting defaults to true. I've filed a bug report, and I'm sure once a Debian maintainer has a chance to look at it we'll find out the answer.

[https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=928433](https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=928433)

Edit2: It should go without saying, but please do not spam this bug report with "me too" and its ilk.

<comment><USER10> I had mine disabled. So let's think about this for a second. If I disable a security hole that you can drive a semi-truck through, I remain foobar'd. If I run my "secure" firefox configuration, with the security hole enabled, then they un-foobar me first. Before anyone else. So I could effectively get rewarded, for always keeping a security hole open. But I didn't keep it open, so... yeah... they'll get around to me sometime.

?

>:-(

Grrr.

I'm just getting old and curmudgeonly maybe? I've decided though, I'm starting an animated security blog to show people the ludicrousness of all this kind of stuff in plain language. I'll be Statler, and I just need someone to be Waldorf. Because this stuff really is getting Statler and Waldorf level ridiculous.

<comment><USER11> > So I could effectively get rewarded, for always keeping a security hole > open.

That's the way it always works, isn't it? Security and convenience are opposing concerns.

<comment><USER12> Good security designs increase convenience (eg ssh, touch id, single sign on).

The three goals of computer security are integrity, confidentiality and availability.

All three of those expand the usefulness of the system to the end user.

<comment><USER13> But the point here is not about integrity, confidentiality, or availability. It is about whether you trust Mozilla, and how much trustworthy they are.

A configuration where Mozilla cannot push remote updates is neither more secure nor less secure. Mozilla is often under fire for not allowing a privacy conscious, minimal trust use case.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> Just discovered the same message in the Tor browser, and it seems that NoScript got disabled. So people running Tor are a lot more vulnerable right now.

Also, wow, the web has a ton of ads. I've been running uBlock origin so long I forgot how bad it had gotten :(

<comment><USER15> > Also, wow, the web has a ton of ads. I've been running uBlock origin so long > I forgot how bad it had gotten :(

Try turning it off. I got rid of ublock after arstechnica complained about a lot of their users blocking ads years ago and it honestly isn't that bad. Every once in a while I do back out of a page for maxing out one of my cpu cores but otherwise, nothing ever bad happens. With ads: either it takes me half a second to tell I'm not interested in an ad, or I actually am interested and i follow the ad because I am interested and I want to support the website.

The alternative is websites charging insane amounts of money with paywalls (Wall street journal has their "best" price for 12 months at $360 a year). That is horrible because it means only rich people can pay for high quality news as ads are one of the most progressive forms of payment (rich people ads are way more valuable than poor peoples and yet everyone gets the same quality services/news with the ad model despite their income/net worth).

<comment><USER16> The alternative is those websites not using third party ads with third party trackers on it. Adblockers _already_ do not block those (cause they're indistinguishable from image links). If they really just want my eyeballs they know how they can get them.

But they really want to _track_ me. And I'm not having that. The moment they stop tracking their users through third party ad networks, most adblockers stop blocking (because there's no AI involved and they wouldn't know what to block except images in general).

It's in their hands, really. If they want to show me ads they can do it in a normal and decent manner.

News websites should in fact be the first to adapt this model, because it's _exactly_ the same thing as ads in print media. But they _chose_ to get those disgusting third party tracking networks involved. And not just one or two.

I don't have to put up with that, but I really don't see why there would be an action required on my site to stop blocking those tracking ads.

<comment><USER17> Just FYI, that's not really true. Adblocker use mostly all the same filter lists and those do regularly block ads that just are regular images, and even text notes. [https://www.troyhunt.com/ad-blockers-are-part-of-the- problem...](https://www.troyhunt.com/ad-blockers-are-part-of-the-problem/) is an example, even if that specific one got resolved

Adblock Plus has the ability to not block ads that conform to a certain standard, but in addition to conform to standards ad publishers need to pay for that. At least that's what they claim.

<comment><USER18> If the images are hosted on the site instead of a third party the list won't matter.

<comment><USER15> Not true, here is an example of easylist blocking OpenStreetMap advertising OpenStreetMap events on openstreetmap.org [0].

[0]: [https://github.com/easylist/easylist/pull/900](https://github.com/easylist/easylist/pull/900)

</comment></comment></comment></comment></comment></comment><comment><USER19> To re-enable all disabled non-system addons you can do the following. I am not responsible if this fucks up your install:

Open the browser console by hitting ctrl-shift-j

Copy and paste the following code, hit enter. Until mozilla fixes the problem you will need to redo this once every 24 hours:

        // Re-enable *all* extensions

        async function set_addons_as_signed() {
            Components.utils.import("resource://gre/modules/addons/XPIDatabase.jsm");
            Components.utils.import("resource://gre/modules/AddonManager.jsm");
            let addons = await XPIDatabase.getAddonList(a => true);

            for (let addon of addons) {
                // The add-on might have vanished, we'll catch that on the next startup
                if (!addon._sourceBundle.exists())
                    continue;

                if( addon.signedState != AddonManager.SIGNEDSTATE_UNKNOWN )
                    continue;

                addon.signedState = AddonManager.SIGNEDSTATE_NOT_REQUIRED;
                AddonManagerPrivate.callAddonListeners("onPropertyChanged",
                                                        addon.wrapper,
                                                        ["signedState"]);

                await XPIDatabase.updateAddonDisabledState(addon);

            }
            XPIDatabase.saveChanges();
        }

        set_addons_as_signed();

Edit: Cleanup up code slightly...

<comment><USER20> I just set xpinstall.signatures.required to True in about:config and that fixed it after a restart.

<comment><USER21> I can confirm this on Ubuntu 19.04 with Firefox 66.0.3. Changing xpinstall.signatures.required from the default 'True' to 'False' resulted in addons working again.

<comment><USER22> Android too.

</comment></comment></comment></comment><comment><USER23> Run this in your Browser Console[1] to delay signature checking for a day:

        function set_xpi_sign_time_now() {
            const {Services} =  ChromeUtils.import("resource://gre/modules/Services.jsm");
            const now = (new Date()).getTime() / 1000;
            Services.prefs.setIntPref('app.update.lastUpdateTime.xpi-signature-verification', now);
        }

        set_xpi_sign_time_now();

EDIT: Changed `Components.utils.import` to `ChromeUtils.import` because apparently Beta and Nightly versions have removed the former, while the latter was introduced in 60.

This does the equivalent of setting in about:config the time of last signature verification to the current time. By default, Firefox re-checks signatures in 24 hours (or so I read somewhere here). I like the temporary effect of this, compared to the permanent disabling of signature verification suggested elsewhere.

\----

1: [https://developer.mozilla.org/en- US/docs/Tools/Browser_Conso...](https://developer.mozilla.org/en- US/docs/Tools/Browser_Console)

<comment><USER24> Outside of about:addons I get this in Firefox 66.0.3 in Linux:

> ReferenceError: ChromeUtils is not defined

Also, this doesn't seem to help with currently disabled add-ons, unless I'm missing something. Trying to reinstall Adblock Plus, for example, still results in

> Download failed. Please check your connection.

<comment><USER23> > Outside of about:addons ...

You seem to be trying in a standard Web Console, not the Browser Console.

</comment></comment></comment><comment><USER25> I'm a bit confused.

I thought that the way signing works in general is that the signer issues a certificate for the thing being signed (domain, code, whatever) that contains identifying information for the thing signed (host name for an SSL certificate, checksum of the code for a code signing certificate), the valid from and valid to dates of that certificate, and assorted other information, and either a reference to or a copy of the signer's certificate, and it signs the whole issued certificate with the signer's certificate.

Someone checking the signed thing is supposed to consider it validly signed if:

1\. The date is in the valid range for the signed thing's certificate,

2\. A check of the signature of that certificate against the signing certificate passes,

3\. The signing certificate is recognized as being from an issuer considered trusted by the checker,

4\. Neither the signed thing's certificate nor the signing certificate have been revoked, and

5\. The signing took place during the valid date range of the signing certificate.

Note there is no "the date of the check is in the valid date range of the signing certificate". A signing certificate expiring should not invalidate things signed by it. It should just prevent signing anything else with it.

So why is a signing certificate expiring for Firefox breaking already signed extensions? Shouldn't it just be stopping new versions of extensions from being signed?

<comment><USER26> As noted, in practice, without additional info, there's no way to tell when a signature was created and so all signatures die when the signing cert expires.

That said, existing signatures that have already been verified, for existing extensions, should still be trusted.

<comment><USER27> Yeah, this response is a bit strange because if I trust my own system's clock, I just have to remember for each signature when I first saw it.

</comment></comment></comment><comment><USER28> They have acknowledged the defect and are working on a fix. While this is a severe impact, I am still with Firefox. The are enough alternative browsers to tide over the problem for now. The fact that alternatives exist is the reason why we should support projects like Firefox.

<comment><USER29> >I am still with Firefox

that's kinda the problem. there's plenty of reasons to be "with" firefox still, but you shouldn't need reasons other than it's the best browser. when it starts requiring loyalty to be a user, that's a big problem.

<comment><USER30> For me, it is the best browser. Yes, this is a big fuck up, but it's not like this has caused me material harm. It's easy for me to switch over to Chrome until this is fixed, and I doubt the same mistake will be repeated in Mozilla.

I expect perfection from plane and car manufacturers, and I pay for that. My browser, I can live with an occasional hiccup.

</comment></comment></comment><comment><AUTHOR> Looks like all extensions have been disabled for all Firefox users.

I think this fail-closed behavior is more of a security issue than the one it is trying to solve. All of my security add-ons - Privacy Badger, NoScript, Decentraleyes, and many more were disabled. Even worse, it happened without notice to the user.

One moment I was browsing the internet (just barely) secured by these add-ons, and the next moment, all of them disappeared (without warning) and I only noticed when I saw my password manager was missing.

<comment><USER31> If it failed open, anyone unlucky enough to update their extensions could end up having a malicious version installed. It also would have taken longer to notice.

<comment><USER32> So why not just disable extension updates instead of disabling the extensions themselves?

<comment><USER33> Presumably because how would it differentiate between a legit "already installed" extension with a signature that cannot be verified, and an extension installed by malware that also cannot be verified?

<comment><USER32> Personally I despise the idea of the software already on my pc being dependent on signatures stored on a remote server. I installed it and Mozilla can fuck right off. It's my responsibility to police what software is on my computer, not theirs.

<comment><USER34> According to [https://news.ycombinator.com/item?id=19824520](https://news.ycombinator.com/item?id=19824520) the signatures are on the extensions themselves, not on a remote server.

</comment></comment></comment></comment></comment></comment><comment><USER35> As a temporary fix, go to about:debugging, and click "load temporary addon", then paste in the download link of the missing add-on. Then just try and not restart Firefox until they fix the broken cert.

<comment><USER36> This works. On a desktop, you can reload installed addons from the firefox profile folder >> extensions.

<comment><USER37> Yes, that seems to work on desktop. Thanks.

</comment></comment></comment><comment><USER38> I'll still keep using Firefox since I recognize the importance of browser diversity and the hazards of a Chrome monoculture (that and vertical tabs), but, yikes.

Still, this type of oversight seems all too common even in large companies. I remember several cases from Fortune 500 companies in the past few years alone. What would be a good way to automate checking for them? Has anyone developed a tool designed specifically to avoid certificate expiry disasters?

<comment><USER39> > Still, this type of oversight seems all too common even in large companies. > (...) Has anyone developed a tool designed specifically to avoid certificate > expiry disasters?

LetsEncrypt renewal is supposed to be automated. [1]

I know of a company that hosted blogs for thousands of customers. They used LetsEncrypt, but the CTO considered automatic renewals a possible security risk, so they did it manually. Problem is, the expiration happened in a weekend and they "forgot" to update the certificates before that. Suffice to say that the next Monday wasn't pleasant. They automated after that.

[1] [https://letsencrypt.org/about/](https://letsencrypt.org/about/)

<comment><USER40> Just curious, are you talking about Webflow? Because I had to hunt down and make sure our Let's Encrypt auto renewal was working until I realized the certificate was served by them. They wait until the last 12 hours to renew the certificate. I have no idea what type of rationalization would lead to that decision.

<comment><USER41> 90 days is 4 times a year. 60 is 6 times, 50% more expensive when you're paying someone to perform the task.

<comment><USER40> I had the same thought, but I still find that absurd. Say they host 500,000 websites with HTTPS. 1,000,000 renewals they save spread across the year, roughly 2 renewals a minute. That is pennies. A t2.medium could handle that type of load increase

<comment><USER42> A bit OT, but what's up with this usage of Amazon EC2 tiers as a unit of computational power?

<comment><USER43> i think it's a combined "fixed cost" rather than just computational power... like you could do it with x, thus it should cost at most y

similar to saying that you could do it with a raspberry pi

</comment></comment></comment></comment></comment></comment></comment><comment><USER44> I don't get why an expiring cert disables the extensions. Shouldn't the browser be checking the cert expiry date against the date the extension was installed, not against current time? As long as there's no way to manipulate the extension installation date that would be fine, wouldn't it?

edit: or even why the browser is checking this at run-time. As long as it checked the cert when the extension was installed, isn't that enough?

<comment><USER45> If Mozilla somehow lost control of one of these signing certs (as in: it got stolen) they would put in on a revocation list. If certificates don't get re- checked, all installations between "cert got stolen" and "noticed that the cert got stolen" would keep installed & running.

</comment></comment><comment><USER46> I have disabled signature checks in Firefox because otherwise it is impossible to install a private extension without uploading the source code to Mozilla.

This is how you allow unsigned extensions in Firefox on Arch Linux, the same files can be edited on Windows and macOS, restart the browser after changes:

      sudo tee /usr/lib/firefox/defaults/pref/config-prefs.js &>/dev/null <<EOF
      pref("general.config.obscure_value", 0);
      pref("general.config.filename", "config.js");
      pref("general.config.sandbox_enabled", false);
      EOF

      sudo tee /usr/lib/firefox/config.js &>/dev/null <<EOF
      // keep this comment
      try {
        Components.utils
          .import('resource://gre/modules/addons/XPIDatabase.jsm', {})
          .XPIDatabase['SIGNED_TYPES'].clear();
      } catch (ex) {
        Components.utils.reportError(ex.message);
      }
      EOF

This method also works for the stable version of Firefox.

<comment><USER47> It's very good that it's possible, but it's literally the worst solution for most people.

</comment></comment><comment><USER48> There have been major organizational problems at Mozilla for a long time that precipitated this. Many of us saw something like this coming, saw gaps and unclear responsibilities, reported these gaps and confusions up the chain, and were reprimanded and financially penalized for asking the tough questions. The questions were never answered, and we all quit, were fired, or lost motivation as a result.

This is a tech problem, yes. Cert renewal has bitten everyone in a high profile way (apple, google, and ms have all had renewal-related outages in recent years). But this was preventable at Mozilla. Ask a Mozillian about IT and Cloud Sevices, and what their respective responsibilities are. Ask Mozilla's VP of IT- who is responsible for cert renewal? Ask Mozilla leadership- why are people afraid to ask questions?

<comment><USER49> You basically just described any sufficiently large organization. Complaining is not helping anyone in these situations, the only thing you can do to change things is to go ahead and try to change things. Reporting things up the chain hardly ever works because the chain is too busy with their own issues and politics. You have to make it worth their while.

<comment><USER50> >the only thing you can do to change things is to go ahead and try to change things

how? Reporting up the chain is THE way to change things. When it doesn't work what are you supposed to do?

<comment><USER51> Take on the responsibility for the gap yourself. If you get assigned something from up the chain that gets in the way of that new found responsibility report up the chain that they need to assign it to someone else first.

<comment><AUTHOR> If I am slowed down in my day-to-day sprint because I decided to take on a task management wasn't willing to fund/approve, I would be in much deeper trouble

</comment></comment></comment></comment></comment><comment><USER52> If everyone's add-ons are disabled, I wonder why mine are not. My computer has been running over night (coincidentally, first time in years) and my add-ons are intact. Does it take a browser restart? Or might I have a setting that prevents this from happening? My system time is correct.

Edit: am on Firefox 66, Linux (Debian Buster/testing), using Firefox from Mozilla directly (not through repositories), and my internet/wifi should not have disconnected. System has been up since 2019-05-03T17:30:00Z, suspended before that.

<comment><USER53> I think it only checks at start up, or something? Mine has been running for days, and all my add-ons are working. As such, I've decided not to restart Firefox until news arrive that it's working again.

<comment><USER54> Mines been running for days and everything was disabled.

<comment><USER52> Oh, darn. Thought I found a way to avoid this. Thanks for reporting!

Just a thought, might it be that you suspend (sleep mode in Windows) your system? So not restart the browser, technically, but iirc applications still receive some event when this happens. At the very least, open connections would break. Any TLS connection would re-validate the certificate. In that case, going offline would also trigger it. Have you done any of those (suspend, be offline / switch networks)?

</comment></comment></comment></comment><comment><USER55> Interesting coincidence, my Ubuntu 12.04 Let's Encrypt certbox-auto updated in March 2019 to a version which no longer runs on Ubuntu 12.04 without major surgery.

[https://community.letsencrypt.org/t/pip-error-with- certbot-a...](https://community.letsencrypt.org/t/pip-error-with-certbot- auto/88200/4)

I wonder if this bit Mozilla and caused this issue.

<comment><USER56> I can beat that. I had a Let's Encrypt cert renewal that upgraded python and itself and borked the system it was running on.

</comment></comment><comment><USER57> In addition to the immediate fix, what needs to happen here and in general anywhere a certificate is used, is the browser should display an informational banner that the certificate is due to expire soon. I'd suggest start warning at T-7 days left.

That way even if the business messed up, they would have a heads up from users to fix it before d-day when everything stops working. This includes website certs and addon signing certs and any intermediaries.

</comment><comment><USER58> Not sure what kind business processes are practiced in Mozilla. Some organizations have the notation of recurring tasks as part of their business processes. Recurring tasks are just like bug reports except they are created and assigned automatically to task owners on a schedule, such as every month, every quarter, and every year.

The goal of recurring task is to get people's attention to review and perform tasks the happen periodically. It could be as simple as reviewing it and marking it done. They will show up as part of the bug report to the assignees, so they can at one place see all the bugs, feature requests, tasks, and recurring tasks.

Cert renewing would fall under the recurring task category.

<comment><USER48> Who at Mozilla is responsible for cert renewal? Is it Mozilla IT or is it the Firefox org? That question has never been answered, and those who asked were often reprimanded. And this is far from the first certificate renewal problem.

<comment><USER58> Reprimanded for raising the question? That's quite dysfunctional. Sounds like that's an ownership problem.

</comment></comment></comment><comment><USER59> I'm not familiar with Firefox extensions (and have pretty much stayed away from the stuff ever since they started making it "mandatory"...) but shouldn't the expiration only mean _new_ signatures won't be valid, yet signatures made _before_ expiration should remain so? At least that's how I understand things like Windows' driver signing works (when that was first introduced, I was quite scared that it would mean perfectly working drivers could just stop working due to the expiration, and asked... but apparently no one at Mozilla asked this question.)

Edit: wow, downvotes? Care to explain what I'm missing?

<comment><USER35> This same behavior is how certs _usually_ work. Stuff with expired certs just _does not run_ after the expiration date; that's because the cert tells you what server to ask for authentication, and if you have an old cert, there's no way to be sure that the original issuer is still the one in control of that domain.

<comment><USER59> You're referring to things like HTTPS and contacting a _remote server_ , and thus your reasoning there makes sense.

I'm referring to traditional code signing, which I assume Firefox extensions are more similar to --- the goal being to ensure that some data has not changed since it was signed, and only the validity of the certificate at the time the data was signed is meaningful; even after the certificate expires, a signature created when it was valid still asserts that the data it signed has not changed.

</comment></comment></comment><comment><USER60> Newbie question: why can't they just renew the certificate, like in 5 minutes?

<comment><USER61> A new certificate can be generated if you have to in a couple of minutes, sure. Of course, you probably defined some procedures to do it properly and securely that require more time.

Then the issue becomes: how to get the new certificate to a few hundreds million users?

If it was a certificate on some server, just replace it there, done. Client software will just pick it up. But not here. A copy of the certificate is shipped in every add-on package file. Oops. Now you have to re-sign all add- ons with the new certificate. And get those resigned files to the users.

Essentially it works like this (which is a slightly modified jar/apk signing mechanism):

\- An add-on package is a zip file and other than the actual files there is also a list of known-good hashes of those files in a file called "manifest.mf" in the META-INF folder

\- Then there is a file "META-INF/mozilla.sf" giving hashes of "manifest.sf"

\- And finally, there is "META-INF/mozilla.rsa", which is a DER-encoded pkcs7 signature and two certificates. The signature verifies "mozilla.sf" was not tampered with and still is the same as when it was signed by mozilla. Which in turn verifies the known-good hashes are still proper.

\- The signature is made with a generated certificate, the first one included in "mozilla.rsa". E.g. "firstname.lastname@example.com" in case of uBlock.

\- The "firstname.lastname@example.com" certificate was issued by an intermediate certificate "CN=signingca1.addons.mozilla.org". This "CN=signingca1.addons.mozilla.org" is the second certificate in mozilla.rsa. It says "Validity Not After : May 4 00:09:46 2019 GMT". Oops. This is where the chain breaks now!

\- "CN:signingca1.addons.mozilla.org" was issued by "CN=root-ca-production- amo". This root certificate is baked straight into the browser and not part of mozilla.rsa.

Therefore, it is not enough to issue another intermediate certificate (e.g. "CN=signingca-number-two.addons.mozilla.org"), but you have to actually generate a new "firstname.lastname@example.com" (or whatever) signed by this new certificate, put those two certificates and a new signature of "mozilla.sf" based on those new certificates into a new mozilla.rsa FOR EACH add-on and ship updated add-on files.

PS:

Try it yourself... Extract some addon package (it's a zip file). Then:

        openssl pkcs7 -in META-INF/mozilla.rsa -inform DER -print

<comment><USER62> The root certificate is also part of the mozilla.rsa. Addons have it included, it can be extracted and then imported into the browser Certificates > Authorities and it will be used to validate addons, including those which are not updated.

A sample procedure doing exactly this, and a fix for Firefox <= 56.0.2 can be found here: [https://www.velvetbug.com/benb/icfix/](https://www.velvetbug.com/benb/icfix/)

The same procedure can be used on newer versions, but the syntax is a bit different (import cert, go to about:addons, open console):

      Components.utils.import("resource://gre/modules/addons/XPIDatabase.jsm");
      XPIDatabase.verifySignatures();

This only makes sense if you really need to fix it while the browser is running, otherwise you can simply restart it after the new certificate is imported.

</comment></comment></comment><comment><USER63> I like the alias name assigned to it: armagadd-on-2.0

Temporary work around till the cert gets fixed: set "xpinstall.signatures.required" to false

<comment><USER64> That doesn't work unless you're on dev or nightly.

</comment></comment><comment><USER65> Active discussion here: [https://news.ycombinator.com/item?id=19823465](https://news.ycombinator.com/item?id=19823465)

<comment><USER66> Even though that's active and has more votes, it's ranked very low (#39 as of right now, and this post is now #1 on the site).

I think HN penalizes non-link posts (or people are flagging it because they think it's just someone asking for tech support).

<comment><USER65> Looks like you're right. I saw the other discussion first and it still has more comments for now, but this one is ranked higher. Perhaps the threads could be merged or something.

</comment></comment></comment><comment><USER67> Honestly, I just have to wonder what the Mozilla devs do. Don't they use extensions? Didn't they notice what was going on? Why does it increasingly feel like a more random Google Chrome-Lite paired with Apple's 'best intentions'?

<comment><USER68> This is a certificate that expired, not some insufficiently-tested change someone deployed. Mozilla devs had their extensions disabled at the same time as everyone else.

<comment><USER69> Well, most of the actual developers probably run Daily and have xpinstall.signatures.required = false, and are therefore not impacted quite as severely.

</comment></comment></comment><comment><USER35> This is a goddamned disaster. I'm just thankful that I use an offline password manager, but even still ...

I _like_ FF, don't get me wrong, but this is going to absolutely fucking destroy user trust in Mozilla. This kind of incompetence, on a _browser scale_ , is breathtaking.

<comment><USER70> I dunno. I'm a typical Firefox user, and I'd rather jump off a bridge than switch to a different browser because of a fuckup like this. People make mistakes, but Mozilla still stands for things that certain other browser vendors don't, last time I checked.

<comment><USER71> That's my thinking too. I went back to Firefox a few months ago and it is back to being a fantastic browser now, and it feels good to use something that is also a force for good. I'm hoping they resolve this quickly and that it all turns out ok.

<comment><USER72> >and that it all turns out ok.

While I agree with you two assuming the bridge is over water and not too high, there are real consequences that cannot be reversed. I cannot unsee the ads I saw in the past few minutes before switching to nightly.

</comment></comment></comment></comment><comment><USER73> I'd be curious to see if there's a marked increase in net sales during the time that this snafu is ongoing just because uBlock Origin is disabled and users are forced to see more ads.

<comment><USER22> Maybe, if FF had a bigger market share. On top of that, I think many FF users know not to browse without it.

</comment></comment><comment><USER74> Patches to workaround the issue were just committed. Those patches were tracked at [https://bugzilla.mozilla.org/show_bug.cgi?id=1549010](https://bugzilla.mozilla.org/show_bug.cgi?id=1549010). At the time of writing this comment, that bug is not chained up to the bug linked by this post.

</comment><comment><USER75> This is highly ungood, of course. The failure to notice an upcoming expiration is terrible in itself, but I suppose we can sort of almost empathize with how such shit may happen. The failure _mode_ is inexcusable. Someone somewhere sometime must once have made a decision, "this will be the right way to fail", or - worse - developed the addon certification system and deciding it _didn 't_ need no stinkin' graceful degradation route in case of disaster.

Still, it's no worse a calamity than most everyone else presents to the world from time to time. I shall stick to my Firefox, partly influenced by the complete absense of any viable alternative out there.

By the way: Writing this a few hours after midnight, May the fourth, local time. Every single one of my 33 Firefox addons is active and working just fine. Wating to see. Breaking out my Waterfox, just in case.

[Edit: May fifth -> fourth]

</comment><comment><USER70> My extensions are still running. I even restarted Firefox a few moments ago. So it's not everyone?

<comment><USER19> My extensions are also still running. Comments on the bugzilla bug restricted so adding details here and figuring that if this is useful someone can forward it to the right people

        $ date
        Fri May  3 22:45:22 EDT 2019

        $ date --utc
        Sat May  4 02:41:47 UTC 2019

        $ firefox --version # Installed from arch repositories
        Mozilla Firefox 66.0.3

about:config

        xpi.signatures.required true
        app.update.lastUpdateTime.xpi-signature-verification 1556920447
        extension.update.enabled true

1556920447 is unix timestamp Fri May 3 21:54:07 UTC 2019.

Edit: I think I know why. It checks the signatures daily, and the timing works out so it hasn't checked since the cert expired for me. Just luck, it will break within the next 21 hours for everyone. From the source code:

        const XPI_SIGNATURE_CHECK_PERIOD      = 24 * 60 * 60;

        [...]

        timerManager.registerTimer("xpi-signature-verification", () => {
          XPIDatabase.verifySignatures();
        }, XPI_SIGNATURE_CHECK_PERIOD);

<comment><USER19> Copying a potential workaround here from my lobste.rs comment, not really tested obviously

If it hasn't broken yet for you, I think (but I'm not very much not sure) setting that preference to 1556940100 should keep it working until 24 hours from now. And if you keep updating that value every 23 hours to the output of date '+%s' until it is fixed via a firefox update it should keep working forever.

I think you need to restart the browser as well after updating the preference for the above idea to work.

</comment></comment></comment><comment><USER76> Mozilla doesn't seem to have communicated the issue well. I could imagine a lot of unsavvy users have tried some wild things in an attempt to fix the problem, and maybe made a mess in the process. Doesn't Mozilla have a mechanism for blasting out a message to all Firefox browsers? Also I have a Firefox account, why haven't I been inboxed about this?

Otherwise I'm not bothered. I won't be switching as long as this gets resolved within the next few days.

<comment><USER77> > Doesn't Mozilla have a mechanism for blasting out a message to all Firefox > browsers?

The cynical side of me says that it must not have this feature because if it did I'd have seen someone complaining about the browser "phoning home" or "forcing Mozilla's opinions into my eyeballs".

<comment><USER78> ...unlike the mothership breaking all addons, and the browser being designed intentionally in a way that prevents me from working around the breakage?

This isn't just a petty snipe borne out of annoyance. Ads being the malware vector that they are, and the degree of tracking and data mining out there, all of those countermeasures being turned off overnight is an exposure that should be treated with the same degree of seriousness as PII breach at a company you have an account with.

God forbid you use FoxyProxy or Tor Browser or something else that masks your connection source - this could have legitimate, real-life consequences if you don't notice the change.

<comment><USER77> On the one hand, I understand the point you're trying to make.

On the other hand, I'm going to be honest, I have trouble reading your post without thinking things like "If you are literally trusting your life to FoxyProxy, you might want to rethink your entire internet safety strategy." Another favorite was "Defense in depth."

I've had _dozens_ of different experiences where my extensions silently and unexpectedly malfunctioned. Configurations getting erased, new extension releases with breakage or different behaviors, maximum compatible versions in the manifest, new permissions, botched keystrokes in the extensions page, profile corruption, incompatibilities between extensions, internal bugs that cause them to crash-loop without doing anything, the works. Like, I run a ton of addons, some of which are a bit esoteric and a couple that I compile from head every few days, but even taking into account my outlier-sized surface area it's a bit silly.

I'm not going to claim that this isn't a problem. That said, blaming Mozilla for a life-threatening failure of _FoxyProxy_ , of all things, is like blaming Cessna because a journalist flew one of their planes into a combat zone. There just wasn't any way it was going to end well.

<comment><USER78> If the nature of this problem didn't also break turnkey distributions like Tor Browser (this kills Noscript, which means your identity can be leaked), I'd agree with you.

There's only so much defending you can do against a failure like this (running Tor Browser is already pretty uncommon) and the blame for it has to be laid squarely at the feet of Mozilla for the way they chose to centralize their plugin architecture.

This is one of those low likelihood/high impact events that tend to catch everyone by surprise.. if you spend all your time as a user thinking about these failure modes (you don't.. nobody does), you'd be unable to get much else done. I'd wager the fact that the browser would suddenly gimp itself is not something the average user (even the average Tor Browser user) thinks about or plans for.

</comment></comment></comment></comment></comment><comment><USER79> Tomorrow (or whenever this gets fixed), ad companies are going to have some great data about what the world would look like if adblock didn't exist. I really home someome does a blog post about it. Yikes, I hope it doesn't play out like a shark smelling chum.

<comment><USER80> Given that Firefox's market share has dropped below 10%, I doubt it will make much of a difference.

<comment><USER79> Yes but there's enough sample size for them to multiply the change by ten to get what would happen if Chrome lost adblock as well.

</comment></comment></comment><comment><USER81> I just ran into this. All my extensions were immediately turned off in the middle of a browsing session, from UI conveniences to important safeguards, with no apparent option to turn them back on again.

Warning about something that can't be verified is one thing, but automatically shutting things off -- particularly things that could affect security and privacy -- is a Windows 10 level of unacceptable interference.

</comment><comment><USER82> This is why users need to be in control of their own computers. Why can't I tell my copy of Firefox to ignore the certificate? Why can't I sign my own extensions?

Mistakes happen, it's okay. But users should be empowered to work around them.

<comment><USER83> > Why can't I tell my copy of Firefox to ignore the certificate? Why can't I > sign my own extensions?

The issue is that if you leave _any_ sort of lever that reduces security, it _will_ be abused by bad actors. This is why browsers are having ever decreasing ways to bypass security and have full access. It is annoying, but at the end of the day, protecting 99.999% of the users trumps what us power users want.

<comment><USER59> _protecting 99.999% of the users_

It is horribly paternalistic to advocate for keeping users ignorant, unlearning, and --- dare I say it --- easily manipulated.

I will refrain from mentioning again that infamous Franklin quote. I am frankly _very fucking pissed off_ by this authoritarian walled-garden trend, and vehemently oppose anyone who helps this industry put the nooses around the necks of others as well as their own.

<comment><USER84> I've been in software development and operations for 25 years.

I still don't want to have to understand everything I ever touch, even if I could.

<comment><USER85> >I still don't want to have to understand everything I ever touch

If you don't understand it, don't touch it. The default settings should work for most users. There can even be a warning against touching without understanding, like with Firefox's about:config. The offensive thing is preventing users from touching even if they do understand.

<comment><USER86> The difficulty is in how to keep them available to end users while keeping them unavailable to malware and bad actors who post "helpful" advice or publish temporarily useful addons that get updated to malware.

I'm not disagreeing with you, but the right mechanism is not straightforward to figure out, and you'll always be in a game of cat and mouse. One that sucks resources from whatever other useful stuff you might be spending your (or Mozilla's) time on.

</comment></comment></comment></comment></comment></comment><comment><USER87> On a slightly different note, is there some curated collection of serious incidents like this somewhere?

Something we could refer to when discussing possible pitfalls?

<comment><USER34> I believe [https://github.com/danluu/post- mortems](https://github.com/danluu/post-mortems) is close to what you want.

</comment></comment><comment><USER37> It's pathetic to see the attitude demonstrated by Mozilla support on this.

 _diox commented 4 hours ago_

 _I 'm locking this like I did in #851 because no new information is being added. We're aware and we're working on it. This conversation has been locked as spam and limited to collaborators._[1]

 _Bug 1548973 (armagadd-on-2.0)_ _All extensions disabled due to expiration of intermediate signing cert_ _NEW Unassigned (Needinfo from 3 people)_

 _Kevin Brosnan [:kbrosnan]_

 _We have confirmed this issue. Extra comments about this being broken will not advance this bug to being fixed._ [2]

Mozilla just left their entire user base unprotected against ads, trackers, and some hostile code. Then they insult their users.

Undoing the damage is hard. First, they have to update their signing certificate. Then they have to re-sign all the add-ons. Then _users have to reload all the addons._ Then, something users won't do - remove all the tracking cookies, etc. that slipped in while Firefox was broken.

[1] [https://github.com/mozilla/addons/issues/978](https://github.com/mozilla/addons/issues/978)

[2] [https://bugzilla.mozilla.org/show_bug.cgi?id=1548973](https://bugzilla.mozilla.org/show_bug.cgi?id=1548973)

<comment><USER19> > Then users have to reload all the addons

I'm pretty sure Mozilla will implement a fix in a way that users only have to update their browser, not do anything to all their addons.

<comment><USER88> I'm curious about how an update will be able to differentiate tracking cookies from legit ones?

</comment></comment></comment><comment><USER89> This is a shocking display of not just incompetence and bad practices but of brazen undisclosed covert control. Why should your local browser depend in this fragile way on some muckup in Mozilla HQ? And people line up to defend this?

Where does it explicitly say Mozilla can disable my addons remotely? When did I give them this power? And this from a so called 'open source privacy focused' browser. This is a mockery of privacy and open source and they shouldn't trade on this goodwill to gain users.

There can be no bigger security hole yet security fear mongers preach exactly this abusive model. This kind of centralized remote power is a far greater security threat that anything they keep on harping about, 'good intentions' and 'good faith' are not remotely something anyone should have to depend on. Why should Mozilla babysit my installation? Shouldn't they be using their resources to do something productive?

There is something rotten in SV culture and we urgently need to think of alternatives that are not infused in this 'know it all' abusive surveillance culture as even after such an egregious abuse of peoples trust and faith all you will get is hand waving, normalization, apologism and snarky entitled comments that trivialize people's concerns and choices made on the goodwill of open source.

</comment><comment><USER90> Sure, anyone one can make mistake. We have seen big companies make stupid mistakes too. But this is Mozilla we are talking about. How this slipped by is beyond me.

</comment><comment><USER91> DAMN! That's quite something!

For a piece of open source software you really have very little control with firefox. It really sucks that the alternatives are worse.

This, likely for almost all of their users, creates more of a security problem than signature checking actually solves. For me noscript no longer works which is (IMO) a critically important extension (between mozilla taking away the disable javascript button and spector.)

<comment><USER92> OTOH, it's opensource, and you can re-compile it in like 20-30 minutes with whatever changes you desire. So you have the control. You can probably even add some hacked up support for multiple signing certificates (and add yours there), if you tried.

It's just that it's more work than having what you want implmeneted and maintained by others.

</comment></comment><comment><USER93> Certificates have been in common use online for maybe two decades now, if not more. This is a common failure mode and it keeps happening. Is there some fix so we don't have to keep dealing with spontaneous failures due to expirations? Or are we doomed to suffer with this until the end of time?

<comment> gator-io Use a certificate monitor [https://letsmonitor.org](https://letsmonitor.org) (free)

<comment><USER94> Is that site legit? Why don't they even have a HTTPS redirect?

</comment></comment></comment><comment><USER95> They don't use cryptographic timestamps with their signatures ? The certificate might now be invalid, but the signatures were done at a time when it was valid...

<comment><USER96> The problem is that "time" is fungible and can be forged. The date on a signature doesn't really mean anything.

<comment><USER97> This is a very bizarre justification for an obvious bug. Code-signing does not work that way anywhere else — neither in Android, nor on iOS, Windows or any other common platform.

There is a possibility that Mozilla implemented their backwards code-signing model on purpose — for example, it allows them to oust unwanted extensions without explicitly recalling their certificates. But personally I think that they just didn't give the matter enough thought.

</comment></comment></comment><comment><USER98> What a spectacular failure... it's hours later, and still not working. The only thing firefox has going for itself stopped working. Already one family member and one friend switched to chrome.

<comment><USER99> Just switch for a day. You're making this out to be a much bigger issue than it has the capacity to be.

<comment><USER100> NoScript and uBlock Origin have stopped working. This opens up a LOT of attack surface for malicious hackers.

If this isn't a _critical_ security issue, what is?

<comment><USER101> The Container extension is no longer working. I'm logged out of mostly everything. There are few sites I don't want to open without container.

<comment><USER101> Update - Container data is lost post addons recovery (I installed Nightly build). This is ridiculous.

My Firefox usage will be so unproductive for few days. I had around 6-8 containers for different purpose, and somehow I'm habitual to using shortcut to launch a container and open whatever I'm supposed to (e.g I've access to 3 different AWS account, and I tend to press shortcut key to launch the relevant container tab)

</comment></comment></comment></comment></comment><comment><USER102> another month, another browser vendor that does something inconceivably bone- headed in "the service of users."

first it was deprecating ALSA for pulseaudio, then it was pocket, then tiles and their suggestions, then that weird video/voice chat thing, and then running "studies" as if my use of the browser was some tacit acceptance of my position as a guinea pig of the internet. Today every extension I use to make the internet even remotely usable is just...deactivated?

without my consent or knowledge?

enough. im switching to waterfox. Icecat is even worthwhile at this point. Anything that respects my freedom and intelligence as a user.

<comment><USER103> This appears to be a mistake.

It's clearly bad that things can break this way. But this happened because a certificate expired.

Mozilla didn't have to require signatures and certificates for extensions. They did so because they want to protect users.

Protecting users with signature schemes, increase complexity and, thus, the risk of debacles like this.

<comment><USER104> People take their privacy seriously. Our addons were disabled. I was browsing for a few minutes until I saw ads and didn't realize what was going on. Unacceptable that the default to an expiring signature is to disable them completely.

<comment><USER103> You would have said the opposite if an expired signature had caused your browser to be compromised!!!

The default behavior seems desired to me.

The problem was that the certificate was allowed to expire.

A have ton of respect for the fact that it was enforced!

</comment></comment></comment></comment><comment><USER32> So that's why I've been struggling for the last hour. It's crazy that it won't even let me I stall extensions from xpi file, even with extension signature checking disabled.

</comment><comment><USER105> Anyone know when they expect to fix this?

I basically can't (OK, won't) browse anything except HN until they do.

</comment><comment><USER106> So:

      * Started day with browser, no problems with extensions
      * At some point in the afternoon, all extensions disappeared
      * A couple of hours (cannot be more specific I am afraid), all extensions came back

However I'm noticing the saved preferences of some of the extensions has gone. e.g. Password manager has kept the username however all "Multi-account containers" are now reset to factory defaults.

Is anybody seeing similar behaviour ?

<comment><USER107> Yep, there's a bug open for that: [https://bugzilla.mozilla.org/show_bug.cgi?id=1549013](https://bugzilla.mozilla.org/show_bug.cgi?id=1549013)

</comment></comment><comment><USER108> Temporary fix without enabling feedback to Mozilla.

Open the address about:config in the Tor Browser address bar At the top of the page, search for xpinstall.signatures.required Set the xpinstall.signatures.requiredentry to false by double clicking it

Note: This workaround should only be used temporarily, as it disables a security feature. Please remember to set the xpinstall.signatures.requiredentry back to true again once the Tor Browser security update is applied.

</comment><comment><USER109> I'm surprised nobody's mentioned addon debugging if you really need an extension working. All of the extension packages are in <your profile>/extensions folder and you can load them for the duration of your session by turning addon debugging on in about:debugging and loading them in. This should be fixed fast enough that this fix will be good enough, just don't close your browser.

</comment><comment><USER110> I'm running the workaround suggested by Reddit use @MeaslyTwerp (the one after the HN solution): [https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_a...](https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_addons_got_disabled_and_they_are_all/)

</comment><comment><USER111> I've been using Firefox since 2002, always loyal because it works as I expect a browser to work. For users like me who see the benefits of Firefox being the only browser worth losing the benefits of native browsers (Safari/Edge), the only real place we'd move to due to sufficient issues is to those native browsers.

The new Chromium-based Edge[0] for someone like me, who would never use Chrome, is the only cross-platform alternative that could pull me away from Firefox. Native power usage advantages on Windows, with portability to other platforms and Chromium's speed advantages makes it a very attractive choice. I hate to see this happen to Firefox, but they don't have a lot of room for error. I've been using "ChrEdge" at work, and it's already a great browser as it is.

[0][https://www.microsoftedgeinsider.com/en- us/download/](https://www.microsoftedgeinsider.com/en-us/download/)

</comment><comment><USER112> Wow this is really bad, breaking millions of people's workflow in one go.

Fixed for now by switching to Firefox Nightly and disabling signing.

</comment><comment><USER113> My password manager and every other plugin disappeared this morning first on one of my Macs and a few hours later on the other. I could not log into anything so I switched immediately to Chrome because I don't have time to fuss with workarounds. I'll return to Firefox when I hear the problem is definitely fixed.

</comment><comment><USER114> Well, a little bit of empathy for mozilla here: I've seen a lot of IT departments that don't have any sort of great system in place for managing certs. A lot of places I worked, I always had a suspicion they were a ticking time bomb. It's not enough work that it's really anyone's full time job to manage them. Also, at larger companies, you might have divisions that do it in different ways without cohesion. And then a lot of times certificate expiration is so far in the future that the people that initially setup a certificate might have left the company and forgot to document it, etc. So that kind of thing can easily fall through the cracks.

Maybe a constructive thing I'm curious about: What is considered best practice for managing certs? How do people do this in a secure way that makes sure they get renewed in a timely way?

</comment><comment><USER115> "One or more installed addons cannot be verified and have been disabled" just showed up at the top of all my firefox tabs. #sadtrombone

[https://i.imgur.com/hUe7Nyn.png](https://i.imgur.com/hUe7Nyn.png)

</comment><comment><USER116> A positive side effect: my 2 years old FF installation stopped freezing constantly.

</comment><comment><USER117> Still working for me in my main profile in Firefox 60.6.1esr. I could even install an update to HTTPS Everywhere!

When I opened my alt profile, extensions still worked, but I can't install updates—including for HTTPS Everywhere.

What's up with that?

</comment><comment><USER118> I'm Interested in a proper root cause analysis and mitigation report.

</comment><comment><USER119> Still beats using chrome.

<comment><USER120> Have fun defending this to your friends who switched overnight to Chrome again.

</comment></comment><comment><USER121> I've had normandy disabled in my user.js for a while, but this was only after thoroughly perusing documentation and some firefox "hardening" projects. Point being, no end user should have to do what I did. Sane defaults, and transparency about things that should be opt-in, are sorely needed. Regardless, I still stand by Firefox, and am thankful there are chromium alternatives. The web is what it is, though I dream of a simpler one.

</comment><comment><USER122> I wonder how it's possible that these kinds of--I assume--easily preventable problems happen at huge corporations and projects.

I routinely renew certs for clients and it never happened that a website was down because of an expired cert.

I would think that Apple, Microsoft, Mozilla would be more efficient than me in avoiding these kinds of fuckups?

I also bet these companies make huge investments to their infrastructure so that their services have a near-100% uptime, and then they let certs expire.

</comment><comment><USER123> All my setup of 24 containers has been reset... Not only it forgot which domains were assigned to which containers, it also removed my manually created containers.

</comment><comment><USER124> What a nightmare! ---- ALL of my extensions have disappeared and have been disabled this weekend (with ZERO warning) ---- Trying to use my Roboform password program is a nightmare---- If the problem is not 100% solved by Monday (May 6), I plan to exterminate every trace of Firefox of every computer I own ---- Shame on Firefox for placing uses in such a nightmare situation with ZERO warning and ZERO solutions ----

</comment><comment><USER125> Advertisements. My eyes, my eyes!

</comment><comment><USER126> i've a strong emotional reaction to this. it feels like a failure on all of us. will be fighting toward a more optimistic interpretation of such a snafu.

</comment><comment><USER127> I never liked change that extensions had to be signed by mozilla.

Why do my personal extensions need to be hooked into some third party service that can go out at anytime?

<comment><USER128> The developer edition allows that just fine.

<comment><USER129> Developer edition is effectively aurora/alpha. It is buggy compared to release. And yes, I do mean that. If it were not it'd be 'release'.

Asking people to either give up control of their software (ie, walled garden release versions) or use buggy and insecure software Dev/Nightly/etc is not acceptable.

It's why I switched to a freedom respecting Firefox fork as soon as they announced walled garden extension signing in Firefox 37.

<comment><USER128> Sure: debranded versions ("freedom respecting") will also do that.

You say walled garden, I see what random WebExtensions people install on their work laptops and think "yeah maybe someone policing this thing isn't the worst thing". But most importantly: it sounds like it's not actually a problem for you?

</comment></comment></comment></comment><comment><USER130> This was probably an unintended consequence of trying to ban Dissenter. [0] [1]

0\. [https://dissenter.com/download](https://dissenter.com/download)

1\. [https://youtube.com/watch?v=f0Cc8RpqH1g](https://youtube.com/watch?v=f0Cc8RpqH1g)

</comment><comment><USER131> Not happy that Firefox didn't ask before it did this. I am getting really tired of my technology trying to tell me how to do things.

</comment><comment><USER132> Can someone explain exactly what went wrong? I don't think I quite understand, but 7/9 of my extensions have been disabled.

<comment><USER128> Firefox requires extensions installed via their "store" be signed with a certificate to make sure they're actually from there. That certificate has an expiry date. It expired, so now all of its signatures are invalid -- and Firefox no longer trusts the associated extensions.

<comment><USER86> A minor added detail is that it wasn't the leaf certificate that expired. I've heard that that would have been handled properly. It was an intermediate cert, and I guess that possibly wasn't fully taken into account? (This is 3rd hand knowledge and speculation, note.)

<comment><USER128> That is accurate :)

</comment></comment></comment></comment><comment><USER133> Amusingly, my surviving extensions are for WAVE, Print Preview and other accessibility tools. Thank you Mozilla, you really do care!

</comment><comment><USER134> This is absolutely going to tank Firefoxs market-share and reputation. I have a site full of tech-illiterate users and they're all uninstalling Firefox and searching for alternatives. None of them are interested in the workarounds presented, most of them even struggle to turn their computers on! Mozilla really dropped the ball here.

</comment><comment><USER135> That's a bad sign on what they do with security

</comment><comment><USER136> Here is a temporary workaround: [https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_a...](https://www.reddit.com/r/firefox/comments/bkcjoa/all_of_my_addons_got_disabled_and_they_are_all/emggvbx/)

</comment><comment><USER137> I was able to temporarily work around this problem by going to "about:debugging" and using the "Load Temporary Add-On" button to load up the disabled add-ons inside of ~/.mozilla/firefox/<my_profile>/extensions/

</comment><comment><USER138> 5 years since Heartbleed, yeah? Was this cert one that was pulled into sync due to being reissued?

</comment><comment><USER139> After learning about Normady thing, I am literally now blocking every Mozilla, Firefox and their affiliate domains and their Ip adresses. The trust is lost. I rely on Debian repo and like other's mentioned, Normandy seems to bypass Debian oversight.

</comment><comment><USER140> For anybody using Firefox Android and ublock origin who now sees ads, worth checking out the duckduckgo Android privacy browser. So far, it seems to be working beautifully so something good for me at least has come out of this mess!

</comment><comment><USER141> adblocker seemed stop working, no idea why, anyone know how to fix them or just wait?!

<comment><USER142> This helped me discover Firefox's Content Blocking setting, which is set to Standard by default, but now I set it to Strict. Works better than an ad block!

Preferences > Privacy and Security > Strict

</comment></comment><comment><USER32> It happened hours ago. Why in the blue blazes of hell isn't this fixed yet?

</comment><comment><USER143> Waterfox works in Windows, no issues with extensions and supposed to have a greater compatibility with legacy extensions.

Fennec works in Android, no issues with extensions and has the third party trackers removed.

</comment><comment> F-B-Aye This problem has screwed up Firefox 53 as well so unfortunately reverting to a superior version won't fix the problem either. At the moment Waterfox is looking like the best fix.

</comment><comment><USER144> What kind of idiot thought that the add-ons I have _personally_ installed on my browser need to have a capability to be remotely disabled despite literally nothing being changed.

This is absolutely inexcusable. I want to see everyone being responsible for this "verified add-ons" fiasco fired from the team (after they roll it back of course).

<comment><USER145> Exactly. If me/firefox has verified the signature (or approved the download) when downloading or updating the addon, that should be all that's necessary. Why does firefox have to check signatures constantly?

<comment><USER146> in case it was revoked? Seems fairly reasonable approach

<comment><USER147> Disabling with no option of user override is not reasonable. It only creates unnecessary dependency on third parties.

When product recalls happen, the manufacturer isn't taking your product away by force.

</comment></comment></comment></comment><comment><USER22> As I'm reading these comments, the Unverified addons warning popped up :) Well, I refuse to use the internet without uBlock Origin. Looks like this is going to be a productive day.

</comment><comment><USER148> This relates to my opinions about encrypted HTTP, which is that it shouldn't be mandatory.

If you have a well-designed system that only works with encryption, then sure, but this idea of using the same mistaken systems as the WWW clearly doesn't work well.

I've never seen a Tor Hidden Service fail because of something expiring.

Much of this nonsense about encrypting everything, without reason and excuse, is to protect advertisements from being modified.

That this hit Tor Browser and disabled NoScript is damning, but I already disable JavaScript in about:config and I'm not even using a version of Firefox this new, anyway.

I can't tell if my opinion of Mozilla is lower or if it can't get lower.

<comment><USER86> Requiring https is a different situation and much more defensible in my mind. It's way too easy to rewrite the web pages of everyone using library or coffeeshop wifi and thereby hack/phish a lot of people's browsers.

</comment></comment><comment><USER149> Does this happen when you open firefox? Forgetting what i need to reenable and what i had disabled is going to make me mad. If i just don't open ff will i be fine?

</comment><comment><USER150> Maybe everybody should set app.normandy.user_id to "anonymous". I am disappointed that Mozilla still has unique tracking ids embedded in Firefox.

</comment><comment><USER151> Firefox extensions installed using apt are not affected by this problem AFAICS, so to get a working ad blocker, apt install xul-ext-ublock-origin

</comment><comment><USER152> Shit, I just switched to lastpass and deleted all my saved passwords in firefox. Good timing. On windows, so the config hack is not working.

</comment><comment><USER103> Wow, that's a bad slip up... I hope users stick with Firefox.

I feel like this sort of thing is a risk when being vigilant about security.

</comment><comment><USER153> > Steps to reproduce: Wait until it's past midnight on 2019-05-04 UTC.

This has got to be one of my favourite bug reports ever.

</comment><comment><USER154> So let's say I'm the IT department in my company. I've already got my root cert on every employee's PC(including Firefox because they can't browse otherwise). Can I act like the Normandy endpoint and let's say remotely disable the ability to install any extension including those pesky VPN ones and also do a lot of other such things I would like, you get my drift? Am I right? Am I right?

Please tell me I'm wrong.

<comment><USER155> You are wondering if an IT admin can admin machines in its network? Yes, an IT admin can admin machines in it's network.

</comment></comment><comment><USER156> And now, all my plug-in / extension settings / preferences are gone after updating...

</comment><comment><USER157> Running Firefox 66 from Ubuntu repositories on Ubuntu 18.04 and all my extensions are enabled.

Does it only occur after a restart or?

<comment><USER117> I believe it happens the next time Firefox goes to check for addon updates. You may want to proactively set xpinstall.signatures.required=false which... I think might work for 66 on Linux? It worked for 60.

<comment><USER157> Ah, checking `app.update.lastUpdateTime.xpi-signature-verification` shows that it hasn't checked since "yesterday"

:/

      jtl@laptop-linux:~$ TZ=UTC date --date="@1556919381"
      Fri May  3 21:36:21 UTC 2019

</comment></comment></comment><comment><USER158> This almost turned me into raging monster. How dare you mozilla disabling adblock? now i want an UNSEE function :D

</comment><comment><USER159> I déinstalled most of my addons and/or extensions How can I return to my previous profile to get them back?

</comment><comment><USER160> The saddest thing today is to realize that I can't use Firefox without extensions on privacy/security.

</comment><comment><USER161> I'm curious to see what kind of impact on ad revenue we'll see in the next couple of days.

</comment><comment><USER162> Is it possible that this wasn't an expired certificate but someone accidentally changed the signing process?

The disabling happened right after the announcement by Mozilla to implement a new policy towards extensions.

Maybe someone didn't realize their mistake, so now everyone thinks it was an old certificate.

</comment><comment><USER163> Is Tor Browser also affected?

<comment><USER14> Looks like it. I had the same yellow "One or more installed add-ons cannot be verified and have been disabled" banner in the Tor browser as in Firefox, the NoScript extension icon is missing, and I went to two different "do I have Javascript enabled" sites and they both said JS is enabled.

</comment></comment><comment><USER164> So this just happened in my browser...when exactly will this be fixed?

</comment><comment><USER165> Ah, so I need not to restart Firefox while they're still enabled.

</comment><comment><USER166> I tell you what: Grammarly is making a mint off this right now.

</comment><comment><USER167> I guess there was no monitoring on cert expiry.

</comment><comment><USER168> Mozilla continues the war against its users.

</comment><comment><USER169> This is huge. This had a huge impact on my work in the last hours. Still not fixed for me. What's going on???

</comment><comment><USER170> its been so long since i browsed the web without adblock. Now I remember why

</comment><comment><USER171> Because of this my Firefox crashed and I lost all my open tabs. Just fking great...

</comment><comment><USER48> Will Mozilla IT finally be held accountable? Whether or not the direct cause of the expired cert is IT failure, the fact is that a healthy IT would have prevented this. Take a look at Mozilla IT leadership. Take a look at all the people who have left in the last few months. For a year people tried to bring attention to the IT leadership disaster, and every person who did that was penalized for it and left Mozilla.

A disaster like this has been brewing for a long time, and Moz leaders didn't listen to the canaries in the coalmine.

It's a very sad day for all the great people at Mozilla who work so hard only to see Mozilla IT let them down.

</comment><comment><USER172> when might the fix be?

</comment><comment><USER172> when is the fix?

</comment><comment><USER173> Well, it was fun whilst it lasted Firefox. Sorry, but this is a deal-breaker for me.

Hello Chromium again...

</comment><comment><USER174> If they can't get this right, what hope is there of browser security?

<comment><USER128> That's silly. Browser security is not one function. It's not even one team.

</comment></comment><comment><USER104> You've fucked up mozilla. I had no option other than to switch over to a browser that would let me install my ad blocker. The web sucks too bad for this. My addons are mine, not yours to disable and enable at will.

</comment><comment><USER64> First they force code signing on everyone without a way to disable it then they break it. This is an extreme level of incompetence I didn't expect from Mozilla.

They'd better have the best post mortum ever, possibly with someone being fired.

<comment><USER175> Oh relax. A cert expired. An intermediate cert at that...

This has probably happened to every major cloud provider and countless companies at least once. Certs are _hard_.

Should Mozilla have had monitoring on their cert expiration? Yes. Will they after this? Probably. Is any one person ever at fault for something like this? No.

Firefox is an open source project. You're welcome to contribute and make things better.

<comment> yjftsjthsd-h > Oh relax. A cert expired. An intermediate cert at that...

Everyone's extensions broke. Including security ones. Including the ones bundled into the TOR browser. And end-users can't fix it. Because Mozilla decided that it was too dangerous to let users choose what extensions to run for themselves. This is an excellent moment to be upset.

<comment><USER176> Being upset is ok! I'm not particularly happy that I can't just override the certificate check on stable. But demanding someone get fired is just pointlessly punitive.

</comment></comment></comment></comment><comment><USER105> So why is this taking so long to fix? From [https://github.com/mozilla/addons/issues/978](https://github.com/mozilla/addons/issues/978)

> diox commented 2 hours ago

> I'm locking this like I did in #851 because no new information is being > added. We're aware and we're working on it.

I mean, two hours? WTF.

<comment><USER88> It was Friday night on most parts of the Western hemisphere. I'm guessing it took some time to get the right people back to work and assess the situation.

<comment><USER105> Yes. I was upset. Sorry. I ought to know to stop when I'm upset.

<comment><USER88> Nah don't worry. We're all upset and a bit disappointed.

</comment></comment></comment></comment><comment><USER177> Is it perhaps a good time to remind folks that the same thing could happen to all your "secure" HTTPS websites that are completely unavailable via HTTP, where the only thing served over HTTP are the 301 Moved redirects, even for sites that don't collect any user information at all, and only serve static and public content, which really hardly benefit from the mandatory encryption?

Or is HTTPS / LetsEncrypt too big to fail? HTTPS still always a good choice? I see…

<comment><USER91> But it's easy to override broken https certificates. Worst case you have trust on first contact style security.

This is just plain bad.</comment></comment>