Verizon and a BGP Optimizer Knocked Large Parts of the Internet Offline - <AUTHOR> https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-knocked-large-parts-of-the-internet-offline-today/ <comment><USER2> Cloudflare managed to get an in-depth blog post, one which has incident details, points blame to other parties, and makes some really quite aggressive (for corporate blog posts) claims, all during an incident, and they did all that in _8 hours_.

I'm impressed. At most other similar size companies, this would take 4 days. And in something like Amazon, it would be 2 weeks of approvals, editing, and review before a watered down version with all specifics removed is published.

<comment><USER3> Regarding those really aggressive claims, I was a bit shocked by that as well.

Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them ... or Tom Strickx (who wrote the blog post) had his beauty rest interrupted early this morning to deal with Verizon's screw-up and was not having it.

<comment><USER4> >"Either Cloudflare has some pre-existing beef with Verizon and is using this as an opportune moment to dump on them"

Indeed. And that's not going to help them or their customer's in the least the next time they need Verizon's cooperation to resolve an issue. You would never see this type of behavior on the NANOG mailing list which has been on the front line of communications between ISPs and providers for BGP issues since the beginning of the commercial internet. It is very much a "community" with reciprocal respect and professionalism, things this blog post was devoid of.

<comment><USER5> [https://mailman.nanog.org/pipermail/nanog/2019-June/101614.h...](https://mailman.nanog.org/pipermail/nanog/2019-June/101614.html)

<comment><USER6> Weird response by the Verizon employee.

> _You guys have repeatedly accused them of being dumb without even speaking > to anyone yet from the sounds of it._

Not for lack of trying...

> _Should they have been easier to reach once an issue was detected? Probably. > They're certainly not the first vendor to have a slow response time though. > Seems like when an APAC carrier takes 18 hours to get back to us, we write > it off as the cost of doing business._

It wasn't a slow response, it was _no_ response. And either is unacceptable for a tier 1 carrier.

> _But this industry is one big ass glass house. What's that thing about > stones again?_

And other carriers are actively working to change that - including, in particular, CloudFlare.

</comment></comment></comment></comment></comment><comment><USER7> My favorite outage when I worked for a voip company was when one of our tech support people told a new customer that she needed to 'add our ip address to your router', meaning add it to the firewall whitelist, but she repeated that verbatim to the telco tech who misunderstood and then escalated her way up the chain at a major telco until some engineer with the wrong rights said 'fuck it' and updated bgp to route all of our traffic down her T-1 line.

That was a fun conference call, and listening to the lady on the phone I could see how the engineer got to that point.

</comment><comment><USER8> AS701 was the UUNET/Worldcom AS for the US and eventually the US/Canada network. Verizon bought Worldcom in 2006 and they became Verizon Business.

From the late 90s to early 2000s I worked for UUNET/Worldcom as an engineer in the network planning and design group. I worked in the international group but among other things we were responsible for the build out of AS701 into Canada, the exchange sites where AS701 connected to the various other international UUNET AS's and the PoP's where dedicated circuits for international customers who wished to connect directly to AS701 would be terminated. The point being that I am familiar with how AS701 was operated at that time.

UUNET's reputation at the time might not have been sterling due to the business decision to basically be a safe haven for spammers but from a technical standpoint the network was operated at a high standard. The basic BGP filtering referenced in the article was certainly in place at the time and if this had happened then heads would have rolled.

<comment><USER9> As they should, this is day-one BGP configuration stuff. Nothing advanced, nothing especially technical or time consuming. Speaks to a gross degree of professional negligence on the part of AS701. Really disappointing to see this degree of ignorance.

<comment><USER10> Networking infrastructure / tools are in the stone age compared to what we have in the software world. I am inclined to cut them some slack.

</comment></comment></comment><comment><USER11> Here's a shoutout to all the on-calls who woke up this morning to deal with "someone else's problem". I think everyone who woke up gets to, at least, order a "fancy coffee" and send the bill to Verizon.

<comment><USER12> Amen. We need a support group.

"Hi, I'm Teejmya, and I was on call last night"

<comment><USER13> Yes, apologies and thank you!

If you email me (matthewatcloudflaredotcom) your shirt size, preference for men's or women's cut, and your postal address with the subject line:

"Verizon BGP Leak On Call Support Group"

I'll send you a Cloudflare tshirt. Least we can do.

<comment><USER14> I think somebody should make some "I Fixed the Internet after Verizon Broke It. 20190624" T-shirts.

<comment><USER15> "Verizon broke the internet and all I got was this lousy T-shirt"

<comment><USER16> I would need a bunch of them

</comment></comment></comment></comment></comment></comment><comment><USER17> _> All of the above suggestions are nicely condensed into MANRS (Mutually Agreed Norms for Routing Security)_

Whoever came up with that name and acronym deserves an award.

<comment><USER18> Wow, thanks for pointing that out, I missed it on my once-over.

</comment></comment><comment><USER19> I appreciate how sympathetic Cloudflare is to the root-cause party because they answered the phone and undid what they shouldn't have done.

(If my understanding is correct, they shouldn't have told Verizon about the better routing, while Verizon should have known better)

<comment><USER20> I read this as Allegheny's fault, actually. DQE published to Allegheny (DQE's customer), who in turn re-published to Verizon (Allegheny's other provider). While most of the 'prevention' section talks about what Verizon didn't do, it doesn't seem to mention that Allegheny should not have re-published the DQE- published routes up to Verizon.

It's startling that Verizon doesn't appear to have any leak mitigations in place, but I feel like Allegheny is getting a pass here because they are small, or something.

<comment><AUTHOR> Allegheny is a steel company. I don't think most people expect them to have the same responsibility for internet health as Verizon, even though they are a $4B company.

(I'm from Pittsburgh, my grandfather and a bunch of my relatives worked for this company for decades. I've been kinda giggling about this intersection of my past and my present all day. I don't work in the parts of Cloudflare that deal with this kind of thing; I'm glad my co-workers were on top of it.)

<comment><USER19> I had to do a double take because they seem to be selling themselves as a technology company nowadays.

I guess "steel" in your name isn't good for share prices.

<comment><AUTHOR> Tech or healthcare, that's the way of things now.

</comment></comment></comment></comment></comment><comment><USER21> Thank you for the summary.

And, a sincere thank you for not mincing words when it comes to something as important as this.

> _However, against numerous best practices outlined below, Verizon's lack of > filtering turned this into a major incident that affected many Internet > services such as Amazon, Fastly, Linode and Cloudflare._

> _IRR filtering would not have increased Verizon 's costs or limited their > service in any way. Again, the only explanation we can conceive of why it > wasn't in place is sloppiness or laziness._

In an attempt to find any statement given by Verizon, I found that The Register was able to get this amazing statement:

"Verizon sent us the following baffling response to today's BGP cockup: "There was an intermittent disruption in internet service for some [Verizon] FiOS customers earlier this morning. Our engineers resolved the issue around 9am ET."" [1]

[1][https://www.theregister.co.uk/2019/06/24/verizon_bgp_misconf...](https://www.theregister.co.uk/2019/06/24/verizon_bgp_misconfiguration_cloudflare/)

<comment><USER22> Verizon's response seems very non-committal and it appears this type of incident may happen again if they don't take any action. Are there ways for companies like Google or Cloudflare to work around ISPs like Verizon without affecting ISP customers, or is this a blocker? Was the 10% of the re-routed traffic from Cloudflare 100% of the traffic from Verizon to Cloudflare?

<comment><USER13> It's worse than that. BGP provides the "map" of the Internet. That map is relayed from network to network. So, as a result, Verizon announcing a bad route can mess up the map not just for them but for any other network that connects to them (directly or indirectly).

We're actually fortunate at Cloudflare because of our scale and wide-spread interconnection. That limited the impact more than it would have for a smaller, less-connected network. The crazy thing about BGP is that any router can announce that it's responsible for a block of IP addresses and, if it's trusted enough, that's what the map of the Internet will reflect.

The long term solution is for networks to implement and enforce RPKI. AT&T, for instance, implemented RPKI and we did not see any drop in traffic to their network today.

Verizon not only didn't implement RPKI, which would be the best-of-breed approach, but also didn't do even basic route filtering. It's as if a trusted traffic cop (Verizon) overheard from a random passing motorist that the main road was closed and, as a result, directed all traffic off a pier and into the ocean.

More about RPKI if you're interested: [https://blog.cloudflare.com/rpki/](https://blog.cloudflare.com/rpki/)

<comment><USER23> The amount of posturing and blaming in Cloudflare's response is breathtakingly unprofessional. If the article was just a few sentences longer, you could have squeezed in a few more statements of blame. We know, they messed up. But Cloudflare isn't making itself look any better by rolling the bus over Verizon again and again.

<comment><USER24> I 100% agree with you, though I am unsurprised that HN is downvoting you. HN seems to revere Cloudflare bigtime despite the fact that Cloudflare often uses HN as their own corporate PR platform. I _absolutely loathe_ Verizon and I'll be the first to line up for a good publish lashing of US ISPs, but even I feel like this blog post is unnecessarily unprofessional.

What strikes me the most is that this whole "event" would have hardly even registered on anyone's radar (it affected less than 10% of their traffic during early hours of the morning. I saw one single news article about it, buried on The Verge, but other than that nothing), _except_ for the fact that Cloudflare's CTO was on HN this morning fanning the flames of the one thread about it. It's like they dug their own hole drawing attention to the "Cloudflare outtage" headline, and now they're overcompensating by going to drastic measures to blame someone else.

And now they keep harping on the fact that Verizon still hasn't responded? Sure, part of that is probably the fact that Verizon is a giant corporation that doesn't want to bother with this stuff, but the other part is that this "event" was hardly even big enough of a deal to register on VZ's PR team's radar, no matter how much CF whines about it.

This blog post (and the accompanying HN comments from Cloudflare execs) just scream "immature company" to me. There's a reason that Cloudflare is the one making this blog post and devoting CEO time to it while the established behemoth is just going about their business as usual.

<comment><USER25> The context (which isn't obvious, and I don't blame you for not knowing it) is that the Internet is held together by spit and duct tape. The only reason it works at all is that major participants are good actors, in the sense that:

1\. They implement basic precautions to prevent dumb things from going wrong.

2\. They're available 24/7, to immediately respond to and remediate whatever does go wrong.

3\. Both of the above are core obligations, which supersede any questions of public relations or maturity or higher-ups not wanting to be bothered.

If Verizon can't be trusted to properly operate their network, that's an immediate threat to the health of the Internet, and many people do need to be made aware of it. It's not just Cloudflare being salty because their customers yelled at them.

<comment><USER24> I know the context, but that's irrelevant here. Whatever the cause, a root cause analysis pointing back to CF is nice for CF to help solve the situation, and is even nice to have for us tech enthusiasts here on HN (though it should still maintain professionalism). But for customers and decision makers at companies that might be looking at considering purchasing Cloudflare, you know what I _don 't_ care about? Who's fault it was. There are multiple buckets of companies here:

1\. Cloud providers that were effected enough to apparently devote not insignificant CEO and CTO time to it (Cloudflare)

2\. Cloud providers that were affected but seemingly not enough for it to even register as anything more than a blip on their status tracker (Google, AWS, etc)

3\. Cloud providers that weren't effected

As a potential customer thinking about buying services from one of these companies, which one do you think I am doing to do with? It certainly won't be CF. And if I am already engaged with CF, I want to know _what CF is going to do to mitigate this situation in the future_ , and _no_ , pointing fingers like a child and saying "it wasn't our fault!" doesn't count.

Cloudflare can't really control Verizon's actions that lead to this situation, but they _can_ control how they respond to it and mitigate it. They had an opportunity to stand up as a leader and improve the internet (which is literally their company motto). As you pointed out, the internet working correctly is a matter of companies working together as good actors, and getting these companies to work together via good, strong relationships is a part of that.

Did Cloudflare do that? Nah. Instead, they made a petty blog post and their CEO is on Twitter telling Verizon they should be ashamed. I don't know exactly what his goal there was, but I assume it has something to do with hoping they'll be better in the future (if that's not his goal, then it really is just petty finger pointing). And if Cloudflare's CEO's method of getting people to improve their work is to _publicly shame them_ , I really feel bad for anyone who works under him.

<comment><USER25> To be frank, your post makes it clear that you don't know the context. CF simply cannot do anything on their own to mitigate the problem where Verizon constructs bad BGP routes to Cloudflare IPs and then advertises those routes to third parties. The only mitigation possible is to contact whoever's advertising the bad routes and get them to stop.

<comment><USER24> Have you read Cloudflare's multiple blog posts regarding BGP? Did you read the tweets from their directors talking about how other customers were unaffected by the event today because of the mitigations put in place? Did you even do the simplest Google about BGP protocols and the plans in place to prevent this from happening in the future?

If you're going to try to impose yourself as the gatekeeper of "knowing the context", you should probably know it yourself. Saying CF "simply cannot do anything" is narrow minded at best, and completely wrong otherwise. In fact, in _this very blog post_ linked in the OP, Cloudflare talks about taking steps to mitigate BGP issues in the future. That's great, if only it wasn't also paired with a childish finger pointing session.

<comment><USER26> AT&T customers were unaffected because of mitigations put in place by AT&T that _Verizon_ hasn't put in place. The steps you refer to in the blog post are ones that _Verzion_ has to take.

<comment><USER24> Yes, and? Cloudflare themselves are the ones pushing their own company as "leaders" in this field, and being a "leader" does not mean "pointing fingers and trying to avoid blame whenever something bad happens". If they fancy themselves leaders regarding BGP, as said on their website, then they need to actually act like leaders.

And as I've said multiple times now, Cloudflare _was_ in a great position here to stand themselves up as a strong leader on this topic to start working together with other companies (a la Verizon) to start to make real headway to fix the BGP problem. As other commenters have noted, the internet is entirely built on multiple organizations acting in good faith towards one another. Verizon failed to do that, and Cloudflare's response _also failed to do that_. I said it in another comment, but I'll also say it here: publicly berating the people that you are supposedly taking a leadership position over _is not good leadership_. This entire episode is going to do _nothing_ to encourage Verizon to work closely with CF to fix this issue. In fact, I imagine it will do the exact opposite.

Today was a display of incompetence from Verizon, and a display of bad leadership by Cloudflare. I have no idea why any objective-minded person would be applauding Cloudflare for this. As I mentioned elsewhere, I would normally love a good public bashing of Verizon, but not when it comes at the cost of professionalism and progress.

<comment><USER27> Companies respond just fine to public scrutiny, caused by them being rightfully and loudly blamed. The way you lead people isn't the same as the way you lead companies.

Verizon was acting _so_ badly that it's clear the pure friendly approach was doing absolutely nothing. And I'm sure Cloudflare is willing to give very real and pleasant engineering help if desired.

If Verizon doesn't want to talk to Cloudflare, that's fine too. This is not a problem that requires active cooperation. They just have to do their job.

<comment><USER24> >rightfully and loudly blamed

There is an enormous difference between assigning fault in a good faith attempt to find a root cause/solution, and casting unnecessary, unprofessional insults such as "Verizon's team should be ashamed of themselves". One is productive, and the other is just being a dick.

>The way you lead people isn't the same as the way you lead companies.

Yes, it certainly is. A company is an organization of _people_ , after all. You don't get to eschew professionalism and start throwing around insults just because a group of people has decided to attach an additional label over their heads.

And just to put an even finer point on it, Matthew Prince's tweets about the issue were not targeted at Verizon "the company". He specifically attacked Verizon's NOC and its team members. Despite everything, this isn't a faceless, soulless corporation that's having insults hurled at them. He specifically went after a specific group of _people_ and publicly shamed them. And then he has the gall to shame them even more for not immediately chomping at the bit to help someone who just aggressively insulted them.

Ask yourself: if Matthew Prince had sent a tweet berating team members from his own company, telling them they should be ashamed of themselves, and spent the rest of the day commenting on the internet insulting their competence, would you still be saying he is a good leader? Or even a good CEO? Of course not. It's Leadership 101 that insulting your team members isn't a good leadership style. And that doesn't change just because Prince isn't the one signing the Verizon team's paychecks.

> This is not a problem that requires active cooperation.

This is clearly not the opinion of those at Cloudflare that are loudly kicking their feet and whining that Verizon didn't devote enough resources to actively cooperate with Cloudflare's troubleshooting today.

<comment><USER27> > A company is an organization of people

Blaming a specific team can get too personal. Blaming an entire company is more about the decision-making structure, and is close to as impersonal as you can get. It's really not the same as blaming a person.

> This is clearly not the opinion of those at Cloudflare that are loudly > kicking their feet and whining that Verizon didn't devote enough resources > to actively cooperate with Cloudflare's troubleshooting today.

They didn't notice, acknowledge, or fix the problem. That's different from a lack of resources devoted to active cooperation. Heck, two messages of "on it" and "it's fixed" would be a pleasant level of "active cooperation", and that takes only a minute or two.

<comment><USER24> > Blaming a specific team can get too personal.

And yet blaming a specific team is exactly what they did.

>They didn't notice, acknowledge, or fix the problem. That's different from a lack of resources devoted to active cooperation. Heck, two messages of "on it" and "it's fixed" would be a pleasant level of "active cooperation", and that takes only a minute or two.

Sure, I'm not defending Verizon's inaction. My point is that regardless of the level of the cooperation, _some_ cooperation is clearly still required. And now because of Cloudflare's hostility towards Verizon after this incident, I wouldn't be surprised if Verizon is much less inclined to participate in _any_ cooperation. That not only seems counterproductive to Cloudflare's goal, it's also bad for all of us that use the internet.

<comment><USER27> > And yet blaming a specific team is exactly what they did.

In this specific case, just blaming "Verizon", it was not personal. (There are a variety of things that can be classified under "blaming a team" so I can't give it a blanket okay/not okay.)

Knowing it's the NOC team, as an amorphous blob of nameless people, is not getting too personal.

Just because something can be traced to a team doesn't mean that shaming the company is the same as shaming specific people from that team.

Going down that road would declare _everything_ as personal, and that's really not how things work.

> I wouldn't be surprised if Verizon is much less inclined to participate in > any cooperation.

The public pressure should be stronger than any pettiness, and if it's not then the solution is to let even more people know it was Verizon's fault.

<comment><USER24> >In this specific case, just blaming "Verizon", it was not personal.

That isn't what they did. They specifically called out teams, which according to what you just said, is too personal.

[https://twitter.com/eastdakota/status/1143182575680143361](https://twitter.com/eastdakota/status/1143182575680143361)

> The teams at @verizon and @noction should be incredibly embarrassed at their > failings this morning ... It's networking malpractice that the NOC at > @verizon has still not replied to messages

Not only does he specifically call out the NOC, he also calls out teams. It is very obvious which "the teams" he is referring to, and "the NOC" is indeed a specific team. In other comments he also calls out Verizon's support team.

This wasn't the case of "tracing it back to a team". CF's CEO specifically addressed them and told them to be ashamed of themselves. That's personal, and it's also being a dick to boot. Was there _anything_ in this situation that was gained by Prince calling these people out in these tweets? Would it not have been just as effective at calling out Verizon (while being less unprofessional and less personally malicious) if those tweets had been less vitriolic?

> The public pressure should be stronger than any pettiness, and if it's not > then the solution is to let even more people know it was Verizon's fault.

So the solution to pettiness is more pettiness? Why does CF have a license to be petty but VZ apparently does not?

<comment><USER27> > according to what you just said, is too personal

That is not what I said!

I said it _can_ be, and then I clarified with: There are a variety of things that can be classified under "blaming a team" so I can't give it a blanket okay/not okay.

I see the tweet. I call this case not personal. He's pointing the blame at large groups inside someone else's opaque company.

If you're pointing at a blob of 100+ people (like you said, support is also being blamed) then you're not making it personal.

> Was there anything in this situation that was gained by Prince calling these > people out in these tweets?

People know what company to blame (a good thing), but nobody outside that company even knows _how many teams_ , let alone specifics about the people on those teams (an acceptable thing). Overall positive.

> Would it not have been just as effective at calling out Verizon (while being > less unprofessional and less personally malicious) if those tweets had been > less vitriolic?

Being less vitriolic would not make it more or less personally targeted.

I'm not sure if the vitriol _helped_ exactly but I think Verizon did enough to deserve it that there's no need to berate Cloudflare for the vitriol itself.

> Why does CF have a license to be petty but VZ apparently does not?

Presuming I even agree with your definition of pettiness, the problem is not the pettiness itself, but the actions they take or don't take.

It's not terrible for VZ to be petty as long as they still fix their broken equipment.

<comment><USER24> >If you're pointing at a blob of 100+ people (like you said, support is also being blamed) then you're not making it personal.

Ahh, I see. So it's okay that he was offensive and insulting, because he was offensive and insulting to many people? It wouldn't have been okay if he was offensive and insulting to only a handful of people, but because it was more than that, it's okay? Is this some weird perversion of "one death is a tragedy, 1000 deaths is a statistic"?

He isn't pointing the blame at a large group inside an "opaque" company. He's insulting _people_. The people at Verizon will know full well that he is talking to them. People that work with the Verizon NOC will know full well that _those specific people_ are being insulted by this CEO. The fact that it was personally directed at multiple people doesn't make it any less personal, it just makes it personal to more people, no matter how much you move the goalposts.

> I'm not sure if the vitriol helped exactly but I think Verizon did enough to > deserve it that there's no need to berate Cloudflare for the vitriol itself.

So it didn't _help_ to berate Verizon, but it was still okay because they "deserved it"? And then you don't apply the same logic to Cloudflare themselves? There absolutely is a need to berate Cloudflare for their unnecessary use of vitriol, _especially_ if you're telling me the bar for berating someone is as low as "it didn't help but that's okay".

It's clear at this point that you're moving goalposts and adjusting your own principles in some weird attempt to defend Cloudflare. Cloudflare did nothing positive here, and your attempt to justify their vitriol and maliciousness is telling.

<comment><USER27> > Is this some weird perversion of "one death is a tragedy, 1000 deaths is a > statistic"?

Nah. If you deliver 100 insults to 100 people, that's terrible. But if you deliver one insult to a vague blob of 100 people, that barely registers. The amount of insult directed at any specific person is tiny. That's why I'm not bothered by it.

> it just makes it personal to more people

No.

> no matter how much you move the goalposts

Really?

Someone disagrees with you so they must be moving goalposts?

Do better than that. I've been consistent on what I consider personal.

Also, I think you're too focused on vitriol. You can single people out and cause them harm while using the nicest and most polite language in the world. The way you target and your underlying meaning is far more important than your choice of words.

> So it didn't help to berate Verizon, but it was still okay because they > "deserved it"? And then you don't apply the same logic to Cloudflare > themselves? There absolutely is a need to berate Cloudflare for their > unnecessary use of vitriol, especially if you're telling me the bar for > berating someone is as low as "it didn't help but that's okay".

Let's put it this way. I regard "impersonal beration" as one tenth the crime of "being obviously and extremely negligent with equipment that can break the internet". And I'm willing to forgive vitriol when it's deserved and impersonal.

You don't forgive that, and want to say Cloudflare acted somewhat badly? Okay, sure.

You want to claim they are failing as a leader, overcompensating with drastic childish measures to blame someone else for something they could and should have mitigated themselves? I completely disagree.

<comment><USER28> Please don't break the site guidelines. Also, please don't do these intense tit-for-tat arguments with another user. They don't help, they lower the signal/noise ratio, and they bore everyone else. I know it's hard (believe me I know how hard it is), but at some point someone needs to be the first to let go.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

<comment><USER27> That's fair. While I haven't had a _huge_ number of arguments like this, I can only name one or two that resolved successfully. I'll leave things earlier.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER29> > One of our network engineers made contact with DQE Communications quickly > and after a little delay they were able to put us in contact with someone > who could fix the problem. DQE worked with us on the phone to stop > advertising these "optimized" routes to Allegheny Technologies Inc. We're > grateful for their help. Once this was done, the Internet stabilized, and > things went back to normal.

It's funny how we have to still use _phone_ to help fixing some internet routing problem, even if _phone_ doesn't literally means the old black curly- wire equipment

<comment><USER30> Gotta have a channel that's out-of-band with the internet to fix problems with the internet.

<comment><USER31> Nowadays with voip "the phone" isn't as out of band as we'd like.

<comment><USER32> Is it time to put an HF ham radio rig in each major provider's office? I shudder thinking of a major outage where even phone communication can't take place.

I'm only half joking.

Edit: and maybe we just give Verizon a toy walkie talkie

<comment><USER33> You joke, but cascade failure ain't no laughing matter, either. Emergency plans aren't for fair weather, they are for sh!tstorms. I don't want to say "we are too reliant on the internet" because it's cliche and connectivity is just part of growing the modern world. But we sure as heck need several layers of backup plans in case things go sideways.

I, for one, hope there is a secret society of HAMs lurking as mild-mannered employees at every telco and ISP, ready to wire things back together when they short out.

<comment><USER34> There is an IRC server (several servers, actually) + channel that a variety of network operators are on that has existed since the very early 2000s for these sorts of events. 414 users on it now, most peoples nicknames include their ASN to make it easier to find each other.

Unfortunately, Verizon is one of those networks that isn't present there. But many other networks are represented there and it provides a direct path to those who have config/enable access on some of the largest networks out there. Cuts out the having to go via formal escalation paths and NOC groups that require a trouble ticket before you can engage them.

</comment></comment></comment></comment></comment></comment><comment><USER35> Long ago, in the mists of time when I was a wee lad, the internet was a simpler place. There were seven buttons around the world, all pressed down by volunteers. If any four of them were released, the world would end. "The world" was defined as "the internet", and at the time that meant "the world" was defined as "men with beards and suspenders and real opinions about Star Trek", and so that wasn't so bad.

Today in 2019, "the world" is defined as "you know, _the world_ ", and there are seven _million_ buttons being held down all over the world.

If any four of them are released, the world ends.

We have made mistakes, is what I'm saying.

(I once had call to explain to nontechnical people how and why the internet is the way it is and why my ops crews tend to be full of people who are a little too calm about things being constantly on fire. This was my best crack at it.)

<comment><USER36> Back in that oft-forgotten age I was privileged to know, work, and chill with the brave volunteers (and, subsequently, paid RIR staff) holding down the buttons. It's worth mentioning that even back then there dwelt in the west a large ugly troll whose name was AS701 (AS701 was not alone, either, having two hideous siblings, AS702 and AS703, that lived in other climes). Everyone tiptoed around the beast, because when angered it would flap its routes and there would be a great wailing and severe packet loss. The brave volunteers tried many times to tame the awful creature and I'm very sorry to see that it is still fucking everyone's announcements even today.

<comment><USER37> For the lazy:

AS701 Verizon Business/UUnet

AS702 Verizon Business/UUnet Europe

AS703 Verizon Business/UUnet ASPAC

</comment></comment></comment><comment><USER38> I remember the early days of the Internet, when I could log in to an ISP router running BGP, with a blazing fast T1 to an early tier 1 Internet provider. We could literally announce any route we wanted, no filtering. We used to regularly black hole spammers, then turn them back on an hour or two later.

<comment><USER39> Please write about your life / these times! Or link me to your blog/writing if you already have. <3

<comment><USER38> It's not terribly interesting. No blog, sorry.

</comment></comment></comment><comment><USER40> CloudFlare continues to raise the bar in terms of communicating technical issues to the public. Thank you for yet another enlightening writeup.

</comment><comment><USER14> It's clear that Verizon broke the Internet this morning through incompetent BGP management, and they could do it again. Who holds them accountable?

<comment><USER41> _glances over at Ajit Pai_

Nobody.

<comment><USER42> It doesn't need government intervention. It needs other companies to hold them accountable.

<comment><USER43> Absolutely. If you are a Verizon service provider customer, please call them and register your feelings on this matter. Make sure you let them know in no uncertain terms that you are considering switching providers based on their lack of following best practices.

<comment><USER44> And will do so, once another provider becomes available in your area.

</comment></comment></comment></comment></comment><comment><USER45> I hope we remember this thread the next time Shady Russians or Crafty Chinese make a BGP mistake and are accused of Active Measures.

<comment><USER46> This is an obvious mistake made by a steel company. The BGP "errors" that make tons of traffic go to Russia or China OFTEN WORK COMPLETELY. Some have worked so well no one noticed for months. That's far, far different than an error, that a large section of a country's traffic being successfully hijacked to another country. Accidents usually break things. What Russia and China are too successful to be accidents.

</comment></comment><comment><USER28> [https://news.ycombinator.com/item?id=20262214](https://news.ycombinator.com/item?id=20262214) is the earlier thread on this.

<comment><USER47> Thanks. And thanks for the help today updating the title on the original post as the real cause came to light!

</comment></comment><comment><USER48> > For example, our own IPv4 route 22.214.171.124/20 was turned into 188.8.131.52/21 > and 188.8.131.52/21\. [...] The prefixes Cloudflare announces are signed for a > maximum size of 20. RPKI then indicates any more-specific prefix should not > be accepted, no matter what the path is.

Did RPKI help reduce the scope of this incident, by stopping propagation of these faulty routes earlier than otherwise? Or did it have no effect in this case?

<comment><USER49> Anecdotal, but:

The article notes that AT&T has implemented RPKI, and a client mentioned to me that he wasn't having problems accessing Cloudflare-hosted infrastructure via his AT&T phone. The rest of his employees were having major issues though via the municipal fiber service provider.

<comment><USER13> Yup. There was no notable impact to AT&T traffic because they rejected the routes because they're filtering based on RPKI. Here's a Tweet from Cloudflare's Head of Network showing the AT&T vs. Verizon graphs: [https://twitter.com/Jerome_UZ/status/1143276134907305984](https://twitter.com/Jerome_UZ/status/1143276134907305984)

</comment></comment></comment><comment><USER50> One would think Cloudflare team would have a direct line of communication to all tier 1 Internet providers.

<comment><USER13> We thought we did. And tried both public and private lines of communication — without reply. Still waiting.

<comment><USER40> Hats off to Verizon for treating everyone equally without discrimination.

<comment><USER51> "We don't care. We don't have to. We're the phone company." \- Verizon, probably

It's good to know that Net Neutrality isn't dead.

</comment></comment></comment></comment><comment><USER52> I am surprised that CF is as aggressive toward Verizon in public as they are. Once you start breaking the Internet for stupid reasons, though, you probably deserve it.

I know very little about BGP operations; I did not know that there was PKI and route validation like they described in the article.

<comment><USER53> They'd probably be less aggressive if they'd been able to reach _anyone_ there or had received any response (none as of 8 hours after the incident). As noted above in this discussion the CF team _thought_ they had appropriate contact information for all top tier carriers, and I suspect they do have what Verizon would call the appropriate contact info. Not much they can do if Verizon ghosts them, though.

I guess they could take steps to null route everything to/from Verizon to see if they could get someone's attention that way.

</comment></comment><comment><USER54> Thank you Verizon for giving me a recent example to talk about while teaching BGP in Computer Networks class. My favorite example will still be Pakistan routing all YouTube traffic to itself while trying to restrict access to it. It is a little bit old though: [https://www.cnet.com/news/how-pakistan- knocked-youtube-offli...](https://www.cnet.com/news/how-pakistan-knocked- youtube-offline-and-how-to-make-sure-it-never-happens-again/) [http://web.mit.edu/6.02/www/s2012/handouts/youtube- pt.pdf](http://web.mit.edu/6.02/www/s2012/handouts/youtube-pt.pdf)

</comment><comment><USER55> Verizon's lucky it's a blog post that doesn't mince words, rather than a lawsuit.

<comment><USER56> Can they get a lawsuit?. Has Verizon broken their SLA?. Is there a manual to mitigate all the edge cases? What about being aware internally this had to be improved but it was delayed due bureaucracy.

<comment><USER57> >edge cases?

This is not an edge case, allowing downstream networks to broadcast routes for networks they do not own is a very well known security and operational issue with operating an ISP. Massive parts of the internet went down in the 90s to teach us this lesson.

Likewise, bureaucracy does not excuse not fixing an issue thats existed since the 90s, and not deploying any 1 of 3 mitigation tricks (let alone all 3).

Negligence causing damage from lost sales/traffic is sue-able.

The case would basically resolve around rather or not V had an obligation to prevent this from happening, and rather or not they were grossly negligent in that obligation.

In my view the answer is yes.

<comment><USER58> Who has standing though?

A Verizon customer might say "my internet was down" but there is 100% a clause in their contract about outages and SLAs.

Any company that lost sales likely doesn't have a contract with them, so what are they going to sue for? "Verizon didn't carry my 1s and 0s for free this morning"? Person A on the freeway having an accident and causing Person B to sit in traffic and miss their sales meeting isn't liable for that...

Maybe their peers (other telcos) have more standing because they couldn't deliver to their customers as a result but they of course all have a clause in their contracts about outages and SLAs that means ultimately they lost no money so there are no damages.

And this is why we need government regulation, either to break up the Telcos or nationalize them

</comment></comment></comment></comment><comment><USER59> Very interesting and easy to understand overview, thanks.

I'm not familiar with this side of networking, but it sounds to me the "BGP Optimiser" product was left largely to its own devices and automated a configuration change without any explicit approval from a human operator (I could be wrong)

With the protocol being prone to problems like leaky routes and sloppy peers accepting them, is it really wise to leave these BGP optimiser products running without some level of supervision?

EDIT: of course I guess the human operator might wave the change through too without fully appreciating the problem...

<comment><USER60> I think the reality of it is that these BGP optimizers really can't be human checked. There's just too much it is doing, and for them to be really beneficial they need to respond quickly to network path congestion. I would be surprised if overseeing such a system could be done with fewer than 6 full time people, as a WAG.

... Which is why you should be really sure that these optimized routes never leak! And on top of it, Verizon should never have accepted those announcements.

</comment></comment><comment><USER61> Nothing is ever new in IT. A similar incident happened 22 years ago. Buggy and/or misconfigured gear disaggregates and reannounces routes and creates a blackhole.

[https://en.wikipedia.org/wiki/AS_7007_incident](https://en.wikipedia.org/wiki/AS_7007_incident)

</comment><comment><USER62> > The RPKI framework that we implemented and deployed globally last year is > designed to prevent this type of leak. It enables filtering on origin > network and prefix size. The prefixes Cloudflare announces are signed for a > maximum size of 20. RPKI then indicates any more-specific prefix should not > be accepted, no matter what the path is.

Does RPKI prevent Cloudflare from announcing additional /22 routes during an incident like this? Any network with RPKI implemented would reject the /22s, but those who ignore it should pick them up over the leaked /21s.

<comment><USER13> We could break our prefixes into smaller routes, but 1) the Internet's routers have limited memory; 2) we have a lot of routes; and 3) we want to be good Internet citizens.

If every network announced all their routes as /24s — the smallest route generally accepted over the public Internet — the routing table would be a giant mess and would overwhelm many routers' ability to store them.

That said, after today we are thinking about ways that, in case of an emergency, we could break the routes down to be more specific than whatever is leaking. Given how broadly peered we are, Cloudflare's network will be as protected as anyone's. However, that's not really a good solution for the Internet generally. Better that we all implement and enforce RPKI.

<comment><USER16> Kudos for a CEO that understands in and outs of Internet routing, making me want to join CF's neteng team

</comment></comment></comment><comment><USER63> Is Verizon's engineer on vacation, or did he just call in sick today?

</comment><comment><USER64> Over a decade ago one of my friends was banned from the Sheffield Uni network for playing around with BGP and knocking the whole campus offline. One kind of has to wonder whether Verizon can suffer the same consequences simply by collective action on the part of other affected parties.

<comment><USER30> Nope - because said collective action would probably involve denying service to tens of millions of Verizon customers.

<comment><USER57> They could do better

<comment><USER58> Can they? How many choices of ISP do you have? I have 2, Spectrum (Time Warner) or some no-name for the same price with 1/10th the speed. I could not practically switch off Time Warner if I wanted to without moving. I imagine that Verizon customers are in the same boat.

</comment></comment></comment></comment><comment><USER65> Amusing callout to pager duty in the screenshot of the call-log :)

<comment><USER66> Which is more interesting, philosophically? The internet had a problem due to a single issue, or that the internet's problem was fixed due to a single person calling various people on a cell phone?

(device interactions versus human interactions..)

</comment></comment><comment><USER67> Every time I see the BGP abbreviation it's about a huge fuck-up. Either somebody hijacks routes intentionally or something like this happens.

<comment><USER40> As any other critical underlying infrastructure of our lives, it's taken for granted and ignored until it breaks.

</comment></comment><comment><USER68> I work from home in NJ and I knew something was screwy this morning. I wish there was a place I could have checked to see it was this issue. I rebooted pretty much everything in my house.

<comment><USER69> I'm in NJ and had issues this morning as well. Figured it was a local outage and promptly forgot about it after I left for work. I never expected it was something so serious. Crazy.

</comment></comment><comment><USER70> "Why is PagerDuty calling me before I've had my coffee?" _Call declined._

<comment><USER40> Well, you just lost yourself a T-shirt Sir. [https://news.ycombinator.com/item?id=20269076](https://news.ycombinator.com/item?id=20269076)

</comment></comment><comment><USER71> I don't fully understand all the networking protocols involved but who/what is responsible to manage this kind of failure in the network? How should this ideally be handled besides cloudflare engineers calling someone?

</comment><comment><USER72> [https://blog.codinghorror.com/coding-for-violent- psychopaths...](https://blog.codinghorror.com/coding-for-violent-psychopaths/)

</comment><comment><USER73> I don''t think Cloudflare is going to get any business from Verizon anytime soon.

This may be as a professional of a "Hey Verizon, you don't know what the fsck you are doing" as I've seen.

<comment><USER39> They're both big enough that they probably can't live without each other, which makes for an interesting relationship because they can probably swear at each other all day long and nothing will come of it.

</comment></comment><comment><USER74> What are the concerns around RPKI?

I'd like to know why some companies aren't implementing it if it solves similar problems. What kind of criticism does it receive?

</comment><comment><USER75> So how many phone calls do you think Allegheny Technologies or that ISP in Pennsylvania received this morning?

</comment><comment><USER76> I saw a few of my static sites that are hosted on Cloudflare and few other auxiliary 3rd party services I use flapping back and forth on PagerDuty, but not all of my Cloudflare sites triggered down.

Is this just because the unaffected Cloudflare sites were not within the CIDR range affected?

</comment><comment><USER77> >It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place.

I see you haven't had much experience with large Telcos. They are all like this.

</comment><comment><USER78> I still get a Cloudflare 1020 error here: [http://shadow.tech](http://shadow.tech) (my location : Scandinavia ) Are these sites waiting for some kind of propagation or cache busting? It's a pretty large gaming service.

<comment><USER79> Works from the ATL DC, what is the airport code that shows up on [https://cloudflare-test.judge.sh/#shadow.tech](https://cloudflare- test.judge.sh/#shadow.tech) ? Might be a local [maybe routing] issue with CF -> shadow's web server.

<comment><USER80> Not working here either (Finland), that page shows HEL for me.

shadow.tech shows "Error 1020 Ray ID: 4ec1c24b2a945b25 • 2019-06-24 21:22:45 UTC Access denied What happened? This website is using a security service to protect itself from online attacks."

<comment><USER79> Oh, 1020 access denied means some firewall rule (any combination of rules [0] and `if`/`or` statements) or access controls (such as blocking IP ranges and countries) blocked access to the website. These are always set up by the site operator, so this isn't caused by any CF issues or the earlier routing issues.

I guess best course of action for those who want to access the site is to tweet them [https://twitter.com/shadow_official](https://twitter.com/shadow_official) with your Ray ID.

0: [https://developers.cloudflare.com/firewall/cf-firewall- rules...](https://developers.cloudflare.com/firewall/cf-firewall-rules/fields- and-expressions/)

</comment></comment></comment></comment><comment><USER81> Would this have affected stuff in the UK? All sorts of sites like MealPal were inaccessible this morning for a bit

<comment><USER82> I wouldn't be surprised if various other transit providers slurped the bogus routes up from Verizon (without filtering), so it's certainly possible.

</comment></comment><comment><USER83> Has Verizon responded in any capacity yet?

</comment><comment><USER84> Can someone explain why the optimizer would split one route into two? Wouldn't it be more optimized to coalesce routes whenever possible?

<comment><USER85> If an ISP has multiple physical connections that it could use to reach Cloudflare's network, it makes sense to distribute the traffic that's addressed to different IPs across different links, instead of using a single route that sends all the traffic over one link and leaves the others idle.

<comment><USER84> Good point. Thanks.

</comment></comment></comment><comment><USER86> They automated a system that relies solely on trust? ...China hackers are surely having a great day.

</comment><comment><USER4> From the post:

>"It doesn't cost a provider like Verizon anything to have such limits in place. And there's no good reason, other than sloppiness or laziness, that they wouldn't have such limits in place."

Is "sloppiness or laziness" really the only possible attribution here? I'm not a big fan of Verizon but I'm a big fan of civility and empathy, two qualities which your blog post lacks. Outages are a really unfortunate fact of life. We've seen them recently with Google, AWS, Dyn - all companies where technical competency is generally not questioned. It's quite possible the cause of of this outage was some "perfect storm" scenario such as an eBGP router rebooted and came up with a stale or incorrect config. "Perfect storm" scenarios even happen at companies with very rigorous engineering cultures as we saw with the most recent Google outage.

Your attempt to shame an organization without knowing all the details reeks of immaturity and pettiness. Ditto for your willingness to turn this into yet another Cloudflare marketing opportunity. Have you forgotten about your own Cloudbleed incident? How would you feel if it a security company took that as an opportunity to shame you for "sloppiness or laziness"? Or some other company's CEO was offering to send people "Cloudbleed Support Group" T-Shirts on HN as your own CEO is doing in this thread?

Lastly RPKI isn't a silver bullet, RPKI authorities can also be misconfigured and attacked[1][2]. This happened with the LACNIC incident in 2013[2]. It's also worth mentioning that RPKI potentially creates new threats[2]. But again it seems more important to you to use this as a marketing opportunity and promote yourself while throwing someone else under a bus while uttering pithy summations.

Also from your post:

>"And, in particular, we're looking at you Verizon — and still waiting on your reply."

Although Verizon is the 400lb gorilla in the room, their NOC and network engineers are still regular people with kids and families and feelings. They are also people who have had a really shit day today. Why you can't extend just a bit of human compassion and feel compelled to try to shame is quite inexplicable.

You may think that your blog post was a marketing coup but I see it as a massive failure in in both leadership and civility.

As a thought exercise maybe Cloudflare leadership could think about how they would like the community to react the next time they are at fault.

[1] [https://www.cs.bu.edu/~goldbe/papers/hotRPKI.pdf](https://www.cs.bu.edu/~goldbe/papers/hotRPKI.pdf)

[2] [https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf](https://www.cs.bu.edu/~goldbe/papers/sigRPKI_full.pdf)

<comment><USER87> Cloudflare reached out multiple times in multiple ways to Verizon, to attempt to resolve the situation.

More than eight hours on, after utilising everything from what they were told was a Tier 1 support line to Twitter, they have nothing.

Even if we're kind to Verizon about the network failure, which was a global issue, they haven't done anything or said anything to suggest that Cloudflare should be treating them kindly in any way.

Not even a "we're aware, we're handling it".

Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons.

Verizon have shown a complete lack of leadership.

<comment><USER4> Have you ever worked for a Tier 1 ISP during a big outage? There is not enough personnel bandwidth in a NOC for everyone to get an individual response.

>"Ghosting one of the world's largest (as in utilised) companies is not wise for administrative, technical or PR reasons"

Oh the Cloudflare marketing machine. Largest by "utilized"? What does that even mean? Cloudflare is not a Tier 1, a Tier 2, or a major eyeball network. They are pretty far down in the pecking order despite what your marketing department wants us to believe. There's always some fuzzy stat isn't there?

Being too inundated to respond to everyone on the day of outage is a human resource problem, plain and simple The fact that you have taken this so personally is kind of embarrassing. What this blog post, the opportunistic marketing ploy and finger pointing have shown is a complete lack of maturity on your part. You want to call out Verizon for their behavior yet your own behavior is unnecessarily aggressive.

<comment><USER87> > The fact that you have taken this so personally is kind of embarrassing.

What? I have said nothing personal.

> What this blog post, the opportunistic marketing ploy and finger pointing > have shown is a complete lack of maturity on your part.

Ah. You seem to be confused. I am not affiliated with Cloudflare, and have not worked with Cloudflare at any point in time.</comment></comment></comment></comment>


Ask HN: Best novels about overpopulation? - <AUTHOR> Do you know (and really like and recommend) sci-fi and dystopic novels with overpopulation as main topic? <comment><USER2> stand on zanzibar, john brunner

the world inside, robert silverberg</comment>


Dropbox: What is Packrat? - <AUTHOR> https://www.dropbox.com/help/113/en

<comment><USER2> Dropbox could identify users who are more likely to need to resurrect a file (either because they frequently delete files, or because they frequently resurrect them), and offer them a discount on packrat.

After all, the man most likely to value hurricane insurance is the man whose house was just blown over by one.

<comment><USER3> Anyone who cares about their data should be storing it under real VCS, not a crappy Dropbox feature.

<comment><USER2> While I agree with you, have you ever tried explaining svn or git to somebody non-technical? It's not at all easy. Now try and explain to them how to commit and push files. Where does their repo live? How do they get at a file from another computer? How do they share it with somebody else?

There's value in a solution that makes copying your data to multiple systems drop dead easy and automatic, and Dropbox is in that niche. If you need more {robust|reliable|scalable|detailed} file history, it's not right for you. But you can bet there are tons of people in Dropbox who delete things a lot (just like I have met people who store "important" email in the trash folder)

</comment></comment></comment><comment><AUTHOR> I can't say I've ever noticed this feature before on Dropbox. I came across it while searching for the correct use of the word/term packrat (with or without hyphen). Nice use of the word on Dropbox's part.

</comment><comment><USER3> Dropbox is yet another NSA whipping boy. Couldn't care less what lame features they're adding.</comment>


Google Web Fonts Typographic Project - <AUTHOR> https://femmebot.github.io/google-type/ <comment><USER2> I find these pairings interesting, and thank the authors. Having knowledgeable and experienced people make suggestions is neat, really great.

FWIW, what would help me most (not that the authors are obliged to provide it; I'm not saying that) is if someone in that category would give some guidelines under what circumstances certain parings are reasonable. Kind of a white-wine- with-fish thing.

For instance, "sober, as with a business report" or "lighter, such as a human- interest story" would help those of us who are, sigh, completely without a clue.

<comment><USER3> This is not quite what you're asking for but [http://practicaltypography.com](http://practicaltypography.com) has lots of useful advice.

<comment><USER4> Did you try clicking on that link? There's some referrer detection. I think the author doesn't think much of HN.

<comment><USER5> Yeah, he has a whole rant about referral sites generating traffic but not interest. It's actually a good read, if a bit sour. However, I don't think that should color the opinion of the book, as it still is one of the best modern, freely available typography resources around (in my opinion).

</comment></comment></comment></comment><comment><USER6> Beautiful Web Type [0] is another beautiful showcase made using the fonts available on the Google Fonts directory.

Font Pair [1] and Font Pairing [2] are pretty awesome too.

[0]: [http://hellohappy.org/beautiful-web- type/](http://hellohappy.org/beautiful-web-type/)

[1]: [http://fontpair.co/](http://fontpair.co/)

[2]: [http://www.ourownthing.co.uk/fontpairing/](http://www.ourownthing.co.uk/fontpairing/)

</comment><comment><USER7> Wow, this is really beautiful. One thing I feel lets it down is that "text- align: justify" still isn't on par with print quality in browsers. It is getting better, but it's still a way behind pdflatex+microtype.

I'm sure there's work being done by talented people, but it's a hard problem, and maybe there's been compromise between speed and quality stuck?

<comment><USER8> See [https://bugzilla.mozilla.org/show_bug.cgi?id=630181](https://bugzilla.mozilla.org/show_bug.cgi?id=630181)

My understanding is that the main problem is incompatibilities between the CSS and TeX layout models. That's not to say that there isn't still plenty of room for improvement, however.

</comment></comment><comment><USER9> My problem with google fonts is not how to pair them but how differently they appear on different OS/browser combinations.

Something like Lato can look great but its relative weight on OSX Safari vs. Chome Windows is so different that I just don't think it can be used in production.

</comment><comment><USER10> Yes! I love typography and try to start the design of my personal projects with font choice and positioning. I'm a developer, though, so I can never match two fonts well. This site is fantastic.

</comment><comment><USER11> Great examples of font pairings—web fonts have come a long way! I like the parallel trend of web sites/frameworks like Medium, WordPress, Bootstrap, GitHub, etc moving to using native system fonts, though[0]. Make websites look good on all platforms and load quickly.

[0] [https://www.smashingmagazine.com/2015/11/using-system-ui- fon...](https://www.smashingmagazine.com/2015/11/using-system-ui-fonts- practical-guide/)

</comment><comment><USER12> I would love to find a list of which google fonts include ligatures

<comment><USER13> Be careful with ligatures in Web pages. They have downsides, which might not initially be apparent.

Ligatures are their own font characters. That is, an s-t font ligature is not recognized as an s and a t except visually. So, some things break:

1) search of text. "step" become a font-dependent codepoint followed by "tep"

2) accessibility for handicapped users. Readers generally cannot correctly translate ligatures.

3) browsers do not guarantee to handle your ligature correctly, especially older versions of browsers.

So, for the small increment in esthetics, you're creating a small pile of annoyances for readers and affecting the ability to find docs on your site via search engines.

For these reasons, you don't frequently find ligatures in Web pages.

<comment><USER14> The trick is to use a font that has ligatures, but not to use the ligature characters directly. Let the presentation choose the glyphs to display, and let the encoded characters remain separate as they should. CSS handles this with "font-variant-ligatures" (works in some browsers). I just did a test in Chrome on OS X; toggling between "font-variant-ligatures: common-ligatures" and "font-variant-ligatures: none" changed the presentation, but it did not negatively impact the accessibility, searchability, etc. as the ligature characters would. Automatic glyph substitution in presentation yay; encoded character substitution nay.

</comment></comment></comment><comment><USER15> Download all Google fonts: [https://github.com/google/fonts](https://github.com/google/fonts)

</comment><comment><USER16> In The Fox & the Grapes, if you try to move the fox over to the grapes they move away. Nice touch :)

</comment><comment><USER17> Are Google fonts free to use in publications/books? Does anyone know?

<comment><USER18> For any given font, the license is specified on the font's specimen page.

For example, near the bottom-right corner of [https://fonts.google.com/specimen/Montserrat](https://fonts.google.com/specimen/Montserrat) we can see that the license used is the SIL Open Font License. And from the SIL fonts FAQ[1]:

 _Please feel free to use SIL fonts in printed publications and PDF documents, including PDF documents posted on web pages. Such use is within the intended use of SIL fonts. No additional license or fee is required to use the fonts in a printed publication._

[1] [http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=...](http://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=FontFAQ_use)

</comment></comment><comment><USER19> Looks very nice, thanks a lot to the authors.

</comment><comment><USER20> Liked the colours

<comment><USER21> Agreed. Typography? Not so much...</comment></comment>


Python-powered "real-world" interface pyMCU giveaway - <AUTHOR> http://www.jcopro.net/2013/05/05/free-pymcu-giveaway/

<comment><USER2> This would be a whole lot cooler if the Python code was actually executed on the MCU itself. As it stands it needs to be tethered to a PC.

<comment><AUTHOR> I see what you're saying, but I really like it for various projects to control stuff with the computer. Like this one: [http://www.jcopro.net/2013/01/30/servo-light-graffiti- pixel-...](http://www.jcopro.net/2013/01/30/servo-light-graffiti-pixel- painting-an-image/)</comment></comment>


Ask HN: Has anyone started a real-money sport fantasy site as a side project? - <AUTHOR> I am considering running my own small site with real money competitions. We have a small unique niche. I am worried about legal problems. Any experiences to share ? <comment><AUTHOR> OP here. I have updated an email address in my profile. Love to get any feedback.

We are aiming at the US market. We will begin by excluding all states which require registration. We are very similar to existing offerings on the market. But we dont have any legal counsel yet.

</comment><comment><USER2> What jurisdiction are you in?

<comment><AUTHOR> I am in Europe. But we are aiming only at the US market right now.

We are very similar to existing daily draft leagues. But i wonder how far we can get before we lawyer up.

We will also exclude states which require registration at this stage</comment></comment>


The Stock Market Will Rally in December and Then Crash – Here's Why - <AUTHOR> http://www.thestreet.com/story/13374815/1/the-stock-market-will-rally-in-december-and-then-crash-here-s-why.html <comment><USER2> Where to park the money if you have any, zero interest does not look attractive and real estate is not cheap either, in addition to the inflations etc.

<comment><AUTHOR> For sure you can do better than zero interest rate. How about bonds, life insurance ?

On a riskier side a passive/long term investment strategy is in IMO a good way to go. Buy a few ETFs and forget about them.

</comment></comment><comment><USER3> Technical Analysis is nothing but hogwash. If you want to make predictions, do fundamental analysis.

</comment><comment><USER4> Okey, see you again next year.</comment>


Handling Overflow condition in calculating p-Norm - <AUTHOR> How can we handle overflow condition when we tr yo calculate p-norm.<p>https://rehanguha.github.io//articles/2019-03/overflow-p-norm <comment><USER2> Clicky: [https://rehanguha.github.io//articles/2019-03/overflow-p- nor...](https://rehanguha.github.io//articles/2019-03/overflow-p-norm)

Post with text instead of a URL have a penalty here, so it's more difficult to get to the front page. Try submitting it again. You can add a comment after submitting, but in this case I think it is not necessary.</comment>


Ask HN: I dropped out. How do I continue my CompSci education? - <AUTHOR> I am a second year CS undergrad who is taking an (indefinite) break from school to start a startup. It's a great opportunity, things are looking promising, and I have no regrets about it.
But I don't want to remain stagnant in my knowledge. What are some topics in CS that I can self-learn? What are some great OCW courses/textbooks/resources I can use to continue learning about advanced CS topics?
Here's what I have taken already: Discrete Math, Data Structures and Algorithms, Intro to Computer Architecture, Intro to Artificial Intelligence. <comment><USER2> "NAND to Tetris" will give you a good grounding in the basics of hardware design, the low-levels of operating systems, implementation of languages. Basically, you get a good overview and practical experience tinkering with the whole thing.

[http://www.reddit.com/r/programming/comments/7s383/from_nand...](http://www.reddit.com/r/programming/comments/7s383/from_nand_to_tetris_in_12_steps/)

I got the same thing from my CS education, but by accident. You can get it packaged neatly in one place from this course. This alone will put you in the top half of the programming community with regards to knowledge and overall understanding.

<comment><USER3> _This alone will put you in the top half of the programming community with regards to knowledge and overall understanding._

Really? Is the bottom half really that dumb? Or people just choose to ignore things which are not immediately of there need. I completed that book and though it was a great read, I wouldn't say I know much more than a person who has a CS degree.

<comment><USER2> _Or people just choose to ignore things which are not immediately of there need._

A sentence that answers its own question! (Left as exercise.)

 _I completed that book and though it was a great read, I wouldn't say I know much more than a person who has a CS degree._

The problem is that often the same material is covered in the process of getting a CS degree, but in the wrong order and in a time-dispersed manner. "NAND to Tetris" gives you an overview of the whole thing in a coherent way. And yes, there are people with a CS degree who would get a few "Ah Ha" moments out of doing the course. Also note that I said it would put you in the _the top half of the programming community_ and not the top half of those who hold CS degrees.

</comment></comment></comment><comment><USER4> I dropped out, too (it was to a programming job). This was quite some time ago (1982).

I tried to finish school later, but work always won. I never had the discipline to do both.

To keep up:

1\. Read CS textbooks and do small projects to make sure you understand the material at a practical level.

2\. Join the ACM and keep up with interesting publications from their electronic library. (The Usenix papers are free, but more limited in scope).

3\. Learn a new programming language, server technology or OS environment at least every year. Keep your level of exposure to new things high and don't get too comfortable in one thing. Don't be a zealot and don't get stuck.

4\. Read a LOT of code. (I read through the Unix sources, the source for TeX and a bunch of other largish packages just because they were there).

More recent things that I didn't have access to: Publically available course videos and material. My guess is that these are as good as the amount of time you put into them.

<comment><USER5> Regarding point 2, it might be worth checking out if any local university has a public library with journal access.

</comment></comment><comment><USER6> Finish school. Startups will wait. In the meantime, see me re an internship.

<comment><USER7> I disagree. If anything, it's school that can wait. Especially considering you can get a decently comprehensive CS education w/o paying mid-5 figures a year in tuition.

Rather than giving all that money away needlessly, spend a fraction of it on room and board and spend the majority of your time teaching yourself. You'll be done way ahead of 4 yrs.

And then jump into the startup world.

<comment><USER8> You don't go to college _just_ to study.

<comment><USER9> I can't agree with this more. I was a bit of an outcast at my school and I still had the time of my life. I played in a few bands, went to a lot of shows, worked at the college radio station, took classes in philosophy, writing and photography _and_ got a degree in Comp Sci. I would argue that the non-programming experiences are just as valuable to someone starting their own company.

</comment></comment></comment></comment><comment><USER10> I might make me here unpopular, but I would say: No, you did it wrong! Let me clarify on this: Yes, in CS you might not learn how to program well, this is true. However, when you study you get knowledge about several interesting computer science problems and how to solve them. You will see that problems later in your programming carrier however now you have a good background to solve them. Just a simple example: You given an array (let's say integeres). How would you remove all duplicates? A naive solution would be to check for every value, if there is more of this in the array. This has a running time complexity of O(n^2). The better way would be first to sort your array and then just iterate and remove all neighbours which are equal. This corresponds to O(nlogn) + O(n) complexity.

What I try to say, as a young "coder" you might not see all the troubles (in this case performance) you make when coding just straight away. You need to understand what you are actually doing, for this CS study gives you a good background.

The other really important point is: Startups fail with very high probability! CS diploma (or whatever this is in your country) gives you a solid background on which you can relay, when looking for a job position after your startup time. So, complete with your study is maybe a very classic way, however this is the most secure way for a long term.

<comment><USER11> You can do even better by inserting each element from the array into a hashtable and either discarding or doing a deep compare on collisions. Ignoring the probability of collisions, this is O(n) at the cost of extra space.

</comment></comment><comment><USER12> I know that some people will disagree with me violently, but you can self- learn pretty much any topic in CS as long as you are willing to put in the effort (any topic in any subject, really, the main factor is motivation). As someone who skipped straight to a masters, I know it is certainly doable to self educate.

MIT open courseware might be a good place to start, although I've never used it myself.

[http://ocw.mit.edu/courses/electrical-engineering-and- comput...](http://ocw.mit.edu/courses/electrical-engineering-and-computer- science/)

I would personally focus on topics that are either historically difficult, or pragmatic.

So stuff like compilers, computability and complexity, operating systems, database systems, cryptography, machine learning, web applications, linear programming, networking. Learning the basics of hardware is important too, although it is more electrical engineering and math theory.

<comment><USER13> Same for Stanford. <http://see.stanford.edu/see/courses.aspx>

Youtube lectures, download through torrents available on the site.

Some are good, and all have a lot of documents attached (all accept the book, often)

</comment></comment><comment><USER14> Doing the startup will be an education in and of itself. Just follow the lead of what needs doing. Need to scale? Learn about scaling? Need algorythms? Learn algorythms. Etc.

</comment><comment><USER15> Learn you some PLT. Specifically, study this:

<http://www.paultaylor.eu/stable/prot.pdf>

and this:

<http://folli.loria.fr/cds/1999/library/pdf/curry-howard.pdf>

and this:

<http://www.cis.upenn.edu/~bcpierce/tapl/>

very, very carefully.

</comment><comment><USER16> More algorithms and theory. In my experience the first algorithms course you take glosses over a lot of things and outright omits a lot of other (very cool!) things. I also enjoyed compilers and OSes, but I ended up doing high level stuff so I haven't had to care about either since school.

These topics aren't strictly computer science, but I highly recommend studying number theory, abstract algebra, and graph theory.

As far as resources go, I would use CLRS/OpenCourseWare/whatever you can find. I don't think the texts I used to study the above topics were exemplary, so anything you find on them will probably be as good.

Also, this place: [http://www.topcoder.com/tc?module=Static&d1=tutorials...](http://www.topcoder.com/tc?module=Static&d1=tutorials&d2=alg_index)

</comment><comment><USER17> Congratulations on your startup!

In my opinion, you could check out what kind of courses they typically offer in their curriculum at MIT/etc and check out their corresponding courses on OCW. Since you're doing this out of sheer fun, don't be bothered getting too caught up with the structure of it. If you see anything you like, just read up on it.

Usually the ones you've mentioned are a good stepping stone towards more advanced ones though you mentioned you don't want advanced ones. Here are some you can look into:

1\. user interface designs (helpful with any startup I think) 2\. software engineering (teaches you a little bit of structure in a development process) 3\. courses about testing, testing methods 4\. operating systems (not crucial but it's fun to read, kinda helps with ideas about distributed computing)

Good luck!

</comment><comment><USER18> Links to free ebooks:

<http://news.ycombinator.com/item?id=1327632>

Which programming book you just couldn't stop reading from start to end?:

<http://news.ycombinator.com/item?id=1944913>

</comment><comment><USER19> _> What are some topics in CS that I can self-learn?_

Any. Pick ones that interest you and start googling.

</comment><comment><USER20> I'd suggest you look into machine learning. It's probably going to be the single most important area of computing in the years to come. Start with the Stanford course online.

Also, if you haven't already, learn functional programming.

<comment><USER21> I hear this a lot: "learn functional programming", so I did a Haskell module and I still don't understand what all of the fuss is about.

Can you attempt to explain it to me?

<comment><USER22> I think PGs Common Lisp is a good book for functional programming. It really forces you to solve problems in new ways. While it is not difficult to do the problems in each chapter, doing them the "lisp" way will definitely challenge a C programmer.

</comment></comment></comment><comment><USER23> An operating systems course and a compiler course are what would complete your fundamentals. I have no idea how good the OCW courses are on these subjects (or if they even exist at all), but if they sound something like "build your own OS/compiler" then you're probably on the right track.

Bear in mind these two courses will contribute minimally (if at all) to your startup but its a nice to have a basic knowledge of how these two things work.

</comment><comment><USER24> Some familiarity with Numerical Analysis is, in my opinion, a must to a practicing Software Engineer. The official course book where I took the course was Dahlquist,G.L., Bjorck,A. Numerical Methods. I didn't use it too much so I can't comment on readability, but it does cover the material well.

</comment><comment><USER25> I would highly recommend iTunes University, its has many high quality recordings of lectures from many of the top universities. I've been programming for 10+ years and am self-taught, however I'm currently 'attending' the Programming Methodology course from Stanford University.

</comment><comment> tomh- Those seem atleast the core fundementals! Maybe pick up compiler design, functional programming and software engineering/architecture

</comment><comment><USER26> Go to your local community college and take some classes. Maybe not a full load, but a few.

</comment><comment><USER27> Congratulations, you made absolutely the right choice, even if the startup is a total failure.

As someone who has worked with many right-out-of-school CS graduates, I'd strongly suggest you pursue a lot of code writing. I'm sure you'll be doing this at the startup. But spend as much time as you can doing it, and learning from errors in production, and in review.

The real thing that CS programs lack is teaching people how to program. I mean that seriously. They focus on theory, but rarely on actual programming. If your startup does not consume absolutely all of the programming energy you have, then work on a cutting edge open source project on the side as well.

If you wish, there are classes in iTunes university, and you can surely peruse the catalog of the university you left, find the classes you would have taken, and buy the books, or comparable (cheaper) books on the same subjects and study them. I've done this in the past, but found that books targeted at practitioners in the field are generally more effective (and at the same cost or less) than those targeted as textbooks. (Back when I was younger, when I'd be going into a new area and wanted to research it, I'd wander into college bookstores looking for things like machine vision, etc.)

The industry oriented books won't have tests and problem sets for you to do... but you don't need them because you're pursuing this knowledge to help you complete some goal, right? For instance, maybe you want to build a robot that can compete in GPS Magellan and you need computer vision expertise for that? (That's what I did.)

When you are excited about a project, you will go and devour what you nee to learn, and you will learn it much more deeply and fundamentally than any class can teach you.

So, find projects and pursue them.</comment>


Google announces Android Studio: built off of IntelliJ IDEA - <AUTHOR> http://thenextweb.com/google/2013/05/15/google-announces-android-studio-an-ide-built-just-for-android-developers/

<comment><USER2> Curious: what happens to the Eclipse interface and ADT? Do those go off into the sunset or is Google planning to maintain 2 independent development toolkits?

</comment><comment><USER3> Finally.... IntelliJ FTW! :)

Best decision ever IMHO</comment>


Opscode wiki and ticket user database compromised - <AUTHOR> http://www.opscode.com/blog/2013/08/01/security-breach-user-information-for-tickets-opscode-com-and-wiki-opscode-com-compromised/

<comment><USER2> Just speculating again, but I bet this is the same security issue that I speculated took out the Apple Dev Center[1].

Given that the software compromised was JIRA and Confluence, my guess is they didn't heed Atlassian's notice to upgrade last week :(

[1] [https://news.ycombinator.com/item?id=6081428](https://news.ycombinator.com/item?id=6081428)

</comment><comment><USER3> I received this email.

I am quite surprised I haven't seen a single word of sorry/apologize/mistake in the email.

<comment><USER2> The posting says: "We're very sorry about this incident. We take security seriously and are addressing the incident as our top priority."

<comment><USER3> I am referring to the email I received.</comment></comment></comment>


I'm never going to retire - <AUTHOR> http://www.sideprojectprofit.com/youre-never-going-retire/

<comment><USER2> tl;dr ... because what I do for work is fun.</comment>