Tech's 'Frightful 5' Will Dominate Digital Life for Foreseeable Future - <AUTHOR> http://www.nytimes.com/2016/01/21/technology/techs-frightful-5-will-dominate-digital-life-for-foreseeable-future.html?ref=business <comment><USER2> The title says it all. "Tech's 'Frightful 5' Will Dominate Digital Life for Foreseeable Future." But since we're talking about technology, "foreseeable" translates to about two years. ;)

<comment><USER3> (Microsoft was dominant for 1-2 decades.)

I think it's more likely we'll see at least another 5-10 years of dominance of the "Frightful 5".

<comment><USER4> And IBM was dominant for at least 50 years ;-)

</comment></comment></comment><comment><USER5> I'm doubtful that FB should be on that pedestal. Social is all about the hip new thing du jour and I wonder whether FB can keep up with buying everything new in the long run since their revenue is still not great. Their shady tactics when trying to take over new areas like video show a certain amount of desperation IMO.

<comment><USER6> Facebook is the most agile of the companies. Two years ago it was a desktop web company, now it is all mobile. With Instagram it arguably made the best tech acquisition of this tech generation - compare that to what Yahoo spent money on. So don't see Facebook as a social app, see them as a powerhouse capable of making big moves and being able to respond to trends better than any other player. Since it has a hard time accommodating new functionality under its Facebook brand, it's perfected the art of managing multiple consumer brands.

<comment><USER7> And let's not forget they own Oculus now. Facebook could be in a much different place in the next ten years depending on how that plays out.

</comment></comment></comment><comment><USER8> " _In the mainframe industry's glory days of the 1960s, it was 'IBM and the Seven Dwarfs': Burroughs, Control Data, General Electric, Honeywell, NCR, RCA, and Univac. RCA and GE sold out early, and it was 'IBM and the Bunch' (Burroughs, Univac, NCR, Control Data, and Honeywell) for a while._ "

[http://www.catb.org/jargon/html/D/dinosaurs- mating.html](http://www.catb.org/jargon/html/D/dinosaurs-mating.html)

</comment><comment><USER9> Sad to see Farhad Manjoo playing the click bait game.

That article is so utterly lacking in content that I would expect it to be on buzzfeed. He might as well have said #8 will surprise you.

<comment><USER10> > _Sad to see Farhad Manjoo playing the click bait game._

It's an article in a mainstream newspaper for a mainstream audience.

> _That article is so utterly lacking in content that I would expect it to be > on buzzfeed._

The purpose is to explain how and why these "tech" companies are expanding into things that most people wouldn't associate with them, and the reason is the idea of the "platform". That's probably not familiar to the layman.

By the way, BuzzFeed does decent work.

</comment></comment><comment><USER11> Interesting they didn't include Netflix. Morgan Stanley talks about the FANG stocks, meaning FB, Apple, Netflix, and Google, and I usually take issue with _that_ for not including Amazon.

<comment><USER12> I don't consider them tech. The article even says "Netflix hosts their movies on Amazon".

<comment><USER13> Yes, Netflix has pivoted themselves as a content company, a la HBO.

Then again, the line for "tech company" is beginning to blur, since every company is so heavily reliant on technology.

<comment><USER14> Netflix is an interesting study in this respect, because they demonstrate both the immense potential if you control a dominant tech platform, and how at the same time relying on platform ownership can become your Achilles heel because you're ultimately just a middle man.

With the likes of HBO launching their own services to distribute their own original content, they have relatively little need for Netflix. Netflix, on the other hand, is nothing without good content, and in the face of declining third party content it has little choice but to move into creating its own to remain relevant. And then suddenly it's "just another" content producer, instead of "the" platform.

</comment></comment></comment></comment><comment><USER15> The problem, IMO, is that any new tech startup that may one day grow to be as these ones, get taken over (for exorbitant amounts) by these very parties once they're starting to gain some traction. In fact, that seems to be the business case for a lot of startups.

<comment><USER16> If you look at any massive company, in the history of business, they all grew and gained market dominance via acquisition.

<comment><USER17> But there is a difference between maintaining dominance through smart strategic investments (like Google acquiring DoubleClick, Android or Keyhole or Facebook acquiring Oculus) and defensive investments because you missed some boat in your segment of the market (Facebook acquiring Whatsapp or Instagram)

</comment></comment></comment><comment><USER18> Honestly, I'd consider five entrenched players with frequent ups and downs to be the signs of a fairly competitive industry. Indeed, each of the five seems to offer roughly comparable products to at least two of the other four players, suggesting that consumers are getting a pretty good deal out of all this.

<comment><USER3> It's a bunch of different industries though.

Social: owned by Facebook.

Search: owned by Google

Advertisement: owned by Google.

Mobile platforms: owned by Apple and Google. Actual competition here. No wonder things have been moving quickly.

Shopping: owned by Amazon.

Cloud computing: owned by Amazon. Google and Microsoft are trying to break in, so there's competition and progress.

<comment><USER19> Facebook is itself sustained by advertising, so how can Google own it?

<comment><USER3> Google's revenues are 4x larger than Facebook's. But sure, Facebook are probably getting there some time.

</comment></comment></comment></comment><comment><USER20> I find it curious, because only of of them produces something I use everyday - Microsoft. It's more like the "Ubiquitous One and the Take Or Leave 'Em Four."

And even Microsoft doesn't "dominate" my digital life. I think it's about the choices you make. You're choosing to be dominated. Maybe you get some benefits using those companies (Prime looks good sometimes), but you don't have to be involved with any of them.

That's not domination.

<comment><USER21> I'd be very surprised if you weren't using an AWS server many times a day every day.

<comment><USER20> That's a good point I hadn't thought of - and you're right. I probably do unintentionally, inadvertently, or, I'm not sure how to say it, but I don't mind, I'm just not conscious of it.

So still, it just doesn't seem like "domination" to me. Yeah, so some of the services I use use AWS. They probably could choose one of a bucket load of web service services out there. Sure seems like there is a lot of choice in that market.

<comment><USER22> AWS has competitors to be sure, but they are definitely dominant in that space.

</comment></comment></comment></comment><comment><USER23> errm - badiu? tencent?

Also the Foreseeable Future is quite a short time in this context.

<comment><USER24> Second that, Baidu(perhaps) and Tencent(certainly), and maybe Alibaba holdings as well. These companies are operating in a country where massive investments are being thrown around, and a product might go from 0 to 90% market penetration in half a year, mobile payments work, and massive fulfilment infrastructure. Wouldn't be surprised if they weren't already eyeing the American market, and suddenly decide to go on a massive acquisition spree. Only that many more C2C startups in China they can buy up or invest into.

</comment></comment><comment><USER16> So I am wondering if "this time it's different" when it comes to market entrenchment because these 5 companies have so much cash that they buy any potential competitor before they get out of hatchling phase.

I mean when you look at the range of capabilities, like the article states, they pretty much have a finger if not a whole hand in what any analyst predicts as future technology: Augmented Reality, Self Driving cars, Radical Life extension, AI, Advanced Logistics etc...

It's hard to see a future when they are unseated.

</comment><comment><USER25> "the giants are among the top 10 most valuable American companies of any kind."

Depends what you mean by "value". If current price Mr Market is willing to pay = "value" then yes. If value is a measure of long term return based on another set of measures like profitability etc then basing the future dominance of these companies on their current stock price seems a little speculative.

</comment><comment><USER26> Foreseeable future, unless .. black swan. And so on.

<comment><USER27> _Fore_ - _see_ -able, indeed. Void where unforeseen ;)

</comment></comment><comment><USER28> So the thesis of the article is that these 5 are permanently entrenched; that none will be run out of business (in the medium term I agree just because of size) and that no new ones will arise (really?).

The explanation is that these ones each control a network-effect-driven platform (and that network-effect-driven platforms arose only during a certain time period).

They list the network-effect-driven platforms: desktop (MS), web search (Goog), mobile OS (Goog, Application), online advertising (FB, Goog), cloud (Amzn, MS, Goog), shipping (Amzn), and "human relationships" (FB).

This thesis is absurd. By that theory, Google is fully dominant and Apple is about to fall out of the club. The idea that no new network-effect platforms can arise is unstated and unsupported.

I usually expect better work from the Times.

</comment><comment><USER29> I am not particularly frightened. I'm not even clear why I should be.

</comment><comment><USER30> Interested to see how this pans out in the next 3 yrs. I see Facebook changing

</comment><comment><USER31> Yet more clickbaity article titles with little to no real content.

</comment><comment><USER22> serious question, since I'm confused, why is the word "frightful" used here?

<comment><USER32> Given how many times the phrase "foreseeable future" was discussed in this comment thread, maybe they should rename them the "Foreseeable Five".

<comment><USER22> seriously. and what is the threshold of "foreseeable"? like 18 months, tops?

</comment></comment></comment><comment><USER33> Yes, probably until one of them invents strong AI.

</comment><comment><USER34> Is it speculation time again? Slow news day?

</comment><comment><USER35> Amazon over Microsoft?

</comment><comment><USER36> Who's losing? The users are.

<comment><USER37> How?

<comment><USER8> Monopolies and oligopolies are always bad for users. Not to mention that many of these companies are in the business of things which directly harms users: Advertisements, Cloud computing, Software-as-a-Service, etc.

<comment><USER37> Advertisements are controversial, but this is the first assertion I've heard that SaaS and Cloud computing directly harm users.

<comment><USER38> The harm comes from users who treat third-party servers as if it was equal safe and private as a server standing inside the company walls.

How many lawyers use SaaS or cloud computing and put information trusted by their clients in the hands of third-party (without ever informing their clients). How many hospitals has data flying around, in plain text, about people very sensitive situations that they entrusted to their doctors? How many political parties and government institution has databases being sent to third-party organization because they move their internal servers to the cloud and didn't consider the impact?

There are laws, and the laws governing how sensitive data did never get enforced before it became to late. Now we are in the situation that enforcing a strict control would paralyze governments and organizations all over the world, and as a result nothing is being done.

</comment></comment></comment></comment></comment><comment><USER39> When three more consumer technology giants exist (perhaps oriented towards the dreadful Internet of Things), can we start to call them The Hateful Eight?

</comment><comment><USER40> There's one issue that can severely damage two of them: Facebook and Google's business value come completely from taking away privacy. Apple, Microsoft, and Amazon also do so of course, but they can offer strong business models that doesn't should the need arise. They won't be _as_ profitable, but they won't die either.

At the moment Americans don't care about this issue, because they balance the lack of privacy against their fear of terrorism. But if they start to care things could turn around in a hurry for those two and we would see an interesting scramble.</comment>


When Academic Disagreement Becomes Harassment and Persecution - <AUTHOR> http://www.stanford.edu/~joboaler/ <comment><USER2> Bullying in the workplace is a big problem and it turns out that smart people are not exempt from participating.

Kenneth Westhues is a leading expert in the field, having studied the practice extensively, which he calls "mobbing". He has published several papers and books about it.

Some references on the topic:

<http://en.wikipedia.org/wiki/Bullying_in_academia>

<http://en.wikipedia.org/wiki/Kenneth_Westhues>

<http://arts.uwaterloo.ca/~kwesthue/mobbing.htm>

The game is often played until the selected victim quits, commits suicide, or in some rare cases, snaps and comes to work with a gun.

</comment><comment><USER3> Let's see what they're fighting about. From Boaler's complaint:

"[...] research evidence that could be used to improve the mathematics achievement of students in the United States and beyond [...]"

From her critics' paper (Bishop, Clopton, Milgram):

"This study makes extremely strong claims for discovery style instruction in mathematics, and consequently has the potential to affect instruction and curriculum throughout the country."

ftp://math.stanford.edu/pub/papers/milgram/combined-evaluations-version3.pdf

Both sides agree that a positive research result has the potential to alter mathematics education in the country. Then the question is: what's true, that Boaler has found a way to substantially improve education, or that her analysis is flawed?

<comment><USER4> Surely the correct response is more research and preparation of further peer reviewed papers?

</comment></comment><comment><USER5> Consider the time that has elapsed in this drama. I have found very few schools that, had they achieved the results claimed in Boaler's study, would not make it very public. How can you turn an urban school around, let alone, have it surpass other schools and not broadcast it to the world?

In science, things become fact by proving them again, again, again, and again. Not by doing one study, claiming stupendous success and that be the end of it, as seems the case in education research.

Without the Bishops, Milgrams, and thousands of frustrated parents, who would provide critical review of these studies? There are dozens of NSF studies claiming success, all with a different methods. I have hunted down most of them and none of them appear in the context of high performing mathematics classes. Like honors algebra or AP calculus where the students score 4's and 5's. It seems that in the education research community, especially in the NSF funded scam community, there is an unwritten rule that researchers do not criticize each other. Boaler seems to be pinning her hopes in this letter on that rule.

If you were Boaler and you possessed the secret to turning around the mathematics attainment of low performing urban schools, would you...

A. Continue to produce stellar results at even more schools? B. Spend ten years arguing with Bishop and Milgram?

Bob Hansen

</comment><comment><USER6> It appears to have been in the semi-public view for quite a long time. Was there ever any action taken?

Other related info:

<http://mathforum.org/kb/message.jspa?messageID=715422>

<http://www.nychold.com/rev-bishop-0302.html>

          In 2006 Milgram claimed that I had engaged in scientific misconduct.
          Milgram and Bishop attempted aggressively to identify my research subjects

That's unacceptable. If either of those are true they should be stripped of their titles and degrees.

<comment><USER2> <http://www.nychold.com/rev-bishop-0302.html>

Is this the actual bullying or is it something more than this?

This particular article by Bishop is not bullying, it's standard academic criticism and it's very reasonable.

If this is representative of the actual issues, then that seriously changes my opinion of what is going on here.

If this is the worst of it, I strongly recommend that everyone read this book review and decide for themselves if Bishop is a villain who is unfairly bullying another academic.

</comment></comment><comment><USER7> I'm not surprised by this academic misconduct. I've met many mathematicians who are hostile to teaching methods not of their own invention. I suspect this comes from a belief that superior mathematical ability implies superior empirical judgment, especially on matters of pedagogy; it also may stem from intellectual independence and unfamiliarity with or suspicion of empirical methods in the social sciences. This doesn't stop the opinionators. Sometimes they embarrass themselves in writing. One imbecile wrote in a Springer Undergraduate Text in Mathematics on linear algebra that one should be a fast note taker. The implication was that less than fast note takers would and should be left behind. Another esteemed author of a book on Fourier Analysis wrote that his book was not intended for the struggling student--a gratuitous and fatuous comment.

<comment><USER8> Mathematicians can definitely be opinionated, but a lot of the problem lies elsewhere. The "suspicion of empirical methods in the social science" is actually pretty well justified, for example the recent replication studies flap in psychology.

In education, I haven't seen any studies showing a marked superiority of a particular teaching method. Sometimes a new idea shows promise, but the effects evaporate when someone else tries it. So Boaler's results were enormously surprising, and it was reasonable to ask for verification. And then it turned into ugly politics.

<comment><USER9> You might be interested in skimming the following paper:

njrp.tamu.edu/2004/PDFs/Collier.pdf

This is a entry-level paper investigating dual-language instruction. These researchers are looking at student outcomes and how dual-language instruction improves those outcomes more than other approaches. It's also not _particularly_ advocating an instructional method that the researchers are claiming ownership of.

The abstract:

"Our longitudinal research findings from one-way and two-way dual language enrichment models of schooling demonstrate the substantial power of this program for enhancing student outcomes and fully closing the achievement gap in second language (L2). Effect sizes for dual language are very large compared to other programs for English learners (ELLs). Dual language schooling also can transform the experience of teachers, administrators, and parents into an inclusive and supportive school community for all. Our research findings of the past 18 years are summarized here, with focus on ELLs' outcomes in one-way and two-way, 50:50 and 90:10, dual language models, including heritage language programs for students of bilingual and bicultural ancestry who are more proficient in English than in their heritage language."

More germane to this discussion, in the paper there are links where the methodology, data collection, and data analysis are discussed in-depth. The authors also call for more specific and follow-up research on "program evaluation research, to refine what particular forms of dual language programs are most effective."

The TL;DR results: "Enrichment dual language schooling closes the academic achievement gap in L2 and in first language (L1) students initially below grade level, and for all categories of students participating in this program."

That's a pretty bold claim, that dual language instruction can bring below grade level student test performance up even for native language speakers.

If we can isolate pedagogies or aspects of dual language instruction that lead to better student outcomes, it would be nice if those approaches could be used in math education.

<comment><USER8> Thanks for the link. It seems like the bold claim is supported by a ton of data, as it should be. If this was what Boaler was offering, I think she would have a much stronger case. I skimmed some of the papers on Boaler's site as well, and they are almost completely unlike this one. They are full of anecdotes and quotes from students and very little data. To be fair, her study that is cited her later papers is not available on her web page so I have not read it.

</comment></comment></comment></comment><comment><USER10> I've been following the "math wars" for years, and used to be a regular participant in some of the email lists with messages quoted by other participants in these threads. I've read writings about mathematics education by Boaler and by Milgram, and have used those for preparing the mathematics classes I teach locally, and I have had a lot of email correspondence (mostly on email lists) with Wayne Bishop. I'm amazed that the root page of Boaler's faculty website at Stanford is being used for this kind of writing. I'm trying to get to the bottom of the actual facts involved. For that purpose, I'm sharing this same link among my Facebook friends, who include several innovators in mathematics education.

I appreciate the links that droithomme has shared in this thread. Yes, there is a lot of passion about what should be done to improve mathematics education in the United States, and one of the chief issues of contention is how to gauge the effectiveness of new programs. I'm pretty satisfied with the results of the homeschooling materials I chose for my oldest son (who is now a programmer for a startup). Over the years I've learned a lot from some of the same friends I'm asking for responses to this link. I hope we can puzzle out who is really coming forward with better evidence about what effective mathematics teaching is, because the United States could certainly use better mathematics instruction than it now has.

</comment><comment><USER11> No rebuttal of concerns in the "paper" except to say that it was improper to identify the schools?

No explanation of why she contacted the police about Wayne?

No explanation of why she apparently thinks hosting a not-peer-reviewed paper online is wrong?

No sympathy from me.

<comment><USER12> I bet she must be really dissapointed that you have failed to show concern for her in this. Your personal sympathy was probably the one thing she was really counting on. "Where's askimito?", she'll be asking. "I was planning on asking him/her/other to champion my education reforms!", she will then shout, while wondering if how she will possibly get through another day without this fountain of online approval.

</comment></comment><comment><USER13> Looks like someone else has a very similar complaint:

[http://mathforum.org/kb/thread.jspa?forumID=206&threadID...](http://mathforum.org/kb/thread.jspa?forumID=206&threadID=482509&messageID=1476271)

</comment><comment><USER14> _Milgram invited me to his office and told me not to talk about my research results in America as American teachers are "too weak" to be able to work in the ways shown to be effective._

If you believe there are harmful truths that should not be published, you don't belong in research. You belong in a PR department. I know exactly what this guy Milgram is getting at: in my experience as a student, progressive methods were occasionally wielded effectively by a few of my best teachers, but most teachers used them to take a break from engaging with the material and puff up the grades of the kids who were struggling. That's no excuse for wanting to hide research that shows innovative methods _did_ work under certain circumstances, especially since research documents _what those circumstances were._ Anyway, like I said, if you believe there are harmful truths that should not be published, you don't belong in academia. That would disqualify virtually everybody outside the hard sciences, of course, and probably many people inside the hard sciences as well. ("So what if dihydroxomayonnaise isn't an effective twinzoblane above bike pants temperature? If we get distracted by that the field will be set back five years! Dihydroxo-condiment studies _needs_ this grant money!")

It's hard to tell when bullying in academia is just a dramatic word for politics. People fight to entrench certain ideologies and stigmatize others, professors fight over control of departments, careers are threatened or ruined. I knew a girl in college who had her Master's thesis written and practically approved. Then her advisor suddenly left the department, and her advisor's vanquisher told her the paper was unacceptable _in toto_ and had to be rewritten from scratch. My friend had been pondering an academic career, but this made up her mind: she got her MA by examination and decided against graduate school (she got the MA at the same time as her BA.)

You can't overlook that one guy in this story had the campus police show up at his home because he said "nuke 'em" with respect to schools of education. This raises two questions. First, by this standard, how many people here have been careful enough with language to avoid police investigation? Second, can you imagine the pressure that must have been put on the police to force them to take time away from their real work because one faculty member wanted to stigmatize another as potentially violent? The other justification for co- opting the police into this dispute was "personal attacks on my work," which doesn't pass the smell test; if you don't distinguish between personal criticisms and criticisms on your work, then you _will_ feel under personal attack all the time. That's a choice. You can separate personal criticism from professional criticism and label the former inappropriate, but if you don't make an effort to distinguish the two, don't come begging for sympathy.

So here we have two groups of academics: each casting aspersions on the others' research; each trying to sabotage the others' careers; one group calling the other dishonest and the other group responding with accusations of bullying and illegal conduct. Both sides have accused the other of subverting academic integrity, by data fabrication on one side and by trying to identify research subjects on the other. I think both sides are doing a splendid job of bullying, no doubt glowing in the self-righteous pleasure of fighting against evil. I think I know which side I would agree with about the research (though it _might_ not be a coincidence that I tend to agree with the one side whose case has been presented to me) but I wouldn't say one side was better than the other just by virtue of being right about the academic point they're fighting over.

EDIT: It was bugging me to have only one side of the story, so I did a quick google search. It isn't much, but perhaps it's better than nothing: [http://mathforum.org/kb/servlet/JiveServlet/download/206-116...](http://mathforum.org/kb/servlet/JiveServlet/download/206-1167514-3866987-217284/att1.html)

I think the safest conclusion here is that we have two people engaged in your usual petty no-holds-barred academic playground fight. I'm prepared to believe that they're both harmless and both completely civilized in any other context. Getting the authorities involved still strikes me as creepy, but they were just campus cops, so I'm prepared to see even that as childish harassment rather than a serious threat.

<comment><USER15> Does anybody else find this part odd?

 _"This is an allegation that could have destroyed my career had it been substantiated."_

Surely the point here is that the allegations are troubling _because_ they could not be substantiated, not _despite_ that. Had they been substantiated, the authors career would indeed have be be destroyed, but the author would deserve no sympathy and Milgram no criticism.

This is a factual statement that does nothing to reinforce sympathies with the author, _unless_ perhaps you don't read it critically. It seems to me to be an emotional appeal.

<comment><USER14> Yeah, the allegations are flying fast and thick between these two, and nobody is impressed who doesn't have a political stake in the outcome.

</comment></comment></comment><comment><USER16> From what I can tell, Jo's work is based on how low socio-economic-status (poor black / latino) students have trouble in standardized exams when there are questions like:

"A cable crew had 120 feet of cable left on a 1000-foot spool after wiring 4 identical new homes. If the spool was full before the homes were wired, which equation could be used to find the length of cable (x) used in each home?".

<comment><USER12> I think it might be based off a fair amount more than that, she's been publishing since 1993.

<comment><USER17> Don't know if @wisty or @ktizo is right here, but _she's been publishing since 1993_ is not a logical argument.

<comment><USER12> Ok, she's been consistently publishing in academia since 1993 and was asked back to her job at Stanford after leaving to become Marie Curie Chair of Mathematics Education. If her research didn't cover a lot more bases than was suggested, then I don't think she would have held either of those academic positions.

<comment><USER17> Resort to authority. One of her adversaries is a retired Stanford prof. I'm not taking one side or the other here.

<comment><USER12> Well, the guy you are referencing has taken no position that we know of here, as far as what we were currently referencing, so I call shenanigans.</comment></comment></comment></comment></comment></comment>


ZooKeeper: Updating thousands of configuration files in under a second - <AUTHOR> http://code.zynga.com/2011/08/updating-thousands-of-configuration-files-in-under-a-second/

<comment><USER2> tl;dr: Zynga uses zookeeper.

I was extremely disappointed with this blog post. It reads like a ninth grade composition assignment, and is extremely lacking in details.

</comment><comment><USER3> Well I'd kind of like to know _how_ they extended ZooKeeper to do that.

<comment><USER4> Maybe just zk clients that listen to updates (watchers) on specific znodes? Business/validation logic can be on clients that can either be embedded in the app or write out to a config file consumed by another process.</comment></comment>


SecBSD: Unix-like operating system focused on computer security - <AUTHOR> https://www.secbsd.org/ <comment><USER2> IMHO stock OpenBSD is a better start. Concise and secure. Sending all traffic through a VPN is probably a bad idea, it is better IMO to put /some/ traffic through the VPN that you want separated. Private browse through VPN, preferably in a VM. The "I'm not a robot" thread ([https://news.ycombinator.com/item?id=19155643](https://news.ycombinator.com/item?id=19155643)) from a few days ago showed just how much can be gleaned by javascript, that's probably enough to work out if your private browser is on the same computer as the non-private.

<comment><USER3> I think this is more geared at pentesters and such, think bsd-backed kali?

I'm not so keen on the VPN by default thing though.. I hope they plan to upstream metasploit back to ports at least.

</comment></comment><comment><USER4> It says nothing about who the Dark Intelligence Team is, but this Google cache[0] has a _little bit_ more info:

>We are security enthusiasts from China, Germany, France, Netherlands, Norway, Switzerland, Mexico, India and Russia. Some of us have used *nix systems since 1999.

>Coding && DeCoding && Talks && Beers && Wine && Pizzas && Good Music && Stuff && Hacking.

So at least nine contributors, unless they're counting one person as being from multiple countries.

[0] [https://webcache.googleusercontent.com/search?q=cache:E2OiV-...](https://webcache.googleusercontent.com/search?q=cache:E2OiV-7ZhGIJ:https://secbsd.org/dark- intelligence-team.html)

<comment><USER5> Heh, good catch.

</comment></comment><comment><USER6> So pretty much Kali for OpenBSD? Also, why use a VPN by default when many of the tools are intended to be run on the local network?

<comment><USER7> I suspect the idea is that a VPN will provide encryption for all network traffic by default, preventing others on the local network from packet sniffing.

<comment><USER8> I use non-VPN IPSEC on my LAN. Works nicely

</comment></comment></comment><comment><USER9> Is security not an illusion with the current computer architectures? Sometimes it feels like we have to go back to architectures that separate code from data (Harvard) to really be secure.

<comment><USER10> A Harvard architecture, by itself, doesn't do anything to prevent code reuse attacks.

<comment><USER11> Indeed. The NX bit (which differentiates between executable and writable data) roughly approximates Harvard Architecture-like differences between types of memory, and likewise it cannot prevent code reuse attacks.

</comment></comment></comment><comment><USER12> "OpenVPN + ProtonVPN"? IMHO that is a red flag. Wireguard would be a better alternative.

<comment><USER13> From the WireGuard homepage:

> WireGuard is not yet complete. You should not rely on this code. It has not > undergone proper degrees of security auditing and the protocol is still > subject to change. We're working toward a stable 1.0 release, but that time > has not yet come.

I agree that WireGuard will be great when it's done. Is there something I'm missing?

<comment><USER14> Maybe he meant that ProtonVPN has a pretty shady reputation on HN due to their business connections to TesoNet, which is a data mining company. Its a lot more complicated than that, but if you see ProtonVPN being shot down on HN, that's why.

Further reading: [https://news.ycombinator.com/item?id=17258203](https://news.ycombinator.com/item?id=17258203) [https://news.ycombinator.com/item?id=17775326](https://news.ycombinator.com/item?id=17775326)

Personally, even if there isn't anything actually shady going on, I would want my VPN provider to be beyond reproach. Any smart VPN provider wouldn't want even tenuous connections to data mining companies. It feels a bit dirt to recommend Private Internet Access since they were the ones who pointed this out on HN, but so far AFAIK they are the only ones that have have been court- tested. Other options would be TorGuard or Mullvad VPN. Mullvad even already supports WireGuard!

<comment><USER13> I just run my own VPN from a $5/month DigitalOcean droplet... I feel like all the public VPNs are like big honeypots I'd rather stay away from.

> ProtonVPN has a pretty shady reputation on HN due to their business > connections to TesoNet, which is a data mining company

Does that mean ProtonMail also is no longer trustworthy?

<comment><USER14> > I just run my own VPN from a $5/month DigitalOcean droplet... I feel like > all the public VPNs are like big honeypots I'd rather stay away from.

I guess that depends on your threat vector. I mainly want copyright hounds and data miners (including my ISP) to stay out of my way. For this a public VPN is perfect. Hell, in a weird way, if PIA somehow turned out be a NSA honeypot they would be even better for that purpose since they'd essentially be untouchable by copyright holders. In general, I guess a personal VPN is more private on a micro level (no VPN provider that can spy on you) but less private on a macro level (any determined actor can trace your DO VPN back to you since you are the only user)

> Does that mean ProtonMail also is no longer trustworthy?

That is, again, for yourself to decide. Personally I think the Proton company isn't malicious and just really bungled up the launch of ProtonVPN by going at it together with / through TesoNet, and their VPN efforts will forever be tainted by that. But, that has very little to do with their mail branch, which preceded ProtonVPN and which so far seems a pretty good offering to me if you want your mail to be encrypted-at-rest.

</comment></comment></comment></comment></comment><comment><USER15> They said they want contributors, but I don't see any link for contributing. And their twitter account basically is only focus on raising money

<comment><USER4> The code isn't public yet. I imagine that if you want to get involved at this stage you'd need to contact them directly. Their email address and public-key are available here: [https://www.secbsd.org/dark-intelligence- team.html](https://www.secbsd.org/dark-intelligence-team.html)

<comment><USER16> _> The code isn't public yet_

Can't take it seriously from a security perspective, then.

</comment></comment></comment><comment><USER17> I am getting a _lot_ of mental "red flags" over this one. its a bunch of distros of things, some of which are pentest, some of which are less clear to me, it has confusing statements about VPN. Kali is understood. It has a sense of purpose, a community, quite strong public statements of intent and purpose.

This one. I mean sure, that lock-pick you bought from a guy in the pub, he _said_ it was only for testing padlocks, but now you see other people testing doors along the hall and you're wondering what you just walked into...

</comment><comment><USER18> Why can't they just be honest and call it an OpenBSD distro, rather than a Unix-like operating system.

</comment><comment><USER19> Just when I google about Tails alternative..

<comment><USER20> this is not a tails alternative. it's a workstation for pen-testing. like Kali

</comment></comment><comment><USER21> >not microkernel, multiserver

Huge TCB, they already fucked up. So much for security.

<comment><USER19> [https://www.allacronyms.com/TCB](https://www.allacronyms.com/TCB)</comment></comment>


The FAA Ordered This Beer Drone Delivery Service to Cease And Desist - <AUTHOR> http://www.smithsonianmag.com/smart-news/faa-ordered-beer-drone-delivery-service-cease-and-desist-180950122/?no-ist

<comment><USER2> This is such an obvious publicity stunt, I'm disappointed at SmithsonianMag for giving it further publicity. I have trouble believing the drone had sufficient battery power to carry a full case of beer farther than across the street in freezing cold weather, and the zip ties that were holding the box to the frame seem to have magically disengaged themselves on delivery too. I wouldn't be surprised if the Cease and Desist letter from the FAA is equally imaginary.</comment>


Systems Thinking in Practice (2016) - <AUTHOR> https://www.ryanfrantz.com/posts/systems-thinking-in-practice.html <comment><USER2> It's worth reading the original 1999 essay in which these 12 leverage points were introduced. It's a shame the article does not link to it.

[http://donellameadows.org/archives/leverage-points-places- to...](http://donellameadows.org/archives/leverage-points-places-to-intervene- in-a-system/)

</comment><comment><USER3> [https://ncase.me/loopy/](https://ncase.me/loopy/) is a great aid in systems thinking and was inspired by the Thinking In Systems book.

<comment><USER4> This is kinda awesome and deserves its own post. There are examples to get started.

<comment><USER5> There's something... not quite right about loopy. I think it's just that it has inverters but not diodes?

You can see for example if you try to form an ecosystem with a Sun node feeding one or more plant nodes that feed herbivore nodes that feed carnivore nodes, with Death nodes that kill off living things.

Eventually once the thing is running smoothly you can just turn off the sun and turn up the death and somehow those plants will still be thriving, apparently because the death of the herbivores is the same as sunshine to the plants.

This means that loopy seems to have real trouble with describing dynamic equilibrium states...

</comment></comment></comment><comment><USER6> strategic questions: engaging people's best thinking: [https://thesystemsthinker.com/strategic-questions- engaging-p...](https://thesystemsthinker.com/strategic-questions-engaging- peoples-best-thinking/)

Leverage Points: Places to Intervene in a System: [http://donellameadows.org/archives/leverage-points-places- to...](http://donellameadows.org/archives/leverage-points-places-to-intervene- in-a-system/)

Tools for Systems Thinkers: The 12 Recurring Systems Archetypes: [https://medium.com/disruptive-design/tools-for-systems- think...](https://medium.com/disruptive-design/tools-for-systems-thinkers- the-12-recurring-systems-archetypes-2e2c8ae8fc99)

Cynefin framework: [https://en.wikipedia.org/wiki/Cynefin_framework](https://en.wikipedia.org/wiki/Cynefin_framework)

</comment><comment><USER7> A game I love playing is Democracy 3[1], I believe it's an excellent example and exercise of simulating democratic systems. There are so many toggles you can tweak, dozens of policies to enact, setting whatever budget you want for each policies. Maybe around ~50 different situations (think GDP to pollution to gang wars) all interconnected with each other, and a bunch of voting groups - every person belongs to several groups (religious, rural, commuter, socialist etc...)

As a newbie, it quickly becomes evident how difficult it is to make an overall positive change in the society - because most tweaks that would make something better by X% will make something else worse by Y% - and since everything is connected to everything that leads to chain reactions. More often than not I end up getting assassinated by the second term. :D

[1] [http://positech.co.uk/democracy3/](http://positech.co.uk/democracy3/), [https://store.steampowered.com/app/245470/Democracy_3/](https://store.steampowered.com/app/245470/Democracy_3/) on steam

</comment><comment><USER8> Are there any related books that should be read along with Thinking in Systems?

<comment><USER9> Here is an extensive list i posted previously:

[https://news.ycombinator.com/item?id=19835208](https://news.ycombinator.com/item?id=19835208)

I have a much larger set of resources. If others would be keen i could structure and categorize it and post it.

<comment><USER10> Thanks! Interested / +1 for "structure and categorize and post".

</comment></comment></comment><comment><USER11> FYI: This is from 2016.</comment>


Sleep is awesome - <AUTHOR> http://inoveryourhead.net/images/sleep-is-awesome.png

<comment><USER2> i think it's a shame how many people think coffee will make up for <7 hours of sleep on a regular basis. i wish a chart like this was more publicized.

if i get less than seven hours, it always throws the rest of my day off. on the flipside, if i ever sleep in on a weekend more than nine hours, i'll feel lethargic and tired like i'm sleep deprived.

thanks for passing along, now i'll be doing the same :)</comment>


The un-celebrity president - <AUTHOR> https://www.washingtonpost.com/news/national/wp/2018/08/17/feature/the-un-celebrity-president-jimmy-carter-shuns-riches-lives-modestly-in-his-georgia-hometown/ <comment><USER2> Great article about a wonderful honorable man who didn't let Washington DC corrupt or change him.</comment>


Facebook while black: Users say talking about racism is censored as hate speech - <AUTHOR> https://www.usatoday.com/story/news/2019/04/24/facebook-while-black-zucked-users-say-they-get-blocked-racism-discussion/2859593002/ <comment><USER2> And so the left's campaign for censorship comes back to bite them. Who could have predicted that?

I guess this is a good thing in that the worse censorship gets, the more people will realize we need decentralized web services.

</comment><comment><USER3> Any statement of the form "[Racial identifier] men/women/people are [criticizing adjective]" is _per se_ racist.

If one wants to argue that wide-ranging systemic racism is worse than incidental one-off racist comments, I totally get that and I'm inclined to agree. But trying to redefine "racism" to exclude racism against a certain group is not acceptable.

 _1984_ was a cautionary tale, not a how-to guide. If a proposed method of political argument requires redefining the English language, that should give one pause.

<comment><USER4> Incredible. You honestly spun it so that the person who is the victim of racism is in fact to blame.

<comment><USER5> The person in the article said "white people are so fragile". That's a racist statement, so no spinning needed.</comment></comment></comment>


Ask HN: Do you use OmniOutliner? - <AUTHOR>

I'd love to know if you're using the new OmniOutliner for iPad, and if so, what do you find it useful for? I've never used it on any platform, and am curious what benefits it provides over, say, just writing notes in Simplenote. <comment><USER2> I've been using outliners since 1984 and rarely use anything else for notes, writing, etc. I love being able to collapse and expand my work, and to indent and move text. The OmniOutliner for iPad is only the first version, and I'm sure it will improve as time goes on.</comment>


Why and How Educators Use GitHub - <AUTHOR> http://alexeyza.com/blog/2015/09/10/embracing-participatory-culture-in-education/ <comment><USER2> When I was teaching at General Assembly, I pushed to us to use Github for _all_ of our assignments, workflow, etc. Homework turn-ins were pull requests, feedback was given via comments per-line that any instructor (or student) could make, etc.

The process to get students fluid at the basic flow took a few days, but after a while it was great and I couldn't imagine doing it any other way. Any programming course using this is obvious, but I'd even do it for non- programming things if I could get the student trained on it in a minimal amount of time.

The only roadblocks with git are that there's so many basic ways to screw up, which are hard to fix. Let's say a student does a `git init` in their `~/code` folder that holds all of their git projects... and then makes a bunch of commits, and then can't figure out why they can't push to github. I wish there were some better failsafes to prevent such things, but they happened by accident once a week at least.

<comment><AUTHOR> Thanks for sharing your experience!

In fact, the blog post doesn't include all the details, but some of the educators we interviewed were not from CS. You can see a table describing all the participants in our research paper (the link is provided at the end of the blog post).

Just like you, I believe it is completely possible to use this for non CS classes, and still gain all the benefits. However, like you said, there should be a basic understanding of Git.

<comment><USER2> One of the hardest things to overcome (even in a programming class) is the _why_ of doing things in Git. Many of the things we do seem arbitrary to students who are just learning. Why do you make multiple commits? Why use branches if you're the only one using it? Why make decent commit messages? I just want to turn in my work!

Many of git's capabilities seem esoteric to students, and for someone just starting indeed they are (students aren't managing a huge FOSS project, just their homework).

<comment><USER3> Maybe you don't need to do some of those things for assignments? They are all intended as aids in maintaining a long-running software project with many contributors, which is a very different use case than personal homework.

<comment><USER2> Oh very very true. They start becoming more useful for group projects.

We taught what what needed, but students (in learning to be self resourceful) will often google things, try tutorials, read on stack overflow, etc... and sometimes get themselves turned around. Git is a big piece of software to understand completely I guess, and even if you only show someone a small bit, they are likely to get confused at some point when they are trying to go deeper.

As an interesting and certainly unintentional thing, we discovered at one point that the words git uses to describe commands are often daunting to students. Commit and blame in particular. At one point we found many students weren't making enough commits, or sometimes none and were reluctant to turn in their homework. Through meeting in some 1:1's, we discovered that some of the students felt that if their work wasn't perfect, they didn't want to "commit" to it and that the word held too much power seemingly- they thought maybe there was something deeper to it.

This probably sounds silly, but when teaching people who are new to programming there are all sorts of preconceived notions that can really hold people back, and this was one of them.

<comment><USER3> That doesn't sound silly at all. In fact what sounds silly is taking techniques designed for large professional software projects and applying them to things that aren't anything of the sort.

It reminds me of the problem with teaching Java as a first language – you _must_ start with `public class Foo` in a file called `Foo.java`, containing a method declared as `public static void main(String[] args)`, which means you _must_ either explain tons of concepts (what "class" means, what "public" means, what "static" means, etc.) long before you can motivate any of them, which makes everything seem complicated and hard, or hand-wave them away as incantations that you'll learn about in due time, which makes programming seem magical and mysterious.

None of which is to say that Java isn't a great language for doing professional work, just that it isn't great to start with. Git seems quite similar to me.

On the other hand, I get that it is valuable to teach techniques that will actually be useful to students in their professional life, even if they are clumsy in a non-professional setting. There's a real tension here.

Apropos of nothing else I wrote: I've always enjoyed seeing people alias `blame` as `praise` or something similarly positive (although `annotate` is the built-in neutral option, which is also pretty good).

<comment><USER2> You're 100% right across the board. The tension between 'real job skills' and 'things that are good for the sake of learning' are very difficult to balance in a program where an overwhelming majority of students are looking for jobs immediately after the program. This was something I thought about every single day.

<comment><USER3> Yeah, it makes me feel like a curmudgeon, or like a job-protectionist, but I think it's a shame that so much of the "learn to code" movement is focused on training for professional software jobs, rather than on learning how programming works so that it can be used as a base tool, more akin to writing or arithmetic. Teaching all these professional techniques up front seems to me like training people for statistics or accounting jobs before they've taken middle school math.

<comment><USER2> That focus comes largely from the students. They want/need a job asap. Unlike a college freshman who has 4+ years to learn, and has set aside that space in their lives, it isn't uncommon for a student leaving these programs to be down to their last 30-60 days of savings after this program and need a job immediately.

Anything you teach them that is simply theoretical, you have to fight them to get them interested in. This isn't to say I didn't do it. I taught students basic C, binary conversion, arduino, basic electronics, game programming, noSQL databases, computer science, etc... in various classes. But there was always a group of students who wanted nothing outside of what they would need on the job, immediately.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER4> This is a timely post, Alexey.

Today, we released Classroom for GitHub: [https://classroom.github.com](https://classroom.github.com)

Classroom for GitHub automates repository creation and access control, making it easy for teachers to distribute starter code and collect assignments on GitHub.

You can read more about it on our blog: [https://github.com/blog/2055-teachers-manage-your-courses- wi...](https://github.com/blog/2055-teachers-manage-your-courses-with- classroom-for-github)

<comment><AUTHOR> This sounds great and I'm sure it would resolve some of the barriers to entry for educators.

</comment></comment><comment><USER5> I teach a college UNIX systems programming class. This semester I decided to switch to GitHub, and so far, I can't believe I didn't do it earlier.

I like to write code in class, and it's great to be able to push every couple of minutes so students can follow along on their laptops if they want. Of course that's more about git itself, but the GitHub interface is good and I think that it lowers the frictional cost for both me and the students, so everybody gets a little more done.

<comment><AUTHOR> I used it to teach my class as well. But I have to be honest, it wasn't obvious at first how to use it - especially how would I manage the course website there (which needed to accommodate different semesters). Eventually, I figured out that I don't really need an HTML website, but using Markdown files would get the job done (and would be better, as it makes it easy for students to commit PR with corrections and suggestions to the course website and material). I really love our setup now, but it took some figuring out the best way to do it. BTW, if anyone is interested here the course website (just as our example):

[http://startup-programming.com](http://startup-programming.com)

<comment><USER5> I am definitely feeling that: I've already had to do some reorganizing as I discover that some of my initial decisions weren't perfect, as to organizing the class repo.

By the way, your article has encouraged me to get the students using issues for eg. homework questions they'd normally email, thanks for that.

<comment><AUTHOR> That's great!! I'm happy it helped.

Can I ask what initial decisions you had, or what did you reorganize with time?

<comment><USER5> Sure. Basically, we go through the first half or so of Stevens' Advanced Programming in the UNIX Environment fairly sequentially. My first take was to group the example code by topic, but I've realized that it might make more sense to group it by book chapter. So that's my current reorganization plan. The thought is, with lots of topics, it will help orient people better if they have an explicit cue for what chapter to review if they are confused.

That's pretty specific to the more or less sequential approach with the book, though. It's a good book, so it can support that kind of thing.

I think it will also help with presenting the supplemental material that is linked with each chapter/section. Again, that might be specific since in the Stevens book, there is always a manpage or Single UNIX Specification reference to give for almost any section.

</comment></comment></comment></comment></comment><comment><USER6> Perfect timing for this post. Github is realeasing Classroom for GitHub. Excelent post by the way!. Thanks for sharing your experience!.</comment>


Microsoft: Apple has it wrong - <AUTHOR> http://mobile.theverge.com/2012/7/11/3151472/microsoft-kevin-turner-apple-wrong-pc-plus-era <comment><USER2> Who else agrees with Tim Cook with regard to Windows 8 merging PC and tablet ...

"You can converge a toaster and a refrigerator, but those aren't going to be pleasing to the user."

</comment><comment><USER3> Above the headline is an ad for the Samsung Galaxy S III which speaks volumes about where we are: Microsoft should have created the Android market, and it may be too late for them to catch up. Can you imagine DEC declaring in the early 80s that we were in the Mainframe+ era?</comment>


Thanks Jamie, Sumon, and Tim  - <AUTHOR>

I appreciate you guys opening your doors last night for an entrepreneurial get together.  I had a lot of fun and look forward to watching everyone's ventures.
<comment><USER2> Hey Dave, great to meet you too. While we're in the san francisco area we hope to do another one of these meetups. Just leave a message if this would be of interest and any feedback you had on making it better.

Everybody is welcome for the next one which we will put on soon :) shoot me an email.

<comment><USER3> Mike and I enjoyed it alot too - especially seeing the infamous Yscraper for the first time and meeting the Weebly's in person. If the group gets any bigger, we could also host it at our office at 2nd/Folsom (right across the street from cnet.com). We could probably hold up to 50 or so. (There is another conference room half the size of our main work space in this picture <http://streamfocus.wordpress.com/about/> which only has a conference table in it) Maybe we could have a two pronged approach to this - a purely informal side like the get together at your place which was great, and a "demo day" where people could demo their apps.

At the techcrunch party, we got a lot of value out of pitching what we were doing to people (shaking Guy Kawasaki's hand was a bit of fun too). At first we did a bad job, but with practice, we got better. What would be interesting about a "demo-day" version is you would get practice pitching plus some excellent geek feedback on the product. The practice could be a really big deal for those times when demoing is serious business. Just a thought :)</comment></comment>


Show HN: Display images in the terminal - <AUTHOR> https://github.com/ichinaski/pxl <comment><USER2> I have a comparison of these tools going here: [http://www.xkyle.com/a-comparison-of-image-to-ascii- conversi...](http://www.xkyle.com/a-comparison-of-image-to-ascii-conversion- tools/)

I can add this to the list, but it looks like it does not take advantage of the half-block trick to double the effective vertical resolution. (like other tools like img2xterm)

<comment><USER3> If you need monochromatic images (ie for logos or charts) a pretty cool trick is using braille. With a little bit of hacking, the xkcd example of this repo [https://github.com/asciimoo/drawille/](https://github.com/asciimoo/drawille/) works pretty well :)

I'll be using this for my email signature.</comment></comment>


Show HN: Interactive visual solver for the traveling salesman problem - <AUTHOR> https://tspvis.com <comment><USER2> Really cool visualisation, thanks for sharing.

Mentally I tend to further categorise heuristics into "construction" and "improvement" heuristics. Your "shortest path" is a construction heuristic that I normally call "nearest neighbour". Other common heuristics are "greedy" (iteratively add the shortest feasible edge) and "farthest insertion" (iteratively add the farthest point from the subtour to the best position). There's an interesting synergy between construction and improvement strategies, e.g. 2-opt might be more worthwhile after nearest neighbour than greedy. It would be extra awesome to be able to choose both construction as well as improvement and see how it pans out.

<comment><AUTHOR> Thanks so much! I haven't formally studied these so the terminology still gets me sometimes.

When you say "choose construction as well as improvement", I'm curious what you mean. It may not be clear, but if you run shortest path, the resulting best path is fed into whatever algorithm you run next. Is that what you mean?

<comment><USER2> Ah I see, I didn't realise you could combine heuristics. I still think it would be cool if you could pick from multiple construction heuristics (shortest path, farthest insertion...), followed by which improvement heuristic/s you want to use (2-opt, 3-opt...).

The choice of which heuristics you use and what order you do them in is sometimes called your meta heuristic strategy.

</comment></comment></comment><comment><USER3> Really nice!

If the optimal result is known maybe you could present that somehow? To see how good/bad the current solution is.

</comment><comment><USER4> Very cool! FYI, double clicking the map is throwing an error for me and crashes the app to a white screen.

<comment><AUTHOR> thanks for the heads up! should be fixed

</comment></comment><comment><USER5> I've been thinking about this problem for a while with a little twist. What if you wanted to run/walk all the streets in your city? What is the shortest path to do it? How do you calculate the number of kilometers of streets in a city?

<comment><USER6> This problem is called Chinese Postman Problem and can be solved in polynomial time using matching algorithms:

[https://en.m.wikipedia.org/wiki/Route_inspection_problem](https://en.m.wikipedia.org/wiki/Route_inspection_problem)

</comment></comment><comment><USER7> Nice. If I can suggest a cosmetic improvement, you list number of possible paths, it would be cool to see how many paths algorithm tried so far until it finishes (and how many paths it tried in total).

<comment><AUTHOR> really like this idea - I'll look into it, maybe with a progress bar or something</comment></comment>


How 1500 bytes became the MTU of the internet - <AUTHOR> https://blog.benjojo.co.uk/post/why-is-ethernet-mtu-1500 <comment><USER3> For.. reasons, I found myself having to make a 'driver' for a PoE+ sensing device this month. The manufacturer had an SDK, but compiling it requires an old version of Visual Studio - a bouquet of dependencies, and it had no OSX support. None of the bundled applications would do what I needed (namely, let me forward the raw sensing data to another application.. _SOMEHOW_ ).

The data isn't encoded in the usual ways, so even 4 hours of begging FFMPEG were to no avail.

A few glances at wireshark payloads, the roughly translated documentation, and weighing my options, I embarked on a harrowing journey to synthesize the correct incantation of bytes to get the device to give me what I needed.

I've never worked with RTP/RTSP prior to this -- and I was disheartened to see nodejs didn't have any nice libraries for them. Oh well, it's just udp when it comes down to it, right?

SO MY NAIVETE BEGOT A JOURNEY INTO THE DARKNESS. Being a bit of an unknown- unknown, this project did _not_ budget time for the effort this relatively impromptu initiative required. An element of sentimentality for the customer, and perhaps delusions of grandeur, I convinced myself I could just crunch it out in a few days.

A blur of coffee and 7 days straight crunch later, I built a daisy chain of crazy that achieved the goal I set out for. I read rfc3550 so many times I nearly have it committed to memory. The final task was to figure out how to forward the stream I had ensorcelled to another application. UDP seemed like the "right" choice, if I could preserve the heavy lifting I had accomplished to reassemble the frames of data.. MTU sizes are not big enough to accomodate this (hence probably why the device uses RTP, LOL.). OSX supports some hilariously massive MTU's (It's been a few days, but I want to say something like 13,000 bytes?) Still, I'd have to chunk and reasemble each frame into quarters. Having to write _additional_ client logic to handle drops and OOO and relying on OSX's embiggened MTU's when I wanted this to be relatively OS independent... and the SHIP OR DIE pressure from above made me do bad. At this point, I was so crunched out that the idea of writing reconnect logic and doing it with TCP was painful so I'm here to confess... I did bad...

The client application spawns a webserver, and the clients poll via HTTP at about 30HZ. Ahhh it's gross...

I'm basically adrift on a misery raft of my own manufacture. Maybe protobufs would be better? I've slept enough nights to take a melon baller to the bad parts..

<comment><USER4> What does it sense that changes >=30 times a second?

<comment><USER5> I was curious about that to. Lots of references to video related standards that imply its a PoE camera, but then why isn't the data encoded in the usual ways? What does that mean?

<comment><USER3> What codec would you use for a camera that captures not RGB, but poetry of the soul?

CONTEXTLESS, HEADERLESS, ENDLESS BYTE STREAMS OF COURSE, where the literal, idealized (remember udp) position of each byte is part of a vector in a non euclidean coordinate system.

<comment><USER6> > What codec would you use for a camera that captures not RGB, but poetry of > the soul?

I would love to read a collaborative work between you and James Mickens -- this genre of writing seems sadly under-present in the computing world...

<comment><USER3> I appreciate the interest in listening to a simulcast of Harvard-Professor- Collaborates-With-A-Nobody-Hobo. I'll forward this to my agent.

My agent is a tin can. I think she used to hold beans. Sometimes I put a few smashed nickels in her and rattle. While I do this, I pretend she's reading me my messages, and I'm like "oh no, I would never consent to a biopic directed by THAT charlatan." and then we laugh and laugh.

Oh how we laugh.

</comment></comment></comment></comment></comment></comment><comment><USER7> For 802.11, the biggest overhead is not packet headers but the randomized medium aquisition time so as to minimize collisions. 1500 bytes is way too small here with modern 802.11, so if you only send one packet for each medium aquisition, you end up with something upwards of 90% overhead. The solution 802.11n and later uses here is to use Aggregate MPDUs (AMPDUs). For each medium aquisition, the sender can send multiple packets in a contiguous burst, up to 64 KBytes. This ends up adding a lot of mechanism, including a sliding window block ack, and it impacts queuing disciplines, rate adaptation and pretty much everything else. Life would be so much simpler if the MTU had simply grown over time in proportion to link speeds.

<comment><USER8> > Life would be so much simpler if the MTU had simply grown over time in > proportion to link speeds.

The problem is that the world went wireless, so _maximum_ link speeds grew a lot but _minimum_ link speeds are still relatively low. A single 64kB packet tying up a link for multiple milliseconds—unconditionally delaying everything else in the queue by at least that much—is not what we want.

<comment><USER9> > _The problem is that the world went wireless, so maximum link speeds grew a > lot but minimum link speeds are still relatively low._

I would argue: the problem is that the MTU isn't negotiated at all, but especially not based on link availability.

<comment><USER10> IPv6 tries to solve this with path MTU discovery.

<comment><USER9> Yes, but IPv6 is still at a higher level than Ethernet, Wifi, et al and is therefore subject to the limitations of the lower level framing

<comment><USER11> Sure, I mean that's what pMTUd is all about. One big difference with IPv6: Routers can't fragment packets. They either send or they don't.

<comment><USER12> I thought so too, but apparently there is an IPv6 fragmentation extension and it's implemented by several operating systems.

<comment><USER11> Only the endpoints can fragment.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> Software developer talking confidently about electrical engineering issues he knows nothing about. How cute. /s

All Ethernet adapters since the first Alto card had self-clocking data recovery [1].

Clock accuracy was never a problem, as long as it was withing the acceptable range required for PLL lock/track loop.

The reason for 1500 MTU is that for packet-based systems, you don't want infinitely large packets. You want _small_ packets. but large enough so that packet overhead is insignificant, which in engineering terms means less than 2%-5% overhead. Thus 1500 max packet size. Everything above that just makes switching and buffering needlessly expensive, SRAM was hella expensive back then. Still is today (in terms of silicon area).

Look at all the memory chips on Xerox Alto's Ethernet board (below) - <USER2> chips were already taking ~50% of the board area!

[1] Schematic of the original Alto Ethernet card clock recovery circuit: [https://www.righto.com/2017/11/fixing-ethernet-board-from- vi...](https://www.righto.com/2017/11/fixing-ethernet-board-from-vintage.html)

EDIT: Lol! Author has completely replaced erroneous explanation with correct explanation, including link to seminal paper about packet switching. Good.

<comment><USER14> > How cute. /s

> EDIT: Lol! Author has completely replaced erroneous explanation with correct > explanation, including link to seminal paper about packet switching. Good.

Don't be a jerk. Being right doesn't give you the right to make fun of people.

<comment><USER13> Noted. I find the over-confidence over a completely imagined issue funny and interesting. I make that mistake too. It's always interesting to do a post- mortem: why was I so confident? how did I miss the correct answer? I respect the author for doing such a fast turn around :)

<comment><USER15> Good to know that your intent wasn't malicious, but fwiw, I also didn't find the tone particularly appropriate for HN, either.

<comment><USER16> The hardware world is full of this.

<comment><USER17> I think because in the hardware world the cost for being wrong is so much higher since you can't push out an over the air update or update your saas or whatever. So if you don't know you stay quiet instead of being wrong.

</comment></comment></comment></comment></comment></comment><comment><USER18> It would be nice to corroborate this reason with another source, because my understanding is that clock synchronization was not a factor in determining the MTU, which seems really more like a OSI layer 2/3 consideration.

I am surprised the PLLs could not maintain the correct clocking signal, since the signal encodings for early ethernet were "self-clocking" [1,2,3] (so even if you transmitted all 0s or all 1s, you'd still see plenty of transitions on the wire).

Note that this is different from, for example, the color burst at the beginning of each line in color analog TV transmission [4]. It is also used to "train" a PLL, which is used to demodulate the color signal transmission. After the color burst is over, the PLL has nothing to synchronize to. But the 10base2/5/etc have a carrier throughout the entire transmission.

[1] [https://en.wikipedia.org/wiki/Ethernet_physical_layer#Early_...](https://en.wikipedia.org/wiki/Ethernet_physical_layer#Early_implementations)

[2] [https://en.wikipedia.org/wiki/10BASE2#Signal_encoding](https://en.wikipedia.org/wiki/10BASE2#Signal_encoding)

[3] [http://www.aholme.co.uk/Ethernet/EthernetRx.htm](http://www.aholme.co.uk/Ethernet/EthernetRx.htm)

[4] [https://en.wikipedia.org/wiki/Colorburst](https://en.wikipedia.org/wiki/Colorburst)

<comment><USER19> I also don't believe this is the reason. Early Ethernet physical standards used Manchester encoding to recover the data clock.

<comment><USER20> I would agree given I worked on an Ethernet chipset back in 1988/9 keeping the PLL synched was not a problem. I can't remember what the maximum packet size we supported was (my guess is 2048) but that was more of a buffering to SRAM and needing more space for counters.

The datasheet for the NS8391 has no such requirement for PLL sync.

[https://archive.org/details/bitsavers_nationaldaDataCommunic...](https://archive.org/details/bitsavers_nationaldaDataCommunicationsLANUARTsHandbook_53722793/page/n73/mode/2up)

</comment></comment></comment><comment><USER21> As others have said, with Manchester encoding 10BASE2 is self-clocking, you can use the data to keep your PLL locked, just as you would on modern ethernet standards. However I imagine with these standards you may not even have needed an expensive/power-hungry PLL, probably you could just multi-sample at a higher clock rate like a UART did (I don't actually know how this silicon was designed in practice).

Futher PLLs have not got a lot better, but a lot worse. Maybe back when 10BASE2 was introduced you could train a PLL on 16 transitions and then have acquired lock but there's no way you can do that anymore (at modern data rates). PCI express takes thousands of transitions to exit L0s->L0, which is all to allow for PLL lock.

My best guess for the 1500 number is that with a 200ppm clock difference between the sender and receiver (the maximum allowed by the spec, which says your clock must be +-100ppm) then after 1500 bytes you have slipped 0.3 bytes. You don't want to slip more than half a byte during a packet as it may result in duplicated or skipped byte in your system clock domain. (200 _1e-6)_ 1500=0.3.

<comment><USER22> I thought most Ethernet PHYs don't lock actually to the clock, but instead use a FIFO that starts draining once it's half way full. The size of this FIFO is such that it doesn't under or overflow given the largest frame size and worst case 200 PPM difference.

I figured this is what the interframe gap is for - to allow the FIFO to completely drain.

<comment><USER23> IFP is really more to let the receiver knows where one stream of bits stop and the next stream of bits start. How they handle the incoming spray of data is up to them on a queue/implementation level.

</comment></comment></comment><comment><USER24> The original MTU was 576 bytes, enough for 512 bytes of payload plus 64 bytes for the IP and TCP header with a few options. 1500 bytes is a Berkeleyism, because their TCP was originally Ethernet-only.

<comment><USER25> Yeah, didn't T1 and ISDN use 576 to limit serialization delay and jitter? The backbone probably switched to 1500 when OC-3 was adopted.

<comment><USER26> The default MTU for a T1/E1 was usually 1500. The default for HSSI was 4470 which meant the default for DS3 circuits was 4470. This was also the usual default MTU for IP over ATM which is what most OC-3 circuits would have been using when they were initially rolled out for backbone use. This remained the usual default MTU all the way through OC-192 circuits running packet over sonnet.

I left the lSP backbone and large enterprise WAN field around that time and can't speak to more recent technologies.

</comment></comment></comment><comment><USER27> IEEE 802 history is disappearing without a trace? Afaik it's pretty well documented, you just need to be a member for some of the stuff.

[http://www.ieee802.org/](http://www.ieee802.org/)

I feel like the last piece we're missing in this story is the performance impact of fragmentation. Like why not just set all new hardware to an MTU of 9000 and wait ten years?

<comment><USER28> > Like why not just set all new hardware to an MTU of 9000 and wait ten years?

The hardware in question is Ethernet NICs. However, for you to set the MTU on an Ethernet NIC to 9000, _every_ device on the same Ethernet network (at least the same Ethernet VLAN), including all other NICs and switches, including ones which aren't connected yet, must also support and be configured for that MTU. And this also means you cannot use WiFi on that Ethernet network (since, at least last time I looked, WiFi cannot use a MTU that large).

<comment><USER27> Sending a jumbo frame down a line that has hardware that doesn't support jumbo frames somewhere along the way does not mean the packet gets dropped. The NIC that would send the jumbo frame fragments the packet down to the lower MTU. So what's the performance impact of that fragmentation? If it isn't higher than the difference in bandwidth overhead from headers of 9000 MTU traffic vs. 1500 MTU traffic then why not transition to 9000 MTU?

<comment><USER29> But how does the NIC know that, 11 hops away, there is a layer 2 device, which cannot communicate with the NIC(switches do not typically have the ability to communicate directly with the devices generating the packets), that only supports a 1500 byte frame?

Now you need Path MTU discovery, which as the article indicates, has its own set of issues. (Overhead from trial and error, ICMP being blocked due to security concerns, etc...)

<comment><USER30> If you block ICMP you deserve what you get. Don't do this. (Edit: don't block ICMP)

<comment><USER31> So now you're trying to communicate from your home machine to some random host on the internet (website, VPS, streaming service), and you're configured for MTU 9000, the remote service is also configured for MTU 9000, but some transit provider in the middle is not, and they've disabled ICMP for $reasons.

They blocked ICMP, do you deserve what you get?

<comment><USER30> Transit providers should push packets and generally do. With PMTU failures it's usually clueless network admins on firewalls nearer endpoints. And no, you don't and I wish the admin responsible could feel your pain.

<comment><USER31> > Transit providers should

Agreed

> and generally do

Agreed.

Now if you can make it 'will always just push packets', we'll be golden.

Unfortunately, there are enough ATM/MPLS/SONET/etc networks being run by people who no longer understand what they're doing, that we're never going to get there.

To make matters more entertaining, IPv6 depends on icmp6 even more.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER32> The article talks about how the 1500 byte MTU came about but doesn't mention that the problem of clock recovery was solved by using 4b/5b or 8b/10b encoding when sending Ethernet through twisted-pair wiring. This encoding technique also provides a neutral voltage bias.

EDIT: As pointed out below, I failed to account for the clock-rate being 25% faster than the bit-rate in my original assertion that Ethernet over twisted- pair was only 80% efficient due to the encoding (see below)

<comment><USER22> > Ethernet through twisted-pair wiring only provides 80% of the listed bit- > rate

Actually, they already accommodated for this in the advertised speed.

In other words, a 1 GbE SerDes runs at 1.250 Gbit/s, so you end up with an actual 1 Gbit/s bandwidth.

The reason you don't actually hit 1 Gbit/s in practice is due to other overheads such as the interframe gaps, preambles, FCS, etc.

<comment><USER32> You're absolutely correct ... it's been a long time since I was designing fiber transceivers but I should have remembered this. Ultimately efficiency is also affected by other layers of the protocol stack too (UDP versus TCP headers) which also explains why larger frames can be more efficient. In the early days of RTP and RTSP, there were many discussions about frame size, how it affected contention and prioritization and whether it actually helped to have super-frames if the intermediate networks were splitting and combining the frames anyway.

</comment></comment></comment><comment><USER33> I used to glue stuff together to FDDI rings and Token Ring networks back in the day (I used Xylan switches, which had ATM-25 UTP line cards among other long-forgotten oddities), and MTU sizes always struck me as being particularly arbitrary.

But I'm not really sure about the clock sync limitations being a factor here. It was way back in the deepest past.

What I do remember vividly is the mess that physical layer networking evolved into over the years thanks to dial-up and DSL (ever had to set your MTU to 1492 to accommodate an extra PPP header?).

And something is obviously wrong today, since we're still using the same baseline value for our gigabit fiber to the home connections, our 3/4/5G (scratch to taste) mobile phones, etc.

<comment><USER34> > ever had to set your MTU to 1492 to accommodate an extra PPP header?

Ah, I was always wondering why my ISP configured my fiber modem's mtu to 1492. So it's due to using PPPoE? Is there no way to use bigger mtu when using PPPoE?

<comment><USER35> Nowadays, there's PPPoA (over ATM) which wraps at a lower level, and allows 1500 byte ethernet payloads through. But running the ethernet over ATM at 1508 MTU so that PPPoE would be 1500 was probably out of reach --- when PPPoE was introduced, the customer endpoint was often the customer PC, and some of those were using fairly old nics that might not have supported larger packets.

Sadly, smaller than 1500 byte MTUs still cause issues for some people to this day. It's all fine if everything is properly configured, or if at least everything sends and receives ICMP, but if something is silently dropping packets, you're in for a bad day. These days, I think it's usually problems with customers sending large packets, as opposed to early days where receiving large packets would routinely fail, but a lot of that is because large sites gave up on sending large packets.

<comment><USER33> Yes, PPPoA was also a thing I dealt with, and another source of irritating MTU issues.

</comment></comment></comment></comment><comment><USER36> Unreliable IP fragmentation, and the brokenness of Path MTU Discovery (PMTUD), is causing the DNS folks to put a clamp on the size of (E)DNS message size:

* [https://dnsflagday.net/2020/](https://dnsflagday.net/2020/)

</comment><comment><USER37> When networks were new, computers connected to each other using a shared trunk that you _physically_ drilled into. It's a non-trivial problem to send data over a shared channel; it's very easy for two systems to clobber each other. A primitive, but somewhat effective mechanism is ALOHA ([https://en.wikipedia.org/wiki/ALOHAnet](https://en.wikipedia.org/wiki/ALOHAnet)), where multiple senders randomly try to send their message to a single receiver. The single receiver then repeats back any messages it successfully receives. In that way the sender is able to confirm its message got through -- an ack. After a certain amount of time with no ack, senders repeat their messages. As you can imagine, shorter packets are less likely to cause collisions.

Ethernet uses something similar, but is able to detect if someone else is using the wire, called carrier sense. A short packet of 1500 bytes reduced the likelihood of collisions.

<comment><USER38> Does multiplexing over Ethernet exist?

<comment><USER39> Not anymore for all practical purposes, but it once did for the very old 10Base-2 standard for Ethernet over coaxial cable. This is practically why the old MII Ethernet PHY interface protocol had the collision-sense lines to indicate to the MAC to stop sending data if it detects incoming data, in attempts to minimize collisions.

[https://en.wikipedia.org/wiki/10BASE2](https://en.wikipedia.org/wiki/10BASE2)

<comment><USER38> This is very cool history, and something I never would have stumbled upon myself. Thank you for sharing! :-)

</comment></comment></comment></comment><comment><USER40> Minor factoid the article does not mention. ATM is an alternative to Ethernet that's used in many optical fiber environments. The "transfer unit" size of the ATM "cell" is 53 bytes (5 for the header and 48 for the payload). This is much smaller than 1500.

Another quirky story from the past: Sometime around 20 years ago I was having a bizarre networking problem. I could telnet into a host with no trouble, and the interactive session would be going just fine until I did something that produced a large volume of output (such as 'cat' on a large file). At that point the session would freeze and I would eventually get disconnected. After troubleshooting for a while I identified the problem as one of the Ethernet NICs on the client host. It was a premium NIC (3Com 3C509). Nonetheless, the NIC crystal oscillator frequency had drifted sufficiently that it would lose clock synchronization to the incoming frame if the MTU was larger than about 1000.

<comment><USER41> Speaking about ATM: the 48 byte payload was a standardization compromise between Europe and the US.

US companies had prototypes using 64 bytes, while European companies used 32 bytes. To avoid anyone giving a competitive advantage, they decided on a middle ground of 48.

There were trade-offs between 32 and 64 bytes as well: a 32 byte payload had a higher overhead than a 64 byte payload, but it had a shorter transmission time which made it easier to do voice echo cancellation.

Or so I was told many decades ago when I got introduced to ATM systems...

</comment></comment><comment><USER42> I remembered something different related to shared medium and CMSA/CD where 1500 ensured fairness, and the minimum of 46 related to propagation time in the longest allowable cable

More at:

[https://networkengineering.stackexchange.com/questions/2962/...](https://networkengineering.stackexchange.com/questions/2962/why- was-the-mtu-size-for-ethernet-frames-calculated-as-1500-bytes)

</comment><comment><USER43> I think the author may have made a mistake in some of the math. The frame size distribution plots are likely based on the number of frames, not the amount of data contained in said frames. The 1500 byte and other large frames should therefore account for the lion's share of the actual data transferred. Correcting this error will totally change the final two graphs.

<comment><USER44> Yes. But only the "AMS-IX traffic by packet size range" graph is wildly inaccurate. Ethernet frame overhead is per-packet and presumably right.

<comment><USER43> Ah yeah, that's probably true. According to some back of the envelope math, it seems like the distribution should be more like 5%, 1%, 1%, 3%, 50%, 39%, ignoring the first and last size bins.

</comment></comment></comment><comment><USER45> I find that technology cements in strata (the archaeology term) just as the layers that accumulate as the result of natural processes and human activity. The dynamics are not exactly the same but the tendency is similar. I wonder whether we'll always be capable of digging down deeper to the beginnings as things get more and more complicated.

</comment><comment><USER46> I don't think the Ethernet Frame Overhead graph is correct. Surely the overhead is proportionally higher, per amount of data, for smaller packets. That graph shows that the overhead is just proportional to the amount of data sent, irrespective of the packet size, which can't be right.

<comment><USER43> The graph above that one is totally wrong. The frame overhead graph may be correct, though.

</comment></comment><comment><USER47> Kind of expected an article titled 'How 1500 bytes became the MTU of the internet' to tell us how 1500 bytes became the MTU of the internet.

Even I could of told you, 'the engineers at the time picked 1500 bytes'.

</comment><comment><USER48> IIRC it was called "thinnet" (10B2). I loved the vampire taps on thick net.

</comment><comment><USER49> Not my field, so I might be making an obvious error here, but:

If there are efficiency gains to be had from using jumbo frames, wouldn't setting my MTU to a multiple a 1500 still be of some benefit? If my PC, my switch and my router all support it, that would still be a tiiiny improvement. If the server's network does as well and let's say both of our direct providers, even if none of the exchanges or backbones in between do, that would still be an efficiency gain for ~10% of the link, right?

<comment><USER50> Locally you can set your MTU to larger than 1500, but if you (generally) try and send a packet towards the internet larger than 1500 it will be dropped without a trace, or it will be dropped and an ICMP message will be generated to tell your system to lower the MTU. Assuming you have not firewalled off ICMP ;)

As a handy feature on Linux at least, you can set your MTU to 9000 locally, and then set the default (internet generally) route to have a MTU of 1500 to prevent issues:

ip route add 0.0.0.0/0 via 10.11.11.1 mtu 1500

<comment><USER51> Over-sized packets can (and generally will) be fragmented by your router. It shouldn't be dropped unless you've intentionally set DNF.

<comment><USER50> Fragments are very hit or miss on the internet, [https://blog.cloudflare.com/ip-fragmentation-is- broken/](https://blog.cloudflare.com/ip-fragmentation-is-broken/)

</comment></comment></comment></comment><comment><USER52> Off-topic but looking at that old network card picture reminded me of a very vague memory of more than one card with a component that looked like a capacitor, except it looked cracked.

Is my mind playing tricks? Were they faulty units or was there meant to be a crack?

This picture could be the same thing:

[https://www.vogonswiki.com/images/3/37/Viglen_Ethergen_PnP_2...](https://www.vogonswiki.com/images/3/37/Viglen_Ethergen_PnP_2000A.jpg)

<comment><USER53> Old network card eh? Its a 3Com 3C509:

[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/3com?h=v5.6-rc2)

I still have a load of them gathering dust somewhere. However a system with ISA on it is a bit rare now and I'm not sure I can be bothered to compile a modern kernel small enough to boot on one. Besides, it will probably need cross compiling on something with some grunt that has heard of the G unit prefix.

<comment><USER54> For Old network card try 3C500, or something made by Excelan ;-)

[http://www.os2museum.com/wp/emulating- etherlink/](http://www.os2museum.com/wp/emulating-etherlink/)

</comment></comment></comment><comment><USER55> So they just picked an arbitrary number that felt right? I expected the story to be more interesting than that, given the title. Still, there was some interesting trivia surrounding the core question.

Reminds me of the IPv6 adoption problem: [https://news.ycombinator.com/item?id=14986324](https://news.ycombinator.com/item?id=14986324)

</comment><comment><USER56> > If we look at data from a major internet traffic exchange point (AMS-IX), we > see that at least 20% of packets transiting the exchange are the maximum > size.

He's so optimistic. My brain heard this as " _only_ 20% of packets […] are the maximum size"

What are all of those 64 byte packets? Interactive shells, or some other low bitrate protocol?

<comment><USER25> Probably mostly ACKs.

<comment><USER56> Well now I feel dumb.

</comment></comment></comment><comment><USER57> I've always wondered how 9000 became "jumbo". Technically anything over 1500 is consider ju!no and there is no standard. The largest I've seen is 16k. I think there are some crc accuracy concerns at larger sizes but 9k still seems quite arbitrary for computer land.

<comment><USER28> The explanation according to [https://web.archive.org/web/20010221204734/http://sd.wareone...](https://web.archive.org/web/20010221204734/http://sd.wareonearth.com/~phil/jumbo.html) is: "First because ethernet uses a 32 bit CRC that loses its effectiveness above about 12000 bytes. And secondly, 9000 was large enough to carry an 8 KB application datagram (e.g. NFS) plus packet header overhead."

That is, 9000 is the first multiple of 1500 which can carry an 8192-byte NFS packet (plus headers), while still being small enough that the Ethernet CRC has a good probability to detect errors.

</comment></comment><comment><USER58> This reminds me of various Windows applications back in the day (Windows 3.1 and 95) that claimed to fine tune your connection and one of the tricks they used was changing the MTU setting, as far as I can recall. Could anyone share how that worked?

<comment><USER59> If your computer sends a larger MTU than the next device upstream can handle, the packets will be fragmented leading to increased CPU usage, increased work by the driver, higher I/O on the network interface, higher CPU load on your router or modem, etc depending on where the bottleneck is. For example if you connect over Ethernet to a DSL modem, or to a router that has a DSL uplink, all your packets will be fragmented. This is because DSL uses 8 bytes per packet for PPPoE authentication. So if you send a 1500 byte packet to the modem, it will get broken up by the modem into 2 packets: one is 1492+8 bytes, and the other is 8+8 bytes.

But your PC is still sending more packets.. the modem is struggling to fragment them all and send them upstream.. its memory buffer is filling up.. your computer is retrying packets that it never got a response on..

By lowering your computer MTU to 1492 to start with, you avoid the extra work by the modem, which can have considerable speed increase.

</comment></comment><comment><USER60> Probably a dumb question, why the maximum size (and the one has most of packages) in the AMS-IX graph 1514 bytes instead of 1500 bytes that got discussed in the article?

<comment><USER61> 1500 bytes is the MTU of IP, in most cases. It often excludes the Ethernet header, which is 14 bytes excluding the FCS, preamble, IFG, and any VLANs.

If have a 1500 byte MTU for IP, then we need at least a 1514 byte MTU for IP + Ethernet. We often call the > 1514B MTU the "interface MTU". It's unnecessarily confusing.

</comment></comment><comment><USER62> Actually, MTUs below 1500 bytes are pretty common, e.g. with PPP over Ethernet or other forms of encapsulation/tunneling.

</comment><comment><USER63> I think you're saying that the smallest bucket of packets are all packets that would have been combined with a larger packet of that had been an option... but that doesn't make sense. That class of packets includes TCP SYN, ACK, RST, and 128 bytes could fit an entire instant message on many protocols.

</comment><comment><USER64> Looks like a ripe low-hanging fruit for SpaceX Starlink to pick..

<comment><USER64> Why the downvote? possibly facilitating the end-to-end transport will allow them to offer jambo packets

<comment><USER65> This would only be possible if you were talking from a jumbo-configured client (let's say you've set up your laptop correctly), across a jumbo-configured network (Starlink, in your scenario), to a jumbo-configured server (here's the problem).

The problem is that Starlink only controls the steps from your router to "the internet". If you're trying to talk to spacex.com it'd be possible, but if you're trying to talk to google.com then now you need Starlink to be peering with ISPs that have jumbo frames, and they need to peer with ISPs with jumbo frames, etc etc and then also google's servers need to support jumbo frames.

Basically, the problem is that Starlink is not actually end to end, if you're trying to reach arbitrary servers on the internet. It just connects you to the rest of the internet, and you're back to where you started.

This is also true for any other ISP, Starlink is not special in this regard.

<comment><USER66> True, you'd expect endpoints to support Jumbo Frames as well, but why not start at least making it possible. It's a dead loop otherwise. IPv6 was the same at start.

</comment></comment></comment></comment><comment><USER67> Nah, it's 1492 forever!

<comment><USER68> Found the ADSL user.</comment></comment>