Performance Engineering with React, Part 1 - <AUTHOR> http://benchling.engineering/performance-engineering-with-react/ <comment><USER2> I've been using React to make a 60fps music game, and this article covers basically all of the issues I ran into:

\- Originally, the game was rendered with SVG elements, so it was very important to use the shouldComponentUpdate hook to ensure elements weren't rendering unnecessarily (and using CSS transformations to position them where possible, as that's way faster than updating SVG coordinates!)

\- I had something similar to the function.bind problem since some of my components had a map of hotkey handlers that I was (naively) returning from a `getHotKeyHandlers` function, meaning the handlers were redefined every render.

\- I wasn't caching/memoizing calculations done to the Redux state as well as I should have (Reselect is very useful for this!)

I ended up deciding SVG wasn't fast enough for my use case and rewriting the core game loop rendering to use Canvas, which was so much faster that I could be a lot lazier about optimizing things. Everything around the canvas is still rendered through React components (i.e. in game UI and menu screens), and the state is still handled through Redux. It's a very fun way to write a game :)

<comment><USER3> Do you think React Canvas is a thing you could have used?

[https://github.com/Flipboard/react- canvas](https://github.com/Flipboard/react-canvas)

I've still not had a chance to play around with it, but the Flipboard mobile site looks phenomenal.

<comment><USER2> That looked interesting but I didn't need the most important functionality it provides (click/touch handling, fancy text rendering, etc) so I stuck with just using the Canvas API. It's pretty simple to do in React: [https://gist.github.com/thomasboyt/bcc496c823a01ac5f648](https://gist.github.com/thomasboyt/bcc496c823a01ac5f648)

<comment><USER4> I might be missing something here: what benefit does React bring in this example? If you're using canvas to render without any kind of virtual DOM, why use React at all?

<comment><USER5> Can't you combine? Thinking it could be used for UI. Makes sense when you need more support, like localisation and other builtin stuff. Just guessing.

</comment></comment></comment></comment></comment><comment><USER6> Thanks for the valuable tips!

Currently, my biggest 'bottleneck' when developing React applications is managing complex forms. Right now I'm using redux-form [0] which is a great tool, but not very performant on dynamic deep/complex forms (more info and partial hacks here: [1]).

Other than that, I don't have performance problems, but I'm curious when will I hit Redux limitations (single store sounds great from the conceptual point of view, but I'm not sure about the performance one). Also, can second ImmutableJS recommendation, after a while of using it's completely natural to use.

[0] [https://github.com/erikras/redux-form/](https://github.com/erikras/redux- form/)

[1] [https://github.com/erikras/redux- form/issues/529](https://github.com/erikras/redux-form/issues/529)

<comment><USER7> > single store sounds great from the conceptual point of view, but I'm not > sure about the performance one

But the Redux "single store" is actually a composition of multiple stores, each with its own reducer function. Performance-wise, it's actually better than vanilla Flux, because, when an action causes N stores to change with vanilla Flux, you can get up to N renders of your components (depending on how many stores they subscribe to). At least, with Redux there is only one render when the whole store has been updated.

<comment><USER5> When I was looking around and comparing I think most allowed for some form of aggregation and syncing them into one call. Not sure though. Still learning.

</comment></comment></comment><comment><USER8> The biggest performance issue I've encountered is with long lists. The most basic implementation means you will have to instantiate a new component for every item in the list even if only one of them has changed. I haven't spent that much time looking into it, so the bottleneck might be function binding (I hadn't thought of that before) but if anybody has an approach to this any ideas are appreciated. Maybe only pass an id as a prop and grab the rest from context?

<comment><USER9> I'm going to cover this in my next post, but I've found that using PureRenderMixin with fairly large number of items (~100s) works. The non- trivial step is often identifying why your subcomponents are still re- rendering, which I solved by writing a mixin that logs to the console what props/state is deep equal but not shallow equal (i.e. where PureRenderMixin should theoretically work).

But I agree, it's still not perfect. You avoid doing the more expensive virtual DOM comparison, but you still have to iterate over all the items to do the PureRenderMixin checks. And it is still O(n) time, just with a smaller constant.

Another solution is to not actually render all the children, just the ones that are visible on the screen, i.e. using some React implementation of infinite lists (see: [https://facebook.github.io/fixed-data- table/](https://facebook.github.io/fixed-data-table/) )

Another idea that I've been thinking about is to arbitrarily fragment your list into lists (possibly of fixed size, or a fixed number of fragments). Each fragment gets its own subcomponent, so we can avoid rendering fragments that haven't changed. For example, instead of a list of 100 items, we treat it as 10 lists of 10 items. If we change one item, we end up instantiating 10 components, and then recursing into the single fragment that has changed, instead of over all 100 items.

<comment><USER8> I've thought about doing the list of smaller lists but it could cause issues with the DOM (if you care about it) since components must return a single element. Will look into that fixed data table.

<comment><USER10> See also react-list ([https://github.com/orgsync/react- list](https://github.com/orgsync/react-list)), which does a surprisingly good and hassle-free job of rendering partial lists.

</comment></comment></comment></comment><comment><USER11> I want to see an article which shows how to code a fast rich text editor in React.

I tried to implement one by representing the text as a long list of character components, thinking react could handle this by only executing small incremental updates, but the whole thing became slow anyway.

Am I missing something, or is react not up to this task?

<comment><USER9> Building a fast rich text editor in React is no easy task, and would be at least a series of articles :-P. At Benchling we've built a fairly fast rich text editor in React--but it was a considerable amount of work and still requires quite a bit more work. You may have heard that Atom was originally written in React, but they abandoned it to avoid its overhead[0]

There are open-source projects worth looking at like Ritzy[1] which is quite fast and written in React from what I've read.

As per your specific issue, I've found that using PureRenderMixin with each line of text represented as a component works for hundreds of lines, but this still ends up with performance issues as you scale to thousands of lines. Using a list of one-character components will run into performance issues much sooner as there is per-component overhead.

[0] [https://github.com/atom/atom/pull/5624](https://github.com/atom/atom/pull/5624) [1] [http://ritzyed.github.io/ritzy/](http://ritzyed.github.io/ritzy/)

</comment></comment><comment><USER12> I don't see enough mention of this, but if you want immutability (to make shouldComponentUpdate easy) icepick is a better choice than Immutable.js for a great many applications:

[https://github.com/aearly/icepick](https://github.com/aearly/icepick)

It's tiny but offers a number of utilities that are more convenient than what you get with ES6 spread syntax. The tradeoff with Immutable.js is that if you have huge single objects (arrays with many elements, objects with many keys), then making copies will get more expensive than with Immutable. Otherwise, icepick is all win.

</comment><comment><USER9> Author here, happy to answer any questions!

<comment><USER13> With the ES6 class syntax, is it possible to avoid the function.bind/anonymous function issue (and subsequent need for the whole IntermediateBinder thing) by redefining methods that will be passed into a subcomponent in the constructor to their bound equivalent? That is, in the constructor: `this.foo = this.foo.bind(this)` and in render: `<SubComponent foo={this.foo} />`?

<comment><USER9> What you described is the current replacement for auto-bind (that React.createClass used to do for you). The issue arises when you have not just one SubComponent, but n SubComponents that require the parent method bound to the subcomponent's index, e.g. you have a deleteItemAtIndex method, and you only want to expose {deleteItem: deleteItemAtIndex.bind(this, i)} to each child component. ES6 React classes don't have any special way of handling this.

To re-iterate the possible solutions I've considered:

\- Don't pass in a bound method; give the component the unbound method and its index and the child component can call it with the index passed in itself. \- Generalize this to a re-usable intermediary component that does for you (it does feel a bit dirty) \- Write your own bind function that annotates the bound function with original function + params allowing you to do a "deep" equality check, and then use a variant of PureRenderMixin that does this "deep" equality check.

Honestly, all of the solutions feel a bit hacky, but I've gravitated towards the first and second options.

<comment><USER13> Ah, of course. Thank you for the clarification.

</comment></comment></comment></comment><comment><USER14> These will _probably_ be useful but I did not understand any of the terms mentioned in the article. I guess I haven't hit any major bottlenecks yet. Bookmarked!

</comment><comment><USER10> I just went through a rather large profiling/optimization run on the my site, BitMEX (we're a Bitcoin derivatives exchange). Performance is paramount in trading, so our site has to be fast. Here's some additional tips:

0\. shouldComponentUpdate (SCU) is everything. Consider creating a base React.Element extension class. I define a base shouldComponentUpdate and some useful utilities on it. Having it in one place is really useful because you can log its output from every component and watch how your whole application updates. It makes it easy to see useless rerenders.

\- One very helpful addition in our app has been a static 'SCUSelector' property, which tells our shallow shouldComponentUpdate to ignore certain props/state/context. This helps in combination with rich componentWillMount() or getInitialState() functions to precalculate expensive data that updates rarely/never.

1\. It seems the `connect(component, selector)` model [0] encouraged by Redux to hook up application data to deep(er) components is much better for performance. You want to avoid renders in as many tree branches as possible. To this day, the root element (`<App>`) still holds most of our data and fans it out to the Router, Header, Sidebar, etc. This root-level render must be _extremely_ fast because it runs on every tick. Hoisting any and all data that can be made constant is a huge help.

2\. We saw massive (50%+) performance gains from Babel's constant-elements and inline-elements optimizations [1]. Careful with it - there's is a nasty bug [2] that _only manifests in production_ you need to be aware of if you're deploying this.

3\. Immutable-JS is kind of a wash. I'm not a big fan of the API or how easy it is to nest mutable objects and arrays inside immutables. The new array/object spread syntax in ES6 makes it easier to construct unique objects every time data updates, so you can do a shallow shouldComponentUpdate.

4\. Watch out for function identity because it will blow SCU [3]. Either ignore the property explicitly, or use a binding helper like the one I posted on that issue [4].

5\. If you're doing expensive data computation, such as update/insert/delete/image on websockets, profile it. If you're accidentally causing deopt, you'll feel it. Make sure the libraries you use don't cause deopt either. A few simple tweaks [5] can give you 2-10x performance boosts on hot functions [6].

6\. If you haven't already, watch videos on how V8 optimizes. It will help you write fast code in the future.

7\. Keep watching your app! The new React devtools can highlight components as they update. This is really useful to see what work is being done and how you can improve.

\---

Links:

0\. [https://github.com/rackt/react- redux/blob/master/docs/api.md...](https://github.com/rackt/react- redux/blob/master/docs/api.md#connectmapstatetoprops-mapdispatchtoprops- mergeprops-options)

1\. [https://babeljs.io/docs/plugins/transform-react-inline- eleme...](https://babeljs.io/docs/plugins/transform-react-inline-elements/)

2\. [https://github.com/facebook/react/issues/5138](https://github.com/facebook/react/issues/5138)

3\. [https://github.com/facebook/react/issues/5197](https://github.com/facebook/react/issues/5197)

4\. [https://github.com/facebook/react/issues/5197#issuecomment-1...](https://github.com/facebook/react/issues/5197#issuecomment-177991647)

5\. [https://github.com/petkaantonov/bluebird/wiki/Optimization-k...](https://github.com/petkaantonov/bluebird/wiki/Optimization- killers)

6\. [https://github.com/AmpersandJS/ampersand- state/pulls?utf8=%E...](https://github.com/AmpersandJS/ampersand- state/pulls?utf8=%E2%9C%93&q=is%3Apr+strml)

</comment><comment><USER15> > This post is for those of you with a complex React application. If you're > building something smaller, you might not need to focus on performance yet. > Don't prematurely optimize! Go build things!

This is bit meta, but it's really, really awesome to see an article on performance begin with this. Well done author :-)

It instantly inspires confidence that the material to follow is worthwhile reading. Knowing the author has started out with this philosophy I trust more that what's covered will focus on things that actually matter, rather than just being a crawl through every and any technique or quirk that they've come across, regardless of its relative impact.

Performance is one of those things where more is obviously better, all other things equal, but that last 20% of effort invested is almost always time that could be better spent improving other aspects of the average _use case_ of any given product, let alone the average product.

</comment><comment><USER16> TLDR for ClojureScript users: most techniques described here do not apply to you. Whatever React library you use (Rum, Om, Reagent, or other), most work has already been done for you through immutable data structures. Unless you go out of your way to create lots of components that react to something different than their parameters, shouldComponentUpdate will be handled quietly behind your back and you'll get impressive performance right out of the box. And no need to even learn about the differences between deep/shallow comparisons, either.

<comment><USER17> This is true unless you're using cljs-datetime. The library's use of goog.date objects under the hood means that storing one in your state tree invalidates the fast equality checks that everything in the clojurescript space is assuming.</comment></comment>


Monetizing a SMS Notification Application - <AUTHOR>

I'm working on the finishing touches of an application that's main feature will be sending out real time SMS notifications.  For the purposes of this question, I'll use the example of a sports score text message update service (Not my actual idea, but similar in function).
I'm planning on offering two different account options, one free, basic, and ad supported, one paid, customizable, and ad free.
The free service would only send messages like "the score is now 7 to 3" while the paid version will have options like sending a message saying "Manny Ramirez hit a double for 2 RBI's".
So my question is, what kind of monetization options are there for the free account?  I've contacted www.movoxx.com, a company that embeds 40 character ads into every text you send, and that seems like a good option so long as they accept my idea and allow me to sign up.  Does anyone have any experience with them?
The service should appeal to users who are looking for a good amount of SMS updates so there's a potential for sending out thousands of messages a day if the product catches on.
What other options are there for SMS message monetization? <comment><USER2> sms messages have a built in monitization scheme: per txt and people are well conditioned to this IMHO. I'd be surprised if you tick anyone off by using it, and I'd heavily question any deviation from this.

I'd probably go for something like a free trial 10 messages, after that sign up and start paying. Just stick with the single paid account.

<comment><AUTHOR> That was actually my exact model before I found out what Movoxx was. I haven't gotten any actual CPM projections from them yet, but their service made it seem to me that it would make it easier to pick up more users because it would allow a long term free option for them.

But I originally had planned on 20 free alerts, then you would have to upgrade for something like $20 for 6 months.

</comment></comment><comment><USER3> Ads are a good way of doing it, but I have never been a fan of the per message ad. I would suggest leaving the messages clean and then sending messages dedicated to an ad. Clearly full message ads will generate more revenue per and be less frequent.

<comment><AUTHOR> Do you know of any other services that implement this advertising model successfully?

Before discovering Movoxx, this was how I pictured a successful advertising campaign operating through SMS messages...

</comment></comment><comment><USER4> I don't know how movoxx works, but I can say I always get a really uneasy feeling when I think about my phone number being close to an advertiser's database. Even if they give you the ads and they never see the numbers, as your customer I can't be sure of that, and it would make me unhappy.

<comment><AUTHOR> I see what you're saying. From my first few emails with them, I have learned that they have an API for ad placement so I think all ads would be handled on my end. They did say they have a free SMS gateway though, which would obviously open up my phone number database to them, which would fall in line with your concerns...

<comment><USER5> What are you using now for your SMS gateway? I think the easiest way is to have the user choose their service provider when they provide their phone number and then you can send an email to firstname.lastname@example.com or whatever the text message email address is for their provider. I'm pretty sure that's free and easy to do. That way you could also use an email marketing platform and just make your ads tiny enough to fit in a text message.

<comment><AUTHOR> I'm using the exact system you just mentioned. What email marketing platform would you recommend if any?

</comment></comment></comment></comment><comment><USER6> Check out zong.com - I met the CEO not too long ago and they seemed to really have a clue.

<comment><AUTHOR> They definitely do seem to have a pretty nice platform built. But my product will be more passive in interaction and there seems to be a pretty big cut taken out from using their service (looks like you're only getting about 45% of the total sales).

I guess I would need to weigh out using a full featured service like that or piecing together my own for a better value.

I'm currently delivering the text messages via the phone number email that all phone's are assigned. Obviously this will have to be analyzed soon to make sure it can scale, not to mention I'm not sure if sending that many emails a day would make me spam-list worthy.</comment></comment>


What are the best law firms for startups today? - <AUTHOR> I recently formed an LLC* and now I need to set up an Operating Agreement. I want to preserve the ability to convert to a C Corp down the line, as recommended in another HN question, if I pursue VC funding.
Which law firms should I reach out to at this early stage to help create my Operating Agreement and advise on next steps? LegalZoom comes to mind, but I think the experience there is a little impersonal?
Thanks.
* An LLC was my only option for reasons I won't go into here. <comment><USER2> [https://seedlawyers.com/](https://seedlawyers.com/) "A platform dedicated to helping early-stage startups find legal help" was posted to Show HN a couple of times, e.g. [https://news.ycombinator.com/item?id=17801136](https://news.ycombinator.com/item?id=17801136) Can't tell (not affiliated) if they're still in beta.</comment>


Why did Symbolics fail? - <AUTHOR> http://dlweinreb.wordpress.com/2007/11/16/why-did-symbolics-fail/ <comment><USER2> DLW's thesis that Symbolics lost as part of the general losing of custom hardware (including all the parallel computer companies) is basically correct. Lucid on Suns was as fast as ZetaLisp on Symbolicses.

But not so fast that that alone made me switch. What made me switch was that Lisp machines (both Symbolics and LMI) were so gratuitously, baroquely complex. The manuals filled a whole shelf. Each component of the software was written as if it had to have every possible feature. The hackers who wrote it were the smartest and most energetic around. But there was no Steve Jobs to tell them "No, this is too complex." So the guy in charge of writing the pretty-printer, for example, would decide. "This is going to be the most powerful pretty-printer ever written. It's going to be able to do everything!"

I learned how to program a Symbolics myself, from reading the manuals. I sometimes suspected that I was the only person who'd ever done this-- that everyone else who knew how to use the damn things had either learned how from the guys who invented them, or had learned from someone else who had. The libraries were so hairy that I generally found it was faster to write something myself than find and understand the documentation for the built-in way of doing it.

Unfortunately this complexity persists in Common Lisp, which was pretty much copied directly from ZetaLisp. In fact, both of the worst flaws in CL are due to its origins on Lisp machines: both its complexity and the way it's cut off from the OS.

<comment> dr-steve I never used LMs, but I wrote my dissertation in CL. What a monster (this was after 5+ years of Lisp programming writing a couple of interpreters, teaching it). A baroque morass, similar in X to complexity. (Btw, if you're comparing graphics APIs, X is a heck of a lot worse than the Windows API.)

To those commenting upon C++ and its metaprogramming complexity -- a lot of your complaints here really focus on the complexity of metacode WRITTEN in C++/templates, not in the template strucure itself. Even with STL, C++ is simpler and as rich as CL. If you'll exclude the paradigm shift, of course...

<comment><USER3> C++ is simpler and richer than CL? I've worked in both languages and would say exactly the reverse. In fact, I'm stunned by this statement; I would have thought it impossible from anyone with sufficient experience in both. Our minds must work very differently!

My feeling when I started working in Lisp (CL) was that of a fish who had spent years swimming in oil, glue, and other fluids, finally experiencing what it was like to swim in water. C++, by contrast, was my wet cement (and cement doesn't stay wet for very long). So while it makes sense to me that a Lisp connoisseur would find CL overly complex compared to other Lisps, I am flummoxed by the idea that anyone would think this in comparison to C++.

Programming language preference is far from rational. Not only is there a strong intellectual bias toward what one already knows (viz. Blub), there is a strong emotional bias toward what one /likes/, and that is conditioned by personal experiences. For example, if a language is imposed on you (by a professor or manager), you'll probably dislike it and look for things to hate about it. Conversely, if you discover something for yourself and associate it with the freedom to do and create what you want, you'll probably feel strongly about defending it against criticism. Of course, we spend a lot of time arguing that our emotionally driven choices are strictly rational. Such is human nature.

I don't think anyone is immune to this, and that's why working in a language that one loves will always be more important than working in an objectively better language (by some criteria).

</comment></comment></comment><comment><USER4> 1\. This sounds an awful lot like the story of NeXt, from making their own custom hardware, to the super advanced software development environment that they eventually tried to sell by itself for other platforms. Too bad there was no struggling, mainstream company to buy them that they could have then taken over from the inside.

2\. Half way through, I was thinking, it would be fun to contrast this with ITA as bad and good examples of how to start a Lisp company with a bunch of guys from MIT. And then he says he works at ITA! So, lesson is, if you're going to start a Lisp company, don't make Lisp the product, make it your secret weapon.

</comment><comment><USER5> In all fairness, the manuals that filled a whole shelf documented a lot of major applications. On my desk right now, I have a copy of the O'Reilley book on Subversion (a source control system). I have another book on Emacs. And so on. ALL of those things were covered in that shelf.

Regarding simplicity versus complexity, please see <http://www.joelonsoftware.com/items/2006/12/09.html>. Different people want different things; you can't just provide the common 20%.

Over the last few days, I have been surveying the WWW for criticisms of Common Lisp. The two that I see most often are: (1) it's too big, and (2) it's missing so many important features like threads, sockets, database connectivity, operating system interoperability, Unicode, and so on. Ironic, no?

It is really too bad that Common Lisp was not defined as a language core, plus libraries. We did originally intend to do that (they would have been called the "White Pages" and "Yellow Pages"), but we were under too much time pressure.

There is no question that Common Lisp is a lot less elegant that it could have been, had it been designed from scratch. Instead, it had two major design constraints: (1) it had to be back-compatible with MacLisp and Zetalisp in order to accommodate the large body of existing software, such as Macsyma, and (2) it had to merge several post-MacLisp dialects, in a diplomatic process (run magnificently by Guy L. Steele Jr) that made everyone reasonably satisfied. It was quite literally a design by committee, and the results were exactly what you'd expect.

But the imperative was to get all the post-MacLisp implementations to conform to a standard. If we failed, DARPA would have picked InterLisp as the reigning Lisp dialect, and we would have all been in a great deal of trouble. (Look where InterLisp is today; actually there's nowhere to look.)

You wonder how other people learned to use Symbolics machines. Some of them took courses - we had an extensive education department. Before you say "that proves that it was too complicated", keep in mind that the system was very large and functional because that's what its primary target market wanted. We did not get feedback from customers saying "make it simpler"; we got feedback saying "add more features, as follows". I bet the people who maintain the Java libraries are rarely asked to remove large amounts of the libraries.

I'm not sure what the reference to Steve Jobs is about. Look at how many features the Macintosh has now. It takes a long time to learn all of them. Their documentation is much shorter because they don't give you any; you have to go to the book store and buy David Pogue's "The Missing Manual" books.

I admit that some (not most) of the complexity was gratuitous and baroque, but not because we liked it that way. The complexity (mainly the non-uniformity) of Common Lisp was beyond our control (e.g. the fact that you can't call methods on an array or a symbol, and so on). Some of the subsystems were too complex (the "namespace system", our distributed network resource naming facility) comes to mind.

In summary, I'm sympathetic to what you're saying, but the reasons for the problems were more involved.

\-- Dan Weinreb</comment>


Show HN: I created a window manager for Mac that isn't insanely over priced - <AUTHOR> http://ginkapps.com/splitscreen

<comment><USER2> Good luck, you have an interesting price point but currently a too limited feature set.

I'm a huge fan of Divvy and it is worth the $14. <http://mizage.com/divvy/>

Disclaimer: I am not associated with the company

</comment><comment><USER3> I didn't know there were any that _were_ insanely priced. Your headline might better be "I'm selling an app at a foolishly low price, take advantage of me while you can!"

<comment><AUTHOR> I think 99c is a fair price. It allows me to cover my development costs with a bit extra on top. The app isnt designed to make lots of money, its designed to give people an alternative to paying $10 for an app that does exactly the same for ten times the price.

</comment></comment><comment><USER4> you mean like this gpl app that does the same thing? <http://code.google.com/p/shiftit/>

<comment><AUTHOR> Unfortunately shiftit is no longer in active development. The last release was in May 2010 and I have heard reports that it does not play well with lion.</comment></comment>


Note takers, how do you take concise notes? What are your rules? - <AUTHOR>

I think it may be my perfectionist tendencies, but I find myself wanting to note down too many things. (From books, articles, etc.)
Most of the time I understand the main idea but it's the extra stuff I have a problem with. Things that add to it, are interesting, or explain why the main idea happens. I don't know what I should NOT write.
A good example might be something to do with improving productivity. I know the actionable idea may be "Do this" but I'll want to also include the psychology and reasoning behind why it affects humans.
So, note taker, how do you take your notes? Do you have any rules you follow? What do you seek to write and purposely leave out?
(Any examples of notes you've taken from an article or something would be really helpful as well!) <comment><USER2> I suck at note taking, can't do it effectively, but of people who are good note takers I think the ones who do visual note-taking seem to get the most out of it [http://austinkleon.com/category/visual-note- taking/](http://austinkleon.com/category/visual-note-taking/) [http://www.core77.com/posts/19678/sketchnotes-101-the- basics...](http://www.core77.com/posts/19678/sketchnotes-101-the-basics-of- visual-note-taking-19678)</comment>


Correcting Intel's Deep Learning Benchmark Mistakes - <AUTHOR> https://blogs.nvidia.com/blog/2016/08/16/correcting-some-mistakes/ <comment><USER3> Ok, its officially the new new thing when corporate communications types are sniping at each other :-)

Once we start seeing press releases of products that are not available yet being shown to kill the existing competition we'll know that the hype train has officially gone super-product-sonic (that is a hype wave travelling faster than the product releases can support it).

<comment><USER4> I think both Knights Whatever and DGX-1 are already there. They seem to be in "order now, receive sometime" mode.

<comment><USER5> Xeon Phi is in 23 of the Top500 supercomputer list, so it's not like they're not shipping. The next version, Knight's Landing, should be launching soon, and hopefully will have better availability.

<comment><USER6> Intel was comparing Knights Landing to Kepler, which was released a few years back.

<comment><USER7> Kepler was the default GPGPU product for many tasks until GP100 released - <USER2> GP100 still has not hit general availability yet.

Essentially, anything that needed dynamic parallelism (launching kernels from within kernels, i.e. tasks where you don't know where the difficult/interesting needles are within a haystack), advanced/concurrent scheduling capabilities, or FP64 is going to be much, much better off with Kepler until users can get their hands on GP100 cards. Maxwell is good at neither of those things - it's actually only good at specifically deep learning/neural nets. _Which is not every task within the GPGPU space_.

<comment><USER8> Nvidia blog post is about Intel deep-learning benchmarks only.

</comment></comment></comment></comment></comment></comment><comment><USER9> > Titan uses four-year-old GPUs

... as does nearly every public cloud provider. I agree with most of the article, but you can't fault Intel for benchmarking the hardware that cloud providers are actually offering.

I'm not sure what exactly NVIDIA is doing with their Tesla product line but whatever it is, it's really restricting the availability of recent GPU hardware. Even Azure's GPU instances released this month are using the Kepler architecture from 2012. It's fully two generations out of date now, and that's sad.

<comment><USER10> I mean Intel is comparing their publicly unavailable product against NVIDIA's publicly available product. Now NVIDIA is replying with the benchmarks on their (as of yet) publicly unavailable product.

I think this blog post is fair game.

<comment><USER11> Not even wrong. I have two PCs with 4 Titan X (Maxwell) GPUs and a third PC with 4 Titan X (Pascal) GPUs. Both of these systems are available today (I built them myself, total BOM about $7K), and both will destroy 4 Xeon Phi servers at Deep Learning.

The benchmark Intel presented here is as disingenuous as their infamous white paper from 2010: [http://pcl.intel- research.net/publications/isca319-lee.pdf](http://pcl.intel- research.net/publications/isca319-lee.pdf)

In comparison, a single Knights Landing Xeon Phi will be ~$7K. I know where I put my money. Caveat Emptor.

But Xeon Phi and I go way back here. They've been trying to beat my AMBER GPU code since 2013 or so. Many man years later I believe that a Knight's Corner is now ~35% faster than 2 Xeon CPUs with 1M atoms or more (source: [http://adsabs.harvard.edu/abs/2016CoPhC.201...95N](http://adsabs.harvard.edu/abs/2016CoPhC.201...95N))

Meanwhile, the CUDA code has continued to scale with the GPU roadmap and a Titan XP is arguably 9-10x faster than 2 Xeon CPUs. No data is supplied at the low-end for Xeon Phi and I think we can safely assume it's because performance there sucks. (source: [http://ambermd.org/gpus/benchmarks.htm](http://ambermd.org/gpus/benchmarks.htm))

Xeon Phi? IMO avoid avoid avoid until they start winning head to head 3rd party benchmarking fights like Soumith Chintala's fantastic convnet benchmark data: [https://github.com/soumith/convnet- benchmarks](https://github.com/soumith/convnet-benchmarks)

<comment><USER12> So, a specific MD code may or may not work well with KNL -- we don't have data. KNL looks quite attractive for other chemistry, given all the vector units, large amount of fast memory, and ability to run realistically-sized examples without the network, or potentially the network-on-chip. We'll see how it pans out.

<comment><USER11> I prefer to look at it the other way, why don't you point out an existing and important chemistry application where KNL bested its contemporary GPUs, say the best of Knight's Corner versus the best of Kepler (K40 or K80). I'm also open to Knight's Landing versus GP100 (vaporware versus no longer vaporware but hard to get)

I'm genuinely interested here because I can't find this anywhere. I don't think it exists personally.

<comment><USER12> KNL is not Knights Corner, and I have limited information on either. I'm interested in data and, more to the point, insight -- not just single benchmark numbers or specific programs, especially if they've had a lot of GPU effort and no tuning for KNL. I don't expect KNL to be particularly good for applications that aren't highly vectorizable, though the memory system may help.

If I manage to access the KNL here, I'll probably run cp2k and gromacs, though single node performance is of limited interest, and ELPA doesn't currently have AVX512-specific support.

<comment><USER11> Here's your answer for GROMACS (and it sucks)...

[http://www.prace-ri.eu/IMG/pdf/wp120.pdf](http://www.prace- ri.eu/IMG/pdf/wp120.pdf)

Even so, right now, little would please me more technologically than a competitive Xeon Phi offering, but while KNL is better than KNC, my inside info says it sucks too (it would have been a lot more interesting, just like Altera's Stratix 10, if it had shipped before GP100 and GP102).

Right now, I have more confidence in AMD GPUs right now than I have in Xeon Phi. This 3rd party benchmark is particularly interesting (and it doesn't look like anyone at NVIDIA is paying any attention to it):

[https://techaltar.com/amd-rx-480-gpu-review/2/](https://techaltar.com/amd- rx-480-gpu-review/2/)

Sure, NVIDIA is still in the lead, but not with the ~10x margins they used to have over AMD.

Finally, I figuratively feel like punching the next person who makes the BS scaling argument over raw performance. GPUs scale too if they're coded correctly. And cloud datacenters are the worst place for that given their craptastic ~10 Gb/s interconnect subject to arbitrary network weather effects.

Or butchering Seymour Cray: Your life depends on winning a race, would you bet your life on a 1,350 HP Venom GT or on 20 179 HP Scion FRSs? I mean collectively that's almost 3600 HP, right? Except it's even worse because for GPUs vs CPUs, it's like they priced the Scion FRS like a Venom GT and vice versa.

I wish you luck finding Xeon Phi winning anything but synthetic tests against yesterday's news:

[https://www.xcelerit.com/computing-benchmarks/libor/intel- xe...](https://www.xcelerit.com/computing-benchmarks/libor/intel-xeon-phi-vs- nvidia-tesla-gpu/)

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> @imaleppert Agree getting the details on the testing would be helpful. I think NV was more pointing out that Intel were putting their latest against NV's oldest. It'd be like testing a RX480 against a GTX660. What's the use of that?

@modeless the new Azure instances have M60's or you can purchase a 1080 or new TitanX which are both available (although stock has been tight).

[https://azure.microsoft.com/en-us/blog/azure-n-series- previe...](https://azure.microsoft.com/en-us/blog/azure-n-series-preview- availability/)

<comment><USER9> Yeah but Azure is also introducing new K80 instances (based on four-year-old Kepler) becuase the M60 is worse in memory capacity and memory bandwidth, and not much better in FLOPS. NVIDIA is just not putting their best hardware out there for cloud providers to use.

Sure I can buy a Titan X for myself (already have), but I can't rent a hundred, or even one, on EC2 or Azure or GCE. And I can't get a P100 yet at all. I don't want to hear NVIDIA claiming unfair benchmarking and citing P100 numbers until P100s are actually available either to buy (and ship immediately) or in the cloud.

</comment></comment><comment><USER14> Why don't they provide a link to their testing methodology? They need to back up their claims (on both sides) with the actual configuration, all versions, and sample datasets for people to independently verify.

A docker container that runs their performance suite would be ideal.

<comment><USER15> Except that Docker containers play terribly with virtualization solutions. Still, some sort of configuration/infrastructure-as-code would go a long way.

<comment><USER16> What about NVIDIA-docker?

[https://github.com/NVIDIA/nvidia-docker](https://github.com/NVIDIA/nvidia- docker)

</comment></comment></comment><comment><USER17> `each with one or two sockets of less-capable processors, like Xeon Phi.`

Corporate smacktalk...

</comment><comment><USER18> I do not usually believe in any vendor provided performance benchmark unless I fabricated it myself. On a more serious note, benchmarking is pretty hard and people usually discount things that seem minor until you found it the bottleneck, like the interconnect in this case for example. Another problem with synthetic benchmarks is that you can always optimize for your exact use case and it usually yields to pretty good improvements, comparable to buying faster equipment. The ultimate quiestion which is more cost efficient, buying faster CPU/GPU or hiring a performance expert.

</comment><comment><USER19> I'm a bit surprised that Nvidia mentioned nothing about performance per watt in their reply.

<comment><USER20> Or that they didn't mention cost for performance. They equated 4 Xeon Phi servers to ONE DGX-1. The DGX has a 140k price tag.

<comment><USER5> That was for performance scaling, not raw performance. Although 140k probably includes a nice interconnect.

</comment></comment></comment><comment><USER21> I actually came away from the article thinking, heh Phi looks pretty good.

</comment><comment><USER22> Well that was a pretty concise explanation of mistakes. Benchmarking is indeed difficult to get right.

<comment><USER6> Especially when you compare on purpose against your competitor's 4 year old product.

</comment></comment><comment><USER23> They do not mention the version of Caffe used to test the Intel systems, Intel claims its numbers based on an optimized branch of Caffe and not the public (BVLC) version

</comment><comment><USER20> Shots fired, wow Nvidia is pissed that Intel is trying to edge into the market with their Phi series.</comment>


FedEx, Walmart Are Being Investigated for Punishing Workers That Take Time Off - <AUTHOR> https://www.vice.com/en_us/article/v7gpq8/elizabeth-warren-investigates-report-that-fedex-walmart-punish-workers-for-taking-time-off <comment><USER2> I may be in the minority here, but no-fault attendance scores seem preferable to having companies investigate whether employees were 'actually sick' or had other 'legitimate' reasons for absence.

It also seems like companies with time-sensitive and seasonal businesses need to have some way of penalizing employees who frequently take leave during critical times (just before major holidays).

<comment><USER3> Didn't take the time to read the article, but ~1 year ago Walmart added a system where all employees would earn a few days/year of time off that did not count against an attendance score, and could not be denied by management. You can still request time off and get it approved/denied, and you can still take time off without permission and accept the hit to your attendance, but there is this other way as well.</comment></comment>


An Open Letter to Mark Zuckerberg on Net Neutrality-By Zuck's Host in Chandauli - <AUTHOR> http://www.livemint.com/Opinion/NHdyVN9njNBvUtisGhTXSI/An-open-letter-to-Mark-Zuckerberg-on-net-neutrality.html

<comment><USER2> To give some context about this, Mark Zuckerberg recently responded to the Net Neutrality debate currently raging in India: [https://www.facebook.com/zuck/posts/10102033678947881](https://www.facebook.com/zuck/posts/10102033678947881)

He starts off with a warm-fuzzy-feel-good story about his visit to Chandauli, a village in North India and then goes on to describe how Internet.org would realize the dreams of the people he met there. The submitted article is a response to that by the person from the foundation who hosted him.</comment>


The future of advertising: Many, lightweight interactions over time - <AUTHOR> http://www.thinkoutsidein.com/blog/2012/03/many-lightweight-interactions-over-time/

<comment><USER2> Seems like the advertising industry is about collapse. This article is trying to redefine the job at hand and lower expectations.

tl;dr No one is clicking through anymore.</comment>


Ask HN: Is it too late to start learning to code? - <AUTHOR>

I'm 23 and have tried to learn coding a few times in my life without any success. I want to try again. I have a design for something I want to build that I think will make a difference in the world and want to start coding it. I know there must people in this community who are 'late bloomers' and started learning in their 20's or maybe later. If you are one of those people, can you please share some advice and guidance on how you got started and how you kept going?<p>It would mean the world to me. Thanks. :) <comment><USER2> For the rest of your life, no matter how old you are, if you want to do something then fucking do it.

Here endeth the lesson.

</comment><comment><USER3> You're 23. It's not too late to start doing anything. I was a sysadmin who started programming and building things around age 30. Last year I quit my senior dev position in S.F. to freelance and now I make more money working only 20 hours per week. So yeah, it's not too late.

</comment><comment><USER4> At the age of 24, I decided to learn to code so that I could create stuff I thought of. Since then I have learned 7 different languages, created several awesome sites, projects, games, and frameworks. I learned it all entirely through Google Search.

</comment><comment><USER5> Seriously? You're only 23. Just do it. You literally have decades ahead of you to enjoy coding. When I saw the headline I thought I'd be responding to a senior citizen...in which case my response would have been the same.

</comment><comment><USER6> I took a programming class (Fortran IV) at 18, and found it interesting enough, but frustrating, given the i/o of the day (hand over punch cards at the operators' counter, wait for the cards to come out a chute with the printout wrapped aroud them.)

At 31 or 32, I discovered that there were things that I could do with a computer that would make my life (and many others') much easier, and I taught myself assembler, since I didn't have acess to a compiler or interpreter. Then I went back to school....

I would suggest taking a few classes. Get the hang of data structures and algorithms. Learn a few languages. The resources that are available to you, for free, are astonishing. If you are using a PC, you can download systems for Python, Scheme, and Common Lisp, you can get Microsoft Visual Studio. If you are using Linux, you can get all manner of systems installed

Good luck, and enjoy yourself.

</comment><comment><USER7> Programming is about solving problems. Is it too late to start solving problems? No. It just takes dedication and perseverance. The learning curve is exponential so the first few months will be slow and confusing, but once you understand the basic concepts you'll start to become more productive.

</comment><comment><USER8> I had some BASIC programming classes in high school, loved everything about it. Although I didn't follow through with programming at the time, chose to be a recording engineer/producer instead. many failed starts with java and html books in the 90's. I find myself 45 and with a passion to really learn this time. I started learning Processing 5 months ago with the only online resources and a book.

It's absolutely one of the best things I've set out to learn. I find so many similarities between music engineering and programming it's kinda crazy. As many have said, It's never too old to learn... anything.

</comment><comment><USER9> I agree with all of the comments here. This is the perfect time for you to learn how to code, especially if you already have an idea/project you want to work on. It won't be easy, but stick with it and you'll get there sooner rather than never.

There are a lot of different resources (free and paid) that will teach you the basics/fundamentals of coding and more...here are a few of the well-known ones to get you started:

1- www.udacity.com 2- www.udemy.com 3- www.lynda.com 4- www.codecademy.com 5- www.teamtreehouse.com

</comment><comment><USER10> Pleeease! You are only 23! I'm 42 and learning Erlang and Haskell! As it is commonly said, never its too late.

Maybe you didn't choose the right programming language.

</comment><comment><USER11> If you have a hard time finding a structured way to learn and staying motivated, you could try out Udacity's courses at www.udacity.com. They're set up so that they're interactive, and they have courses for people of all skill levels. You can start with CS101 if you feel that that's right for you and then move on to whatever else you're interested in!

</comment><comment><USER12> Just do it. Your biggest problem is that you feel the need to come on here and ask if it's what you should do. It is. Now do it.

</comment><comment><USER13> I started at 24, probably same boat as you.

I wrote a blog post about how I learned: [http://coryliu.com/post/42990233356/how-i-learned-to- program...](http://coryliu.com/post/42990233356/how-i-learned-to-program-aka- the-surest-way-to)

</comment><comment><USER14> Why people think that with age the ability to learn new things diminishes?

You can ALWAYS learn new things, no matter how old you are.

I learned programming after the age of 25.

</comment><comment><USER15> I am 33 and started to learn a month ago, I need some time to realise if I "bloom". I am starting with front-end developing, it looks much more rewarding to start of...

</comment><comment><USER16> Yep, it's too late. In fact, you shouldn't try to learn anything anymore, especially if it involves fulfilling your dreams :)

</comment><comment><USER17> Before I opened this post I was expecting you to be very old. Get started today. Honestly, it's never too late.

</comment><comment><USER18> If you are serious about learning, and I mean really learning, send me an email.

</comment><comment><AUTHOR> thanks everyone. I'm going to give it another go. I'm overthinking it. No matter what happens, I'll always have HN for feedback and guidance.

</comment><comment><USER19> It's never too late to start learning anything.

</comment><comment><USER20> Noooooooo. Just start right away.....

</comment><comment><USER21> I doubt age has anything to do with it. Poor fit with your learning style is far more likely.</comment>


BMG: A Production Ready Relational Algebra in Ruby - <AUTHOR> https://github.com/enspirit/bmg <comment><USER2> This is great looking! I don't use Ruby, but it's good to see libraries like this. It still surprises me how few libraries implement the relational algebra. Building on proper mathematical theory that's conducive to programming languages really removes a lot of impedance mismatch and leaky abstractions over ORM's.

<comment><USER3> The Rails ORM (ActiveRecord) is built atop of a relational algebra gem (arel, which was eventually pulled into the AR library). I'm pretty sure the arel functions are considered private API, so it's not necessarily best practice, but you have the option to call those functions directly when you find the ORM lacking and don't want to pass around raw SQL.

<comment><USER4> AFAIK Arel is not a real algreba, since it does not really allow composing relations the way we want.

More info here: [http://www.try-alf.org/blog/2013-10-21-relations-as-first- cl...](http://www.try-alf.org/blog/2013-10-21-relations-as-first-class- citizen)

</comment></comment></comment><comment><USER5> Honest question: does it make sense to learn Ruby in 2020, with the hopes to actually have a job with it for the next 5 years? I want to leave JS world for something different, and I'm evaluating Elixir, Python and even PHP. What I'm looking to work on are basically SaaS sites, nothing too crazy. Would appreciate any thoughts.

<comment><AUTHOR> My personal experience is that the ruby job market is quite strong. For context, I've been a ruby developer (among other languages) for 5+ years.

Have you tried looking for some ruby work?

</comment></comment><comment><USER6> If you are using Java or a JVM language (e.g. Ruby, Clojure, etc.) it may be worth checking out Apache Calcite [https://calcite.apache.org/](https://calcite.apache.org/).

</comment><comment><USER7> Is there any such library for Go? I was searching one for simple algebra stuff while back and only prominent math library for Go seems to be gonum[1] which for lack of better term seemed 'too large' for my purpose.

I didn't look properly or is Go's math scene not that great?

[1][https://github.com/gonum/gonum](https://github.com/gonum/gonum)

<comment><USER8> Note that this library implements relational algebra, a specific set of set- theoretical operations on an object called a relation — not general-purpose mathematical algebra.

</comment></comment><comment><USER9> Neat! One obvious question: Is this intended to integrate with Rails?

<comment><USER4> (lead dev here). Not _intended_ , as I don't use Rails myself. But I would of course support every PR making Bmg compatible/great with it.</comment></comment>


The bandwidth of a Boeing 747 and its impact on web browsing - <AUTHOR> http://blog.cloudflare.com/the-bandwidth-of-a-boeing-747-and-its-impact

<comment><USER3> Hey, everybody bitching about how close a call it is for 1TB, just make it 10TB. There. Problem solved. 100Mbps internet link now no longer even remotely close. Stop nitpicking, it's pointless. As the amount of data to move increases, moving physical storage around wins increasingly more, nitpicking won't change that.

<comment><USER4> The whole comparison is flawed/irrelevant anyway. Their point is valid irrespective of the airplane comparison in that having CDN servers closer indeed mean faster downloads, but I could have a perfect 100Mbps connection to the building across the street and try to download a 1TB file, it would still be insanely faster to travel to the other side of the road and lug the hard disk back.

<comment><USER5> Of course it's relevant... Any time you're considering moving a large chunk of data to a remote datacenter, it makes sense to do the math and determine whether it's faster/more cost-effective to move the data there digitally or physically. Why do you think AWS allows you to ship them your portable storage? <http://aws.amazon.com/importexport/>

</comment></comment></comment><comment><USER6> This was actually utilized at some point in Usenet's history to transfer posts between the US and Australia, as I recall. Googling isn't turning up citable references, so I'll call this apocryphal until verified. I do find traffic stats showing ~4GB/day posts as of 1996 (<http://en.wikipedia.org/wiki/Usenet#Technical_details>).

Australia for a long time was a worst-case major economy networking situation with few, slow, and high-latency links (and still suffers from high costs due to limited links and telco monopolies).

For an understanding of how limited international data links were as recently as the early 1990s, one of the key public data links between the US and Europe at the time was a 9660 baud link according to a dated, but fascinating, compendium of networking at the time (lost somewhere in the Krell stacks, I'll dig a reference on request).

Jon Bentley's classic _Programming Pearls_ discusses the problem of transferring graphical data images between Mountain View and Santa Cruz, California, in the 1970s. The high-bandwidth, cost-effective, sufficiently reliable solution? Carrier pigeons. Really.

<comment><USER7> Wasn't there an RFC entitled "IP datagram transport via avian carriers", or something similar?

<comment><USER8> Yup, RFC 1149: <http://www.ietf.org/rfc/rfc1149.txt> Extended by RFC 2549 to add QoS: <http://tools.ietf.org/rfc/rfc2549.txt>

But the best thing is that this was actually implemented at least once: <http://www.blug.linux.no/rfc1149/>

<comment><USER9> and there is DTN using IP over avian carriers: <http://dl.acm.org/citation.cfm?id=1614234>

</comment></comment></comment></comment><comment><USER10> In my day we called this Sneaker net.. The bandwidth of a floppy vastly out paced modems back then.. :)

<http://en.wikipedia.org/wiki/Sneakernet>

Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. —Tanenbaum, Andrew S. (1996). Computer Networks. New Jersey: Prentice-Hall. p. 83. ISBN 0-13-349945-6.

<comment><USER11> "Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway."

How far we've come. Now we use 747s as the ironic competitor to digital transfer.

<comment><USER12> A Volvo full of LTO-5 tapes still has a very respectable bandwidth. LTOs come in 20 tapes boxes, about 10 litres in volume; a Volvo break with backseat down fits about 1500 litres, so you should be able to fit 150 boxes of 20 tapes = 150 x 20 x 1.5 = 4.5 Petabytes.

</comment></comment></comment><comment><USER13> There are actually special file transfer protocols for high-bandwidth high- latency situations where you want to transfer gigabytes/terabytes across the world.

The main example I've heard is special effects companies like Weta Digital in New Zealand wanting to deliver video to the US across a 200ms latency link.

There are a few open protocols in various stages of development or you can pay for a product like FileCatalyst <http://www.filecatalyst.com/> ( some interesting case studies on their website ).

<comment><USER14> "There are a few open protocols in various stages of development"

Can you, or somebody else please name some? It always irks me a little when someone just mentions that "these projects" exist, and then fail to mention any of them.

<comment><USER15> We had a discussion about this over on Server Fault (<http://serverfault.com/q/111813/7200>) and UFTP (<http://www.tcnj.edu/~bush/uftp.html>), UDT (<http://udt.sourceforge.net/>), and Tsunami-UDP (<http://tsunami-udp.sourceforge.net/>) all came up as viable open source offerings (along with FileCatalyst and Aspera in the commercial space).

</comment></comment></comment><comment><USER16> This is why Amazon allows you to physically send them disks to upload data. <http://aws.amazon.com/importexport/>

</comment><comment><USER17> I came here to say that my father works for a company called MTU Areo Engines as an FAA RS-DER. What that means is he can write and design repairs for the hot-sections of most engines. Why this is interesting, and related to this story is something that comes along with that.

All engine data that the pilot can see is available realtime on the ground at the repair stations.

What this means is that an engineer can call up a GE-CF6 in the second position on the starboard wing and see how it is performing. While that might not sound all that interesting, there are literally tens of thousands of measurements taken a bunch of times a second and transmitted to the ground for each engine.

For instance, if the flight was 10 hours (and every package was 100 Kb) they'd send about 35GB of data per engine. This is just the engine measurements. This does not include avionics, communication, position.

Most of the information is propriety for how they store, transmit, and calculate these things so I do not know exactly how this works. I have seen one of these workstations and it is really impressive to know the exact RPM of the number two engine compressor is spinning at in real-time of a flight crossing the Atlantic.

Also, you can two-way text-message with the pilot from this workstation to pass perceived information about the engine performance.

<comment><USER18> FAA RS-DER: Federal Aviation Administration Repair Specification Designated Engineer Representative

GE-CF6: A model of turbofan engine

</comment></comment><comment><USER19> I wonder how suitcase-full-of-truecrypt-disks compares to pgp/tls transport of data over the internet in terms of probability of detection, interception and access.

I recall the US now has a relatively free hand in confiscating storage media for analysis/cloning during border crossings, and you might not get your copy back for a couple of months if they take a serious interest.

So, even though the aeroplane travel infrastructure itself is quite reliable, there are points of failure that would require another copy of the data to be created and shipped, with a much greater latency (It might take days to purchase more disks, transfer all the data onto them, find a willing courier, book their flight, etc).

</comment><comment><USER20> A friend at a botanical institute sends and receives a 500GB hard drive weekly, exchanging data across the Atlantic. Back of beermat calculations at the time demonstrated huge efficiency compared to similar cost broadband solutions!

</comment><comment><USER21> Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. — Andrew Tanenbaum

<comment><USER22> Eyup, not sure why this blindingly obvious truth has to be rediscovered once in a while.

This is also how astronomers work, radiotelescopes produce terabytes of data per day (the LCH produces ~15PB/year) so sending that over the wire makes absolutely no sense. Instead, they ship external disks for smaller datasets[0] and for bigger ones some go as far as shipping complete NFS/CIFS servers[1].

[0] [https://science.nrao.edu/facilities/evla/data- archive/data-s...](https://science.nrao.edu/facilities/evla/data-archive/data- shipment)

[1] <http://queue.acm.org/detail.cfm?id=864078> middle of the article

</comment></comment><comment><USER23> This also makes the assumption that you'd have zero time copying the data once you arrived in London. More realistically, you'd probably make a copy. In this case, you need to also take into account the amount of time copying the files at the other end. Depending on your external drive's interface, this is a non- trivial amount of time.

<comment><USER24> Well if we're speaking more realistically, you'd use this for shipping 100tb worth of disks (which would fit in your luggage in both size and weight), and it would blow away just about any internet link you could lay your hands on with a remotely similar budget.

That is a great point though. A 4TB drive, hooked up to your internal SATA port, will do about 130 megabytes per second, which is around 9 hours for a full copy.

Drive speeds increase linearly with track density, but drive capacity increases quadratically, so this is only going to get worse. Hard drives are the new tape.

<comment><USER25> For a point of reference -- LTO5 pushes data at up to 140 megabytes per second. (But Tape has the same problem -- Capacity increases faster than speed)

</comment></comment></comment><comment><USER26> > Boeing 747 has a bandwidth of 1 TBps.

Hmm, not sure about those numbers.

Say we limit it to this trip, at 10 hours.

A Boeing 747 can carry ~30,000 kg (<http://www.boeing.com/commercial/747family/pf/pf_facts.html>) You can get get 3TB 3.5' drives now, and they weight about 1kg ([http://www.amazon.co.uk/WD-Caviar-Green-WD30EZRX- SATA-600/dp...](http://www.amazon.co.uk/WD-Caviar-Green-WD30EZRX- SATA-600/dp/B0067LA6CS)), so 30,000 _3TB is 90,000TB in 10 hours.

10 hours is 36000 seconds, so that means that the speed of the data on a 10 hour flight is actually 90000/36000 = 2.5TB/sec.

That said, if we were to account for the next generation jump ([http://www.techspot.com/news/47860-seagate-60tb-hdds- possibl...](http://www.techspot.com/news/47860-seagate-60tb-hdds-possible- with-next-generation-storage-tech.html)) so 60TB for about a kilo, we'd get 60_30000 / 36000 = 50TB a second.

Obviously, the longer the flight, the more inefficient this transfer becomes.

<comment><AUTHOR> I'll modify the post to make that point. You are correct that the real bandwidth of a single 747 is more like 220 Mbps.

<comment><USER27> In either case, it's an interesting exercise but missing some key calculations...

When I transfer data from machine to machine, the data is immediately "usable" on the receiving machine.

When transferring data by physical means, the data needs to be brought to the final point of consumption, and possibly loaded into the host machine if it is not in some form that could be immediately connected/mounted.

So, the real comparison would be transferring data from the host machine to some form of removable media, taking that media to the airplane (early enough to meet all the pre-clearance stuff), waiting for the plane to takeoff, fly, land and then handoff the physical media at the other end.

In many cases, I'm sure the plane is still faster, but if we're talking about its theoretical bandwidth we need to look at the entire "trip" of the packets/data, not just one segment of it.

<comment> excuse-me Unless you take the rackspace crates off the 747 and hook them up at the destination end. But compare that to ordering blank disks, having them delivered, configuring them and then copying network data onto them.

</comment></comment></comment></comment><comment><USER28> I'm assuming 747 option is using carry on luggage otherwise at one stage the underlying infrastructure changes to packet transfer by baggage handler which, in my experience, can be an incredibly high latency protocol with a possibility at times of zero error correction.

</comment><comment><USER9> There is whole field of study called delay/disruption tolerant networking (<https://tools.ietf.org/html/rfc4838>), which uses data mules or physical mobility to overcome connectivity outages in constraint environment ([http://en.wikipedia.org/wiki/Routing_in_delay- tolerant_netwo...](http://en.wikipedia.org/wiki/Routing_in_delay- tolerant_networking)).

However, there is one thing that the delay (in the cited article) does not take into account is that the flights do not take-off all the time, so if you have data ready to be sent and no one to take it, then there is waiting delay before it can be sent. so if there is only one flight in a day then some data will have to wait between 24+10h to 0+10h before it can arrive. Since storage cost is very low, one could ship a lot of data but this data cannot be delay sensitive else.

on the other hand, using TCP one can prioritize data and send more important data before the rest. Of course there may be intrinsic problems with TCP receiver window and one of the proposal is to increase the initial window but the research community and the IETF is a bit split on the real benefits of it. The fear is that it may cause a congestion collapse. Moreover, the current buffer-bloat in the routers and the increase in the window size may only aid in causing a congestion collapse ([http://tools.ietf.org/html/draft-gettys- iw10-considered-harm...](http://tools.ietf.org/html/draft-gettys- iw10-considered-harmful-00)).

</comment><comment><USER29> 28 crashes a day is a lot. But author assumes that storage magically appear inside 747 during takeoff add disappears during landing. But if we will include this strange institution called customs then 28 item loss per day won't be unrealistic at all.

Items detained indefinitely in the customs, items detained until bribe is paid, items lost, items stolen during load/unload, items broken (hits, pressure changes, magnetic gateways and handheld scanners etc.). Lots of problems in general, all are multiplied if your country customs and airport personnel are incompetent.

</comment><comment><USER30> While it makes for a good blog, this idea ignore cost ($1200? to transfer 1tb data) and it ignore that you still need to get the 1Tb _off_ the drive once you get to London.

And you need to sit in London traffic.

<comment><USER31> Typically drop shipping something on an airline is substantially cheaper than a full fledged seat fare. You go to the counter, say you want to ship something, they toss it in the belly and your friend picks it up at the counter at the destination. We used to do this all the time with equip in Alaska.

</comment></comment><comment><USER32> Old example based on the one in Tanenbaum's book (a little van against a 747, but that's the same...) [http://books.google.it/books/about/Computer_Networks.html?id...](http://books.google.it/books/about/Computer_Networks.html?id=Pd-z64SJRBAC&redir_esc=y)

</comment><comment><USER33> I enjoyed the premise behind this post, and perhaps this is pedantic or I am simply confused, but why does the it conclude with, "The speed of light really is a limiting factor in downloading!" and talk about what Cloufare does to "fight the speed of light" after smartly explaining network latency at the layers _above_ the physical one?

I guess fighting the speed of light sounds cooler, but that's not really the problem they are solving or speaking too, is it?

*edit just for fun: 5MB data storage transfer, 1956 <http://mlkshk.com/p/AW8A> (img)

</comment><comment><USER15> All this makes me think of is "In order to transmit the same amount of information on paper, they would have to arrange for a 747 cargo freighter packed with telephone books and encyclopedias to power-dive into their unit every couple of minutes, forever."

<comment><USER34> One of the wonderfully vivid images that so enamored me with Snow Crash as a young college student.

</comment></comment><comment><USER35> OK... but the nice thing about computers and networks is that you don't _necessarily_ have to transfer 1TB globs of data to from SF to London to use it or compute upon it. There are computers in SF too. And of course, if you're talking about media files you can't watch more than one at a time and anyway you don't need "random access" to the media file (you can stream them).

</comment><comment><USER36> Does anyone know why the system doesn't send over 10 packets at once, then check those packets, then ask for the packets that are not received.

Wouldn't doing it that way cut down on the latency signifigantly?

(or send the entire file, 1000s of packets) determine which don't make it and just ask for those?

This isn't my area of expertise, so I'm just curious.

<comment><AUTHOR> Two things:

1\. TCP wasn't designed like that. It uses a rather simple scheme of sending back a byte count telling the other side up to what byte number it has received. So there's no provision for telling the other side that a bit in the middle is missing.

2\. We don't send 1000s of packets at once because doing so would cause congestion on the Internet. So there's a whole subsystem in TCP that tries to determine the capacity of the link between two points and avoid creating congestion.

3\. But the scheme you describe was actually added to TCP and it's called SACK: [http://en.wikipedia.org/wiki/ACK_(TCP)#Selective_acknowledgm...](http://en.wikipedia.org/wiki/ACK_\(TCP\)#Selective_acknowledgments) Note that the original article isn't talking about the case where packets are being lost.

<comment><USER37> Note that SACK is enabled by default on all modern TCP implementations, so #1 really isn't an issue.

And the 1000's of packets thing isn't quite right either. With windows scaling (another option pervasively enabled) and a fat pipe (it does take some time for the algorithm to ramp up that far), you can have about 1GiB in flight on the wire at any time, that's hundreds of thousands of packets.

</comment></comment></comment><comment><USER38> This calculation works if you can directly plug in the disk and start using the data that way without copying. If, when the disk arrives, you still have to plug it in and copy it at 50MB/s via an IDE cable then that has to be factored into the calculations.

</comment><comment><USER39> _One thing British Airways ensures while flying the 1TB of data is reliability._

Modulo baggage reclaim, of course.

</comment><comment><USER40> The analogy would have worked better if you used a Fedex jet instead of a BA one. That said, I don't think the latency of transferring large data is the best selling point for Cloudflare, especially since the acks are asynchronous and good load balancers perform quite well over long latency links.

A better selling point for distributed delivery is the round trip when you deal with multiple transactions, for example requesting a web page and then the included objects, or submitting data and waiting for a response.

</comment><comment><USER41> Given that the OP doesn't factor in delays due to transit to and from the airport, clearing immigration and customs, and other delays, a more apt comparison would be the relative bandwidth of transferring 1 TB from one part of town to another. At that point, you're only facing traffic. If it takes 1 hour from one part of the a town to another part of the same town (4 if you are in LA), then the relative bandwidth is much more remarkable.

</comment><comment><USER42> Two things nobody is considering: price and time to transfer data to the disks. I'd rather pay for a 100Mbps connection than 10 hours of jet fuel.

</comment><comment><USER31> See, I always told my friends it was faster to burn to CD and then toss the disc their way. I mean, the last time I had to make a null modem cable it took me hours just to find the right tools!

<comment><USER43> With friends we do the same when we meet to LAN play. We copy the games into a USB stick and pass it around. It's waaaaay quicker than downloading from steam _, and quicker than copying through the local net.

_ If you know the right folder you just copy it there, buy the game if you didn't already have it, and just wait a little for it to check the integrity of the install.

This article has me almost wishing to start a data courier service. Flying around the globe transporting data.

</comment></comment><comment><USER44> Reminds me of the broadband vs carrier pigeon story a few years back - <http://www.bbc.co.uk/news/technology-11325452>

</comment><comment><USER45> What about environmental impact of delivering data via plane vs. cable? It's probably too complex to compute precisely, though.

</comment><comment><USER46> s/bandwidth/throughput/g

</comment><comment> excuse-me Easy to beat a 747

A rackspace portable storage center fits 12,000 hard drives (=36Pb) of data, into a 40ft shipping container.

A container ship like the Emma Maersk can carry 15,000 TEU (20foot containers) across the Atlantic in 5days

So 12,000drives * 3Tb * 7000 containers = 250 million Tb, and it takes 5days = 450,000 seconds

So a data rate of 250,000,000Tb/450,000s = 600 TBytes/s

Although you do have to watch out for pirates!

<comment><USER47> A Boeing 747 LCF <http://en.wikipedia.org/wiki/Boeing_747_LCF> can cross the Atlantic in 3.5 hours and carry 180,530 kg.

3TB drives weigh ~1 lb = 180,530kg/1lb * 3Tbytes / 3.5 hours in seconds ~= 235TBytes/s which is rather close. Especially when you consider how long it takes to load those container ships and how limited the destinations are.

PS: As to pirates your 84 million drives at 150$ a piece would be worth ~12.6 billion dollars which might actually be a tempting target even if you can only get 10% of their actual value in ransom.

Edit: Even this is small next to MicroSD cards. ~64GB/gram = 29Tbytes/lb = 2,270Tbytes/second across the Atlantic for ~12.64 billion.

<comment><USER48> How did you do get your result? I calculated 758 Tbits/second [1] or 95 Tbytes/second [2]. I think you or I may be missing a conversion somewhere.

[1] [http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tb...](http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tbytes+%2F+%283.5hours*3600sec%2Fhr%29)

[2] [http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tb...](http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tbytes+%2F+%283.5hours*3600sec%2Fhr%29+in+terabytes%2Fsec)

<comment><USER47> Heh, I just did it again and got your numbers so I think I made the mistake.

PS: It's off by ~2.4 so probably did the kg -> lb conversion twice + some rounding errors.

Let's see if I can redeem myself Heathrow Airport averages a little over 1300 flights per day. If 1 flight can handle 1.2 million TB's [http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tb...](http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tbytes+) then Heathrow Airport in theory is a router than can handle 17,965 terabytes / sec [http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tb...](http://www.wolframalpha.com/input/?i=180%2C530kg%2F1lb+*+3Tbytes+*+1300+%2F+24+%2F+60+%2F+60+in+Tbytes) (Not that they are setup to handle freight, but at least flight time is not important only throughput.)

<comment> excuse-me But the rackspace containers you just need to hook up power and water - <USER2> you are done!

</comment></comment></comment></comment></comment><comment><USER49> This doesn't take into account UK customs. I recently send a nicely packaged 1Tb drive to my place in tb uk by express mail from San Francisco. The drive arrived in customs two days later, sat there for a week, then they mailed a postcard saying the import duty was 550 GBP. Now we are up to about the third week... Obviously I didn't pay that so the drive went back, taking over two months. Using scp would have been far quicker...

So next time you're considering physical transport include the time spent in the destination country's customs, with their highly incompetent and inefficient crews.

</comment><comment><USER50> I find this a rather misguiding post.

First off, do you really think it's more practical to ship data per airplane than a simple cable that's already connecting you? Not that bandwidth is completely for free, but most of the time you're paying a flat fee and transferring this 1TB won't cost you anything extra. I bet that Boeing is quite a bit more expensive.

Secondly, at some point the article assumes the 'transfer rate' of the Boeing to be 222Mbps. While this might be true for this practical example--let's ignore that you actually need to get the data on the plane in the first place --I bet that you could load it up from bottom to top and reach practically a speed of many gigabits ifnot terabits per second.

Thirdly, the Boeing flies with physical stuff. If the plane crashes, the data is lost. You would need backups, which take time. Over the internet it's not even possible to transfer the original medium; you'd need to first transfer and then delete it.

Fourthly, while it might be true that a packet gets corrupted 28 times during this 1TB transfer (I myself do not have statistics on this, but let's assume), on the internet you can retransmit a packet. Try retransmitting that one sector on the harddrive in under a second.

Fifthly, the article goes on about how TCP needs to wait for an acknowledgement (or 'ack' in short). I've made this mistake before when choosing for UDP instead of TCP because "it would not have to wait for acks and thus make my game much faster". It's wrong because there is a so-called congestion window. I do not know the exact figures, but it's like this: The server sends packet one. Packet one is in transit. The server sends packet two. Packet one and two are in transit. The server sends packet three. Packet one arrives. Packet two and three are in transit. The server sends packet four; the client sends ack 1. Packet 2, 3, 4 and ack 1 are in transit. Etc. The thing is that the server can send up to X packets ahead, without receiving an ack.

Sixthly, the article finally concludes that this is why you should choose for Cloudflare--so it was an advertisement after all. That explains why it's told in this story-like way and why it contains so many technical errors.

<comment><AUTHOR> To address some of your points:

1\. Yes, physical shipping is more practical than using the Internet for large transfers and this is done frequently. For example, Amazon Web Services offers the possibility to send them disks: <http://aws.amazon.com/importexport/> Shipping of disks is also common in the movie industry. For example, a lot of digital movie distribution to cinemas relies on shipping of hard drives.

2\. I used a statistic of 0.1% packet loss on the Internet and 28,000 flights per day to arrive at 28 flights lost per day.

3\. There are two interacting things happening in TCP: the congestion window (which limits the number of outstanding _packets_) and the receive window which limits the number of outstanding bytes. This blog post does not address the congestion window at all and I assume that the receive window can always be filled. Thus is _overestimates_ what might be possible on a real link.

4\. For very large transfers there are improvements that can be made using UDP and there are protocols that do just that. See, for example, TixelTec.

<comment><USER22> > Shipping of disks is also common in the movie industry.

As well as sciences based on massive amounts of data (astrophysics and particle physics for instance, where experimental systems can produce terabytes per day), financial data analytics, or petroleum seismic surveys.</comment></comment></comment>