Tell HN: Your social widgets are losing you visitors right now - <AUTHOR>

Recently a lot of sites are getting slow for me. The reason? They add a lot of social submit widgets that use non-async scripts.
If you see a substring <script src="http:// or <script src="https:// in your HTML source - you are killing your site slowly.
Using Twitter's official ReTweet button? You've slowed your site down by 60 seconds per each page (I don't know how many people are affected by this hiccup that lasts more than 5 days now for me, but you can easily fix it for everybody, see below)
Just to be clear. I'm on 35Mbps line in Russia near Moscow (4.3MBytes/s - very fast! 4ms ping to national traffic exchange point in Russia).
Yet some sites load up to 2-5 minutes for me? Why?
According to Chrome Dev Tools I receive main blog content, including all images within 1-2 seconds. (It's a 35Mbps!), but I don't see anything from your site on screen (even though it has finished loading), because...
"platform.twitter.com" responds in 49-62 seconds! Uses <script src="http://... for their "retweet" button. Your site is STUCK until "platform.twitter.com" loads (1 minute).
Facebook's CDN responds within 30-50 seconds. The site doesn't load until it's loaded.
"www.stumbleupon.com"'s button loads in 20 seconds.
I'm not sure what the problem is, but I can tell you for sure - it takes minutes to load some sites with those buttons, it takes less than a blink after I add "127.0.0.1 platform.twitter.com" and others to /etc/hosts (that's not a way to solve it, it's a way to diagnose the problem, see below for solution).
Many of you use a lot of those buttons in hope that they will bring you visitors. But while they load - they lose you visitors that have to wait 2 minutes for your page to load.
WordPress' social submit plugins are often have the same effect on your site.
The solution? Use async code and ask your plugin developer to move to async code.
It's not some futuristic HTML5 goodie that works only in modern browsers. It works everywhere.
Facebook has async code - use it!
Google Analytics has async - use it!
Twitter doesn't give out async, but it's easy to do it, based on FaceBook and Google Analytics code:

  <a href="http://twitter.com/share"
  class="twitter-share-button" data-count="horizontal"
  data-via="WhitePostsCom">Tweet</a>

  <script>
    (function() {
    var src = document.createElement('script');
    src.async = true;
    src.src = document.location.protocol + '//platform.twitter.com/widgets.js';
    document.getElementsByTagName('head')[0].appendChild(src);
    }());
  </script>

 Replace the first part with your own code instead of WhitePostsCom one.
StatCounter doesn't give async, adapt it from the code StatCounter gives you:
(There was a day when StatCounter didn't load in 2 minutes! Your site is stuck again if you don't do async)

  <!-- Start of StatCounter Code -->
  <script type="text/javascript">
  var sc_project=[YOUR CODE HERE];
  var sc_invisible=[YOUR CODE HERE];
  var sc_security="[YOUR CODE HERE]";
  (function() {
      var ga = document.createElement('script');
      ga.type = 'text/javascript'; ga.async = true;
      ga.src = document.location.protocol + '//www.statcounter.com/counter/counter.js';
      var s = document.getElementsByTagName('script')[0];
      s.parentNode.insertBefore(ga, s);
  })();
  </script>
  <noscript><div
  class="statcounter"><a title="web analytics"
  href="http://statcounter.com/" target="_blank"><img
  class="statcounter"
  src="[YOUR CODE HERE]"
  alt="web analytics" ></a></div></noscript>
  <!-- End of StatCounter Code -->
 StumbleUpon? Adapt it from the above codes.
Seeing someone asking you to insert '<script src="http://' into your code? Tell them to do better engineering and stop slowing down your site.
P.S. The reasons for hiccups of Twitter and FB's CDN might be poor peering, bad servers, anything really. You can't fix Twitter's and Facebook's software and servers, but you can let your visitors see your site without depending on how good do engineers at those companies do their job.
P.P.S. There is a good question in comments about how to diagnose your own site's problems: http://news.ycombinator.com/item?id=1771755
Problematic hosts list (the srcs that cause sometimes huge slowdowns for me) - just in case you need it:

  widgets.digg.com
  platform.twitter.com
  static.ak.fbcdn.net
  www.stumbleupon.com
  i.ixnp.com
 <comment><USER2> In general I find these buttons obnoxious as hell (they are tasteful on some sites like the NYTimes). This just adds insult to injury.

<comment><USER3> Or perhaps this adds injury to insult.

<comment><USER4> I wonder if injuries and insults commute...

</comment></comment></comment><comment><USER5> Incredibly annoying. I have a quad core machine, 16GB RAM, SSD, 100Mbit broadband yet it takes 14 seconds for TechCrunch to fully load due to the Post Up widget they use. Once it loads the page scrolling is not smooth at all. It definitely makes me less likely to visit.

<comment><USER6> I just adblock that shit. I know I should just not visit their site or whatever, but if everything I wanted to read had to live up to my standards, there would be nothing to read.

(Incidentally, my own blog didn't live up to my own standards, and it's down pending me fixing that :)

<comment><USER7> This is my primary reason for running adblock. I can't tell you how many clients have e-mailed me saying 'My site is down!' or 'My site takes a few minutes to load', even though the site loads in about a second in Safari. When I switch over to my (un-extended) Firefox install, I suddenly see the problem - literally dozens of Javascripts, image requests, tracking pixels, and other junk.

It's ridiculous. At one company we had a client who spent the last year contracting us to add widgets and pixels and add-ons and buttons and everything, until suddenly they found out that Google would be factoring in page load times into their search results. Suddenly it was a rock and a hard place - do they keep adding content without any proof of its value? But that would hurt their search rankings - but but but, it adds value! It encourages engagement!

It's ridiculous. They didn't even have any metrics showing that any of the stuff they were adding was helping, or even being used. Likewise with the search widgets. People are told that these things will 'drive user engagement' and 'encourage social interaction', so they throw it in and assume that it's made their site better.

</comment></comment></comment><comment><AUTHOR> I have written a very simple tester for these problems that you can use to diagnose your own site: <http://whiteposts.com/not-async>

Now also shows the code to be replaced to make calls async.

If you like the tool - please upvote it at <http://news.ycombinator.com/item?id=1772169>

</comment><comment><USER8> Google and Amazon have published research to the effect that even small increases in page load time have a massive effect on traffic.

Social sharing is intended to boost traffic virally, but I wonder if these stupid widgets are actually reducing traffic by slowing the page load so much.

Many of these submit tools could be replaced (with less functionality) with an image button or text link.

<comment><USER9> Absolutley. When I set up caching and a cdn on one of my sites, I saw a 25% increase In visits literally overnight. When I asked the developer of the caching plugin, he said that many people report the same thing.

</comment></comment><comment><USER10> Warning: while the above async loading code works for Twitter, it won't work for any JavaScript containing calls to document.write() as they'll append the data to the end of the page depending on the browser (or even replace the whole page!)

All is not lost though, as you can patch document.write to do the right thing and write to an element's .innerHTML or equivalent. If you are a JavaScript API provider though, please take heed and don't use document.write(). I'm looking at you, PollDaddy!

<comment><USER11> How would you handle waiting simultaneously on multiple scripts that all want to use document.write?

I use some ad networks which want to call document.write to insert the ads. First I patch the document.write to a function that writes to the innerhtml for the first ad tag, then after that ad loads I change the function and start loading the next one.

Obviously slower than doing it simultaneously. I suppose I could look at the contents of how the function is called to determine which ad it came from, but did you have some better idea?

<comment><USER10> That's a tricky one. I haven't tested this, but I think you should be able to listen for the 'load' event on each of the script elements that you inject, and patch document.write differently each time.

Let me know how you get on! I plan on doing something like this soon for some of my sites that have AdSense (another document.write culprit).

<comment><USER12> I just pushed up a class I've been using to do this to github. It's a little rough around the edges, but works in Firefox and IE. I've trialed with the AddThis. <http://github.com/joshduck/Injector.js>

</comment></comment></comment></comment><comment><USER13> Meebo made a presentation at Velocity this year about truly asynchronous loading, among other performance secrets of the meebo bar. (FF can still block using the suggested technique)

Well worth the watch: <http://www.youtube.com/watch?v=b7SUFLFu3HI>

</comment><comment><USER14> There is another widget which shows a bottom toolbar that looks like the Facebook one (and it offers sharing and chats and other crap). It makes the website fucki*gly slow and sometimes interferes with the website JavaScript and add nuisance like bad scrolling, selecting...

I finished up closing these sites as I open them. I would also recommend that everyone that has a blog, implement the most minimalistic widgets, less JavaScript and a simple design. The purpose of a blog is to read, if I want to chat, I'll open Skype.

Coding Horror is a good example.

<comment><USER15> Is it the Meebo bar?

<http://bar.meebo.com/>

<comment><USER14> Yeah, that one!

<comment><USER16> The bar from www.wibiya.com is even worse. Not only is it slow to load but it has notification pop ups and a bunch of other crap.

I'm sick and tired of all the sites that have decided to add bars to the top and bottom of their pages and a billion social network buttons all over the page. I'm at the point where even though I'm blocking most of them via adblock new ones keep showing up so I just give up and stop going back to sites that use them.

</comment></comment></comment></comment><comment><USER10> LABjs is a useful JavaScript library that lets you load JavaScript on-demand and specify dependencies: <http://labjs.com/>

<comment><USER17> LABjs is great for making javascript not block the rendering of your page. It's not just a simple switch though. Depending on how your code is structured, and the various assumptions you've made there may need to be some code changes. The biggest thing, is that if you load everything in LABjs - including jQuery - then the document.ready event fires BEFORE ALL THE JAVASCRIPT IS LOADED/EXECUTED. This is incredibly important because many many people code with $(document).ready being the signal to run all sorts of stuff. However, with LABjs the meaning of document.ready changes so you can't assume that things like plugins have been declared when document.ready fires.

</comment></comment><comment><USER18> This is really good advice. I'm surprised these big sites aren't using something like CloudFront to serve their files.

<comment><AUTHOR> Well, they kind of do. It's just that sometimes their CDNs have serious hiccups and ALL sites that use non-async code are affected (when they shouldn't be).

I'm really surprised at Twitter. They have good engineers, why are they giving you non-async code, when the above code does the same thing exactly?

</comment></comment><comment><USER19> Did I miss something, or why are these in your head tag? Why not put them in the body tag so they don't load until after the browser has rendered the page?

<comment><AUTHOR> Sorry, but you aren't right. Those are mostly located in body tag. They are requested as soon as encountered by browser, because they might issue "document.write", so the execution of rest of HTML is delayed until that "src" is loaded. That's exactly what causes the problem. Putting script into body doesn't defer its execution.

I'm not sure how browsers render

(Use 'async="async"' for XHTML.)

<comment><AUTHOR> Not that simple, won't work on most browsers. Also if script has document.write inside - will fail.

[http://stackoverflow.com/questions/1834077/browser- support-f...](http://stackoverflow.com/questions/1834077/browser-support-for- script-asynctrue)

<comment><USER23> More and more support for it every day:

<http://webkit.org/blog/1395/running-scripts-in-webkit/>

So there's also "defer," which is better-supported. The document.write issue is also an issue for JavaScript element building, but the assumption is that you won't blindly use these methods without testing, first.

</comment></comment></comment><comment><AUTHOR> It turns out (just read an article in Russian) that recently there was a huge DDOS attack (100+ Gbit/s) that dropped major providers in Europe, DARPA(.mil), major Russian search engine. 100GBit/s is enough to fill major landlines and backbones (even in US, darpa.mil failed too). That might have been the cause of delays. Nobody is sure whether the attack is over, but major datacenters and DDOS protection providers failed to do anything against it.

Still, making your site independent with async scripts is good idea.

</comment><comment><USER24> Those kinds of sites haven't gotten slow for me at all, and I'm only on 8Mbps broadband. Thanks for bringing it to everyone's attention though, hopefully it'll get fixed.

<comment><USER25> Broadband penetration in the US was at 20% in 2007. 8Mbps is positively luxurious. (When I lived in Ohio I was at 3Mbps advertised, which of course translates to significantly less.)

Now I'm on a shared satellite link which is something like 8Mbps for 30 users.

</comment></comment><comment><USER26> Delays of 30-90 seconds are rarely at the network level. Asynchronous is generally better than synchronous, but it seems more detective work should be done before declaring war on synchronous scripts.

Are other users in Russia affected? Are users on other ISPs affected? How widespread is this problem? Are your ping times the same to "www.digg.com" as to "widgets.digg.com"?

If there is any truth to this post, the problem should be fixable.

<comment><AUTHOR> It's not at network level, it's probably at service level. Service takes too long to respond (overloaded, not enough bandwidth, real-time replication, deadlocks - you guess it). I don't know how widespread is the problem, but I'm on a major provider and connection to everything else is very fine.

The ping is fine BTW - 50ms for widgets.digg, 200ms for digg. Which just helps to prove that it's a service problem.

</comment></comment><comment><USER27> I built some fairly clumsy code to fix this on one of the blogs I run. It loaded the buttons in the footer of the page (so you can be reading the article before all the nonsense has loaded) then moves them to the correct places (the foot of the article) once ready. By the time you came to retweet or what have you, it was loaded.

It's not that dignified, but did what I needed it to do.

</comment><comment><USER28> I have problems with our adserver tags loading much slower than our pages. I often use a CSS hack to that the adserver code loads at the bottom of the page but is position in CSS at the top. However this can't be applied in 100% of situations......

Has anyone managed to load adserver code with this method successfully? I'll be trying myself this weekend and will report back....!

</comment><comment><USER29> I have these problems developing with backend Facebook calls. In theory facebook servers and our servers shouldn't have a load of latency between them but it really makes the site noticeably slower. Definitely slow enough that I would have to remove all the back end calls on page load if we were to push facebook connect more.

</comment><comment><USER30> Does page caching help? I have that setup for my personal wordpress blog. I really don't wanna have to modify all the javascript on my page. I've got bigger fish to fry.

I also don't want to have crap slow page loads. It loads for me in under a second.

<comment><USER31> Page caching doesn't matter if you're making a request to an external server.

</comment></comment><comment><USER32> Even worse, the buttons hosted on wd.sharethis.com set frequent timeouts that use up a measurable amount CPU (and laptop battery life).

(This is particularly bad when I open a large number of articles in tabs, to read later.)

</comment><comment><USER6> Do the ads load quickly? If the ads load quickly but the content loads slowly, people are more likely to click the ads.

Of course, they'll never visit your site again, but hey... that's what "social media" is for!

</comment><comment><USER33> I've seen some metrics drop by as much as 10% because of this issue. The drop was pretty bewildering at first, but then I A/B tested including (but not using) the javascript and saw the difference.

</comment><comment><USER34> Is there a service that let you check your site loading time and whether you are using asynchronous loading? Would be useful to solve this problem.

<comment><USER35> If you're using Firefox, the Firebug extension's Net panel (when active) shows the Gantt chart of downloads.

<comment><USER36> You can also team Firebug up with Yahoo's fantastic benchmarking utility YSlow.

</comment></comment></comment><comment><USER37> Are the delays in Russia due to a satellite? Those seem like very long delays. I am usually annoyed by 5-15 second delays in the US.

<comment><AUTHOR> No. I have 180ms _stable_ ping to news.YC, _stable_ 40ms ping to Google. It's not a line problem.

Chrome Dev Tools show me that I receive content within 300-400ms of hitting the page with all the images loading in about 2-3 seconds... and then there's one huge 1 minute line associated with "platform.twitter.com", "static.ak.fbcdn.net", "www.stumbleupon.com" (everything else being a short ticks with 50-100ms receive time - it's a real 35Mbps line)

Any site that I measure with Chrome Dev Tools in any part of the world is fast (seconds, milliseconds), the only parts that are slow quite sometimes are those guys - T, FB, SU.

And yes - those are very long delay. Maybe Twitter and FB have some peering problems - I don't know. I don't need to know. It's just that those can be fixed easily so that we don't have to find out why T/FB have peering problems.

<comment><USER38> For what it's worth, I often see this problem in Canada - it's not always the offenders you list, but third-party widgets do significantly slow down page loads relatively often.

</comment></comment></comment><comment><USER39> Or just put them after  but before

<comment><USER40> In most cases for social media buttons supplied by these web services, you can't. You will typically have to reference a JS library at the position of the document you want to render the button at. You put them after  and they'll render in that position in the document. The above tip is smart and works around that by asynchronously updating the DOM when the scripts are ready (i.e., downloaded).

If you meant "put the OP's scripts after the body element" -- yeah, you could do that (at least for sure with GA async code, you can), and that would be a good idea so that you serve and render CSS first, making your pages feel more responsive. But that's not what your comment reads as.

</comment></comment><comment><USER41> could it be china's great firewall? these sites all load snappy for me and i am on a 45mbps local ethernet loop in san francisco. sounds like a network edge problem.

<comment><AUTHOR> I'm not behind China firewall. I'm in Russia (near Moscow).

<comment><USER41> yes, and there is not a direct connection between the US and Russia, the internet will take the shortest traced route. So, if one site traces well through china. Then an IP that is nearby, might actually start the connection using that route. It's hard for me to believe that Russia has opted not to allow any traffic what so ever to route through China.</comment></comment></comment>


Stephen Hawking: AI, Potentially the Worst Thing to Happen to Humanity - <AUTHOR> https://uk.news.yahoo.com/stephen-hawking-artificial-intelligence-potentially-worst-thing-happen-110523390.html#r6ovIz7 <comment><USER3> I don't really want to come across as hostile to the views of such a well respect physicist, but this article represents the view which is often held by those who are very intelligent yet lack the actual study of AI.

Anybody who has actually, or is currently, studying AI will know that there are fundamental differences in programming "AI" and some self aware magic computation which is almost entirely unfathomable. With current knowledge, at least, the only way to have any sort of learning is to design and implement algorithms to do so. There can of course be incredibly abstract knowledge base designs and the computation can assimilate incredibly complex knowledge from those, however it is almost a logical fallacy to suggest that we would ever lose control of something autonomous due to some sort of "rogue agent".

Of course, there are very real risks with something like AI, but it's much less sinister than what the article suggests. For example, faulty code or errors in the knowledge base could lead it to make a bad decision, but humans can also do that.

I don't think we should be dismissing the risks necessarily, but I do believe that it is completely far fetched to say we would be making the worst mistake in history if we are to dismiss a film as science fiction!

<comment><USER4> On my side, I am surprised that so many AI researchers are unable to take the long-term view in this discussion. It's presumably because AI in the laboratory is still relatively primitive.

No one is talking about magic. The human brain is not magic, neither is that of chimpanzees, rats, dolphins or gorillas. Intelligence is a purely physical phenomenon, which means it can be emulated by computers. Natural brains are also a product of evolution, which means (1) the development happens very slowly, (2) development is directed only towards evolutionary success and (3) there is no flexibility in how the thinking organs are constructed. Computer intelligence does not in principle have these limitations. It would be terribly anthropocentric to believe that humans are the most sophisticated intelligent entity that can exist in the physical world - after all, we are as far as we know the first such entity to emerge, so from our perspective the evolution of intelligence has now stopped.

That's the feasability argument. The risk argument is that the consequences of an independent, runaway intelligent entity significantly more capable than humans would have such devastating consequences for humanity's future that even a small risk merits a significant effort to map out the territory. Respected scientists have said "it is impossible" to hundreds of things that proved to be quite simple, so this is not an argument. Even if you don't buy the previous part of my reasoning, there is still risk here. In principle, there are any number of things that could preclude advanced AI in the near future even if the above reasoning is correct (too difficult, requires too much computing power, uses different computational techniques), but seeing as we don't know the unknowns here, taking the cautious route is the correct thing to do. This has been a scientific principle for decades, no reason to drop now.

Do AI researchers have any arguments opposing this that don't amount to "the AI we have created up until now is not very good"?

<comment><USER5> > _" I am surprised that so many AI researchers are unable to take the long- > term view in this discussion."_

In my opinion, the Middle Age alchemists who made gunpowder and first primitive bombs, didn't need to establish research programs to worry about advances in bomb-making leading to the threat of Mutually Assured Nuclear Destruction.

If they had tried (maybe they did?), maybe they would have come up with ideas like very heavy regulation of the trade of saltpeter, so that no one has enough to make a verybigbomb.

I agree that everything you described will happen in the future. But in my opinion we are at a Middle Age alchemist's level in AI research (no offense to AI researchers) and we can safely wait a 100 years, and let those people worry about the existential threat. They will be in a much better position to do so appropriately, because they will know much more than we do. And they will not be late, either.

<comment><USER6> In this case the worst case would be very fast development of dangerous AI, and I'm not sure the 100 years is anywhere close to the lower bound. Low-end estimates for the computation power needed to run something equivalent to full human cognition are around 1 petaflop [1]. Google's total computing power in 2012 was estimated at 40 petaflops [2]. Of course it's split wide to a wide network of computers, but the human brain we're looking for comparison looks like a pretty parallelizable design. So we seem to already be at point where it might just be the lack of very clever programming that keeps us from getting a weakly superhuman AI running in the Google internals.

It looks like we've got ways to go there though, current programs don't seem to even begin to act anything like an adult human. So if the problem was to engineer an out of the box adult human level AI, we might again assume that there's obviously decades of work left to do before anything interesting can be developed. The problem now is that that's not how humans work. Humans start out as babies and learn their way up to adult cleverness. I can tell that an AI is nowhere near having an adult human intelligence out of the box, but I'm far less sure how to tell that an AI is nowhere near being able to start a learning process that makes it develop from something resembling a useless human baby towards something resembling an adult human in capabilities.

[1] [http://www.nickbostrom.com/superintelligence.html](http://www.nickbostrom.com/superintelligence.html) [2] [https://plus.google.com/+JamesPearn/posts/gTFgij36o6u](https://plus.google.com/+JamesPearn/posts/gTFgij36o6u)

</comment></comment></comment></comment><comment><USER7> This is what MIRI (Machine Intelligence Research Institute) is investigating.

[http://intelligence.org/research/](http://intelligence.org/research/)

The scariest part of the problem to me is this: [http://yudkowsky.net/singularity/aibox](http://yudkowsky.net/singularity/aibox)

If an AI can't be contained, then there has to be some failsafe that prevents _any_ self-modifications which could turn it evil OR remove the failsafe. Or you have to believe that you can neutralize an evil AI before it destroys humanity.

<comment><USER8> Regarding the AI box experiment, I'm somewhat dubious about it, considering there doesn't seem to be any transcripts from when the AI 'escapes'.

Which makes me think they use blackmail or something similar in the transcript.

<comment><USER9> I couldn't imagine how the AI could escape an hour ago, but after thinking about it for a while I am more and more convinced that it is actually hard to contain the AI. Containing the AI is like imprisoning a human for no reason and this will usually conflict with the gatekeepers moral values. Just imagine talking to human suffering in a tiny prison cell without having done anything wrong instead of the trapped AI. Can you see how it would be the right thing to open the cell's door? You are cruel not opening the door.

</comment></comment></comment><comment><USER10> I share Stephen Hawking's concerns. I think AI is already at a point where maybe 10 or even 15 years down the road, artificial intelligence will be everywhere. Autonomous public transport, taxis, driver-less vehicles, autonomous crime prevention systems. The military is said to always be at least 10 years ahead of public tech research and development, look what they've already got now, imagine another 10 years.

We've become a society that is too big and intelligent to fail, but that ignorance of being too smart and big to fail could be that our downfall is.

<comment><USER11> I share the concern partially. It's similar to other technologies which have downsides. Nuclear power for example. AI will have the power to help humanity prosper or cause its downfall. The balance will be for us to decide, again just like Nuclear power; for with great power comes great responsibility.

</comment></comment><comment><USER12> What would the AI's motivation be? Will it understand what existence means and have the "desire" to survive? Is it possible that a self aware AI has already occurred in some form many times in the past, but since it doesn't have the anthropomorphic trait of wanting to exist, it never actively persisted?

So what we are really afraid of is developing an AI with human motivations. But do the intelligence we want to artificially develop require human motivations? Aren't they two mutually exclusive traits?

Maybe part of the fear is how we train the AI. If we are to imbue it with the total knowledge of our history, then maybe it will become more "human" and have those motivations. Is it possible to train it with hard data that aren't anthropomorphized in any way?

<comment><USER4> An AI's motivation would be whatever it is programmed to do. Which is why it is so important to get this right the first time. Human motivations are incredibly complex, human moral systems have not yet been formalized and any attempt at doing this breaks down at the edge cases.

This is not by any means a solved question, but there have been quite a few publications on this already. There are a hundred subtle errors of reasoning that ruin naïve and anthropocentric reasoning around morality and AI safety. Have a look at the lesswrong blog, for instance. Lots of interesting reading.

<comment><USER13> If human motivations can be quantified and programmed into an AI system, my guess is 'loyalty' or 'patriotism' will be one of the first.

"It is lamentable, that to be a good patriot one must become the enemy of the rest of mankind." Voltaire

</comment></comment></comment><comment><USER14> I recommend reading "Shadows of the Mind" by Roger Penrose. In that book he argues that strong artificial intelligence isn't achievable on classical computers, since the human brain seems to have the ability to do quantum computing in certain molecular structures. These processes probably can't be simulated efectively. So AI in today's computers probably won't work at all (although quantum computers or artificial neural networks might).

<comment><USER15> This has been discredited (along with Penrose's incompleteness theorem based "AI is impossible" conjecture before it)

[http://en.wikipedia.org/wiki/Quantum_mind#Roger_Penrose_and_...](http://en.wikipedia.org/wiki/Quantum_mind#Roger_Penrose_and_Stuart_Hameroff)

<comment><USER16> Although specific claims might have been discredited, quantum processes are relevant to biology, e.g.

[http://www.nature.com/nature/journal/v446/n7137/abs/nature05...](http://www.nature.com/nature/journal/v446/n7137/abs/nature05678.html)

<comment><USER15> This is true, and it's also true that such quantum processes exist in the brain.

Penrose's claim was that those quantum processes are a significant part of the computation that is consciousness. It was discredited by getting better upper bounds on how much quantum computation the brain is capable of (ie very little).

</comment></comment></comment></comment><comment><USER17> I have a great deal of respect for Hawking, but... Lately it seems like he has said a couple of crazy things. We (as in most humans) are NOT going to get off the earth. Earth will always be the best chance for a planet for humans. Let's take care of it. And AI is NOT anywhere near becoming really intelligent. Anyone who has interacted with Siri knows that she has no real understanding. And the Google car and other autonomous driving systems are already safer and probably better for traffic than almost all drivers, but they still, and will most likely for a long time, need a human driver to take over under circumstances that actually require thought. AI is advancing, but it has always advanced slower than predicted. A lot slower than predicted. And while there are signs that AI will continue to be more and more useful as time goes on, and in some circumstances like recognizing driving, captchas and playing chess and jeopardy it will outperform humans in simple tests, it is not anywhere near, nor will it be for a long time, outperform the human brain in a general way. And certainly it is NOT similar to an alien (as in from outer space, not from another country on earth) civilization.

</comment><comment><USER18> Intelligence is not magical. What matters is algorithmic complexity. At some point only throwing more computing power can help. Today, yes, maybe we could build an ai with close to human level intelligence, running on a very expensive super-computer(s) eating enormous amounts of energy. So what?

Human-level ai isn't going to figure anything that humans can't figure out themselves, so it's not going to design new almost-magical ways to enhance itself. Especially because the entire human race is basically one giant parallel data processing system, although communication leaves much to be desired. Moore law isn't going to help - silicon technology is very close to stagnation [1].

I don't think singularity is possible until potential ai-accessible artificial computation capability is at least several orders of magnitude higher than human brain's, which is impossible with silicon-based technology. Nothing to worry about.

[1] [http://www.extremetech.com/computing/178529-this-is-what- the...](http://www.extremetech.com/computing/178529-this-is-what-the-death-of- moores-law-looks-like-euv-paused-indefinitely-450mm-wafers-halted-and-no-path- beyond-14nm)

</comment><comment><USER19> One thing I look forward to is greater logical input into high-level decision- making. Emotions, egos, racism, discrimination, nationalism, aggression - <USER2> these are things that we just don't need any more.

I do think we need exceptional care when working on such a system but I also think the human mind of 50000 years ago leaves a lot to be desired when running in the 21st century. A mind that is ultra-rational would be a good adviser, helping to point out our blind spots and guide us to better thinking. We shouldn't have to learn from all mistakes every generation.

(I'm certainly equally wary of blind acceptance of whatever science and technology can create. A favourite article of mine is [http://archive.wired.com/wired/archive/8.04/joy.html](http://archive.wired.com/wired/archive/8.04/joy.html) \- has some very serious warnings and consequences, even about my abovementioned thoughts.)

</comment><comment><USER20> Link to the actual article: [http://www.independent.co.uk/news/science/stephen-hawking- tr...](http://www.independent.co.uk/news/science/stephen-hawking- transcendence-looks-at-the-implications-of-artificial-intelligence--but-are- we-taking-ai-seriously-enough-9313474.html)

</comment><comment><USER21> Even if AI is harmless, there is still the risk of brain emulation. To me this is a greater threat, because if we don't know how a fully artificial mind would behave, we can get an idea of how an emulated human brain could. The problem is that human beings are potentially dangerous: they have volition, and they can enjoy destroying things. Basically they can be narcissistic psychopaths.

All it would take would be a human being rich enough to have his mind uploaded in a machine, and then this man will get greedy for more computing power, even if to do so he would have to take over the World and extinct humanity in its organic form.

Sounds very far-fetched, but as Hawking puts it: there is nothing in the physical laws that prevents a man-made device to perform as well or better than a human brain.

<comment> T-A The computational requirements are overwhelming though. There is a short discussion of this in section VIII of [http://www.fqxi.org/community/forum/topic/2077](http://www.fqxi.org/community/forum/topic/2077) (click the PDF link); the currently top-ranked supercomputer turns out to be roughly five million times too slow to simulate a human brain in real time.

<comment><USER22> This might be a retarded question, but a simulation of a thing must always run more expensively / slower than the reality of the thing, right?

And couldn't they build an organic material AI with inorganic interfaces for external manipulation?

</comment></comment></comment><comment><USER23> Anyone know of a larger discussion related to this issue?

<comment><USER24> The Less Wrong community, various subreddits (/r/singularity, etc). Good terms to search for are "x-risk", "Jaan Tallinn" and "Nick Bostrom".

</comment></comment><comment><USER25> I think the biggest problem as far as AI and automation is concerned is, what is left for us humans to do? At the moment, we work 42 hours per week (UK full time average), with many millions of people unemployed. Ultimately, our goal as a species, should be to build our infrastructure to be as efficient as possible. In an ideal world, we wouldn't have to go to work for 42 hours per week and many wouldn't need to work at all. We'd just enjoy ourselves 24/7\. I think this shift from the 'work hard mantra' to an efficient world where this is much less 'manual' work to do (that includes many office jobs), is one that society, government and business underestimates.

</comment><comment><USER26> Assuming strong AI arises, which I think likely, and given human nature I imagine people will build all sorts of AI's, 'evil' and 'good', though those terms are a bit subjective. Probably the likes of Kim Jong Un would love to have an army of self reproducing killer robots and Google will try to build nice ones and the US defence department will continue with drone like things. I agree with Hawking it could be a problem and hope the nice ones win out as they probably will. Fingers crossed.

</comment><comment><USER27> Bill Joy's article Why The Future Doesn't Need Us comes to mind: [http://archive.wired.com/wired/archive/8.04/joy.html](http://archive.wired.com/wired/archive/8.04/joy.html)

</comment><comment><USER28> so, Mr. Hawking hasseen 'The Animetrix,' I gather?

</comment><comment><USER29> Simplistically:

If Win = Continue

If Loose = Change

Basic evolutionary principle.

Then giving the tools of doing, would be required to consume the formula.

I argue that an AI with limitations would not work.</comment>


29-Year-Old Gets $15 Million to Decipher Song Lyrics - <AUTHOR> http://finance.yahoo.com/blogs/power-pitch/29-old-gets-15-million-decipher-song-lyrics-124202754.html

<comment><USER2> My gut reaction reading this article was "Oh great, someone didn't learn from the 90s."

But once they started talking about Sandberg and Andreessen annotating books, speeches, reports, etc... that has to be where they're headed. I can see some sort of monetization strategy there.

If the site focus eventually shifts (and I doubt this was planned from the beginning) to "true experts explaining things," I can see a lot of potential there. Maybe not "bigger than Google," but certainly far "bigger than a lyrics explanation site."

<comment><USER3> Speeches and reports are pretty worthless, no one would move into worthless little niches like that. Popular media like they are in now is a goldmine, though. Pretty trivial to monetize as well. Many of my friends who do music tied apps are at the point of swapping out the affiliates and common payment providers for direct credit card exchange deals just to optimize percents even.

</comment></comment><comment><USER4> They are simply trying to own a particular, important part of human consciousness: Lyrics.

The founders court popularity as their primary business function, and they hype Rap Genius lavishly.

Monetization is completely an afterthought.

I am convinced that Rap Genius is the purest consumer play.

</comment><comment><USER5> OK so maybe some of his claims are comically grandiose.

But I've found Rap Genius useful recently for finding explanations of poems by Yeats,Wordsworth and many others. The convenience of being able to see the explanation right there without having to scroll or navigate away really helped me get into the poetry. I can see this approach being extended into many other areas.

Theres a lot of potential in the idea of being able to annotate select parts of a document instead of having to comment on the whole document. I think we need an annotator for the web. Maybe it would work as a browser extension and users could annotate any part of any website with extra comments , photos etc.

Another approach might be to develop an embeddable widget which could allow anyone to annotate any part of a web page.

But I couldnt figure how to handle changes to the original document if the model isnt under our control.

<comment><USER6> The question (and this applies to Wikipedia as well) is _whose explanation are you reading_?

I'd throw you a quotation about the Power of Language, but I'm sure you can imagine it already. This site marks the end of 19th / 20th Century propaganda, which relies on an ignorant audience to swallow the lies without being able to fact-check them[1] (e.g. The Protocols of Zion or the "re-imagining" of Founder Father's texts to be anti-Semitic or (to be even) the notion that America was founded as respecting all of its citizens as equal - you'll want to read the 13th closely, you're all breaking the Law somehow, which is the hack they're going to use on you).

Modern Propaganda doesn't do this: it uses truth, but with hooks.[2]

The More You Know.

Footnotes, because I forget that not all readers share similar thought patterns:

[1] If this is true, what does that tell you about the content of most mainstream American News, and its consumers, and the future of these companies? [2] Is my statement propaganda? Answer: of course it is, _everything_ you consume online is.

<comment><USER7> Whose? The Voice's. The ultimate truth who's always right. Wouldn't that be convenient? I'm sure if Rap Genius is going to be bigger than Google and Facebook (combined, obviously), they call pull that off.

</comment></comment></comment><comment><USER8> Does RapGenius have the proper licensing to republish these lyrics? Is that what the funding is for? Lyric websites have been shutdown many times for this.

<comment><USER9> (disclaimer: posting anonymously because I work for a competitor)

 _Does RapGenius have the proper licensing to republish these lyrics?_

Nope. I'm always a bit surprised this never gets brought up.

 _Is that what the funding is for?_

Doubtful. If it was, they would have licensing by now - and you don't need anywhere near that much money for it, countless smaller sites have licensing.

 _Lyric websites have been shutdown many times for this._

Yup. Most of the big "illegal" lyric sites are outside the US, but RapGenius is not. Should be interesting.

</comment></comment><comment><USER10> I'm 'out' \- this site in my opinion is going to fail. No revenue model? Why are people entertaining this as 'bigger thn google' are you kidding me.

<comment><USER11> The thing is you don't know how the site is going to playout. IMHO their biggest advantage is having artists annotate the lyrics. It gives fans an insight in to the song like never before.

Monetization will come in time. But first, they need to be sure what they are creating.

</comment></comment><comment><USER12> The arrogance is so intentionally outlandish it's hard to even find it comical.

Bigger than Google, yeah ok. $300 billion market cap, $50 billion in cash, $50 billion in sales, within 15 years.

The outcome for Rap Genius is far more likely to be that of a Hunch, than a Google. I've yet to see anything about Rap Genius that suggests their technology or approach will yield such a breathtaking commercial windfall as Google. Even their reach is pretty mediocre for a nearly three year old service (speaking in terms of the huge future outcome they pander to for headlines). See Pinterest for a comparison on size (they're 15 times larger than Rap Genius), and those guys don't go around claiming to be the next zillion dollar company. Zero chance Rap Genius can springboard that to Google like results. Google was at the center of the most important Web utility of its time: the ability to accurately and quickly find anything online; and their technology was mind boggling good. Rap Genius is more like Wikipedia - it's a content solution provided by the user base, and there isn't anything earth shattering about that (see: Answers.com or Quora).

I'd say that within 24 to 36 months they'll pivot or be sold. Reality usually hits like a hammer when you proclaim yourself the next Google (or Facebook).

<comment><USER13> Every rapper needs a hype man. Rap Genius is no different. Obviously Rap Genius will not be bigger than Google.

<comment><USER14> That's a very poorly executed hype.

</comment></comment></comment><comment><USER15> Feels like a pretty obvious winner...

Not as big a winner as they thing though but that posturing may just be due to having been in fundraising mode or preemtively positioning themselves as a juicy aquisition target..the mentioning of Instagram among Facebook, Google, Wikipedia was rather strange.

"Annotating stuff in a way that doesn't suck" and "getting stuff annotated by others" are two major pain points that they seem to solve rather well. The rest is automatic as long as they can figure out legal hurdles in the domains they want to tackle. If they can't they'll be the Napster and someone like Amazon will probably build the iTunes.

Annotations by domain experts/creators is probably the biggest thing. Basically allows them to extend certain products i.e. deluxe editions of books with annotations from authors (like those director/actor comments tracks for movies)

<comment><USER16> Annotation is useful, but might not be able to be monetized like they hope. Personally, I feel that RapGenius falls short when it comes to other current enterprise collaboration methods such as Word or Google Docs.

</comment></comment><comment><USER17> The annotation aspect of Rap Genius always gets me dreaming of a site with "completely commented code" through crowd-sourcing. Every single line/block of code of various open source projects explained thoroughly.

When I started picking up programming, I learned by copy-pasting code bits from everywhere, and reverse-engineering it to fit my learning projects. If, along with that, I could have gotten thorough explanations for each block of code I copied, I feel I'd be much more proficient in understanding the "why", not just the "how" of programming.

<comment><USER18> There's already other versions of RapGenius for Rock/Poetry/News, so why not a CodeGenius.

Having this kind of commenting mechanism on Github would be fantastic.

</comment></comment><comment><USER19> So basically, Rap Genius is just a website explaining what lyrics of a song mean with no revenue model? Not only sounds this like a not very significant idea, the approach to music is very questionable. It doesn't matter what the lyrics mean to the artist or some expert. The only thing that matters when listening to music is what those lyrics mean to YOU!

Seems to me that Rap Genius is corrupting that concept.

<comment> 1234567890-- "The only thing that matters when listening to music is what those lyrics mean to YOU!"

A lot of lyrics are full of obscure references and complex ideas not easily understood. Perhaps especially in rap. So explanations save you hours of work in trying to understand and appreciate the lyrics. For example: [http://rapgenius.com/Lupe-fiasco-dumb-it-down- lyrics](http://rapgenius.com/Lupe-fiasco-dumb-it-down-lyrics)

</comment></comment><comment><USER20> That dude on the panel looks like a 'know-it-all-douchebag'. His posture and everything seems very aggressive - and he was wondering why Ilan was being defensive.

</comment><comment><USER21> I can't tell you how difficult it was to keep the tab open after I read the words "bigger than Google".

</comment><comment> mh- not sure whether to congratulate Rap Genius or Heroku,

</comment><comment><USER7> Why just rap?

<comment><USER3> Did you read the article? It isn't just rap.

<comment><USER7> I skimmed the article, but read the web app's title: "Rap Genius: Discover the Meaning of Rap Lyrics"

Glad to know it isn't just rap, but then the name is very misleading.</comment></comment></comment>


Ask HN: What seeds you sowed years ago that you are reaping fruits of now? - <AUTHOR> This could be technical, social, financial or anything -- what was the seed and what fruits are you reaping now? <comment><USER2> I took the first nice car I bought three years after I got my first job to the dealer to replace brakes. They charged me $550 for it. After that I taught myself how to fix things, starting with cars, plumbing, drywall, electrical and each and every thing in my daily use. I haven't paid for any repairs or upgrades to my house or my cars in over 10 years. I invested in tools and educating myself rather than pay someone else.

I can now fix anything and everything that I use daily.

<comment><USER3> I did the complete opposite.

I paid a company to buy furniture from IKEA that I selected, deliver it, assemble it and get rid of all the cardboards and waste. I just sat there for three hours and everything was done. Best money I spent.

I also did the same while moving to a new home. I paid the mover to do everything, dismantle beds and furniture, pack everything, then assemble everything in the new property. I didn't feel the pain of moving at all.

I get that sometimes saving money and learning something new like fixing cars is a good thing. But as a 45 years old, spending money to get peace of mind and reduce stress has become the priority for me now.

<comment><USER2> For me it started because of money, but then it became a hobby. Besides, I can go for an oil change and have everything done for me. However, it's fun researching about different oils, finding a best price online, getting it shipped and then doing the oil change with my young kid. I consider it nice father/son activity. Having my son learn about things is a bonus.

Similarly, when I finished some unfinished part of the basement, I planned it with my kid, went to Menards to buy drywall, picked a color, painted together. He got a sense of accomplishment and so did I.

<comment><USER4> I can see your point. I'm currently fitting new doors in my house and teaching myself how to do it. It's a pain in the backside however, but it's satisfying and I feel a sense of pride every time I open and close the door to go to the bathroom. I did that!

<comment><USER2> Yep. I am writing code professionally for long time, so that pride and sense of accomplishment is hard to come by. But fixing anything, even small, gives me that satisfaction every time I use it.

</comment></comment></comment></comment></comment><comment><USER5> In 2014, after retiring from full-time work in the SF Bay Area, I decided to use my savings to purchase some undeveloped land in southern Rhode Island and build a custom home there. The results have exceeded all my expectations and I'm very grateful.

[https://www.corememorymusic.com/this-new- house.html](https://www.corememorymusic.com/this-new-house.html)

<comment><USER6> Very nice! Would you mind sharing ballpark figures on how much a project like this costs?

<comment><USER5> About the same as a 2br townhouse in Mountain View. Feel free to email me (see my profile) if you'd like more information on costs.

</comment></comment></comment><comment><USER7> Answering in the present as a mid-20s person who is currently sowing the seeds of financial independence by saving/investing over half of my after-tax income. Having compound interest working in my favor early is key to my FIRE goals.

Want to set myself up to have the freedom to demand certain things at work, whether that's taking a sabbatical or working only 4 days a week without being chained to my work for a living. And eventually I'd like to consider early retirement.

</comment><comment><USER8> A solid understanding of math. Not so much the specifics of certain subjects, but the way of thinking.

<comment><USER9> Agreed. Numeracy is a life-long skill whose applications pervade every aspect of one's life. Making decisions and navigating the world become much simpler tasks when estimation, analysis and forecasting are familiar tools at one's disposal.

</comment></comment><comment><USER10> I inherited some code written by a junior. It was messy, with lots of code (and bugs) copy pasted everywhere and gigantic god methods which took up 4/5 of the file. It would take 4 hours to copy a thing like loading a paaginated endpoint, because of the heavily coupled nature.

I realized it would only take about 8 hours to refactor some of those parts to 30 minutes, and about a week to refactor that 30 minutes of work to 5 minutes. The pages now not only load better, they're free from copy-paste bugs, and easy to add new modules. At peak refactoring, I was removing thousands of lines of code a week, with some good teamwork from QA to keep it clean.

A year later, and I'm now able to integrate new features in half a day compared to about a week, leaving me about 20 hours/week free to do whatever, which I'm using to upgrade the whole codebase to reactive and native code, as well as learning iOS programming. It is very nice to go from a week behind schedule to several days ahead of schedule.

My plan for next year is to keep cutting down the cost for new features. Everything that takes me 20 hours to do today should take me 2-3 hours in the future.

<comment><USER4> Well done. That's the perfect attitude to writing and maintaining code.

</comment></comment><comment><USER11> I had an idea for an app and I taught myself how to code so I could make the app. I poured thousands of hours into learning. For a while it was like having two full time jobs. At a certain point I knew it was what I wanted to do for work. It was hard to get my first job, but I was lucky and got it. The rest is history.

I never got around to building the app, but I did build a career.

</comment><comment> noman-land I taught myself how to pick locks when I was a kid and I've saved the day on numerous occasions in my adult life where people were locked out at inopportune moments and I was able to get us in.

<comment> non-entity Haha, I've tried to teach myself a few times, but only when I was locked out myself. Always ended up just calling a locksmith.

</comment></comment><comment><USER12> I married the right person 30 years ago. I reaped 30 years of a pretty sweet life. When I faced difficulties, I did so with someone firmly in my corner that I could trust and count on.

Thanksgiving weekend 1984 I spent reading K&R. That's added up to a career that has paid (and still pays) pretty well.

<comment><USER12> My wife said, "That's not actionable." What I did from her point of view is, I always treated her with compassion. And I listened with empathy, even if I didn't agree. The fruit is how much she trusts me.

</comment></comment><comment><USER13> I got good at networking on LinkedIn. I realized that it didn't need to be weird, any more than meeting new people at events is weird. Over time, it became just as useful a tool as actually meeting people in person, only I could do it anywhere.

It helps if you've not got the option of discovering new people at events, for example like the way things are now. It's making it hard for people who rely on face-2-face to develop relationships with new prospects.

Separately, I always had this sense that it helps to have a trio: skills, domain expertise, and a professional network. It took a while to build up all three, and it's a process that's never done, but I always sensed to work towards having not just one or two but all three.

<comment><USER3> How do you network on LinkedIn please? Do you just hit people up and add them? How would one start?

<comment><USER13> > Do you just hit people up and add them Yes and No. Yes, you reach out to > people that are not in your network, and that have never met you. No, it > helps if you connect with a message tailored to that person that makes the > connection of where your professional interests overlap.

> How would one start? I actually wrote an article about this when everyone > went into lockdown:

[https://medium.com/skill-strong/how-to-network-on- linkedin-w...](https://medium.com/skill-strong/how-to-network-on-linkedin- when-youre-shy-4c60f02b3afd)

</comment></comment></comment><comment><USER14> I forced myself to get into my own bootstrapped SAAS business in 2014 and it is now in a reasonable shape where I can continue to grow it at any pace I want while I enjoy a stability of enough revenue to sustain a decent lifestyle. I m no way to close to being done :) but I can say with comfort that I achieved the bare minimums and have replaced my meaningless corporate life successfully. Has been a tough 6+ years but was worth it for me.

On a personal side, I taught myself how to cut my own hair few years ago and boy it has been amazingly fruitful especially right now. My first 4-5 cuts were nothing short of embarrassments but I am a pro now with the limited way I give myself a buzz cut and scissors in front.

</comment><comment><USER15> About 20 years past, helped a friend of a friend build a manufacturing process, get accredited, and wrote some templates for his QA, QC, NRTL submittals, etc. Did this mostly gratis. Got many good references, which became the beginning of a side business.

Got my pilot's license and instrument rating through the old military flying club system (while all else were getting drunk on liberty). It has oft been the difference between servicing a happy client or perhaps getting hit with a tort (the latter has yet to happen directly to me, but it has happened to a member of the consortium because he was not able to get to a site before a critical deadline.

</comment><comment><USER16> White collar crime, decades ago. Unless laws change, it'll follow me for the rest of my life.

<comment><USER10> Good fruit or bad fruit? As in cash or curse?

</comment></comment><comment><USER17> Not years ago but recent seeds

I wrote a blog post on filament manufacturing at home.

That blog post got over 25k views, many people contacted me on telegram. While all the information is available for free in my blog post, few people asked me to be their consultant for 10% share of profit.

So I ended up helping them setup their own filament manufacturing unit and now I am recieving 10% of profit. We aren't competiting with each other as we only target one country.

Right now money from review share is not a lot it's few thousands of dollars. But we are combining efforts and helping each other get more sales.

This business is local, for example I found 1 guy from Germany, Italy and United States and one university from India which are now making their own filament. One guy from UK is also working on this but it's still in progress.

Here's the blogpost if anyone is interested:

[https://medium.com/endless-filament/make-your-filament-at- ho...](https://medium.com/endless-filament/make-your-filament-at-home-for- cheap-6c908bb09922)

</comment><comment><USER18> Spent $10 on node.js course, now earning my living from it.

<comment><USER19> That's a pretty sad view into the state of hiring.

<comment><USER20> It's sad that someone with self-taught skills can earn their living?

</comment></comment></comment><comment><USER21> Technically not from seeds, but apples, cherries, peaches, raspberries, blackberries, and asparagus.

</comment><comment><USER22> Fitness: lost 50lb, which moved me from obese to normal BMI, then put on enough muscle to squat 1.5x my bodyweight for reps. Then got into running just enough to be able to run 3 miles on demand and 10 miles with a little lead time to prep. This all took maybe five years. Now I'm hopefully 18 months out from starting a thru hike of the Appalachian trail. The feeling that I can just trot into a run at any time just feels so good after a lifetime of obesity. Plus now exercise just feels so good, my biggest issue is overuse injuries so I'm constantly mixing up my routine to fight that.

Finances: started saving more than I spend, so I can take off time as needed to do big expedition projects like a 6-9 month thru hike. Also have learned a huge amount about finance and economics, which helps a lot with knowing what to do with saved money.

Location: moved from nowhere TX to the big city. Best decision ever. I was making 29k a year as a programmer there, immediately doubled that when I moved. The more I made, the more I could pay off debt.

Got debt free ASAP: paid off 200k student loan debt by living with my grandmother as her cook, handyman, and cleaner for 3 years. This was awesome, I got to learn a lot from her in those few years she was still pretty coherent.

Food: started writing down everything I eat, this helped me break my emotional attachment to eating that plagues my whole family. Now I'm 5+ years out from losing 50# and still fitting in my new clothes. Also got good enough at cooking that I can feed myself without getting bored, so I only really ever eat out socially, maybe 2-3 times a week.

Drastically reduced drinking alcohol and caffeine: drinking makes me feel bad, costs a lot of money, and caffeine makes me anxious. Today I drink less than a beer a week, and I feel like crap when I do, so I rarely do. It's a bummer too, because I'm a pretty handy bartender and barista after years of food service work.

Engineering: got serious about my programming skills 10 years ago, and worked hard to get much better at it. Finished SICP, Let Over Lambda, a couple of Haskell books, and a million TDD katas. Now I'm extremely fluent in several languages, and can solve most technical challenges encountered in a typical software shop. I'm surely not a PhD in CS, nor am I master leetcoder, but I'm pretty handy with unit tests and a repl in pretty much any environment. This makes my day job so much more rewarding, and I'm able to work a chill job with some fantastic coworkers.

Marriage to the right person: I'll never marry anyone else, but I choose to remarry this woman every single day, 13 years later. She doesn't care about money and isn't materialistic. She isn't addicted to social media. She likes to work hard, then loaf on the sofa with me and play WoW. We're effectively 100% financially compatible. This is so important it's silly. This takes out maybe 99% of potential stress between us. She's effectively already had kids (raised her two much younger sisters, woot they just graduated college!), so we're childless and really happy with that.

Career: learned progamming, made really good money doing it. Taught friends and family how to program, now they've got careers too and are escaping poverty. Feels pretty amazing to help my family really learn how to fish.

Art: picked up violin at age 30, taking lessons every week, I've gotten good enough I can reasonably site read a lot of easy pop pieces. Learned enough painting and drawing to really enjoy figure drawing sessions and portrait painting. Both have helped complement my life considerably.

Materialism: been working on reducing this in myself. I've gone from a medium house filled to the brim with crap down to a small apartment with just a few select high quality items. My day to day emotional enjoyment has increased substantially. It's easier to keep tidy, which helps a lot.

Mental health: I realized that I suffer from anxiety and an in progress with doctors and therapists to sort out how to let go. I'm expecting high dividends from this, so far it's better but not great.

Location: just moved a few miles away, but now I'm surrounded by forest. This is amazing and I love it.

Old car: I just don't care one wit about looking cool or socially signaling with a car. This saves money for actually cool stuff like violin lessons.

What didn't pay off as much as I'd hoped:

Writing a blog then some books: eh it was fine, but didn't even scratch 1% of what I could earn in the open market as a developer. That time better would have been spent focusing on my career.

Baking: contributed to my obesity in my mid twenties, even baking bread is still a hell of a lot of empty carbs

Winter sports: eh they're fine, but really expensive and ultimately something I enjoy just once or twice a year. Also I now have a limp when I sit for too long from a pretty bad snowboarding accident. It goes away when I warm up, but that freaked me out enough to say mostly it's just not worth it for me.

Moving upwards in the tech ladder: became a director of a whole office a few years ago in medium software company. It largely made my life worse for little benefit. I'd previously read the Gervais-Mclod series by michaelochurch and believed it, but wanted to see for myself. Just started the process to step down last month, couldn't be happier. I'd rather be a "loser" than "clueless". Plus engineering is awesome, managing managers and budgets just isn't as appealing to me. As Jack in Stargate says, "I've spent my whole life sticking it to the man. If I do this, I'll be the man. I don't think I can be the man."

Buying a house: this was a wash, netted about the same, and I was pretty careful about it all. Houses have a lot of unexpected costs. I'm glad I did, and glad it's gone.

What I'm hopeful for:

Sailing: I want to get a liveaboard small sailboat. I expect this to be a decent amount of work, but I love sailing and being near the water. Hoping to try this for a year in 2023. Helps that my wife is am A&P mechanic, and she's excited at the prospect of having a project always to work on.

Thru hiking: I plan to attempt the AT in 2022, I'm hoping this becomes a big part of my life. I'm expecting to get into the culture of it, then earn money to live on over winters. I've started several years ago doing more overnight and weekend hikes to get used to it. I'm crushing a 45lb ruck every day to build up tendon/muscular strength, with the goal of building up to a consistent 60lb every day. Already my endurance is really increased, so I'm hoping it's a good way to prepare in advance. Also my gear is pretty locked in at <25lb total, so I'm hoping the heavier rucks translate easily into endurance.

</comment><comment><USER23> Created a subscription based Chrome extension: [https://www.checkbot.io/](https://www.checkbot.io/)

It was a lot of work to get out of the door and I nearly went with one-off payments but it's been going well.

</comment><comment><USER24> Tech stocks, bought them and sat on them.

AMZN 2014

NFLX 2014

TSLA 2014

NVDA 2016

Frankly valuations are getting bubbly now but I'll probably just hold for the next while.

</comment><comment><USER25> Learnt Python to get started with webdev a decade ago.

It let me put bread on the table all this time.

</comment><comment><USER26> Nvidia shares. Put all my savings in them. 8x return if I sell now.

<comment><USER27> What made you do it?

<comment><USER26> That was when I realized deep learning is here to stay, and Nvidia owns the DL hardware market. At the time, everyone said I was crazy to put all my eggs in one basket.

<comment><USER28> "Put all your eggs in one basket and then watch that basket." \- Andrew Carnegie

Always depends on the quality of the basket and the eggs, of course.

<comment><USER29> Also survivorship bias. We wouldn't be having this conversation if NVIDIA had a major catastrophe like Wirecard recently had and owner of basket was unable to sell shares quickly enough for some reason.

All eggs in one basket takes some serious bravery.</comment></comment></comment></comment></comment>


Guix Workflow Language - <AUTHOR> https://www.guixwl.org/ <comment><USER2> A workflow language is only as good as it's engine.

Nextflow was mentioned. I think what most people want is probably closer to Airflow, although it takes some time getting it up to production in a cloud environment (there is astronomer.io and a GCP product).

HTCondor via DAGMan has existed a long time, and there's even engines built on that (Pegasus, Wings).

There's Swift ([http://swift-lang.org/main/](http://swift-lang.org/main/)) and it's successor Parsl. Cray has Chapel. These are a bit different, in that they are more like a distributed computer program. Of course, so is Julia, but built into these languages is the assumption you can be using unreliable, in some way, computing. Makeflow and GNU Parallel are closer to this category too.

Then there's Beam, but that's dataflow.

The crappy thing about this is it's hard to understand when to use a solution and when to not use a solution. Why are there so many solutions? Because there's a ton of different needs, and a lot of these focus on a few in particular:

Latency

Scalability or workers

Dynamic Scalability of workers

Throughput

Polyglot

Integration with existing Schedulers

Workflow Code Management (container support)

Maintainability of very large DAGs

Testability of DAGs/Development support

Execution Management support/Web APIs

Error recovery (especially for long running workflows)

Re-execution capabilities

Provenance tracking

Domain Specificity

Data Management (next to data processing)

... the list goes on.

<comment><USER3> Just regarding Airflow: unless Google has done a lot of work upgrading the internals since embracing Airflow as a supported cloud provider, I would think twice about using it.

It's amazing it works at all in my opinion.

This file [0] contains much of the complexity as a messy, stateful, monolithic block of Python. Having had to chase down deep bugs / limitations in this software, I'm now convinced that Python, with it's GIL, weak typing, lack of concurrency primitives, and generally OOP / imperative style is just the wrong tool for the job.

[0]: [https://github.com/apache/airflow/blob/master/airflow/jobs/s...](https://github.com/apache/airflow/blob/master/airflow/jobs/scheduler_job.py)

<comment><USER4> I'm using Airflow for a lot of critical tasks and it works really well. But I agree that Python may not be the best language to implement a workflow engine.

<comment><USER3> It's fine for moderate workflows. We ran into several hard limits when scaling up, and thought to try to patch some limitations. I think it's got a number of edge cases / scalability issues that will be very hard for them to fix without a full rewrite of the internals.

</comment></comment></comment></comment><comment><USER5> I wonder why it is not in Guile? I thought one of the selling points of Guix was one config language to rule them all. Or is it some syntactic sugar on top of Guile? It's not clear. The page doesn't really explain the syntax clearly anywhere so I'm a bit confused.

<comment><USER6> I think it is actually in Guile. Even though there are not parens in sight, this is a whitespace-based syntax for Scheme.

I don't have a reference at hand now, but I vaguely remember such a comment from the FOSDEM presentation.

<comment><USER7> It is implemented in Guile. The language is just syntactic sugar via macros and utilizes WISP.

[https://git.savannah.gnu.org/cgit/gwl.git/tree/gwl/sugar.scm](https://git.savannah.gnu.org/cgit/gwl.git/tree/gwl/sugar.scm) [https://srfi.schemers.org/srfi-119/srfi-119.html](https://srfi.schemers.org/srfi-119/srfi-119.html)

</comment></comment></comment><comment><USER8> For bioinformatics, take a look at Nextflow. I personally think it is miles ahead of the competition having reviewed about a dozen options out there.

This looks useful, but can it submit jobs to cloud compute clusters or HPC systems and operate locally? Maybe I'm missing the point in terms of the purpose.

<comment><USER9> Nextflow is indeed the best option today. It's sad that it's based on Groovy though, which seems past its heyday as a language/community.

I'm very excited that this is based on Scheme -- I've long wanted a lispy workflow language!

I'm a little concerned about the coverage of the Guix package manager though. I guess instead of writing Dockerfiles the user would have to learn to write Guix packages.

The "Getting Started" example uses samtools so I guess this is oriented towards a similar bioinformatics audience. However without HPC/Cloud support it's probably not too practical, yet.

Addendum: Listened to the FOSDEM 2019 talk, seems like it does support Docker and HPC. However I need AWS Batch support for it to be really useful to me, hopefully that will be implemented at some point.

<comment><USER10> If there is Docker/HPC/AWS support in the system through its local commands, there will be support.. Check the cluster mode setup in guix documentation ([https://www.gnu.org/software/guix/manual/en/guix.html](https://www.gnu.org/software/guix/manual/en/guix.html))

Edit: adding gnu guix manual link

</comment></comment></comment><comment><USER11> I've found that combining Nix with Luigi provides a solution to managing complex reproducible workflows.

[https://luigi.readthedocs.io/en/stable/](https://luigi.readthedocs.io/en/stable/)

"Conceptually, Luigi is similar to GNU Make where you have certain tasks and these tasks in turn may have dependencies on other tasks."

Having this directly type of functionality integrated into Guix (an alternative Nix) looks very interesting. I'd encourage the Guix workflow developers to study Luigi and SciLuigi for inspiration on design ideas.

I will be sure to follow this effort and see how it progresses.

</comment><comment><USER12> For the curious, 'Guix' is pronounced the same way as 'geeks' [0].

0: [https://www.gnu.org/software/guix/manual/en/html_node/Introd...](https://www.gnu.org/software/guix/manual/en/html_node/Introduction.html#DOCF1)

<comment><USER13> Nobody is going to do that.

It's goo-icks.

<comment><USER14> Debian, Ubuntu, UNIX, Linux — none of these have an obvious pronunciation to a native English speaker from USA.

Why take such a defeatist attitude? I'm sure you can get the pronunciation right with a little effort.

<comment><USER15> I also have a hard time saying gaaaa nome or matey like all aboard matey with a straight face or hey you should use the gimp.

</comment></comment></comment></comment><comment><USER16> What is a workflow language exactly? What benefits do they bring vs using say Python?

<comment><USER8> The two main things you want out of a workflow language are re-entrancy and a DAG of job dependencies. Re-entrancy is the ability for the workflow to pick up where it left if something crashes (basically via caching or detecting the presence of expected output files). The DAG is a directed acyclic graph of job dependencies: First do A, then B, then C. The DAG is worked out by the workflow manager, and jobs can be managed accordingly.

A good workflow manager builds on these ideas further by managing environments, job submission, parallelization, cloud/cluster submission, and other options that make processing large amounts of data a lot easier and more efficient.

</comment></comment><comment><USER17> I see a surprising lack of parenthesis for something related to Guix. What's the story behind it?

<comment><USER18> Lisp doesn't need to use s-expressions (i.e. parentheses), that just-so- happens to be the most popular serialisation format. These examples look more like I-expressions to me ( [https://srfi.schemers.org/srfi-49/srfi-49.html](https://srfi.schemers.org/srfi-49/srfi-49.html) ); i-expressions and s-expressions are equivalent and can be converted back and forth trivially.

Lisps can also support arbitrary input formats using reader macros, so it might be using that (I haven't looked at the implementation yet).

<comment><USER19> But with sexps, if your code suffers some formatting catastrophe (such as all instances of whitespace being reduced to a single space), the sexp code is trivially recoverable/reformattable.

Depending on whitespace and indentation creates such fragile code that I can't understand why the trade-off would be made.

<comment><USER18> If that's a concern then serialise using s-expressions. You can still edit using i-expressions or something equivalent if you like (it's trivial to convert, after all). Code on-disk doesn't need to be the same as code in- editor (for example, syntax highlighting isn't saved to disk either)

</comment></comment></comment></comment><comment><USER20> [https://www.commonwl.org/](https://www.commonwl.org/) is pretty good and supported by several workflow engines

</comment><comment><USER21> What are common use cases for this?

</comment><comment><USER22> It would be nice if this language was extensible to running on various compute cluster managers. From what I can tell, these workflows only run on one machine. I like the bioinformatics tool examples though... you can tell who their target market is ;P

</comment><comment><USER23> I've recently been using nipype [0] for workflows, which is fairly domain specific, but pretty nice.

0: [https://nipype.readthedocs.io/en/latest/](https://nipype.readthedocs.io/en/latest/)

</comment><comment><USER24> Would something like this make sense for defining machine learning/data science processes? Like obtain, clean, reformat, split (train/test) datasets.

</comment><comment> 1-6 I love workflow languages. I would really want to make a visual one with input output nodes (in a true GUI fashion). See Autodesk Dynamo for a close concept.

<comment><USER25> There are already a few visual workflow composers, e.g. Rabix,KNIME

</comment></comment><comment><USER25> Q:How many workflow languages/engines does the world need? A: As many as possible: [https://github.com/common-workflow-language/common- workflow-...](https://github.com/common-workflow-language/common-workflow- language/wiki/Existing-Workflow-systems)

<comment><USER26> Is it really so surprising that people continue to iterate and explore the space of possible DSLs -- literally, _domain specific_ languages -- especially when people are solving problems from many different _specific domains_?

<comment><USER25> but many, if not most, are not domain-specific.

<comment><USER27> Depends what you mean by that. For example, in the bioinformatics space it's super common to parallelise a workflow over genomic regions and then merge the results. So I use a tool that has a top level construct for that, literally language syntax which makes that both utterly trivial and extremely robust (for example, deals with the annoying problems of edge effects, overlapping regions, trying not to create breaks in important regions, etc). You can argue all of that is basic parallelism and not domain specific, but in practice it's extremely useful to have these constructs at the language level.

<comment><USER26> That's a really excellent example -- crossing the regionality information of genomics with an otherwise-basic parallelization problem definitely makes it nontrivial. Thank you :D</comment></comment></comment></comment></comment>


JetBrains Space – Early Access Program - <AUTHOR> https://www.jetbrains.com/space/#request-eap-invite <comment><USER2> I really like the idea of a new competitor on this market, and will be trying this soon, but please please stop doing a one-stop tool for everything and just let me plug and play other things, or your own things.

Used to really love things like GitLab, but I dont like bloated sites/tools and IMO things like Space or the direction GitLab has taken create bloat as they start to heavily couple their own tools and "workflows".

</comment><comment><AUTHOR> As a fan of JetBrains products, I thought this looked pretty interesting.

It seems to have everything that GitHub and Azure DevOps have, and more. The fre plan looks to be enough for OSS and small projects, with the paid tiers looking very reasonably priced.

Really interested to give this a try!</comment>


Exfiltrate Files with DNS Queries - <AUTHOR> http://16s.us/dns <comment> X-Istence Dan Kaminsky a couple of years ago did a talk about this, as well as using DNS to cache files...

[http://byteworm.com/2010/10/27/free-content-delivery- network...](http://byteworm.com/2010/10/27/free-content-delivery-network- using-dns-cache/)

Exfiltrating using DNS, or VPN over DNS and the various other techniques are not new.

They do show how difficult it is to police data from leaving ones network.

<comment><AUTHOR> This is an actual working example, not theory on how to do it. IMO, that's what makes it significant. Lot's of people talk about how this can be done, few show actual working examples (with source code) that others can re-create on their networks.

<comment><USER2> Sorry to rain on your parade, but Dan Kaminsky additionally wrote an implementation called ozymandns in 2005 or so:

[http://dnstunnel.de/](http://dnstunnel.de/)

I use it when I need an ssh or web connection in extremely hostile environments that only allow free DNS queries out, like some planes, buses, establishments, etc.

There are also links to other implementations here: [http://en.cship.org/wiki/OzymanDNS](http://en.cship.org/wiki/OzymanDNS)

<comment><AUTHOR> Those examples tunnel traffic over DNS. They do not specifically break large files up into small chunks and exfiltrate them off of a secure network with simple DNS queries.

<comment><USER2> Sure, yes. All those examples (ozyman, iodine, etc.) tunnel arbitrary traffic over DNS, whereas yours is more traffic-efficient in essentially being a static file server via DNS, I suppose?

</comment></comment></comment></comment></comment><comment><USER3> This seems to use base64, DNS is case insensitive so really it should use base32 or some other encoding scheme. However DNS is usually case preserving so it will likely work.

Unless the recursive nameserver in use happens to implement this hack for improved security: [http://tools.ietf.org/html/draft-vixie-dnsext- dns0x20-00](http://tools.ietf.org/html/draft-vixie-dnsext-dns0x20-00)

<comment><USER4> Supported in Unbound with `use-caps-for-id` [http://www.unbound.net/documentation/unbound.conf.html](http://www.unbound.net/documentation/unbound.conf.html)

I love unbound.

</comment></comment><comment><USER5> Don't just get files, use DNS for Command and Control too.

[http://blog.strategiccyber.com/2013/06/06/dns-command-and- co...](http://blog.strategiccyber.com/2013/06/06/dns-command-and-control- added-to-cobalt-strike/)

<comment><USER6> After I wrote that blog post, I also added the ability to tunnel traffic through Beacon when its checking in several times each second. Recently, I added the ability for it to download a large file, a piece at a time, with each checkin. The size of the piece depends on the data channel (DNS vs. HTTP). It's all encrypted too.

\- [http://blog.strategiccyber.com/2013/06/20/thatll-never- work-...](http://blog.strategiccyber.com/2013/06/20/thatll-never-work-we-dont- allow-port-53-out/)

\- [http://blog.strategiccyber.com/2013/07/09/hacking- through-a-...](http://blog.strategiccyber.com/2013/07/09/hacking-through-a- straw-pivoting-over-dns/)

Cobalt Strike is a commercial tool, so it better include the bells and whistles. The OP does a good job of showing code that anyone can play with, right now.

Dan Kaminsky's BlackHat presentations on OzymanDNS are excellent as well.

<comment><AUTHOR> I really like your blog post and your ideas.

With new versions of BIND 10 allowing Python scripting, PowerDNS with Lua scripting and Unbound with Python, I think we'll start seeing more corps controlling DNS queries (or attempting to do so) with whitelists/blackslists, but for the time being, things are mostly wide-open.

</comment></comment></comment><comment><USER7> The author of sqlmap added DNS exfiltration for blind SQL injection last year. Really creative technique (DNS stack doubled the size of sqlmap code-base).

Paper: [http://arxiv.org/pdf/1303.3047.pdf](http://arxiv.org/pdf/1303.3047.pdf)

Slides: [http://www.slideshare.net/stamparm/dns-exfiltration-using- sq...](http://www.slideshare.net/stamparm/dns-exfiltration-using- sqlmap-13163281)

</comment><comment><USER8> 8 bytes at a time.

Wouldn't that mean 100's, 1000's or 10's of thousands of requests for nonsensical subdomains of the same domain name (and that domain name is probably a silly one if you got it recently for 10 bucks).

This is not anomalous DNS traffic? My imagination just does not stretch this far. If the title was just "Transfer a file via DNS", maybe I could play along.

I think nstx preceded iodine.

Here's another one no one has mentioned yet:

[http://www.skullsecurity.org/wiki/index.php/Dnscat](http://www.skullsecurity.org/wiki/index.php/Dnscat)

What I'd really like to see is an implementation of lcamtuf's old, pre- cloud/dropbox idea: daemon caches, specifically recursive DNS caches, as free, (temporary) distributed storage. Anyone can store data for free on 100's of 1000's of networked computers worldwide, otherwise known as recursive DNS caches. Currently we only store "domain names" on these servers, but as the OP shows, it's possible to encode more information into requests than just domain names.

Imagine if the encoded data was an image. With most recursive DNS servers, the data expires upon the TTL expiry. Snapchat via DNS.

</comment><comment> ryan-c This is a hack of mine that stores files in public DNS caches. It's a horrible, hack and slow but it does work.

[https://github.com/ryancdotorg/dnsstore](https://github.com/ryancdotorg/dnsstore)

<comment><USER8> Some caches, like dnscache (and thus OpenDNS), may not respond to non- recursive queries. Would this break your dns_peek function?

<comment> ryan-c Correct, it requires the cache to respond to queries with the recursion bit disabled.

It will also probably behave in "interesting" ways when run against a resolver that's on an anycast IP and doesn't synchronize.

</comment></comment></comment><comment><USER9>
        "When/if the network security team figures this out and
        blocks it, I'll demonstrate a few other ways in which data
        can be exfiltrated."

I loved this line.

He mentions blocking there, but given the technique, could forensics show that this has been used? For example, could some future whistleblower for a national security agency (ours or anyone else's for that matter) use this to exfiltrate files without risk of discovery after the fact?

Could an organization like wikileaks or the guardian use this as a technique for whistleblowers to leak files safely?

<comment><AUTHOR> In a well-monitored environment, it should draw attention and cause them to investigate the internal host making the queries.

</comment></comment><comment><USER10> People have been doing covert channels over DNS in the wild since <2001\. Fast forward 12 years, and this is the new 'my first socket app'.

</comment><comment><USER11> there was a related talk [1] at the usenix 2013 in which this (quite old i might add) method of information ex-filtration was analyzed.

bottom line: amateurs get caught.

[1] [https://www.usenix.org/conference/usenixsecurity13/practical...](https://www.usenix.org/conference/usenixsecurity13/practical- comprehensive-bounds-surreptitious-communication-over-dns)

</comment><comment><USER9> Could gifsockets be used to exfiltrate a file as well?

[https://github.com/videlalvaro/gifsockets](https://github.com/videlalvaro/gifsockets)</comment>


Looking back on "Look Back" videos - <AUTHOR> https://code.facebook.com/posts/1474977139392436/looking-back-on-look-back-videos?hn=1

<comment><USER2>
      - Obtain enough computing capacity

      - Stay within the electrical power limits of our data centers

Somehow it feels like the world is so backwards. Here you have all these smart hard working people with billions of dollars of resources at their disposal and they made a load of kooky videos that people could 'like'.

The videos I'm sure are cool, and I realise that it was quite an achievement, but still, our priorities as a Hunan race are pretty much fucked up.

<comment><USER3> What are the priorities of the Hunan race?

<comment><USER4> Food for everyone, diversify our habitable planets portfolio, higher education en masse, stop killing each other, prevention of diseases, keeping up with the Kardashians, a few others as well...

<comment><USER5> Or at least some spring rolls.

<comment><USER2> This is turning into the funniest thread I've ever started on HN because of a typo. Not very PC, but amusing nevertheless.

<comment><USER3> :P

</comment></comment></comment></comment></comment></comment><comment><USER6> The people from Statigram[1] came up with this idea last year. I have always been interested in how they did it. It wasn't the same scale as Facebook, but I assume it was a big challenge for them, especially not having the resources.

The feature consisted in generating a 15 second video with your most liked Instagram photos and sending it to you as and MPEG email attachments. They had to do this because third party apps cannot post to Instagram.

I initiated my video generation probably at their peak, and still, in less than five minutes, the generated video was in my inbox. If someone from Statigram is reading this, I would really love to hear your side of the story, being able to compare it to facebook's approach now will make it even more interesting to me.

Here's an example of the generated video: [http://instagram.com/p/ibh7zXmvqm/](http://instagram.com/p/ibh7zXmvqm/)

[1] [http://statigr.am](http://statigr.am)

</comment><comment><USER7> To get a sense of scale, the largest DDoS ever recorded by Cloudflare is 400Gb/s[1]. Lookback video was 450Gb/s.

[1] [http://arstechnica.com/security/2014/02/biggest-ddos-ever- ai...](http://arstechnica.com/security/2014/02/biggest-ddos-ever-aimed-at- cloudflares-content-delivery-network/)

</comment><comment><USER8> Really annoyingly, Facebook's selection of photos was way off for me. I've been married over 3 years, with my wife for over 4.5 years, yet it still chose to include photos of ex girlfriends. Facebook has all the data it needs to discern that was a bad idea. Quite an oversight in my opinion as I know several people who it affected too so they didn't publish the video.

<comment><USER9> Ex-girlfriends are part of your past whether you like it or not. I can speak with most of my ex-girlfriends, and it was a nice trip down memory-lane to watch the pictures of me and them.

I am not running from my past, and I appreciate the time that we had together.

<comment><USER10> That's a nice sentiment but do you think your wife or their new beaus feel similarly?

What exactly is the value of the "social graph" if it is never applied by the one company in the prime position to do so?

<comment><USER11> A HN-specific analogy would be: would you work for a startup who can't accept what you did prior to employment?

A wife or new beau who can't accept that you had a past may not be a good idea for the future.

<comment><USER12> touche

</comment></comment></comment></comment></comment><comment><USER13> I assumed the videos were created lazily, on first view. This article makes it pretty clear that they decided from the beginning that they needed to render all the videos in advance. I wonder how many videos they rendered which were never viewed, and whether rendering on first demand would have been viable.

<comment><USER14> I was also expecting amazing optimization tricks, like:

\- Designing an MPEG encoder which can take a couple of pictures and render them on the fly, so only saving the pictures,

\- Carefully spreading the views over a period of time,

\- Associating the soundtrack at view time, since the soundtrack is the same for everyone,

\- Using EC2 for the purpose, deleting all vids afterwards,

But no, they just used the brute force method.

It would be a good job interview question: "How would you serve 700m videos?"

<comment><USER15> \- Using EC2 for the purpose, deleting all vids afterwards

Facebook has their own infrastructure and very well capable of hosting their own. And of course deleting the videos isn't a smart idea

</comment></comment></comment><comment><USER16> Why didn't they just create an html version of this? No need to render a gazillion videos

<comment><USER17> Probably to ensure compatibility for old browsers and mobile users. Obviously the more elegant solution is to do the animation client side and avoid rendering millions of videos but it's not the solution that would have worked for the most people.

<comment><USER18> Considering how facebook has more than a billion users from all the corners of the world accessing it from literally every internet connected device, they had to make something that would work everywhere. Doing it the video way seems like a good decision in this case, at the scale of facebook.

</comment></comment></comment><comment><USER19> Is there really the need to use 80 people for this feature? Why it seems to me that is just because of excessive programming hours in total available at Facebook rather than the real need for this new feature. Just curious.

<comment><USER20> There are no excessive programming hours at Facebook. I assume most of those people worked on the massive backend challenge, not on making the videos (but that's my guess).

<comment><USER19> I understand that the difficulty of making this new feature is scaling the backend, but at this point wouldn't Facebook already have sort of some processes or "frameworks" to follow when making a new feature.

</comment></comment></comment><comment><USER21> > We had anticipated that about 10% of the people who saw their video would > share it. Instead, to our surprise, well over 40% of the viewers were > sharing.

That 10% estimate seems way too low to me. If something like this catches on, it catches on big time. If they didnt expect it to catch on, then why even bother. That said, I never had a problem with videos loading etc during this look back craze. And hardly ever any other time for that matter - it is remarkable how stable that site is.

<comment><USER22> That estimate might be a tacit admission to the problem of all the fake accounts out there.

<comment><USER23> If the estimate was for people who would see the video, why would fake accounts see the video?

</comment></comment></comment><comment><USER24> An impressive engineering task, for sure. But this shows how far we are from the fundamental ideas of the internet and the web, and instead fall back to the good ol' feudalism.

<comment><USER25> Looks like that Schneier article made a dent...

<comment><USER24> In France we had that thing called Minitel [0], which was heavily widespread before the WWW took off (I think it was given to everyone or something like that). The economic (and technical) model was completely different than the internet: you had a passive, stupid terminal, and you connected to a server that offers services for a fee. So you paid for connecting and using a service that could technically not be done on your own minitel (at least in the earlier version). Just like a browser that is completely stupid and allows you to connect to Facebook, Twitter, Google et al. for things you can't do on your own machine. You were depending on a third-party to do things.

We speak a lot about the Minitel model and how it is after all these years a model for all the big WWW players and ISPs to monetize their services. But it's not a common reference, so 'feudalism' works well in this context.

[0] [http://www.reddit.com/r/netsec/comments/1shi77/textsecure_no...](http://www.reddit.com/r/netsec/comments/1shi77/textsecure_now_with_10_million_more_users/ce1uwc1)

</comment></comment></comment><comment><USER26> It's easy to forget given some of the press they get just how active their users are, 720m users who care enough about your product to watch a video about how they use it must be incredible. The algorithm seemed fairly naive but it certainly seemed to work well for some I saw, and really impressive engineering too.

<comment><USER27> > 720m users who care enough about your product

With the product (and the videos) being (about) the users themselves, it starts to make sense.

</comment></comment><comment><USER28> It unbelievable to think about how much synchronization must have gone into this project for all the pieces to come together successfully. Just some more evidence that goes to show facebook has some of the best engineering teams in the world.

</comment><comment><USER29> While I understand they want their own server farms, it is interesting that Facebook does not use any cloud compute. This is almost the perfect usage example and pattern for cloud compute.

<comment><USER23> Facebook has some of the largest data-centers in the world, with plenty of spare capacity. In the article they describe that running these HUGE jobs put them at risk of not potentially not being able to provide power to all the machines at once.

If you're thinking they could have just started up a ec2 job to process a billion videos, I really doubt there's any cloud solution in the world right now that can handle this scale without capacity planning before hand.

</comment></comment><comment><USER30> They should make a video on how they served a billion Facebook likes and where they came from for the people who bought them.</comment>