Most Reported Genetic Associations with Intelligence are Probably False - <AUTHOR> http://dash.harvard.edu/bitstream/handle/1/9938142/Most_Reported_Genetic.pdf?sequence=1 <comment><USER3> Note that the article is _not_ saying that intelligence is not heritable. In fact it makes it quite clear that despite the controversy, the opposite is true:

> Although the exact figures have been the topic of much debate, the claim > that IQ is at least moderately heritable is widely accepted. IQ may in fact > be similar in heritability to the physical trait of height

What the article _is_ saying is that intelligence cannot be tied to _particular_ genes:

> We sought [and failed] to replicate published associations between 12 > specific genetic variants and g using three independent, longitudinal > datasets

It just goes to show that there isn't necessarily one particular factor you can pick out to predict intelligence; it is a combination of many things.

(I shall attempt to avert the impending discussion about the word "intelligence" by clarifying that the sense I am using that word is in the same specific manner that the article is: as an objective measure that is statistically correlated with other measures).

<comment><USER4> For all practical purposes, "intelligence is largely not heritable" is actually closer to the truth than the opposite.

As long as you can't tie intelligence to particular genes, as long as you can't tell what amount of intelligence in an individual is inherited, and how much isn't, there is no use for this in education, politics or social welfare.

What I see the argument being used for, mostly, is to justify racism and entitlement issues in wealthy people...

<comment><USER5> IQ is highly heritable: [http://en.wikipedia.org/wiki/Heritability_of_IQ#Estimates_of...](http://en.wikipedia.org/wiki/Heritability_of_IQ#Estimates_of_the_heritability_of_IQ) (estimates of adulthood heritability are 0.7-0.8, similar to height).

We can reject the existence of an observed natural phenomenon because we can't explain it. We still haven't unified gravity with the standard model, but that doesn't mean we reject that gravity exists (for practical purposes or otherwise).

The fact that a natural phenomenon might be socially inconvenient is even less of a good reason to reject its validity.

Finally, I don't even see why it should even be so socially inconvenient. Say the experiments are right, and intelligence is mostly heritable rather than a result of education, upbringing, etc. Why should that justify entitlement on the part of the wealthy? Indeed, it should cut across that grain. If its mostly heritable, being intelligent shouldn't be seen as intrinsically virtuous any more than is being tall. Moreover, the heritability of intelligence may very well imply a need for a change in our priorities when it comes to social welfare. E.g. maybe we need to de-emphasize education, and divert resources to building up social and family networks.

<comment><USER4> I qualified my statement with "for all practical purposes". I have often claimed that there is no practical relevance and no reasonable way to use this "fact" in practical real world decisions. Nobody was able to tell me one that holds up to even superficial checking.

"Socially inconvenient" doesn't even come into it, because twin studies don't tell us anything about differences between two ethnicities. For that matter, genetic diversity within populations might have a larger effect...

The "intelligence is genetic" "fact" is often used to justify refusal to tackle socio-economic problems (like "under-achievement" in blacks or hispanics in the US, or the turkish minority in my home country) with socio- economic solutions. Because hey, it's all genetics, so we can't change it. Even if measures for improvement have been proven...

<comment><USER6> "intelligence is genetic" and "intelligence is heritable" are completely different statements. The notion that IQ is fixed and encoded in genes is nonsense. What is true is that for a given environment, the variation in observed IQ is strongly explained by genetics. It is possible, and likely true, that lower intelligence among minorities is caused by the environment, but that the variation among those in this environment is explained by genetics.

The fact that intelligence is heritable says nothing about social-economic policy for changing the environment. It is as absurd as saying that because "height is genetic" there is no reason to worry about malnutrition.

</comment></comment></comment></comment></comment><comment><USER7> This whole field seems likely to be a case study supporting [http://xkcd.com/882/](http://xkcd.com/882/).

But the research does have a questionable assumption. On p 5 it says, _" Such a result would not likely be due to differences in the methods used to generate g in the various datasets under comparison, since g is consistently measured by a wide variety of tests."_

This is true. However it is also true that, as [http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/523.htm...](http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/523.html) points out, that all evidence we have is consistent with g being a correlation between different ways of weighting a large number of independent factors. If so, then it is easy to have two test measuring g that are fairly well correlated with each other, one of which picks up a particular factor and the other does not. If a gene affects that factor, then the test you use to measure to find these correlations will matter.

<comment><USER8> > This whole field seems likely to be a case study supporting > [http://xkcd.com/882/](http://xkcd.com/882/).

The scientifically interesting thing is that it's not at all like that XKCD cartoon. Multiple hypothesis testing is built into the design of the studies, and has been from the very start. When it comes to proper interpretation of stats in science, geneticists are at the top of the hierarchy.

Your second point is far closer to what I think is going on, but I still think that single-gene studies are the wrong way to go, as you might guess from my username. They're testing for the wrong thing, single locus or single gene causes, of a very complex phenotype with the involvement of many many different loci and genes. So if there are 500-600 genes that are important, maybe one passes the test in your particular subsample, but in a different subsample the rest of the 500-600 genes mediate the effect. I.e. there's not enough complexity in the model to account for the signal. What's worked very well for simple diseases and traits may not work well for complex traits with the influence of many many different genes.

<comment><USER7> _Multiple hypothesis testing is built into the design of the studies, and has been from the very start. When it comes to proper interpretation of stats in science, geneticists are at the top of the hierarchy._

It is built into the design of the studies. But studies that do not find results do not get published. Thus there is a lot of room for exceptional results to sneak through despite the attempts to get the stats right.

<comment><USER8> GWAS studies are extremely expensive studies, often in the hundreds of thousands of dollars in SNP chips alone, and will get published somewhere, somehow. If they don't find results they get published in lesser journals, because there's always _something_ to be said even if there are not significant associations.

It's the same with academic clinical trials, even negative results get published, because for those academics they still need something to show, and a publication of a negative result is better than nothing.

Non-publication bias is more common in less expensive, more exploratory settings, where there are lots of little experiments going on all the time. This is quite different from GWAS.

</comment></comment></comment></comment><comment><USER9> Heritability is related to the degree to which a trait is genetically determined. The relationship is far from trivial.

Skin color makes this much easier to understand. Imagine a couple, where the man and the woman each have one Scandinavian parent and one Bantu parent. Their children could have almost any skin shade.

If they have identical twins, those twins will have the same shade of skin. Skin color is genetically determined but the heritability varies.

Imagine two people who have the same skin color as our first two, but they come from an island population founded by Norwegians and Nigerians exclusively. They are 20th generation, or 50th. Now, it is likely the skin color of their children will be the same as their own.

Intelligence inherits upon many factors. Even identifying those factors proves to be elusive and hard. The good evidence that intelligence is genetic is twin studies: the good evidence that it is variable in heritability is every family you've ever known.

</comment><comment><USER10> Boring. The effect size of the variants is too small to find with n=10,000, hence any positive findings are almost certainly false positives which will fail to replicate. That's why the 3 SNPs found recently ([http://infoproc.blogspot.com/2013/05/first-gwas-hits-for- cog...](http://infoproc.blogspot.com/2013/05/first-gwas-hits-for-cognitive- ability.html)) were in a sample of n=101,000.

There's nothing here to undermine the view that genetics causes much of variation in intelligence, it's just that the variation wasn't as clustered on a few genes as people hoped because it would make the research so much easier.

</comment><comment><USER11> The now famous paper _Why Most Published Research Findings Are False_ ([http://www.plosmedicine.org/article/info:doi/10.1371/journal...](http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124)) singles out genetics as a field particularly prone to false positive findings.

</comment><comment><USER12> I would assume that many behaviours (including language itself) are inherited - and yet have nothing whatever to do with genetics. How would you control for this?

Only last week, we had an article on here about how vocabulary in very young children was a function of how much parents talked to their children - <USER2> marked differences between economic groups[1]. There are many many systematic errors in this kind of data, and I have no good (ethical) suggestions in how to eliminate them.

[1] [https://news.ycombinator.com/item?id=6480155](https://news.ycombinator.com/item?id=6480155)

<comment><AUTHOR> _How would you control for this?_

The classic way to control for cultural and in-family influences in studies of human behavior genetics is to look for the rare cases of monozygotic ("identical") twins brought up in separate households after adoption. Those studies of monozygotic twins reared apart (MZA twins) set a MAXIMUM figure for the heritability of any trait of interest--because the genes are as similar as they can be between two different human individuals, but the environments, even in separate households, may not be as maximally different as they could be in influence on the trait in question. There is a whole book about the most famous study of identical twins reared apart, which took place at my alma mater while I was an undergraduate and postgraduate student there (in other subjects).

[http://www.hup.harvard.edu/catalog.php?isbn=9780674055469](http://www.hup.harvard.edu/catalog.php?isbn=9780674055469)

As some people will remember from the press reports at the time, the most culturally variant upbringing two of the separated twins, Oskar and Jack, studied by the Minnesota researchers received was one twin being raised as a Jew, and the other twin being brought up in Nazi Germany and becoming a member of the Hitler Youth.

[http://online.wsj.com/article/SB1000142405270230355210457743...](http://online.wsj.com/article/SB10001424052702303552104577436333754014866.html)

<comment><USER12> I guess I was saying that the rarity of this happening in the real world makes good statistics difficult (which are _essential_ in biological systems,) and of course intervening to force better statistics is a rather horrific proposition.

Thank you for the links though, I'll have a read!

</comment></comment></comment><comment><USER13> " _These and our other results, together with the failure of whole-genome association studies of g to date, are consistent with general intelligence being a highly polygenic trait on which common genetic variants individually have only small effects._ "

This doesn't seem like a surprising result to me. What is being claimed is not "intelligence is not genetic" but it is that "heritability of intelligence is complex". For something that appears as complicated as human intelligence, wouldn't this be expected?

<comment><USER14> Things that are "expected" still need to be tested. Also, [http://lesswrong.com/lw/im/hindsight_devalues_science/](http://lesswrong.com/lw/im/hindsight_devalues_science/).

</comment></comment><comment><USER15> Nature vs nurture is hard to separate, especially when genes and culture tend to be 'handed down' to the same individuals.

However I'm curious that they chose to use the very-powerfully-determined-by- genes features of Alzheimers and BodyMassIndex as their control. Surely there are other genetic traits that are not 'disorders' which could have been referenced? Breakdowns in physical function would naturally have larger measurable effects than, say, alcohol metabolism rates or whatever.

<comment><AUTHOR> _Surely there are other genetic traits that are not 'disorders' which could have been referenced?_

Studies of the issue of "missing heritability" have often investigated height (bodily stature), which has a very high calculated heritability (resemblance among close relatives) but which must be influenced by hundreds of genes, many of very small effect. Even very extensive genome-wide association studies (GWAS) in large samples find NO gene with large effect on height, and fail to detect most of the presumed associations with height in gene associations investigated in such studies.

The review article Johnson, W. (2010). Understanding the Genetics of Intelligence: Can Height Help? Can Corn Oil?. Current Directions in Psychological Science, 19(3), 177-182

[http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/...](http://apsychoserver.psych.arizona.edu/JJBAReprints/PSYC621/Johnson%20Current%20Directions%20Psych%20Science%202010%20\(G%20and%20E%20in%20IQ\).pdf)

looks at some famous genetic experiments to show how little is explained by gene frequencies even in thoroughly studied populations defined by artificial selection.

<comment><USER4> One example for the trouble with genetics in intelligence is the issue of Malaria: If a child gets Malaria, it will have a lower intelligence quotient. Resistance/resilience of Malaria sure has genetic factors. Would those factors show up in genetic studies about intelligence? Betcha.

Should we accept this variance in IQ as genetic and unavoidable? Or should we just buckle down, eradicate Malaria and stop worrying about these factors?

</comment></comment></comment><comment><USER4> No surprise there...

Knowing a lot about genomic animal breeding made me wonder how people can be so sure about genetic basis of intelligence, while we understand neither intelligence nor the genetic basis behind it. For that matter, even plain body height in humans is barely understood...

</comment><comment><USER16> Do we even have a working definition for intelligence at this point that can't have holes poked in it from every direction?

<comment><USER17> No. A unified definition may be impossible.

But not having a concrete definition doesn't make certain inferences impossible.

</comment></comment><comment><USER18> Genes are to humans in the same way CPU instructions are to computers [1]. So this experiment is analogous to measuring the capability /throughput of a computer only by inspecting existence of arbitrary instruction sets. They may be the atomic enabling factors, but can't really say much about something so complex as intelligence.

[1]: Richard Dawkins

</comment><comment><USER19> Wouldn't intelligence be 100% genetic. I mean humans have intelligence and insects have way less. And all that programming to make one or the other is encoded in the DNA.

<comment><USER20> You could just as easily argue that it's 100% environment, since a human who is never provided with food will end up dead, and therefore much less intelligent than an insect.

</comment></comment><comment><USER21> "African Americans currently score lower than European Americans on vocabulary, reading, and mathematics tests, as well as on tests that claim to measure scholastic aptitude and intelligence. This gap appears before children enter kindergarten (figure 1-1), and it persists into adulthood. It has narrowed since 1970, but the typical American black still scores below 75 percent of American whites on most standardized tests. On some tests the typical American black scores below more than 85 percent of whites"\- [http://www.nytimes.com/books/first/j/jencks- gap.html](http://www.nytimes.com/books/first/j/jencks-gap.html)

"Intellectual and personality measures were available from unwed mothers who gave their children up for adoption at birth. The same or similar measures have been obtained from 300 sets of adoptive parents and all of their adopted and natural children in the Texas Adoption Project. The sample characteristics are discussed in detail, and the basic findings for IQ are presented. Initial analyses of the data on IQ suggest moderate heritabilities." \- [http://www.ncbi.nlm.nih.gov/pubmed/496798](http://www.ncbi.nlm.nih.gov/pubmed/496798)

"Adolescents' IQ test scores were similar to those of their parents and siblings only if they were biologically related. Our interpretation of these results is that younger children are more influenced by differences among their family environments than older adolescents, who are freer to seek their own niches." \- [http://www.ncbi.nlm.nih.gov/pubmed/6872626](http://www.ncbi.nlm.nih.gov/pubmed/6872626)

Notice how carefully worded the "interpretation" of that last one was. Rather than suggest that genetics has to do with older adolescence not having IQ scores that match their adoptive parents their "interpretation" is that the adolescences simply chose not to perform as well as their adoptive parents...Anything to avoid controversy.

White guilt is getting in the way of a lot of racial/genetic/intelligence/violence studies. People don't want to go back to a time when one group of humans were seen as superior and another inferior. The funny part is, Intelligence wise, whites are NOT superior. Jews are.. Their intelligence levels are higher, achievement levels higher, hackers should know this. Many of Silicon Valleys most prominent companies were started by jews. Facebook, Craigslist, Google. Also a report by Joel Stein from the New York Times showed that every major Hollywood studio was founded by jews. In the fashion world: Ralph Lauren, Levi (Jeans), Guess, Mark Ecco, Men's Warehouse, and most of the brands you see in stores today were also started by jews.

"In 1921 Budapest, 88% of the members of the stock exchange and 91% of the currency brokers were Jews, many of them ennobled. In interwar Hungary, more than half and perhaps as much as 90 percent of Hungarian industry was owned or operated by a few closely related Jewish banking families. Jews represented one-fourth of all university students and 43% percent at Budapest Technological University. In 1920, 60 percent of Hungarian doctors, 51 percent of lawyers, 39 percent of all privately employed engineers and chemists, 34 percent of editors and journalists, and 29 percent of musicians identified themselves as Jews by religion. Resentment of this Jewish trend of success was widespread." \- Yuri Slezkine. The Jewish Century. Princeton, 2004. ISBN 0-691-11995-3

Since 1921 theres been a lot of catching up by Whites, Blacks, Asians, etc... but Jews are still disproportionately successful and score high.

<comment><USER21> Continued: I think it's pretty obvious that genes have quiet a bit to do with intelligence and achievement. There have been too many studies done on race and iq, race and crime, race and achievement to just throw out. I'm not the smartest but at least I'm living in reality. And I'm proud to see so many HNers here doing the same and acknowledging the important role genetics has in our lives. It's good to know there's more people following truth and science no matter how blasphemous it may seem to a delusional emotion-oriented general population hell bent on believing that everyone is "100% equal" in order to feel good about themselves.

Nature is racist. Us finding out about it is not.

<comment><USER22> It's only racist if you think the value of a person is tied to his intelligence. Achievement is definitely related to intelligence, but I don't think achievement makes one better than another.</comment></comment></comment>


Tokenising the english text of 30TB common crawl - <AUTHOR> http://matpalm.com/blog/2011/12/10/common_crawl_visible_text/ <comment><USER2> Having the full Common Crawl uncompressed in disk with some indexing and RAID-1 is becoming quite feasible (easily under $10k). The size of the text part of the web is growing a lot more slowly than the price of storage for maybe 10 years. At this rate anyone would be able to have a full crawl of the web in their mid-range desktop computers in just a few years. Here's hoping for good weather in Thailand. :-)

You need how much right now, 200TB tops? including index files, uncompressed data and replication. With some clever compression and data structures you can probably cut that by half or less. Financially speaking this is already on local club territory.

</comment><comment><USER3> Very good to see an example of working with the common crawl. I would encourage YC applicants to think about what kind of opportunities it presents.

</comment><comment><USER4> Very interesting. Surprising to see Lithuania as the second most popular language. Oddly enough, I was having a conversation with someone this weekend when they mentioned that Lithuania had the highest fiber-to-the-home penetration in Europe and one of the fastest average internet connections in the world. Does anyone know why this is?

<comment><USER5> I once tried to identify the language in (supposedly) English-language Facebook posts. The second-most frequent language was Estonian. Looking at actual posts classified as Estonian, it turned out they contained text like "Soooooo nice". Because the language identifier looks at character sequences, and Estonian tends to have unusually common double vowels, such posts were classified as Estonian. Dutch and Norwegian were other two languages for which English was mistaken - apparently some character sequencies there have frequency distributions that are similar to English.

</comment></comment><comment><USER6> A bit premature to post this - no results yet, he's still working on it.

<comment><USER7> +1</comment></comment>


Ask HN: How would you market this site? - <AUTHOR>

I made a site that allows you to find out if a crush is mutual without having to embarrass yourself. Something I perceived to be a useful service (which may or may not be true). How should I market this site? How can I validate that the site actually has value for people? (I created a test Facebook ad campaign - we'll see how that goes) http://www.luckychat.biz <comment><USER2> Personally I think a FB ad or pitching to friends is a little too impersonal and not a high enough likelihood of many crushes between that group. Instead I'd aim for a small but well interconnected network first, like a single dorm building. The next week, the next one over. Eventually, it could own the campus' attention for a bit, then maybe the next school nearby. Getting a single dorm on board could be as simple as setting up a table in the lobby with a couple laptops and giving out donuts to people who sign up.

<comment><USER3> This is very good advice. You need to get a community that is intimate enough that there will be crushes found. That's your successful customer. People aren't likely to invite their crush for fear of outing themselves, so they would have to be exceedingly lucky to get their crush there through random invites.

<comment><AUTHOR> I think I might be able to do this. Its getting the first users on the site that is that hardest part. But in a few hours, I will go out and see if I can start recruiting users. (Coding is far easier than motivating people)

</comment></comment></comment><comment><USER4> I think having a bit of an explanation of what the site is and how it works would be very helpful. Having a video of you using the website to show some features while also explaining how the site can help people will go a long way. Other than that I would recommend starting small and getting everyone you know to talk about it. Perhaps you should also try to market it on a local college campus if there is one near. College kids are always looking for ways to meet people who are interested in them (read: get laid) and I'm sure would be glad to check your stuff out. Or perhaps you could try and get in touch with the guys from like a little and see if you could do something with them.

<comment><AUTHOR> I think a video is clearly needed. Although it might take me a while to put one together.

<comment><USER4> All you really need, at least at the beginning, is you using the service with a voiceover explaining what you are doing.

</comment></comment></comment><comment><USER5> I think you need to add an explanation of the site, the concept, and address how it works. I think most people may be scared away by not having a good understanding of how it works, especially being that it has to do with their personal lives.

<comment><USER6> ...and that it requires giving access to all sorts of Facebook permissions when all the user knows about the site beforehand is "LuckyChat - Connect with someone special"

A general concern that I would have is that there have been various websites over the years that address this problem (the first one I remember was called something like CrushMail back in 1999 or so and the most recent was the pivot made by Breakup Notifier after Facebook shut them down (<http://apps.ycombinator.com/item?id=2264660)>). But often, they end up being mostly a way for people to spam their friends instead of people getting value out of it. (Typically, they ask you to enter email addresses of who you think has a crush on you. The site then uses that information to notify those people that someone has a crush on them. So when you find a 'match' it could just be that both people entered the others' name on there thinking that they might be the one to have a crush on them.)

<comment><AUTHOR> Comments like these is why I love HN. Thanks!

</comment></comment></comment><comment><USER7> Hey! When I saw your post for luckychat on Hacker News, I thought it was a good idea. A while back, I had an idea for something similar to this, and was glad to see someone finally implementing it. My suggestion for marketing the site would be to have every one of your Facebook friends start using it (if you know them all), and have them post that they are using it. Start small, and then try to grow the app.

</comment><comment><USER8> The tagline should be a bit more clever and tell you what's unique about the service. I get the abstract sense that this site has some connection to dating, but you haven't given me a reason who I should take the time and sign up with my Facebook ID. Also from a branding point of view the site feels a impersonal a minimal to me.

</comment><comment><USER9> Doesnt this idea exist.. everywhere? Orkut had this feature.

<comment><USER9> to advertise though. I would target a specific middle school or highschool.

</comment></comment><comment><USER10> Make a landing page that explains what the app actually does.

<comment><AUTHOR> I'm going to work on making a video clearly demoing the site and probably rework the entire landing page somewhere along the way</comment></comment>


iOS 9 allows access to photos and contacts on a passcode locked iPhone - <AUTHOR> http://www.idownloadblog.com/2015/09/20/ios-9-access-photos-contacts-locked-iphone-security-flaw/ <comment><USER2> Reminds me of a similarly Rube—Goldberg-esque login screen circumvention in Windows 95:

[https://media2.giphy.com/media/lN0A3biIDKepW/giphy.gif](https://media2.giphy.com/media/lN0A3biIDKepW/giphy.gif)

<comment><AUTHOR> Right, we saw some of bypasses like this lately.

And I believe we'll see more, because we add more and more features to the lock-screens.

</comment></comment><comment><USER3> "This _only_ allows users to view your contacts, and look at your photos (not videos) through a limited interface. Photos cannot be forwarded or shared from your iPhone"

 _only_? Isn't that bad enough? Anecdotally I really couldn't care if any of the messages I have get out in the open as for some reason I don't have sensitive content in them, but some pictures on the other hand are really not meant for everybody, especially when they aren't even sorted out yet, and I'm sure I am not alone.

Also: taking a picture of a phone with a picture still preserves most of it and so they can in fact be - what I would call - forwarded.

</comment><comment><USER4> There have been quite a few of these exploits over the years, where you can trick a program running in locked mode into thinking it's in unlocked mode. It seems like they really need to have completely separate programs to run in locked mode that can't access anything except via defined APIs.

<comment><USER5> I think object-capability based access control would solve the problem as well. If the login screen is only given the capability to talk to the authentication program, then even if you could force opening photos, you wouldn't have the capabilities to hand to the photos app to actually view them.</comment></comment>


Windows 8 Usage Sinks Below Vista Levels - <AUTHOR> http://www.dailytech.com/Windows+8+Usage+Sinks+Below+Vista+Levels/article29546.htm

<comment><AUTHOR> essentially ...

"Market research group Net Applications, a research service that tracks traffic across 8,000 affiliates' sites and 3 million registered users, reports that Windows 8 at the end of December accounted for a mere 1.72 percent of traffic."</comment>


Ask HN: Best open-source tool to help create a network architecture diagram? - <AUTHOR> Hi HN - I'd like to be able to map my network and understand what software is deployed (along with versions, connections / APIs, and other details). Are there any open-source solutions? Ideally the output is a list of tools along with a visual which I could use as the starting point for a network architecture diagram. Thanks! <comment><USER2> Use [https://github.com/digitalocean/netbox](https://github.com/digitalocean/netbox) to record your network information. Then you can create topology maps with dot/graphviz. [https://netbox.readthedocs.io/en/latest/additional- features/...](https://netbox.readthedocs.io/en/latest/additional- features/topology-maps/)

Someone has a tutorial on using graphviz directly: [https://mikegriffin.ie/blog/20110308-a-graphviz- tutorial](https://mikegriffin.ie/blog/20110308-a-graphviz-tutorial)

<comment><AUTHOR> Thank you - this is very helpful

</comment></comment><comment><USER3> I have used [http://draw.io](http://draw.io) before, it works well. This is the source [https://github.com/jgraph/drawio](https://github.com/jgraph/drawio)</comment>


Deep Learning Framework Caffe Releases RC4 - <AUTHOR> https://github.com/BVLC/caffe/releases/tag/rc4 <comment><AUTHOR> Hmmmm, it looks like the version string in RC4 hasn't changed...

$ grep "VERSION_" Makefile

      DYNAMIC_VERSION_MAJOR 	:= 1
      DYNAMIC_VERSION_MINOR 	:= 0
      DYNAMIC_VERSION_REVISION 	:= 0-rc3

...</comment>


Content paywalls on the agenda for digital news sites - <AUTHOR> http://www.ft.com/intl/cms/s/0/a8daf5d0-7892-11e5-933d-efcdc3c11c89.html <comment><USER2> Ironically behind a paywall.</comment>


The Fiction of Self-Exposure - <AUTHOR> http://www.newyorker.com/books/page-turner/the-fiction-of-self-exposure

<comment><USER2> The short version: _" Sex and food and self-congratulation: this could also serve as an ungenerous summary of most social media."_

<comment><USER3> And for that matter of a lot that is published in print, in The New Yorker among other places.

</comment></comment><comment><USER4> The title should be, _The Self of Exposure Fiction._</comment>


The Law of Leaky Abstractions (2002) - <AUTHOR> http://joelonsoftware.com/articles/LeakyAbstractions.html <comment><USER2> I continually find this particular point useful and enlightening. (Joel's whole corpus of stuff is worth a read. At the risk of causing a hacker news "joel storm" of submissions, check out the pricing/segmentation piece: [http://www.joelonsoftware.com/articles/CamelsandRubberDuckie...](http://www.joelonsoftware.com/articles/CamelsandRubberDuckies.html))

For example, after discussing the subject of promises (in the sense of the tool to manage and abstract asynchronous program flow) with a colleague, it occurred to me that the _reason_ we care about async behaviour is that a "function call" is a leaky abstraction, in that it takes a variable amount of time.

We think of a function call as "doing X, Y and Z". The fact that it takes _time_ to do it, blocking our 'thread of execution' causes us the problem that we can solve with the overhead of async programming (or threading, or...)

(By 'function call' I'm glossing over system call, library call, embedded cpu- eating loop.)

From this perspective, you can see that an alternative to working around the leak is plugging (some of) the leaks - which is the approach golang takes, by shuffling around threads of execution to avoid them being stalled by blocking system calls.

That perspective is useful to me and helps me understand why I think I prefer the golang approach, "more robust abstraction".

<comment><USER3> Agreed about Joel. When I first read his "corpus", it was already a few years old. But whenever you get to it, get to it.

My only complaint is that once you find some _really good_ writing in this field, you realize how hard it is to come by (in print, anyway). I happened to find "Joel on Software" in the "Professional Computing" section at Barnes & Noble. That and "Coders at Work" are the _only_ books worth reading I've ever seen in decades of browsing that section. Still, I'm glad I came across them.

</comment></comment><comment><USER4> Leaky abstractions are precisely why it's important to understand what's going on at least one level below where you're working at. This has always been true of computers, whether you're working at a script level, system language level, assembly, or even hardware. Eventually something will break the abstraction, and on that day you'll be there looking at that stack trace or register + memory dump or burnt out nand gate or cold soldered capacitor or faulty wall circuit, and not knowing what to do about it.

<comment><USER5> The constant churn of web technologies make this really frustrating, because you have an entire ecosystem of people that doesn't know what their tools do. "Just run re-war again, dude, that will fix it."

</comment></comment><comment><USER6> This is a concept that I've discussed frequently with team members. Related: if you abstract away the differences between A and B, you likely lose some of their unique advantages. A and B could be databases, hard drives, etc. It may be worth it, but it's a tradeoff.

</comment><comment><USER7> Can every word be considered an abstraction?

An abstraction in mathematics is a simple, small idea that we use to describe and understand a large variety of objects. For example the concept of an algebraic group is an abstraction that can be applied to numbers, functions, matrices, etc. We can understand all these objects using the same concept.

Are all words this way? Take the word "orange". Are all oranges the same? They all have variations in taste, texture, color, etc. They are all physically separate objects. But we use the same idea to understand and describe them. Maybe this is more clearly seen with the word "fruit".

This applies to verbs too - think of the verb "to run". Usain Bolt runs in a very different way from normal people, but we use the same concept to understand what they do.

These abstractions "leak", i.e. cause confusion when the underlying reality doesn't match up. Is a tomato a fruit? If I trained as an Olympic sprinter would I have to abandon misconceptions about how to move my body, misconceptions caused by a lifetime of using the word "run"?

Is every word an abstraction?

</comment><comment><USER8> The examples don't support the argument. It's not like writing your application using UDP and expecting some packet loss is going to mean that your application continues to work when someone unplugs the network cable. And the other examples aren't really _leaks_ , just performance disadvantages. I might accept an argument that any sufficiently powerful abstraction imposes a performance penalty, but that's a lot less interesting.

I've seen a single-platform JVM bug. _once_. The only definite OS failure I've ever seen was bad RAM, and you could write a bootable program that ran without an OS and that wouldn't protect you from bad RAM. I've seen _one_ CPU flaw, and linux abstracted over it seamlessly. I've never seen a failure that was the result of analogue chip behaviour. There are plenty of non-leaky abstractions around.

<comment><USER9> I think saying that some examples are just performance disadvantages rather than leaks misses the point. "Leaky" is a metaphor here, so assuming that any kind of actual data loss is necessary for it to be the case seems incorrect to me.

For example if your SQL query is too slow from a pragmatic point of view and then you go ahead and fix it by using optimizer hints then the abstraction of the procedural querying process provided by SQL has just failed you as it hasn't worked.

I agree that some abstractions are very good, but that does not make Joel's point moot. The key thing is to understand that abstractions are always some sort of tradeoff between the virtues of directness and the virtues of productivity.

Btw: I am really surprised that this essay had never been posted to HN before.

<comment><USER8> > For example if your SQL query is too slow from a pragmatic point of view and > then you go ahead and fix it by using optimizer hints then the abstraction > of the procedural querying process provided by SQL has just failed you as it > hasn't worked.

Well sure. But I've never had that happen; is the A = B and B = C and C = A problem the article talks about one that causes problems in real-life code?

<comment><USER5> I've had to rewrite _lots_ of SQL queries (for Postgres) that an intelligent database should have just handled for me. I'd prematurely exclude certain columns that it could be mathematically proven from the query I would never need and see amazing speed-ups.

(I'm not picking on Postgres, it's a wonderful DB, but you need to understand how your tools are working.)

</comment></comment></comment></comment><comment><USER10> A true leaky abstraction is where the behavior of the interface is forced to reflect some of the complexity of the implementation, but in a way that is surprising given the rest of the interface.

I think most of his abstractions are not leaky, but they come at the cost of poor performance in certain cases.

In any case, his point is correct: it's hard to use complex systems to implement a simple interface, because it's hard to create the glue code so that the simple interface drives the implementation in the "right" way in every circumstance.

There is, of course, another approach that the author didn't mention, which is providing a mechanism in the interface for "hints", e.g. "please store this matrix in column-major format". That is not a perfect solution, but it is another tool for better abstractions.

</comment><comment><USER11> Here's the actual law:

        > All non-trivial abstractions, to some degree, are leaky.

It itself is somewhat leaky: it never defines "non-trivial"! Perhaps "non- trivial" just means "complex enough to be leaky", rendering the whole thing a tautology.

I've been thinking about different philosophies behind abstraction for a while. My ideas are a little unformed, but I think they're still useful. Basically, as I see it, there are two different ways to define abstractions.

There's the usual way: you come up with a new notion that has additional capabilities. The abstraction _adds_ to the underlying concept. Perhaps it also hides some detail, but the main action is _adding_. Some of the leaky abstractions from the post really fit this: TCP, for example, adds new capabilities and behavior over the underlying protocol; C++ strings add a bunch of string-specific capabilities over char arrays; NFS adds filesystem capabilities to a network connection.

These abstractions are largely "leaky" because what they add does not always map nicely to what's underneath. They're also interesting because each abstraction only fits _one_ thing--TCP is just for network communication, C++ strings are just text in char arrays and so on. Ultimately, all these abstractions are just more structure over whatever they're based on. Again, the key idea is _adding_.

The other sort of abstraction is what I think of as "algebraic" or "Haskelly". Instead of _adding_ , these abstractions _take away_. They expose some feature of the underlying concept and throw everything else away. A monoid, for example, just exposes an associative operation from whatever set (or type) it's defined over. It can't really be "leaky" (at least in the same way) because it's showing what's already there. The normal sort of abstraction is like a structure built on top of a foundation; this sort of abstraction is like a lens or a view into a property of its foundation.

A very important result is that these abstractions are very independent of any basis or use-case. Monoids come up _everywhere_. Strings form a monoid. Numbers form a monoid (in many different ways, even!). Booleans form a monoid. Certain classes of functions form monoids. Note how I said "form": even if we ignored it, the monoidal structure would still be there. Talking about these things as monoids doesn't add anything but just highlights something already present.

Instead of adding capabilities to the underlying object, this sort of abstraction instead exposes structure shared by otherwise distinct objects. Monoids represent things you can combine; groups represent things that are somehow symmetric; rings represent things that are sort of like numbers; fields represent things that are really quite a bit like numbers; F-algebras represent things that are like any of the other abstractions I've listed (meta-abstraction!); functors represent things that can be transformed while preserving structure and so on.

These abstractions also let us talk about different things uniformly and generically. We can reason about and write code against these abstractions without worrying about any particular underlying object. By doing this, nothing _but_ the abstracted structure can do anything--details of any given thing can't leak through because we have to be generically compatible with _every_ instance of the abstraction.

Functors are a great example of this. If we wrote code using lists, we could do all sorts of things: always return an empty list, add elements to the end, drop elements from the front, change the order... If we wrote a code just using a _functor_ , it would still work for lists but all we'd be able to do is map a function over the elements. Nothing else. Always returning an empty lists wouldn't even be a meaningful construct! Clearly, using this abstraction has taken away our capabilities, but it turns out this is often a good thing.

But this is where "non-trivial" comes to bite us. Are any of the abstractions I listed actually "non-trivial"? I'm not sure I could claim that. To be generally applicable, all of them have as little structure as possible. Ideally, you want to limit your capabilities as much as possible to express whatever you care about. This usually means limiting them _a lot_. All of the things I listed can be expressed as a handful of functions and a handful of algebraic laws--a few lines of description is enough to understand them _completely_. They are nowhere _close_ to something like TCP!

Ultimately, I am not even sure it really makes sense to compare the two sorts of abstractions. But they are certainly both approaches to _abstracting_ , albeit in different ways. It's something that's worth thinking about the next time you have to design or use an abstraction yourself.

<comment><USER12> ... and in the end you still have talk over TCP and deal with that leaky abstraction, or you accidentally write O(N^2) code that could be expressed easily in O(N) (compare [https://github.com/nominolo/HTTP/commit/b9bd0a08fa09c6403f91...](https://github.com/nominolo/HTTP/commit/b9bd0a08fa09c6403f91422e3b23f08d339612eb) for a recent HN submission) because the "taking away" approach at abstraction also abstracts out some important performance considerations.

<comment><USER13> TCP is done via the IO monad and this is an excellent example of what the IO monad is used for---it forces you to realize that there's another layer of (this time quite leaky) abstraction over your potentially simple inner abstraction.

</comment></comment></comment><comment><USER14> I used to think all abstractions were leaky until someone explained parametric polymorphism and free theorems to me.

[http://ttic.uchicago.edu/~dreyer/course/papers/wadler.pdf](http://ttic.uchicago.edu/~dreyer/course/papers/wadler.pdf)

Every programmer or computer scientist should read that paper. Wadler is clear enough that even without knowing typed lambda calculus, you can understand his point.

</comment><comment><USER15> This is why I think Bret Victor's stuff, while incredibly cool and interesting, will have a tough time living up to his vision.

<comment><USER16> On the other hand, think of how much power a program like PowerPoint gives the average user without forcing them to learn the underlying technologies. Or the iPhone. I think that user interface design solves most of the leakiness problems, and the illusion of seamlessness is critical to good interface design. Remove as many features as you have to so that the abstraction doesn't leak.

PowerPoint came out in 1990 - what's the PowerPoint of 2020?

<comment><USER15> All good points, however Bret Victor has specifically been showing off alternatives to low level programming by building very high levels of abstraction. He has even apparently said that he thinks the development community is purposely keeping programming complicated to maintain the demand for their specialized skills. I think the law of leaky abstractions is why it is simply not possible to elevate programming to a level of abstraction that allows people without specialized skills to implement arbitrary applications.

</comment></comment></comment><comment><USER17> (2002)</comment>


How we use Google Analytics and Website Optimizer - <AUTHOR> http://blog.kashflow.com/2009/10/21/how-we-use-google-analytics-and-website-optimizer/

<comment><USER2> Summary:

a) They started tracking conversions b) They did tweaks to improve it c) They split test the headline to increase conversions

Simple but effective steps which many online businesses fail to take.</comment>


The 80/20 rule for storage systems - <AUTHOR> http://www.cohodata.com/blog/2014/12/10/8020-rule-storage-systems/ <comment><USER2> Seagate tried to take advantage of the 80/20 rule with their SSHD "hybrid drives". Unfortunately, they decided to use only 8GB of flash on all of their SSHDs. That's a far cry from 80/20\. The solid-state cache only covers 1.6% of a 500GB hybrid drive, 0.8% of a 1TB drive, and 0.2% of a 4TB drive (yes, Seagate makes 4TB SSHDs). Those Seagate drives don't even have enough flash to hold a recent version of Windows and some commonly used apps, not to mention any games.

The laptop version is even worse because the non-flash part only spins at 5400rpm. Somehow they thought this would be okay in a performance-oriented drive. The slow platters make Seagate SSHDs even slower in general than regular 7200rpm laptop drives from WD and Hitachi/HGST. The only improvement is slightly shorter boot time.

Since the day the first generation of SSHDs came out, reviewers have said that you need at least 32GB of flash in a hybrid drive to see much real-world benefit. Everyone has also been complaining about the choice of 5400rpm. But three SSHD generations later, Seagate still hasn't listened.

Now the SSHD technology is on the way to irrelevance because (1) SSDs have become much cheaper than three years ago, and (2) even if you need a lot of space, the mSATA form factor now allows most laptop users to install an SSD _in addition to_ , not instead of, a 2.5" hard drive. These days, I wouldn't even consider the SSHD as an option. I'd just buy a large hard drive _and_ an mSATA SSD, there's no need for a hybrid anymore.

It's a sad tale of corporate boneheadedness killing off an idea that had so much potential.

<comment><USER3> The world needs a single-HDD form factor explicit HD + SSD for laptops. No changes in laptops needed, use a single cable and SATA's standard PMP (port multiplier) feature in the device. OS sees two separate drives.

</comment></comment><comment><USER4> The article embeds a great talk by the author: [https://www.youtube.com/watch?v=zDuxd1Enxj8](https://www.youtube.com/watch?v=zDuxd1Enxj8)

He talks about using HyperLogLog to store information about the IOPS a workload is causing. This lets them do (at 4kb granularity) analysis on _just how performant different cache sizes are_. Currently solutions are just measuring filesystem last-accessed times, which hide a lot of important data (notably storing VM images).

The tools he shows off are pretty awesome too, like letting you replay old workloads and determining how best to allocate faster storage.

<comment><USER5> Thanks! We're pretty early in our use of HLLs within the system but are already managing to get some really cool data off of them. I'm excited to see where this all goes as we build out the system over the next year.

Your last-accessed time point is exactly right. Storage systems used to be able to do a lot with file system-level metadata, but with the size and opaqueness of VM image files, those techniques have become a lot less effective. We're currently exploring how we can use HLLs in combination with a couple of other techniques to do things like clustering co-accessed data and then managing operations like prefetching and demotions over much longer time frames than are typically done in OSes and storage systems.

<comment><USER6> I havent read the paper yet, but Im a little surprised by the hyperloglogs. I was under the impression that HLLs break down when your symbol frequency varies by orders of magnitude. Those are exactly the patterns id expect to see in block/page access frequncies over time. Are the HLLs only tracked on a smaller temporal scale to increment the distance matrix? Or is there something else Im missing?

<comment><USER7> The state of an HLL is completely determined by the set of distinct symbols that appear, not the order or the frequency of those symbols. So, inserting a billion A's and a single B into an HLL will have exactly the same outcome as inserting a bllion B's and a single A, or even just a single A and a single B.

Does this address your concern, or did I misunderstand your point?

</comment></comment></comment></comment><comment><USER8> Can a 'perfect' algorithm actually give end-users the consistent feel of flash performance with 80% of data stuck on platters? I doubt it.

Access patterns having some statistical distribution is one thing. Writing the allocator -- and always guessing right in where you allocate a file is another thing altogether.

Flash is massively higher performance, so every time you are wrong the user will definitely know. All we hear about these drives is that they aren't consistently fast. Or they're decent until you actually push them too hard.

SSDs themselves used to have poor consistency. But drives are manufactured to ever increasing standards. The Samsung 10 year warranty for example is striking.

80% of your data on a medium which is less reliable over time?! More complex moving parts increases failure rates. There are downsides here to consider I think?

<comment><USER9> _Can a 'perfect' algorithm actually give end-users the consistent feel of flash performance with 80% of data stuck on platters? I doubt it._

Of course not. That's not the point!

What may be possible is to give the "consistent feel of flash performance" some very high percent of the time for a _dramatically_ lower price.

If it's possible to get flash-like performance 95%+ _of the time_ for <50% of the price then I'd find that pretty compelling.

<comment><USER8> But flash will have higher performance and higher capacity, at least that's the projections. I don't know why flash shouldn't be able to win at price/capacity, longevity, total capacity, robustness, performance, etc. At some point, there just shouldn't be any reason to manufacture storage in any other form. In a few generations, isn't that the level of dominance of flash?

<comment><USER10> >I don't know why flash shouldn't be able to win at price/capacity, longevity, total capacity, robustness, performance, etc

yes, you dont, and this is why you think this: >But flash will have higher performance and higher capacity

longevity, robustness, endurance, it all goes down with every new generation of flash. Performance at silicon level is already as good as it gets, we only win by making wider busses and assessing more data in parallel. Price per gigabyte comes at a price (haha) or endurance. Speed comes at a price of more chips per disk (or more wafers/structures inside chips) = money.

Want that sweet new 1TB PCIE SSD able to write at 2GB/s? it will shit itself after ~24hours of continuous write.

There is no free lunch with SSDs, they are a dead alley.

</comment></comment></comment></comment><comment><USER11> From the article:

"...there are now more than three wildly different connectivity options for solid state storage (SATA/SAS SSDs, PCIe/NVMe, and NVDIMM), each with dramatically different levels of cost and performance.

So even if disks go away, storage systems will still need to mix media..."

<comment><USER12> The point is that once storage is all-flash, people outside the storage industry have little incentive to think about differences in storage speed. What will matter broadly is:

1) Code that scales well over a cluster. 2) Code that distinguishes between RAM and storage well. 3) Code that distinguishes between cache and RAM well (for really computationally intensive stuff).

</comment></comment><comment><USER12> Irrelevant - storage is going to be all-flash pretty soon anyway!

<comment> jude- I doubt it. If your access patterns follow a power law distribution, why should you pay more for less space, if the performance boost gained by doing so (i.e. putting the "long tail" onto fast storage) is negligible? Moreover, the need for storage space certainly isn't decreasing, and the storage/cost ratios for disk and tape certainly haven't stopped increasing. If your access patterns follow a power law distribution, a multi-tiered storage system lets you find the optimal cost/performance trade-off.

<comment><USER8> Mostly reliability. I would never carry spinning rust in my laptop again, those days are gone. But also form factor, simplicity, consistency of end-user experience, and eventually solid-state will be cheaper than spinning platters in any case. Every graph I've seen of storage price with series for platters and flash show those lines eventually intersecting.

<comment><USER13> > Mostly reliability. I would never carry spinning rust in my laptop again, > those days are gone.

Exactly. Spinning rust drives are now far inferior to SSD's for reliability. No 2.5" spinning rust drive I have bought in the last 7 years has lasted for more than 2.

I have yet to have an SSD fail on me.

<comment><USER14> I've had 2 ssd's fail on me. To the point that the data is unrecoverable, ssd's fail worse than hdd's from my limited experience so far. Hope you have backups either way.

I also have at least 2 2.5" hard drives that are running on 3 years. Anecdotes make for poor data sets. (note one of those 2.5" drives is getting smart errors now after 12800 hours of use, so probably going to die in short order)

Besides, with spinning rust getting to 8TB now, hard drives are just taking the old place of tape in a proper backup solution. That is memory->flash->hdd->tape->optical (last two could possibly be the same layer).

<comment><USER8> Would you trust a drive to carry that kind of lifespan? I think solid state will ultimately make the rest irrelevant, and in just a few more generations time.

Then again, when storage and compute are integrated, even solid state should some day become irrelevant. So I just think 'hard drives' are going away, and 'store' will be a resource, like 'compute', and most likely implemented in flash for bulk store, and backed up, on separate flash.

<comment><USER14> Carry what kind of lifespan? Generally I don't trust anything anymore and have a 2 local copy + one offsite policy for data I care about.

If I don't have that I don't care enough about the data generally.

As for storage and compute you mean memristors or fpga like?

Solid state has a rather huge size difference to overcome. Hard to beat spinning rust at the moment and at least near future for bulk storage.</comment></comment></comment></comment></comment></comment></comment>


Google offers researchers 1 billion computing core-hours - <AUTHOR> http://googleblog.blogspot.com/2011/04/1-billion-computing-core-hours-for.html <comment><USER2> A rancorous debate broke out inside Google once about the suitability of Google's infrastructure for solving 'real' problems. The people who felt it was inadequate pointed out that everyone who built "super" computers did so with lots of shared state and epic low latency bandwidth. But Google's computers were designed with 'shared nothing' in mind, the architecture [1] so clearly supports web search but is useless for scientific computing.

I and a few others were of the opinion that the scientific community didn't build the equivalent of LinPack or scientific simulation packages on 'Google like' architectures because they _didn't have access_ to such architectures rather they had "Beowulf" clusters [2] which had been built to be more like Supercomputers. It wasn't that such problems couldn't be worked on shared nothing architectures, it was just that nobody was making any real progress along those lines.

This problem reads like a typical Google response to such an argument "Ok, if we made some hardware available for this sort of thing, would academics be willing to apply some serious thinking to it?" No doubt buried in the details somewhere there will be some language about Google owning or at least getting free perpetual access too any code or techniques that emerge out of this experiment.

You have to admit, if they could pull it off it would put a huge crimp in any supercomputer type system.

But perhaps more interestingly, after I left Google one of the things that caught my eye was some of the articles that have been written about what quantum computing might look like. And I realized that with enough cores sitting around you could imagine how stuff learned programming that for 'real' problems might inform how you would program a quantum computer.

[1] <http://labs.google.com/papers/googlecluster-ieee.pdf> [2] [http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book/beowulf_bo...](http://www.phy.duke.edu/~rgb/Beowulf/beowulf_book/beowulf_book/node9.html)

<comment><USER3> >I and a few others were of the opinion that the scientific community didn't build the equivalent of LinPack or scientific simulation packages on 'Google like' architectures because they didn't have access to such architectures rather they had "Beowulf" clusters [2] which had been built to be more like Supercomputers. It wasn't that such problems couldn't be worked on shared nothing architectures, it was just that nobody was making any real progress along those lines.

Whats is the difference between 'Google like' and Beowulf or other similar Linux clusters for scientific calculations? A Beowulf is just a software which clusters together a bunch of cheap computers connected by whatever Ethernet is currently available for normal money (of course you can throw more money if you have it). On the other side, people have no problem running physics or bioinformatics calculations onto hundreds of Amazon nodes.

>it would put a huge crimp in any supercomputer type system.

thats what Beowulf and the likes already did 10 years ago. It is one of the reasons why "supercomputers" (SMP nodes connected by extremely fast backplanes in big cabinets or as you said "lots of shared state and epic low latency bandwidth") had the low rate of survival into 21st century. Of course there is still Top500 supercomputers - big rooms with a lot of racks and, frequently, very expensive/fast networks. Yet, if your distributed program significantly depends on the speed of the interconnect, ie. it have significant message passing component, it usually wouldn't scale effectively, ie. it may scale, yet with quickly diminishing return, Amdahl law style, even with extremely low latency / expensive interconnect.

</comment></comment><comment><USER4> Read the ending; it sounds like a cloud computing offer:

 _In the future, we think this service could also be useful for businesses in various industries, like biotech, financial services, manufacturing and energy. If your business can benefit from hundreds of millions of core-hours to solve complex technical challenges and you want to discuss potential applications, please contact us._

</comment><comment><USER5> Interesting! I have a few ideas on some awesome things I could do with a billion CPU hours, though I'm not one hundred percent sure whether I can get a proposal together by the end of May.

The eligibility is a little unclear though... at one point it says "up to 10 distinguished researchers and postdoctoral scholars worldwide", while elsewhere it says "Awardees will participate through Google's Visiting Faculty Program; faculty members need to have full-time status at an academic institution". So are postdocs invited to apply or not?

edit: Oh, the page could also use some information about memory per node.

<comment><USER6> Postdocs are full-time faculty members. They just aren't permanent.

<comment><USER7> No, postdocs are not faculty. They may be classified as staff, as fellows, or even as students [1-2], but they aren't members of faculty senates.

[1] [http://sciencecareers.sciencemag.org/career_magazine/previou...](http://sciencecareers.sciencemag.org/career_magazine/previous_issues/articles/2000_11_10/noDOI.10424127276070392774) [2] [http://sciencecareers.sciencemag.org/career_magazine/previou...](http://sciencecareers.sciencemag.org/career_magazine/previous_issues/articles/2000_11_17/noDOI.10424127276085878637)

<comment><USER6> At my current university all postdocs in my department carry the official title of Visiting Assistant Professor. They are listed under the tab 'Visiting Faculty' on the departmental website. At least in name they are faculty, and they are professors.

<comment><USER8> That's unique. Having many friends who were, or are still, postdocs, they were almost always staff, but not faculty in any sense (in the US at least)

</comment></comment></comment></comment></comment><comment><USER9> Really interesting that your submission has to be in the form of something that can run on Native Client. I wonder if they're using this to stress-test implementations of NC.

<comment><USER10> Or they're just using NativeClient as the sandboxing mechanism.

<comment><USER11> It's almost certainly this, and I would love to see them release _lightweight_ grid software that uses NaCl on compute hosts. The footprints of existing grid software stacks, both in terms of volume of software and sysadmin time to setup/maintain, are ridiculous.

<comment><USER12> I and another computer science student spent an entire semester internship just trying to get Globus properly installed and configured in the CS lab for one of our professors. It (the cluster computing software infrastructure of Globus) was a ridiculous sprawling mess of poorly integrated components. I've gone through both Gentoo and Linux-From-Scratch and can honestly say it is easier to _assemle the parts for an entire operating system_ than to get Globus working (at least that was the case in 2008). I could see a definite need for a lightweight solution with a simple install package, but I wasn't in the position to write one.

</comment></comment></comment></comment><comment><USER13> You can also go for the INCITE or ALCC programs from the DOE: <http://www.doeleadershipcomputing.org/>

They award O(1 billion) hours to O(100) projects, and you don't need DOE funding to apply (or even be a US-based project). The catch is that your code has to be very scalable (to roughly 40k cores), and your science problems should be in line with the DOE mission.

</comment><comment><USER14> Has Google been putting out really cool stuff consistently over the last few years or is it just because I'm constantly on HN that I see this stuff?

Anyway, I'm really starting to like Google again.

</comment><comment><USER15> That's effectively 114200 cores devoted continuously for a year. Not bad. An experiment I work with uses ~600 cores over 85 hosts, although the average load is probably only around 20%, but there's times where it's pegged for a few months.

<comment><USER16> _That's effectively 114200 cores devoted continuously for a year._

Or 256 x Quadro 6000's @ 99.6% up time. Granted, Google can hand out the real CPU time not just a bank of GPU's. Still, I don't think it will be that long before someone pulls that "switch".

<comment><USER17> Quadros don't have 448 cores each. nVidia's "core" numbers count each portion of a vector processor (a "thread warp") as a core; by this definition, a single Core i7 core has 48 "cores".

</comment></comment></comment><comment><USER18> This is a truly _amazing_ offering. I hope the applications in the area of biochemistry research will find creative ways to work around the two important limitations of google: its GFS filesystem and its latency.

</comment><comment><USER19> The question I would like to research is: 1 billion core-hours on Google's infrastructure yields exactly how many bitcoins?

<comment><USER20> Well, if the Google infrastructure is so massive that it will dominate the bitcoin generation: 50 coins every 10 minutes.

Not worth the electricity I think. Let's talk real science instead of self- enrichtment for a change.

<comment><USER19> I'd heard the bitcoin system auto-adjusted for more 'mining'... but is its production rate really constant, no matter how large the arrival of new computing power under a single authority?

(My 'proposal' was a thought experiment; I'd definitely agree that given that much computing-power there are more beneficial and/or profitable things to do... among them running a market-dominating search engine.)</comment></comment></comment>


An 'Accordion' of Wood and Glass - <AUTHOR> http://online.wsj.com/article_email/SB123872378357585295-lMyQjAxMDI5MzA4NDcwMjQzWj.html

<comment><USER2> "'My books and my house are my twin legacies,' Mr. Stewart says. 'If I hadn't commissioned this house, I'm not sure what I would spend the money on.'"

$24 million dollars? I can think of a few things, including education, feeding starving children around the world, healthcare research, etc. I have no problem with a man spending every penny he earned to build a house. I only take issue with the preposterous statement above. I think saving hundreds of lives, sending hundreds of kids to college, building schools in impoverished areas, etc. is a much more powerful legacy. And I would argue that the house is ultimately the architects' legacy and not his.

That said, it is a beautiful home.

</comment><comment><USER3> I looked up the homeowner's calculus book and realized it was the one I used in college: <http://www.amazon.com/Calculus-James-Stewart/dp/0534359493></comment>


Extinq - LINQ extensions for the everyday programmer - <AUTHOR> https://github.com/holmstrom/Extinq

<comment><USER2> Actually, some of these are quite handy!</comment>


In Praise of a Nomadic Life - <AUTHOR> https://www.wsj.com/articles/in-praise-of-a-nomadic-life-1490582016 <comment><USER2> Nomadism is lovely but I believe that natural comfortable state for humans is to be Migratory. Find several comfortable places that you belong in specific seasons and move amongst them. You can develop ties and a sense of ritual belonging while having most of a benefits of a nomadic life.

Snowbirds, Celebrities, Jet Set, Aristocrats, Fashion industry have all figured this out as the most comfortable form of human life at some point or another.

</comment><comment><USER3> I traveled around for a period, and my feeling is that any joy to be found in a nomadic lifestyle and travelling cannot compare to the feeling of belonging and connectedness that being a part of a real physical community gives you.

</comment><comment><USER4> While the nomadic life does sound appealing to me I see it as the ultimate form of instant gratification. You sacrifice all stability and ability to effectively change the world around you in order for some momentary pleasure and as they say in the article they just seem to keep pinning their happiness on future events. It seems to me that pinning your joy on future events is a sure way to accumulate happiness debt (much like tech-debt) that you're effectively unable to pay down.

</comment><comment><USER5> What do people think about a real solo nomadic life?

I'm a huge introvert and used to being alone and lonely all the time yet I cannot quite imagine myself enjoying being a moving ghost from an exotic location to another alone and getting much out of it.

I hear people talk about finding local friends and so on however I cannot imagine that is going to be that satisfying or practical. Making some broken language hello-how-are-you conversation and never seeing them again.

I think the feeling of not having someone to share any of the experience with would overwhelm me and dominate my experience.

I see people talk some feel-good stuff about learning to enjoy things on their own and so on but I don't really see practical examples of it. And as I said, if there was one person who would be comfortable spending time on their own that would be me, yet I don't find the idea of solo nomadic life appealing.

The few people that I have seen appear to be homesick and isolated they claim they are loving the solo trip but in reality they seem to be desperate to share it with someone and they seek that on Twitter and Instagram and on their blogs.

</comment><comment><USER6> >>>I kept pinning my happiness on future events.

This is true. I don't think you have to go to the extreme of living a nomadic life to address it, but I definitely feel where the author is coming from.

The first time I realized this, in conjunction with how much of what my expectation of happiness comes from media, it was a really jarring experience.

</comment><comment><USER7> _> Nor will we have children_

This is no way to run a civilization.

<comment><USER8> How might I be incentivised to have children?

<comment><USER7> If I were in charge I would institute a basic income for all citizens over the age of 18 (no additional money for children) and would make each child someone had a significant tax break against income. People who could afford them would be incentivized to have them.

<comment><USER9> Why incentivize people to have children when there is no shortage of them?

<comment><USER10> Actually, in many Western countries there is, much of Europe for example. Sustainable demographics are why there are subsidies for having children in Italy for example.

<comment><USER9> I live in Germany, the country with the lowest birth rate in the world. The population is still slowly growing due to immigration.

Even once the developing countries reduce fertility below replacement (which will not happen for a long time yet) with increasing life spans and improving medicine the world's population will still continue growing(slow growth is still projected in the year 2100), then stagnate and then one day very far from today slowly decline...

but it will still take centuries to reach the population levels of the early 20th Century (only reached 2 billion people in 1927). In the meanwhile we have too many, not too few, people around.</comment></comment></comment></comment></comment></comment>


Programming less - <AUTHOR> http://www.scripting.com/stories/2007/11/28/programmingLess.html

<comment><USER2> _Your code at its kernel level must have this simplicity. But at the edges, where you're accommodating the minds of users, inevitably it gets a little messy._

Absolutely. Your core code is usually targeted to programmers, that is, (hopefully) formally thinking folks, so you may choose to be as succinct as possible here. At the same time UI code may become illogical, non-linear mess only because their users can't think formally. Bookkeeping/accounting software is a perfect example.

Those developers who prefer working on compilers, libraries or operating systems are actually those who want to get the most out of programming. Maximum joy, that is.

</comment><comment><USER3> I agree that simplicity and elegance are key, but in my experience the UI does not _inevitably_ get messy. The same rules apply, but you have to work at different levels of abstraction. The central structure and algorithms of a UI are no longer about the data itself, but about transforming the data (and transforming the transformer!). There's a special brand of programmer crack reserved for those who create elegant and flexible UI code.

</comment><comment><USER4> I wish people would test their pages with a text-only browser more. The article whole article was interspersed with the phrase "permalink to this paragraph"

<comment><USER5> Even in a graphical browser, I found the permalink icons really intrusive. I'm used to things like that being used to sign off posts (<http://www.theregister.co.uk/2007/11/29/erskine_steps_down/>), and it breaks the flow to see them at the end of every paragraph.

</comment></comment><comment><USER6> "The design of the central data structure of an app determines the quality of the app, in every way."

Wow!

As Edward Everett wrote to Abraham Lincoln the day after the Gettysburg address:

"I wish that I could flatter myself that I had come as near to the central idea of the occasion in two hours as you did in two minutes."

</comment><comment><USER7> dicts FTW.</comment>


Do You Know What an Angel Investor Is? - <AUTHOR> http://www.dilbert.com/blog/entry/do_you_know_what_an_angel_investor_is/ <comment><USER3> Scott Adams is wrong. There are angel investors who invest in unproven companies, even at the concept stage. I'm one of them, and I prefer to invest in people. There are people who I'd invest in before even knowing what their companies plan to do.

<comment><USER4> Every Scott Adams post about CalendarTree makes me wonder if I'm reading marketing for CalendarTree.

<comment><USER5> So just like any other post on hacker news by someone with a product?

The "my experiences with product X as diving board for tangential point" is probably one of the less annoying positions on a scale that runs all the way from "pitching hard" to "cleverly designed to disguise relationship to the product."

Don't get me wrong, a lot of them are incredibly gross to witness, but to me this one barely registers.

</comment></comment></comment><comment><USER6> I'm still not sure Scott Adams knows what an angel investor is. That, or I don't!

As I understand it, the real difference between angels and VCs is that angels invest without locking down a valuation, and angels invest without demanding control. A VC round comes with a firm valuation and with a board seat.

<comment><USER7> Angels sometimes invest in priced rounds. Convertible notes are a newer thing, and not yet 100% universal.

The biggest difference, traditionally, is that angels invest their own money or that of buddies, and that VCs invest the money of LPs. This is a little weirder these days, as there are super-angels/micro-VCs/whatever-their- business-card-says where that starts to become a distinction without a difference.

<comment><USER6> So the answer is, mostly, that I don't know what an angel is. :)

<comment><USER8> You do, it's just that the term is pretty encompassing :)

</comment></comment></comment></comment><comment><USER9> There's a few points which sound right in this essay, but I'm surprised he didn't discuss the more technical definition and its impact on the ecosystem:

1) an angel investor is someone who invests their own money 2) a venture capitalist invests out of a fund, which is mostly other peoples' money (OPM!) while taking a fee + economics from the fund (the famous "2 and 20" model)

But that's not too useful, because what's important are the behaviors that come out of the situation.

Because most angels are investing their own money, they usually don't have crazy amounts of capital to work with. Thus, they usually invest early so they can get a better percentage at a lower valuation.

Many VCs also invest early, sometimes exclusively so with smaller funds, but they are often called "seed funds" to make that distinction. A fund which invests OPM is never called an "angel" regardless of what stage they invest at.

And finally, big funds (managing 100s of millions of dollars) are what we think of usually as venture capital.

(Funds that invest even larger amounts at higher valuations are often referred to as "late stage venture capital" or "growth capital." And there's more specialized terminology later stage since more specialized financial instruments can be brought into play - SPVs/debt/mezzanine/etc)

Where Scott's essay rings true is that idea that investors of all classes are more risk averse these days- they prefer to see traction since the cost of building an app/website is rapidly decreasing. Thus, they are all behaviorally acting like "traction investors" rather than "idea investors" whereas in the past, traction investors purely consisted of the growth capital guys.

This might be worse for the ecosystem since people want you to have everything built before taking in our first dollar of investment, but you could argue it means the ecosystem's $s are being allocated more efficiently also.

</comment><comment><USER10> _" I have a degree in economics, an MBA from Berkeley, and over 30 years of business experience. Do you know what question I hear in Silicon Valley nearly every time I meet with potential investors for CalendarTree?"_

I would have supposed that Scott Adams has become quite wealthy from the Dilbert strip, his many books and all the spin-off merchandise (calendars, toys, etc.). Why would he need to look for outside investors to fund his app rather than just investing his own money? And if he's not willing to risk his own money to start his new business, what message does that send to potential investors?

<comment><USER11> Investors understand risk management. It is possible he knows he needs 15MM to go for the home run, but doesn't want put in the 15MM himself and go bankrupt in the process. Investors allow him to shoot for the moon.

Also, investors help with accountability, pushing through tough spots, connections, advice, etc. Investors also add credibility that 'rich white guy throws money at project' doesn't give.

<comment><USER10> Seems like you could write a shared calendar app for way less than $15M. And it's already in beta, so a lot of the development has been done already.

I also wonder how much success he's going to get selling to enterprise customers. Any company with over 200 employees already has a calendar system (Outlook or some cloud equivalent).

<comment><USER12> Pretty sure the $15 million referred to a long-term bet, that takes years to get through the burning cash phase and into profitability.

If you hire 20 people for the first year, that's going to cost three million dollars (if you're lucky) for everything all-in, without calculating potentially high customer acquisition costs early on.

Writing the calendar app is the least expensive, least difficult and least time consuming part of building a large scale business around it.

If you wanted to go big, and had three years to do it, you could easily burn through $10 or $15 million trying to get there before generating your first quarterly profit.

</comment></comment></comment></comment><comment><USER13> "If I had to guess, I'd say Silicon Valley would be among the easiest for women to penetrate"

This is not an opinion grounded in any sort of reality. Attrition for women in technical roles is nearly double that of other industries. In finance, women make up around 20% of corporate boards. Not so in tech. Not by a long shot.

This is Scott Adams saying "hey guys, here's some stuff I think but haven't really tested even a little bit."

<comment><USER14> > This is Scott Adams saying "hey guys, here's some stuff I think but haven't > really tested even a little bit."

For non-regular readers: that's _exactly_ what Scott Adams' blog is about - throwing ideas into the wild and discussing them in the comments. It's a feature, not a bug.

The problems come when people read it without knowing that, and assuming he's advocating for <issue>, while he's more of a "what do you think about this crazy thought?" kind of writer. Check this post[1] (specially the disclaimer) for an example.

[1] [http://dilbert.com/blog/entry/proof_almost_of_intelligent_de...](http://dilbert.com/blog/entry/proof_almost_of_intelligent_design/)

<comment><USER13> That may be so, but I think it's not a little pernicious for someone with standing and reach to say "Hayyyyy, so, I'm sure everything's FINE" when everything is not at all fine. Sloppy and counterproductive.

</comment></comment></comment><comment><USER15> No. Angels invest their own money.

I assume that he can't get interest in his idea because folks are dubious about the idea, but are willing to be disproven ("come back when you have some traction")

</comment><comment><USER16> ...Who is Nikki _Durbin_?

<comment><USER17> Agreed, would be nice if the author used the correct name.

</comment></comment><comment><USER18> Boo popups

</comment><comment><USER19> _If I had to guess, I 'd say Silicon Valley would be among the easiest for women to penetrate_

In the average relationship, the male is 4 years older than the female. That means that women (as a group, even if their individual dating patterns differ) have a "look-ahead" insight into which careers turn out well (medicine, law) and which promise the moon but fall flat (startups, academia).

Among a group of 21-22 year-old college-senior women, at least a few have dated 24-27 year-old guys, so they know a lot more about the career landscape than the 21-year-old men, who've been dating 18- to 20-year-old women.

That's why there are no women (except for hand-picked pretty tokens) playing the VC-funded game. They have the look-ahead information, and they're smart enough to see the VC-funded "tech world" as a fraud and not get involved. It's clueless men who get in, wreck their careers, and find themselves with no other options.

<comment><USER20> Law is a terrible career these days. Talk to folks just getting out of law school about what their career prospects are like, don't talk to folks who are lawyers. The latter have already cleared numerous weed-out processes (getting into law school, getting through law school, getting a job after law school) that chew people up and spit them out. About 20-25% of T14 law students (and virtually none from lower-tier law schools) make it to BigLaw positions paying $160K/year. The rest become small-town attorneys, public defenders, assistants in the DA's office, or go to work for boutique law firms. These positions pay about $40-75K/year, barely more than a public schoolteacher, and yet they have law school loans to pay off. Even if you do get a BigLaw position, it involves being some partner's bitch for 7-10 years until ( _if_ ) you make partner yourself.

Medicine is a decent career if you get through it and can deal with the hours, but it also has a massive weed-out funnel. At my (top liberal-arts college) alma mater, the admissions department had a joke: "95% of you are pre-med. By graduation, about 2% of you will be pre-med."

This is also why your main thesis, about women having an information advantage about the career world because of their relationship partners being older, doesn't hold. There is a strong selection bias in relationship partners. Lawyers find it much easier to find girlfriends than unemployed law-school graduates who went back to live with their parents. Anyone who uses their personal relationship choices as data on how the employed population as a whole functions is getting seriously biased data.

<comment><USER21> I can't really disagree that law is an unappealing career choice these days. However, I'd say getting getting _through_ law or med school at the elite is pretty well assured once you get in. Here are the numbers for boalt

[http://berkeley.lawschoolnumbers.com/](http://berkeley.lawschoolnumbers.com/)

First Year : 2.0% Second Year : 1.7% Third Year : 0.3% Fourth Year : 0.0%

here they are for stanford

[https://www.law.stanford.edu/facts/enrollment-data- transfer-...](https://www.law.stanford.edu/facts/enrollment-data-transfer-and- attrition-rates)

1st year 0 3 3 1.7 2nd year 0 2 2 1 3rd year 0 0 0 0 4th year 0 0 0 0

Here are the graduation rates for UCSF med school

[http://saa.ucsf.edu/sites/saa.ucsf.edu/files/PDF/WASC/Append...](http://saa.ucsf.edu/sites/saa.ucsf.edu/files/PDF/WASC/Appendix%2033.%20Program%20Graduation%20Rates%202006-2009.pdf)

(UCSF-medicine has a 99% graduation rate).

<comment><USER20> Yeah, I should probably de-emphasize "getting _through_ " law school in favor of getting into law school and getting a job after. Last I heard getting into a T14 law school was still quite challenging, and even then getting a well- paying job was far from assured.

<comment><USER21> I'm, very, very late responding to this, so apologies for that.

I agree that getting into a top law or medical school is exceptionally difficult. Here's the admissions page for ucsf

[http://meded.ucsf.edu/admissions/successful-applicant- profil...](http://meded.ucsf.edu/admissions/successful-applicant-profile- class-2017)

Interestingly, they don't give admissions rates, but we can see that only 6% even get an interview. The numbers themselves are high, too. Looks like about 95%ile MCAT scores, and a high (nearly 3.8) GPA overall and in the sciences.

Law has similar numbers, though LSAT scores tend to be a bit higher - <USER2> 98%ile. My guess is that this is because law doesn't have the "weeder" courses in pre-med, so you do have a lot of high GPAs in history or other relatively easy majors.

Engineering is a very, very difficult comparison to make, because it truly is not the same thing. Licensure is unusual, largely limited to a couple of specialties, and happens (with exceptions) at the undergrad level anyway. Legally speaking, you don't _need_ an MS or PhD for anything, and you rarely need a BS either. This of course is a big factor when thinking about admissions and attrition rates.

Still, there is a large PR effort in the US to convince the public and young people that there is a shortage of US Citizens going into "STEM" graduate programs. I've posted a link to a RAND study that concludes that this aversion is rational and market driven when you consider the pay and career prospects relative to the "professions" (medicine and law, as well as dentistry, MBA programs, even nursing and pharmacy... in SF, dental hygienists earn, on average, about 98% of a software developer's median salary).

Here's a link the RAND study

[http://www.rand.org/pubs/issue_papers/IP241.html](http://www.rand.org/pubs/issue_papers/IP241.html)

It's very difficult to compare with graduate engineering programs, since they tend to be specialized at the degree level (rather than through residency in medicine or career choices later with law). But looking at entering class profiles, top engineering programs appear to be very selective. Not _as_ selective in terms of admissions rates as medicine or law, but I think this is more a function of very heavy course requirements for admission (you can major in anything for law, and you can do pre-med in 2 years, whereas to apply to an MS or PhD program in engineering, you tend to need essentially 4 years of exceptionally difficult specialized coursework).

So admissions aren't really notably easier for graduate work in engineering, undergrad prep is very hard... and yet, attrition rates are brutal. Overall, engineering is the best of the bunch where PhD attrition rates are concerned, at about 35%. In hard sciences, these typically go up to about 50%. This is for a cohort that has gotten top grades in majors like physics and scored at least above the 90%ile on standardized tests, usually quite a bit higher.

In short - I'm willing to agree that getting into law school is tough, but not any tougher than getting in to a top grad school in engineering (probably much easier, when you consider the coursework you need to take for engineering), and getting _through_ law school, based on attrition rates, appears to be _far, far easier_. Even med school appears far easier from an attrition rate point of view. They're hard working, sure, but so are grad students in STEM. I just don't think hard work can explain a difference in attrition rates between 50-100 times higher.

Of course, like I said, you don't need a PhD or an MS to do anything in engineering, so maybe they're not really useful as a point of analysis here. The reason I bring it up is that there is still a strong PR push to get more young US citizens (or these days, women) into STEM graduate programs, and yet the numbers are horrendous for that path. Difficult coursework, sky high attrition, and - at the elite level, since we're talking about "best and brightest" \- mediocre outcomes compared to medicine and, at the elite level, perhaps law as well (and many other fields).

</comment></comment></comment></comment></comment><comment><USER22> nice pun by the end.. I see what you did there ¬¬

</comment><comment><USER23> Every single real or perceived slight is a result of one's gender/color/orientation. We all know that white men are never slighted.</comment>