Large home libraries may have a long-term impact on proficiency: study - <AUTHOR> https://www.smithsonianmag.com/smart-news/growing-surrounded-books-may-bolster-skills-later-life-180970523/?no-ist <comment><USER2> I grew up in a household that had passed that 350 book mark, and beyond all of the actual meaningful improvements it has made to my life, it also meant that reading comprehension on standardized testing was laughably easy. So teach your kids to love books, and enjoy reading. It doesn't have to be high-brow intellectually challenging shit, just make sure that they associate books with fun.

<comment><USER3> >> it also meant that reading comprehension on standardized testing was laughably easy

I'm actually curious to hear how various HNers have done on on standardized reading comprehension tests. I grew up with massive bookcases in the house, I've now populated my own, I read a lot, get a lot out of it, and enjoy it. On the ACT I had perfect scores on every category but a literal 0 on reading comprehension. As far as I'm aware, that's always been the case on my standardized tests, and I wasn't surprised because the questions seemed ludicrous to me (I nodded vigorously while reading [1]). I primarily read non- fiction and have wondered if I just don't grok fiction. What are other people's experience on that?

[1] [https://www.washingtonpost.com/news/answer- sheet/wp/2017/01/...](https://www.washingtonpost.com/news/answer- sheet/wp/2017/01/07/poet-i-cant-answer-questions-on-texas-standardized-tests- about-my-own-poems/?noredirect=on&utm_term=.980b67e62e34)

<comment><USER4> I read a ton as a kid, and I always struggled a bit on reading comprehension tests. To me it generally seemed like the "reading comprehension" questions would focus on minutia I didn't consider important, and simply didn't bother to remember. It's been a long time since then, but I want to say I'd read a story about a kid sledding on a hill with his friends, and then later to test your reading comprehension they'd ask you what color his shirt was.

Maybe that _is_ reading comprehension and I am just bad at it, but to me shirt color was unimportant.

<comment><USER5> I can't imagine why just memorizing shirt colors would be worthwhile.

But if, for example, the author made it clear that the other kids were all wearing the same color shirt and the main character was not, you might infer something about their relationship from that--like maybe that the other kids all went to the same school, but the lead character went to a different school.

Details do matter for reading comprehension, because every detail is chosen. The real world is full of superfluous detail... writing, especially good writing, is not. Every word on the page represents a choice by the author--a choice to put that detail in, and to use that particular word instead of a different one.

<comment><USER3> And yet standardized test writers seem to think they understand those choices better than the authors themselves.

<comment><USER5> Critics have an opportunity to understand those choices better than the authors themselves because critics have the opportunity to look at the work from a different perspective.

Just because every bit of writing is a choice, that doesn't mean the author him- or herself fully understood _why_ they made that choice. We're all shaped by our own life experiences, our family, the time in which we grew up, etc. Often, it's easier for a third party to connect those dots. That's why coaching and therapy works, for instance.

That said, when we're talking about standardized tests, they tend to (and should be) sticking to pretty well-understood and obvious examples of this sort of thing.

And of course, sometimes they're just not well done. Standardized test writers face the same challenges as every other writer, and I think we can guess that the very best writers probably do not end up in that job...

<comment><USER3> I'm sorry you completely lost me in the second paragraph. If reading comprehension comes anywhere near connecting an author's life experience to their word choice, then I'll gladly take my 0 and call the whole thing garbage. Sure, coaching and therapy work. They're also fields that churn out massive amounts of pseudo-science when it suits the author's ego.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> Halfway through the article, when I saw the part about how the number of books varied between nations, I thought, "oh cool! They're going to use that to address the possible confounding affects, since otherwise you might just be measuring whether parents who make sure their kids are well educated end up with a lot of books in their home. If kids in the 90th percentile for their country (for number of books in their home) have less similar outcomes than kids with the same number of books (from different countries), this would be indicative that the books that actually matter." If that point is in the article, though, it's sure not in Smithsonian's article, and since it's Elsevier I can't look at the original. :(

<comment><USER7> I'll bet you equally good correlations can be found when the parents have a Kindle library that the kid doesn't have access to. You'd expect articles like this to have a paragraph, "The researchers controlled for biological heredity by ________" but it seems they don't want you to take them seriously.

Edit: Oh, they also show that growing up in a home full of books makes you more numerate, too. So, no surprise, there's a general intelligence factor involved that has nothing to do with books.

</comment></comment><comment><USER8> My parents would basically (with a few exceptions due to price) buy me any book I wanted growing up (without it hitting into my allowance); they thought it was incredibly important to grow up well-read, and even more important to learn how to teach yourself stuff, and similarly every standardized test I took was insultingly easy.

I'm definitely not some pillar of intellectual vigor either (I don't even have a college degree), so the fact that my reading comprehension is good, I think, is because of the exposure to books.

That said, I realize that one anecdote doesn't mean anything.

<comment><USER9> >That said, I realize that one anecdote doesn't mean anything.

Nevetheless, I appreciate the hack you suggested of excluding books from allowance. As a parent, that's something I'll now implement. Thanks for sharing.

<comment><USER10> Do you charge your kids' allowance for food? Rent? If you all go out to eat?

It's odd to me that anyone would ever think of charging their child for a book.

<comment><USER11> I have heard of parents making kids buy their own clothing out of allowance, as a way of teaching them the value of money and how to shop responsibly.

</comment></comment></comment></comment><comment><USER12> I completely agree with this article, based on my own anecdotal experiences (I don't think the science behind this article was the best, but I'll save that for another post).

My great grandma never had the chance to go to school, but she loved to read. She would work, save up, and buy books whenever she could. Books were all she ever wanted for her birthday or Christmas. Even though she had no formal education to speak of, she was an intelligent woman and wrote prose and poetry as good as any I experienced in college.

My great-grandmother's love of reading has been passed down through four generations now (down to my children). I have over a thousand books in my house, some of which belonged to my great-grandmother. I learned software engineering by reading books I borrowed from my local public library and built a successful career through reading and studying these freely available tomes on my own. When my kids were in school, I told them to put all their efforts into reading. I knew from personal experience that if they could read well they would be able to learn anything they wanted to. So far, that has been the case.

If I could only have one life skill, it would be reading. Having access to hundreds of books growing up certainly helped me and made me who I am today. Of course, having books around does nothing for you unless you crack them open and read. The only downside has been schlepping all those books around every time we move.

</comment><comment><USER13> The study this summary refers to studied books as an indicator of a "culture of scholasticism." It didn't even really try to disentangle socioeconomic effects, because it was looking to use books as a proxy for them.

It did not say "books in the home make you smarter." It was arguing "books are a reasonable cross-cultural proxy for orientation towards scholasticism in the home."

This summary, to my reading of the actual article, completely confuses cause and effect.

<comment><USER14> Their confusion goes beyond using books as a proxy for scholasticism.

No doubt its just a proxy for IQ itself (higher IQ people tend to read more books, thus own more books, thus score better on scholastic measures).

</comment></comment><comment><USER15> I grew up in a house that had a library with 1000+ books. Unfortunately I didn't read much of any of them. Nobody ever tried to get me to read them. However, I blame myself for not reading much as a kid.

My point being that the mere presence of books is not enough.

<comment><USER16> I don't think it makes sense to blame yourself. It is absolutely the job of parents to encourage their kids to read from a young age.

To kids, reading looks terribly boring, until they actually find a book they like - then they devour everything in sight.

How much I miss the combination of childhood imagination and fantastic stories!

<comment><USER17> > To kids, reading looks terribly boring, until they actually find a book they > like - then they devour everything in sight.

 _So_ important and always ignored. My mom read to us often, and I and my brothers hated reading. Later I encountered an Animorphs book and that was my gateway. One of my brothers, several years after that, discovered Redwall, and that did it for him.

<comment><USER18> Redwall and Salamandastron were great stories. Go Matthias.

</comment></comment></comment></comment><comment><USER19> I am unable to read the actual study (paywall) and this article doesn't really talk about the methodology, but did they account for socioeconomic factors?

I would assume _on average_ that the more books your family could afford, the more money they have.

<comment><USER20> Here's the paper: [https://sci- hub.tw/10.1016/j.ssresearch.2018.10.003](https://sci- hub.tw/10.1016/j.ssresearch.2018.10.003)

<comment><USER19> Great, thanks! It looks like they didn't account for family wealth unless I'm mistaken.

In the omitted variables section:

> However, prior research using longitudinal data from Australia with > corrections for measurement error showed that a substantial impact of home > library size on adult education existed net of the effects of academic > ability (IQ) of adults or their father's scholarly habitus (i.e. employment > in occupations where use of books is common) as well as family income or > wealth (Evans, et al., 2010).

</comment></comment></comment><comment><USER21> I wonder how outdated/lagging these studies are.

When I was a kid I was reading nearly every waking moment between say 3rd grade and 9th because It was the only form of entertainment I had (until I got a computer). That is because my parents thought that TV rotted a child's brain and didn't have one in the house until after I left.

Today, my kids read, but they have phones, TV's, computers, and a ton of other distractions that have proven nearly impossible to control without an outright ban in the whole house.

<comment><USER22> > and a ton of other distractions that have proven nearly impossible to > control without an outright ban in the whole house.

So your parents were right.

</comment></comment><comment><USER23> Leaving aside the poor science of this study, I wonder if there is any impact to the fact that I have a large collection of books that sit on my kindle. Living in my parents house I was surrounded and influenced by their books. My children read a lot, but they may not be as aware as I was of what their parents are reading. And there is a lot less of that social signaling that happens when a guest sees a book on the bookshelf that they read or are interested in.

</comment><comment><USER24> This particular instance of correlation and causation has always been fascinating to me.

Is it possible to have a non causal correlation that is impossible to bring about without increasing (causing) the effect.

For instance suppose you tried an experiment of getting 500 books into a household with a new infant and zero books. Let's say you did this with no special attempt to produce the cause, but made no attempt to ensure the process of getting 500 books over there had no influence either.

How would you go about this? Ask for donations? Collect books? How would you find donors, how would you find families to donate to?

My guess, which I make purely on a hunch, is that you would have to make dramatic (perhaps unconscionable) efforts to ensure that this process would not bring about the effect of higher literacy on average.

Not sure by how much but I think (just a thesis) that it would be meaningfully measurable

<comment><USER25> Books are cheap. I'd get 500 books from a used bookstore and put them in a hut in some primitive village. Note that books would not be in the native language, and nobody in the village can read anyway. My hypothesis is that books will make no difference.

An interesting and useful experiment would be to place books in the homes of poor people who can read but don't. Turns out cities all over the US are doing this experiment on a large scale right now (often paid for by United Way). While the experiment isn't completely controlled anyone who cares to can collect data and analyse it.

<comment><USER24> Sounds like they're running the experiment. The first scenario you described is a good illustration of the kind of effort it would take to ensure no effect. The second? I'd be interested in knowing how they find families and decide what books to include. I'd expect (and I admit I hope) it will have a measurable efect.

I'll search for it but please post a link if you have one.

<comment><USER25> [https://1000booksbeforekindergarten.org/](https://1000booksbeforekindergarten.org/)

They claim studies support the idea, but I'm not able to figure out which

</comment></comment></comment></comment><comment><USER26> Or it could be only a correlation. Smart parents have books, but they also have smart children.

<comment><USER27> Maybe.... natural predilection is no replacement for experience, though, and reading is a type of experience. Arguably a high quality one.

Also, there are obviously countless examples of seemingly intelligent people having kids which aren't.

<comment><USER26> Yes. Genetics is messy and non-exact, but on the _average_ IQ is highly heritable.

<comment><USER27> On the average, IQ is 100. That is quite heritable. I'm not sure that's meaningful genetically.

Even correlated high IQ across generations can easily be a nuture over nature deal—see the above article for one hypothesis.

Finally, even if high IQ is genetically heritable, it may still require cultural or experiential "activation" to nuture intrinsic ability.

There are simply too many variables to casually state any causal relation.

<comment><USER26> No, the average of white people was normed at 100. I'm saying the average case is that people are the average of their parents. I'm not a genetic determinist and I am not an environmental determinist, but there is vast amounts of evidence that IQ is, for a large proportion of the causality, a heritable trait.

<comment><USER11> Wikipedia says: "The heritability of IQ for adults is between 47% and 53% with some more-recent estimates as high as 70% and 76%."

</comment></comment></comment></comment></comment></comment><comment><USER28> My house didn't have that many books (less than 100), but I didn't have a computer (I'm from a developing country), so when I got extremely bored I just picked up whatever book I could to keep myself entertained for a couple of hours. This is how I fell in love with Asimov's stories and eventually got into programming.

</comment><comment><USER29> As a child in a boarding school I actually got to grow up in a literal library. I would say most of what I am today, the good parts, come from me getting the opportunity to be among books, and that too of such diverse nature.

</comment><comment><USER30> I liked how Matt Yglesias summarized the quality of this research: "Kids who grow up in households with lots of pants suitable for tall adults grow up to be tall as adults."

Making causal claims without accounting for genetic confounds is simply pseudoscience, there are no excuses.

<comment><USER31> > Making causal claims without accounting for genetic confounds is simply > pseudoscience, there are no excuses.

This study was done in response to the Scarborough and Dobrich paper that found that how much parents read to their children (generally) isn't correlated with children's future reading ability, but how many books are in their household _is_ a positive predictor of future reading ability.

It's one of the most famous and surprising findings in all of social science.

I'd suggest rather than the authors not "understanding science" or whatever, perhaps it's more likely that their target audience wasn't people on HN who would complain without knowing anything about the field.

[https://www.sciencedirect.com/science/article/pii/S027322978...](https://www.sciencedirect.com/science/article/pii/S0273229784710100)

<comment><USER32> That's the whole point - there is research supporting the point that "how many books are in their household is a positive predictor" not only of future reading ability but also a bunch of other things, but there seems to be no research supporting that "Growing Up Surrounded by Books Could Have Powerful, Lasting Effect on the Mind". These are very, very, very different things.

In essence, a very likely explanation for the observed "childhood books vs adult success" observation is that growing up surrounded by books is an indicator that your parents may have higher socioeconomic status than average, or that they have an above average educational background and IQ, or possibly all of it, and for _those_ things it's well known that they have hereditary (both nature-hereditary and nurture-hereditary) effects that indicate that the kids are somewhat more likely to be successful in most aspects of life.

That observation (both in your link and in the studies described in the main article above) is not a reason to assume any causal relationship; the original studies don't claim that, but the article we're discussing _does_ try to imply such claim without any appropriate justification, and _that_ is inexcusable pseudoscience.

<comment><USER31> > there seems to be no research supporting that "Growing Up Surrounded by > Books Could Have Powerful, Lasting Effect on the Mind"

Maybe not in this particular study, but the whole point is that any given study is going to be done in the larger context of a one or more fields of pre-existing research.

If you look at the paper I linked to, the proposed model is that in order for kids to be good at reading, they need to read. And the kids who read the most are going to be those who love reading. Now when you as a parent read books to your kids, some kids will love that and others will hate it. And if you read to them if they don't like it, that can actually turn them off from reading and lead to them being worse at reading in the long term. Whereas if you just leave books around the house, sooner or later they're likely going to read some of them, and are more likely to develop a love of reading just by reading things they're interested in when they're interested in them rather than having reading being forced on them before they're interested.

No one is arguing that there isn't an SES component or whatever. But the actual proposed explanation comes from the context of the last 50+ years of research on intrinsic vs extrinsic motivation, and is interesting on its own.

Any one paper isn't going to look at more than one specific thing, but if you're only looking at that one paper without understanding the larger context of the research then you're not going to understand what's interesting about it or why it was worth doing.

</comment></comment></comment></comment><comment><USER33> Even without reading them, the books on my parents' shelves had an affect on me due to the wacky titles. They stuck in my mind and to this day, I can still remember them: "Blind Ambition" or "The Winds of War" or "The Greening of America" \- seeing these as a kid caused me to ask lots of questions, i.e. who is John Dean and why is America turning green? :)

</comment><comment><USER34> I never had much of a home library my formative years (would estimate < 50, maybe < 30), but I always had such easy access to public libraries (in my schools, and more importantly one within walking distance in my town) that I still easily consumed 1-3 new books a week for most of my childhood and teenage years. While 500 books in a home library is impressive, I almost think that would have been more constrictive to my learning options compared to the selections available from a public library system.

One thing I noticed, and a reason that I depended on my local public library so much more than the school ones, is that the selection for libraries in schools before high school is way too age restricted. I once tried to donate a few of my favorite paperbacks (Stephen King novels) for future students, but the librarian told me the school system would just end up transferring those to a high school library.

Anyhow, my point is, I would be interested to know if living within walking distance of a public library is similarly correlated with better outcomes.

</comment><comment><USER35> There is supposedly a huge correlation between reading speed and standardized test scores, like the SAT/GRE/GRE, which while not the point of this article is still a powerful argument for reading books as a kid. And, this could be any book, it just matters that you're practicing reading.

<comment><USER36> This could make sense. The hardest part of the English SAT section was my abysmally slow reading speed.

</comment></comment><comment><USER37> When I would get in trouble as a kid my mom would ground me for a certain number of books. As I got older/wiser/in-more-trouble the requirement moved from # of books to # of pages and she'd test me on random sections of the readings. A++ productive way of dealing with a difficult child!

<comment><USER38> I would've loved to see the creativity that went on between you trying to get through as many pages as possible and your mom trying to ensure you read them

</comment></comment><comment><USER10> I'll be the devil's advocate. There's this idea out there that you should get your kids reading, no matter what they read. I see the idea expressed here, and I see it expressed in my kids' school.

I wonder what that idea is based on. Any actual experimental studies that imply a causal link between any kind of reading and intelligence or success or some such metric? Anything to show how a heavy-reading childhood compares to a heavy-videogames childhood?

The answer is of course no. You can't run such a study and have it be experimental. So I'm extremely skeptical of this idea.

Not that I keep books from my children. Reading is one of life's great joys, and that's enough of a reason to encourage it, in my opinion.

</comment><comment><USER39> How much of this is simply attributable to parents who love to read inculcating their kids with a love of reading -- as opposed to the presence of the books themselves?

<comment><USER12> I would say it is entirely parents inculcating a love of reading in their children. Kids could live in a library, but if they never read they wouldn't receive any benefit from their environment. That was the issue I had with the article. The author wrote it in a way that suggested the mere presence of books produced an effect. I believe the presence of books offers no benefit whatsoever unless kids read them.

</comment></comment><comment><USER40> How would growing up in a household with only ebooks, or a household with only physical books make a difference? (Assuming an equal number of both)

<comment><USER40> Perhaps completely irrelevant as long as an equal number are read?

<comment><USER41> Maybe, maybe not. These social studies are so poorly executed that you're just waiting for them to be debunked.

</comment></comment></comment><comment><USER42> This feels like another version of nature vs nurture. That is, if your parents had books, lots of books, there's probably a reason(s) for that.

I would think, the ultimate study would be of adopted and/or children who do not share DNA with their parents, or other siblings. Is the environment an equalizer, so to speak?

This is certainly interesting, but it's more correlation than cause.

</comment><comment><USER43> Definitely. When I was little, my parents had this amazing book about each of the planets in the solar system from the late 80s. It had all these theoretical paintings of what the surface of each planet looks like (or the interior of Jupiter). I didn't understand the science since it was more adult- focused. But it was definitely good to have around.

<comment><USER44> National Geographic Picture Atlas of Our Universe (1980). I loved that book as a kid!

</comment></comment><comment><USER45> I was fascinated by D&D rulebooks when I was a kid. The systems captured my imagination as much as the fantasy aspect. Because of that, my vocabulary included excellent words like "melee", "flail", and "barrage" while I was still in the single digits of age.

</comment><comment><USER36> Wide availability of access to the internet could have a similar effect, unless time is mostly spent on sites like Facebook. Reading became a lot easier (wouldn't space out) once taking acetyl L-carnitine + alpha-GPC. SEMAX + selank + alpha-GPC is even better. Iodine protocol.

</comment><comment><USER46> > Respondents, who ranged in age from 25 to 65, were asked to estimate how > many books were in their house when they were 16 years old. The research > team was interested in this question because home library size can be a good > indicator of what the study authors term "book-oriented socialization." > Participants were able to select from a given range of books that included > everything from "10 or less" to "more than 500."

In order words, they just used the number of books as an indicator that very likely correlates with how much people used to read as kids, how many books their parents could afford and how much their parents cared about education and so on. If you accept this correlation, the results of the study become somewhat trivial.

It's amazing how the people at Smithsonian Magazine maganaged to spin this as a causal relationship, though, in the sense of "Just surround yourself with books and it'll have a powerful, lasting effect on your mind." Relevant xkcd: [https://xkcd.com/552/](https://xkcd.com/552/)

</comment><comment><USER47> maybe being surrounded with 350+ books correlates with rich parents. And rich parents have an opportunity to pay for higher education for their children.

</comment><comment><USER48> Growing up in homes advantaged enough to have large libraries could indicate you are, in fact, advantaged!

</comment><comment><USER48> I could just say correlation != causation and move on.

...but I thought I would at least give it 5 minutes of googling.

Here's the first randomized controlled trial I could find testing whether access to libraries had any impact on literacy. TLDR (for this one study): access to libraries does not improve literacy.

"We conduct a randomized controlled trial of an Indian school library program. Overall, the program had no impact on students' scores on a language skills test administered after 16 months. The estimates are sufficiently precise to rule out effects larger than 0.053 and 0.037 standard deviations, based on the 95 and 90 percent confidence intervals. This finding is robust across individual competencies and subsets of the sample. The method of treatment, however, does seem to matter--physical libraries have no effect, while visiting librarians actually reduce test scores."

I will repeat the last line "visiting librarians actually reduce test scores." That seems counter intuitive. Maybe we should write articles w/ titles "LIBRARIES INCREASE ILLITERACY." That would be a dump article to write, but only slightly more dumb than the current article ;)

[https://www.nber.org/papers/w18183](https://www.nber.org/papers/w18183)

p.s. I like libraries.

</comment><comment><USER49> Obligatory Schopenhauer: [https://ebooks.adelaide.edu.au/s/schopenhauer/arthur/essays/...](https://ebooks.adelaide.edu.au/s/schopenhauer/arthur/essays/chapter3.html)

On reading and books.

</comment><comment><USER50> I never enjoyed reading fiction. Fiction trains your mind to look for fictitious narratives in real life; it distorts your perception of life and makes the world a worse place for others who don't follow those narratives.

<comment><USER51> Fiction certainly can have that downside.

But good fiction is also one of the most effective ways--apart from deep listening--of training empathy because it lets you step into someone's inner life. Rarely do we get a chance to see the world through someone else's eyes for long stretches.

If you're not an immigrant, you can vicariously live the life of one through fiction and see what they have to deal with.

If you're not a news reporter or a cop or a tycoon during the Great Depression, you can tag along with one through fiction.

The point of reading fiction isn't about content, but how they change you as a person.

<comment><USER52> I might think that of biography, or autobiography. But fiction writers often don't know what it's like to be that sort of person, they just make it up.

<comment><USER53> This is kind of analogous to photorealism versus cartooning. A history or biography would attempt to adapt events and experiences directly to the page: "this happened, and then this happened, and it felt like that". Fictional work transforms those same elements into exaggerated characters and scenarios. Both kinds of works require some creativity to build up a coherent story, but fiction has a freer hand to manipulate elements and choose its focus.

Just making it up without adapting from anything real usually isn't something that good fiction is built up from, though. That path goes directly to cliche, and it's easy to find cliched fiction.</comment></comment></comment></comment>


Stanford Computer Science '10-'11 Salary Survey Results - <AUTHOR>

I'm an undergraduate at Stanford and I recently got this mass email from the CS department. I figured it might be interesting to the Hacker News audience (with all the talk of bubble and stuff). No one said the survey results were not to be distributed so I assumed that it was okay to do this.
<comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment></comment><comment>--
CS/EE Undergrads
Data: I received 140 responses which described 360 job offers. 95% of the job offers were primarily located in the Bay Area, 5% were from the Midwest and East Coast. 10% of the job offers were from start-ups.
Salary offers ranged from $64,400 to $100,000. The average salary offer was $79,914. The median salary offer was $ 82,200.
About 70% of students were offered stock options. About 80% of students were offered signing bonuses. And about 60% were offered relocation assistance and there were others who did not report the statistics since relocating did not apply to them. Relocation assistance ranged from $2,000 to $10,000 with an average of $3,000. Bonuses ranged from $5,000 to $25,000 with an average of $5,700. I did not calculate the range of stock options because stock options offered by companies are so different in their actual and potential values.
Students who replied averaged about 2 job offers. However, students may not have reported on all the offers they received. The average student who replied to the survey all had some job experience, nearly all of it through summer internships and averaged 3 summer of work.
Location, scope of work, salary/benefits, environment/culture, company were the important factors in accepting the offers for the undergrads.
CS/EE Masters
Data: I received 145 responses which described 330 job offers. 94% of the job offers were primarily located in the Bay Area, 6% were in the Midwest and East Coast. 15% of the job offers were from start-ups.
Salary offers ranged from $66,000 to $120,000. The average salary offer was $94,634. The median salary offer was $93,000. The candidates who were offered $120,000 was reporting directly to the VP of his research group. Also, one of the candidates was offered $36,000 because the individual was doing an internship instead of a full-time position out of state which I did not include into the calculation.
About 78% of students were offered stock options. About 66% of students were offered signing bonuses. And about 43% were offered relocation assistance and there others who did not report the statistics since relocating did not apply to them. Relocation assistance ranged from $1,000 to $8,000 with an average of $1,740. Bonuses ranged from $5,000 to $35,000 with an average of $4,102. I did not calculate the range of stock options because stock options offered by companies are so different in their actual and potential values.
Students who replied averaged about 3 job offers. However, students may not have reported on all the offers they received. The Masters had a little more summer experience than the undergraduates, an average of 3 summer internships.
Like the undergrads, location, scope of work, company, and salary/benefits, and environment/culture seem to be the important factors for the MS grads.
CS/EE PhD's
Data: I received 26 responses which described 60 job offers. 79% of the job offers were primarily located in the Bay Area, 21% were in the Midwest and East Coast. 30% of the job offers were from start-ups. 5% of the job offers were from a university.
Salary offers ranged from $132,888 to $145,000. The average salary offer was $123,972. The median salary offer was $138,944. One of the candidates was offered $43,000 because the individual was pursuing a Post-doc at a university which I did not include into the calculation.
About 45% of students were offered stock options. About 45% of students were offered signing bonuses. Bonuses ranged from $5,000 to $19,600 with an average of $6,150. Relocation assistance ranged from $5,000 to $10,000 with an average of $7,500. However, they may not have reported on all the benefits they received. I did not calculate the range of stock options because stock options offered by companies are so different in their actual and potential values.
Students who replied averaged about 3 job offers. However, students may not have reported on all the offers they received. The PhDs had about the same amount of summer experience as the Masters, an average of 2 summer internships and with the exception of 2 full-time experience.
Like the undergraduates and masters, location, company, environment/culture, salary/benefits, scope of work seem to be the important factors when it came to accepting their job offer. </comment><comment><USER2> I'm curious how well the study results reflect actual numbers of offers, salaries, and benefits. I would expect that since it's a voluntary survey, there would be some amount of the 'high school reunion effect' going on here and that the survey results would skew higher than the actual. In the same way that those who look good and have done well are more likely to attend a high school reunion, I would assume that those students who have had relatively successful job searches would be more likely to respond to the survey.

</comment><comment><USER3> This is very interesting, thank you so much for posting. I must say that I cringe when I see "Low numbers were discarded because of X" in surveys. There is always a reason for X, but if possibilities for X are not determined and justified prior to examining the data, then we are cherry picking.* I am glad though that the elimination of numbers was at least disclosed.

* People graduating with a degree from Stanford can be presumed to look for full time work at a salary sufficient to pay off student loans and live in a reasonable shelter and buy food and health care. Earning less than a restaurant manager as an intern is not what people choose, it's what they take when there are no better options. Discarding it as invalid salary information is not well justified ex post facto. If we decide in advance of gathering data to discard people doing internships because these are unfavorable situations and not average ones, why not also discard all the favorable situations that are not average ones as well, such as high salaries received in offers where the student has a relative or friend who helped him find an especially favorable position. Or any other arbitrary criteria chosen after the fact upon examination of the data.

<comment><USER4> > _People graduating with a degree from Stanford can be presumed to look for > full time work at a salary sufficient to pay off student loans._

I just wanted to point out that Stanford has a no-loan policy. Most Stanford graduates owe either nothing or very little in student loans (anecdotal evidence so correct me if I'm wrong).

<comment><USER5> I don't know where this claim comes from, but as a Stanford CS grad (MS) I never heard of such a thing. Stanford definitely offers loans.

<comment><USER6> I believe this is only true for undergrads, and they started doing this in 06-07 or 07-08. I was undergrad '09 and definitely had loans for the first year or two, but not in subsequent years after they instated the "no required loans" policy.

</comment></comment></comment></comment><comment><USER7> _Salary offers ranged from $132,888 to $145,000. The average salary offer was $123,972. The median salary offer was $138,944. One of the candidates was offered $43,000 because the individual was pursuing a Post-doc at a university which I did not include into the calculation._

This makes me sad.

<comment><USER8> $43,000 is actually a bit higher than what Post-docs in other science fields (esp. biology) can expect to make. Unfortunately, Post-doc salary is often fixed by the institutions making the grants (typically NSF or NIH) so there is no competition in the Post-doc market. This wouldn't be so bad if Post-docs still served their original purpose of bridging a Ph.D. from mentored researcher to independent investigator. Unfortunately, with fewer and fewer tenured positions (and the increasing life-span of currently tenured faculty, especially in the US where there is no mandatory retirement like some other countries), this is no longer the case. Originally, Post-docs were expected to spend 2 years until they could establish their own lab. Today, Post-doc periods of 6 or 8 years are not unheard of.

<comment><USER9> CS isn't a field where a postdoc is a mandatory step on the way to a professorship (unlike, say, physics). Postdocs in CS are not too common, so it's hard to generalize.

<comment><USER10> It wasn't common in areas like systems until recently. However, it has gotten common in the past 3 years. In "Theory", I thought most people did post-docs after their PhDs.

<comment><USER11> The post-doc is now effectively mandatory in computer science academia like other fields. I don't know where this figure of 75k for an astrophysics post- doc comes from. The only 70k+ post-docs I've ever heard of are at government labs like Los Alamos. At Harvard, for example, post-docs make less than 50k per year (I'm a grad student there and have post-doc friends). If Harvard's Smithsonian center for astrophysics cannot pay market demanded prices for post-docs, then I just don't buy this claim that Santa Cruz could.

I second the sentiments mentioned above that the years and low-pay of a post- doc are depressing now that it is overwhelmingly likely that it won't lead to a tenured position.

<comment><USER12> I wouldn't say it's mandatory; about half of my grad-school friends did post- docs, and half didn't. It depends strongly on your sub-area, and in part on how flexible you are on location and type of school. In some cases, people who take post-docs _could've_ gotten professorships, but not at top-tier research universities, and preferred a year of post-doccing at Stanford to taking a professorship at a low-ranking school.

</comment></comment></comment></comment></comment></comment><comment><USER13> Wow. Very high salaries indeed. What I find particularly interesting is that the PhD's clearly have absolutely no incentive of staying in academia. I wonder how bad an effect this has on academia..

<comment><USER9> There's certainly no financial incentive for a Ph.D. graduate to remain in academia. That said, there is no financial incentive to do a Ph.D. in the first place. My understanding is that the opportunity cost (salary you didn't earn while doing your Ph.D.) is larger than the total expected extra earnings over your lifetime. My understanding is that by contrast a Masters degree is a good investment.

There are other reasons to stay in academia --- mostly to do with lifestyle. You have a lot more freedom to work on whatever interests you than in many other career paths.

<comment><USER14> There is certainly incentive to get a Ph.D. in many other studies (where the comparable industry salaries are much lower). Archaeology, East Asian Studies, etc.

</comment></comment></comment><comment><USER15> One thing I'm curious about is: at a larger software company like Google, Facebook, Twitter, Amazon, Microsoft, how much of a raise do you typically get every year?

Also, does Google's 10% across-the-board raise last year not apply to new hires?

<comment><USER16> In google if you are really good you could maybe get promoted every 1.5 years, but realistically this is probably more like every 2-3 years.

Your pay then gets bumped to the next grade (or whatever grade you move in to in cases where you transfer laterally) and from what I can tell the increase can be 20-30% increase.

You also will get regular stock option/unit refreshes each year based on performance, so those will start to add up over the years and can be a substantial part of your income assuming the stock is doing ok (and historically it has).

</comment></comment><comment><USER17> > The candidates who were offered $120,000 was reporting directly to the VP of > his research group

The candidates or the candidate? It seems like this candidate is easily identifiable... How many masters' grads from Stanford (145 responses) was offered 120,000 and reports to the VP of his research group?

This makes me wonder: How taboo is it to talk about your offers/comp details? When I get offers, I'm told to keep them confidential, but everyone knows that the numbers are openly discussed between friends.

<comment><USER18> This is a real problem. There were TWO of us at Drexel who interned one summer at Google. They published salary details BY EMPLOYER! I was livid when a professor made a snarky remark about how my internship paid better than his career. I gave the dean quite an earful over that.

</comment></comment><comment><USER19> I would be interested to know what % of students would state if asked whether they would prefer to stay in the Bay Area vs. relocate (assuming assistance) to another city. I know some companies on the East-coast don't focus too much energy on recruiting in West-coast schools because it is assumed there will already be a much lower % which would be willing to relocate in the first place. I'm just wondering what the actual survey #s would say and how tied the students are to the Bay Area tech culture.

</comment><comment><USER17> > Salary offers ranged from $132,888 to $145,000. The average salary offer was > $123,972. The median salary offer was $138,944.

How can the average be that much lower than the min?

<comment><USER20> I believe the colloquial term for this kind of event is "typo." Also "that much" is redundant because the average can never be lower than the min at all ;).

</comment></comment><comment><USER21> abi, sounds like all of these responses are for fresh grads (be it BS, MS, or Phd). It would be interesting to see the stats for offers after 1, 2, up to 10 years in industry. There was some element of 'hockey stick growth' for me from the year I graduated (unemployed for some months), to my first job (nothing to brag about, salary wise), to my second job (which is much better). I wonder if it will be the same for Stanford grads since they're already starting relatively high.

<comment><USER22> It might also be interesting to control for prior experience. Some people enter the masters/PhD program with years of industry experience, as I did.

I'm a little sad to see that after five years in industry I made less than the median Stanford masters grad and was pretty much on par with the recipients of bachelors degrees. I should complete my masters by the end of the academic year and I'm starting to wonder how much I could conceivably be making so I know what to ask for during salary negotiations. I'm expecting my experience should help some, but but I bet the degree will be the eye catcher.

</comment></comment><comment><USER23> Bay Area, Midwest, and East Coast are the three locations for each option, and they all add up to 100%. What about Southern Cali or the Pacific Northwest, both of which have at least some amounts of demand for CS graduates? I can't imagine they are 0%, so I'm guessing they are clumped with the Bay Area.

</comment><comment><USER24> "Salary offers ranged from $64,400 to $100,000. The average salary offer was $79,914. The median salary offer was $ 82,200."

Friends of mine got offers for 64k in Boise, ID. 64k seems low for the bay area. Or is 64k in a place like Boise, ID just a really good offer?

<comment><USER25> I believe the cost of living in places like Boise is much less than places like SF/NYC, so that's relatively decent.

<comment><USER24> With housing costs at almost half what they are in the valley, the cost of living is much lower in Boise.

I was there last summer on an internship and paying $550/month for a decent one bedroom.

</comment></comment></comment><comment><USER26> The question is also how this data was "collected". I once was asked to fill out some survey of my school were I had to enter a salary number.

Knowing that this is going to be used most likely for some inter-school benchmark it is making very tempting to polish up the numbers.

<comment><USER9> Since this is a copy of an email sent to CS students at Stanford, I'm not sure it's intended to be public information. One of the staff members in the CS department emails graduates every year to survey salary data and to ask about how useful the recruiting resources provided by the department are. I'd imagine the numbers accurately reflect the responses received, although as others have mentioned there may be biases in who chose to respond.

</comment></comment><comment><USER27> damn, i dropped out, so i guess this make me unemployable.

<comment><USER28> When I see these surveys from the top-tier universities, I often wonder what the ceiling is for those of us who are self-educated.

<comment><USER29> I have no degree. Base salary 160k roughly 50-60k in equity/year. 10 years experience. I'm not at the top of my game yet.

I started at 70k with no experience no degree.

<comment><USER30> I would perhaps argue that 70k starting with no experience or degree is very much not the norm. I'm not questioning you, but how/where?

<comment><USER31> I have no actual work experience prior to my current job and no degree. I am 23 years old and I have a 6 figure salary and equity in the Bay Area. If you know people that know what you are capable of and can refer you to a company then an amazing resume is completely irrelevant. Last week I turned down a job offer from Blizzard Entertainment (was by referral) because they could not match my current salary.

<comment><USER32> I'm jelly, you should post some your story for inspiration for all of us young people without a 6 figure salary :)

</comment></comment></comment></comment></comment></comment><comment><USER33> If anyone is interested for UK data, then the UK government has a neat table here: <http://data.gov.uk/appathon-2011>

It is called "Salary of graduates by course topic (National Student Survey)"

</comment><comment><USER34> Average starting salaries for the "top" universities in California (not arranged by degree): [http://www.payscale.com/best-colleges/best-schools-in- califo...](http://www.payscale.com/best-colleges/best-schools-in- california.asp)

</comment><comment><USER35> "I received 140 responses"

Uh... how many did you send out? How many people are unemployed? I had similar issues with my university, where I and everyone I knew made less than half the lowest quoted value, but had never received such a survey.

</comment><comment><USER36> For PhD's: "Salary offers ranged from $132,888 to $145,000. The average salary offer was $123,972."

how can the average be lower than $132,888?

<comment><USER37> fuck this law school nonsense - wish I had computer science background

</comment></comment><comment><USER38> Thanks a lot, its very useful information. I wish I had this kind of information when I graduated.

</comment><comment><USER39> It would be interesting to see how many started their own startup.

</comment><comment><USER40> would be interesting to know how many graduates would be willing to join very early stage startups. humble salary, good equity, great idea, great work.

</comment><comment><USER41> Carnegie Mellon CS undergrad stats: [http://www.studentaffairs.cmu.edu/career/Students/gps1/explo...](http://www.studentaffairs.cmu.edu/career/Students/gps1/explore/survey/pdf/scs.pdf)

Max $120k, Min $45k, Mean $85k, Median $87k

Much more diverse by region (39% CA, 21% WA, 17% NE)

MIT's is more difficult to parse: <http://web.mit.edu/career/www/infostats/graduation10.pdf>

<comment><USER42> I assume NE in the above stands for North East (of the US), not Nebraska?

<comment><USER43> Yes.

</comment></comment></comment><comment><USER44> You have an error with your PhD stats.</comment>


(take 5 kevin-lynagh) - <AUTHOR> http://clojure.com/blog/2012/02/28/take5-kevin-lynagh.html <comment><USER2> I've been using Clojurescript with a Noir backend to build an internal tool for work, and I thought I'd share my experience so far. Some background: I'm a Java guy by trade, this is my first FP language, and I'm not that well-versed in Javascript. In general, I've found the whole experience to be rather fun. A bit of a learning curve for sure, but once I ramped up I felt really productive. I don't feel like I get stuck as much as other times when I've tackled a new framework/language. And the nature of the language lends itself to just start writing functions rather than get caught up trying to abstract everything. I feel like my code flows better, is easier to refactor, debug, etc. Clojurescript may still be considered new/beta/cutting-edge, but I never found it to be buggy or frustrating to use. A few minor things I got hung up on:

goog.dom.query - This isn't included by default, I'm guessing because it's a third-party wrapper around dojo.query. I had to follow a few threads on Google Groups to figure out how to include it. Another option is to use Chris Granger's jayq. But there's no way to do CSS-type selector queries by default (that I know of).

Converting between Javascript objects and Clojure data structures - I had to scratch my head a bit the first few times I ran into this, but there are lots of code snippets on Stack Overflow and various other places around the web. I was able to quickly develop a few reusable idioms to handle this.

Lazy sequences and side-effects - people with more experience in FP might not have gotten stuck on this, but this was another head scratcher for me. Basically any type of DOM manipulation is a side-effect, so if you're app relies on that, you'll need to occasionally force evaluation of a lazy seq.

The browser-connected REPL takes a few manual steps, but it's well documented, and extremely useful. It's just not as easy as running "lein repl". The default compiler is not integrated with lein either, and is not automated, but lein cljsbuild is available as Kevin mentions. I ended up using Chris Granger's cljs-watch.

In any case, if you can't find it through Google, it takes all of about 15 secs to get your question answered on #clojure IRC.

My current setup includes VimClojure and a browser-connected REPL in my terminal. I'm using Noir on the backend, and the included Google Closure libraries on the front-end. I tried to minimize the number of additional dependencies (perhaps irrationally), so I opted out of using the remotes from fetch (formerly pinot), and just used goog.net.XhrIo calls to the noir backend to acheive a similar effect. I did find the crate library to be perfect for dynamically creating DOM objects. You can write something like [:form [<form body...>]] instead of "<form> <form body...></form>", and it's nicely highlighted with Clojure syntax. For debugging, I just use Chrome's debugger. It's compiled JS, but it's easy to figure out where you're at. All of your functions are at the bottom of the script. You're not going to have a lot of state to track in the actual Javascript either, so most bugs are easily sniffed out, at least in my somewhat simple app. Sorry for the long post.

<comment><USER3> For jQuery-like selector functionality without sacrificing maximal code compression you have Domina - <https://github.com/levand/domina>

</comment></comment><comment><USER4> This interview is from November 2011; ClojureScript tooling has improved since then.

Evan Mezeske's lein cljsbuild plugin (<https://github.com/emezeske/lein- cljsbuild>) is a great place to start.

</comment><comment><USER5> Kevin will be talking about using ClojureScript and d3 to create graphics on the web at Clojure/West (Mar 16-17, San Jose, CA) - <http://clojurewest.org/sessions#lynagh>

<comment><USER4> For anyone who's tempted to dive into my ClojureScript d3 wrapper (<https://github.com/lynaghk/cljs-d3>), hold up---there just might be Clojure/ClojureScript-native data visualization library open sourced at Clojure/West...

</comment></comment><comment><USER6> Clojure compiles to JVM bytecode; what's stopping browser-builders from creating a similar bytecode format (or even JVM) so developers can build in their language of choice? Time/effort?

<comment><USER7> I found this article [1] that makes the argument that JS doesn't need a bytecode format, and sorta-but-not-really comes to the conclusion that a compact AST format would be a better choice as an intermediate, and mentions JSZap [2] (which itself seems to be more about compression than being used as an IL).

I think an AST format that v8 and other JS engines could read could be awesome.

Ultimately, I'd like to have a Clojure that can compiled to this AST. On servers, the ast files could be cached a la Python .pyc files.

[1] <http://www.2ality.com/2012/01/bytecode-myth.html>

[2] [http://www.slideserve.com/elina/jszap-compressing- javascript...](http://www.slideserve.com/elina/jszap-compressing-javascript- code)</comment></comment>


TLC, a tiny lua cocoa bridge - <AUTHOR> https://gist.github.com/2211379

<comment><USER2> That is a spectacular amount of functionality in one small source file. It appears to open all objective C classes and their methods to Lua.

<comment><USER3> I looked at these comments before the code and assumed it was going to be an ugly first-principals dump of code connecting lua to objc.

What I found instead is beautiful.

</comment></comment><comment><USER4> What's the reason to use this over <https://bitbucket.org/ewing/luacocoa> ?

</comment><comment><USER5> Amazing. Can you use this to run a LuaCocoa example? Like [https://bitbucket.org/ewing/luacocoa/src/e939d422b62a/LuaCoc...](https://bitbucket.org/ewing/luacocoa/src/e939d422b62a/LuaCocoa/Samples/MinimalAppKit/MinimalAppKit.lua)

Can you load a framework with this? I know hardly anything about Objective-C.

<comment><AUTHOR> There's no need to load a framework since linking to a framework means all of it's classes are added to the runtime. You simply request the class you want. That is, if you are embedding lua in your app.

I haven't tried using TLC from the standalone luajit interpreter, but in that case you would probably have to use fii.load() to load the Framework & libobjc before being able to use objc. But that is not what it was designed for

To translate a LuaCocoa example you'd in most cases just have to add objc_loadClass calls for any class you want to use, and use dots instead of colons to call methods (obj:method() -> obj.method() ).

</comment></comment><comment><USER6> Hmm, this looks very interesting. Could it be used as a scripting layer within a OSX/iOS application or could you whip up a complete application with this ?

<comment><AUTHOR> It was designed for use for scripting within an objc app. (<http://github.com/aptiva/tranquil> to be precise)

It doesn't support subclassing objective-c objects so that would probably limit you if you tried to write your app exclusively in lua.

</comment></comment><comment><USER7> Anyone know any real world examples of Lua being used with Objective-C, and what use cases it is good for? Sounds very interesting.

<comment><USER8> Take a look at the app that the OP is building. It's pretty interesting and uses this scripting bridge for generating OpenGL visuals. (<http://github.com/aptiva/tranquil>)

</comment></comment><comment><USER9> Question: how is this different than iPhoneWax?

<comment><AUTHOR> Well, the biggest difference is that it's written in Lua. And it uses LuaJIT not vanilla Lua. (LuaJIT being orders of magnitude faster)

And from what I can tell, iphonewax is intended for writing entire apps in lua, this project is intended to be used when embedding bits of lua code in your objc app. (But there's nothing stopping you from writing most of your app in it)

</comment></comment><comment><USER10> Is it using LuaJIT or any another ffi library?

<comment><AUTHOR> Yes, it does indeed require LuaJIT. (I probably should indicate this more clearly in the source)

<comment><USER11> Did you try it in vanilla Lua with the FFI port linked in a sibling of your post?

<comment><AUTHOR> Nope, can't say I have.

</comment></comment></comment></comment><comment><AUTHOR> It's moved to <https://github.com/aptiva/TLC>

</comment><comment><USER12> Amazing .. off to pack it into an iOS bundle and wire it up to an editor. ;)</comment>


Detailed guide to mouse sensors - <AUTHOR> https://sensor.fyi/info/ <comment><USER2> Excellent write up! Wish I saw saw this before I bought my mouse :)</comment>


Bfxr. Make sound effects for your games. - <AUTHOR> http://www.bfxr.net/ <comment><USER2> This is the most awesome thing I've seen here in a while. I'm in serious danger of blowing off the rest of my day to make a video game to stick these into.

<comment><USER3> It really opens some doors doesn't it :)

This looks like a great implementation of the sfxr tool that's been around for a long time. I've seen it in Flash and there's an EXE version too. This one has some nice new add-ons I haven't seen yet, and a GREAT interface.

If you REALLY want to go nuts, here is a massive list of indie game development tools that can keep you busy for a while:

[http://www.mangatutorials.com/forum/showthread.php?742-The-U...](http://www.mangatutorials.com/forum/showthread.php?742-The- Ultimate-Indie-Game-Developer-Resource-List)

<comment><USER4> If this is your idea of a great interface then you need help :-)

Seriously, as someone that works with sound and software synths a lot, it looks awful to me. What is it that you like about this?

</comment></comment></comment><comment><USER5> Related: <http://thirdcog.eu/apps/cfxr> (aka Cocoa sfxr) better UI than sfxr and bfxr, but simpler than bfxr.

Original project: <http://www.drpetter.se/project_sfxr.html>

<comment><USER6> SFXR and CFXR have been two of the most useful tools I've had while developing games. So simple yet so flexible. I really recommend them for any game developer.

<comment><USER7> SFXR/CFXR are so much fun to play with. I'm glad this dev listed them as inspiration.

</comment></comment></comment><comment><USER8> And this is the javascript one <http://www.egonelbre.com/js/jsfx/> :)

</comment><comment><USER9> Well... was there ever a shortage of 8-bit sounds you can put into your games? These creators have been around years, i remember fiddeling with one 10 years ago when i was playing with RPG Maker 2000...

A High Def Sound Creation Tool would be nice...

<comment><USER10> ... like, say, any synthesizer?

For free, I suggest: Synth1 - www.kvraudio.com/get/214.html

You'll need a VST host, for which I suggest www.reaper.fm

</comment></comment><comment><USER4> Those of you who are simply enjoying the bleeps and bloops should drop by a music store and buy yourselves a Monotron (or two!) for Christmas. $60 each and you can download the circuit diagrams from Korg, who has designed them to be hackable.

<http://www.korg.com/monotron>

</comment><comment><USER11> I just set up a cache of sounds to use in place of verbal responses today - mostly Coins, Lasers and Jumps. They just seem to say so many things better than I can. It has already gotten me into trouble. In other words, it's awesome. Thanks.

</comment><comment><USER12> This has been around for a while, it's pretty cool nonetheless. I used it for sond effects on a pong game I made while learning C++.

On a side note that's the best way to learn to make games (or a new language for that matter), recreate old games from scratch. You'll also learn some trig and calculus while you're at it.

</comment><comment><USER13> The one problem with the *fxr family is that it is a cliche already. It's like the official soundtrack to Ludum Dare...

</comment><comment><USER14> Wow, that is just incredible.

Just randomly hitting a few of the buttons, it produced really excellent 2 x Atari 8-bit style explosions, and as near as damn it, a perfect Defender gunfire effect.

I haven't looked to see if it can do musical notes, but if it can, surely this would be excellent for chiptunes.

</comment><comment><USER15> Wow cool! Reminds me of this standalone I used to use to make a ton of simple drum sounds and blips to put in trackers...

<http://www.threechords.com/hammerhead/drumsynth.shtml>

<comment><USER16> Have you seen Sunvox? Tracker/synth/sequencer all in one:

<http://www.warmplace.ru/>

Lots of fun.

</comment></comment><comment><USER17> This is pretty awesome. I hit randomize and got this <http://goo.gl/EuMtA> , I can totally see it as a warp engine, or something like that. I'm going to play with it for a while...

<comment><USER18> Did same thing and got this: <http://bit.ly/vjDZDG> This is so awesome. Woah!

</comment></comment><comment><USER19> Really neat extension of sfxr, but man I wish there wasn't that inner scrollbar.

</comment><comment><USER20> I am going to use the hell out of this site.

I kept hitting randomize and got this: <http://t.co/Av3WNjzB>

Now I want to make a game around it.

<comment><USER21> Interesting. By the way, no need for URL shorteners: [http://www.bfxr.net/?s=4%2C0.5%2C0.7518%2C0.0237%2C0.1582%2C...](http://www.bfxr.net/?s=4%2C0.5%2C0.7518%2C0.0237%2C0.1582%2C0.4515%2C0.8135%2C0.5157%2C%2C%2C0.0403%2C0.2709%2C0.0128%2C0.2663%2C0.8745%2C0.3809%2C-0.8757%2C0.8916%2C-0.4377%2C0.5863%2C0.1847%2C0.0447%2C0.4012%2C-0.8036%2C-0.9988%2C0.7131%2C-0.3451%2C0.6465%2C0.8841%2C0.0155%2C0.6218%2C0.0079%2CmasterVolume)

(People avoid URL shorteners here. Keeps the Rick Rolling low.)

</comment></comment><comment><USER22> Wait what just in time for Ludum Dare 22?

Sweet!

<comment><USER23> I've been finding the LD entry posts all quite interesting; they often have stuff (like bfxr) that I didn't know about, or at least, didn't know was that popular:

<http://www.ludumdare.com/compo/>

</comment></comment><comment><USER24> Whoo! They finally added history so you can keep track of a few sfx that interest you, very cool. :)

</comment><comment><USER25> Truly awesome, but could need some interface improvements.</comment>


Is the Higgs just two photons? Einstein's equation explains - <AUTHOR> http://particality.org/ <comment><USER2> I tried to copy the permalink, but I got this: [http://ec2-52-205-143-146.compute-1.amazonaws.com/particalit...](http://ec2-52-205-143-146.compute-1.amazonaws.com/particality/?p=8)

<comment><USER3> Thanks for pointing that out</comment></comment>


News Corp launches new ad network to take on Facebook/Google - <AUTHOR> https://www.axios.com/news-corp-to-fight-google-facebook-2514912101.html <comment><USER2> >The new platform, called News IQ, will pull audience data from sites like The Wall Street Journal, New York Post and Barron's and give advertisers a way to reach specific audiences around safe content.

>While none of these new ad networks are as big as Google or Facebook's, they are hoping to offer more brand-safe content at scale — major selling point for advertisers spooked by terrorist content and suicide videos.

These guys think that they have a chance? Advertisters go where they get most bang for their buck. Time will tell how they do, but it's really hard to compete with Google and Facebook who know their audiences better then the audiences know themselves.</comment>


Why recursive data structures? - <AUTHOR> http://raganwald.com/2016/12/27/recursive-data-structures.html <comment> Const-me Apparently, every computer science problem has easy to understand solution that's easy to implement but awfully slow in practice.

This is one of them.

Rotating images that way isn't CPU bound problem. It's RAM bandwidth and RAM latency bound problem. The optimal data structure for that is not recursive, it's 2D array of small square blocks of pixels. If pixels are bits, I'd bet on either 8x8 or 16x16 blocks. If they are RGB, one good format for those images, optimized for rotations, is BC7 [https://www.opengl.org/registry/specs/ARB/texture_compressio...](https://www.opengl.org/registry/specs/ARB/texture_compression_bptc.txt) the decompressor being implemented right in the graphics hardware.

> If we were writing an image manipulation application, we'd provide much > snappier behaviour using coloured quadtrees to represent images on screen.

That statement was true 30 years ago, no longer a case. Branch misprediction is expensive, memory latency is huge, RAM is a block device now (the block size typically being 16 bytes for dual-channel DDR). In addition, some displays now show 8+ megapixels, and most show 16M colors.

For an image manipulation application, that recursive thing ain't good at all.

<comment><USER3> > The optimal data structure for that is not recursive, it's 2D array of small > square blocks of pixels. If pixels are bits, I'd bet on either 8x8 or 16x16 > blocks.

If the image is very large and contains large contiguous regions of solid colors, then a quadtree that doesn't subdivide homogeneous regions can be a much faster approach than a flat pile-o-pixels.

Either way, this article isn't about rotating images.

<comment><USER4> > _If the image is very large and contains large contiguous regions of solid > colors_

In other words, if the image is not very image-y? (but more vector-y).

<comment><USER5> It's an entirely plausible scenario. It will rarely be the case with photographs, but graphical images often feature large solid color blocks. It's large reason why the PNG file format exists.

</comment></comment></comment></comment><comment><USER6> Recursion is often times less efficient than a well designed and concrete imperative solution, even if the imperative solution uses more memory. This is because most languages, the stack is not free. There's a lot of meta data and house keeping that goes on when you invoke a function to support things like stack traces and introspection.

The provided example (at least to me) is also convoluted and fails to illustrate the elegance of using a recursive data structure.

In general parlance, an affine transform or rotation matrix would both be more elegant and readable.

<comment><USER7> Are you aware of a programming language/compiler that optimizes on recursively written code? I'm finding it difficult to imagine a PL where stack _isn't_ free except tail-recursive code.

<comment><USER8> My understanding is that it's the opposite -- function calls expend a cost to save (and later restore) the registers and push the locals onto the stack. Tail recursive functions on the other hand can be implemented as a jump to the top of the function, right after all the stack bookkeeping, and so saves on that expense.

<comment><USER9> Tail-merging doesn't only apply to recursive functions. As soon as you don't need to store intermediate result, you can jump to another function without allocating a frame.

</comment></comment></comment></comment><comment><USER10> I really enjoyed this article because it introduced me to a new data structure and some very elegant and (subjectively) beautiful algorithms. I came onto HN expecting some insightful commentary and instead got 80 comments griping about how it wasn't 100% optimized.

Sigh.

<comment><USER11> Author here. Nevertheless, there's learning gold in those comments. Take the gold, leave the mud.

</comment></comment><comment><USER12>
      maxIndex = size - <USER2>
      for x in range(0, size):
        for y in range(0, size):
          # swap diagonally, invert horizontally
          out[x][y] = in[maxIndex - y][x]

</comment><comment><USER13> The algorithm to rotate images can be illustrated by running

        /usr/lib/xscreensaver/blitspin

_blitspin - rotate a bitmap in an interesting way_

 _The blitspin program repeatedly rotates a bitmap by 90 degrees by using logical operations: the bitmap is divided into quadrants, and the quad‐ rants are shifted clockwise. Then the same thing is done again with progressively smaller quadrants, except that all sub-quadrants of a given size are rotated in parallel. So this takes O(16×log2(N)) blits of size NxN, with the limitation that the image must be square, and the size must be a power of 2._

Movie here, for those without XScreenSaver installed:

[https://www.youtube.com/watch?v=UTtcwb- UWW8](https://www.youtube.com/watch?v=UTtcwb-UWW8)

<comment><USER11> Thanks, I have included the video and a credit to you in the essay.

</comment></comment><comment><USER14> Quadtrees are neat. Though I guess general purpose image processing would be better served by either raster or vector representation. Maybe a better showcase would be something like collision detection or geospatial indexing.

<comment><USER15> I have a bit-rotting ray tracing library where I put all geometric primitives in an oct-tree. When you fire a ray through a 2x2x2 cube it intersects at most 4 of the child nodes. You can see that because the ray must originate in one octant, and it must cross one of the 3 partition planes to reach another octant and it can only cross each one once. There are lots of other interesting properties.

One of my favorite algorithms is one that takes an oct-tree and a triangle and inserts the triangle into the tree. I allow primitives to occupy more than one tree node, so first I decide what size node they will occupy and then I use a recursive algorithm to "voxelize" the triangle. I found a heuristic for determining optimal node size.

The ability for node to contain and arbitrary number of primitives and for each primitive to be in an arbitrary number of nodes resulted in a very interesting data structure that allows fast deletes and inserts of the primitives. But I'm really starting to ramble now.

<comment><USER16> Could you go on a bit more (or post a link to somewhere you do)?

<comment><USER15> I've been hoping to put the code online for a long time, but I'm suffering from "it's not cleaned up enough" syndrome. But I'll talk a little here.

I'd been after real time ray tracing since the 1990's so I had a goal of putting both static environments and dynamic objects/players in the same acceleration structure. A lot of the old BSP-tree stuff or other BVH are used only for static geometry and I didn't like that. The problem with those structures is that if you move some things around it may require regeneration of the entire structure. Or at the very least, local updates that result in a different structure than if you regenerated the entire thing.

Part of the reason for that is trying to partition the geometric primitives roughly in two groups so a partition roughly cuts the scene in half. I decided that with a regular octtree structure, which nodes a primitive is contained in could be independent of the other primitives. They may share tree nodes, but I never decide whether to split a node based on content - instead each primitive is inserted into the nodes that make sense for it. This does lead to a few nodes having a lot of content - thing of the node that encloses a triangle fan - but those are rare and you won't typically be shooting lot of rays through them.

Also, with the primitives deciding what nodes they occupy, we can now do local deletes and inserts without having to regenerate the whole world structure.

For inserting a primitive we take its bounding box. If that is outside the root box of the world, I repeatedly double the size of the world by adding a new root node and placing the existing roots children and some new parents inside it - that's a neat recursion too. Next, we traverse the tree from the root calling on only the children that overlap the bounding box until we reach the node size that was requested. This decent is trivial for bounding boxes, but doing it with high precision specifically for triangles was harder.

Once the nodes that need to contain a primitive are reached, it's a matter of adding that primitive to a linked list off the tree node. But each primitive will need to exist in multiple lists. This means not putting the primitives directly in a list, but having a list of little link structures that are added to the nodes list and point back to the primitives.

For deletions, each primitive will need a list of link nodes that point back to it. Deletion consists of following this list of links and deleting them from the list that contains them.

So the links themselves will each be a part of two lists. They are part of a list coming off a primitive and they are part of a list coming off the tree node. It took a lot of thought to shrink the link node data structure, but at present it is 4 pointers in size. It needs to contain two "next" pointers, a "previous" pointer (to allow deletion from the tree nodes list of primitives), and it also has to have pointers to both the tree node AND the primitive.

The reasons for those last two pointers is thus: when checking which primitives a ray intersects, we want to start at the tree node that contains it and quickly get to the primitives for ray/object intersection tests. This is done by following the list of links via their "next" pointer which also following their links to "primitive". For deletions, we start at the primitive and follow the other set of "next" pointers and do deletes from the other list by using both the next and previous pointers. We never need a previous pointer for the list off the primitive because that entire list is always deleted, where the other list may still contain other primitives. There remains a problem that when an Octree node becomes empty, I need to prune back the tree - this is why the links need to also contain a pointer back to the tree node that contained the link. So now we're looking at a link structure that contains 5 pointers {2 nexts, one previous, one primitive, and one treenode}

Then I had the insight that no matter which list we're traversing, we always have a pointer to either the treenode or the primitive. So those two pointers can be combined via sum or XOR into a single value. If your deleting a primitive you just XOR a pointer to the primitive with that value to recover a pointer to the tree node. When you're doing intersection tests, you XOR the treenode pointer with that value to recover a pointer to the primitive. This change resulted in no significant performance change while reducing the structure to a nice size of 4 pointers.

Another problem was that having a linked list with a previous pointer meant the octree node needed to always contain one of these link structures because deletion depended on that "back" pointer pointing at another link. Turns out you can have the back pointer point directly to the "next" pointer of the previous link. That means the octree node now only needs a single pointer to the first "link" node in the list and that links back pointer points directly to that pointer.

So for fitting an arbitrary number of things (primitives) each into an arbitrary number of boxes (tree nodes) we add a single pointer to the thing structure, a single pointer to the box structure, and then use these little link structures that are the size of just 4 pointers.

It was a long journey to arrive at that data structure, but for a while it was real point of pride for me. The structures are nice, but the code is still a bit messy with the remains of several experiments still around. Also notice that the list in one direction does not have a back pointer, so you can never delete a box directly - well you could but it's not as easy. There isn't complete symmetry between the boxes and things.

Is that enough rambling for now? ;-)

<comment><USER14> Nice puzzle! I think you can allow both kinds of deletions while still using four pointers per link node:

* prev1 XOR next1

* prev2 XOR next2

* prev1 XOR prev2

* data1 XOR data2

Because while traversing you always know one of {prev1, prev2} and one of {data1, data2}. Right?

</comment></comment></comment></comment></comment><comment><USER17> Before reading this, I would have said that we need to use recursion to model constructs like programming languages that are specifically designed to have a recursive structure. They are naturally recursive because someone designed the language that way. (By contrast, building a basic database-backed website doesn't require any recursion, unless you're modeling a hierarchy.)

But with a compiler, there is the same pattern of converting to a recursive data structure (parsing), performing manipulations, and then converting back to a non-recursive structure (object code generation).

</comment><comment><USER18> Reginald is definitely hinting at hylomorphisms here. I'd be curious to see if stream fusion could be achieved in JS for a hylomorphism.

<comment><USER11> Isn't `multirec` a function for making hylomorphisms? The recursive dividing is the anamorphism, and the combining of the results is the catamorphism.

As written, however, `multirec` does not efficiently compose. For two constructions to be compatible, they'd have to share their decomposition strategy, including `indivisible` and `divide`. `value` probably composes neatly, but `combine` is a little thorny as the current implementation does two jobs: Processing the data and reconstructing the form of the data.

A composable `multirec` does sound interesting, however. Hmmm...

</comment></comment><comment><USER19> This is worth it, if only for the footnotes. Don't skip them!

</comment><comment><USER20> I liked this article because I enjoy reading about data structures that can be used in places where the "obvious" implementation is a list of lists. A lists of lists data structure is general enough for many applications and readily available in many languages, but when using it, I often get the sense that I am spinning wheels writing far more code than necessary to do the desired manipulations. It makes me happy to read about more novel solutions.

</comment><comment><USER21> The argument would be stronger if he wasn't swapping an O(n) algorithm for O(n log(n)). With the recursive implementation you have to touch every element once for each recursion level.

<comment><USER11> For quadtrees come into their own when there are optimizations that fit the problem domain, such as the coloured quadtrees explained at the end of the post: They are much faster for images with large blank areas.

They aren't discussed in the post, but quadtrees are also very amenable to memoizing common operations.

</comment></comment><comment><USER22> The footnotes say that all the functions can be adjusted to deal with non–power-of-2 squares, but it's not obvious how this is done when working with region quadtrees.

<comment><USER11> One way is to represent squares as a square-that-is-a-power-of-two, plus a "clipping size" that is used when the square is rastorized. If you're using a lot of optimizations for blank regions, the extra padding has a negligible effect on performance.

<comment><USER22> I suppose that would work, but all the operations you do on the region quadtree have to be aware of the "clipping size" as well. For example, the rotation transformation that this article implements would also have to rotate that "clipping size" as the unwanted portion of the square is now in a different location.

<comment><USER11> That depends If you can live with evenly-sized squares, you can always center the clipping square, and you're good for rotations, flips, and so on.

If you want odd-sized squares, or other shapes, then yes, you'll need to know how to transform the clipping path/description as well.

All in a day's work!

<comment><USER23> One can use recursive properties better if the structure divides equally. So if not having a strict quadtree is okay, they can divide the square into 3x3, 5x5, etc

</comment></comment></comment></comment></comment><comment><USER24> The 'multirec' function here looks to have a lot in common with Clojure's transducers. Can anyone more knowledgeable do a compare and contrast?

<comment><USER11> `multirec` is a combinator, it transforms functions into other functions. `transducers` are "composable algorithmic transformations."

`multirec` as presented here is not composable the way transducers are composable. It is, however, an extremely stimulating thing to consider.

[https://news.ycombinator.com/item?id=13305176](https://news.ycombinator.com/item?id=13305176)

</comment></comment><comment><USER25> Today I learned a different, much more harsh meaning for "bespoke"

I've never picked up on a hipster connotation from the word, though I did feel it was coming back into vogue.

<comment><USER26> It's been popular in the derivatives space for a while now.

</comment></comment><comment><USER11> FYI, this article on recursive combinators from 2008:

[http://www.lazutkin.com/blog/2008/06/30/using-recursion- comb...](http://www.lazutkin.com/blog/2008/06/30/using-recursion-combinators- javascript/)

</comment><comment><USER27> Interesting article. While reading it I wrote a version of the code in Racket. YMMV but I think it turns out cleaner, especially towards the end.

[https://gist.github.com/bon/801fdb6dbced0efdb83a358bb4f6285f](https://gist.github.com/bon/801fdb6dbced0efdb83a358bb4f6285f)

</comment><comment><USER28> Got an opportunity to do this recently using a graph mapped on to a K^2 Tree to find out if a relationship existed or not. See [https://maxdemarzi.com/2016/12/27/connected-2/](https://maxdemarzi.com/2016/12/27/connected-2/)

</comment><comment><USER29> Umm... or, I could just rotate a square a quarter-turn by iterating over the square's elements:

for (x = 0; x < N; x ++) { for (y = 0; y < N; y ++) { newSquare[x][y] = oldSquare[y][N-1-x]; } }

Of course, in certain representations of the square (such as using a 1-d array) even simpler algorithms can be used.

</comment><comment><USER30> As a web developer, I appreciate this rare use of JavaScript in a CS lesson.

<comment><USER11> JavaScript is the language most likely to be familiar to the audience I'm trying to reach. But at a certain point, it gets in the way of explaining an idea rather than making it easier.

I think this essay stays on the right side of that trade-off.

</comment></comment><comment><USER31> Wait, the way he defines the quad tree color property requires that you manually input the color for each region. If you wanted this to be part of a general purpose algorithm in real software then you'd need another algorithm that determines the color of each region - a very expensive operation if you wanted the maximum benefit.

</comment><comment><USER32> Raganwald is the poet of code.

</comment><comment><USER33> But why recursive data structures?

(Both a recursion joke and a Zoolander joke)

<comment><USER34> [https://news.ycombinator.com/item?id=13308230](https://news.ycombinator.com/item?id=13308230)

</comment></comment><comment><USER35> I stopped reading at the first sentence when he improperly used `isomorphic`. I thought it made no sense at all so I looked up the footnote... just to get a big middle finger from the author.

What a con artist. Moving on.

<comment><USER36> If you suspend your disbelief until the end, it's an inductive anti- hypothesis. Induction is recursion under intuitionism.

</comment></comment><comment><USER37> I love the word "isomorphic". It's by far the most efficient word in the English language.

When someone else uses it in conversation, it's absolutely guaranteed that the person is a pretentious asshole.

When you use it in a conversation, the same is also true.

It's only ever acceptable in written contexts or if you happen to be Douglas Hofstadter.

<comment><USER38> > When someone else uses it in conversation, it's absolutely guaranteed that > the person is a pretentious asshole.

I don't get your angle here. You realise an isomorphism is actually a well- defined mathematical concept, right? Are you advocating we use another, plainer word for an isomorphism, or do just naturally get angry when you hear words you don't understand?

<comment><USER12> Pretentious or not, the term is used erroneously here.

Recursive data structures and recursive algorithms are, quite plainly, not isomorphic. They are not two equivalent representations of the same thing.

Something that is isomorphic to a recursive function that works on a quadtree would be an imperative function that works on a quadtree with identical functionality.

tl;dr: I think the author meant "synergistic".

<comment><USER11> I don't know that an algorithm and a data structure have to be equivalent representations of the same thing to be isomorphic. In the case of the algorithm, is it the code that must be equivalent? Or the runtime behaviour?

The runtime behaviour of the 'whirling regions' algorithm clearly is not an equivalent representation of the two-dimensional array representation of an image. However, the runtime behaviour of the quadtree algorithm exactly matches the quadtree data structure.

This is my proposition: That what matters is whether the algorithmic behaviour is equivalent. I would say the same thing about `binrec` matching a binary tree.

...Or at least, I would have, up until today. I am open to being disabused of this notion.

<comment><USER39> Isomorphic means "have the same shape". It means that there is some structural property that the two objects have in common. The more interesting/complex the property is (and the more superficially different the two objects look), the more defensible the use of the word is.

</comment></comment></comment></comment></comment><comment><USER40> That was a lot of words to say "because the languages we use to solve problems is naturally recursive."

<comment><USER41> true, but sometimes people need to be convinced by a preponderance of evidence.

</comment></comment><comment><USER42> Using the same didactic style as the author, I would like to show how it is possible to replace the image rotation algorithm with a NOOP. First, look at any of the pictures shown. Next, turn your head to the side 90 degrees. Although the computer's output hasn't chamged, by redefining requirements we can achieve the the desired change. The fastest way to rotate a square is a noop (or O(1) message to the user): turning your head. Never forget it!</comment>


An Introduction to Cache-Oblivious Data Structures - <AUTHOR> https://rcoh.me/posts/cache-oblivious-datastructures/ <comment><USER2> For everyone interested in this, I would like to recommend the MIT Advanced Data Structures course, by prof. Erik Demaine (mentioned in the article). The part of the course about cache oblivious data structures starts on this lecture:

[https://www.youtube.com/watch?v=V3omVLzI0WE](https://www.youtube.com/watch?v=V3omVLzI0WE)

<comment> quantum-qaem I watched the lectures last week but it was difficult to understand. Can you recommend any beginner data structures course?

<comment><USER2> As @Cursuviam said, Introduction to Algorithms is a pretty good introduction:

[https://www.youtube.com/playlist?list=PLUl4u3cNGP61Oq3tWYp6V...](https://www.youtube.com/playlist?list=PLUl4u3cNGP61Oq3tWYp6V_F-5jb5L2iHb)

There are two things that make the lectures difficult to follow, IMO:

The first thing is that you need to be comfortable with the names that are given to things, understand well the simpler data structures like different kinds of trees, linked lists, graphs and so on, with their corresponding operation running times. Go watch Algorithms and you'll be fine :)

The second is that the subject is _really hard_. During the course, most of the time we're learning about (close to) state of the art data structures, so don't expect to understand everything in one go.

If you want to understand it in depth, you'll probably have to more time studying each lecture. Try thinking about how you'd solve the problem, try to understand the proposed solution and go through the video very slowly making sure you understand the details, try to implement at least some of them.

I used these lectures as a way to get comfortable with the field and get a broad overview of the kind of solutions that usually work, so I still have to go through most of Step 2.

</comment></comment></comment><comment><USER3> Trivia: Why they are called "cache-oblivious" and not "cache-aware"?

The paper that coined the term is here: [http://cacs.usc.edu/education/cs653/Frigo-CacheOblivious- FOC...](http://cacs.usc.edu/education/cs653/Frigo-CacheOblivious-FOCS99.pdf)

Basically, there were first cache-aware algorithms that assumed certain cache sizes and other properties. "Cache-oblivious" algorithms were a refinement that worked well for many cache sizes.

All in all it's silly that the "cache-oblivious" term was the one that survived, because now cache-unaware and cache-oblivious algorithms mean the opposite things - contradicting the dictionary definition of oblivious.

<comment><USER4> It all become clearer if you replace it in your head by

"cache sizes oblivious" algorithms.

<comment><USER5> Nope. Still sounds like "having no awareness of cache sizes and lacking a strategy for dealing with them."

No matter how you prefix it, 'oblivious' means you are missing something you are expected to be responsible for. It is a word expressing negative sentiment.

</comment></comment></comment><comment><USER6> Another fun one is radix sort[1]. We used to use it for sorting alpha'd quads(think particle systems/ui/etc) from front-to-back. Easily annihilates anything else in terms of raw performance for datasets < 10,000 with int/f32 keys and it's stable(!) for identical values.

This talk[2] from Herb Sutter(starting at 22:30) is one of my favorite talks and does a great job of explaining why this is so important(and why dense arrays are still king in many areas).

[1] [http://stereopsis.com/radix.html](http://stereopsis.com/radix.html)

[2] [https://channel9.msdn.com/Events/Build/2014/2-661](https://channel9.msdn.com/Events/Build/2014/2-661)

</comment><comment><USER7> The article looks great but a doubling of speed actually seems kind of disappointing (What I'd like to see is would something like "big O" improvement).

I have been investigating cache-oblivious data structures for a while. They're impressive, complex and a bit difficult to fathom (for a cache-oblivious tree, you have to consider both relational structure created by pointers and the layout of all the pointer in memory - the article seems like it gives a nice overview compared to academic articles I've looked at).

The thing is, in most of the large applications I've tried to optimize, you could squeeze several speed doublings out of fairly easy inner-loop rewrites as soon as you started profiling the app. After that, you got more marginal results from optimizing this and that. Consider; if 25% of the time is spent retrieving data, a halving of time to do this results in a 12.5% increase in performance. Is that 12.5% worth the complexity of implementing a cache oblivious structure? Even if you application is a full database, I'm not sure if the trade-offs are worth it.

<comment><AUTHOR> It actually may be an order as you suggest -- the difference gets bigger as the tree gets bigger. I just stopped benchmarking at about 300MB of data where the perf was almost 2x. If you run it on a 1GB tree it might be 4x

<comment><USER7> Sure, maybe. The problem is that doesn't seem enough to necessarily compensate for the added complexity.

The thing is a self-balancing binary tree is _so_ much faster than the alternatives for large chunks of data that you really have to use it. Here, I'm not so sure.

Anyway, thinking about this lead me to this discussion about fractal trees and LSM trees.

[https://www.quora.com/What-are-the-major-differences- between...](https://www.quora.com/What-are-the-major-differences-between-LSM- trees-used-in-BigTable-etc-and-Fractal-Trees-used-in-TokuDB?share=1)

Another question here is how will this play out if the high-performance DB is going to be moving to a GPU where cache levels and scans would be quite different.

<comment><USER8> _> Sure, maybe. The problem is that doesn't seem enough to necessarily compensate for the added complexity._

A 50% improvement could be massively significant if processing a huge data-set on thousands of nodes and it is still taking days/weeks/more of wall-clock time. Given the choice between waiting longer, buying more computer power, or using a more complex algorithm, the best of those choices will depend a lot on the circumstances of the project. If it is a one-off then waiting or paying for power will likely win out. Of course in some circumstances paying for more power won't help: you can only get so much out of each node and for some processes factors like interconnect throughput & latency might eat into the ROI significantly as the number of nodes grow.

In some cases a 10% or less boost might be worth the extra complexity. And if the algorithm can be generalised and packages as a library, you aren't actually having to deal with that complexity each time the method is useful.

</comment></comment></comment></comment><comment><USER9> Neat--I wrote a C++ implementation of a cache-oblivious (implicit) van Emde Boas binary search tree a [long] while back:

[https://github.com/lwu/veb- tree/blob/master/veboas.cpp](https://github.com/lwu/veb- tree/blob/master/veboas.cpp)

</comment><comment><USER10> Dumb question: is it appropriate to describe fractal trees as an "extreme" version of a log-structured merge tree? At a glance (i.e. if you squint really hard), the buffer flushing process in a fractal tree seems kind of similar to the tablet merging process in LSM trees in that they both batch operations to amortize secondary memory access costs, except fractal trees generalize it to all operations.

<comment><USER11> Yes, additionally fractal trees have a predictable and compact memory structure while LSMTs are more sparse. Average case for queries is also better for fractal structure and implementation is actually easier.

</comment></comment><comment><USER12> Really enjoyed this post, thanks.

I'm not sure if I really understood how the cache-oblivious solution works though in this case, I can't get an intuition for how the code linked in the github relates to the description in the article.

<comment><AUTHOR> OP here. It is admittedly a little hairy. The place to start is [https://github.com/rcoh/treelayout/blob/8adb9bb9727bdaee671d...](https://github.com/rcoh/treelayout/blob/8adb9bb9727bdaee671d86e73eb7fbffee82889d/treelayout.go#L71)

The rough idea is that we layout the top chunk, then the bottom chunks, and then wire them together

</comment></comment><comment><USER13> Do fractal tree indexes lend themselves to copy-on-write designs? (B-trees do.)

EDIT: Also, are fractal tree indexes roughly like COW b-trees?

</comment><comment><USER14> Are fractal tree indexes in the public domain, or are they patented?

<comment><USER15> They are patented but there is a patent grant in place.

[https://github.com/Tokutek/ft- index/blob/master/PATENTS](https://github.com/Tokutek/ft- index/blob/master/PATENTS)

Disclaimer: IANAL. Consult proper legal professionals etc.

<comment><USER16> One thing I've always wondered with patent grants in open source: it seems to say only that it's okay to use _this library_ , nor necessarily to implement the patented algorithm yourself.

Probably open sourcing and including a patent grant signal that the owner is benevolent and isn't likely to sue you for re-implementing their algorithm, but I don't think that's spelled out anywhere.

<comment><USER15> In the linked document they go out of the way to specify that the patent grant only applies to "THIS IMPLEMENTATION" of the algorithm. I'm not one hundred percent sure what that means in regards to reimplementing the algorithm.</comment></comment></comment></comment>