P values are not as reliable as many scientists assume - <AUTHOR> http://www.nature.com/news/scientific-method-statistical-errors-1.14700 <comment><USER2> This is probably the most annoying problem in my daily life (yeah I know, first-world problems). I have daily conversations with biologists where I've analysed some data and associated (say) a posterior probability with each condition in the model. They insist I give them p-values or something they can present as though they were p-values. At the beginning of my PhD I complied. Then they throw out all the nuance in the data, put one asterisk for p < 0.05, two asterisks for p < 0.01, etc., and (this is the horrifying part) _believe_ that an asterisk indicates that something is true. They put stupid asterisks all over my beautiful plots and then think their arbitrary cutoffs mean something biologically meaningful. I die a little inside every time I see an asterisk on a plot.

Now I refuse to use p-values and deliberately construct analyses that are incompatible with Fisherian statistics. And rather than giving people raw numbers, I produce a massive document of interpretation. Takes a huge amount of time, but I'm hoping it will mean my publishing track will contain significantly (ha!) fewer false results than most biologists'.

<comment><USER3> Noble, but it sounds risky. Don't you need false results to pad your CV to keep up?

<comment><USER4> He may not be long for this (scientific) world. As they say, _p_-ublish or perish.

<comment><USER2> Publishing is easy. It's just a bit easier for the people who are (perhaps unconsciously) cheating. I'm pretty confident my loudly-refuse-to-cheat strategy will prevent me perishing before my time.

</comment></comment></comment></comment><comment><USER5> I was recently trying to explain Bayesian logic to a friend, and came up with the following analogy. I would be interested to hear feedback on it.

\---

Imagine everyone in the USA gets sudden amnesia. We want to find out who the President is, but no one can remember.

A scientist comes up with a test to determine if someone is the President.

If they _are_ the President, there is a 100% chance the test will say they are the President and a 0% chance the test will say they are not the President.

If they are _not_ the President, there is a 99.999% chance the test will say that are not the President, and a 0.001% chance the test will falsely say they are the President.

Giving the test to the person sitting in the big chair in the Oval Office is useful, because it's already quite likely this person is the President. If the test is positive for Presidency, it's extremely likely that person is the president.

Giving the test to the 10 people nearest the oval office is useful, because it's fairly likely the President is one of these people. A positive result will indicate strongly that that person is the President, and if no-one in that group is actually the President, there's a 99.99% chance the test will say so.

Giving the test to the 1000 people in the White House is pretty useful, because it's pretty likely the President is in the White House, and if none of these people are the president, there's still a 99% chance the test will be correct. A positive result for any one person will indicate quite strongly that that person is the President.

But giving the test to everyone in America is not very useful at all, because it's very unlikely that any particular person is the President, and we can expect the test will give a positive result for around 3200 people. For any particular person in this group, it's much more likely they're not the President than they are.

\---

Is this a broadly correct, if non-rigorous, analogy? I realize most HNers will be much more familiar with this stuff than I am, I'm interested chiefly in whether or not I misled my friend.

<comment><USER6> Right. And the frequentist version is also useful.

If you test one person and the the test is positive, then that person is the president (p=0.00001).

If you test a thousand people, and the test is positive for one of them, then that person is the president (p=0.01).

So you don't really need Bayesian logic to reason that you should test fewer people if you want a more significant result. (Note I'm not saying you don't need Bayes' Theorem, which _everyone_ uses.)

Edit: I think most people on HN get their knowledge of frequentist and bayesian statistics from XKCD #1132. That's sad.

<comment><USER7> > So you don't really need Bayesian logic to reason that you should test fewer > people if you want a more significant result.

That's misleading by the use of the word "significant" which apparently means something in frequentism than it does in normal speech. I certainly wouldn't use "significant" in that way as a non-frequentist, I would instead rephrase what you said as:

> So you don't really need Bayesian logic to reason that you should test fewer > people if you want _more confirmation bias_ in your result.

And that's a statement I can definitely get behind!

<comment><USER8> It can definitely be used misleadingly, but it's not too out of line with normal scientific usage of the term. The "significant" in "significant figures" is the same: if a number has "11 significant figures", it doesn't mean the 11th digit is significant in the sense of being important or having a big impact, just that the 11th digit is within the measurement precision (as propagated through any subsequent calculations).

</comment></comment></comment></comment><comment><USER9> This is the reason why I don't include P values on [http://www.correlated.org](http://www.correlated.org).

They would muddle my otherwise irreproachable statistics.

<comment><USER6> I bet the p-values on that site would be very high if properly calculated. Generally, if you are reporting correlations between a large number of variables, the p-values shoot through the roof.

Of course, TONS of people forget this and publish a p-value as if those two variables are the only ones under consideration. Which is just sad.

<comment><USER10> With correlation statistics the p value is proportional to the sample size.

<comment><USER6> I think we're talking about different experiments. I'm not talking about, "here are the correlation statistics for 100 variables", I'm talking about, "I tested 100 variables, and these are the five pairs which are most strongly correlated."

<comment><USER10> At which point you need to do some variant on principal components analysis on the covariance matrix.

</comment></comment></comment></comment></comment><comment><USER11> The last sentence in this paragraph is hilarious:

P values have always had critics. In their almost nine decades of existence, they have been likened to mosquitoes (annoying and impossible to swat away), the emperor's new clothes (fraught with obvious problems that everyone ignores) and the tool of a "sterile intellectual rake" who ravishes science but leaves it with no progeny3. One researcher suggested rechristening the methodology "statistical hypothesis inference testing"3, presumably for the acronym it would yield.

<comment><USER12> As is:

"One researcher suggested rechristening the methodology "statistical hypothesis inference testing", presumably for the acronym it would yield."

<comment><USER13> I'm guessing you didn't read the last sentence of the paragraph in the post you were replying to.

<comment><USER12> Oops!

</comment></comment></comment></comment><comment><USER14> Statistics are descriptive, not predictive--period.

I'm continually surprised at how many people either don't know, or don't internalize, that. Look at how often "risk factors"\--which are a descriptive concept--are converted to advice--which is predictive.

Doing so in the absence of a causal hypothesis is a basic violation of "correlation does not equal causation."

If you want to construct a scientific theory you must be able to articulate some predictive tests, and that means you must hypothesize a causal mechanism.

<comment><USER15> As in "I used to think correlation implied causation. Then I took a statistics class. Now I don't..."

[http://xkcd.com/552/](http://xkcd.com/552/)

<comment><USER14> The reason that comic is funny is because there _is_ a known mechanism that we would expect to cause that effect: teaching.

<comment><USER16> Which is why the alt text on that particular comic is so key.

Correlation _suggests that if you 're looking for causation, it might be somewhere over here_. It doesn't _insist_ that the two are the same, but if you're looking for clues, it's a hell of a dowsing rod.

</comment></comment></comment></comment><comment><USER4> Further reading: [http://lesswrong.com/lw/g13/against_nhst/](http://lesswrong.com/lw/g13/against_nhst/)

</comment><comment><USER17> The root of this problem is that most data sets in psychology, anthropology, and epidemiology are not as large in terms of sample size as what computer scientists and electrical engineers encounter. p-values are a surrogate to explicitly describing the data using probability distributions or as random processes. In essence, you sacrifice granularity for simplicity. If you look at the original works of Fisher, etc. and their widespread utility, a large part of early statistics is intended for 'practical statisticians' who seldom encounter data-sets that are 'small' in terms of sample size. As someone who works in electrical engineering/computer science, I've never used the p-value because:

1\. The field, in general, demands far more mathematical rigor when dealing with statistics.

2\. The demand for mathematical rigor is justified because most data sets we deal with are many orders of magnitude larger than what psychologists and others encounter. So predictions based on limit theorems, etc. are often testable.

<comment><USER18> As an Epidemiologist, some of the most appalling uses of statistics I've ever seen have been in electrical engineering and CS.

<comment><USER17> Can you elaborate?

<comment><USER18> A little late in coming (blame the Southeast U.S.'s snowstorm), but I work in a very stats heavy field and interact with a number of CS types because I work on computational models.

I've gotten a fair amount of "I just need a p-value" requests, and some assumptions that everything can be hit with a t-test or an ANOVA and it'll all work out fine.

</comment></comment></comment></comment><comment><USER19> I'd love to see a comprehensive article that shows what a research paper's analysis would look like using Bayesian methods. I've seen plenty of general hints about Bayesian methods, discussion of priors, and similar, but I haven't found any specific guide on how to apply those methods to the types of research papers that would traditionally use a null hypothesis significance test with a p value.

<comment><USER20> Not articles but there are two very excellent books on the subject that I can't recommend enough:

If you read calculus with about the same fluency as the comic books then "Data Analysis: A Bayesian Tutorial" is awesome [http://www.amazon.com/Data- Analysis-A-Bayesian-Tutorial/dp/0...](http://www.amazon.com/Data-Analysis-A- Bayesian-Tutorial/dp/0198568320)

And if you would like a little more exposition (but still a mathematically sophisticated treatment) "Doing Bayesian Data Analysis: A Tutorial with R and BUGS" is fantastic [http://www.amazon.com/Doing-Bayesian-Data-Analysis- Tutorial/...](http://www.amazon.com/Doing-Bayesian-Data-Analysis- Tutorial/dp/0123814855/)

The latter will also give you more details of how to approach classical, frequentest tests and summary statistics with their Bayesian equivalent.

Honestly I would say get both books as they're cheap and provide different insights. You only need to read a few chapters of each to see how you approach basic experiments from a Bayesian perspective.

</comment></comment><comment><USER21> As the article reports, "Perhaps the worst fallacy is the kind of self- deception for which psychologist Uri Simonsohn of the University of Pennsylvania and his colleagues have popularized the term P-hacking; it is also known as data-dredging, snooping, fishing, significance-chasing and double-dipping. 'P-hacking,' says Simonsohn, 'is trying multiple things until you get the desired result' — even unconsciously."

Simonsohn, has a whole website about "p-hacking" and how to detect it.

[http://www.p-curve.com/](http://www.p-curve.com/)

He and his colleagues are concerned about making scientific papers more reliable. You can use the p-curve software on that site for your own investigations into p values found in published research.

Many of the interesting issues brought up by the comments on the article kindly submitted here become much more clear after reading Simonsohn's various articles

[http://opim.wharton.upenn.edu/~uws/](http://opim.wharton.upenn.edu/~uws/)

about p values and what they mean, and other aspects of interpreting published scientific research. He also has a paper

[http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2259879)

on evaluating replication results with more specific tips on that issue.

"Abstract: "When does a replication attempt fail? The most common standard is: when it obtains p>.05. I begin here by evaluating this standard in the context of three published replication attempts, involving investigations of the embodiment of morality, the endowment effect, and weather effects on life satisfaction, concluding the standard has unacceptable problems. I then describe similarly unacceptable problems associated with standards that rely on effect-size comparisons between original and replication results. Finally, I propose a new standard: Replication attempts fail when their results indicate that the effect, if it exists at all, is too small to have been detected by the original study. This new standard (1) circumvents the problems associated with existing standards, (2) arrives at intuitively compelling interpretations of existing replication results, and (3) suggests a simple sample size requirement for replication attempts: 2.5 times the original sample."

</comment><comment><USER22> "If your experiment needs statistics, you ought to have done a better experiment." \-- Ernest Rutherford

</comment><comment><USER23> I don't understand how did they get to the number 71% percent for 0.05 p-value?

0.05 p-value means that there is 5% probability that (for a t-test as example) a difference in averages of two sequences (the statistic) is by chance and not because of difference in means of their underlying normal distribution.

I assume that the "toss-up" means that there is no difference in the means in reality (so the null hypothesis is true). Am I understanding it correctly? Shouldn't in this case the probability of getting p-value < 0.05 be in fact less than 5% and not 29%?

How did they get the 29%?

<comment><USER24> No, you've got it wrong. A _p_ -value of .05 means that given a _true_ null- hypotheses, you have a 5% chance of getting this result anyway. That is, if there is _no difference_ in the population mean, than you still have a 5% chance of finding a sample that contains this difference anyway.

Do you see the difference. The _p_ -value doesn't tell you anything about the population, it only gives you information about your sample -- a confusion that this article was pinpointing.

So what the 29% is telling you is that, the chances of _finding_ a statistically significant result ( _p = 0.05_ ) given a population factor of _µ = 0_ are up to 29%. Whereas given a population factor of _µ = 0_ the chances of _getting_ a significant result ( _p = 0.05_ ) is 5%.

EDIT: erased a wrong and confusing claim.

<comment><USER23> Thanks, can you also explain why 29%?

EDIT: also, what is population factor µ? The difference in means?

</comment></comment></comment><comment><USER25> If you're interested in the subject, some additional reading (warning, PDF):

[http://www.deirdremccloskey.com/docs/jsm.pdf](http://www.deirdremccloskey.com/docs/jsm.pdf)

</comment><comment><USER26> Most of the objections here and in the article are not inherent problems with frequentist p-values.

First, the reported p-value might be wrong. E.g. basing it on assumptions of normality when the data is non-normal. However modern non-parametric approaches like the bootstrap can avoid this issue.

Second, testing multiple hypotheses. If you test 10 hypotheses then you cannot reject the null (that all 10 null hypotheses hold) simply because one single hypothesis is rejected in isolation. But this is well known, and failing to account for it is an issue with the researcher, not with frequentist statistics. I actually think that the main practical difference between Bayesian and Frequentist statistics is whether accounting for the issue of multiple hypotheses is done formally or informally.

<comment><USER27> The article doesn't bash the p-value as a statistical test specifically, more its use and interpretation by scientists over the years.

You're absolutely correct about using non-parametric tests, and more scientists should be using them. The normality assumption is flat out laughable when using real-world data most of the time.

You're also correct about multiple hypothesis testing. Accounting for familywise error (e.g., Holms adjustments) can help to keep your p-value reporting honest.

That doesn't negate the underlying problem, though. A p-value is simply an indication, nothing more. The p-value never promised to be more than that. The issue isn't in the p-value's construction, the issue lies in its misuse and how easily it can be abused in statistical reporting (see: p-hacking).

The p-value as a test statistic is perfectly honest in my opinion. But like many other statistical methods, it comes with its own set of baggage that I feel gets conveniently glossed over more often than it should.

</comment></comment><comment><USER28> I fully agree with the critics about the p-values, but what are the best alternatives to analyze and compare data? Most of the time, scientists have to compare the outcome of treatment 1 versus treatment 2; how should they do it "properly"?

What is the HN recommandation?

<comment><USER18> Effect measures. Don't just report your p-value, report the actual effect measure, and a measure of uncertainty around it, be it a frequentist confidence interval, Bayesian posterior distribution, etc.

More information is better.

<comment><USER29> Agreed. A good reference is Geoff Cumming's "Understanding the new statistics," although its focus on analysis in Excel may put off the typical HN audience.

Effect sizes and confidence intervals are much more useful than a p value or two.

<comment><USER30> Cumming had a short treatment of the subject published in Psych Science recently: [http://pss.sagepub.com/content/25/1/7.short](http://pss.sagepub.com/content/25/1/7.short) ; doi: 10.1177/0956797613504966.

</comment></comment></comment></comment><comment><USER31> I have always thought people rely on the normal distribution too much.

Does it work? Sometimes.

The problem is that people tend to believe something they use a lot.

Even the 0.05 threshold is sort of made up.

Correlation does not mean causation.

<comment><USER32> p-values need not arise from the normal distribution - but any distribution - selecting the right one is another cause for error when producing them.

the normal distribution is also quite well justified by the central limit theorem.

i do however agree that a p-value of 0.05 is not worth very much.

</comment></comment><comment><USER33> > In 2005, epidemiologist John Ioannidis of Stanford University in California > suggested that most published findings are false; since then, a string of > high-profile replication problems has forced scientists to rethink how they > evaluate results.

That's what is supposed to happen, though, right? You publish your findings. Others try to reproduce. They publish THEIR findings, etc. etc. If most published findings are false, it sounds like the process is working as designed.

<comment><USER34> Bad papers can be generated. published, and cited a lot faster than failures to replicate.

<comment><USER33> So, doesn't that mean we should be blaming idiotic news media outlets that tend to generalize and publish scoops on scientific papers too quickly?

<comment><USER29> Yes and no. Yes, the media sucks, but no, most scientific papers are not replicated -- or if they are, it's by a hapless grad student using the results in their research, only to find they don't hold up. The hapless grad student usually doesn't get to publish this, because negative results are boring and not usually published in prestigious journals.

Most scientists have better things to do than replicate previous findings, unless that previous finding directly bears on their own work.

</comment></comment></comment></comment><comment><USER35> Another reason to be skeptical of the statistics thrown around in popular news.

<comment><USER36> Unfortunately it's not just the nightly news one needs to be skeptical of, it's also (amongst many others) your doctor—who probably isn't particularly well educated w.r.t. statistics, but who will nonetheless give advice and prescribe treatments based on them.

<comment><USER37> This is a great -- and scary -- point. These statistical fallacies and misunderstandings are so deeply ingrained into our scientific and medical systems that it's hard to see how and when they will be removed. I can attest from personal experience that many scientists 1) don't understand these statistical tests, 2) don't care to find out, and 3) don't think there's a problem.

</comment></comment></comment><comment><USER38> This is why, to be on the safe side, in particle physics we have a requirement of a p-value of 0.0000003 for a discovery.

<comment><USER39> But isn't the whole point that no matter how low the P-value is, it is not a reliable measure?

</comment></comment><comment><USER40> i routinely get jobs from doctors at prestigious universities who say, "here's at study with 3 samples, see if we can get p < 0.05"

</comment><comment><USER32> a p-value of 0.05 or even 0.01 is stupidly high. it only takes a little thought experiment about what that means in reality to realise how permissive it is and you can find demonstrations of this without going particularly far, looking very hard or being especially well educated...

consider the wikipedia example with heads vs. tails.

[http://en.wikipedia.org/wiki/P-value#Examples](http://en.wikipedia.org/wiki/P-value#Examples)

the idea that 5 coin tosses can produce a p-value < 0.05 that 'demonstrates' that the coin is biased towards heads is intuitively 'obviously wrong'. even if we take it to 10 coin tosses (the p-value you get is 0.001 - which looks really strong if we accept that 0.01 is acceptable) it clashes with my own ideals for what statistical significance should mean. this is in a loose way a proof by contradiction that p-values of 0.05 or 0.01 do not have utility (at least for these kinds of small n).

aside from that consider running the experiment 5 times or 20 times. how many false positives do you expect? what is the expected number of false positives? is that significant?

it also bothers me how connected to the problem formulation that the value itself is. if we analyse the same situation with an identical test but a different formulation of the problem that the values differ?

why is five heads in a row less significant as a result when the test is whether a coin is biased at all rather than a test that it is biased towards heads only? sure i understand the probability involved there that we have all these potential coins biased towards tails that mean nothing in the first case - but there is something very deeply wrong with that.

shouldn't this be the other way around? if 5 consecutive heads is good evidence that a coin is biased towards heads, isn't it equally good evidence that it is biased at all? classical logic says that it is because being biased towards heads is a subset of being biased in either direction. the truth is that it really is equally good evidence - i challenge someone to explain why it is not! ( actually i kinda want to be wrong about that because i might learn something new then :) )

probability is counter-intuitive and useless for the kinds of small n usually used in experiments - the intuition about it recovers when we deal with sensible n - numbers like 1000 or 10000 - but these are still small n really if you need to scale up, or be confident that your result is correct. even at 100 samples its obvious that our idealisation of percentage and what happens in reality do not marry up neatly...

to make a very crude software analogy what about those 1 in 10,000 bugs? they are still a very real problem if you have millions of customers...

or - IMO even 10,000 is a very exceedingly small n to try and draw robust conclusions from.

<comment><USER41> 0.05 == 5% == 1/20\. If you flip a coin 5 times and get heads every time, do you intuitively feel that there is more than 1-in-20 odds that the coin is fair?

You should really get used to the idea that stating a different problem will give you a different answer. You need to be very careful when asking a question, or your answer might not mean what you think it means.

<comment><USER32> maybe my presentation was unclear. i seem to have given the impression that i am really quite dim...

> You should really get used to the idea that stating a different problem will > give you a different answer

this is entirely normal and expected since ever i can remember... what i'm saying is that you can analyse the same data in two different ways and reach differing conclusions because of the nature of the p-value (vs. the nature of, well... nature)

what i was mainly trying to get across is that a coin being biased towards heads /logically infers that/ it is biased. so the idea that 5 heads in a row is less evidence of a coin being biased than it is that it is biased towards heads is not only counter-intuitive but in disagreement with a much stronger and more intuitive form of reasoning.

the fact that the p-values are different in these cases leads me to expect that p-values on their own are not a good indicator of strength of evidence without a lot more context - and /really/ understanding what that context is and means - in which case why use the value at all? nobody else is likely to interpret it correctly unless you lay it out that way which then negates the supposed utility of the p-value...

and yes, i intuitively consider 5 heads in a row to be unspectacular for a fair coin certainly not a 19/20 chance that it is biased (maybe i am very, very wrong though).

<comment><USER41> Well, try it a few times and see if you can convince yourself :)

The evidence is different because there are different outcomes. For the two- tailed test the possible outcomes are: biased toward heads or tails, or not biased at all. For the one-tailed experiment, the outcomes are: biased toward heads, or not. Getting 5 tails in a row would be evidence in favor of the coin being biased, but not being biased toward heads.

Think of it this way: the two-tailed test is running 2 experiments at the same time (one for heads and one for tails) with the option of picking the one that gives you better results. So obviously the standard for significance has to be higher, because you're cherry-picking results. [https://xkcd.com/882/](https://xkcd.com/882/)

</comment></comment></comment></comment><comment><USER42> Many frequentists tend to Bayeslessly trust them.</comment>


The Benefits of a Network - Gina Bianchi - <AUTHOR> http://www.nytimes.com/2008/12/28/jobs/28boss.html?ref=technology <comment><USER2> I was part of a team that created a similar platform years ago. We knew, even before we launched, that it could be an enabler for evil as easily as it could be an enabler for good. Private social nets are as useful to neo-nazis as they are to parents of children who are terminally ill. And it wouldn't hurt if someone from Ning would acknowledge that instead of making Ningland sound like the land of milk and honey where sugar ferries dance on-top-of candy coated clouds.

When our service was finally shut down by the company that acquired us, our users we generating 20 million+ page views per day (back in the year 2000 no less) and I can tell you one thing for sure: most of those page views were not generated by church groups, little league clubs, fans of pop singers, or people wanting to change the world in a positive way.

The amount of spin going on around Ning these days has my head spinning!</comment>


Boustrophedon Order - <AUTHOR> http://en.wikipedia.org/wiki/Boustrophedon <comment><USER2> According to the Wikipedia image, the "backwards" row (from right to left) was not only written from right-to-left, but the characters were also inversed horizontally ([http://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Bou...](http://upload.wikimedia.org/wikipedia/commons/thumb/c/c8/Boustrophedon.svg/400px- Boustrophedon.svg.png))

I can understand the mentality behind alternating rows, especially on non- uniform surfaces but inversing the characters seems like more effort than what it's worth. Can anyone explain this further?

<comment><USER3> I recall that the Moai (<https://secure.wikimedia.org/wikipedia/en/wiki/Moai>) were transported horizontally quite a ways from their quarry to their pediments. >IF< the inscriptions are on these prone stones then perhaps it was so that two stone carvers could carve two lines simultaneously thereby halving the time to complete their work. By inverting the second line the second carver doesn't have to write upside down. The carvers just face each other while squatting on the stone's face and start at opposite sides of the face working past each other.

If this style of inscription occurs frequently on stone surfaces not likely to be transported then this theory is unlikely.

EDIT: answering the reply below - the above technique can be applied to any number of lines. As in pipeline processing for vector computers ([https://secure.wikimedia.org/wikipedia/en/wiki/Pipeline_%28s...](https://secure.wikimedia.org/wikipedia/en/wiki/Pipeline_%28software%29)) the last operations do not fully utilize the available capacity of the system. In the case of inscribing in stone - the last line only needs the labor of one carver.

As for frustrating - I'd find carving in stone quite tedious in any case.

I may still be mistaken but I don't think it's for the reasons you gave.

<comment><USER4> That's not practical at all. It could be applied only for an odd number of lines, if they even happened to know in advance how many lines it'd take. Also, it'd be amazingly frustrating: imagine having to "switch" your focus from the start of the text to the end, and then a little further from the start, and then a little further from the end...

I don't think our ancestors were masochists or dumb, really.

<comment><USER5> > It could be applied only for an odd number of lines, if they even happened > to know in advance how many lines it'd take.

I think when you are working in _stone_ , you have your text prepared in advance.

(But what do I know? Just the other day, I took a week and dashed down my grocery list on some granite. Turned out to be an even number of lines.)

<comment><USER4> Hey, I might be horribly wrong. It's not like I can put myself in their position.

It certainly isn't the first or last thing about ancient times that we come up with half-witted explanations for..

</comment></comment></comment></comment></comment><comment><USER6> Remember that word spacing was invented only in the 12th century. Ifyoutrytoreadseverallinesoftextwrittenlikethis, you'll quickly find that you lose track of where you are in the text fairly often. That's why Boustrophedon was invented and used, and that's why it became definitely obsolete after word spacing was invented.

You can read more about this in Paul Saenger's and Frances Yates' books.

</comment><comment><USER7> I'm not surprised that this method of writing didn't catch on, since it could be quite confusing. It would tend to minimize saccade distances when reading, but the down side is that you need to memorize 2N characters (normal and mirrored versions) plus you also need to memorize which is the current reading direction.

</comment><comment><USER8> maybe an effective way to parallelize writes?</comment>


Government contracting database lost track of $8 Billion - <AUTHOR> http://www.wired.com/dangerroom/2013/03/iraq-waste/

<comment><AUTHOR>
      >Government contracting databases didn't even have "an information management system that keeps track of everything built,"

This brings record-keeping to a new low.</comment>


Should an Amazon Echo Help Solve a Murder? - <AUTHOR> https://www.technologyreview.com/s/603278/should-an-amazon-echo-help-solve-a-murder/ <comment><USER2> This is a dupe. This story was posted like 20 times in the last few days. Here are the most popular discussions...

* Police seek Amazon Echo data in murder case | Hacker News || [https://news.ycombinator.com/item?id=13263894](https://news.ycombinator.com/item?id=13263894)

* Amazon refuses to let police access US murder suspect's Echo recordings | Hacker News || [https://news.ycombinator.com/item?id=13269930](https://news.ycombinator.com/item?id=13269930)</comment>


Motorola XOOM priced at $800 - <AUTHOR> http://www.engadget.com/2011/02/06/best-buy-ad-prices-motorola-xoom-at-800-affirms-february-24th/ It is surprising that motorola priced xoom at $800 when iPads are starting at $499. 3G iPad version starts at $630. What do you think? <comment><USER2> I really don't understand this at all. According to iSuppli[1], Apple pays $229.35 in parts and labor to make the low-end iPad. They develop their own operating system, buy incredible amounts of ads, and yet they still have massive profit margins selling iPads for $500.

Meanwhile, Motorola and Samsung and other manufacturers get a free operating system and get cellphone carriers to give them a bunch of money to sell the device with a data plan. Yet their devices are $600-$1000, they often have smaller screens, and are (probably) loaded up with crapware. I fail to see why anyone would ever buy something like this, and I fail to see why anyone thinks someone will buy one.

Even if no company has a supply chain as good as Apple's (which I have trouble believing, considering how many components are manufactured by Toshiba and Samsung, for instance) and they need to pay an additional 20% for parts, they could still make the same device, sell it for slightly less and still make plenty of money.

[1] [http://www.isuppli.com/Teardowns/News/Pages/Mid- RangeiPadtoG...](http://www.isuppli.com/Teardowns/News/Pages/Mid- RangeiPadtoGenerateMaximumProfitsforApple,iSuppliEstimates.aspx)

<comment><USER3> However, the Xoom has beefier (i.e. costlier) hardware than the iPad:

      * 1 GB vs 256 MB RAM
      * 1 GHz dual-core vs 1 GHz single-core processor
      * 10.1" 1280×800 vs 9.7" 1024x768 display
      * 2 cameras vs 0
      * and a freakin' barometer?!

But how much will these hardware improvements affect Joe Consumer's purchasing decision when he wants read Facebook and play Angry Birds?

<comment><USER4> "*and a freakin' barometer?!"

Wow, that makes it really interesting. The huge problem with GPS is height information. My GPS devices have commonly 10-15 m practical error in height on cities (urban canyon effect).

This could be very interesting for tunnel inertial navigation as makes the inertial problem a 2d issue from 3d.

<comment><USER5> "Warning, there's a storm coming. You should probably put your $800 tablet in a bag so it doesn't get wet."

</comment></comment></comment></comment><comment><USER6> I think the products that compete with iphones and ipads point to a fundamental shift in how Apple do things these days: Apple now compete on price.

I'm not saying they've given up on their high margins on everything policy. Like other commenters have said, they probably use purchasing power to make up the difference, but they are using price nonetheless.

We're so used to Apple being the premium option that we keep on being surprised that they aren't being beaten on price and then assume it's temporary. Well, it's several years into iphone and they aren't the macbooks off mobile phones yet. You actually need to pay for a non Apple alternative.

They still feel like the premium option though so it seems just silly to pay a $100 premium for the non-apple alternative, like paying a premium for an ipod knock-off.

It's quite a strategy. Build a company around products need to command a premium. Then remove the constraint. It's like having a runner run with stones in his pocket all his life then suddenly remove them.

<comment><USER7> I think what we're seeing is that Apple always _did_ compete on price. We just didn't know it until we saw competitors trying and failing to match their prices.

<comment><USER6> I'm not a hardware expert, but I don't think so. There are two ways to think about this:

a- What would it cost to build a more or less exact hardware replica?

b- What would it cost to build a decent substitute?

I don't even think they were competing on price in the 'a' sense.

But either way, 'b,' is, I think, the important one and Apple certainly weren't competing on price in that sense pre-iphone. You could always get a decent mp3 player or laptop roughly as good as but different to an Apple for a big discount. Still can. If you want to run Windows or Linux, you can do it without hassle on 1/2 - 3/4 of the budget. If you want a music player for jogging, an ipod nano knockoff will do the trick just fine.

If you want an android tablet or phone, you seem to need to pay more for (arguably) less.

<comment><USER8> You can get Tegra2 tablets running Android for half the price of the cheapest iPad for roughly the same techie mucking around effort as putting Linux on a netbook right now.

I personally wouldn't buy one till it was confirmed you'd be able to get Honeycomb on it, but they've been out there for a few months now, mostly held back by Google's lack of approval for shipping with the Android Market unless you're selling a phone. I believe that policy's about to change though.

</comment></comment></comment></comment><comment><USER9> ... and you have to purchase a data plan before you can use WiFi.

The iPad is still the only tablet in the history of personal computing that makes any sense.

<comment><USER10> At the low end $20/month plan, assuming a two year contract, that's $1280 total. Yikes.

The pricing psychology of the iPad starting at $500, then relatively small incremental updates that increase the size and give it 3G makes for a much easier sell.

The Xoom does look like nice hardware - I just wonder if they'll sell enough of them to reach economies of scale and drive the price down.

<comment><USER10> After further reading, it appears that you only need to buy one month of Verizon service (no 2-year contract), so the total is $819.

(Would have fixed the parent post but can't edit for some reason...)

<comment><USER11> That's better, although still a bit confusing. It's supposed to be a "Google experience" device meaning no manufacturer/carrier crapware or deliberate crippling, so it's not clear how they'd prevent you from using wifi if you never sign up for data.

</comment></comment></comment></comment><comment><USER12> This is the practical downside of being too cozy with carriers. A cheaper wifi-only model would be great for consumers but cuts the carriers out of the loop entirely. Probably the same reason there's no 'iPod Touch killer' out there yet. All these companies are in the business of selling equipment to carriers. What the consumer wants is a secondary consideration at best.

<comment><USER13> This is a huge frustration to me as a developer, I wish I could buy an iPod touch style android device.

<comment><USER14> Archos has an entire line, from 2" screens to 10" screens.

<comment><USER13> Thanks for the suggestion. The Archos 32 [http://www.archos.com/products/ta/archos_32it/index.html?cou...](http://www.archos.com/products/ta/archos_32it/index.html?country=us&lang=en) seems like the closest to an android phone in terms of size.

</comment></comment></comment></comment><comment><USER15> This seems pretty crazy, as the comparable iPad (16GB WiFi+3G) retails for $630. It will also be difficult to build up a large install base without a cheaper, wifi-only model. And while at the moment the Xoom has some features the iPad does not (cameras, for example) the inevitable release of a new iPad in a couple months will no doubt erase that advantage, at the same (or lower, per GB) price.

I can't imagine what Motorola/Verizon were thinking. They're not going to sell more than a few of these without lowering the price drastically.

Edit: Oops, I misread the ad as saying it had 16GB storage. The more reasonable price comparison is then the 32GB iPad, which is $730. More reasonable, but still too much (especially given that you must sign up for a very expensive data plan to even use WiFi).

<comment><USER16> Everybody is getting hung up on the price of this device, which I admit is high, but a comparison to the iPad's price is not fair. As stated below, the closest iPad is $730. Also the hardware is a generation and a half ahead of the iPad. The iPad doesn't even have a camera, very basic need for any kind of video conferencing (Xoom comes with 2). Please see the comparison chart [1] for more details.

But yeah, having to buy the Xoom from a carrier and paying atleast one month of data plan rate before cancelling it is kinda ridiculous.

[1] [http://static.skattertech.com/media/2011/02/infographic- play...](http://static.skattertech.com/media/2011/02/infographic-playbook-vs- streak-vs-ipad-vs-xoom-large.png)

<comment><USER17> You can't buy a $500 Xoom. It's just not possible. I think that matters a lot, I doubt many people will check specs.

Edit: To expand a little on this point, Apple sold 7.33 million iPads in the holiday quarter and made $4.4 billion in revenue which means that consumers paid about $600 (±$8) on average for an iPad. Considering that, I don't think it is a wise idea to not at least have a base model that is priced at $600 or less.

</comment></comment></comment><comment><USER14> Times have really changed when the knock-offs cost more than the real thing...

Someone just needs to figure out how to install Android or Chrome OS on an iPad, and it will be the tablet without any compromises.

</comment><comment><USER18> I'm a huge Android fan and while I would buy a laptop from Apple I wouldn't touch an iOS device due to disagreement over how the Market is handled.

However it doesn't take an Apple fanboi to know this pricing is horrid.

This isn't a laptop. People are not going to shop around on specs. $800 vs $500 for similar functionality is fscking retarded, especially when the $500 device is from the BMW of computers.

</comment><comment><USER19> This will bomb really badly. I am simply in awe at how far ahead of the competition Apple remains after all these years. I would have thought eventually the smartphone and especially the mp3 player market would've normalized by now but Apple is now telescoping their lead. By the time the iPad 2 ships every Android tablet not even on the market yet will be obsolete by an entire generation.

</comment><comment><USER20> I am especially bothered by the Verizon details here. First off, how can they prohibit WiFi use? Second, it seems like the Xoom is unavailable in Canada, if it really is Verizon-locked. Ouch and ouch.

</comment><comment><USER21> Not sure how they can compete with the iPad at that price range.

</comment><comment><USER22> I suppose now would be a bad time to mention Viewsonic's G-Tablet which has a Tegra 2 chip (just like the XOOM), a 10" display (not quite as nice as the XOOM's), and 16 gigs of RAM for a bit under $400? :-)

<comment><USER23> Have they announced honeycomb support for it yet? I haven't seen any news on it.

<comment><USER24> No, and they probably never will...officially The official software story on the G-Tablet is a total disaster that Viewsonic will likely never fix.

However the hardware is great and 3rd party developers are supporting the device with proper software. You can currently run CyanogenMod7 (which is based on Gingerbread/2.3) on the G-Tablet, though it is fully a community project that Viewsonic has nothing to do with. I'm sure the same will be true of Honeycomb soon after it hits full ASOP status.

If you're willing to hack around with the device and not rely on the official software, the G-Tablet is great at <$400. My only complaint is the viewing angles on the LCD aren't that great (the panel is not IPS), if not for that it would be a pretty ideal Android tablet platform for people willing to install community roms.

</comment></comment></comment><comment><USER25> Would someone care to explain why these things still cost double what an equivalent netbook with additional parts does?

<comment><USER4> Netbooks use way cheaper hard disks, instead of ssd. Netbooks don't have touchscreens. Netbooks don't have GPS, accelerometers, and gyros. Netbook don't have a battery that last 9 hours.

If there is a netbook with this specs,witch they are, it is going to be as expensive as tablets.

So, there is no "equivalent netbook" with half the price. Putting a pixel qi screen to my netbook cost me $275, a touchscreen $100 plus shipping cost, plus my time, plus the risk I break my netbook(I nearly broke it when bending one of the screen connection pins).

<comment><USER25> You certainly can get netbooks with 9h battery life. [http://www.compuplus.com/ASUS-Eee-PC-1015PED- MU17-1166505.ht...](http://www.compuplus.com/ASUS-Eee-PC-1015PED- MU17-1166505.html) for example.

GPS - when I can get a standalone satnav unit for $70 with its own processor, screen battery et al, that's hardly adding up to a huge extra cost. The accelerometer and gyro can be handled by a single relatively cheap chip from what I remember.

Netbooks also have keyboards, touchpads and extra interface parts such as USB and SVGA that tablets don't tend to have - not expensive components in themselves but extra parts to integrate into the design that aren't supplied with tablets. And, in a large number of cases, they come with a Windows license.

So, we're left with an SSD rather than HDD (retail, marginal cost about $70 for bigger drives than tablets use) and a touchscreen rather than a non- touchscreen (don't know the component suppliers for them to research it).

I still don't see why they're so much more expensive, beyond 'because we can' novelty value. The numbers simply don't stack up for me.

</comment></comment></comment><comment><USER26> The way Motorola has handled updates for the Droid and the high price will make the Xoom flop.

</comment><comment><USER27> i think a big question is how much is B&N losing on each nook color sale? 7 inch IPS panel with a decent processor.

10 inch tabets over $300 don't make any sense to me. archos 70 and 101 seem like the only sane products for the casual computing market that doesnt already have an ipad.

<comment><USER28> Engadget's impression of the Archos 70 and 101:

 _"incredibly chintzy feel [...] the 70 and 101 look and feel very cheap. Both are primarily made of black plastic [...] We just don't have much confidence in the longevity of the device [...] In hand, the materials feel far from solid, and even after wrapping it in a scarf, we worried about keeping it safe from bumps and bruises in our bag. [...] As you'd expect, the cheap make isn't limited to the surrounding parts of the tablet -- the plastic displays on the 70 and 101 suffer from the same issues. We weren't expecting Gorilla Glass quality here, but both of them are flimsy and when pressed firmly, they start to cave in quite a bit. [...] vertical viewing angles are incredibly bad. Tilting the screens slightly off axis to about 30-degrees caused a ton of color distortion, and if you happen to be standing above the displays while they are propped up on the kickstands, it's nearly impossible to see what is on screen. [...] The VGA cameras on both tablets are incredibly sluggish, and when we finally were able to snap some stills, they came out blurry and grainy. Similarly, when we shot some video, the footage was very choppy and the audio wasn't in sync with the video at all."_

[http://www.engadget.com/2010/12/24/archos-70-and-101-interne...](http://www.engadget.com/2010/12/24/archos-70-and-101-internet- tablet-review/)

</comment></comment><comment><USER29> And with this, all interests I had on the XOOM are gone.

</comment><comment><USER30> No thank you, drop the price $500, then we're talking.

</comment><comment><USER31> I am sorry Moto, this would not fly. Too expensive. I am just pissed to think I have told everyone around to wait for it. The next iPad will have all this features at a far better price. I have to give it to Steve Jobs. They certainly had a better iPad but decided to release something slightly basic first at an awesome price point, make loads of profit then have the ability to match the specs of the new comers who will think they can better the iPad1 specswise

</comment><comment><USER32> $800?

And if you read the subtext of the picture carefully, it says, "To activate WiFi functionality on this device, a minimum of 1 month data subscription is required."

My god, is it rude to just point and laugh?

<comment><USER33> > "To activate WiFi functionality on this device, a minimum of 1 month data > subscription is required."

I'm gobsmacked by that - is there any precedent for it? Anywhere? I've literally never heard of any manufacturer locking Wifi support. I'm almost not believing it yet until I see some kind of other confirmation - I can only think that it was meant to say "Wireless" or "3G" or something else, because locking Wifi is just ridiculous.</comment></comment>


Innovative technology for speed reading on wearable tech screens - <AUTHOR> http://www.businessinsider.com/spritz-speed-reading-gifs-2014-2

<comment><AUTHOR> Their website also has a cool demo: [http://www.spritzinc.com/](http://www.spritzinc.com/)

And one of their promotional photos implies that it would work on google glass as well.</comment>


Ask HN: What do you use for making webservices in Java? - <AUTHOR> <comment><USER2> At schezzle ([https://schezzle.com](https://schezzle.com)) we use Spring. If you don't want to go the whole DI IoC path, just use Jersey (JAX-RS) directly. Dropwizard isn't a bad implementation either, but can be a bit verbose. Check out swagger for docs as well.

</comment><comment><USER3> Spring (normal or boot) MVC, mostly.

Or Jax-RS/Jax-WS with Jersey/CXF/whatever, if you want to stick to standard EE technologies.

95% of our stuff is Java/Spring (big SOA, banking sector). ESB/Gateway is normally Mule or Camel.

<comment><USER4> do you have experience with other stacks (e.g. Play, Node, Ruby)? What's your experience with them?

<comment><USER3> We used Python with Django (and django-rest-framework) in my old (non- financial) shop, and it was fine.

</comment></comment></comment><comment><USER5> We're moving away from the Play! Framework to Spring Boot as we were porting and using more and more Spring based libraries in our codebase.

Below is pretty much a default list of dependencies for every web service.

* Spring boot

* Lombok

* Javaslang

* Feign

* WireMock

* RestAssured

* AssertJ/Junit/Mockito

</comment><comment><USER6> Spring Boot is the way everything is going for us these days.

<comment><USER3> +1 for spring boot.

I dont use the rest repositories/hateos stuff much.

But mvc/data/security, are solid.

The netflix cloud stuff is really nice too, for circuit breaking/routing/discovery.

</comment></comment><comment><USER7> Spring Boot.</comment>


My no-iPhone no-smartphone experiment - <AUTHOR> http://projectgus.com/2010/09/my-no-iphone-no-smartphone-experiment/

<comment><USER2> I love my seven-year-old LG flip phone. No camera, no MP3s; it's a phone. (I admit, I do use the alarm clock feature, too.) I've heard from lots of people that iPhones and their brethren are pretty decent as tiny computers, but mediocre phones; and with the tablet/pad form factor coming into its own I think the "smart phone" will soon feel more constraining: now it's a mediocre phone _and_ a mediocre tablet.</comment>


Why It Took So Long to Invent the Wheel (2012) - <AUTHOR> http://www.scientificamerican.com/article/why-it-took-so-long-to-inv/ <comment><USER2> One of the most fascinating books I ever picked up from the library was _Ancient Inventions_ ([https://www.amazon.com/Ancient-Inventions-Peter- James/dp/034...](https://www.amazon.com/Ancient-Inventions-Peter- James/dp/0345401026)), which goes over the details of stuff like this.

The historical figure that strikes me as "most likely to be a time traveller" is Archimedes, who designed incredibly advanced machines that destroyed the Roman fleet and protected his home city. Sadly he was accidentally killed during the war. The Romans wanted to capture him and use him for their own purposes. It's possible that could have changed history quite a bit.

Heron of Alexandria is another candidate that gets a lot of space in that book. He invented the first steam engine _way_ before his time. But also many other insanely elaborate mechanical devices. Including an entirely mechanical little theater, with mechanical puppets and music.

Unfortunately almost all his inventions were for entertainment value only. Sometimes I wonder if the greatest inventor of our time might be applying his work to video game engines.

<comment><USER3> > Unfortunately almost all his inventions were for entertainment value only.

I read somewhere that one of the problems in antiquity to develop useful machines was the abundance of slave/captive labor. Why do you want to create a complicated inefficient first version of a machine when you have cheap humans being to do the task?

<comment><USER2> Yes that's probably why the industrial revolution didn't happen. There's a good comment below that explains other societal reasons why the industrial revolution didn't happen, like the lack of contract law and investors.

I also read a great theory once (but I can't find it again through google!) that the industrial revolution really required the invention of the cannon first. Building an _efficient_ steam engine in roman times would have been incredibly expensive and dangerous, because of their primitive metal technology. The cannon created a tech race between nations to develop more/better/cheaper/lighter cannons. Which resulted in better metal working tech. Once you can make big pieces of metal that can withstand and direct the pressure created from explosions, steam engines aren't too far away.

But Heron's invention was still useful, and would have been the fastest moving thing ever built up to that time. Perhaps it could have been adapted to military applications, like flinging tons of rocks really fast at high speed?

<comment><USER4> I've heard it said that the bicycle basically needed the car, despite seeming to be much simpler: accurate mass production of ball bearings, chain drives, and pneumatic tires were all driven by car manufacturing but needed for really practical bicycles.

<comment><USER5> But bicycles are useful on their own (possibly more so in a world where cars do not exist), so the existence of bicycles alone could easily have driven innovation.

In fact, Dunlop invented the pneumatic tire for his son's tricycle and they were initially used for bicycles ([https://en.m.wikipedia.org/wiki/John_Boyd_Dunlop#Pneumatic_t...](https://en.m.wikipedia.org/wiki/John_Boyd_Dunlop#Pneumatic_tyres))

<comment><USER6> Bicycles are _hugely_ useful -- they were the first mode of transport that was affordable to an ordinary person (though relatively about as expensive as a car today), and _didn 't_ require feeding, shoding, tackle, or stabling. With a bike, a person could move beyond 3 miles in an hour, or get to and from a railroad station allowing for much longer commutes, visits, or recreational travel.

As of the late 19th century, horses consumed about 25% of the grain production of the US. Much of that was to provide drayage of people and goods to and from railroad stations or train yards.

<comment><USER7> The first mode of transport affordable to an ordinary person was probably an animal like a donkey or an ox, etc..

<comment><USER6> Not within an urban context, and not much elsewhere. A donkey or ox required grazing access -- which meant either owning your own land or having access to a commons, as well as a stable. Bicycles, despite many other issues, don't poop.

A horse was a major capital investment. And a donkey can't carry all that much weight -- a light adult, perhaps, but only just.

The US horse population in 1915 was 20 million, contrasting with about 100 million people. Keep in mind that most of those horses were in commercial use -- drayage, farm traction (a large combine would be drawn by 40 horses), etc.

Numbers for England/UK are harder to come by, but a Google Answers thread gives 3.3 million for late Victorian Britain, and 14 million in Europe in 1800, the latter against a human population of 203 million.

The typical person, or family, owned zero horses.

[http://answers.google.com/answers/threadview?id=144565](http://answers.google.com/answers/threadview?id=144565)

[http://www.geohive.com/earth/his_history1.aspx](http://www.geohive.com/earth/his_history1.aspx)

Though you raise an interesting question: what was ownership of draught animals throughout time, and how did that compare with bicycle sales in, say, the US, England, France, and Germany from 1880 - 1920 or so. Keeping in mind that mass private automobile ownership wasn't really a thing until the 1950s, even in the US.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> I often like to fantasize how would I fare if I found myself suddenly transported into medieval times or earlier (spoiler: poorly).

Articles like this make me appreciate even more how little difference there is between us and humans thousands of years ago. I wouldn't have figured out how to make a working wheel, and I've _seen_ one. I have no idea where I picked up the idea that "ancient civilizations were dumb", but the more I read, the more I appreciate how little difference there was.

BTW, if you haven't, you should definitely check the "Primitive technology" channel on Youtube.

<comment><USER9> The "what would I try to invent?" question one is fascinating. If I found myself transported back to Roman times, I'd aim to make a lens. The Romans had glass so the raw material is there. With a lens you can fairly quickly go a telescope, which has immediate military application, so would perhaps be widely adopted. Astronomy might then follow, heading off the whole "Earth is the centre of Universe" thing that held back modern thinking. You can also make a microscope, which fairly quickly leads to the discovery of microscopic organisms, and perhaps the germ theory of disease.

For bonus points I'd give them the decimal number system.

<comment><USER10> The problem is with the assumption behind "which fairly quickly leads to the discovery of microscopic organisms". In your scenario, who learns about microscopic organisms, and when?

As a point of comparison, who discovered the existence of North America? Was it Christopher Columbus in 1492? Or was it the Vikings in 900 AD? Or was it some mix of people who first melded together in Beringia, somewhere around 20,000 BC?

Has a fact been discovered if one person learns of it? Or is it only discovered if the news spreads to all of society? If the latter, then science is a social process, not an individual one.

The historian Fernand Braudel points out that an Italian engineer, wanting to show off his skills, built an automated saw mill in Italy, back during the 1200s. The logs were slotted automatically and run through the mill automatically. This engineer had created an important part of the Industrial Revolution, 500 years early!!! Did it matter? No. The project was merely a demonstration. The abundance of wood was never used. How could it be? There was no system of transportation for the abundance of wood, there was no finance to cover operating expenses, nor any finance to support a building boom that might use the wood, nor any modern labor market from which the mill owner could hire more workers to do the initial cutting of trees, nor was there the modern strictness regarding contracts, nor were commoners allowed to bring civil suits against the nobility (in case of default, supposing a nobleman contracted to have a manor built), nor was the judiciary independent, nor... In other words, the Industrial Revolution was a social process, involving the creation of dozens of important modern institutions, and the public's acceptance of the legitimacy of those institutions. No single engineer, however brilliant, could jump start the process.

<comment><USER11> That's impressive, but the invention of sawmills didn't need to wait until the industrial revolution. I've just come back from The Netherlands, which had massive uses in wind powered technology in the 1500's, including windmill powered sawmills that led to the Dutch golden age because they could build boats faster:

[https://en.wikipedia.org/wiki/Cornelis_Corneliszoon_van_Uitg...](https://en.wikipedia.org/wiki/Cornelis_Corneliszoon_van_Uitgeest)

<comment><USER10> You are referencing an event more than 300 years later than the event I am speaking of. But yes, to your point, much of what was necessary to the Industrial Revolution was pioneered by the Dutch during the 1600s: contracts, fair and impartial courts, the end of legal immunity for the nobility (regarding civil contracts), modern finance, modern insurance, advanced transport, modern storage facilities, etc. This is also covered by the historian Fernand Braudel, who is worth reading.

</comment></comment></comment></comment></comment><comment><USER12> If you think INVENTING the wheel took a long time, wait till you learn how long it took to put them on suitcases: [http://www.worldatlas.com/articles/did-you-know-that-we- went...](http://www.worldatlas.com/articles/did-you-know-that-we-went-to-the- moon-before-we-put-wheels-on-suitcases.html) (spoiler alert: we got to the moon first!)

<comment><USER13> That's more a mass production thing isn't it - we had trolleys of various forms you could load your bags on to.

Strong plastics, probably made such mass production cheap enough.

People rich enough to have wheeled luggage in the past would have just used servants, it's seemingly economics rather than technology that prevented such "inventions" as attaching wheels to things permanently instead of temporarily.

A modicum of googling suggests Templar Knights had wheeled trunks in the 12th Century though.

<comment><USER14> > we had trolleys of various forms you could load your bags on to

and a man or boy you paid to push the trolley. Much more civilised.

</comment></comment></comment><comment><USER15> The wheel is one part of a two part invention:

1) The wheel

2) smooth hard-ish surfaces that are large enough to be worth going anywhere

Without either, the wheel on its own isn't worth much. It turns out roads are a much older, more useful invention without which the wheel doesn't make much sense.

<comment><USER16> Settlers beat trails across the American west. Granted, much of it is hard and flat, but it didn't take a lot of improvement to get that way.

There's also still lots of relatively major roads in undeveloped countries that are not smooth or hardish. [http://www.flickriver.com/photos/globalizepeace/3832727846/](http://www.flickriver.com/photos/globalizepeace/3832727846/)

</comment></comment><comment><USER17> I heard the story differently. Wheels are only really useful if you have good enough streets to use them on and building streets is a huge upfront investment. It is also a chicken egg problem - wheels are only useful with streets and streets are only useful for wheels. Without streets you are better of carrying your stuff around than trying to push it around on wheels in natural terrain. That is also why the wheel first appeared in toys, because you can push them around on a flat table. So the wheel was probably well known before it saw widespread adoption for transportation which additionally to the lack of streets had to overcome the engineering problems mentioned in the article that come up when you scale the wheel from a toy to a wheel burrow.

<comment><USER18> Streets? Where we came from, we needed no.. streets ;)

I mean, didn't "streets" emerge really from commonly-travelled paths carved by feet and horse hooves and time, and really most non-paved paths are quite fine for most wheels? Perhaps they last longer on pavement, but other than that..

<comment><USER17> I didn't want to imply pavement when I said streets. The question is what commonly traveled paths look like before the wheel. If they were just desire paths, then they were not suitable for carts because they are to narrow even if they were used with horses or other animals instead of just walking. Could they already have been wide enough for example because of several people walking next to each other? Certainly, but were they? I personally can not think any path I have ever used that was wide enough for carts but never intended or actually used by any sort of vehicle.

<comment><USER19> Packs of large animals carve impressively road-like paths between their points of interest in the scrub.

</comment></comment></comment></comment><comment><USER20> Wheels were certainly in use well before 3500BC.

[http://mentalfloss.com/article/62357/who-actually- invented-w...](http://mentalfloss.com/article/62357/who-actually-invented- wheel)

(A casual Google check will reveal other examples as well)

Also parent article mentions metal tools being needed to bore holes for the shaft. Other methods could do this also... people worked intricately with wood long before metal was used (although metal would certainly speed up mass production.)

As far as roads, you don't really need roads for a wheel to be useful. You just need a flat semi hard surface. People crossed the US in horse drawn wagons without roads. Not sure where the "roads" argument comes from other than, sure, better roads means more wheeled vehicles.

<comment><USER21> The oldest artifact mentioned in the article at "5650 to 5385 years ago" is right around 3500 BC.

</comment></comment><comment><USER22> _The invention of the wheel was so challenging that it probably happened only once, in one place_

i disagree with this premise based on kevin kelly's simultaneous discovery phenomenon.

 _The fact that pretty well every invention and discovery you mention has occurred to two different people at possibly the same time. Possibly three. Possibly four. My favorite example of this is the light bulb, which in my part of the world a man named Joseph Swan gets the credit for inventing the light bulb and a terrible fraud called Thomas Edison came along and ripped him off. Well, if I live in Russia, I give the credit to Lodygin, and I 'm equally cross with Edison. But actually if you drill down into history, in the 18th century there are 23 people in that decade alone who deserve independent credit for coming up with the idea of the incandescent light bulb. It was an idea ripe to be discovered. It was inevitable that it would be discovered in that decade. And that's true of almost everything. And then of course famously evolution itself, the idea of natural selection occurs to Wallace and Darwin at the same time, and Darwin has to rush into print to prevent himself being preempted. Even relativity--we tend to think of Einstein as unique in coming up with this idea that nobody else, that it took the world by surprise and nobody believed him. Well, that's true, but if you look at what Hendrik Lorentz was doing at the same time, he was well on the trail. He would have got there if Einstein had been run over by a tram. The double helix of DNA (deoxyribonucleic acid), incredibly important discovery--big race going on to find it. That, the technology had reached the point where we're going to find the genetic code around that time. So, there's a sort of complete dispensability of scientists and inventors that really surprises people when you think about it._

[0] [http://www.econtalk.org/archives/2016/02/matt_ridley_on_1.ht...](http://www.econtalk.org/archives/2016/02/matt_ridley_on_1.html)

</comment><comment><USER23> The article surprised me in that it didn't mentioned the potter's wheel even though the principles are mostly the same.

<comment><USER24> If I remember correctly, the pottery wheel came later.

</comment></comment><comment><USER25> Is there any realistically conceivable alternate-history/universe where cable- suspended wheel-less trams (like [1][2][3] but closer to ground level) became the dominant method of transportation instead of huge roads and cars?

It's something I've been musing on to incorporate into some fiction, inspired by all the recent news and discussions about self-driving cars and the prospect of shared fleets.

[1] [https://en.wikipedia.org/wiki/Cable_transport](https://en.wikipedia.org/wiki/Cable_transport)

[2] [https://en.wikipedia.org/wiki/Aerial_tramway](https://en.wikipedia.org/wiki/Aerial_tramway)

[3] [https://en.wikipedia.org/wiki/Gondola_lift](https://en.wikipedia.org/wiki/Gondola_lift)

<comment><USER26> Try [http://worldbuilding.stackexchange.com/](http://worldbuilding.stackexchange.com/)

</comment></comment><comment><USER27> > _However, from that place, it seems to have spread so rapidly across Eurasia > and the Middle East that experts cannot say for sure where it originated._

Well the very nature of the wheel itself probably helped with that - I wonder if that holds true for all transportation-related inventions?

<comment><USER6> Lateen sails took a particularly long time to spread. They had been introduced to the Mediterranean region as early as the 2nd century BC (and had been developed much earlier in China and Arabia), but were not brought into widespread use in European sailing ships until the 1400s. This is _hugely_ significant as a lateen-rigged ship can sail far closer to the wind than a square-rigged ship, giving it much better maneuverability, allowing it to navigate complex waters (e.g., harbour entrances) under less-than-ideal conditions, and increased the length of the sailing season -- November through May was "no-sail" season from Roman times (with numerous interesting impacts on historical events, messages, etc.).

The rig seems to be among the technologies lost during the middle ages.

The big hitch, as noted in James Burke's _Connections_ , was how ships and sailing voyages were financed. A ship was a sufficiently sizeable investment, and financing sufficiently difficult to arrange, that conservative financiers opposed any radical explorations of such tomfoolery as them thar lateen-rigged newfangled boats.

As for other transportation innovations ... there mostly _weren 't_ any until quite modern times.

Once introduced, though, their advantages were immediately obvious, and they were adopted quickly throughout Europe.

[https://en.m.wikipedia.org/wiki/Lateen](https://en.m.wikipedia.org/wiki/Lateen)

Yes, you saw further advances in ships and navigation. _Especially_ navigation -- position-taking and timekeeping were critical, and several innovations followed catastrophic losses of entire fleets run aground and shipwrecked _into the 20th century_ \-- see the Honda Point disaster in which _seven_ US destroyers sailed at full speed into rocks just offshore the Santa Barbara coast in 1923.

[https://en.m.wikipedia.org/wiki/Honda_Point_disaster](https://en.m.wikipedia.org/wiki/Honda_Point_disaster)

After the Lateen rig and excepting other ship designs, the major transport innovations were canals (major construction from the 1500s onward in Europe, though much earlier in China, and some earlier European/Mediterranean instances), the railroad (1825), steamship (1810s), safety bicycle (1870s), automobile (and truck and bus -- 1880s), and airplane (1900s). These all spread essentially instantly throughout the world wherever industrial capacity (and fuel) capable of producing such vehicles existed.

<comment><USER28> And yet European naval architects continued using square rigging for most large vessels until the end of the age of sail even when they were fully aware of lateen rigging. So clearly there were advantages to square rigging beyond just conservative finance.

<comment><USER6> The vessels made use of square-rigged sails, but _also_ carried lateen, gaff, or fore-and-aft rigging, allowing for better handling.

Just to give a sense for the significance of this (and later steam): ships could spend weeks, or even months, awaiting favourable winds to enter harbour. Improved sailing capabilities were directly beneficial to all aspects of the shipping business.

[https://en.m.wikipedia.org/wiki/Sail- plan#/media/File%3AShip...](https://en.m.wikipedia.org/wiki/Sail- plan#/media/File%3AShip_Rigging_differences_in_schematic_view.png)

[https://en.m.wikipedia.org/wiki/Sail- plan#](https://en.m.wikipedia.org/wiki/Sail-plan#)

</comment></comment></comment></comment><comment><USER29> It could also be related to the fact that in order for wheels to become widespread you need to have terrain that they can roll on. The only obstacle to the 'need something to roll on' hypothesis is wheelbarrows which in theory should have appeared long before.

</comment><comment><USER30> Also, the wheel is not very helpful without compact, step-free pathes.

<comment><USER13> Water wheel, spinning wheel, pottery wheel, grinding wheel (eg for wheat flour), ... seems useful.

<comment><USER6> The domain under discussion is transportation, not use of the wheel in industrial applications.

Spinning, water, and grinding wheels all postdate the wheel-and-axel in transportation. It _is_ thought that the transport wheel may have evolved from the potter's wheel.

</comment></comment></comment><comment><USER31> Because middle managers in meetings were accusing innovators of reinventing the cube.</comment>


404 Not Found - <AUTHOR> http://null.kiwi.net/foo/404.php <comment><AUTHOR> Interestingly, this got several upvotes very quickly (thank you) and then within a few minutes plummeted down the rankings to the mid-500s. Clearly a few people think it doesn't belong on HN and flagged it. To you I also give my thanks, for reminding me I have much to learn about the HN "community," and that I really don't understand what you will find interesting, fun, useful or engaging.

<comment><USER2> I nearly flagged it based on the title alone (until I saw who'd submitted it). It screams "someone tried to post a link to HN but pasted the url in wrong".

<comment><AUTHOR> Thank you - I've submitted it again (with an explanation) in case you're right: <http://news.ycombinator.com/item?id=4123010>

</comment></comment></comment><comment><USER3> I wish I understood myself well enough to know _why_ I had to read it through until the end.

<comment><USER4> I read the first 8 or so, then clicked "view source"....

<comment><USER3> I read it on my iPad. No obvious view-source option.

</comment></comment></comment><comment><USER5> This is like the Marvin of web servers. I bet a 500 error really ruins his day...</comment>


Bankruptcy judge turns back FirstEnergy reorganization plan - <AUTHOR> https://www.utilitydive.com/news/bankruptcy-judge-turns-back-firstenergy-reorganization-plan/552137/ <comment><AUTHOR> Notable due to the judge preventing FirstEnergy from shedding the costs to decommission the firm's coal and nuclear generation assets as part of BK proceedings.</comment>


Why the Web Is Useless in Developing Countries - And How to Fix It - <AUTHOR> http://mashable.com/2011/02/04/web-developing-world/ <comment><USER2> Are Google/Groupon/Facebook acting in this space? Wouldn't it be to their advantage to improve access to the web and bring in more users? Or is there still no ad money to be made in developing countries?</comment>


Show HN: Structured drawing app – my first indie iOS app is on sale this week - <AUTHOR> http://tanglelibrary.com/

<comment><USER2> Bought this app. Looks fun, but took me a while to figure out that it wasn't actually a drawing program but a template library to use as a guide when drawing with paper and pencil.

I happen to like drawing with paper and pencil, so I'll try it out later.

<comment><AUTHOR> Thanks for checking it out - I'll work on making the description clearer.

If you have more feedback after playing with it you can email me directly from the address on tanglelibrary.com/support</comment></comment>


Ask HN: Engineering roles in sales orgs? - <AUTHOR> For the past few years the company I work for has been trying to recruit engineers to join their sales org. We started by recruiting for a very general engineering position and have recently tried to be more specific in the roles we're looking for, i.e Solution Architect.
I've observed that engineers do not stay in these roles for very long. Some complain that their technical skills are not kept sharp, others that the work that they do day to day is not satisfying. Some feel like their not seen as 'real engineers' by the engineering department -- for many of them it's easier to get an engineering role at another company rather than transfer internally. There is a general view that very little career development is possible.
We've cycled through a couple of different managers (all non technical) and have tried to define what career advancement looks like but haven't hit on something that aligns the engineers belief that they can add the most value by building tools with the metrics of the sales org.
Are there any of companies that have successfully staffed engineering positions in sales (or more generally, non-engineering) orgs?
If your company does this, how is it structured and what processes do you have in place to keep engineers happy? <comment><USER2> At my org we call them "Opportunity Engineers". We do this to position them mentally that their job is to create new opportunities with every client. This means that they have to use whatever technology they want to solve the clients requests as simple as possible. They are still umbrella'd under the Engineering department and attend the same standups and meetings as all the other engineers. They work closely with our engineering leads to make sure their solutions won't cause a disconnect in our grander technology stack. Often times the solutions they prototype for a client end up becoming a longer term solution that we try and market to others. So when that happens they are brought back into the Engineering team and lead the effort in building that solution into a full product.

We also make sure it is as seamless as possible if someone wants to transfer internally. Our Engineering department is relatively flat, in that Team Leads are more of Project Leads and will shift from engineer to lead from project to project. And engineers are encouraged to join whatever projects they find fascinating</comment>


Mac OS gets multi-tasking - <AUTHOR> http://www.folklore.org/StoryView.py?project=Macintosh&story=Switcher.txt

<comment><USER2> This story is especially notable for the little tidbit about both Bill Gates and Steve Jobs. I'm always impressed by their business acumen.

<comment><USER3> Also notable as an early anecdote on the development of user interface/experience. It's funny how complicated simplifying the user experience can be from the engineering perspective.

Another interesting tidbit was Microsoft's habit of storing bytecode (to be interpreted) making the developer's life much, much harder. Hmmm, sounds like curiously similar to more recent news...

</comment></comment><comment><USER4> I don't think it's really fair to call this "multi-tasking". Curtainly a predecessor, though. A more accurate description would be "task switching", whereby each application had reserved memory, but only one application ran at a time. At the user's request, another application would take over and run. There was no CPU time-slicing. That came later, when Apple introduced the Multifinder extension in System 7.

</comment><comment><USER5> Andy rocks.

</comment><comment><USER6> awesome story. thanks for posting it!

</comment><comment><USER7> Can I be the first to say, "About time!"

</comment><comment><USER8> Hmmm ... wonder if the Memory Shift guys ever sued Apple for IP/copyright infringement ... fast forward 25 years, you know Steve definitely would ...</comment>


Nedtries: an ordered container faster than hash tables and red-black trees - <AUTHOR> http://www.nedprod.com/programs/portable/nedtries/ <comment><USER2> This is basically a radix/patricia/critbit tree. Those might be O(1) for the number of keys but are O(n) for the length of the key. Note this is true for hashtables as well.

Such trees are pretty excellent in practice; they give hash-like fast lookup while preserving ordering. Hashtables do have the advantage of being less prone to cache thrashing as you traverse the tree, which is probably why nedtries is best for <10,000 elements. Not to mention the key distribution may affect tree balance and hence lookup times. Probably an issue when you're allocating memory from sequential addresses.

I have a unoptimized and under-tested implementation of such a binary trie, which uses string keys. <https://github.com/j0sh/radixtree>

edit: another big plus to these types of trees: they use a minimum of space, unlike a hash table.

<comment><USER3> There has actually been work on cache-conscious tries, such as the HAT-trie.

<http://crpit.com/confpapers/CRPITV62Askitis.pdf>

<comment><USER2> Those are way complicated, though. I read that HAT-trie paper some time ago and I still don't understand it 100%. Compare this to djb's crit-bit description which fits nicely in a paragraph and tells you most everything you need to know.

I suppose it's one of those 80-20 issues: compressed binary tries only have a few drawbacks, but addressing those requires a ton of effort.

<comment><USER3> FWIW, I don't think nedtrie has path-compression, and it is only level- compressed in that it has an array at the top that indexes based on leading zero count (which is likely to lead to a very unbalanced root node in comparison to just doing level compression up there).

I pointed out the HAT-trie thing because you claimed that that was a bug with tries, and it is definitely something being worked on. And, as someone who had spent time implementing this data structure who sounded like the hadn't heard of that work, you might have found it fascinating ;P.

(Also, "stratified indexes" are a concept that, while technically applying to most unbalanced structures, was designed for patricia trees and supposedly leads to good balancing properties. It was designed for disk storage, though, so it not quite ideal for in-memory use, but will at least therefore be not /ludicrous/ at memory performance if you stick to the page level.)

<comment><USER4> Since you are here, what do you think would be a suitable data-structure for SHA'ed key (the first 64-bit) associative array? I implemented a simple tries structure (no compressing at all) to handle this, but constantly struggled that maybe I over-engineered on this: (<https://github.com/liuliu/ccv/blob/release/lib/ccv_cache.c>)

<comment><USER3> Unfortunately, my interest has mostly been at the algorithms level: real-world implementations are usually dealing with so few elements that you are more affected by "silly" concerns like cache locality and instruction timing than anything else.

Therefore, it is really critical to find out how many items you want to be able to store. If you want to store a couple hundred you are going to do really well with something incredibly "stupid" like a changing hash table.

Another important question is regarding your expected concurrency. I mean, with a really strong random hash function and a data set that reaches a reasonable "steady state" on the number of objects it contains, hash tables are going to be ludicrously efficient if you have a really good hash function.

(edit: I meant to say: if you can tolerate the hashtable going offline entirely during a resize before it reaches steady state, although there are some algorithms for doing online concurrent resize; memcached actually uses a concurrent resizable hashtable, from what I remember, and is generally considered "very fast")

<sidenote> And, SHA is going to be a ludicrously good hash; arguably a /too/ good hash ;P. Out of curiosity, why /are/ you starting with a cryptographic hash (SHA)? Did your data already come in that form (maybe you are storing iPhone UDIDs ;P)?

These kinds of hash functions are actually /designed/ to be slow (so you can't use brute force) and are generally not needed for search structures (as you hopefully have a plan for organic collisions anyway). ;P </sidenote>

I will say that, before going out and doing a lot of research on this, I was in the position of building something similar-sounding (but using a fast 64-bit hash), and what I ended up with was similar to an "LPC trie" (which I luckily discovered quite quickly, and was able to read up on).

[http://www.nada.kth.se/~snilsson/publications/TRASH/trash.pd...](http://www.nada.kth.se/~snilsson/publications/TRASH/trash.pdf)

[http://www.nada.kth.se/~snilsson/publications/Dynamic- trie-c...](http://www.nada.kth.se/~snilsson/publications/Dynamic-trie- compression-experiment/text.pdf)

(However, again back to the hash function: the fact that I was doing the hash beforehand is not really how most of these algorithms are through through by the people who are developing them. Once you use the hash function you are now storing a fixed size element with a known-random distribution, which makes "use the first X bits as a hash function and store it in an array" an irritatingly good choice.)

I also generally recommend keeping a lot of the stuff the HAT-trie guy has published in mind (I've got it on my "omg must read" list, but had to get back to actual work). His PhD work was on cache-conscious data structures, and he simply went through and optimized the performance of everything under the sun for cache locality.

<http://www.naskitis.com/>

(By the way: I'd love to talk to you further at some point on IRC. I'm on irc.saurik.com; send a private message, and make it clear who you are in your initial message: don't just say "u there?" as I get a million of those ;P.)

<comment><USER5> _These kinds of hash functions are actually /designed/ to be slow (so you can't use brute force)_

Not so much. I think they are designed to have good mixing, be impractcally hard to reverse, and have other properties. This just makes them slower than hash functions generally used in hash tables. My understanding is that both MD5 and SHA were designed to be efficient under these constraints. bcrypt is designed to be slow, and can beat other cryptographic hashes by ridiculous margins. (Parameterized, actually)

</comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> These are binary tries with some interesting optimizations, in particular using special instructions supported on all modern cpus to skip common prefixes at the start of looking up.

According to the author's benchmarks they offer very consistent performance between insert and access operations, which seems to be key to why they perform better than R-B trees. Since tries don't need to re balance on insert we avoid memory operations R-B trees spend reorganizing interior nodes. Even though these are amortized in the R-B, we're still saving work.

Likewise, these are faster than small hash tables since we avoid resizing- rehashing overhead. Searching the binary trie is also likely faster than following a collision linked list we might find in a naive hash table implementation. However, for larger containers hashes begin to catch up, since at these sizes the trie involves increasingly more memory accesses.

I'd like to see these tries compared to generalized cuckoo hashing.

I'd also like to see them compared to the bagwell array mapped tries used by clojure and others.

<comment><USER3> (For the record, the reason why I have stared at this as much as I have just now is because I study these kinds of data structures and therefore want to understand what is new and compelling about this specific one.)

So, tries have the property of being only as balanced as the distribution of their alphabets. It is very easy to end up in a situation where worst case performance of a trie is linear in the number of items in the trie. There are some researchers who were doing work on a concept called "stratified indexes" that allow you to index into deeply skew'd tries that you may find interesting.

Regardless, the claims in this article are somewhat off. For example: they state that red-black trees "write out a lot of memory". Any full binary tree with N leaf nodes has N-1 non-leaf nodes. That statement is just as true of red-black trees as it is of this data structure. It is definitely true that the STL red-black tree implementation is inefficient (allocating memory for empty nodes), but that should not be held against the algorithm.

The documentation is even weirder. "Unlike either red-black trees or most hash tables, nedtries can store as many items with identical keys as you like." That is certainly not true, and isn't even true of the STL, which has a red- black tree without unique restriction called std::multimap.

The documentation also claims that hashtables can store arbitrarily large keys and nedtries can't, which is only as true of hashtables as it is for nedtries. As hashtables aren't ordered, and in fact require fixed-length keys, you can (and do: normally this statement would be going in the other direction of causality) use hash algorithms to hash the information to fit into the key length and then verify the value when you find it. The same thing could be used for nedtries.

I'm also having a difficult time understanding why red-black trees are not an "in place algorithm" and nedtries are. I'm not even certain I understand what it would mean for a search tree to be "in place", but the documentation makes claims about "dynamic memory allocation" being its key distinction, and I see no reason why a red-black tree need require that for search or deletion. Insert requires a single allocation, but so does a trie; and, even with the STL most people use a custom pool allocator for that purpose.

To be clear, I am not at all stating that "nedtries aren't faster than other implementations", just that that has nothing to do with their algorithm. The algorithm seems to just be an array of tries, where the array is designed to help with the common case of "there are a large number of zeros at the top of the key" (caused by using small numbers as keys).

Unfortunately, that makes the distribution among the bin-level tries drastically uneven, making that first "which bucket do I end up in" question severely unfairly unbalanced.

That specific optimization, then, doesn't seem to offer anything over a patricia trie, which itself would dramatically decrease the number of non-leaf node traversals (as nedtries does not describe using any kind of path compression at all). These path-compressed tries are what people normally use for high-performance lookup like routing.

<comment><AUTHOR> I agree that his prose is a little confusing. Here's my understanding of his intent:

"red-black trees write out a lot of memory"

I think this refers to how R-B trees require re balancing on insert, particularly accessing siblings of ancestors. This is doubly bad on modern processors, as it introduces a chain of control hazards dependent on these accesses. That can burn a lot of cycles on cache misses. In contrast, the trie structures only access nodes along the direct line down to the appropriate leaf. This has a control hazard (ie when to end the loop) and a data dependency between iterations. The control hazard can be easily predicted (assume search recurses) and modern processors are quite good at unrolling the data hazards across the shadow registers (or equivalent). Basically, modern processors and compilers are much happier with the sort of inner loops you see in tries vs restructuring trees (and are happiest with the inner loop of accessing a hash table).

"The documentation also claims that hashtables can store arbitrarily large keys and nedtries can't, which is only as true of hashtables as it is for nedtries."

I believe his implementation assumes size_t sized keys. This limit comes from his root table, which needs as many entries as bits in the key. While you could code up a variant that used arbitrary length keys I think you'd lose most of the performance advantages he measures (remember, we're mostly talking about constant factors here, not complexity class). He doesn't care about this limit, as his primary application is storing free lists in an allocator, which will always have size_t keys.

"I'm also having a difficult time understanding why red-black trees are not an "in place algorithm" and nedtries are."

I'm also confused by this, but I'm assuming he's looking at concurrent access. Tries only ever push down toward leaves, so with careful ordering it's possible to ignore many locking concerns. Or put a different way, you're only ever allocating new nodes or filling in previously nil slots in existing nodes. Trees with structural re balancing on insert like R-B could invalidate a concurrent traversal with a rotation.

"That specific optimization, then, doesn't seem to offer anything over a patricia trie, which itself would dramatically decrease the number of non-leaf node traversals (as nedtries does not describe using any kind of path compression at all)."

I think for his purposes there are some assumptions about distribution: namely if he's storing memory addresses in a nedtrie, it's likely he's allocating them with brk() or the like, and so they're clustered in one or two slots of the root table. Likewise, if he's storing block sizes, then the root table acts as a segregated fit similar to dlmalloc and its many offspring.

I'd agree that the way he over generalizes the advantages is a bit confusing, but I do think there's something interesting here vs vanilla patricia tries.

<comment><USER3> Re: assumptions about distribution

But if the assumption is that they are all using a couple slots of that root table, then the root table is not a performance advantage. It would be faster to just start the trie immediately and switch off the first bit.

The point of having a structure like that at the top is that it is a cheap-o form of level compression, and increases the fan-out of the root node, allowing you to cut the search space down much faster given a single very cheap lookup.

As for segregated fit, the bucket array itself is probably actually good enough for that. What /could/ make that interesting is if you /also/ stated that "small" (many leading zero) keys happened to be much more prevalent, and in fact twice as prevalent, as those in the next bucket up.

Otherwise, it is just further unbalancing his trie and is getting reasonably the same tradeoff as if he were using a patricia trie (which will use path compression to eat large numbers of 0s and otherwise will be reasonably as unbalanced without the cost of the bucket array, which is going to be many internal nodes worth of memory).

<comment><USER3> In fact, it is even worse than that: in the case of random keys (which he admits in the documentation is important for trie balance without helping the user obtain them) he'd be much much better off just chopping the first five bits off the key and using that to index into his top-level array.

Doing that would let the top-level array segment the search space perfectly into 32 sub-buckets, avoiding the fact that the higher order array elements are increasingly meaningless (as they are exponentially smaller).

At that point, however, the datastructure is now a level-compressed trie ;P. (LPC tries seem to be at the state of the art for this kind of structure).

</comment></comment></comment></comment></comment><comment><USER6> djb's "critbit" trees are much more interesting, I think: They are basically binary patricia trees (like nedtries, if I understand correctly). However, djb's observation is that in order to locate a specific string, you only have to look at the "critical" bits - the bits that are different between different entries in the database.

So, for example, if key 1 differs from key 2 (top two keys in the tree) in bit 317, you check bit 317, and you know which one of them it is NOT (and also, which branch of the tree it is not). Proceed recursively -- most keys will, in practice, be identified by log2(n) bits where n is the number of keys in the table, NOT the length of the key.

<comment><USER3> For the record, patricia tries in the literature are defined to have this "critbit" property. The wikipedia article on that subject is a little broken, but djb's article is correct when it states:

"This idea was introduced by Morrison in 1968 under the name PATRICIA, and independently by Gwehenberger at about the same time."

<comment><USER4> Also, I found this link to be a good reading for anyone interested in the implementation detail: <https://github.com/agl/critbit/blob/master/critbit.w>

</comment></comment></comment><comment><USER7> I like the concept, but the author haphazardly combines asymptotic analysis with unsubstantiated performance benchmarks and uses some imprecise language.

What does "nearly O(1) complexity" mean? Why does he mention a complexity of "O(8 _sizeof(void_ ))"? Shouldn't that constant factor be collapsed?</comment>


Show HN: Kixker – The social soccer news app (curated by r/soccer community) - <AUTHOR> I am a huge soccer fan, but nowadays I could rarely find the time to go from web to web to read soccer news and watch highlights.
Recently, I've found the reddit r/soccer community (~400k redditors) to be a great source for soccer news and highlights. The problem is that it's painful trying to read news and filter out game highlights on mobile devices (especially on game day).
Over the holidays, I've spent time to build the apps (Android & iOS) and this is my way to follow the latest/important soccer news.
Android: https://play.google.com/store/apps/details?id=com.talatululabs.kixker
iOS: https://itunes.apple.com/us/app/id1067496083
Would love to get your feedback! <comment><USER2> Can this be made generic? How about one for /r/nba?</comment>