Babylonian Mathematics (2003) [pdf] - <AUTHOR> https://www.math.tamu.edu/~dallen/masters/egypt_babylon/babylon.pdf <comment><USER2> Related unsolved mystery in ancient mathematics - the Biblical genealogy from Adam to Noah in Genesis 5 contains 10 individuals. For each of the individuals, the genealogy reports the age at which their descendant was born, the remaining number of years in their life, and their age at death. 30 numbers in total. In the Masoretic Text (often considered the most reliable), those 30 numbers all end in a 0, 2, 5, 7, or 9. The chances of this happening by chance are effectively 0.

Possible explanations include either 1) that the numbers were corrupted, possibly when being converted from a sexagesimal system into decimal, or 2) that the author never intended the numbers to be "real" numbers in the sense we would understand them, but was instead intending to use a pattern to teach symbolic meaning. However, to my knowledge, nobody has ever offered a completely convincing interpretation of the puzzle. If #1 is correct, what were the original values and what was the process by which they were corrupted? If #2, what is the pattern that the author was constructing, and what is it intended to communicate?

Other genealogies in Genesis have similar weird features, and some have proposed theories that involve the other genealogies as well, but that's beyond the scope of this comment.

See - [https://en.wikipedia.org/wiki/Genealogies_of_Genesis](https://en.wikipedia.org/wiki/Genealogies_of_Genesis)

<comment><USER3> _" those 30 numbers all end in a 0, 2, 5, 7, or 9. The chances of this happening by chance are effectively 0."_

What are the odds of those 30 numbers all ending in some numbers in a random set of 5 single-digit numbers?

Also, considering all the numbers in the Bible, what are the odds that some arbitrarily chosen set of them will contain some sort of unusual pattern?

<comment><USER2> I don't have the math in front of me for the first question, but I've seen it worked out in the past. It's on the order of 1 in a billion.

On the second question, considering "all the numbers in the Bible," an arbitrarily chosen number or set of numbers from the Bible will contain a pattern very, very often, because that's how people thought in that culture.

Jesus didn't have 12 disciples by accident. They symbolically represent the 12 tribes of Israel.

The genealogy in Matthew 1 doesn't contain 3 sets of 14 generations by accident. The author had to skip at least a few kings from the Jewish tradition to make it work out. And the fact that it's intentional is underscored by him explicitly saying "so, there were 3 sets of 14 generations..." (He's making a theological point, but the details of that aren't relevant to the question at hand.)

I could go on... Intentional numerical patterns are, in fact, quite common! And this gives us more, not less, reason to suspect that the Genesis 5 genealogy has an intentional pattern.

</comment></comment></comment><comment><AUTHOR> So one item on my "bucket list" is to write some math texts for either curious school students or people who home school, or for older people interested in learning math. And I think a good approach that I haven't seen before is to intersperse the math with the history of it's discovery. So kids could spend a chapter doing egyptian math. Then a chapter doing Babylonian math. Then a chapter doing Greek math with a ruler and compass. Then some more. And in this way, build math up together with fun facts and pictures of the historical places where the math was created. That's I love documents like this.

<comment><USER4> I have a similar desire but for science in general!

<comment><AUTHOR> That would be awesome, too!

</comment></comment></comment><comment><USER5> What blows my mind is that Babylonians knew about Fourier transform. What was humanity doing for like 3000 years?

<comment><USER6> I didn't see anything about fourier transforms in TFA? Do you have links?

<comment><USER7> I too got curious, but the only thing that I have found is this paper from the University of Babylon:

[http://www.uobabylon.edu.iq/eprints/paper_2_29545_76.pdf](http://www.uobabylon.edu.iq/eprints/paper_2_29545_76.pdf)</comment></comment></comment>


Robots will not take your job - <AUTHOR> https://www.wired.com/2017/08/robots-will-not-take-your-job/ <comment><USER2> This article is straight-up nonsense. I'm shocked that an intelligent, educated person wrote it.

Its main argument is that technology has had a net-positive effect on jobs in the past, therefore it will in the future. Here's a key excerpt:

> _A rigorous study [...] found that robots did reduce the hours of lower- > skilled workers—but they didn't decrease the total hours worked by humans, > and they actually boosted wages._

Many years ago, robots replaced the lowest skilled jobs. That's not a huge problem because, with enough education, those workers could move somewhere else in the economy.

But technology gets better at an exponential pace. We're already seeing robots that can perform surgery better than the best surgeons. Some of these things aren't happening in 25 years, like L5 autonomous cars -- they're happening in 5-10 years.

We're also going to see workers replaced in quantities that will absolutely dwarf those of the past. What happens when sewing is automated in the US? There are small countries which will spiral into economic collapse, and they won't have "time to prepare" because time isn't what they need to prepare. They need resources and infrastructure that they just don't have.

The dire warnings are true. There's a good reason Zuckerberg, Gates, and Musk (among many others) have all warned about this being an impending crisis.

</comment><comment><USER3> "that industrial automation has been responsible for the loss of up to 670,000 jobs since 1990. But just in the period between 1999 and 2011, trade with China was responsible for the loss of 2.4 million jobs: almost four times as many. "If you want to know what happened to manufacturing after 2000, the answer is very clearly not automation, it's China,""

I think globalization in general would be more accurate than "China".

Yet I also think that automation is also a factor which even though might not be the major one yet it can very well be in the near future. I'm in IT industry and I've witnessed at least one line of job going out of the window - namely manual testers- due to automation. I bet everyone has similar stories from his own walk of life. Yet most of the jobs lost in my locality was due to them being shipped to low cost countries. So - all in all- these two factors work in parallel and probably globalization is holding up automation up to a point (where automation cost > low wage worker).

</comment><comment><USER4> Sure they won't.

I remember coming to my mothers office when I was a kid. There was entire big hall full of desks with typewriters and typists - like 50 of them. If some documents needed new version it had been passed there. Xerox machines had a special crew. Almost every mid level manager had a personal assistant.

Nowadays you have to do it yourself with email, printer and a smartphone with calendar reminders I guess.

Soon it's going to be the same with security guards watching all day CCTV's in shopping malls, doctors (sometimes in 3rd world outsourcing center) who check your USG and CT scan, low level lawyers going through tons of legal documents. And your visit to the doctor will take 5 instead of 15 minutes cause you would be pre-screened by robot assistant first.</comment>


5 GPU Ethereum Mining Rig Build Guide - <AUTHOR> https://github.com/fogonthedowns/5-gpu-ethereum-mining-rig <comment><USER2> Quite funny the whole thing is about 5 GPU's and the image on the repo shows 6.

There are too many good guides on the internet already, this one doesn't even come close providing all needed info. But I see the author has put his ether account address in the build string, that might be the main reason for this post.

Btw, don't go mining on linux with nvidia, it's really bad advice as overclocking all cards can take days to get it working if ever! Just use Windows for nvidia, it's a breeze to setup and you'll get overclocking for all cards out of the box.

<comment><USER3> Mind to share those good guides?

</comment></comment><comment><USER4> It's probably more profitable to just buy 1 gpu, and play games with.

</comment><comment><USER5> How do people even buy on this "mining money" thing? For me, it's just like thousands of people burning tons of electricity for nothing, which they could direct to really useful calculations in astronomy or physics for example.

</comment><comment><USER6> You will top out at generating $900 from this rig: [http://www.mycryptobuddy.com/EthereumMiningCalculator/path?h...](http://www.mycryptobuddy.com/EthereumMiningCalculator/path?hashrate=150&power=900&chartView=profits)

Don't use the CyrptoCompare profitability calculator for anything other than 1 month projections. It doesn't take into account increasing difficulty. The MyCryptoBuddy calculator I linked to does.

</comment><comment><USER7> I am completely new to ETH. Is this profitable?

<comment><USER8> Yes, the hardware manufacturers will certainly profit from you.

</comment></comment><comment><USER9> So how much ETH would this generate today?

<comment><USER10> Roughly 1 per month

<comment><USER11> I googled some of the hardware mentioned in this guide and it seems like it would cost several thousand dollars, at least, to buy it all (I could be totally wrong, I'm not much of a hardware guy). At the current price of Ethereum, you're not going to break even for six months, which doesn't strike me as a fantastic get-rich-quick scheme. I would be delighted to be proven wrong by someone more knowledgeable, though!

<comment><USER9> Plus with difficulty increases you will soon be earning less than 1 ETH per month so break even will get further and further away. As with most cryptocurrencies that are actually worth having you are better spending the money you would spend on mining equipment actually buying ETH.

This is great if you happen to have 5 x GPU's lying around to put to use and cheap electricity to run it.

<comment><USER12> Or if you _want_ to have a 5 GPU system as a way to offset some of the costs before using it for it's real purpose..

</comment></comment></comment></comment></comment><comment><USER13> what's a good guide for AMD GPUs R280</comment>


TOS requires users to give up first born, people sign-up anyway - <AUTHOR> http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757465 <comment><USER2> People vote for Brexit, Donald Trump, etc. Humans are capable of autopilot</comment>


Larry Lessig moving to Harvard - <AUTHOR> http://www.boston.com/news/local/breaking_news/2008/12/harvard_snags_a.html

<comment><USER2> <http://www.rei.com/product/752777>

<comment><USER3> Am I missing something, or does Lessig's move to Harvard come with a free jacket?

<comment><USER4> That REI link was meant to say that it's cold in Boston.</comment></comment></comment>


Failing Subway Threatens New York's Financial Future, Transport Chief Says - <AUTHOR> https://www.nytimes.com/2017/11/20/nyregion/subways-new-york-lhota-mta.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=second-column-region&region=top-news&WT.nav=top-news <comment><AUTHOR> Discussion on an antecedent story: [https://news.ycombinator.com/item?id=15737333](https://news.ycombinator.com/item?id=15737333)</comment>


Ask HN: 802.11ac-enabled routers for OpenWRT, zRouter, Tomato, pfSense, etc.? - <AUTHOR>

I'm about to move into an apartment with a 100 Mbps connection from WebPass, so I'm looking to buy a new router and/or access point. As far as I can tell, they use Large Scale NAT[1] on IPv4, so I'd like something that allows me to install firmware where I can configure and tinker with IPv6.<p>I stumbled upon this thread[0] from last December, but was curious as to whether people feel the advice in that thread still holds true and/or if people have had any experience with newer 802.11ac routers/APs (especially devices from Buffalo, TP-Link, and Ubiquiti) now that they've been on the market for a few months.<p>[0]: https://news.ycombinator.com/item?id=6828699 [1]: http://webpass.net/blog/ipv4-exhausted <comment><USER2> If you care about IPv6 I suggest CeroWRT but it does not support ac hardware yet.</comment>


5 Reasons You're Probably Not Going To Big Omaha Next Year - <AUTHOR> http://www.beyondthepedway.com/5-reasons-youre-probably-not-going-to-big-omaha-next-year

<comment><USER2> Reason number six... never heard of Big Omaha.

<comment><AUTHOR> Now you have. :)</comment></comment>


Schneier on Security: Cargo Security - <AUTHOR> http://www.schneier.com/blog/archives/2010/10/cargo_security.html <comment><USER2> Per Schneier's point, focusing on "threat scenarios" is useless if you have a diffuse threat and unlimited targets.

As an aside, try to look at the history of the definition of terrorism: <http://en.wikipedia.org/wiki/Definition_of_terrorism> . It's quite difficult to find an all-encompasing term that does not include the US or her allies.

</comment><comment><USER3> I agree that you should fight terrorism using intelligence, but I don't agree with this statement:

"If a passenger plane blows up, it affects a couple of hundred people"

What about the scenario where a cargo plane is blown up over a highly populated area or the centre of a major city? I think he is forgetting that on 9-11 the planes were used as weapons, and the lives lost on the planes were a fraction of the overall total on that tragic day.

<comment><USER4> If a cargo plane blows up over a highly populate area it will kill about a dozen people (see example in blog comments). This isn't nothing, but its pretty hard to get your bombs set to blow up at the exact right time.

As to 9/11, that was a hijacking rather than a bomb and an entirely different attack vector.

<comment><USER5> > its pretty hard to get your bombs set to blow up at the exact right time.

You can use GPS tracking and a program running on the phone. With this setup, you can guess when the plane will pass over a major city and blow it up according to population density.

I would assume someone would already have sent the phones with their programs to acquire more information on signal strength, cell towers and all the data to make something like this work before attaching explosives to the devices.

Someone probably is right now going over other packages sent by similar routes, possibly by associates of this guy and those who picked them up at the destination. I am quite sure this was not his first attempt.

<comment><USER4> GPS won't work inside a metal shell, which is what cargo compartments on airplanes are (passenger airplanes have windows).

<comment><USER5> Not sure about that (planes being radio-sealed). Planes are getting more and more composite materials to save weight. One of the key features of the 787 is the composite fuselage that I am not sure will effectively block GPS signals. Also, many cargo planes are remissioned or dual-use passenger planes that still have windows:

<http://picasaweb.google.com/lh/photo/XMBDcG2dmr-Z1QoGBdXu2g>

If all else fails, some phones are coming with gyroscopes...

</comment></comment></comment></comment></comment><comment><USER6> I'm not one for conspiracy theories, but I do find it strange that this new type of threat comes days after the BA chief complained about overly-strict US security measures.

This response says, "See, we do need them! And we probably should add a few more measures too!"

<comment><USER7> It seems like you _are_ one for conspiracy theories to try to make that work.

The British airport guy complained about shoe checks and laptop scans. This threat was about cargo shipped bombs. They are two entirely different things with an minuscule Venn overlap ("having something to do with some sort of transportation device...").

The "cargo shipped terrorism" was all the focus during the anthrax worries. It isn't new.

</comment></comment><comment><USER8> The crazy thing about this is that this type of terrorism is completely pointless. There are no demands, no hostages, no agenda, it's just let's blow something up in America. At best someone is just trying to score some points back home by having managed to kill some Americans, but are they really terrifying anyone? Is there anything we could do to in terms of policy to end this "threat"? If there was would we ever? The answers are no.

The only way to end terrorism is to bring education and prosperity to the middle east, to all of them, not just the few already rich from oil money. To give them something to do other than blow themselves up.

<comment><USER5> Not necessarily.

If they succeed blowing up enough cargo planes, this would create a non- trivial economic pressure to deal with whatever demands they have. If they manage to blow up one with debris falling on a big city, more points to them.

</comment></comment><comment><USER9> _[this is a repeat of my post on his blog]_

From what I've read, the devices were wired to detonate via cell phone, and some speculated that the devices were to be triggered when the flights were over major cities. This raises three questions in my mind:

1\. Did the flight paths of these planes actually go over major cities? Or just on the periphery?

2\. Will cell phone signals reach flight altitude?

3\. (Although Bruce advocates doing nothing) Would cell phone signal jammers on cargo planes be a cheap deterrent in this case?

<comment><USER5> What if the device is rigged to explode when the phone loses signal?

<comment><USER4> Then it explodes when loaded into the cargo hold, in which case only the loaders die and the terrorists could have done much better with just a bomb on a street corner.

<comment><USER5> > the terrorists could have done much better with just a bomb on a street > corner.

Not sure. Do you know how much money would it cost to replace the plane and shut down the airport until officials make sure that was the only bomb?

Besides that, the loss of signal could start a countdown.

</comment></comment></comment></comment><comment><USER10> _Cargo that is loaded on to passenger planes should be subjected to the same level of security as passenger luggage._

It should be more. It's more costly for a terrorist organization to put a bomb on a plane they also have to ride on. You have to find, train and forfeit one "martyr" per bomb.

After 9/11 I seem to remember something about not allowing people to put luggage on a flight they weren't going to be on. As in, if you didn't show up at the gate to get on, they would remove your luggage too. What happened to that?

</comment><comment><USER11> I've heard report that attack was actually revealed by whistleblower from Saudi Arabia. Good job unknown US and UK secret agencies for taking credit for saving the World, again!

<comment><USER12> British never discus sources or capabilities.</comment></comment>


The Ramanujan sum of all positive numbers up to infinity is -1/12 - <AUTHOR> http://kottke.org/14/01/the-sum-of-all-positive-integers

<comment><USER2> According to the Wikipedia article [1], it equals -1/12 (R), except the R is written a rather fancy script.

Now, I'm no mathematician, but I'm betting that (R) is important, and affects the true value of the -1/12, and probably should not be omitted.

[1] [http://en.wikipedia.org/wiki/Ramanujan_summation](http://en.wikipedia.org/wiki/Ramanujan_summation)

<comment><USER3> There's only one way to add a finite list of numbers, but several ways to "add" infinite lists. Ramanujan summation is just one way. There are also other ways to define "addition" on infinite lists: Abel, Borel, Cesaro, Euler, etc. summation.

They're different from each other because they disagree on certain infinite sums, but they're still considered "summations" because, for finite lists, they give the same answer as normal addition.

<comment><USER2> So the title should instead read "The sum of all positive numbers up to infinity (for a certain definition of 'sum') is -1/12".

</comment></comment></comment><comment><USER4> According to some people...

The sum of alternating +1 and -1 equals 1 if you stop on an odd, and 0 if you stop on an even. Infinity is treated differently in different fields. I suppose in physics, it's probably treated as a superposition of even and odd, so statistically, you could interpret it as 0.5. And I'm sure that works great for physicists and it fits a number of scientific models. But there are probably other fields involving mathematics that would find this interpretation to be silly and not very useful.

<comment><USER5> You're referring to Grandi's Number. Numberphile covered this pretty well on their youtube channel: [http://www.youtube.com/watch?v=PCu_BNNI5x4](http://www.youtube.com/watch?v=PCu_BNNI5x4)

</comment></comment><comment><USER6> The Ramanujan Summation is not a sum in the traditional sense. So the title is missing a script R after the -1/12.

<comment><USER7> And a non-linkbait title for the post would be:

 _The Ramanujan sum of all positive numbers up to infinity is -1 /12_

Which is not sensational, exciting, or even that interesting.

</comment></comment><comment><USER8> I asked this question on the mathematics stack exchange: [http://math.stackexchange.com/questions/633285/is-the-sum- of...](http://math.stackexchange.com/questions/633285/is-the-sum-of-all- natural-numbers-frac112)

It is only true if you redefine what the "sum" of an infinite series is.

</comment><comment><USER3> Achilles runs at 1 unit/s and Tortoise at x < 1 unit/s. Tortoise gets a head start of 1 unit. How long until Achilles catches up?

Well first, Achilles has to travel 1 unit because Tortoise got a head start. That takes 1s. By then, Tortoise managed to run x units, so Achilles has to catch-up, which takes another x seconds, and by then... etc: The time it takes Achilles to catch up is 1 + x + x^2 + ... Since x < 1, this converges and the answer is 1/(1-x). It takes 1/(1-x) seconds for Achilles to catch up.

Plot the lines: Achilles runs along A(t) = t, the tortoise T(t) = 1 + x t. Solving for t: t = 1 + x t = t (1 - x) = 1; t = 1/(1-x).

Ok, now what if x > 1? Let's use x = 2.

1/(1-2) = -1. It takes -1 seconds for Achilles to catch up. Don't believe me? Plot it: A(t) = t, T(t) = 1 + 2t; Achilles catches up with Tortoise 1 second _before_ the race starts!

There you have it, a physical interpretation of why 1 + 2 + 4 ... = -1. (I have no example right now for 1 + 2 + 3 ..., sorry.) So you see, this isn't nonsense, though I admit it DOES depend on definitions!

<comment><USER9> I am fairly certain sum=1/(1-x) holds true only for x<1\. when x>1, it is a divergent series... and I can't tell if you're joking or not, or I would just laugh...

</comment></comment><comment><USER10> This video has been showing up all over the internet, and the math in it is completely imprecise. Numberphile has lost a lot of respect in my eyes due to how much misinformation this video has spread.

<comment><USER11> They also created a pretentious video where they explained that the NSA hacked people's e-mails through a rigged random number generator, complete with an attempt at explaining elliptic curves and modular arithmetic, that ignores the fact that barely anyone used that PRNG and most e-mail isn't encrypted properly in the first place.

Either I'm a really grumpy person, or their videos are actually terrible. I think there are enough interesting things in math that can be explained without falsely embellishing uninteresting results and ideas.

</comment></comment><comment> dj-wonk Unintended consequence: my confidence in string theory just plummeted.

</comment><comment><USER12> Wikipedia says the sum (1 - 1 + 1 -1 ...) diverges.

[http://en.wikipedia.org/wiki/Grandi's_series](http://en.wikipedia.org/wiki/Grandi's_series)

<comment><USER13> Well yeah it doesn't converge on any particular number so it must diverge.

Like what has already been said I think this is helpful in certain fields, but bollocks in others.

</comment></comment><comment><USER14> He shifted the series over and added it to itself, which I've seen be called out as an invalid thing to do on another sum 'proof' with a weird result.

</comment><comment><USER15> there was a good discussion of this on /r/math a week ago:

[http://www.reddit.com/r/math/comments/1usu93/1_2_3_4_5_112_n...](http://www.reddit.com/r/math/comments/1usu93/1_2_3_4_5_112_numberphile/)

<comment><USER16> I was visiting this comment thread with the intention to post that link if nobody else had already. For all the hate reddit gets, I love it for content such as the discussion contained in the above.

Reddit has major issues due to its user base, but one thing I do is browse certain subreddits once a week (month) by filtering by top posts in the past week (month). The day by day drone in most subreddits is tiring, but quality threads hit the top posts.

</comment></comment><comment><USER17> If 1+1-1+1-1+1... can be approximated to 0.5, then 1000+1000-1000+1000-1000+1000... must be approximately 500, but shouldn't 1+1-1+1-1+1...be equivalent to 1000+1000-1000+1000-1000+1000...?

<comment><USER18> Note that 1+1-1+1 doesn't fluctuate between positive and negative, it fluctuates between 0,1,0,1,0,1.

</comment></comment><comment><USER19> What is presented here is actually a reductio ad absurdum proof that 1 - 1 + 1 -1 ... does not equal 1/2 (or any other number).

</comment><comment> dj-wonk Lose 20 pounds -- and your sanity -- using this one weird trick!

I suspect this kind of reasoning is useful in some fields for some particular kinds of infinite sums, but for the particular summation problem as stated, as most people understand it, this is rubbish.

<comment><USER20> You should watch the linked video. The logic isn't that hard to follow, and finds real-world application all over the place.

<comment> dj-wonk I was using humor there. Yes, I am open to this kind of thinking being useful. I did watch the video. I followed the logic. (That logic is contradictory from other perspectives -- see [http://www.quora.com/Mathematics/Theoretically- speaking-how-...](http://www.quora.com/Mathematics/Theoretically-speaking-how- can-the-sum-of-all-positive-integers-be-1-12)).

Summary: as most people understand it, the sum of all natural numbers is (a) positive (b) very large (c) not finite (d) certainly not -1/12.

</comment></comment></comment><comment><USER21> Did Euler overflow his abacus or something?</comment>


Ortega hypothesis - <AUTHOR> https://en.wikipedia.org/wiki/Ortega_hypothesis <comment><USER2> I'm not all that sure that you can generalize between the time periods

a) prior to 1900

b) 1900 to 1970

c) 1970 to the present

In particular, the academic enterprise was so much smaller in period (a) and it was dominated by people who were born of nobility.

In the period (b) it was rapidly expanding and it was possible for people (good or bad at science or politics) to get in relatively easily.

In period (c) it has been a game of musical chairs in which being good at politics trumps being good at science.

An example of science in period (c) is high-energy physics.

People talk about Edward Witten, for instance, as if we was comparable to Feynmann or Einstein, but Ed Witten has never predicted that something would be observed that later got observed. (Nobody has in period c!)

There is a huge amount of research in "anti-desitter spacetime" because string theory might work in ADS, but the part of the universe we can see seems to be a "desitter spacetime" where it doesn't seem string theory will work.

This doesn't seem to bother period (c) scientists for some strange reason.</comment>


Clojure Concurrency  - <AUTHOR> http://blip.tv/clojure/clojure-concurrency-819147

<comment><AUTHOR> copy of slides is available here: [http://unina.stidue.net/Politecnico%20di%20Milano/Elettronic...](http://unina.stidue.net/Politecnico%20di%20Milano/Elettronica%20ed%20Informazione/Matteo.Giovanni.Rossi/Didattica/FunProgLang4Parallelism/04-ClojureConcurrencyTalk.pdf)

<comment><USER2> I found the source to the ants simulation on GitHub, made a gist: <https://gist.github.com/1494094>

</comment></comment><comment><USER3> Note that this is pre clojure 1.0. With some of the newer features some things could be simpler (futures).

</comment><comment><USER4> This is an old recording.

It's my favorite of the many recordings of Rich giving presentations on Clojure in the early days.

It's where he presented his famous ants program that runs on ~64 native threads.

<comment><USER5> I was in the room for this presentation, it was the first time I had heard of Clojure.</comment></comment>


De-Obfuscating the Statistics of Mass Shootings - <AUTHOR> http://www.charlespetzold.com/blog/2015/07/De-Obfuscating-the-Statistics-of-Mass-Shootings.html <comment><USER2> Interest groups lying with statistics is a specific example of confirmation bias. It's sadly so prevalent that I generally assume that any statistics I find are intended to lead me in a particular direction.

I am skeptical of the final implication of the blog entry, that improved gun- safety legislation would solve this issue. For someone taking such a strong approach to statistics, it's a shame that they focus on only one facet of such a complicated situation. American culture, mental health treatments, social programs, and gun laws are all different from most other countries on their list - it seems disingenuous to mention only the last.

<comment><USER3> Your comment is actually a perfect example of why I have issues with using per capita as the basis of the analysis.

I adjusted the OCED numbers to be on a per-gun basis as opposed to a per capita basis and non US OCED countries combined were more likely to have rampage shootings per gun than the US.

There are not issues in the US with social programs, mental health, etc as compared with other countries when looking at mass shootings. The only number that plays any significance is the number of guns per capita to influence the number of shootings per capita.

If other countries had as many guns per person as the US, the data indicates they'd proportionally have the same amount of shootings per person or more.

<comment><USER4> In the USA, people who own guns are likely to own more than one gun. They may be hunters or collectors. Per-gun is fairly meaningless.

<comment><USER5> I recently read statistics that the number of households owning guns has decreased from 50% to ~30% since 1977, while the number of guns increased over that time period.

<comment><USER6> I did some back of the envelope checks on those statistics, and they would require us gun owners to have arsenals worth $100,000 or so.

Note that one's willingness to tell an anonymous voice on a phone that one owns guns changes with the political climate towards guns.

<comment><USER7> How do you get $100,000? If 30% of households own guns, that's about 37 million gun-owning households. If there are 300 million guns, that's about 8 guns per gun-owning household. If that's $100,000, that means a gun is worth on average $12,500. I'm hardly an expert on guns but that seems about an order of magnitude too high from what I do know.

<comment><USER6> This was in reference to a statistic about who's buying newly manufactured "assault weapons", particularly the mass quantities of AR-15 pattern rifles and the like that have been selling like hot cakes since the 2008 election.

It insisted that only us "gun nuts", not e.g. the large number of households that only have one or two guns for self-protection, were buying almost all of these new guns; to review and improve my estimate we'd have to find that particular statistic/study.

<comment><USER7> I see. Is there some context in the gun world that I'm missing out on? Since the comment you replied to said nothing about AR-15s or indeed anything beyond the number of gun-owning households and number of guns overall changing over the years.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> This brings up more questions than answers in my mind. Do we care about mass killings, or whatever is qualified as a "rampage shooting"?

For example, Mexico has 1 "rampage shooting" in the data referenced in the article. Yet you have cartels killing dozens of people on a regular basis. The question in my mind is that if we don't have lots of "rampage shootings" in countries with lots of access to weapons, why does the US have so many? Why aren't developing countries with ubiquitous AK-47s experiencing this phenomenon?

<comment><USER9> And if you included all the killings in Chicago during the same period, it would probably make most of Mexico looks like Disney Land.

<comment><USER10> Sure you could find a city in Mexico that would look like Disneyland next to Chicago. But compare Chicago to Ciudad Juarez and the murder rates are about the same (457 in 2014 in Chicago, 424 in 2014 in Juarez). And a few years ago (2010) Ciudad Juarez was the murder capital of the world with over 3000 murders.

Mexico as a country still has about twice as many murders per year as the United States while having less than one-third the population. Mexico has FAR stricter gun laws btw.

<comment><USER6> _Mexico has FAR stricter gun laws btw._

Mexico has exactly _one_ legal gun store for the entire civilian population. Ownership of guns in "military" calibers (previously adapted by a country's military) are illegal, once reason .38 Super was developed for the M1911. The whole country probably makes NYC look like a legal gun-owners' paradise (less than 60,000 each for handgun and long gun ownership permits for 8 million residents).

</comment></comment></comment></comment><comment><USER11> Isn't the fact that Norway appears near the top of this list only due to a single incident with a large number of victims, i.e. [https://en.wikipedia.org/wiki/2011_Norway_attacks](https://en.wikipedia.org/wiki/2011_Norway_attacks) which skews the statistics. This is what he means when discussing small-N statistics being meaningless.

</comment><comment><USER12> I would be interested in seeing the data correlating SSRI drugs to mass shootings statistically analysed so neatly

<comment><USER13> I've been panning these comments with Ctrl+F for "SSRI" and you're the first one.

You and I would be interested in seeing that data, and you and I will never see that data. ;)

Also the media never ever mentions most of those shooter kids were on those hardcore FDA drugs.

What's the point? We know the gun was responsible for the killings. /s

<comment><USER6> Seriously, we _do_ believe these shooting are correlated with just plain mental illness. Which are largely "treated" with "those hardcore FDA drugs" (which SSRIs, due to the low side effect profile, really aren't), so therefore I'm not surprised there seems to be a correlation between the two. However, after a fair degree of study, and discussion with one of my psychiatrists a dozen or so year ago, I don't think there's causation, just correlation. Or at least he didn't believe the "antidepressants give you enough energy to be violent" thesis based on all three generations of them.

<comment> norea-armozel I can say that sadly I'm one of those people that don't do well on SSRIs (at least the withdraw from them). I literally wanted to stab my own mother and sister when I was coming off of SSRIs (Zoloft). I literally felt so angry at everyone and everything. So, I'll never go on SSRIs ever again.

</comment></comment></comment></comment><comment><USER14> I believe the statistical lesson here is widely applicable, and may well be of use if data analysis - particularly of rare events - is part of your job.

<comment><USER15> This statistical lesson is actually a textbook error, similar to taking a p=0.05 test on multiple subjects separately.

What makes this interesting is the political context.

<comment><USER14> Certainly, but I have seen _many_ people tasked with analysing data in a work context who have never picked up a stats textbook.

</comment></comment></comment><comment><USER16> Sherlock (or Doyle) says it best. "It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts."

Petzold does a really good job of debunking the original claim that the US is not statistically significant when it comes to the number of mass shootings. However, all of the effort he goes through only points to one conclusion: the number of shootings we have in the US isn't statistically random. All this means is that there's something causing the rate of shootings to be higher here than is expected by random influences. It doesn't point to what that cause is.

It's odd to see how statistics are abused by interest groups/media. Intuition is usually a pretty good yardstick for the validity of a statistic, insofar as knowing what kinds of conclusions should be able to be drawn from what kinds of data. If you've got a bunch of data about who likes what flavor of ice cream, it's not likely that data says anything about /why/ each person likes what they like. In the same way, it doesn't matter if you say that the US is normal when it comes to number of shootings or to say that it's abnormal as Petzold does, when you try to say /why/ this is the case, you better be drawing direct conclusions from your data.

<comment><USER17> Speaking of validity, it's not immediately obvious to me that each citizen of an OECD country constitutes a valid Bernoulli trial. Applying a uniform empirical prior of shooting rampage probability across the entire population strikes me as questionable. Presumably the results would be roughly convergent if he had done this with an inhomogeneous Poisson or something, but it seems a little handwavy to just fire off a binomial and call it good.

</comment></comment><comment><USER18> This article demonstrates the danger of thinking statistics is logic - you can only use statistics with the analytical framework firmly set up...not as a replacement.

The United States is just that - United. States. 38 mass shootings, however defined, means there is /less than one per State/. Guess what? That makes Finland's two /much/ higher. That's...statistically significant!

Explaining how completely incapable frequentist statistics are at describing anything at a low sample size doesn't mean that something succumbs to "random fluctuation." There is nothing random about a person deciding to take arms and shoot lots of people. Not only that, this data is based on a full population sample - it's not a survey of 5,000 people extrapolating to the whole population - it IS the full population.

And he removed the idea that the original chart was designed show - lack of data that shows compelling evidence that more restrictive gun laws prevent either mass shooting incidents or mass shooting deaths.

He should step away from the statistics - it's a loaded gun and he's not trained to use the weapon properly.

</comment><comment><USER3> It's disingenuous to compare shootings purely by a per-capita basis. It should be per-capita among gun owners.

The data as shown indicates the US has a cultural mass shooting phenomenon, but it's simply not accurate. The US just has more guns, and mass shootings happen at extremely rare rates among gun owners everywhere.

This approach would be like comparing drunk driving fatalities in Los Angeles vs Venice, Italy.

Edit: I was curious and just ran the numbers with the full OCED countries. The US had 0.136 rampage shootings per gun (compared to 0.120 rampage shootings per capita), and non-US countries had 0.177 rampage shootings per-gun (compared to 0.246 rampage shootings per capita). I used the author's numbers and the Wikipedia page data for guns/100 residents.

<comment><USER19> > It's disingenuous to compare shootings purely by a per-capita basis. It > should be per-capita among gun owners.

First off, I don't live in the US and I don't have a horse in this race.

You may have a point. However, wouldn't the fact that having a large number of gun owners directly correlating to mass-shooting fatalities still come down to the same conclusion: that guns should be regulated further? Or do you think that if the analysis was done it'd show something different to what the analysis in the original article shows?

<comment><USER20> > First off, I don't live in the US and I don't have a horse in this race.

Not living in the US presumably subjects you to position bias. Those who live in countries where gun rights are strong tend to favor gun rights. Those who live in countries where gun control is strong tend to favor gun control. This is true of most things, as the moral compasses of most are drawn from what is currently normative to their existing environment. This is true of most places in which socio-political biases are codified into law, whether regarding religion, gay rights, whatever; Change in these areas tends to come slowly if at all.

> guns should be regulated further

Perhaps they should, perhaps they shouldn't. I have my bias, as I'm sure does everyone engaged in the discussion, but, at the end of the day, the document of the highest possible authority suggests that the citizenry should have access to guns, and the only way to overturn that is by ratifying that document. Thankfully, the document itself offers prescriptive measures for such ratification, but at the end of the day, without a strong enough amount of support throughout the nation and states for such a measure to pass.

At present, such support does not exist, so our politicians play different games on what does or does not constitute "infringement".

In the past, the argument was that it wasn't infringing if the person who wanted to exercise gun rights wasn't a member of a militia. We can prove historically and grammatically that this isn't true, but it took the Supreme Court's agreement for the legislature to accept it.

Then, the argument was that sure, it's an individual right and not limited to those in the militia, but hey, the second amendment says you can keep and bear arms, but it doesn't say where, so instead, the government insists that the right is only applicable to within the home. Have as many guns as you like, and carry them anywhere you want, as long as it's inside your home.

It took Chicago v. McDonald getting that one overturned in 2012 by the Supreme Court.

So, legally speaking, we know that the right to keep and bear arms is an individual right, disconnected from membership in a militia, and it extends outside the home. To those who favor gun rights, the matter might settled, but to those who favor gun control, it's a challenge to come up with a new rule that doesn't directly contradict those recently affirmed rules by the Supreme Court.

In D.C., the argument is currently that the right to keep and bear arms is only available to those with a heightened need... but what other right in our Constitution merits such a high bar to exercise? Would it be okay to restrict voting only to those harmed by the current administration? Would it be okay to limit speech only to those who can show that they have been victimized?

Anyway, I'm obviously ranting, and there's certainly more of my bias in this post than I intended, so I should shut up. The point, that I'm sure I made poorly, is that the law here says that we can have firearms, and that not even the elected legislature has the power to overturn that without ratifying the Constitution... unless they can come up with something legally clever, or wait until there exists a more favorable court to their cause.

<comment><USER6> Nits:

 _In the past, the argument was that it wasn 't infringing if the person who wanted to exercise gun rights wasn't a member of a militia. We can prove historically and grammatically that this isn't true, but it took the Supreme Court's agreement for the legislature to accept it._

I don't see any legislators changing their mind about this, or any laws being passed based on it, although I might have forgotten some state ones. Except for:

 _Then, the argument was that sure, it 's an individual right and not limited to those in the militia, but hey, the second amendment says you can keep and bear arms, but it doesn't say where, so instead, the government insists that the right is only applicable to within the home. Have as many guns as you like, and carry them anywhere you want, as long as it's inside your home.

It took Chicago v. McDonald getting that one overturned in 2012 by the Supreme Court._

Errr, _McDonald v. Chicago_ just applied _Heller_ to the states and specifically Chicago, and further Federal court action, not appealed to the Supreme Courts, enforced a shall issue concealed carry regime on the state ... which I'll note for a blue state is much better balanced between the big cities and "downstate" for it.

Not counting California and Hawaii, where this is still being litigated, _every_ other Federal judicial challenge to a restrictive concealed carry regime has failed, and the Supremes have then denied cert, despite the circuit split. As it stands in the Federal courts, we now have a right to keep arms in our homes, and to bear them there (that was actually an issue litigate in _Heller_ and is elsewhere one, based on so called "safe storage" laws), but not outside our homes.

<comment><USER20> "I don't see any legislators changing their mind about this"

That's more to rebut the perennial tide of "Well, if you read the second amendment it clearly states that it belongs to a well regulated militia" argument that invariably crops up in lesser discussions on the subject.

"Errr, McDonald v. Chicago just applied Heller to the states and specifically Chicago"

Correct. I crossed it with Moore v. Madigan. It's late, and they're in the same jurisdiction, which I don't live in. Apologies for the error.

"Not counting California and Hawaii, where this is still being litigated, every other Federal judicial challenge to a restrictive concealed carry regime has failed"

And D.C., as Pena v. Lindley is being reheard en banc, and the new panel has stayed their previous injunction on enforcing "good and substantial"... otherwise, yes, agreed.

</comment></comment></comment></comment></comment><comment><USER21> Intresting note:

Nordic countries (Finland, Norway) and Swizerland rank very high in gun ownership among OECD countries. [http://www.nationmaster.com/country- info/stats/Crime/Gun-own...](http://www.nationmaster.com/country- info/stats/Crime/Gun-ownership/Guns-per-100-residents/2007)

<comment><USER22> Switzerland is biased because of the militan army approach. Every able-bodied Swiss male has an assault rifle at home, without any bullets, however.

<comment> at-fates-hands >> Switzerland is biased because of the militan army approach. Every able- bodied Swiss male has an assault rifle at home, without any bullets, however.

This was exactly the purpose of the text in the second amendment when they talk about having a "well regulated militia":

 _The phrase "well-regulated" was in common use long before 1789, and remained so for a century thereafter. It referred to the property of something being in proper working order. Something that was well-regulated was calibrated correctly, functioning as expected. Establishing government oversight of the people's arms was not only not the intent in using the phrase in the 2nd amendment, it was precisely to render the government powerless to do so that the founders wrote it._

source: [http://bearingarms.com/well-regulated/](http://bearingarms.com/well- regulated/)

Essentially this meant every citizen should be properly armed should they be called into duty for defending the country. Sure, a lot has changed since then, but most gun owners I know still take this wording very seriously and believe it is their duty to defend against government tyranny.

<comment><USER23> > Essentially this meant every citizen should be properly armed should they be > called into duty for defending the country. Sure, a lot has changed since > then, but most gun owners I know still take this wording very seriously and > believe it is their duty to defend against government tyranny.

I don't see how you get from point A ("well-regulated" means keeping in proper working order) to point B (the purpose of the militia was to defend against _government_ tyranny).

One of the motivating purposes of the Constitutional Convention was to respond to the failure of the state militias to put down armed rebellions _against the government_. Whether or not the second amendment guarantees the right to own personal firearms, the idea that the framers intended those firearms be used against the government is ludicrous. The most reasonable conclusion from the primary materials is that those well-regulated militias exist to put down those who would take up arms against the government!

<comment><USER6> Errr, how about Thomas Jefferson in 1789, after the Revolutionary War just to be clear? From Wikiquote:

 _God forbid we should ever be twenty years without such a rebellion. The people cannot be all, and always, well informed. The part which is wrong will be discontented, in proportion to the importance of the facts they misconceive. If they remain quiet under such misconceptions, it is lethargy, the forerunner of death to the public liberty. … What country before ever existed a century and half without a rebellion? And what country can preserve its liberties if their rulers are not warned from time to time that their people preserve the spirit of resistance? Let them take arms. The remedy is to set them right as to facts, pardon and pacify them. What signify a few lives lost in a century or two? The tree of liberty must be refreshed from time to time with the blood of patriots and tyrants. It is its natural manure._

But he was most certainly a firebrand, and you do have a point about motivations for our current Constitution if this book is to be believed: [http://www.upenn.edu/pennpress/book/13777.html](http://www.upenn.edu/pennpress/book/13777.html) (it's new (2003) and sounds revisionist, but that doesn't mean it's incorrect, just that as with anything historically political like it it has to be checked, see e.g. _Arming America_ ).

<comment><USER23> Thomas Jefferson was a lot of things, but one of the things he wasn't was at the Constitutional Convention. His writings evidence strains of thought that existed at the time of the founding, but to the extent that you can divine some sort of "intent" on the part of the 40 people who signed the Constitution, his writings do not clarify that intent.

</comment></comment></comment></comment></comment></comment><comment><USER24> I'm to the right of president Obama, own guns (yes plural) and yet agree with him whole-heartedly. We, as a nation, need to figure out how to stop this violence. Please don't paint all conservatives with the radical right-wing brush.

As an aside, I don't think Finland is celebrating in the streets with shouts of "we're number one".

</comment><comment><USER25> You know what Twain says - there are lies, damned lies, and statistics.

([https://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statist...](https://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statistics))

</comment><comment><USER26> Intentionally or not, the author has cherry-picked the data in a way that completely undermines the article. The conclusion that "lives are at stake" is obviously true and supported by the data, but the implication that the US has too many guns is not. An analysis is only as accurate as the data. Here, the missing data makes it painfully obvious that, while we should whatever we can to reduce gun violence, we cannot morally accomplish it by disarming US citizens.

If you merely look back to about 1939, you would find numbers for Poland alone that dwarf any statistics for the US, even ignoring differences in population! Poland isn't the only such example. Around that time, certain groups of people suffered from numerous horrific shootings, decimating their populations, despite those groups having been recently disarmed almost entirely. Of course, I'm being sarcastic. These people suffered at the hands of their own governments _because_ they were disarmed, not despite it. There are some still alive who remember.

An honest assesment of history, even very recent history, makes it clear that America's current level of gun violence is a mere rounding error next to the real threat: descent into tyranny. We're lucky that we live in such a relatively peaceful time, but let us not delude ourselves. No human will ever live without the threat of tyranny. No government of the people or by the people will ever, by it's own internal guidance, remain civilized indefinitely. We must never surrender the tools needed to provide the most drastic of course corrections for a government gone astray.

We are fortunate that by keeping and maintaining arms, we make it _more_ likely that we will never have to use them. Armed violence is always the absolute last resort, but if we give up our last resort, we give up all the other ones too. Speech, for example, is of little use against a tyrant with absolute power. But speech from the mouths of well-practiced riflemen, with their weapons close at hand? Those words cannot be safely ignored by anyone.

It is for this reason that the right to keep and bear arms is not unique to just the United States. Rather, it is a fundamental and irrevokable human right. The Constitution of the United States does not and cannot _grant_ this right -- though it does explicitly _recognize_ it, lest we forget.

Yes, gun violence is awful, and we should work hard to reduce it. But let us not forget that is is the price we pay for a civilized society, and it is a small price compared to the alternatives. I grow tired of articles that completely miss this point.

</comment><comment><USER27> Petzold is awesome. His blog has several articles he's written about his political views. I'm glad that someone with that kind of notoriety isn't afraid to write about politics.

<comment><USER28> I'm sorry, what notoriety?

<comment><USER29> He's known, amongst certain people in the industry - [https://en.wikipedia.org/wiki/Charles_Petzold](https://en.wikipedia.org/wiki/Charles_Petzold)

</comment></comment></comment><comment><USER30> The reason we care about mass shootings is because people die. Petzold drops the one variable we _care_ about and calls it de-obfuscation? Neat.

<comment><USER31> If you care about people dying then mass shootings aren't an important issue. (Compared to almost every other issue.)

</comment></comment><comment><USER32> Excellent article. I wonder what software was used for the computer simulation in the second part. Can one do that in R relatively easily ?

<comment><USER29> He mentions that it's a C#/WPF program that he wrote :) The article includes a link to download the code and play with it yourself: [http://www.charlespetzold.com/blog/2015/07/RampageShootingSt...](http://www.charlespetzold.com/blog/2015/07/RampageShootingStatistics.zip)

</comment></comment><comment><USER33> Spree killings/mass murders/rampage shooters/whatever are all a canard anyway. A canary in the coal mine. Add up all the spree killers in the US and they'll not break 1% of the murders. Just like the debate over automatic weapons is a canard, since they're very rarely used in murders.

The US has a murder problem. It's murder rate is four times that of it's industrialised contemporaries. Most of those murders (~75%) are done with firearms, and most of those (>75%, can't recall exactly) are done with handguns. And by someone who knows the victim closely.

The stereotype of the inner-city gangsta gunning down a gangland enemy with an automatic weapon is an outlier. And the same is true of the spree killer. As long as debates on either side center on these kinds of outliers, little will be achieved. The murder rate isn't going to be affected by open carry, since most murders aren't of strangers. And banning automatic rifles isn't going to do anything either, because it's handguns that do the dirty work.

The problem is that these details aren't sensationalist. Being gunned down by a stranger on a rampage makes the news. Being killed by a crazy relative doesn't.

<comment><USER20> I hate to be pedantic in discussions like this one, and please know that I certainly don't mean any offense beyond correcting a technical error that perhaps nobody really cares about.

"Just like the debate over automatic weapons is a canard, since they're very rarely used in murders"

There _is_ no debate surrounding automatic weapons, or, at least there isn't one on the national stage. Automatic firearms are dramatically different from semi-automatic firearms, and it's been a (specifically expressed) political maneuver to confuse/conflate the two.

The number of deaths attributable to automatic firearms is effectively nil, as they've been highly controlled since the days of Al Capone and his ilk.

When people are referring to "automatic firearms" in today's political arena, they almost always mean to say "semi-automatic firearms", which most closely maps to the sorts of weapon used in the Newtown and Aurora shootings.

It's a subtle, but important distinction, but as those who are trying to conflate the terms generally intend to do so to confer additional scariness to semi-automatic firearms that are simply not there.

<comment><USER34> "The number of deaths attributable to automatic firearms is effectively nil, as they've been highly controlled since the days of Al Capone and his ilk."

Well, since we're going to be pedantic, this should really say "the number of deaths attributable to _legally owned_ automatic firearms", since there is quite a bit of criminal violence (as in, violence to support criminal activities, like drive bys, rip deals resulting in shootings etc) and deaths involving automatic weapons (although it's quite hard to find statistics on this, to my surprise). MAC 10's and Mini Uzis are rather popular for that sort of shenanigans, because of their small size (thus easily concealable). An AR15 style weapon (which is usually the type of weapon people talk about when the discussion is about 'assault rifles') is very impractical for such a thing, as per your point.

<comment><USER35> > MAC 10's and Mini Uzis are rather popular for that sort of shenanigans, > because of their small size (thus easily concealable).

Since you have provided no citation, I can only judge this statement by my own anecdotal experience.

As someone who is closely involved with "gun culture", I'm aware of only a handful of cases where automatic weapons were confiscated in the US - and of no cases they were used in a murder in my adult lifetime.

<comment><USER6> You might be younger than me ^_^, but in my lifetime two legally civilian owned automatic weapons were used in crimes, both murders as I remember, one by a policeman, though. I can't recall any illegal ones being used in murders, but maybe some escaped my notice. Legal or illegal, they are demonstrably not a problem; in fact, the NFA of 1934 used them as a wedge to try to severely restrict handgun ownership, but that failed in the Congress.</comment></comment></comment></comment></comment>


Exploiting URL Parser in Programming Languages (2017) [pdf] - <AUTHOR> https://www.blackhat.com/docs/us-17/thursday/us-17-Tsai-A-New-Era-Of-SSRF-Exploiting-URL-Parser-In-Trending-Programming-Languages.pdf <comment><USER2> Am I correct to understand that the main exploit here is:

A) A server that makes HTTP requests. B) It does so based on a white or blacklist C) These white or blacklists are circumvented due to bad parsing D) This can be used to call unintended services (and sometimes protocols).

I had an idea to build a generic Webhook service. My thought was that the actual machine that does the requests should be 100% isolated from our VPS and only have public network access. Would that largely stop this attack? Perhaps it should also have a rate limiter to prevent DDOS attacks.

Waiting for every url parser to be fixed seems impracticable. This to me is a great example of why the WHATWG url specification is so terrible. It's so much harder to implement than RFC 3986.

Things that aren't browsers should just implement RFC3986 and reject anything invalid.

<comment><USER3> Check out the TLS SNI exploit. The attack allows embedding CRLF in a TLS payload. The entire handshake leading up to it is ignored by an SMTP server, and the next command is valid SMTP. Meaning that if you ask something to e.g. load an image from, or send a callback to "[https://smtp.targetcorp.com<space><CR><LF>...smtp](https://smtp.targetcorp.com<space><CR><LF>...smtp) instructions to send mail here...:25/" it'll actually work. As far as I understand this is the TLS handshake so even before any HTTP is done; post, get, whatever. All works.

Good moment to lock down outgoing requests, I guess. At least to port 25 :)

<comment><USER4> I've not communicated with SMTP servers manually all that much, but I remember my ISP's SMTP server would just close the connection if you sent it something it didn't understand.

<comment><USER5> That sounds like most SMTP servers - they just reject what they don't understand.

The wrinkle here that might be, in the opinion of some, worth noting is that this gives SMTP servers something they understand fully. The HTTP server has been convinced to send perfectly well-formed and valid SMTP commands.

</comment></comment></comment></comment><comment><USER6> Saw this talk at Defcon last year. Left the room .... pretty nervous ....

This isn't it, but same title and content [https://www.youtube.com/watch?v=D1S-G8rJrEk](https://www.youtube.com/watch?v=D1S-G8rJrEk)

<comment><USER7> Yet another one:

[https://www.youtube.com/watch?v=2MslLrPinm0](https://www.youtube.com/watch?v=2MslLrPinm0)

</comment></comment><comment><USER8> This doesn't seem designed to be read without the accompanying speech

</comment><comment><USER9> Just fuzz every parser you write. And the problem is solved.

<comment><USER10> I agree with the idea, but I don't think it is enough: A fuzzer does not necessarily find all problems.

<comment><USER9> I wrote several parsers that I fuzzed with tools like AFL or gofuzz. And I promise you that fix lot ugly corner cases.

</comment></comment></comment><comment><USER11> Wow this is brutal! I wonder how safe apache and nginx are?

</comment><comment><USER12> It was posted before [1] and thus I know the paper isn't from 2018. Should it be added to title?

[1] [https://news.ycombinator.com/item?id=14879488](https://news.ycombinator.com/item?id=14879488)

<comment><USER13> Added now. Thanks!

<comment><USER14> It was only a year ago. Would it be substantially different if written today?

<comment><USER13> It's just an HN convention to have the year in the title when the article isn't the current year. Not only does it not imply badness, it's associated with goodness, since most things from previous years have lost their interest.</comment></comment></comment></comment>


Don't Panic! Better, Fewer, Syntax Errors for LR Parsers - <AUTHOR> https://soft-dev.org/pubs/html/diekmann_tratt__dont_panic/ <comment><USER2> I have a somewhat controversial opinion on this.

This paper, like some others that I've seen, treats parsing as an academic/algorithmic topic. Given some token stream we'd like to minimize some error criteria. However, these papers ignore the fact that people don't write token streams: they write code that is formatted using whitespace. I don't know any programmer that writes their source code on a single line. On the contrary, pretty much every programmer formats their source code to some reasonable (if not complete) degree. In other words: parsing error recovery is not primarly an algorithmic problem: it's a usability problem.

Newlines and indentation are a source of extra information that we can use to infer what the programmer meant. Why throw all that away during tokenization? That's crazy! We can totally use it for error recovery/error message generation.

I decided to play around with this idea in the design my programming language Muon [1]. The language uses redundant significant whitespace for parser error recovery. This simple approach by itself has turned out to work surprisingly well! In my admittedly biased opinion, it frequently surpasses the error recovery quality in mature compilers such as the C# compiler, which has had tons of effort poured into error recovery.

Of course, a purely whitespace based recovery scheme is not perfect: there are rough edges, like having to deal with tabs vs spaces, and recovering inside a line is not usually possible. But the fact that such a simple approach has led to good results makes me think this would be a great area for future research, that can perhaps combine the best of both worlds here.

[1] [https://github.com/nickmqb/muon](https://github.com/nickmqb/muon)

<comment><USER3> An even bigger flaw with most academic research into parser error recovery is that the vast majority of syntax errors occur from modifying a valid program to produce an invalid program, but the recovery algorithms are oblivious to this.

<comment><USER4> Have a look at this writeup by the author of lezer, if you haven't already: [https://marijnhaverbeke.nl/blog/lezer.html](https://marijnhaverbeke.nl/blog/lezer.html)

<comment><USER5> tree-sitter is excellent stuff! It's heavily inspired by Tim Wagner's PhD thesis (original site seems to be down, but [https://web.archive.org/web/20150919164029/https://www.cs.be...](https://web.archive.org/web/20150919164029/https://www.cs.berkeley.edu/Research/Projects/harmonia/papers/twagner- thesis.pdf) works). IMHO more people should know about that work, and the sequence of work from Susan Graham's lab that led up to it. We have also been heavily inspired by Tim's work and Lukas's thesis extends and updates a number of aspects of that seminal work including, in Chapter 3, error recovery ([https://diekmann.co.uk/diekmann_phd.pdf](https://diekmann.co.uk/diekmann_phd.pdf)).

All that said, it's surprisingly difficult to compare error recovery in an online parser (i.e. one that's parsing as you type) to a batch parser. In the worst case (e.g. load a file with a syntax error in), online parsers have exactly the same problems as a batch parser; however, once they've built up sufficient context they have different, sometimes more powerful, options available to them (but they also need to be cautious about rewriting the tree too much as that baffles users).

</comment></comment></comment></comment><comment><USER5> Co-author here. If you want to quickly play with this on the command-line with your favourite Yacc grammar, a simple way is to use nimbleparse [https://crates.io/crates/nimbleparse](https://crates.io/crates/nimbleparse) ('cargo install nimbleparse' should do the trick; though note that you will probably need to munge your lexer a bit). You can see this in use in the paper itself: the examples at the end are direct output from nimbleparse [https://github.com/softdevteam/error_recovery_paper/tree/mas...](https://github.com/softdevteam/error_recovery_paper/tree/master/examples). Some example grammars are at [https://github.com/softdevteam/error_recovery_paper/tree/mas...](https://github.com/softdevteam/error_recovery_paper/tree/master/examples) if you want to get going quickly.

If instead you want to use the full set of Rust libraries, a good place to start is [https://softdevteam.github.io/grmtools/master/book/quickstar...](https://softdevteam.github.io/grmtools/master/book/quickstart.html).

</comment><comment><USER6> Awesome. I love good research into parsing, since parsing is famously "the solved problem that isn't": [https://tratt.net/laurie/blog/entries/parsing_the_solved_pro...](https://tratt.net/laurie/blog/entries/parsing_the_solved_problem_that_isnt.html)

I hope to study this paper well (which is by the author of the above post) and understand what their algorithm is. I feel that good error messages make or break a language.

Here is the arxiv link of the same paper: [https://arxiv.org/pdf/1804.07133.pdf](https://arxiv.org/pdf/1804.07133.pdf)

<comment><USER7> In theory it's "the solved problem that isn't", in practice it is --- there's a reason the compilers for the most popular programming languages either moved to or have always been recursive-descent with perhaps a bit of operator- precedence/precedence-climbing to handle the repetitive cases of binary operators. GCC, Clang/LLVM, Java, and C# are the ones that immediately come to mind; and no doubt there are plenty of others.

<comment><USER8> There are _way_ more parsers out there in production than the top-ten programming language implementations. Just think about serialization formats (json, bson, protobuff, et.al.): There are maybe 10 or so widely used. Then think about message formats (http, xml, yaml, ...): Maybe another 10 or so? Then think about configuration languages (.ini, .toml, ...), let's call that 5?. On top of that we have a bunch of widely used scripting languages (Make, sh, various lisps), I would say again 5 or so.

So for any new language, say rust, you want (ideally) parsers for about 30 or more languages in real-world use cases. And that does _not_ include language prototyping or consider the fact that recursive-descent parsers are usually not available as libraries (say for use in an editor).

In conclusion: Having good automatic parser generators readily available is very relevant. And "good" can obviosly include usability, IMO.

</comment></comment></comment><comment><USER9> I've thought about doing something like this when using matching/parsing with derivatives[1]. When the graph collapses to ∅ (indicating an error), you could back up one character, then ask the graph "what are the next valid token(s)?" This would give you a list of possible things to insert (`,` and `=` for the example given in the article, though naively implementing this would also suggest stuff like `;`).

But this article takes it a step further and suggests deletions, too. That's actually really cool. My biggest gripe with people making DSLs is the crappy error messages you get when you have a syntax error (GCL, I'm looking at you). Hopefully research like this will improve the lives of those of us that have to use DSLs.

[1]: [http://matt.might.net/articles/parsing-with- derivatives/](http://matt.might.net/articles/parsing-with-derivatives/)

</comment><comment><USER10> Most grammars define a very specific starting point for languages. This is fine of course, but what if we want to parse, say, only an expression, only a function body, etc?

I feel most parsing APIs tend to expose a single method, something like:

        parser.parse(source)

... which is kind of a limited API to do the things I ask about above.

With a more flexible API, an structure aware editor [1] could keep different areas of the buffer separated, and know if the user is typing a function body, a variable definition, etc, and only parse that part w/o breaking the rest of the parser progress (is this a good idea? I think so but I don't think most editors work like this, so maybe not :-)

\--

An area of parser APIs that I've recently seen trouble with is lack of metadata on the productions. I'd like to be able to do something like:

        production.meta() => { :line 23, :column 10 }

... or something like that. This is very useful while error reporting, for instance. I was trying to find a parser for RDF Turtle that would do this, but couldn't find any!

1: [https://en.wikipedia.org/wiki/Structure_editor](https://en.wikipedia.org/wiki/Structure_editor)

<comment><USER8> You can always start from a different place even with an LR(k) parser. You just need to make sure that your starting point is sensible.

</comment></comment><comment><USER11> In the first figure, it gives "int x y;" and says that the new algorithm gives the complete set of minimal cost repair sequences, which is in this case "Delete y", "Insert ,", and "Insert =".

I haven't read the article, so maybe this is covered later, but why isn't "Delete x" included?

<comment><USER12> Just a guess: "int x" could already be comsumed so there is no need to change anything about it. It's only the next token that can not be processed.

<comment><USER8> Intriguing, so if we design a language that allows for:

int single x;

and

int x y z;

The error for

int single x y z;

Would be to delete "y" and "z", but not to delete "single"?

</comment></comment></comment><comment><USER13> Trying to repair bad syntax by editing the tokens stream is decades obsolete. It was an important technique when compiling had long turnaround times which included having to stand in line to submit a deck of punched cards to a clerk in a job submission window. Naturally, programmers wanted to catch as many errors as possible in a single round trip.

That said, it could be useful again, for an IDE to get the most amount of information (for completion and whatnot) even in the face of a bit of bad syntax.

The technique opens up possibilities for the compiler to get into a bit of a loop generating a large quantity of error messages, most of which result from its repair attempts. If any of the recovery strategies insert tokens, there is a potential getting into an infinite loop, if the code isn't careful.

In the 1970's, _Creative Computing_ magazine held a contest for producing the longest ream of error messages from a single error in a program, or something like that. That would have been fueled by error recovery.

</comment><comment><USER8> That is a very well-written paper and an interesting topic! Kudos to the authors! I'd really like this to turn into a thesis and further research.

Edit: it also follows the de-facto standard for the naming of cs-papers, which demands a pun followed by a quick explanation. Brilliant!

I also just saw that it might be too late for the thesis part. A pity.

<comment><USER14> > the de-facto standard for the naming of cs-papers, which demands a pun > followed by a quick explanation

I'm reminded of the following wordplay. (I'm ashamed to say I was rather slow to catch the pun when I first heard it.)

 _The aim of the semantic web is to save the world._

</comment></comment><comment><USER15> > A more grammar-specific variation of this idea is to skip input until a pre- > determined synchronisation token (e.g. ';' in Java) is reached [8 , p. 3], > or to try inserting a single synchronisation token. Such strategies are > often unsuccessful, leading to a cascade of spurious syntax errors (see > Figure 1 for an example). Programmers quickly learn that only the location > of the first error in a file – not the reported repair, nor the location of > subsequent errors – can be relied upon to be accurate.

      C.java:2: error: ';' expected
        int x y;
             ^
      C.java:2: error: <identifier> expected
        int x y;
               ^

This is a common problem, but the solution here for the example shown is for the parser _not_ to attempt to recover that statement and _ignore_ everything until it reaches the "synchronization point" (I call them "landmarks"), which sounds like what their were arguing for at first. Doing that would make the AST be marked with a node along the lines of "some int named x with errors", and would completely _skip_ everything between the `x` and the `;`. Because an error has been reported, there will be no output binary, but the second error will _not_ be emitted.

Beyond the parser, the AST needs to keep around a lot of extra information for error recovery that would otherwise not be needed. An example from rustc[1] (what I'm familiar with) is structs and struct variants that have had some parse error. We recover the parse and keep a node for the variant in the AST signaling the existence of a parse error, which means that later passes of the compiler can type and borrow check, but if it finds a _use_ that is either missing or _adding_ fields we _do not_ emit an error for it[2][3] under the assumption that the parse error caused the field being used to not be captured in the AST node for the variant. This can have quite dramatic impact on the quantity of errors being emitted.

[1]: [https://github.com/rust- lang/rust/blob/7e11379f3b4c376fbb9a6...](https://github.com/rust- lang/rust/blob/7e11379f3b4c376fbb9a6c4d44f3286ccc28d149/src/librustc_middle/ty/mod.rs#L1969-L1971)

[2]: [https://github.com/rust- lang/rust/blob/7e11379f3b4c376fbb9a6...](https://github.com/rust- lang/rust/blob/7e11379f3b4c376fbb9a6c4d44f3286ccc28d149/src/librustc_typeck/check/expr.rs#L1292-L1295)

[3]: [https://github.com/rust- lang/rust/blob/7e11379f3b4c376fbb9a6...](https://github.com/rust- lang/rust/blob/7e11379f3b4c376fbb9a6c4d44f3286ccc28d149/src/librustc_typeck/check/pat.rs#L1085-L1094)

<comment><USER5> I think we are in agreement that skipping over large chunks of input is rarely a good idea. In Section 7 ("Using error recovery in practice") we show how you can make fine-grained decisions about what to do when recovery has happened in our Rust parsing system. The more fine-grained you go, the more code you have to write, but it allows you to do exactly the sorts of things that rustc does with structs if you want. [https://softdevteam.github.io/grmtools/master/book/errorreco...](https://softdevteam.github.io/grmtools/master/book/errorrecovery.html) is a more approachable version of the same stuff.</comment></comment>


Why Software Doesn't Follow Moore's Law - <AUTHOR> http://www.forbes.com/sites/ciocentral/2014/05/19/why-software-doesnt-follow-moores-law/ Thoughts? <comment><USER2> In some ways it does. I am working on a Database system in Django by myself. Higher level languages and frameworks will eat at performance in terms of speed, but give a true developer advantage compared to not using them. 10 years ago it would probably have taken a team of 5 to achieve the same thing. Depends what you are measuring.

Go back 30 or so years and people were using punch cards. Do you think it would be possible to "knock up a quick web app" programming like that?</comment>


China's new 'social credit system' turns Orwell's '1984' into reality - <AUTHOR> https://nypost.com/2019/05/18/chinas-new-social-credit-system-turns-orwells-1984-into-reality/ <comment><USER2> You get denied from jobs, school, and are essentially outcast from society because of your low score and then sent to a re-education camp because your score is too low and likely to commit a crime.

After destroying a person's livlihood, friends, and family...why wouldn't they resort to crime? It will most likely be the only way to survive.</comment>