Structure and Interpretation of Classical Mechanics (2015) - <AUTHOR> https://mitpress.mit.edu/sites/default/files/titles/content/sicm_edition_2/toc.html <comment><USER2> Sussman is one of the very few classical mechanics textbooks that gives a reasonable definition of the Legendre transform. Most physicists cannot actually tell you what that transform is, even though it sits at the heart of both classical and quantum mechanics.

[http://blog.jessriedel.com/2017/06/28/legendre- transform/](http://blog.jessriedel.com/2017/06/28/legendre-transform/)

<comment><USER3> The description in this article is great, but the why is still rather mysterious. How would somebody come up with that?

If you are familiar with the method of Lagrange multipliers, then what's happening can be explained as follows. Given the Lagrangian L(x,v) the problem of classical mechanics is to find a trajectory x(t),v(t) that extremises the integral of L(x(t),v(t))dt under the constraint x'(t) = v(t). Lagrange multipliers are a method to deal with constraints in optimisation problems. Usually it's taught in the finite dimensional case, but it also works in the infinite dimensional case. We introduce a Lagrange multiplier p(t) and add the constraint to the objective: integral of L(x(t),v(t)) + p(t)(x'(t) - v(t)) dt. To solve the problem we minimise this over x,v,p. If we carry out the minimisation over v first then we're left with two variables x,p. That's the Hamiltonian formulation of the problem, and it's called the dual problem in convex optimisation. So the momentum p is the Lagrange multiplier for the constraint x' = v.

In more detail: we rewrite L(x(t),v(t)) + p(t)(x'(t) - v(t)) = L(x(t),v(t)) - p(t)v(t) + p(t)x'(t). Now we separate out H(x,p) = min_v L(x,v) - pv, so the original problem becomes to minimise the integral of H(x,p) + p(t)x'(t). After applying the Euler-Lagrange equations we obtain Hamilton's equations:

dH/dx = dp/dt

dH/dp = -dx/dt

<comment><USER2> Agreed, thanks. That blog post was just trying to explain _what_ it is, not why it is, mostly as a basis to complain about education rather than teach physics.

<comment><USER4> The why is actually quite simple- and the same reason the Fourier transform is sometimes used. Some problems are simply more elegantly expressed in a particular basis system. Nobody tries to express a ball's motion in flight via Fourier analysis, but you certainly could.

In the same way, sometimes solutions for the position(s) of a system is the most natural basis system for describing/investigating a problem (use the Lagrangian) and sometimes solutions for its momentum are (use the Hamiltonian).

Of course, they're intrinsically linked since the evolution of one determines the evolution of the other.

</comment></comment></comment></comment><comment><USER5> You can play with the software online here: [http://io.livecode.ch/learn/namin/scheme- mechanics](http://io.livecode.ch/learn/namin/scheme-mechanics)

The notation chapter with live snippets is reproduced here: [http://io.livecode.ch/learn/namin/scheme- mechanics/chapter9](http://io.livecode.ch/learn/namin/scheme- mechanics/chapter9)

</comment><comment><USER6> At first glance, this looks like an excellent textbook.

Personally, my favourite theorem in classical mechanics is the so called 'tennis racket theorem', sometimes known as the 'intermediate axis theorem'.

It explains why objects with roughly three different moments of intertia have unstable rotation about their intermediate moment.

It can be easily demonstrated with a tennis racket, or even most smartphones (be careful not to break it though).

[https://en.wikipedia.org/wiki/Tennis_racket_theorem](https://en.wikipedia.org/wiki/Tennis_racket_theorem)

<comment><USER7> There's a link on that wikipedia page to this youtube demo which makes it really obvious

[https://www.youtube.com/watch?v=4dqCQqI- Gis](https://www.youtube.com/watch?v=4dqCQqI-Gis)

</comment></comment><comment><USER8> Here is a link to the unofficial html edition : [https://tgvaughan.github.io/sicm/toc.html](https://tgvaughan.github.io/sicm/toc.html)

<comment><USER9> I came to say how horrible the formatting of the original post is. But this site fixes that. Much better! Thanks!

</comment></comment><comment><USER10> I love this book! Prior to reading it, I had been getting confused when trying to learn classical mechanics. The book writes out everything explicitly in code, which let me use the software engineering part of my brain, and made everything easy to follow. Apparently I had been being held back by unfamiliarity with math formalisms!

If you're interested in the topic and know how to program, it might be worth a read in case it turns out you're in the same boat.

</comment><comment><USER11> Apparently written by the same person who wrote Structure and Interpretation of Computer Programs - Gerald Jay Sussman .

<comment><USER12> Who also wrote "Functional Differential Geometry": [https://mitpress.mit.edu/books/functional-differential- geome...](https://mitpress.mit.edu/books/functional-differential-geometry)

</comment></comment><comment><USER13> For the curious, 2015: [https://news.ycombinator.com/item?id=9560567](https://news.ycombinator.com/item?id=9560567)

2013: [https://news.ycombinator.com/item?id=6947257](https://news.ycombinator.com/item?id=6947257)

2010: [https://news.ycombinator.com/item?id=1581696](https://news.ycombinator.com/item?id=1581696)

</comment><comment><USER14> Unfortunately, the code in this book uses a modified version of scheme which is no longer maintained.

<comment><USER15> Looks like MIT Scheme (on which the ScmUtils system runs) still gets some regular maintainence releases, though I don't know how active development is these days:

[https://www.gnu.org/software/mit-scheme/](https://www.gnu.org/software/mit- scheme/) [https://ftp.gnu.org/gnu/mit- scheme/stable.pkg/](https://ftp.gnu.org/gnu/mit-scheme/stable.pkg/)

Installation instructions for the ScmUtils package:

[http://groups.csail.mit.edu/mac/users/gjs/6946/index.html](http://groups.csail.mit.edu/mac/users/gjs/6946/index.html)

As far as I know, Gerry and Jack still teach the course every year (can someone currently at MIT verify?) and still use the system.

<comment><USER16> Yep! [https://groups.csail.mit.edu/mac/users/gjs/6.945/](https://groups.csail.mit.edu/mac/users/gjs/6.945/)

I'm in his group at MIT.

<comment><USER15> Good to hear! Please say hi to everyone. I used to hang out in the group back when it was still at Tech^2.

</comment></comment></comment></comment><comment><USER17> I haven't read beyond the first two chapters, but the clarity and expressiveness of the notation used is amazing.

I do wish more maths/physics etc would adopt similar, clearer, unambiguous notations.

</comment><comment><USER18> What are the books you need to read, before this book, so it makes sense?

I'm an EE, but I admit I made no progress. Clearly I lack some deeper understanding - where can I best look to fix that?

<comment><USER19> It's not a book that I would recommend but the lectures from a course: [https://theoreticalminimum.com/courses/classical- mechanics/2...](https://theoreticalminimum.com/courses/classical- mechanics/2011/fall)

This should give you plenty of intuition to then tackle SICM.

</comment></comment><comment><USER20> Excellent! Good companion to this is Richard Fitzpatrick's Computational Physics notes

[http://farside.ph.utexas.edu/teaching/329/329.html](http://farside.ph.utexas.edu/teaching/329/329.html)

</comment><comment><USER21> > _We hope others, especially our competitors, will adopt these methods, which > enhance understanding while slowing research._

:)

</comment><comment><USER22> Is there a PDF-version?

</comment><comment><USER23> Is there something similar for Quantum Mechanics? Both MIT and Harvard use Griffith's "Introduction to Quantum Mechanics," but it seems to emphasize computation and symbol manipulation over physical intuition.

<comment><USER24> There's [1], also by Sussman, which is "just" about differential forms and related math, and in the summary they refer to QM: "An explanation of the mathematics needed as a foundation for a deep understanding of general relativity or quantum field theory."

[1] [https://mitpress.mit.edu/books/functional-differential- geome...](https://mitpress.mit.edu/books/functional-differential-geometry)

<comment><USER25> Having read through that it's not really directly relevant to quantum physics (at least what I learned in undergrad). It is super applicable to general relativity and electrodynamics though! It's a much shorter book than SICM but I still have yet to understand it through to the end. Tricky stuff, but fun to think about!

</comment></comment></comment><comment><USER26> "If you can't explain it simply, you don't understand it well enough," followed by "For example, we write Lagrange's equations in functional notation as follows: D(∂2L ∘ Γ[q]) − ∂1L ∘ Γ[q] = 0" Whoa Nellie! Gonna get bumpy.

</comment><comment><USER27> Interesting -- I'm super rusty on my physics -- is there discussion of the exercises anywhere? It'd be fun to go through this and feel like I "got" it.</comment>


Automata in the Ancient World (2007) - <AUTHOR> http://cabinet-of-wonders.blogspot.com/2007/07/automata-in-ancient-world.html <comment><USER2> Hero's robot video is interesting

[https://www.youtube.com/watch?v=xyQIo9iS_z0](https://www.youtube.com/watch?v=xyQIo9iS_z0)</comment>


Kroes Throws in Towel on ACTA - <AUTHOR> http://blogs.wsj.com/tech-europe/2012/05/04/kroes-throws-in-towel-on-acta/?mod=google_news_blog

<comment><USER2> The conference where she delivered the speech was at the re:publica in Berlin (<http://re-publica.de/12/>). I've seen her talk in entirety on Friday, and I was deeply impressed with her opinions. Usually, when politicians (particularly older politicians) talk about Internet topics, their opinion is mingled with half-knowledge and the hope to enslave the internet and make it a read-only thing. However, Kroess' speeach was fantastic and it made me almost happy to see such a level of understanding of the importance of the internet, and the questions, dangers, and opportunities it raises for the future to come, in a politician. Especially someone who has such an important position in the EU.

<comment><USER3> She's great (or at least the dutch think she is), and did a good job as well when she was commissioner on competition, handing out some big fines to business cartels. see

[http://www.ft.com/cms/s/0/a854010e-dd84-11db-8d42-000b5df106...](http://www.ft.com/cms/s/0/a854010e-dd84-11db-8d42-000b5df10621.html)

I think we first were a bit disappointed that she was no longer keeping businesses on the straight and narrow, when she moved from competition to the digital agenda: it seemed like she was rearranged to a position where she would be less dangerous. In retrospect, that impression might have been wrong, and she might have just decided that keeping the digital world from being corrupted (by roaming fees, broadband prices, copyright agreements) was a more important thing to go after now than business cartels :-).

<comment><USER4> I wasn't so happy about her work as competition commissioner. They went very far in the direction of price fixing. For instance, the EC tells telcos what to charge for roaming even though no one can claim that the telecom sector isn't very competitive. It's cut throat actually.

<comment><USER3> I understand the the telco's are very competitive, but for some reason, free market competition appeared to fail to have an effect on prices like texting or data roaming. Oddly enough, you roam through various network using the internet, and there free market seems to have worked between fiber providers into keeping the prices low and efficiency high.

It seems that Kroes only launched the inquiry into telco's as part of her commision on competition. The price regulations came after her time there (see [http://en.wikipedia.org/wiki/European_Commission_roaming_reg...](http://en.wikipedia.org/wiki/European_Commission_roaming_regulations))

<comment><USER4> The price of something being "high" doesn't mean there is a market failure. It may just be an expression of customer preferences. If high roaming fees are used to subsidise cheap local calls and data plans, that may be a problem for EU officials who travel a lot, but it may be exactly what regular folks want.

The job of the competition commission is to make sure there is real competition, but what they have done is plain old price fixing. In this case, their actions make low income people subsidise the mobile life-style of high income people.

<comment><USER3> Isn't it actually the reverse?: don't the EU actions actually stop the previous practice of Telco's to subsidize the calling behavior of the majority by adding additional fees to a smaller group that heavily uses roaming?

I'm not sure what the free-market theory on that is (it is an interesting topic though) - making a smaller group of customers pay (and have limited alternatives but to pay) to be more competitive in other parts of your market.

<comment><USER4> Yes, I think that's exactly what happened before the prices were fixed and it's a completely normal facet of markets. You could make a case that governments should protect minorities. But I see a couple of issues in this case:

The competition commission doesn't have a legal mandate to give preferencial treatment to minorities. They have a very clear mandate and they should stick to it in the interest of seperation of concerns.

The particular minority concerned in this case is a rather wealthy one that benefits from free markets in other respects more than the average person. They don't need extra help from governments.

Helping minorities by fixing individual prices tends to be inflexible because any change in the underlying economics or social culture has to be to be compensated with new regulation which often doesn't happen in a timely fashion.

Preferencial treatment for minorities should, in my view, focus on life and death issues, freedom of expression, legal matters, etc.

</comment></comment></comment></comment></comment></comment></comment><comment><USER5> Thats great. But I smell a slight of hand trick here. Lead the audiences attention away while you stuff cards down your sleeve.

Stay wary, stay vigilant.

<comment><USER6> I agree. This cat and mouse game between lawmakers and the people is quite exhausting. If it isn't SOPA, it's PIPA, CISPA, ACTA... and they'll just keep coming until one of them makes it into law.

I think we should be focusing our power into pushing for legislation that prevents these kinds of bills to be passed, rather than fighting them off time and time again.

<comment><USER7> The only way to limit what congress can do is with a constitutional amendment. Anything else, and they can just overwrite it.

<comment><USER8> Yes and no. It's true that you can't just have Congress pass a law that says "future Congresses can't do xyz," because future Congresses can just repeal it.

But you can do something even better. Pass legislation that takes a step in the other direction. Repeal the DMCA anti-circumvention clause and demand an international treaty that requires other countries to do likewise. Reduce copyright terms. Eliminate statutory (as opposed to actual) damages for noncommercial infringement.

This would do three things. First, all of those things are good policy, and passing them would help us. Second, it would put the copyright extremists on the defensive and makes them spend their political capital to try to prevent a series of sensible bills from being passed. And third, when we succeed in passing them, it shifts the status quo, so that the next outrageous bill they propose looks like an even larger departure from the baseline, making it easier to defeat.

<comment><USER9> All of this is predicated on having a government that makes wise policy choices as opposed to acting in the self-perceived interests of its sponsors. The USG is not such a government.

<comment><USER8> They say you get the government that you deserve. If you don't think you deserve what you have, maybe you should get something else.

<comment><USER9> I'm Canadian; I'm more or less happy with our government. It's about the best you could hope for of a product designed by a committee of 15M.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> CISPA is way worse and its backed by 800+ massive corporations:

[http://www.digitaltrends.com/web/cispa-supporters- list-800-c...](http://www.digitaltrends.com/web/cispa-supporters- list-800-companies-that-could-help-uncle-sam-snag-your-data/)

So they can talk all they want... its still the same kind of thinking in the system. Dirty old men wanting to get more power.

</comment><comment><USER11> Way to go, Europe and elsewhere! Don't let Hollywood screw you over like they continually screw us over in the States.

</comment><comment><USER12> Amazing that it ends with:

"The controversial agreement, which opponents say was conducted behind closed doors, aimed to fight against counterfeiting at international level through greater co-ordination of anti-counterfeiting measures and tougher enforcement."

WSJ is a great place for propaganda and framing it wrong.</comment>


Show HN: Turn form submissions into emails (without the spam) - <AUTHOR> http://www.mailthis.to <comment><USER2> How's this better/different then Formspree ([https://formspree.io/](https://formspree.io/)) or Sent API ([http://sentapi.com](http://sentapi.com))?

</comment><comment><USER3> How do you prevent form submission spam?

<comment><AUTHOR> Thanks for asking! There's a verification step (one email) for new destination email addresses.

<comment><USER3> Oh, but that does not prevent the filling of the form by spambots after it is online and functional.

At Formspree we struggle every day against this problem, and it seems very difficult to solve while keeping a simple API.

Also, displaying the email in the action attribute automatically adds your email to spam lists and you're doomed forever (unless you use Gmail, they have pretty good antispam technologies that cannot work outside of their ecosystem). I know you are providing an alias so people do not have to fall into this, but you should be more clear about it instead of saying aloud that your service is spam-free.

<comment><AUTHOR> You guys have a really awesome product, good work!

I think I misinterpreted your question as "How do you prevent the form API from being used to send unwanted email", not email filtering. We also apply a spam filter to outgoing email requests, but as you stated, it's a challenge to balance that with usability.

Thanks for your feedback!

</comment></comment></comment></comment><comment><AUTHOR> Let me know if you have any questions!

</comment><comment><USER4> This looks for very good. Thanks</comment>


Bad JSON Parsers - <AUTHOR> https://github.com/lovasoa/bad_json_parsers <comment><USER2> The JSON spec [1] clearly states, in section 9:

      An implementation may set limits on the maximum depth of
      nesting.

So, this is kind of a pointless exercise. The author would do well to inform themselves better before slinging epithets such as "bad" for a library.

[1]: [https://tools.ietf.org/html/rfc7159](https://tools.ietf.org/html/rfc7159)

<comment><USER3> I would be perfectly OK with that answer _if_ they documented that they use the C stack and therefore have a small finite depth limit, and/or threw some sort of TooMuchNesting exception before they hit that limit.

AFAICT, many/most of these libraries do neither of the above. There's nothing in the documentation to suggest they'll have trouble with deeply nested inputs, and their response to such a situation is to simply crash the process. I'd call that a bug.

I don't consider it acceptable for a library to simply say "the spec says we're technically allowed to 'set limits' on what we accept, so here's some nasal demons". If you've got limits, you've got to document them, or at least make them recoverable.

<comment><USER4> It's hard for a library to portably know when the C stack will exhaust or give a concrete estimate that makes sense for documentation purposes.

Firstly, that concept is not exposed at all to standard complaint C code. The standard is vague enough to allow for many possible variations of the program stack abstraction. So you would not be portable, standard C.

Second, the library author doesn't know ahead of time how much stack space _you_ , the caller, have consumed before calling the library. Do you call it deep inside some ui framework with a big stack? With a huge i/o buffer on the stack? They don't know.

Translating all of this into "this is how deeply nested your json can be" is not a sane exercise, at best you can give a machine-specific guess that will vary a bit with your own application's stack needs.

But perhaps one can side-step all of that and get most of the way with an artificial, arbitrary-constant runtime depth check that is safely below likely actual stack limits.

<comment><USER3> Sure, that's all true. So don't use the C stack for parsing. It's not difficult.

</comment></comment></comment></comment><comment><USER5> It would've been interesting to know which platform was the JavaScript test performed on (judging by source code, likely Node.js). However, on my current machine Chrome chokes to stack overflow above nesting level 125,778.

More amazingly, Firefox is currently happily parsing a JSON array with 50,000,000 levels of nesting at full CPU single core utilization and consuming about 12 GB of memory. This was after an array of depth 10,000,000 was successfully parsed after peaking around ~8 GB of RAM. So Firefox's parser seems to be only constrained by available memory!

Also - while this is running in the console the Firefox UI is completely usable and I can browse the web in another tab. Mozilla has come a long way!

E: The 50,000,000 nesting level array parsing eventually failed with "Error: out of memory".

<comment><USER6> Great. Is anything else usable too?

<comment><USER5> Pretty much, but RAM utilization is of course closing at 100% so starting any new programs would fail on OS level.

</comment></comment></comment><comment><USER7> My takeaway from this is that you can submit a long string of brackets to REST apps built with Haskell and Ruby/Oj and crash the process and/or machine by consuming all available RAM. Whereas the smarter JSON processors near the top of the list will abort processing pathological inputs before serious harm occurs.

Wait, which JSON parsers are supposed to be "bad"?

<comment><USER8> Just because a structure is deeply nested doesn't mean that it consumes a lot of memory.

And just because your payload is massive and consumes a lot of memory doesn't mean it's deeply nested.

I doubt any of the parsers are resilient against storing large documents in memory. What makes them bad is that structures that would easily fit in memory are unparsable.

In most cases your web server imposed a limit on the size of the HTTP payload so you wouldn't normally be able to deliver a 128GiB JSON file anyway. It's not like nested structures are zip-bombs.

<comment><USER7> My comment was mostly tongue-in-cheek but there's some truth to it. Since the receiver creates a navigable data structure with 64-bit pointers, every two bytes sent ("[]") becomes 20+ bytes in RAM (low estimate!). An order of magnitude magnification helps.

</comment></comment></comment><comment><USER9> "The tar archive spec does not specify an upper limit on archive size, but my kernel keeps saying 'no space left on device'. It's obviously a bad kernel."

<comment><USER10> On one hand, yes. On the other hand, the standard Ruby "json" gem can be made to overflow on just 202 bytes of valid JSON input.

<comment><USER11> I mean, it'd be a pathological 202 byte JSON, but yeah (I guess it could be a DOS attack of some kind, actually, hmm...)

Ruby is a good example, because the `oj` gem, on presumably the same system, is listed as "infinite." Obviously not truly "infinite", eventually it'll run out of RAM -- but this shows it is _not_ an issue of machine resources really.

As the OP notes, it's because if you implement using recursion, you'll run out of stack in most languages (now let's start talking about tail call optimization...), but this isn't the only implementation choice, you _can_ implement to not have this limitation, and probably should.

<comment><USER12> Tail call optimization wouldn't help with parsing JSON.

<comment><USER13> Usually the solution is to create an array on the heap and manage the stack yourself.

</comment></comment></comment></comment></comment><comment><USER14> The JSON:XS documentation has this to say about the $max_depth field setting for the parser:

        Second, you need to avoid resource-starving attacks. That means you should
        limit the size of JSON texts you accept, or make sure then when your
        resources run out, that's just fine (e.g. by using a separate process that
        can crash safely). The size of a JSON text in octets or characters is
        usually a good indication of the size of the resources required to decode
        it into a Perl structure. While JSON::XS can check the size of the JSON
        text, it might be too late when you already have it in memory, so you
        might want to check the size before you accept the string.

        Third, JSON::XS recurses using the C stack when decoding objects and
        arrays. The C stack is a limited resource: for instance, on my amd64
        machine with 8MB of stack size I can decode around 180k nested arrays but
        only 14k nested JSON objects (due to perl itself recursing deeply on croak
        to free the temporary). If that is exceeded, the program crashes. To be
        conservative, the default nesting limit is set to 512. If your process has
        a smaller stack, you should adjust this setting accordingly with the
        "max_depth" method.

So accepting an unlimited depth structure by default is probably not a good idea. There's a flag you can set if you need to parse something like that, but the defaults are set to parse anything sensible and reject documents that may be malicious.

<comment><AUTHOR> My takeaway would be "using the C stack to parse JSON is probably not a good idea", not "accepting an unlimited depth structure by default is probably not a good idea".

This limit is entirely artificial, and due only to the design of the parser.

As the documentation you quote says, you should limit the size of JSON texts you accept. If you don't, then you are at risk of a memory exhaustion attack. But this is completely unrelated to the maximum depth you accept. If your parser doesn't use the C stack to parse recursive structures, then deeply nested payloads will parse just fine, without using an abnormal amount of memory.

<comment><USER15> I disagree with your conclusion. I would instead say that the _combination_ of using the stack to parse JSON and allowing unlimited depth is the problem. In practice, almost no one ever deals with JSON structures even a hundred levels deep (I'd quite seriously guess it to be less than one in a million applications that desire it), so if you artificially cap it to protect against stack overflow you can then safely use the stack, thereby getting better performance by reducing the required allocations, while also protecting against one DoS attack vector.

<comment><AUTHOR> You never "artificially cap it to protect against stack overflow": Most libraries that cap it artificially cap it to a fixed value, that may or may not protect against stack overflow depending on how large the stack already is when you call the parser.

<comment><USER15> These caps we're talking about are normally a small fraction of the total stack size, and few applications run anywhere _near_ the total stack size. I acknowledge your point (and had it in mind when I wrote my first comment), but maintain my position that this capping _is_ protecting against the stack overflow DoS attack: if you're able to get a stack overflow with your capped JSON parser, you were probably already able to get a stack overflow, because it's not adding overly much to the stack.

The problem that's being protected against is _user-controlled_ stack overflow—that's what makes it a DoS _attack_.

</comment></comment></comment></comment></comment><comment><USER16> I'd argue this is a pitfall that languages can and should solve. There's no reason why writing a naive recursive descent parser should cause this problem. The fix, using a separate stack data structure, is telling, since it's the same algorithm, with the same space complexity, you're just saying "my language's call stack isn't a usable implementation, so I need a different stack." Why not just fix the call stack?

In languages like rust/c/c++, this might be hard, since you can't grow the stack by moving them (since you'd need to update pointers, which are too visible in those languages, and even identifying a pointer at runtime is undecidable), and segmented stacks cause the "hot-split" problem.

But most languages could just make this problem go away by growing the call stack when needed. A few of them (incl. Haskell) do. I'm following suit in a language I'm working on.

<comment><USER17> Most purely functional languages (like Haskell) don't really have a "call stack", but the computation is carried out by graph reduction, so it can obviously detect cycles and do some optimizations there, depending on the type of recursion.

If you want the gory details [https://www.microsoft.com/en-us/research/wp- content/uploads/...](https://www.microsoft.com/en-us/research/wp- content/uploads/1992/04/spineless-tagless-gmachine.pdf)

<comment><USER16> Yeah, I've read the STG paper, though its been a while. IIRC there isn't a call stack as we'd normally think of it, but there's still a stack of pending pattern matches, so you actually do hit the same issues. I vaguely recall making deep recursion "just work" in GHC was something that happened just in the past couple years.

</comment></comment></comment><comment><USER18> Related: "Parsing JSON is a Minefield" [http://seriot.ch/parsing_json.php](http://seriot.ch/parsing_json.php)

Discussed on HN:

• in 2016: [https://news.ycombinator.com/item?id=12796556](https://news.ycombinator.com/item?id=12796556)

• in 2018: [https://news.ycombinator.com/item?id=16897061](https://news.ycombinator.com/item?id=16897061)

</comment><comment> bloody-crow The repo is calling it bad, but in production I'd actually prefer a parser that throws an approptiate error based on a setting that can be configured with some reasonable defaults over something that just silently chugging along until it runs out of memory.

I can imagine it'd be a lot easier to overwhelm and DDoS a server that attempts to parse incoming JSON requests without any depth bounds too.

<comment><USER8> Just because your data structure is deeply nested doesn't mean it takes a lot of memory.

[[[[]]]] is still only 4 objects worth of memory on a stack.

</comment></comment><comment><USER19> I think the reason that the nesting level limit is 128 and the file size is 256 bytes for serde_json in this test is that the recursion_limit attribute is not configured.

[https://doc.rust-lang.org/reference/attributes/limits.html](https://doc.rust- lang.org/reference/attributes/limits.html)

The default value of this attribute is 128 which would explain both results, I'm guessing with configuration you could significantly increase both values at the expense of compilation time.

<comment><USER20> That's for the recursion limit on compile time stuff, like how many layers of macros it will expand.

serde does have this though: [https://docs.rs/serde_json/1.0.41/serde_json/struct.Deserial...](https://docs.rs/serde_json/1.0.41/serde_json/struct.Deserializer.html#method.disable_recursion_limit)

<comment><USER19> For some reason I thought serde was using macros to perform deserialisation. Now I know that isn't the case and it's a coincidence the default recursion limit and the serde default nesting limit is 128.

<comment><AUTHOR> Even if serde does indeed use macros, they run at compile time. The limit on macro recursion depth has an impact on what programs do compile, but it has no impact on the runtime behavior of these programs.

</comment></comment></comment></comment><comment><USER21> Is this relevant/are there situations where people nest JSON even 100x/1000x? I have not come across something like that.

<comment><USER22> The question isn't necessarily what might be found in nature, but what might be intentionally constructed in order to attack something. Without knowing more, the segfault in the C++ library is somewhat alarming.

<comment><USER23> On the topic of malicious input, the serde_json example seems to be deliberately limiting recursion depth in order to head off DoS attacks:

 _" serde_json has a recursion limit to protect from malicious clients sending a deeply recursive structure and DOS-ing a server. However, it does have an optional feature unbounded_depth that disables this protection. It still uses the stack though, so it will still eventually blow the stack instead of using all available memory"_

[https://github.com/lovasoa/bad_json_parsers/issues/7](https://github.com/lovasoa/bad_json_parsers/issues/7)

This begins to suggest that it may be downright desirable for servers to toss out deeply-nested JSON, regardless of the JSON specification.

<comment><USER24> Why not implement parsing without recursion? It doesn't look like a particularly hard task for me. Why restrict users with artificial limitations?

<comment><USER25> You'll need a stack anyway, because the structure is logically a tree. Better to use the call stack, that is already there and heavily optimized.

<comment><USER8> Naive person here. Isn't the call stack a lot heavier than pushing data structures on a std::stack?

<comment><USER26> A recursive parsing call stack has the overhead of pushing/popping return addresses and maybe a couple other registers. Maybe a bit more if there are security protections like stack cookies that didn't get optimized away, or if shadow stacks are enabled. It's not free, but is pretty cheap. A non-recursive use of std::stack dodges some of that, but instead has the overhead of dynamic allocation, which I'm generally going to consider more expensive - but does have the advantage of being more of a once-off cost - so perhaps it'd be cheaper for long but shallow JSON objects? The cost of the call stack vs std::stack is probably dwarfed by the cost of allocating arrays/objects/strings and decoding in either case.

If you want to over-optimize for that specific case _anyways_ , you can take a hybrid approach with smaller stacks kept on the stack in a single linear root blob, with larger stacks falling back onto the heap, using a non-recursive style.

However, if you're capping JSON depths as an anti-DoS measure _anyways_ (your parser isn't the only thing possibly vulnerable to blowing the stack via recursion - anything dealing with the parsed result is also in danger if it can be recursive), the heap fallback may be entirely dead code anyways... so why bother coding it?

</comment></comment></comment></comment></comment></comment></comment><comment><USER27> I know, Halloween is over, but currently I'm working with an API where the returned JSON is created by manual string concatinations. Sometimes commas and quotation marks go missing.

<comment><USER28> Does the API specify under which conditions you can expect missing characters? There should be an undefined behaviour angle here somewhere

<comment><USER27> I wish.

The dev simply writes new string concatination code every time a new endpoint is needed, so I can't even hope for some bugs being gone for good after he tested his implementation with a few endpoints.

</comment></comment></comment><comment><USER29> They've changed the title[1] to remove the phrasing "bad," I think the title should be updated here as well, even if the repo name still reflects the original title. It's not reasonable to use this single criteria as a measure of "badness," as there are _many_ different variables. I would place security/memory safety 1000x higher than recursion limits. I have never hit JSON recursion limits in production.

[1]: [https://github.com/lovasoa/bad_json_parsers/commit/16f6ed75b...](https://github.com/lovasoa/bad_json_parsers/commit/16f6ed75b4edcac4998878229783eadaa7212f18)

</comment><comment><USER30> Coincidentally, I just wrote a simple JSON parser the other day as a toy exercise. A simplified snippet of parsing code (handling only arrays) relevant to the discussion here would be something like:

      def parse(text):
          stack = []
          index = 0
          result = None
          while index < len(text):
              char = text[index]
              val = None
              if char == '[':
                  stack.append([])
              elif char == ']':
                  val = stack.pop()
              index += 1
              if val is not None:
                  if stack:
                      stack[-1].append(val)
                  else:
                      result = val
          return result

Using the test utilities in the repo indicate that the parsing logic can handle arbitrarily nested arrays (e.g. up to the 5,000,000 max in the test script), bound by the limits of the heap.

It seems like the main criticism here is against recursive implementations. Or am I missing something?

<comment><USER31> A better question to ask is: Is it a success of CS education since developers know when to make tradeoffs in software engineering and go for the most easily understandable code (recursive implementation used in most pseudo code) since deeply recursive cases are rare, or is it a failure that software engineers cannot figure out how to convert a recursive algorithm into an iterative implementation for greater memory efficiency?

<comment><USER32> It doesn't necessarily mean anything that deeply nested cases are rare. If you're making a general library, someone is going to throw user input at it, and some users will try to break your system. If it can avoid a crash in that case, it usually should.

</comment></comment></comment><comment><USER33> One of JSON's selling-points is to be human-readable. I would struggle reading any JSON document with a nesting level greater than 6 or so. Why would I ever want a nesting level > 100?

<comment><USER34> This doesn't really make sense to me. Just because JSON is human-readable doesn't mean that humans are meant to read any possible serialized data nor that it should always be an ergonomic read. Not everything you serialize is just a config file or hand-edited data. It may be arbitrarily dense with arbitrary data.

To me, this is kinda like seeing a JSON array of user IDs coming over the network and going "Huh? Why not usernames? I thought JSON was supposed to be human-readable!"

<comment><USER33> I see your point; although if you look at the specs for a lot of easy-to-parse formats for computers, a stated design goal is also easy-for-humans (e.g. Markdown, YAML).

Large, complex object hierarchies with lots of nesting might make more sense represented in binary (e.g. Avro).

I realize I'm making a little bit of a McLuhan-esque argument in a largely computer science-oriented context, but I hope you can see what I'm getting at.

</comment></comment></comment><comment><USER35> Recursive parsers need not use the stack for recursion, they can use the heap instead through trampolining, and not have an issue until memory is fully consumed. That would impact performance somewhat - so users should also be allowed to choose when they need the trampolined version. They should still limit recursion depth and allow the user to set a value higher than the default.

I can't help but think that the lazily evaluated Haskell version score of infinity is probably somewhat misleading, as once you access something within a large enough tree you'll probably run out of memory pretty quickly.

<comment><USER36> It'd be nice if it was possible to annotate recursive functions as using the heap instead of the stack for recursion. I've seen too many implementations of people writing a poor man's stack frame to store on the heap and it just totally destroys any sort of elegance the original code had.

</comment></comment><comment><USER37> Shameless self-promotion: I wrote a JSON parser for node.js that specifically solves this problem of parsing deeply-nested and/or huge JSON payloads.

[https://www.npmjs.com/package/bfj](https://www.npmjs.com/package/bfj)

It's entirely asynchronous, yielding frequently to avoid monopolising the event loop. And it uses a fixed-length buffer so it never exhausts available memory. The tradeoff is that it's slow. But afaik it parses JSON payloads of any size and any depth.

</comment><comment><USER38> I thought i'd try some other Java parsers, but the Maven build doesn't work with current Maven, and it's not clear what you'd feed to test_parser.sh anyway, as there's no driver script for the Java case.

One of the things i'm passionate about is always setting up a project in such a way that someone new to it can download, build, and use it with an absolute minimum of fuss. That means making the most of the build tools, and scripting everything you can't do through the build tool. I don't always meet this standard myself, because i am a weak and wretched human being, but it's great when it's done.

Gradle is good here, because the wrapper means you always get a specific Gradle version, although it can't control the JDK version, which is increasingly painful in the post-8 age. Cargo is pretty good, because the user just needs to install rustup, then they get a precisely controlled Rust and Cargo version via the rust-toolchain file. I'd love it if there was a rustup- wrapper which i could check in, which would obviate the need for a global rustup installation. Ruby, Python, and Node do alright, but you have to ask the user to install the version manager of your choice. C is absolutely miserable.

Since this project has all of the above and more, making it as self-sufficient as i would like would be quite an effort!

<comment><USER39> How are C build systems miserable? If you're building for a general Linux distro, you'll not have much control over the dependencies, but that's not a problem with C, but with how programs are distributed in the GNU/Linux distribution model.

<comment><USER38> AFAIK, there is no tool that lets you specify a compiler version in your project, then automatically build with that compiler. The equivalent of rustup, rvm, nvm, etc.

There is also no standard package manager.

It is very hard to persuade GCC to use a specific set of libraries that you have obtained using a package manager, or vendored or whatever, rather than getting distracted by random libraries it happens to find in in /lib etc. The root of the pain is that you really want to use the host system's platform libraries, the ones which implement the POSIX API, but once the compiler can see those, it tends to go off looking for other things in the same place. In my experience, at least!

These deficiencies are tied up with the historical interrelationship of C and the operating system, but that doesn't mean that they aren't real.

</comment></comment></comment><comment><USER40> Related question: why is the stack limit so tiny in modern operating systems? Why do we allow programs free reign of the heap but not the stack?

Edit: I hadn't considered that all of a processes's threads share the same address space. In a 32-bit address space, a 4MB stack limits you to 1024 threads (with nothing left over for heap).

However, perhaps the question still stands: why use a small stack size on a 64-bit system.

<comment><USER41> One of the reasons is that if you momentarily use a very large amount of stack to calculate something (say gigabytes), and then return, the large part of the stack which is no longer used stays allocated, full of unused data taking up RAM for the remaining life of the process, which might be a long time.

If a heap-allocating algorithm does the same thing, it's likely to free the memory when it's finished with it.

</comment></comment><comment><USER42> the sort order is incorrect. he says they sort from worst to best, but the worst one is the one that can segfault and the perfectly good ones are all the other ones that, if i read this correctly, just throw an exception.

</comment><comment><USER43> In Clojure, we've taken to using a macro to generate custom parser drivers (using Jackson underneath) from schemas. If we encounter an unexpected token or invalid (according to the schema) value, we drop the whole input.

</comment><comment><USER44> I expecting a report on parsers that crash for challenging syntax. eg: { hello: 'world\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\' }

<comment><AUTHOR> Independently of the backslashes, this is not valid JSON: keys have to be quoted, and JSON uses double quotes, not single quotes...

<comment><USER45> Hey look, a human-based JSON parser. I wonder what their maximum nesting level is...

<comment><USER46> Over-under on when the first "ML-based approach to parsing JSON" story hits HN: 1.5 years from now.

</comment></comment></comment></comment><comment><USER47> Just couple of weeks ago I had to debug a production problem when Jackson parser failed on parsing prod-specific configuration (in JSON). Turns out the problem was a trailing comma in the array of values. IntelliJ IDEA JSON syntax highlighter did not flag it, due to some obscure setting.

I also had problems with some JSON parsers allowing NaN and Infinity as double values and others - failing on them.

<comment><USER43> For future reference, if your JSON parser accepts NaN, Infinity, or trailing commas, it is broken.

I suspect IntelliJ is using their JavaScript grammar for JSON.

</comment></comment><comment><USER48> For a moment, I was worried that someone had found the JSON parser I wrote in Python years ago. My career would be over.

</comment><comment><USER49> Regardless of the quality of this post, the author does highlight a security issue here. The RFC states that it's up to the parser to define the limits and different parsers may behave differently in the face of malicious inputs. That's why you should probably test them...

</comment><comment><USER50> Nice effort, but seems to miss things. RapidJSON for C++ is a very popular library. And Go has both the standard library JSON parser and a couple of other popular implementations. I would gladly add implementations for Go, but it is unclear how do do that.

<comment><USER51> Since the document is on GitHub, it's perhaps a safe assumption the author may be amenable to pull requests (and if they're not, you can always fork their repo and add additional information; they MIT licensed the whole thing).

<comment><AUTHOR> Indeed, I'd love to get pull requests. It's mentioned in the "Remarks" at the end.

</comment></comment></comment><comment><USER52> Guile scheme expands the call stack until until memory is exhausted and then raises an exception. That is pretty damn comfy, because it means that building lists with recursion is just as fast as using tail recursion and a reverse! at the end

</comment><comment><USER53> Interesting results e.g. on Ruby. I did give the javascript test file a whirl and was able to run it just fine with a nesting level of 1,000,000, so I wonder if something else is happening there.

</comment><comment><USER54> JSON nested more than 100??

Why calling a JSON parsers are "bad" when the JSON object is already gone far beyond "bad"?

PLEASE don't call anything bad just because it doesn't fit in your own standard.

</comment><comment><USER55> Much more detailed comparison of JSON parsers: [http://seriot.ch/parsing_json.php](http://seriot.ch/parsing_json.php)

</comment><comment><USER56> For the Java test, I'd love to see how Jackson compares to Gson.

<comment><AUTHOR> Someone just contributed a pull request with results for Jackson in java

<comment><USER56> And it's already been merged (nice turnaround time!). It narrowly beats Gson.

</comment></comment></comment><comment><USER57> PicoLisp successfully parsed 20M nested brackets, its maximum i can do on 8GB laptop.

</comment><comment><USER58> "Ubuntu Linux 4.10.0-35-generic SMP x86_64 with 8Gb RAM" Gb != GB.

</comment><comment><USER59> if you allow a JSON parser use all of memory to go deep, you know what attackers will do... As other commenters mentioned JSON spec allows setting a limit on the depth.

<comment><USER16> It isn't a matter of memory usage; the memory usage is still linear in the size of the input regardless of the depth. If an attacker can get you to oom by feeding you a deeply nested data structure, they can do the same with a shallow one just as easily:

"[0, 0, 0, 0, 0, <...gigabytes of zeros>, 0 0, 0, 0]"

So you need to limit the overall input size regardless of whether you already have a depth limit in place.

</comment></comment><comment><USER60> Has anyone run this for Newtonsoft.Json?

</comment><comment><USER61> there's a .net parser in the source code but not in the results

<comment><AUTHOR> Would you be interested in making a PR adding the .NET project to travis and the results to the README ?

</comment></comment><comment><USER24> Java Jackson is bad too.</comment>


Nvidia  education, and embedded architectures - <AUTHOR> http://lists.linux-foundation.org/pipermail/ksummit-2012-discuss/2012-June/000304.html

<comment><USER2> Interesting answer by Matthew Garrett[1] via reddit [2]

1: [http://lists.linux- foundation.org/pipermail/ksummit-2012-dis...](http://lists.linux- foundation.org/pipermail/ksummit-2012-discuss/2012-June/000336.html)

2: [http://www.reddit.com/r/linux/comments/vgasz/nvidia_employee...](http://www.reddit.com/r/linux/comments/vgasz/nvidia_employee_asks_what_they_should_be_doing/c548day)</comment>


Level 3 are now hijacking failed DNS requests for ad revenue on 4.2.2.x - <AUTHOR> http://james.bertelson.me/blog/2014/01/level-3-are-now-hijacking-failed-dns-requests-for-ad-revenue-on-4-2-2-x/ <comment><USER2> Seems reasonable. Those servers were never intended for widespread public use, so they may as well make back some funds for upkeep, and maybe encourage some more-technical users to switch away.

Here's a blog post with some background on these servers: [http://www.tummy.com/articles/famous-dns- server/](http://www.tummy.com/articles/famous-dns-server/)

<comment><USER3> I'm fairly sure that the "4.2.2.x was never meant to be public" line is a myth. Though the NANOG thread cited in that post is good historical background, it is contradicted by more modern sources:

"[...] DNS infrastructure is largely split into two types; open (public) and closed (private). Open DNS is provided by companies like OpenDNS, Google and Level 3. You can use it wherever you are on the Internet with no restrictions or authentication required."

\- Mark Taylor, VP at Level3

[http://blog.level3.com/level-3-network/a-flawed-study-of- cdn...](http://blog.level3.com/level-3-network/a-flawed-study-of-cdns-and- dns/)

I can't find any cite where anyone else who I would consider a reliable source in the DNS world (Vixie, &c) repeat this claim. To the contrary, Level 3 is often grouped with Google, OpenDNS and others in discussions of open public resolvers [1][2][3], and those in the know never seem to speak up and say otherwise in these discussions.

That being said, I have absolutely no personal knowledge on any of this.

[1] [http://www.maawg.org/system/files/Fergie_DNS_Open_Resolver_M...](http://www.maawg.org/system/files/Fergie_DNS_Open_Resolver_MAAWG_India_SANOG.pdf)

[2] [http://markmail.org/message/gh7f2wvfbn5mpvuq](http://markmail.org/message/gh7f2wvfbn5mpvuq)

[3] [http://www.circleid.com/posts/87143_dns_not_a_guessing_game/...](http://www.circleid.com/posts/87143_dns_not_a_guessing_game/#4234)

<comment><USER4> Presumably it takes non-0 costs to maintain 4.2.2.x DNS servers. While I'd want to believe Google and L3 just try to help out the public at large with free DNS services, I suspect they are not doing that just purely out of altruism.

<comment><USER5> Google's motivations are long-range, but simple - more ad dollars. Faster DNS means more people using the web (rather than give up in disgust - and if don't believe that happens, let me introduce you to comcasts's DNS servers...); more people using the web means more page view which means more ad revenue.

So no, definitely not altruistic.

<comment><USER6> I just thought it was to track user actions via DNS - they can see which sites you visit without needing tracking bugs on those sites. Better profiling means better ad serving for Google ... profit.

<comment><USER7> [https://developers.google.com/speed/public- dns/privacy?hl=en](https://developers.google.com/speed/public- dns/privacy?hl=en)

"We built Google Public DNS to make the web faster and to retain as little information about usage as we could, while still being able to detect and fix problems. Google Public DNS does not permanently store personally identifiable information."

</comment></comment></comment></comment></comment></comment><comment><USER8> I heard years ago that Level 3 were trying to encourage people (non- customers?) not to use these DNS servers. I guess this is one way to ask people not to use them.

Having said that, 220.127.116.11, Google DNS, has been planted firmly in my memory as my go to "is this machine up?" IP.

<comment><USER9> My issue is that level3's dns has always been very fast, and more importantly, up... When google's dns is slow, level3 is fast... when my isp's dns goes down or wonky, level3's is up... I'd pay them $10/year to use them without the ads.

<comment><USER10> Why not use OpenDNS then? If you have an account you can configure them to behave however you'd like.

<comment><USER11> And then they have your entire history of internet activity, _matched to you name, address and CC number_. Fine if all you want is reliablility. For those who want to imped surveillance it is as bad as the ISP.

<comment><USER12> So does Google, for what it's worth.

<comment><USER13> No, they very clearly and explicitly promise otherwise [https://developers.google.com/speed/public- dns/faq#privacy](https://developers.google.com/speed/public-dns/faq#privacy)

</comment></comment></comment></comment></comment></comment><comment><USER14> To be perfectly clear: It is not hijacking when you are sending them queries for which you should have no reasonable expectation that they service. If you are actually a Level 3 customer, call your sales rep, but I believe this is only for non-customers.

EDIT: By the way, this is the actual company operating the "service" behind the scenes for Level 3 [http://www.xerocole.com/searchguide/](http://www.xerocole.com/searchguide/)

<comment><USER15> If you're not going to service a query, there are perfectly good ways to do that. Since they didn't do one of those, I'm happy to call this hijacking.

<comment><USER16> and one of them is returning bogus junk. why should they (the NSP) care? especially if you aren't paying them for a network service? it's not hijacking -- 126.96.36.199 isn't a public resolver and never was. it just happened to become one.

<comment><USER15> I'd say they should care because the Internet only survives through cooperative effort. Breaking something so they can pocket money is greedy and dickish.

But if they are being jerks rather than just being thoughtless, then maybe that isn't enough. In which case, my fallback answer is "bad PR". It would have been easier for them just to deny service to anybody they didn't want to serve. They went to a lot of trouble break something in a profitable way. To me, that says they might not be a trustworthy vendor, and thousands of nerds are now aware of that.

</comment></comment></comment></comment><comment><USER17> I have used the DNS servers of the Swiss Privacy Foundation for some time. The IP addresses are not easy to remember but it is great to have uncensored DNS from a Swiss non-profit organization:

188.8.131.52 (Ports: 53, 110; DNSSEC), 220.127.116.11 (Ports: 53, 110; DNSSEC) and 126.96.36.199 (Ports: 53, 110; DNSSEC).

[https://www.privacyfoundation.ch/de/service/server.html](https://www.privacyfoundation.ch/de/service/server.html)

(The Swiss Privacy Foundation operates Tor exit nodes too.)

<comment><USER18> I have a newbie question, what would an end user do with the (Ports: 53, 110; DNSSEC) information?

I've set my machine to use those three IP addresses as the DNS servers, is there something else I'm missing? Thanks!

<comment><USER11> Normally DNS is on port 53, but if your ISP is preventing you from DNS requests to servers other than theirs on port 53 you can use the other one.

'DNSSEC' means DNSSEC is supported by the server if your resolver can use it - it's a digital signature regime to prevent DNS forgery (disclaimer: look up criticisms of it as well as selling points).

</comment></comment></comment><comment><USER19> Yep, I'm seeing what should be NXDOMAIN results returning the IP 188.8.131.52 which brings me to a page like [http://searchguide.level3.com/search/?q=http://188.8.131.52...](http://searchguide.level3.com/search/?q=http://220.127.116.11/&=)

Does anyone know if actual Level3 customers see this page, or is it only for off-network requests? Up until the end of last year, my employer had a Level3 internet connection and we legitimately used 22.214.171.124 as our DNS recursive resolver. I'd be pretty pissed if they returned spammy results to their customers, but to non-customers, well, I don't care: That's what you get. Use a DNS server that somebody says you're allowed to (220.127.116.11, maybe)

<comment><USER20> Querying from one of my personal servers on a Level3 DIA circuit, I am getting NXDOMAINs for non-existent hostnames.

</comment></comment><comment><USER21> Suggestion: if your network provider's recursive DNS service sucks so much that you cannot bear to use it (and even if it doesn't, quite frankly) your next best bet is probably to install unbound ([https://unbound.net/](https://unbound.net/)) listening on localhost on your workstation.

Not only does this give you known-good DNS resolution, but you can also enable DNSSEC validation and be fairly confident that it'll actually do its job in preventing your local machine from resolving poisoned zones.

<comment><USER22> I set up a caching recursive DNS server on my lan, but had to add a second entry for a caching public DNS server since:

1) Most DNS entries these days seem to have _very_ short TTLs

2) Occasionally the recursive queries would fail

<comment><USER21> Would you mind clarifying what you mean by "second entry"?

</comment></comment></comment><comment><USER23> When their DNS seemed to go down a few days ago I also noticed this behavior and immediately switched to a local independent ISP's DNS that namebench spit out.

Doesn't seem too out of character, if they're returning this for actual customers, from the company that most likely allowed the US government to tap into Google and Yahoo's fiber lines.

[http://www.nytimes.com/2013/11/26/technology/a-peephole- for-...](http://www.nytimes.com/2013/11/26/technology/a-peephole-for-the- nsa.html)

<comment><USER14> That is a pretty big, and very uneducated, accusation.

Since the 1970s the CIA has been installing taps on undersea cables using specialized submarines. This is well documented, and it is well known that the NSA prefers to use methods that involve the least amount of interaction from uncleared individuals even if it is at a much greater expense.

<comment><USER16> the NSA is the one installing the taps. CIA tried to do a radio tower once -- or maybe that was the FBI. anyway, when it comes to sigint, it's NSA all day every day.

<comment><USER14> Nope. The Special Collection Service is responsible for deployments. It is a joint program with the CIA providing the field resources and management and the NSA providing the toys.

</comment></comment></comment></comment><comment><USER24> I have to plug OpenNIC[1] anytime I hear of a DNS hijacking story. OpenNIC is a peer run network of DNS servers that are open for public use.

[1][http://www.opennicproject.org/](http://www.opennicproject.org/)

<comment><USER25> This looks cool, but why should I use and trust this over something like Google's Public DNS?

<comment><USER24> Honestly, I have no argument for OpenNIC over Google's DNS that will convert you right now. It really depends on who you trust more.. strangers over the internet or a large corp who was the target of espionage. That's really up to you to decide the lesser of two evils.

A cool thing about OpenNIC is that they offer alternative TLDs that aren't part of ICANN's gTLDs. Also, the owners of the public servers strive to be as open as possible with their policies and features, such as no logging or using DNSCrypt. One of them even offers DNS level ad blocking, though I don't like it because I prefer the internet at its purest form and that policy doesn't seem to flow well with their anti-censorship mantra.

</comment></comment></comment><comment><USER26> Can everyone else reproduce this problem? People from different locations and ISPs should try it.

I'm not a Level 3 customer in a any way and I'm on a German VDSL connection provided by Deutsche Telekom. And here the Level 3 resolvers still return normal NXDOMAIN answers:

      ; <<>> DiG 9.8.3-P1 <<>> thisprobablydoesntexist.com @18.104.22.168
      ;; global options: +cmd
      ;; Got answer:
      ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 44948
      ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

<comment><USER14> Lots of well meaning kids go and setup Level 3 resolvers on grandma's home network, then the ISP has to deal with support calls when someone elses DNS servers go down.

As a result quite a number of networks "hijack" 18.104.22.168/24 and route it locally to their own resolvers.

<comment><USER11> How can one detect whether this is happening?

<comment><USER27> I'm not sure if it's a certain way to tell, but try running a traceroute. If your traffic seems to go into Level3's network, that's a good sign that it's not getting rerouted.

Here's what I see from my DigitalOcean droplet.

root@derpy:~# traceroute -I 184.108.40.206 traceroute to 184.108.40.206 (188.8.131.52), 30 hops max, 60 byte packets 1 220.127.116.11 (22.214.171.124) 12.055 ms 12.123 ms 12.314 ms 2 xe-10-3-3-100.edge3.Newark1.Level3.net (126.96.36.199) 0.948 ms 0.959 ms 0.959 ms 3 ae-31-51.ebr1.Newark1.Level3.net (18.104.22.168) 1.396 ms 1.477 ms 1.478 ms 4 ae-10-10.ebr2.NewYork1.Level3.net (18.104.22.168) 1.530 ms 1.630 ms 1.659 ms 5 ae-62-62.csw1.NewYork1.Level3.net (220.127.116.11) 1.465 ms ae-82-82.csw3.NewYork1.Level3.net (22.214.171.124) 1.464 ms ae-62-62.csw1.NewYork1.Level3.net (126.96.36.199) 1.390 ms 6 ae-1-60.edge2.NewYork1.Level3.net (18.104.22.168) 1.363 ms 1.389 ms 1.395 ms 7 a.resolvers.level3.net (184.108.40.206) 1.456 ms 1.466 ms 1.421 ms

</comment></comment></comment></comment><comment><USER28> > At the least it's leaking, in clear text on the wire, things that I expected > to be sent to an encypted DDG search. If there was sensitive search terms or > information in that query, it just dropped into Level3′s logfiles.

He must not realize that even if the DNS server was working correctly, the original request that should result in NXDOMAIN is also passed in clear text over the wire and naturally potentially logged by the DNS server. The lesson is not to rely on DNS security. Your ISP can see what servers (IPs) you communicate with anyway.

</comment><comment><USER29> I'm rather curious. I thought this kind of predatory network shenanigans was par for the course in the US?

<comment><USER30> By residential ISPs, sure. Level 3 is not a residential last-mile provider. I'd be surprised if you could get any sort of service out of them for less than $1k/month.

</comment></comment><comment><USER31> Why Am I Here?

The Example Net Web Helper has been enabled to provide helpful searches from web address errors. You entered an unknown name that the Example Net service used to present site suggestions which you may find useful. Clicking any of these suggestions provides you with Yahoo! search results, which may include relevant sponsored links. Why should I use this?

The Example Net Web Helper makes finding what you are looking for easier and more convenient. The service uses the entered non-existing website name to determine useful search results. Often, you will see a desired website or page that meets your needs. Do you track my Internet usage?

No. The Example Net Web Helper simply redirects queries to non-existing domain names to a useful search results page instead of a cryptic error message page or browser-defined page.

The "Example Net" huh?

</comment><comment><USER32> don't use L3's 4.2.2.x resolvers, as they aren't meant for public use, unlike google's public dns

<comment><USER33> You say that, but if they weren't meant for public use, they wouldn't be accessible to the public. Time Warner's DNS, for example, are not available from outside their network.

<comment><USER13> And if your car is unlocked it's meant for joyriding?

<comment><USER33> If my car became very well known for being unlocked and immensely popular with people taking joyrides, then yes, my continued unlocking of the door would constitute tacit permission.

</comment></comment></comment></comment><comment><USER34> Comcast also did this to me. Not one of the several tech support people I talked with seemed to be aware of Comcast's non-hijacking DNS servers at 18.104.22.168 and 188.8.131.52.

<comment><USER35> Fortunately they haven't done this for 2 years. They killed it when they flipped on DNSSEC because the practice of NXDOMAIN hijacking is incompatible:

[http://dns.comcast.net/index.php/help#faq2](http://dns.comcast.net/index.php/help#faq2)

Now no customers from Comcast suffer this.

... JavaScript and HTML injection when you reach a cap limit in a throttled market or when you get a cease and desist for pirating, however, is another matter.

</comment></comment><comment><USER36> I use mdnsresponder nomulitcastannounce -> dnsmasq 127.0.0.1#53 -> dnscrypt- proxy 127.0.0.1#54 -> an encrypted DNS proxy that does dnssec. All of which is locked down by a minimal whitelist leak-preventing fw ruleset like little snitch. I have a script which checks for authentic internet access to allow captive portals to work which leaks temporarily (which I prob need to toggle between rulesets to only allow the captive portal agent to work and deny everything else).

</comment><comment><USER37> Can't really blame them since they have been telling the public not to use their service forever.

Still, I am guilty of using them too.

Not really thrilled with the idea of using Google DNS.

<comment><USER38> > Not really thrilled with the idea of using Google DNS.

You can use your own dns server. Just install a recursive dns server on your own network like [https://www.powerdns.com/recursor.html](https://www.powerdns.com/recursor.html) or [https://unbound.net/](https://unbound.net/)

<comment><USER39> unbounds https cert is broken. The identity of this website has not been verified. • Server's certificate is not trusted. • Server's certificate cannot be checked.

Not a good first sign.

</comment></comment></comment><comment><USER40> I noticed this a few weeks ago (not sure exactly, but I don't really mistype domain names all that often either) and I can remember the first time that 198.x IP showed up I was rather shocked since the domain I mistyped was my own site's (!), but they seem to have stopped doing it now.

Perhaps L3 themselves have a lot of stuff both within and outside their subnet that depends on these servers behaving correctly.

</comment><comment><USER41> Is this only affecting Level3's 4.2.2.x nameservers (when it happens), or are their 209.244.0.x ones doing this as well?

</comment><comment><USER42> I can't reproduce this behavior from the UK. In any case, DNSmasq (the DNS cache daemon that is part of OpenWRT) has an option to filter bogus NXDOMAIN responses if you can get a list of the IPs.

Alternatively, run your own recursive resolver and cache, it's worth it.

</comment><comment><USER43> While in search of a new DNS server because of this and for lower latency, namebench is your friend: [https://code.google.com/p/namebench/](https://code.google.com/p/namebench/)

</comment><comment><USER44> The tracker should now be blocked also in Easyprivacy.

[https://hg.adblockplus.org/easylist/rev/303e65d3a2bd](https://hg.adblockplus.org/easylist/rev/303e65d3a2bd)</comment>


Pulsars are a secure time source - <AUTHOR> http://unenumerated.blogspot.com/2007/09/secure-time-broadcast.html "Nature provides clocks that are oblivious to the malicious intentions of any outside parties. In the case of a remote high-energy system such as a pulsar, this means anybody. Pulsars are many orders of magnitude more accurate than random delays that face attackers on the Internet. If critical Internet servers were synchronized to natural clocks in a secure and timely fashion, they would be immune to attacks that relied on uncertainties in timing." <comment><AUTHOR> "Nature provides clocks that are oblivious to the malicious intentions of any outside parties. In the case of a remote high-energy system such as a pulsar, this means anybody. Pulsars are many orders of magnitude more accurate than random delays that face attackers on the Internet. If critical Internet servers were synchronized to natural clocks in a secure and timely fashion, they would be immune to attacks that relied on uncertainties in timing."

Forget GPS, syncing your clocks to a pulsar is much sexier!

<comment><USER2> Yeah, until we realise the aliens are exploiting us by manipulating the pulsars ;)

Ever read Iain M Banks? I like how the ships subtly re-arrange star formations so that 10000 years it spells out a rude word from the viewpoint of some star across the galaxy. (of course these ships originally came from human AI, and thus inherited and encourage the eccentric human sense of humour).</comment></comment>


Ask HN: Can we get a hide submissions from domain button? - <AUTHOR> Frankly, I'm sick of reading through news on HN.
Sites like bloomberg and nytimes that hold an almost permanent spot on the front page but seldom satisfy my personal curiosity.
I would like to be able to stop them from showing in the page ranking, and I believe there are others in the community that would feel the same. <comment><USER2> Here's a crappy userscript that might help

(function(shitlist) { [].__proto__.forEach.call(document.querySelectorAll('table.itemlist tr.athing'), (n) => { let s = n.querySelector('.sitestr'); if (s && shitlist.indexOf(s.textContent) !== -1) { n.parentNode.removeChild(n.nextSibling); n.parentNode.removeChild(n.nextSibling); n.parentNode.removeChild(n); } }) })(["nytimes.com", "crapsite.com"]);</comment>


Hong Kong newspaper raided, publisher Jimmy Lai detained under new law - <AUTHOR> https://apnews.com/2ce4e0aac2bbb360f06e481dffaf5149 <comment><USER2> [https://news.ycombinator.com/item?id=24105311](https://news.ycombinator.com/item?id=24105311)</comment>