Solving Every Sudoku Puzzle (2006) - <AUTHOR> https://norvig.com/sudoku.html <comment><USER3> Sudoku solvers are near and dear to my heart.

They're my go to problem when learning a new programming language because they're just complex enough to exercise a whole bunch of different language features.

I built this when I wanted to learn Swift and dip my toe into machine learning: [https://twitter.com/braddwyer/status/910030265006923776?s=21](https://twitter.com/braddwyer/status/910030265006923776?s=21)

<comment><USER4> I know someone who likes to create a fractal generator when learning a new language. He says it gives you a bit of exposure to the strengths of quirks in the language from implementing algorithms, working with data structures, and displaying a GUI to render an image to screen.

<comment><USER5> My go-to is the first fifty problems on Project Euler. It ramps up quite nicely from "can you write a for loop?" to problems with some nice complexity. Great for learning a language!

</comment></comment></comment><comment><USER6> An oldie but a goodie.

The other day we were at a restaurant and they gave my four year old a kids menu. One of the activities was a modified sudoku that went up to 6. My first thought was that I'd never realized you could do sudokus with other values of N.

But the really cool part was that once I taught her the rules (the constraints) she was actually able to figure it out without much help at all!

Turns out sudoku is a good way to teach kids logic, even at four years old.

<comment><USER7> You might want to check out KenKen puzzles. They are similar to Sudoku and were created by a math teacher as a fun learning tool. IMO, they are quite a bit more interesting than Sudoku puzzles.

<comment><USER8> As well as Kenken, I also like Killer Sudoku as a more interesting option than vanilla Sudoku. It's basically 9x9 Kenken with addition as the only operator, it uses a bunch of the same thought processes as Kakuro which I like too.

</comment></comment></comment><comment><USER9> Wrote a solver using basically the same idea back in college. Was kind of fun. Probably some stupid things in there that I would do differently now with more experience.

[https://github.com/war1025/sudoku- solver/blob/master/Sudoku/...](https://github.com/war1025/sudoku- solver/blob/master/Sudoku/Sudoku.vala)

<comment><USER10> Same. Didn't save the code; wish I had just so I could see if I got a better execution time than his 1439 second example.

Amusingly, my rationale was much like the author's; I freakin' hate Sudoku. So the fact it's such an obvious candidate to have an automated solver for meant I wrote one (it was also an excuse to do something more meaningful than the intro to CS exercises that class had me doing).

<comment><USER9> I left my solver running for 28 minutes and it never found a solution to that puzzle. Guess whatever corner case it's hitting in his program is the same as my program runs into.

<comment><USER11> Hmm it's weird. I tried my solver on it and it tells me that it can't find a solution (in 13 seconds). My solver is pretty well tested (on some very difficult sudokus as well). Maybe this guy's solver is bugged?

<comment><USER12> It's trivial to know that it has more than one possible solution since less than 8 of the possible numbers are on the initial board meaning any valid solution that is found has a mirrored solution with all of one number swapped for all of another number.

<comment><USER9> Theoretically a board with multiple solutions should be easier to find a solution for though, right?

<comment><USER13> Theoretically a board with multiple solutions isn't a sudoku.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> I love it how with the Z3Py solver you just write down a few constraints (rows, columns, 3x3 boxes and the given values) and you immediately get an answer.

[0] [https://ericpony.github.io/z3py-tutorial/guide- examples.htm](https://ericpony.github.io/z3py-tutorial/guide-examples.htm)

[1] [https://yurichev.com/writings/SAT_SMT_by_example.pdf](https://yurichev.com/writings/SAT_SMT_by_example.pdf)

<comment><USER15> Raymond Hettinger has a great video introduction to SAT solvers in python [47m]

[https://www.youtube.com/watch?v=_GP9OpZPUYc](https://www.youtube.com/watch?v=_GP9OpZPUYc)

</comment></comment><comment><USER16> A thread from 2015: [https://news.ycombinator.com/item?id=9901214](https://news.ycombinator.com/item?id=9901214)

2011: [https://news.ycombinator.com/item?id=3096922](https://news.ycombinator.com/item?id=3096922)

</comment><comment><USER17> The key insight when writing a sudoku solver is that it is just depth first search. Once someone told me that it just sort of clicked. Just guess each possible number at each open square and if you ever get to an un-solvable state, just undo the last change you made and try the next one. I was extremely satisfied with myself after writing my first really clean solver with backtracking.

I'm sure there are other, more efficient, ways to write sudoku solvers, but the simplicity and clarity the DFS solver offers is pretty cool IMO.

<comment><USER18> I don't think there are, actually. Some algorithms might run faster in practice, but, since the game is NP-complete, brute force is pretty much as good as it gets for exact solutions.

<comment><USER9> The constraint propagation bit is the main key. It still ends up being a DFS, but at most steps along the way, finding one number leads causally to several other spaces being found.

In most easy / medium puzzles, there are actually no points where you need to guess (i.e. recurse down the tree).

In many hard puzzles, there are only one of two guesses, which is where you would recurse down the decision tree.

Things get a bit more intense once you move into the puzzles specifically designed to be a pain in the neck for programs, but often those inputs aren't actually valid Sudoku puzzles.

</comment></comment></comment><comment><USER19> Contrast this with the TDD approach: [http://ravimohan.blogspot.com/2007/04/learning-from- sudoku-s...](http://ravimohan.blogspot.com/2007/04/learning-from-sudoku- solvers.html)

<comment><USER20> Do you mean the series of posts linked to from the one you give, beginning with [1]? They are an excellent demonstration of two of the pitfalls of naive TDD:

1) The things you can test first (before you have figured out the central problem) are not necessarily the things that matter most. The author should have addressed the algorithm for solving the puzzle first; everything else is secondary.

2) This is an example of those cases where having tests for what you are going to write does not provide much guidance into how to solve the problem.

[1] [https://ronjeffries.com/xprog/articles/oksudoku/](https://ronjeffries.com/xprog/articles/oksudoku/)

<comment><USER21> When making this comparison, it's worth noting that Jeffries is showing his work process. Norvig is showing a cleaned up version of his final result after throwing away some earlier attempts. (Source: I spoke with Peter about Sudoku solvers shortly after he'd published.)

<comment><USER22> I think that's understood. But what's telling is that no single step of Jeffries' series of articles seems to be approaching any kind of sensible solution. It reads like an amateur -- e.g. me -- trying to write a game and getting stuck in the low hanging but inconsequential fruit.

Because Jeffries' series is about TDD, and not really Sudoku, the _process_ is important. And the process failed him spectacularly.

</comment></comment></comment></comment><comment><USER23> I love the quote near the bottom: "Sudoku is a denial of service attack on human intellect."

You could substitute "Sudoku" with most social media platforms and it would be equally true. ;)

<comment><USER24> Similar to how tetris was (jokingly ?) described as a plot of the soviet to ruin the productivity of the west by hijacking the minds.

Second-hand Reference: [https://www.filfre.net/2017/06/a-tale-of-the-mirror- world-pa...](https://www.filfre.net/2017/06/a-tale-of-the-mirror-world- part-3-a-game-of-falling-shapes/)

</comment></comment><comment><USER25> Much faster if constraints are represented by single bits in an integer mask and binary masking operations are used - even the hardest soduko solves instantly.

</comment><comment><USER26> My first ever CS assignment was to build a sudoku solver using MIPS. I remember reading this then and being inspired by the beauty of computer science!

</comment><comment><USER27> This is pretty much exactly the algorithm that [https://www.ioccc.org/2005/aidan/aidan.c](https://www.ioccc.org/2005/aidan/aidan.c) uses, except obviously much more clearly expressed. There's also a reasonably straightforward extension of this technique that can test for solution uniqueness, and so generate random valid puzzles; I can't remember how it was implemented exactly but it involves continuing the search after finding the first solution and seeing if it succeeds.

</comment><comment><USER28> I wrote a TypeScript implementation of this algorithm [https://www.khrapov.org/sudoku/solver.html](https://www.khrapov.org/sudoku/solver.html)

</comment><comment><USER29> I always thought one of the requirements for published Sudoku puzzles was that no trial and error should be required; it should be possible to deduce the single unique solution from the clues given. This would mean the "search" part of Norvig's algorithm would not be required; constraint propagation alone would be enough.

<comment><USER30> I think technically, if constraint propagation can't solve the puzzle, that means it's either contradictory (zero solutions) or ambiguous (multiple solutions).

However, it seems Norvig was feeding his program with all three types of puzzles -- zero, one, _and_ multiple solutions -- so perhaps that led him to implement a more robust search.

<comment> T-hawk If there is one unique solution, then constraint propagation always works, yes.

But a propagated constraint isn't always just a single value at a time. There are cases where a solver (human or machine) needs to identify that a group of two or three or more cells mutually creates a constraint. ("Naked twins" and other such configurations, in Sudoku jargon.) Which can recursively cause to exist another such mutual constraint, and so on. Identifying every possible way such a constraint can arise isn't a simple operation either to implement or to execute, computationally speaking.

For both Norvig's programmatic solver and for a human, it's possible and common for trial and error to be faster and more expedient than trying to identify and execute every possible means a constraint can arise. Norvig's aim was for implementational simplicity more so than computational elegance, deciding to care about only the simplest possible type of constraint, and let trial and error handle everything else.

</comment></comment></comment><comment><USER31> FWIW, here's what I call "Boring Sudoku" in Prolog.

It is just 81 vars constrained to be 1..9 and then twenty-seven applications of the Pigeonhole principle.

[https://swish.swi-prolog.org/p/Boring%20Sudoku.swinb](https://swish.swi- prolog.org/p/Boring%20Sudoku.swinb)

</comment><comment><USER32> A while back I wrote a solver that uses no backtracking, just constraint propagation, and it wasn't substantially more complex than this.

One of the key insights comes from noticing a feature of groups of numbers in cells. For example, suppose you notice two cells in a given row and they're the only cells in that row that the numbers 1 and 2 can be placed in - <USER2> one of them is going to have 1 and the other will have 2, and you don't know which but you can eliminate any other possibilities in those two cells. This generalizes to groups of any size in rows/columns/squares. I think that and maybe one other heuristic along with the basic rules about number placement is enough to solve everything.

<comment><USER33> This sounds like Crook's Algorithm ([https://www.ams.org/notices/200904/tx090400460p.pdf](https://www.ams.org/notices/200904/tx090400460p.pdf)), which is a perfectly reasonable thing to do, especially for pencil-and-paper puzzle solving since pigeonhole inferences are easy for people to spot. But this inference rule does not suffice on its own for many hard puzzles. Crook's algorithm still requires backtracking.

</comment></comment><comment><USER34> > I can copy values with values.copy() which is simple and efficient. If I > implemented a possibility as a Python set or list I would need to use > copy.deepcopy(values), which is less efficient.

The array deepcopy isn't too bad, at least when writing code, it compressed into a neat slice [:]

I wrote a hinter[2] which was more fun than the actual solver too, because it shows the "minimum remaining values" as a colour coded chart.

[1] - [http://notmysock.org/code/sudoku/solver.py](http://notmysock.org/code/sudoku/solver.py)

[2] - [http://notmysock.org/code/sudoku/](http://notmysock.org/code/sudoku/)

</comment><comment><USER35> On a related note, Knuth has a cool algorithm to solve exact coverage problems like sudoku (similar logic as Norvig's DFS but implemented very differently). It's more work to code it than Norvig's but worth it for the education:) (and it solves faster)

Here's the paper describing the algorithm: [https://arxiv.org/pdf/cs/0011047.pdf](https://arxiv.org/pdf/cs/0011047.pdf)

<comment><USER36> While dancing links is cool, when benchmarking it turns out using bits to represent values, and just copying, ends up being faster. This is because on modern CPUs about the worst thing you can do is lots of pointer following.

<comment><USER37> I wrote a (straightforward) dancing links implementation that solves 17.000 puzzles/sec/core. It also validates uniqueness of the found solution.

How much faster does it get? I'm genuinely curious

<comment><USER38> There is a collection of benchmarks of the fastest sudoku solvers here: [https://github.com/t-dillon/tdoku/tree/master/benchmarks](https://github.com/t-dillon/tdoku/tree/master/benchmarks)

How fast they are depends on the difficulty of the sudoku and on your machine, but it's typically a few thousand/s on the hardest known and several 100k/s to a million/s on very easy.

<comment><USER33> I'm the author of Tdoku and maintainer of those benchmarks.

If anyone has a well optimized DLX implementation I'd love to add it to the comparison. The few I've sampled from github were not competitive, but I have no idea how much effort when into them.

This page from 2011 has some comparisons of backtracking and DLX solvers from that era, but it would be interesting to update: [https://attractivechaos.wordpress.com/2011/06/19/an- incomple...](https://attractivechaos.wordpress.com/2011/06/19/an-incomplete- review-of-sudoku-solver-implementations/)

<comment><USER39> Seems pretty competitive to me if the third fastest in your benchmark uses dancing links.

I do think it's probably true that dancing links wont be literally the fastest, but the neat thing is that it's a generic algorithm that solves ALL total cover problems, not just Sudoku. Of course a tuned Sudoku solver could probably beat it (optimizing around the Sudoku-specific parts), but it's nice to see that it's at least in the running.

<comment><USER33> Bear in mind that JSolve was 5x faster than the fastest DLX solver in the attractivechaos 2011 benchmarks, and today there are several solvers that are 2x faster than JSolve ([https://github.com/t-dillon/tdoku/tree/master/benchmarks](https://github.com/t-dillon/tdoku/tree/master/benchmarks)). If the difference is really an order of magnitude, that's pretty non- competitive.

I certainly don't mean do disparage DLX. It's a powerful and general algorithm. But if you need a fast Sudoku solver (for puzzle mining or studying Sudoku research questions) then it's not the tool I'd reach for. The backtracking solvers just get a lot of mileage out of tuning their representation to play nicely with hardware. They also blow general purpose CDCL SAT solvers out of the water, at least for 9x9 Sudoku.

</comment></comment></comment></comment></comment></comment></comment><comment><USER40> > As computer security expert Ben Laurie has stated, Sudoku is "a denial of > service attack on human intellect". Several people I know (including my > wife) were infected by the virus, and I thought maybe this would demonstrate > that they didn't need to spend any more time on Sudoku.

That's hilarious! But what puzzles, or even activities wouldn't be classified by Ben as a DOS on humans?

<comment><USER41> What I wonder is: In what way is chess different from sudoku that it is not a DOS?

<comment><USER42> There's beauty in chess.

</comment></comment></comment><comment><USER43> For a mind-melting good time, check out this sudoku solver written in APL: [https://www.youtube.com/watch?v=DmT80OseAGs](https://www.youtube.com/watch?v=DmT80OseAGs)

</comment><comment><USER44> I'd like to see a solver that works in a similar way to how a human would solve a puzzle.

"This square must be a 5 because a 5 can't appear in any other cell in this row."

<comment><USER45> The algo described in the 'constraint propigation' section was basically how I implemented my sodoku solver, and it basically can do 99% of all puzzles thrown at it. And it wasn't particularly complicated.

</comment></comment><comment><USER46> First time I saw a sudoku, was so intrigued, I wrote a solver using simulated annealing. Solved any puzzle in a couple second.

</comment><comment><USER47> last year's US Sudoku Team Qualifier was held online at some point. Possibly in May or July.

[http://wpc.puzzles.com/ussq2019/](http://wpc.puzzles.com/ussq2019/)

</comment><comment><USER48> A lot of people here have built solvers. What about Sudoku generators?

<comment><USER49> [https://sites.math.washington.edu/~morrow/mcm/team2280.pdf](https://sites.math.washington.edu/~morrow/mcm/team2280.pdf) Is my favorite paper on the subject. Section 8.3 describes a generator.

<comment><USER48> Awesome - thx!</comment></comment></comment>


How Reorganizing Our Content Increased Leads By 800% - <AUTHOR> http://hub.uberflip.com/h/i/3318006-beyond-blogging-how-reorganizing-our-content-increased-leads-by-800

<comment><AUTHOR> Great long and transparent post by Hana.</comment>


How AI and Machine Learning Work at Apple - <AUTHOR> https://backchannel.com/an-exclusive-look-at-how-ai-and-machine-learning-work-at-apple-8dbfb131932b <comment><USER3> None of this really answers the overlying question that Jerry Kaplan and Oren Etzioni raised. The question raised by most in the field isn't whether Apple use AI/ML internally, the real question is why they avoid the research community so strongly.

For me, the greatest thing about the ML/AI community is how open it is and how strong a sense of camaraderie there is between people across the entire field, regardless of whether they're from industry or academia.

Employees from competing companies will meet at a conference and actually discuss methods. Papers are released to disseminate new ideas and as a way of attracting top tier talent. Code is released as a way of pretraining students in the company's stack before they ever step through the company's doors. Papers are published on arXiv when the authors feel they're ready - <USER2> free to access - without waiting for a conference for their ideas to be spread.

This entire push of camaraderie has accelerated the speed at which research and implementation have progressed for AI/ML.

... but Apple are not part of that. They publish little and more broadly don't have a good track record. On acquiring FoundationDB, they nixed it, with little respect to the existing customers. Fascinating pieces of technology lost. If they aren't using the exact thing internally, why not open source it? I fear the same is likely to happen to Turi, especially sad given the number of customers they had and the previous contributions that many of Turi's researchers made to the community via their published papers.

Apple may change in the future - they may become part of the community - but a vague article of self congratulation isn't going to sway me either direction.

"We have the biggest and baddest GPU farm cranking all the time" ... Really? ¯\\_(ツ)_/¯

<comment><USER4> What makes you think this is self-congratulation? This is a response to Marco Arment, Ben Thompson, and other pundits (largely spurred by the PR efforts of Google and to a lesser extent Facebook and Microsoft) who claim that Apple is "behind" in this field or that it somehow poses an existential threat to Apple. This is Apple saying "uh no".

They'd be perfectly fine with not talking about it as they've done until now if they weren't trying to counter these media narratives.

As to your point about Apple not engaging with the research community, how is this a surprise at all? Their M.O has been secrecy ever since Jobs returned to Apple.

<comment><USER3> This article lacks any real evidence they're ahead though, which is essentially equivalent to "behind" if (a) the majority of the advances are democratized by your competitors and (b) you lack the ability to attract top tier talent. If AI/ML is an existential threat to Apple, then this article doesn't provide any comfort beyond being a PR puff piece.

The talent war for AI/ML is truly insane. Given that the top researchers and engineers can get similar benefits at other entities, whilst also continuing to publish code and research out in the open _, Apple aren 't that attractive.

They may try compensating for this via acquisitions but that still leaves a fundamental issue when it comes to long term retention.

_ As I mentioned before, this is a fairly universal trait. Many come from academia or have open source affiliations / rely on open source tools for their skill set, so the idea of putting their ideas out there to be used, ...

<comment><USER4> If what AI/ML talent wants is to do research and draw a Silicon Valley salary then you're right, they're not going to work at Apple.

Some people don't want to work at Apple because they can't describe their job on their LinkedIn pages or tell their friends and family about what they do. Others don't like that they have to buy their own meals at Caffe Macs. I don't think Apple worries about this too much.

What they do worry about is finding people that fit Apple's culture and want to work on products used by hundreds of millions of people.

In any case, being perceived or not to be at the top of the field and being able/unable to hire the best ML/AI talent isn't an existential threat. ML/AI isn't magic, it's a technology. If there's one thing that the tech industry should have learned from Steve Jobs, it's that you make good products by working backward to the technology. In that respect, what Federighi mentioned about not having a ML/AI division was reassuring.

<comment><USER5> > If what AI/ML talent wants is to do research and draw a Silicon Valley > salary then you're right, they're not going to work at Apple.

Let me simplify that further: "If what AI/ML talent wants is to _publish_...then you're right, they're not going to work at Apple". Money may or may not be a factor, but Apples secrecy won't allow the talent to publish their research. Given that AI is highly driven by research right now (no "if" about it), what talent would want to work somewhere where they can't publish research papers?

<comment><USER6> "what talent would want to work somewhere where they can't publish research papers?"

There are lot of people out there who don't particularly merit the publish-or- perish ethos of academia and would rather publish only after they have something worthwhile to publish and not just footnotes to an established scheme.

E.g. [https://www.theguardian.com/commentisfree/2014/feb/14/higgs-...](https://www.theguardian.com/commentisfree/2014/feb/14/higgs- boson-publish-or-perish-science-culture)

and so on.

</comment></comment></comment></comment></comment></comment><comment><USER7> I like how Apple can't win here.

If they publish, they aren't doing anything new and they haven't innovated since Steve died, and they should really just give up because there's obviously no point to anything they do and hasn't been since 1997.

If they don't publish, they're evil secretive bastards who don't contribute to the ML community and probably drown puppies or something because who knows what goes on behind closed doors?

I don't really have a dog in this fight, except inasmuch as I'm a generally satisfied iPhone owner. I just think it would be really neat if people would settle on one narrative or the other, instead of keeping on with both at once.

<comment><USER8> If they were serious, they'd provide objective evidence for their claims. Papers, source code, numbers. Not "we have the biggest, baddest GPU farm" and refuse to say any more as policy. Under this veil of secrecy and doubletalk, all they have is marketing: Trust the brand rather than objective reality.

Many other companies are mature enough to show their cards but Apple keeps declaring victory. Apple writes its own narrative instead of participating.

There are mature ways for this to shake out and they don't include your strawman dichotomy.

<comment><USER9> > If they were serious, they'd provide objective evidence for their claims. > Papers, source code, numbers. Not "we have the biggest, baddest GPU farm" > and refuse to say any more as policy.

But why? Who cares about papers, source code or numbers beyond the tiny segment that HN caters to? Apple has always been about the user experience. When they discuss a vast majority of prior accomplishments they discuss them, many times, in terms of the layman. Does that mean they can't manufacture? Nope. AI is simply harder to show without the papers / source code you mention but this isn't typical Apple to release that.

I honestly don't think they care to go beyond the level of details outlined here. I interviewed with the Siri team twice now and they certainly have some incredibly smart people. Whether they're "winning" against Google, Microsoft or whoever? I say who cares. They want to control their narrative without divulging too much detail like they have always done.

<comment><USER10> > But why? Who cares about papers, source code or numbers beyond the tiny > segment that HN caters to?

I don't know, say, the thousands of future engineers and researchers, that, you know, make the product?

<comment><USER7> Yes. That tiny segment.

<comment><USER11> In my mind, this article is squarely aimed at that segment. They are trying to tell one of the most competitive labor markets in the world, AI developers, that they're hiring, but more importantly, that they're doing interesting things.

</comment></comment></comment></comment></comment></comment><comment><USER12> I realize this is going to get voted down, but piece reads like a Trumpism "we have great AI, trust me, you would not believe, if you could see what we're doing".

I don't think you get to claim PR credit for advances in ML unless you publish. In general, for R&D, IMHO, you need to publish. There's product R&D, and there's fundamental R&D. If you make an advancement in something fundamental, but that helps your product, then publish it. If it is specific to your product only and can't be transferred elsewhere, then maybe it's ok to keep it secret.

Apple and Google's competitive advantage now arises from scale and path dependency. I think they need to let go of this idea that somehow they derive a competitive advantage by keeping these things secret. The Open AI community is going to advance at an accelerated rate regardless and IMHO, it's better to be part of it than to be seen as a kind of parasite that consumes public R&D, but doesn't give back improvements.

<comment><USER13> > If it is specific to your product only and can't be transferred elsewhere, > then maybe it's ok to keep it secret.

wouldn't it be the other way around? if the competitors can benefit from your knowledge, you'd want to keep it secret.

<comment><USER12> No, if everyone can benefit, that, IMHO, is precisely why it should be contributed back to the community. If you want a selfish altruistic reason to do it, well the likely improvements that the external community will make will benefit you, including those contributions made by your competitors.

I think a culture of secrecy yields local optima. Only if you believe (and is true) your company has unique geniuses that can't benefit from other people reviewing their science will secrecy benefit you.

IMHO, only research that is useless to your competitors is research to keep secret in the sense that it is too specific to your own proprietary dependencies.

</comment></comment></comment><comment><USER14> Finding these heavily curated advertorials Apple has been pushing out (This and the recent wired advertorials come to mind) a bit of a sign that not everything is sunny at One Infinite Loop.

<comment><USER15> Probably no more so than Google. Steven Levy is basically the go-to when you want this sort of stuff. For example, [https://backchannel.com/how-google-is- remaking-itself-as-a-m...](https://backchannel.com/how-google-is-remaking- itself-as-a-machine-learning-first-company-ada63defcb70#.3gjh9zpy1)

</comment></comment><comment><USER16> IMO the most profound quote in the interview: "Our practices tend to reinforce a natural selection bias — those who are interested in working as a team to deliver a great product versus those whose primary motivation is publishing," says Federighi.

</comment><comment><USER17> Nice article, but in my perception, Apple is way behind in AI vs. Google on mobile. Siri's current speech recognition is still far behind Google on my iPhone. Half of the time it Siri doesn't recognize something but Google recognizes it right way. Apple Maps continues to lag in the following features: biking and public transit and even things like which lane to take on a big interchange. As a result I end up using Google (Now) and Google Maps by default.

<comment><USER18> Never mind biking. For example in Switzerland Apple Maps doesn't even seem to take traffic information into account when it gives you ETAs. It shows heavy traffic as red spots, but it doesn't know the delay. Google meanwhile gets all of this perfectly.

</comment></comment><comment><USER19> Why should Apple researchers contribute back to the AI community on their shareholders' dime? What makes AI special as opposed to any other field of computer science?

<comment><USER20> The long term goal is loftier than almost all others except for extending human life and populating other star systems.

<comment><USER21> That's an opinion. I don't think you could get a random group of 3 AI researchers to agree on what the long term goal is, or if there even is a long term goal as opposed to this simply being a very buzzword- and marketing- friendly area of research at the moment.

</comment></comment></comment><comment><USER22> > If you're an iPhone user, you've come across Apple's AI, and not just in > Siri's improved acumen in figuring out what you ask of her. You see it when > the phone identifies a caller who isn't in your contact list (but did email > you recently). Or when you swipe on your screen to get a shortlist of the > apps that you are most likely to open next. Or when you get a reminder of an > appointment that you never got around to putting into your calendar. Or when > a map location pops up for the hotel you've reserved, before you type it in. > Or when the phone points you to where you parked your car, even though you > never asked it to. These are all techniques either made possible or greatly > enhanced by Apple's adoption of deep learning and neural nets.

This whole paragraph must be a joke. Google started doing this since way too long and they don't even publish these as their best features.

</comment><comment><USER23> I've never had a single one these happen to me. Has anyone actually seen these behaviors out in the wild? Is it because I use gmail, chrome, google maps, and shut off most of the siri/recommended apps/etc functions in favor of serious battery life gains?

>You see it when the phone identifies a caller who isn't in your contact list (but did email you recently). Or when you swipe on your screen to get a shortlist of the apps that you are most likely to open next. Or when you get a reminder of an appointment that you never got around to putting into your calendar. Or when a map location pops up for the hotel you've reserved, before you type it in. Or when the phone points you to where you parked your car, even though you never asked it to

<comment><USER24> Nope, but it often tells me how long it will take to drive to work (I never drive to work), even when I'm on the other side of the country, on holiday, and has been for two weeks.

<comment><USER25> So glad I'm not the only one. How do you turn this crap off?

<comment><USER24> It will take 524 minutes to drive to work. Traffic is normal.

</comment></comment></comment></comment><comment><USER26> i am really disappointed by apple. I respect Apples wish to develop products in secrecy and i understand that you can't just open source your secret sauce. I also really like their products.

But not publishing your advancements harms the community greatly. Its like building your product entirely with open-source software (the published work of other researchers) and not contributing back.

<comment><USER27> That's what they did with osx.

<comment><USER28> OSX is much more than the kernel. Darwin is open source, even. What are you talking about?

<comment><USER29> How much of Apple's investment in OS X was in Darwin, and how much was in proprietary technology?

Apple built a proprietary product using open source technology, and are now building proprietary products using open ML research.

But it seems they're doing so for their own profit, not to benefit the open source and research communities.

<comment><USER30> > _Apple built a proprietary product using open source technology_

You say it as its somehow contradictory. Open Source licences allow (LGPL etc), and some even welcome (BSD, MIT, Apache) creating proprietary software based on open source technologies.

Besides, Apple also did extend open source technologies (as open source) a hell of a lot. Webkit past Apple is 100x bigger/fancier/better than the puny KHTML it started from. Tons of LLVM work, especially all the early stuff, has been sponsored by Apple. Swift was made Open Source just recently...

<comment><USER29> Apple's core "product" is the user experience. Darwin is open source, Carbon and Cocoa are not.

I think that philosophy will continue as they use ML tools. They might share the ML equivalent of plumbing like Webkit/LLVM/Swift, but probably not improvements to the user experience like Siri's brain.

<comment><USER30> > _Apple 's core "product" is the user experience. Darwin is open source, > Carbon and Cocoa are not._

So? Why should they open source and commoditize their core product?

Besides, I don't know any company that did it and got much out of it, except for some gratitude from OSS fans.

<comment><USER29> I'm not saying they should, I'm saying they wouldn't (and probably shouldn't).

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER31> Apple sends their employees to ML/AI conferences with fake company names on their badges to avoid leaking a single bit of their knowledge. I don't know how any AI researcher resists working at Apple!

<comment><USER32> So - I don't need a source on this to grill you and score internet rhetoric points; I need a source to share this juicy factoid with skeptical friends.

</comment></comment><comment><USER33> I have a big problem with articles like this.

Apple's PR is notorious for cracking the whip, which means that the "inside story", if they give it to you, comes with a warning to the journalist to behave and be nice. Levy's piece is generous with flattery and cautious with criticism. He quotes Kaplan and Etzioni high and briefly in the piece, and spends the rest of it refuting them. Apple will give him another inside story down the road.

Apple has a big question to resolve for itself about the tools it's going to use to develop this. It can't go with Tensorflow, because TF is from Google. It's kind of at another turning point, like the one in the early 90s when it needed it's own operating system and Jobs convinced them to buy next and use what would become OSX.[0]

The most pointed question to ask is: What are they doing that's new? The use cases in the Levy story are neat, and I'm sure Apple is executing well, but they don't take my breath away. None of those applications make me think Apple is actually on the cutting edge. There's no mention of reinforcement learning, for example; there is no AlphaGo moment so far where the discipline leaps 10 years ahead. And the deeper question is: Is Apple's AI campaign impelled by the same vision that clearly drives Demis Hassabis and Larry Page?

We see what's new at Google by reading DeepMind and Google Brain papers. Everyone else is letting their AI people publish, which is a huge recruiting draw and leads to stronger teams. Who, among the top researchers, has joined Apple? Did they do it secretly? (This is plausible, and if someone knows the answer, please say...) The Turi team is strong, yes, but can they match DeepMind? If Apple hasn't built that team yet, what are they doing to change their approach?

Another key distinction between Apple and Google, which Levy points out, is their approach to data. Google crowdsources the gathering of data and sells it to advertisers; Apple is so strict about privacy that it doesn't even let itself see your data, let alone anyone else. I support Apple's stance, but I worry that this will have repercussions on the size and accuracy of the models it is able to build.

> "We keep some of the most sensitive things where the ML is occurring > entirely local to the device," Federighi says.

Apple says it's keeping the important data, and therefore the processing of that data, on the phone. Great, but you need _many_ GPUs to train a large model in a reasonable amount of time, and you simply can't do that on a phone. Not yet. It's done in the cloud and on proprietary racks. So when he says they're keeping it on the phone, does he mean that some other encrypted form of it is shared on the cloud using differential privacy? Curious...

> "How big is this brain, the dynamic cache that enables machine learning on > the iPhone? Somewhat to my surprise when I asked Apple, it provided the > information: about 200 megabytes.."

Google's building models with billions of parameters that require much more than 200MB, and that are really, really good at scoring data. I have to believe either that a) Apple is not telling us everything, or b) they haven't figured out a way to bring their customers the most powerful AI yet. (And the answer could very well be c) that I don't understand what's going on...)

[0] If they have a JVM stack, they should consider ours: [http://deeplearning4j.org/](http://deeplearning4j.org/)

<comment><USER34> Good comment - a few thoughts:

AlphaGo is impressive no doubt but has DeepMind done anything really key to Google's bottom line yet? A lot of the really sweet stuff they do doesn't have immediate commercial utility that I can see. Apple might be waiting to strike once Google has found the killer app (remember they're never really first at anything and focus holistically on the product).

Apple is benefiting from other companies releasing their research. If everyone but Apple releases the community is nearly as good, and Apple gets the pick of external and internal research while not needing to give up any of their own ideas. I know they send people to conferences and it can be a bit weird talking to someone who won't tell you anything about what they do.

Regarding researchers, I don't know of any top trend setters who've joined but they do have some very good applied ML people through direct- or aqui-hires.

Tl;dr: Apple doing what they usually do and keeping their powder dry/free loading off other's work until they can execute the product.

<comment><USER35> DeepMind reduced Google data center cooling bill by 40% in an experiment. If this effect is real, it probably more than justified the $500m tag Google paid to acquire DeepMind. Google might not have come up with a killer app, but perhaps the real killer app is not in the consumer space.

[https://deepmind.com/blog](https://deepmind.com/blog)

[https://news.ycombinator.com/item?id=12126298](https://news.ycombinator.com/item?id=12126298)

<comment><USER36> To be fair, the HN discussion was mostly focused on the lack of substantial details or an actual research paper to back up that claim. Google would probably need to show that only DeepMind, or an equivalent device, would have been presently likely to find the energy savings that it ended up finding.

</comment></comment></comment></comment><comment><USER37> There are a lot of criticisms here about Apple's secretive AI/ML development practices. But it's not unusual to Apple's long culture heritage of secrecy.

From consumer's perspective, I applaud their firm believe of customer privacy as well as pioneering consumer products based on AI/ML development AND with differential privacy in mind.[1][2]

[1] [http://highscalability.com/blog/2016/6/20/the-technology- beh...](http://highscalability.com/blog/2016/6/20/the-technology-behind-apple- photos-and-the-future-of-deep-le.html)

[2] [http://www.imore.com/our-full-transcript-talk-show- wwdc-2016...](http://www.imore.com/our-full-transcript-talk-show- wwdc-2016-phil-schiller-and-craig-federighi)

</comment><comment><USER38> THis feels so much like a story done as a marketing/PR angle

</comment><comment><USER39> Is there a way to op-out of this on iOS and macOS? That's really scary.

ML and AI on OS-level should run decentral on the device itself, and don't leak data at all. The spirit of the 1990s was that way, and we older desktop software works fine that way (even on Pentium 1 hardware), so it would run like a piece of cake on a modern smartphone.

The "differential privacy" technology may sound good, but without an independent audit who knows how good it works.

</comment><comment><USER40> siri has gotten worse over time, or at least that's why friends and i have noticed

<comment> k-mcgrady I've noticed it get drastically better. I never used to be able to use it at all. It was just a pointless exercise. Now it works for everything I use it for, which isn't a lot, but I don't get any issues.

</comment></comment><comment><USER41> Off topic a bit, but this article makes me wonder where does one start with Deep/Machine Learning/AI? I've seen a few posts the past few days talking about the topic (Deep Learning with Python, etc.), but what are the core requirements regarding math, statistics, programming, etc? Where should a web developer start?

<comment><USER42> Check out the stanfard MOOC class taught by Andrew Ng... that's where most people (I know of) started. [https://www.google.com/search?client=safari&rls=en&q=Andrew+...](https://www.google.com/search?client=safari&rls=en&q=Andrew+Ng+machine+learning&ie=UTF-8&oe=UTF-8)

Web developer to ML? Doable but you may have to work harder than some other folks have a background in statistics, probability, signal processing etc. Plus, you don't have to be an algorithms developer... doing the backend, compute cluster is hugely valuable, too.

<comment><USER41> Thank you, I actually just found him on Coursera([https://www.coursera.org/learn/machine- learning](https://www.coursera.org/learn/machine-learning)) before your reply. I will definitely check it out.

I don't think that I want to make the switch, just know what is what and dabble a little bit.

</comment></comment></comment><comment><USER43> has anyone considered the possibility that perhaps Apple has made a judgement that the recent resurgence in AI/ML is, (like all of them before, over the past 50 years) overblown, and Apple would rather spend their time and resources on other things?

There's no doubt that the recent advances in deep learning have improved ML/AI in certain specific domains ... but it seems like every 15-20 years or so we see an advance and an accompanying narrative that "AI is back! fully automated future is near!"... which fizzles out, again

Also, Apple has a more humanist tradition than Google, FB, etc, and it's my impression that they value the human element perhaps more.

Sure, there's Siri, but Siri strikes me more like an ongoing experiment than a fully fledged whole hog "let's put all our eggs in this ML/AI basket"

</comment><comment><USER44> They were certainly at NIPS. I got a nice Apple pen and ski hat...

</comment><comment><USER45> It isn't clear from the article do they still use Nuance?</comment>


iPhone Call Recording: It Makes Too Much Sense Not to Do - <AUTHOR> http://www.louisgray.com/live/2009/08/iphone-call-recording-it-makes-too-much.html

<comment><USER2> I always record calls that include word-of-mouth contracts, from negotiating auto repairs to canceling paid subscriptions. I have never had to use one of these recordings, not even in case of disputes, but I have always felt comfortable that if push comes to shove I have their word on tape.

(In Denmark it is legal to record a phone conversation without the consent or knowledge of the other party, you'd probably want to check your local laws regarding this.)

<comment><USER3> The legality of this varies from state to state in the US. Where I live, Kansas City, MO only one of the parties on the line has to be aware of the recording, so this would be fine.

</comment></comment><comment><USER4> Imagine if someone came if with a generic idea for a computer program and they described it as 'Windows Blah Blah'. Many would see this as reflecting poorly on the author's knowledge of computing. What about Linux or OSX? Why are you so close minded as to only think about Windows? they might say. In the same way I think that the use of the word 'iPhone' for generic smartphone concepts should be deprecated in favor of more generic terms.

<comment><USER5> In this case, I referenced the iPhone because of their recent introduction of voice memos. It seems like the tech is there, and they can marry it to the standard interface. I also understand Android has this available.

<comment><USER4> I'm guessing that you used it because many others before have also used it. The problem is it tends to propagate an untrue and unfortunate meme that the iPhone is the only mobile computing platform. Now that many are starting to doubt the 'perfect golden future' of the iPhone I think it is time to squash this meme.

The other thing is that many phones have had the capability of doing much of this for years. I had a phone 5 years ago that could record phone calls. I think it was an Ericsson.

The ironic thing is that there is a high likelihood Apple wouldn't allow you to implement these ideas where other platforms would.

</comment></comment></comment><comment><USER6> I consider things like these scary privacy problems.

Imagine it becoming a standard and then your friends going "Hey, you gotta hear what X said about Y on the phone. Here, I recorded it."

It would probably be considered weird if you had something against recording, after all you are talking with a friend and you trust him/her, right?

If you want to record a phone call, you can always do so (maybe not on a locked-down thing like the iPhone, well, your choice). But making it a prominent feature screams for abuse.

<comment><USER7> An armed society is a polite society.

We already use plenty of other self documenting communication mediums (email, instant messaging, SMS).

<comment><USER8> A polite society is a conforming society. Opinions offend.

</comment></comment></comment><comment><USER9> Google Voice has phone recording. I use it quite often for business calls. It chimes before the recording begins with a message informing both parties what's going on.

<comment><USER10> Unfortunately it doesn't (yet) record outgoing calls.

</comment></comment><comment><USER11> Mobile phone audio quality is barely a photon's cough above usable. Add in mobile phone signal transience and the suggested periodic beeps to indicate that it's being recorded ... would that be good enough to be a podcast?

If that is good enough, then don't wait for Apple to code it, use the mentioned conference call/third party feature to dial an Asterisk server that announced "Recorder present" and then records the conversation.</comment>


Windows Kernel-Mode Drivers Written in Rust - <AUTHOR> https://github.com/pravic/winapi-kmd-rs <comment><USER2> This is a neat project but with a caveat: Microsoft has solid, verification tools for C drivers but Rust doesn't have the equivalent. The tools are especially important for finding behavioral problems at the interface between the driver itself and Windows.

So, my suggestion for production by Rust developers is to simultaneously develop the driver in Rust and a C subset. Keep the two logically equivalent. Rust's checker will catch problems in C program possibly. More importantly, Microsoft's analysis tools will probably catch problems in both if it's a logical error. Should result in some pretty robust drivers.

<comment><USER3> As someone who's never worked with the Windows kernel, I'm curious about the tools you mention. How do I get them?

<comment><USER4> I think the most significant are the SAL version 2 annotations. This is a nice introduction : [https://www.cs.cmu.edu/~fp/courses/15122-f10/lectures/25-sal...](https://www.cs.cmu.edu/~fp/courses/15122-f10/lectures/25-sal.pdf).

Basically, there's an additional step after compilation, called code analysis, which runs this static verification. I think it's implemented on top of the Z3 theorem prover.

<comment><USER5> Here's a list of the driver code analysis warnings. I wonder how many of these could be codified as compile-time checks in the winapi-kmd-rs library. For example, the KIRQL type is a u8 in winapi-kmd-rs, but the code analysis warnings the valid IRQL range is 0–31.

[https://msdn.microsoft.com/en- us/library/windows/hardware/ff...](https://msdn.microsoft.com/en- us/library/windows/hardware/ff550572\(v=vs.85\).aspx)

</comment></comment></comment></comment><comment><USER6> This looks very cool, definitely something I'm tempted to play with (having never done any dev in this area before).

However, the feedback loop for debugging drivers on Windows seems quite manual ([https://msdn.microsoft.com/en- us/library/windows/hardware/mt...](https://msdn.microsoft.com/en- us/library/windows/hardware/mt269367\(v=vs.85\).aspx#Section4_Install)). On Linux I'd have SSH at my disposal - are there any best practices for speeding up this copy/install process? Cygwin sshd could be an option, off the top of my head?

<comment><USER7> First step I'd say is to import your certificate into the the debugged computer, so it doesn't warn you every time you re-upload the driver [1]. Second, use powershell remoting [2]. Pretty much all of the other steps can be established and automated with a powershell remote connection. I'm in fact surprised that microsoft hasn't updated that page to use powershell, because driver debugging is a task where it could be invaluable in helping.

[1] [https://blog.jourdant.me/ps-installing-drivers-from- untruste...](https://blog.jourdant.me/ps-installing-drivers-from-untrusted- publishers/)

[2] [https://technet.microsoft.com/en- us/library/dd819505.aspx](https://technet.microsoft.com/en- us/library/dd819505.aspx)

</comment></comment><comment><USER8> Judging from the screenshot it even works on Windows XP. This is really impressive, since Windows kernel drivers typically have to be written in plain old C.

<comment><USER9> It's just the kind of area Rust does really well in. From a certain point of view, Rust can produce executables that are pretty much "written in plain old C" in that they can provide linkage that works like C linkage, don't necessarily need heap/GC, don't necessarily need big std library presuming userspace context.

I'd wager you could do something similar with D (soon if not now), since its no-std support is catching up w/Rust's.

<comment><USER10> I have a "framework" (really just a build tool and some examples) for bare- bones Win32 support for D[1]. I haven't tried writing kernel drivers with it though.

I used it to create a few Fallout 4 mods[2]. Avoiding the stdlib was not necessary there, but I didn't want the bloat. One interesting application of D's metaprogramming has been automatically recursively instrumenting the DirectX COM API just from the D interface definitions[3] (not IDLs).

      [1]: https://github.com/CyberShadow/SlimD
      [2]: https://github.com/CyberShadow/csfo4
      [3]: https://github.com/CyberShadow/csfo4/blob/master/dxlog/d3d11.d

</comment></comment></comment><comment><USER11> It is very nice to see this effort. I have one suggestion - integrate with WDF APIs instead of the lower level and legacy WDM APIs.

<comment><USER12> "Legacy" is as much said. It still forms the basis interface that is used by WDF. Also you still need to use the WDM directly if you are writing a filesystem driver.

Besides that, the WDF source is available under a MIT license, so it might make more sense to just port it to Rust.</comment></comment>


Adobe quits Flash, goes full HTML5  - <AUTHOR> http://www.bitsandbuzz.com/article/adobe-quits-flash-goes-full-html5/

<comment><USER2> That'd certainly be nice.

</comment><comment><USER3> News was from 1 April. Just sayin'.</comment>


Every startup needs this due to lazy employees - <AUTHOR> https://www.youtube.com/watch?v=O1D07dTILH0 <comment><USER2> I saw a video maybe a year ago of the same technology, also using Nissan's platform, for automated lines. Sit in a chair, and it'll move you down the queue until its your turn, then the chair returns to the back of the line.</comment>


WebAssembly Troubles Part 4: Microwasm - <AUTHOR> http://troubles.md/posts/microwasm/ <comment><USER2> I really like how it has only CFG control flows which would have made life much easier for people generating WASM and for people writing AOT compilers. Too bad WASM had to cater for the requirements of V8.

<comment><USER3> Perfect example of why the Chrome/V8 monoculture is a bad idea.

<comment><USER4> Indeed, if V8 has a deficient design, well, it's no reason to stop the world. Adults can't be cry-babies.

</comment></comment></comment><comment><USER4> Note the previous 3 parts are also make very interesting points.

</comment><comment><USER5> Haven't all of these problems already been solved if you use GO as your compiler?</comment>


California moves toward open-source vote counting software - <AUTHOR> http://www.truthout.org/092908VA

<comment><USER2> How does posting the source to vote counting software help anything? The real issue isn't that people want some source that's capable of counting votes accurately. The problem is that the machines aren't necessarily counting all the votes; having some open source software laying around somewhere doesn't guarantee that the machines themselves are running audited and trusted code. How can anything other than a physical, tamper-resistant audit trail help make vote tallies trustworthy?

<comment><USER3> Security is always a matter of degrees, just because it doesn't entirely solve the problem doesn't mean it is bad step to take.

</comment></comment><comment><USER4> I saw Debra Bowen speak at USENIX Security this year and it struck me how completely intimate she was with all associated technology. She _understood_ PKI, various encryption methods and their caveats, open source, operating systems, issues of intellectual properties and her drive to get it right the first time was eye-opening.

The stereotype of politicians being technology-inept and low-bidder-first type people was totally shattered with her. Not only did she truly care about the voting process (and dedicates a huge portion of her time to it), she actually took the time to understand the backing technology and made contacts with mostly everyone who matter in e-voting.

</comment><comment><USER5> it isn't California where vote accountability will be needed.

<comment><USER6> Step in the right direction though. If it works in California and is cost effective it'll spread to other states.

Shouldn't have a negative attitude against something just because it's used in a non-ideal environment. I imagine most things wouldn't have been created or invented without the first experimentation steps.

<comment><USER5> how can this not be cost effective? You could write an open source voting script in 5 minutes. Why does it take them 4 years to approve a 10-20 line piece of code?

<comment><USER6> People fear what they don't understand. Also, most politicians are of an older generation that doesn't embrace computers as easily as the tech crowd.

I would also think when things are done the same way for so long, it's hard to change - look at how certain people are handling the new facebook layout. Now magnify that to an election system to people who don't easily embrace technology.

They will have to spend money on reeducating the people, testing the system, and a host of other ideas that only politicians can think of.

I'm just playing devil's advocate here. I'm all for it and hope it gets to the East Coast soon.

</comment></comment></comment></comment><comment><USER7> <http://news.bbc.co.uk/2/hi/europe/4343374.stm></comment>


Simulation of Ideal Gas Particles - <AUTHOR> http://www.eeweb.com/project/steven_hochstadt/simulation-of-ideal-gas-particles

<comment><USER2> "To keep track of just 500 balls, we have to make (500 _501/2 comparisons)_ (60 frames per second) = 7,515,000 comparisons per second."

This is incorrect. Space partitioning makes the distance search much faster.

Ideal gas simulations started in the 1950s. One of the standard optimizations is to compute time until next collision and use a priority queue. Pop the smallest item off the queue and advance everything by that step, compute the collision, then do the 500 or so line intersection tests to figure out when the next collision will occur, and put that back into the queue.</comment>


Interesting Data Sets for Statistics - <AUTHOR> http://rs.io/2014/05/29/list-of-data-sets.html <comment><USER2> In related news, the Global Database of Events, Language, and Tone (GDELT) is now available in bigquery, for free. [1]

[1] [http://googlecloudplatform.blogspot.com/2014/05/worlds- large...](http://googlecloudplatform.blogspot.com/2014/05/worlds-largest- event-dataset-now-publicly-available-in-google-bigquery.html)

</comment><comment><USER3> The Reddit data isn't actually the top 2.5 million posts - it's the top 1000 posts of each of the top 2500 subreddits. An important distinction to make if anyone's planning to do statistical analyses on the set.

</comment><comment><USER4> Surprisingly it doesn't mention HN itself which is a treasure trove of data. I know there is APIs to download HN content but is there a permanent location for HN data dump (like StackOverflow do their data dump on Internet Archive)? This is a great article, BTW, anyone who wants some cool projects to do in data mining and machine learning.

<comment><USER5> I currently have a subset of one year's worth of all HN stories and comment trees, organized by story, but it's on my local machine. Where is a good place to post it? It's quite big, on the order of multiple GB.

The problem (if you want an easy scraper) is that the HN API limits you to 1k requests per hour. So it took me about 10 days of continuous running and restarting because of random crashes to get all the data.

</comment></comment><comment><USER6> Looks like an interesting list of datasets, but it's such a large number that it's tough to get a feel for what all is in it (without reading a lot of the entries). I wonder if some sort of organized table might be a way to present the information in a more skimmable fashion.

</comment><comment><USER7> I think it will be really interesting in a few years when people start some in-depth analysis about the bitcoin blockchain(though some is going on today). If Bitcoin hits mainstream adoption it may be the first time ever someone can run analysis on a complete financial system. Not even including the applications built on top of the block chain.

<comment><USER7> followup, this is the kind of stuff i was alluding to:

[http://www.technologyreview.com/view/527906/data-mining- reve...](http://www.technologyreview.com/view/527906/data-mining-reveals-the- factors-driving-the-price-of-bitcoins/)

</comment></comment><comment><USER8> For starters and people who miss their R sample sets there is a pretty good maintained archive of 731 of them available as CSV at [http://vincentarelbundock.github.io/Rdatasets/](http://vincentarelbundock.github.io/Rdatasets/)

Index: [http://vincentarelbundock.github.io/Rdatasets/datasets.html](http://vincentarelbundock.github.io/Rdatasets/datasets.html)

Github: [https://github.com/vincentarelbundock/Rdatasets](https://github.com/vincentarelbundock/Rdatasets)

</comment><comment><USER4> So someone put 2.5 million Reddit posts on Github. I was thinking about doing same for the HN data I've downloaded (1.3 million stories ~ 1.7GB of json).

Does Github has any restrictions on hosting data files like this?

<comment><USER9> Their guidelines suggest to use dropbox for that:

[https://help.github.com/articles/what-is-my-disk- quota](https://help.github.com/articles/what-is-my-disk-quota)

[https://help.github.com/articles/working-with-large- files](https://help.github.com/articles/working-with-large-files)

</comment></comment><comment><USER10> I have always wanted to create a Data Science training course which finds the right dataset to expose the power of the technique in question. I think this dataset will give me a good start. Thank you!

<comment><USER11> I would be way interested in your course!

</comment></comment><comment><USER12> This looks like a good resource. But be sure to understand all the implicit assumptions in each data set before announcing your amazing discoveries!

</comment><comment><USER13> Some stuff I hadn't seen before.

</comment><comment><USER14> This is a gold mine, thank you!

</comment><comment><USER15> Good article!</comment>


Interview with Founder of RentHop (YC S09) - <AUTHOR> http://www.examiner.com/examiner/x-2901-NY-Business-Strategies-Examiner~y2009m7d27-An-interview-with-Lee-Lin-cofounder-of-RentHop

<comment><USER2> I love reading stories of people dealing with complex backend stuff and translating that experience into an easy-to-use consumer tool.

</comment><comment><USER3> _I purchased two condos, living in one and renting out the other. That might sound pretty crazy for a guy fresh out of university_

Raise your hand if 50% of your income at that age went into partying and having good times; I can't even remember what I did at that age :-(

</comment><comment><USER4> I used to read Lee Lin's blog through Facebook -- very smart guy. I think at one point he designed a custom search on Zillow for condos that were commutable from Microsoft, and in the price range that their newly-hired engineers could afford.

</comment><comment><USER5> I should have said _SOME_ 2br townhouses in 2004 were selling for $120K, as in, the starter homes I wound up buying. The median near MSFT campus was about $180K.</comment>


Apple Promotes Strange New Game Type in iOS App Store, 'Pay Once and Play' - <AUTHOR> http://www.forbes.com/sites/insertcoin/2015/02/13/apple-promotes-strange-new-game-type-in-ios-app-store-pay-once-and-play-2/

<comment><USER2> Isn't this the standard way that thousands of games were sold since the Atari area? I see no innovation here.

<comment><USER3> You also missed the satire, even though it was expressly pointed out.

<comment><USER2> Yes, and it seems obvious now. I have no excuse other than too many stimuli.

</comment></comment></comment><comment><USER4> Glad they're promoting it. Separating pay once and pay as you go was always hard to tell</comment>


H-1B visas really do drive down salaries and deprive U.S. IT workers of jobs - <AUTHOR> http://www.itworld.com/it-managementstrategy/182967/h-1b-visas-really-do-drive-down-salaries-and-deprive-us-it-workers-jobs

<comment><USER2> As a guy waiting for his H1B visa, I dislike this article. Personally Im going to be paid as much as US employees, took no one's job. But I'm already EXTREMELY limited by the current visa. No green card. Cant change company for 3 years. Can get visa - maybe - after 6 month wait. Enormeous amount of info to give to get it delivered. Costs a lot.

So if a company lays off someone for a H1B I guess the previous employee was just bad. Cause the earning doesn't sound right, its possible to fire people quickly in the US and replace them. I know for sure that many americans would take the jobs at the salary foreigners would take it.

</comment><comment><USER3> If the US starts limiting highly skilled immigrants from coming to your country, you really have no chance to establish the US as the hub of the information economy. I actually think the reverse should be the case.

<comment><USER4> Did you read the article? In only one place did it mention "immigration issues".

The H1B Visa is "a non-immigrant visa in the United States under the Immigration and Nationality Act, section 101(a)(15)(H).

[via <https://secure.wikimedia.org/wikipedia/en/wiki/H-1B_visa>]

<comment><USER5> "Non-immigrant visa" is a term of art. It is relatively straightforward to immigrate permanently into the US if you have an H-1B and can stay employed for a few years.

<comment><USER2> few = 3 years minimum then probably another 2 to get the green card (due to delays) about 10 to be a citizen</comment></comment></comment></comment>


$25k signing bonus for any Yahoo employee who joins Yammer in the next 60 days - <AUTHOR> https://twitter.com/davidsacks/status/180399269801504769

<comment><USER2> Sick!! No more ridiculous lawsuits... Yammer talking action on behalf of the whole tech community!! Much good karma to you..</comment>


Brit start-ups invade the Valley - <AUTHOR> http://news.bbc.co.uk/1/hi/technology/7362022.stm <comment><USER2> These articles make me cringe. They are typical of my (former) country's need to sound like it's important. Here we have a _government_ scheme to send some people on a trip to San Francisco. Hell, with the dollar/pound ratio the way it is any hacker worth his code is going to make that trip just for the hell of it.

It reminds me of a scene in the truly awful BBC TV series Spooks in which one character is given a very small tracking device and the MI5 man says with pride "Even the Americans don't have technology like this".

The "Web Mission 08" web site tells you all about the 'requirements' to get in this program:

DO I QUALIFY?

Web Mission 2008 is open to web companies that:

\+ Are innovative \+ UK headquartered \+ Have 2 years trading history, or failing that, compelling early-stage fast-track potential \+ Can provide references from key sponsors/industry players \+ Can Demonstrate some commitment to sustainable business practices (eg, positive environmental or social outputs) \+ Are ready to do business in the US OR potentially attractive to US investor

References? Sustainable business practices?

Poppycock!

The full list of companies is here:

<http://www.webmission08.com/whoisgoing.html>

I have nothing against the companies involved, but surely you didn't need a government leg up to get you to SF?

</comment><comment><USER3> "a government-backed project to provide a window into the UK's start up scene"

Spot the obvious contradiction, right there. There are startups in the UK and just about anywhere you care to look, but there isn't a "scene" per se. If there were, it wouldn't need a "government backed" road-show to promote it, particularly not in Silicon Valley. If the UK "scene" is so vibrant, why would anyone from over here care too much about over there?

</comment><comment><USER4> Urghhh, I'm British and can't stand stuff like this. I got off my ass and went to San Jose and San Francisco on my own, met people via online networking and found it very successful.

</comment><comment><USER5> Auctomatic was founded by UK based founders. It was the first UK YC startup.</comment>


Work Is Work - <AUTHOR> https://codahale.com/work-is-work/ <comment><USER2> As a small part of a massive digital transformation project last year, it was clear to everyone involved that no amount of good organizational design could succeed without significant adjustments to organizational behavior and culture.

We made all these logical plans but implementing them was about as effective as dipping them, bloodsoaked, into the piranha pit of legacy culture.

> A better model for staying informed of developments as the organization > scales is for groups to publish status updates as part of the regular > cadence of their work.

I mean, it sounds great. But politics just eats this stuff alive. And the more I felt empathy for the various factions, the more I realized that real, honest change takes unrelenting and sheer force of management will. Which is itself a kind of culture.

<comment><USER3> >> honest change takes unrelenting and sheer force of management will

Having also experienced this firsthand, I've come to learn that:

\- there are people in the world who have difficulty changing, despite best of intentions

\- there are people who are narcissistic, psychopathic, or toxic in other ways

\- there are people who are lazy and are just clawing and scratching to keep their paycheque because it's basically free money due to not adding commensurate value back to the organization

\- there are people who are in over their heads and find it difficult to improve, no matter how much money you may provide for training budgets, you're basically paying their salary to be full-time students at that point, not employees, and they still fail to graduate from school

Take all of the above, let it rot in a stale culture for a decade, you can easily see why an organization might get into some difficulty. Change is hard and unfortunately sometimes involves layoffs if the organization is to thrive (note I said organization, not company, affects organizations of all types).

Change is hard because people are hard. People are hard because... many reasons. Not everyone can be a driven Steve Jobs to achieve success, damn the torpedoes and other people's feelings, especially if they're just the average joe.

The easy answer is to fire these people. I don't like it, but I don't have a better answer. It sucks. I went through it before, refused to fire anyone for years, believed in people's ability to change. The scar tissue has caused me to completely change my mindset moving forward. If you can't change, you need to be on a performance improvement plan, if you can't achieve it, you're out the door, but hopefully we work at an organization willing to give you some nice severance so that you don't immediately end up on the street.

<comment><USER4> I was discussing this issue with somebody that lead an engineering transformation effort to drag an organization of around 2000 developers towards modern development practices. They had accreted so much tech debt and bad ways of working that their velocity was too slow to allow them to thrive in a changing market, it took forever to get things into production.

I would have expected that at least the developers would have been on board with this change but the amount of resistance was surprising. They ended up closing 2/6 of their offices, letting the bulk of those employees go and then opening a new office.

</comment></comment></comment><comment><USER5> This is a very interesting article, and it contains something that seems true at face value but that I feel compelled to disprove:

> The work capacity of an organization scales, at most, linearly as new > members are added.

Let's go back to separating cotton fibers from its seeds, a laborious process. If you had a factory of 100 workers doing this and added another one, then you can expect the work capacity to increase 1%. But what happens if you add a worker that is called Eli Whitney, who makes a machine that automatically separates those elements? You definitely don't get a 1% increase in capacity since the new combination of everyone's capacity is more than 101%.

So the author goes on explaining that capacity is not the same as productivity, that the important task is to change the work itself. But is capacity measured _only_ as the hours worked? An extra CPU processor does not do the same as an extra GPU or TPU processor. I've been playing with Coral.ai and I'm totally mindblown at how my raspberry pi can do ML!

Going back to the company, if your bottleneck in the company is e.g. devops and you hire a devops person, then productivity will increase super-lineally; because with this being the bottleneck it means that _other resources were underused_. So the new person will not add +1/n of capacity. Sure they might add 1/n of the _working hours_, but the capacity itself (and in all likehood the productivity) will increase more than 1/n.

<comment><USER6> I don't think what you're saying contradicts the article.

Especially the example of the devops bottleneck -> hire a devops team. That's more or less explicitly covered: "Prioritize the development of force multipliers".

The idea is that adding a new person to a team increases the work capacity of that team linearly _to do the work that the team has been doing, in the same way they 've been doing it_. Finding new, more efficient ways to do the work is how you increase productivity, not work capacity. Hiring an Eli Whitney means that your work capacity is now devoted to doing something else (operating a gin rather than separating by hand), something that has higher productivity per time unit.

<comment><USER7> This doesn't remove the contradiction, it only moves the contradiction into the article.

It's not as though the article makes a soft claim which should be interpreted generously, it makes a very strong claim - "at most linearly" \- despite the fact that there are all sorts of exceptions. That is certainly not "at most linearly", perhaps "at base case linearly".

</comment></comment></comment><comment><USER8> This post reeks of mathiness and pseudoscience and analogies taken too literally. Sure there are some interesting ideas, but without evidence, there's not much to it. You want to say that communication costs grow quadratically? You could "show" it with an argument about dyads, but that's not the kind of thing to believe without data. Or that productivity per person is at best constant as an org grows. Or most assertions in this post.

<comment><USER9> One of the obvious issues is that human communication is not just point to point.

We can all get in a room together or read an email chain at once.

</comment></comment><comment><USER10> I feel I myself come from a similar background to the author. We both think heavily from time to time on the theory of work in philosophical, management theory, mechanistic and (computer) scientific frames. We see correlations. However, I always fear I'm falling into some trap of reductive reasoning or worse, finding I'm really an Anti-realist at heart.

The author would do well to look into management theory, initially Harold Leavitt's Diamond in "Applied Organization Change in Industry" now more commonly distilled down to golden triangle of "People, Process and Technology".

I don't know the answer, not can I offer any overarching counter-argument but I feel a deeper understanding of lessons learned in management theory and operating models for the author might help in their next iteration.

</comment><comment><USER11> One of the basic parts of the author premise is just wrong:

> The work capacity of an organization scales, at most, linearly as new > members are added.

Maybe asymptotically, but usually - not at all. Different hires have specialty skills which existing employees/collaborators simply don't, and would have taken extremely long, or forever, to do what the new person can achieve handily. Also, some tasks are simply better geared towards multiple people collaborating; simplest example: assembling large IKEA furniture...

</comment><comment><USER12> Roughly, if an org is considered as a system, then entropy of it will continue to increase. To keep the order, more and more energy needs to be spent. This energy is in the form of developing internal tooling, more explicit communication which is repeated often, hiring more, establishing processes (which in turn itself adds more tasks on people's plate), etc

</comment><comment><USER13> I believe there are two sides I'm aware of when it comes to people. Firstly there exists persons that consider work as another extension of life with social merit. Secondly the persons that think of work as work and disregard any social merit because they care more about it outside of work.

In my opinion if we shut out the first classification we're making these persons miserable because they see social merit in work. I also believe work has become overly optimized compared to previous years and where people should still be able to enjoy social merit if they desire it.

</comment><comment><USER14> > Keep the work parallel, the groups small, and the resources local.

That how the military gets things done, by having a trained officer corps which can think on a local level and the higher up give general guidelines for how to proceed in different situations.

<comment><USER15> I guess it depends what you mean by that exactly. Military "group" sizes range from squads (~30 people) to corps (~50,000 people). According to the article, the dyad sizes here would be huge; but my hypothesis was that point-to-point communication is often _not_ necessary in military organizations due to the rigid structure.

<comment><USER16> The military - and many essential services like police, fire, healthcare - differ in that a majority of the work is not "doing" but "preparing". The military does not optimize towards cheaply increasing daily metrics of death and destruction, rather, it aims to use a targeted amount of force to achieve a particular national objective at the exact moment demand for that objective arises. Most units sit idle most of the time, in a runtime system with very low latency and variable throughput. The same analogy is obvious when you think of police(we don't want crimes committed), fire(we don't want lots of fires to fight), healthcare(we don't want tons of patients).

And that leads towards the extensive investments in training, doctrine, equipment, logistics etc. And so the majority of military employees are not "doing work" in the sense that a tech startup is, where every employee is directly engaged in optimizing the system's value chain - that kind of work occurs only in the core command, around the generals and planners and architects and researchers of the next war. The majority of the work is simply maintenance of the system so that the needed response is possible when called for.

And I believe this examined difference in organizational purpose also applies across a lot of businesses. The value chain is always shifting in surprising ways, but generally in the direction of lower maintenance.

<comment><USER15> Excellent explanation! The difference between these kinds of "work" is a very poignant distinction.

</comment></comment></comment></comment><comment><USER15> > The work capacity of an organization scales, at most, linearly as new > members are added.

This is going to sound kind of grim, but a counterexample here is, e.g., a Southern plantation where children of slaves were also born into slavery, yielding a superlinear work capacity. With that said, even though I intuitively agree with the statement, I think other (more contemporary) counterexamples might exist.

> Coherence costs grow quadratically as new members are added.

I see the point here, but I'm not sure how I can reconcile it with the reality of modern-day armed forces: where millions of organizational members function extremely efficiently (possibly due to very rigid organizational structures -- which the article doesn't really touch on). It seems there's a ratio between dyads and "organizational strength" which probably yields "true" coherence.

<comment><USER17> I don't think your slavery example is really a counterexample... those children are new members. The linear scale is about people to productivity, not 'cost of a new worker' vs productivity.

Also, the military is different; it is not really trying to produce anything new, so isn't the same type of work.

<comment><USER15> > The linear scale is about people to productivity

According to the article, the scale is about "hiring" to "work capacity" \-- the author specifically points out that productivity is another ballgame altogether. (FWIW, I'm not sure what "hiring" means in the context of slavery -- I suppose a Roman-style post-war enslavement could be one equivalent.)

> Also, the military is different; it is not really trying to produce anything > new, so isn't the same type of work.

Yeah, this is a good point. The military is not as "creative" as your run-of- the-mill corporation and, as such, probably doesn't need the coherence.

</comment></comment></comment><comment><USER18> I feel like this is what an engineer who doesn't really understand businesses would write. The models he writes about don't even come close to the complex realities of politics, externalities, market factors, accounting, funding, etc... When he started talking about workers like cpu cores, I knew immediately, this is not going to go anywhere.

<comment><USER19> If you know more, please share some of what you know so the rest of us can learn. It's less helpful just to talk about what someone else doesn't understand.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

</comment></comment><comment><USER20> I will note that I have written the following in jest, due to the title of the post. I have not read your post, before writing the following statements: Arch biet Mach Frei - There was a man once, who believed as such. Although there were a great many more under him, who didn't much like what he had to say. Even though quite many simply went along with it, because it seemed easy, comforting, one might even contend that it was the right thing at the right time, while the others vehemently opposed such thinking and critique.

<comment><USER21> > I have not read your post

Then don't comment?</comment></comment>


Why Rust is the future of robotics - <AUTHOR> https://medium.com/pollenrobotics/why-rust-is-the-future-of-robotics-81d7fb68fe37 <comment><USER2> > Race conditions, deadlock, data corruption can cause bugs that are complex > to debug.

> Rust can address all these problems, and more.

Rust does not prevent deadlocks.</comment>


80,000 job losses, unemployment spikes - <AUTHOR> http://money.cnn.com/2008/04/04/news/economy/jobs_march/

<comment><USER2> "It's not going to be a lot of fun. Recessions are never fun"

I laughed when I read that.

What I would actually be interested in, though, is a study on how difficult it is for the unemployed sector to find jobs if they are completely open to relocating, retraining, etc. It was one of the things I couldn't understand when watching Moore's "Roger & Me" (though not the only thing I was frustrated with, to be sure): Why, if you can only pay for one more month's rent, at which point you will be kicked to the curb, and there is absolutely NO hope of finding a job in your own town, would you pay that month's rent? Why wouldn't you take the money and make a last ditch effort a few towns over, where the chances of becoming employed are higher?

<comment><USER3> There are lots of reasons someone would not want to leave their current desperate circumstances for another location. People have family and social support networks which won't make the transition. Sometimes even finding transportation is a barrier. Fear of the unknown can also be very strong. Hopelessness can make one believe that no amount of effort will fix the situation, and that moving to a new area will do no good. And some people just... give up. It's so easy to get caught up in our struggles that we forget to look for a way out.

Personally I've moved many times - since I was a child my family moved every three or four years. We all have the wanderlust. Some of it was for education, some of it was for new jobs, and honestly sometimes it was because we wanted a change of pace. I have a hard time fully comprehending it but I've met many people over the years who are born, and will live, retire, and die in the same town where their parents did the same.</comment></comment>


Serverless Architecture – All you need to know about caching - <AUTHOR> https://theburningmonk.com/2019/10/all-you-need-to-know-about-caching-for-serverless-applications/ <comment><USER2> So I've got a site on S3 Cloudfront. I have an API gateway origin. I need to call an external API. The auth works by constructing a JWT, POSTing it to a token endpoint, and including that access token in requests to the API. I need a way to cache the access token.

1\. Call API with Lambda proxy with the access token stored as a global variable outside the handler. Would be reused but like article mentions not guaranteed and can miss often. 2\. Store access token client side (cookie). This would require one call per client even if the access token already generated. 3\. Store access token externally (S3 or secrets manager?). Latency would increase, but could be combined with #1.

Any other way to do this?

<comment><USER3> Proxy the calls to the external API via your own gateway. You'll get better control over caching behavior, push complexity away from the client, and have more control over how you handle dependency failures. Also, you should load test lambda with expected traffic patterns to actually validate that container-based caching is insufficient. Dynamo also has good in-dc latency, with very good scaling behavior.

<comment><USER2> interesting. yeah I'm definitely proxiying all API calls through my own gateway. and yeah I'll have to run some tests.

</comment></comment></comment><comment><USER4> 2 modal opt-in popups. 1 is standard. 2 is really pushing it.</comment>


A quick intro to writing a parser with Treetop - <AUTHOR> http://thingsaaronmade.com/blog/a-quick-intro-to-writing-a-parser-using-treetop.html <comment><USER2> This page would be more readable if there wasn't a huge black bar overlaid across the middle of it.

<comment><AUTHOR> Would you be able to tell me what your screen resolution is?

I decided to go with the bottom-attached menu because it makes navigation painless, on the resolutions I've tested the site with the difference between the above-menu area and the menu area made reading seem fairly natural.

<comment><USER3> Even at 1680x1050, I find attached menu's really grating. I think it's the lost screen real estate because the larger they are, the more irritating I find them. On the other hand, Twitter went ahead and implemented a top attached menu, so maybe I'm in the minority.

Anyways, back on topic, do you (or any HNers) have any recommendations on how to get started with formal grammars and parsers? Is there a canonical introductory text?

<comment><USER4> Niklaus Wirth's _Compiler Construction_ (free online, <http://www- old.oberon.ethz.ch/WirthPubl/CBEAll.pdf>) has a good intro to parsing, though for better or worse it's skewed towards recursive-descent parsing and has a "hit the ground running / focus on theory later" style.

After that, you could dig deeper with Andrew Appel's _Modern Compiler Techniques in {ML,C}_ books. The ML one is better, IMHO. Those cover other methods (LL, LALR, SLR, etc.) in greater detail, and I'd also recommend either for learning compilers in a heartbeat. (Appel's _Compiling with Continuations_ is also excellent, but doesn't cover parsing.)

Following that, Dick Grune's _Parsing Techniques: A Practical Guide_ (<http://www.few.vu.nl/~dick/PTAPG.html>) is a good reference...and thorough. If you're reading it to learn the basics, it might seem a bit dry, but I think it's at a sweet spot between depth of coverage and deference to the extensive bibliography. I have the second edition; the first is free online. Not sure about the differences, but the coverage of fundamentals probably haven't changed much. Also, while the other two are compiler books with chapters on parsing, this one is 100% parsing, and gets to a lot of interesting parsing algorithms (e.g. Earley parsing) that don't usually get much love in compiler texts.

Some people will also recommend the Dragon book, but I think those three will be more helpful. I haven't read the new edition, but the old seems drier & less thorough than the Grune book, less direct than the Wirth book, and less modern than the Appel book.

Also: For learning lex and yacc, the intros in the _4.4BSD Programmer's Supplementary Documents_ ("PSD", included with OpenBSD and probably the other BSDs, and not hard to find online) are hard to beat. The O'Reilly _Lex and Yacc_ book somehow manages to be roughly ten times as long yet less informative.

And if you have full control over the syntax used, S-expressions (Lisp), RPN (Forth), or Lua/JSON will let you dodge the issue of parsing entirely.

<comment><USER3> That is likely the most thorough, informative response I have ever received. I almost qualified that with "on the internet", but then I realized, hell, it's probably more informative than any response I've ever received _anywhere_.

I recently saw a news piece about a philanthropist helping people in poor African villages by simply building wells. The tribesmen were so grateful, one said "I wish he lives one hundred and fifty years so that he can continue helping people like us."

Well, Scott, thank you. I hope you live one hundred and fifty years.

<comment><USER4> There's some real gems in the archives here. For starters, see <http://news.ycombinator.com/item?id=835020> . More generally, try googling _site:news.ycombinator.com keywords_.

And, thanks. :)

</comment></comment></comment></comment></comment></comment><comment><USER5> This example uses Treetop. The same variety of parser, PEG (parsing expression grammar), but not code-generating, but dynamically defining ruby code is Citrus. It was really a pleasure to work with. The difference is that you don't need a preliminary compile step in a rake file e.g., which I like better for a language as dynamic as ruby.

<http://github.com/mjijackson/citrus>

<comment><AUTHOR> The intermediate compilaton step is actually optional when using Treetop. You can see in the code in the article that Treetop is directly loading and interpreting the grammar at runtime. (Of course there is always going to be some compilation 'behind the scenes', but there's no need for it to be explicit.)

Citrus looks interesting! From what I can see the PEG syntax used by Citrus is _very_ similar to Treetop. I'll definitely check it out more later, I'm particularly interested in performance difference between the two.

<comment><USER5> In fact, you're right, it doesn't necessarily write a source file. I missed that last time when I looked. But Treetop will still create ruby code, write it into a string and then eval that. I didn't find that approach really elegant.

<comment><AUTHOR> I do agree to an extent. I'd be interested to see if Citrus is faster... I'll write up a test in a week or two and we shall see!

</comment></comment></comment></comment><comment><USER6> My favorite right now is PEG.js: <http://pegjs.majda.cz/>

</comment><comment><USER7> ctrl-f grammer

Really?

<comment><AUTHOR> Fixed. It must have been a late night!</comment></comment>


PeerTube v3: Progressive Fundraising - <AUTHOR> https://framablog.org/2020/05/26/our-plans-for-peertube-v3-progressive-fundraising-live-streaming-coming-next-fall/ <comment><USER2> If you feel like contributing, here is the link [https://joinpeertube.org/roadmap](https://joinpeertube.org/roadmap)

(I'm not affiliated to PeerTube, I'm just a user)

</comment><comment><USER3> I just put 10 EUR on it

I think this project has a chance in ten of working. But I hope it does.

I set up Riot with a group of friends in an old used laptop last year, and it has been working surprisingly well for us. I hope we can try this out soon!

</comment><comment> vertex-four Note re live streaming: a minute of delay between streamer and audience makes interactivity (even just Q&As at the end of a presentation) essentially impossible, and Peertube's particular design makes it unlikely that'll improve by much. It does make it somewhat cheaper to host though.

I'm working on a self-hostable live video streaming platform, building on solid, well-known technology like MPEG-DASH's low latency mode and a relatively dumb (and easily replaceable) content delivery network.

Preliminary tests suggest that my system can get to about 3 seconds, which makes interactivity between streamer and audience a lot more feasible. The downside is that this costs money - about 12 cents per GB through e.g. Fastly, on top of hosting and origin bandwidth costs.

<comment><USER4> Whatever happened to the plausibly Twitch-killing, super-low-latency startup "Beam" that won TC Disrupt circa 2016?

<comment> vertex-four Apparently it got bought by Microsoft and renamed Mixer.

<comment><USER4> Thanks. I asked, bc I had a vague recollection of their having open-sourced a key part of their astonishingly low-latency streaming infr, and that's the 1st thing I'd look into if I were entering this space.</comment></comment></comment></comment>


Google Caffeine - Faster realtime search index - <AUTHOR> http://www.techcrunch.com/2009/12/28/google-caffeine-faster-search-index/

<comment><USER2> Google's indexing is already scary fast, I'm excited to see what this new update they've been testing for a while will do once they roll it out completely.

</comment><comment><USER3> "SEO firm Summit Media suggests..."

Prediction: the next batch of SEO snake oil/spam trap techniques will involve making pages look "fresher" than they are.</comment>


List of eponymous laws - <AUTHOR> https://en.wikipedia.org/wiki/List_of_eponymous_laws <comment><USER2> This list should be named after someone, so that we could refer to their eponymous list of eponymous laws.</comment>


Ask HN: Why most of the Game Engines were written in C/C++? - <AUTHOR>

I was going through the list of game engines on Wikipedia (http://en.wikipedia.org/wiki/List_of_game_engines) and was amazed to see that the primary programming language for most of them was C/C++. <comment><USER2> Another factor is predictability. When you write a line of code in a low-level language like C, you have a reasonably accurate approximation of what the machine is actually going to do for you at that point. Moreover, things like memory management are manual and explicit, and you have a reasonably good idea of when they happen and how long they take. In comparison, a lot of higher languages have a significant and not necessarily predictable overhead in the runtime and/or VM.

</comment><comment><USER3> Uhm performance?

<comment><USER4> Could you be a little more specific? What makes C++ faster than, say, Visual Basic 6? Just curious.

<comment><USER3> C is designed and optimized to be very close to the hardware, in fact it's called high-level assembler. You can access specific hardware features directly. C++ is the OO version of C (yes I know this is an oversimplification). Other languages, like for example Java creates a logical layer (known as the java virtual machine or JVM) which adds more complexity and makes it harder to tweak your code to fit the particular hardware.

</comment></comment></comment><comment><USER5> They don't have garbage collection. C++ is still the most capable no-garbage- collection language.

</comment><comment><USER6> OpenGL/DirectX APIs are the reason

</comment><comment><USER7> because they are the best languages.</comment>