To Remember Everything You Learn, Surrender to This Algorithm (2008) - <AUTHOR> https://www.wired.com/2008/04/ff-wozniak/ <comment><USER2> Few people have tried Anki (a free alternative to SuperMemo). Even fewer have tried its add-ons, which is where it truly shines! Vanilla Anki is mostly good for text flashcarding, with the phone app helpful for making use of those little bits of downtime throughout the day.

[https://www.youtube.com/watch?v=c-4xOe79epU&list=PL3MozITKTz...](https://www.youtube.com/watch?v=c-4xOe79epU&list=PL3MozITKTz5YFHDGB19ypxcYfJ1ITk_6o)

[https://www.youtube.com/watch?v=QzBoDe3PgAc&list=PL3MozITKTz...](https://www.youtube.com/watch?v=QzBoDe3PgAc&list=PL3MozITKTz5Y9owI163AJMYqKwhFrTKcT)

Glutanimate has taken Anki to a whole new level. He created a painless UI for occluding any part of an image and making Anki flashcards out of it (Image Occlusion Enhanced). I used it to efficiently create and memorize thousands of anatomy flashcards. Ordered lists can be a real pain to memorize, but when not missing steps is crucial (as it is for medical OSCEs), Cloze Overlapper has worked very well.

Anki lets you share your decks with others, and medical students have collaboratively created very high quality study decks for studying for the various STEP exams.

To be honest, I was always a fairly mediocre student. Big-picture concepts and figuring things out on the fly, no prob. But I've always struggled to nail down bits of information long-term or learn sequential information, meaning calc and ochem required inordinate amounts of time. No longer the case. Spaced repetition plus visual/text/auditory learning is a recipe for success.

The possibilities here should not be underestimated. Folks on Hacker News have talked about the value of information commons, and this right here is the next best example behind Wikipedia I have ever encountered. While quality collaborative decks exist for things like medical school, learning languages, and more, none exist specifically for the program I am in (PA school). I am currently creating my own with the goal of it being the core of a quality collaborative deck. I am using Gephi to spatially organize concept maps for diagnoses complete with incidence and strength of associations, when such information is available. I then export these maps to Anki, use Visual Occlusion Enhanced to block out the information I want to recall, and voila: time-efficient first-time learning and long-term retention.

<comment><USER3> I'm using DroidAnki on my phone as part of my efforts to learn Mandarin. It took a bit of time to figure out how best to configure it, but now that I have it tuned to my liking, it's a great tool. It's not the only tool I've been using, but it's one of the most helpful.

<comment><USER4> Yeah I've got the iPhone version of the Anki app. There's a good Mandarin deck called Spoonfed that gives you sentences instead of just words. Much better. But, by far the most effective way to learn the language is to practice with natives. Which I have been doing for a few weeks and the results are amazing.

<comment><USER5> Can you link to the iOS app? I see several in the App Store and I'm not sure which one to try.

<comment><USER6> Download links for all versions of Anki can be found on [https://apps.ankiweb.net/](https://apps.ankiweb.net/) (And yes, Anki for iOS costs money.)

<comment><USER5> Thanks!

</comment></comment></comment></comment></comment></comment><comment><USER7> I used Anki (similar to SuperMemo) to study for my PhD qualifiers and it definitely helped. I kept up the cards for a while after that, adding more and more. I'm convinced I would have not passed the measure theoretic probability theory class I took without spaced repetition software. The software also proved very useful when I would have a technical idea and had no reference on me as I could remember the details of many results. It seemed to accelerate my thoughts in a way too by removing the delay to look up some information.

Eventually, I did not have the time or need to keep reviewing the cards, so I stopped. But I intend to pick it up again after cleaning up the cards after my PhD. I highly recommend the software.

One piece of advice: I treat these flash cards as basically fast tests that trigger a review if you fail. I recommend this approach as many people seem to think flash cards can only be definitions. Some of my cards are definitions, some are short problems, some are proofs, etc. The definitions are unit tests. Problems and proofs are integration tests. Both are necessary.

<comment><USER8> I credit Anki with allowing me to successfully leap from ivory-tower academic to professional software engineer. I recognized right away that most academics would scoff at lowly "plebian" tasks like "actually learning the details of a programming language". So I forced myself to push through that prideful bottleneck and essentially memorized several programming manuals cover-to- cover using Anki.

<comment><USER9> Well, IMHO (no offense meant), memorizing programming manuals makes one a good lonely programmer.

To become a 'professional software engineer', one must learn to be a team- player within a group of persons tasked in building and maintaining a piece of software that will stand the test of time. Which requires a very different skill set. One can be a good SE and still be relying on auto-completion and inline help in one's favorite text editor.

Remembering the API of your languages definitely helps, but actually building softwares that will be used by real users, for years, that's what makes you a 'professional software engineer'.

<comment><USER8> Sure, but teamwork isn't everything. Or studios would hire cheap communications grads and there'd be no need for CS programs.

</comment></comment></comment></comment><comment><USER10> > "To this day," Bjork says, "most people think about forgetting as decay, > that memories are like footprints in the sand that gradually fade away. But > that has been disproved by a lot of research. The memory appears to be gone > because you can't recall it, but we can prove that it's still there. For > instance, you can still recognize a 'forgotten' item in a group. Yes, > without continued use, things become inaccessible. But they are not gone."

This was quite eye opening to me when I first heard of it. Apparently, our long term memory is virtually limitless. You never truly lose something you once had learned well.

Another fact about memory that I found equally interesting: memory is not a "thing" but a process. When you recall something, you are actually reconstructing the memory from various factors associated with it, pretty much like how a paleontologist might reconstruct the image of a dinosaur from fossil remains.

<comment><USER11> It seems to me that there are two gates, and they move with age, and the rate at which they move differs from person to person. The first gate is the time until losing something from associative memory. The second gate is losing the memory completely. Yes, this does happen, and it happens to everyone. But the quote still has some validity in that many things we think we forgot are still buried deep inside.

</comment></comment><comment><USER12> The best and most practical introduction to spaced repetition is Michael Nielsen's _Augmenting Long-term Memory_ ([http://augmentingcognition.com/ltm.html](http://augmentingcognition.com/ltm.html)), already discussed in [https://news.ycombinator.com/item?id=17460513](https://news.ycombinator.com/item?id=17460513).

</comment><comment><USER13> It's quite interesting that Anki(It's a Japanese 暗记 for those who don't know where the name comes from) is mentioned a few times but no body mention "Memory palace"[1] and "Major system"[2] yet. Both methods try to hack some features/bugs of our brains built by mother nature during million years of her evolutionary development. Most memory athletes use some variation of these methods to achieve astonishing performance. There's a Dr. Yep can remember a 65 thousands of English word and the locations (i.e. page number and the sequence on that page) of each word on a Chinese/English dictionary.

The trick is investing some time to build a personalized memory system, then use the system to remember other things like passport number, credit card , phones etc. It might not be attractive to most people because today we have smart phone to help us. But for learning new knowledge, combination of both space repetition and memory system is quite interesting. Here's what I did: I use a Anki alternative called Memrise[3] to build a Major system. (I also have Anki installed but just prefer Memrise). Then I use Major system to review the concepts I've learned. Major system here is the equivalent of Anki as a tool for short term place holder of space repetition, not directly used for learning. I found brain Anki is better than smartphone Anki/Memrise although I built former Anki with later Anki.

Come on fellow hackers, hacking your brain is very rewarding

[1][https://artofmemory.com/wiki/Method_of_Loci](https://artofmemory.com/wiki/Method_of_Loci)

[2][https://artofmemory.com/wiki/Major_System](https://artofmemory.com/wiki/Major_System)

[3][https://www.memrise.com/en/](https://www.memrise.com/en/)

<comment><USER14> [https://artofmemory.com/wiki/Person-Action- Object_(PAO)_Syst...](https://artofmemory.com/wiki/Person-Action- Object_\(PAO\)_System)

I considered working on this, but I'm just not excited enough about remembering. Thanks google. Oh and Einstein said (according to Shaq) you should never remember what you can go lookup.

<comment><USER15> The beauty of the human memory system is not whether or not you can remember a thing, but the correlations and relationships between them. A convenient lookup mechanism doesn't help there.

</comment></comment></comment><comment><USER16> For language learning, consider Pimsleur, who discovered 'the algo' previously:

 _"Graduated Interval Recall - Dr. Pimsleur's research on memory was perhaps one of his most revolutionary achievements. He discovered that if learners were reminded of new words at gradually increasing intervals, each time they would remember longer than the time before. He documented the optimal spacing for information to move from short-term into long-term, or permanent, memory. This theory is at the base of all the Pimsleur programs."_

[https://www.pimsleur.com/the-pimsleur-method](https://www.pimsleur.com/the- pimsleur-method)

<comment><USER4> I'm a big fan of the Living Language package.

</comment></comment><comment><USER17> > _how to become a genius: You must clarify your goals, gain knowledge through > spaced repetition, preserve health, work steadily, minimize stress, refuse > interruption, and never resist sleep when tired._

<comment><USER18> Maybe it's just me who after years at uni is sick of viewing everything from this impoverished academic point of view but I think these statements are not helpful at all. People worship advice/knowledge these days and totally forget about the doing.

The statement does not deny that actually doing stuff is important - "work steadily". However, these are just empty words to someone who tries to become a genius by hitting the books first.

My advice (yes, I do see the irony here) in this age would be to postpone seeking advice or reading about a topic as long as possible. As a result, your common sense sharpens and you may even come up with fresh, creative ideas that others perhaps didn't have.

<comment><USER19>
      > postpone seeking advice or reading about
      > a topic as long as possible

Oh gosh, the number of inefficient solutions, horrible architectures, brain- dead encryption systems I saw, because of the people who did not bother to open a book.

To come up with creative solutions you need to have a base for said creativity. I.e. to connect the dots in original way you need to have the dots already.

I have nor proof, but I suspect that the number of useful solutions created by people who know nothing about the field is close to zero.

Somehow I doubt one would see much further when instead of climbing onto the shoulders of giants they prefer to stay in the pit of their own ignorance.

<comment><USER14> > Oh gosh, the number of inefficient solutions, horrible architectures, brain- > dead encryption systems I saw, because of the people who did not bother to > open a book.

If you hadn't come across horrible architectures you would have learned less about good ones. It would be nice if this all happened inside of one person, that's what the parent is suggesting.

</comment></comment></comment></comment><comment><USER20> This type of learning, combined with applying the knowledge learned, is easily one of the biggest lifehacks I've encountered.

Growing up, I (falsely) internalized that I somehow had a hard time learning languages. However, with the help of Anki, that has an algorithm based on SuperMemo, I realized that this wasn't the case - just that the traditional textbook methods weren't for me when it comes to cramming in "arbitrary" information.

While living in Japan, I used Anki to supplement my recognition of kanji. I averaged roughly 7 minutes of studies for a year, internalizing the meanings and readings of roughly 1000 characters with a 95% retention rate.

Judging from my peers, the only ones that managed to pick up this volume were those that studied the language full-time. Of course, they learned the language overall at a much higher level than I did - this methodology purely focused on the recognition aspect.

Naturally, I got to put the characters to use living in the country, and I also spent time making up mnemonics for each character. Your mileage may vary.

<comment><USER21> I'm curious what your mnemonics are like, could you give an example or two?

I'm not a student of Japanese, but AIUI there are pictographs and ideographs amongst the characters, presumably these are effectively already mnemonics?

<comment><USER6> That might have been true of the original bone script characters, but over the millennia they have been stylized beyond recognition, and many new characters were created. Modern Japanese or Chinese is about as mnemonic as the typical icon set used in a GUI application.

However, most characters are composed of a few common components, called radicals, which can sometimes provide hints about the meaning. For example, the Chinese words 海 "sea", 波浪 "wave" and 渴 "thirsty" all contain the water radical ⺡. Similarly for 丝绸 "silk" and 缠绕 "wind around", which contain the thread radical ⺓.

Personally, I don't use any mnemonics, but I know someone who does, and he'd assigned each radical a memorable representation (e.g. the thread corresponds to Spiderman) and then he had a story linking a word to the components, so that 丝绸 would be about two Spidermen, one of which wraps himself in silk to use it as a ghost costume.

<comment><USER21> Cool, thanks for the lesson and info.

</comment></comment></comment></comment><comment><USER22> I use Anki. It really does work. A few things I found:

1\. It takes discipline to understand the material properly before attempting to just bang it all in Anki, but learning directly from Anki can be miserable, so it really is best to try to grasp it fully before going near Anki

2\. Creating the right questions can be difficult - sometimes I found myself able to regurgitate the answer but other tests proved I didn't really understand the meaning. One way to approach this is to test different aspects of the answer, or to add the inverse of the question. Still even if you don't grasp the meaning properly just having a rote answer in your head can improve your capability

3\. Life often gets in the way of the discipline required!

</comment><comment><USER23> It would make sense if you could get electronic textbooks which are already in the question/answer form required by spaced repetition software, ready to "load and learn". Rather than reading a traditional textbook, you would take an equivalent question/answer set and learn it by spaced repetition. Do such things exist?

Following on from that, maybe such question answer sets could be automatically generated from the semantic web? A simple example would be a question/answer set about capital cities, but could one pick an arbitrary subject and manufacture a list of questions about it?

<comment><USER24> > It would make sense if you could get electronic textbooks which are already > in the question/answer form required by spaced repetition software, ready to > "load and learn". Rather than reading a traditional textbook, you would take > an equivalent question/answer set and learn it by spaced repetition. Do such > things exist?

I am working to solve exactly this. A form of learning where the learning happens due to interaction between a bot and a human. Machine teaching humans basically. Been working for over 20 months. Will be releasing within two weeks. :)

<comment><USER25> I would like to know more about this. Do you have a mailing list I can add myself to for a notification when you release?

<comment><USER24> Thanks. You can sign up at this mailing list: [https://tinyletter.com/primerlabs/](https://tinyletter.com/primerlabs/) here. I created it just now.

</comment></comment></comment></comment><comment><USER26> This is by far my favorite piece of software I've ever purchased (and one that's keeping me on Windows). It's very obtuse to use and not really user friendly at all unless you're just making flashcards with it. I get the impression that it's somewhat dogfooding taken to the extreme, and it's almost as if they're selling it as a side project.

There are so many random features like the sleep tracker ([https://www.supermemo.com/help/sleep.htm](https://www.supermemo.com/help/sleep.htm)) that don't really make much sense to have but are there anyway.

There's also a web version if you just want flashcards that's mainly focused on language learning. They sell some courses and you can add your own cards too.

The biggest selling point for me is the incremental reading ([https://www.supermemo.com/help/read.htm](https://www.supermemo.com/help/read.htm)) feature. It's the only way I can manage to keep track of so many articles that I find interesting, and is vastly superior to leaving hundreds of tabs open or a bunch of bookmarks I'll never look at. It's superior to instapaper,readitlater etc because I don't have to decide what to read when I open the app, and I don't have to worry about forgetting about the article if I stop in the middle of it.

If I see anything remotely interesting, I just chuck it in Supermemo, and I'll see it eventually. If it's really important, I'll set its priority higher so I see the article sooner.

If I'm doing research on something and have 15 tabs open, all 15 tabs go into Supermemo, and will eventually get whittled down into flashcards. Start with an article -> extracts -> smaller extracts -> cloze deleted flashcard.

Long comment chains on Hacker News? Supermemo. Youtube videos? Supermemo. Email? Supermemo. Images? Supermemo.

I check my RSS feeds once or twice a week, grab all the interesting articles, import into supermemo, and I'm done.

Supermemo for me replaces: Evernote/OneNote, most bookmarks, Anki, and various todo list apps.

If I just wanted to make flashcards from set material, I'd honestly use Anki, because of the mobile apps. It's really easy to do a few repetitions when you have downtime through the day.

Every person I've shown this app too doesn't really care about it, and I don't really know why. If I had to guess, it's probably because the UI is horrific. I can effortlessly keep up with so much information, and it takes an hour or two out of my day.

<comment><USER27> So for memorizing "stuff" \- does this provide anything over Anki? It looks like Anki does the same thing.

<comment><USER26> No not really, the algo might be slightly "better". But if you only want flashcards, go with Anki. I'd also recommend you make them yourself.

</comment></comment></comment><comment><USER28> I've been doing spaced repetition with Anki since 2013. I wrote about my experience here: [https://www.blinkingcaret.com/2016/05/04/hack-your-brain- lea...](https://www.blinkingcaret.com/2016/05/04/hack-your-brain-learn-faster- better/)

If I could only give one advice to someone about improving their learning skills, it would be to learn about spaced repetition

</comment><comment><USER29> It took me several attempts to ascend the learning curve of Anki, but once I did, it changed my life. Anki Essentials by Alex Vermeer [0] was the introduction that got me over the hump.

Now I use Anki almost daily for learning programming, math, vocabulary, and retaining books I've read.

On books, I originally wrote a script that scraped the Kindle highlights page and converted them into a file that could be imported into Anki. I'd Cloze delete passages and review maybe 20 a day. That practice was so beneficial that I've since started developing it into a standalone product called Readwise. [1]

Readwise doesn't yet incorporate hardcore spaced repetition and active recall principles, but it will soon.

[0] [https://alexvermeer.com/anki-essentials/](https://alexvermeer.com/anki- essentials/) [1] [https://readwise.io](https://readwise.io)

<comment><USER30> Thats awesome! Have you considered Diigo integration?

</comment></comment><comment><USER31> As soon as I read this, I wondered if someone had written some elisp to do it. Sure enough, org-drill[0] exists and implements spaced-repetition learning. In true emacs fashion, it actually implements three different algorithms which one can choose from!

Just looking at it briefly, it appears to be pretty nice.

0: [https://orgmode.org/worg/org-contrib/org- drill.html](https://orgmode.org/worg/org-contrib/org-drill.html)

<comment><USER32> While I'm a pretty great emacs fanatic myself, I think that this is one of the few cases where you'd be doing yourself a disservice by using the emacs alternative. Anki is extremely keyboard-friendly (more so than anything bar Emacs and vim), pretty extensible in a decent language (python), and unlike Emacs in this case, has a huge ecosystem behind it. The only thing missing is the possibility of using Emacs keybindings in text input fields (which is Qt's fault) — if that's important for you, you can use Emacs for input into Anki — google for it, as they're seem to be several options...

FWIW I read somewhere that Anki started off as an Emacs mode, initially, though the only source I could find now (quickly), is this:

[https://news.ycombinator.com/item?id=10925971](https://news.ycombinator.com/item?id=10925971)

Edit:

The Web archive helps (as always):

[http://web.archive.org/web/20091001153225/http://ichi2.net:8...](http://web.archive.org/web/20091001153225/http://ichi2.net:80/flashcard/)

</comment></comment><comment><USER33> Spaced repetition software? A requisite post is gwern's excellent and comprehensive article on them [https://www.gwern.net/Spaced- repetition](https://www.gwern.net/Spaced-repetition)

</comment><comment><USER34> Memrise.com (primarily for learning the vocabulary of another language) combines spaced repetition with mnemonics. When learning a particular piece of vocabulary it will offer up an ordered list of suggested mnemonics based on their popularity in other sets. Extremely effective for its use case.

</comment><comment><USER35> TL;DR: The "algorithm" is spaced repetition. God this was a frustrating article to read.

Spaced repetition does work though, I used a tool called Anki which has a free online sync service and free apps for every platform (bar iOS, it costs £15 on that platform).

<comment><USER36> From Anki manual[0]:

Internally, Anki's spaced repetition system is based on an older version of the SuperMemo algorithm called SM2

[0] - [https://apps.ankiweb.net/docs/manual.html](https://apps.ankiweb.net/docs/manual.html)

</comment></comment><comment><USER36> I just found out that you can use SuperMemo without a computer.

[https://www.supermemo.com/articles/paper.htm](https://www.supermemo.com/articles/paper.htm)

I thought it would be interesting for me to do this method while I'm studying a scientific paper. Any thoughts?

<comment><USER37> I've used the paper method in the beginnings of my English study and it worked well for me. SuperMemo was a really great tool and I liked it very much. Managed to keep using it for maybe 10 years, and I'm still sorry that I've stopped when the program and its community were in decline. But the method and algorithm remain the best and it's great to see it's used in modern applications - not just Anki but perhaps in all major language teaching apps. The fact that this article made it to HN front page speaks for itself.

As for using it for scientific paper study - depends what you expect from it. Spaced repetition is great for remembering and retention of facts. When I'm reading a scientifi paper, I rather struggle with the basic understanding of it :)

<comment><USER38> Ahh, but if you could better remember everything you have understood, surely that would help with understanding the next paper!

</comment></comment></comment><comment><USER39> Is there a version of this or some alternative for learning+remembering music?

I learn a piano piece, memorize and internalize it, but then if I stop playing it for a few weeks, it completely vanishes from my head. Super frustrating, it's like starting over every time.

On the other hand, I have very good auditory memory -- I can "hear" and recall music in my head without problems. But that's unhelpful to me when I try to _play_ music.

<comment><USER40> Play very slowly through the piece at least a couple times every time you practice. So slowly that it's hard to discern what piece is being played.

Repeat the same process when you have your piece memorized. Then play your piece from memory but hands alone. You will build up the muscle memory required to keep the piece in your fingers, rather than your head.

<comment><USER39> No, I did that already. I practice slow and methodically. I keep a measure-by- measure logbook of progress. Muscle memory established. I can then play expressively from memory.

But eventually I'll move on to the next piece, and wind up forgetting to re- visit the other one for a few weeks or months. Gone!!

<comment><USER41> > and wind up forgetting to re-visit the other one for a few weeks or months

Maybe adapt your practice to the forgetting curve, revisit a piece the next day, then in 3 days, then in 7 days, then in 2 weeks, then 1 month, then 3 months, etc. If ever you mess up the piece, then it's back to 1 day.

<comment><USER39> This makes sense. Thanks!

</comment></comment></comment></comment></comment><comment><USER42> been researching spaced repetition and anki/sm and other stuff... I decided to try it because my memmory is quite bad when learning new stuff. I am also a daily weed smoker so my short-term memmory is quite bad but since positives outweigh the negative sides I don't plan on stopping. What do you guys think am I in big disadvantage?

</comment><comment><USER43> I used to use supermemo a lot but when I found out I couldn't export my cards (at the time anyway) I got nervous and switched to Anki

Does anybody know if the original creator is still in charge? Supermemo.com got reworked really heavily a couple of years ago and I wondered if he may have sold his company, or maybe just hired a team to make some apps

</comment><comment><USER44> Piotr Wozniak just posted a new article about SRS [https://www.supermemo.com/pl/articles/history](https://www.supermemo.com/pl/articles/history) (June, 2018)

</comment><comment><USER45> I heard a doctor online once say, "watch one, do one, teach one." He was talking about medical procedures but I've focused on the "teach one" part to much success in software.

</comment><comment><USER46> Feynman technique [https://qz.com/849256/how-to-master-a-new- subject/](https://qz.com/849256/how-to-master-a-new-subject/)

</comment><comment><USER47> So can anyone summarize the memorization algorithm for me? I don't have time right now to read through the full article.

How often should repetitions be spaced, roughly speaking? Is there a rule of thumb?

<comment><USER48> This is one flavor of these types of algorithms that you can viably implement yourself: [https://en.wikipedia.org/wiki/Leitner_system](https://en.wikipedia.org/wiki/Leitner_system)

<comment><USER47> That's funny. I developed a crude version of that algorithm by separating my flash cards into easy, medium, and hard stacks. I spent most time on the hard stack and would downgrade the difficulty of cards after I increasingly mastered them.

Nothing groundbreaking. I'm sure other people have done the same things out of sheer efficiency - no need to keep going over the easy cards.

</comment></comment></comment><comment><USER49> Obligatory shout out to Learn with Texts - [http://lwt.sourceforge.net/](http://lwt.sourceforge.net/) \- which interfaces with Anki.

</comment><comment><USER50> The TLDR: the algorithm is spaced repetition. Notable apps that implement it are SuperMemo (this one got it popular) and Anki (the free one).

<comment><USER51> Next article will be 6000 words about how moderate exercise improves general well being.

</comment></comment><comment><USER52> > "To Remember Everything You Learn, Surrender to This Algorithm"

> "The winter sun sets in mid-afternoon in Kolobrzeg, Poland, but the early > twilight does not deter people from taking their regular outdoor promenade"

I find titles promising a specific piece of useful information combined with longform, author storyline based content to be one of the most fustrating things about reading traditional journalism on the internet. A form of bait and switch.

<comment><USER53> My writing teacher called it "verbal diarrhea".

Journalism students spend so much of their time doing pointless writing exercises, they feel like they should continue with it, as if their reputation depends on it.

In the modern world the opposite is true - attention spans are short, you must get to the point.

A similar thing is true in video - I can't stand vloggers who dance around or have to explain the whole history of the subject before getting to the point.

I blame Google - they incentivise long articles with lots of keywords and longer videos.

<comment><USER54> Long form content actually does very well online when it is an interesting subject and well-written (and, increasingly, creatively designed).

For example: this article has been voted to the front page of HN, which is more than thousands of terse, AP-style inverted-pyramid news articles can say.</comment></comment></comment>


Tech workers at UCSF face layoff, training their low paid replacements - <AUTHOR> http://www.mercurynews.com/2016/11/03/after-pink-slips-ucsf-tech-workers-train-their-foreign-replacements/ <comment><USER2> "We're under a great deal of pressure," Joe Bengfort, chief information officer at UCSF, told employees at a staff meeting"

Should probably start with this guy's pay first. He was paid over $460,000 in 2013.

[http://compensation.universityofcalifornia.edu/reports/2013-...](http://compensation.universityofcalifornia.edu/reports/2013-annual- report-executive-compensation.pdf)

The number of administrator jobs and their outrageous pay in public school system is shameful. Just look through the above PDF.

<comment><USER3> I'm sorry, but why is that salary out of the question? Assuming he's actually worth his salt, he could make that at Google, probably easily, as an executive. Do you propose replacing him with someone who's worse, simply because they're paid less?

<comment><USER4> According to PayScale [1] he's making 3x the median for a CIO. In understand that $150k seems crazy low but even at the high end of those numbers he seems overcompensated.

[1] [http://www.payscale.com/research/US/Job=Chief_Information_Of...](http://www.payscale.com/research/US/Job=Chief_Information_Officer_\(CIO\)/Salary)

<comment><USER5> 1\. He's not just CIO, he's also a Vice Chancellor.

2\. Median pay for the US is a poor proxy. Using your same site, the median pay for Software Engineers is $80K/year. It doesn't reflect market (CA) or expertise (healthcare).

</comment></comment></comment></comment><comment><USER6> I rarely see in coverage of these mass replacements what kind of positions are being affected, just "IT workers". The article says they're mostly back-office staff but I don't know what that means at UCSF. Are these software engineers, systems admins, middle-managers, etc? Are they low level help-desk positions? Or is it a broad mix across the entire IT hierarchy? How does the answer in this case compare to other recent incidents (e.g., Disney)?

I don't ask to insinuate "lower-skilled" workers deserve to be replaced or should have prepared for it; I'm merely trying to get a better grasp on the scope of these recent outsourcings.

<comment><USER7> It probably means some of their developers, the ones who glue systems together --also maybe the sysadmins for mundane things but will probably keep key people like LDAP/Auth dev, InfoSec, network architect and then move all the other routine operational stuff off-shore.

</comment></comment><comment><USER8> > In a move that could spread to other universities, about 80 information tech > workers at UC San Francisco are facing layoffs and have begun training their > replacements — lower-paid tech workers from an Indian outsourcing firm.

This should work out well...

<comment><USER9> Is there any evidence that paying more for IT workers results in better quality of service?

It probably won't succeed to complain about this on the basis that the result might be poor, if the real problem is that outsourcing firms are abusing the H1B visa to reduce wages.

<comment><USER10> What happens is that those with the skills, even in India, demand enough that they aren't working for these outsourcing companies.

This is why there are different connotations between "outsourcing" and "remote workers". One implies the lowest bid.

</comment></comment></comment><comment><USER11> I have worked in many companies with H1B workers, and the old adage always applies: "You get what you pay for." I don't mean this as a dig against H1B employees, but UCSF will definitely notice a substantial drop in quality of H1B work quality compared to their American counterparts.

<comment><USER12> That's my experience as well. Considered opening an office in Bangalore as I was thoroughly impressed by some candidates I had interviewed. Performance on the job was significantly different... like by several orders of magnitude.

Taught me a lesson. Interviews can be gamed!

</comment></comment><comment><USER13> Simply... public funded jobs should remain public not offshored.

The UC system should face significant fines for this move.

</comment><comment><USER14> When US companies (and universities) can operate and profit by operating in other countries, like India. Why should we not allow employees from other countries (like India an China) to seek employment in US, if they can do the job better and at a competitive price point. Why should it be just a one sided gain model that always benefits US employees.

<comment><USER12> Because those companies are US companies, their very existence is a byproduct of this political community's stability, of the infrastructure that was paid with taxpayer money, sometimes of the direct subsidies of the US states and their federal government and generally speaking by the direct or indirect economic activity that US citizens generates.

Simple as that. There is a social contract that exists and that holds together the civic tissue of this country. Of course, some companies are going to try cheating it. That's fine, it is the reason we also have law enforcement agencies.

In the case of UCSF, a public University that almost exclusively relies on public funding to survive. I think people have an overwhelming amount of reasons to be pissed off.

<comment><USER14> > Because those companies are US companies, their very existence is a > byproduct of this political community's stability.

Engineering talent (in US and other countries) is also a byproduct of that society's investment into educating its children. Arguably graduate education is also funded by tax payer money in these countries. So when US firms hire an engineer from India or China, they essentially are getting an engineer without funding his education.

> There is a social contract that exists and that holds together the civic > tissue of this country. Of course, some companies are going to try cheating > it.

Seeking merit (searching for a person who can do the job at a competitive price point) is not cheating. If UCSF does not offer good courses at great prices someone else will. That someone else could very well be a Indian or a Chinese university.

</comment></comment></comment><comment><USER15> Well, if we want to see the long declines in healthcare costs we all love so much continue, we are going to have to accept these sorts of capital efficiencies, now aren't we?

</comment><comment><USER16> As someone on an H1 visa (although not working for an outsourcing company, but rather in a product org), this makes me mad. Not only due to the job losses, decline in quality, but also it will ruin a lot of people's lives who now have to attend daily status meetings at 9pm PT. In addition, these outsourcing companies don't even bother to hire local folks for the "onsite" positions. Even those positions need to be imported from India. That doesn't make any sense and I am not sure why the USCIS do not reject those applications.

</comment><comment><USER17> surprised to learn courts threw out the lawsuit against Disney for their similar H1B scandal

[http://www.orlandosentinel.com/business/brinkmann-on- busines...](http://www.orlandosentinel.com/business/brinkmann-on-business/os- disney-visa-lawsuit-dismissed-20161014-story.html)

H1B applier must certify no current American employees harmed by the visa. But that isn't transitive from the outsourcer to Disney. :(

<comment><USER18> It's a fairly nonsensical concept, though. There's always _someone_ who's harmed, and someone else who benefits. Why focus in particular on current employees? What makes them so much more deserving than future employees (who are hurt by the visa), or current customers (who benefit)?

<comment><USER17> the intent and spirit of law is that the companies are hiring H1Bs for positions that could otherwise not be filled in sufficient quantity by Americans. Companies and Customers win, and American labor should experience no measurable harm. Not that I would but one could even argue that the way that AppleGoogleFacebookMicrosoft use them to hire students is even okay since they probably cannot fill the volume they need with American STEM grads.

This end-run with the outsourcers is a particularly egregious perversion of the law because they're bringing H1Bs with the _intent_ to fire and replace Americans. There's also an angle of age-ism where they are firing older IT professionals in their 40s, 50s, 60s who have accrued higher pay-grades , and replacing them with less productive but much less costly young foreigners. Only losers are employees who need quality IT help, but who cares about that.

</comment></comment></comment><comment><USER19> Why does the company not consider offering pay cuts in the first instance before jumping straight to outsourcing?

<comment><USER20> my guess: unions

</comment></comment><comment><USER21> What I don't get from these articles is why are they only talk about outsourcing IT services and how it hurts US employees? Outsourcing manufacturing to countries like China also hurt employment - I don't see this to be much different than that. Comoanies should do whatever it takes to keep them profitable and in this era of globalization, the competition is not only from companies based in US but worldwide.

As an aside though, I do feel that H1-B visa is being misused. I know a couple of friends who recently completed their MBA from reputed colleges in US but weren't able to get the visa due to the cap since Indian outsourcing companies nabbed majority of them. I feel there should be a separate category for visas for students who study in US so that they get a fair chance to participate in US workforce. Also, this will encourage talented folks worldwide to come to US since their future no longer will be decided based on a lottery

<comment><USER22> On your first question, I think the issue is that Americans have gotten accustomed to the loss of manufacturing sector jobs, but the argument had always been that we were slowly switching to a service and/or "knowledge" economy. The kind of outsourcing described in the article challenges the claim that Americans are competitive even in those employment sectors.

As an aside, UCSF is not a company, so your point about anything-goes competition is in this case a non-sequitur.

<comment><USER23> It isn't a skills problem, its a salary problem. Cost of living in the US is astronomical compared to any of the nations you import IT labor from. The only leg up the US (still) has is that there is no accountability in bad business done overseas - if you get crap products from your outsourced developer team, you don't have a floor to go down the elevator to where you can yell at a project lead in person.</comment></comment></comment>


Snorelax – A smarter way to wake up - <AUTHOR> http://www.snorelaxapp.com

<comment><AUTHOR> Hi Guys, Im the founder of Snorelax.

Snorelax is an IOS alarm app which analyzes traffic and wakes you up earlier in case of a delay.

If you guys have any questions, I'm happy to answer them!

<comment><USER2> Snorelax or Snorlax[1]? Just a heads-up to avoid any naming confusion/infringement stuff. (Though I should note, I have no idea if there's even a risk of that; I'm basically the opposite of a lawyer).

Seems like a cool idea! I'm on Android though, so I can't give it a test.

[1]bulbapedia.bulbagarden.net/wiki/Snorlax_(Pokémon)

<comment><AUTHOR> Hi, It's SnoRELAX, its partly based on the pokemon snorlax but also on RELAX.. and on SNOR.. Because its an alarm app.. And it has to do with Traffic and you know how Snorlax always would block the way like modern day traffic.. Im planning to build also the Android version, but first Ill have to see how it will be received by IOS users. You could already signup if you want. And I would appreciate it if you could share it with your friends who have iPhone's. Thanks!</comment></comment></comment>


Can North Korea Nuke Me? - <AUTHOR> http://CanNorthKoreaNukeMe.com <comment><USER2> The thing I really enjoy about these types of things is the alternate map projections / focus points from other countries (if you click through to the NYT page). For instance, Seattle / SF are both "just up the coast" from NK.

<comment><USER3> The 2 times I flew between Seattle and Japan, I'm pretty sure the plane more or less took the route along that coast.

</comment></comment><comment><USER4> NO! NORTH KOREA CANNOT NUKE YOU* You are 6509 miles from North Korea, 9 miles out of range

<comment><USER5> I am 92 miles out of range, according to the site.

I kinda feel left out.

</comment></comment><comment><USER6> The IP geolocation database they are querying is returning a 429 too many requests error.

</comment><comment><USER7> What a cruel twist of irony that here in Seattle, probably the major city that hates Trump more than any other, is closest to NK's nukes and will probably be first to go when his deranged ranting sets off Kim once and for all. Don't let him use us martyrs for his cause.

</comment><comment><USER8> Greetings from America's capitol city, Washington, DC! Worry not! North Korea is 5,862 miles away, and we are safely 362 miles out of range! Your government isn't going anywhere!

<comment><USER9> ... unless somebody figures out how to launch a weapon from a ship.

<comment><USER10> ...and every one of the dozen missile defense systems between NK and here fails.

<comment><USER9> They have ships in the Atlantic, Pacific, and the Gulf (maybe the arctic as well).

</comment></comment></comment></comment><comment><USER11> It just says "calculating..." for me.

iOS safari.

<comment><USER12> Working for me on iOS Chrome....unfortunately I'm just within range in Chicago so I started flat out sprinting westward

<comment><USER13> Don't you need to run East from Chicago to get out of range...?</comment></comment></comment>


I wrote a beginners guide to Kubernetes - <AUTHOR> https://www.learncloudnative.com/blog/2020-05-26-getting-started-with-kubernetes-part-1/ <comment><USER2> IMO uni-tools like kubectx are pointless

Huge fan of fzf + bash history

Trigger fzf, search "context cluster-name" done

Learning a bespoke tool to run one bash command is poor craftsmanship, won't be worth knowing about once Kube is entirely wrapped by some providers HTTP API, and this is all managed by GUI based services, obsoleting "SRE" teams at places that aren't Google scale

The excitement of Kube will go the way Puppet and Chef have. It's just HTTP commands putting some YAML or JSON which is metadata for how to manage cgroups

Oh right but Google and Oreilly flooded the world with books, so kiss the rings.

<comment><AUTHOR> I try to be focused on the results, not the tools per se.

For me, kubectx makes it easier to view and switch between context. I run "kubectx 'context-name'" and it's done :)

<comment><USER2> The results I'm chasing are fewer uni-task tools

Ctrl+r to get fzf, searching "context prod" pulls up my last use of the kubectl form of the change context command

Desired results are subjective

</comment></comment></comment><comment><USER3> Nice article, I liked the heavy focus on networking. Looking forward to the next part. If I could make a request it would be for the inclusion of Istio somewhere in in the following parts

<comment><USER4> IMHO, its a beginners guide and the focus on networking was to make beginners understand basic concepts - heavy focus would mean the guide would explain how k8s networking works under the hoods which is often important in real production environment, where you will face various issues.

When it comes to Istio - its a complex solution by itself and deserves a separate guide. I would say beginners should try to learn Istio when they are no longer beginners but rather advanced k8s users or even maintainers.

<comment><USER5> We are looking at deploying Kubernetes on GCP as a managed service (GKS) - will we have to deal with networking issues in production, or is this something abstracted away by the managed service?

I know that ELBs for external traffic ingress are automatically managed by GCP, but not sure what other things we need to consider.

</comment></comment></comment><comment><USER6> You mention talking about stateful workloads later on, but I don't see it. Is that in a separate article?

<comment><AUTHOR> I am still working on it; it will be a separate article.

</comment></comment><comment><USER7> Thanks! You can add that if the docker-desktop context is not available, you can enable it in Docker preferences -> Kubernetes -> Enable Kubernetes (I had to do this)

<comment><AUTHOR> Thanks for letting me know!

</comment></comment><comment><USER8> very nice, can you share the tool that you used to create the images?

<comment><AUTHOR> Thanks! I used [https://www.figma.com/](https://www.figma.com/) to create all diagrams.

<comment><USER8> Thanks for the response. You seem to write well.

I am about to release a new Kubernetes based machine learning product (using new CRDS, operators, etc).

I wonder if you can you do a paid review? we can talk offline if needed.

<comment><AUTHOR> Feel free to contact me - my info is on the website.</comment></comment></comment></comment>


Chrome gmail delays on Linux fixed by running Firefox? - <AUTHOR> My gmail was freezing up with dismissed ads and other functionality in Chrome running on Ubuntu Linux.  I switched to Firefox, and now it works great.  Clearly there is support in Chrome for gmail that is affecting performance on Linux.  As anyone had a similar experience? <comment><AUTHOR> Mine is paid as well, but still hangs in Chrome on Ubuntu. i should look into bug reporting with Google ...

</comment><comment><USER2> Nope, but then I use paid Gmail. Just got the scheduled send today!</comment>


Starting today, anyone can turn their Xbox One into a dev kit for free - <AUTHOR> http://www.polygon.com/2016/3/30/11318568/xbox-one-dev-kit <comment><USER2> Why did it take so long, finally! ID@XBox was pretty much supposed to be this soon after launch.

 _Developers will still rely on the existing ID@Xbox system in place for getting their games onto the console._

 _That means that a developer will have to go through Microsoft 's concept approval, which usually takes about two weeks, before the game is eligible to be published on Xbox One._

Looks like ID@XBox approval is still needed to launch on the device but at least you can build. Maybe that will go away for an approval process more like mobile markets or Steam in the future, more open. 2 weeks and a devkit from any device is a nice step.

Sony you are next, open dev markets sell more hardware and create better games at the top being open.

Open markets like Apple Appstore and Google Play have so many games that the best 1-2% are better than any walled garden/curated list could be. Though, like the internet, there are lots of bad quality/garbage games/apps as well but the quality and innovation at the top is better.</comment>


Nobel Prize in Economics awarded to Roth and Shapley for matching algorithm work - <AUTHOR> http://www.nobelprize.org/nobel_prizes/economics/laureates/2012/popular-economicsciences2012.pdf

<comment><USER2> Their work has nothing to do with central planning in the sense of committee deciding on production quotas---it's actually about how to allocate indivisible resources (school positions, kidneys, residency slots etc.) in situations where using money isn't feasible and/or the market has broken down due to problems like congestion or lack of market thickness.

<comment><USER3> I think results like Shapley's and Roth's present an interesting commentary on the usual "philosophy of economics".

1\. These results are all about finding good solutions to allocation problems without using money or in cases where money isn't or can't be applied.

2\. These results do tend to show that a "centrally planned" solution can be better for everyone than a "free market" solution. But as you say, by central planning I really just mean that some entity takes in everyone's preferences and outputs a global allocation, with the nice property that everyone is happy with what they got and nobody can find a way to deviate from their assignment and be better off. One might argue that a "free market" would find the same solution eventually, but it might be slower.

These two results are sort of the opposite of the Invisible Hand stuff as well as the common assumption that, for instance, everyone values money the same (which underlies the justification for much of economics). So I think they're really cool in that respect.

<comment><USER2> well put.

FYI - Here's a link to an interactive illustration of the deferred acceptance algorithm that I posted a few years back on HN: <http://mathsite.math.berkeley.edu/smp/smp.html>

</comment></comment></comment><comment><USER4> There is no Nobel Prize in Economics

[http://www.alternet.org/economy/there-no-nobel-prize- economi...](http://www.alternet.org/economy/there-no-nobel-prize-economics)

<comment><USER5> Wasn't that due to Nobel girlfriend leaving him for an economist? Read something about that but can't find the surce at the moment (bad connectivity)

And I don't see that in the first 2 pages. Or was that about the math nobel prize ?

<comment><USER6> I believe that was the reason for the math nobel

<comment><USER7> <http://www.snopes.com/science/nobel.asp>

</comment></comment></comment></comment><comment><USER8> I find it interesting that they awarded the prize to Shapley after Gale's death[0]. I always thought that there was some rule that prevented them from awarding the Physics Nobel Prize to Aharonov after Bohm died[1].

[0] <https://en.wikipedia.org/wiki/Stable_marriage_problem>

[1] <http://en.wikipedia.org/wiki/Aharonov–Bohm_effect>

<comment><USER9> You are correct, Nobel prizes are not awarded to the dead, unless they die just after announcing the prize and before the ceremony ([http://www.guardian.co.uk/science/2011/oct/03/nobel-prize- aw...](http://www.guardian.co.uk/science/2011/oct/03/nobel-prize-awarded-dead- scientist?intcmp=239)). Gandhi was denied the prize for this reason.

In this case, it is simply the post title being misleading, as it's not the Nobel prize these folks won, but the Nobel memorial prize in economics, which is a relatively new prize which is not part of the 5 Nobel prizes which had been given by Nobel's estate for more than a century.

Edit: oh I see they didn't reward a dead in this case, misread your comment..

</comment></comment><comment><USER10> I recognized their names from the first example in the classic book "Algorithm Design" by Kleinberg and Tardos. But there it was called Gale-Shapley Algorithm.

<comment><USER11> The algorithm is due originally to Gale and Shapley, in a 1962 publication in the _American Mathematical Monthly_ , but Gale isn't eligible due to the rule against posthumous awards. Roth is included because he was the first to apply it practically in economic-matching settings, quite a few years later.

</comment></comment><comment><USER12> Josh Gans provides a good summary on Roth and Shapley over at Core Economics <http://economics.com.au/?p=9423>

</comment><comment><USER13> The so-called "nobel" prize in economics does not come from the estate of Alfred Nobel. It comes from the central bank. Thus it is not surprising to see it given to people that central banks would favor (often people whose theories try to justify inflationary policies.)

This one seems to be given for work that could be said to claim that central planning really can work after all.

<comment><USER14> Both Shapley and Roth are microeconomists, so the fact that you are bringing up banking and inflation is a clear signal that you don't understand anything about economics. If you want to harp on Nobel Prize winning economists regarding their work on macroeconomic theory, you're 1 year too late.

Shapley and Roth deal with topics that fall, at the highest level, under the field of game theory, but more specifically market design. Market design relies heavily on mechanism design, which can also be called reverse game theory. In game theory, you're interested in determining the best strategy given a set of conditions (the players, their strategies, the timing of their moves, their payoffs and the information they have [or don't have]). On the other hand, mechanism design is concerned with what the conditions have to be in order to achieve a specific objective or outcome. Hence, mechanism design problems map well onto many problems of regulation.

For example, let's say there is a Senator, call her Mrs. X, who is interested in passing a bill that achieves outcome Y. The mechanism design solution is then the set of rules that lead to outcome Y occurring. Setting the right rules refers to Mrs. X drafting a bill that includes mechanisms such that every actor in the game HAS AN INCENTIVE to play the strategy/strategies that lead to outcome Y.

How does this fit in with market design? Well, besides generally using mechanism design to align the incentives of both the buyers and sellers, there is a deeper point to be made about market design here. All so often in economic theory it is assumed that a market already exists, and the analysis is then on understanding the inner functions of that market. However, what that type of analysis leaves out is how that market even came into existence in the first place. In what ways were incentives aligned such that both buyers and sellers found a match for one and other to begin with? In other words, why is this market here and how did it get here? These are the nature of the questions that market design tries to answer. By understanding how and why the actors in the market are doing what they are doing, you begin to understand how markets form. And if you understand how markets form, you can apply that know-how to the formation of new markets, or to improve old ones.

<comment><AUTHOR> Great explanation, thanks for posting that!</comment></comment></comment>


The Toughest Companies for Job Interviews  - <AUTHOR> http://www.fins.com/Finance/Articles/SB130826852769924133/The-Toughest-Companies-for-Job-Interviews?Type=0

<comment><USER2> No-Hiring somebody after they get one wrong answer is ridiculous. A wrong answer during an interview and how it's followed up on is one of the most telling parts of the whole process.

If you want to hire people that are good at taking tests then by all means require every answer to be correct. If you want people that understand themselves enough to recover and think critically about their own mistakes then treat a wrong answer as what it is: being a human being in a stressful situation.

edit: also any company that still asks "brain teaser" questions in 2011 should not be one that you consider working for.

<comment><USER3> I did a study of Wikipedia, and 75% of contributions from registered users came from someone who got a contribution reverted at least once.

[http://slightlynew.blogspot.com/2011/05/vandalism-second- cha...](http://slightlynew.blogspot.com/2011/05/vandalism-second-chances-and- bots.html)

<comment><USER4> A reversion on Wikipedia doesn't necessarily indicate that the reverted edit was wrong.

Wikipedia is a highly politicized and point-of-view pushing place, so it's quite common for edits get reverted by others who simply have a different opinion or a contrary agenda.

</comment></comment></comment><comment><USER5> Some of this list is pretty much expected (McKinsey, Bain, Jane St, Palantir, Bridgewater), but some of it seems pretty ridiculous. BP? Really? My sister interned there (and was in charge of on-campus recruiting for oil companies at her grad school), and thought they were a bunch of idiots, something apparently confirmed by Deepwater Horizon. Amazon and eBay ahead of Google and Apple? I think anyone in Silicon Valley would think that a bit off. And the only hedge funds were Jane St and Bridgewater - where's D.E. Shaw, or Renntech, or Citadel?

<comment><USER6> For what it's worth, my Amazon interview was probably the hardest one I ever did -- much harder than my Google interview. At Amazon, every single one of the 7 or 8 people I interviewed with asked me a coding question, including the guy that had done the phone screen, and he'd asked me a coding question in the phone screen too. I also got a coding "homework" problem in the phone screen. I don't have any problem with coding questions in interviews, but it seemed like total overkill.

<comment><USER7> Were the coding questions themselves extraordinarily difficult? It just doesn't seem unusual to be consistently asked coding questions during interviews for a software engineering position (assuming that's what you were going for).

<comment><USER6> No, it's not unusual for all your interviewers to ask you coding questions. It is, however, a lot more work to have 7 or 8 people do that rather than 4 or 5, which is the point I was getting at. That was just a really long and tiring interview.

</comment></comment></comment></comment><comment><USER8> I don't see how you can compare interviews at companies that do different things against each other in terms of difficulty. In fact I don't even know what this means for interviews _within_ the company. For instance, Amazon: is it a difficult interview for a programmer, any technical person, or even for a marketer? If it's for a software engineer, is it as difficult for new grad entry positions as it is for higher positions? Which is being evaluated?

This comes to mind mostly because I'm surprised at who ended up on the list in terms of tech. Palantir, for instance, is indeed one of the hardest interviews out there, if only because the variety of programming questions they ask you is so large.

Salesforce on the other hand, while not a cakewalk, is definitely not on the level of difficulty of a Google or Facebook technical interview, at least for their university hires.

Lastly, I'm glad to see Teach for America up there. It's become a pretty competitive program to get into and this just further proves it, and I hope it gets expanded so more people can be out there doing good.

</comment><comment><USER9> Coding questions are often not the toughest interview questions.

Tough questions could be some "soft" non-coding questions like: design questions (how to design a class hierarchy for a blog), behaviour questions (if your boss is wrong, what to do?), experience questions (why did you use tool X in your previous project Y? I think tool Z is better.)

Those questions are tough because your performance on those questions are very subjective. It depends highly on if the interviewer likes you or not.

Coding questions can be tough, but they are much more objective. Google's engineering interview mostly asks coding questions. Questions can be difficult, but at least, if you write great code, you will pass the interview.

In this sense, Google is not a very tough company for job interview.

In general, the degree of interview toughness depends on the job supply/demand ratio. When many people apply for positions at company X, the company has to apply a high-rejection rate, based on whatever (sometime very random) criteria.

</comment><comment><USER10> I'm surprised not to see D. E. Shaw here. Unlike, say, Bain, which actually needs to hire people, D. E. Shaw seems to hire about three people a year but actively collect thousands of CVs just so they can brag about how many people they reject.

</comment><comment><USER11> Having interviewed with Amazon before, they ask lots of optimization type questions and if you can make it faster. For instance, take some algorithm and can you make it linear? They do the same thing with their coding questions, they'll have you solve a problem and then ask you to make it faster. I wouldn't say it was the hardest interview I've seen, but it was definitely more challenging than most others I've done.

<comment><USER12> My favorite interview questions are the ones where they have you figure out an algorithm for something and then say, "Ok, but you can do it better (faster or with less memory)."

It's slipped into how I do my work. I write something that I want to be optimized and say "Ok, but I can do it better", regardless of whether I know it can actually be done better or not. Very fun.

</comment></comment><comment><USER13> As someone who has recently interviewed at most of those software related companies and got offers from most of them, I can confirm that some of these ranking seem really arbitrary.

Amazon was definitely one of the easier interviews I had. Microsoft had one of the harder ones (but this might have just been a case by case thing).

Jane Street and Palantir, however, I can say for sure had two of hardest and most intense interviews I've ever done.

I've now learned, after being trained to conduct interviews, that these interviews aren't meant to be fair. Companies are afraid of getting any single person that isn't competent because people tend to hire others like them. A single bad person is a cancer. Thus, the hardest interviews are meant to reject 99% (or even more) of the qualified candidates so their false positive rates are ridiculously low. This is how scared these companies are of false positives.

</comment><comment><USER14> I did 2 rounds of interviews, 9 interviews in total if I recall before starting at McKinsey. Though I have to say, the whole case interview thing is a hack that can be taught to anyone. I knew a lot of not very brilliant people who got in because an alumni pretty much showed them exactly how to do it. Like any test, you can study the fuck out of it and get in.

</comment><comment><USER15> I interviewed at Amazon once. They gave me three phone screens but I didn't get invited for the 1:1. I greatly respected the interview and it was my first wake up call that I had lost some of my technical edge as a manager.

At Google, at the same time, I almost got a job offer. I got all the way through the interview (phone-screen, all-day in-person interview) and then they said they didn't know what to do with me could I come back in 6 month and make a presentation on a business idea. I took them up on it 6 months later but they passed on my business idea (2004, a rich text wiki).

</comment><comment><USER16> What I don't understand is that this report seems to contradict other reports of software engineers being in high-demand, which presumably should lead to _lower_ interview barriers?!

<comment><USER17> It's a market for lemons. When devs are in demand, that brings in applicants who are only in the industry because the wages went up, while the good ones are more likely to be taken. These each lead to a greater proportion of incompetents to filter out, and the cost and risk of accidentally hiring one have not improved.

<comment><USER18> What risk though? Instead of these days-long interviews with multiple people (and the inevitable meetings after to discuss their results) why not just hire people provisionally and see how they do? Especially if you pair-program, code review, or anything that will expose them to evaluation.

I'd advertise with something like "Job coding X at a company that does Y. Finish the Ruby Metakoans and be prepared to discuss your answer. You will be asked to code in Ruby and C during the interview."

These huge processes just seem like HR trying to justify their existence.

<comment><USER17> After a phone screen, we spend maybe five hours of developers' time on each candidate and reject 80–90% of them. If we provisionally hired all of them expecting useful work, we would invest far more time ramping them up on our system, risk outages from their early mistakes, disclose trade secrets to a bunch of devs who end up unhappy with us, and lose out on all the employed candidates up front (who aren't going to quit just for an extended interview).

Then there's the moral hazard of paying anyone who can get into our filter, rather than just the ones who can get all the way through it. It's the same reason we can't expect candidates to compensate us for the cost of interviewing them—it's better if a bad match is lose/lose than if anyone could reliably expect to be rewarded for wasting everyone else's time.

<comment><USER18> I wasn't suggesting you hire everyone who walks through the door, but I bet my hour-long test is within 80% as accurate as those crazy all-day stress tests, and much cheaper. Asking for code (even a few lines) in the application itself would further reduce the number of applicants.

Your system/circumstances seem like they'd make trusting and benefiting from new hires quite a hassle. Ideally you'd have some public and open-source app to have new hires hack at, but there should be tests to be written, etc, which shouldn't leak trade secrets. (If you don't have docs for which files/components implement your trade secrets, and therefore which ones do not, I don't see how you have any hope of keeping anything straight.) If their early mistakes cause an outage I think it says more about your build processes than them.

As for people not quitting for what's in essence an extended interview, that's what they are doing for you and others already. You fully intend to dump them if they don't prove to be good, but you don't have a quick process to determine this so you aren't as up-front about it.

But yes, not every place has it as easy as Github, for instance. They have a ton of publicly available code an applicant could work on, and in doing so they'd show they could use the company's products and tools. I don't know, but would bet they have a comparatively easy interview process.

</comment></comment></comment></comment></comment><comment><USER19> I think the toughest companies are where _interview == quiz_

</comment><comment><USER20> The interesting thing is that almost all of those companies are companies where I wouldn't want to work - with the possible exception of Jane Street (though that depends on whether or not that company pays is programmers 10% of what it pays its traders) and Amazon (assuming that the stories Steve Yeggy tells are exaggerated and they have brought actual desks for their programmers, instead of the doors they used to have).

</comment><comment><USER21> I interviewed at Salesforce.com in mid January 2011. It was at 300 Market Street San Francisco, bright and early 8:30AM. Linda Ellis, Raja Bhamidapati, Allison Nishioka, Kei Tang, Ranjani Salur, Rhoita Misra, Sagar Wanaselja, Sam shoute. I didn't get the job. I was put through the ringer. I completely agree with this (difficult interviewers) report. Here are some notes:

I met with 10 people during the interview. A complete battery of psychological analysis. In that group was every kind of co worker I've ever dealt with. I could tell some were acting the part to try to throw me off base and test me for any sociopathic or personality problems. Some were insulting me in my face, some were super charismatic, some people with impossibly hard questions. Some questions were strangely easy. The whole gambit, it gave my personality side a workout (Programmers usually don't work hard on dealing with the 50 personality types).

They planned the sit down coding test an hour after lunch, to test if you know how to deal with the 'afternoon dip' in energy. The coding test I did well on, it was to create functionality to replace the "String" class with char * in java. And more importantly is to unit test. Do this in one hour.

On the plus side, they paid my entire way. and got me into the "Omni hotel" one of the best hotels I've ever stayed in. Paid for the Car. It was by far one of the most exciting experiences of my life.

They analyzed my personal life as well and ask questions about what I'm doing when I'm here, what I like to do, what kind of person I am.

The majority of the questions they asked me were a subset of the questions in the book "Cracking the coding interview" by Gaylle Laakman. They were tough questions, but if you are prepared you can give them canned responses.

After that Interview, I felt like I've never been so probed in my life. From stem to stern. Every part of me, everything that makes me a human was analyzed. I left feeling like I've just been scanned by the Borg. I didn't get the job. but I made it to the 'short list' of 13 candidates up for consideration. I left my heart in San Fran California.</comment>


Why Question Answering Is Hard - <AUTHOR> http://nicklothian.com/blog/2014/09/25/why-question-answering-is-hard/ <comment><USER2> Also there's the problem of context. When someone asks you a question you generally know what kind of answer they expect.

In this classic example from Richard Feynman [1] the answer to "Why is Aunt Minnie in the hospital" can be "She broke her hip", "She slipped on the ice and broke her hip", "An ambulance took her there", "Because her husband is concerned for her well being".

What's interesting about the Watson approach is that, by taking intelligently composed web pages as input (as opposed to randomly generated texts, or an arbitrary list of atomic facts) the context is already "baked in". Put another way, people make web pages with sentences like "Aunt Minnie is in the hospital because she fell and broke her hip", and don't make web pages with sentences like "Aunt Minnie is in the hospital because the ambulance was headed there and the friction between Aunt Minnie and the bed in the ambulance was high enough to keep them from becoming relatively displaced".

[1] [https://www.youtube.com/watch?v=MO0r930Sn_8](https://www.youtube.com/watch?v=MO0r930Sn_8)

(Edit: I notice, of course, that I have now created a web page with that exact sentence.)

<comment><AUTHOR> I'm not convinced that context is as hard as people think. I think the answer "she broke her hip" is the appropriate answer there.

The other answers are appropriate for other questions ("How did Aunt Minnie get to hospital" etc).

OTOH.. I'm disagreeing with Feynman here. That gives me some pause.

<comment><USER3> The answer from an intelligent alien who was context-naive might be something like so:

Aunt Minnie is a human being living in the 21st century Western culture of Earth. When humans in this region and time period become sick or injured, relatives and friends, or medical workers working on their behalf, will transport them to a hospital for treatment. Aunt Minnie was injured on a sidewalk made slippery by seasonal ice. Such injuries aren't uncommon but are treatable with various surgical and therapeutic techniques. This demonstrates the humans vulnerability and heralds our inevitable victory to harvest them as a source of food.

</comment></comment></comment><comment><USER4> Question answering is actually much harder than was is presented in the article because most questions people ask are not factoid questions. Consider the following that Watson needed to address at a recent customer engagement (an insurance company):

* Can you help me with life insurance?

* What benefits will I lose when I leave?

* What last minute things should I take advantage of in the military before I go?

The answers to these questions are not entities (people, place, dates) as in factoid questions. They may be procedures, they may depend on the user, further clarification/dialog may be required before being answered, they may not have a single answer or require a combination of answers, etc.

Also Watson does not try to formalize all knowledge even when answering factoid questions. Many of the candidates generated do not actually come from a formal knowledge graph, instead they are generated on the fly from analyzing all the pieces of text that seem contextually relevant.

<comment><AUTHOR> There isn't that much publicly available information about how Watson works - I'm pretty sure I'd read everything there was in January this year. There might be some more now?

I can't even imagine where to start with "Can you help me with life insurance" \- beyond maybe triggering standard sales pitches.

How does Watson decide when to use the knowledge graph and when to look more broadly?

<comment><USER4> The actual Watson program that performed on Jeopardy is pretty well described in publications. You can start here: [http://www.aaai.org/ojs/index.php/aimagazine/article/view/23...](http://www.aaai.org/ojs/index.php/aimagazine/article/view/2303/2165) for example.

The right way to answer "Can you help me with life insurance" is to start asking the user questions to learn more about their needs - that's one of the way we are currently trying to go beyond the Jeopardy use case.

<comment><AUTHOR> Interesting.

I'm yet to see a good public implementation of conversational question answering. It's pretty clear Google is headed down that path with Google Now, and I guess Siri is going the same way.

Traditionally question refinement is done using filters and clustering, but of course this focuses the result sets, and is the opposite of what you need in conversational question answering.

</comment></comment></comment></comment><comment><USER5> Question answering is an amazing problem. I built factoid QA systems using a mix of NLP, open relation extraction (dependency relation classification via SVMs), semantic web stuff via triple stores (virtuoso) and answer candidate ranking algorithms (classify the questions in to different types via keywords) and using those to determine whether I should use a search engine, or lookup system.

I think for those who want to get started in the field understanding some of the more complex NLP techniques, information retrieval, and some idea of how triple stores work gets you to building a QA system.

This is actually why I jumped in to deep learning. The hardest part of NLP for open ended question answering is coming up with all of the different features.

Deep Learning with textual representations fixes this.

The tech stack was [http://uima.apache.org](http://uima.apache.org) for those who are curious. I'd be happy to answer anything specific. QA systems are a little esoteric due to the combination of techniques used to build them, but they aren't as mysterious if you put in the effort to understand what they actually are.

</comment><comment><USER6> Question answering is hard!

A couple of years ago I wrote a question answering system too; it's just sitting on my drive. I haven't decided what to do with it. Maybe open source it?

You can see a demo here [http://youtu.be/PIxAsVDBfzs](http://youtu.be/PIxAsVDBfzs)

Is there any interest in open sourcing it?

<comment><AUTHOR> Open sourcing would be great, but I'd be interested in hearing what techniques you used.

</comment></comment><comment><USER7> ... but also fascinating to work on.</comment>


Tesla Raises the Bar for Self-Driving Carmakers – Nvidia - <AUTHOR> https://blogs.nvidia.com/blog/2019/04/23/tesla-self-driving/ <comment><USER2> "And only one of these is an open platform that's available for the industry to build on."

Wow, this hits me. The need for Tesla to open its doors for its hardware platform is a constant reminder of who you let in can kill you.</comment>