Luje 0.1 – a pure Lua JVM - <AUTHOR> https://cowlark.com/luje/doc/stable/doc/index.wiki <comment><USER2> David Given is one of those prolific thinkers and doers who seem to be overrepresented in the Lua community.

He's probably best known to HNers for his critical article "On Go" and Objective Lua.

[http://cowlark.com/2009-11-15-go/](http://cowlark.com/2009-11-15-go/) [https://cowlark.com/objective-lua/index.html](https://cowlark.com/objective- lua/index.html)

<comment><USER3> Thanks for the link! That's a really interesting article.

Though out of date, I still think his point holds water. It's shocking how little difference there is between Algol-68 and modern Go or C. Go has changed quite a bit since his article, but not all of these changes have been well received (a great example being automatic semicolon insertion).

</comment></comment><comment><USER4> The article makes a rather hefty claim, that it is comparable or even in some ways faster performancewise than Sun's hotspot JVM.

From what I've heard the JVM is one of the most heavily researched and optimized bytecode VM out there. Could it really be that some guy can write a competitor in a scripting language in his spare time?

Not saying it's impossible, just curious if there's anything to back that claim up, besides the very specific microbenchmarks.

edit: I was a bit too quick to comment, reading again he doesn't really claim to be faster at anything other than small loops. Still interested if it's fast at any real world things though.

<comment><USER5> HotSpot is indeed one of the most optimized runtimes around. It is, however, optimized for large, complex applications. It's quite possible to beat in a carefully designed microbenchmark. Beating it in a complex application is a whole other matter (in fact, it's pretty hard to beat even in C++).

<comment><USER6> > in fact, it's pretty hard to beat even in C++

If there's one thing I've learned, it's that the wishful thinking will never stop.

<comment><USER5> Yes, but only if you call 15 years of experience in both C/C++ and Java systems wishful thinking. And I'm talking big-ass, several MLOC, some soft and some hard realtime systems.

But I guess some people must hold on to their beliefs, while some do extensive research (well, to be fair, it was government funded, so we had the means to write the same 2+ MLOC system twice, and we've had direct access to the Sun team working on their realtime Java project at the time). I guess you just don't have the data.

<comment><USER6> > I guess you just don't have the data.

Basically every piece of performance-critical open-source infrastructure (ie. what we can actually all see and evaluate) is written in C or C++. That is empirical data.

You say that long-running servers that don't have startup cost constraints will perform better in Java, particularly when there is concurrency. Why then, is the entire web served by web servers written in C/C++ (Apache, IIS, nginx, LiteSpeed), mail servers written in C/C++ (Exim, Postfix, qmail, Sendmail, Microsoft), DNS servers written in C/C++ (BIND, PowerDNS), and database servers written in C/C++ (MySQL, PostgreSQL, Microsoft SQL Server, MongoDB, Riak)?

You attempt to explain all this away, as if OS's, VMs, web browsers, etc (which you admit are better off written in C or C++) are somehow different from the applications you work on, and that somehow the Java you write in your applications would be hard to beat in C++.

The only "data" you have offered is your own personal experience of working in the defense industry. A couple problems though. First of all you never write the "same" 2+ MLOC system twice, you'll always use the lessons gained from the first go at the second design. Something tells me you wrote the C++ version first. Your experience is also something that we can't see for ourselves, and involves lots of variables that we can't evaluate. It seems likely that your company didn't have a strong culture of best practices in C++, or maybe just weren't that good at it. Your C++ design might have had lots of opportunities for improvement that good C++ people would immediately recognize.

It's not hard to write good and high-performance C++ when you have a good set of guidelines like the Google C++ style guide ([http://google- styleguide.googlecode.com/svn/trunk/cppguide.x...](http://google- styleguide.googlecode.com/svn/trunk/cppguide.xml)) and a solid set of core libraries, but without these it's easier to go astray. There's no doubt that Java's reasonable set of "batteries included" libraries makes this harder to screw up.

I can say from my brief stint in defense that I was not super impressed. Everything was based around extremely heavy infrastructure like CORBA. There are lots of decisions like this that can skew the results you were getting.

<comment><USER5> > Basically every piece of performance-critical open-source infrastructure > (ie. what we can actually all see and evaluate) is written in C or C++. That > is empirical data.

Beside the fact that this is an appeal to an irrelevant authority, it is also false.

MongoDB is about as far from being performant as you can be; Riak is not written in C++, but in Erlang; PostgreSQL and MySQL predate Java (and predate Java 1.4 – when it started getting performant – by many years), as do Exim, Postfix, qmail, Sendmail and BIND (don't know about PowerDNS); Microsoft has been an enemy of Java for political reasons.

On the other hand, recent Apache projects are all Java: Tomcat, ZooKeeper and the list is long. Cassandra and Voldemort are Java; Twitter Storm; most benchmarks show Java web-servers (of non-static content) to outperform or match C and C++ servers ([http://www.techempower.com/benchmarks/#section=data-r6&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r6&hw=i7&test=plaintext)).

> It's not hard to write good and high-performance C++ when you have a good > set of guidelines like the Google C++ style guide ([http://google- > styleguide.googlecode.com/svn/trunk/cppguide.x...](http://google- > styleguide.googlecode.com/svn/trunk/cppguide.x...)) and a solid set of core > libraries, but without these it's easier to go astray.

This may have been true 5 or 6 years ago. But modern multicore hardware requires a high level of concurrency, and a high level of concurrency requires low contention, and low contention requires non-blocking data structures, and nonblocking data structures require either a GC, or some other method of automatic memory reclamation that's _extremely_ hard to achieve in C++ (or any other language, for that matter). Even C++'s natural way of pointer sharing (reference counting) is a huge contention point and a performance killer. Not to mention the fact that until C++11, C++ didn't even have a memory model (which, BTW, was designed by Java people).

So while it's not too hard to make single-threaded C++ code run a little faster than Java, it's _extremely_ hard to make multithreaded C++ code outperform Java, even for experts (and it was almost impossible prior to C++11).

<comment><USER6> > most benchmarks show Java web-servers (of non-static content) to outperform > or match C and C++ servers > ([http://www.techempower.com/benchmarks/#section=data-r6&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r6&hw=i7...)).

Umm, there are no C or C++ servers in that benchmark, except for one extremely slow one I've never heard of.

> and a high level of concurrency requires low contention, and low contention > requires non-blocking data structures

That really does not follow.

Lock-free data structures are not about reducing contention, they are about eliminating blocking and improving performance when contention _does_ occur. Contention occurs whenever multiple threads attempt to access the same resource, regardless of whether access to that resource is blocking or non- blocking. Lock-free data structures can still experience contention, and while they do not block, their performance does degrade under contention. Indeed later in your message you say that reference counting is a "huge contention point" \-- refcounting a la C++'s shared_ptr is a lock-free algorithm.

So while lock-free algorithms don't affect the _amount_ of contention in the system, they do exhibit better performance under high contention, for the (small) subset of data structures that can be made lock-free.

But this is only an issue when contention exists to begin with. Many highly parallel systems exhibit very low contention; for example, MapReduce. Even in systems with inherent contention, there are many strategies for reducing it, like partitioning and affinity schemes.

> nonblocking data structures require either a GC, or some other method of > automatic memory reclamation that's extremely hard to achieve in C++ (or any > other language, for that matter)

I'm not sure what is so hard about Hazard Pointers ([http://www.research.ibm.com/people/m/michael/ieeetpds-2004.p...](http://www.research.ibm.com/people/m/michael/ieeetpds-2004.pdf)). Maged Michael is a smart guy who figured this stuff out 10 years ago.

But you don't even have to do it yourself. Non-blocking algorithms are extremely subtle by their nature (I hope you don't implement them yourself in Java; that would be a waste of effort). Just use an existing library like Thread Building Blocks, Grand Central Dispatch, etc. that are specifically designed to achieve maximum multi-threaded performance in C++.

<comment><USER5> Contention is not binary, or, rather, its effects aren't binary. A lock that is held for 10 microseconds potentially causes a lot more contention than a CAS that takes a few nanos. OTOH, thousands of short contentions are as bad as, or worse, than few long contentions (there's a whole great talk by Doug Lea about how Java's fork/join tricks modern CPU out of crazy power-saving optimizations to reduce thread wakeup time).

Intel's TBB were modeled after java.util.concurrent, BTW. Later, the Java memory model inspired the C++11 memory model (and the Java people were called to help with that, too). When it comes to concurrency, Java is usually at least 5 years ahead of C++ (perhaps with the exception of fork/join, which was inspired by Cilk, but has since come to surpass it).

> I'm not sure what is so hard about Hazard Pointers

Hazard pointers are expensive without RCU (the hazards need to be visible to other threads, which means store-load barriers), and RCU requires kernel cooperation, and besides this entire class of techniques is either very limited or _incredibly_ complicated (what do you do if you have thousands of threads?). A good GC vs hazard pointers with or without RCU is like driving down a highway vs stumbling in a swamp when it comes to lock-free data structures.

Here's Herb Sutter on the issue (source: [http://herbsutter.com/2011/10/25/garbage-collection- synopsis...](http://herbsutter.com/2011/10/25/garbage-collection-synopsis- and-c/)):

 _All existing C++ garbage collectors, notably the Boehm collector, aren't enough for many interesting uses for several reasons, the main one being that they're conservative, not accurate. See Hans Boehm's page Advantages and Disadvantages of Conservative Garbage Collection for more discussion of tradeoffs._

 _As usual, there's something to standardize iff we want to enable portability. So in this case the question is whether it's interesting to enable portable programs that rely on GC — and note that typically if they rely on GC, they rely specifically on accurate GC, not conservative "maybe- GC." For example, C++11 atomics enable writing many kinds of portable lock- free code, but not all; we need a standard for #2 if we want to enable people to write portable lock-free singly-linked lists, where the only known practical algorithms require automatic (non-refcount) GC and so they cannot be implemented in portable C++ today without some nonportable platform-specific code underneath._

 _Again, appeals to hazard pointers fail because the obscure convolutions and limitations of hazard pointers only expose the limits of even the most heroic efforts to work around the lack of true automatic GC. The main contribution of hazard pointers, unfortunately, is that they thoroughly motivate why direct support for automatic GC is needed._

<comment><USER6> > A lock that is held for 10 microseconds potentially causes a lot more > contention than a CAS that takes a few nanos.

See, if you're trying to compare those two things, you're comparing a lot more than locking vs lock-free; you're comparing two completely different designs. All you can do with a single CAS is a very primitive operation like queue enqueue/dequeue, stack push/pop, set add/remove, etc. If you're holding a lock for 10us you're doing a whole lot more than that; you're doing something that isn't possible to do lock-free, and your design is inherently less concurrency-friendly.

If your experience comes from taking a coarse-grained mutex-heavy design in C++ and reimplementing it in Java with a design that communicates between threads only with lock-free data structures, then your results aren't indicative of locking vs. lock-free _or_ C++ vs. Java.

If you really want to compare locks vs. lock-free apples-to-apples, then use data structures that have the same API as your lock-free data structures, but are implemented by taking a lock for a very short amount of time -- just enough time for the operation you would have performed lock-free.

And if you want to compare C++ vs. Java, use an equally concurrency-friendly design with both. If you're using high-performance lock-free data structures in Java, use the same in C++, since they are available in both. The memory management complications can be confined to the container library; at the application level exclusive ownership of objects can be passed from thread to thread.

<comment><USER5> Obviously. But lock-free data structures in C++ are not as good as their Java counterparts (because of GC), and besides, most lock-free DS are often implemented in C++ several years after they're implemented in Java, if at all.

For example, Java's nonblocking hash map (by Cliff Click) is still not available for C++, 6 years after its release. There are a few very limited/half baked implementations, but nothing "canonical" (and nothing so thoroughly researched, optimized and used).

BTW, part of the reason is that the world's leading low-level concurrency researchers and implementers (Doug Lea, Nir Shavit and others) do most of their work in Java (most of my own work is designing non-blocking data structures, BTW; not that I'm comparing myself to Lea or Shavit in any way).

To get a sense for the quality of Java data structure implementations, I strongly suggest you watch this talk by Doug Lea: [https://www.youtube.com/watch?v=sq0MX3fHkro](https://www.youtube.com/watch?v=sq0MX3fHkro)

His work is incredible. Even if you don't use Java, there is so much to learn in the talk. One of the coolest things there (which I already hinted at), is how to spin properly on modern Intel processors.

<comment><USER6> I appreciate the link to the talk, I learned a lot. It just reinforces my beliefs though; so much of the talk is about having to work around the idiosyncrasies of the JVM. Quotes from the talk:

4:15 "The main moral of this talk is that things are never what they seem, and they are increasingly not even _close_ to what they seem as we have more and more layers and systems. So there's you on top of core libraries who live on top of JVMs, and the reason I know all the JVM folks so much is because I yell at them pretty much all the time."

10:30 "Oh my god, [because we derive from java.util.Collection] we have put more lines of code into supporting a method that nobody in their right mind would call for a concurrent queue, than almost everything else combined. Between iterators and interior remove, that's probably 80% of our concurrent queue code. The algorithms themselves, they're maybe 20 lines, they're cool but they're not a lot of code."

0:57:10 "How do you get a good CAS loop? CAS until success? Don't do while, do do/while! Why? Safepoints! Remember safepoints I mentioned long ago? Yeah, they're not your friend. [...] You know the great thing about JITs is that they work so well, and the bad thing about JITs is that, you know, sometimes they're idiots! [...] Boxing, oh my god I hate boxing. Classloading appearing at mysterious times."

Maybe to you the JVM gives enough benefits to be worth giving up all this control, but to me it it's just nuts that you'd put yourself on top of a platform that you have to work against so much.

And I still don't buy that GC makes for better lock-free data structures. Lock-free data structures may need to have a GC-like scheme _internally_ to memory-manage their nodes, but I can't see any reason why this requires the entire program to be subject to GC.

<comment><USER5> > I can't see any reason why this requires the entire program to be subject to > GC.

It doesn't. Lots of Java low-level programs (DBs, caches etc.) manage most of their heap manually. But an "internal" GC would have to be just as good as a "whole program" GC anyhow in order to make the non-blocking data structure robust (I generally don't use the term lock-free, as it's a very specific case of a non-blocking data structure; many interesting nonblocking DSs are only obstruction-free rather than lock-free).

Also, GCs have other advantages, such as a much higher memory allocation- reclamation throughput than any general manual scheme, especially (though not only) in the multi-threaded case. Many GCs may introduce pauses, but all the good ones would give you a much higher throughput than anything you can achieve manually.

> Maybe to you the JVM gives enough benefits to be worth giving up all this > control, but to me it it's just nuts that you'd put yourself on top of a > platform that you have to work against so much.

Yes, to anyone doing hardcore multicore programming, the JVM is probably the best choice. You are not giving up control, though, because C++ has far worse problems in those areas (forget about safepoints - C++ doesn't even have a memory model, or didn't until C++11, and most C++ programmers don't even know what ordering guarantees their particular compiler provides). Also, if you think you're not "giving up control" in C++ (or C; or Assembly), then you're sorely mistaken. You have little or no control over ILP, prefetching, branch- prediction, outstanding cache misses and all the magic modern CPUs do at runtime. You could think of the JVM as an extension of the CPU only in software – modern CPUs do so much crazy stuff to your code anyway, that you can't possibly know how exactly it will be executed even if you program directly in machine code. You can no longer get truly close to the metal, so being one step rather than two steps removed is not much of an advantage any more.

> "Oh my god, [because we derive from java.util.Collection] we have put more > lines of code into supporting a method that nobody in their right mind would > call for a concurrent queue, than almost everything else combined. Between > iterators and interior remove, that's probably 80% of our concurrent queue > code.

Sure. The code is often complex to support moot use-cases. Still, it's leaps and bounds beyond anything in pretty much any other language. Again, the reason is mostly coincidental. From its beginnings, more or less, Java (or Sun) has attracted the most prominent concurrency researchers. Java 1.5 was the first general purpose programming environment with a memory model, iron- level access to concurrency primitives (fences, CASs), and some world-class GCs, all requirement for state-of-the-art concurrency research. To this day, concurrency researchers prefer the JVM, despite its annoyances (and what language doesn't have those?) to any other platform out there. That's why most concurrent DSs get implemented in Java first, and their Java implementation is usually the most polished and robust.

But to quickly go back to where we started, a higher-level PL often brings about performance improvements. There are famous examples of sorting algorithms in C++ easily outperforming their C implementations. Why? Because of templates vs. function pointers. More precisely - due to inlining, as inlining is "the mother of all optimizations" (it opens the door to other optimizations). Now, of course you could achieve the same in C, by writing the sort function many times, but it's cumbersome and error prone.

The JVM is "the inlining magic machine". It inlines code that a C++ compiler never could. There are some areas where C++ is more performant, namely computations on arrays of complex numbers, but hopefully that will be addressed in Java 9. But this inlining capabilities of the JVM is why it often outperform C++ even in single-threaded code. Again, you could achieve the same in C++ by writing the same code several times (or by generating machine code at runtime, which is what the JVM does). Also, this effect is usually perceived in larger applications, because then you use virtual methods.

Anyway, that's pretty much what I have to say on the matter. I have a lot of respect for C++. I was a C++ programmer for many years. The JVM brings many interesting performance improvements over C++, as well as performance regressions in comparison. All in all, Java usually comes up ahead in large codebases, and falls behind in specific use cases (mostly numeric HPC). In addition, there is no other programming language/platform out there that even comes close to the JDK's power when it comes to hardcore concurrency; some successful implementations find their way to TBB a few years later, and some don't. If you're trying to push the envelope on concurrent software development (my day job), Java is pretty much the only sensible way to go.

Still, there are valid reasons to prefer C++ over Java in many circumstances, but sheer performance is seldom one of them.

<comment><USER6> > Also, GCs have other advantages, such as a much higher memory allocation- > reclamation throughput than any general manual scheme

A benefit of using C/C++ is that you are not limited to general schemes. If you have an allocation/deallocation hotspot, you can use more specialized schemes like arena allocation, which will beat a GC handily.

Also, latency and predictability are often more important than throughput. GCs fare poorly on these counts.

> forget about safepoints - C++ doesn't even have a memory model, or didn't > until C++11

Aka it _does_ have a memory model. And one of the points Doug Lea made in the talk you linked is that the Java memory model is actually broken and no one knows how to fix it, whereas C++'s is not (though it's broken in ways that apparently don't affect anyone).

> Also, if you think you're not "giving up control" in C++ (or C; or > Assembly), then you're sorely mistaken. You have little or no control over > ILP, prefetching, branch-prediction, outstanding cache misses and all the > magic modern CPUs do at runtime.

This is a tired and weak argument.

A mispredicted branch costs 5ns. A main memory reference (cache miss) costs 100ns. And you actually have a lot of control over these things; you can make your branches more predictable and the working set of your tight loops smaller so it will fit in cache. And there are actually instructions that give control over prefetching.

A Java GC pause on the other hand is often on the order of 200ms. That's six orders of magnitude in difference, and can be even longer in degenerate GC situations. There is really no bound to the worst case.

The loss of control also comes from the fact that the effects are highly nonlocal. If your C/C++ code is in a tight loop, the microarchitectural effects are mostly a function of your own code and its behavior. But whether a safepoint triggers a GC pause in Java is a function of the entire program's behavior.

So you really can't equate these two things. Java's loss of control is many orders of magnitude larger and less local.

> But this inlining capabilities of the JVM is why it often outperform C++ > even in single-threaded code.

I don't think you've made this case at all.

> All in all, Java usually comes up ahead in large codebases

Coincidentally it is large codebases where it is most difficult to compare apples to apples, because it is so unlikely that you can really get two large programs that do the same thing with the same design. I'm deeply skeptical of this claim.

> and falls behind in specific use cases (mostly numeric HPC).

Java falls behind it tons of cases. It loses in nearly all of the benchmarks game, only some of which are numeric: [http://benchmarksgame.alioth.debian.org/u64q/java.php](http://benchmarksgame.alioth.debian.org/u64q/java.php) It loses very badly in this benchmark published a few years ago by Google, which is not numeric: [http://static.googleusercontent.com/external_content/untrust...](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/pubs/archive/37122.pdf)

Given comparable programs, C++ is nearly always faster. You really haven't offered any evidence that contradicts this basic fact. The only benchmark you cited didn't even have any C++ programs in it.

<comment><USER5> > A benefit of using C/C++ is that you are not limited to general schemes. If > you have an allocation/deallocation hotspot, you can use more specialized > schemes like arena allocation, which will beat a GC handily.

Absolutely. And you can do the same in Java with about the same amount of effort (lots of Java projects do that).

> it does have a memory model.

It does, and it probably fixes some mistakes in the JMM, but C++ got it, what, 10 years after Java? That's about how far behind C++ is in terms of concurrency. I know this because that's my business (concurrent data structures and high concurrency on many-core hardware in general - you really can't do C++ well on a many-core system; it doesn't scale unless it's embarrassingly parallel).

> A Java GC pause on the other hand is often on the order of 200ms.

It could actually be much worse. Fortunately, you could get a commercial GC that offers exactly 0 pauses (in fact, I suggest my customers do exactly that if they need 0 pause but fall short of true hard real-time; for hard real-time there's real-time Java with built in arena allocation with correctness guarantees of no pointers from the heap into the arena).

> Java falls behind it tons of cases.

I know I won't be able to convince you, but those benchmarks are completely irrelevant. I can easily make any single-threaded algorithm perform better in C/C++. But, as I've said in the beginning, Java comes out ahead when you have a large codebase with many team members, which requires software engineering that precludes good optimizations. This realization is a result of many years experience with both Java and C++. I don't have much hard evidence (some, but it's possibly anecdotal and also proprietary), but in our field, as in medicine, sometimes hard-earned first hand experience means a lot.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> It's an impressive project that shows the extraordinary work of a single developer, Mike Pall. LuaJIT is not only an incredibly beautiful JIT, the current project also demonstrates it can run generated code efficiently, which is probably the hardest thing to do for a language.

When implementing Opa in OCaml, we run into tons of problem since the OCaml runtime was unable to run generated code efficiently. One of its author, Pierre Weis told us at the time: "it's normal, OCaml is not meant to run generated code at all".

As a side note regarding Luje source code: Fossil must be great and all that but having the code on a publicly available Git repository would probably bolster contributions.

<comment><USER8> Luajit is of course awesome, but let's not get too carried away. It doesn't sound like this JVM has been used for running anything beyond extremely simple benchmarks. Of course you're not going to run into difficulties with the translation of a 15 line method.

I saw a talk Mike gave on Luajit last week, and one thing he credited Luajit's success on was that it was working on a rather high abstraction layer. Stuff like the loop optimizations and alias analysis work because the optimizer can essentially understand the code at a fairly high level, and is somewhat specific to Lua. When dealing with java translated to jvm bytecode naively translated to lua source you lose much of that context. (And at the time he didn't seem too convinced about using luajit as the backend for other languages.)

But even if you ignore the loss of context, I'm sure that various luajit limits would get hit with real life Java code. E.g. I'd guess there's a fundamental 65k limit for the number of constants and bytecode instructions in a single function.

<comment><USER9> JVM bytecode is quite nasty from this point of view; one of Mike's comments in the mailing list was to try Dalvik instead. It loses a lot of loop structure.

<comment><USER10> [http://www.freelists.org/post/luajit/ANN- luje-01](http://www.freelists.org/post/luajit/ANN-luje-01)

</comment></comment></comment></comment><comment><USER11> There's an interesting paper about a similar project that was done using Self [1]. "Design and Implementation of Pep, a Java Just-In-Time Translator written in Self"

It goes through details with benchmarks. The PEP code is available with the Self distribution. They were able to get faster than the Java implementation at the time. Back then Java was interpreted though.

[1] [http://bluishcoder.co.nz/self/97-pep.pdf](http://bluishcoder.co.nz/self/97-pep.pdf)

<comment><USER12> Languages are nor interpreted or compiled. It is all about implementations.

</comment></comment><comment><USER13> Isn't this going to be intrinsically restricted to one numeric type? Lua numbers are all the same type (double by default).

<comment><USER9> No, LuaJIT has an ffi interface with support for any type, and you can just use them as if they are native types.

<comment><USER14> Seems a bit like cheating if you do it that way, though.

<comment><USER9> Its not really, they are just native types, but defined with a C syntax so they are compatible with C. The code generation knows about them natively and optimises them.

</comment></comment></comment></comment><comment><USER15> Is LuaJIT bytecode a good target for toy languages?

</comment><comment><USER16> is there any java/lua interop?</comment>


Decoding the Thought Vector - <AUTHOR> http://gabgoh.github.io/ThoughtVectors/ <comment><USER2> When I was working on a recommender for television shows, I ran SVD on a large User/Item matrix to create a low rank approximation, essentially reducing thousands of user features (TV show preferences) to user vectors representing twenty or thirty abstract "features". Then I looked at the actual item preferences of users who expressed each feature at the greatest and least magnitude. The features, in some cases, mapped to recognizable constructs. There were distinct masculine and feminine features, several obvious Hispanic / Latino elements, and strong liberal versus conservative indicators. Others were less explainable using common labels.

It struck me at the time that the qualities that were expressed most strongly were the ones that ended up having names in our language. But there were others for which I would say to myself, there is something about this group (e.g. those with the greatest expressed value of F124) that I recognize, but can't quite put my finger on.

Of course, I was looking at people through a keyhole, their TV viewing preferences being the only information I had.

Also, I noticed that these "came into focus" most clearly at a certain level of compression (rank).

FWIW

<comment><USER3> A question: is this much better / different than a principal component analysis (or a factor analysis)?

<comment><USER4> It's a bit of an apples/oranges comparison to compare SVD to PCA. SVD is a numerical technique, whereas PCA is a method to analyze a dataset. You can use SVD to perform PCA (although there are other ways to perform PCA without explicitly doing a SVD). I'm guessing that the GP performed PCA using SVD. There's a good Stack Exchange answer to exactly this question here:

[http://stats.stackexchange.com/questions/121162/is-there- any...](http://stats.stackexchange.com/questions/121162/is-there-any- advantage-of-svd-over-pca)

</comment></comment></comment><comment><USER5> > Rather curiously, it turns "airplanes" into "knives". I do not understand > why this happens.

I thought it was pretty obvious. The atoms are complected (defined - by Rich Hickey of Clojure - as, basically, a semantic that contains multiple interdependent concepts (for example how variables complect state, values, and names)). In fact that's the conceit of the whole idea, the thought vector is being extracted from the sparse matrix, sometimes that sparse matrix isn't that sparse and you will get complected concepts. It was obvious in the earlier pics. One atom might contain a piece of information needed by a hat, that when combined with other atoms makes a hat, but when combined with a different set makes a headband.

For example if you look knives is shared with scissors. It's one atom describing roughly "handheld sharp objects". The airplane atom + many items atom is actually a special mutation for many knives. Where as the many items atom + sharp objects atom is likely scissors. And the sharp object + airplane atoms are probably knife. They are all complected and interdependent.

Sure an atom may generally mean a specific concept but sometimes it will fall back to a combination specific mutation. For example there aren't often many planes, and many planes looks rather like many knives. And there probably aren't ever more than one pair of scissors, and one pair of scissors looks rather like two knives. It's a way to describe 3 things and their number (knives, scissors, plane) using 2 things, an existing counting mutator, and the fact that scissors and planes are often singular. It's a form of semantic compression, quite interesting, and I would imagine domain dependent.

That's my hypothesis anyway.

<comment><USER6> This explanation reminds me of a concept in ASL (American Sign Language) called Classifiers:

[https://seattlecentral.edu/faculty/baron/Summer%20Courses/AS...](https://seattlecentral.edu/faculty/baron/Summer%20Courses/ASL%20223/ASL223ASLClassifiers.htm)

They are "class" modifiers that modify different nouns in different ways.

</comment></comment><comment><USER7> This is incredible and infinitely useful. For many years, I've had this hunch that Human symbols and abstractions had an algebraic quality. And, I have always wanted to substitute tags, category, labels, and attributes with global, permanent indices. Mainly, I wanted to do this because the 'view' of a thing is mutable: e.g. a word for a concept changes over time even when the conceptual meaning remains constant. I can't wait to get home and play around with this.

</comment><comment><USER8> Much of AI and Machine Learning all boils down to the vector and vector space in addition to how well those features are engineered, constructed, scored and ranked.

</comment><comment><USER9> Nice interactive examples but I'm afraid the basic setup here doesn't make sense to me. The "atom" is defined as the average encoding of inputs with the feature ("faces with a smile"), but I'd think the proper definition should subtract off inputs without the feature (i.e. "smile" = "faces with a smile" minus "faces without a smile"). The way it's defined you end up adding an extra "average face" along with the feature of interest, which is clearly seen in "The Geometry of Thought Vectors" example -- the non-smiling woman isn't so much forced to smile as to have her face merged with that of a generic smiling woman.

</comment><comment><USER10> Is there any way to read these responsive sites on mobile when they are cut off by the column margins?

<comment><USER11> Request desktop site

</comment></comment><comment><USER12> The "atom" terminology is a bit confusing to me. Isn't an "atom" just another thought vector? If "sunglasses" \+ "smiles" = "smiling-while-wearing- sunglasses", then "sunglasses" and "smiling-while-wearing-sunglasses" are both just vectors. Is there a reason for the distinction?

Also, if all thoughts can be described as vectors and linear combinations of vectors in "thought-space", I wonder what the axis represent and how many dimensions there are. Are all thoughts just a combination of 100 "unit thoughts"?

Really interesting post!

</comment><comment><USER13> I find it counter-intuitive that thought vectors in should have "Linear Structure" in multilevel autoencoders.

Since the whole appeal of neural networks is that they can model non-linear functions. Why would the autoencoder end up with an encoding that is essentially linear?

<comment><USER4> The goal of a neural network is to take a complicated manifold and, through each of its layers, flatten it out into progressively more and more linear manifolds. If you are building a classifier, then the inputs to the last layer will necessarily have to be linearly separable because the final layer is essentially linear --- the softmax operation just transforms the pre- activation values from logits to probabilities.

So if the NN is well trained, the second to last layer will have linearly separated the different classes (as much as possible, anyway). Earlier layers may not have completely linearly separated their inputs, but they are probably going to lie along simpler manifolds than even earlier inputs.

There's a good blog post on this here: [http://colah.github.io/posts/2014-03-NN-Manifolds- Topology/](http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/)

<comment><USER13> Yes I understand that conceptually. But the middle autoencoder layer is not really an output layer, what would constrain it to linear representations? I assume normal output layers are constrained to linear representations by the supervised training process.

</comment></comment></comment><comment><USER14> From the post:

 _> Rather curiously, it turns "airplanes" into "knives". I do not understand why this happens._

I would venture to guess that this happens because of the existence of an plural ambiguous "thought" bridging the _" airplane"_ and _" knives"_ concept vectors, along the lines of _airplanes - > propellers -> blades -> knives_, and the _" a group of"_ vector is causing the system to jump over that semantic ambiguity.</comment>


Google CEO Sundar Pichai Testifies Before the House Judiciary Committee - <AUTHOR> https://www.youtube.com/watch?v=zIniYSkAWo0 <comment><USER2> Who is the person wearing a _hat-mustache-eyeglass_ setup behind and right of Sundar Pichai?

<comment><AUTHOR> [https://www.newsweek.com/monopoly-man-made-appearance- equifa...](https://www.newsweek.com/monopoly-man-made-appearance-equifax-and- wells-fargo-hearings-heres-why-677984)</comment></comment>


Did you call them autocrap tools? - <AUTHOR> https://www.varnish-cache.org/docs/trunk/phk/autocrap.html

<comment><USER2> If his replacement build system lets me a) build varnish under SUA on a version of windows that didn't even exist when he wrote the project b) cross- compile varnish for arm, then I'll agree with him. The "standard" gnu autoconf layout, for all its m4 insanity, gets you both those, and none of the various wheel-reinventions I've seen do.

</comment><comment><USER3> Yes they're a pain in the ass, but a necessary one. `uname -s` obviously doesn't cater for the divergences and bugs within a platform.

Have a look at gnulib for things one has to deal with. BTW, one can just use gnulib and forget about lots of portability issues

</comment><comment><USER4> Some interesting discussion in the previous submission (18 months ago): <http://news.ycombinator.com/item?id=1499738></comment>


Madrid taxis block major road in biggest anti-Uber protest yet - <AUTHOR> https://www.reuters.com/article/us-uber-spain-strike/madrid-taxis-block-major-road-in-biggest-anti-uber-protest-yet-idUSKCN1PM117 <comment><USER2> They are doing similar protests here in BCN too.

The taxis offer an inferior service at a higher price - adapt or die.</comment>


Auto detecting page content via PHP - <AUTHOR> http://getpagecontent.com/?url=http%3A%2F%2Fwww.foxnews.com%2Fus%2F2015%2F05%2F30%2Fsavopoulos-assistant-allegedly-lied-to-police-about-details-40000-cash-delivery%2F

<comment><AUTHOR> Sources here [https://github.com/mihailShumilov/readability](https://github.com/mihailShumilov/readability)

</comment><comment><USER2> Nice - I was looking for a tool to scrape page content length.</comment>


No Founders Required: How Flagship Starts Companies - <AUTHOR> http://bostinno.streetwise.co/2013/04/15/no-founders-required-how-flagship-starts-companies/ <comment><USER2> I've always wondered if a tweaked version of the DARPA model might work for VC's. The gist is that DARPA hires a bunch of PhD's (to limited terms) to serve as program managers. The PhD's have an area of expertise and a research agenda, but not specific ideas. During their tenure, they think of specific projects and get a few tens of millions to fund them. Obviously for a VC you'd want to pin down ideas into more concrete, less blue-sky sorts of things with correspondingly smaller budgets.

I think the real benefit to the model is that you decouple idea generation from execution. The guy who is an expert in a particular field and can analyze the literature to see what can be done isn't necessarily the right guy to bring a product to market. Indeed, that expert doesn't necessarily want to take all the risk entailed in bringing a product to market, but may be the person best positioned to determine what can and can't be done.

<comment><USER3> My field (broadly, infosec) is disproportionately favored by DARPA, and I feel like the results from DARPA-funded work have been iffy at best. There's some standout work, but it floats in a sea of mediocre (or worse) stuff.

<comment><USER2> I'm probably biased, but I've been impressed by their work in the field I used to work in (white space wireless). I spoke to a VC the other day who seemed surprised that we were working on stuff back in the early 2000's that he's only been seeing commercially in the last couple of years.

I guess the nature of the model is hit and miss...

</comment></comment></comment><comment><USER4> So here is the fundamental flaw in this model. Conflict of interest.

If I, as an entrepreneur, walk into the VCs doors to pitch them on something I have done and research I am doing - presumably without knowing that they have invested (or are investing in similar research with a view to taking that research into a commercial business) why should I feel comfortable knowing that they won't take my research and use it without me.

I know that a successful business is more than just 1 idea, but if two sets of eyes are looking at the same problem, both trying to build a business around it and 1 has some insight that the other doesn't....that's a competitive advantage. That could be used against the entrepreneur - given that VCs tend to not want to sign NDAs & non-competes before hearing pitches (understandably so).

So this whole VC investing in their own ideas, while also hearing pitches from founders....I am not too bullish on.

The potential for conflicts are just too great.

This is like having an EIR but worse....from the pitching entrepreneur's perspective anyway.

</comment><comment><USER5> Lerer Ventures is doing this too, albeit in the consumer space, with their SoHoTechLabs efforts.

<comment><USER6> I believe Redstar is too.</comment></comment>


Minard / Morse / Tufte and Authenticity on the Web (2018) - <AUTHOR> https://www.masswerk.at/nowgobang/2018/minard-morse-tufte <comment><USER2> I think the bigger issue is removing the context and cropping the complete work here, which quite likely is already a grave deviation from Minards intent (note the Northing of the Hannibal map to align with Napoleon's).

It's originally a 2 panel/figure page evoking a comparison between Hannibal's campaign over the Alps with vs. Napoleon's French Army in Russia 1812-1813

Sarah Rendgen just recently got a book published on the topic: _The Minard System: The Complete Statistical Graphics of Charles-Joseph Minard_

Here is the whole print in question:

[https://i2.wp.com/geekdad.com/wp- content/uploads/sites/3/201...](https://i2.wp.com/geekdad.com/wp- content/uploads/sites/3/2019/01/PAP_MinardSystem_Spread6.jpg?ssl=1)

from [https://geekdad.com/2019/01/the-minard-system-charles- joseph...](https://geekdad.com/2019/01/the-minard-system-charles-joseph- minards-innovative-use-of-statistical-graphics/)

[Book] [https://www.papress.com/html/product.details.dna?isbn=978161...](https://www.papress.com/html/product.details.dna?isbn=9781616896331)

Landsteiner makes notes on this, with a section in a previous article too:

[https://www.masswerk.at/nowgobang/2018/observing- minard](https://www.masswerk.at/nowgobang/2018/observing-minard)

[https://news.ycombinator.com/item?id=17808302](https://news.ycombinator.com/item?id=17808302)

<comment><USER3> (OP here.) Yes, I think this is really the most important issue. Minard's graph was never meant to be observed in isolation, but is part of a greater visual rhetoric. It is also in this context that I did observe the various sources of the graph a bit closer. – I guess, we may say what has become "our Minard" isn't necessarily the "authentic" Minard.

</comment></comment><comment><AUTHOR> What is most interesting to me is that someone already did a similar research in the English Wikipedia talk page for that file [1], but concluded that it was indeed Minard's (oops) and the error still persists today. It is really understandable as the presented differences are minor and only discernible for the most determined people though.

[1] [https://en.wikipedia.org/wiki/File_talk:Minard.png](https://en.wikipedia.org/wiki/File_talk:Minard.png) (note that it is actually a Commons image, so there are two talk pages for the same file)

<comment><USER4> Welp. I have no recollection of what research I did to conclude that it was "legit", but I believe that was my post on Wikipedia. As I don't have memory or note of that research, I can't really stand behind either my initial claim or my counterclaim.

</comment></comment><comment><USER5> If Elaine Morse is still around she could perhaps shed some light. Perhaps Tufte knows her.

(Many years ago I went to a talk by Dennis Ritchie about *roff. He showed the graphic and said that someone at Bell Labs had taken the time to code it up. I have no idea if that person is Ms Morse, though.)

</comment><comment><USER6> I received physical promotion for Tufte twice in my personal letterbox (while in USA) too my high surprise. Of course it made me check him out online, but I think his presentation is not something for me.

Edited for clarity since I apperently got downvoted

<comment><USER7> I went to one of his seminars and walked out. There was very little substance. At one point he told everyone they should bring a technical document when bringing their pet to the vet.

edit: Some other things he suggested: don't use powerpoint. Write a technical document and have everyone read it before the meeting. He also showed how he himself could have saved the space shuttle discovery.

Sorry if I'm coming off as a bit negative. I like the books, they have some good insights but the seminar itself was terrible.

<comment><USER8> That's unfortunate that you had such a bad experience at the seminar. I went to one early in my career (maybe 2001) and I consider it a pivotal moment. The seminar and books have influenced my thinking on design, experience, UI, decision-making, and how to present effectively, whether on paper or in person. Learning to use the concept of the data-ink ratio alone has paid off a thousand-fold for me. The other concept that sticks out to me: "good design cannot save failed content." I also agree with his stance on presentation software.

<comment><USER9> I also attended his seminar in 2001 and it heavily influenced my life and work. As an example, when my wife was giving birth to our second child I made a plot, a simple one with paper and pen, of the time between contractions. I did that because the first time they tried to hurry her along with drugs. The second time when the doctor came in to whine that we were "not making progress" (read: the doctor has dinner reservations at 6) I was able to show convincingly that progress was indeed steady. The simple act of visualizing data can influence decisions, which is the point of his chapter on the Challenger disaster.

<comment><USER10> Sitting on my bed in Cape Town, South Africa and knowing that I'll probably never be able to attend his seminar, it's nice to read your folks good experiences.

It's nice to think that something you dream about is in fact something fantastic.

Tufte had a huge impact on me too, around the same timeframe (2001.)</comment></comment></comment></comment></comment>


Facing meat shortages, some Americans turn to hunting during pandemic - <AUTHOR> https://www.reuters.com/article/us-health-coronavirus-usa-hunting/facing-meat-shortages-some-americans-turn-to-hunting-during-pandemic-idUSKBN22F0G4 <comment><USER2> I think a lot of these folks underestimate what it takes to hunt an elk, field dress it, bring it back to their vehicle, and put away 300 lbs of meat. Even if you borrow a gun as one person interviewed here would, you need a truck, knives, a bone saw, camouflage, maybe a tree stand, and a large freezer, among other things. It is reasonable to hunt for food, but it's not reasonable for your typical "hospital manager" to go out in the woods and take his first shot at a large animal.

Lots of people from NYC suburbs and Long Island head up into the nearest forest every hunting season and accidentally shoot each other, maim animals and don't have the stamina to chase them to complete the kill, etc.

If you're concerned about food shortages, it would be much easier to raise a few backyard chickens or rabbits if you have the small amount of space required for it. Much less likelihood of a gun accident that way too.

If you're set on going out in the woods for a "mental cleanse" and to kill an animal, you could do worse than starting with a read of Jackson Landers' excellent Beginners Guide to Hunting Deer for Food ([https://www.amazon.com/Beginners-Guide-Hunting-Deer- Storey/d...](https://www.amazon.com/Beginners-Guide-Hunting-Deer- Storey/dp/1603427287)).

<comment><USER3> It's also a good excuse to learn to hunt. Every single experienced hunter had their first hunt at some time.

My family has hunted for generations and I have hunted for nearly a decade now. You are right, there is an elevated risk of accidents. But the experience of hunting is nothing like raising chickens. Staying out in the woods for days, becoming more and more tuned to the animals and environment around you. Feeling the adrenaline when you hear a huge animal tromping through the woods, just to look over and see a couple squirrels playing. There's nothing like it.

If this pandemic draws more people out to try hunting, and in so doing, they have similar experiences to those I've had, then I'm fine with it.

<comment><USER4> To echo this: Hunting is good for the animals.

I know, I know, that seems like an oxymoron, like jumbo-shrimp or old-news. But in the US at least, a lot of national parks are funded via hunting permits [0]. The same is true in many parts of Africa. Those photos you see of some rich person with an elephant or a 5 point buck? The permit to kill that animal cost a _lot_ of money and the parks are very grateful to have the cash [0].

NPR had a good story on how conservation in part depends on hunters: [https://www.npr.org/2018/03/20/593001800/decline-in- hunters-...](https://www.npr.org/2018/03/20/593001800/decline-in-hunters- threatens-how-u-s-pays-for-conservation)

Yes, the issue of hunters/poachers is a very complicated one. But what the sportsmen take, generally, is less than what they give back.

[0] The funding of parks is very complex, but permits do play a large part, generally. Each park is unique though.

<comment><USER5> In our state the conservation efforts, the DNR, everything comes from hunting and fishing licenses. So-called environmentalists who criticize hunting (at least here) are ironically doing little or nothing while the hunters are supporting the cause.

</comment></comment></comment></comment><comment><USER6> This pandemic is really bringing out the folly of concentration at the expense of all other business considerations. Sure, it makes food cheaper but at what cost? We deserve systems that are more resistant to shocks. We've allowed oligopolies to form in many industries, perhaps especially agribusiness. This has brought prices to rock bottom and commoditized living beings, putting pressure to further reduce prices. "Ag-gag" laws and the might of huge consolidated meatpacking firms makes it difficult to do investigative journalism.

Maybe it's time for people to try going meatless instead? People are so disconnected from their meat supply and the horrific conditions ([https://www.washingtonpost.com/business/2020/04/25/meat- work...](https://www.washingtonpost.com/business/2020/04/25/meat-workers- safety-jbs-smithfield-tyson/)) imposed on the people who provide that meat to them. If we had a different political climate, we could think about sensible reform in meat.

Meat should cost 3-5x as much as it currently does ([https://www.theguardian.com/environment/2019/jul/21/the- true...](https://www.theguardian.com/environment/2019/jul/21/the-true-cost-of- eating-meat-if-we-want-change-we-have-to-pay-for-it)) to allow for a safe and healthy habitat for the animals, and a living wage to the farmers and packers.

Finally - if you, like me, grew up eating meat and never considered what a meatless diet would look like, now's a perfect time to try while you're stuck at home learning new things. There's a large community of former meat-eaters that would be happy to help you start.

<comment><USER7> But meat is _sooo_ good, and there's nothing else even close, that I'm willing to pay more for it. I'd rather see the supply chain issues worked on, even if it just means more expensive meat. Going meatless is like cutting of the nose to spite the face.

</comment></comment><comment><USER8> The article is more about legal hunting. Not uncommon for 'guides' to provide a full service experience, including the gun in your hand, the clothing on your back, coffee thermos at your side, and all the cleaning/butchering while you have a beer. Not an economical way to fill your freezer, but it is a bonus if you were looking to 'try' hunting. Hunting elk like that is closer to you going deep sea fishing on a charter.

I would have expected an article more along the (legal) varmint hunting for wild hogs down in the south or straight up increases in poaching. I know some farmer have discovered cows .. dead and missing only a few choice bits.

</comment><comment><USER5> There is a tiny fraction of the available game to feed America. A single deer would feed a family of four for, what, a week? So, every family needs 52 deer per annum. Forget it.

This is sort of why civilization is based on agriculture. Not hunting-and- gathering.

<comment><USER9> I don't think anybody is proposing to replace the entire existing food supply chain with nothing but hunted deer.

There is no reason however, to think that hunting can't supplement a family's food stocks if other sources start to become thin. And there's a lot more to hunt than deer.

There's also more in play than hunting. A family might feed themselves through a combination of any or all of: hunting, trapping, fishing, foraging for wild edibles, home gardening, AND buying from whatever is available on the market.

One interesting thing about this is that those living close to the coasts may be in better shape in one or two regards. Most notably, seafood is one area where "direct to the consumer" sales is a real thing, and there isn't a lot of processing involved. At least where I grew up on the NC coast. It's common for seafood houses to sell oysters, clams, shrimp, etc. directly to consumers.

<comment><USER5> Again, there is every reason to believe (know) hinting/foraging is woefully inadequate. Simple numbers. Natural 'foods' are pitiful fractions of modern cultivars. It takes a quarter acre(?) of Jerusalem Artichoke to gather a few baskets of roots, which must be boiled and peeled down to a few score pounds of edible material. Even a bonanza 10-acre plot in a meadow would feed 40 people for a few days.

Early Americans spent the whole summer struggling/working like slaves to put up enough food for the 3 months of winter.

Seafood - that sounds like continuing to rely upon the food supply line (fishing boats and their supply chain). That used to be pretty good return. Not sure stocks today could feed a fraction of America for long. Most of us eat seafood a meal a week/month that way even in the best of times.

<comment><USER9> I don't really understand what point you're trying to make, w/r/t the post you're replying to.

Again:

 _A family might feed themselves through a combination of any or all of: hunting, trapping, fishing, foraging for wild edibles, home gardening, AND buying from whatever is available on the market._

I'm saying that hunting, etc. are individuals components of a broader strategy for providing food - not that any one of those things is sufficient in and of itself. As as of today, it doesn't appear to me that the commercial market for food is going to just disappear overnight. But if things devolve to the point that you can't buy _everything_ you need, then some of these other approaches can come into play.

 _Seafood - that sounds like continuing to rely upon the food supply line (fishing boats and their supply chain)._

There isn't just one "food supply line." The supply chain for the steak you buy at Food Lion involves a lot of intermediate steps and processing. And those are exactly the steps where we've heard news reports of the processing plants closing down, etc.

My point about seafood is that there are fewer such intermediate steps involved. A boat comes in, docks, and sells their shrimp to the seafood house, and the seafood house sells to the public. The main processing is weighing. And some boat captains even bypass the "sell to the seafood house" part and sell direct.

<comment><USER5> Responded to the comment directly above (?) As a supplement to a family's food supply, hunting and foraging are totally insufficient. Not if everybody was doing it. The numbers aren't there.

Fewer steps in the seafood supply chain, doesn't offset the fact that in America anyway, it currently provides a whopping total of 1 meal a week per. That's gotta be dropping too. So still the numbers don't support any idea of supplementing food supplies to any great degree.

The OP was just a puff piece about an uptick in hunting. That was fun and interesting. I just posted to provide the insight that, this isn't going to ever matter to our situation in any significant way. We have to support/restart our commercial food chain, or its going to get very bad. No alternatives are reasonable.

<comment><USER9> _As a supplement to a family 's food supply, hunting and foraging are totally insufficient._

You can't _really_ make that claim without quantifying the degree of supplementation in question. And that's an unknown. So really this is all just speculation. But I can say that for myself, I'd rather have the ability/knowledge/skills to acquire food by hunting, trapping, foraging, fishing, etc. than not, all things being equal. If you want to take a defeatist attitude and not pursue those things, well... that just means you'll be one less person I'll be competing with for food eventually.

 _I just posted to provide the insight that, this isn 't going to ever matter to our situation in any significant way. We have to support/restart our commercial food chain, or its going to get very bad._

I think we're ultimately talking about two different things here. Of course we have to restart the commercial food supply chain. That, to me, goes without saying. It's also somewhat irrelevant to that I'm talking about (which I think is different from what you're talking about). I'm talking about things to do _in the interim_ for base survival... _until_ the commercial food supply chain is working again. It sounds like you're assuming that we're talking about depending on hunting/foraging/fishing etc. as the default way of life for perpetuity. On that point, I agree with you... it's not reasonable to sustain a population of our current size for an indefinite period of time.

<comment><USER5> Agreed.

If it comes to it, good luck out there hunting with all the amateurs who are hungry, unskilled and outnumber trained hunters 100:1.

<comment><USER9> Let's hope it doesn't come to it though. There's a reason I have a lot of dried beans and rice stored away... :-)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> It's strange that meat processing facilities in the US have been hard-hit by Covid-19.

These are places that have strict hygine standards during normal times, with employees wearing gloves and face masks as a matter of course. So you'd think they wouldn't be somewhere that diseases would spread easily.

This has also not been a significant issue in other countries hit by Covid-19, that I'm aware of. What is it about the USA's meat processors that have made them vunerable?

<comment><USER6> You would certainly expect that these places should have strict hygiene standards, but investigative journalists have found time and time again that slaughterhouses in the USA are filthy places where human and animal disease alike are concentrated by the time pressure to go faster, to make meat cheaper and cheaper.

Fast Food Nation ([https://www.litcharts.com/lit/fast-food- nation/chapter-9-wha...](https://www.litcharts.com/lit/fast-food- nation/chapter-9-what-s-in-the-meat)) profiled this in the early aughts.

Countless other investigative journalists have tried, even against the "ag- gag" laws we have in farming states, to uncover even more in recent times:

[https://www.theguardian.com/world/2020/may/02/meat-plant- wor...](https://www.theguardian.com/world/2020/may/02/meat-plant-workers-us- coronavirus-war)

[https://www.thestar.com/news/insight/2010/05/14/probing_the_...](https://www.thestar.com/news/insight/2010/05/14/probing_the_link_between_slaughterhouses_and_violent_crime.html)

The consolidation of the meat industry has given them extraordinary lobbying power. They now have broad authority to regulate themselves:

[https://www.newsweek.com/trump-administrations-new-hog- slaug...](https://www.newsweek.com/trump-administrations-new-hog- slaughterhouse-rules-may-risk-more-contamination-food- safety-1459922?fbclid=IwAR13lxfSA9I_9WtJWlaJWfCoOBPTyKTpzxv_fpi6Z1_sIt3jWxxsyKHB5BQ)

</comment></comment><comment><USER11> Calling it a "meat shortage" is glossing over the situation. In the US we're killing 160,000 pigs a day just to throw their bodies in the landfill.[1]

So the solution is to go kill more wild animals. Awesome.

[1] [https://www.bloomberg.com/news/articles/2020-04-28/closed- jb...](https://www.bloomberg.com/news/articles/2020-04-28/closed-jbs-plant- will-be-used-to-euthanize-hogs-peterson-says)

<comment> kube-system Most shortages are caused by logistics, not supply.

</comment></comment><comment><USER9> I can relate to this, to an extent. I never hunted in my youth, despite having grown up in redneckville. I was all about bass fishing in farm ponds, not hunting. I was enthusiastic about shooting though, and I'm a lifelong gun owner and have done a fair amount of sport shooting. So at least I already know how to use a gun and do so safely.

Anyway, even before this COVID thing started, I had gotten into the whole prepper mentality a bit and had started thinking "I should learn to hunt in case I need to bring home some food one day." I went and took the NC Hunter Safety Class back in January, just so I could get my hunting license. I had planned to team up with some family and friends who hunt this coming fall, to learn the ropes. Now I find myself wondering if I'll be forced to try and find a way to accelerate the learning curve.

At least I do know how to fish. I wouldn't necessarily want to be forced to support myself entirely on fishing, but I could make up at some some portion of my diet on fish if push came to shove.

<comment><USER12> I've had similar thoughts, the reality for me is that, should it come to needing to fish/hunt for food, old methods aren't going to work. People are going to fence in/trap big game, they are going to net/troll/spike water even in ponds. If the rules of subsistence change enough that hunting becomes required, accessing land in camo and a 4x4 will be outdated. My 2 cents.

</comment></comment><comment><USER13> I was so mad when i saw that meat workers were being forced back to work i stopped buying meat

</comment><comment><USER14> "Rent a gun" to "shoot some elk".

This is as close to the proverbial foot gun you can get.

<comment><USER15> Yup. This is a classic case of something making the news because it is atypical.

I found it funny that the article stops just short of implying that hunting is driving gun sales when despite the obvious 800lb gorilla (Covid driving the sales of any/all emergency preparedness products).

I didn't even know gun rentals outside a range (pay money to shoot stuff you can't afford to own) or B2B (e.g. security or armored car company leases handguns from a gun shop) context were a thing but it makes sense in states where the only thing you'd need a full sized rifle cartridge for is assigned tags via a lottery so every season you have plenty of hunters who suddenly have a tag that they might not be equipped to fill.

</comment></comment><comment><USER16> Resurgence in hunting... air travel is rare.... home hair cuts... families are playing board games....

We've gone back to the late '40s!

</comment><comment><USER17> I wish that was an option for me. I dont live near elk. i could eat the rabbits in my yard but that wouldnt go aong way.

<comment><USER18> Wrong time of the year to eat wild rabbits. Wait until a few weeks after the first hard frost, else you run a risk of contracting tularemia.

Unfortunately, I missed the small window to get a hunting license in my state this year, it closed in Feb, before CV-19 was much of a thing. Fortunately, no license is needed here to hunt rabbits. From what I hear, the same is true for feral hogs, which are becoming a bit of a problem in some parts. Next year, I am definitely putting in for an elk tag. I got a 4x4 largely so I can get into the areas where they hide.

<comment><USER19> Don't rabbits die if they are infected with tularemia? Is there actually a correlation between temperature and tularemia spreading in rabbits? It seems incredibly rare.

I can see there are more cases in humans during warmer months, but that is probably due to insects that are the most common disease vector, no?</comment></comment></comment>


SPA-like user experience with back end only: Phoenix.LiveView: - <AUTHOR> https://www.youtube.com/watch?v=Z2DU0qLfPIY&feature=youtu.be&t=918 <comment><USER2> This is very interesting and a project that I'll track to see how it evolves. I agree fully that for many applications a SPA and the mess of a technology stack that it usually involves is overkill.

Part of the current impl doesn't seem like it is going to scale as the DOM in a page gets larger and larger: he mentions that right now they are sending the full rendered HTML from the server to the client on every update, then doing the diff client-side. They are looking at optimizations, hopefully sending diffs from the server is one of them.

</comment><comment><USER3> See also Drab [https://tg.pl/drab](https://tg.pl/drab) The extension library to Phoenix Framework for providing an access to the browser's User Interface (DOM objects) from the server side.</comment>


Google Plus Tells Pseudonym Lovers to Shove It - <AUTHOR> http://www.readwriteweb.com/archives/google_plus_tells_pseudonym_lovers_to_shove_it.php

<comment><USER2> I'm not going to throw a hissy-fit over every social application that doesn't cater to my perceived personal needs, but as someone who is well-known on the Internet by my pseudonym, I have no use for Google+ and its "circles" unless I can be raganwald to some people and Reg Braithwaite to others. Or until they drag me into it my getting practically everyone I know to use it instead of Facebook.

</comment><comment><USER3> Disclaimer: I am a Google fanboy.

I am as unhappy about this development as anyone. I don't understand the insistence on real names, not just because I think it's wrong, but I just don't understand what the motivation is. What's the upside?

Here's the only thing I could come up with:

Community. This early in its life, they don't want Google+ to become a cesspool of fake/joke accounts, youtube comments, spam, and "brands." They want it to be a real place for real people to share about real things. They don't want it to become Myspace. [Edit: I think you can also see this emphasis on noise reduction in how Circles are set up and how game notifications aren't part of your stream unless you want them to be.]

But I hope the plans are bigger. I hope they're just going for a nice clean introduction for a large enough user base, while temporarily marginalizing edge cases in order to give the mainstream a pleasant initial experience, so they can leverage that user base into a more successful version of what they tried with Buzz and Wave: an open, standards-based interoperable social networking environment that's more like the open web than a closed social network.[1]

The fact is, if they're planning on opening it up eventually, there's no stopping pseudonymous/anonymous activity or brands or even spam to some degree.

That's my optimistic fanboy take on it. I just don't see what else they get out of the policy. I definitely don't buy the "it's for advertising" line. They don't need your real name for that. And it just seems like too much of an obvious backlash and logistical mess to have to deal with unless they have a good, large, long-term reason.

[1]: [http://radar.oreilly.com/2011/07/google-plus-social- backbone...](http://radar.oreilly.com/2011/07/google-plus-social- backbone.html)

<comment><USER4> I've said in other threads, I think the average Facebook user would be put off commenting on a post where "joebadmo" is a participant.

I don't see anything wrong with having a division between real people and brand/pseudonyms. I think this is one thing Facebook got right.

<comment><USER5> "the average Facebook user"

I think G+ could aspire to so much more than being an average Facebook clone.

</comment></comment></comment><comment><USER5> Nothing with real consequences at stake relies on a name, real or otherwise. We rely on social security numbers and credit card numbers in the US. Drivers license numbers. Telephone numbers. Account numbers.

No organization that really cares who you are gives a flying eff about your name, they rely on the relevant number, because _names are not unique_. _Names do not identify, they merely suggest_.

It's so bizarre. Even if Google's motivation is to uniquely identify you to advertisers, they already have that information (your unique Google account ID) and yet cannot guarantee that unique you is the same John Smith as other John Smiths in the advertising universe.

They seem to be getting nothing from this, nothing but lost good will.

EDIT: Changed money to consequences.

<comment><USER6> > _[…]because names are not unique._

Mine is. There's nobody in the entire world with the same first and last name.

<comment><USER5> Yes, some specific names are currently unique, but naming in general is not unique, so you can't rely on it to uniquely identify a population.

</comment></comment></comment><comment><USER7> They have that right, no matter how much I may disagree. But everyone I was actually talking to has been suspended or left preemptively, the organization I work for got booted, my non-tech-savvy friends and family aren't budging from Facebook, and my circles are dead. I was really excited about Google+ when it launched, but from where I sit, they've done a much better job of discouraging people from using their service than of actually building it up.

</comment><comment><USER8> From the company that advocated being able to re-invent your identity when you come of age. So obviously they recognize that there is a problem, they just don't know what to do about it.

Has anybody else been getting the feeling that the company vision has been very confused since they changed CEOs?

</comment><comment><USER9> If I wait long enough is it going to be hip again to have your own website?

You can have any name you want on your own website and run it any way you want.

<comment><USER10> That's why I never stopped having my own site, and posting EVERYTHING (even photos, bookmarks, likes, check-ins) there, and using scripts I made to propagate the info on Facebook, Twitter, etc.

It's the only way to be safe against any company shutting you down.

</comment></comment><comment><USER5> In following threads pointed to by the various +pseudonym threads, I've learned that many states allow common law naming, i.e. your name is what you say it is based merely on usage for N years.

It would be refreshing for Google to be as tolerant as government.

</comment><comment><USER11> so this doesn't apply to those famous enough to be recognized by their pseudonyms? e.g. Bob Dylan, Bono, The Edge, 50 Cent, Eminem, Dr. Dre, etc.

Sounds like selective entitlement on the part of Google.

<comment><USER12> I read it the other way around, that it very much applies to them, and thus might make it less interesting for Mr David Howell Evans and others like him to join Google +.

<comment><USER13> Hasn't stopped Lady Gaga.

</comment></comment></comment><comment><AUTHOR> There's also discussion on Google+ at [https://plus.google.com/117421021456205115327/posts/BPKvVJm7...](https://plus.google.com/117421021456205115327/posts/BPKvVJm7kiM)

<comment><USER14> There's more discussion on the official announcement: <http://news.ycombinator.com/item?id=2877442>

</comment></comment><comment><USER15> And from reading the comments here, I think people don't realize that the reason Google is insisting on real names is so they can be a central identity management and login service which Facebook is trying to do now.

</comment><comment><USER15> How did Facebook make it to now without this controversy- or did they?</comment>


8pen version 1.1 is out And the 8pen is now free - <AUTHOR> http://www.the8pen.com/changelog.html

<comment><USER2> In landscape mode, I would prefer to have the swiping area on the right/left side of the screen rather than at the center bottom (I use my dell streak mostly in landscape). It would be easier to reach as I would only be using my thumb for input.

Would be great if this could be made an option in the settings.

</comment><comment><USER3> i bet this while paid resulted in something like: 1\. download 2\. use for some hours 3\. rate one star and comment how difficult it is to lear 4\. refund

some apps really should have way more time than one day for refund. this clearly is one.</comment>


Ask HN: Alternative to Heroku that doesn't enforce 12-factor - <AUTHOR> Is there a PaaS that is as easy to scale as Heroku but does not enforce 12-factor?<p>I believe docker has changed some of the things and there is less need to enforce 12-factor for a heroku-like PaaS to work? <comment><USER2> Yes, I think the newer container services like Triton, IBM Containers, ECS, and GCE intend to be more flexible than Heroku. It will probably take some time for them to mature, especially in storage.</comment>


(╯°□°)╯︵┻━┻ - <AUTHOR> https://github.com/zeroedin-bill/flip

<comment><USER2>
        Please respect tables! ┬─┬ノ(ಠ_ಠノ)

This did give a chuckle.

Edit: I'm surprised that PHP accepts these characters ((╯°□°)╯︵┻━┻) as a valid function name. Perhaps there's a way to hack the same into JS (that would be messy).

<comment><USER3> Not verbatim. But you can do window["(╯°□°)╯︵┻━┻"] = function() { return ... }; window["(╯°□°)╯︵┻━┻"]()

<comment><USER2> Enclosing it within quotes seems like defeat.

Edit: You can assign the property to be without quotes using a bit of a hack, you just can't call it.

        var o = {}; // Creates a new object
        Object.defineProperty(o, "(╯°□°)╯︵┻━┻", {value : "Please respect tables! ┬─┬ノ(ಠ_ಠノ)"});
        //=> o = {(╯°□°)╯︵┻━┻: "Please respect tables! ┬─┬ノ(ಠ_ಠノ)"}

Trying to call o.(╯°□°)╯︵┻━┻ returns an error though.</comment></comment></comment>


The seduction secrets of video game designers - <AUTHOR> http://www.guardian.co.uk/technology/2011/may/15/video-game-design-psychology

<comment><AUTHOR> "There's an emphasis on the pleasure of choosing and planning. We've moved quite dramatically away from the action games of the 80s and 90s, where the primary mode of engagement was reaction to events."

"Again, this comes back to the central appeal of games – authority. Game stories are often pretty hokey, but they're compelling because we're in control."

"Another important game design facet is "disproportionate feedback", in which players are hugely rewarded for achieving very simple tasks."

"Conversely, designers have discovered that failure can be an important factor in keeping us entertained. "

"That loop of agency, learning and disproportionate feedback is at the heart of something very important,"</comment>


California publishes robocar "intervention" reports – Google/Waymo is far ahead - <AUTHOR> http://ideas.4brad.com/california-publishes-robocar-intervention-reports-google-waymo-so-far-ahead-its-ridiculous <comment><USER2> It's very early in the game for most of Waymo's competitors, and next year's disengagement reports will be more telling, as it will give us an indication as to the rate of progress being made by the various players in this race.

GM's Cruise appears to be on a solid trajectory.

Tesla appears to be in a state of clusterfuck. They made the committed decision to build expensive sensor suites and a PX drive 2 computer worth 'a few thousand dollars' (according to Jen-Hsun Huang) into every new Tesla. This marries Tesla to an incremental development approach that certainly good for generating excitement and engaging users near term, but the holy grail is L4, and Tesla is putting a whole bunch of unnecessary obstacles between themselves and that longer term objective.

So long as it isn't safety critical, it isn't hard to demonstrate impressive proof of concept, but what is clear by Waymo's gigantic numbers is that mopping up those last remaining edge cases and validating their system against them is actually a monstrous undertaking. Not impossible, just really hard. The closer you get to the finish line, the harder it gets.

So my guess is that whatever progress Tesla makes with HW2, they will have to set up a separate dedicated division for testing and validating a proper L4 vehicle if they wish to keep pace with the rest of the peloton. Every other big player has abandoned any notion of taking an incremental approach towards full _robotaxi-grade_ autonomy for good reasons.</comment>


The High Price of Stale Grievances - <AUTHOR> https://quillette.com/2018/06/05/high-price-stale-grievances/ <comment><USER2> Quillette is absolute gold. One of the few outlets left doing journalism and rational commentary.</comment>


Mental processes of chess masters revealed how people become experts (2016) [pdf] - <AUTHOR> http://web.cecs.pdx.edu/~tymerski/ece101/Expert_mind_scientificamerican0806-64.pdf <comment><USER2> Related - "Deliberate Practice and Performance in Music, Games, Sports, Education, and Professions: A Meta-Analysis" \- [http://scottbarrykaufman.com/wp- content/uploads/2014/07/Macn...](http://scottbarrykaufman.com/wp- content/uploads/2014/07/Macnamara-et-al.-2014.pdf)

Gives some context for the kinds of fields where deliberate practice improves performance.

<comment><USER3> I'm having a bit of trouble interpreting these numbers.

If I look at deliberate practice in musical proficiency amongst the general populace, it'll look like deliberate practice is responsible for almost all of the proficiency. However, if I look at the top 100 musicians in the world, they would have all practiced thousands of hours and I expect practice to have diminishing returns, so it should be responsible for very little of the variation in their ability.

I would expect also that in more multifactorial activities which are often the basis of professions that it is both A) difficult to ascertain what expertise is (often reducing to an empty social credential) B) difficult to evaluate what good deliberate practice is.

The meta-analysis talks about such factors as the predictability of the environment correlating with the effect size of deliberate practice. I think that's just the tip of the iceberg.

<comment><USER4> As basically a lifetime musician (and second generation) I'd like to give some counter-points to your hypothesis regarding the "diminishing returns" notion and deliberate musical practice.

I'll try to keep this light hearted - so as a guitarist, one learns the rules, and practices them, then grows and grows and grows. Eventually, one learns how to break a rule, but over time, bend it into something that resembles an existing rule or otherwise still interests an audience, and create rules whereby none existed before. Sometimes minor mistakes or oddities become wholesale innovations in the right hands (ex: Jimi Hendrix, Tom Morello) but are very much a byproduct of deliberate, routine, intense attention and practice.

My experience is that music performance has an incredible power to both be- shaped-by-and-reciprocally-shape the human brain. The inherent elasticity and plasticity cross cultures. Generations. One of the most consistent observational effects though is that practice makes better in most cases.

<comment><USER5> There's one or more quotes about this, one of them supposedly by Picasso: "Learn the rules like a pro, so you can break them like an artist."

</comment></comment></comment></comment><comment><USER6> If you want to improve this chess, I recommend this playlist by International Master John Bartholomew:

[https://www.youtube.com/watch?v=Ao9iOeK_jvU&index=5&list=PLl...](https://www.youtube.com/watch?v=Ao9iOeK_jvU&index=5&list=PLl9uuRYQ-6MBwqkmwT42l1fI7Z0bYuwwO)

I recommend [https://lichess.org](https://lichess.org) over chess.com for a far superior UI.

<comment><USER7> Lichess is excellent. I run my in person games through it's analysis. They have a great feature where you practice the correct move where you had made a mistake or "blunder".

And it's open source: [https://github.com/ornicar/lila](https://github.com/ornicar/lila)

<comment><USER6> Open source, awesome!

I'm so impressed with their UI. You can watch games in real time and there's a dashboard showing you your Ping + Server Processing Time per Move (usually 2ms).

</comment></comment></comment><comment><USER8> Fyi... this a 2006 publication. (Not sure why HN submission says (2016).)

Being 11 years old isn't an issue because the chess memory observation is interesting data. The age is significant because virtually every pop psychology & brain book in the last 10 years mentions this chess study as one of the anecdotes. (Similar to how all the pop psych books mention the Invisible Gorilla, Marshmallow Experiment, Stanford Prison Experiment, etc)

<comment><USER9> Hey want to provide some quick clarification on your point.

First, you are dramatically, 110% right that pop psychology books plunder really valuable research such as this for ultimately banal insights that they manage to hide in a 300 page book that could be 10 pages.

K. Anders Ericsson, the absolute giant in the field of expertise research, is quite dismissive of Malcolm Gladwell and upset at his mis-interpretations of his work. Go look at his book, "Peak", which goes into depth on this (and funny enough looks like every other stupid pop psychology book, although it is not).

Finally, if you want to explore the best research we have from the present through the last 40 years, highly recommend you check out "The Cambridge Handbook of Expertise". It's 40 chapters written by nothing but PhDs.

They compare what expertise means across History, Firefighting, Surgery, Ballet, even Truck Driving and Software Design.

<comment><USER10> Got any links for when K. Anders Ericsson slams Gladwell? Was it in a blog post or a podcast or something?

<comment><USER9> Just looked at the physical book. I work in DevOps but just in general want to say physical books are great and highly underrated for how quickly you can access info / other things.

[https://www.dropbox.com/s/mqmmlf1jpdexmav/Gladwell- Peak.jpeg...](https://www.dropbox.com/s/mqmmlf1jpdexmav/Gladwell- Peak.jpeg?dl=0)

Lol you'll see in the comments I even wrote "hates Gladwell!".

In fairness to Ericsson, he is very civil and it's a very respectable "takedown".

</comment></comment></comment></comment><comment><USER11> One of the books that has had the biggest influence in my life is "The Art of Learning" by Josh Waitzkin, the child chess prodigy of "Searching for Bobby Fischer" fame.

[https://www.amazon.com/Art-Learning-Journey-Optimal- Performa...](https://www.amazon.com/Art-Learning-Journey-Optimal- Performance/dp/0743277465/ref=sr_1_1?ie=UTF8&qid=1503698225&sr=8-1&keywords=art+of+learning)

<comment><USER12> My experience was the complete opposite. I genuinely do not understand why this book is recommended. It's been years since I've read it but whenever i see it recommended it irritates me. Especially as people who do never list anything of actual significance that's to do with learning, other than clichés.

<comment><USER11> You're not the first person who has told me that.

It's been ~5 years since I read it, so I can't remember too many specifics. I was ~6 months into starting to learn to program and I think I would have probably given up if I hadn't read Waitzkin's book. He talks about how certain people tend to give on something if they struggle in order to _protect_ their intelligence, to keep from feeling stupid or like a failure. It may be common sense but it's something that really clicked with me as I realized that it was something that I had done my whole life up until that point.

I kept going and eventually made a career transition into engineering. But something else happened that was actually much more important, and that was that I developed a much deeper understanding of and appreciation for the whole learning process. I was so tuned in to the whole thing that I would have these insights where I would I could pinpoint almost all of the things he talks about in the book in my personal experience. Years later, I'm able to make parallels in my present experience to all the things I experienced then, which in many cases provides some encouragement because I know I'm on the right path.

</comment></comment></comment><comment><USER13> So what kind of deliberate practice is required to become a highly skilled developer? Not just to know a language and library. Code kata used be a thing years ago.

<comment><USER14> IMO code katas are a waste of time. Practically speaking, prolific development means getting large chunks of the system done quickly, and the best way to do that is to reuse what others have done. Don't code your CSS from scratch, use Bootstrap or similar. Don't create an auth system, use the one that comes with your framework.

Counterintuitively, don't automate all the things until that automation is sure to pay off. Sometimes manually testing something is just faster, if you only need to test it a few times before the code never changes again (when was the last time you rewrote your password reset flow?)

To get prolific, don't do katas. Katas are the same problem solved in different environments. Do different problems in the environment in which you are productive. That is what will make you better and faster.

<comment><USER15> "Katas are the same problem solved in different environments."

Are you sure about that? I thought it was more solving the problem, then solving it again more efficiently / differently in the same environment. Like using recursion the second time around, not changing languages.

"Don't code your CSS from scratch, use Bootstrap or similar. Don't create an auth system, use the one that comes with your framework."

Definitely true for productivity, but a lot to be gained in experience by knowing how to roll your own.

</comment></comment></comment><comment><USER16> In the introduction I would begin telling the story of Najdorf playing blind chess: [http://www.blindfoldchess.net/blog/2011/12/after_64_years_ne...](http://www.blindfoldchess.net/blog/2011/12/after_64_years_new_world_blindfold_record_set_by_marc_lang_playing_46_games/)

<comment><USER17> Apparently he did that one also as a means to inform family in Poland that he was still alive ([http://www.poloniachess.pl/najdorf2017/index.php?page=najdor...](http://www.poloniachess.pl/najdorf2017/index.php?page=najdorf&lang=en)).

I post-war times, doing something highly newsworthy was pretty much like today making it to HN front page.

</comment></comment><comment><USER18> So...(start internal monologue)

\- The 10x engineer is so because of 10x practice (numbers not precise nor accurate). Companies - need to keep challenging people to get experts. Bootcampers - keep practicing.

\- As with the soccer study in the article, boys ended up as the "bigger kids" in computer science and due to motivational factors, there's more boys in the tech industry today [1]

\- Refutes a large portion of [2]; did the google engineer read these studies before?

[1] [http://www.npr.org/sections/money/2014/10/21/357629765/when-...](http://www.npr.org/sections/money/2014/10/21/357629765/when- women-stopped-coding)

[2] [http://gizmodo.com/exclusive-heres-the-full-10-page-anti- div...](http://gizmodo.com/exclusive-heres-the-full-10-page-anti-diversity- screed-1797564320)

</comment><comment><USER19> "Talent is Overrated" is a great pop exploration of this and similar findings. If you find this interesting, I highly recommend that book as a jumping off point for exploring similar findings.

<comment><USER20> Another very good book is "Peak: Secrets from the New Science of Expertise" by Anders Ericsson. I'm in the middle of reading it now. As I suspected (since he mentions studies on chess masters), he's part of this article, and in fact he's the author of 2 of the 5 "More to Explore" sources listed at the end of the article.

<comment><USER9> Just posted above on K. Anders Ericcson and the book Peak is great.

Main point is the 10,000 hour rule comes from Ericcson's study that the best violinists in the world practice about 7,000 hours by the time they are 18. They average between 3-4 hours a day. This last point is really interesting because the ceiling for what qualified as good practice consistently was no more than 4 hours. So if you do 4 hours of top notch practice / studying, it would seem like no matter your field you are done for the day.

Continuing on, the 10,000 hour rule was stupidly expanded by 3,000 hours just to make it easier to remember, and additionally that number is COMPLETELY ARBITRARY anyway.

The only reason violinists practice that much is because everyone else is competing that hard. Old, well defined fields like Violin or say Chess have pretty clearly defined ways on how you get better, meaning the "secrets" are somewhat known. Therefore the only way to get better is via practice, which of course everyone does, which drives up these insane numbers.

In other fields which are much newer and far less defined, expertise is up for grabs and there are no well-defined broadly agreed on ways to get better. If you do 800 hours of the "right" practice you can be far ahead of anyone else, since no one really knows what's right.

That being said, Ericcson's research still shows lots of correlation before time and skill, so alas no easy shortcuts.

EDIT: between time and skill _</comment></comment></comment>