Plot to steal cryptocurrency foiled by NPM - <AUTHOR> https://blog.npmjs.org/post/185397814280/plot-to-steal-cryptocurrency-foiled-by-the-npm <comment><USER3> NPM foils plot that was only possible due to ridiculous house of cards built by NPM.

I notice that the NPM blog post also fails to mention that it was actually _successful_ and resulted in about 1 million KMD (current value ~$1.7M) being stolen. The Komodo blog post contains that information: [https://komodoplatform.com/update-agama- vulnerability/](https://komodoplatform.com/update-agama-vulnerability/)

Even worse, they had also successfully stolen about _9 times more than that_. The only reason it was prevented was because the seeds were being sent to a public server, and Komodo was able to access and use them to "steal" the 8 million coins (and 96 BTC) from those wallets before the attacker did.

This was a successful $10M+ theft that NPM is somehow trying to spin into a positive.

<comment><USER4> I don't think it's fair to single npm out here. The truth is any package manager that lets anyone publish package updates is vulnerable to this type of attack.

<comment><USER5> npm encourages an auto-upgrading versioning scheme: take version x or higher.

It's great for keeping dependencies up-to-date, but this makes it immensely vulnerable to the kind of malware injection, as publishing a library version with payload will immediately propagate throughout the ecosystem.

In most other version managers (e.g. maven) every library and application specifies a hard-coded version, which requires users to manually upgrade, giving a window were a malicious version can be detected before it affects anyone.

<comment><USER6> For a security critical app not locking down the dependencies to audited versions seems very dangerous indeed and I believe there is some blame to be had there

<comment><USER4> You're still vulnerable if you lock down dependencies during the period the malicious code is wild though. You have to actually audit your locked dependencies to consider yourself safe.

<comment><USER7> Isn't that expecting the car to be invented before the wheel?

From what I've seen, quite a few of these have been targetted attacks - <USER2> assailants have targetted specific variables that existed in a specific codebase that they knew imports their module. Of course, if you've only just started writing that codebase, your dependencies have no way of knowing where you're storing private keys (short of somehow scanning the object space for variable names like "private_key")

But yes, in theory I would love for any *-sensitive code to go through a thorough audit after every dependency update before a version gets released in the wild. I'm guessing that next-to-nobody has the resources for that kind of an effort though.

</comment></comment></comment></comment></comment></comment><comment><USER8> "The attack was carried out by using a pattern that is becoming more and more popular; publishing a 'useful' package (electron-native-notify) to npm, waiting until it was in use by the target, and then updating it to include a malicious payload."

Gary Bernhardt has an interesting comment: [https://twitter.com/garybernhardt/status/1137459482135416832](https://twitter.com/garybernhardt/status/1137459482135416832)

<comment><USER9> re: Bernhardt's tweet —

> This is exactly what many of us predicted when NPM introduced a world of > thousands of separate dependencies per app. The main reactions to that > skepticism were "stop being so negative" and "don't try to stop progress."

I'm sure "many of us" predicted planes would crash when they first took flight, but nonetheless the benefits of commercial air travel vastly outweigh the risks.

npm, having been a huge benefit to developers and the JS ecosystem for many years, should not have its problems dismissed with an "I told you so" attitude.

<comment><USER10> The problems with the JS ecosystem probably could have been solved in a way that doesn't involve every nontrivial project dynamically downloading thousands of transitive dependencies by god-knows-who. It's not like package repositories and dependency management were uncharted waters before npm came along.

<comment><USER11> But "this one goes to eleven".

<comment><USER12> You mean it goes to 10.9999999998

</comment></comment></comment></comment></comment><comment><USER13> Here's a free idea for somebody at NPM or any other third party security provider: _Tracking changes to network or fs access across versions_

It should be possible to scan JS modules to determine either the fixed set of dependencies, the fixed set of system dependencies (ex: require('fs')), or whether it's not determinate (code contains "eval" or otherwise invokes require(...) with a non-constant). Including a package that has fs or net access would automatically include that taint as well.

The idea for this is that a malicious package would be more easily noticed as something like leftpad(...) suddenly requiring net access should be flagged. It's not a panacea as something malicious that already has fs or net access could do something new, but it would help add some sanity checks and give a smaller set of packages and versions to manually review.

I think deno (Node.js creator's new project using Typescript + Golang) has a similar idea but built in to the individual packages themselves where they need to explicitly include permissions for fs, net, etc. That'd be a great idea but getting to that point from the current (and growing) Node.js/NPM world is going to take a while.

<comment><USER14> Have a look at JSFuck: [http://www.jsfuck.com](http://www.jsfuck.com)

</comment></comment><comment><USER15> Does anyone know how this was done? Was it some sort of automatic AI system? Dedicated people watching update feeds 24/7? Random person noticing github commit and notifying npm?

<comment><USER16> Obvious way to do this would be with some variation of ranking the edit distance or simple entropy differences, and alerting/prioritizing significant updates for further scrutiny in another workflow pipeline. Bug fixes are relatively low distace, where a new module would show up as a bump. It would FP on code cleanups, but who actually does those. There are more sophisticated methods than these used in products today, but this is something an amateur could implement. Compute is huge, but cost/benefit is a product problem.

Some string searches would be easy wins as well, given wallet file names, detecting new public keys, etc. Can think of a few AV products that would benefit users by being able to state an app has added an update that resembles a new module with functionality they might not have signed up for in the original EULA.

If you are an enterprise customer, I'd ask your AV vendor for it as a feature. Realistically, the best control is don't put a cryptocurrency wallet on a device that has automatic updates from untrusted sources.

</comment></comment><comment><USER17> These issues are entirely caused by NPM in the first place. Anyone who starts a node project ends up with a massive web of unaudited code which can only be dealt with my moving off NPM and adding packages to your own private repo after youve audited them yourself. Which is an almost impossible task.

</comment><comment><USER18> I expected the article to explain how they detected it, but it's reduced to just:

> After being notified by our internal security tooling of this threat

</comment><comment><USER19> I posted this a couple of days ago, but I guess at the wrong time of day - when I saw it got no interest, I thought "I'll bet this is one of those articles that someone else posts later, that the mods let through"!

This comment is a bit meta, but I actually really like this "second chance" feature of hacker news. Mods could in theory "boost" such stories by making them sticky, but I like this kind of hands-off, no-meddling approach :)

<comment><USER20> I received an email from Dan once recommending I repost a Show HN submission I had made that had no hits. Hacker News is pretty well moderated overall.

</comment></comment><comment><USER21> I cannot wait for Entropic[1] to take off so I can just use that (and also replace my self-hosted Verdaccio with it).

[1] [https://github.com/entropic-dev/entropic](https://github.com/entropic- dev/entropic)

</comment><comment><USER22> Doesn't add up if you look at the dates they claim the malware was injected and the dates of the most recent transactions of all the addresses swept. Some of those addresses hadn't been used in over a year.

</comment><comment><USER23> Good write up and great job of NPM security team. Thanks!

</comment><comment><USER24> if you're dealing with money, you shouldn't install any dependency unless you run a security check.

you should never update a dependency unless you run a security check.

this isn't npm's fault.

<comment><USER25> I get where you're coming from, but doesn't that mean you need at least one full time security auditor just to keep pace? Node projects seem to pretty quickly hit hundreds or thousands of dependencies, which means updates happen most days.

<comment><USER26> > But doesn't that mean you need at least one full time security auditor just > to keep pace

Crazy idea - maybe you shouldn't be writing financial software if you're not going to audit the code that you're using</comment></comment></comment>


The code I'm still ashamed of - <AUTHOR> https://medium.freecodecamp.com/the-code-im-still-ashamed-of-e4c021dff55e#.yx291w59i <comment><USER2> My first job out of University was at a debt collection firm. There's nothing more surreal being in a call centre listing to 200 people on government aid, on phones collecting debts from other people who were also on government aid.

Years later I'd run into the people who worked there. Many of them told me they hated it; felt like scum.

There was a ton-of-unethical stuff that went on there ... probably why they folded and had to tell to a larger shit shop.</comment>


Memo to DOJ—Drop the Apple E-Books Suit - <AUTHOR> http://online.wsj.com/article_email/SB10001424052702303740704577527211023581798-lMyQjAxMTAyMDEwNzExNDcyWj.html <comment><USER2> We hate it when this logic - "it's not in the public interest to prosecute them, whether or not they broke the law" - is applied to big banks. Apple should be treated no differently.</comment>


"Bifinexed" Twitter account suspended following spam reports - <AUTHOR> https://www.reddit.com/r/Buttcoin/comments/7vv1gc/after_failing_to_spam_my_twitter_with_400000_fake/ <comment><USER2> Might be suspended because someone posted details of an alleged new bank account on their support forum and Bitfinexed shared a screenshot of that post on Twitter.</comment>


Reified Generics: The Search for the Cure - <AUTHOR> https://gbracha.blogspot.com/2018/10/reified-generics-search-for-cure.html <comment><USER2> I'm nowhere near qualified enough to critique this article in a meaningful manner, but I've been thinking recently that types and compile time functions go hand in hand.

Type systems are essentially an attempt to embed a logic that can prove various invariants about the code. What varies across type system to type system is A. what you can express and B. how much you can check. The issue is, most programming language designers decide these two things for you. That's why stuff like reification vs type erasure bites designers; the decision is irreversible.

But what if it wasn't? And this seems to be what this article is proposing. That if you can write compile time functions that deal with types as a construct, you can leave decisions like type erasure and reification to the programmer. Granted, that essentially forces programmers to act as language designers. And I doubt most programmers want to learn stuff like type rules or how to prove soundness.

However it does have some really cool implications. Like right now, Ruby has this whole debate over adding types. And sure, Ruby could just add normal static typing with some annotations and it'd work. But it kind of doesn't make sense for such a dynamic language to use such a static construct. What'd make more sense (to me) is some sort of metaprogramming on code that allows for checking invariants. Like being able to load a class, then run it through a checker at runtime. I've been reading a book on dependent types (The Little Typer) and the whole idea of types having types with a meta-type of sorts (universes in dependent types, meta classes in Ruby) seems very applicable to Ruby.

<comment><USER3> The main issue is that designing a good type system is a dark art.

The more powerful a type system, the easier it is to manipulate terms but the harder it is to reason about types. That's the fundamental tradeoff.

Being able to specify your own type language is like having an untyped language governing your term language.

There are promising directions though. In 20 years they might even have matured.

<comment><USER4>
       That's the fundamental tradeoff.

I'd like to refine this to: _the fundamental tradeoff is between expressivity of the typing system and computational complexity of type-inference_. Hindley- Milner's polymorphic typing system "has hit a sweetspot" (B. Pierce) between expressivity, type-inference. Almost every extension to Hindley-Milner makes full type inference undecidable.

<comment><USER3> Type inference is just one part of it. That's taking a term and asking "what type could this have?" Another thing you can do is turn it around and ask "what terms have this type?". That question also gets complicated easily.

And that's _principled_ type systems. If you put ad hoc garbage into the types it might be inferrable and whatnot but still be ridiculous to reason about.

<comment><USER4>
       what terms have this type?

This is a very interesting question, but one that is rarely asked by compilers or by programmers, or indeed by theoreticians.

I conjecture that the complexity of both questions is related, although I admit that I'd be hard-pressed even to define properly what the complexity of "what type could this have?" might be.

<comment><USER3> Oh, it's more common a question than you suggest. If I am writing a block of code and I come to some hole in context, for example `foo (bar _) baz`, and I know what type of thing has to go in that hole, then the very next thing I ask is "What term do I need there?".

This is a big question for theoreticians working on new languages. The problem is called canonicity in that context. Here is an example of such a proof of "What terms have this type [in this context]?": [https://arxiv.org/abs/1607.04156](https://arxiv.org/abs/1607.04156)

And finally, compilers _will_ ask this question if they're nice to you :) Here is GHC recently gaining the ability to ask this: [https://phabricator.haskell.org/D3361](https://phabricator.haskell.org/D3361)

The questions are related by proofs of cut elimination / normalization, which is the compatibility between a term and its context. Complex types describe more complex interactions between terms and contexts. Higher rank polymorphism has undecidable inference because a higher rank type can be used in _very_ complex contexts.

<comment><USER4> I know that this question is being asked and reseached -- indeed I do research in this direction. But it's not well understood, and it's not nearly as widely asked a question as the type-inference question which the java compiler asks every time you type something like

       int n = 2 + 3;

Note that 2+3 needs to be type-inferred.

       cut elimination / normalization,

I'm not sure this can be reduced to cut-elimination, because the question "What terms have this type?" can be asked for non-terminating programs as well -- indeed it becomes much more interesting.

       https://arxiv.org/abs/1607.04156

Aside: I'm not following HoTT closely, I guess the problem whether canonicity of the natural number type holds in HoTT in general (= can be derived from the axioms of HoTT alone) is still open?

<comment><USER3> > I'm not sure this can be reduced to cut-elimination, because the question > "What terms have this type?" can be asked for non-terminating programs as > well -- indeed it becomes much more interesting.

Oh please share some references :) I think in that context what matters is the progress and preservation parts of cut elimination.

> Aside: I'm not following HoTT closely, I guess the problem whether > canonicity of the natural number type holds in HoTT in general (= can be > derived from the axioms of HoTT alone) is still open?

I haven't either, but I think that's the case.

<comment><USER4>
       share some references

The PBE / IPS (= programming-by-example / inductive program synthesis) literature cares about this a great deal. The approach to PBE / IPS in the tradition of [1], essentially seeks a unique representative of each equivalence class of programs w.r.t. to the usual typed notion of program equivalence.

Since most interesting examples of PBE / IPS use recursion, [1] goes through a hierarchy of recursion principles, e.g. map, filter, fold, full recursion. The order of choosing recursion principles matters a great deal for controlling overfitting, the _bête noire_ of machine learning.

The basic problem is this: the answer to the question "What terms have this type?" is almost always an infinite set with a great deal of redundancy. E.g. 2+3 and 3+2 are both an answer to "What terms have type Int?". But both answers are contextually indistinguishable, so in practise we only one to see one of them. Since contexual equivalence is typically undecidable there are limits to how clever you can be in a computer program to avoid redundancies. Moreover, in PBE / IPS you typically don't want to have just values, you are interested essentially in _short_ programs inhabiting a type. So cut- eliminiation, at least as conventionally understood, doesn't quite do the job.

[1] J. K. Feser, S. Chaudhuri, I. Dillig, Synthesizing Data Structure Transformations from Input-Output Examples.

<comment><USER3> Thanks! This looks like a really interesting part of PL theory that I haven't looked at before. The problem you describe immediately reminds me of Equality Saturation; I wonder if it's been applied in this space: [http://www.cs.cornell.edu/~ross/publications/eqsat/](http://www.cs.cornell.edu/~ross/publications/eqsat/)

<comment><USER4> That's interesting, although from scanning the abstract, I don't immediately see the connection between ES and PBE.

<comment><USER3> Well the common technical problem is that of representing program fragments along with their equalities so that they can be utilized in the case I linked, and pruned in the case you suggested. It's just a musing though. I look forward to reading up on PBE in depth.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER5> Haskell has shown that you can have erased generics and have reified generics opt-in (i.e. Typeable and Generic). Works perfectly.

And the vast majority of generic Haskell code does _not_ have either of those constraints, which makes you think how important reified generics as-default really is.

<comment><USER6> > Haskell has shown that you can have erased generics and have reified > generics opt-in (i.e. Typeable and Generic). Works perfectly.

Haskell has little to no RTTI, which makes erased generic a non-issue. Java/C#/Dart have _a lot_ of RTTI, which makes erased generics problematic.

Without RTTI/reflection capabilities, generics reification become a performance tradeoff, with it they become a logic/correctness issue.

<comment><USER7> It's not just RTTI. There are some idioms in OOP languages that legitimately ought to apply to generics, but don't if you erase.

For a fairly common example, a class implementing several specializations of the same generic interface. Suppose we have Convertible<T> for something that can make a T out of itself - it would be perfectly reasonable for an object to be both Convertible<int> and Convertible<string>, but you can't do that in Java. And this doesn't require RTTI checks at call site at all - it might be that the caller is itself generic, and is just trying to accept something that's Convertible<U> for its own U.

<comment><USER8> It also happens with method overloading. In Java, you can't have a class that has both of these methods at the same time:

      Integer sum(List<Integer> values) { ... }

      float sum(List<Float> values) { ... }

because, as far as Java is concerned, both of these methods have exactly the same argument list.

You also have no way in the language to express things like "Map<String, Option<Integer>>.class", which has some ergonomic implementations. (e.g., see: [http://gafter.blogspot.com/2006/12/super-type- tokens.html](http://gafter.blogspot.com/2006/12/super-type-tokens.html))

The gotcha here, though, is that Haskell doesn't have these problems, or the one with implementing multiple specializations of the same generic type, even though it erases types much more completely than Java does.

I'm not much of an expert on language design, but that leaves me thinking that there's probably a lot of stuff we lay at the feet of Java's type erasure that is actually more properly attributable to having a type system that is broken in some fundamental way.

Or maybe it's the way that type erasure interacts with dynamic dispatch in an object-oriented language? Since the dynamic dispatch means that there are decisions about datatypes that a language like Haskell or ML can sort out at compile time, but that an OO language like Java must delay until run-time?

<comment><USER7> In this case I suspect Haskell just passes the appropriate "view" of the value, since it knows what the target type class is at any given point, and - most importantly - it can't be "downcast" later through that same value; you can't do a _runtime_ check for whether a given Convertible<T> is also Convertible<U> (either explicitly, or observable via dynamic dispatch). In Java, it's possible, so even if the code never actually does that anywhere, it always has to be pessimistic and assume that someone somewhere might.

But I'm not sure that truly counts as type erasure. If that's how things are done, then the type class of every such value is effectively encoded by the set of function values that are passed around, and could be queried at runtime - the Java equivalent would be to tag every reference with its class, including all generic parameters used to instantiate it.

</comment></comment></comment></comment></comment><comment><USER9> I haven't seen anyone mention it yet, so I'll just point out that the kind of userland reification described in this article is also possible in Scala using implicits:

        class List[T](implicit tag: ClassTag[T]) {
          def elementType: Class[_] = tag.runtimeClass
        }

You can do something similar for your own types in Java, albeit with less help from the compiler:

        public class List<T> {
            private final Class<T> t;

            public List(final Class<T> t) {
                this.t = t;
            }

            public Class<T> elementType() {
                return this.t;
            }
        }

The problem is that you rarely own the definition of all the types you want/need to reify.

</comment><comment><USER10> It's not directly related, but this reminds me a bit of Zig's treatment of compile-time parameters [1], which seems somewhat more natural than generic parameters in most languages.

Zig doesn't go all the way, though. Instead, a type parameter could be any combination of a compile-time and runtime parameter, depending how it's used.

I do wonder how that affects calling conventions, though? If have separate linking then we probably need to avoid changing the calling convention if a function is modified so that a parameter is used in a different way.

[1] [https://andrewkelley.me/post/zig-programming-language- blurs-...](https://andrewkelley.me/post/zig-programming-language-blurs-line- compile-time-run-time.html#compile-time-parameters)

</comment><comment><USER11> So I totally buy the argument that you can reify your own generics with a system like this. It's nice, I like it.

...but I don't see how this feeds into the compiler without a "shadow type system"[1]. If you have to specify the types everywhere, or don't have compile-time checks, it's not really generics. It's just an Object-container that has validation funcs executed at runtime, which can already be done relatively easily in basically every language. Did I miss something?

[1]: or compile-time code generation, but that isn't mentioned. and that strikes me more as a macro system than a type system (though I don't know if that's a real distinction, if your macros are sufficiently powerful™)

<comment><USER12> > If you have to specify the types everywhere, or don't have compile-time > checks, it's not really generics.

Specify which types everywhere? If you mean specifying the concrete types (e.g. `Integer` and `List[Integer]`, then I'd agree. If you mean type variables (e.g. `T` and `List[T]`) then I disagree: that's a perfectly valid way to do generics, and having to annotate stuff is just a deficiency of inference (no worse than, say, Java).

The key is that we're allowed to use the argument names and function/method names in our types, thanks to the 2 given rules. We don't need to lift those functions up to the type level, since we're only checking that the inputs and outputs line up.

To me, this seems to be a form of dependent types. Note that "dependent types" only refers to the ability of having types refer to values, e.g. dependent function types like `(n : Int) -> Foo n` or dependent pairs like `(n : Int, Bar n)`. Most dependently typed languages _also_ treat types as first-class language constructs, allow the type-checker to run arbitrary code (which usually implies that computations are pure and everything terminates/coterminates). Those latter aspects don't seem to apply here: it's just about using value-level names in our types.

I think this works quite nicely, from a theoretical point of view; but I wonder what would happen in more complicated situations. When type checking is purely syntactic, we can easily get stuck with mismatched things like "should have type 'T', got type 'Pair[T, U].first()'". That's where we need type-level computation, to reduce things down to normal forms so that the syntactic checks (are more likely to) work.

<comment><USER11> > _If you mean specifying the concrete types (e.g. `Integer` and > `List[Integer]`, then I 'd agree._

Yep, that's exactly what I mean (since that's the only example in the article that seems to imply any kind of compile-time safety).

re dependent types: interesting! yeah, it does seem like normal generics could be a type-dependent type in a case like this - I hadn't really thought about that before. though that also seems like a "shadow type" tho, which the article is trying to eliminate... so we're back at the starting point :|

<comment><USER12> > that's the only example in the article that seems to imply any kind of > compile-time safety

The type-checked example is this:

        var lst : List[Integer] := List(Integer).new();
        var i : Integer := lst.add(3) + 4;

To me, this isn't so different from something like Java:

        List<Integer> lst = new List();
        Integer i = lst.add(3) + 4;

The major difference is the extra argument in `List(Integer)` (compared to `List()`). That's kind of unfortunate, but note that it's a _value_ , not an annotation. It's still generic, since we could just as easily pass in some variable (e.g. `T2`) that we've received as an argument, or pulled out of some other method, etc.

This extra argument (a run-time representation of the type) is actually to be expected. Values (including functions) which are generic (in the Java sense), also called "parametrically polymorphic", or as you say "type-dependent types", can be understood as taking those parameters as extra arguments (which is precisely what this article is talking about); although the value itself _cannot_ change its behaviour depending on that argument. For example, `List<T> reverse(List<T> input)` cannot work different for, say, `List<Boolean>` compared to `List<Integer>`.

This is really clear in Haskell. The Glasgow Haskell Compiler uses an intermediate language called "GHC Core", where generic values are literally represented in the same way as functions. There's a nice overview at [https://alpmestan.com/posts/2013-06-27-ghc-core-by- example-e...](https://alpmestan.com/posts/2013-06-27-ghc-core-by-example- episode-1.html) (the "id'" example is the first generic one).

> it does seem like normal generics could be a type-dependent type in a case > like this

Generics are definitely type-dependent types. However, when I said "dependent types" I was referring specifically to value-dependent types, e.g. [https://en.wikipedia.org/wiki/Dependent_type](https://en.wikipedia.org/wiki/Dependent_type)

Note in that Wikipedia article's introduction "dependent types are used to encode logic's quantifiers like 'for all'". Compare that to the generic example in that GHC Core link above, where the keyword "forall" gets inserted by the compiler ;)

The Wikipedia article also states "Deciding the equality of dependent types in a program may require computations". That's what I was getting at with the system described in this article: it doesn't seem to allow for type-level computations, without which we may end up getting errors that "expected type X, given type Y" when those types are 'obviously' the same (to us, but not the the checker, since it's not able to compute).

I know Bracha is a fan of 'pluggable type systems', where we can use a variety of type systems to check things about the same program (perhaps requiring different annotations). However, I'm not sure if these 2 rules are enough to perform meaningful reductions on types (e.g. to spot that `Pair(T1, T2).secondType()` is equal to `T2`). I'd imagine we'd have to "lift" more and more expressions into the type level (e.g. `return`, then `if/then/else`, and so on) until we have a full-blown theorem prover like Agda!

<comment><USER12> Reading this back, I'd also suggest a comparison with Hindley-Milner type systems ( [https://en.wikipedia.org/wiki/Hindley%E2%80%93Milner_type_sy...](https://en.wikipedia.org/wiki/Hindley%E2%80%93Milner_type_system) )

These type systems are used by ML and its descendents (OCaml, Haskell, Rust, Reason, etc.) and are generally seen as a "sweet spot". They feature:

\- Types depending on types (AKA type constructors)

\- Values depending on types (e.g. type classes)

\- Type-generic values (AKA parametric polymorphism)

\- Higher-kinded types (AKA higher-order types, or generics at the type-level)

\- Inference of all types

\- Erasure of all types

Things which Hindley-Milner _can 't_ do include:

\- Inference of values (AKA implicit arguments, as found in Scala and Idris)

\- Erasure of values

\- Types which depend on values (AKA dependent types)

These things are important for this article, since it represents types as values (which the two given rules let us access from our type annotations). In particular, if we could infer (some) values, we could avoid having to specify arguments like in `List(Integer)`.

Going "beyond" Hindley-Milner is a very active area of research. There's a nice talk about this at [https://www.reddit.com/r/haskell/comments/390dyx/conor_mcbri...](https://www.reddit.com/r/haskell/comments/390dyx/conor_mcbride_type_inference_needs_revolution/) (which i admit is a bit symbol-heavy); full disclosure, I was in the audience for that talk ;)

<comment><USER11> Thanks for all the info! I definitely do want to learn more about this kind of thing, it's always interesting and I'm always glad for more links :)

To circle all the way back around tho, from the intro to the article:

> _A while back, I discussed the problem of shadow language constructs. I gave > examples of shadow constructs such as Standard ML modules, traditional > imports etc. Here is another: reified generics._

The article seems to be about _trying to eliminate_ shadow types, e.g. meta- type info carried by the compiler but not otherwise visible. By turning generics into type-generator funcs. My question / confusion is around "it seems like you can't type-check that at compile-time without being either so explicit it's no longer generic, or with shadow types, so I don't see how this helps". Or indeed changes anything.

Generics as type-generating funcs makes sense, it lets you choose erasure-or- not however you like. Perfectly on board[1]. But how do you type-check it at compile-time without "shadow types"? (if the answer is "read my comments", I'm happy to! I haven't gone through them super carefully yet.)

[1]: well... in concept. I'm not sure how libraries would be able to use this in a standard way, seems like it'd lead to multiple competing "generics systems". But that's also useful sometimes. I like languages that let you do that kind of thing, though the ecosystem does not always benefit from it.

<comment><USER12> > Thanks for all the info! I definitely do want to learn more about this kind > of thing, it's always interesting and I'm always glad for more links :)

No problem. Most of my blog posts start out as rambling HN comments ;)

> how do you type-check it at compile-time without "shadow types"?

From my understanding of the article, the 2 rules allow type checkers to refer to the value names. To me, the key parts of the (typed) example are these lines:

        public var List = (function(T : Class) {
          ...
            public datum : T;
            ...
            public add:(e : T) : T {
            ...

In the expression (type annotation) `T : Class`, we're introducing a function argument called `T`. `T` refers to a _value_ , which we can tell because it's on the left-hand-side of the type annotation (`Class` is on the right-hand- side, so we know `Class` is a type).

In the other two lines, `T` is being used as a _type_ (again, since it's on the right-hand-side of a type annotation). This is what the first "rule" lets us do.

As far as I can tell, this (along with the second "rule", which lets us refer to method names in our types) avoids the need for "shadow types" (or a "shadow language" for types), since we only introduce/define/pass-around things using the normal value-level machinery (functions, methods, classes, etc.), and our type annotations simply _refer to_ those value-level names (we don't re- implement functions, methods, classes, etc. for types).

My concern is that "rule 2" lets us write things like:

        function pickAType(T1 : Class, T2 : Class) : Class {
          if (canProve(collatzConjecture)) {
            return T1;
          else {
            return T2;
          }
        }

        var x : pickAType(Int, String) = 42;

This is well-typed if and only if the Collatz conjecture is provable (e.g. by enumerating all proofs in some formal logic and checking whether any is a proof or disproof of collatzConjecture). If it's provable, then `pickAType(Int, String)` reduces to `Int`, which matches the type of `42`. If it's not provable then `pickAType(Int, String)` reduces to `String`, which doesn't match the type of `42`.

This is just an example: we can replace "prove Collatz conjecture" with any arbitrarily-difficult calculation. The point is that _checking_ these types requires _running_ the function which, in this case, requires the type checker to understand `if/then/else`, functions like `canProve` which don't fit the constraints of "rule 2" (that inputs and output should be `Class`); presumably `canProve` will use things like recursion, and `collatzConjecture` will be some sort of datastructure, which a type checker would _also_ need to understand. In short, we'd need an awful lot more rules than just the 2 in the article. The straightforward way to do this is to just have our type checker just invoke the normal language interpreter, but that raises more issues.

For example, side-effects: what if `pickAType` chooses its output based on, say, reading a random number from the OS kernel? It seems unwise to allow side-effects when type-checking (except maybe for "type providers" [https://docs.microsoft.com/en- us/dotnet/fsharp/tutorials/typ...](https://docs.microsoft.com/en- us/dotnet/fsharp/tutorials/type-providers) ). However, if we're running arbitrary value-level code as part of the type-checking process (e.g. by invoking the normal interpreter), that would require the value-level language to _also_ be free of side-effects (or else we would need an alternative interpreter, which raises the "shadow language" problem again).

In particular, object oriented systems, especially those like Smalltalk, Self and the author's Newspeak, model function calls, property lookup, value construction, method calls, etc. as 'sending messages between objects', which is an I/O effect that can have all sorts of repercussions for the meaning of other code. For example, imagine the method call `True become: False` ( [https://gbracha.blogspot.com/2009/07/miracle-of- become.html](https://gbracha.blogspot.com/2009/07/miracle-of-become.html) ; or in Newspeak, where booleans are immutable "value objects", some equivalent behaviour-changing call). Even in a system with no I/O, I think this would make types non-compositional, i.e. if the expression `f` type-checks to type `A -> B` and `x` type-checks to type `A`, the expression `f(x)` might not type-check, and even if it does it might not have type `B`, since the code run while type-checking one sub-expression may alter the behaviour of code run while type-checking the other.

Note that these aren't insurmountable problems: we could approach it like most other things in dynamic languages and say "yes it can be broken by doing that; so don't do that".

Alternatively, perhaps more in keeping with Bracha's ideas of pluggable type systems, we might be OK with type systems that are limited (e.g. unable to reduce `pickAType(Int, String)` to either `Int` or `String`), or uncomputable (if we _do_ run `pickAType`), or even containing "shadow" interpreters which may, for example, over- or under-approximate types (e.g. allowing `42` since it _might_ have type `pickAType(Int, String)`, or disallowing it because it _might not_ ), or which implement some alternative evaluation method or effect system (e.g. calling out to SMT solvers or proof assistants, mocking I/O calls, etc.). If they're pluggable, then we're free to ignore those which aren't appropriate for our project (based on how much confidence we want in the result, whether we'll be doing shenanigans like `pickAType`, etc.)

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> This sounds very academic. Meantime in the real world, people in Java-land are constantly killing themselves trying to invent workarounds for type erasure induced problems. The latest example I know of being Micronaut [1] which touts itself in some sense as being a whole platform for re-reification.

[1] [https://objectcomputing.com/news/2018/10/08/micronaut-10-rc2](https://objectcomputing.com/news/2018/10/08/micronaut-10-rc2)

<comment><USER14> This is because the decision to type erase was very weird for a strong static typed language...

You inject weak typing (unexpected conversations) into a strong type system by doing that.

<comment><USER15> "Weird" is a strong word. Once the compiler has proven the codes satisfies certain type constraints you usually don't need the type information anymore ... except when you do of course.</comment></comment></comment>


The Unofficial Hacker News FAQ - <AUTHOR> https://jacquesmattheij.com/the-unofficial-hn-faq <comment><USER2> > Though it shouldn't be the case, but there definitely seems to be some > evidence for it disagreeing with prominent HN'ers that have a large number > of fans is another way to get downvoted, that shouldn't happen but there is > definitely evidence for it.

This makes me sad. I have experienced that... so I just stopped placing such comments here. The toxicity of the community is just too high here. You can either agree with the crowd, or be forced to leave. Your experiences or arguments don't matter.

<comment><USER3> I have the same feeling regarding the meta group on Stack Overflow (and some other prominent ones).

There is so much hatred, looking down on people who ask a question that I decided to never get back there anymore (and the two bugs I found are lost). This is truly a toxic environment (and apparently happy with that).

What I find amazing is that many of the people who behave like 4 yo playing in the sand when in meta give fantastic answers on the regular site.

</comment></comment><comment><USER4> This appears to have been posted in response to my list of undocumented Hacker News features ([https://news.ycombinator.com/item?id=16437973](https://news.ycombinator.com/item?id=16437973)).

This FAQ is from 2011 and reading through it, I'm not sure how much is applicable in 2018, or not otherwise covered in the current FAQ/Guidelines. Hacker News has changed a lot since then. (particularly after Paul Graham stepped down)

</comment><comment><USER5> This is from 2011 and contains broken links (in section "How do I know if my comment has been replied to") and links to (now) malicous sites (in section "how do I search Hacker News?")</comment>


12 hour battery life in a high-end laptop? Asus says yes - <AUTHOR> http://arstechnica.com/gadgets/news/2010/01/asus-laptop-makes-intel-nvidia-gpus-work-together.ars

<comment><USER2> Laptops are becoming potentially useful as servers, especially considering that they have what amounts to a built-in UPS that can run for up to 12 hours (in this case), without the problems inherent in a conventional UPS (price, single point of failure, etc.). ACPI even offers flexible event-driven power management to protect your data. The only problem is that I haven't found a laptop that offers server-oriented options in the BIOS. For example, I would like to tell the laptop to hibernate when power is disconnected and battery is dangerously low, then wake when AC power is restored. Extra points for providing a sleep stage before hibernation, depending on battery levels. I'm not a big fan of WOL for this purpose.

<comment><USER3> > For example, I would like to tell the laptop to hibernate when power is > disconnected and battery is dangerously low, then wake when AC power is > restored. Extra points for providing a sleep stage before hibernation, > depending on battery levels.

I'm confused. Unless I misunderstand you, this is exactly how my MacBook Pro behaves—as the battery drains, it warns me with an icon in the menu bar when the power is low (less than 1/4 charge, I think), then with a pop-up dialogue at 10 minutes, then hibernates. If I plug it in, it wakes back up effectively instantly. (It doesn't offer the sleep period by default, but I think it can be configured to do so.) Is this what you're looking for, or is it something more subtle?

<comment><USER2> My MacBook Pro is in the red at this very moment. Let me test this out...

<comment><USER2> Well, the MacBook hibernated (and the Windows XP VM running in Fusion even went into standby first), but reconnecting the power supply didn't bring it out of hibernation, which is the behaviour I want. I had to hit a key to wake it up. I had ruled out using a MacBook as a server, mainly because I want to operate it with the lid closed most of the time, which might cause it to overheat (an issue shared by some models of other brands). If I can find a way to run Linux, keep the lid closed, and bring the laptop out of hibernation when power is restored, I'll be happy.

<comment><USER3> Oh, sorry, you're right! Indeed I was missing the bit about having to hit the power key. In Leopard, under Energy Saver, there's a selection "Wake for Ethernet network administrator access"; will that provide what you want?

(Actually, I'm a little confused: If you're physically present to plug in the machine, what's the problem with pressing the power button? Is it just that you don't want to open it?)

By the way, according to this article ([http://forums.macnn.com/69/mac- notebooks/359432/how-get-your...](http://forums.macnn.com/69/mac- notebooks/359432/how-get-your-mb-p-run)), which links to this Apple KB article (<http://support.apple.com/kb/HT3131>), it is perfectly OK to run the MacBook Pro with the lid closed. Have you had trouble when you actually do so?

<comment><USER2> Plugging in the machine was merely to simulate power restoration (a more accurate test would be to leave the DC connector attached to the laptop and cut off power to the AC plug, which sometimes creates different behaviour). I want a server to come back online unattended, so I can enjoy my vacations. :)

I've only read that MacBooks overheat with the lids closed (I definitely feel the warm air being expelled near the hinge, and the area surrounding the track pad can get pretty hot), but if that's not the case, I might reconsider using them as servers.

</comment></comment></comment></comment></comment></comment><comment><USER4> I think laptop makers are going in the right direction with long battery life. I have an Acer Aspire Timeline ( <http://www.acer.com/timeline/> ) and I usually get 8.30 - 9hrs of life just browsing. I love the battery life and it has completely changed how I work with a laptop.

</comment><comment><USER5> Any company who can build a laptop whose batteries last all day is going to make a lot of money, because it frees users from the constant worry that they're going to run out of battery power soon, and would enable genuinely mobile working. No longer would you always need to do your work somewhere in the vicinity of a mains socket.

<comment><USER6> Agreed. My gf has the Nokia Booklet, and its 11 hour battery life really is sweet. She charges it at night (some nights), and then can just use it all day without any worries. The downside is that a 1280x720 display just doesn't show all that much data. My Sony gets about 7 hours of battery life, which isn't quite enough to get through the day, but its 1600x900 resolution is actually enough to get some work done. I'm sure looking forward to the end of that compromise.

<comment><USER3> > My Sony gets about 7 hours of battery life, which isn't quite enough to get > through the day, but its 1600x900 resolution is actually enough to get some > work done.

What model of Sony is this? A big factor in my current laptop shopping is battery life, but I didn't know that we were beyond the 6–7 hour manufacturer figures now.

<comment><USER6> The vaio z series (mine is the z590, but I think the current one is z610 or something) have an optional larger battery that gives a claimed 9 hours of battery life. If I shut off all radios, dim the screen, and work in terminal mode, it can even do that, but for honest usage with wifi enabled, it does get about 7 hours.

The battery sticks out the bottom, but it's actually really nice because it tilts the keyboard nicely, and it gives a nice thing to hold onto when carrying the laptop :)

<comment><USER3> Thanks! I just went to their website and checked out the F Series (although I couldn't find any way to customise it). I found it very funny that, in addition to the standard choices like a mouse, keyboard, or spare battery, one of the 'accessories' that it prompted me to buy was a netbook! (I know that they're commodity items, but not _that_ much so.)

</comment></comment></comment></comment></comment><comment><USER7> The i7 is incorrectly reported to be a dual core in the article, it is actually a quad core with hyperthreading, so it looks like it has 8 cores to the os.

<comment><USER8> There are dual-core i7s with hyperthreading as well:

[http://en.wikipedia.org/wiki/List_of_Intel_Core_i7_microproc...](http://en.wikipedia.org/wiki/List_of_Intel_Core_i7_microprocessors#Dual- Core_Mobile_Processors)

</comment></comment><comment><USER9> For the record, Asus already has a earlier version of this laptop (without an i7) for $800 available on amazon (search for UL80VT). It has the same form factor, battery life, and the video cards perform almost exactly the same.

</comment><comment><USER10> Battery life is the most overrated of all laptop specs. How often are you really without an electric outlet? Other than an airplane or the occasional wifi-capable coffee shop (one without outlets, that is), I can't think of anywhere I'd be likely to be using my laptop. It's really rarer that you might assume.

<comment><USER11> if the history of capitalism proves anything it's that people will pay far more for convenience than anyone ever suspects.

<comment><USER10> If we can change "convenience" to "what they perceive to be convenience"...then sure, totally agreed.

<comment><USER12> If it feels convenient it is convenient.</comment></comment></comment></comment>


Ask HN: Why aren't the price of online services based on user location? - <AUTHOR> I am a software developer in Turkey. Salaries in Turkey are lower than USA but we have to pay the same as US citizen for online services or international books. For example, in Turkey standards, $40 for a book or $500 for an coursera specializations is so expensive.<p>What do you think about this situation? <comment><USER2> Please define _online services_. Renting a VPS has a fixed cost associated with it regardless of your location, so the hosting provider can't do much to differentiate prices based on the customer's location.

You are probably referring to digital products that have an initial development cost, but no or negligible transport and manufacturing costs. Buying a digital game on Steam is a good example of this (this goes for films, ebooks, and music as well).

Game prices on Steam can be based on the GDP of your country, because if they used the same prices as in Western Europe or the US everywhere, a lot of people would not be able to afford the games. For a game developer on Steam a lost sale means less profit, whereas a sale, _any sale_ , means profit — even if the price was heavily discounted — at no, or almost no, additional cost for the service rendered. Not everyone does this though, because that is how a free economy works. If a game studio thinks they can sell more games in Turkey by lowering prices there, they will probably do it.

On the other hand, if someone is selling an ebook that no one else sells, and they know that you are willing to pay the full price even though it costs you more (relatively speaking), then why would they lower the price? They are a business, not a philanthropy.

<comment><AUTHOR> Most of the books about software development in amazon can be downloaded from libgen.io. Many course in pluralsight can be downloaded via torrent. Almost every programs for windows are hacked. I accept those are not ethical but unfortunately most of the people in countries like Turkey don't pay for the products. I think those are not affordable and it increase s the use of pirate products.

</comment></comment><comment><USER3> There are two ways to think about the price of something: cost and value. If the services or good don't provide enough value to Turkish people, they won't buy them!

I'd say you are right to point out that it costs more-or-less the same to deliver digital services like Coursera. But perhaps because the content was largely produced in the US (I'm assuming it is) their costs align with US wages and costs.</comment>


Ubiquitous new workplace surveillance tools to accompany office reopenings - <AUTHOR> https://www.wsj.com/articles/lockdown-reopen-office-coronavirus-privacy-11588689725 <comment><AUTHOR> [https://archive.vn/vraQv](https://archive.vn/vraQv)</comment>


Envoy: Elegantly greet visitors to your office - <AUTHOR> https://signwithenvoy.com

<comment><USER2> "Your lobby is the first thing visitors see when arriving at your office. It's a reflection of your brand and culture. Make sure they don't deal with a human being"

</comment><comment><USER3> It's so exciting that going to a meeting will soon be like going to my doctors office, filling out tons of forms and I can spend more time poking at a little screen.

Technology can be super useful (obviously), but this is just total overkill.

<comment><USER4> I for one am looking forward to a day where strangers don't have to see my awful handwriting. Although the thought of touching a tablet that all of the infected patients before me have touched might keep me to the pen and paper.

</comment></comment><comment><USER5> I'm not a fan of this idea, that said I've seen it done right and done poorly.

Done poorly - Google - Typing on a screen to get your badge is always a slow, painful process. Trying to have an event with several people who all have to sign in demonstrates how much of a painful bottleneck this can be.

Done well - Hacker Dojo - Multiple kiosks with real keyboards. You can sign in and (at some events) get a badge super quickly.

The code for the hacker dojo kiosks is open source and available here ([https://github.com/hackerdojo](https://github.com/hackerdojo)) along with a bunch of other useful tools.

</comment><comment><USER6> What is going on? This is the third Envoy post to hit the front page immediately after posting and then go nowhere in the past couple of hours.

[https://news.ycombinator.com/item?id=8325634](https://news.ycombinator.com/item?id=8325634)

[https://news.ycombinator.com/item?id=8325701](https://news.ycombinator.com/item?id=8325701)

<comment><AUTHOR> Weird. The URL on the second link is identical to the one I submitted. Wonder why the HN link de-duper didn't catch it.

<comment><USER7> Not quite identical. Trailing slash.

We're working on a new de-duping system. The current one is deliberately left porous so that reposts of good stories that didn't get attention won't be killed as dupes.

<comment><AUTHOR> They both look like they have a trailing slash to me.

<comment><USER7> Huh. At our end, 8325701 has a trailing slash and this one does not.

<comment><USER8> I am seeing the opposite--701 has no trailing slash, 34 does have a trailing slash.

</comment></comment></comment></comment></comment></comment><comment><USER9> Why not just connect to their Google calender and pull it ahead of time</comment>


An Aspiring Scientist's Frustration with Modern-Day Academia (2013) - <AUTHOR> http://crypto.junod.info/2013/09/09/an-aspiring-scientists-frustration-with-modern-day-academia-a-resignation/ <comment><USER2> 90% of everything is crap. And there's no guarantee that the other 10% isn't crap as well. I get it, we need the social dialogue, so that people know how and why it's crap, but really the only way forward that doesn't result in eternal loss of control of your efforts to sociopaths with no interest in your happiness or society's well-being is to blaze your own path forward.

Academia is crap. Most jobs are crap. Most people do not make any dents in the universe. Most relationships end.

One should learn to recognize situations where irrationality is the norm. The easy one to spot is music. Way more people want to be musicians than the market for their output can bear. These market realities will dominate your life if you decide to become a musician. Doesn't mean you can't make it, just means that if you want a semblance of control over whether you'll make it or not, you have to create that control yourself and you'll have nobody to help you that can really help you because they have to deal with the market realities too. Even if you do make it you'll be similarly powerless to change the realities.

Once you understand the dynamic you will immediately see the pattern everywhere. Film. Academia. Comedy. The Apple App Store. The dating market for very attractive women. People who want me to work on their websites or otherwise hire me. Silicon Valley. You have to deal with all the other assholes in the world who want the same thing you want, and there's only so much of it to go around. The ones that succeed learn how to increase the demand for the services they want to provide. The ones that don't bitch about how unfair everything is.

<comment><USER3> Companies live or die by the value that they create for their customers. Films win or lose by the entertainment they provide. Musicians get rich or stay poor by the number of fans who enjoy their music etc. The post is saying that people in academia (Except those at the top institutions) are not judged by the value that they create for science and that this gets in the way of good science in many institutions. Science is not the kind of market where the most beneficial outcomes are automatically selected and the author is upset with how incentives and low odds of success are aligned to make his life in science anxious and unfulfilling. One can scold the author for complaining or one can take him seriously on his points about how society should better benefit from the money it spends on science.

<comment><USER4> Your examples are about popularity, whereas with science you switch to value. These are far from equivalent. Films win or lose by the entertainment they provide, but do the best ones win? No, rather the ones that can convince the most people to watch them win. It's quite possible that an unpopular movie _would have_ provided a lot more entertainment if more people had gone to see it, but since they didn't, it loses. Likewise with music: just because some musician attracted a lot of fans doesn't mean those fans wouldn't enjoy some obscure musician more, they just never really made the choice.

I think it's useful to ask whether science is actually different in this respect, or whether this is merely an aspect of a universal problem.

</comment></comment></comment><comment><USER5> This part gets best at why I jumped ship last year, at the end of my math postdoc:

"(4) Academia: Where Originality Will Hurt You "The good, healthy mentality would naturally be to work on research that we believe is important. Unfortunately, most such research is challenging and difficult to publish, and the current publish-or-perish system makes it difficult to put bread on the table while working on problems that require at least ten years of labor before you can report even the most preliminary results. [...] "Ideally, the academic system would encourage those people who are already well established and trusted to pursue these challenges, and I'm sure that some already do. However, I cannot help but get the impression that the majority of us are avoiding the real issues and pursuing minor, easy problems that we know can be solved and published. The result is a gigantic literature full of marginal/repetitive contributions. This, however, is not necessarily a bad thing if it's a good CV that you're after."

I came away with the impression that there are a few people given extremely wide latitude to work (or not) on problems of real significance, who had generally been granted 'rock star' status by the math community in one way or another. It's a 1% whose academic freedoms serves as a motivation for the other 99%, who will try to write a bulk of irrelevant papers while dealing with a massive teaching load. A job in industry ultimately felt like it might provide more freedom in the long run than the academic track, with less danger of ending up in a adjunct teaching gig in Wichita.

<comment><USER6> >A job in industry ultimately felt like it might provide more freedom in the long run than the academic track [...]

Does having a full-time industry job allow you to seriously pursue your own research interests on the side?

For mathematics research especially, why don't more research-oriented mathematicians pursue this dual path? I understand part-time research necessarily implies a slower publication rate, but since you're already eschewing this anyway, are there any other drawbacks? Two that spring to mind are a restriction to travel to mathematics conferences and socializing with mathematicians during the day -- are these big show stoppers?

<comment><USER7> I've actually been playing with the idea of a "monastery" for mathematicians; we could get a group of mathematicians together to live simply and cheaply somewhere, subsisting on a small amount of savings and freeing up time to study and solve problems.

<comment><USER8> I thought about something similar, but more like a "monastery-style research institute" where people, with some amount of provable expertise, regardless of credentials or previous publications, would be able to check-in for pre- determined period of time and just do "open ended research" (the kind of things no sane university or corporation would ever pay for...) and have living expenses plus other basic needs provided, and spared from having to waste energy in the meaningless and draining "dance" we call "social life", for with the sole condition that their research is "open source" and that they are actually staying there working, not just using it as a free/cheap rent.

The "monastery" could also be run as a business, having a for-profit section developing products based on the ideas produced inside. Something like the members of the monastery signing an agreement that they are ok with their results being published with one year delay, time that would give an edge to the monastery over any competitors that would also want to develop products based on them - think of it like a "research monastery self-financed by coupling with a sort of startup incubator" (with some really intransigent and incorruptible people working to keep the two camps of people apart and to stop the "business idea developers" from exploiting the "monks").

And for research fields that require some expensive equipment, like bio- medical research, the "monastery" may make the "monks" sign contracts giving the patents and other right to their discoveries to the monastery, in exchange for having access to a lab where they can work on anything they want. It's not that utopic since you can a pretty usable top notch molecular biology lab plus an animal experiments facility for under $1M (probably not in the US because you have too many regulations and stuff...) if you find alternative channels of buying equipment of user other "hacks" that regular research institutes can't use...

But it's the kind of idea that it's hard to convince other people it would make sense... so for now I'll keep it on my list of to-dos for after I get filthy rich so I can finance this myself.

<comment><USER9> That sounds a great deal like what Universities are supposed to be. (Or did I miss the funny?)

</comment></comment></comment></comment></comment><comment><USER10>
      "I sometimes find it both funny and frightening that the majority
      of the world's academic research is actually being done by people
      like me, who don't even have a PhD degree. Many advisors, whom you
      would expect to truly be pushing science forward with their decades
      of experience, do surprisingly little and only appear to manage the
      PhD students, who slave away on papers that their advisors then put
      their names on as a sort of "fee" for having taken the time to read
      the document..."

That's a bit like saying: "What's so great about Henry Ford? He didn't _build_ the damn cars."

It appears that the author's definition of "true science" is akin to knowledge-that-needn't-be-effectively-communicated-to-anyone, which, in my estimation, is not science at all.

<comment><USER11> A lot of labs are very much like this... with the actual science work getting done by the post-grads.

What the author doesn't realise that while the lab heads might not be doing a lot of science on a day to day basis - most are overloaded with a bunch of other stuff that is forced upon them... endless committee meetings, grant writing, teaching and teaching administration etc...

<comment><USER12> Can't all of that, except teaching, be done by a secretary?

<comment><USER13> That'd require a secretary with a PhD. Looks more like most of it should not be done. But then, I have no idea how to change it.

Teaching is probably the easiest one to solve. Some kinds of classes should be attributed to teachers, not researchers (those are very different kinds of expertize).

<comment><USER14> It is increasingly common for large labs in the life sciences (25+ members) to hire multiple support staff (with PhDs) to help offset this load. This is in addition to positions like Senior Research Scientists (senior postdocs) who often serve as 50% "project managers" and 50% independent postdocs.

</comment></comment></comment></comment></comment><comment><USER15> In the comments, Lee Smolin says something I agree with:

 _A final word: where ever you go if you leave science you are likely to face the same fight, because it is in the nature of modern life and modern organizations._

I've thought before that much of the criticism of academia (which I've certainly engaged in) mostly reflects disappointment that it was ultimately not all that different from the rest of the world.

<comment><USER16> My grad school stipend was ~$500 every two weeks pre-tax. I worked like crazy. The football coach made ~250k. I now make over 5 times that.. after taxes.

When I left grad school I found many of it's problems present in the corporate world; except that now, compensation is commiserate with responsibility.

<comment><USER17> Commensurate, not commiserate.

You commiserate (sympathize) with the author's plight. And now your compensation is now commensurate (measurably equal) with your responsibility.

<comment><USER18> During my stint in the corporate world, my compensation was also commiserate with my responsibility. How could you tell GP wasn't being both ironic and Freudian?

<comment><USER17> Because I didn't think he was adjectiving a verb. :)

</comment></comment></comment></comment></comment><comment><USER19> As a current CS PhD, I feel there are elements of truth in the letter, on the whole, it's a bit too dismissive of the value of academia.

Yes, a lot of research in academia ends up going nowhere, but that's not always obvious beforehand. Research is an inherently chancy endeavour with a low chance of success. But academia is still a source of important research. To name just two examples, much of the major deep learning research which everyone is jumping on these days was kept alive and done at universities. The CRISPR/Cas9 breakthrough in biology was done at universities.

It is true that there is a huge pressure to publish which leads to a lot of crap being published but that doesn't mean the academia is not valuable and it seems like a hard problem to figure out the right incentives (which we should of course work on). Maybe something that gives a weighted rank of the papers would be better able to put value on the research you've done and make purely the # of papers less of a unit of currency in the academia. Citations do that to some extent but they're still not a great solution.

Egos are a huge problem, but that's true in every sphere. Academics are humans too...

<comment><USER20> Yes, that's what's missing from this article: recognizing academics as humans and not perfect idealistic beings.

To me criticisms like this need to at least suggest paths towards solutions. If the system is already as good as it can reasonably work, how can we criticize it?

It needed at least to suggest paths for researchers outside academia: will we get more research done outside? Will it fit better those idealistic goals?

</comment></comment><comment><USER21> I was similarly frustrated when I left a Math PhD program.

I realized that my best-case outcome was probably a tenure-track job at a #100 ranked school, and my worst/likely-case outcome was being a permanent adjunct with no other marketable skills. So I left to work at a software career.

<comment><USER22> I feel like this is why undergraduate research is absolutely key.

I had the opportunity to work in a lab while an undergrad and to do a couple industry research internships, and it set my expectations on what the Ph.D really is (and why I didn't go for it).

The problem is that many do not know what to expect or have expectations that differ from the reality.

<comment><USER21> In Math, it's almost impossible to do serious undergraduate-level research. It takes about 2-3 years of graduate level study in a specialty to get to the point where you are at new territory. Almost all the easy theorems have been proven already.

Also, it wasn't until I was in graduate school that I realized people that were comparable/smarter than me were struggling with mediocre jobs or not finding any permanent job at all. I also saw people who were really weak but picked a hot specialty doing rather well.

Also, everything is so specialized in Math that nobody really understand anyone else's work. If everything is so specialized, then how can anyone decide whose work is good and whose work is junk? If objectivity is completely out the window in Math, what hope is there in other areas of science?

So it's essentially the same as being a programmer, where the skill of promoting yourself is more valuable than the skill of doing good work. At least I get paid more, and still have spare time to read about things that interest me.

</comment></comment></comment><comment><USER23> Cached copy as I'm not getting a response from that page:

[http://webcache.googleusercontent.com/search?q=cache:U3-ruJk...](http://webcache.googleusercontent.com/search?q=cache:U3-ruJk3abcJ:crypto.junod.info/2013/09/09/an- aspiring-scientists-frustration-with-modern-day-academia-a- resignation/+&cd=1&hl=en&ct=clnk&gl=uk)

</comment><comment><USER24> "it is important because I've spent too many years working on it"

This hits me really close to home. When I left graduate school, a lot of my peers were unhappy about a lot of the same things I was. The reason a majority of them stayed was because "I put so much time in already and what's another 3 years." I almost stayed for the same reason.

Sunk cost is a fallacy. Be happy now :)

</comment><comment><USER25> Previous discussion, from two years ago: [https://news.ycombinator.com/item?id=6355579](https://news.ycombinator.com/item?id=6355579)

</comment><comment><USER26> I didn't see anything mentioning chemistry/science in the commentary, so forgive me if I'm repeating something I missed...

The PhD is not "just for academia". It's the entry ticket for working in the biotech/pharma business. It's a 4-6 year process that demonstrates that one can do independent work in an organized fashion and write it up. Quite similar, except for scale, as how medicinal chemists (and various support disciplines) work.

One learns the jargon, often combines a complementary post-doc to round out training (such as synthetic PhD following up with natural product p-doc). This extra nudge improves the chance of being hired and can usually lead to more $$ upon hiring. PhD's usually know 1-2 things well. P-docs, maybe, push that up to 3-5. It's a basis for getting started.

Is the entry-ticket aspect of a PhD fair to BS/MS students, who tend to be the lab denizens who actually do the work? No, probably not. There's a huge status difference between "PhD" and "pair of hands"... But then, as the difference is only 2-ish years, carrying through to PhD should be the typical choice (barring real-life issues forcing termination).

Due to job/geographic stability, it's not the best career choice, but it's a useful one (as, for example, they actually DO work towards curing or alleviating disease). The pay is reasonable, the chance for personal recognition is present and there's chances for "financial renumeration" if one scores big-time. And if groups like YC get really into funding longer-horizon projects, the jobs just might reappear.

</comment><comment> white-flame I had always wanted to be a university professor.

I do a ton of practical algorithmic research in the course of personal and professional software development. I love teaching and tutoring. I love definitively and completely solving hard problems with new solutions. I love devising proven partial solutions when that's not feasible, and looking forward to the rest. When I get a bug up my nose about a problem, I do not rest until it's solved, and solved properly.

It's probably around 15 years ago when I had my own realization that success in academia is driven primarily by publishing metrics. Since that's the reward mechanism, academia has optimized for that case above all else. As the article lists, only the current hot topics for publishing get any authorization or funding, so who knows what you'd actually be working on. And whatever you're working on demands _everything_ you have to avoid failure. Beyond that level of graduate indentured servant, it's all posturing and politics, still chasing the hot publishing topics.

I have been able to work on projects that map well to sub-problems of my long term interest field. Hobby and work blends nicely. I love what I do. While I'm not yet reaping significantly financially, I do what I want and do not have the stress and crunch of either academic process or a soul-sucking job, and that was my tradeoff to decide.

I still have unbounded financial potential as a business owner and curator of ground-breaking technologies. But I also know that technical merit does not equate to success in the market; that's another set of skills, and its own challenging industry.

</comment><comment><USER27> This is the last rant out of numerous appearing on the web. I wonder if, from a historical perspective, wasn't always like that. Galileo risked his life for science. Only undisputed geniuses of their time like Newton, Gauss or Archimedes could think and say something _original_ without being ridiculed by their peers.

<comment><USER28> Newton was ridiculed by his peers (or at least his contemporaries). Part of the reason why people tell the story of Halley convincing Newton to publish the Principia is that Newton hadn't published anything after the reception of some of his early work on optics.

For example, one of Newton's observations was that the spectrum cast by a prism he'd bought at a local fair was elongated. Being Newton, and therefore given to extreme of care, he had arranged the prism in a room that was about twenty feet long and was struck by sunlight coming through a pinhole in a shutter. This is not a simple thing to arrange, and because the Earth is rotating the correct geometry lasts quite a short time. But it allowed him to see the elongation of the spectrum.

The problem was that received wisdom said the spectrum should be circular, as it was cast by a circular beam. Some of the replies to Newton's paper were on the order of, "Well, I held up a prism and took a quick look and it seemed ciruclar to me! Mr Newton is a fool."

Science is always going to be like this. People are mostly idiots (I certainly am, most of the time.) But the author's point is that it is worse now, and that the institutional incentives that make it worse are pretty clear. Get a couple of beer into an older scientist and they'll often tell you that they couldn't have gotten an academic position in the current climate.

Furthermore, similar incentives are in play all over the world--althouhg a friend in the Far East tells me things are better there because universities are still trying to build reputations so they are engaged in more fundamental research rather than just grant-writing.

Given this, and given we depend on science to solve some of our biggest problems, and given we have a lot of smart, passionate, dedicated people going into the sciences to solve those problems, an institutional structure that fails to deliver on the promise of giving those people a meaningful career solving problems that matter is a serious problem for Enlightenment civilization.

<comment><USER27> > Newton was ridiculed by his peers (or at least his contemporaries). Part of > the reason why people tell the story of Halley convincing Newton to publish > the Principia is that Newton hadn't published anything after the reception > of some of his early work on optics.

It doesn't look like had it rough though. From his early years he was kinda distinguished[1] at age 27. But thanks for the heads-up and your Newton story. These stories are the ones I enjoy the most :-)

ps. You certainly don't look like an idiot to me. But I see what you mean, as Shakespeare often hinted, the wise men allows _doubt_ into any conversation while the fool men never does. :-)

[1] [http://en.wikipedia.org/wiki/Isaac_Newton#Early_life](http://en.wikipedia.org/wiki/Isaac_Newton#Early_life)

<comment><USER29> > _as Shakespeare often hinted, the wise men allows doubt into any > conversation while the fool men never does._

This is also known as impostor syndrome [1] vs. Dunning-Kruger effect [2].

[1] [http://en.wikipedia.org/wiki/Impostor_syndrome](http://en.wikipedia.org/wiki/Impostor_syndrome)

[2] [http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect](http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)

</comment></comment></comment></comment><comment><USER30> Server seems to have been hugged to death.

Google cache: [http://webcache.googleusercontent.com/search?q=cache:U3-ruJk...](http://webcache.googleusercontent.com/search?q=cache:U3-ruJk3abcJ:crypto.junod.info/2013/09/09/an- aspiring-scientists-frustration-with-modern-day-academia-a- resignation/+&cd=1&hl=en&ct=clnk&gl=de)

Note that the link is from 2013.

</comment><comment> bra-ket Cached: [http://webcache.googleusercontent.com/search?q=cache:U3-ruJk...](http://webcache.googleusercontent.com/search?q=cache:U3-ruJk3abcJ:crypto.junod.info/2013/09/09/an- aspiring-scientists-frustration-with-modern-day-academia-a- resignation/+&cd=1&hl=en&ct=clnk&gl=us)

</comment><comment><USER31> Yes, fine, we have all heard the complaints. I've thought similar things myself.

But when are we going to put our heads together to analyze the problem into its mutable points and immutable points, the better to find alternative arrangements and solutions?

<comment><USER32> I'm getting some people to flesh out a graduate program to focus on this problem, if you want to join..

<comment><USER31> Not sure if ironic...

</comment></comment></comment><comment><USER33> "The Higher Learning in America", by Thorstein Veblen, 1918. If this poor fellow had only read this book before commencing in his studies, perhaps it would have steeled his nerves somewhat, or sent him in another direction.

</comment><comment><USER34> I sympathize with a lot of these complaints, but the thing that really bothers me about a lot of these academia resignations, or complaints about the academic job market, is that there is definitely an expectation that a path (e.g., a job) should be provided for a young researcher to do whatever he or she wants, research-wise. I really wish that graduate school was viewed less as a job tunnel/funnel and more like undergrad, where it is an opportunity to learn (and network, and signal, and so forth) but ultimately, it's up to the student to pick a path, with the skills and reputation earned along the way.

To provide a counter to the hopelessness:

As much as I love the academy, and research, I never wanted to be a professor. After my postdoc I started a research and consulting company (thanks HN for many wise words of advice about going solo). I firmly believe that, in contrast to the hierarchical and managerial state of academic (and a lot of industrial) research, there is a valuable role to be played by professional scientists. So many of the badass scientists I know have such a small fraction of their time that they can devote to actually carrying out research[1], that all of their wisdom, efficiency and so forth is somewhat wasted as they go about their managerial lives as professors or industrial scientists. I don't think it has to be this way.

I haven't been doing this forever, but so far it's been great. I have been doing some interesting, research-oriented consulting (and some not so interesting, too), which puts money in the bank. It also lets me see some of the problems that industrial scientists deal with. I'm in geoscience and the consulting so far is all in energy, and unsolved problems there are just as cool if not cooler than the question du jour in academic work--and some of the problems are exactly the same... And it's very rewarding to be brought in on some of these projects, and help hard-working people out.

I also spend a good chunk of my time, half maybe, doing my own research, collaborating with others, and writing proposals to NSF and other public agencies. I of course have more ideas than I can ever hope to see through, but I feel very free to work on whatever I see fit, even if it's a bit outside of my experience and comfort zone.

In the end, I am hoping (and betting) that my accumulating strengths as a professional researcher, combined with the falling costs of a lot of science, particularly computational science and online journal access, will allow me to be as productive or more as I would managing graduate students and post-docs, and a lot happier. I also think that by doing consulting instead of teaching (which I enjoy but not enough), I can have a bit higher maximum potential income as well, and take a month off to raft through Peru when I get the hankering.

OK, so just to summarize, this ended up more biographical than I wanted: I think there is a market for professional, non-academic scientists, both in academic-type (i.e. gov't-funded) and industrial research. It seems to be the road less traveled but if you're not into that then science might not be your thing. It might be very hard at times, and hopefully you have a good support network.

[1]: I do believe that writing proposals is actually doing a very important part of science by making you get your shit together and vet yourself. It'd be fun if it wasn't for the existential pressure.

</comment><comment><USER35> Academia fucked up by copping an attitude toward the most important, but least glamorous, aspect of the job: teaching. In the 1960s, the U.S. had a strong research culture. Unfortunately, the professoriate took the attitude that research was their real work and that teaching was just grunt work to be sloughed off to the people at the bottom of the heap (graduate students and juniors). Eventually, they turned that work over to low-paid adjunct faculty, occasionally taking on an upper-level undergraduate course.

Teaching is important for a number of reasons, but it also functions as a sales front for research. Research is important, but only educated people can see its importance. If 40% of the next generation's leaders learn calculus and 5% learn complex analysis, then they'll have an appreciation for the beauty of mathematics and an understanding of its importance. However, if they're taught by overworked, underpaid adjuncts who've lost all enthusiasm for the subject, then the rising generation will lose sight of why the work is importat.

This is, of course, bad for science but even worse for the humanities, where the transfer of knowledge from one generation to the next (i.e. teaching) is almost the entire point, because commercial applications are nonexistent. I won't call most humanities research unimportant, because I don't understand most of it, but I will say that if you have a leadership class as ignorant of the humanities as the American upper and upper-middle classes are, you're going to have a hard time selling anyone on the importance of it. Most people are quick to decry what they don't understand as useless and unnecessary.

When the research rock stars started de-emphasizing teaching and offloading the work, the rest of society (including university administrators) called their bluff. If they were so ready to slough off the teaching (which is what most of the world sees as the real work) to overworked adjuncts, then why have professors at all? Of course, a few politically adept stars (grant-writing rain-makers) managed to hold on... but the quiet researchers are done.

The injustice of the tenure system is that the people to suffer for this attitude were the young-- not the entrenched tenured professors who created that culture. The jobs got taken away, but not from the people who copped that negative attitude toward teaching. It was the rising generation (from 1985-ish to today) that got fucked. At this point, you have a toxic culture of scarcity and the false poverty mentality (i.e. the "I'm 75% poorer than my friends, so I'm living a life of service and the rules don't apply to me" attitude).</comment>


New artificial intelligence beats tactical experts in air combat simulation - <AUTHOR> http://magazine.uc.edu/editors_picks/recent_features/alpha.html <comment><USER2> News discussion: [https://news.ycombinator.com/item?id=11993366](https://news.ycombinator.com/item?id=11993366)</comment>