This Function is Not Tail Recursive - <AUTHOR> http://codemonkeyism.com/function-tail-recursive/ <comment><USER2> No one addressed the parallelization claim, so here goes:

The ability to parallelize purely functional code comes from the Church-Rosser theorem (<http://en.wikipedia.org/wiki/Church%E2%80%93Rosser_theorem>), which states that you can do beta reductions in any order in the lambda calculus and get the same term.

The thing with tail-recursive code is precisely that there is only one way to do a reduction - the function has to call itself at the end. That's what makes tail-recursive code "iterative."

As soon as you introduce multiple reduction paths, you need a stack for the intermediate results - think of for example a recursive divide-and-conquer algorithm.

This Kornelius Fuhrer dude shouldn't be writing articles on parallel programming, he should be reading books about it.

<comment><USER3> I can, however, see a compiler turning simple cases of tail recursion into recurrence relations, and re-extracting the loop, but it's a relatively long way to go to get back that iterative bit of the problem, especially as a language in which tail recursion is often used probably won't be working with mutable arrays as often - a loop over input and output arrays being probably the easiest parallelization candidate.

<comment><USER2> I never thought about it that way. That's quite clever and out of the box!

Guy Steele gave a very good talk about the problem (basically, "tail recursion considered harmful") and solution ("divide and conquer") at ICFP last year: [http://docs.google.com/viewer?a=v&q=cache:bBDVKW8DP6UJ:l...](http://docs.google.com/viewer?a=v&q=cache:bBDVKW8DP6UJ:labs.oracle.com/projects/plrg/Publications/ICFPAugust2009Steele.pdf+foldr+considered+harmful&hl=en&gl=ca&pid=bl&srcid=ADGEESiHWEmfK12_eOyYM0465fA2ltUZLfPwQxtlNADEVwhJgdiIFKB1_YRvSv064kacRgLtLX7CWHrBBi72WaPrlXcrQqRXaqJB6-DSA10Sej9jFkXJM7GVPVx- VUfkmho-s5sduHga&sig=AHIEtbTLZ4NCoMphAYt6CkQ7PuspHJqNCw)

APL-style operators (map/reduce, paralations, NESL, etc.) are definitely the way to go. Not only do you get parallelization "for free," but optimizations like map fusion (see [http://www.randomhacks.net/articles/2007/02/10/map- fusion-an...](http://www.randomhacks.net/articles/2007/02/10/map-fusion-and- haskell-performance), and Rich Waters' earlier work on SERIES: <http://series.sourceforge.net/>) become easy - something like that is almost impossible to do with loop analysis. All that, and you end up with clearer, shorter code. Due to all these reasons, I'm convinced that automatic loop parallelization is the biggest waste of time in parallel computing research.

</comment></comment></comment><comment><USER4> In the example case, and many others, the tail-recursive version is the simple application of an accumulator. So for the factorial example, an accumulator version would be (in haskell since I don't know the example language very well):

      fac1 0 = 1
      fac1 n = (*) n (fac1 ((-) n 1))

      -- tail-recursive
      fac2 0 acc = acc
      fac2 n acc = fac2 ((-)n 1) ((*) acc n)

To me, something in with a pattern of:

f v = g v (f (t v))

where f, g and t are functions (t is a transform on a, in the factorial case it is (-),

Can usually be turned into the form (mechanically even):

f v acc = f (t v) (g v acc)

Does this hold for all cases or am i missing something? If it does hold, could a language just implement the TCO for it, and therefore couldn't we just consider the very first implementation of fac1 to be tail-recursive?

edit: It is ugly but changed my exmple code to match the pattern i describe exactly.

<comment><USER5> I believe this has some issues if the operation isn't commutative.

For example:

blah 0 = []

blah n = (++) [n] (blah ((-) n 1))

\-- blah 5 -> [5,4,3,2,1]

blah2 0 acc = acc

blah2 n acc = blah2 ((-) n 1) ((++) [n] acc)

\-- blah2 5 [] -> [1,2,3,4,5]

If we changed the tail to: blah2 (n-1) ((++) acc [n]) it would work in this case, but then we've flipped the arguments, and I have a feeling we could come up with a function where simply flipping the arguments won't work, though I can't think of one at the moment.

<comment><USER4> Good point! It becomes hard to do mechanical transforms like I suggested when you have to keep track of properties like this for every function. Thanks!

<comment><USER5> Ya that's the problem. Here's what I originally intended to write, but couldn't think of how to do cons in haskell so I used list addition. With cons (:), you can't rewrite it with the arguments swapped, so no simple mechanical transform will be able to fix this.

blah 0 = []

blah n = (:) n (blah ((-) n 1))

\-- blah 5 -> [5,4,3,2,1]

blah2 0 acc = acc

blah2 n acc = blah2 ((-) n 1) ((:) n acc)

\-- blah2 5 [] -> [1,2,3,4,5]

</comment></comment></comment></comment><comment><USER6> A comment on the site itself noted that tail recursion does not imply parallelization -- in fact, I would make a stronger statement that it _forces_ serial execution, at least as far as funneling results of one "iteration" into the next.

The benefit, as "everybody" here of course already knows, is to reduce memory (call stack) usage.

Knowing how to implement tail recursion allows you to write in an "assign only once" type of language, without blowing up the stack. And using such "invariant symbols" in a routine allows the compiler/interpreter to seek parallelization at other points in the function on your behalf. OK, so I just contradicted myself, but the point is there is an rather indirect connection between tail recursion and parallelization.

</comment><comment><USER7> One way that would make that particular example clearer is to convert the multiplication from an infix operation to a function call:

return n == 0 ? 1 : n * fac(n -1);

becomes

return n == 0 ? 1 : multiply(n, fac(n -1);

This makes it a fair bit clearer, IMO.

<comment><USER3> I suspect a better approach is to make expected tail recursion a language feature, and have the compiler emit an error if the returned expression is not in fact valid for a tail call.

Imagine if it were 'treturn' or something.

It's not something that bites people who have been around this mulberry bush before, but I think it can very likely catch problems that novices have, thinking they're writing an O(1) space solution but actually using O(n) stack.

<comment><USER8> OK. But then, even if somebody is using `return', and its tail-recursive, it should be optimized. (`return' should just not raise an error, if it's not tail-recursice.)

</comment></comment></comment><comment><USER9> Wow, this is basic stuff. I'm surprised that a researcher would get this wrong.

Also, let us not forget that tail recursion is a specific form of the more general _tail call optimization_. For example, the following is two TCO-able functions calling each other:

        functionA(...):
            // do stuff
            return functionB(...)

        functionB():
            // dostuff
            return functionA(...)

Whether or not the compiler optimizes this, however, is a completely different question.

</comment><comment><USER10> Perhaps I'm missing some larger point here, but if all tail recursive operations can be represented as loops, why not code them that way in the first place? It's probably better to know that recursive calls take up stack space on each call, than it is to know that tail recursion optimizes to a loop.

If somebody could provide an example of a tail-recursive function that would be much more difficult to code as a loop, I'll eat my words.

<comment><USER11> As it happens, I've actually grown to like Clojure's _recur_ more than tailcalls. And I much loved tailcalls over imperative loops earlier.

 _recur_ is still functional, it resembles a loop but isn't one so nothing is mutated, and you can even nest them. And you can recur over a _loop_ or the function itself. Handy.

<comment><USER12> That's fairly common style in Scheme as well, fwiw; a "named _let_ " is very similar to Clojure's _recur_ , and some programmers prefer it over tailcalls, at least for many common recursion/iteration patterns.

</comment></comment></comment><comment><USER13> On the JVM, wouldn't * be an operator rather than a function? Would it even make a difference?

<comment><USER14> At the assembly/bytecode level, you still have to save the return value of the recursive call and then do something to it, whereas a tail call wouldn't need to save any intermediate value because it would immediately return it as soon as it got it.

I guess Java's infix notation kind of obscures this. This is one of the reasons why functional programmers prefer to consider _everything_ a function. It's also why some languages (like Clojure) have special syntax for tail calls, with special error messages if you goof up the semantics.

<comment><USER8> Yes, in Lisp there's no real difference between * or car.

In Haskell, which has more syntax, you can use * in infix notation, but so you can with other function. E.g. the function `on`. Quoting from the documentation:

      on :: (b -> b -> c) -> (a -> b) -> a -> a -> c
      (*) `on` f = \x y -> f x * f y.

      Typical usage: Data.List.sortBy (compare `on` fst).

</comment></comment></comment><comment><USER15> The correction is incorrect.

A function is tail recursive when the last thing it does is call some function, _possibly different_. Yes, you can have a mess of different functions that are mutually tail recursive. For an example of how this comes up, think "state machines" with each call being another state transition.

<comment><USER16> I am used to the term _Tail Call_ for a function that returns the result of calling another function. _Tail Recursion_ is then a special case where a function makes a Tail Call to itself. _Mutual Tail Recursion_ is another special case where multiple functions make tail calls to each other in such a way that a function will end up calling itself indirectly.

Not a tail call:

        var even = function (x) { return x == 0 || !even(x-1); }

Mutually Tail Recursive:

        var even = function (x) { return x == 0 || odd(x-1); }
        var odd = function (x) { return x == 1 || even(x-1); }

<comment><USER8> Your first example is actually a tail call. It's just a tail call to the || operator. Every function has a tail call by definition.

<comment><USER8> And the next two examples are tail calls of even and odd thanks to shortcutting of ||.

<comment><USER16> Absolutely the calls to even and odd are tail calls because of the semantics of || in Ruby (and most other languages). I hope we agree that even and odd are still mutually tail recursive, because Mutual Tail Recursion is a special case of tail calling.

</comment></comment></comment></comment></comment><comment><USER17> I don't understand what his last statement means:

"So please, do not call all functions where the last function call in your source code is a call to itself, tail recursive. A function is tail recursive, if the last operation is a function call to itself."

It reads to me as "Don't call functions that call themselves last tail recursive. A function is tail recursive if it calls itself last."

That's saying the same thing, if my reading of it is right. His oddly-placed commas don't help.

<comment><USER18> I can't read the article (silly work web filters), but if you're quoting him correctly then I have an idea. (

      function foo(n) {
        ...
        return n+foo(n-1);
      }

is not tail recursive because the last operation is not the recursive function call, but rather the addition.

      function bar(n,acc) {
        ...
        return bar(n-1,acc+n);
      }

is tail recursive because the last operation is now the recursive function call.

</comment></comment><comment><USER19> It seems like a difference without distinction to me. I mean, sure, it doesn't fit the precise dictionary definition of tail recursion, but the function would be fairly trivial to implement in a purely tail recursive manner. As such, its close enough for most purposes.

<comment><USER20> Tail recursion means that there's nothing in the continuation of the recursive call besides a return from the outer call.

This means that a compiler can optimize away the saving of a new stack frame for a recursive call, because the old stack frame of the enclosing call will never be used again. That's tail-call optimization.

If there's stuff in between the return from the recursive call and the return from the enclosing call (like the multiply in this factorial example) then the old stack frame will have to be kept around which means you must use unoptimized recursive calls.</comment></comment>


What Rust Can Do That Other Languages Can't - <AUTHOR> http://robert.ocallahan.org/2017/02/what-rust-can-do-that-other-languages.html <comment><USER2> I spent an afternoon messing about with Rust and found it infuriating. The compiler kept (quite rightly) telling me how crap my code was and wouldn't compile it for me. I found it frustrating but thoroughly educational.

For systems programmers, Rust looks a fantastic option. For line of business apps, it is inaccessible and the safety dial is turned up too far. But then I guess that is because Rust isn't intended to be a general purpose language for writing line of business apps?

I can see it being useful for writing platform neutral, gnarly code and offer easy hooks for popular, managed languages such as JavaSript, C#, Java et al to hook into.

Outside of systems programming, I wonder if Rust will be part of a silent revolution?

<comment><AUTHOR> I am a volunteer who works on Rust & I do almost all of my open source work in Rust. I am paid as a Ruby web developer. I personally have little interest in systems programming & want Rust to be a productive language for application development.

I frequently witness this rush to judge Rust as too complex to be used for business applications based on a few hours of hacking. For this to be a valid judgment, I would think one of two things would need to be true: a) Rust remains this difficult to use once you have experience with it, or b) most business applications are worked on for only a few hours.

In my experience, neither of those premises hold.

I believe that the rules of Rust not only give you exceptional performance without memory issues (as this article tries to demonstrate), but also that its rules engender a higher degree of correctness and 'well-factoredness' than most languages. I believe Rust is an excellent language for writing applications at every level of the stack when 'craftmanship' matters.

<comment><USER3> For me personally Rust is not very interesting because it doesn't have a garbage collector. Manual memory management is just a pain in the ass and really not worth it for the vast majority of applications - games and realtime audio processing are an exception.

Besides, if I don't want automatic memory management, I could also use Ada or Freepascal, who are both reasonably safe, though not generally as safe as Rust, _and_ are mature and have a well-established toolchain.

<comment><USER4> I have never done anything with Rust but I was under the impression that Rust's borrowing and ownership system made manually freeing memory more or less obsolete except for a few edge cases where the compiler isn't intelligent enough. Is this impression false?

<comment><USER5> Manual resource management in Rust is about as convenient as manual resource management in modern C++. That is, you don't really do any "manual" memory management at all in >90% of your code, it's all taken care of by RAII.

<comment><USER6> Yes, but with the added benefit not requiring external tooling to validate those RAII pattern are used correctly and forcing not so security savvy developers to actually use them.

<comment><USER7> So it's better, safer, but still manual memory management.

<comment><USER6> Not quite, because the rules are enforced by the compiler.

In C++ the rules are enforced by optional tooling that most developers don't use.

At CppCon 2015, only 1% of the audience acknowledged using such tools at Herb's talk about the core guidelines.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> In D, the example looks like (if I understood it correctly):

      struct X {
        Y y;
        ref Y getY() return { return y; }
      }

This tells the compiler that getY() returns a referenced based on the implicit 'this' reference. It's available now (DIP25 was the proposal for it).

We're now implementing DIP1000, which adds similar support for pointer values (DIP25 only dealt with references).

To see it in action:

      alias Y = int;

      struct X {
        Y y;
        ref Y getY() return { return y; }
      }

      ref Y foo()
      {
        X x;
        return x.getY();
      }

Compiling it yields:

      test.d(11): Error: escaping reference to local variable x

<comment><AUTHOR> All due respect, but I would be extremely surprised to learn that this feature is as capable as Rust's lifetime system. Whiel you've shown that you can't have an escaping reference to a local variable, what Rust provides goes far beyond that. You can return the references to Y (for example if your function has a reference to X as an argument), you can store them in the heap, you can reassign them to be references to different Ys, etc, all so long as the Ys they point to live longer than they do.

<comment><USER8> > I would be extremely surprised to learn that this feature is as capable as > Rust's lifetime system.

So would I. It isn't. But we're working on much more than this in order to provide guaranteed memory safety (although using a very different approach than Rust's).

It's just that Rust does not stand alone with this particular example.

<comment><AUTHOR> I think a key fact about this example is that `&Y` is just another type of value which can be used like any other; if you can't return it our put it in a vector or whatever its really not comparable.

</comment></comment></comment></comment><comment><USER9> D is working on it, if I understand DIP1000 [0] correctly. The example would look like:

      struct X {
        Y y;
        scope ref Y getY() { return y; }
      }

[0] [https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md](https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md)

<comment><USER10> How would it be used from user code ?

<comment><USER9> See Walter's reply. He is actually doing it. [https://news.ycombinator.com/item?id=13578315](https://news.ycombinator.com/item?id=13578315)

</comment></comment></comment><comment><USER11> I have no idea what this means, and I have a masters in Computer Science and 20 years of industry experience. Enlighten me?

<comment><USER12> Let's translate this code to C++:

        class X {
          public:
            Y* getY(void) {
              return &y;
            }
          private:
            Y y;
          };
        }

`getY` returns a pointer to the struct field, this field is allocated as a part of the struct. There is no way (in C++) to verify that this pointer does not outlive the struct itself. In other words, your pointer may become stale, and the compiler has no way to check for it.

Rust has a resource ownership model (borrow checker) that allows it to check at compile time that references do not outlive the data.

<comment><USER13> A little closer would be to use a shared_ptr<X> and shared_ptr<Y> that share the same reference count (1). It's not exactly the same thing, though, since Rust gives you some safe usage checks that you don't get with shared_ptr. That is, vanilla Rust objects have lifetime semantics closer to unique_ptr than shared_ptr.

(1): See constructor number 7 here, which was designed for this purpose: [http://en.cppreference.com/w/cpp/memory/shared_ptr/shared_pt...](http://en.cppreference.com/w/cpp/memory/shared_ptr/shared_ptr)

<comment><USER14> It also isn't the same in that it comes with the cost of allocation and reference counting.

</comment></comment></comment></comment><comment><USER15> > Most other languages simply prevent you from giving away an interior > reference, or require y to refer to a distinct heap object from the X.

Go lets you return an interior reference. But Go uses a garbage collector instead of lifetime checks at compile time (which is an advantage or a drawback, depending on your requirements).

<comment><AUTHOR> I don't know much about the Go compiler internals, but it sounds like in most cases (possibly all) the object being referenced in that way will be heap allocated; the entire point of this article is that no matter what (across function boundaries, the reference itself put into the heap, whatever) the member `y` of `X` will be inline with its stack representation.

[https://golang.org/doc/faq#stack_or_heap](https://golang.org/doc/faq#stack_or_heap)

<comment><USER16> Go does escape analysis like most (all?) good GC implementations to detect optimization opportunities. You can allocate an array of objects, for instance, and Go will try to make a contiguous, dense allocation for all the members, enabling extremely efficient pointer arithmetic to resolve member offsets thus avoiding an indirection, and precluding the GC bookkeeping for individual elements, and perhaps improving processor cache utilization.

But the key point stands wrt Go; Go is among the languages that "can't do this." First, there is no way to get a reference to a field inside a struct in Go (the workaround is reflection which has runtime overhead, and therefore doesn't count) and if you could get such a reference the compiler would be forced to reference count the member via the GC to satisfy the Go memory model.

<comment><USER17> > First, there is no way to get a reference to a field inside a struct in Go

what?

[https://play.golang.org/p/1OJ5U5ZwFJ](https://play.golang.org/p/1OJ5U5ZwFJ)

> if you could get such a reference the compiler would be forced to reference > count the member via the GC to satisfy the Go memory model

no reference counting in go.

<comment><USER18> I'm not that familiar with Go, but my guess is that "reference count" here refers to the action that the garbage collector does when it pauses the world (i.e. "count as a reference") rather than the actual paradigm "reference counting".

</comment></comment></comment></comment></comment><comment><USER19> Can anyone explain the reason why C or C++ compilers can't do this? Obviously the language specs allow you to do the unsafe thing, but suppose we add some "-Wreference-lifetime" flag to gcc that warns if it can't statically verify that a reference or pointer doesn't outlive the referenced object, and then compile everything with "-Wreference-lifetime -Werror" from now on.

What aspect of the language in particular makes that impossible? Or is it?

<comment><USER12> In principle, nothing stops C++ compilers from implementing this feature.

However, this

\- requires additional input from the programmer, such as lifetime parameters in Rust.

\- C and C++ don't have ADTs which help working with lifetimes A LOT (e.g. having Option<Box<T>> as nullable pointer).

\- will reject 99.9% of useful libraries and programs written in C.

\- would be equivalent to using Rust.

<comment><USER20> For the record C++17 does introduce std::variant. It's horribly cumbersome but it does at least exist.

<comment><USER12> It's horribly cumbersome AND it doesn't solve the same problem.

        // Rust
        let v: Result<i32, i32> = Ok(1);
        // std::variant (doesn't compile)
        std::variant<i32, i32> v = 1;

<comment><USER21> I had to RTFM a bit to understand this example, so here's a more explicit description of this case for those playing along at home:

AFAICT the idiom for Result<i32, i32> is a pair of "numerical-result-if- success"/"errno-if-failure". Seems like a great idea, allows us to not overload a simple integer result and end up ignoring the failure path by accident. "Ok(n)" would get assigned to the "numerical-result-if-success" and "Err(m)" would get assigned to the "errno-if-failure". I think this means that "v.unwrap()" would panic if it had been assigned "Err(m)", which is what most people would want/expect.

AFAICT std::variant can't do overload resolution correctly when both of the union types are the same. I'll wager that if you use an aliased type it will still choke on this ambiguity. So if you wanted something similar in C++ maybe you'd have to wrap it so it's not POD?

<comment><USER12> It's not quite a "pair". It's a "tagged union" in C-speak, "enum" in Rust- speak and "sum type" in CS-speak. In this case, numerical result and errno will be located in the same memory, but have different tag.

</comment></comment></comment></comment></comment></comment><comment><USER22> There's one area where Rust's typesystem will probably be counterproductive, and that's flexible and efficient graph algorithms. (Someone please prove me wrong).

<comment><USER23> Which is where the 'get out of jail' card in the form of 'unsafe' code is useful. You can always fall back to not having those guarantees checked if you're writing core, low level, well tested code. You can then ensure the right guarantees are made by typing at the interface, and ensure the code is used correctly.

This is why writing data structures in Rust is considered an 'advanced' topic.

<comment><USER24> This last sentence sounds a bit worrying. Writing data structures such as graph is just as common as writing if then else... if this is an advanced topic, then maybe an intermediate library providing common patterns for building elaborate structure would be welcome ( that's usually what stdlib types are about, but with the borrow checker, maybe those aren't enough).

<comment><USER25> I've been wrestling with this very concept. I often find myself writing custom variants of datastructures and relatively uncommon datastructures, and a language that makes this difficult isn't very motivating to get into.

What is really needed is good documentation on this specific aspect.

 _If on the hand if the messaging is "if you find yourself reaching for unsafe, it is likely you need to rethink your code's design" and then also "common datastructures are hard to write without unsafe" there is some slight dissonance there (imho, ime)._

I'm not looking for 'blessed libraries' of prebuilt datastructures. I want to code up datastructures (e.g quadtrees, graphs etc), and good guidance on when exactly unsafe is the only way to get this done.

I don't really want to pause in the middle of a project and have to spend serious amounts of time coding up a (custom/unusual) datastructure just because there is no way without using unsafe (etc).

Right now there doesn't seem to be good _guidance_ on _this aspect_ (writing datastructures in Rust) which is presented as an "advanced topic" and "the wrong way / the hard way" to learn Rust. (there is the linked list book, which serves as a starting point, and using Cell etc seems to work - I'm still struggling - which is fine, that is how people learn.)

PS: None of the above is really a criticism of Rust the language or the team. I think Rust is great, I'm just a bit frustrated with the 'build common datastructures being "advanced" ' aspect, but in the end it is probably just that I haven't been able to wrap my head around Rust yet

<comment><USER26> I think the messaging is like this because folks approach Rust and often try to write Rust code as if it were C. You can do this in C++; you can't do this in Rust.

unsafe is highly discouraged, but designing the innards of abstractions like datastructures is one of the places where you basically need it.

Be aware that good generic datastructures are just as hard in C++. See [https://news.ycombinator.com/item?id=13580418](https://news.ycombinator.com/item?id=13580418) . Naive datastructure implementations often have strict aliasing bugs or mess up on the destruction behavior. Using unsafe to write datastructures in Rust isn't particularly hard; it's just as tricky as it is in C++. Having to use unsafe is annoying, but it's a minor annoyance. The nomicon helps teach how you're supposed to use unsafe.

</comment></comment></comment></comment></comment><comment><USER27> Seeing the comments here i fear that Rust will end up binned alongside the likes of Ada. Because people want to write code that runs and thats it, correctness and safety be damned...

<comment><USER28> So long as correct programs are incorrectly rejected, and it takes extra time to fix those mis-matches, many people will simply avoid the pain in the first place.

At the end of the day, many of us are paid to ship features. For many industries, correctness and safety just can't compete with new features. It's the reason Lisp, Python, Ruby, and Go are so popular - you can quickly write programs which are both fast enough and memory safe.

The gaming industry, one of the bigger C++ consumers, will also probably not move to Rust for similar reasons: hitting a schedule is much more important than not crashing or not having memory leaks. You can always patch a game.

<comment><USER29> The counter-argument to this being that the safety checks imposed by Rust prevent errors in the first place, saving debugging time and getting features out more quickly.

<comment><USER28> > the safety checks imposed by Rust prevent errors in the first place

Not all errors, only a particular class of memory errors. Most of those memory errors are those that a GCed language doesn't have to worry about, and C/C++ have a bevy of tools to find those errors before they leave the developer's hands as well.

That said, Rust's borrow checker will also help protect against a class of shared memory mutation errors as well. Is the time cost of pleasing the borrow checker for every memory allocation worth this benefit? Probably something only individual developers can answer. My answer is, for now, no.

</comment></comment></comment></comment><comment><USER30> > _pointer addition_

Isn't it a bit confusing to talk about pointer addition when the example struct is just

        struct X {
          y: Y
        }

and thus the addresses &x and &y are (probably) the same. Maybe the author thinks that &y is obtained as &x+0, but.

<comment><USER26> The author is generalizing to when there are more fields here.

</comment></comment><comment><USER31> In what situation would you want to do this?

<comment><USER32> This like this are extremely common in performance-critical code. In this case, you want to:

1\. Allocate both X and Y in the same block of memory, maybe on the stack, perhaps in a heap. In inner loops, making calls to malloc is pure performance poison. Most common garbage collected languages are completely unable to do this, with the admirable exception of the C# and the other .NET languages (and possibly some others).

2\. You want to access Y by reference, so that you can work with it without needing to make a copy. Again, this is very common in performance-critical code. You can do this in C, C++ and other existing "systems" languages.

3\. You don't want to accidentally keep using Y once X (and hence the underlying storage for Y) is destroyed. This is a subtle and vile bug that can memory corruption, once-a-week crashes on production, and week-long debugging sessions, among other headaches. This is where C and C++ fail, and where Rust nails it.

<comment><USER33> > In inner loops, making calls to malloc is pure performance poison. Most > common garbage collected languages are completely unable to do this, with > the admirable exception of the C# and the other .NET languages (and possibly > some others).

Modern GCs use bump allocators and (ideally) combine allocations of objects and their subobjects where that is possible.

<comment><USER32> Even with a bump allocator, you're still putting large amounts of data into the nursery generation. If you're in a performance critical inner loop (for each line of a 50 gigabyte CSV file, or for each scan-line in an image codec), this is still a bad idea, performance-wise. You still can't beat static stack allocation, where you increment a pointer once to create a stack frame, and decrement it to free memory.

In really fast Rust code, my goal is usually to entirely eliminate all heap allocation in favor of zero-copy parsers. This can be tricky—especially for streaming I/O using buffers, because a single chunk of input might get split over two buffers—but Rust's borrow checker makes it possible to maintain references into other people's buffers without shooting yourself in the foot.

<comment><USER34> With java scalar replacement (possible from escape analysis/partial or not) calling new Point(x,y) in a hot loop does not allocate an object on heap at all (or even the stack) unless it escapes.

In the newer JITs that escaping allocation will be deferred until the last possible point in time.

This is actually a nice point about JVM languages, _new_ has defined behavior and can thus be elided if the effect is not observable. malloc (or new in C++, although I am not sure about the spec there) do not and eliding a call to new may break the language specification as function is not called that should have been.

I don't know what Rust does here, and what is allowed in regards to the new operator. i.e. must a local variable be on the stack or can it generate direct values in machine registers?

<comment><USER35> Rust doesn't have a new operator (its equivalent to malloc is Box). When that's used, it forces the value onto the heap- but that's not what you would write most of the time.

Local variables that are not explicitly heap-allocated (and don't have their address taken) can be on the stack or in registers, and this is true of both Rust and C/C++.

</comment></comment></comment></comment></comment></comment><comment><USER36> Rust seems to be a good language for game development. I watched this guy ([https://www.liveedu.tv/gexon/videos/evAbX-dotakiller- gamedev...](https://www.liveedu.tv/gexon/videos/evAbX-dotakiller-gamedev- indiedev-onedev-rust-42)) and it made me want to try it.

</comment><comment><USER37> C can do exactly the same, it just won't perform the lifetime checks for you. I haven't written much Rust, but if I understand correctly you can write code that is valid, but won't compile because you're not 'following the rules' of the constraints/type checker.

I find Dafny to be a more elegant and nice solution, since this language allows you to actually prove your code is valid when it's not obvious from the static analysis. But this might be because I'm somewhat more mathematically inclined (and i admit that Dafny is still by no means easy to use)

</comment><comment><USER38> Other languages can definitely do it:

Ruby:

        class X
          attr_accessor :y
        end

Crystal (full working example):

        class Y
        end

        class X
          def initialize(y : Y)
            @y = y
          end

          def y
            @y
          end
        end

        y = Y.new
        x = X.new(y: y)
        puts x.y

<comment><USER39> The article is not about returning a reference to a struct member, its about returning a reference to something on the stack rather than heap.

<comment><USER16> Rust ownership applies to heap allocated objects as well; if you return a reference to a member of a heap allocated struct that reference cannot outlive the struct allocation. The point of the article is more general than "stack vs heap;" other languages are either incapable of expressing a reference to a member of a heap allocated struct or incapable of ensuring the memory safety of such a reference.

<comment><USER39> As the article notes, garbage-collected languages can ensure the memory safety of a pointer into a heap object.

<comment><USER40> But not at compile time, which is what Rust provides.

</comment></comment></comment></comment></comment><comment><USER41> > What Rust Can Do That Other Languages Can't

But other languages are Turing-complete too...

<comment><USER22> Yes, but Rust might still be able to do what other languages can't. For example, to express a given program using at most N input symbols.

<comment><USER42> Perhaps more relevantly, we can talk about the relative sizes of the sets of invalid Rust programs that rustc will fail to reject, and invalid C programs that gcc will fail to reject. The halting problem tells us that neither set is empty, but there's still a lot of room for differentiation.

(Actually, upon further thought, it may be that rustc rejects all invalid Rust programs and some valid ones. gcc certainly accepts some invalid C programs.)

<comment><AUTHOR> My rough approximation is that the difference between a sound type system and an unsound type system is that a sound system rejects some valid programs and unsound system accepts some invalid ones.

</comment></comment></comment></comment><comment><USER43> I don't think this is a particularly useful example of what rust can do.

Other languages can return inner pointers; its not an interesting feature.

The only interesting thing is that it is both safe and has no gc... but really, this is just another example of safety in rust, via a relatively obscure code snippet.

If you want to pitch rusts safety features, we can do something a bit more interesting, surely?

<comment><USER44> Most safe languages are unable to return inner references. Heap-allocating the inner object doesn't count; object inlining is important for systems programming.

You're absolutely right that there are much more interesting examples, e.g. [https://t.co/HJV7G4n0PD](https://t.co/HJV7G4n0PD). Doesn't fit in six lines though :-)</comment></comment>


Show HN: I created a file transfer website that doesn't involve a server - <AUTHOR> It uses WebRTC to allow peer-to-peer sharing, so you can send files to devices on the network using super fast WiFi speed. You can send to anyone on the web though.
Drag files, get a download link and send it. Enjoy
http://cend.me <comment><AUTHOR> Use cases I found this exceptionally useful:

* I wanna send files from my phone to my computer that's under the same network.

* I wanna send pictures from my device to someone else's device that is 2 feet away from me without losing quality via WhatsApp or Facebook Messenger.

* I wanna send files from my computer to my father's computer without uploading it to some cloud service or establishing a shared folder.

* I wanna send personal/confidential files from my computer or device to someone else without it being logged or saved somewhere, in a direct communication manner.

</comment><comment><USER2> HN has a "Show" section for showing off projects and stuff, you can submit to it by starting your title with "Show HN".

<comment><AUTHOR> Did not know that, thanks!

</comment></comment><comment><USER3> Feel free to "Show" on [http://rtc.news](http://rtc.news) too

</comment><comment><AUTHOR> Forgot to mention, Chrome works 100%, FF is good up to 16MB, in end January FF will be 100% as well.</comment>


I got access to all of BitTorrent Inc's source code and sensitive documents - <AUTHOR> http://forum.ragezone.com/f10/bittorrent-source-dns-reward-948810/ <comment><USER2> Looks like he acquired login information, he didn't exploit some sort of major vulnerability that needed to be fixed which is generally what bug bounties seem to be for. He's insulted he didn't get paid for not being an asshole, what a strange world we live in. Facebook (or any other company) wouldn't pay out if one of their support staff accidentally made a random user an admin, why should this be any different.

<comment><USER3> He didn't get credentials, they never secured their admin panels:

> A friend was looking for a hosting provider and wanted it all secure, I > checked a network range and found the Jenkins panel of BitTorrent. This was > all 100% accidental, truth! At the end of the day If I wanted to 'hack' > BitTorrent I wouldn't even know where to begin. I mean, there is no real > skill or talent involved in what I did to find the information on > BitTorrent. They forgot to set a user/pass to the admin panel, that had > access to github from a master account. Github accounts had user/passes that > was linked to everything.

> No hacking, just random find. Mad huh?

</comment></comment><comment><USER4> I find the author's righteous indignation that he did not receive the ransom he thought he deserved amusing at best.

<comment><USER5> I contacted BitTorrent and told them EXACTLY what I did , what information I had access to and what user/passes needed updating, including SSH keys and more. Whilst I gave a small generalization of what was at hand on the thread, there was a considerable amount of data available to me.

\- MentaL (RZ Thread Starter).

<comment><USER6> Scenario 1: You find a boy who is lost and take him to the police station. The family gives you $500 in reward.

Scenario 2: You find twin boys who are lost and take them to the police station. The family gives you $500 in reward.

Would the fact that you get the same reward regardless of the number of people lead you to act differently in the future? So if next week you found lost twins again, would you let them stay lost or try to sell them on the black market because you know you'll only get $500 when you think it should be $1000?

Or is doing the right thing its own reward regardless of whatever amount you get in recognition for acting honorably?

<comment><USER7> Who wouldn't help a lost boy for free ?!

<comment><USER8> Many otherwise sweet young women (or men!) several of us probably went to school with, for a start. :|

Edit: Was trying to give some help to somebody puzzled about this, which is why I bring it up. Turns out that, if you're a lost young man, there's probably a point in your travels where you zigged instead of zagging and thus lost your way.

<comment><USER9> In the "lost child" scenario, it seems to me there's some ranking like:

1) helping out 'cause it's the right thing to do.

2) helping out with some hope there'll be a reward.

3) not helping out, 'cause there's probably nothing in it for you.

4) helping out with an expectation of a reward.

5) helping out with an expectation of reward then complaining publicly 'cause it's "not enough"

…

VERY_LARGE_INT) Abducting the child.

While there's a long way between 1-5 and the biottom of that list - there's no doubt in my mind that 1 & 2 speak significantly better of the "helper" than 3, 4, & 5\. (and interestingly, there's no "outside" difference between 1 and 2 or 4, unless you choose to admit your mercenary inclinations to onlookers.)

Things are a bit murkier in the guy in the article's case, since "bug bounties" are known about and perhaps "expected", and I think Bit Torrent handled this badly, but further down the thread he admits "This was all 100% accidental, truth! At the end of the day If I wanted to 'hack' BitTorrent I wouldn't even know where to begin. I mean, there is no real skill or talent involved in what I did to find the information on BitTorrent." That makes it hard for me to feel sympathy for him failing to get a windfall "bug bounty".

</comment></comment></comment></comment></comment></comment><comment><USER10> Just because someone leaves their car door unlocked and you tell them about it, doesn't mean they are obligated to give you any of its contents.

<comment><USER11> On the other hand, if you find a wallet, you are entitled to 10% of the cash.(1) (Bad fit as an example, I know, but the question is what is a good comparison..)

(1) at least in Germany.

<comment><USER12> Is this German law? I wish that were the case in the US, as I've found a number of wallets over the years and would have made at least a few hundred USD by now.

<comment><USER13> It is. But only if the wallet contains >50€ or so.

</comment></comment></comment></comment><comment><USER14> You got my hopes up that we'd finally be able to audit Bittorent Sync.

<comment><USER15> lol I was hoping for that too. A great piece of software but a waste since it can't be trusted for anything secure. To make it worse, this news just adds more fuel to the fire. How can you trust them with security if they can't even remember to add a basic authentication to their Admin Panel?

<comment><USER9> I've got it syncing EncFS encrypted directories - which in the broad view doesn't help - I've got un-audited code running with my user privs, which _claims_ to "not do anything evil" without being (easily) verified.

At least it does mean I've got some extra encrypted-on-the-wire and encrypted- at-rest data floating around - I strongly suspect BTSync's SHA256 encrypted wire protocol moving AES encrypted EncFS files means that data is secure against the NSA's ubiquitous surveillance programs - the cleartext of those files won't be showing up in XKeyscore in response to bored or curious NSA (or Boos-Hamilton) staff. (While acknowledging that if the NSA becomes interested in _me_ specifically (or probably interested enough in any of my social network) - my privacy will be easily and wantonly violated. And I'm actually _mostly_ OK with that.)

</comment></comment></comment><comment><USER16> You have to be proud of yourself for not abusing it no mater how they responded. You did right thing here. Hope Bitorrent Inc will fix this misunderstanding.

</comment><comment><USER17> If one guy could accidentally stumble upon this, what are the chances others have too? Would it be possible or even likely that the source code or binaries or web servers or private keys have already been compromised or trojanized?

</comment><comment><USER18> Some thoughts: From the perspective of the company, what kind of financial impact would they have suffered should that information have fallen in different (malicious) hands? Providing an adequate award not only shows appreciation but it sets a precedent should something similar happen again. The 'finder' who might otherwise usually go the malicious route will be more likely to do as MentalL has.

</comment><comment><USER19> By submitting this information to BT, he indeed closed a access point to all this data, thus reducing the likelyhood of it being badly used. Isn't this exactly why bounties were created? Because its better to give a few bucks and increase the system's security than to have it compromised for worse costs.

</comment><comment><USER20> Stay classy!

<comment> cupcake-unicorn Indeed...This seems rather petty to me. They did offer him some money, which he's not even automatically entitled to in the first place...companies can chose whether they do bug bounties, or how much they pay.

<comment><USER21> You don't seem to understand that access to something that widely used would be worth tens of thousands of dollars on the black market for those sorts of things. For every "good" person that reports the information for free, there are people out there actively looking for this sort of thing that are not good people. If this company sets a precedent that they won't make it worth your while to "turn in the evidence", that sort of thing will just go to the highest bidder out there on the Internet and undoubtedly used for something extremely malicious.

tl;dr: small payouts and jerking this guy around sets a bad precedent that will encourage future people to sell on the black market.

</comment></comment></comment><comment><USER22> It's not the amount of money so much as the "so invoice us" attitude.

</comment><comment><USER23> whilst i agree they are ungrateful and inept i can't help but notice that you want a reward for doing the right thing?

get over it... life isn't this nice, you are entitled to nearly nothing. deal.</comment>


RePress - WordPress plugin turns your site into proxy for thePirateBay - <AUTHOR> http://all4xs.net/

<comment><USER2> Context, from the Dutch press release: the Dutch copyright organization ( _not quite_ as evil as MAFIAA) won a court case. Soon Dutch ISPs (including XS4ALL, started by hackers to bring internet to the masses) will be forced to block the Pirate Bay. A Dutch web host (<https://greenhost.nl/>) made this plugin to make it easier to set up proxies.

</comment><comment><USER3> Is this really an open proxy? Or do you configure it to proxy only certain websites?

<comment><USER4> You can configure it for any site, see some examples here: <https://all4xs.net/uncensored-website/>

<comment><USER3> Check out the error page :) <https://all4xs.net/repress/news.ycombinator.com>

</comment></comment></comment><comment><USER5> This title was not very inspiring for the HN crowd. It should have been:

"Magically uncensor any website on the internet from your own WordPress installation"</comment>


Huawei Signs Deal with African Union - <AUTHOR> https://www.ft.com/content/30ec5c54-83aa-11e9-b592-5fe435b57a3b <comment><USER2> for pw|lazy: [https://qz.com/africa/1632111/huawei-african-union-sign- deal...](https://qz.com/africa/1632111/huawei-african-union-sign-deal-to- boost-5g-ai-cloud-computing/)</comment>


Show HN: Simple CSS Button - <AUTHOR> https://vladocar.github.io/Simple-Button/ <comment><USER2> I'm genuinely interested wether this is a sarcastic submission or actually serious!

<comment><AUTHOR> It is a simple solution for very common problem: styling a CSS button. There are many CSS buttons libraries but they usually force you to use their style or size. That is why this is interesting and simple solution born with my necessity not to recode and reinvent the CSS button for every new project.

<comment><USER3> The "hard" part about styling buttons is making them look the same for , <button>, <input>, which is solved like a million times.

Your solution only addresses
, which is a accessibility nightmare.

</comment></comment></comment><comment><USER4> Pretty cool! Small thing: you should use button tags for the examples – you're losing a lot of built-in accessibility by using div tags (for instance keyboard navigation).

How does these handle being applied to anchor (a) tags? Some times you want links to appear as buttons :)

<comment><AUTHOR> It also works with (a) tags:  Button Link I need to add this in the examples.

</comment></comment><comment><USER5> This looks pretty cool, very simple. What is flexy.css? There is 1 button color? Feel free to submit it to my new GitHub repo curation site: [http://icodedis.tool.cards/](http://icodedis.tool.cards/)

<comment><AUTHOR> This is flexy: [https://github.com/vladocar/flexy](https://github.com/vladocar/flexy)

Yes, just 1 button color if you need more colors just add .btcolor1 , .btcolor2 ...</comment></comment>


Google's half-truths, and a plea for perspective - <AUTHOR> http://andreyf.tumblr.com/post/332303404/googles-half-truths-and-a-plea-for-perspective

<comment><USER2> What is this, an episode of Lie To Me? :-)

Government spies in the Shanghai office stealing Google's source tree seems to have been the straw that broke the camel's back. I guess China will just build their own Google now...

<comment><AUTHOR> _Government spies in the Shanghai office stealing Google's source tree seems to have been the straw that broke the camel's back. I guess China will just build their own Google now..._

The NYT article explicitly says the opposite: "attackers may have succeeded in penetrating elaborate computer security systems and obtaining crucial corporate data and software source code, though Google said it did not itself suffer losses of that kind". In general, that's not the language I hear them using - "broad", "sophisticated", and "targeting infrastructure of a variety of major corporations".

<comment><USER3> FWIW, Wikileaks is saying that they were 'infiltrating' the Google repos: <http://twitter.com/wikileaks/> (and specifically <http://twitter.com/wikileaks/status/7688415363> ). The NYT is simply parroting Google, so I give Wikileaks more credence.

(Colloquially, you can be 'stealing' something while haven't yet actually absconded with it yet, and be foiled if discovered.)

<comment><USER4> Wikileaks: "Several rumours from google sources that China accessed google's US-gov intercept system which provides gmail subjects/dates"

I'm remembering recent news about convenient gov't web portals to telecom records...

</comment></comment></comment></comment><comment><USER5> "How did Google find out the other companies were being targeted?"

Google runs data centers, and probably sees a lot more traffic than for its own services. Also, who is to say these companies are not using Google products (Gmail, Docs, etc.)? In other words, I think Google might be in a position to know.

<comment><USER6> Google supposedly hacked the hackers and found evidence that they had infiltrated the other companies.

<comment><USER7> Do you have any evidence for this?

<comment><USER6> It was in MacWorld's report and I've seen it a few other places, but no first hand information.

</comment></comment></comment></comment><comment><USER8> Here's my guess. Google initially detected an attack coming from Chinese IP ranges and subsequently checked what other requests the IPs had made.

This is what led to the accounts that were accessed not "through any security breach at Google, but most likely via phishing scams or malware placed on the users' computers".

Although they mention gmail in the statement, the expansion to other industries is made on the Google Enterprise blog (mainly aimed at promoting Google Apps to large companies), so it seems likely the same Chinese IPs accessed users of Google Apps.

So this isn't the Chinese government targeting other industries so much as dissidents that might work there. Google's statement still seems consistent to me, although they have shied away from mentioning Google Apps.

<comment><USER9> > and subsequently checked what other requests the IPs had made.

By how?

<comment><USER8> Gmail stores the last IP that accessed your account - it's shown at the bottom of your inbox:

Last account activity: 14 minutes ago at X.X.X.X

My guess is they used that.

<comment><USER10> Yes but how do they get the _other_ IP addresses that had been accessed without having that traffic go through Google?

<comment><USER11> That's the six hundred million dollar question in my mind.

How exactly did Google know that those other institutions were under attack?

Unless the attack resulted in rooted Google machines that then attacked outward, or all those institutions were running on Google Apps.

Major institutions running on Google Apps seems unlikely (Are _any_ major institutions outside of academia running Google Apps hosted by Google?). And attacking outward from a rooted box inside Google's network seems careless, unless the attackers were leveraging some dedicated pathways that made their job easier. Which suggests the breach was far more serious than indicated.

<comment><USER12> _Google's security team eventually managed to gain access to a server that was used to control the hacked systems, and discovered that it was not the only company to be hit._

<http://news.ycombinator.com/item?id=1050300>

</comment></comment></comment></comment></comment></comment><comment><USER13> I'd never heard of FACS before. The Wikipedia page he links to doesn't contain much info. Wikipedia says nothing about which AUs correspond to which emotions. Does anyone know any good articles on this? It's interesting.

<comment><AUTHOR> Paul Ekman is the guy who popularized it. Here is a good overview of his life/work: <http://www.youtube.com/watch?v=IA8nYZg4VnI>

Regarding reading expressions in particular, good start is "Emotions Revealed" [1], which was pretty comprehensive, but not as comprehensive (or as expensive) as his METT system: <http://face.paulekman.com/products.aspx> The half-smirk when talking about the other companies is contempt. The raised chin is shame. A lot of fear and sadness in the beginning, with exceptions of talking about the technical details of the attack and Google's desire to stay in China towards the end - lots of confidence there. If you watching him carefully with the sound off, I think you can pick up on the emotions intuitively.

In a more cynical light, this is good to keep in mind as well: <http://en.wikipedia.org/wiki/Facecrime>

1\. [http://www.amazon.com/Emotions-Revealed-Recognizing- Communic...](http://www.amazon.com/Emotions-Revealed-Recognizing- Communication-Emotional/dp/080507516X/)

</comment></comment><comment><USER14> I doubt that lie detection voodoo actually works when the person being interviewed is new to the whole being on TV thing.

<comment><USER15> That's pretty much accurate. Lie detection methods almost all base around detecting stress, and large amounts of stress (like being on TV for the first time) tend to throw them way off. In order to really have a baseline to compare lies against, you need to have a baseline. If they're not stable enough to do that, the science of it gets lost, and it's basically up to totally subjective methods.

</comment></comment><comment><USER16> A quote in Macworld suggests it's Google's law enforcement backdoor that was compromised. Google didn't mention that in their press release, which is a mighty big omission.

[http://www.macworld.co.uk/digitallifestyle/news/index.cfm?ne...](http://www.macworld.co.uk/digitallifestyle/news/index.cfm?newsid=28293)

</comment><comment><USER17> Andreyf, is this original content you have written or was it borrowed in another social experiment?

<http://news.ycombinator.com/item?id=1049430>

<comment><AUTHOR> All original, just for you :) I'd appreciate it if you didn't hijack this conversation into another 30-comment unrelated thread, though.

</comment></comment><comment><USER18> I read an interesting article yesterday about the news coverage of North Korea's drive towards nuclear weapons. Looking back on it now, it's obvious that the North started a nuclearization program in 1996 and pursued it each year no matter what was going on at the negotiating table. (Both political parties and the Clinton and Bush administration were taken to task in this article)

Looking at the press coverage, however, was even more interesting. There were some quite prominent columnists who insisted that the North was misunderstood, that they weren't committed to continuing to develop weapons, and that the various tactics used by both administrations were really what was at the root of the North's belligerence.

Sometimes pleas for perspective can be good, and sometimes they can all be so much noise. I'm not going to characterize this piece, but I'll note that there is a great similarity between how NK was covered with nukes and how China is being covered with CyberWarfare.

I find it disconcerting.

<comment><AUTHOR> Oops, is "plea for perspective" not the right phrase? I meant to say that a coordinated attack on a variety (probably all) major US corporations might be _more_ significant than what Google does with their .cn properties...

<comment><USER19> If that was your goal, your reference to "google half truths" was a big distraction. If what they do with their cn properties is less signifigant, then don't devote so many words to them, and instead focus on your message.

</comment></comment></comment><comment><USER20> just speculation, Adobe might be on the list: <http://www.dailytech.com/article.aspx?newsid=17387>

<comment><AUTHOR> Hm, the article seems to jump between "Adobe was attacked" and "Adobe software was used as attack vector", in a way that inspires the least of confidence. _sigh_

<comment><USER20> They were definitely attacked: [http://blogs.adobe.com/conversations/2010/01/adobe_investiga...](http://blogs.adobe.com/conversations/2010/01/adobe_investigates_corporate_n.html)

Both could be true

</comment></comment></comment><comment><USER21> I don't think you could ever mine a chief council's interview for much information. However, human rights activists are certainly targeted but you are way off base to suggest companys are not.

I am in a rush so I can't pull up a story I recent heard about where a foreign business man was brought into custody. A large portion of billionaires in China are children of party officials that were installed in those positions.

The party very much wants to stay in power in China and they are not only worried about idealistic, which I am sympathetic to, and not so wealth opposition. They are worried about business leaders and all evidence indicates they are keeping an eye on them as well.</comment>


OS.js – JavaScript Cloud/Web Desktop Platform - <AUTHOR> https://www.os-js.org/ <comment><USER2> This is progressed a bit. Love how you can play castle wolfenstein.</comment>


The men who stare at airline passengers - <AUTHOR> http://www.economist.com/blogs/gulliver/2010/06/does_airport_security_deception_detection_work <comment><USER2> _He says he now avoids peer-reviewed journals because they're read closely by scientists in countries that America considers to be threats._

Wow. Terror as an excuse for draconian policy, I have come to expect. This is the first time I am seeing terror as an excuse for flawed and unscientific 'research'.

<comment><USER3> The military has its own classified, peer-reviewed journals to provide an outlet for people with legitimate analysis that they'd like to keep out of the hands of foreign agents.

His excuse is remarkably thin.

</comment></comment><comment><USER4> A few years ago when I was about to board a flight to London from Boston, I was grabbed by the arm by a plains-clothed security type immediately before the airline scanned my boarding pass at the jetway.

He flashed his badge and I was pulled to one side, and he asked if his partner could look through my carry on. As she did this, he asked me about where I had been eating in Boston. He asked many, many questions about Santarpio's, saying he'd never heard of it, asking what they sold, etc.

Given that Santarpio's is immediately next to the airport and sells amazing pizza, it's extremely unlikely that someone who worked at Logan was unfamiliar with it.

At this point, the lady looking at my bag said "he's clean" or similar, the first guy lost all interest in Santarpio's, and they put me back in to the front of the queue.

<comment><USER5> That's that joint in Eastie, right?

<comment><USER4> It's basically the last non-airport thing you see before you get to the airport (erm, depending on which of the 4 or so approaches you use to get to the airport, I suppose).

EDIT: it's really good and worth a visit. Get there early and/or be prepared to queue.

<comment><USER5> Cool, thanks! Will do :) When I'm in Boston, I stay in Southie, so I'll have to get over there.

</comment></comment></comment></comment><comment><USER6> Kudos to the person who came up with that title. Those of you who haven't seen the similarly titled movie or read the book are probably baffled, but I assure you, it very nicely sums up the ridiculousness of this particular program of the TSA (a cynic might even go so far as to say it describes the TSA as a whole, but none of us here are cynics, right? Right? Guys...?)

<comment><USER7> The economist has always had a good eye for titles and frontpages. Some of them are just amazing.

<comment><USER8> Also their image captions are nicely done.

As a student of English as a second language, I find it rewarding whenever I spot the references.

</comment></comment></comment><comment><USER9> This reminds me of part of Stephen Jay Gould's _The Mismeasure of Man_ , which talks about "trained" employees at Staten Island (or other points of entry for immigrants) in the nineteenth century, who could spot mental deficiency by looking at the shape of the head (craniometry, but without measurement, just eyeballing). I'm writing this from memory, so I can't go into detail, and I don't have a link to this particular passage. But a significant number of immigrants were put on boats back to Europe just because of the snap judgment of these spotters.

<http://en.wikipedia.org/wiki/The_Mismeasure_of_Man>

</comment><comment><USER10> Paul Ekman's METT and SETT Microexpression Training Tool and Subtle Expression Training Tool, have been in use by many government agencies not just the TSA. <http://www.humintell.com/about-us/>

From a washington post article: _"But the day I spent at Logan confirmed for me that SPOT violates no one's civil rights. Few people were identified. Nearly always, the answers to initial questions made further investigation unnecessary. No record was made, and the passenger lost no time.

Observational techniques are not a substitute for all the other techniques we now use to catch would-be terrorists. But they add another layer to transportation security. They are now being used at fewer than one in 10 major U.S. airports. We need to use them everywhere."_ How to Spot a Terrorist on the Fly [http://www.washingtonpost.com/wp- dyn/content/article/2006/10...](http://www.washingtonpost.com/wp- dyn/content/article/2006/10/27/AR2006102701478.html)

<comment><USER11> "But they add another layer to transportation security."

That's not the point. The question is whether the efficiency of that layer in any way justifies the cost and hassle.

</comment></comment><comment><USER12> Kind of like the 'divining rod for explosives' used in Iraq: [http://www.nytimes.com/2009/11/04/world/middleeast/04sensors...](http://www.nytimes.com/2009/11/04/world/middleeast/04sensors.html)

<comment><USER13> To figure this out all you have to do is follow the money. :) I love the last part of the story:

 _During an interview on Tuesday, General Jabiri challenged a Times reporter to test the ADE 651, placing a grenade and a machine pistol in plain view in his office. Despite two attempts, the wand did not detect the weapons when used by the reporter but did so each time it was used by a policeman.

"You need more training," the general said._

</comment></comment><comment><USER14> _...would achieve similar hit rates if they flipped a coin..._

 _...No scientific evidence exists to support..._

 _...the TSA is "unsure" whether SPOT has ever led to the arrest of an actual, real-life terrorist._

Which probably makes this program fit in perfectly with everything else that TSA is doing.

<comment><USER15> Israel puts heavy emphasis on human behavioral screening and it's fantastically successful. It seems SPOT is a heavily watered down version. I'm sure all the expensive technology vendors will love to point out SPOT is a failure.

<http://www.forward.com/articles/122781/>

Thankfully we have very few attempted attacks in America so we don't have the will to implement racial profiling.

If it gets ever gets bad enough we will and it will probably work.

<comment><USER16> I wouldn't equate the Israeli security with racial profiling. That's a part of it but it mainly involves asking passengers questions and looking for things that are out of the ordinary.

I got stopped leaving Israel once and it wasn't because of racial profiling. I gave them my family's phone number and they called around and it all got resolved.

</comment></comment></comment><comment><USER17> I'd like to know how much money is invested in this.

Since inception the DHS has been a huge money pit of questionable efficacy. This level of incompetence smells like corruption to me.

40 Billion dollars in funding for 2010. <http://www.wallstats.com/deathandtaxes/>

Realizing the costs associated with TSA security theater are frustrating. We all fund it.

</comment><comment><USER18> I have a vision of a bunch of officials in ill-fitting uniforms giving passengers the Larry David Stare.

</comment><comment><USER19> It reminds me of China, where they have armed guards at train stations "randomly" questioning anyone they "think" might not have the necessary documents to be moving to (living in) that particular city.

</comment><comment><USER20> The interesting thing about this technique is that unlike most security techniques it actually might work _better_ the more the enemy knows about it.

Imagine the expression on the face of the terrorist as he's thinking "Must not make suspicious-looking micro-facial movements... must not make suspicious- looking micro-facial movements..."</comment>


Why are there so few women programmers in the software industry? (2015) - <AUTHOR> http://chaource.livejournal.com/131649.html <comment><USER2> Sweden collect statistics on peoples job titles as a part of the process of collecting taxes, and based on that I find it slightly odd that there is such focus on programming. Only about 10% of professions is considered gender "balanced", and a profession with 80% balance is quite in the middle. There is a bunch of professions which has 90%+, with several that has 99% of a single gender (primarily female professions, with some exceptions). The top one, never spoken about in gender politics, had about 4000 working females professionals to 1 male.

Since the article provided their personal theory, I will provide mine. I do not think its the work activity itself that matter, but that a person who picks between multiple choices of potential future professions, the deciding factor when everything else has been considered is the amount of people with similar life experience that one is likely to find. If you are a young parent, you have a minor preference for places which employ a lot of other young parents of the same gender. When everything else is equal, such preference can have a huge impact on gender statistics.

</comment><comment><USER3> Odd article. Lots of interesting history in there, then the explanation is simply "Today's programming resembles design and construction of large mechanisms a lot more than word processing. The majority of women just don't find these things as interesting as men do."

Seems like the author did a lot of research then just jumped to conjecture for his conclusion.

<comment><USER4> There's a lot of that in there:

> Today's programming resembles design and construction of large mechanisms a > lot more than word processing. The majority of women just don't find these > things as interesting as men do.

Do the majority of men find those things interesting? Do women like word processing and menial tasks?

Is it possible that as the job of software engineer garnered more money and status, women were shut out?

I don't know, but this article doesn't answer these questions.

</comment></comment><comment><USER5> The article provides a lot of interesting questions, but occasionally without enough data to answer them. But maybe the questions are a good starting point, from the perspective of logical deduction.

On an additional note, I find it odd that software always gets so much focus: for example, 99 percent of construction jobs are held by males in many countries (and a large portion of these jobs do not involve heavy physical labor, if that is a concern - i.e. heavy machinery operators, foremen (forewomen?) etc ).

[http://jobs.aol.com/articles/2010/07/27/where-women- work/](http://jobs.aol.com/articles/2010/07/27/where-women-work/)

See link above. I think we might be asking the wrong question, focusing on too narrow of a field. The better question might be, why do certain fields get dominated by women, and certain fields are devoid of them? I think there is a much broader hypothesis here to be tackled then one concerning a single field.

</comment><comment><USER6> The premise of this article and many like are based on flawed data. They only take into account the people with actual CS related degrees working in the industry. When in reality you have people with everything from English Lit. to psychology, to no degrees at all.

A more interesting study to me would be why so many "programers" seem to exhibit traits commonly associated with OCD or Asperger syndrome(some times both). Wired even called it "Geek syndrome" [http://www.wired.com/2001/12/aspergers/](http://www.wired.com/2001/12/aspergers/)

</comment><comment><USER7> Hmm, immediately flagged off the front page.

Burn the heretic!

</comment><comment><USER8> I find that Java things like Ant or Maven bring tedium back to software, but not quite the same way that punch cards did.</comment>


Neil Harbisson - the World's first cyborg - <AUTHOR> http://en.wikipedia.org/wiki/Neil_Harbisson

<comment><USER2> This seems more like a stunt. The device has no direct interface to human biology, nor does it seem very practical. The "officially recognized as a cyborg by a government" seems bit fishy too. It sounds like government recognized the device as prosthesis, which isn't that unusual.

<http://en.wikipedia.org/wiki/Visual_prosthetic> is far more interesting imho, especially when a close friend of mine is losing his eyesight. To be actually able to tap into nervous system opens quite a lot of doors, both capturing data, and sending data.</comment>


What I don't like in your repo - <AUTHOR> https://medium.com/@olegkovalov/what-i-dont-like-in-your-repo-a602577a526b <comment><USER2> If you don't like my repo, you don't have to use it. :)

If you wish to add tests and a Dockerfile, feel free to open a PR. I don't use either and am not going to waste my valuable personal time making them to suit your desires.

Open source developers should not work for free, so they should implement what they personally are enriched by doing, and let you do what you wish to do, or pay them to do what you wish done.

</comment><comment><USER3> That's a good list. Pretty sensible advice. I wish all programmer would follow the README advice. With all that said was it really necessary to add the same emoji at the end of every point ?

<comment><AUTHOR> Thank you, emoji was added for no reason, article have been updated.

</comment></comment><comment><USER4> having "recent version" and "CI status" in the readme seems a tad unnecessary to me. No commits should be merged to `master` which fail CI, and the releases are only one click away anyway.

depending on the type of project, it can certainly be very helpful to have a link to a working demo. But presumably having a dockerfile wouldn't be worth the hassle if it were automatically possible for docker to use the config environment specified for CI and run the latest release?</comment>


Meet the 'Mann' who registered 14,962 domains in 24 hours - <AUTHOR> http://news.cnet.com/8301-1023_3-57418250-93/meet-the-mann-who-registered-14962-domains-in-24-hours/ <comment><USER2> Does anyone know good and recent research on the value of domains? Thinking back to the Read Write Web & FB login disaster a few years back, it seems like the merger of the search & address bar would make the names much less valuable.

That said, perhaps they're still valuable for the credibility provided?

</comment><comment><USER3> DUPE

       https://news.ycombinator.com/item?id=3873741 332 days ago | 91 comments</comment>


Goodhart's Law and how systems are shaped by the metrics you chase - <AUTHOR> https://whyisthisinteresting.substack.com/p/why-is-this-interesting-the-goodharts <comment><USER3> One of the best examples of Goodhart's Law I think is what has happened over the past 20 years in sports like gymnastics and figure skating.

The International Skating Union changed the figure skating scoring system after the 2002 Olympics scandal. The goal was to make the system more transparent and objective. Instead of a couple of 6.0 values, each element in a program would get a score based on difficulty and quality of execution.

The end result is that things that are essentially easily measured (like the number of revolutions in a jump) went up in value, while many feel like the overall quality and artistry of skating has suffered greatly. I'm not going to argue about how the "artistry" has suffered, even though I agree; artistry is subjective, and it is supposed to be a sport, after all. What I think is tragic about the scoring system, though, especially in women's figure skating, is that it is greatly biased towards _girls_ , not women. The top "women" skaters these days are a trio of 16 year olds from Russia who excel at some of the quadruple jumps that were rare even among the men just a decade or so ago.

The issue is that it's nearly impossible for a fully developed woman (i.e. with breasts and hips) to do quadruple jumps - the physics just make it extremely difficult. So, sadly in my opinion, it has turned ladies figure skating into girls figure skating. Gone are the old days where you could follow a skater and see her improve and mature over multiple Olympics - <USER2> it's basically become a sport of mastering the difficult jumps and quads as quickly as possible until physical development eventually takes over.

I see more of this in society in general now, where with so much data and analytics there is a rush to "measure" everything. But I think it's important to acknowledge what is lost when "subjective" becomes a dirty word.

<comment><USER4> Thank you for the last paragraph.

I feel that a lot of people have this assumption, unconscious or not, that the world is entirely objective. That everything can be measured, and what isn't measurable doesn't exist. I think this belief (which essentially elevates the scientific method to a religion) is causing a huge amount of grief in the world.

<comment><USER5> Agreed. We had a doctor who refused to believe that my partner's symptoms were relieved by a specific procedure - because there were no reports of other people's symptoms being relieved by this procedure. But by this logic, no such report could ever get recorded!

It is critical that subjective, experiential data is cross referenced with objective data in order to control for errors in both. Both kinds of data are fallible for different reasons.

In my opinion, the erosion of subjectivity from science is probably a driver of the loss of objectivity in politics.

<comment><USER6> I'd say it is the reverse. Less influence of science in politics is a driver of loss of objectivity in politics.

<comment><USER5> Well, ISTM that science has had a pretty good run in politics, especially relative to the rest of history - but now, particularly in the US, science based governance is well and truly out the door.

Why is this? In my opinion it's because science has rejected people's subjective experience for too long, to the point where science has become just "someone else's opinion". I can think of three examples From the past decade, off the top of my head.

Gluten make you feel yukky? Nonsense, said Science, only people with celiac disease can be gluten intolerant... oh wait, there are other types of gluten intolerance? Who knew?

Qualms about GM? Nonsense, said Science, what we're doing is perfectly safe... oh wait, GM crops can unexpectedly spread into non GM paddocks, ruining a bunch of farmers? Who knew?

Prefer organic food? Nonsense says Science, pesticides are safe and harmless... oh wait, all the bees are dying from Neonicotinoids? Who knew?

It's not that there is anything wrong with the scientific method, but scientists who run around and demean people's subjective view of the world - right or wrong - have IMO really contributed to the current crisis of faith in science. People's subjective opinion and experience must necessarily be taken into account, since that's part of their lived experience and what we are here to explain. You don't have to believe everything people tell you, but you do have to listen and be tolerant.

It's like a special application of Goodhart's law, "if we haven't already seen it, it doesn't exist". As a huge fan of the scientific method, this attitude makes me very sad.

<comment><USER6> I think the main reason is that science has simply become too hard to be explained in laymans terms in a way that you can still connect all the dots. Even plenty of scientists are now so narrow in their knowledge that within the same domain they won't be able to keep up with their peers if it isn't their exact specialization.

It's logical: in the renaissance a single individual could still hold 'all of science' in their heads, by the early 1900's that had split up into a whole bunch of domains each of which a single individual could still comprehend in their entirety. By the mid 1970s I don't think any single scientist still had complete command of their domain and that has only gotten worse.

It was bound to happen sooner or later, and this is not a sad thing per se, more of a measure of how far we have come and how quickly we've done that. The scientific method is the most amazing invention we've done, it succeeded where everything else has failed at explaining how the world and in fact much of the universe works.

</comment></comment></comment></comment></comment></comment><comment><USER7> Short, well-written, and interesting.

One question that popped in my mind as I was reading this was, how about alpha, correlation, and volatility in financial markets? Has the behavior of markets changed over time as alpha, correlation, and volatility have become dominant metrics to be chased?

<comment><USER8> A lot more has changed in the financial markets than just the way perofrmance is measured and which alphas are known.

Some examples (not in chronological order, mostly US equities focused): electronic market makers, electronic order execution (which means large orders are harder to detect now), retail access, the way retail flow is directed around, ZIRP + QE, the rise of indexing, Volcker Rule.

My point is that it would be quite hard to isolate the impact of just how people think of performance.

<comment><USER7> Thank you. Yes, I agree.

My question was only whether the behavior of the whole system has been shaped by the chasing of these metrics.

Your comment implies that you think the answer is yes, even if you won't venture a guess of the impact of these metrics in dependent of other changes (understandably, in my view; I'm not sure I would venture a guess either).

</comment></comment></comment><comment><USER9> There was this concept of objectives and counter objectives which was meant to combat Goodhart's law. The idea was that you define what you want to measure and then define another metric which acts as regularization for the first. A good example is if you define a metric of WAU only, you might be motivated to bring in as many new users each week as possible. Setting a counter metric of retention makes you take into consideration that it's important they stay.

I don't remember who suggested this - I vaguely remember it was someone at a16z, but can't find the original place I read it

<comment><USER10> _There was this concept of objectives and counter objectives which was meant to combat Goodhart's law._

Metrics and counter-metrics, IIRC originally coined by Julie Zhuo.

It's a great framework for making sure that you consider Goodhart's law, but it's also only as good as the person thinking about it.

The problem I have seen in practice is that the same person whose job it is to think of the counter-metric (usually the PM) often looks better and gets promoted/paid more when the real costs of chasing the KPI stay hidden. PM leadership doesn't have the bandwidth to make sure every PM is being rigorous.

I've felt this pressure as a PM (I don't think I gave into the temptation but that's not for me to judge) and as a developer I've seen my PM invest only nominal time on the counter-metric, where arguably you should be spending _more_ time thinking about it than the metric itself. In practice, that resulted in things like security-through-annoyance-and-unreliability.

I've been in those shoes, so I know defining metrics to track business goals is really hard. But I do think people on the ground can tell when it's a matter of difficulty as opposed to a matter of cutting corners or storytelling spin.

<comment><USER11> > security-through-annoyance-and-unreliability

What's that? How does it work / What does it mean?

</comment></comment></comment><comment><USER12> I love this, but it struck me in the opening paragraph how similar it is to the challenges around training AI on higher-level abstraction goals, with tons of examples of AI's basically "cheating" and passing information discretely to itself to yield answers, rather than actually solving the problem as the human had hoped!

</comment><comment><USER6> You can see this in the Netherlands: people are actively discouraged from filing reports of criminal activity resulting in lower crime on paper but rising (small) crime in real life. Car burglaries, bike thefts, pickpocketing shoplifting and so on are strongly subjected to this pressure.

</comment><comment><USER13> This is an excellent example of Goodhart's Law in practice.

Every time it comes up, I'm reminded of a Dune quote: "The problem isn't thinking machines, it's letting the machines do the thinking for you."

IMO the Law comes into effect as people stop also paying attention to the externalities in favor of "letting the [metric] do the thinking for them.".

</comment><comment><USER14> We don't even need a fictional nail-factory example, because an almost-similar real example exists: the Backyard Furnaces of China during the Great Leap Forward [1]

> Pots, pans, and other metal artifacts were requisitioned to supply the > "scrap" for the furnaces so that the wildly optimistic production targets > could be met.

[1] [https://en.wikipedia.org/wiki/Great_Leap_Forward#Backyard_fu...](https://en.wikipedia.org/wiki/Great_Leap_Forward#Backyard_furnaces)

</comment><comment><USER15> We need to rate the performance of our economy not by how much we produce, but by how little.

<comment><USER16> A better metric would be something related to happiness. If we produce less and and GDP goes down but everyone is happier then who cares?

<comment><USER17> There are many ways happiness can be manipulated though. People tend to be happy when things are improving. The most obvious way to manipulate this metric is to deliberately hit rock bottom and then slowly improve from there.

Another problem is drugs. It's possible to manipulate people's happiness rather explicitly with that.

Edit: these are out there examples, but here's a more realistic one: borrow money from future generations. Set up a system that will eventually fail, but provide a lot of benefits/happiness in the present. When the long-term consequences arrive politics have already changed.

<comment><USER18> Drugs: yup, Huxley's dystopian "Brave new world" scores remarkably high on a subjective wellbeing scale. I think this is a sort of obvious limit to the validity of the metric though; it doesn't invalidate the metric within sensible limits (though we could have a chat about antidepressants and adhd I suppose)

On your last point, Wales legislated for wellbeing as a guiding principle in the Well-being for Future Generations Act which incorporates that concern.

My other issues with wellbeing are ...

1\. it's shaped by expectations. There are well known cases of people who have horrific life changing accidents yet after adapting to their new lifestyle are just as happy as before. We can also imagine very well off people being picky because the champagne and caviar are not up to standard.

2\. we know how to measure it at individual level but don't have enough data on how we as a society think we should trade off between average levels of wellbeing and inequality of the same.

More on all this on my link in sibling comment. I'm interested to hear feedback on all the above as we're working on the next stage of this project now.

</comment></comment></comment></comment><comment><USER19> What gets measured gets treasured.

alternate form: The uncountable is of no account

</comment><comment><USER20> >But some of these chiefs started to figure out, wait a minute, the person who's in charge of actually keeping track of the crime in my neighborhood is me. And so if they couldn't make crime go down, they just would stop reporting crime. And they found all these different ways to do it. You could refuse to take crime reports from victims, you could write down different things than what had actually happened. You could literally just throw paperwork away. And so that guy would survive that CompStat meeting, he'd get his promotion, and then when the next guy showed up, the number that he had to beat was the number that a cheater had set. And so he had to cheat a little bit more.

Why does the story end here? Why not put someone else in charge of taking crime reports?

</comment><comment><USER21> While many here would hate it, Goodharts law is a powerful argument for things being obtus, vague and abstract. The less measurable something is, the less it can be easily hacked.

Also, interestingly, what will stand in the way of AGI - no definable loss function for these vague cases and scenarios.

<comment> swagasaurus-rex Approval of managers in charge of payroll is a very obtuse, vague and abstract thing.

It can also be quite demoralizing and damaging as an incentive structure, even to very productive employees.

<comment><USER22> Approval of managers should, IMO, be the main way employee performance is measured.

If it's damaging then you've got a bad manager. Either his manager or HR training is at fault. Both can be prevented with the right systems in place.

</comment></comment></comment><comment><USER23> So the sample story was about how the police achieved their metrics through unsavory methods because it was too hard/too much work to do it the "correct" way?

Is the problem then how you go about achieving the metrics? Let's say that you always pair metrics with an approval process for how you plan to achieve those metrics. Presumably when the police chief gives a presentation on how he plans to reduce crime rate by throwing out crime reports, the committee would be none too pleased.

Does that theoretically handle the majority of cases where enacting metrics has adverse effects, or does it fall on the other side of 80/20? Or maybe actors are always so inclined to lie and cheat that there's no point in even exploring such an approach.

<comment><USER24> Presumably the optimal solution would be to prepare fake but easily executed plans (tuned to appeal to whoever's on the committee) to go with your fake numbers. But probably no amount of regulating police will help if your regulators are also chasing metrics (say, to make themselves look better than their opponent for voters).

Random sampling is one effective (if adversarial) way to validate numbers. For policing I guess that looks like "secret citizens" reporting crimes and sending the results back up the chain? But that doesn't solve the second half of the problem.

<comment><USER25> You optimize what you measure, don't you? It is crucial to that any set of metrics considers this effect. DOne right, a metrics system points an organisation is the "right" direction and sets priorities for everone. Done wrong, and the organisation will run, sometimes ruthlessly efficient, in the wrong direction.

For policing, just counting the number of arrests and tickets written (something German police is guilty of to certain degree), pushes investigations and police work to pursueing "easy" crimes. And sometimes harassing people for minor stuff to meet numbers. No idea what could be a better way, absennce of issues is so hard to quantify and faces the same risk of optimizing for one thing at the cost of another.

</comment></comment></comment><comment><USER26> in some sense the history of the SAT examinations is a deep example of Goodhart at work.

</comment><comment><USER27> Im curious what "deeper interventions" look like? What metrics systems work?

<comment><USER28> a) Pick a single measure that is very very hard to game (life expectancy, for example). But that's non-trivial, even something like life expectancy can be gamed by deciding who's included in the stats, by changing the start date (e.g. for cancer diagnosis), by ignoring quality of life, etc.

b) Use a robust mix of different measures that is harder to game. Continuing the example of the above, one common metric is the "disability-adjusted life year", which combines quality of life, length of life, economic productivity, and subjective ratings of happiness

c) Socialize your workers to believe in "the cause", so they are more likely to do what you mean rather than what you measure (hard to do when you're not a governmental or a political organization). One of the big motives of early Soviet purges (back when they just involved kicking someone out of the Party) was to remove people for which this didn't work.

d) Not exactly "fair", but often effective - investigate and punish people when they do counterproductive things to optimize their stats. This was _another_ reason for said early purges of the CPSU membership rolls.

These are all _hard_.

</comment></comment><comment><USER29> I personally don't believe Goodhart's Law overpowers the benefit of the metric, ever.

It's a quaint factoid, but just as the soviets would just execute a factory manage gaming the system for producing pins or spikes, people will see you at some level gaming the system and call you on it. (and make a TV show like The Wire)

Just like other factoids (ie Risk compensation), the effects will be real, but the factoid is so misused at extreme interpretations the factoids do more harm the good.

<comment><USER30> Instead of downvoting you I thought I'd ask, how likely do you think people do see you gaming and don't call you on it? I've been in many more situations where you don't have the power or ability to call someone out on something because it's more complicated than just "I'll tell on you".

<comment><USER29> It's your boss not being fooled by your gaming and not promoting you.

Its not something you report.

Like taking a long lunch break.

People aren't blind, they can see what you are doing, people will talk about it behind your back.

Goodharts law relies on a mythical assumption everyone in the organisation is just out to scam it.

It's all measured in real life. If people are evil and out to game metrics why won't they just game the system when there are no metrics?

A good version of Goodharts law might be, people will work towards metrics so some times it's worth tweaking metrics to reduce incorrect outcomes, but sometimes worrying about the inefficiency is worse than the inefficiency itself.

<comment><USER31> I think your last paragraph gets to the point.

Goodhart's law is not about people "gaming" the system. Sure that could be one of the effects of aligning incentives to metrics, but it's really about about the dangers of aligning _goals_ to metrics (of course incentives are often tied to achieving goals).

For most non-trivial systems it can be difficult to correctly identify and describe all of the variables that affect the system and in what ways. For example, at a high level a company's singular goal may be something as abstract as "make more money". It's pretty difficult to just _do_ that, so we break it down into smaller, more concrete goals that we feel will contribute to the above. How? Good news! Last quarter we hired a business analyst who started recording detailed metrics about our business and processes. Now we can simply look at the metrics from last quarter and try to improve upon some key numbers.

And this is where Goodhart's law comes in. You see last quarter we were not _trying_ to improve upon any specific numbers. Although we can see that we achieved poor efficiency in our inventory processes last quarter, it turns out that focusing on improving this metric actually net lost us money because it created more inefficiency in shipping!

No one has to be "gaming" the system for the above to occur. A system full of good actors can fall prey to turning metrics into goals (at the expense of the system as a whole). Goodhart's law is just a warning. Of course we still want to measure things!

<comment><USER32> If it isn't a goal (or a subgoal) then why measure it? Groups have many goals but what usually happens is that certain goals are prioritized.

Unfortunately, because of incentives, the anointed goals (or the metrics) get over prioritized causing worsening externalities until the goals or metrics get adjusted to correct the externalities. "Move fast and break things" turns into "Move fast with stable infrastructure".

Of course this is a dialectic, so expect the goals and metrics to shift again as new externalities are exploited. The ultimate correcting mechanism is a god who isn't a slave to the incentive structure and who resets the system by getting rid of bad actors.

The world is the Matrix.

<comment><USER33> > If it isn't a goal (or a subgoal) then why measure it?

Often the goals are hard to measure, so you measure a proxy. This can work well if people are actively focused on the goal, and check the measure to see how they're doing. It stops working if people forget that the measure isn't the goal, and work to increase the measure.</comment></comment></comment></comment></comment></comment>