Continuous Compliance - <AUTHOR> http://www.davefarley.net/?p=285 <comment><USER2> I suspect most of HN hasn't ever worked in a compliance-driven, regulatory environment. They've never had to deal with the possibility of audits by government, insurance, or financial regulators.

This is the world I live in - building pipelines for financial and health care giants. I agree with this article 100%. Not only _can_ continuous delivery provide regulatory compliance consistency, but also more traditional human- bureaucracy approaches _cannot_ provide it. There are too many potential leaky spots, and far too much potential for human error - in particular, people signing off on things they don't understand. And this _includes_ compliance auditors!

<comment><USER3> And when someone asks you for proof that some check that you happen to have automated was run, you get to ask "and how many reams of paper's worth of proof would you like?" instead of saying "umm, uhhh..." and rummaging around trying to figure out what you can send.

<comment><USER2> I love being able to show the lines of code that say "This code cannot advance to a prod-deployable state if the tests haven't passed, because here's where it gets rejected".

<comment><USER4> The problem is that you're often showing those lines of code to someone who became an auditor because they're incapable of authoring or even _understanding_ what those lines of code are doing.

<comment><USER2> Or you're showing them to technical management, architects, etc, who are quite capable of understanding their importance and see how those lines benefit them, the ones on the front lines for compliance issues.

</comment></comment></comment></comment></comment><comment><USER5> >> What are the goals of Regulatory Compliance? All of the regulatory regimes that I have seen are, in essence, focussed on two things: 1) Trying to encourage a professional, high-quality, safe approach to making change. 2) Providing an audit-trail to allow for some degree of oversight and problem- finding after a failure.

No and no. The goal of regulatory compliance is to avoid liability. You maintain professional standards because that helps keep everyone safe, but you obey the rules to avoid the punishment associated with not obeying them.

This really matters when regulatory compliance conflicts with good judgement. Sometimes you do the bad thing because the bad thing is mandated in the rules. If a regulation says that you have to have "antivirus software" on the machine then you have antivirus software on the machine.... even if no antivirus software exists for that machine. You shoehorn something because the rules say you need it. You don't do this to increase security. You do it because the lawyers tell you that not doing it will get you sued.

<comment><USER4> Which leads to things like anti-virus software being installed and configured to not run. Or anti-virus software being selected based on the ability of the software to be so configured.

<comment><USER5> The most common issue is normally password managment. Lots of little regs say things like "passwords must be changes every 30 days", which might have made sense decades ago but is very anachronistic today. (I have an alert at the bottom of my screen right now saying my password expires next week.)

</comment></comment></comment><comment><USER6> In general I agree -- but it means your pipelines will have to support every customer you deliver to and be version-aware.

Part 11 of the FDA regulations that oversee software compliance enforce expensive re-validation processes for every major release of software. It's not something your customers are going to want to do very often. So set up a CD for each customer on version X.y.z where X is static and make sure that you don't accidentally ship a backwards-incompatible major-version change on that pipeline.

It's an interesting challenge for operations-focused teams but I agree that CD is a valuable tool to have.

<comment><USER7> This is a major problem and one of the artifacts of the reg not having been updated since 1997. The industry expectation for the validation process is for it to take four to six months, and it's specific to one version. We have 99.99% of our software requirements specification testing covered by several thousand automated tests that run in several minutes. Nonetheless, auditors have been wary of frequent updates, even though by their own admission our test strategy is far superior to any manual testing strategy and, critically, is the only realistic way to monitor for regressions specific to our usage of software that the vendor might not catch.

<comment><USER6> I don't disagree. The validation process is hilariously outdated as are the compliance requirements for software vendors. But it's ambiguous enough in places that we can adapt _some_ modern operations tooling and practices to it.

</comment></comment></comment><comment><USER8> In my experience, putting CD into place makes a significant improvement to the SDLC only when automated unit tests are a part of that life cycle. I have never seen the benefit of implementing a system like Jenkins if the only automated testing portion is 'does it compile without error? Yes? Then, good to go!'. Without the automated unit tests it just doesn't seem like a worthwhile endeavor to me.

<comment><USER2> Unit tests are hardly the only kind of testing needed. First, they're completely decontextualized (hopefully), so they don't show the code working in any sort of broader context. Second, they only show the code is consistent with itself, not consistent with requirements. More to the point, unit tests are tests by programmers for programmers, rather than by business for business.

For any mildly complex modern app, you need integration tests to show context, and behavioral/functional tests (Cucumber) to show the code actually does what it's supposed to do according to the customer.

<comment><USER9> > they don't show the code working in any sort of broader context

If you're testing a REST interface, then how much additional context you need? They're supposed to be stateless, so they can be tested in isolation just fine, because that's how they're supposed to operate. Or does it stop being a unit test once your test includes DB interaction?

For your typical web app, front end testing is the primary place you need anything more complicated, and even then you could debate how much of that should be manual.

<comment><USER2> Do you test timeouts? Connection drops? REST itself may be stateless, but it exists in a complex and failure-prone environment with numerous moving parts. What's the routing like between interfaces? Are there proxies? Do you control their configuration? Are your certs properly managed?

"Unit tests should be sufficient" is a fancy way of saying "Throw it over the wall and pray it works".

<comment><USER9> Most of that sounds more like application performance monitoring than pipeline testing.

<comment><USER2> Most of that sounds like things that can be tested for, rather than just tossing code over the wall and waiting to see if it catches fire.

<comment><USER9> Where's this "throw it over the wall" stuff coming from? I write tests, if I push and my pipeline fails, I have to fix it. I'm rostered on call, if my app breaks, I have to get up in the middle of the night.

For a simple web app (which is probably what most engineers on HN work on), unit testing is most of the testing you probably need to do (unless you think that a unit test that involves a DB interaction becomes an integration test). Front end code needs a little bit more than that, but automating front end tests is not particularly reliable to begin with, so you could really debate how much of that should be done by hand anyway. Some people swear by BDD frameworks, but that's how you end up with a 4 hour test pipeline, without getting all that much additional confidence in your work.

<comment><USER2> The original discussion is about working in regulated environments like health care and banking, not youtubeforcats.com. BDD frameworks and four hour test pipelines can start sounding pretty good after things reach a certain level of complexity.

I used to work on a federal banking system that moved a hundred billion dollars a DAY. Believe me, unit testing is _not_ sufficient for that kind of world.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> " In fact it is quite hard to imagine a Pipeline that doesn't give you access to this information"

I have seen multiple pipelines that don't tell you who manually tested this change - or at least, don't _always_ tell you. The author here is assuming the existence of a lot of practices already, which makes his pointing to CD as magic look a lot weaker.

</comment><comment><USER11> We have had a lot of success using Inspec and for a while Chef compliance in validating the infrastructure elements remain compliant continually.

For instance, you can check a folder remains encrypted or certain ports remain closed in dev and test before promoting your config management to production.

Quite involved to setup but a big tick for auditors.

<comment><USER12> Years back, I was brought in as a consultant for a large online provider of automated loan application systems to provide Chef automation for confirming that the CIS-CAT controls had been properly applied.

This was for a firm that handled over 80% of all online loan applications, from all the major banks, insurance companies, credit unions, etc.... From the moment you typed in the first character of your name to when you hit the final submit button, everything was handled on their site, and proxied through to the customer systems via an iframe or other technology.

I tried to convince them that they should have automated compliance testing and confirmation on a regular and frequent basis (like every thirty minutes), but they refused. They said it wasn't necessary according to their auditor, and they weren't going to open themselves up to additional risk by having the tool run too frequently.

They had quarterly audits, and therefore they wanted the tool to only be run quarterly, and to only be kicked off manually at that.

While us normal human beings might be convinced of the need and desire to always be protected, but banks and financial institutions don't think like that. For them, they only ever do the absolute minimum required by law, because anything more just creates more risk.

</comment></comment><comment><USER13> Your link is not working:

 _Resource Limit Is Reached The website is temporarily unable to service your request as it exceeded resource limit. Please try again later._</comment>


Small project's life depends on his owner - <AUTHOR> http://www.nicosphere.net/small-projects-life-depends-on-his-owner/

<comment><USER2> There have been attempts to organize this, including one I know off the top of my head[1].

There are two problems. One is that an orderly transfer of control requires a certain amount of planning and coordination, which are precisely the resources that aren't available when a maintainer has left (or is on the way out). The other is that people tend to become maintainers of other peoples' projects by climbing the open source 'career ladder' from user to minor contributor to major contributor; projects are stable when they have enough contributors that the most-productive non-maintainer can be promoted to maintainer status when the original author leaves.

GitHub helps by making it easy and obvious to fork something and eventually have your repo promoted to be the 'official' one, but as with many problems in open source, finding a particular, specific person who actually does that is the hard part.

[1]<http://thechangelog.com/post/1986814704/stillmaintained>

<comment><USER3> Ironically, <http://stillmaintained.com> is closing down.

</comment></comment><comment><USER4> This would be a nice feature for github. Also, additionally, if there was an abandoned open source project that had no activity on it after a certain time period, and the project owner went AWOL, a user could petition to take it over, get X votes and it would be auto transferred over. The project owner could stop the process simply by logging into their user account.

</comment><comment><USER5> Isn't that exactly why GitHub is great? Love it, fork it, maintain it!

</comment><comment><USER6> I really like this idea of an orphanage for abandoned projects.

</comment><comment><USER3> Debian keeps track of packages that need new maintainers. With the large amount of software they package, it seems like the Debian project would be in a good position to create a similar list of packages where the upstream has become inactive.

<http://www.debian.org/devel/wnpp/></comment>


Ask HN: Will the net-neutrality be looking to throttle illegal downloading? - <AUTHOR>

It occurred to me, probably a lot later than it should, that this will be a nuclear weapon against freedom of speech and illegal file-sharing. Is this possible? <comment><USER2> Net neutrality is nearly, but not entirely, orthogonal to efforts to control _unlawful_ internet activity; its about neutrality among _lawful_ uses.

</comment><comment><USER3> It's more about ISPs acting like cable companies where you are forced to pay for different service tiers. If you want to watch youtube or netflix you have to pay extra otherwise they will throttle your connection to those sites. A lot of ISPs where I am (not in US) already throttle default torrent ports for most of the popular clients like utorrent/bitTorrent so p2p file-sharing is already a target.

<comment><AUTHOR> I'm not a fan of south park (a popular comedy show in the US) but the creators Trey Parker and Matt Stone are pretty smart guys. They will never censor content for anyone. If they do, it will lead to a slippery slope where others will demand the same privilege. If we give up even an inch on this then pages and files will become so latent they won't even be worth accessing.

</comment></comment><comment><USER4> No. If anything, I would expect net neutrality to benefit pirates.</comment>


The worsening journalistic disgrace at Wired (2010) - <AUTHOR> http://www.salon.com/2010/12/27/wired_5/

<comment><USER2> Salon criticizing the integrity of Wired? That's truly ironic. Salon is maybe one step up from a blog of ghost-writers.</comment>


Hacked. Hard - <AUTHOR> http://www.loopinsight.com/2012/08/04/hacked-hard/

<comment><USER2> Original post was posted here --- <http://news.ycombinator.com/item?id=4337938> \--- with 271 comments so far, currently #13 on the front page.</comment>


"Transducers" by Rich Hickey at Strange Loop [video] - <AUTHOR> https://www.youtube.com/watch?v=6mTbuzafcII <comment><USER2> I enjoyed the discussion of the types. I dig haskell (and clojure), and I think this is perhaps the perfect lens with which to view how to make choices between them. You can have an insanely complex typesafe haskell transducer, a still very complex but unsafe haskell transducer, a weaker and less flexible version of transducers with a simpler type encoding in haskell, some combination of those ideas, OR...

you just test your code out in the repl while developing in clojure and just kinda rely on the fact that core infrastructure or popular libraries will generally work.

<comment><USER3> Personally, I don't think the gist posted by tel qualifies as "insanely complex" by any stretch.

But a more serious question is: how do you know if "will generally work" applies to your particular case? The short answer is that you don't and you end up having to test library-provided functionality as part of your own test suite. You can argue that you'd have to do the same in e.g. Haskell, but really the surface area of potential failures is _hugely_ reduced by the fact that you _know_ whether a function "f: a -> b" can have side effects and that given something shaped "a" it _will_ give you something shaped "b" (modulo termination, but that applies in any non-total language).

This is not merely a theoretical issue: Compare the amount of documentation you need when using some random library in Haskell vs. e.g. JavaScript. The types act as compiler-checked documentation. In JavaScript it's really a crapshoot if the library has sufficient documentation and it's always specified in an ad-hoc manner (which naturally differs from library to library).

<comment><USER2> I think if you sit down with the code clojure uses to implement transducers and compare it directly against the haskell code and _don't_ come to the conclusion that the clojure code is simpler, there might be some motivated cognition at work. I don't think it's trivial for someone with Haskell experience to even verify that code implements the same (or a similar) thing.

I also believe it lacks sufficient polymorphism, for instance surrounding the output of the step function, and lacks safety around (notably, but not exclusively) the application of the step function (i.e. to only the last value of the transducer, not just something of that type). So this would be squarely in the tries-to-be-simple category, despite it's use of profunctors (don't know why that was used here, it's not a super-standard abstraction).

But this is all beside the larger point. Things generally working is learned through philosophical induction in the case of clojure -- just seeing something work a bunch of times and understanding the mechanisms in some level of detail. That's not the same as having a machine-verified proof, but it's also not the same as not knowing at all.

<comment><USER3> It depends on what you mean by "simpler", I suppose.

> there might be some motivated cognition at work.

Did you really just say that?

> I don't think it's trivial for someone with Haskell experience to even > verify that code implements the same (or a similar) thing.

No, not to verify that it does the same thing. For that you'd have to understand exactly what the Clojure version does too. I'm a quite rusty on Clojure, so I can't make a fair comparison on how easy it is to understand vs. the Haskell version. However, you'll note that I didn't actually say anything about the Clojure version being harder to understand.

In fact, my point is that I don't even _have_ to understand the implementation of the Haskell version: I just have to understand its interface (i.e. the types) and have a general idea of what it's supposed to do (in fuzzy terms).

</comment></comment></comment></comment><comment><USER4> First, to be clear, I really liked this presentation. The criticism below is both technical and small---all in all I greatly enjoy Rich Hickey's work and generally admire his ability to talk compellingly about complex technical topics.

That said.

I somewhat disliked Hickey's presentation of typing transducers here. I feel as though he builds a number of strawmen toward typing and then tries to knock them down and suggest that either Clojure has some kind of mystical mechanism that is ineffable to types or that the exercise of typing transducers is wasteful. I disagree on both accounts, I suppose. I think types are useful for analysis and teaching.

The two major points he seems to make is that in order to "properly type" transducers you must

        1. Index the type of the "accumulation so far" so
           that it cannot be transformed out-of-order
        2. Implement early stopping "without wrapping anything
           except for the reduced value"

There may be other critiques as well, but I want to examine these two in the context of Haskell.

With respect to the first point, the major concern appears to be prohibiting behavior loosely described as "applying the reducing function, say, 3 times and then returning the first resulting accumulation". In some sense, the idea is to force us to be faithful in passing on the accumulating parameter. In code, a pathological setting is the following:

        transduce :: (r -> a -> r) -> (r -> a -> r)
        transduce reduce accu0 a =
          let acc1 = reduce acc0 a
              acc2 = reduce acc1 a
              acc3 = reduce acc2 a
          in  acc1

The concern is unfounded in a pure language, however, since calling `reduce` can have no side effects. This entails that all possible effects on the world of calling `reduce` are encapsulated in the return and, therefore, we can completely eliminate the steps producing `acc2` and `acc3` without worry.

        transduce :: (r -> a -> r) -> (r -> a -> r)
        transduce reduce accu0 a =
          let acc1 = reduce acc0 a
          in  acc1

Now, there may be concern here that we still want to index the `r` type somehow to allow for changes of accumulation to occur. This is not the case (in _this_ simple model!) as in order to achieve the "baggage carrier independence" property the `r` type must be left unspecified until the transducer is actually applied. The cleanest way to do that is to use a higher-rank type (Hickey mentions these briefly and offhandedly toward the end of his talk)

        type Transducer a b = forall r . (r -> b -> r) -> (r -> a -> r)

which thus prohibits the implementer of a Transducer from affecting the values of `r` in any way whatsoever---they must be left anonymous until someone decides to _use_ the Tranducer on a particular collection of values `a`.

(It must be noted that the model given above is isomorphic to a "Kleisli arrow on the list monad" which I described a little bit here [http://jspha.com/posts/typing-transducers/](http://jspha.com/posts/typing- transducers/). It should also be noted that this model includes neither (a) the ability to use local state to capture time-varying transductions or (b) the ability to terminate early)

With respect to the second point, I'd like to suggest that there is a difference between the semantic weight of wrapping the result types in an Either in order to indicate early termination and the implementation weight. I completely agree that using an Either to implement early stopping (as it's easy, if finicky for the library implementor, to do) will involve wrapping and unwrapping the "state" of the transduction continuously. I also would like to suggest that it's a very natural way of representing the "accumulation | reduction" notion Hickey uses in his own "Omnigraffle 8000 type system".

We really would like to capture the idea of the transducer state as being "either" in-progress or fully-reduced and act accordingly. If Clojure's implementation of that requires fewer runtime tags than an Either, so be it, but I personally fail to see a semantic difference except in the way one can play fast-and-loose with dynamic types over static types to begin with.

\---

So, I gave above an implementation of Transducers in types which has some of their properties, but certainly not all. In fact, I abused the fact that there is no ambient state in Haskell in order to ensure that a certain property would hold (notably this doesn't require a type system at all, just purity). I also argued that using Either is a perfectly natural way to implement early termination in such a transduction pipeline.

I've also made an extension to the `(r->b->r) -> (r->a->r)` mechanism which enables local state to be enabled for various components of the transduction pipeline. A version without early termination is available here:

[https://gist.github.com/tel/714a5ea2e015d918f135](https://gist.github.com/tel/714a5ea2e015d918f135)

Notably, this uses most of the same typing tricks as `(r->b->r) -> (r->a->r)` but adds a "reduction local hidden state" variable which lets us implement `take` and `partition`. This takes Hickey's notion of needing to be explicit about the state being used to a whole new level.

\---

So what is the point of all this?

I'd like to argue that Transducers do not present such a mysterious mechanism that they cannot be sanely typed in a reasonably rich language. I believe that I can capture most of their salient features in types without using the dependent indexing Hickey suggested was necessary.

More than this, the compartmentalized, hidden reducer-local state in the Gist implementation allows for each reduction step to include fairly exotic local states in their state machine. You could implement a kind of type indexing here if desired and no end-user would ever know of its existence.

I also absolutely concede that many type systems people regularly use could not achieve this kind of encoding.

Finally, what I really want to say is that type systems are not something to be denigrated. I believe some of the earliest "transducers v. types" argumentation took a nasty turn as amateur type theorists (myself included) rushed to write things like "Transducers are just X".

I want to apologize for any kind of bad feelings my own writing in that thread may have stirred up. I try not to be haughty or dismissive with this kind of writing, but I also make mistakes.

So what I'd really like to suggest is that types should not be taken as reductivist on interesting techniques like Transducers but instead as a tool for analyzing their construction and either improving it or better teaching it. Hickey himself often turns to some kind of "pseudotyping" to talk about how Transducers work---formalizing those notions should only lead to greater clarity.

Of course, implementations will differ in small ways. As I've noted abundantly here, a major difference between the Haskell and Clojure implementations is driven more by Haskell's purity than its typing. Hopefully, however, exploration of alternative implementations and the rich analysis produced by their typing can help to introduce new ideas.

For instance, the Gist implementation, if you strip the types away, is an interesting divergence in raw functionality from Clojure Transducers---if Clojure Transducers are "reduction function transformers" than the Gisted Transducers are "Moore-style (Infinite) State Machine transformers" and that difference allows the implementer to be extra explicit about the use of local state.

I'd rather see discussion about whether such InFSM transformation techniques have a place in Transducers literature than a fight over whether or not its possible or reasonable to "type transducers".

<comment><USER5> > higher-rank type

Im not a type person. So here is my question, how many languages other then haskell have this?

<comment><USER6> Any dynamic language should do, kind of :)

For example, the following code needs Higher Ranked Types to typecheck:

        id :: forall t. t -> t
        id x = x

        -- mk_pair's type is a rank-2 type, because its
        -- f parameter is a polymorphic function that gets applied
        -- to different types inside mk_pair.
        mk_pair :: (forall t. t -> t) -> (Int, String)
        mk_pair f = (f 10, f "str")

        a_pair :: (Int, String)
        a_pair = mk_pair id

But it can obviously be written in an untyped language if you wanted to. Just erase all those type annotations:

        function id(x){ return x}

        function mk_pair(f){ return [f(10), f("hello")] }

        var a_pair = mk_pair(id);

The deal with higher-ranked types is basically a tradeoff between the flexibility of the type system and the power of type inference. If you restrict yourself to rank-1 types (that is, none of your function arguments are polymorphic) then the compiler can infer all all types in the program without you having to write any type signatures. If you want to use higher ranked types then the compiler can't infer everything anymore so you might need to add some type annotations yourself.

<comment><USER4> The "kind of" being really pertinent. In particular, this forgoes the guarantee that the only thing the transducer is allowed to do is return a value from the reduction. Hickey notes that this is a law of writing a proper transducer. Higher rank types ensure that this law is properly encoded directly into the system.

That said, I'm pretty sure you know this :)

</comment></comment></comment></comment><comment><USER7> Correcting two flaws with this leads to an interesting result: 1) no error handling, at all, anywhere, not just missing from the presentation, but not in the code. transduce just wraps reduce. FAIL!!! 2) the 'result' parameter unnecessarily pollutes the interface. reduce can be reduced to foreach (doseq) with state and be implemented like other tranducers that require state.

Correcting for these two errors: a) the 0-arity init function is removed b) the 1-arity completed function becomes 0-arity c) the 2-arity reducing function becomes 1-arity d) a 1-arity error function is added. Since these functions cannot be distinguished by arity alone, we give them names: call the reducing function 'onNext', the error function 'onError', and the completed function 'onCompleted', and optionally, group the three functions into an object and voila, we have Rx Observers.

Hickey's twist here is composing Observers instead of Observables. Whether this buys you anything worthwhile is debatable.

Two derivations of Rx often accompany it's introduction: 1) Observable being the dual of Iterable 2) Observables being Promises with multiple return values. Thanks to Hickey, we can add Observers being an abstraction from reduce/fold (along with it's many other names).

</comment><comment><USER8> Great talk by Rich on transducers, instrumental in understanding the "hows" and "whys" behind the concept.

</comment><comment><USER9> I'm still a little confused and will have to go over some code or something to really understand the limitations of what transducers can do... can any transducer be used in a "parallel" context (like map and filter) or are they limited to a linear context (like the fold makes me suspect)?

<comment><USER10> I think it could be parallelized by chunking up the input, however the transducer doesn't know how to do that by itself, so the step function would have to be modified somehow.

<comment><USER11> Exactly! core.async had the pipeline mechanism added which allows for the parallel application of a transducer to a channel.

[https://github.com/clojure/core.async/blob/17112aca9b07ebba6...](https://github.com/clojure/core.async/blob/17112aca9b07ebba6ce760ca01d117c24c80cc9a/src/main/clojure/clojure/core/async.clj#L500)

</comment></comment></comment><comment><USER12> Removing conj is what finally made it click

</comment><comment><USER13> I threw together a toy implementation in Lua:

[https://gist.github.com/scythe/d28c3f4933ff2f1e5c47](https://gist.github.com/scythe/d28c3f4933ff2f1e5c47)

Granted, none of the cool out-of-order-iteration is there, but the reverse composition looks natural to me now, so I can sleep at night.

<comment><USER14> Took me a while to understand the reverse composition too so I hacked on it in Python

[https://gist.github.com/colinsmith/3140e7cb57c4095ed83f](https://gist.github.com/colinsmith/3140e7cb57c4095ed83f)

</comment></comment><comment><USER15> Transducers really sound like monads. Are they the same thing?

<comment><USER4> In almost every meaning of that sentence—no.

One simplified model of transducer happens to be similar to half of a particular monad. It's very disingenuous to say that "transducers are monads" or even "transducers are a particular form of monad", however.

<comment><USER15> :( it was an honest question. I really didn't know. I watched the talk. I didn't get everything that was said, so I came here to ask a question. Man, HN is use to too much snark.

<comment><USER4> Oh! Sorry, I may have written that poorly. I meant that technically: "in _almost_ every sense, no". There is one sense in which "yes" they kind of are: a simplified model can be seen as

        Transducer a b ~ (a -> [b])

where the right side is a pretty fundamental type when talking about the list monad. In fact, composition of transducers is exactly "monad-arrow composition" or "composition in the list monad Kleisli category".

So in that sense, they are a particular form of monad. But it's a bit of a stretch.

Thus, I really meant it in "almost every way", not every way entirely.

<comment><USER15> So by what you wrote, it sounds like they're related, but a completely separate concept. Where can I find an intro to transducers that's readable for beginners? I don't know what "list monad Kleisli category" means. Searching for "transducers" came up with a physical transducer.

<comment><USER4> If you know what monads are then you may know of

        join :: Monad m => m (m a) -> m a

Since list is a monad then we know that join can be specialized to

        join :: [[a]] -> [a]

which is just `concat` then. Now Transducers are a bit of an elaboration over functions like

        Transducer in out === in -> [out]

and so we might compose them a little like

        comp t1 t2 in = concat (map t2 (t1 in))

which if you follow in types ends up working out. Finally, that concat/map business is just monadic bind in the list monad. We could also write

        comp t1 t2 in = t1 in >>= t2
                      = (concatMap t1 t2)
                      = bind t2 (t1 in)

depending on what syntax is most familiar. That's why they're (an elaboration of) "monad composition", i.e. a Kleisli category.

The only thing left is just that Hickey did a transformation called "Church Encoding" or "Continuation Passing Style" to the regular list functions. This eliminates the need to generate concrete lists and turns this fancy composition into reverse normal function composition. It's a clever trick.

</comment></comment></comment></comment></comment></comment><comment><USER16> I think somebody just reinvented data flow programming.

<comment><USER17> yeah, is this going to become a 'thing'? i'm finding it really kind of annoying that terms from the fundamentals of theoretical & applied compsci ([weighted] finite-state transducers WFSTs) are being appropriated like this.</comment></comment>


FBI Contracts Cellebrite to Break into San Bernardino iPhone - <AUTHOR> https://www.fpds.gov/ezsearch/fpdsportal?amp;templateName=1.4.4&q=cellebrite+CONTRACTING_AGENCY_NAME%3A%22FEDERAL+BUREAU+OF+INVESTIGATION%22+PIID%3A%22DJF161200P0004424%22&sortBy=SIGNED_DATE&s=FPDSNG.COM&indexName=awardfull&desc=Y&&templateName=1.4&indexName=awardfull <comment><USER2> I suspect that contract is unrelated. Details of the contract are visible at:

[https://www.fpds.gov/common/jsp/LaunchWebPage.jsp?command=ex...](https://www.fpds.gov/common/jsp/LaunchWebPage.jsp?command=execute&requestid=66872289&version=1.4)

Specifically, the "description of requirement" says:

        IGF::OT::IGF  UFED software renewals for seven machines.

Which sounds like a routine license renewal... unless, of course, they're deliberately being misleading with that line item.

<comment><USER3> Looks like that link has a transient id and returns a blank page..

Real link here: [https://www.fpds.gov/ezsearch/jsp/viewLinkController.jsp?age...](https://www.fpds.gov/ezsearch/jsp/viewLinkController.jsp?agencyID=1549&PIID=DJF161200P0004424&modNumber=0&transactionNumber=0&idvAgencyID=&idvPIID=&actionSource=searchScreen&actionCode=&documentVersion=1.4&contractType=AWARD&docType=B)

Agreed that this doesn't look like the actual phone contract. Cellebrite runs contracts for the DOJ every few weeks and that amount is quite low:

[https://www.fpds.gov/ezsearch/fpdsportal?q=cellebrite+CONTRA...](https://www.fpds.gov/ezsearch/fpdsportal?q=cellebrite+CONTRACTING_AGENCY_NAME%3A%22FEDERAL+BUREAU+OF+INVESTIGATION%22+VENDOR_FULL_NAME%3A%22CELLEBRITE+USA+CORP%22&s=FPDSNG.COM&templateName=1.4&indexName=awardfull&x=0&y=0&sortBy=SIGNED_DATE&desc=Y)

Wouldn't be surprising if the DOJ had called them up, though, and also wouldn't be surprising if any purchase order surrounding the case was sealed for reasons of national security.

</comment></comment><comment><USER4> Is $15k a weirdly low number for this, given the importance the FBI has attributed to it? Anyone with similar contracting experience who can comment?

<comment><USER5> Well, from the FBI's perspective, they may already know the data on that work phone is worthless, but they just pretended it's not to get the precedent against Apple.

From Cellebrite's perspective, they should've probably asked more like $1.5 million to solve "such a valuable case", as the FBI has made it to be in public (and if this is indeed that contract, which it may not be). Apple also helped there by talking about how much of a burden it would be to unlock it.

<comment><USER6> Truth is the community knows how much of a relative non-burden it is and the contract was probably bid upon.

</comment></comment></comment><comment><USER7> Danny Yadron, a reporter for the Gaurdian, recently tweeted "Reliable sources with a variety of biases saying Cellebrite rumor is bunk."</comment>


Apple Looks to Streamline Clarification of Awkward Autocorrect Messages - <AUTHOR> http://www.macrumors.com/2016/04/29/apple-autocorrect-messages-patent/ <comment><USER2> how about it just determines you didn't mean that, and correct it before the message is even sent? Machine learning should be able to do this rather easily.</comment>


"Counterterrorism in the airport is a show designed to make people feel better" - <AUTHOR> http://www.theatlantic.com/doc/200811/airport-security <comment><USER3> Remember to consider the incentives of the 535 individuals who vote on funding the TSA year after year.

It doesn't matter if it provides any security. It keeps us sheeple under the illusion of safety, we keep buying plane tickets and a whole economy springs up surrounding the airport (not to mention the firms supplying the TSA with equipment). That's good for a ton of voter approval. Even better, if you don't vote for the TSA your opponent can call you out: 'Senator so-and-so wants our innocent children to die in a fire at the hands of scary men with beards!'

Yes. I'm jaded. I know.

</comment><comment><USER4> No matter how many times I try, the security guards never let me take toothpaste or bottled water through the gate. They also had a tendency to confiscate my hair gel when i used to gel it up.

However, I was able to take my laptop + two batteries + 360 (and cables) + two external hard drives (and all the cables) + router + other misc electronic stuff through as carry on (Since I didn't trust the TSA not to steal it out of my suitcase when flying home and all) without any issues. Not even a question or odd look.

</comment><comment><USER5> So many more people die in car accidents than in terrorist attacks - I would say that the feeling of safety in flight is not false in any account. The article has it completely backwards - the airport security theater is what creates the false feeling of unsafety.

<comment><USER6> planes are the safest way to travel PERIOD. What other method of travel gets as much safety testing? And has a second pilot just in case the first one is knocked out.

From 1999 to 2007 there's been a total of 11,000 deaths from plane crashes world wide. With an average of 1,200 deaths per year.

Car accidents? According to world health organization, there are a total of 1.2 million deaths world wide per YEAR. In addition to 48 million injuries per year.

So in 2007 when there were 965 deaths from plane crashes, you were 1,243 times more likely to die in a car accident than you were in a plane crash.

</comment></comment><comment><USER7> I've been saying this for years. I routinely just stick my toiletries in my pockets to avoid having to do that stupid little baggie. I used to fly a pretty decent amount (50k miles per year) and never once had a problem that way.

Also, I messed around with cheap homemade plastic explosives in high school enough to know that the amount you could legitimately stick in plastic tubes and then cram into a ziploc bag is more than enough to cause some serious damage.

Luckily we're all safe because very few people actually want to kill a plane full of strangers, and even fewer want to kill themselves too.

<comment><USER8> I flew recently, at Gatwick they say "no containers of liquids bigger than 100ml". At the other end they said "no more than 1L of liquids in total". But, if you were working in a team, you could get any amount of any liquid you wanted aboard.

<comment><USER9> True, but working in teams is hard. The larger the team must be, the harder it is to recruit, train, organize, supply, maintain confidentiality, keep out spies, and avoid detection (hence the cell structure). Forcing terrorists to work in teams significantly increases the cost of conducting an attack, so it's an excellent security strategy.

<comment><USER7> That's an interesting thought. You're probably right. I imagine doubling the number of terrorists required probably more than doubles the chance of them getting caught. Might be exponential.

Unfortunately, one ziploc bag full of plastic explosive is probably more than enough to blast a hole in the side of a plane. And one of those beer belly things I would think could take a whole plane out.

</comment></comment></comment></comment><comment><USER10> You say it like it's a bad thing. It works very well - people feel safe.

<comment><USER11> Does it? Everyone I know, and a few strangers I've met while taking off my shoes for the X-ray machine, _hates_ airline security and doesn't really believe that it helps keep anyone safe.

<comment><USER10> It does. Proof - if it didn't make majority happy it would be scrapped.

You are not a typical user of the system. You also hang out with like-minded people which gives you illusion of being normal which ain't true - <USER2> people aren't reading HN, aren't starting startups and aren't working deeply immeresed into unforgivingly precize digital machines.

<comment><USER11> > Proof - if it didn't make majority happy it would be scrapped.

That's not a proof at all. Representative democracy doesn't mean the majority gets its way on every issue - otherwise, it's a fairly good bet that CEO pay would be capped, the Wall Street bailout would never have happened, and we wouldn't still be in Iraq. Rather, it means that when the government screws up on _enough_ issues and pisses _enough_ people off, we get to throw them out of office.

Granted, my experience isn't a proof either, for the reasons you point out. (Although - talking to random strangers in line for the checkpoint is about as close to a relevant random sample as you can get.) But then, I didn't claim it was.

</comment></comment></comment></comment><comment><USER12> If it is true, it means that counterterrorism in the airport actually decreases security, as it decreases motivitation to find real solutions to the problem.

</comment><comment><USER13> Surely it's also at least partly a show in order to make terrorists feel like they won't get away with it?

It's easy to complain about airport security, but you gotta admit: terrorists seem to have more-or-less given up on targeting aeroplanes.

<comment><USER14> Is that because of the show or some actual, intelligent counterterrorist work?

<comment><USER15> Or is it because it would be stupid of them to target airoplains again. As the article mentioned, they are watching against what happened last weeks.</comment></comment></comment>


Corinthian to close its last schools; 10,000 students in state displaced - <AUTHOR> http://www.latimes.com/business/la-fi-corinthian-shutdown-20150427-story.html#page=1

<comment><USER2> "schools" and "students"</comment>


Launch of the Biggest Rocket Ever From the West Coast Today - <AUTHOR> http://spaceflightnow.com/delta/d352/status.html

<comment><USER2> Is there a live feed anywhere? I love to watch this thing launch in real time!

<comment><AUTHOR> This might be it: <http://www.ulalaunch.com/site/pages/Flash_Webcast.shtml> (Use the "Launch" links on the right.)

FaceBook page: <http://www.facebook.com/30thSpaceWing>

Also, live text updates here: <http://twitter.com/#!/spaceflightnow/></comment></comment>


Ask HN: What are some biz-friendly US cities similar to Brighton or Bristol, UK? - <AUTHOR>

I'm scouting out some locations for a 2-3 person US office for my UK online orientated business.
For comparison, I've found secondary cities in England like Brighton or Bristol to be particular good for setting up a small business because:
- Good universities and lots of graduates milling around after university - attractive cities to live with good life/work/urban/rural balance - Strong local freelancer/small business scene and entrepreneurial vibe.
- Relatively cheap wages and office rents compared to capital cities. - Shortish commute to nearby major city when needed.
So which US cities would you consider to be in the same kind of category? <comment><USER2> Take a look at the top 100 places to live. This is a great place to start your hunt, you can exclude some obvious cities and do some additional research for cities that sound interesting. Anything East of the Mississippi is very easy to travel around in, you can plan a road trip vacation and visit visit all the cities that caught your interest.

[http://livability.com/best- places/top-100/2015?page=1](http://livability.com/best- places/top-100/2015?page=1)

I can vouch for #13 and #96 both being great places and pretty affordable, however Im not sure how strong the free lancer scene is. #13 is a little higher on the arts scale, but #96 is Beer City USA... Cant beat that

<comment><AUTHOR> that's a great place to start. Thanks very much</comment></comment>


Douglas Crockford: The JSON Saga [video] - <AUTHOR> http://www.yuiblog.com/blog/2009/08/11/video-crockford-json/ <comment><USER2> 40 minutes into this talk is the best bit (watch the 3 minutes between 40" and 43" even if you don't care about javascript or JSON). My shouted question from the audience is inaudible, but they included a transcript:

\</comment><comment>--

 _When I put the reference implementation onto the website, I needed to put a software license on it. I looked up all the licenses that are available, and there were a lot of them. I decided the one I liked the best was the MIT license, which was a notice that you would put on your source, and it would say: "you're allowed to use this for any purpose you want, just leave the notice in the source, and don't sue me." I love that license, it's really good._

 _But this was late in 2002, we'd just started the War On Terror, and we were going after the evil-doers with the President, and the Vice-President, and I felt like I need to do my part._

 _[laughter]_

 _So I added one more line to my license, which was: "The Software should be used for Good, not Evil." I thought I'd done my job. About once a year I'll get a letter from a crank who says: "I should have a right to use it for evil!"_

 _[laughter]_

 _"I'm not going to use it until you change your license!" Or they'll write to me and say: "How do I know if it's evil or not? I don't think it's evil, but someone else might think it's evil, so I'm not going to use it." Great, it's working. My license works, I'm stopping the evil doers!_

 _Audience member: If you ask for a separate license, can you use it for evil?_

 _Douglas: That's an interesting point. Also about once a year, I get a letter from a lawyer, every year a different lawyer, at a company – I don't want to embarrass the company by saying their name, so I'll just say their initials – IBM…_

 _[laughter]_

 _…saying that they want to use something I wrote. Because I put this on everything I write, now. They want to use something that I wrote in something that they wrote, and they were pretty sure they weren't going to use it for evil, but they couldn't say for sure about their customers. So could I give them a special license for that?_

 _Of course. So I wrote back – this happened literally two weeks ago – "I give permission for IBM, its customers, partners, and minions, to use JSLint for evil."_

 _[laughter and applause]_

 _And the attorney wrote back and said: "Thanks very much, Douglas!"_

</comment><comment><USER3> "Fortunately nobody listens to Dave Winer"

Thankfully that's been true most of the time -- a world in which anyone but him used Frontier is not one in which I want to live. But a few of his clusterfucks did slip out: XML-RPC, SOAP, his epic mismanagement of RSS (which only ended with a headshot from Atom).

The man singlehandedly set the web's progress back about a decade! If not for him, the nightmare of WS-* would not have existed -- the likes of Microsoft and IBM would have done something stupid in a big way, but they wouldn't have pretended it had anything to do with the web.

</comment><comment><USER4> Netscape 4 was indeed a crime against humanity. Great video and very informative. I always enjoy Douglas Crockford's talks.</comment>


The Social Graph as Crude Oil  - <AUTHOR> http://www.forbes.com/sites/venkateshrao/2011/10/21/the-social-graph-as-crude-oil-go-ahead-build-that-yasn/

<comment><USER2> a</comment>


Spy access to NZ used as bargaining tool - <AUTHOR> http://nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=10910701 <comment><USER2>
        [former CIA director and National Security Agency director
        General Michael Hayden] was quoted as saying: "Because of
        the nature of global telecommunications, we are playing
        with a tremendous home-field advantage, and we need to
        exploit that edge.

I really hope the rest of the World understands the implications here and works to eliminate the home field advantage for the US and makes sure that no other country ever gets this advantage. No single country should ever have that power.

</comment><comment><USER3> Gods, could the NSA get any more short-sighted? Once they started tapping into all of this data, how long did they think they could keep it secret, especially when tens of thousands of contractors knew about the program? Once the news leaked, what did they think other countries were going to do?

Whenever people in power setup perfect systems of abuse, the people being abused will change that system by going out-of-band.

<comment><USER4> This is the problem with having so much unchecked power. It's extremely tempting to abuse it and take full advantage of the benefits it gives you, and not think of the consequences _years_ down the road, because you know you have the whole government backing you (even if it's unconstitutional) - at least at the time.

<comment><USER5> It's not even abuse in the sense that it will be done with evil intent.. It is, in fact, a _natural consequence_ that even good people will do 'bad things' when put in the position.

Imagine the following: You, mtgx, are put in charge of protecting the mid- Atlantic states from an attack. You have been given complete access to all of the legally obtained intelligence that the nation has about the region and potential conspirators for said attack.

Do you _not_ use it? What are your options? If you don't use it, and the attacks are successful, it will be your fault. You will have to live with the knowledge that your inaction, for whatever higher purpose, cost those lives.

I posit that even good people like you would take all of the steps you have available to do your job. Even if by day you are writing letters to the editors begging the nation to take those powers away from you.

For a reak example, look no farther than what happened in the case of the Boston Marathon bomber Tamerlan Tsarnaev. It turns out that we had a lot of intelligence on him; the Russians even alerted us that he was trouble. And yet, we did not feel we had enough evidence to invade his privacy. We didn't stop him (this is why I think that the people doing this are basically trying to do the right thing). And when we found out after the bombing that this could have been stopped? The shit hit the fan: People were declaring the FBI/CIA incompetent, etc.

The main reason to act on this privacy issue is to help the good people who have to do these jobs, not to stop evil people. The rules are now such that good people have to do evil things in order to be good.

We want to keep this in mind so that we solve the right problem.

</comment></comment></comment><comment><USER6> _" It seems that we now have the choice between taking the time to understand and implement secure encryption or choosing services based on which governments we don't mind spying on us."_

I prefer the first of those two options but it'll only have an impact if more people make the same choice.

</comment><comment><USER7> Is this why they killed Pacific Fibre?

<comment><USER8> I think the official line is that having investment from both China and the US was considered a dealbreaker by the US side, and since the US wouldn't fund it completely, the deal couldn't go through. Guess this explains why.

I'm a resident but not anyone important, so clarifications welcome

<comment><USER9> China investment meant a preference for Chinese equipment/vendors, which in turn would mean getting a landing in the USA would be very hard, essentially impossible.

The primary contract was signed with a US vendor, TE Subcom (actually HQ is in Switzerland for tax reasons), and whether the Chinese vendor could even do the job aside even from landing rights was a big question.

Funding from any source never amounted to enough to get it away. Which is a shame as the business case was - and still is - compelling.

Source: me - a Pacific Fibre founder

<comment><USER7> I still don't understand what legal justification the US has for making it hard to get landing rights in the USA just because you use Chinese equipment.

If I ever have a spare $300mm or so, I know what I'm funding.

<comment><USER10> If there weren't any existing, it could make some. There are some very protectionist policies in the US and if you have the lobbying power you can get even more. That said, how many local industries aren't protected by their country's government? Some protection is good, but finding the balance can be a little tricky.

</comment></comment></comment></comment></comment><comment><USER11> So McCain says today US needs to start poking digits(pun?) into Putin's eye, and this NZ fragment leads me to wonder what fist or carrot incentive makes every(?) country in the world pipe their domestic surveillance back to US? Now, how then, do you apply all the world's intel to acquire whatever you want? What new US intel agency with an eagle logo orchestrates the `parallel construction' to `git 'er done'?

</comment><comment><USER12> The way things are going, I wouldn't be surprised if they bundled it in as a secret requirement for international trade deals.

</comment><comment><USER13> This wouldn't happen under a Labour led government... 2006 oh shi..

<comment><USER10> Clarke wasn't perfect, but Key's willingness to stoop to any level for the rich or powerful is shameless. Pandering to the US (Afghanistan, Dotcom), Sky city deal, Power company sell off and directors massive pay bumps, Kiwi rail shares owned by the PM whist the future of Kiwi rail hangs in the balance, bailing out Rio Tinto last week. How many dozen have I missed?

[http://www.scoop.co.nz/stories/PA1305/S00195/skycity-hits- ja...](http://www.scoop.co.nz/stories/PA1305/S00195/skycity-hits-jackpot-with- keys-dodgy-deal.htm) [http://m.nzherald.co.nz/business/news/article.cfm?c_id=3&obj...](http://m.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=10875466)

[http://www.3news.co.nz/Key-lied-about-ownership-of-Tranz- Rai...](http://www.3news.co.nz/Key-lied-about-ownership-of-Tranz-Rail-shares- Labour/tabid/419/articleID/72617/Default.aspx)

[http://m.nzherald.co.nz/business/news/article.cfm?c_id=3&obj...](http://m.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=10910225)

<comment><USER13> Anti-sea protest laws at the request of Royal Dutch Shell, 3 strikes law prototyped for the US and the attempted inclusion of software patents at the request of IBM and MS to name but a few. And most obviously the GCSB law change.</comment></comment></comment>


Racial Diversity In Tech By The Numbers - <AUTHOR> https://www.beamjobs.com/diversity/racial-diversity-in-tech <comment><USER2> It seems at the same time white employees are under-represented in tech overall (see Racial distribution of tech employees relative to US population).

</comment><comment><USER3> As a PoC, who the f cares. Fix the root of the issue. Anyone can point at the fruits and say they're bad but that doesn't fix anything.

<comment><AUTHOR> Ya, I agree it doesn't fix anything. I think that the first step in addressing the issue is to have companies be more open with this data so you can actually track if things are changing for the better. That's my goal with this article.

<comment><USER4> I think defining "for the better" is the contention, not the existence or visibility of the data. There are people who say that a workforce must be exactly proportionate to the racial makeup of the population. There are people that say no, we need more minorities to be hired to make up for injustice. There are people that say no, it should be purely meritocratic. Resolving this tension is key IMO.

<comment><AUTHOR> That's a fair point. Personally I've been convinced by research that diversity improves business and financial performance.

Value of diversity in leadership: [https://www.mckinsey.com/business- functions/organization/our...](https://www.mckinsey.com/business- functions/organization/our-insights/is-there-a-payoff-from-top-team- diversity#)

Another article: [https://hbr.org/2018/07/the-other-diversity- dividend](https://hbr.org/2018/07/the-other-diversity-dividend)

</comment></comment></comment></comment><comment><USER5> It's interesting - mentally I break up "Tech" companies into many subdivisions. I think the Tech-Tech portion of a "Tech"company (i.e. SW engineers, chip designers) is probably decently good at being a meritocracy.

If you have a decent track record of being able to code/implement/ship features that are on-spec with few defects, on time, with a decent attitude, its a gross generalization but some company out there will probably hire you - particularly if you have a solid GitHub profile and recs.

And if you don't feel the urge to grow career wise outside of Tech-Tech then you'll be fine although your pay/leadership opportunities will probably reach a ceiling.

But a "Tech" company also consists of Tech-Sales, Tech-Finance, Tech-Law, Tech-Marketing, Tech-VC, Tech-Management, Tech-Product Management, Tech- Management.

All of these other subdivisions that make up the rest of a "Tech" company are actually much more like their counterparts in non-"Tech" companies with all their much more plentiful issues of bias, discrimination and harassment. Perhaps this is because a lot of the work is even less measurable on binary terms and therefore graded based on subjective reviews by existing leadership who as humans that have biases. They are also outward facing where you have to deal with existing biases of society.

<comment><USER6> >I think the Tech-Tech portion of a "Tech"company (i.e. SW engineers, chip designers) is probably decently good at being a meritocracy.

I don't think that's true at all. Engineers are not some sect of logical beings that don't have human emotions and foibles. I think they're just as likely to be discriminatory as anyone else.

</comment></comment><comment><USER7> Part of the issue is lack of entry-level positions. When everybody is hiring seniors, you're competing with the same existing pool of talent. You need to grow the talent pool.

</comment><comment><USER8> Comparing to US population is a bit disingenuous when good tech jobs are not available in most of the country.

</comment><comment><USER9> Good, now run the numbers for the world... us is a melting pot for the world after all and projects power and currency all over the world. May as well project social justice for all as well.

</comment><comment><USER10> This is a great but how does everyone put half the worlds population under "Asian"??! Lol.

<comment><USER11> Well... It's a race right? ;) Personally I think racial divisions are stupid. Cultural divisions are much more exact and useful

<comment><USER12> Chinese, Nepalese, Indians, Turkics, Armenians, Arabians, Jews, Iranians etc. are all "Asians".

</comment></comment></comment><comment><USER13> Tech leadership really means the middle management, i.e. folks hired to manage the line workers. Wait until they find out about the race of most capital owners!</comment>


Laura Loomer Loses Lawsuit Against Twitter, Google - <AUTHOR> https://lawandcrime.com/high-profile/conservative-activists-lose-lawsuit-that-accused-twitter-google-and-facebook-of-censorship/ <comment><AUTHOR> Note: The <h1> title is "Conservative Activists Lose Lawsuit That Accused Twitter, Google and Facebook of Censorship" but that's too long for HN, so I used the the title from the <title> element in the <head>.

</comment><comment><USER2> To no one's surprise:

> The Plaintiffs do not show how the Platforms' alleged conduct may fairly be > treated as actions taken by the government itself. Facebook and Twitter, for > example, are private businesses that do not become "state actors" based > solely on the provision of their social media networks to the public.

</comment><comment><USER3> They tried every banal free speech absolutist claim (1st Amendment! Monopolistic collusion! Discrimination against conservatives!) and lost on _all of them_. A good reminder that despite the constant bombastic comments about these companies, there is no legal basis for any of these complaints. I'm surprised the court even entertained Freedom Watch with standing to sue.</comment>


The best board game of 2017 is a wildly entertaining romantic comedy generator - <AUTHOR> https://www.vox.com/2017-in-review/2017/12/17/16764344/best-board-game-2017-fog-of-love <comment><USER2> Actually, the best board game of 2017 was Kingdomino.[0] It's a good lightweight euro, easy to learn but decent strategy for those who want it.

Arranging dominos in a 5x5 grid is harder than it should be, when you're trying to maximize points one domino at a time. I start off with a great plan, but by the end: d'oh, last one won't fit!

[0]: [https://en.wikipedia.org/wiki/Spiel_des_Jahres#2017_awards](https://en.wikipedia.org/wiki/Spiel_des_Jahres#2017_awards)

<comment><USER3> This is a far better authority on tabletop gaming than Vox in the OP.

Also not super impressed with: > Fog of Love is available exclusively from Walmart for some reason

<comment><USER4> It should still be taken with a grain of salt. E.g Istanbul beat Concordia. I have both games. Istanbul is good. But Concordia is a modern classic.

Not all of the good games in 2017 necessarily even qualify for the Spiel des Jahres or the Kennerspiel des Jahres. E.g. Gloomhaven, 7th Continent.

<comment><USER2> Right. I enjoy Istanbul, but for the most part, it seems like you just pick one or two economic engines as soon as you see the board layout, and then work on the one that has the least competitors on a given turn. The strategy changes enough between turns to make it fun, but not enough to make it one of the greats.

SdJ isn't always right, but still a better choice for best game of the year than J. Random Blogger's pick.

<comment><USER5> Agreed. When I started out getting a gaming collection, I read about the winners and recommends of SdJ, and picked among those. Ended up with 7 Wonders, Codenames, Dixit and Spyfall. Got to say it gives you some perspective on Risk and Monopoly.

</comment></comment></comment></comment></comment><comment><USER6> Also for consideration: Gloomhaven - [https://boardgamegeek.com/browse/boardgame](https://boardgamegeek.com/browse/boardgame)

Now the top ranked board game on BGG.

<comment><USER7> Yup, I don't think anyone can argue against Gloomhaven being the game of 2017. It totally dominated the BGG hotness rankings and the BGG geek rankings for the entire year. It was also on almost every reviewer's game of the year lists.

<comment><USER4> It's a legacy game. I don't like my board games to permanently change :)

<comment><USER8> It does have permanent world state changes, but it's nowhere near as extreme about it as capital L Legacy games.

It doesn't throw in rule changes the way the actual Legacy games do. Also, anecdotally, there's about 200 hours worth of play time in a single party running through the campaign, so the chances of anyone actually exhausting the game are slim. There's also a replay mechanic and a random dungeon mechanic, so you're not totally locked in.

</comment></comment></comment></comment><comment><USER9> Any recs for cooperative games suitable for kids? Have just started playing Forbidden Island with the kids, and they love it, but Burgle Bros is a bit too complicated for them.

<comment><USER10> Forbidden Desert has similar game mechanics but is sufficiently different from Forbidden Island to be worth getting as well.

<comment><USER11> My family has found that Forbidden Island is far more fun than Forbidden Desert. (Obv, ymmv). We just don't feel the need to have both of them, so one has been relocated to the grandparents house.

<comment><USER12> I have both, I liked Forbidden Island more, and my GF and her son both like Forbidden Desert more. I think the main difference is Desert involves a lot more luck, you can't really plan ahead much because the game could change considerably on one play. So Island would appeal to people that like to plan ahead, and Desert would appeal more to people who like chaos and drama.

</comment></comment></comment></comment><comment><USER13> I don't want to derail the thread, but I have discovered the 7th Continent thanks to last year discussion on HN. I ended up buying it (but I have not received or played it yet). This is a kind of remix of "Choose Your Own Adventure" in board game set in a early twentieth century with some fantastic elements (inspired by Lovecraft, Poe, Verne, etc) that can be played solo or in group.

If you were interested by the game, the 15th of January, so in less than 24 hours is the last day to pre-order it (and at this stage it probably won't be released by any other channel, so the last chance to get the game)

[https://app.crowdox.com/projects/1926712971/the-7th- continen...](https://app.crowdox.com/projects/1926712971/the-7th-continent- what-goes-up-must-come-down)

<comment><USER14> While that sounds fun, buying something that can only be pre-ordered is crazy (especially for fifty quid). What incentive is there for the seller to ensure it's not crap? Even pre-orders for something that will be on general sale is fairly suspect in my opinion.

<comment><USER6> It's fairly common in the boardgame industry, especially for boutique games like 7th Continent, Gloomhaven, or Kingdom Death: Monster.

These games don't fit well into the general mass production/sell in your local game store/keep stock on hand long term and sell online models. Too heavy, too many specially cards, too weird of a form factor, etc.

The 2017 Kickstarter campaigns for all 3 of these games were actually their second runs - previous KS had run and generated a lot of hype for the games, and were massive hits the first time around.

A lot of the time you are betting on a designer with a past track record. Other times you get drawn in by a concept that sounds interesting. But for these second production runs, it's more just pre-ordering a known quantity that doesn't lend itself to constant general availability.

</comment></comment></comment><comment><USER15> Really? I Player this back when it was first sent out, and honestly the only thing it's got going for it is the premis. It's not fun to play, it isn't an interesting game and not one of the people I played with wanted to ever play it again.

<comment><USER16> Sounds like just about every romcom i have ever endured...

</comment></comment><comment><USER17> For anyone wondering: It's also available for purchase on walmart.com (seemingly with free shipping)

<comment><USER18> I believe it's the first in Wal-Mart's exclusive board game line (so you can _only_ buy it there, for now at least).</comment></comment>


Indeed: AI job listings are up over 99% since 2015 - <AUTHOR> https://venturebeat.com/2018/08/23/indeed-ai-job-listings-are-up-over-99-since-2015/ <comment><USER2> I am surprised it has only doubled in 3 years. I feel like every company thinks they need a data scientist, and AI/ML is listed on every one of those postings. Plus almost every startup job listing I see has ML as one of their "differentiators"...</comment>


Kubernetes local development cluster in one command - <AUTHOR> http://www.charliedrage.com/kubernetes-dev-in-one-command <comment><USER2> Looks like a layman's version of Spread[1] Great and simple solution!

[1] [https://github.com/redspread/spread](https://github.com/redspread/spread)

<comment><USER3> You mean localkube, not spread - [https://github.com/redspread/localkube](https://github.com/redspread/localkube)

</comment></comment><comment><USER4> I'm not sure I find this that useful. [http://kubernetes.io/docs/getting- started-guides/docker/](http://kubernetes.io/docs/getting-started- guides/docker/) is pretty simple.

<comment><USER5> It looks like all the commands from that page put into a single script to be easier to use.

</comment></comment><comment><USER6> Nice.

Adding sky-dns would be a nice enhancement.</comment>


Show HN: Afrostream Media Server – create MPEG-DASH videos without the headache - <AUTHOR> https://github.com/Afrostream/afrostream-media-server <comment><USER2> I would like to see a demo/sample, with the proper vanilla HTML5 markup to use the different streaming methods. Since I don't know anything about how DASH works, a demo page telling me _which streaming type is being used currently_ would be really helpful so when I visit the demo with my different devices, I can visually see what it's doing different each time.

<comment><AUTHOR> We will do that :)

</comment></comment><comment><USER3> Thank you for this. Will surely test this out at work tomorrow because I've been trying to get a headache free dash or any smart streaming setup. So far all open source options i tried have given me nothing but headaches.

<comment><AUTHOR> Don't hesitate to give us feedbacks

</comment></comment><comment><USER4> Does this only work with pre-recorded video or does it support live-streaming (like Twitch too)?

<comment><AUTHOR> We tested on prerecorded vidéo now. We are working on live. Keep in mind it's a early alpha version

<comment><USER4> Looking forward to it. Keep it up!

</comment></comment></comment><comment><USER5> Looking forward to trying this. There is certainly a gap in the market for this.</comment>


Bitstamp is open for business - <AUTHOR> https://www.bitstamp.net/article/bitstamp-is-open-for-business-better-than-ever/ <comment><USER2> Have they figured out how the attackers got in? Have they competed the forensic analysis to determine they can't get back in through the same or similar vectors?

There's nothing really here about what happened or how they have mitigated it. Perhaps it's too early and they don't really know, but then isn't going back live fairly risky?

Aside from a persistent firmware threat, which has nothing to do with the attack vector, I don't see why "yay we're on AWS now" is any different unless they know the compromise was due to a specific piece of infra which is no longer present. Isn't it much more likely to be a bug in their software logic which would certainly not be fixed by restoring a backup...

<comment><USER3> I'm managing the incident response.

The investigation continues in parallel. First priority was safety, which meant shutting everything down to prevent any additional issues or destruction of evidence. Second was giving people the ability to access their funds and to trade - a complete redeploy of the infrastructure which took us working 22/7 since Monday until just now.

An investigation is underway and we have some internal speculation from the first bits of information - the real statements will come after we aren't guessing. This is the first minute we've had to breathe this week.

<comment><USER4> So eh, are you sure they couldn't get back in?

<comment><USER3> Yes.

</comment></comment></comment></comment><comment><USER5> The real question is whether Bitstamp is now insolvent. They lost $5 million. Did they get new funding to replace that, have enough capital of their own, or are they dipping into customer funds? Their statements have carefully avoided mentioning this issue.

If a real stockbroker lost funds like that, and became insolvent (debts > assets), they would have to stop operations immediately. In some jurisdictions it is a felony for a broker to continue to accept funds once insolvent. They don't get to "fix it later". That's because, historically, the temptation to fix it by speculating with customer funds has been a big problem.

Bitstamp now needs a full audit by an outside auditing firm.

<comment><USER6> Considering Bitstamp's fees, $5M shouldn't be a problem for them. They are a huge exchange and they make a lot of money.

<comment><USER5> Bitcoin exchanges aren't very profitable. The numbers for Mt. Gox are available now. [[http://www.businessinsider.com/mt-gox- financials-2014-2](http://www.businessinsider.com/mt-gox-financials-2014-2)] The numbers for 2013 are actuals; the future-year numbers were pure fantasy. For the year ending March 31, 2013, Mt. Gox, the biggest Bitcoin exchange in the world at the time, made $286,000. That's all. For comparison, the average profit for a single McDonald's location is about $200,000.

The Bitcoin exchange business has low commissions, and they're heavily discounted for big traders. Sometimes all the way to zero. It's hard for an exchange to raise prices. So exchanges tend to look for other ways to make money. Those ways usually aren't good for customers. Trading on one's own exchange and front-running are tempting. Mt. Gox probably did that; many users observed that, during busy periods, trades did not appear to be first in, first out.

That's why no major financial firm has entered the Bitcoin exchange business. It's not very profitable.

<comment><USER6> Bitstamp's 6-month volume is 2,179,978 BTC. Their fees start at 0.5% and go progressively lower down to 0.2% for large volume customers.

Their volume and bitcoin prices were crazy high a year ago, I estimate they made around $6-10M.

[http://bitcoinity.org/markets/list?currency=ALL&span=6m](http://bitcoinity.org/markets/list?currency=ALL&span=6m)

[https://www.bitstamp.net/fee_schedule/](https://www.bitstamp.net/fee_schedule/)

</comment></comment></comment></comment><comment><USER7> > We can assure customers that any bitcoins held with us prior to temporary > suspension of services on January 5th at 9am UTC are completely safe and > will be fully honored.

I, for one, am glad to see a Bitcoin story turn out with a somewhat positive outcome, at least from the users' perspective.

That said, the article says a breach resulted in the loss of ~19,000 BTC, or around $5.6 million. If they're honoring all user funds, does that mean they are just eating that loss out of their own profits, and perhaps try to file an insurance claim? Or were they actually responsible enough to set enough profits aside in cold storage self-insure the total amount floating in the "hot" wallets? On that note, I wonder if profits and hot wallet demands scale proportionately or not, or if their margins are just at a scale where this is a non-issue?

<comment><USER3> > does that mean they are just eating that loss out of their own profits,

Yup.

<comment><USER8> So you are essentially claiming a profit to date >> $5.6M.

There will be lots of parties interested in that information.

<comment><USER9> It is probably much higher. If we take a conservative estimate of 10k btc traded per day, on average, and a trade commission of 0.008% (0.004 two ways), we get to $5.6m in little over half a year. Now yes, the have some operational costs etc, but they've been in business for several (?) years.

tldr: back of the envelope calculations show that they must have made more in profit to-date.

<comment><USER10> The trading commission is 0,8% (0,4% two ways), and it is lower the more you have volume. In addition, they have withdrawal fees.

In addition to operational costs probably fraud is also a big cost, but it is very difficult to estimate how big it is for them.

And, they don't exactly need to take it from their profits, they can just do fractional reserve... They are a bank, as are any bitcoin services that hold money for the users.

<comment><USER11> How is fractional reserve profitable except by loaning BTC with interest (which I don't think they do)?

<comment><USER12> It's probably more profitable than going out of business (which is probably what would happen if they admitted to being insolvent).

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> To me this marks a turning point in the maturity of bitcoin - major breaches may still be a cost of doing business, but end users can finally hope for some limited insulation from such failures.

<comment><USER14> Perhaps not this particular event, but definitely this period was a turning point. Multi-sig is a great example. Here's a chart on the amount of bitcoin stored in multi-sig addresses:

[http://p2sh.info/p2sh-ratio](http://p2sh.info/p2sh-ratio)

Nine months ago it was sub 0.1%. Three months ago it was sub 1%. Today it's over 6% and growing rapidly.

Multi-signature is just one example of potentially radically improved security versus the single key methods we had before.

We've also seen in the past 6 months 3 companies come out with proper insurance. Circle, Xapo, Coinbase. All at a sub 1% cost. (Coinbase and Circle free, Xapo I think 0.12% annually). Combined with multi-sig vault products from Coinbase, which has done a lot of interesting security stuff. (the basic 2fa stuff of course, but I think also things like finterprinting devices to detect stolen credentials from users.)

And with that, we're also seeing enterprise-products come out which offer bitcoin management in a secure way. I wouldn't be surprised if such a company like BitGo starts marketing insured warm wallets to exchanges on a percentage cost basis, and let exchanges offer users to pay for the option of this insurance.

And we've seen more steps taken regarding the bitcoin ETF on the NASDAQ, which settles manually twice a day and allows (industrial) trading and liquidity but gets rid of the issue that Bitstamp and everyone else has: running a warm wallet on a server that automates transactions. The vast majority of Bitstamp's coins were in cold storage, but if you want to run automated transactions you can't do it on cold storage.

So this period has definitely been a turning point I think. Will be interesting to see things develop the next few years.

</comment></comment><comment><USER15> > On January 4th, some of Bitstamp's operational wallets were compromised, > resulting in a loss of less than 19,000 BTC.

That's a nice way of saying "we're out five and a half million USD"...

<comment><USER16> Apparently they were the ones moving the famous transaction of 200k bitcoins. So, yes, for them this breach could look like: "Meh, we've lost some pennies".

[http://www.coindesk.com/194993-btc- transaction-147m-mystery-...](http://www.coindesk.com/194993-btc- transaction-147m-mystery-and-speculation/)

<comment><USER9> Those are all (vast majority) client funds. 5 million is coming out of their pocket.

<comment><USER10> Who knows if it comes from their own pocket, or from the customer funds? It is just their internal accounting in the end. The same way the banks work.

</comment></comment></comment></comment><comment><USER4> Wow they have multi-sig support now, that is pretty amazing. Would easily be worthy of its own article in other circumstances.

If I understand it correctly it means money can not be withdrawn without both the account holder and bitstamp signing the transaction. This means that neither can the money be stolen by hackers, nor could bitstamp run off with it. There are actually some bitcoins in the real blockchain "with your name on it", as opposed to you just having a claim on bitstamp.

<comment><USER17> I doubt that. Having the funds jointly controlled by the exchange and the user, and outside the full control of the exchange, would require each fund transfer to be represented as a transaction on the Bitcoin network, which is simply impossible - the volume of transactions on an exchange is a few magnitudes larger than what Bitcoin can currently handle. To keep up with that volume, exchanges must use an internal off-chain accounting system, which precludes the use of multi-signature (or any other smart contract based on Bitcoin scripting).

In addition, this would require the user to be online and actively sign transactions when a matching order is found.

I think that they most likely meant that they're using multisig internally to protect their funds, in a setup where all the keys are controlled by them.

</comment></comment><comment><USER16> He doesn't say in the post that they have ditched the previous PHP site in favour of Django/Python... Lesson learned? xD

[http://oi60.tinypic.com/20sj3b9.jpg](http://oi60.tinypic.com/20sj3b9.jpg)

PS: Still wondering why they use Apache instead of Nginx which is more efficient/robust/secure

<comment><USER10> They have always been django/python and have never used PHP, please don't spread this kind of bullshit (unless you have solid facts to back it up. I remember checking the setup in 2012 and it was already django back then).

<comment><USER16> Ooops, I didn't know that. I guess I rely too much on wappalyzer (which showed PHP). Sorry for the confussion...

</comment></comment></comment><comment><USER18> I think the conclusion from all this heists is that virtual money is much easier to steal. Next time you will see physical break-ins and so on. We've watched all those action movies where people risk their lives to get hold of $5M. Nowadays, you can do this remotely from the convenience of your home... but it will be easier for the old crooks to steal a key vs bags of cash, so, let's see how this goes forward.

<comment><USER14> It's trivially easy to secure bitcoin, it's just data after all, a private key, one you could write on a dollar bill if you wanted and secure no different from a dollar bill.

The difference is securing bitcoins in high-risk environments: live automated servers. But that only goes for a small minority of bitcoins as only about 0.5% of bitcoins are traded on a daily basis, the rest can be in what is called cold-storage which is way, way easier to secure than a bunch of physical money, gold, diamonds, whatever. The 0.5% is a bit like the security of wallets in people's pockets: not very good and not comparable to bank-grade security of those same people's savings in a bank. Or comparable to lone ATMs which can be raided.

Beyond that, bitcoins exist on an open ledger. While it may be hard to trace that now, we're seeing blockchain analysis tools develop all the time. Who knows what they might uncover later? I certainly wouldn't be excited as a criminal to leave a permanently recorded trace of every single transaction ever since the moment of the theft, with the prospect of decades of cheap supercomputers doing analysis of this open source data set which might reveal yourself. Bitcoin is great for pseudo-anonymity when buying por n online, but it probably won't stand the test of time in protecting you from every criminal investigation, a bit like a randomly generated pseudonymous email address gives a large extent of privacy, but not if you're a criminal.

<comment><USER19> It's not easy to secure it. You have to do two things very well:

1) Don't let anyone find the private key

2) Don't let yourself lose the private key

The more of 2 you do, the harder 1 gets and the more of 1 you do, the harder 2 gets. If you have only a single copy on paper, it's hard for anyone to find, but also easily damaged or lost. If you make a backup on a computer, a virus might find it, making the paper copy worthless.

Not only that, but from the moment you generate the key, you have to be completely confident about the security of the computer you're using. People recommend a clean install of OS from the original CD and never connecting to the internet. That's not impossible but not easy, furthermore you have to carefully destroy all record of it left on that computer. So unlike gold, not only do you have to keep it secure in the future, but you have to have made sure that it was always secure throughout the past. You can't transfer ownership to anyone else without them going through the same rigmarole too, unlike gold which you can physically give to someone and they can be confident it won't evaporate in front of their eyes one day.

I agree with you about blockchain analysis. Who knows what new techniques might be able to dig up. They might not have much certainty but perhaps enough to put suspicion on someone.

</comment></comment></comment><comment><USER20> Nice, they gave me good odds on fairlay. Betting 0.001BTC on 3.5 odds did not earn me much money, but It made Bitstamp drama more thrilling. :)

</comment><comment><USER21> Is the address of the stolen wallet known?

It would be interesting to watch where that money is moving and how the thief goes about anonymising it.

<comment><USER22> Yes, it's trackable, subject to the pseudonymous limitations and opportunities for obfuscation inherent in blockchain technology.

[http://blog.cryptocrumb.com/2015/01/bitstamp-theft- bitcoins-...](http://blog.cryptocrumb.com/2015/01/bitstamp-theft-bitcoins- being-spent.html)

BTW, in that title, the word 'spent' is used in the formal sense for blockchain transactions. It doesn't mean the money is gone or ''used up'. It really just means 'moved'.

After reading that, it looks like the person who stole this money is actually panicking and making a lot of dangerous mistakes. I guess they are in a state that's between terror and ecstasy. I doubt they had any sleep recently, and they're going to be looking over their shoulder for a very long time.</comment></comment>