Facebook open-sources LogDevice, a distributed storage for sequential data - <AUTHOR> https://logdevice.io/ <comment><USER2> Can someone from FB chime in with some info how much storage is needed for the logs/data? Say, for 1 GB of raw input logs from a http server (nginx/apache), when stored in LogDevice would they take notably less space on disk (compression), or more (overhead)? This interests ne for evaluating resources/costs I'd need to prepare if I were to deploy it...

<comment><AUTHOR> These numbers really depend on the compressibility of the content, compression scheme and the type of batching used. The metadata overhead is fairly minimal. LogDevice allows you to configure this on either the client, sequencer or rocksdb level.

<comment><USER2> Is some form of compression enabled by default, without having to tweak options?

<comment><USER3> No, compression is disabled by default. You can enable compression either on the storage layer, or by enabling batching and compression on the sequencer, or by using the buffered write API with compression on the client.

</comment></comment></comment></comment><comment><USER4> Happy to finally see LogDevice open. We have been working on this for years now.

<comment><USER5> Can you give an overview over the difference to eg Apache Kafka?

It seems very similar.

<comment><USER4> It's a very different architecture and design. You can head to [https://logdevice.io/docs/Concepts.html](https://logdevice.io/docs/Concepts.html) to learn more about how LogDevice works.

In terms of function. LogDevice is similar to the core of Apache Kafka.

<comment><USER6> True, but Kafka has two very annoying features built into it:

\- There is no many-to-many log recovery whereas -- for example in Pulsar/DistributedLog -- logs are stored in small segments and distributed to multiple nodes.

\- Read scalability. Since all the log is stored in one node (with some replicas) the readers are bound to single disk sequential read capacity. Again Pulsar stores logs in segments that are distributed among broker nodes which helps a lot when there are many readers.

<comment><USER7> I'm not sure how accurate your comment is regarding Kafka's annoying features given that Kafka has partitions, which "solve" all of the problems you stated.

<comment><USER6> No it doens't, since a single partition is stored sequentially on one disk which limits the consumers to bandwidth of single disk (say c1 reads beginning of the partition and c2 end of the partition). But in the case of Pulsar c1 is most probably connected to a different node than c2.

<comment><USER8> LogDevice has this concept of "node set", which is the set of storage nodes that can be selected by the sequencer as recipients for a record or a block of records. A typical node set size is around 20-30 in our deployments. Each storage node in the node set contains a subset of the records (or blocks of records) of the log, we call that subset a log strand. The amount of IO capacity available to append records to a log or read records from a log scales with the size of the node set.

All of this is done while preserving the total ordering guarantee thanks the separation of sequencing and storage.

The operator could for example set a bigger node set size for logs that are known to have multiple consumers and require more IO capacity.

At facebook, we have use cases where a single consumer will need to replay a backlog of records in a log, sometimes hours or days worth of data to rebuild its state. We call this a backfill. Node sets allow the IO to be spread across multiple disks which improves backfill speed and helps reduce hotspots.

\-- Adrien from the LogDevice team.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> Very interesting! I hadn't heard of this before but I'd love to see it in action.

If anyone from the FB team or anyone using LogDevice wants to test performance with Optane SSDs (and compare to a NAND SSD), make a request by submitting an issue on our GitHub page: [https://github.com/AccelerateWithOptane/lab/issues](https://github.com/AccelerateWithOptane/lab/issues). I'll hook you up with a server hosted by Packet.

</comment><comment><USER10> Martin Kleppmann seems to point out technologies for problems of similar patterns already exist - [https://twitter.com/martinkl/status/1039938408393662465](https://twitter.com/martinkl/status/1039938408393662465)

<comment><USER11> Those are streaming/pubsub services though, this actually claims to be a store. I feel that's an important difference.

Do people just point their system journal at Kafka and wait for something to break?

At my previous job we built something similar to this out of rabbitmq and mongodb. I always wondered what the other big log companies used. Mongodb seemed like a pretty good fit, but a pure append only database might be even better. Trimming performance in MongoDB was subpar so we worked around it by creating a new collection for each day, trimming became a simple operation of dropping a collection at the end of each day.

<comment><USER12> > Those are streaming/pubsub services though, this actually claims to be a > store. I feel that's an important difference. > Do people just point their > system journal at Kafka and wait for something to break?

Kafka can be used as a data store if you like, so long as you're happy with the data management and access patterns it gives you - it is, after all, optimised for large sequential reads.

LogDevice looks to be very similar for most use cases to Kafka, hell, they even use RocksDB, which is used by stateful operations in Kafka Streaming, and of course, Zookeeper.

Where it differs is that it looks like it was designed for you to be able to work against a single "cluster" that could well be running across multiple data-centres. Which is very much a Facebook problem to solve.

So yeah, Kafka was a distributed log built for LinkedIn size problems, LogDevice is a distributed log built for Facebook sized problems.

Most of us don't have Facebook sized problems.

<comment><USER13> What's a good distributed log for 10-dev sized companies? :)

<comment><USER14> OKLog, Humio, and Splunk are all worth checking out.

<comment><USER15> OKLog has been abandoned by the author (the project is now read only on GitHub).

Humio is not self-hosted or open source, so not really a fair comparison. It also seems targeted towards _operational_ logs, i.e. system logging, traffic logging, auditing. Not things like data pipelines. Kafka and friends can be used for that kind of log, but they are more like databases; they use the term "log" in the sense of sequential and append-only.

Same goes for Splunk, which does have a self-hosted version, but is extremely expensive, last I checked. The SaaS version is also extremely expensive.

<comment><USER16> Humio does have a self-hosted version - but is closed source. You can download a trial and it supports most logs types via support for opensource log shippers, ie logstash and beats..along with other popular formats including Kafka.

[https://docs.humio.com/integrations/](https://docs.humio.com/integrations/)

The UI is what simplifies analysis and visualization with live, real-time query and db.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> I had just stumbled across [https://github.com/facebookincubator/python- nubia](https://github.com/facebookincubator/python-nubia) and am anxious to try it out. Was wondering about the internal project it was factored out from. This appears to be it.

<comment><USER4> Correct. LDShell in logdevice was the starting point of python-nubia.

</comment></comment><comment><USER18> The use cases overlap neatly with Kafka's. Everything from it's usage of zookeeper, time-and-storage-based retention tuning are similar

The announcement does not clarify the reason they use this over kafka. Is it because Kafka doesn't scale to millions of logs on a single cluster or is it because kafka is not sympathetic to heterogeneous disk arrays containing SSD and HDD. I strongly suspect it may be latency of writes at scale but this is pure speculation.

I don't know. If I understand why anyone might use this I'd contribute to building language bindings for the APIs.

<comment><USER19> > Is it because Kafka doesn't scale to millions of logs on a single cluster

I doubt that's it, since Kafka can certainly do that.

<comment><USER20> Millions of separate topics on a single Kafka cluster? The way it's designed requires opening files for all of those topics and their partitions so good luck if you're trying that. You'll run out of file handles, then memory, and then the disk access will completely freeze up.

<comment><USER19> I didn't think we were speaking of millions of topics here; only millions of logs. You can certainly have logs numbering in the millions using a single topic. Mux/demux would have to happen at the producer/consumer side, of course.

<comment><USER20> Do you mean log segments then? In that case I don't see what's special about it because that's just rolling files and all of these systems can handle millions that way.

As far as millions of topics, if you have to do it at a logical layer yourself, then you might as well use a system that supports it natively.

</comment></comment></comment></comment></comment><comment><USER20> Great to see this released. Some similar architecture decisions to Apache Pulsar as well with the separate of compute (in this case the sequencer) from the storage.

Kafka has done well so far, especially in making streaming systems more common, but it's about time for the next-gen systems.

<comment><USER21> How does LogDevice differ from Kafka?

<comment><USER20> Kafka brokers handle both the computation (partition/topic management, sequencing, assignments, etc) and storage together. This coupling creates scaling and operational challenges which LogDevice removes by separating the layers. Storage nodes can be as simple as object stores (but optimized for appending files) and use multiple non-deterministic locations for a given piece of data to randomize placement. They read, write and recover data very quickly by working together in a mesh.

Meanwhile the compute layer becomes very lightweight and almost stateless, which is easy to scale. In LogDevice, the Sequencers are potential bottlenecks but generating a series of incrementing numbers is about the fastest thing you can do so it'll outpace any actual data ingest to a single log, while giving you a total order of all entries within that log. The numbers (LSNs) follow the Hi/Lo sequence pattern so if a Sequencer fails, another one takes its place with a greater "High" number, so it's guaranteed that all of its LSNs will be greater than the previous Sequencer as a result. This also provides a built-in buffer to still accept messages and assign the permanent LSNs to them after recovery in case a Sequencer fails.

Apache Pulsar is similar to LogDevice but goes further where brokers manage connections, routing and message acknowledgements while data is sent to a separate layer of Apache Bookkeeper nodes which store the data in append- optimized log files.

<comment><USER22> Interesting. Microsoft's Tango paper had some interesting things to say about sequencers/sequences as well.

</comment></comment></comment></comment><comment><USER23> Awesome, I have been waiting for this since seeing the @scale talk about it. [https://atscaleconference.com/videos/logdevice-a-file- struct...](https://atscaleconference.com/videos/logdevice-a-file-structured- log-system/)

<comment><USER9> Is there supposed to be a replay of that talk on the site you link to or is it just not loading for me?

<comment><USER20> The event is hosted by Facebook and there is an embedded Facebook video player on that page. Here's the direct link: [https://www.facebook.com/atscaleevents/videos/19602876909109...](https://www.facebook.com/atscaleevents/videos/1960287690910992/)

<comment><USER9> Thanks! My script and adblockers must've broken the site.

</comment></comment></comment></comment><comment><USER24> The amount of great quality open source projects dein Facebook just keeps growing. I really like the consistency guarantees:

[https://logdevice.io/docs/Concepts.html#consistency- guarante...](https://logdevice.io/docs/Concepts.html#consistency-guarantees)

And it uses RocksDB under the hood:

[https://logdevice.io/docs/Concepts.html#logsdb-the-local- log...](https://logdevice.io/docs/Concepts.html#logsdb-the-local-log-store)

</comment><comment><USER25> Thank to Open Source that, it looks a great project.

Could a LogDevice give a bit of informations about the scale they use that at facebook ?

\- How many record this thing can injest per day ? \- Any limitations on the maximum number of storage nodes ? \- What would be your maximum and advise size of record for a production usage ? \- ZooKeeper seems to be the center point used as epoch provider. Did you encounter any scaling limitations or max number of client due to that ?

<comment><USER4> I cannot give you exact numbers, but here are some information that might be useful: \- LogDevice ingests over 1TB/s of uncompressed data at Facebook. This already has been highlighted in last year's talk in @Scale conference. \- The maximum limit as defined by default in the code for the number of storage nodes in a cluster is 512. However, you can use --max-nodes to change that. There is no theoretical limit there. Each LogDevice storage daemon can handle multiple physical disks (we call them shards). So, If you have 15 disks per box, 512 servers. That's 7680 total disks in a single cluster. \- The maximum record size is 32MB. However, in practice, payloads are usually much smaller. \- Zookeeper is not (currently) a scaling limitation as we don't connect to zookeeper from Clients (as long as you are sourcing the config file from filesystem and not using zookeeper for that as well).

Hope that helps.

</comment></comment><comment><USER26> Very interesting!

I like the idea of decoupling compute from storage for streaming/log data.

I wonder if it would be easy to make it run under Consul, instead of ZooKeeper.

<comment><USER4> We use Zookeeper primarily for the EpochStore. This is the abstraction that you can you use if you want to replace Zookeeper. It shouldn't be that hard as long as Consul offers the same guarantees as zookeeper.

</comment></comment><comment><USER27> Am i the only being puzzled by

 _Scalable_

 _Store up to a million logs on a single cluster._ ?

This sounds pretty confusing / low volume.

<comment><USER20> logs = topics, so they mean 1M separate topics on a single cluster.

<comment><USER27> Makes more sense. Thank you!

</comment></comment></comment><comment><USER28> What benefit to facebook is there from open sourcing technology they have developed?

<comment><USER29> Facebook's competitive advantage doesn't come from having the best reliable streaming data store at scale, or from its software in general. Even if MySpace, Friendster or Google + got their hands on the whole software stack & started running it, people would stick with Facebook.

So there's no cost to open sourcing. The benefit comes from being known as technically innovative in general, and for recruiting, being known as having interesting, meaty, challenging projects to work on.

The impetus usually comes from team members who want to do the work. It could be to become known for having worked on the project, or a sense of giving back to the community, or a hope that you'll get bug fixes & features from outside contributors. In my (very limited) experience, managers "passively encourage" it -- they generally don't push the team to do it, but when the team asks, they encourage it.

<comment><USER30> If that's true, why haven't they open sourced Haystack? Clearly they're holding onto it due to competitive advantage.

<comment><USER29> I don't know, but my guess is nobody associated with it wants to put their other work on hold to make it happen. From my limited experience, nobody pushes you, and nobody blocks you. So it depends a lot on the motivations of the engineers on the project.

</comment></comment></comment></comment><comment><USER31> Awesome to see this finally happening :)

Previous discussion in HN: [https://news.ycombinator.com/item?id=15142266](https://news.ycombinator.com/item?id=15142266)

</comment><comment><USER32> I don't see anything about trust requirements or verification. Does LogDevice assume that all devices in my cluster are trusted?

<comment><AUTHOR> LogDevice uses SSL for authentication. This can be enabled for both clients and servers [1].

[1] [https://logdevice.io/docs/Settings.html#security](https://logdevice.io/docs/Settings.html#security)

<comment><USER32> That's not what I mean though. What if I have a cluster with devices I don't trust, but I want to let them emit logs if they conform to a particular protocol. Like, will this thing check signatures for me and such?

Since it doesn't say anything about trustlessness, I assume that it assumes that all nodes are trusted.

<comment><USER33> LogDevice is crash fault tolerant not byzantine fault tolerant if that's what you're asking. This fault tolerance is in regards to where the logs are placed not who's emitting them though. If you want to analyse logs for inconsistencies or attack patterns you should look into something like SEAMS/REAMS, it's completely out of scope for LogDevice.

</comment></comment></comment></comment><comment><USER34> "bin/logdeviced"

All daemons and system administration utilities belong into sbin, because bin is for end-user applications.

Historically, the "s" in sbin meant something else, but it always contained applications and scripts only root could run.

When I see these examples, it's depressing to see just how much understanding of UNIX is missing.

<comment><USER4> Maybe sending a PR would help?

<comment><USER34> It's for Linux only, and I run illumos-based SmartOS on my own infrastructure.

That's not the point. The point is that all these generation Y kids grew up on PC buckets and still don't understand UNIX and the concepts behind it, and yet they use it to power their applications. This can only end badly unless they start making an effort to understand the concepts behind the substrate they are writing software for.

<comment><USER31> A few things. First, can't you run Linux inside a Solaris Zone? I don't know much about Solaris stuff (although I do like it very much, I grew up mostly with Linux, which you so much despise, and I'm not too familiar with other Unixes). So... I think you could probably run Logdevice if you really wanted.

Then, here's my two cents. When engaging in conversation and civil dialog, please try to avoid being so dismissive and so proud of yourself and of how much you think you know about stuff. You come across as abrasive and entitled. It's not nice to just jump into a conversation and talk trash about the work of others just because you dislike the operating system that they use.

Finally, if you really care, work on porting it to your operating system of choice and engage in civil conversation doing pull requests, etc. Everybody will be thankful for that.

<comment><USER35> > You come across as abrasive and entitled.

I recognize the good intention here, but if you're going to post this kind of comment, please try to eliminate the personal provocations. They don't help, and do hurt.

</comment></comment></comment></comment></comment><comment><USER6> External logging service is my favorite way of doing replication. It provides nice features. Specifically:

\- Cross vendor replication which makes migration much easier.

\- No dependency on vendor provided replication protocols.

\- Ability to use in-app databases such RocksDB, SQLite, ...

\- Upgrading DB nodes becomes way easier since they are totally separated from each other.

</comment><comment><USER36> How does that fit in a ML training pipeline? (this is mentioned on the page)

<comment><USER20> It's just streaming data but more scalable and with total ordering which can be important for ML.

</comment></comment><comment><USER37> Sounds like it might have been influenced by the MSR CORFU project (separate sequencer, write striping). Can anyone confirm?

<comment><USER38> It's hard to deny that there is at least some influence there. Like LogDevice, the zlog project [0] is influence by CORFU (separate sequencer, write striping), but both use different storage interfaces / strategies.

[0]: [https://github.com/cruzdb/zlog](https://github.com/cruzdb/zlog)

</comment></comment><comment><USER39> Is there any comparison with other similar storages?

</comment><comment><USER40> This lot more similar to apache bookeeper.

</comment><comment><USER41> this is like....the harder half of a whole blockchain project :D super interesting

</comment><comment><USER42> Is this a Kafka competitor?</comment>


Show HN: Sesame, a self-aware digital assistant that protects your messages - <AUTHOR> http://sesame.chat <comment><AUTHOR> We created Sesame because we believe private ownership and nuanced control leads to better communication. Sesame's Cognitive Messengers audit every message and their attendant permissions independently for you, and give you the ability to take back your messages from any device, whenever, forever.

</comment><comment><AUTHOR> ...So if you break up two years from now, you can take all of your messages and pictures back :)

</comment><comment><USER2> Any plans for an Android app? This sounds like something I want to try out.

<comment><AUTHOR> Yep, that's next :-)</comment></comment>


Dear Microsoft - <AUTHOR> https://slackhq.com/dear-microsoft-8d20965d2849#.3wj6j4611 <comment><USER2> What a patronizing and condescending piece. All I got from this is that Microsoft has launched something in this space that is really worth taking a look at, if it prompts a competitor to write and publish something like this.

<comment><USER3> >> What a patronizing and condescending piece.

That's why it is so great. Of course, only time will tell if their display of strength is of fear or confidence.

<comment><USER4> If it was confidence, they wouldn't have to share their confidence with anyone outside their office.

They could have sent a cake. Or took an ad like the ones Spielberg and Lucas did when the other broke box office records.

</comment></comment></comment><comment><AUTHOR> This was published in the Times today and it came across as passive aggressive and insecure to me, but I'm curious to hear what others think of this stunt...

Link to Times ad: [https://mobile.twitter.com/stewart/status/793811616760496128](https://mobile.twitter.com/stewart/status/793811616760496128)

<comment><USER5> Historically, letters like this don't bode well for the company writing them. People are drawing comparisons to Apple's "Welcome, IBM. Seriously" ad from 1981. Rdio sent one to Apple when Apple Music was launched too.

<comment><AUTHOR> Yes! I was racking my head trying to remember where I had heard this before, thanks!

</comment></comment></comment><comment><USER6> This is really terrible. Since nobody's name is actually on it, I assume their marketing team wrote it.

I understand that Slack's valuation is based in part on the idea that they can convert large portions of the enterprise market, and that Microsoft has a pretty great track record of burning that market down. It makes sense that they'd want to write something to maintain their claim on the enterprise market.

But this piece can't decide who it's talking to, and so it's probably pretty cringe-y to everyone.

<comment><USER7> Microsoft already has the beachhead into the enterprise, with their Office 365 offerings. We're seeing most of our Fortune 500 customers move their email, at least, to Office 365, and almost none of the smaller businesses go through the hassle of managing their own Exchange infrastructure.

Once they are on Office 365 for email and Office, they basically get Skype, Yammer, and all the other things thrown in for free. I'm sure that this will fold into that same pattern, assuming it's, say, 75% as good as Slack.

<comment><USER8> I work in a Fortune 500 company, our CTO has already announced on Yammer that we will evaluate Microsoft Teams, because anyway, it is included in the Office 365 package. So yes, the bundling strategy seems to work.

</comment></comment></comment><comment><USER9> "We're glad you're going to be helping us define this new product category."

Wow, and folks give _Apple_ a hard time about acting like they invented stuff. The hubris of Slack thinking there's some moral high ground for them to take is astounding.

But their post does have me searching the web for Microsoft's offering. Because I don't know what the hell Slack is on about, and I'm thinking what Microsoft has might be better than the disjointed web-app-in-a-native-wrapper thing Slack has going.

EDIT: found it, thanks to _whalesalad_ 's post in this thread. Okay, given that the MSFT client looks a hell of a lot like Slack, I can see where they're coming from. Still, it strikes me as someone who wrote the ubiquitous to-do app and then buys an ad in the NYT when Apple includes the Reminders app in iOS, welcoming Apple as they "define this new product category".

<comment><USER10> While I would love more competition in this space, I doubt if MS will offer a native app. They have recently moved Skype to an electron app (basically the website in a wrapper)[0] so I don't expect this to be any different. While this is in alpha (and only for linux), I expect them to move all their other clients (apart from Windows) to this electron app.

[0] [http://www.omgubuntu.co.uk/2016/07/skype-linux- alpha](http://www.omgubuntu.co.uk/2016/07/skype-linux-alpha)

<comment><USER11> The Slack desktop app works pretty much the same way - native wrapper around an HTML UI that is almost identical to their web UI. (I think they may use the platform's native webview though, not Electron).

It has its pros & cons... one nice thing is that if you use the desktop app most of the time, using the web UI on the go doesn't feel like much of an adjustment. It has basically the full power of the desktop app because it's the same UI code.

</comment></comment></comment><comment><USER12> Anyone else find it a bit ironic (?) that they say "an open platform is essential" but haven't open sourced their core code base? That feels like a bit of a misstep to me.

<comment><USER13> They're referring to an open 3rd Party API platform - not giving away their secret sauce.

<comment><USER14> Can you explain the difference between an open and a closed 3rd party API?

<comment><USER15> To develop an app using the Slack API, you do not need to register with Slack and download an API key for your app. Anyone can just develop whatever they like without having to ask permission first and there's no easy way to selectively shut down apps that Slack doesn't like.

</comment></comment></comment></comment><comment><USER16> When did Slack become an "open platform"?

Ugh, really hate work going this way. Slack is an all-day all-hands meeting with no agenda. I miss async email.

<comment><USER17> How did slack take away your email?

I treat slack as an async alternative to voice and video... maybe I'm doing it wrong?

<comment><USER18> Because people use it as an alternative to email? They obviously weren't saying "Slack literally took away my ability to email" they were referring to how everyone is actually using Slack to talk to each other

</comment></comment></comment><comment><USER19> For those wondering what this is in response to, [http://www.theverge.com/2016/10/25/13405200/microsoft- teams-...](http://www.theverge.com/2016/10/25/13405200/microsoft-teams-slack- competitor-launch)

<comment><USER20> Here's a video, "Introducing Microsoft Teams" [https://www.youtube.com/watch?v=nKU- FMzZFF0](https://www.youtube.com/watch?v=nKU-FMzZFF0)

<comment> yoz-y I think the application can be quite cool, but this video... ugh, it reminds me somehow of the infomercials where people struggle to do the basic tasks.

</comment></comment></comment><comment><USER21> Not really sure what the message is here. Somebody from MS could just answer: "Thanks for the advice. We've been there before. We swallowed whole industries without chewing and we have killed more competitors that we can remember. We hope that you will still exist in the year to drive our creativity. Sincerely, MS"

</comment><comment><USER22> Bad move on Slack, accept your competition and work to stay ahead of the game that you no longer have a monopoly on.

<comment><USER23> They never were.

We only use a mix of Microsoft Lync and Lotus Sametime.

<comment><USER7> > Lotus Sametime

You're hanging onto that still? We used to build a number of products for Sametime, but IBM appears to have written it off and is hemorrhaging customers jumping ship, to Lync/Skype mostly, or Jabber, to a lesser extent.

<comment><USER23> That is what happens when IT dictates the rules.

</comment></comment></comment></comment><comment><USER24> This comes across as sour grapes from Slack.

Also, if you want to talk about "open solutions" then make your own solution open.

</comment><comment><USER25> I would describe the tone of this whole thing as "shit-eating"

<comment><USER26> or more like "pants-shat"

their only hope is what they hope for in this writing, MS f's up somewhere in the process and create another yammer. I hope they won't, competition is healthy. ;)

</comment></comment><comment><USER27> Ironic. I don't think Slack is particularly well designed.

Use it every day. And it's always unpleasant.

Why does it say "New mentions" when I wasn't actually mentioned, just someone said something in some other chat.

Also, I find it pretty tedious to switch back and forth between discussions.

In terms of UI design, I suspect they don't have a good team.

Maybe it just tries to please too many people?

</comment><comment><USER28> Now I'm interested in Microsoft's product. I'm certainly not in love with Slack. I'd love for someone to eat their lunch.

</comment><comment><USER29> Several commentators here seem unaware that "open platform" and "open source" are distinct concepts. From the Wikipedia article on "open platform" [1]:

" _In computing, an open platform describes a software system which is based on open standards, such as published and fully documented external application programming interfaces (API) that allow using the software to function in other ways than the original programmer intended, without requiring modification of the source code. Using these interfaces, a third party could integrate with the platform to add functionality.[1] The opposite is a closed platform._ "

An open platform can be closed source. A good example is Microsoft Windows.

[1] [https://en.wikipedia.org/wiki/Open_platform](https://en.wikipedia.org/wiki/Open_platform)

<comment><USER30> The confusion comes from Slack acting like Teams won't also have an open api for integrations.

</comment></comment><comment><USER31> Microsoft Teams [1] is apparently what this is in response to. I would not have even heard of it had this piece not gone out, ironically enough.

[1] [https://products.office.com/en-US/microsoft-teams/group- chat...](https://products.office.com/en-US/microsoft-teams/group-chat- software)

</comment><comment><USER32> I think the time they spend agonizing over how to slip some humor into their release notes would be better spent making their platform faster or improving their convoluted message search.

<comment><USER33> Agreed, and not making me create a new account every time I join a new team. Now I have to remember which account I used for which team. It's silly.

<comment><USER34> I can see how that's annoying for sure.

Some people like me see the discrete accounts as a definite positive though!

</comment></comment></comment><comment><USER35> If MS can nail the creation and organization of "micro channels", that would be a huge innovation in this space that Slack has not been able to figure out yet.

We are a 100% remote company and we use Slack and Teamspeak as our primary communication tools. Our biggest problem with Slack is when multiple conversations are happening in the same channel. There should be a way to segregate those convos into separate "theads", without having to make new channels.

That would be a huge win for whatever chat product can solve this pain point. Unfortunately, innovation is probably not a huge payoff since the competition would simply copy the feature =(

<comment><USER36> Teams does allow you to respond to a specific message, linking messages together into threads. So you can have multiple threads into a single channel.

</comment></comment><comment><USER37> Dear Slack. No one cares.

<comment><USER38> Beginning of this year Slack was the new internet and now no one cares. For me it is just another chat client/server: not sure what the fuss was about. And not open source does not help nor does the non native client. Good to see companies continue trying as the revolutionairy collab tool does not exist yet.

<comment><USER37> Yeah! Everyone was foaming at the mouth about it. My BS detector was strong, now when I log into slack, look at my various channels, including the company. I just see tumbleweeds.

</comment></comment></comment><comment><USER39> A new dimension of passive aggressiveness.

Honestly though, that's hilarious.

</comment><comment><USER40> Wow.

Okay so, definitely not applying to work for Slack anytime soon. This is not the kind of attitude I would want to support.

</comment><comment><USER41> > _I love your new coat, Microsoft, it doesn 't even make you look that fat!_

Slack

</comment><comment><USER42> Of all the platforms my team has used, Slack has been the most "polished". Lots of things come under that, but its hard to define exactly. Speed, Design details, consistency etc. Product quality is just very high with Slack. Only Whatsapp comes close in some of those matters.

<comment><USER43> Right now, I am in 3 slacks, and have maybe 50 channels open. Slack is currently chewing 2+ GB of RAM. Its not that polished.

I also have irssi open. it has 7 IRC networks, and ~ 120 channels - its taking 144MB of RAM.

<comment><USER42> Good to know! I only have two teams on a Macbook. No issues so far. These are things that will matter if they want to continue growing at the rate they were used to.

</comment></comment></comment><comment><USER44> Translation: we're scared.

</comment><comment><USER45> #Yawn

Completely pointless. To think someone got paid to write that shite...

</comment><comment><USER46> Dear Slack,

Try sending them an email - it's a better medium (pun-intended) for this, than an open opinion piece pushed to the general public.

Unless of course, you're marketing, in which case honesty can be a much better policy.

<comment> xkcd-sucks Better yet, send it as a daily message from @slackbot

</comment></comment><comment><USER35> What is this letter in response to?

<comment><USER36> Microsoft Teams (Slack competitor) will be launched in a few minutes: [http://news.microsoft.com/november-2016-event/](http://news.microsoft.com/november-2016-event/)

<comment><USER47> Can ms do something better than the pile of crap that is Skype for business?

<comment><USER23> Just use Lotus Sametime for a few hours and you will worship Skype for business for life.

<comment><USER7> One slim advantage of Sametime, at least the ancient, pre-Eclipse clients, was that you could actually run multiple instances and sign into two accounts at the same time.

Lync has a stupid mutex that prevents this.

<comment><USER23> Ah, we don't have that problem because that is anyway forbidden, we are only allowed to use our official account.

</comment></comment></comment></comment></comment></comment><comment><USER48> Anyone feeling defensive or threatened today?

<comment><USER49> If I was Slack I would be terrified. My only USP is a chat client. Microsoft has an easy to integrate with other MS products chat chat system.

</comment></comment><comment><USER10> While I was aware of the MS product, I never took it seriously (I mean, it's from the same company that rebranded Lync to Skype for business).

Now that Slack has posted this, I intend to give this a look and a trial (hopefully it does not require an O365 subscription) as it just tells me that Slack is either annoyed or scared of Skype teams (either one being a good enough reason for consumers to check it out)

<comment><USER50> I think they're scared that Microsoft Teams will take part of the enterprise market segment. The part that already uses a lot of other Microsoft products. So Slack will likely lose out on that now. People who don't use anything Microsoft will likely not take a second look at Microsoft Teams.

I'd personally probably rather look at something open source and self hosted, if I was going to move away from Slack.

The only real thing that I image Microsoft Teams would be able to do that Slack can't, is integrate with MS tools better and easier.

</comment></comment><comment><USER49> I wonder when the Google apps suite will bring a rich chat feature to enterprise. The G suite is promising.

</comment><comment><USER51> Ugh.

If you want to do something like this _right_ , the way to do it is the way Data General wanted to do it when IBM, then the 800-pound gorilla of the computer world, entered into DG's minicomputer market. (Which is described in Tracy Kidder's classic book _The Soul of a New Machine_ ([https://www.amazon.com/Soul-New-Machine-Tracy- Kidder/dp/0316...](https://www.amazon.com/Soul-New-Machine-Tracy- Kidder/dp/0316491977))

The ad they proposed was much simpler -- a full page that said only the following:

 _They say IBM 's entry into minicomputers will legitimize the market._

 _The bastards say, welcome._

</comment><comment><USER52> Sometimes it's worth not writing things like this ...all Slack managed to do here was draw more attention to Microsoft's announcement. A way of marketing $$ if you ask me.

</comment><comment><USER53> To me Slack was always just a shiny IRC clone where your sensitive communication data sits inside an American company.

I find it ironic that Slack calls Microsoft out for making something like Slack for the enterprise.

I sincerely hope one day we'll have something like [https://matrix.org/](https://matrix.org/) and [https://riot.im/](https://riot.im/) as the standard.

</comment><comment><USER54> Since all the posts here seem pretty negative about Slack's response, I'm genuinely curious...how do you think a smaller company should respond when a huge company decides to enter their competitive space?

I ask because I've been an employee at multiple start ups before and seen this kind of situation happen multiple times. I have not yet seen a response that gets received well. And silence gets perceived by existing customers as weakness or fear.

</comment><comment><USER55> > So welcome, Microsoft, to the revolution.

Honestly slack doesn't feel much different than IRC and wasn't it built on this originally. What revolution is this anyway? Lame.

And MS has had yammer for a while now (2012).

[https://www.yammer.com/](https://www.yammer.com/)

[https://en.wikipedia.org/wiki/Yammer](https://en.wikipedia.org/wiki/Yammer)

<comment><USER7> Yammer died the minute that acquisition went through. I was working on a data compliance tool for Yammer at the time, and my employer had a handful of large enterprises lined up to purchase it. When the news broke that Microsoft had bought them, they all went _noop_ and abandoned Yammer.

Four years later, Yammer looks exactly the same, and the data access API that I was using, which was supposed to be facing imminent deprecation, is still there.

<comment><USER55> > Yammer died the minute that acquisition went through.

Somewhat comically yammer was just rolled out to my company this year. Perhaps we got free licensing for it. I'm not sure.

For offices with mostly remote users slack or even yammer might be a great option for dialog. I'm in the email camp for a running dialog of well thought out replies or a group IM for rollouts or some other conversation that can't be done face to face.

</comment></comment></comment><comment><USER56> What a disaster.

They speak about an open platform (forget open-source), just being open to communities that might benefit from their expertise.

Here is a piece from ages ago, what slack means for huge communities: [https://medium.freecodecamp.com/so-yeah-we-tried-slack- and-w...](https://medium.freecodecamp.com/so-yeah-we-tried-slack-and-we- deeply-regretted-it-391bcc714c81)

</comment><comment><USER57> News to me that MS is launching a new product. I hate slack to begin with, but after using 'Skype For Business' my expectations are pretty low with this new product.

I'm assuming it's going to fail much like Yammer did - the screenshot looks like a rip-off of Slackanyways...

</comment><comment><USER58> Slack somehow became the de facto standard for "internal IRC" but i do think it does many things not particularly well. Then writing an article like this as if their product is the reinvention of human communication feels patronizing and VERY cringy (to me anyway).

</comment><comment><USER59> This was to expect. I don't wanna know how much companies didn't use Slack just because it seemed like something crazy and hip.

Now, before Slack has the time to become "old, known and reliable" Microsoft jumps in and steals these potential customers :D

I like it.

</comment><comment><USER60> Flickr tried to patent something that I had invented. After the acquisition the Yahoo lawyers caught this and had to add me to the patents.

The "we are better" thing they claim is just marketing. They are as cutthroat as anybody else.

I have plenty of other stories.

Beware.

</comment><comment><USER61> The biggest keyword in this "ad" is their competitor's brand name.

</comment><comment><USER62> This was Slack's Liam Gallagher, moment. At the peak of their fame Oasis shits on the Beatles and every great band that has come before.

McCartly responds- "Now go and do better than The Beatles did"

</comment><comment><USER25> How did this get buried from the front page so quickly? Hmm.

</comment><comment><USER63> "When we ... agonize over the best way to slip some humor into release notes, ..."

Ugh

</comment><comment><USER64> Narc Rage, anyone?

</comment><comment><USER65> This was one of the most stupid post I've seem a company publish. I don't know about what he is talking but I'm going to search it with a feeling that this guy is a looser.

Edit: more stupid than I thought! They bought a full page ad. Edit 2: I hate M$, but Slack doesn't help either.

</comment><comment><USER66> Thanks to Slack we now know that MS is launching a competitor.

Never mention the competition, always focus on your shit.

But yeah, we get it - first Facebook Workplace, now MS Team. Slack might get pushed into the feature/addon category faster than it can move into full product. Hard to reach the crazy valuation that way.

<comment><USER67> Yes, I found that strange. After reading the first few words ("Congratulations on today's announcements.") my first thought was, "What announcements?" .. then after reading for 30 more seconds the second reaction was, "Slack is scared and not happy about this", which is something that wouldn't have crossed my mind had they not written this.</comment></comment>


Machine Learning Crash Course: Part 2 - <AUTHOR> https://ml.berkeley.edu/blog/2016/12/24/tutorial-2/ <comment><USER2> Now that there's a bunch of AI/ML-related links in the front page, probably now is the best time to ask:

As I learn deep learning, from the practical point-of-view, I found that the idea is simply to feed some "black box" with labeled data so next time it can give you correct label given unlabeled data. In essence, it's pattern recognition. What do you think?

And then, as I try to find use cases for ML (you know, finding problem for the solution), I found that actually, many problems that can be solved with ML can actually be solved with rules. For example, detecting transaction fraud. You just need to find the right rules/formula. Forget ML, if you can't hardcode if-else, just use rules engine. What do you think?

So, I'm starting to think that ML is good for solving problems where (1) we're too lazy to formulate the rules, or (2) the data is too complex/big to analyze by rules (as in, understanding image or voice). What do you think?

<comment><USER3> ML is about creating a process that produces models from data in a way that is likely to generalize to new data.

For example if you just start trying lots of rules by hand on your fraud data set there is a good chance you'll come up with a rule that looks good on your data but doesn't generalize to new data.

The number of models (or rules or formulas) you try by hand increases this chance (this related to multiple testing) and worse the process that generates them isn't repeatable since one you know a feature worked on the data set you're biased towards finding it again.

So in ML you try to come up with a model generating process that is entirely automated and repeatable. This means you can do it repeatedly in cross validation, over bootstraps, out of time etc on the same data set and be fairly sure that it will generalize.

The goal can still be a simple rule or formula but you achieve this simplicity for simplicity by penalizing complexity (as in lasso) or doing explicit simplification (as in pruning).

The reason that complex "black box" models are so popular is that they often have really nice statistical properties in terms of generalization. It's fairly intuitive that averaging over a bunch of slightelly perterbed simple models will give you a nice combined model as in a random forest, gbm or other ensemble.

Deep neural networks are less intuitive but it's been hypothesized that the depth makes them less prone to overfitting than a shallow network.

A shallow network will have one global optimal set of weights that's simple and really good but it also has lots of locally optimal states that aren't as good...and the chances are high your training procedure (or manual search for a simple model that works) will get caught in one of these.

It's been shown that the for deeper networks these local optima tend to be much closer in performance to the global optima so in effect making the model more complex makes it less likely you'll end up with a bad model.

</comment></comment><comment><USER4> Here's _Part 1_ : [https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/](https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/)

</comment><comment><USER5> A comment I made within this post: Fitting a formula to data implies that there are reasons for the formula to be this and not something else. For example one can prove, within the current model, that the speed of a falling object is v(t) = 1/2 g t^2 and it is reasonable to fit experimental data to this formula to determine g. It may be a false assumption (it is, actually) and this is the whole idea of modelling and understanding limitations of a model.

In contrast, the article assumes that a linear expression will be fitted to any kind of data as if they behaved, by miracle, in a linear fashion. Any kind of deduction from this will be false, except if, again by miracle, the data actually behaves linearly.

I am a big fan of clusterisation and data behaviour discovery - the process which highlights relationship between data we do not know anything about. I believe this is a huge win in ML. Fitting something (1D, 2D, ...) to data without a model and drawing conclusions is at least perilous.

</comment><comment><USER6> does anyone have recommendation on a text that is mathematically heavy but also looks at modern approaches/appleications?

<comment><USER7> You can order the standard machine learning texts from most to least math-y, and least to most modern:

\- Pattern Recognition and Machine Learning (Bishop 2007)

\- Machine Learning: A Probabilistic Perspective (Murphy 2012)

\- Deep Learning (Goodfellow, Bengio, Courville 2016)

If you want cutting-edge material, read the Deep Learning book (which is still quite technical, though some of its content may be outdated in a few years). If you want timeless mathematical foundations very clearly presented, read Bishop. Murphy is a good middle ground.

If you're self-teaching and have trouble focusing on a textbook for long periods, the Stanford CS229 lectures combined with Andrew Ng's course notes and assignments are probably the best resource. They are still quite rigorous and working through them will give you a solid foundation, after which you'll be more prepared to understand the deeper content in any of the texts above.

<comment><USER8> Where would you place "Elements of Statistical Learning" in relation to these, if you know?

<comment><USER9> You want ISLr ([http://www-bcf.usc.edu/~gareth/ISL/](http://www- bcf.usc.edu/~gareth/ISL/)), not ESL. ESL is the prototype for the former.

<comment><USER8> Interesting, thanks - I wasn't aware of this text.

But according to your link, ESL isn't a prototype of ISL, but a more 'advanced treatment'. The R applications in ISL seem like they might be very useful, though.

<comment><USER10> ISLR was written after ESL to be an easier to read version. I guarantee you will have trouble if you try to teach yourself ML with ESL.

</comment></comment></comment></comment></comment></comment><comment><USER11> I also have a question regarding ML. Are there resources where I can see how could I treat video sequences (series of images / spatial/temporal continuity) as inputs? Trying to find a starting point for learning and use case I have in mind.

<comment><USER12> There's a really straightforward approach to turning video into feature vectors which you can readily plug into any old ML algorithm. You can turn every decoded frame (which is essentially a PNG, right?) into a (Width*Height)x1 vector, where each cell is an RGB pixel. You can then compose these vectors into a matrix, or perform further operations on them, e.g. SVD. Whether or not this is a good approach, though, will depend of course on your application. For face detection, for example, this is not adequate, and you'll need more sophisticated algorithms. For a good overview of that, see Jason Lawrence's slides[1] and also take a look at some of the other stuff he covered in his Computer Vision course[2].

Sorry if that's too basic for what you were asking. If you want to see some messy code that does this using OpenCV, here's some I wrote a while back with a friend, starting on line 127: [https://github.com/sprestwood/CompVisionS2015/blob/master/te...](https://github.com/sprestwood/CompVisionS2015/blob/master/test_scripts/people_detector.py#L127)

[1] [http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/le...](http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/lectures/lecture07_recognition.pdf)

[2] [http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/sc...](http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/schedule.html)

<comment><USER11> Thanks for pointers! I'm basically below/at 101 with ML, but have background in computer graphics.

I was wondering if one could utilise ML for either or both of two things: object outline spatial/temporal (feathering can solve for motion blur) and better Chroma key.

<comment><USER12> For outlining, my initial reaction is to reach for two things:

1\. Kalman filters to track an object in motion within a frame [1]

2\. Edge detection on the subframe you got from (1) [2]

Both appear to be available out of the box in OpenCV[3][4], though you'll have to fiddle with parameters I'm sure.

[1] Example: [https://www.youtube.com/watch?v=K14SK4v3-IY](https://www.youtube.com/watch?v=K14SK4v3-IY)

[2] [https://en.wikipedia.org/wiki/Canny_edge_detector](https://en.wikipedia.org/wiki/Canny_edge_detector)

[3] [http://docs.opencv.org/trunk/dd/d6a/classcv_1_1KalmanFilter....](http://docs.opencv.org/trunk/dd/d6a/classcv_1_1KalmanFilter.html)

[4] [http://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/ca...](http://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/canny_detector/canny_detector.html)</comment></comment></comment></comment>


Top bottled water brands are contaminated with plastic particles, study finds - <AUTHOR> https://www.japantimes.co.jp/news/2018/03/15/world/science-health-world/top-bottled-water-brands-contaminated-plastic-particles-study-finds/ <comment><USER2> Better reporting with links: [https://www.theguardian.com/environment/2018/mar/15/micropla...](https://www.theguardian.com/environment/2018/mar/15/microplastics- found-in-more-than-90-of-bottled-water-study-says)

The study was commissioned by Orb Media, who have more information on methodology and the full lab report: [https://orbmedia.org/stories/plus- plastic](https://orbmedia.org/stories/plus-plastic)

</comment><comment><USER3> I used to buy a lot of water and can taste the plastic. Large multi-gallon jugs are the worst. Volume packaged small bottles can be bad too. If you wonder what to look for - drill a hole in a plastic bottle (had to, since the lid wasn't opening) and look at at your own risk taste the water afterwards. You won't forget the taste and will be able to detect it elsewhere. With BPA causing estrogen surges, it can be valuable to have some kind of detection mechanism, especially at conferences and events.

</comment><comment><USER4> I drink tap water instead because the minerals are not filtered out of it.

<comment><USER5> Unfortunately it is not suited for drinking everywhere, even with a passive water filter.

</comment></comment><comment><USER6> looks like a subtle ad for San Pellegrino water (has the least, hence assumed to be the safest)

</comment><comment><USER7> Most important line in the article, by far: (edit: if you happen to live in much of Europe, Japan, Canada, or the United States, among other places, like I do)

 _"Tap water, by and large, is much safer than bottled water," said Mason._

<comment><USER8> How about lead in old pipes in parts of the city where the water might pass through? Lead seems more dangerous than plastic particles.

Any way to filter lead out with certainty?

<comment><USER9> > Whataboutism (also known as whataboutery) is a variant of the tu quoque > logical fallacy that attempts to discredit an opponent's position by > charging them with hypocrisy without directly refuting or disproving their > argument, which is particularly associated with Soviet and Russian > propaganda.When criticisms were leveled at the Soviet Union, the Soviet > response would be "What about..." followed by an event in the Western world.

<comment><USER10> I'm not a big fan of accusing people of whataboutism in general, but in this case, it's not even relevant. Your parent comment asked, "Ok, bottled water has plastic in it. I know that some tap water goes through lead pipes. Is tap water going through lead pipes still more healthy than bottled water with plastic in it?"

The question was a very reasonable one, and just happened to contain the phrase "what about".

<comment><USER9> Yeah, you're right. I understood their question differently as I brought along a defensive stance from a previous discussion.

</comment></comment></comment></comment></comment><comment><USER11> >"It's more urgent now than ever before to make plastic water bottles a thing of the past."

Great way to end good research with retarded political statements. What exactly will replace plastic bottles?

<comment><USER12> Maybe we could build some kind of system of pipes which would convey potable water from a central treatment facility out to homes and offices?

<comment><USER13> But where exactly is the end-game profit in that? I'm not seeing how this idea could grow to be the next Uber.

Maybe if you had an app with a social or sharing economy component?</comment></comment></comment>


Ask HN: Have offer, still talking other companies - <AUTHOR> I have an offer than I would be happy to take however I am still discussing opportunities with other companies. Should I tell the company that has given me an offer that: I accept and give my notice, possibly quitting before starting? That I have other offers I am working on, hold on?<p>Thanks <comment><USER2> Don't tell them you have other offers. Set a reasonable date by which you should decide (a few weeks at most) and conclude all your negotiations by then, tell the 'winning' party that you accept their offer and hand in your notice.</comment>


The birth and death of a bike company: What happened to SpeedX - <AUTHOR> https://cyclingtips.com/2019/06/what-happened-to-speedx/ <comment><USER2> Is it just me, or are there seemingly far more people trying to invest and profit off bikeshares than there are using them?

I've lived in several cities in europe and i'm not actually sure i've ever seen anybody using them. Yet there seems to be tons of competing companies putting out bikes in the street.

On top of that, they don't even seem to bother marketing or anything. The first I'll hear of a new company is seeing a new colour of bike.

The idea is sound in practise, but people seem to place great value owning their own transportation and the flexibility it affords them.

<comment><USER3> I've pondered that question too. Something like Uber or Lyft works partly because of the cost and burden of car ownership. It's attractive to use someone else's equipment if you don't have to buy, maintain, finance, fuel, insure, and store (park) similar equipment yourself, to the tune of thousands of dollars per year.

But then you look at bike ownership. I almost want to put "bike ownership" in quotes because even the mere phrase makes it sound so much more serious and demanding than it is. It's so easy and cheap, it seems obvious you would just do that. Or put it this way, I don't find myself going "I just don't want & can't handle all that responsibility of a bike!" It's like, boy you better not get any houseplants then, that much responsibility will stress you right out!

<comment><USER4> The 'problem' you're looking to solve is not the hassle of bike "ownership" but bike access. If I take the car or bus to work I can't then take my bike to meet a client. If I take my bike into town I can't then take the bus home if the weather turns bad or I have a lot shopping to carry. If I take my bike into a 'bad' part of town to go to a bar I have to worry about the very real chance that it won't be where I left when I want to go home at night. Bike sharing solves all these problems.

<comment><USER3> If you take the bus home because of bad weather one day, and leave your bike at work, it's there for you to go meet a client the next day! :P

No but seriously. Sounds like your transit agency doesn't have bike racks on the buses. That's what enables the integration with transit. Around my area you can take a bike on the bus or light rail.

All the other problems have pre-existing solutions though too:

Groceries to carry → Get panniers, and don't buy more than they can hold (you can always get more tomorrow, and it'll be fresher).

Rain → Wear rain gear.

Bad part of town → Own a modest bike and a good lock.

The extreme mode flexibility you mention sounds nice, but I've never found I've needed it. With a little advance planning and the extra bits above, you can just commit to one mode.

</comment></comment></comment></comment><comment><USER5> I always find it shocking to hear that so many people think crowdfunded projects have no risk, that it's somehow similar to ordering stuff from Amazon. Especially when people invest more money than they are actually willing to lose. Whose mistake is that? The customer for not understanding what they are into? Platforms like Kickstarter for not explaining it clearly enough?

</comment><comment> at-fates-hands I can't think of an industry that hasn't been affected by Chinese manufacturers trying to undercut American or European companies by short cutting the manufacturing process.

I just see the same thing with this bike company. Ignore all the rideshare stuff, the push to grow so fast and not having a sustainable product and its the same story. They took a standard Aero fram and tried to make it appear as a "high end" product. The problem is, once you've ridden a high end road bike you suddenly realize the amount of engineering and R&D that goes into those bikes to get them to perform at that level. You can't just reproduce that overnight, it takes time and a lot of money. From the article (and the scathing reviews of the bike) it didn't sound like they really tried to make a quality product, just another carbon Aero knockoff with a cool integrated stem. At that point, it was easy to undercut your competition on price.

The lessons are all too clear and which you see a lot of. You can't cut corners on products like this and sell it off as "high end" and expect to get away with it.

Source: I worked in the bike industry for almost 15 years. Had various stints with Specialized and Cannondale.

</comment><comment><USER6> I didn't know this CEO was the same as from Bluegogo. When that bike share company entered the Bay Area market, they helped push out artists from a West Oakland warehouse studio called Lo-bot and were occupying it for maybe six months before disappearing. The place is still empty.

</comment><comment><USER7> It's a shame that there's been little innovation from road bike manufacturers in the past 10 years or so. The last major innovation was aero carbon frames. Since then everyone is just copying each other and trying to improve by fractions of a percent for weight, stiffness, and efficiency.

Most of the value and innovation has shifted to the groupset suppliers, with Shimano taking the lion's share of the market and a few smaller players picking up scraps. When all the important components come from Shimano it barely matters who manufactures the frame and assembles the complete bike.

At this point road bikes are about as good as they will ever be within the limits of available materials and UCI rules. It would be nice if one of the big bike manufacturers would make their own high-quality components to compete with Shimano. Or throw out the UCI rules and build a fast, comfortable bike for customers who don't care about competing in sanctioned races.

<comment><USER8> Why bother innovating when its not needed? Expensive bicycles are in the same market as fancy watches. Its all just wealth signalling. Nobody needs 10% lighter frames, or 5% more aerodynamic shapes. If you want to drive faster or burn less energy you go electric, If you want health benefits you buy the shittiest heaviest bike you can find and pedal harder.

<comment><USER9> Don't get a shitty bike if you want exercise. Shitty bikes are unpleasant to ride, making the experience not fun, and discouraging you from doing more (and longer duration) riding.

Also, safety is important, and shitty bikes are less safe.

<comment><USER10> This response to the person ahead of you makes me think you either don't know much about bikes or you missed their point.

I think we all agree, buy something shitty and you get shit, especially with things you put between you and the earth. A medium quality bike, which costs an order of magnitude less than a "fancy" bike, will do literally everyone who isn't a cat 1-2 racer fine, objectively. Price difference is $1-5k for a mid range vs $10-20k for top of the line.

I don't think anyone is advocating for $300 walmart bikes here.

<comment><USER11> You could win the tour de france doing the road stages on a medium quality bike. (if you put good tires on it). Also given your price range, some of the objectively best bikes (meets weight minimum, good aerodynamics) are in the $5k range, and the more expensive ones are worse but have big name value!

<comment><USER12> That's definitely not true...at the level of the TDF, every gram of excess weight matters, and there's a vast weight difference between a cheap $5k bike and a "fancy" $20k bike.

If you're not at the TDF level, there's no point in spending more than $2k on a bike since the difference in weight won't matter.

<comment><USER13> If every gram matters why do they wear those little hats under their helmets?

<comment><USER9> Not sure what you're talking about? Can you provide photos?

<comment><USER13> Sorry, don't do photos. Feels like an invasion of their privacy.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> A very well written article. I like the point about how the follies of the bike sharing business have actually got people to ride bikes. Before it was not an option to hire a bike for a day, nowadays that is a thing.

It takes a while for things to find their level. The 'Boris bike' is now a feature of London, the other bikes haven't established that niche in the streetscape.

The road bike was also innovative, if flawed. Plenty of bikes and plenty of bike groupsets are flawed when they first come out. It takes a couple of iterations to get right and the costs of getting it wrong are terrible.

Years ago in the days of the mountain bike craze a brand came out with a few of their own components including a bespoke freehub body. This did not last long on any given bike and it did not take long before the warranty returns aspect of the business fixing whole wheels took over. There was no scope to develop the brand when all hands are literally swapping tyres and tubes onto new wheels and getting them out to customers.

Sometimes bikes that don't actually work don't bounce back. During the mountain bike boom there was a shortage of Shimano components and a rising Yen. This made it hard to get the gears and brakes needed to build a bike. So European components might be substituted. And they would not actually work together, no matter how skilled the mechanic, no matter how new everything was, the gears would not shift and the brakes would not stop the bike.

Very few of these frankenbikes made it back. They just sat in sheds and garages, never to be ridden!

The SpeedX bike was not a bike that would be just dumped in the shed or bought as a well meaning gift. They could not rely on 90% of the problems not coming back.

They also were never going to do well with the cycling press. Or the industry. The cycling business is actually really conservative and there are chunks of it controlled by people who never actually ride..

</comment><comment><USER15> > Bluegogo's contribution alone was 800,000 bikes, impounded in cities across > China – wave after wave of blue bikes, piled two high, sitting in > construction sites and empty lots on the outskirts of megacities, waiting. > Decaying.

> Zoom out from the specific, and you quickly start to drown in the enormity > of it all. Millions of bikes, assembled by thousands of hands. Entire towns > like Wangqingtuo, built around servicing a rapacious demand. Restaurants and > hotels and supermarkets, constructed and opened and shuttered in what are > now virtual ghost-towns. Countless families relying on the bikeshare > industry to survive. Seats and spokes and tyres beyond number, consigned as > fast as they were made to technicolour scrapheaps around China. All of it, > this vast mound of human toil, left to rust and rot among bamboo and rubble.

I am continually facinated with China and it's people. But in instances like this I get a real feeling of looking into the eyes of Yog-Sothoth, the Gate and the Key.

</comment><comment><USER16> This feels very typical of a Chinese "startup." Any company marketing its leadership on being some "whiz kids" is a red flag.

A gigantic amount of Chinese entrepreneurship is just about people trying to game the insane capital market in China under every pretext possible, without any genuine concern for the business being sound. Not much different from what you see in the US, except for even less curbs on outright fraud. After all, all of that came to China along with the first wave of "returnees" from US West Coast.

In this case, you have to admit, the company simply ran out of money, and you can not put that in any other way. Whether they would've or wouldn't have been descended upon by the communists wouldn't have changed anything about that.

I noticed a tendency that in China, whatever company can be called a substantial, solid business is much more likely to be privately owned, low profile, and kept as far away from capital markets as possible.

Best example — Bu Bu Gao group, the second in line after Samsung in smartphones, but is almost invisible.

Espressif recently went with a placement, but also very low profile, without much noise, and on a 3rd tier burse oriented to institutional buyers.

Allwinner, Rockchips — both companies are past their prime, but still super duper secretive with regards to ways of company's leadership.

<comment><USER17> > In this case, you have to admit, the company simply ran out of money

You seem to have skipped the paragraph with the little tanks. Arbitrary plot twist it out of a clear blue sky, would be too absurd even for an episode of South Park.

<comment><USER16> No I didn't. That has nothing to do with them simply not having a way to get profitable, and running out of initial capital. That would've happened invariably of then being raided or not.

</comment></comment></comment><comment><USER18> I wonder how does Kickstarter deal with credit card chargebacks? Their support site basically says "talk to us directly please"[1], but it's common for chargebacks to be awarded to customers regardless of arguments provided by a merchant. Would they just consider it the cost of doing business or try to recover the money from the backed project?

[1] [https://www.kickstarter.com/disputes](https://www.kickstarter.com/disputes)

</comment><comment><USER19> The article challenges Kickstarter to refund their 5% commission when companies go bankrupt. Seems like the right thing to do and could win back more customer loyalty.

<comment><USER17> It would be seen a nice symbolic act by some, but as an admission of guilt by many others. This would not end well.

</comment></comment><comment><USER20> Another reason to not support large ticket items on Kickstarter.

</comment><comment><USER21> There are no noteworthy novelties in this kickstarted bike. I'd personally venture to lose some money if I saw some real innovation and I wouldn't bother even if they failed to deliver the end product or the company went bankrupt. Here we see only some Chinese copy work.

</comment><comment><USER22> Oh man, I remember when SpeedX first came out. I can't describe it exactly, but it was an immediate red flag for me. Basically they tried to fix _everything_ about buying a bike in one fell swoop and said it was going to cost less than other providers. If they were able to provide a superior experience to bike sizing, bike computer, wheels, frame, etc all in one fell swoop, then I would expect the overall product to be more expensive...not cheaper. Huge red flag.

<comment><USER17> > I can't describe it exactly, but it was an immediate red flag for me.

The universal playbook of all bike related crowdfunding campaigns: take a tiny innovation (or none at all, it's not strictly necessary to have something original), put it in a blender with everything that has become state of the art within the last decade or two and use it to impress those who have never experienced anything better than a walmart clunker from the falling edge of the original mountain-bike wave.

</comment></comment><comment><USER23> > At the time, early 2016, aero was the next greatest advance in the cycling > tech space; like the Giant Propels and Specialized Venges it was pitched > against, the Leopard featured a snugly tucked rear wheel, a wind-cheating > profile and hidden cabling.

As a road cyclist - I'm no expert on this, but even in 2004 such a thing was becoming commonplace in Chinese CF frames and companies that were trying hard to copy Cervelo.

<comment><USER24> I think you're mistaken. In 2004 Cervelo had just began offering their Aero bikes, but they were aluminium, not carbon. See the Cervelo Soloist Team.

The manufacturers using carbon like Time, Look were not producing anything 'aero'. Giant's TCR (ridden by T mobile) in 2004 was carbon front triangle with aluminium seat and chain stays. And again, not 'aero'.

Lots of interesting details from here: [http://autobus.cyclingnews.com/teamtech04.php?id=tech/2004/p...](http://autobus.cyclingnews.com/teamtech04.php?id=tech/2004/probikes/stevenson_bianchi%20)

</comment></comment><comment><USER25> Meanwhile incredibly Detroit Bikes appears to be doing quite well. So well in fact that they're attracting multiple competitors in Detroit. Perhaps because they're not competing on price and some people still value American made?

[https://detroitbikes.com/](https://detroitbikes.com/)

<comment><USER26> I got my C-Type from Kickstarter, and while it's more expensive than a big box stores, I felt it was still pretty competitive for the quality.

It's only had preventative maintenance in the years I've ridden it. I grew up biking, and I've never had one more reliable.

I just wish the frame had rack mounts for a pannier rack.

</comment></comment><comment><USER27> I'm pretty sure that Playa Bike Repair bought a ton of those bikes last year for renting out at Burning Man. Seemed to work pretty well!

</comment><comment><USER28> Fascinating story. I wonder what happened to that game company with the little tank, it probably shutdown by the police as well... 4th June is a very sensitive word in China...

</comment><comment><USER29> That was one expensive mistake

</comment><comment><USER30> I wonder if the boss is still driving the lambo. (actually I shouldn't have to guess)</comment>


Battle of the Clipper Chip - b-man http://www.nytimes.com/1994/06/12/magazine/battle-of-the-clipper-chip.html?pagewanted=print&src=pm

<comment><AUTHOR> _The Government understands the impossibility of eradicating strong crypto. Its objective is instead to prevent unbreakable encryption from becoming routine._

Wow! While this article was from 1994 this is a telling quote.</comment>


How the Pentagon punished NSA whistleblowers - <AUTHOR> http://www.theguardian.com/us-news/2016/may/22/how-pentagon-punished-nsa-whistleblowers <comment><USER3> That it's unconstitutional is bad enough. But the ineffectiveness of the NSA is the most damning aspect of all:

> Drake had discovered a shocking example while researching his postmortem > report on the September 11 attacks. Months beforehand, the NSA had come into > possession of a telephone number in San Diego that was used by two of the > hijackers who later crashed planes into the World Trade Center. But the NSA > did not act on this finding.

> As Drake later told the NSA expert James Bamford, the NSA intercepted seven > phone calls between this San Diego phone number and an al-Qaida "safe house" > in Yemen. Drake found a record of the seven calls buried in an NSA database.

> US officials had long known that the Yemen safe house was the operational > hub through which Bin Laden, from a cave in Afghanistan, ordered attacks. > Seven phone calls to such a hub from the same phone number was obviously > suspicious. Yet the NSA took no action – the information had apparently been > overlooked.

If you are going to shred the foundation of jurisprudence, you should get something for your trouble.

<comment><USER4> Everyone seems to presume the NSA has one overarching goal in what it does, but I thought it was obvious that it answers to the demands of the POTUS. Shouldn't conclusions about "effectiveness" and "competence" stem from what a POTUS's intent is when employing it? These aren't going to be satisfactory arguments against it (viz., "ineffective" or "incompetent").

<comment><USER5> I bet it is just the opposite. The likely problem is that it exists beyond purpose. The middle managers that would be the people proposing and creating the structures that would notice and utilize the 9/11 intelligence are focused on their careers.

<comment><USER6> I've started using the phrase The Silent State for that phenomenon.

[http://www.goodreads.com/book/show/7755493-the-silent- state](http://www.goodreads.com/book/show/7755493-the-silent-state)

To see this in action, pick a hot topic, and watch any public hearing where our representatives are trying to get straight answers from department heads and middle management. It's infuriating.

Effective governance, transparency, accountability are still unsolved problems. Meanwhile, more eyeballs and fact checking can help, given enough effort and endurance.

</comment></comment></comment></comment><comment><USER7> > "Hillary Clinton argue that Snowden broke the law when he should have > trusted it. "He could have gotten all of the protections of being a > whistleblower," Clinton said in the first Democratic presidential debate > last October. "He could have raised all the issues that he has raised. And I > think there would have been a positive response to that."

Tell that to Thomas Drake. Tell it, for that matter, to John Crane."

<comment><USER8> Not to mention that Hillary also improperly handeled classified documents.

<comment><USER4> But she's on the side of the MIC, so she'll always be treated with kid's gloves about that. Proof? She's still running for office, not being pursued for law violations.

<comment><USER7> If you live in CA, NJ, Peurto Rico, ND, SD, MT, the Virgin Islands or New Mexico, and are eligible to vote in the Democratic primary you can still vote against her and vote for Bernie Sanders, a man who once called for the abolition of the CIA just prior to the Church committee.

Contrary to media meme, it is not mathematically impossible for Bernie Sanders to win the majority of pledged delegates, he would just need around 68% of the remaining vote.

It would take a decisive swing in the polls for sure, but not impossible. There are two weeks left and with Trump beating Hillary in many national polls, perhaps enough Democratic voters will see that a vote for Hillary is almost the same as a vote for Trump.

The FBI investigation is yet to have concluded as well.

And if that decisive swing doesn't come, I hope that the superdelegates, who will be the deciding vote in any event, will do what they were put in place to do in the 80's, and that is prevent the nomination of a candidate that would lose in the Fall election.

Are they undemocratic and a disgrace to the party? Yes. But the rules are the rules, and the process is the process. They were made unbound delegates by the party for this exact reason.

Make no mistake, if Bernie Sanders was ahead in pledged delegates, but behind in head to head polling with Trump, the superdelegates would not hesitate to deny the will of the majority of voters, that is why they were there in the first place.

They are also there to create a psychological advantage to the establishment candidate, who gets to include over 500 delegates in their count based on a poll of super delegates, who don't even vote until the convention.

The unwritten rules are different for the connected elite than they are for outsiders and whistle blowers.

My analysis is that at this point Trump is the heavy favorite to win the Fall election. Trump's negatives are already known and baked in, and enough people just don't care, while Hillary's negatives have not yet hit a floor. Trump has not even begun his attempt to define her.

Trump is about to bring up in an unavoidable way her ME intervention as Secretary of State, questionable Clinton Global Initiative deals, Whitewater, Filegate, Travelgate, her paid Goldman Sachs talks, as well as comments made against women who accused Bill of misconduct.

Trump will continue to be an asymmetric opponent, with tighter OODA loops than Hillary.

She will try to define him, and he will morph. She will make an attack, and he will dominate the next news cycle with an expertly planted submarine.

She won't be able to pin him down, and she will be constantly on the defensive.

Hillary's best and only hope is demographics and identity politics, but my bet is that Trump shaves off enough economically aspirational voters that identify or want his success to win.

<comment><USER9> _If you live in CA, NJ, Peurto Rico, ND, SD, MO..._

I already voted for Sanders in Missouri, over two months ago. I hope that no Bernie supporters in this state thought they could wait until now, because he only lost by 1500 out of 620k votes cast.

I do agree with you that at this point Trump is certain to beat Clinton by wide margins in the upcoming general election. She will attempt the same tactics that already failed for the Republicans. They'll fail for her too. I suppose I'll vote for Gary Johnson again. He might have more of a chance this time around, but mostly I don't want to waste a vote on the _status quo_. (Don't kid yourselves, Trumpets, Comb-over Donnie is as inside as inside gets.)

"Super"-delegates are kind of awful, but they would be less awful if they didn't declare their preferences until after the final state primary.

<comment><USER7> Corrected to MT.

And superdelegates change their preference, as they did in 2008, so there is no reason to include their numbers in media graphs until their vote is cast at the convention.

<comment><USER9> Well yes "media graphs" are _part_ of the problem but we have social media now so any "super" who wants to make her choice known to the public can do that without help from traditional media outlets. The Democrats really should impose this silence on themselves. I can see exempting e.g. governors from the gag, but others who are only "super" by dint of their positions as associate assistants to state committeepersons should just be quiet until the voters are done speaking.

</comment></comment></comment></comment></comment></comment></comment><comment><USER10> Whenever a story about Snowden is in the news, some people complain that some of the documents he released were "off topic". They accuse Snowden of releasing too much[1]. I agree, in an ideal world Snowden's bulk-copy methods shouldn't have been necessary.

Unfortunately, we don't live in an ideal world, and the "proper" methods of whistleblowing didn't work. When the proper methods fail, fallback methods are used. When legitimate channels fail, whistleblowing has to appel directly to the people. This is practically guaranteed to have collateral damage.

Anyone who is angry with Snowden's methods should work hard to create safe and effective whistleblower mechanisms.

[1] The cutoff point for "too much" varying with personal opinion.

<comment><USER11> > Whenever a story about Snowden is in the news, some people complain that > some of the documents he released were "off topic".

Which is odd, because I personally have not found any of the Snowden publications off topic or unnecessary. When I press these people about what they think shouldn't have been published, they always give me vague answers about "military secrets" and the like without citing anything specific.

<comment><USER5> There's a stage of disclosure prior to publication, where Snowden gave the documents to journalists. So it isn't necessarily possible to evaluate what he leaked by looking at what has been published.

It's a problem that the internal whistleblowing mechanism isn't working, but contractors shouldn't be picking reporters that then filter national secrets.

<comment><USER12> Why not? You use the word "should" without a good counterargument to the parent's point, that there was no other way to blow the whistle.

<comment><USER5> I don't see where they made that point.

Anyway, Snowden could have made sure to personally review everything he turned over. It's certainly an open question if such a leak could have been as effective, but it would obviously be an improvement in some ways.

<comment><USER13> And how is it that Snowden is more qualified than reporters to do that? The implication behind your argument is that giving the information to the likes of Glen Greenwald was dangerous. Are you asserting that it was plausible Greenwald would have sold it to Al Qaeda?

<comment><USER5> It's plausible Greenwald could have lost control of the documents (read [http://www.nytimes.com/2013/08/18/magazine/laura-poitras- sno...](http://www.nytimes.com/2013/08/18/magazine/laura-poitras- snowden.html?pagewanted=all)). Snowden is hopefully more qualified along those lines.

And yes, I do think that the person with security clearance minimizing their leak is a better than a reporter doing it.

<comment><USER14> Do you really think that Snowden could have kept unencrypted documents secret while still avoiding arrest? Would he have been able to keep them away from the Russians, and been able to transmit them to trustworthy reporters?

<comment><USER15> He did long enough to read and organize all them per interviews. All tgat's needed to filter it down to only illegal stuff withoug blowing ops.

Further, argument falls apart when he hands it over to journalists clearly unable to stop nation-state opponents per the very leaks. Of course someone hacked into them or bribed/coerced a copy. It should be assumed even without evidence given what we know.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> This was my main concern about how Snowden approached the whistleblowing - <USER2> seems that in practice, the whistleblowers got punished heavily for doing their job, so I have to say that I was wrong to question Snowden's methods, and the government needs to reform how it handles whistleblowing.

</comment><comment><USER17> I've learned about US's Office of Special Counsel - an agency that's tasked to protect government whistleblowers - from an episode of Person of Interest. In it, a low-ranking NSA worker tried to ask his employees about what he discovered, only to get terminated and framed for drug use; he called the OSC and in response to that the OSC sent his coordinates to a squad of black-ops hitmen.

While obviously work of fiction, it somehow _feels_ like an accurate description of how this works, and this article only confirms it. If you stumble upon a secret that could endanger important people in the government, the government ain't gonna be helping you. I think Drake and Crane are lucky to be alive today.

</comment><comment><USER18> I don't have a strong opinion on the whole article, but almost from the beginning it's less than honest.

Of all the whistleblower cases of the last 10 years, Thomas Drake has the most sympathetic. The conventional narrative about Drake appears to be mostly correct. But there are two very important details that this story not only leaves out, but implicitly contradicts.

1\. Drake did not follow NSA whistleblower protocol in revealing details about Trailblazer. Unless NSA whistleblower protocol includes "create a Hushmail account and then send classified documents to journalists over it".

2\. Drake was concerned about dragnet surveillance. But his concern wasn't "why" or "whether". It was "how". Drake blew the whistle on Trailblazer after a program he sponsored, ThinThread, was pushed aside for Trailblazer. ThinThread sounds a lot better in magazine articles, but it too was a large- scale dragnet surveillance system. The differences were that ThinThread was less expensive, and that ThinThread had better technical controls. But nobody on HN would be OK with ThinThread either.

For pretty much all of the whistleblower cases, you can go to the Federation of American Scientists (FAS.org) to get the original case documents.

<comment><USER19> Ah, I see, the agency which appears to not protect whistleblowers at all and fabricates information has official documents which claims he didn't follow protocol, which is probably because he was never taken seriously.

It seems with every agency with poor oversight finds some important reason to punish whistleblowers for some comparatively trivial infraction to the one they reveal. The information they reveal rarely leads to punishment but thr small (possibly made up) transgression ruins there career and gets them crucified.

By all accounts this belies a broken system.

<comment><USER18> You've missed my point. I'm not litigating Drake. I'm criticizing the article.

Again: the documents on FAS suggest that he _created a Hushmail account and used it to send classified documents to reporters_. "Not following protocol" doesn't capture that.

<comment><USER19> Perhaps for once someone is actually trying to tell his story rather than following official documents, which as I said, seem to be prone to a lot of bias and wrong doings.

It doesn't seem to invalidate the account, nor does it seem like the most important comment to give to this story.

<comment><USER18> Telling Drake's side of the story is a fine exercise, but it's not what this article purports to do. Instead, it's loaded with copy intended to bolster it's claims of being heavily reported and of breaking new news. But it doesn't do that.

Here's the kernel of what seems to have happened, synthesizing both the Crane reporting from the Guardian today and the Drake indictment:

1\. Drake favored a competing dragnet surveillance system that was pushed aside in favor of Trailblazer.

2\. Drake lodged a complaint with the NSA IG that Trailblazer was a waste of money with inadequate technical controls. He is joined in his complaint by William Binney and several other former NSA employees.

3\. Concurrently, Drake began leaking to reporters --- presumably, James Risen and Eric Lichtblau, who broke the first major NSA surveillance story for the NYT --- using Hushmail.

4\. In handling Drake's formal complaint, the NSA managed to leak Drake's identity to DOJ. The article is fuzzy about exactly which law this violates (it refers to the Whistleblower Protection Act, much of which does not apply to national defense and classified information, and not the ICWPA, which has different procedures and protections). Either way: the White House is apoplectic about the Risen/Lichtblau scoop and uses the leak to raid Drake.

5\. In the course of investigating Drake, DOJ discovers (perhaps by first ruling out Binney and the other original complainants) Drake's Hushmail leak. Drake is prosecuted.

The Guardian article would have you believe that Drake was prosecuted directly for blowing the whistle through formal channels. But that is not at all what seems to have happened. Yes, in the process of investigating Drake's complaint, there was a procedural foul (Drake's name shouldn't have ended up at DOJ). But it wasn't Drake's formal complaint that got Drake in real trouble!

Again: I think Drake has the most sympathetic case of all the NSA whistleblowers. His leak was extraordinarily targeted, not particularly political, and clearly in the public interest. I'm not writing to litigate whether Drake was right. I'm saying: the Guardian did a poor job of reporting this story.

<comment><USER20> You're also incorrect on many points here.

1\. You're making Trailblazer sound similar to Drake's favored system. This is extremely misleading, and is a long discussion on its own. But making this claim paints Drake's motives as career politics and seeks to diminish his character by inferring ulterior motives. If nothing else, that isn't a productive debate.

2\. Drake's complaint was not merely that it was a waste of money with inadequate technical controls. He was also not joined in his complain by William Binney.

Binney made the initial complaint, and it was to the Pentagon Inspector General, not the NSA Inspector General (if that is a thing), and he was anonymously joined by Drake and 3 others. This was in Two of the other complainants were former NSA employees who worked on the original program, and one was Diane Roark, a staff member on the House Permanent Select Committee on Intelligence, who had been trying to investigate the NSA on these matters (which is her job), and was getting stonewalled. Drake was still an NSA employee and did not want his career to be impacted, so his name was left off. Others, including Binney, who originally made the "competing system" initiated this because they felt that TrailBlazer was a modified version of ThinThread with all of the privacy protections removed and deployed domestically.

3\. Drake did not concurrently leak to reporters. His contact with reporters was several years later. The first report was internally, then he went to congress, then in 2005 he finally went to the media. The Binney complaint was filed in 2012, and the internal escalations within the NSA and Congress had already been happening for years before (ultimately leading to Binney's 2001 resignation in protest).

Drake's hushmail communication was to Gorman of the WSJ about Trailblazer, not James Risen and Eric Lichtblau of NYT, who instead had a totally different story, which was the AT&T warrantless surveillance issue. He set some ground rules for his communication with Gorman, one of which was that he would not reveal anything classified. He maintains that he never did. The outcome of the trial was that he never did. Yes, this contradicts the indictment, for very obvious reasons.

Risen/NYT on the warrantless AT&T issue was a totally different story and in fact had nothing to do with Drake! The investigators of the leak thought the sources might have been the same, and they were wrong.

<comment><USER18> No, that's not what I believe. As I keep saying: Drake has the most sympathetic of all the NSA leaker cases. What he revealed was very targeted and motivated by the public interest (something you cannot say of some other leakers who, for instance, leaked counterproliferation secrets, or details in preparation for a book they were promoting). I do not believe Drake's leak was careerist.

I bring up ThinThread because the article paints Drake as someone who was motivated to blow the whistle based on the same concerns Snowden had. But that's not what happened. Both ThinThread and Trailblazer were in large part motivated by concerns that more and more SIGINT had a nexus in the US, and that large-scale systems needed to be implemented to capture intelligence that had a US nexus.

When you look at the Drake complaint to the NSA IG through the lens of Snowden, your natural reaction is that the NSA was pushing the concern aside out of C.Y.A., to prevent the public from learning that it was monitoring US comms. But in fact, Drake's complaint is about _how_ Trailblazer was monitoring US comms; the privacy concerns it had were procedural and not substantive. It wasn't stark, the way the article makes it out to be.

I was imprecise about the word "concurrent". My frame of reference was "between the numbered steps in my outline". You're right to correct me. Either way, the point I'm trying to make is that it isn't the IG complaint that got Drake in trouble; it was the communication with reporters.

You can believe that the communications with the press were proper (I don't agree) or in the public interest to the extent that they overrode "proper" (I do agree). But either way, you should be able to agree with me that in an article that is entirely about what happens when insiders at NSA raise concerns through official channels, the fact that Drake got in trouble primarily for what he did through _unofficial_ channels is important to the story, and should have been prominently included.

<comment><USER20> Considering that you conflated the Risen/NYT leak with the Drake/Binney/ThinThread/WSJ story, I am not sure how you feel comfortable enough to say that Drake got in trouble for doing things through unofficial channels. If that is your central point, you got the most important detail here completely mixed up with another case. I charge that it is _you_ who is doing a poor job of reporting (to say nothing of the Guardian).

<comment><USER18> That's an association the article made, not me. The indictment and court filings aren't specific about which reporters were involved. Either way, can you rebut the argument I actually made?

I agree that if we're going to hold Guardian stories to the standards of Hacker News comments, I have much less of a case. That article would make a fine Hacker News comment.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER21> Yes, we need to punish people trying to bring systemic wrongdoing to light, and protect poor innocents like Ollie North from the consequences of their choices. /s

</comment><comment><USER22> _it didn't have to be a problem if everyone was a good team player._

The eternal line that makes law inefficient, and "government secrets" or "spy agencies" utterly untenable.

</comment><comment><USER23> Not surprising. All these guys want to protect their own fiefdoms.

</comment><comment><USER24> The larger lesson of stories like this is that trying to fight the system is mostly not worth it. Even if you in the end "win", it will have been at the cost of your career and life savings. And if you lose, you're in prison for a long time.

If you uncover something that seems very wrong, you are probably safe pushing back a little and making the matter known to your superiors. But if that isn't working, the best course of action is to get out. It won't stop the crap from happening, but you did all you could do without taking some very big personal risks, and at least the blood won't be on your hands.

</comment><comment><USER25> Maybe if the whistleblowers hook up with the militia movement...</comment>


C++ Is a Lie - <AUTHOR> http://harmful.cat-v.org/software/c++/I_did_it_for_you_all <comment><USER2> This reminds me, in spirit, of [http://cppclock.org](http://cppclock.org)

Who knows.. Python 2 didn't die until there was a countdown to tell us all to switch.</comment>