State.of.dev – Explore the current state of development - <AUTHOR> https://stateofdev.com/ <comment><USER2> I love the _idea_ of this: that there is a place you could go to get a pulse on the opinions of technology professionals regarding the tools they're working with.

So I apologize for the negativity that follows; it's a cool idea, and you've put work into it and made it look very pretty. That being said: unfortunately, without it being backed up by _data_, this is just one person's opinion and ultimately not very worthwhile as is.

Furthermore the presentation of the data along a single axis 'adoption' (or sometimes 'expectation') is even more confusing. Technology comparisons are not helpful plotted on a single variable as the comparator. It also leads to some very bizzare plots where somehow 'mariadb' has more 'adoption' than 'mysql' or 'postgres'

<comment><USER3> Yeah, this is ThoughtWorks' Technology Radar[0] with an axis missing.

[0]: [https://www.thoughtworks.com/radar](https://www.thoughtworks.com/radar)

<comment><USER4> This is an epic resource. Thanks for sharing.

Direct link to the most recent report with tools and commentary (PDF): [https://assets.thoughtworks.com/assets/technology-radar- nov-...](https://assets.thoughtworks.com/assets/technology-radar- nov-2016-en.pdf)

<comment><USER5> No organization could adopt what they recommend. 75% of the "adiot" class disappears in six months

It is good for buzzword bingo awareness.

</comment></comment></comment></comment><comment><USER6> I have a lot of trouble believing that the author of this page has much experience with iOS development. As such it calls into question the rest of their evaluations. Cocoa programming is the place I can speak intelligently, so I'm going to limit my remarks to that.

Let's start with Objective-C which is being listed as "Enlightenment" on the cusp of "Productivity". Objective-C has been around for 30 years. Check out the Swarm Library. An Objective-C library developed first in the 90's for research into CA. I won't go into the whole history, but it's been a productive language since the mid 90's.

In Text Editors, Xcode is listed as "Challenged" by what? There's no mention of AppCode at all. The Cocoa IDE marketplace is so small I cannot name another editor without going to google. Listing it as "Challenged" may appeal to people who aren't experienced with it, but it's still the accepted tool for Cocoa development.

For iOS Development... I know React Native is popular among newcomers, almost exclusively because they'd rather not learn a new language, but I can name exactly 0 mobile professionals who use it. Yes, I know Facebook uses it. Facebook apps are objectively terrible. They get away with it because React Native is moderately better than mobile web.

/rant

<comment><USER7> And GraphQL is listed as a database... I would take these charts with a huge grain of salt, they seem to basically be a single person ballparking things on a curve.

</comment></comment><comment><USER8> the problem with this is it's opinions of the creator, presented as imperial fact - as if it's actually representative.

This is dangerous, because people who are easily mislead will see this and become java developers - or hire only .net developers, because they saw that graph.

I've browsed around - every one of these graphs is completely irrational, tells nothing substantive, or are misleading at the least. There are _some_ truths scattered about, but by and large it's largely nonsense.

Graphs tell a story - and every one of those stories is either entirely made up, or has some pretty disputable results.

there IS a story to tell, but it takes more than a graph to tell it. There's a lot of "it depends"

I don't mean to gutpunch the author. I strongly disagree with this project - but commend the effort. the design is clear and straightforward, and it's another project shipped. That's certainly worth something.

edit:

from the related writeup:

> What makes this visualisation so attractive, is that it provides you with a > starting point for understanding and exploring a certain market and it's key > technologies.

> That information can help you decide whether to either invest your time and > money into an emerging technology, or wait for the hype to pass.

this is _exactly_ the kind of takeaways you should not be making from these graphs.

<comment><USER9> It would have been better to host an industry survey first and then plot results based on it.

<comment><USER8> This is a great idea! I really like this. But the audience needs to be clear, and the method of gathering unbiased.

Maybe I would say, something like "you would use PHP when _____" or "you would reach for flexbox when _______" as a survey methodology. Because that's kinda how it is for a lot of things. The give & take needs to be captured.

Maybe some way of check boxing priorities. Like, I care about maintainable code, or cheap developers, or high concurrency, or good track records. All that would be really useful in helping establish a direction for a developer and a company researching directions.

</comment></comment></comment><comment><USER10> Whoever wrote this doesn't seem to have much experience with the .NET ecosystem.

Referring to F Sharp as "#F"; saying Visual Studio is in the "growth" phase; no mention of IIS under web server or SQL Server under database or Xamarin under any of the relevant mobile sections.

Overall I like the visualizations though

<comment><USER6> I had similar feeling regarding their experience with iOS.

</comment></comment><comment><USER11> I build this as a side project. I enjoy doing research for certain topics, and usually end up drawing a graph that positions all concepts so I get a good understanding of a market.

I build a tool around that, which became State.of.dev.

Read more about its future plans: [https://medium.com/@yvoschaap/state-of- dev-c609cc12084b](https://medium.com/@yvoschaap/state-of-dev-c609cc12084b)

<comment><USER12> Thanks for building this! What data did you base the graphs on? Did you extract these data from Gartner reports? If so, which ones?

Although I see a bunch of things on the graph, I don't know what they imply. For example I see Heroku in the "challenged" state -- challenged by whom? As far as I know Heroku is still big and growing.

And I see PostgreSQL being "more challenged" than MySQL. What backs up this claim? I get the feeling that PostgreSQL is getting significantly more buzz and development after the fall of MySQL (as a result of the Oracle takeover).

I also see containers being on the verge of being challenged. But I thought containers are still very much up and coming? So I'm confused.

More explanation would definitely help.

<comment><USER11> All valid questions.

re: context chart and states. The y-axis mentions "adoption". Which is the market adoption of an item. If you click a state, I explain what that state mean. I'm certain this can be improved. So for your example Heroku: Yes they could be growing, I'm not saying they aren't, but they've come into a state where their service is strongly challenged by competitors (usually left in the chart, AWS, Google Cloud).

Re: PostgreSQL and Containers. Those are my opinion in this release. I would see state.of.dev evolve into expert/community consensus based on arguments and actual chart releases.

<comment><USER13> Containers is still very much in growth, for the majority of deployments.

VMs would be mature, and bare metal would challenged / superseded.

Remember that 90% (a number that is completely take out of thin air, but a large majority) of deployments are not what we see on HN, lobsters etc.

They don't understand why you would have a VM, let alone a container.

I have seen feature requests recently for zero downtime migrations of containers, as "some apps cannot be stopped and started" \- that is how enterprise sees containers.

<comment><USER14> Hate to break it to you, but for high performance only specialised hardware works. Sometimes even only local hardware. Or even a full cluster that only uses VMs or containers as a nice migration of availability layer.

Internet bandwidth and latencies kill clouds while there is no real gain to be had by running multiple heavy compute services on one machine. (as much as certain gpgpu companies want you to believe)

Bare metal superceded, ha. It is just augmented with an availability layer.

</comment></comment></comment></comment></comment><comment><USER15> Visual Studio in growth?

Atom/Vim/Sublime in maturity?

Eclipse/Xcode/Emacs challenged?

Emacs almost superseded? Superseded by what exactly?

At least the text editor section doesn't make any sense to me. Atom more mature than Emacs/Eclipse/Visual Studio? Visual Studio Code is in growth, Visual Studio is a very mature product.

<comment><USER16> >Emacs almost superseded? Superseded by what exactly?

VIM HA! That whole chars is completely dumb. It's comparing oranges to pears. VisualStudio works on Windows, Emacs, Vim work on every platform, are hackable and opensource etc. etc. The chats is called 'text editors' ha, VS and Eclipse are not really text editors, they're full-featured IDEs, with integrated debugger and stuff. I can't stop seeing those charts as someone's imagination how tech looks like, it's not even supported by anything like data, google trends or even survey.

<comment><USER15> Exactly. I think there should be a disclaimer saying that those charts don't represent the actual consensus/reality and should not be taken at face value. They're not accurate and even conflate categories/products which are not comparable.

<comment><USER17> if you focus on the y axis and ignore the x axis then it kinda of makes sense, mostly because the adoption trend is not related to the lifetime of the subject in this case, like it can be for programming languages and the likes.

</comment></comment></comment></comment><comment><USER18> Nice project, needs a bit of work though, mostly in the form of explaining the data in the graphs.

Also, two points: * It's SQLite, not SqlLite. * Yeoman has been around for ages and I've never met a person who actually uses it. How is it under "Release"?

<comment><USER19> I've used Yeoman in a few projects, but realistically I've only seen it used to bootstrap a project, and I happen to do longer running projects so there's not much bootstrapping. A second use case of Yeoman is generating components, and we do have a set of templates to do that in our project, but honestly I'm old-fashioned and inefficient and just copy / paste an existing component instead of try and figure out how Yeoman works again and iterating on the templates. Might be different if we added new components more often.

</comment></comment><comment><USER20> It's a bit strange that most of the used graphs imply some kind of chronoligical order.

First Hype, than disillusionment etc.

But some things, like the waterfall model were considered productive once and aren't now.

Also, GraphQL isn't a databse but an API layer/protocol.

<comment><USER21> The Hype Cycle Chart you refer too is based on the Gardner model (which is a decade old): > [https://en.wikipedia.org/wiki/Hype_cycle](https://en.wikipedia.org/wiki/Hype_cycle)

Waterfall is stuck in 'disillusionment', and didn't move forward into enlightenment/productivity. Again, this should get more context on the arguments and context of why something is where :)

<comment><USER20> Ah okay, even if it's old, I still think this form of representation is misleading.

</comment></comment></comment><comment><USER22> Could anyone clarify why PostgreSQL is _challenged_? Challenged by who?

No sarcasm intended, I am a bit confused by this.

<comment><USER23> By NoSQL.

<comment><USER24> Oh you and your jokes...

<comment><USER23> I think that's what he meant, this is not what I think.

</comment></comment></comment></comment><comment><USER25> I think the idea of having some site like this has some merit to it, but this idea of plotting them on these graphs is misleading and inappropriate.

The graphs are drawn as line charts, with everything being points on the line. This creates the illusion that every technology goes through a lifecycle between all of these states in the exact same path. This is utter nonsense.

Take for instance, your technology graph, where you list "Augmented Reality" as being in the "Disillusionment" phase. Being a point on the curve, along with every other point on the curve, it's implicit that someday "Augmented Reality" may move to be more widely adopted in the "Productivity plateau." But that may never happen; it may just end up being a fad that Nintendo tried to make money off of and never really succeeded at. Nevermind that it's debatable what "productivity" is for a concept like AR.

These graphs also give no real useful information on their Y-axis. It's not really useful to know if the author thinks that Swift is about to peak and that Java has plataued. It may be extremely useful to know how many people are starting new Swift projects, relative to other langauges in the same niche (e.g. Obj-C, maybe Java if you want to include Android? Idk it's not my niche). That might be information someone can use to decide whether they should start a project in Swift or do something else.

Also, the selection of these charts themselves is a little odd. Why a chart devoted just to PHP frameworks? Is this the State of Dev or the state of people who only know one language and read about the rest in school?

<comment><USER26> State of Dev as the author saw fit. No one can come up with State of Dev that will accommodate everything ,not even the ThoughtWorks tech radar.

Also, the last line:

> State.of.dev is still a work in progress. Have feedback? Ping me at > @yvoschaap.

<comment><USER25> > State of Dev as the author saw fit. No one can come up with State of Dev > that will accommodate everything ,not even the ThoughtWorks tech radar.

I disagree. There are ways you could come up with the State of Dev in ways that are more objective than one guy's opinion. Like, the StackOverflow dev survey.

>Also, the last line: >> State.of.dev is still a work in progress. Have feedback? Ping me at @yvoschaap.

Yvoschaap is here reading the comments and has already replied, so he can benefit from my feedback here or ignore it as he pleases.

Even then, that doesn't preclude me from discussing it here; this is a place where people discuss things they see.

<comment><USER11> Yes, reading and absorbing the feedback.

The current charts are a starting point. To re-emphasize (from the linked blog post) I would love it to evolve to an community/expert consensus based on transparant arguments.

</comment></comment></comment></comment><comment><USER27> Gartner/Forrester approach is great but I think a little more useful for the business planning side (the time dimension is more about market value). Similar vein but more on the 'tech worth checking out/playing with/deploying in-prod' side (panel opinion): [https://www.thoughtworks.com/radar](https://www.thoughtworks.com/radar)

<comment><USER14> Read up the verification of the Forrester approach. It has failed to predict anything important yet. A simple differential model like this is no good.

Likewise gaussian cupola models are often wrong. You might be facing a sigmoid instead. Or a stepwise distribution.

Both of these are used by alleged econometrists and planners who do not know better and hate drawing real histograms of opinions, because that takes actual data collection.

Thoughtworks one is even less descriptive and more limited.

</comment></comment><comment><USER28> A fun little site. But how is Sublime in "Release"?

<comment><USER11> I would love to see these charts evolve into a expert/community effort with releases. Including arguments why items are in certain states.

Sublime's position is a bug!

</comment></comment><comment><USER29> That there is no JS framework in "mature" after a decade or so of SPAs says a lot :-)

I wouldn't call Ember "challenged" though, since it's evolving constantly and adding features inspired by a lot of new tools.

<comment><USER30> It might, but I think it's equally possible he's right. For example, had Angular 2 not come out the way it did... it probably WOULD be in the mature column (I personally have found myself extremely productive in it), but adoption was hindered after people found out about the breaking changes in 2.0 (I personally hesitated for the last 2 years to start a new project in it). React is close to mature, but I kind of get the feeling it's accelerating not because of it's own merits, but rather it filled a vacuum. Adoption not earned by being "the best", but rather "the product everyone is talking about" does not bode well in the long term. Don't get me wrong, I think React has a lot of positive qualities, but I think a good framework should make a problem no more complex than the problem itself is. If you start using flux frameworks etc (as is expected with React) I really start to feel like I'm wasting time on things that are not the problem I'm trying to solve.

</comment></comment><comment><USER31> What no c++? Perhaps it should be stateofwebdev.com. Or web.stateofdev.com ;)

</comment><comment><USER32> I love this. I especially like how insights fall out of the presentation. For example, looking at the Product Development graph, there isn't anything that's really an innovation trigger there. Not seeing something there makes the mind want to put something there, and there is your insight on the status of Product Development methodologies; a space ripe for disruption.

</comment><comment><USER33> Are those statistical distributions? One looks like a bell curve, but I don't recognize the other.

<comment><USER11> Actually based on the Hype Cycle [https://en.wikipedia.org/wiki/Hype_cycle](https://en.wikipedia.org/wiki/Hype_cycle)

</comment></comment><comment><USER23> I like it, but it is just your opinion, right? It is strange to have a website for that.

</comment><comment><USER34> Vey interesting compilation; I really like it; Thank you;

</comment><comment><USER35> What is the data source for the graphs ?

</comment><comment><USER36> GraphQL is _not_ a database.

</comment><comment><USER37> Emacs challenged?! Nonsense! ;-)

</comment><comment><USER38> Looks like no build tools on maturity plateau, also no design patterns there.</comment>


Mailinator family now includes talkinator - <AUTHOR> http://www.talkinator.com/ o if there is no sign-up, what name will I have when I enter a room? One will be generated for you. You can change it with the "name change" button or just type "/nick yourNewName".<p>How can I tell if someone in a talkinator room is a noob? Their nick will be "yourNewName". <comment><AUTHOR> from the FAQ: o if there is no sign-up, what name will I have when I enter a room? One will be generated for you. You can change it with the "name change" button or just type "/nick yourNewName".

How can I tell if someone in a talkinator room is a noob? Their nick will be "yourNewName".

classic.</comment>


Zeptolabs Spying on You? - <AUTHOR> http://locusdementia.blogspot.ro/

<comment><USER2> One man's "spying" is another man's "analytics".

It's ironic to see these tracking services launch to much fanfare and pats on the back on HN every few weeks, only to be followed by the "WTF is this in my logs" posts like this one.

</comment><comment><USER3> Good lord, it's 99.99% likely it's just for some ad targeting, and it's not like they don't disclose it in their privacy policy (<http://www.cuttherope.ie/privacy.html>):

 _We may also employ third-party ad serving and/or analytics technologies that use certain methods to collect information through the Services. These technologies may be imbedded within the Services and may collect demographic and location information as well as information from your device including, but not limited to IP address, UDID, software, applications, hardware, browser information, internet and on-line usage information and in-game information._

If you don't want this sort of thing to happen, stop agreeing to it. It's not like you _need_ to play Cut the Rope.

<comment><AUTHOR> No you don't, but I find it "strange" to see DEVELOPERS getting targeted. And this, by monitoring what apps a user is installing (and reinstalling). This includes and possibly even targets beta, internal, stealth mode builds. This is done by an app market vendor (Getjar) who has every motive of knowing what are the currently developing apps and who does them. This is not related to "analytics" but resembles more to some form of "insider trading", IMHO.

<comment><USER3> Everyone's being targeted. GetJar is undoubtedly receiving data from millions of phones, not just yours - you only _noticed_ because you're a developer. GetJar has no idea you're a developer, and if they did, they wouldn't care.

The 'spy on currently developing apps' thing is a nice-sounding theory, but it's wrong. I've worked extensively in this industry (although not for GetJar), and the only developers any company in this space is interested in are the ones that are already moving a ton of units, not ones working on applications that statistically are unlikely to see the light of day or make a single sale. And they're only interested in the big developers because they want to sell them some advertising or get them to run ads in their apps. The money you can make from advertising is huge, while the money you could make from some sort of corporate espionage is tiny. (How would you even sell it?)

The data collected is almost absolutely related to ad targeting. The advertising company fetches a list of installed applications, ignores all the stuff it knows nothing about, and uses the rest to construct a profile about what you like which it can use to show you better-performing advertisements. You use Fandango? Great, we'll show you movie ads. You play Cut the Rope a lot? Great, we'll slot you into our casual gaming segment, and show you casual games.

There's still plenty to object to, if you like - sticking this in a paid game is kind of lame, even if they did disclose it - but it's not at all what you think.

<comment><USER4> Ok, you may have point, but it also is plausible what has been said, that being an app store (like Google Play is) what you are actually looking for is not an ad profile, but a developer profile.

If I would be running an app store, I would certainly be interested in the new games/apps that are going to come out and see if I can convince the developers to list it on my store first, given certain incentives (because if I have apps I make money, so my actual target as an app store are the developers).

Notice that GetJar does not describe itself as an add network but an app store of some sort. If they are indeed an ad network, this behavior is somewhat expected, otherwise...

<comment><USER3> GetJar's been trying the third-party / bundled app store schtick for a while, but the money in this corner of the world is in driving application installs through advertising. Here's an example from an industry news site:

[http://www.gomonews.com/even-more-ad-funding-getjar- monetise...](http://www.gomonews.com/even-more-ad-funding-getjar-monetises- its-app-downloading-service-with-getjar-ads/)

</comment></comment></comment></comment></comment><comment><USER5> Getjar is spying on you, not Zeptolabs.

I don't see the "sendInstalledApps()" lines in my logs, neither the "FOUND_INSTALLED" part in the "Sending POST data" line.

I have the com.zeptolab.ctr.paid app too.

Maybe they changed that in the last versions?

<comment><USER6> Original post was from January, enough time for a change

</comment></comment><comment><USER7> Please. Everyone online company is spying on you.</comment>


Show HN: Cybernotes - <AUTHOR> http://cybernotes.herokuapp.com <comment><USER2> Please provide a demo or example so interested users can view without signing up.

<comment><AUTHOR> Like a video?

<comment><USER2> It looks like a video has since been added. Thanks, looks sweet!

</comment></comment></comment><comment><AUTHOR> Added video by the way.</comment>


Using logs to build a solid data infrastructure - <AUTHOR> http://blog.confluent.io/2015/05/27/using-logs-to-build-a-solid-data-infrastructure-or-why-dual-writes-are-a-bad-idea/ <comment><USER2> I've been really interested in this architecture since Jay Kreps' blog post on it. One part that I'm less clear on is how this fits in with request-response style communication between, say, a Web browser and a Web server.

In a simple Web-app-writes-to-DB scenario, it's easy to read my writes, but with a async log processing system, how am I supposed to organize my code so I can read my writes and respond with useful information?

Maybe the solution is to eschew request-response entirely and have all requests return 200, then poll or use two-way communication?

Alternatively, I could have my log-appending operation return a value indicating the position in the totally-ordered log, which I could pass to the query interfaces as a way of indicating "don't return until you've processed at least to here." Does anyone do that?

Am I totally off base here? I'd love to hear from anyone who is using these kinds of systems today.

<comment><USER3> You're quite right. Switching from a db to a log as the "master" loses the simplicity of the app-to-db model.

As the article says: "For now, a better option is to extract the log from a database" \- i.e you use some tooling to generate a log from the db.

Indeed, you can now see tools that go in this direction usually by using the replication stream. (eg [https://github.com/shyiko/mysql-binlog-connector- java](https://github.com/shyiko/mysql-binlog-connector-java) and [https://github.com/xstevens/decoderbufs](https://github.com/xstevens/decoderbufs))

</comment></comment><comment><USER4> I've really learned a lot from these Confluent posts about building log-based architectures, but I feel like they're rehashing the same high-level architectural ideas again and again.

I'm already sold on this idea, and would really love to see more posts that get into the nitty-gritty details of how to integrate Kafka, how to migrate an existing infrastructure, case studies, sample code, etc. It all seems very handwavy otherwise.

</comment><comment><USER5> Has anyone thought of how the distributed commit log can be extended to client-side FRP?

Elm has this notion of signal – [http://elm-lang.org/learn/What-is- FRP.elm](http://elm-lang.org/learn/What-is-FRP.elm) – which is really a stream of changing values, that are used to construct varying model and its rendered view (virtual DOM and all that).

I am wondering if we can merge these two notions – signals and commit logs. Consequently, this would replace the traditional "request-response" model in REST API with nothing but signals, thus leveraging the simplicity of FRP for the whole application. Client side Elm code does a "send" on the signal that is connected to the server-side commit log, and also reads from another signal connected to another log that receives new data (added from various places).

</comment><comment><USER6> Martin's talks and blog posts are aways awesome. I'm really excited to see how this plays out for real applications.

The one thing I'm always somewhat confused by though is how a "totally ordered log" intersects with the reality of a partitioned log. The simplicity of a log seems to break down a bit when you partition.

For instance, imagine I want to implement multi-key transactions on top of a distributed datastore. With a totally ordered log this is easy. But with a partitioned log, it becomes much harder.

Alternatively, imagine I want to implement a collaborative editing app like Google Docks or something like Slack. A natural design would be to to have millions of independent logs. I can then replay logs to get current state and watch logs to keep it updated. But as far as I'm aware, partitioned logs like Kafka do not actually support millions of topics. So there's no way to replay a log for something like a channel or document.

<comment><USER7> One thing to realize is that a partitioned log is a generalization of an unpartitioned log (i.e. if you set # partitions = 1 in a partitioned log you have an unpartitioned log).

In Kafka the purpose of partitions is to provide computational parallelism not model entities in the world. So if you have 100m users you would map that into a number of partitions based on your computational parallelism (maybe 10-100 machines/processes/threads). In other words you would have a single topic partitioned by user id, not a topic per user.

If you have a centralized relational database that maps reasonably well to a single partition log (both in terms of scalability and guarantees).

For distributed databases you generally don't have a total order over all operations. What you usually have is (at best) a per partition ordering, which maps well to a partitioned log as well.

For applications that record events (logging or whatever) it is natural to think of each application thread or process as a kind of actor with a total order.

<comment><USER6> Yeah, this is a good way to look at it. But it's also my point. Partitions are about parallelism and don't always fit the data model or domain. While you can reduce the partitions to 1, this is limits parallelism. It's not always easy to design a partitioning scheme that preserves the replication semantics you want. And these semantics vary from having something totally ordered, to having something that you can replay on a very fine-grained level (to replicate to the client for instance.) Most discussions of the advantages of logs really emphasize how amazing a totally ordered log is for replication, but that's not actually what production deployments look like so you still need to think carefully about what happens when writes are being applied to your datastores without a clear order.

<comment><USER8> What Kafka gets right is that it explicitly addresses the fact that if you add parallelism, order goes out of the window (modulo partition key).

This is something that tends to surprise developers early on (myself included, years ago). But plenty of people still use queue solutions like RabbitMQ without thinking it all the way through.

Unfortunately, partitioning introduces a design step that makes it a little harder to make processing generic. With RabbitMQ you just post to an exchange and let queues (ie., consumers) filter on the routing keys; if no queues have been bound, for example, messages don't go anywhere. If you want, or don't want, parallelism, you just run either multiple consumers or just one. With Kafka, you need to decide beforehand, and design the "topology" of your log carefully, not just for the producer, but for each consumer. When producers and consumers are different apps, this starts smelling like a violation of the principle of "separation of concerns".

I rather wish Kafka had a better routing mechanism, actually. I don't see any reason why it couldn't have routing keys, just like RabbitMQ.

</comment></comment></comment></comment><comment><USER9>
      if you want to build a new derived datastore, you can just start a new consumer
      at the beginning of the log, and churn through the history of the log, applying
      all the writes to your datastore.

For high-throughput environments with lots of appends to the log, how do you get around the ever-increasing size of your log file? I know the traditional answer is to take a periodic snapshot and compact the previous data, but is that built in to tools like Kafka?

<comment><USER10> There's a log compaction cleanup policy yes. Never used it myself but if I'm not mistaken it works like this: for each message you send to Kafka, you set a key with it. When Kafka does log compaction, it keeps only the last value for each key.

The other cleanup policy is to just have a retention time. After X minutes/days/weeks segments of the log are simply deleted.

<comment><USER9> That sounds great if your messages in the logs are the complete state for that key, but I'm not seeing how to use that compaction system if the messages are change events.

Is there a system designed for snapshotting the aggregate and logging the delta?

<comment><USER11> A common pattern is to publish a "checkpoint" message. Not sure if the concept is built into Kafka or not.

</comment></comment></comment></comment><comment><USER12> The proposed architecture really works well for me. I've used it for a couple of projects now.

To throw around some terms for those interested in reading up/background: \- The separation of writes (through log) from reads (through any of the consumers) is sometimes called: CQRS (command query responsibility separation) \- having a centralized log as the defining store for updates/ change events is sometimes called: eventsourcing \- as mentioned in article: elastic search as a consumer of the log, which only gets updates through the log, is an example of an Eager Read Derivation.

All defined on site of Fowler.

Glad to see this getting more attention. Asked about usage for Kafka as an eventsource here some time ago. Includes insightful answer of Kafka author. [http://stackoverflow.com/questions/17708489/using-kafka- as-a...](http://stackoverflow.com/questions/17708489/using-kafka-as-a-cqrs- eventstore-good-idea)

<comment><USER13> > having a centralized log as the defining store for updates/ change events is > sometimes called: eventsourcing

To clarify slightly: Event Sourcing isn't just emitting events for others, it means that your system is _re-reading its own log_ to derive current state from past-events.

For performance reasons you try to avoid this with stuff like periodic "snapshots", but the _capability_ of has to be there.

</comment></comment><comment><USER10> That talk was great. We introduced Kafka at my work probably 3-4 months ago, at first only to track events from our webservices, but eventually it became the backbone of communication between our services.

The Java library for the consumer part, still based on the Scala code, is not that great though. They're rewriting a Java-only library, which is much nicer to use, but I'm not sure when it'll be stable.

</comment><comment><USER14> I have actually found myself thinking a lot about logs lately and how I can end up using them for a lot of problems, and are sometimes very simple to implement. But I always wondered if I was actually using the right tool for the job...

I had no idea they had such far-reaching implications and so many jobs where this _is_ the right tool.

Neat!

</comment><comment><USER15> Isn't this just what datomic automatically does?

</comment><comment><USER16> What is used to create those images?

They look like whiteboard pictures but not quite.

<comment><USER17> I'm pretty sure it's Paper by 53, more information is available here: [https://www.fiftythree.com/paper](https://www.fiftythree.com/paper).

If you look at some of the screenshots on the site, you might notice some similarities.

</comment></comment><comment><USER18> Could someone compare Apache Kafka to Eris Industry's centralized blockchain model?

<comment> compleat-ang __Disclaimer __I am the CEO of Eris Industries.

Firstly, we are not doing centralized blockchains, we're doing permissioned blockchains which work most efficiently when actors within the system have been identified. These blockchains are simply one element of the Eris Blockchain Application platform we are developing.

Comparing a smart contract enabled blockchain to a distributed messaging broker is a difficult proposition as they are very much apples and oranges in their capabilities, usage, and goals.

That said let me give it a go.

## Similarities

* Kafka and eris:db both work on a distributed model * Kafka and eris:db both are inherently scalable and persist their data * Kafka and eris:db both include a distributed messaging layer optimized for their system which is not http

## Differences

* eris:db persists using merkle-ized encoding schemes with all data having been digitally signed via a node using an ECC-PKI and verified by a logic gateway (known commonly as a smart contract) prior to being persisted -- eris:db further builds logs into blocks and chains those together using traditional blockchain techniques; kafka uses a relatively sophisticated, but not inherently as verifiable "traditional" logging mechanism * eris:db, as with any blockchain, includes a [fork choice rule]([https://eng.erisindustries.com/blockchain/2015/04/30/on-bloc...](https://eng.erisindustries.com/blockchain/2015/04/30/on-blockchains/)) for resolving differences within the nodes on the network as to prior history as well as adding a layer of byzantine fault tolerance; from my (admittedly cursory) overview of kafka while it is distributed it is not byzantine fault tolerant * kafka is fast and meant as a near real time message broker; eris:db is not built for speed and sacrifices some availability for consistency

## Overall

Message brokers are great complements to blockchains actually as they can provide an additional speed layer for messages which blockchains are not particularly good at.</comment></comment>


The Der Spiegel journalist who messed with the wrong small town - <AUTHOR> https://spectator.us/der-spiegel-small-town/ <comment><USER2> People were right in the previous discussion [0] to criticize Der Spiegel's self-serving sanctimonious tone in their writeup of the scandal. It's impossible to read the allegations by the Fergus Falls residents here and square them with Der Spiegel's purportedly high-performing fact-checking operation. It's not just only that they failed to detect their reporter's malicious deception. But they failed to do the most basic kind of fact- checking. For instance, I could see them failing to get to the bottom of an ephemeral fact like whether someone had put up a sign saying "Mexicans Keep Out". But calling the city to confirm that the official city sign says "Home of Damn Good Folks", and the theater to confirm that they were playing "American Sniper" in 2017? Those are the kind of facts that at a bare minimum professional fact-checkers should be double-checking.

Even something like the claim that the city administrator was a gun-toting virgin is something that a fact-checker is expected to call and independently confirm. And even if the guy gives a non-committal denial, e.g. _" That's ridiculous, I don't want have anything to do with your left-wing fake news article!"_, the fact-checker should be prying the reporter for all the documentation/notes that support the claim. In fact, in this scenario, the fact-checker should be highly suspicious that the city official is so standoffish, if the reporter managed to gain his trust enough in the first place to reveal that he's a virgin (and is ostensibly embarrassed about it).

Der Spiegel has a lot of questions it has to answer for how it failed so thoroughly for so many years here.

[0] [https://news.ycombinator.com/item?id=18716613](https://news.ycombinator.com/item?id=18716613)

<comment><USER3> Most workplaces operate on a high amount of trust though, especially in Western Europe. If you've achieved star-reporter status at der Spiegel, as such that you're sent 3 weeks to America to do investigative journalism, it goes to reason, that you also meet a high amount of trust. So I think it's easy to see how the editorial staff might have missed the problem.

I work in the public sector of Scandinavia, we do a lot of internal reviewing of employee behaviour. When things go wrong, it's possible to steal 111 million danish kroner after all[0]. Despite our best efforts, things still do go wrong though, and that's because, even in the heart of bureaucracy, we still rely on trust, and when people intentionally abuse it, shit happens.

It's unfortunate, but things aren't perfect. Everyone is pressed for financial resources and then everything from financial security to editorial journalism becomes patch-work "good enough" type systems. Because the perfect system simply doesn't exist. One of my user accounts has a lot of privileges for instance, more than it rightly should. It's like that because I need access to so many different things, that our IT-department would need a half-time hire if they were to keep my user rights for our 500 different IT systems up to speed. It's an imperfect solution that only works as long as I don't misbehave, and while we do a lot of internal reviewing, the truth is I could abuse those rights.

At least Der Spiegel fired the fraudster when they found out.

[0] [https://www.information.dk/debat/2018/10/111-millioner- krone...](https://www.information.dk/debat/2018/10/111-millioner-kroner- senere)

<comment><USER4> Just want to say this:

The fact that European readers even believed half of the fabrications in this story is telling of how ignorant they really are about the USA.

Just because you have seen movies which are the product of LA and NYC doesn't mean you know anything about the USA. Let me be clear: people in NYC and LA themselves know nothing of their country. This is blatantly obvious when you watch ANY Hollywood creation set in rural America.

It is frustrating to deal with self righteous assumptions of my culture like this based on movies and TV shows.

<comment><USER5> I have had a bunch of experiences in Europe where I get told what's wrong with America. I like how America is so self critical and wants to improve itself. But most of these Europeans have no idea what they're talking about. I had a Brit tell me how sickened he was that 40% of Americans were Christians that literally only support Israel in order to create a holy land that will survive the apocalypse. He was so convinced that when I returned home I asked my religious friends what they thought. I can't find a single person who believes anything remotely close to that. And definitely not a part of the calculus for supporting Israel.

</comment></comment></comment></comment><comment><USER6> I'm fairly convinced it's not remotely limited to Der Spiegel. If primary sources aren't cited in downloadable attachments, it's just best to assume the thing is a pure work of fiction.

I have absolutely zero respect for "journalism" at this time. It is, as far as I'm concerned, an entirely dead art, all life sucked from it by the great destroyer we call _the Internet_.

This network we created, and in particular most of the shiny things at L7, is not healthy for even an above average psyche. It attenuates all feedback loops that our bodies require to form empathy, while amplifying the worst parts of us through appeals to fame, fortune, and notoriety.

I don't even hold this particular journalist in particular disregard. The sooner we recognize that he was so outmatched by the _construct_ that _he didn't even have a chance_ , the better.

<comment><USER7> I'm inclined to agree and the sad part is, journalism is an essential NGO organ of civil society.

The 4th Estate is a very important thing for us all.

Watching Cable News these days is crazy because you can watch journalists, spin, lie and misrepresent ... but at the very same time they are presenting facts that do matter.

Go take a look at 'waybackmacine' and click some links from even 10 years go - the news was not nearly as crazy and clickbaity.

I truly hope that things can settle down a bit maybe in the next political cycle.

Finally - I should say that almost all MSM left and right completely misrepresent middle America. The reality in Cleveland, rural Ohio, Utah, Houston, Alabama ... it's just not anywhere to be found on the news unless something terrible that happens where the press wants to use that piece of information to construct a narrative.

Edit: I should add that there is a very real financial pressure right now for journos - newspapapers are falling apart everywhere, this is a real thing. Individual journos are not natural capitalists. We used to pay for the news we don't now, and we still haven't figured out a model. This is a major factor.

<comment><USER8> I'd be interested to hear how you describe the current situation in Cleveland. I just finished my first semester at a school nearby but spent no time in Cleveland aside from the airport, so even though I've been nearby, I really haven't been exposed.

<comment><USER7> Cleveland is just an example.

I currently live in Canada and get the local news from Vermont. It's sublime and cathartic: local girls basketball team highlights. Interviews with part- time stat reps who also run the local store. No clickbait language between state reps, governor etc.. No culture war.

The national news is a circus, with each clown playing a part. It's ideological, clickbait, narrative-driven and parsed to remove anything that could upset corporate sponsors.

'Normal' is simply too boring for television.

<comment><USER9> "Ideological" and "parsed to remove anything that could upset sponsors" are incompatible goals.

</comment></comment></comment></comment></comment><comment><USER10> The latest accusation against him is embezzlement [0]. Also, the Spiegel reporter who tried to unmask Relotius was threatened with a termination of his contract when he talked to supervisors about his findings. [1].

Also in the FAZ, there was a very good commentary on the whole case by one of the publishers [2]. He identifies the same narrative smoke-bombs ("story- telling kitsch") in the Spiegel's writeup of the whole case as in Relotius' original articles.

[0] [https://www.faz.net/aktuell/feuilleton/medien/betrug-beim- sp...](https://www.faz.net/aktuell/feuilleton/medien/betrug-beim-spiegel- claas-relotius-veruntreute-offenbar-auch-spendengelder-15955791.html)

[1] [https://www.faz.net/aktuell/feuilleton/medien/fall- relotius-...](https://www.faz.net/aktuell/feuilleton/medien/fall-relotius- spiegel-reporter-mit-rauswurf-gedroht-15954983.html)

[2] [https://www.faz.net/aktuell/feuilleton/debatten/ueber-die- fa...](https://www.faz.net/aktuell/feuilleton/debatten/ueber-die-faelschungen- von-claas-relotius-im-spiegel-15954144.html)

<comment><USER11> I came across [2] as well, yesterday, and particularly liked the conclusion:

Slightly improved Google translation:

"One does not write about arguments, ideas and concerns, but about the people who supposedly have them and about how they are, how they look and how they live, where one has met them, how dangerous, or at least exotic it was to meet them at all, and that the wind lashed against the quay wall, when splendor and misery came together at night in the room no. such-and-such. But as long as one writes in such a way, because one thinks in such a way, the case of Relotius will only lead to appeals to conscience and to the condemnation of those without conscience. The underlying issue, the storytelling kitsch, will not change."

<comment><USER12> Could've been written about the con in general. I recently read a book about this called The Confidence Game: Why We Fall for It Every Time by Maria Konnikova [1]. I can highly recommend it.

[1] [https://www.goodreads.com/book/show/25387895-the- confidence-...](https://www.goodreads.com/book/show/25387895-the-confidence- game)

</comment></comment></comment><comment><USER13> Besides the obivious lack of oversight on Der Spiegel and ethics of the journalist. I believe that there are a few lessons we can take away here:

1\. We value honest hard work. However, we rarely value the people who do it. (The journalist was a "star" journalist and was required to perform at that level constantly.. does this ring a bell to any developers?) 2\. Confirmation Bias is rather nasty, and that has been used to propagate some bad narritives. Right now the political deviviness has encouraged by the media. This article: Look at these "racist", "sexist", *ist people in some low-class people in this rural area. At least this is what I get from it. (That's been the garbage tauted by the left) 3\. This guy was trusted as a leader way too much. I would not be surprised if he's a psychopath. (He has no concerns over the people he screws over, and the only reason he seems sorry is that he got caught).

<comment><USER14> As this guy [0] (a TV moderator who handed Relotius several of his awards) says, that journalist was under no economic pressure, at all. And the interviewed person also calls out this worldview in which everybody is a victim, nobody is an actor, and many journalists are more concerned about them being victims than being victimizers.

And it really is telling that the perpetrator so easily gets styled as kind- of-also-a-victim, as if having won a few prizes created this pressure/temptation no human could possibly withstand.

(I started to write my comment in response to someone who replied to you and said you using the word sociopath was "dehumanizing", and that they don't want to defend the journalist, but that "any normal and well adjusted person" could have gotten addicted to having won prizes -- that as well as some comments on the first story about the subject on HN is what I meant with styling them as victim, not your comment as such)

[0] [https://www.youtube.com/watch?v=8TrfRWnYCo0](https://www.youtube.com/watch?v=8TrfRWnYCo0)

</comment></comment><comment><USER15> Maybe they were too willing to believe his depiction of small town America as a bunch of poorly educated, bigoted hicks? Perhaps because it confirmed their own biases?

I know it's a German publication, but this is perhaps a symptom of the monoculture that permeates a lot of organizations in the "West"

<comment><USER16> When I read news articles about topics with which I am familiar, I encounter plenty of statements that strike me as basically false. Consider this excerpt from the New York Times' piece of Donald Knuth [1]:

> Following Dr. Knuth's doctrine helps to ward off moronry. He is known for > introducing the notion of "literate programming," emphasizing the importance > of writing code that is readable by humans as well as computers — a notion > that nowadays seems almost twee.

Calling the idea that code should be readable by humans "almost twee" just makes me think that the writer has no idea what they're talking about. Sure, there are times when people write crap code, but being readable by humans _is the whole reason why we built coding languages_. Not to mention, the average readability of code has probably increased over time, not decreased. The author is basically just pulling this statement out of her behind to embellish Knuth's reputation, to the detriment of the credibility of the rest of the article.

It's nowhere near the magnitude of the fabrications in Der Spiegel of course, but it does seem to be the case that writers are willing to trust their own prejudices without fact checking. I was able to spot when those prejudices in the above example but only because I have relevant domain knowledge. It makes me wonder, how many times do I swallow the BS without knowing it? It's just a matter of how small (or big) of a statement they're willing to make without verifying it's veracity.

1\. [https://www.nytimes.com/2018/12/17/science/donald-knuth- comp...](https://www.nytimes.com/2018/12/17/science/donald-knuth-computers- algorithms-programming.html)

<comment><USER17> Gell-Mann amnesia effect: [https://en.wikipedia.org/wiki/Gell- Mann_amnesia_effect](https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect)

<comment><USER18> I immediately thought the same thing. But then I thought, not being a subject matter expert doesn't give you an excuse to write bullshit.

I think people who study the arts are less likely to have experience with and understand complex systems and, because of the different structure of knowlege in a subject like journalism, it's easier to be overconfident in one's knowlege. Especially when extended to topics like physics which require mathematics to truly understand.

<comment><USER2> And yet a journalist with no tech startup, venture capital, or biomedical experience was able to understand and expose Theranos, which ran its scam right out from Palo Alto and Stanford.

<comment><USER12> After reading Bad Blood, the exposure had almost failed was it not for Carreyrou's enduring persistence. He met both internal and external pressure to call off his investigation. Not every journalist is cut for that kind of pressure.

</comment></comment></comment></comment></comment></comment><comment><USER19> Kinda offtopic: I find it interesting and a little bit disturbing to read "Der Spiegel". "Der" is the German male form of "the" and the Spiegel uses it in the title like "The New York Times".

For me as a German the comments here read like: A "The New York Times" article stated... which is odd, everyone usually drops the "the", if it doesn't fit.

<comment><USER20> Hmm interesting. So it would make more sense to you to read "The Spiegel's article..."?

<comment><USER19> Yes.

We also change the "der" in German newspapers. E.g. "at the Spiegel" "bei dem Spiegel" or "in an article of the Spiegel" "in einem Bericht des Spiegels".

</comment></comment></comment><comment><USER21> A few years ago the Columbia Journalism Review published an article about how great Der Spiegel's fact checkers were.

[https://archives.cjr.org/behind_the_news/inside_the_worlds_l...](https://archives.cjr.org/behind_the_news/inside_the_worlds_largest_fact.php)

It's quite obvious that article about that small town had not passed through the most cursory fact checking.

The CJR owes some explanation.

<comment><USER22> My impression from that interview is that the fact checkers are not doing things like calling movies theaters, mayors, or owners of restaurants and verifying the statements they allegedly made in their interviews with the reporter are true. Instead they are cross referencing claims made by their reporters, things like whether or not a politician has been quoted in a reputable newspaper as saying something, or how many jobs were created in a time period, with their database of open source intelligence. Is it expected that a fact checker contact everyone a journalist interviewed and go over each sentence and claim line by line and verify that it is true?

<comment><USER23> > Is it expected that a fact checker contact everyone a journalist interviewed > and go over each sentence and claim line by line and verify that it is true?

Yes, this is exactly what fact checkers at top newspapers were traditionally paid to do.

I have no idea how many true fact checkers are left, but you described the job precisely.

</comment></comment></comment><comment><USER24> As somebody who does a lot of writing and used to write news copy, this is a story I've been following for a while. In my mind it's easily the biggest story in journalism this year.

The internet has made everybody a publisher, from your cousin who thinks Elvis isn't dead to that MIT scientist with 17 degrees. And they're all online.

It's also created an instant feedback loop, where you instantly measure how "good" your publishing is by tracking clicks, page time metrics, and referrals.

It even provides a zero-overhead revenue model. The money's not what publishing used to be, but hey, you can tap into it without having to do much.

Finally, because of the previous items, it has destroyed traditional journalism. What do I see when I consume journalism today? I see fear. Everybody knows it's dead and internet eyeballs is the only thing that does anything. The person and their emotional connection over Twitter, FB, YouTube and the like is the brand, not the paper or magazine. Everybody knows that.

This means the old guard is gone, the people who used to sit in an office and tell you that your story was crap. They cost too much in overhead and nobody cares if your story was crap as long as you correct it the next day. (It's all self-correcting, right?)

So instead of news, we now have celebrities who give their personal impressions of how it all makes them feel. People reward or punish publishers by their ability to draw them into some event and make them feel one way or the other. This reporter (I use the term loosely) knew how to do a great job of that. He also knew that the emotional content far outweighed the rest of it. People loved him. His publisher loved him.

Until they didn't.

The only reason he was fired, the only reason BigNetCorp X would care if they're screwing you over or not, is because enough _other people_ were able to raise a big enough rukus that it couldn't be ignored. He could have kept doing this for years as long as he was willing to scale his fabrications in accordance to how much of a PR stink people could have made.

While interesting as a stand-alone story, my feeling is that each of these just makes the next bunch more clever.

It's not a very good time to be alive for people who like journalism.

</comment><comment><USER25> The irony is that nearly every prestigious American publication has run dozens of versions of this story, and while most of those should be presumed to be factually accurate, they are fundamentally the same work of fiction.

<comment><USER26> It's not difficult to push your agenda with it technically being "true". Just interview 100 people, focus on 1 or two, maybe take their quotes out of context, and congratulations you can portray an entire town or demographic in a negative light that fits with your readers preconceived notions

<comment><USER7> In 2018 you don't need to interview anyone.

You just cruise Twitter to find some random poor soul who said something consistent with the narrative you want to construct.

Now you say: "People are saying XYZ"

And deflect the source of the narrative from yourself.

</comment></comment></comment><comment><USER27> I remember an interesting discussion I read either here or on reddit between an American and a German where the German stated that he believed he had an accurate idea of what life was like in the US. I didn't have an opinion on who was right or wrong in that discussion at the time but if the lies stated in this article about this town did not immediately send off the bullshit detector for most Germans I'm afraid they've been fed a significant amount of misinformation and propaganda and really have no idea how we live here.

<comment><USER28> I have lived in Europe for about 5 years now, and sadly this is correct. German and French media in particular seem to delight in disparaging America and Americans.

Most Germans I've talked to are incredibly misinformed about what life in America is actually like. They think the entire country is more like a small southern town and that police roam the streets shooting citizens at random.

We have our problems of course and are far from perfect but the extent of them is grossly exaggerated by European media.

<comment><USER29> I spent a month in Germany in the mid-90's as a high school student. At the time, Bay Watch was big thing, and the (friendly, nice) people I met had a pretty tough time wrapping their heads around the idea that my experience, living in California, did not include warm beaches, that I didn't surf, etc.

It's really easy to pick up caricatures of life in foreign countries. I don't think anyone is immune to imaging that foreigners are more exotic and strange than they really are. I remember someone on an American forum confidently saying that in anime, a character's family owning a car was supposed to indicate great wealth because car ownership was super rare in Japan -- a vast exaggeration of a minor demographic difference.

<comment><USER30> > It's really easy to pick up caricatures of life in foreign countries.

I'm a German who's lived abroad for the majority of his life. This statement is so true. I've heard of Africans asking if Hitler was still in power, Americans thinking Germans all lived in castles, and Germans wondering whether we had electricity in Zambia. If you've never lived there, how _can_ you know?

I have dozens of American friends and frequently read American publications. I daresay I have some understanding of US culture from this exposure - but even so I still don't know what it's like to actually live in the United States.

</comment></comment></comment></comment><comment><USER31> Wow, Der Spiegal's lack of fact checking is sad, and it seems like pure fraudulent writing. I love the breakdown of counter-investigative journalism here.

<comment><USER32> The have nearly 100 people in their fact check department. Legendary German efficiency.

</comment></comment><comment><USER33> The original article, with the same title, researched and written by two residents of the town, is here:

[https://medium.com/@micheleanderson/der-spiegel- journalist-m...](https://medium.com/@micheleanderson/der-spiegel-journalist- messed-with-the-wrong-small-town-d92f3e0e01a7)

I think that's a better link than the one above, which is to a partisan political website.

<comment><USER34> Plus it is not behind a pay wall.

</comment></comment><comment><USER35> I wonder if this guy is just a compulsive liar who happened to fake his way into a journalism career; I wonder if we'll hear stories from his acquaintances about this.

<comment><USER36> >I wonder if this guy is just a compulsive liar who happened to fake his way into a journalism career

That accurately describes one of the established ways to bookend a stint in politics. In the UK, this caliber of bullshit would either tip them for a future cabinet post should they ever stand for election, or indicate that they are a recently retired MP, fabricating scandal to preserve some remaining public relevance, which they know only how to measure in inches of print.

</comment></comment><comment><USER37> To be honest, my impression of modern journalism at established news organizations is:

      * business model crashing, so experienced staff laid off

      * smart millennials don't opt for a career in journalism

      * the ones who do emerge inculcated by the college monoculture, largely unable to question authority

      * growing up with net-access, they are ill-equipped for any method of fact-finding except google searches

      * laziness, incompetence and docile conformity make them soft targets for vested interests

      * many stories are therefore just regurgitated press releases

      * many 'reactions' are strikingly uniform across independent organizations

On the bright side, the internet allows real information to route around the established press, greatly reducing their ability to shape public opinion or suppress stories that don't fit someone's agenda. We could be entering a golden age of independent journalists whose careers live and die by their reputation for integrity. But even if we are not, _wikileaks_ style journalism -- the transmission of source documents -- should see us through.

</comment><comment><USER38> Some of you might be interested in a good example of an article about a small town done right... >> [http://america.aljazeera.com/multimedia/2015/7/richland- wash...](http://america.aljazeera.com/multimedia/2015/7/richland-washingtons- atomic-legacy.html)

<comment><USER39> Thanks - that is a very good and BALANCED article.

</comment></comment><comment><USER40> Ha, Fergus Falls. Driven through there many a time. Don't know if I've ever stopped... too close to Fargo/Moorhead on the drive up/back.

This whole article reads like that guy from the fifth season of The Wire.

<comment><USER41> I've taken to stopping at the Fergus Falls Walmart on my way through. I guess I got a bit old and need that stop after Fargo for gas and food. Its basically half-way between Fargo and Alexandria which means its going to have lower traffic because both of them are bigger. Plus, along that route Fargo & St. Cloud have cheaper gas prices. It is a good stop point if you are having car trouble or the weather gets a bit too much to handle.

1) Fargo, ND often has the cheapest gas in the area because it has 3 pipelines going through the town.

<comment><USER40> Yea fair enough. Alexandria always made a better halfway point for me, halfway ish between TC and F/MH.

<comment><USER41> When I was younger, I was with you since Alexandria also had a better gas station and coffee shop. I've had to adjust as I got older.

</comment></comment></comment></comment><comment><USER42> My favorite tidbit of this story:

In 2014, Relotius sold two stories to the monthly magazine of a Swiss paper, both interviews with hairdressers. The second one (still online: [https://folio.nzz.ch/2014/februar/blondinen-faerben-ihr- haar...](https://folio.nzz.ch/2014/februar/blondinen-faerben-ihr-haar- dunkel)), a supposed interview with a Finnish hairdresser immediately received a comment from someone in Finland ("this report seems to be fiction"), complaining that the mentioned salon doesn't actually exist, the mentioned prices were all wrong and the mentioned name had the wrong gender. The magazine printed a correction and decided to no longer work with the author.

No offense to Finnish hairdressers, but if someone completely fabricates a story this meaningless, it's reasonable to assume that he's a pathological liar and not a single written word of his can be taken seriously.

Given that the guy was such a pathological liar, even about trivial matters, I find it very hard to believe that no one supposedly ever had any suspicion about the truthfulness of writing.

</comment><comment><USER43> David Kriesel (the guy who detected the Xerox issue 3 years ago) has been data mining Spiegel Online (the online version of Der Spiegel) for a couple of years now. It will be interesting to see what other articles are full of fiction: [https://www.youtube.com/watch?v=-YpwsdRKt8Q](https://www.youtube.com/watch?v=-YpwsdRKt8Q)

Maybe David is willing to hand out the data he has downloaded?

</comment><comment><USER44> Here's a great twitter thread from a German-based American fact checker. [https://twitter.com/spoke32/status/1076219412485677056](https://twitter.com/spoke32/status/1076219412485677056)

>> As far as I can tell from their mea culpa, this vetting process is limited to what can be Googled from Hamburg.

<comment><USER45> Oh dear, one of those Tweets that should have been some sort of proper blog post.

The damn thing doesn't even load properly with cookies disabled.

I apologize for the toxicity but the content got lost to me.

</comment></comment><comment><USER46> If it weren't because the article came out in 2017, I'd have thought that this Relotius fellow played too much "Far Cry V" and thought it was a documentary.

Also, TIL that many germans haven't watched "Game of Thrones", apparently.</comment>


Timing attack on Arduino - <AUTHOR> http://eax64.github.io/2015/04/10/timing-attack-on-arduino.html

<comment><USER2> While I realize this was an example of how side channel attacks work, timing attack happen over and over again in the real world, for example back in 2008 the xbox 360 per console key could be recovered allowing downgrading to a vulnerable kernel.

[http://beta.ivc.no/wiki/index.php/Xbox_360_Timing_Attack](http://beta.ivc.no/wiki/index.php/Xbox_360_Timing_Attack)

For the curious, the solution is to use a constant time comparison function, which can usually be done by not returning until the end of the comparison loop

<comment><USER3> or-of-xor's is the traditional approach, from PHP's password_verify:

      for (i = 0; i < hash_len; i++) {
        status |= (ret[i] ^ hash[i]);
      }

This works well enough, and doing something more clever runs the risk of the compiler optimizing out padding code.</comment></comment>


Bitcloud: proof of bandwidth - <AUTHOR> https://github.com/wetube/bitcloud

<comment><USER2> Before somebody asks the payments take place with "Cloud Coin" a BTC derivative for just this project (which means it can't be taken down via It slightly begs the question of: "Why not just use an already established currency instead of inventing a new one causing more fragmentation within the electronic currency market?"</comment>


Google's new messaging app translates your voice into emoji - <AUTHOR> http://mashable.com/2017/03/16/google-supersonic-voice-emoji/#Lkxjojbl9OqO <comment><USER2> _Google is working on another experimental messaging app_

Does anyone want to buy in on a betting pool on how many messaging apps we're going to have from Google by Christmas?</comment>


ChaosKey v1.0 Released – USB Attached True Random Number Generator - <AUTHOR> http://keithp.com/blogs/chaoskey/ <comment><AUTHOR> > ChaosKey is free hardware running free software, built with free software on > a free operating system.</comment>


Investors may have lost 1 Trillion US$ due to Russian policy - <AUTHOR> http://www.economist.com/news/finance-and-economics/21608754-bad-governments-cost-investors-fortune-trillion-dollar-boo-boo

<comment><USER2> Russia used to be a BRIC (Brazil, Russia, India, China), now the trillion dollars is a DOG = Discount for Obnoxious Government.</comment>


Apps that send packages with the Shippo API - <AUTHOR> https://getputpost.co/apps-that-send-packages-with-the-shippo-api-6184d909ccb1#.tn2wzmtmu <comment><USER2> I love Shippo as a product and even more so as an example of startups solving unsexy problems with technology.

Payments before Stripe and Square were a pain in the ass. Shipping is a similar backbone to the economy that, for businesses, just sucked.

Building one universal shipping API and taking advantage of carrier bulk discounts even for the smallest businesses makes a much more demonstrable impact than a chat bot.

Good on them. Also amazing blog name (Get Put Post.)

<comment><USER3> Thanks :) You can subscribe here for future interviews: [https://tinyletter.com/getputpost](https://tinyletter.com/getputpost)

</comment></comment><comment><USER4> I had the pleasure of getting to meet the Shippo team recently. I have to say, I was incredibly impressed. Everyone I met there was incredible: down to earth, really nice, highly motivated. I'm sure this company / team will go really far and do really great things. The two founders I spoke with seemed like genuine, great people. They have a really strong vision =)

I'm also a huge fan of the way Shippo's API service works, and the way they price their product. They're one of the few developer companies charging really fair prices for their service, and pricing is straight forward, and encourages usage. None of that enterprise BS that everyone is so eager to provide now-a-days.

I hope to use them in a future project someday =)

</comment><comment><USER5> Is Shippo comparable to easypost? What are the differences/benefits? Seems like Shippo is more for small businesses and offers a richer interface for managing things without having to first integrate the API? Easypost seems more directed at developers, but they also seems like they are hiring a lot of ux people so I would assume they are building their own management interfaces aswell. Just wondering if both serve the same use cases. I will be looking at integrating something like this into a product primarily for shipping label creation and I am curious how the two services differ.

<comment><USER6> Sawyer from EasyPost here. Yes, we're generally upstream of Shippo - we power a bunch of their competitors in the app / shipping UI space. We focus more on developer APIs and all of the tools and performance considerations that come with that.

Our UI developers are focused mainly on analytics for the platform - we're happy to leave the click n' ship piece to Shipstation, ShipRush, Shippo, etc.

</comment></comment><comment><USER7> Shippo is great, we use it from our Weebly-based site. Between Shippo, Stripe and Square we've got pretty much what we need to run billing and shipping. Thank you, interwebs!

</comment><comment><USER8> I'm sure this is not the intended use case, but considering the trouble I've had trying to use UPS/Fedex/USPS to print shipping labels for my personal use (many of the websites suck), it'd be great to see a desktop app that used this API to find the lowest cost service, print you a label for it, and track your shipment.

<comment><USER9> Actually! We do offer a web interface for this exact thing. Many of our customers are small businesses who don't have developers to integrate the API, so we built a dashboard for them to compare shipping prices, print labels, and track shipments. (You can also use it even if you're not a business)

You can see a small gif of it working under "How you make money" in the interview. [https://cdn- images-1.medium.com/max/800/1*GDC26ygGdl49WKsb9Z...](https://cdn- images-1.medium.com/max/800/1*GDC26ygGdl49WKsb9Z83ZQ.gif)

<comment><USER10> Are you from Shippo? Would you mind if I contacted you directly to ask some questions? As it happens I'm in the process of comparing a few services like Shippo to purchase for a 3PL company and I'd love to be able to chat with someone. Email in my profile if you don't mind reaching out.

<comment><USER9> Yes, from Shippo here. Will shoot you an email. :)

edit: if anyone else would like to chat, email@example.com is the best place to get to us. That goes directly to a few of our inboxes, so we're get back to you in no time.

</comment></comment></comment></comment><comment><USER11> Looks like they offer free tracking api even if you don't buy shipping through them, which is interesting: [https://goshippo.com/pricing/](https://goshippo.com/pricing/)

<comment><USER12> Yes, we do!

<comment><USER13> How is the eta of a shipment determined when tracking a shipment this way?

<comment><USER9> We get that data from the carriers directly at the moment.

<comment><USER13> OK. Thanks for the reply. So that might be a continually updated value, as opposed to an originally scheduled date? As a seller we have an interest in knowing when a package is delivered later than originally expected. I'm doing some programatic, carrier-specific things to monitor this internally, but would be interested in finding a way to this through something like Shippo's API. Basically, show me delivered and undelivered shipments that are beyond the originally scheduled/promised delivery date. Possible?

<comment><USER9> Yes, this should be continually updated (since it's (for the most part) pull from the same API that they have on their own tracking sites).

It won't ever say "this is later than expected" so you'd need to cache the originally scheduled date, but what you're doing should be possible.

</comment></comment></comment></comment></comment></comment><comment><USER14> Thanks for this article. I'm interested in the way you marketed your product to developers in early daus. What problems did you face and how fast did you grow in the first year? I'm currently looking for kind of a benchmark to compare if our API-first platform is on a good track.

<comment><USER9> Hey sedzia - from shippo here, saw your comment.

We actually just published this piece on Mattermark on how we sustained the business and build up our API in the early days. [https://mattermark.com/focused-long-tail-get-traction- enterp...](https://mattermark.com/focused-long-tail-get-traction-enterprise/)

Doesn't necessarily have benchmarks, but maybe the story will trigger some ideas.

</comment></comment><comment><USER15> I like the data science angle where they're using their large number of deliveries to make predictions for customers on when packages will arrive depending on carrier, route, day/time. Would be nice for a future blog post to show some stats on how accurate their predictions are.

<comment><USER9> Great idea! When we're ready to share the stats we'll definitely do something like that.

</comment></comment><comment><USER16> Site appears to be down - mirror?

<comment><USER3> Sorry, Medium and/or GoDaddy DNS sometimes has hiccups with the custom URL. This is the direct Medium link: [https://medium.com/get-put-post/apps-that- send-packages-with...](https://medium.com/get-put-post/apps-that-send- packages-with-the-shippo-api-6184d909ccb1)</comment></comment>


ThinkTank to launch a startup this weekend - <AUTHOR> http://thinktankstartup.com

<comment><AUTHOR> Seeking submissions for a startup to be launched this weekend. This venture will have the backing of attendees from the ThinkTank 2008 conference. See www.purposeinc.com for a list of attendees. ...50 of the brightest minds of internet marketers and entrepreneurs today.</comment>


Ask HN: The good that can come out of these times - <AUTHOR> Hi HN.
Hope you're all well.
I'd really like to hear/read the thoughts of this community about the good (in any field of life) that can come out of the hard times we're in nowadays. Thank you. Wish you well. <comment><USER2> I would have to say one positive concerning people having to work from home is that folks will be far more prone to question the legitimacy of not being offered the option to work from home. Kind of like the Microsoft rep noted, this could be a massive inflection point for the way we work.

</comment><comment><USER3> "Never let a good crisis go to waste"

In India, where there is currently a huge political divide, this crisis can actually bring people together. The right and left wing bigots will realise this soon enough.</comment>


Mental health in startups – we are not alone - <AUTHOR> https://medium.com/@anonent/mental-health-we-are-not-alone-2345240b0c3f#.716nn8jtg <comment><AUTHOR> Hi everyone,

Last week I posted an article on mental health in startups (my first HN post!)

I was blown away by the response it got, over 10k reads in 2 days, which is huge for me.

I've written a follow-up post on what I've learnt and shared some of the comments I've had.

There are so many people who want to discuss their issues but just do not know where to turn or are afraid of being judged.

Thanks so much for the reads and shares on the last one (HN was my biggest referrer!)

I've created a short survey within the post on mental health in startups and would love to learn some more about these problems.

James</comment>


My Job Interview at Google (2008) - <AUTHOR> https://catonmat.net/my-job-interview-at-google <comment><USER2> > The questions were technical but not very challenging or difficult.

I might be really bad, because few of the questions were challenging and difficult. I was expecting a rant how bad the google interview process is, which is usually what I come across. I do agree the questions are fun and are helpful to someone who is preparing for an interview. The interview and programming questions covered lot of topics.

</comment><comment><USER3> > The questions were technical but not very challenging or difficult. So THATS > why Google products are so consistantly shite...</comment>


Researchers Uncover Vulnerabilities in LTE Wireless Protocol - <AUTHOR> https://www.securityweek.com/researchers-uncover-vulnerabilities-lte-wireless-protocol <comment><AUTHOR> Direct whitepaper link:

Touching the Untouchables: Dynamic Security Analysis of the LTE Control Plane

[https://syssec.kaist.ac.kr/pub/2019/kim_sp_2019.pdf](https://syssec.kaist.ac.kr/pub/2019/kim_sp_2019.pdf)</comment>