Event Sourcing is Hard - <AUTHOR> https://chriskiehl.com/article/event-sourcing-is-hard <comment><USER4> I agree with the author completely. I worked on a fairly large system using event sourcing, it was a never-ending nightmare. Maybe with better tooling someday it will be usable, but not now.

Events are pretty much a database commit log. This is extremely space innefficient to keep around. And not nearly use useful as you might think.

Re-runs need to happen pretty often as you change how events are handled. Even in our local CI environment, it eventually took DAYS to re-run the events. It was clear that the system would never survive production use for this reason alone.

De-centralizing your data storage is a bad idea. We ended up not only with a stupidly huge event log, but multiple copies of data floating around at each service. Not fun to deal with changes to common objects like User. Sometimes you would have to update the "projection" in 5-10 different projects.

In practice ES amounted to building our own (terrible) database on top of a commit log that's actually a message queue. Worst technology I've worked with in years, eventually the whole project collapsed under it's own weight.

Some of these problems are fixable in theory. Perhaps a framework to manage projection updates, something to prune old events by taking snapshots, a DB migration style tool to "fixup" mistakes that are otherwise immortalized in the event stream. But right now, seriously stay away :)

<comment><USER5> I agree that it's hard, however doable and pays benefits if you know what you're doing. I worked on 3 successful implementations for finance sector and we could replay a few million messages per second. Have a look at how we achieved that in LMAX: [https://martinfowler.com/articles/lmax.html](https://martinfowler.com/articles/lmax.html)

Sorry to say it, but clearly you must have been doing something wrong or employing event sourcing where it does not belong.

<comment><USER6> > LMAX's in-memory structures are persistent across input events, so if there > is an error it's important to not leave that memory in an inconsistent > state. However there's no automated rollback facility. As a consequence the > LMAX team puts a lot of attention into ensuring the input events are fully > valid before doing any mutation of the in-memory persistent state. They have > found that testing is a key tool in flushing out these kinds of problems > before going into production.

I'm sorry, but this is saying "catch your bugs before they reach production" which just isn't feasible on non-critical software development (i.e., most software development). The important part that is left out here is: what happens when one such errors slips in? How do you deal with it after the fact?

That being said, your system is impressive and I loved being able to read about it. Please keep up the good work and specially sharing your findings! :)

<comment><USER7> Not quite! I don't think that's what they're getting at.

The idea is this: Say you have a record A with fields f1, f2, f3. When an even comes in you run a function F with steps s1, s2, s3 each of which may modify a field of record A.

Here's the issue, if s3 fails (due to "invalid input"), the modifications to A from s1 and s2 are incorrect and A is now corrupt.

There are a bunch of ways to handle this but the one described here is to avoid touching data that persists between requests until you're at a stage where nothing can fail anymore.

<comment><USER6> > until you're at a stage where nothing can fail anymore.

... and then there's a NullPointerException because you forgot to check something that could indeed fail (i.e.: you have a bug).

In other words: they advise you to not have bugs in that part of the codebase, which was precisely my objection.

<comment><USER7> Absolutely. But doing things in this style protect you from large classes of the especially hard to reproduce bugs. Nothings perfect but it helps a lot!

I'd never heard it articulated before but I personally discovered this style over the years as well.

</comment></comment></comment></comment></comment></comment><comment><USER8> Event Sourcing = everything that happens is an event. Save all the events and you can always get to the latest state, as well as what things look like at any time in the past. What an "event" is depends on your business domain and the granularity of processing. It's very common in enterprise apps with complex workflows (like payment processing or manufacturing). Good fit for functional programming techniques and makes business logic easy to reason about since everything is in reaction to some event, and usually emits another event.

With global ordering, you can have a single transaction ID that points to a snapshot of the entire system. The actual communication is usually handled by "service bus" style messaging like RabbitMQ that can support ordering, routing, acknowledgements, and retries. Kafka or a RDBMS can also be used but requires frameworks on top.

This concept is used pretty much everywhere. Redux is event sourcing front-end state for React. Most database replication is event sourcing by using a write- ahead log as the stream of mutations to apply. All that being said, I completely with this article that it's the wrong solution for most cases and creates far more problems and limitations than it solves.

<comment><USER9> What about when event structures change? Now you're having to push versions into your events and keeping every version of your serialisation format.

Redux often does not have to keep track of versions, because the event stream is consistent for that session.

<comment><USER8> The most common approach is to add versions to the events. The good thing is that with event sourcing, the exact cutover and lifetimes of these schema versions can be known (and even recorded as events themselves).

Downstream apps and consumers that don't need to be compatible with the entire timeline can then migrate code over time and only deal with the latest version. You have to deal with schemas anytime you have distributed communications anyway, but event sourcing provides a framework for explicit communication and this is one area where it can make things easier.

<comment><USER10> For me an issue, not usually made explicit, is that those benefits seem to require a pretty stable business and a stable application landscape.

Why? Because a changing business requires changes to the Events. Since a change in an Event requires that all consumers to be updated, immediately or at the schema version deprecation, the cost of change seems to increase faster in comparison to an application landscape without event sourcing. At the same time, the cost to know who needs to be changed also grows since "any" application can consume any Event.

An stable application landscape also seems to be required because if the number of an Event consumer grows quickly the ability to update, and deprecated schemas, seems to be related with the number of Event consumers (which require update).

<comment><USER11> If your org is anything like mine, mostly things (data) are "additive" onto the existing structure. When you want to deprecate something, you can notify all the consumers like a thirst party would, if you were going to change something enough. But this later happens much rarer for us, though tends to leave traces of technical debt....

</comment></comment></comment></comment></comment><comment><USER12> First we need to agree on what "Event Sourcing" means. In my view you don't need to implement every single Event Sourcing pattern to have an "Event Sourced" system. Say you have a TODO List app (yes, pretty cliche but that's ok). In that TODO list you have a <form> in which you post the state of the TODO list to the server. That state is stored in the database in the form of an event "TODO_LIST_SAVED". When you want to "replay" back, just list all the events from the DB in chronological order while filtering the ones the user has access and pick the last, then rebuild the HTML using that one event converted into the view model.

Kaboom, you have an event sourced system that doesn't even use a queue.

The problem with trashing the idea is that people have a bad experience with it, either by over-engineering and trying to apply all the solutions with tools or hand-made implementations instead of using a Lean approach, or storing events in a type of business model that doesn't even require a database.

"Event Sourcing is Hard" is a statement as true as "web development" is hard, or "distributed systems" is hard, or "API" is hard, "eventual consistency" is hard... yet, we build those things every day doing the best we can. In fact, anything, any practice, any technique, any architecture can be hard because software is hard. Even harder is to not over-engineer something that can be very simple.

Simplicity is hard.

<comment><USER13> Kafka-oriented streaming folks talk about stream-table duality; the idea that one form can be expressed as the other. There is usually a little lip service paid to this idea before some heavy hints that _actually_ , the _stream_ is the _true_ reality are dropped.

My own view is that there are dimensions for any data of interest, expressing some ability to show an evolution of it. Frequently that dimension is time, or can be mapped onto time.

But neither the stream nor the table is the truest representation. The truest representation is whatever representation makes sense for the problem. Sometimes, I want to clone a git repo. Sometimes I want to see a diff. Sometimes I want to query a table. Sometimes I want a change data capture stream. Sometimes I want to upload a file. Sometimes I want a websocket sending clicks. Sometimes you need a freight truck. Sometimes you need a conveyor belt. Sometimes a photo. Sometimes a movie.

Sometimes I talk about space vs time using the equations for acceleration, or for velocity, or for distance. These are all reachable from each other via the "duality" of calculus, but none of them is the One Truest Formula.

And so it is for data. The representation that makes the most sense for that domain under those constraints is the one that is "truest".

<comment><USER14> I believe this is where CQRS plays nice with "event sourcing". You write all your events in one model, but can read them in multiple ones, that is if you can tolerate some read latency... and most systems are usually ok with that.

<comment><USER15> CQRS also alleviates a lot of the pain people experience around event sourcing and distributed systems with event evolution and manipulation. There are many cases where it's inappropriate for consuming services to be aware of the internal representation of events. Giving them a 'snapshot-centric' view of the data can be a simplification in both directions.

</comment></comment></comment></comment><comment><USER16> The great thing about HN is that it consistently shoves into my face how many, seemingly common, dev tools or frameworks etc... that I've never heard of.

Event sourcing isn't anything I've ever heard of, let alone something for which broad marketing promises need debunking.

How common is this framework/archecture/product? Google isn't helping me determine how widely used it is.

<comment><USER17> EventSourcing is not a Framework, but a concept.

The idea is to store not the current state of your app, but the transitions (events) that derive into the current state.

Think about how git stores your source code as a series of commits.

In theory it is a beautiful idea; in the real world, it is hard to implement.

<comment><USER18> In fact, git stores a full snapshot of your entire repo with every commit. It does not store diffs from the previous commit. When you do "git show <COMMIT_SHA>" it generates a diff from the parent commit on the fly.

There's a huge optimization though: it uses a content-addressed blob store, where everything is referenced by the sha1 of its contents. So if a file's contents is exactly the same between two commits, it ends up using the same blob. They don't have to be sequential commits, or it could even be two different file paths in the same commit. Git doesn't care - it's a "dumb content tracker". If one character of a file is different, git stores a whole separate copy of the file for it. But every once in a while it packs all blobs into a single file, and compresses the whole thing at once, and the compression can take advantage of blobs which are very similar.

<comment><USER15> Gits bi-modal nature is a wonderful representation of a sanely architected Event Sourced system. When needed it can create a delta-by-delta view of the world for processing, but for most things most of the time it shows a file- centric view of the world.

IMO a well-factored event sourced system isn't going to feel 'event sourced' for most integrated components, APIs, and services because it's working predominately with snapshots and materialized views. For compex process logic, or post-facto analysis, the event core keeps all the changes available for processing.

Done right it should feel like a massive win/win. Done wrong and it's going to feel hard in all the wrong places :)

</comment></comment></comment></comment><comment><USER19> Datomic is, at its core, an event sourced datastore, and it works really well. I don't think event sourcing is something that should be solved in application space — you wouldn't write a database from scratch for your products yet implementing an in-house event sourcing system is for some reason more acceptable.

All of these stories of failure are stories of teams bandaiding event sourcing on top of other databases not optimized for the usecase.

<comment><USER20> I was thinking about Datomic when I read this, and yet came to exactly the _opposite_ conclusion :)

Datomic may be something like 'event sourcing' internally, but it works so well because it abstracts and _hides_ the naked overreach that the article pounds against.

Datomic users think of it as a DB, not a stream of events. So I don't think that Datomic users are following the 'event sourcing' model.

<comment><USER21> Fair point... but, events in the style of "User address changed" are very repetitive, wasteful, fragile etc.

So stopping seeing user-address-change as (primarily) an event is a win. One always can fall back later to perceiving user-address-change as an event.

For "real" events (e.g. "Transaction fraud detected") possibly I wouldn't use Datomic at all even if it was my primary DB. There are more suitable pub-sub systems.

And that results in a neat separation of concerns: data changes in one store, actual business-valuable events in the other.

</comment></comment></comment><comment><USER22> Good summary of the drawbacks of ES.

I think one thing can not be repeated often enough:

Eventsourcing can be an incredibly valuable approach, but almost always only for a very SMALL SUBSET of your system.

Most of the problems with ES materialize from trying to build your whole architecture around it.

I think many developers fall into this trap because theoretically the concept sounds so appealing and elegant (immutable, reproducible, modular, ...)

<comment><USER23> Event sourcing is REALLY hard to figure out how to do "right." A lot of getting it right is modeling knowledge/experience, understanding your domain.

That said, you can succeed at building your entire arch around it and once you do, it's glorious. Kafka Streams makes the technical aspects easy once you figure out how to model correctly.

<comment><USER22> Out of curiosity, how do you deal with consistency guarantees across aggregates? (which is much more relevant when your whole architecture is ES)

I realize this is highly domain dependent. Some will be much less affected than others. But it's another drawback not mentioned, because now you start to need sagas/managers that coordinate across services with commit/rollback patterns , conflict resolution., etc.

<comment><USER23> As zenpsycho said, all you get is eventual consistency across aggregates, if you're talking about projection aggregates. As you say, for domain aggregates you can wire up transactions by writing your own 2PC on top of Kafka exactly once semantics.

I would recommend only to people who are really committed to the idea or know what they are doing and modeling. It took me a long time because I went from zero knowledge of Java/DDD. First I had to learn Java, then Kafka, then KS, but still I was lost. Learning the most basic DDD was enough for me and did the trick though. Writing a 2PC to coordinate across aggregates wasn't pleasant but also wasn't hard with KS. The hard part was learning all the other stuff and the modeling.

I think a well done ES framework based on Kafka streams would maybe be the first ES framework that would have a chance. The primitives in KS seem just right.

<comment><USER24> DDD in this context = Domain Driven Design?

<comment><USER23> si

</comment></comment></comment></comment></comment></comment><comment><USER25> I just finished the last 2.5 years replacing a somewhat complex legacy system. It's a lot of Spring Integration, JMS Queues in between (so technically, 'events') and a traditional relational DB. The system was deployed 6 months after conception and piece meal migration and feature addition to support a full decommission of the legacy system.

It runs very well and the business is happy. However, I feel I need to convert it to Event Sourcing/CQRS not because for any technology constraint, or business requirement...but I feel if I don't have buzz words on my resume, I won't be marketable.

Ideally, systems are created to meet current and future business needs within time and budget constraint. In reality, I need to also maintain marketability. JMS queues and 2 phase commits....old school!

<comment><USER26> > It's a lot of Spring Integration, JMS Queues in between (so technically, > 'events') and a traditional relational DB.

Ah, i wonder if you're working on the project i worked on at my previous job ...

> It runs very well and the business is happy.

Apparently not.

</comment></comment><comment><USER27> At Qbix, early on we made the decision to do replication via logs of events, which we call "messages". This was at a time when Parse and Firebase were all te rage and collaboration was done by things like Operational Transformations and various diffing.

We figured that a social activity would be best represented as a linearly ordered set of messages, interpreted by various actors (and front end components) depending on the type of activity.

The thing is, you usually don't want global ordering of events across all activities. That introduces problems that Google solves with Spanner and CockroachDB works hard to solve. Global Byzantine Fault Tolerant Consensus is even harder to achieve with any scalability, as can be plainly seen from Bitcoin and Ethereum. You can do it if you trust most of the nodes (like Ripple) but otherwise it's infeasible.

But it's also overkill. You need vector clocks locally for activities only. Like a chess game or a chat. You don't NEED to know which message came first across chats, or unrelated transactions across countries. It's a bit like quantum entanglement — only if actors activities and transactions start being entangled do you need to start caring about conflicts. And it's a bit like relativity — if events happen far enough apart then it doesn't matter which happened "first", it will depend on the location of the observer.

So anyway... the primitive for us is the Group Activity, and they can reference each other, forming Merkle Trees and DAGs if needed, just like in Git and so on.

For more info see qbix.com/platform/guide/streams

</comment><comment><USER28> Interesting- Sounds to me like you were reinventing a database - <USER2> has taken over 30 years, has 400 contributers and 1.1 Million LOC, no surprise to me, it was a bit tricky! PG has a rock solid ACID enabling transaction log - the Event Source - you can access this easily via logical decoding functions. You can easily replicate this data into tables, if you need to keep it, you can also add system and applicable timestamps to get a bi-temporal db, to enable queries to go back in time. You mentioned impedence mismatch with GUI, relational databases are famous for ths same problem, fantatic tools like PostGrest and recently GraphQL integration have already been built to address this.

<comment><USER29> just wanted to say that I don't like your comment. too shallow, wrong direction - something along those lines

</comment></comment><comment><USER30> I've worked in a couple of ES based trading system (matching engines and algo trading both) and it was definitely the way to go, especially on the equity side were you can restart or snapshot the system every day. In my startup, I've designed the core transactional system and it's serving us well for 2 years now. It comes with its own challenges and you need more senior devs than your average crud system hence you use it only for parts which are better specified and less subject to change. As everything it's a trade off.

<comment><USER31> I think this is the kind of pragmatism that is needed when taking about event sourcing. It is a _very_ advanced architectural pattern. There is no easy 'Event Sourcing made simple' way to use it.

I think one of the reasons for this is that the systems people often describe when they talk event sourcing are actually _three_ different, but interrelated, architectural patterns:

\- event sourcing (build your models based on immutable facts) \- event driven (side affects triggered by messages, often delivered by queues) \- workflow / state machine

It takes a long time to get these concepts straight. In our case once it was untangled, our framework can quite clearly demonstrate how they relate: messages update the state in a workflow engine, this triggers side effects, the results are captured as facts and used to build the model, repeat.

This worked for our use case, in which our transactions have a strong workflow, it may not work in other cases.

Finally the one point I'd certainly reinforce from the article is: _don't reach directly into the event stream_. This causes huge amounts of coupling. Instead, we ended up using bounded contexts to define our systems, and then treat key events as our API. It sounds counter to some of the ideals of event sourcing, but it is absolutely needed once you grow past that toy phase.

<comment><USER30> Very good points. Cleary defined bounded contexts are the way to go and also having a clear distinction between internally and externally visible domain events. Other tricky points to get right how to distribute events and the relationship order (as in stream ordered, totally ordered, entity ordered) based on the scalability and "correctness" requirement. For instance in algo- trading is not uncommon to have totally ordered events throughout the system.

</comment></comment></comment><comment><USER32> Event sourcing is still the best paradigm for exposing valuable service data to an undefined number of services downstream, especially when there's enough data to make querying painful for some use cases. They are like a NoSQL buffer between services and databases. Not always necessary but sometimes useful.

Event sourcing brings its own complexities (eg Kafka clients) but it's still better than having one huge shared database or even RabbitMQ fanout exchanges. In the best cases, developer experience of Kafka can be very good and comfortable. You can write short python scrips that ingest data and dump to other databases and services, it's very nice. In some cases you want services writing directly to databases but sometimes you don't.

<comment><USER33> This has nothing to do with event sourcing. What you're describing is event- driven integration. Rarely the events used to source the domain model should be the same as the ones used for the messaging integration. Unfortunately, due to the repetition of the term "event" the purpose gets confused most of the time. The "Data on the Outside versus Data on the Inside"[0] paper makes a good distinction even though it doesn't use the same terms.

[0]: [http://cidrdb.org/cidr2005/papers/P12.pdf](http://cidrdb.org/cidr2005/papers/P12.pdf)

<comment><USER32> This is a good point (about how messaging integration requests differ). I think the choice comes down to whether at least once or exactly once makes more sense for a specific unit of shared data / functionality. For exactly once, APIs and RabbitMQ shine for connecting services.

I'm still wrapping my head around when to do what in SOA. Thanks for the link btw.

</comment></comment></comment><comment><USER34> Never heard of event sourcing before, but I've used this pattern myself (as another comment mentioned, when you need it, it's a fairly obvious way to do it). Looking it up, I am guessing it's from Martin Fowler's book about patterns and architecture design? And on a related note, would you recommend the book to the average developer?

</comment><comment><USER35> This is question more than a comment, as I have only casual knowledge of event sourcing...

"""

You wouldn't let two separate services reach directly into each other's data storage when not event sourcing – you'd pump them through a layer of abstraction to avoid breaking every consumer of your service when it needs to change its data

"""

Isn't the event itself precisely that layer of abstraction? That is, you're not publishing the details of your data store. You're publishing an event which is a thin slice or crafted combination of details that ultimately reside in that store, but which you are hiding...

Am I misunderstanding the quote?

<comment><USER36> > Am I misunderstanding the quote?

I don't think you are misunderstanding the quote, I think you are misunderstanding the nature of the problem.

If you tip your head sideways, you may notice that the persisted representation of your model is "just" a message, from the past to the future. It might describe a sequence of patches, or it might be a snapshot of rows/columns/relations. But it is still a message.

The trick that makes managing changes to this message schema easy is that you own the schema, the sender, and the receiver. So coordinating changes are "easy" \-- you just need to migrate all of the information that you have into its new representation.

If the schema is stable, the risk of coupling additional consumers to the schema is relatively small. Think HTTP -- we've been pushing out new clients and servers for years, but they are still interoperable, because the schema has only changed in quiet safe ways.

But if the schema _isn't_ stable, then all bets are off.

Because of concerns of scale/speed, we normally can't lock all of our information at once. Instead, we carve up little islands of information that can be locked individually. The schema that we use are often implicitly coupled to our arrangement of these islands, which means that if we need to change the boundaries later, we often need to change schema, and that ripples.

And all of this is happening in an environment where business expect to change, and there is competitive advantage in being able to change quickly. So it turns out to be really important that we can easily understand how many modules are going to need to be modified to respond to the needs of the business, and to ensure as often as possible that the sizes of the changes to be made are commiserate with the benefits we hope to accrue.

</comment></comment><comment><USER37> I've been using event sourcing as an exclusive data store for a variety of toy modules, which are slowly becoming something large... I am using all bespoke infrastructure, both for managing the logs and for building UI.

I don't think what the author did sounds like event sourcing, as I think of it. His setup sounds more like pubsub. In all honesty I'm probably the one doing it wrong though.

My event stream isn't typically consumed or listened to by arbitrary listeners. (Although it can be, in some rare cases) Each event is namespaced to a specific module. When you load a log, you provide singletons for each module. Each message is only received by one singleton. If you want to broadcast that message to another module you have to do that from the parent module.

Everything is totally imperative so there's no ambiguity about what causes what.

If I need data on a different machine, or in the browser, I just send the log down as a procedure and run it with fresh singletons on the client.

I also don't have one mega stream, I only mix modules that actually interact. If I have two pieces of data that aren't connected, they each have their own log.

I don't know, maybe I'm the one who's not really doing event sourcing.

Like I said, I'm still somewhat at the toy stage, so maybe I will get to where OP is eventually. I do plan on doing log rewriting for compression. That might lead to some pain.

I also haven't yet had to deal with sharding. I suspect there's pain there.

My plan is to use the one-singleton/one-module per message namespace rule to keep things from getting too complex. I figure if the same module that consumes messages is responsible for rewriting them, maybe it won't be too weird. We'll see. I have a lot more prototyping to do.

If anyone is interested, the core module is "a-wild-universe-appeared" on NPM. It's still in the 0.x.0 series so it could break. But the basic API is pretty simple and hasn't changed much in a few months even though I've been using it regularly.

[https://www.npmjs.com/package/a-wild-universe- appeared](https://www.npmjs.com/package/a-wild-universe-appeared)

I agree with the OP that it's perhaps too soon to throw this kind of thing at production problems. We need more research on how stores like this integrate with other layers.

</comment><comment><USER38> Event sourcing is often described in the context of a rich domain model where every event has slightly different semantics with respect to the domain. As a result, there is often a need to adapt these event types over time as the domain changes. This could involve revising an existing event type (which is fine for adding new attributes for capturing more information) or creating a _new_ event type to model some that has changed in the domain. If you simply incorrectly modeled the events for the domain, then you will have the same challenges of migrating a poorly designed database schema.

The arguments that Git and Datomic are both event sourced systems are good examples of successful application of this pattern. However, these are poor examples when it comes to event sourcing where the events are "domain events". In both Git and Datomic, the data model of the _event_ are pre-defined. With Git you have a changeset and with Datomic you have datoms, both of which are composable by design and where every changeset and datom is equivalent (from a structural standpoint).

Applying event sourcing to an arbitrary domain model means every event is both semantically and possibly structurally different. That is, how event of type X affects the state compared to an event of type Y is different, where as applying two different datoms or changesets in Datomic or Git, respectively, change the state in the same way.

So I think there are two fundamental challenges faced when using event sourcing with a domain model. First is that the type and structure of events need to adapt to the domain (business, organization, etc). Datoms and changesets never change.. they are fixed and therefore don't have that challenge. Second, and related, is that as new events are introduced or existing ones are adapted, etc. the code that processes those events are unique as well (not just the model of the event itself). This challenge is exacerbated if there are multiple downstream consumers of this event stream where now there is "coupling" on the event/data side of things and likely semantic coupling as well.

Again, this is not something you run into with Datomic or Git simply because the data model of the _events_ in those systems (the fundamental unit being left-folded) has a fixed data model and semantics.

</comment><comment><USER39> Some contrarian opinion: [https://vvvvalvalval.github.io/posts/2018-11-12-datomic- even...](https://vvvvalvalval.github.io/posts/2018-11-12-datomic-event- sourcing-without-the-hassle.html)

Basic idea is to make the event log dumb. Instead of capturing business logic, make them as dumb as SQL commit logs: this row was inserted at this time, that row was deleted at that time. Sort of like a glorified WAL file. Benefits of this scheme? Querying old database states and auditing.

</comment><comment><USER40> I've never read or heard that event sourcing was supposed to be all sunshine and rainbows.

Microsoft wrote about their adventures going down this path [0]. It's well worth the read if you're considering it for your project.

For a few features in my company's current platform we use event sourcing. We collect plenty of small data points over time that is aggregated into rows to provide users summary information of reams of operational data. We tried aggregating the data in queries and despite our best efforts to optimize our indices, tables, and queries there wasn't any way to compute it in any reasonable amount of time.

The pain points for us:

Our UI team wasn't in sync with how data flows in an event-based system. There's a lot of friction there. We're slowly updating the team on task- oriented user experiences and breaking down our UI components to transmit their commands directly. For now a lot of our control plane has to break up the huge form data we receive into commands and send back partial responses to the client. As we move forward and find better UI patterns this has improved.

The control plane controls some data models that are not event-sourced and are mutable. Our users tend to expect to be able to rename and delete objects in the system that our event-sourced models refer to. It caused some confusion when certain views in the application that are built from our projections wouldn't see updated name of the object they had just renamed. And so we ended up doing the event-sourcing no-no of emitting the CRUD events so that our projects could appear in the manner our users expected. This is partly because of the aforementioned problems with the UI team but is also a problem with event-sourced models referring to data that can mutate over time.

However it hasn't been a hellish experience either. I took the liberty of developing some models of our event-sourced infrastructure and features in TLA+. This has been helpful to ensure that certain properties of the system under development would hold: consistency, availability, etc. You may not be willing to go down the path of learning TLA+ but the key take-away there was that a little planning goes a long way with a project like this: simple unit tests and whiteboard diagrams are not going to cover all of the things that can go wrong in an event-sourced system. If anything it might convince you to keep it simple, limited, and constrained as we did.

 _edit: forgot link_

[0] [https://www.microsoft.com/en- ca/download/details.aspx?id=347...](https://www.microsoft.com/en- ca/download/details.aspx?id=34774)

</comment><comment><USER41> It is hard to separate DDD, Event Sourcing/CQRS - they seem to all be joined up concepts promoted by a small circle of people.

I worked on a blockbuster project financed by a local billionaire in a Gulf State where the entire shebang was "mandated" by the CTO and a well-known Scala consultancy. Event sourcing, CQRS, with DDD to define architecture.

Let me describe one simple issue that was almost intractably complex - <USER3> sign up.

We had one service which was authentication, and another which was user preferences.

The front end sends a create new user _command_ (set up a new user in the auth system with a password, email, etc), we also needs now to send some kind of message to initialise the user info records (e.g. home address, telephone, language preference, etc.)

We need to implement a saga for this, e.g. implement a process coordinator. Or maybe the authorisation system should on first request where the data is not populated, fill in a blank, default record?

We genuinely achieved organisational paralysis over this, as there was no clear emergent right way to do this, and a bunch of ways that really smelled.

Additionally, regarding the a code organisation perspective, we had a repo that had the central commands and events defined as code, which every service linked to. So we had a huge central dependency which had a high velocity of change.

What was meant to be a distributed, loosely connected system was in fact the most coupled system I've ever worked on.

Architecture is about clear communication, and the developers were simply perplexed. We were bringing in 10 developers a month, and communicating the architecture was impossible. Lofty ideals were espoused.

The first basic rule of delivery in a software engineering project, KISS, was universally ignored, in favour of using "sexy" architecture.

Fitfully, the CEO was first sacked after 18 months at the helm, a few months later the CTO and 95% of the dev staff. The year long coding effort was turfed. Estimates are $50m-$100m was burnt.

The winner of course was the DDD consultant that mandated everything, whose word was seen as law, who didn't actually seem to have very much pragmatic, practical experience - he was getting paid $2k/day, and found it difficult to listen to any idea which watered down his architecture in any way - I think he would eeked out enough cash from this shitshow to purchase a small dwelling.

The biggest losers were the hundreds of staff that had relocated to the region sometimes with families, that had made genuine plans to be in the region for years, that would have all had their visas cancelled.

I think the entire thing is a con - my experience was in ideologues and purists pushing it and making themselves niche consultancy/speaking/writing careers in it.

Now this is just one apocryphal story - but it really is an architectural style that can totally wreck a development or even a company - people are ideological about it - it is not low-risk, and the entire Saga/Process Coordinator stuff is just a tack-on to try to (unsuccessfully in my opinion) answer things that simply don't work properly. The message I mean to communicate is that the async-everywhere nature of CQRS/ES is super complex where coordination is required.

If you are going for loosely connected services, I far more prefer the microservices/RPC architecture - such as Netflix - as the synchronous model with distributed load balancing is a lot more sound.

<comment><USER42> I might be misinterpreting your example, but wouldn't you have both the auth service and the preferences service listening for a 'new user' event (which would contain all the details needed by both systems), and both if them acting on it? Of course it would get more complex to handle error conditions, e.g what do you do if there is validation on the preferences that fails...

<comment><USER41> There are commands and events in CQRS/ES - where you need commands and success reported from multiple subsystems from front end interaction, now you need a transaction coordinator for your "saga"

Typically the front end needed to get a success message to say user created, or user creation rejected (e.g. backend checking on valid postcode, valid username, etc - rejections which would come from two separate services.)

<comment><USER43> I am completely confused by your description ...

I have also not used a 2pc transaction coordinator in well over a decade.

Also you use the term "saga" which is correct in that what people are often doing here is not in fact a saga (which actually has a pattern description) it is usually a process manager which is a different pattern.

<comment><USER41> If you're confused, welcome to how everyone felt. The nuances and abstract definitions made for confusion for the staff. I've forgotten what I knew about it, you may well be right in your definitions, however what was clear was that we needed to implement asynchronous pub/sub comms on the Kafka queue to get this work. What might have been simple with an RPC (e.g. a POST/PUT) was turned into a system which needed to track the state of its asynchronous RPCs, and listen to the queue for responses.

All of this was for no real purpose, and as I said the proof was in the pudding - tens of millions of dollars wasted, hundreds of staff hired and then fired. The domain itself was not complex (e-commerce) but the implementation was ludicrous.

There were basic delivery problems in the project as there was a complete lack of keeping things simple, and massive overengineering.

</comment></comment></comment></comment></comment><comment><USER44> I'm of the mind that the usefulness of event sourcing as an architecture is directly correlated with how easy it is to determine what an event is.

I'm also of the mind that most challenges with event sourcing are ultimately tooling problems, and that we will "get there" eventually, for some fairly pleasurable definition of "there."

</comment><comment><USER45> It is worrying that a central figure to Event Sourcing & CQRS like Greg Young reduces the "framework" to a function, a pattern match & a left fold.

Linked in the article [https://youtu.be/LDW0QWie21s?t=1926](https://youtu.be/LDW0QWie21s?t=1926)

<comment><USER46> Why do you find that worrying?

I interpreted that section as "the core bits are easy, and the frameworks people have built don't really help you with the non-easy problems that come later, so they provide little value", which (assuming the statement about the frameworks is true) seems like a reasonable position?

<comment><USER45> It is not how I interpreted it. What I heard was salesman pitching an expensive product as a bargain.

Even though frameworks have lots of drawbacks, I think it solves one problem really well, it gives you, the team, a direction.

Doing Event souring & CQRS correctly takes years of experience, this can be concluded by reading articles like this one or watching any of video by Greg Young.

In a sense the origin of this article stems from the notion that no framework is needed. I think that is a setup for disaster by selling developers the idea that this is easy, when it isn't.

In my experience frameworks have often learned me how _not_ do things. Frameworks are condensed experiences that you don't need to learn yourself, someone else has already done the mistakes for you. This is a huge time saver & gives you, the developer, experience at a lower cost.

With that experience going frameworkless can then be achieved if necessary.

However my interpretation of it can be exaggerated due to the fact it was a short statement without much context.

</comment></comment></comment><comment><USER47> I think Event Sourcing is neatly defined as an algebraic group and provides a nice abstraction to reason with. Perhaps the problem is not recognizing this notion and not proving the correctness of a system before implementing it?

</comment><comment><USER48> Although terminology differs, storing the canonical source of truth in Kafka has worked great for many of my clients. If that is Event Sourcing, then it can be made to work easily. I do get asked many, many questions about this, often from inexperienced teams. I took their questions, and my answers, and posted them here:

[http://www.smashcompany.com/technology/one-write-point- one-r...](http://www.smashcompany.com/technology/one-write-point-one-read- point-one-log)

</comment><comment><USER49> If people are indeed directly accessing a stream, they are violating a fundamental service oriented principle - Teams must communicate with each other through their service interfaces. It is worth understanding this concept with clarity. Assuming we go with Kafka, even if we don't need any additional functionality other than what Kafka provides out of the box, it would still need to be wrapped in a service and treated as an actual service. Otherwise, it becomes a shared resource.

<comment><USER50> This is one of the core issues that make many ES systems complex. A service owns an event in the same sense that it owns internal state. It's internal to that service and only expose what it finds appropriate, in the way it finds appropriate. That is almost never directly as an event on a public eventbus.

</comment></comment><comment><USER51> I agree that event sourcing (and also CQRS) are not so simple, in practice. The coupling the author mentions is something I definitely experienced, and I think the answer is to separate your internal event representation from both the input (e.g. command) and output representations. I got seduced by the simplicity of having them all be the same, but I definitely found I wanted to be able to vary these things independently.

In the first project where I applied event sourcing, I treated my commands (system inputs) as events, but ended up regretting it. The problem was that many commands cause multiple effects and as the number of commands increased, the complexity of the logic for deriving the state started to accelerate.

If I could do it again, I would have strictly separated the command and event ontologies, and adopted the concept of a command processor. The command processor takes a command and an event list and returns feedback on the validation of the command along with a new event list that is at least as long as the input. Rejected commands would result in the same event list. Accepted commands that map 1-to-1 with events would result in an event list one item longer. Complex/composite commands would result in more events. I probably would have logged the commands to retain the command-event relationship (for things like undo), but largely that would be a separate thing.

If commands and events are separated, CRUD doesn't require the event sourcing paradigm to bleed through to the UI. Each operation is just a command. Valdiation comes directly out of the command processor and can easily be mapped back to user-visible feedback.

The other mistake I made was concentrating too much logic in my state calculator. My calculator produced all derivable facts from the event log, as well as housed the validation logic (the command processor). In retrospect, I should have figured out what the most fundamental derived facts were and then moved higher level facts into their own calculation logic. I think this would have made maintenance and testability far easier.

While event sourcing does come with its baggage, I find that in projects that aren't using it, the cost is a bunch of ad hoc solutions to the problems of "first-class change", which you get for free. It's an extremely helpful essential technique when modeling stateful workflows.

<comment><USER52> > If I could do it again, I would have strictly separated the command and > event ontologies,

Your use of "I" instead of "we" is interesting here.

I find that when there is a separation of two two things that naively look like they can be collapsed into one things (e.g. because all the simple cases have 1-1 mappings) then someone will either collapse the two -- or (more likely) force you to collapse them by building in an assumption about the 1-1 mapping.

Your beautiful separation then becomes useless. Do you have experience about how to set up teams that don't do this?

<comment><USER51> I say "I" for a couple reasons. First, I'm no longer at the same company, so that's just my personal reflection. Second, I was the architect of the project and definitely the person who sold the bill of goods on the benefits of event sourcing :). It was largely successful, but with those lessons learned.

I'm not opposed to collapsing things that have a 1:1 mapping. It's often a reversible decision, when/if you find the simplification is no longer actually simplifying things. The problem is that as these representations cross boundaries between modules and systems, reversing the decision becomes far more difficult. This isn't limited to event sourcing at all, though. It's the fundamental concept of encapsulation and coupling in system design.

I have found it difficult to socialize the benefits of encapsulation in a team, because the upfront cost is easy to see, but the downstream benefits are not. Sometimes, I've made the judgment to just step back and let people learn from their own mistakes. I've learned the hard way that it's actually not the worst thing in the world.

</comment></comment></comment><comment><USER53> There's sort of a middle ground between event sourcing and ordinary mutable entities: versioned entities.

[http://higherlogics.blogspot.com/2015/10/versioning- domain-e...](http://higherlogics.blogspot.com/2015/10/versioning-domain- entities.html)

The particular schema described there isn't suitable for highly concurrent entities, but a more suitable schema could be employed that achieves the same goals.

<comment><USER54> We did something similar to this for a CRUD app that needed to become append only, have a full change log, have approve/deny events, and the ability to be rolled back. We still had an event table, but instead of having event data it just had a reference to a 'shadowed' (versioned) entity in the entity table. Once an event is approved, you project the shadowed entity on to the real one. That way the ID of the real entity never changes. This worked really well for our very specific use case (simple CRUD events, monolithic app.)

</comment></comment><comment><USER55> One of the problems I _think_ I see with event sourcing is its inability to scale. You have to guarantee the order of events, right? How do you do that in a large scale distributed system with eventual consistency, without incurring an insane synchronization time penalty? I'd genuinely love to hear if you have a good solution for this, because if you do I have a use case I need it for, so this ain't a troll comment!

<comment><USER56> [https://stackoverflow.com/questions/41082938/event- sourcing-...](https://stackoverflow.com/questions/41082938/event-sourcing- microservices-how-to-manage-timestamp)

<comment><USER55> that basically says Vector Clock, which is a good solution for where it's usually applied, but I think this is different. Having everything bottlenecks through your vector clock isn't going to be feasible.

</comment></comment></comment><comment><USER21> I don't think one can do proper, robust Event Sourcing without having first a few years experience in hardcore functional programming (particularly of the variant that strictly segregates side-effects), something most of us lack.

OO and loose FP are fine for a (huge) variety of problems, but the hardest problems need the next level of correctness and elegance.

Else you end up authoring yet another Rube Goldberg machine.

</comment><comment><USER57> Here's my take on Event Sourcing: it's not particularly well defined what an "event" is. Did the event happen yet? Did it succeed? Which part? I didn't like having an event ledger say "comment created," where my application is then meant to consume this, handle validation, potentially fail on the db operation, etc.

So here is what I do: I basically combine Event Sourcing architecture with CQRS. Whenever a client makes a write request, that "command" gets written to a log. Then when things happen as a consequence of that command, for example, a successful DB write happens, I write another log that references the command.

So I'll have an event that says:

        id: '1234'
        type: 'command'
        data: {type: 'createComment', text: 'Ayy', userId: '1234', postId: '6543'}

And then that can get picked up and processed by the application, which will create another event like:

        id: '2345'
        type: 'dbWrite'
        data: {model: 'comments', data: {text: 'Ayy'}}
        commandId: '1234'

There's a lot of redundancy, and you can instead rely on something like your DB WAL for some of the consequences of the commands, but separating out the commands from the results like this has made Event Sourcing work quite well for me.

<comment><USER58> Having it clear not just in your own head, but your entire team's head what an event is, quite important, especially to keep in mind what happens when you "play back" a log. Do transactional emails get sent out again? Do upstream services record the playback as duplicate events? How are transactions handled? What external state are you unknowingly depending on for that playback to produce the same result?

<comment><USER13> > _Do transactional emails get sent out again? Do upstream services record the > playback as duplicate events? How are transactions handled?_

The narrow problem here is that Event Sourcing is temporal, but not bitemporal. Or rather, the different kinds of temporality are frequently muddled.

Event streaming in the Akidau/Google style is bitemporal, but mostly accidentally, as a side effect of distinguishing "event time" (fact time) and "processing time" (transaction time / belief time).

(Snodgrass later proposed _tri_ temporal models, including a timeline for when a fact-belief was viewed, which I find both brilliant and slightly terrifying).

The problems of evolving data models, which is the third-order problem, is often hardest when you put a log or stream at the centre of your design. Databases using SQL, for example, struggle with first-order (current state) and second-order (historical state) evolution. But they do much better on third-order evolution, since DDL is built into every relational database.

</comment></comment></comment><comment><USER59> Related to the topic, does anyone have experience with Axon Server? [https://axoniq.io/product-overview/axon-server](https://axoniq.io/product- overview/axon-server) It claims to solve a number of the concerns over adopting an event sourcing model.

<comment><USER60> From what I've seen in demos, it addresses low-level plumbing elegantly but I don't see how a relatively unobtrusive framework (I mean it as a major compliment) can help against making plain old design mistakes with wrong events, commands etc. that don't fit together and are unable to satisfy requirements.

</comment></comment><comment><USER61> Event Sourcing is not hard (as compared to not event sourcing).

But event sourcing is not for every application. Event sourcing solves some, otherwise hard, problems at the cost of added complexity. You need to judge whether it pays off.

Event sourcing assumes particular size of an application. Too small application will pay a lot in complexity with no added benefits because the problem would otherwise be easily solvable without having to use event sourcing. Too large an application will pay a lot in complexity because your write path will be complex due throughput requirement.

Event sourcing requires dedication. You can't go half-way, mix event sourcing here with direct inserts there, for example. This is going to be hell of a complex environment to live in with worst of the two worlds. Event sourcing only solves problems if you use it 100%.

</comment><comment><USER62> But I suppose there is no other alternative for high traffic systems? Concept of "Event sourcing for everything" belief might be the issue itself, not the selective use.

</comment><comment><USER63> If you want to do event sourcing. Look at persistent actors from akka (jvm and clr). It solve a lot of feaures as discussed here with replay, snapshotting, event upgrades etc.

</comment><comment><USER64> Everything in the world works off event sourcing, from your database commit logs to your react. Hard or not, we are stuck with it</comment>


The Digital Age Is Stamping Out Serendipity - <AUTHOR> http://www.nytimes.com/2009/08/02/business/02ping.html?em

<comment><USER2> I do see the argument, but I often come across things that I may never would have before with such applications described in the article. The internet appeals to my need to learn, and speak to those in interesting fields. I suppose it just a different kind of serendipity. I may be looking at an article or blog and discover something entirely new and unexpected within it that I was not looking for in the first place. Quite serendipitous.</comment>


The Programmer's Stone - <AUTHOR> http://www.reciprocality.org/Reciprocality/r0/index.html <comment><AUTHOR> An exploration of the art of computer programming.</comment>


Show HN: A new interactive crowdsourcing social map - <AUTHOR> http://www.doozyrama.com/ <comment><AUTHOR> I have made this app as free platform where people (not businesses) decide which places should be on the map.Most likely google maps will win, but I'm sure that people should have a viable choice.</comment>


The Developer Ecosystem in 2020: How C++ is Doing - <AUTHOR> https://blog.jetbrains.com/clion/2020/06/dev-eco-cpp-2020/ <comment><USER2> _> Speaking of unit testing, at the beginning of 2020 we can see 34% of C++ developers don't write unit tests_

That's just sad.

<comment><USER3> And also really low. I'm astounded it's even under 50%.

</comment></comment><comment><USER4> Well what else can you do but upvote?</comment>


Predicted market correct last 5 days in row - <AUTHOR> http://www.twitter.com/dynofuz

<comment><USER2> There is about a 3% (0.5^5) chance of this occurring with random guesses isn't there? There are enough hedge funds that predict the market that many of them are bound to be right many days in a row. I will be impressed if you get a whole month in a row.

<comment><AUTHOR> Yup thats the correct %. Of course these weren't random guesses. I'm only correct ~2/3 of the time historically (past 1.5yrs) but that's way better than any hedge fund out there. You can see more at dynofuz.com. I'm looking for people to work on this with me, so if you're interested email me at firstname.lastname@example.com</comment></comment>


Is email the ultimate social environment? - <AUTHOR> http://gigaom.com/2007/09/20/is-email-the-ultimate-social-environment/

<comment><USER2> Depends. Does it serve alcohol?

<comment><USER3> More importantly: will it get you laid?

</comment></comment><comment><USER4> how about facebook taking over as the "ultimate social environment" to whatever it is right now? reason why i throw that out there is because it has many functions all in one: email, photos, IM, etc.

</comment><comment><USER5> Maybe, second to the phone (less spam)

<comment><USER6> Agreed. SPAM, phisihing scams, 419 etc is the Achilles heel of email. I often watch/ask how my 12 and 9 year old cousins use Internet... neither of them use or like email. They all use either IM or msg through sites.

Email will never go away but its use will be reduced as the new generation grows up.

<comment><USER7> Maybe, but perhaps your cousins will start using email more as their social circle expands.

<comment><USER8> or facebook

<comment><AUTHOR> ....or their friends live further than 4 houses away.... or they get a job

<comment><USER5> Email is not going anywhere for a while. It might move around in the hierarchy of formality, but that's all.

</comment></comment></comment></comment></comment></comment><comment><USER9> Email is horrible.

</comment><comment><USER10> are they giving out invite codes?

<comment><AUTHOR> We will be rolling out more invites soon. You can sign up to be notified on our homepage www.xobni.com

</comment></comment><comment><USER11> i can twit you, chat you, email you, or send you my blogroll. it's all just formatted text meant to communicate. and albeit mail and chat being the stickiest apps online, to say that email is somehow better than facebook is a farce. there's no set standard mode of communication.

if someone makes an utility that can auto-scale your message though, that'll be nifty. so if i type less than 140 characters, i want it to show up on my twitter list. if i address it singularly i want it to be sent via email. if i type a long winded essay i expect it to be published somewhere somehow.

<comment><USER12> Yeah, exactly. All these mediums are just waiting to be integrated. Just think how similar chat and email are.</comment></comment>


Could you help with selecting a better solution? - <AUTHOR> http://app.beeux.com/testab/perform/19-22

<comment><USER2> I can't choose without a proper mockup. Images don't give the real feel of how it should work like and I don't even know the real use cases here. Are short lists common or is it going to get filled quickly? What kind of projects are these? Do I even need to search my projects?

> Which actions should be available from project list?

I don't know! What are the most common actions? Is "starting a project" common? How could I possibly answer this non-randomly?

These issues made me leave the survey.

Prepare a proper mockup and give simple tasks (abstract enough to be understandable without domain knowledge) to your users. THAT's real testing.

<comment><AUTHOR> Thanks a lot. We'll consider changing our test.</comment></comment>


A Monday rant on various kinds of opinionated people - <AUTHOR> http://people.gnome.org/~federico/news-2012-11.html#a-monday-rant

<comment><USER2> that's right, kill the messenger.

there's been a literal exodus away from Linux towards Mac and Windows on the desktop over the last 15 years because things have gotten steadily worse.

the problem isn't the handful of enthusiasts who are left, it's all of the people who gave up on desktop Linux or never gave it a chance because the people behind desktop Linux just weren't listening.

<comment><USER3> If your timeframe is the last 15 years, it would be wrong to speak of an exodus. In 1997, linux on the desktop was a niche market, only for people who didn't mind editing a text configuration file now and then. Nowadays, you give a desktop with Ubuntu to grandma and she'll be perfectly happy with it. What Ubuntu and RedHat did was copy the "it just works" experience from Apple's MacOSX, not caring about the non-grandma power users who got their arms and legs maimed in the process of fighting NetworkManager, dbus, and GConf-2. Or the *BSD people, who would prefer a desktop environment that runs without Linux emulation. So, as far as I (or anyone) can see it, people are still engrossed with the Egyptian standard of living and don't mind the slavery so much that they would do an exodus. It might still happen, I guess, if a substantial part of the distro-sponsored Linux developers (Ulrich Drepper, Lennart Poettering, whoever) operates with their fingers stuck in the ears, plowing through with whatever semi-evil plan that they set their mind on.

<comment><USER4> _In 1997, linux on the desktop was a niche market, only for people who didn't mind editing a text configuration file now and then. Nowadays, you give a desktop with Ubuntu to grandma and she'll be perfectly happy with it._

The latter point (about Ubuntu being suitable for grandma) is true, but irrelevant. In 2012, Linux on the desktop is every bit as much a niche market as it was in 1997.

That's fifteen years of hard work by lots of people that has won negligible market share. Which is a huge problem, if you care about Linux on the desktop.

Moreover, there has definitely been an exodus of sorts, especially among developers. You see far, far more Apple machines in developers' hands in 2012 than you did in 1997. Some of these folks have switched from Windows, but a significant number switched from Linux or BSD. And losing developers means losing applications, which means even less likelihood of a market-share breakthrough.

</comment></comment></comment><comment><USER5> I really liked this post. His experience - there are folk talking on mailing lists, there are folk actually doing stuff, and the two groups rarely overlap - matches mine. The concern I have with gnome 3 is that there is sufficient change to trip me up when I'm trying to get stuff done ;-)

</comment><comment><USER6> [http://doublebuffered.com/2009/06/16/the-players-are- wrong-b...](http://doublebuffered.com/2009/06/16/the-players-are-wrong-but- listen-anyway/)</comment>


Deprecation notices in Go - <AUTHOR> http://golang.rakyll.org/deprecated/ <comment><USER2> So interesting, 2 nights ago when we launched our product. I was developing almost exclusively locally, and when we deployed all of the cached deprecated packages we were using hit us when we deployed, we still hit our deployment but it was not fun.

[http://launchaco.com/build/](http://launchaco.com/build/)</comment>


Three facts from "Our World in Data" that everyone should know - <AUTHOR> https://www.gatesnotes.com/Development/Max-Roser-three-facts-everyone-should-know?WT.mc_id=20180626180003_MaxRoser_BG-LI <comment><USER2> I think this is the most important thing to retain from reading this article:

>[...] Both are true at the same time: The world is much better than in the past and it is still awful.

>To bring this to mind I need to know both statistics: When someone says we can sit back and relax because the world is in a much better place, I point out that 11 children are still dying every minute. We cannot accept the world as it is today. And when I feel hopeless in the face of this tragedy, I remember that we reduced annual child deaths from 20 million to 5.6 million in the last fifty years.

Keeping this in mind at all times is a very important thing.

<comment><USER3> I agree with you, and that's succinctly said. It reminded me of this article from a couple days ago.

>Trump administration claims only 250,000 Americans live in extreme poverty, despite UN estimates of 18m

[https://www.independent.co.uk/news/world/americas/us- extreme...](https://www.independent.co.uk/news/world/americas/us-extreme- poverty-numbers-un-nikki-haley-trump-a8417001.html)

<comment><USER4> These seem to be two different measurements of "extreme poverty," each of which is different from the world-wide definition used by economists in the original article:

"The UN's numbers come from the official Census definition which has been kept for decades by the US government, defining extreme poverty as having an income lower than half the official poverty rate."

"Citing a recent survey of American households, Heritage found only 0.08 per cent of American households (or about to 250,000) are in "deep poverty," defined by Heritage as living on less than $4 (£3) a day. This statistic does account for government social spending programmes which help the poor – like Medicaid, food stamps, and housing assistance – while the figure cited by the UN does not."

I guess if you're going to memorize facts, you should also memorize definitions.

</comment></comment></comment><comment><USER5> First off, I stand up and applaud the Gates Foundation, and everybody, who work to improve the lives of everybody on the planet.

WRT: Extreme Poverty, a few years back, we had a political party who tackled the problem of poverty by redefining the measuring stick of what constitutes poverty. _POOF_ much celebration and self-handshaking when they announced that during their term, their efforts dropped the number of people living in poverty by a very significant amount.

This has left a bad taste in my mouth ever since to never trust claims from politicians.

So I wonder, in the case of this article, how many of those people no longer living at the adjusted poverty line and now only marginally above it and no longer included. They're still their, but buy grace of a single digit, are now considered much better off.

<comment><USER6> If you are curious here is the larger exploration of global poverty from Our World in Data: [https://ourworldindata.org/extreme- poverty](https://ourworldindata.org/extreme-poverty)

It has been a while since I read it but I found it quite compelling. If i remember correctly, the measures are based on consumption of goods per day to try to avoid complications related to differing economic systems. Also while it is true that measures can be manipulated to tell the story you want, it does not appear that that has been done in a significant way here. Even if it had i doubt that, such an extreme 90% -> 10% drop could be shown even with the most crooked measuring stick if things had not been genuinely improving a lot. Incidentally, my brother used to be a full on marxist until he read this data and completely abandoned that belief system.

</comment></comment><comment><USER7> There is a fantastic video by Hans Rosling about the global population growth and why global population might actually stall around 10 billion people:

[https://www.ted.com/talks/hans_rosling_on_global_population_...](https://www.ted.com/talks/hans_rosling_on_global_population_growth/up- next)

This relates to fact 2 in the article.

<comment><USER8> His foundation's website also has some really nice data visualizations: [https://www.gapminder.org/tools/](https://www.gapminder.org/tools/)

<comment><USER9> Here's a UN chart: [https://esa.un.org/unpd/wpp/Graphs/Probabilistic/POP/TOT/](https://esa.un.org/unpd/wpp/Graphs/Probabilistic/POP/TOT/)

</comment></comment></comment><comment><USER9> Relevant Kurzgesagt video on overpopulation: [https://youtu.be/QsBT5EQt348](https://youtu.be/QsBT5EQt348)

I learnt about the overpopulation myth during my undergrad, but I'm continuously surprised by how many people (including Thanos) didn't know this.

Edit: Here's a chart of expected population growths. It's expected to start slowing down[1]

[1]: Select WORLD from the menu: [https://esa.un.org/unpd/wpp/Graphs/Probabilistic/POP/TOT/](https://esa.un.org/unpd/wpp/Graphs/Probabilistic/POP/TOT/)

<comment><USER10> I don't see why you called it myth. Just because it sort of works short term does not tell you anything about long term consequences. Earth has changed drastically with all the humans on it. You need _a lot_ of optimism about solutions for global warming and pollution. The linked short "feel good" video asks an extreme question "Will this lead to the end of our civilization?" (pretend the other side made a ridiculous claim to make them look foolish and to easily counter what nobody said), takes an incredibly _tiny_ number of data well selected to support the desired conclusion, and of course lots of colorful animations - I'm supposed to take that seriously? I find the myth that the myth is a myth to be too mythical for my taste.

<comment><USER9> I think you misunderstand, the video isn't a happy colorful animation about an optimistic conclusion to humanity's problems. It discusses _one specific problem_ : over-population. And distills a ton of data collected over the several decades to offer the conclusion.

Historical evidence supports that a decrease in mortality rates is followed by a decrease in birth rate. Here, check out the data for yourself[1]

The theory of population growth leading to resource depletion is fairly old. Thomas Malthus published this in 1798[2], but guess what: we're all still alive, despite growing over 7 billion in size, and the empirical evidence debunked this theory. There are a fairly large number of theories why this didn't happen, but that's off topic.

My point is:

\- The shortage of resource because of over-population is an old theory, but we've seen no evidence to it yet.

\- Population growth eventually slows down when the mortality rate decreases.

\- The population growth rate is expected to begin slowing[3]

\- Colorful animations help people understand a point, but that doesn't necessarily mean they're wrong.

[1]: [https://ourworldindata.org/grapher/Demographic-Transition- Mi...](https://ourworldindata.org/grapher/Demographic-Transition-Mitchell)

[2]:[https://www.intelligenteconomist.com/malthusian- theory/](https://www.intelligenteconomist.com/malthusian-theory/)

[3]:[https://esa.un.org/unpd/wpp/Graphs/Probabilistic/POP/TOT/](https://esa.un.org/unpd/wpp/Graphs/Probabilistic/POP/TOT/)

<comment><USER10> > _It discusses one specific problem: over-population_

As I said, they make exactly zero effort to say anything of substance about that issue. Key word is "over". I refer back to my comment. They say something about "population" \- but the vital "over-" part is not addressed at all. For good reason, that's a complex topic with mostly unknowns.

> _Colorful animations help people understand a point, but that doesn 't > necessarily mean they're wrong._

If there is no substance and the topic is complex and big than that alone is wrong already.

<comment><USER9> There are unknowns, yes, but that doesn't mean we can't build estimations and try to reduce the unknowns. That's how econometrics works. And that's how the UN came up with projected expectations: [https://esa.un.org/unpd/wpp/Graphs/Probabilistic/POP/TOT/](https://esa.un.org/unpd/wpp/Graphs/Probabilistic/POP/TOT/)

</comment></comment></comment></comment></comment><comment><USER11> I'm curious about one thing. Like the article alludes to there is currently an extremely strong connection between fertility and low education, low income, and high religiosity. This [1] is a list of nations by _net_ natural population growth. So births - deaths. From the top (according to UN predictions for 2015-2020): Uganda, Angola, Mali, Somalia, Tanzania, etc. From the bottom on nations that are collectively dying: Bulgaria, Latvia, Ukraine, Hungary, Lithuana, Romania, Portugal, Greece, Japan, Germany, Italy, etc.

One of the facts is that 137,000 people/day have been escaping 'extreme poverty'. But what happens when the part of society that is living in economically stable and educated households is not even reproducing to the point of replacing itself, while those living in low education and low income households are massively reproducing?

The natural response here is that if we can just improve the situation of places such as Africa then this situation ought resolve itself with birth rates starting to become comparable thus preventing a condemnation of the next generation. There are two problems with this view. The first is that this relies exclusively on a correlation which to date has proved less than predictive of African fertility levels. More importantly this effect is not just international but also intranational. Those who earn the most in the US have the fewest children with households earning less than $10,000 having a 50% greater fertility rate than those earning $200,000+. [2] It seems we're creating a society where each new birth is more and more likely to come into this world in some of the least appropriate households.

[1] - [https://en.wikipedia.org/wiki/List_of_countries_by_natural_i...](https://en.wikipedia.org/wiki/List_of_countries_by_natural_increase)

[2a] - [https://www.statista.com/statistics/241530/birth-rate-by- fam...](https://www.statista.com/statistics/241530/birth-rate-by-family- income-in-the-us/)

[2b] - [https://www.census.gov/topics/health/fertility.html](https://www.census.gov/topics/health/fertility.html) (the link to statista presents this data in a cleaner format - also adding this as just a more reliable source)

<comment><USER12> > But what happens when the part of society that is living in economically > stable and educated households > is not even reproducing to the point of > replacing itself, while those living in low education and low > income > households are massively reproducing?

Migration is what happens. Followed by integration.

<comment><USER11> Can you explain how you see this as a solution?

We can view this problem on any scale as it seems to be applicable everywhere. On a world scale the fundamental issue is that people who are of low education, low income, and high religiosity are increasingly the ones primarily repopulating our planet. This means that any given child is more and more likely to be born into this sort of circumstance. And many of these characteristics tend to pass from parent to child. It matters not what chunk of land they call home.

<comment><USER13> _low education, low income, and high religiosity_

This describes much of Europe less than a century ago. None of my grandparents had more than four years of schooling. The local priest had enough power to imprison people. People were punished for homossexual acts until quite recently (e.g. Turing). In Ireland, divorce only became legal in _1995_.

Low income and education is not a genetic characteristic, it can and is changeable.

<comment><USER11> Right, but the problem is once people do obtain a healthy income and good education, they are not reproducing even to the point of sustainability. Because of this we end up taking 1 step forward, but then 2 steps back. You bring x units of population to education, out of poverty, and so on. But then they fail to reproduce to the point of sustainability. At the same time, some number much larger than x units that remains in poverty is reproducing like there is no tomorrow.

This is a new factor in an old problem. People reproducing beyond their means of sustenance is nothing new. What is new is that those of means are now failing to reproduce.

<comment><USER14> Why not let people in prosperous regions have a smaller population. I would love a country of smaller cities and increased prosperity

<comment><USER11> These are again the tangential issues. I think you as well as the person I was initially responding to are conflating this issue with the one of decreasing population. I agree with you - I do not see population decline as a problem in and of itself.

The problem is population dynamics. Those that are more well to do and most capable of producing productive offspring are not doing so. By contrast those least able to care for and produce productive offspring are multiplying. Imagine we start with a nation that's incredibly prosperous - there are 9 wealthy families and one poor family. But each generation the wealthy families only have 1.6 children on average - 80% of what's necessary to maintain their population. By contrast the poor family has an average of 3 children - 150% of what's necessary to maintain their population.

Generation 1: 9 wealthy, 1 poor

Generation 2: 7.2 wealthy, 1.5 poor

Generation 3: 5.76 wealthy, 2.25 poor

Generation 4: 4.6 wealthy, 3.38 poor

Generation 5: 3.68 wealthy, 5.06 poor

Generation 6: 2.94 wealthy, 7.59 poor

Of course as has been mentioned some of the poor will become wealthy and some of the wealthy will become poor, but all things being equal even in very socially balanced nations the parents' income is strongly correlated with the child's. People of no means reproducing beyond their ability to sustain themselves is not a new problem, and is something people have pondered for centuries. But what is new here is that people of means are no longer producing enough to even sustain their population. When you combine these effects together, it turns poverty into a sort of virus that spreads and expands rapidly. Our utopia where 90% of people are wealthy ( _somehow.. that doesn 't even make sense if you consider the connotation of wealthy, but that's another topic_) ended up being a nation that was heavily impoverished in just 6 generations.

I also think a somewhat interesting pattern to observe in those numbers is that there was a population decline during periods of prosperity, but as the nation became more impoverished its population began to rapidly grow. The generation where the poor greatly outnumber the wealthy being the first generation to have a greater total population than we started with at generation 1.

The whole point here is that all the work in the world against poverty means nothing if those that escape poverty do not reproduce, while those that remain within it do!

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> If you want more facts like this I'd highly suggest anything but Steven Pinker especially:

[https://www.amazon.com/Better-Angels-Our-Nature- Violence/dp/...](https://www.amazon.com/Better-Angels-Our-Nature- Violence/dp/0143122010/ref=nodl_)

<comment><USER16> I am always confused buy the expression "anything but". I assume you want to suggest that people should read the Pinker's book, but I think it could be interpreted also as read everything except that book. [https://english.stackexchange.com/questions/8061/what-is- the...](https://english.stackexchange.com/questions/8061/what-is-the- difference-between-nothing-but-anything-but-and-everything-bu)

<comment><USER17> I think "anything but" was just a typo for "anything by"

<comment><USER18> For full correctness, it should be "... anything by Steven Pinker, especially..."

</comment></comment></comment></comment><comment><USER19> There's an interesting link between Facts 1 and 2 that the author misses:

> Fact #1: Since 1960, child deaths have plummeted from 20 million a year to 6 > million a year.

About half of this reduction (from 20 - 10 million) is because:

> Fact #2: Since 1960, the fertility rate has fallen by half.

Globally, the population of young people has decreased. Child mortality has decreased, too, but not by a factor of 3.

[1] [https://data.worldbank.org/indicator/SP.POP.0014.TO.ZS](https://data.worldbank.org/indicator/SP.POP.0014.TO.ZS) World Bank reports 0-14 years, rather than 0-5 used for infant mortality.

<comment><USER20> You seem to be forgetting that during this time period the world population grew from 2.5B to 7.5B. That's faster than the fertility rate decreased.

> Globally, the population of young people has decreased.

The proportion of the population which are young people has decreased but the total population of young people has increased significantly: [https://data.worldbank.org/indicator/SP.POP.0014.TO](https://data.worldbank.org/indicator/SP.POP.0014.TO)

<comment><USER21> > You seem to be forgetting that during this time period the world population > grew from 2.5B to 7.5B.

Older people were also staying alive longer - the death rate no longer offset the birth rate to the same extent as before.

</comment></comment></comment><comment><USER22> Looks good right? but all is not so great. Whats happening is a squeezing of the middle. [https://www.youtube.com/watch?v=v1oHJezqBYU](https://www.youtube.com/watch?v=v1oHJezqBYU)

Thats why this is also front page HN: [https://news.ycombinator.com/item?id=17413622](https://news.ycombinator.com/item?id=17413622)

<comment><USER14> What is inherently wrong with squeezing the middle? Seems like you'll get increased competition and innovation as people scramble to move to the top. Could be a net positive for mankind

<comment><USER22> By that logic I think you would want to squeeze the top right? I mean if you remove the middle then people will have to fill the middle before the even attempting at the top (and the top is saturated under current conditions) ;)

Beyond such myopic reasoning I think there is a great deal of economic theory around the collective economic benefits of having a strong middle class. You could try to argue against this but I think its at least a popular mainstream theory.

<comment><USER14> It is a popular mainstream theory. I'm not optimistic that this can continue. With technological innovations, I am convinced that we will end up in a can and cannot society where some people will be able to keep up and some won't.

<comment><USER22> Well if we all go around promoting the destruction of the middle class im confident will be able to create that future that you are unoptimistic about ;)

How it will turn out is kinda unknown. 100 years ago many people couldnt read and worked on farms. We past the first transition quite well, how will we fair in the next transition is indeterminate. I do know that this is the one area where what people THINK actually could have an effect on the outcome. (unlike most science)

</comment></comment></comment></comment></comment><comment><USER23> The book which Bill Gates recently gave a copy of to all 2018 college graduates - Factfulness, is a very well written and comprehensive treatment of this and more by Hans Rosling (Gapminder). Highly recommended, it's not long either - you could read it in a couple of evenings or a weekend easily.

</comment><comment><USER24> The world population has almost doubled from 1960 to 2018. I think this makes the #1 fact much stronger.

<comment><USER25> but the fertility rate has fallen by half which compensates for the doubling of the world population

<comment><USER26> There are still more kids born now than back then, so the decline in absolute child mortality shows progress.

</comment></comment></comment><comment><USER21> I don't think the graph regarding extreme poverty accounts for inflation. I don't know what an international dollar is, assuming its close, but 1.90 USD in 1850 is equivalent to about $55 today (according to Google), or 20,000 USD per year. That's not extreme poverty, I've lived on less than that.</comment>


India's Response to Covid-19 Is a Humanitarian Disaster - <AUTHOR> https://bostonreview.net/global-justice/debraj-ray-s-subramanian-indias-response-covid-19-humanitarian-disaster <comment><USER2> Indian here. This piece may sound biased, but it is true.

The lockdown was a good step. But poorly enforced. /people think it is against their prestige if they are home quarantined/. Not much of rationale could be seen even during the lockdown. People always put emotions above logic and screw up everything.

People also stepped out of homes unnecessarily because they don't understand the seriousness of the situation. But it is like every other country, only that we have much higher population density and disorganised public.

But the darker side is that there are lot of people in India who __Cannot __afford to stop working and stay at home. And this means lockdown can 't be completely obeyed at villages, where most of agricultural activity happens. Village people are harder to keep aware of social distancing as well. (No offence intended, in fact I am from a village). The agricultural activities confined to villages wouldn't have been a big issue if people reduced other methods of contact with cities / other villages. But they don't understand.

Since there is no Govt. support for COVID-19 patients, it costs some multiple times of a village person's income. That means people won't get tested even if they have slight idea about the pandemic and suspect they are infected - because they can't afford treatment even if they get tested.

COVID-19 is currently mostly limited to urban areas. I can't even imagine what happens if it spreads to villages.

<comment><USER3> It has already spread to villages. The ridiculously haphazard implementation of the lockdown led to a widespread migration of people back to their homes - which are often villages. Lucky for the government, theres no one there to challenge the official numbers and story.

In the urban centers at least it's now being questioned [https://scroll.in/article/966898/how-many-people-in-india- ha...](https://scroll.in/article/966898/how-many-people-in-india-have-had- covid-19)

</comment></comment><comment><USER4> For a country with 1 billion people, India is handling the situation better than most countries from a numbers perspective.

As of today, India has around 1.19 million cases with 28,732 deaths and more importantly, 753k recoveries.

Comparatively, the US has around 4.04 million cases with 145k deaths and only 1.2 million recoveries.

What is missing from most conversations around COVID is how to strengthen your immune system to resist the more deadly symptoms of the virus.

The Indian diet, based on Ayurvedic principles, emphasizes the use of spices and other ingredients that are naturally anti-oxidant, anti-inflammatory, and anti-bacterial. It is common in many Indian families to treat seasonal illnesses through home remedies rather than immediately resorting to medication. Doctors in India are recommending COVID-positive patients to do the same. These home remedies include eating "soonth goli" (a mix of ginger, piprimul jaggery, almonds, and ghee), drinking turmeric milk, and various forms of steam therapy, to name a few.

When almost 75% of COVID patients are recovering, clearly India must be doing something right.

<comment><USER5> >As of today, India has around 1.19 million cases with 28,732 deaths and more importantly, 753k recoveries.

Only if you believe the reported number.

India has a bad track record in tracking deaths.

Most deaths go unreported specially in rural areas. Only 20% of the deaths are reported and it was found that most of the deaths where reason was covid19 were reported as death due to other morbidities like organ transplant, diabetes etc...</comment></comment>


WearScript for Glass - JS with Batteries Included - <AUTHOR> http://wearscript.com

<comment><USER2> That looks really cool -- is there a way to get Google Glass from outside the US?

</comment><comment><USER3> This is awesome!!</comment>