Microservices – Combinatorial Explosion of Versions - <AUTHOR> https://worklifenotes.com/2020/03/04/microservices-combinatorial-explosion-of-versions/ <comment><USER3> I was at realestate.com.au when they started writing microservices in 2013 and we were all facing this exact issue. Out of that experience, the (now open source) contract testing tool called "Pact" was written (I am now one of the maintainers). I wrote a blog about the combinatorial explosion problem here many years ago now! [https://www.rea-group.com/blog/enter-the-pact-matrix-or- how-...](https://www.rea-group.com/blog/enter-the-pact-matrix-or-how-to- decouple-the-release-cycles-of-your-microservices/)

From the Pact docs: 'Contract testing is a technique for testing an integration point by checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a "contract".' By focussing just on the messages we get tests which are fast, give us quick feedback, and scale linearly instead of combinatorially.

Some good resources are:

[https://pact.io](https://pact.io) (for information about contract testing and the Pact tool itself)

[https://pactflow.io/how-pact-works/](https://pactflow.io/how-pact-works/) (explains how Pact works)

[https://docs.pact.io/faq/convinceme](https://docs.pact.io/faq/convinceme) (answers the question of why you would want to do contract testing)

[https://slack.pact.io](https://slack.pact.io) (a friendly 1000+ member community which is very experienced dealing with these kinds of issues)

[https://docs.pact.io/pact_broker/can_i_deploy](https://docs.pact.io/pact_broker/can_i_deploy) (addresses how we handle and channel this combinatorial explosion for good instead of evil!)

<comment><USER4> I did a stupid simple thing for a system with a small fanout footprint. Not sure this scales.

The organizing principal was that if my code passed the acceptance tests once, the very same code should still pass with your service/library/data added to the mix. If they don't, it's likely your code and not mine.

This might not seem like much but it cuts hours of finger pointing out of a three party system. I expect this effect would be multiplied for a five or ten party scenario.

And in particular, latency has a huge negative impact on behavioral modification. People will keep doing things that they get yelled at over three weeks later. They will stop doing things they get called on hours later.

Which I would hope offsets the additional test matrix complexity.

</comment></comment><comment><USER5> The diagram of the service in the blog post is absurd.

Anyone who designs a microservice environment like this should be fired. The interdependencies between services in the picture make the entire system fragile and everything will fail if a single service goes down.

A real production microservice environment should be designed so that interdependencies are limited and system wide failures won't occur if a service or two go down. Once you limit the interdependencies then the combinatorial explosion doesn't exist anymore. You might have some services that have a wide range of version used by clients, but you don't have the interdependency that complicates things.

Then, versioning isn't such a big deal.

<comment><USER6> You cannot build software without dependencies.

The dependency pattern around microservices is either a contract on a queue/stream/topic or a version of an API from another depened service.

Don't care how many times you tell me that services should be interdepended and bla bla bla... In the end a service has dependencies and those dependencies must then have a versioning strategy so humans are not in doubt.

Versioning a microservice is no different than versioning a linked library or a Rest API.

Documentation is key. Tell people what versions break.

<comment><AUTHOR> Thank you, I very much second this. Another way of putting this - if we need to ensure identicity between any 2 environments - we have to care about fine details and exact versions.

Otherwise, we're back at "works on my machine" mentality.

And yes - trying to make microservices independent from each other is also a form of pruning and sometimes works well to a point, but also requires good tooling to be done right.

<comment><USER5> My point is your article is based on a fake premise, or a poorly designed premise. Your architecture that creates this combinatorial explosion is only because your supposed microservice architecture is poorly designed. If you look at a real microservice architecture, it won't be nearly as complicated and it's not as big of a deal to version.

<comment><AUTHOR> > My point is your article is based on a fake premise, or a poorly designed > premise.

That's a strong statement which you should at least try to provide counter- example to support - just saying "you design it wrong" is not enough. I've seen this kind of problem irl left-right-and-center - so I take it there is an issue, and it's not happening because "everybody doing it wrong".

> If one service goes down, they all go down

I believe you don't understand the diagram. Lines are not dependencies, but a way to connect components of different versions into single product. (I.e., you pick either v1 or v2 of each component, and it becomes single product along the lines - it doesn't necessarily mean there is hard dependency).

To my point, I treat whole architecture as a product. I don't necessarily speak about dependencies. Instead, I'm taking general view of this as a math problem - first of all, I establish search space - that is number of available versions to the power of number of microservices.

Then I very much support and want to discuss various ways to reduce this search space via pruning - and what you're talking about is just one of the options how to prune it (by reducing dependencies). But as others pointed (rephrasing in my words), you can apply a greedy algorithm to NP-problem, but first of all understand what the real problem you're dealing with is and second of all realize that your algorithm is greedy - meaning it may have flaws in the edge cases and it's better to be prepared to those.

In your specific case I claim that you can't be completely sure that you actually don't have any interdependency between components - I believe it would be impossible to prove for any system. And again I saw really hard bugs irl coming from those assumptions. Recent example (this is not about microservices - but pls try to solve this case): [https://stackoverflow.com/questions/60486853/aws-ecr- uploadi...](https://stackoverflow.com/questions/60486853/aws-ecr-uploading- docker-image-give-below-error) \- Tools in question must be completely independent, but somehow they are not. So assuming something is completely independent is frequently dangerous.

</comment></comment></comment></comment></comment><comment><USER7> A lesson I learned working with microservices that are deployed independently of each other is that you need a policy that ALL changes must be backwards compatible with existing clients.

This sounds crazy, but it's essential if you can't guarantee that you can atomically deploy a new service version AND any other services that call it.

Creating backwards compatible releases means sticking to some rules, things like:

\- you can add fields but you can never delete them

\- you can add new methods but you can't remove old ones

Having really detailed logging helps a lot. If you want to deprecate a method you can remove it from calling clients first and then use the logs to confirm it isn't being called any more before removing it from the service.

It's a lot if work, and teaching a large engineering team how to be productive in this kind of environment is decidedly non trivial.

<comment><USER8> Never say never.

Adding fields or methods is easy. You can do it at any time.

Removing them is difficult. You must first go through a long deprecation cycle and ensure ALL clients are updated.

You need good logging AND good monitoring and analysis of the logs in both client and server to be able to detect access to removed features.

<comment><USER7> A coworker pointed out that this is a huge point in favour of GraphQL: because it forces clients to explicitly ask for every field you can log those queries and use them to spot when a field is no longer in use.

</comment></comment></comment><comment><USER9> The typical approach to catching this that I have seen is to deploy the new version of Service X to only a small percentage of all machines for a certain amount of time. E.g. 1% of machines get v1.1 while all others stay on v1.0. You then monitor error rates in v1.1 (and its dependencies) and stop if there are elevated levels, or crank the release up slowly if the error levels aren't elevated. Obviously some automation in your release management and monitoring tools helps a lot with this.

Of course this does not "solve" the problem of incompatibilities themselves - for that the simple solution that I've seen used is just bog standard versioning of APIs (/v1/... /v2/... /v2019-03-04/... etc) and some procedures for managing the supported versions, e.g. not doing breaking changes in an existing version, only supporting a set number of versions (i.e. previous, current, and next), and having proper sunset-periods for older versions before they are turned off so clients have time to update. Slows velocity to do all this management but that is life anyway in a complex system

</comment><comment><USER10> Don't monorepos solve this? Lets presume I'm not a FAANG company (though some of those have monorepos) with a bazillion services, I'm just a startup with say 10 services. At that scale I think its pretty realistic to maintain a monorepo for all 10 services. That way every release only upgrades the version count by one.

<comment><USER5> A startup has no business running microservices. Microservices are only when the engineering team can't scale properly because they keep stepping on top of each other.

A startup should be using a monolith until it becomes a victim of its own success and then should migrate thoughtfully to microservices only when it has to. It also requires a very heavy investment in devops and tools since microservices will fail in production in every which way possible. It also requires a huge investment in metrics and alerts and logging otherwise you will have no idea what is wrong with your now too-complicated production system.

<comment><USER6> Do you know the domain of the business? Otherwose it sounds like you should reconsider the advice you give out.

<comment><USER5> My statement applies across the board. It doesn't matter what the domain of the business is. Microservices cost more and have a higher impact on productivity and is only worth it if your engineering team needs the flexibility, at the cost of increase devops investment and more down time.

<comment><USER6> No.

<comment><USER5> Ok. Once you have real world experience, then you should continue commenting.

</comment></comment></comment></comment></comment></comment><comment><USER11> Few points based on my experiences with (Micro) services:

Note: None of these are cast in stone. These have worked well for us

1\. Decide in your team/org what constitutes a new version and what it means. Does every change means version change? What if an API adds an optional parameter; do I need to update the version?

2\. Not everything has to be a service. Think if the same functionality can be consumed in the form of a library/sdk instead of a full service.

3\. The dependency amongst services should resemble acyclic graph and not cycles. This limits the version change impact.

4\. Think about abstraction. Can we logically group set of services A, B & C and provide unified API via pass-through service D? Only D needs to handle API breakage most of the time.

</comment><comment><USER12> If you have this problem, you don't have microservices -- you have a distributed monolith.

If each service can talk to each other, the number of possible communication paths grows exponentially -- this is manageable up to a point, but even the largest teams will have to start investing into orchestration sooner or later.

There are many orchestration patterns and mechanisms, and this is not the place to list them all -- which one is the right choice will depend on the complexity and needs of the system.

<comment><AUTHOR> This is not about microservices talking to each other, but rather about seeing customer-facing product as a whole with all its different components and their versions.

And yes we can agree that there are various (usually "greedy") strategies to simplify this - but this problem is a fact of life and it makes thinking process easier to accept it as such. Same as accepting that it is impossible to solve consensus in async system, but there are algorithms that perform fairly well.

\+ I summed this up in more details in the comments down below.

</comment></comment><comment><USER13> Turns out the latest hype is a dud for most companies ^_^

There are good reasons to use microservices, like when you have 100+ engineers working on your planet-scale cloud or social network. At that scale, you need to have internal API documentation to ensure people that never met can still work together productively.

But if you are a normal company, you will probably never have enough engineers to make microservices worthwhile in the first place. So just put everything into one big git repo and call it a day.

Also, the ability to do rolling restarts and no downtime upgrades by running multiple versions in parallel adds a lot of work and complexity. But for most small to medium companies, a planned 5 minute downtime in the middle of the night is completely no problem, so all that zero-downtime-upgrade work is just wasted effort.

I mean, even my bank has a fixed offline maintenance window every night from 3:00 to 3:30 am. As does Amazon RDS.

So don't solve problems that you don't have :) Most companies do fine without microservices.

<comment><USER14> > There are good reasons to use microservices, like when you have 100+ > engineers

Its enough to have 2 that work in different domains with different constraints and different tools. Company scale is completely irrelevant here.

> But if you are a normal company, you will probably never have enough > engineers to make microservices worthwhile in the first place

Many "normal" companies use microservices and common reason is that they simply cannot function otherwise.

> But for most small to medium companies, a planned 5 minute downtime in the > middle of the night is completely no problem,

The fact that some site doesn't function for 5 minutes may not be a problem, the fact that any change requires many people agreeing on deployment time often is.

<comment><USER15> _Its enough to have 2 that work in different domains with different constraints and different tools._

Wouldn't that be more or less be just two monoliths?

<comment><USER14> If they both work for the same organization to achieve common goal, communicate through some interface they agree on, having to coordinate occasional changes that affect both codebases, than no.

If they are completely isolated, than yes, but its really rare scenario.

A common example would be a website that gathers/presents some data from users, and a service that provides analytics of said data. Both domains may require completely different technologies.

</comment></comment></comment></comment><comment><USER16> Is this a real problem people deal with? People actually build these spaghetti services and deploy several versions of them at once? Each of these versions are somehow routable to and from each other version with no abstraction?

Surely this doesn't actually happen.

</comment><comment><USER17> Consumer driven tests can help with this too. Rather than just service providers writing tests, service consumers write them too, which are then given to the service provider as part of their test suite.

</comment><comment><USER18> I don't see how this is a problem. I see that the combinatorial explosion exists, but that isn't an actual problem. The entire point of microservices is independent deploys, meaning you literally only have two permutations per deploy, new and old. If you think deploying 10 services at once is feasible, you don't understand the appral of microservices, and should never have opted for them in the first place.

<comment><AUTHOR> Deploying one at a time may be fine if you always test that each one is working in backward compatible manner. However you still have same search space and same problem as before.

Imagine

a - you have 10 microservices, 1 update per each. Each supposed to be backward-compatible. You start rolling out. One by one. 5 go fine, 6th breaks. You end up in a weird state where 5 out of 10 are updated.

You hope it's fine due to backward compatibility but you never really tested this config exactly. (Which is why I might prefer either converging to deploying all 10 or rolling back fully - if that was a known good state - <USER2> having canary cluster rather than canary microservice in many cases).

b - same as above but now you try to catch this behaviour on test / staging. You still have same hard problem at hands.

Key here is you clearly can't try every possible variation of what may break, so need to make conscious decisions about what to do in the case of failure.

</comment></comment><comment><USER19> Although it's an interesting calculation, you rarely have all your microservices communicating with eachother since you tend to group by areas. And even if that was the case, if good practices are in place (semantic versioning, good test cases, etc) I don't see why that should be a problem.

</comment><comment><USER20> in my experience, most service API grow by accretion and need to break backward compatibility very very rarely. Naturally, clients should always be Tolerant Readers.

So in many case, you will have only a single version per service.

Moreover, imo, when we are talking about internal apis, there should be nothing preventing the service owner from updating the consumers if he wishes to converge more quickly after breaking compatibility - just like he would when refactoring a monolith. The culture should allow and encourage this kind of collaboration.

</comment><comment><USER21> This is an interesting void for micro services that's not yet being met.

</comment><comment><USER22> if you: Deploy new versions of new services and quickly rollout the number of combinations get manageable.

Making he interface of services backwards compatible, either by testing and/or by using stricter contracts like grpc (which had a way to deal with compatibility)

Then you're mostly covered. However integration testing indeed makes sense

</comment><comment><USER23> Microservices only have two versions you should worry about: the one that's currently live and the next one. Two for each micro service you are responsible for. IMHO roll backs are not a thing. Instead git revert and roll forward when you need to deploy a fix. So once a service has been fully deployed, you permanently stop caring about the previous version.

Microservices should not be mass deployed together but be managed separately. That vastly simplifies the combinatorial explosion of versions you need the worry about: the currently live ones.

The rest is just applying SOLID principles to your microservices and avoid having services with poor cohesiveness or tight coupling (most of the SOLID principles boil down to affecting these two metrics). Bad service design with tight coupling where deploying service A also requires redeploying B,C,and D, is basically a design problem and not a micro services problem. There's a lot of bad design in our industry. Monoliths allow you to get away with that but it's also the reason that breaking them up is a hard problem. Just because you are getting away with it does not mean it is not a problem though.

You can further mitigate integration issues by deploying using modern practices like blue green deployments where you gradually move traffic to a new versions and adapt on things like error rates and other metrics, AB testing, etc. So if it breaks, you don't end up breaking it for everyone and you can fix the problem and try deploying that in a controlled way.

Basically your goal is to keep your customers (aka. dependees) happy and make sure you don't negatively affect them (which you should be actively monitoring). Likewise if you have dependencies (to whom you are a customer) and they stop working when you deploy something new, you probably want to detect and fix that before you break all your customers. If that is a regular thing, consider having integration tests, contract tests, etc.

Staging environments are a controversial topic in this context. My view is that they don't make much sense in a properly run micro service deployment since you are not testing in a real environment with real users, real data, and lots of things happening concurrently and features interacting. And of course your customers doing real things they care about. The only realistic environment that has that in most complex micro service deployments is called production. Once you add serverless and edge computing to the mix, these things become even more true. The bigger the organization, the less feasible it becomes to have a staging environment.

Update. I forgot to add this but doing continuous deployment means small deltas that are low risk. Any bigger change can be behind a feature flag. There are a few more strategies.

Also, I'm not actually a microservices proponent for small teams. It just creates deployment and operational overhead (read go to market bottlenecks).

<comment><USER24> Pretty much yes on all of this. I don't mind rollbacks though - there are issues with rolling back but if I can, I will. If there is a database migration, I typically will roll forward as you say.

Staging envs are generally a crutch. They can help but they can also hurt.</comment></comment>


ICO Advertising Has Been Cracked Down by Google – Anecdotal Reports - <AUTHOR> http://www.darkwebguide.net/google-cracked-ico/ <comment><USER2> Even if one's ICO complies with the SEC, Google and Facebook have become faux regulators in the space. Anyone else thinks their influence is unjustly greater than the government and something ought to be done about it, not just with regards to ICOs, but our right to communicate online</comment>


Ask HN: What are some great sites you frequent that others may not know. - <AUTHOR> I find myself visiting the same group of websites by habit, most times typing the URL unconsciously.
Examples HN, news websites, reddit.
I'm looking to expand my range and learn new things.
What are some websites you visit that others may not come across often that you find useful? <comment><USER2> I think [http://www.ribbonfarm.com/](http://www.ribbonfarm.com/) and [http://slatestarcodex.com/](http://slatestarcodex.com/) are sites that are fairly well know among this crowd, but have consistently thought provoking content. If you enjoy meta-thinking or reading about complex societal dynamics, you're in for a treat.

</comment><comment><USER3> I know it's not a secret site, but my reading consists of paid subscriptions: \- NY Times \- Wall Street Journal \- Hacker News for tech related news

A lot of people are against paying for news - they feel it is free elsewhere or they can google the title of the article.

For me I noticed that reading the NYTimes and Wall Street Journal provides me curated news with the in-depth high quality coverage I like and leaves me feeling informed.

I avoid reddit as I find myself spending a lot of time on it, similar to facebook.

Newsletters like [http://allthesmallthings.co/](http://allthesmallthings.co/) is great has also given me some info on design which I find interesting.

</comment><comment><USER4> [https://www.artstation.com/](https://www.artstation.com/)

I've discovered it a couple of month ago, and I'm so happy I did, now I visit it almost every morning. It has so many brilliant digital artists and mindblowing artworks.

Even if you're not that into "art", it's so fun to just watch people doing something with incredible level of expertise.

Just so fucking good. I can't recommend it enough.

</comment><comment><USER5> No ads and no silly comments - [http://skimfeed.com](http://skimfeed.com)

<comment><USER6> I clicked on the small "a" link at the bottom of the page and the site says I'm a spider, and to write "human" if I'm human.

Is this a way to ban site scrapers? how effective it is? (I'm assuming the site is yours - ignore me if it's not).

</comment></comment><comment><USER7> I find Atlas obscura.com to be one fascinating sites.

It's a travel guide for the obscure treated in a way that is respectful and not treated as guide for the weird and spooky.

</comment><comment><USER8> [https://longform.org/](https://longform.org/) I don't visit as frequently as I used to, but it's an excellent source of curated long form articles; brilliant Sunday morning reading.

</comment><comment><USER9> I love [http://hackaday.com/](http://hackaday.com/) and [http://makezine.com/](http://makezine.com/) and visit them often to learn about DIY projects and making things.</comment>


My Impressions of Four NYC Coding Bootcamps - <AUTHOR> https://medium.com/@adamzerner/5da7ddbfe301

<comment><USER2> So you didn't actually attend 4 bootcamps.

<comment><AUTHOR> No, these are my impressions from doing research into which one I should attend.</comment></comment>


Why Banksy Is Probably a Woman - <AUTHOR> http://www.citylab.com/design/2014/11/why-banksy-is-probably-a-woman/382202/

<comment><USER2> Why!? "Banksy's Everything but the Kitchen Sphinx in Queens was dismantled and removed by the owner of an auto-glass shop. (Shannon Stapleton/Reuters)"

Is anyone actively trying to save these for a personal collection?</comment>


The Innovation Whiteboard from NYT - <AUTHOR> http://www.nytimes.com/interactive/magazine/innovation-whiteboard.html?hp

<comment><AUTHOR> They are inviting users to submit their daily innovation ideas. I especially liked the Pop Up Beach Mat, so your stuff doesn't get blown off easily.</comment>


Ask HN: Anyone else not read the questions on SO, only the answers when stuck? - <AUTHOR> I don't know if I could pinpoint exactly when this started for me, but I realized the other day that I never read the questions when I'm trying to debug something and SO pops up as part of my Google results. I just read the title, then scroll down to the answers to see if any of the proposed code will work for me. Maybe 1 in 50 I'll flip back up to the question to see if a specific variable or setup is mentioned and therefore used in the answer.<p>I have been wondering with the discussion around SO culture and community if maybe this is important. For all the discussion about "good" questions and wanting to archive generally applicable programming advice, would it change the assumptions about the value of either if it turns out that people aren't reading the questions anyway? <comment><USER2> With experience you realise you're looking for an answer that matches your question, as opposed to a question that matches your question.

</comment><comment><USER3> Ditto. I almost never read the question. A well written answer restates the question in some form.

</comment><comment><USER4> I usually open like 8 tabs of possibly relevant answers from a google result, look for something that looks right, and if I get frustrated why something doesn't make sense or seem to work, I scroll up and read the question to find it was super specific or stupid and doesn't really match the title.

</comment><comment><USER5> I think you pointed this out yourself: everybody is in fact reading the questions, but in the SERPs, not on SO.

(And I'm sure people are scanning the body of the question there, too, if only subconsciously--I know I do.)

</comment><comment><USER6> I don't do this at all. More than half of the questions that seem relevant from Google actually aren't my situation -- no point reading the answers. Also, there are lots of good questions that list things I should have tried.

I also read the comments on the question and most of the answers to the bottom, so I just read more than most people. I have the gold badge for voting on 600 questions, which only 0.4% of users have. I'm just too curious for my own good.

<comment><USER3> That's nice & all. But you are not being very efficient :)

</comment></comment><comment><USER7> Indeed. I check out the question only if I understood the answer, but need to verify if it solves my problem.

</comment><comment><USER8> Same. I noticed after a while that I stopped upvoting questions when there was a prompt that reminded me.

Also noticed that my gold badge questions (the ones with really high views) weren't getting a lot of upvotes too. Once a question is prominent enough on Google, people just read the answer.

</comment><comment><USER9> I also noticed this in myself, but sometimes Google/the question title misleads me to an answer for another situation. I find that this habit sometimes waste my time

</comment><comment><USER10> Read the question first or the answer. A programming problem is not a narrative, it's a structure. Totally non-linear, man!

</comment><comment><USER11> Why bother reading the question if the answers are lame. It the answers you want after all. Bravo.

</comment><comment><USER12> Yes I do this.</comment>


31% of likely voters think US civil war likely within 5 years - <AUTHOR> http://www.rasmussenreports.com/public_content/politics/general_politics/june_2018/31_think_u_s_civil_war_likely_soon <comment><USER2> I can't actually see how a civil war would work mechanically. The right side of the political divide makes up almost all of the armed forces, first responders, and police; their civilians also own most of the guns; and they control the food supply.

Beyond all of that, though, my reading of history is that civil wars usually have a geographic component. So, there is usually one region (that may be home to a different ethnic group) which elevates it's political grievances into actual war. But how would that even work in the United States? This isn't like the US civil war where it was the North against the South. This would be geographically isolated urban cores versus their surrounding suburbs and rural areas.

And, finally, I don't think Americans have the stomach for casualties. Life is pretty damned good here. So good that we soured on Iraq and Afghanistan after a few thousand casualties (where we never even had to look at dead bodies, just flag-draped caskets). Somehow I don't see us being okay with the corpses of our dead children littering the streets of Los Angeles or Manhattan.

<comment><AUTHOR> To me, the interesting thing about these survey results was not that they convinced me this was actually likely to happen, but rather to contemplate that 31% of Americans (and likely voters at that) could actually believe this is likely in the next 5 years.

What does that say about the state of America? As I noted in another comment, at least somewhat it probably says people aren't very conscious of the immensity of what a US civil war means. So that speaks to civics and educational issues, which are real and relevant.

But I would just say that in a society people's feelings matter, whether rational or irrational. Irrational feelings lead to real world outcomes all the time.

So even if you think that believing this outcome is likely in the next 5 years is extremely irrational, the fact that a significant proportion of the voting population could believe this has real-world implications.

<comment><USER3> > So even if you think that believing this outcome is likely in the next 5 > years is extremely irrational, the fact that a significant proportion of the > voting population could believe this has real-world implications.

We're in a country where a lot of people believe the president when he says _three million people_ voted illegally. [1] If you believe in the existence of those imaginary people, you might wonder what will happen when they take up arms.

[1] [http://www.politifact.com/truth-o- meter/statements/2016/nov/...](http://www.politifact.com/truth-o- meter/statements/2016/nov/28/donald-trump/donald-trumps-pants-fire-claim- millions-illegal-vo/)

</comment></comment></comment><comment><USER4> The only scenario I can see of a civil war happening is if it follows the same template or series of events that occurred in 1860. i.e. Some state or states declare independence and create their own currency. I don't think that is going to happen but nationalized terrorism within seem plausible due to the way some "tribes" are advocating violence against journalists.

<comment><USER5> Sources on the last point please?

<comment><USER4> "…Right-wing provocateur Milo Yiannopoulos says he was trolling members of the media when he said he couldn't wait for "vigilante squads" to attack journalists. ..." [https://www.washingtonexaminer.com/news/milo-yiannopoulos- wa...](https://www.washingtonexaminer.com/news/milo-yiannopoulos-walks-back- original-call-for-violence-against-journalists-says-he-was-trolling)

"After meeting with North Korean dictator, Trump calls press America's 'biggest enemy'" [http://money.cnn.com/2018/06/13/media/trump-tweet- media/inde...](http://money.cnn.com/2018/06/13/media/trump-tweet- media/index.html)

</comment></comment></comment><comment><USER6> The ordering of the questions [0] strikes me as lending to a priming [1] effect, the first two questions evoke memories of media reports of violence on both sides, which makes the leap to civil war seem more likely by the third question.

[0] [http://www.rasmussenreports.com/public_content/politics/ques...](http://www.rasmussenreports.com/public_content/politics/questions/pt_survey_questions/june_2018/questions_civil_war_june_21_and_24_2018)

[1] [https://en.wikipedia.org/wiki/Priming_(psychology)](https://en.wikipedia.org/wiki/Priming_\(psychology\))

<comment><AUTHOR> I agree. There is some priming in the cadence of the questions.

It's also likely that given the state of civic knowledge and education in the country that respondents conflated something like "violence" or "minor fighting with a political tinge" to "civil war" without really understanding how immense of a thing a US civil war actually would be.

But still, to me, even the idea that a much smaller percentage of Americans (let's say 15%) could imagine a civil war happening in the next 5 years was shocking on its face, and worthy of awareness and discussion.

</comment></comment><comment><USER7> Great discussion on [https://www.reddit.com/r/AskHistorians/comments/8t4py1/would...](https://www.reddit.com/r/AskHistorians/comments/8t4py1/would_a_regular_us_citizen_in_1860_be_able_to_see/) recently about the lead up to the American civil war and how people were able to see it coming.</comment>


Country tries to censor YouTube, inadvertently launches a massive DDOS on itself. - <AUTHOR> http://blogs.zdnet.com/threatchaos/?p=548

<comment><USER2> From the article: The first repercussion was that YouTube disappeared from the Internet for almost an hour. I suspect the second repercussion was that Pakistan's Internet access crawled to a halt as all of a sudden they were handling IP requests for one of the busiest sites in the world.

Law and technology is rarely good. However, it seems that religion, law and routing tables is a particularly bad combination.

</comment><comment><USER3> Nothing says "Epic Fail" quite like DDoSing one's whole nation into submission. At least Pakistan got what it wanted: none of its citizens are going to be seeing any blasphemous internet content . . .

<comment><USER4> If I read it correctly, they also successfully screwed YouTube over for an hour... so a pyhrric victory at the least.

Lately Pakistan has been 'Epic Fail' in general...

<comment><USER3> Pyhrric victory indeed. To paraphrase Pyrrhus himself: "One more such victory will undo Pakistan (at least on the internet)."

</comment></comment></comment><comment><USER5> Can someone more knowledgeable about the internets than me please explain how someone can just hijack YouTube's IP address?

<comment><USER6> i asked the same question of a network phd student. i got the simple text version, and i have no idea if it is valid or not, but the guy i know is very smart.

he says in all likelihood, a guy in charge setup a malicious route or "report" saying they could access youtube faster than they really could. which means, everyone trying to get to youtube, went through pakistan because it was the fastest way to youtube. so all youtube traffic went to pakistan, which was the ddos part.

why isn't this a huge problem? it kinda is. the system shouldn't be able to be taken down. But, it isn't a huge problem because nobody actually does this.

Apparently, isps pay a lot for their access, and this kinda thing is a good way of paying a lot more and loosing all your money and not having internet, which is the opposite of the business they are in.

i've never ever heard of something like this happening, or about these 'reports.' so... maybe nobody should listen to me. but my friend is usually right about this stuff.

<comment><USER7> It is a problem if a hacker does this to an ISP in order to push his political / religious / whatever agenda.

</comment></comment></comment><comment><USER8> Should Google look to litigate? After all, they could probably quantify how much they lost during the downtime, and I'd imagine it's a pretty big number.

<comment><USER9> What does Youtube actually make for Google? I thought I read somewhere that it's not that much, considering what was paid for it. For 2006 it looks like it was $12.9m. If that rate is the same for now (it's probably more), then 1 day of Youtube earns $35,000. I doubt they would go after a country for that amount, though they would probably be upset about the outage and use whatever political muscle they have to pressure Pakistan to rethink.

<comment><USER5> Ha, I'm pretty sure YouTube is highly unprofitable currently, in which case Google should be paying Pakistan.

</comment></comment></comment><comment><USER10> The Internet interprets censorship as damage and... fights back.

<comment><USER4> The Internet decided to change it's name... it'd now like to be known by it's porn name... Skye Net.</comment></comment>


Researchers Warn of Physics-Based Attacks on Sensors - <AUTHOR> https://securityledger.com/2018/01/researchers-warn-physics-based-attacks-sensors/ <comment><AUTHOR> Now imagine what this type of attack would do to self driving cars...</comment>


How to Build a Scalable SDK the Right Way (Learnings from AWS SDKs) - <AUTHOR> https://dailyjag.com/technology/how-to-build-a-scalable-sdk-the-right-way-learnings-from-aws-sdks/ <comment><AUTHOR> I used to been an engineer in the PHP SDK for AWS which is open-sourced on Github and anyone can look at the code published by the AWS teams. AWS manages over 8 SDKs including SDK for JAVA, ruby, python, javascript, PHP, go, C++ and few other languages.

Generally, it requires a significant number of engineers to manage just one SDK. To make sure all APIs and features are available across all SDKs is a big feat. In this post, we are going to talk about how to scale the SDK and how AWS is able to achieve perfect synchronicity across all SDKs without sloshing the SDKs team with manually updating each SDK.

Let me know what you guys think?</comment>


Minimal ECG using an Arduino and Xoscillo - <AUTHOR> http://codinglab.blogspot.be/2013/07/my-heartbeat.html

<comment><USER2> That's an absolutely phenomenal trace... Obviously he's working under somewhat idealized noise conditions (as he described), but that trace is _cleaner_ than what we get out of the portable ECG monitors we carry in the back of ambulances (it also has much higher resolution). These are monitors have a price tag that you'd be more familiar with on a new car...

Don't get me wrong, I'm well aware that there's a big gap between some wires in a breadboard and a ruggedized, battle tested 12-lead. I'm not complaining about the cost of our Lifepacks; rather I'm ust blown away at how great that trace looks with such rudimentary equipment.

I'm gonna have to 'borrow' some electrodes from the station and give this a shot myself...

<comment><USER3> Hi, I'm glad to hear that, I'll be happy to help if you plan to build one!

As for the idealized conditions, if I gently pass my finger on my laptop's aluminum case I can feel the 50hz hum, so that is why I have to unplug it to get that signal, it may be a laptop's issue. I will check with a desktop and post my results.

<comment><USER2> Which lead was the picture from? That biphasic morphology of the T wave is a little unusual. Do you have an older ECG to compare it to?

<comment><USER3> Yes i noticed that too, I thought it could be because I put the leads where ever I felt like.

I am using 3 electrodes, the reference one (ground), was on my right thigh, the other two were, one where my right chest meets the shoulder and the other one on my left side at the level where the floating ribs are...

I placed them in non standard positions so that is why i may be getting funny values... But I agree that lower lobe is not in any ECG I have checked so far... I may go to the doctor to double check..

<comment><USER2> That's close enough to the standard Lead II. A biphasic T wave in that lead is technically abnormal, but there are a number of benign causes. Certainly wouldn't hurt to check in with your doctor though (and you get to tell him a cool story of how you discovered it)

<comment><USER3> Hey! Thanks for letting me know! I'll try to get a real ECG as soon as I can... I hope that negative lobe is just an artifact :) Thanks again, and if you need help building one please let me know I'll be happy to help!

</comment></comment></comment></comment></comment></comment><comment><USER4> I'm good enough to follow along a diagram and build stuff I find online, but have no real clue as to the electronics.

Given I have a neglected arduino kicking about and a paramedic I could probably get some electrodes from how far removed is this kit from knocking up a very basic 'brain controller'?

I've often fancied feeding some 'thoughts' into Ableton for some synth tweaking fun.

[edit] thought I should actually do some searching on the subject and I just found the openEEG project which has some info in that direction.

<comment><USER3> Yeah, another thing to try is to connect it to my head to see what I get.. I'd need to increase the gain a bit though

</comment></comment><comment><USER5> I am assuming you do the A/D conversion with the electronic board, and then treat the digital signal with the Arduino?

How difficult would it be to wire your A/D conversion unit to do USB? Then any laptop, or even a smart phone could do the UI part. Or am I missing something?

<comment><USER3> The arduino has an ADC conversor and sends it with no posprocessing at all to the PC. This could be easily used with a smart phone indeed, in fact that is my next step :)

<comment><USER6> There's prior art on that one... (using the microphone input as a ADC, sending power to the preamp via the headphone output)

[http://web.eecs.umich.edu/~prabal/projects/hijack/](http://web.eecs.umich.edu/~prabal/projects/hijack/)

</comment></comment></comment><comment><USER7> Great work! This is why I come here :)

I built a pulse oximeter as my final year project at university and have been rather obsessed with medical electronics since.

I've always wondered if it's possible to build your own ultrasound sensor. Rather complicated devices though.

</comment><comment><USER8> Is it still good practice to have opto-isolators on anything that you're wiring up to your body?

<comment><USER9> I would DEFINITELY use opto-isolators (my EEG BCI had them) and would always run the ECG on a laptop with its AC adapter _unplugged_. Do you really want to risk even a small chance of electrocution?

</comment></comment><comment><USER10> this would be a cool school science lab experiment</comment>


Coordination Problems: What It Takes to Change the World - <AUTHOR> https://fs.blog/2020/06/coordination-problems/ <comment><USER2> I think most of the desirable aspects of government can be framed in terms of such coordination problems; i.e. as a mechanism for groups to enforce collective decisions on members, where deviations benefit that individual but make everyone worse off on average.

For example, _I_ can benefit from polluting (e.g. saving money by using cheap, poorly refined fuels). My pollution costs other people, but it's only small, since I'm just one person. Hence it's rational for me to pollute.

However, I don't want _other_ people polluting, since that will cost me. Since there are lots of other people, it may even add up to more than I would gain by being the only polluter (saving money versus, say, getting cancer).

From the perspective of individuals, this is a coordination problem. The categorical imperative (what we would like everyone to do) is to not pollute; but this is unstable, since there is incentive to deviate (e.g. save money). In a competitive environment, polluters have an advantage, so the polluting strategy will dominate.

Collective government is a mechanism to solve such problems. The group decides (e.g. by vote) that freedom _from_ pollution is worth more than freedom _to_ pollute, and agrees to punish polluters, disincentivising that activity, and thus changing the payoff calculation.

Of course, collective government _itself_ introduces all sorts of new coordination problems, like corruption, revolving-doors, watching-the- watchmen, a disconnected political class, etc. (many of which are particularly evident right now!)

</comment><comment><USER3> Very surprised this article doesn't mention Schelling points, a concept which earned Thomas Schelling the Nobel Memorial Prize in Economics (among other achievements): [https://www.lesswrong.com/posts/yJfBzcDL9fBHJfZ6P/nash- equil...](https://www.lesswrong.com/posts/yJfBzcDL9fBHJfZ6P/nash-equilibria- and-schelling-points)

Actually, I'll even say that this article is _deficient_ precisely because it doesn't mention Schelling points, which are essential to understanding the kind of society-wide coordination problems discussed here.

</comment><comment><USER4> See also: [https://en.wikipedia.org/wiki/Wicked_problem](https://en.wikipedia.org/wiki/Wicked_problem)

</comment><comment><USER5> Another fantastic take on coordination issue is Scott Alexander's Meditations on Moloch [0].

0 - [https://slatestarcodex.com/2014/07/30/meditations-on- moloch/](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/)

<comment><USER6> It's a very long read, but it's worth it. This post gave me a lot to chew on.

<comment><USER7> yes, the title and introduction don't seem too compelling from the outside, but once it gets into the topic it's indeed a great discussion of _the problem_ witholding us from a better world.

so, I have to ask. maybe we haven't coordinated well enough to solve this, but can we coordinate well enough to at least get some bibliography on the topic, attempts at improving the situation...? or is people convinced this is unsolvable? is there any place, site, book or organization seriously trying to tackle this? (I'll search more later, but maybe someone can already add something useful)

<comment><USER8> There are a fair amount of people assembling in numerous loosely associated Discord servers, each of which tends to have it's own take/ideology on the overall situation, but you could probably categorize them all under the "Game-B" movement.

<comment><USER9> I'd like to join these servers. Where can I find them? Googling for "Game-B Discord Servers" doesn't return any useful results.

<comment><USER8> Some of them are listed here:

[https://theportal.wiki/wiki/The_Portal_Discord_Servers](https://theportal.wiki/wiki/The_Portal_Discord_Servers)

The CoffeeHouse is probably a good place to start as it's a bit of a main hub, and over time you will get invited to other ones.

Fair warning though, there is a significant amount of religion/spirituality, psychological woo woo, ironic overconfident naivete, etc. Not really all that different than you'll find in any community consisting of human beings, but the flavor and magnitude in these communities is likely quite distinctly different than what the HN culture is likely familiar with. They could sure use some more strictly rational people, and they definitely need folks with advanced technical skills, particularly programmers.

</comment></comment></comment></comment></comment></comment><comment><USER10> I'm guessing the reason we had all these named revolutions is because the something that inspires the name —phrygian caps, carnations, laban signs, etc— help coordinate the perception that a minority (Outer Party members?) actually believes in Moloch.

But I've never lived through one of these — anyone on HN with firsthand experience?

(Yugoslavia was, I guess, what one gets when coordination is intense but not universal)

</comment><comment><USER11> That idea of a Nash equilibrium is very interesting. I wonder if there really are multiple Nash equilibriums for many systems. At least in the prisoner example the article talked about it seemed more like there is a single optimal N.E. and it only changed once the circumstances/system changed

<comment><USER12> Equilibria concepts in game theory are many and varied. A system may have many Nash Equilibria, in mixed strategies and in pure strategies. Nash equilibria are where no one player has an incentive to deviate from their current strategy.

Game theory is a fascinating topic I highly encourage interested folks to investigate.

</comment></comment><comment><USER13> What are the best software driven coordination systems for promoting collective action? Would love less mainstream examples.

Seems like there's more opportunities than feedback mechanisms (ratings), pooled auctions (Kickstarter), and market making (Uber, eBay, Amazon).

What can we do with pre commitment?

<comment><USER7> yep, I thought the same about pre-commitment, but I guess the problem is not software itself: for proposals that don't need general consensus, but only a certain threshold of support, the problem seems to be getting enough attention. Kickstarter does well in this area, but I don't think the pre- commitment involving money is such a big deal in the collective coordination aspect. I mean, you could post a call to action on a random blog, and if it gathered enough attention you would have managed to promote collective action. That requires nothing new. It has more to do with the inability to gather the attention of enough people on a single focus when there are so many other potential focuses fighting for that same attention. In my opinion, the deadlines in Kickstarter are more interesting than the pre-commitment for coordination.

And then for proposals that require general consensus, it seems to be even more complicated. In theory democracy is about this, and we kinda accept the idea of democratic consensus, but in practice things end up having to be implemented by or pass through the hands of individual actors, and it's there where everything starts to rot. So you need to make sure the distance between consensus and its practical application is short enough too? Also, as the article says, you'd like to guarantee that "those who take a different course of action will not suffer negative consequences". But this is kinda contradictory with general consensus. And you get different groups "coordinating" in different directions. Ahhh... and the less power you have, the bigger your consensus group needs to be.

<comment><USER14> The benefit of the Kickstarter-like crowdfunding model is that it solves the coordination problem. When you bid/pledge on a threshold-funded project, your bid is _only_ paid if the project reaches its preset funding threshold-- if the threshold is not met, you pay nothing. Thus, it is _individually_ rational to pledge on a project if you expect that your pledge will meaningfully contribute to the threshold being reached and you derive individual benefit from the project itself; each contributor is essentially "matching" others' pledges. The project creator herself can then strategically choose the threshold amount and the scope of her project to improve their appeal to prospective contributors.

(There are a few emerging problems with this model, but they're generally related to having to trust that the project will be correctly fulfilled. These problems have nothing to do with the basics of collective fund-raising; they would also arise in the exact same way whenever an individual agent is contracting with a third-party for any sort of good or service. The fact that these are by far the _most_ commonly-cited issues with the threshold-based crowdfunding model is itself proof that the funding aspect works quite well indeed.)

<comment><USER7> I like your comment. I think this point you mention is very relevant for more complex cases:

> if you expect that your pledge will meaningfully contribute to the threshold > being reached

Kickstarter solves the coordination problem for _their_ case (which is great), but coordination is harder or easier depending on the scale. Kickstarter here deals with a "rather simple" coordination problem: value proposition and conditions are very clear and straightforward, risks are low, there are no opposing parties / colliding interests, and thresholds are low enough that individual contributions can be quite meaningful. So even if they have an interesting business around solving that coordination problem, and they managed to reach the critical mass for it to work (which I highlight as trickier than the actual software)... as programmers would say, the solution doesn't generalize. (Even if it might be a good place to start looking).

</comment></comment></comment></comment><comment><USER15> I wonder about the proverbial "friction force" to the dynamics of deception and misinformation and how that could fit in the model. It is certainly possible to generate personal gains through such actions, trivially just pump and dump schemes.

Verification can counter that some but in addition to the burden of fact checking there is the "want to believe" aspect of emotionally motivated thinking causing them to believe something merely because of how it makes them feel even if it is an absurdity.

<comment><USER8> Deception and misinformation can do a lot more than just generate personal gains, done properly it can also control the thoughts of the population to a very significant degree. As an example, look at how the public's _extremely passionate_ opinion (both imprecise aggregate opinion, as well as precise individual opinions) on the _absolute necessity_ of social distancing seems to have changed dramatically in the last two weeks.

> there is the "want to believe" aspect of emotionally motivated thinking > causing them to believe something merely because of how it makes them feel > even if it is an absurdity

This one is particularly interesting, because it is damn near impossible for one to 100% accurately perceive whether they are falling victim to various mental shortcomings (motivated reasoning, state dependent memory, post-hoc rationalization, etc etc etc) themselves - there are just way too many variables and neurological processes (that we have little understanding of) involved, and most processing is not (so they say) done in the conscious mind. If you take the example above, if you were to ask people who happened to hold extremely hardline positions on social distancing 2 weeks ago, their mind would instantaneously be able to produce an absolutely righteous and flawless (in their mind's evaluation) reason for the complete 180 in their stance. And from a more meta-perspective, I suspect there will most likely be some of this very thing happening as some people read these words. This is just how humanity works, for the time being at least.

Perhaps this phenomenon has something to do with the difficulty in changing the world.</comment></comment>