Why Segment Went Back to a Monolith - <AUTHOR> https://www.infoq.com/news/2020/04/microservices-back-again/ <comment><USER3> I think that the problem here was that they were fighting against Conway's Law: [https://en.wikipedia.org/wiki/Conway%27s_law](https://en.wikipedia.org/wiki/Conway%27s_law)

> Any organization that designs a system (defined broadly) will produce a > design whose structure is a copy of the organization's communication > structure.

I think microservices work well in organizations that are big enough to have a team per microservice. However if you've just split your monolith up and have the same team managing lots of microservices you've made a lot more work for the team without the organisational decoupling which are the real win of microservices.

In my experience it is really difficult to fight Conway's law, you have to work with it and arrange your business accordingly.

<comment><USER4> Noo! Building teams around software components cements your architecture and prevents most cross-cutting improvements.

I'll claim that splitting a well-structured monolith into microservices will always make it less maintanable, but it might be worth it if you need to for some reason like elasticity or failure tolerance.

But for the love of god, keep the design open. Don't tie the existence of internal software components to peoples livelihoods.

<comment><USER5> What is your alternative? Tying "the existence of internal software components to people's livelihoods" across the expanse of the entire codebase is the only remotely effective approach I've seen to scaling the SDLC at scale.

<comment><USER6> "What is your alternative?"

Aggressively small teams, with no hands-off middle-management layer.

You can build massive capability around a small number of well-managed message-backbones and a single codebase. By keeping the number of hands small and the structure flat, you force high standards. (Skilled staff won't tolerate distractions caused by bad engineering or inadequate automation.)

Heuristic for analysing firms: who has strategic power in decision-making? Conventional answer: a group of hands-off middle-managers who run on meeting tempo, and who are valued by how many people and systems report into them. Under AST: an engineering effort running on maker tempo in cooperation with a hands-on sales effort.

Microservices tend to have multilateral contracts with other systems in the organisation. This steers all planning towards meetings. This creates middle- management bloat.

<comment><USER7> Is there any example where this works (articles, presentations, etc)? In particular, anywhere with more than a couple dozen developers?

<comment><USER8> Amazon has a famous love for what they call "two-pizza teams" and you can find writeups about the philosophy by searching the term. The joke is that a team should be small enough that you only need to order two pizzas to feed them all. The philosophy is about the number of participants in the decision-making process. Keep teams small and give them total ownership of decision making so that decisions can be made by a small group of people who work with each other every day. That way no meetings (and certainly no cross-team meeting) need to happen for most decisions to be made.

<comment><USER7> Amazon is very well known for having A LOT of middle managers too, so I'm not sure it's a good example?

<comment><USER8> Seems sorta reasonable that if you need a manager for every 6-8 engineers, you would end up with a lot of managers.

<comment><USER7> OP's post was " Aggressively small teams, with no hands-off middle-management layer". 6-8 swe teams + hands off people manager reporting to middle manager, who reports to director, is how Amazon organizes teams, therefore it isn't an example of what their suggestion was...

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I see a lot of places that seem to either think that:

1\. Microservices will let them ship things faster or

2\. It's microservices everywhere or nothing

Microservices might let you ship faster if you are really good at deciding where to draw the lines between services and really good at managing multiple deployment pipelines and all the infra - that's a pretty tough ask.

Also, if you have a monolith it's perfectly fine to pull out one or two parts that need to scale much more efficiently and leave most of your codebase in the monolith, but a lot of times I see companies think once you have created one microservice the monolith is now the worst thing possible and it needs to be broken up entirely.

My general rules for this are to always start in a monolith and break things out as they start to fail or break other parts of the codebase, and don't go all in just because you now have one microservice that works well by itself

<comment><USER10> Starting with a monolith could lead to really difficult refactorings unless you structure the code in a way that it can be easily decoupled.

<comment><USER11> Monolith -> microservices : difficult refactoring

Microservices -> monolith : difficult refactoring

Microservices with poorly chosen context boundaries -> microservices with well chosen context boundaries: very difficult refactoring.

<comment><USER10> "Monolith -> microservices : difficult refactoring" I guess my point is that it doesn't have to be complicated if you architect the monolith carefully. That usually doesn't happen though because frameworks don't necessarily promote the practice and projects are short sighted.

<comment><USER11> It's also really hard. Trying to determine how to split up any code base into logical divisions such that you when adding the next 5 years of functionality you'll have the fewest number of cross division processes is hard.

This is why Martin Fowler recommends starting with a monolith and refactoring into microservices unless you have extensive experience building out very similar applications in the same domain.

</comment></comment></comment></comment></comment><comment><USER12> If you take a look at some of Segment's open source code, it isn't hard to see why they wound up struggling with microservices. It looks like they subscribe to the "left-pad" style of software development. They have tons of repositories that have less than 10 lines of code. They have a two line repository for calling _preventDefault_ [0], a four line repository for getting the url of a page[1], and a eight line repository for clearing the browser state that calls into eight different packages[2].

Disclaimer: I run a Segment competitor. I'm pretty biased, but still...

[0] [https://github.com/segmentio/prevent- default/blob/master/lib...](https://github.com/segmentio/prevent- default/blob/master/lib/index.js)

[1] [https://github.com/segmentio/canonical/blob/master/lib/index...](https://github.com/segmentio/canonical/blob/master/lib/index.js)

[2] [https://github.com/segmentio/clear- env/blob/master/lib/index...](https://github.com/segmentio/clear- env/blob/master/lib/index.js)

<comment><USER13> What does segment.io do and what does your company do?

<comment><USER12> Sure. I'm the founder of freshpaint.io.

The premise of segment.io is that there are lots of tools that take user behavior data from your site and it's a lot of work to integrate them all. For example, when a user signs up, you may tell multiple different tools that a user signed up:

      - You tell Mixpanel so you can create graphs of how many people signed up.
      - You tell Google Ads so Google knows a specific ad just resulted in a conversion.
      - You tell Optimizely so it knows a specific page from an A/B test just converted.

Before Segment, you would need to write code for each tool separately. This doesn't sound so bad, but it becomes a pain when you have dozens of different tools and dozens of different events you want to track. With Segment, you only need to tell Segment that someone logged in. Segment will then send that event to all your other tools. You can think of it as like a multiplexer for user behavior data. Instead of integrating 10 tools, you just integrate Segment.

The challenge with Segment is you need to write custom code for every action you want to send into Segment. This is bad for two reasons. Usually the end user of Mixpanel/Google Ads/Optimizely is a non-technical person that doesn't know how to write code. What they have to do is file a Jira ticket for an engineer to add a new bit of tracking to the website. Depending on the size of the organization, that person can end up waiting two weeks or more in order to start tracking a new bit of data from the website.

The other challenge is people often don't know what to track ahead of time or forget to track something important. For example, if you launched a new feature two weeks ago and forgot to setup tracking on it, there's no way to get that data back.

Freshpaint solves these problems by automatically collecting every user action upfront. Anytime someone clicks a button on your site, that fires an event in Freshpaint that someone clicked that button. You can then use Freshpaint's point and click UI to say that whenever someone clicks that button that is a "login" event. Then you can send that event into different tools. This is great because the point and click UI allows a non-technical user to send data into different tools and because we track everything up front, even if you forgot to track something, Freshpaint will still have recorded every instance of that action. That way, even if you decided you want to start tracking some action today, you can use our "time travel" functionality and recover every instance of that action since you installed Freshpaint.

<comment><USER14> This is both interesting and horrifying when I remember how much we are being tracked.

</comment></comment></comment></comment><comment><USER15> This is a discussion on pretty much every team I've been on for the last 5 or so years. I agree mostly this stuff is done for the wrong reasons.

IMHO it doesn't matter if you replace microservices by components, corba objects, rpc objects, soap services, etc. It all boils down to chopping your software into smaller bits that than immediately start having a need for sending messages between them, finding each other, defending their boundaries, etc.

So, the first mistake would be assuming this is a new problem to think about. It's not. You can find similar debates about how to chop up software ever since people moved beyond just having their code ship in punch card form.

The right discussion to have would be first deciding whether you want to break down by your logical architecture so that your deployment architecture reflects that or your organization diagram (aka. Conway's law). Then the next step is deciding whether your primary goal is network isolation of unrelated chunks of code or enabling asynchronous development of these chunks of code (if so, there are other solutions). Usually it boils down to, again, Conway's law: different teams just don't want their stuff to depend on shit happening in another team because of internal bureaucracy and hassle.

Now say you have a valid business reason or technical reason for actually wanting to have different stuff be isolated (e.g. for scaling reasons or security reasons). The next step is deciding whether this means you also want to break up your code base. Monorepos and microservices are a thing. Look at e.g. lerna for node.js, or multi module gradle projects on the jvm. In Go this is well supported as well. If you're really sure that you don't want micro services because of Conway's law there are lots of valid reasons for having a well structured mono repo with a bit of reuse of shared functionality, a simplified review process and more visibility in what is happening.

IMHO people do this for completely the wrong reasons; like wanting to try out some new language, organizational issues, etc. that ultimately result in fragmented code bases, lots of devops overhead and complexity (it's never simple or cheap), lots of project management overhead, etc. You pay a price.

</comment><comment><USER16> >Shared libraries were created to provide behavior that was similar for all workers. However, this created a new bottleneck, where changes to the shared code could require a week of developer effort, mostly due to testing constraints.

That is a big red flag. Microservices that suffer from shared code changes are not really microservices, but a distributed monolith instead.

<comment><USER17> This is really a time-of-binding argument; the difference between a "library" and a "service" is that one is in-process and accessed over function calls, and the other is out-process and accessed over RPC.

If you change code that other services are using, you can break those other services. No way round that.

<comment><USER18> While that is true, a microservices architecture can (and in my opinion should) rely on messaging and account for message schema evolution. Dependencies between services should be way less coupling than dependencies between an application and a library.

<comment><USER19> Schema evolution is just as big of a dependency hell as managing direct library dependencies. With a monolithic architecture, a lot of those concerns are contained within the context of a single repo, and can be tested much more easily than with many repos.

</comment></comment></comment></comment><comment><USER20> I'm struggling to understand the problem with shared code and the desire to fragment the code repo!

Why can't you have both independently deployed microservices and a shared code base?

If the deployment lifecycle is different for each microservices and each _deployment_ is self-contained, then they can be deployed with different versions of the code - even if they use the same source tree and share code.

Obviously the shared code needs to be properly maintained and evolved, but it seems to me a lot of the software engineering problems occur when people move away from source code dependencies - with great tooling - versioning, diffs, debuggers - to other types of dependencies ( shared libs etc ) where the tools are non-existent or very simple.

Now granted if you needed to fix a critical bug in the shared code - <USER2> would require a redeploy of everything, but that happens much less frequently than the need to deploy a single service with immunity as long as your keep your microservice contract. It also means the discipline of making sure every services is deploy-able at anytime is kept to.

And if you didn't share code - you probably wouldn't be fixing a single bug once, you'd have much more code, with many more bugs.

<comment><USER21> > Why can't you have both independently deployed microservices and a shared > code base?

This is what everyone does, so I can't even comprehend what Segment was doing. Maybe they were deploying a fleet of microservices inside a monolithic deployment? If so, there's no wonder it failed.

<comment><USER22> We do separate code repos, my last place did separate repos, place before that did monolith(s) but still did separate repos for anything not in the same monolith. I'm pretty sure it's more common to do separate repos, rather than mono-repo, for separate services.

Seems to me, though, the problem is people trying so hard to reuse code. That's the main problem cited in the article. People get really gung-ho about reusing code and creating shared libraries, but reusing code is actually bad most of the time. You should strive to only depend on things that you can reasonably expect to not change, and that you don't need to update even if a new version comes out. What you're supposed to do is take that code in the shared library, and make it a microservice, and obey the usual backward/forward compatibility rules.

Using a monolith hides that problem because the code remains easy to update and build, but just as fragile and in need of heavy testing whenever you change code modules that have multiple consumers. That goes against the idea of mono-repos as well.

<comment><USER23> > _People get really gung-ho about reusing code and creating shared libraries, > but reusing code is actually bad most of the time._

Disagree here, in general. I'm not in the ruby hyper-DRY camp, but copypasta is not the solution to dependency management problems.

Creating shared libraries does require discipline; you should do your best to just avoid breaking changes ever, and on the rare occasion you must, you need heavy communication and testing to ensure consumers find out about the change. And you can only change the API of the library; you can never incompatibly change how the library interacts with other services. I get that this is hard, but it's worthwhile if you can do it right.

We have thousands (maybe even tens of thousands) of lines of share library code at this point. Some of it is probably not necessary, but most of it we'd be completely lost without. Reimplementing core logic and utility classes and auth code over and over again is a great way to burn out your developers and create bugs. And these bugs are even worse than your garden-variety bugs, because you have to track them down and fix them over and over, and each fix is slightly different because each reimplementation is slightly different.

I agree that sometimes sharing code is a bad idea, but asserting it's bad "most of the time" is completely antithetical to my experience.

</comment></comment></comment></comment><comment><USER24> I see quite a few people defending microservices; the org is the problem, they must not have written the software correctly, etc. Most org structures are not great. Most software is not great. If you expect the exception to be the rule you're setting yourself up for a career full of disappointment.

Microservices are a modern re-branding of service-oriented architecture, but 'microservices' sounds cuter and less like it belongs in Java-land, and there's some theoretical idea that splitting your app into even smaller pieces will somehow make the whole thing better.

SOA/microservices solves a few problems and introduces a great many. The original SOA proponents were pretty explicit about this. Beware! There be dragons here! But one of the main pieces of prescriptive advice from domain driven design is helpful for splitting into distributed services: split along domain lines with minimum inter-service dependencies. Payments is an obvious one. Microservices seems to buck much of this advice in favor for a blissfully ignorant principle of "small" or "isolated". Good luck isolating something that is not meant to be isolated.

Scaling software is hard. Scaling teams is harder. Trying to scale teams by scaling/distributing software is an understandable goal but extremely hard to pull off because of additional complexities and costs you incur in doing so. Dev gets harder, deployment/ops becomes harder, testing becomes harder. Cross- team communication, documentation, API publishing and adherence goes from being very low impact within an org to suddenly being critically important.

To do SOA/microservices effectively you need complete organizational buy-in, and you have to commit completely to developing tooling and solving all the associated problems in moving to a services approach. Often, it's easier to just put it all back together, organize the code in such a way to minimize merge conflicts and wait for the ungodly slow test suite to run in CI. There are good reasons you rarely hear SOA/microservices success stories outside of enormous companies (Netflix, Facebook, Google, Amazing, etc). Doing this stuff takes an enormous investment and commitment from the entire organization, and there are just lower friction ways to skin this cat if you don't operate at mega web scale.

Growing a monolith is hard. Growing a microservices/SOA architecture is also very hard. Growing is hard.

<comment><USER20> > split along domain lines with minimum inter-service dependencies.

Exactly, and done right that quite often means big 'microservices'.

All too often I see the 'functional programming disease' where the aim is to deconstruct to the smallest possible reusable functions ( 'micro' services right? ), often prematurely, creating high levels of compositional complexity and with zero tools to help you understand how the actual 'app' \- say payment system works if it's distributed across 20 services.

Yep each single microservice is simple - but the payment system might not be and that's what you need to understand - better if your payment system is one thing - with maybe one or two things separated out if you need to scale that part.

<comment><USER25> "Yep each single microservice is simple - ..." but the whole is not.

I always find it more interesting what's _not_ in the single microservices, the stuff you do see. When you make a diagram with boxes and arrows, the interesting stuff would be the arrows, not the boxes themselves.

<comment><USER26> Indeed my loudest prescription to people doing service-oriented architectures of any kind is to simplify these arrows.

The common mistakes that I see are for two services to share read access to a common database, or to discover each other and send RPCs to each other. Both really dangerous for exactly this reason! The common database obscures how the two communicate with each other, and invariably everything connected to a database becomes one service -- call it a "mini-lith" if the overlapping sets created by databases do not cover the whole architecture. The problem is the preponderance of implicit arrows; when I reason about what it means to make this datetime nullable so that I can store such-and-so, I need to consider whether everybody who can read that datetime will be prepared for its nullability.

RPCs and APIs are the same way. I add a contract about what I am outputting and then everybody needs to know about my contracts and I must commit to them or else modify all of my consumers. So because the arrows are bi-directional everything just becomes one monolith again.

Instead, I recommend message brokers -- all that pubsub stuff. A given service tells all the other services simultaneously "this happened," and it is their responsibility in their codebase to listen for that event and then say "okay, then this must happen." Publishing a new version of the event is done by just emitting both the old and the new version of the event and perhaps having a shared standard for deprecation across the codebase so that you get deprecation warnings in your prod logs.

Every service has its own database and they generally only communicate to each other through these broadcasts, makes the arrows into the "stuff you do see".

</comment></comment></comment></comment><comment><USER27> I have this thing about micro-services/complexity in that it follows Conway's Law - the architecture follows the organisational structure.

If you push authority and decision making and responsibility for a _service_ to a (2 pizza) team then guess what, microservices work really well.

If you have vast monolithic centralised production operations teams, and no way in hell is their C-Exec going to assign two of them to look after the user-login service, you might not do so well.

Like most things, the organisation needs to change to get the best out of the opportunities software offers. Those that don't will face increasing friction and eventually die off.

<comment><USER28> Conway's Laws isn't a law, it's just an interesting thought experiment. Organization and architecture bidirectionally effect each other, but not directly, and not completely. I hate how current discourse invokes these different "Laws" as if they are physical properties of the universe. I've worked at places with a strong, hierarchical organization that created a wonderful set of "micro" services, and I've worked at places with a chaotic environment that developed monoliths.

There are shitty hierarchies and shitty flat organizations, just like there are shitty monoliths and shitty microservices.

Sorry if you actually agree with this more nuanced view, it's just that I've seen Conway's "Law" invoked more than once in this discussion and it drives me bonkers. I get the same way when someone ("Medium Developers" I call them, more than green but less than seasoned who swallow everything the read on Medium as gospel and run around quoting it zealously) quoted liskov substitution principle at me as if it was one of Newton's Laws.

<comment><USER29> Conway's Law is a physical law in the same sense as Murphy's Law.

It's also obviously true. The organization builds the architecture. The architecture either helps or hinders the organization. The organization builds a new architecture. There's no indirect connection here. If you've seen hierarchical organizations implement microservices, it's because that organization's complement was a microservices architecture. And likewise for a chaotic organization.

\--well, sidetrack: Aren't strongly hierarchical organizations the best suited for microservices? With all the strongly divided responsibilities and whatnot?

<comment><USER30> > Conway's Law is a physical law in the same sense as Murphy's Law. It's also > obviously true.

It's like a tautology: "In logic, a tautology is a formula or assertion that is true in every possible interpretation."

</comment></comment></comment></comment><comment><USER31> My takeaway from these kinds of stories is that microservices make sense if it's no longer possible to operate a monolith. By existence proof, that was clearly never the case at Segment. The common fallacy seems to be that microservices lead to better software via better architecture, regardless of human factors like team size. My sense is that it's the opposite: microservices are a necessary evil to scale teams past a certain size due to the bottlenecks that emerge with monoliths as more people begin trying to make changes simultaneously, and should be viewed as neutral at best in terms of a software architecture pattern to increase reliability, performance, etc. In practice, it seems wise to keep your engineering team as small as possible for many reasons, one large one of which is that past a certain point you will be forced to move to microservices. All other things being equal, that's a move you don't want to ever have to make.

If you have hundreds of engineers then certainly microservice architecture starts to make sense, since even the idea of transactional deploys of the monolith break down due to queuing at that scale. But jeeze, don't pull that trigger until you actually find yourself backing up on necessary complexity like deploy queues, PRs stuck due to inability to maintain the branch given the velocity of master, etc. Don't let Conways law lead you prematurely to microservices. If I'm ever in a position where I am feeling real pain that leads to an urgency for microservices, I am probably going to first ask the question if I can just fire some people to make the problem go away. The risk of the transition to microservices is just that high.

It's the same rule of thumb with other things like hiring, feature roadmaps, etc: YAGNI. If you are hiring someone before the pain is so high the work cannot be done otherwise, building features before you have people explicitly showing the need for them, or making deep, cross cutting architectural changes that impact everyone before they are strictly necessary due to concrete problems with shipping software, you're probably choosing the wrong use of opportunity cost, capital, etc.

</comment><comment><USER32> This sounds like the old arguments about OOP.

Turning everything into an object can make a small program into a big program, so it's maybe not such a good idea for small-scale stuff.

[http://www.solipsys.co.uk/new/TheParableOfTheToaster.html](http://www.solipsys.co.uk/new/TheParableOfTheToaster.html)

However, in my experience, OOP made it possible to do really big stuff.

It's all about not having a "one-size-fits-all" approach. I don't think it's just about scaling architectures; it's about changing architectures to match scale.

It's difficult as hell to make these changes, because people get invested in methodology, and insist on applying the same lens to everything we do.

It sounds like they had the right idea, but they probably had the wrong people.

<comment><USER33> "Turning everything into an object can make a small program into a big program, so it's maybe not such a good idea for small-scale stuff."

In my experience OOP actually makes programs smaller. Assuming of course they have good programmers/architects and the program itself is larger than "Hello world".

<comment><USER32> Don't get me wrong. I love OOP, and have been using it since before it was cool. It's been a standard wrench in my toolbox for decades.

In fact, I have been running into folks, these days, that don't understand it, as, apparently, OOP is becoming "uncool."

I've always been a "right tool for the right job" kind of guy. I started off with ML (Machine _Language_ , not Machine _Learning_ ). I am quite comfortable, sitting down with a breadboard, and flashing an OS.

But I remember the old days of OOP, where "classic" structured programmers didn't "get" OOP, and designed these horrific chimeras.

I always make it a point to understand my methodology and drivers "to the bone." Just because someone at a conference said it, doesn't mean that I should use it for everything.

<comment><USER34> Please write a blog post called "Horrific OOP chimeras" and post a link on HN ...

<comment><USER32> Oh...the stories I could tell...

But I have made it a point of personal ethos not to post criticism or polemics, denigrating/excoriating the work of others.

I know that could buy me a lot of clicks (and probably some considerable HN Above The Fold time), but I think we have enough negativity and finger- pointing on the Internet.

If you read my stuff, you won't see much of that. I may, in a rather vague way, allude to something that gives me a frowny-face, but I don't want that to be part of my "personal brand," so to speak.

I do take tremendous personal pride in my work; both coding and writing, and hold myself to a high bar. I may even project that bar onto others (only in some circumstances), but I don't think it's helpful to do so in public.

I find it most gratifying to write a "This is how _I_ do this..." post, as opposed to a "This isn't how _you_ should do it..." post.

</comment></comment></comment></comment></comment><comment><USER35> I have always viewed using many micro services as something that adds complexity, something to be used when necessary.

I started working remotely as a consultant in the early 2000s when my wife and I moved to a remote area. I had several development jobs that used the same monolith pattern: I would embed everything in a web app using Apache Tomcat, taking advantage of work threads for background tasks. The only external services were a database and crontab settings to frequently snapshot databases. This pattern was so easy to code to, so easy to debug and deal with any runtime problems. One customer reported that a system ran without stopping for six years (ouch, no OS upgrades??) until they restarted it on a larger server.

Micro services can be great, but not always the best choice when horizontal scaling is not required.

</comment><comment><USER36> Are there any case studies where microservices went well?

From an end user perspective, Netflix runs in "constantly degraded" mod.

From an engineering perspective, they track "number of successful stream starts", instead of percentage of the time 100% of their services are working. That's a huge red flag.

As a researcher, the monitoring and fault-propagation / modeling work they've done to get it to stay up at all is impressive, but it's not clear all of that tooling would be necessary if they didn't have to reason about N^2 fault tolerance scenarios, where N = 100's of microservices. That's on the order of one fault tolerance scenario for each atom in the universe.

<comment><USER37> Amazon does micro services (or SOA) extremely well. In fact they practically invented the concept. It's intricately linked with the 2 pizza team and service ownership concept (you build it, you support it)

<comment><USER38> and the concept of well defined API's for each service.

</comment></comment></comment><comment><USER39> It seems like they introduce microservices for the wrong reason. Instead of having a service per team, the focussed on services to solve a technical problem:

"Having a code repository for each service was manageable for a handful of destination workers"

Microservices should be introduced to make teams go faster, not to decouple external api endpoints....

<comment><USER40> I mean you are right of course, but at the same time I can't knock the superficial idea of having one codebase for one domain-specific application. Applications / codebases like that are usually not the problem; it's integrating them into the larger whole where things start getting fucky.

</comment></comment><comment><USER41> I always felt like the biggest benefit of microservices (for the average company that just jumped on the band wagon) was simply the fact it forced them to break things up. Yes, they could achieve the same result with none of the overhead on a monolith, but it would take... discipline. It's much easier to just enforce a hard external constraint.

Realizing this and circling back is still a useful life lesson.

<comment><USER42> I think this, approaching DDD, is the most common reason engineers push for it these days.

<comment><USER43> It isn't worth the added friction though.

And if you happen to leak concerns in your services (in a monolith), it's really easy to adjust, as opposed to having to coordinate the deployment of 5+ services.

And even then, a distributed monolith is still a risk.

Micro-services add cement to your project. Be prepared to keep boundaries you write for a long time.

</comment></comment></comment><comment><USER44> Without knowing more about their architecture it is difficult to comment beyond the conclusion Alexandra Noonan came to, stated at the beginning of the article. It looks like to me that the architectural assumptions were changing too quickly due to the demands of a fast growing business. Having all their code in a single repository means that they can control dependencies, versioning and deployment centrally, it gives them central control of their software development lifecycle. I can't see how they could not have had the same benefits of the monolith if their microservices existed in a single repo to begin with and had the appropriate tooling to enforce testing, versioning, deployment across all services in the repo. I guess this is the whole monorepo debate and tooling.

This article for me is more about the complexity of managing a large team across different sites where the architecture needs to change rapidly when modularity is absent. They did get a measurable benefit around performance, though. I wonder if Alexandra will comment on the challenges of running a team in an environment of this complexity?

<comment><USER17> If the microservices are in a single repo and tested and deployed together then they are arguably no longer microservices but a "distributed monolith"!

<comment><USER44> I'm referring to having the same testing, deployment,packaging,versioning policies etc being consistently applied across projects within the same repository not deploying, testing and releasing together.

It's the drift and inconsistencies across these concerns across projects that makes deployment and operations less predictable.

</comment></comment></comment><comment><USER45> My experience of moving to a microservice architecture: the most important consideration with microservices is "who will develop, maintain and operate them?". You can split them down functional lines, architectural lines, whatever you like, but if you don't have teams with definite ownership of each microservice (and that aren't swamped by maintaining lots of them like it seems happened with Segment), it will become impossible. The "operational complexity tax" is a real thing but is manageable if your engineer:service ratio is sensible and considered.

</comment><comment><USER46> I see it like organization of large companies: you have to split it into divisions, but you can't make every single person their own division.

Don't make the divisions too large, don't make them too small.

The art is to make them the proper size for the particular company.

If you have a small company you don't need divisions. If you have a large one, you need to make divisions as you no longer can speak to every single employee.

</comment><comment><USER47> Microservice-Architecture is one of these trends where the value is unproven, the upfront costs are high and the unknowns are unknown.

There's also a clear conflict of interest with SAAS and Cloud providers benefiting from the perception that microservices are the way to go.

Under these circumstances, letting _someone else_ figure out all the issues is the wise thing to do. Thanks to the authors for doing just that.

</comment><comment><USER48> They went to 50+ services in a few months, were applying the same policies across all services. It sounds like they didn't plan well enough and jumped straight into it, without any good DevOps or infrastructure mindset. It was a disaster waiting to happen. This shouldn't be an article that people read and say "Oh Im never using Microservices". This should be an article people read and say WOW that is exactly the right way to NOT break apart a monolith.

</comment><comment><USER49> What's notably absent are descriptions of problems with versioning interfaces, failures from network unreliability, or problems managing connecting infrastructure, or poor delineation of service boundaries leading to undesired change dependencies--which is to say it seems like they executed well. There's no sign they fell into common pitfalls.

There are definitely some good insights here that I don't often read about. The idea that with a sufficient number of microservices (say 50+) you not only treat your instances as-cattle-not-pets you have to treat the service types en-masse as-cattle-not-pets. This requires more automation and organized management as pointed out by the need for tuned autoscaling rules. This requires continued investment into automating things you would do manually if you had 50- services.

The other thing to consider is that going to microservices and back to a monolith is not necessarily a failure. Microservices are good for periods of high change velocity, once a platform is mostly built requiring much less new development consolidation completely makes sense. At all points, we're solving for impedance mismatch, whether that's the org structure, velocity of changes, or numbers of developers vs numbers of deployed units.

</comment><comment><USER50> How many places have gone from monolith to microservices and back to monolith? I'm sure there's been quite a few.

<comment><USER51> I bet plenty of them.

A team that fails to understand how to write modular code, is just going to write spaghetti RPC calls, while having to deal with all the traditional failures and performance issues of distributed computing.

Naturally it is a recipe doomed to fail in the large majority of cases, but it doesn't matter because whoever drove the change is no longer at the company and a new consulting team/new hire gets the money to drive everything back to the monolith.

So goes the money around on plenty of consulting gigs.

<comment><USER52> > A team that fails to understand how to write modular code, is just going to > write spaghetti RPC calls

This is interesting. I always assumed we were talking about good developers here.

I wonder what's a more likely cause for a failed attempt at microservices. Is it developer incompetence and lack of discipline, or is it environmental factors related to the product and the organization?

<comment><USER17> For almost all works produced by more than one developer the "good developer / bad developer" dichotomy is just useless social darwinism. Talking about the team, organisation, incentives, or business is far more useful.

(My favourite example is John Romero, part of the very small team that produced Doom - but who also produced Daikatana, which keeps showing up on lists of notoriously bad games.)

<comment><USER53> That was pure hubris, foreshadowing GamerGate (and inventing the self-Pwn)! As you say, it's all about the team, not the technique. And talking about that particular team:

[https://en.wikipedia.org/wiki/Daikatana](https://en.wikipedia.org/wiki/Daikatana)

>One advert for the game became notorious; a 1997 poster containing the phrase "John Romero's About To Make You His Bitch[. Suck It Down.]". According to Mike Wilson, the advert was created by the same artist who designed the game's box art under order of their chosen advertising agency. Originally, both he and Romero thought it was funny and approved it. Romero had second thoughts soon after but was persuaded by Wilson to let it pass. Speaking ten years later, Romero said while wary of the slogan at the time, he went along with it as he had a reputation for similar crass phrases. In the same interview, he noted that reactions to the poster tarnished the game's image long before release, and continued to impact his public image and career. In a 2008 blog post concerning the recent activities of Wilson, Romero attributed him for the marketing tactic. This prompted a hostile exchange of public messages between the two at the time.

At least he apologized, though:

[https://www.youtube.com/watch?v=BF_sahvR4mw](https://www.youtube.com/watch?v=BF_sahvR4mw)

John Romero Apologizes for Trying to Make You His Bitch:

[https://v1.escapistmagazine.com/news/view/100748-John- Romero...](https://v1.escapistmagazine.com/news/view/100748-John-Romero- Apologizes-for-Trying-to-Make-You-His-Bitch)

>I'm going to quote our very own Shamus Young here for a moment: For almost a decade, Ion Storm's Daikatana has been the example of "industry waste, arrogance, and incompetence, as well as a universal punchline for things that suck." The shooter was supposed to be an epic vision, the masterpiece of John Romero - the mastermind behind genre-defining Doom and Quake.

>Then it came out in May of 2000, and it sucked. The arrogance and hubris that crippled Daikatana have been well chronicled over the years, but none of it is quite as infamous as the ad you see here to the right: "John Romero's About to Make You His Bitch. Suck it Down." It was a pretty ballsy statement in itself, but after the game's failure simply became laughable.

John Romero Is So Sorry About Trying To Make You His Bitch:

[https://kotaku.com/john-romero-is-so-sorry-about-trying- to-m...](https://kotaku.com/john-romero-is-so-sorry-about-trying-to-make-you- his-bi-5541406)

>Game designer John Romero and John Romero's hair ruled the roost during the 1990s. With titles like Doom and Quake, he not only helped popularize the first-person shooter, he defined it. Then the unthinkable happened. He made Daikatana.

>[...] Romero, who now says he is resigned to the ad, dished on the ad back in 2008, which evoked a saucy response from the marketer that spearheaded the suck-it-down campaign.

Romero Dishes on the Ad:

[https://web.archive.org/web/20081225071219/http://kotaku.com...](https://web.archive.org/web/20081225071219/http://kotaku.com/345386/john- romero-dishes-on-bitch-ad)

>[...] these are the kinds of jackass stunts he pulled [...]

Suck-It-Down Campaign Marketer's Saucy Response:

[https://web.archive.org/web/20081225070532/http://kotaku.com...](https://web.archive.org/web/20081225070532/http://kotaku.com/346816/gamecock- head-tears-into-john-romero-its-getting-ugly)

>[...] and ill advised breast implants strewn across this fair nation [...]

</comment></comment></comment></comment></comment><comment><AUTHOR> As it is so often the case, the choice between monolith and microservices is not a binary one; rather, it is a sliding scale between two extremes.

On one end we have a real monolith: a single executable binary, with no external dependencies apart from the OS bindings. This is very rare in practice, and most commonly found in games and probably mobile apps; when it comes to Web-based services, even a traditional idea of single-codebase app usually has a SQL database as an external dependency.

On the other end of the scale there is a complex system consisting of hundreds or even thousands [0] of tiny services that require complex orchestration mechanisms such as a service hub or service mesh.

So each team (in a wide sense: could be a company, organisation, department etc) needs to consider where they fall in the continuum, considering a) which architecture will provide most benefit while b) still being maintainable by the team; both the architecture and the team need to evolve together.

[0] [https://qconlondon.com/london2020/presentation/monzo](https://qconlondon.com/london2020/presentation/monzo)

</comment><comment><USER54> Previous discussion from her blog post on the same topic: [https://news.ycombinator.com/item?id=17499137](https://news.ycombinator.com/item?id=17499137)

</comment><comment><USER55> I'm just curious if there is a middle ground somewhere?

On one end you have a giant monolith. All services rolled into one which includes your API, Middle ware and then Database.

One the other end you have microservices which bundle services into individual distinct units with each service being responsible for its API, middleware and database.

Are there any preexisting patterns which seek to combine these two and come up with an architecture which is midway between those two. A few months ago I had read an article about Data Oriented Architecture on HN which comes close though I'm wondering if there are others.

<comment><USER56> > which includes your API, Middle ware and then Database

Layered microservices are an antipattern. In most cases, functionality is best divided by domain.

<comment><USER57> This is why I struggle with microservice architectures. It seems like there is a basic contradiction. On the one hand, it's vitally important that the microservices are carved into the correct modules otherwise you get a nightmare of operational complexity where simple functional changes require coordinated changes across multiple services. But defining the correct way modules requires a bird's-eye architectural view of the entire system, which seems contradictory to idea of self-organizing, independent teams. I can see how it works when the right way to divide things up is obvious or when you are dealing with IaaS or PaaS services, but in a complex business domain who decides how to carve things up?

</comment></comment></comment><comment><USER58> Still waiting for Monzo's following blogpost on cutting down their outrageous number of 1500 microservices [0] and moving some back into monoliths. I'm not sure if I would be too excited over the number of microservices if there is a degree of complexity involved here. That is just too many here.

[0] [https://monzo.com/blog/we-built-network-isolation- for-1-500-...](https://monzo.com/blog/we-built-network-isolation- for-1-500-services)

<comment><USER59> Why do you think/feel that there are "too many"? What's the threshold for an acceptable number of microservices? (Not asking this to be confrontational. Just curious, because it's a sentiment I've seen before, without the reasoning behind it being articulated.)

<comment><USER60> One per developer seems like a fairly loose upper bound.

<comment><USER61> Even then this is risky - if that developer is hit by a bus do you throw the service away and have another developer write it again?

We recently had an interview candidate say this when we questioned the wisdom of having over a thousand microservices: some in languages that only the one developer maintaining them used! For me this is insane, but I digress

Monzo says that they have 800 people, and 1500 services. If we're generous and say 500/800 are developers, then each developer is responsible for 3 services! A team of 6 would have 18 projects in their domain.

<comment><USER62> There is a classic tradeoff here between top down organisation dictates giving consistency vs engineering independence giving flexibility.

Two organisations that I know of who favour the latter are Spotify and Netflix. It has benefits - different languages are good for different jobs and engineers like to be able to choose their tools.

It would be bad if this was taken too far, and something was written in a language only one person knows, but that problem already exists with the technical knowledge if something only has one mantainer.

</comment></comment></comment></comment></comment><comment><USER63> "voices of experience pointing out that most decisions are made based on the best information available at the time."

funny in my experience with digital and particularly larger corporate jabronis. The people who makes decisions are fucking McKinsey consultants who know nothing about the actual project, are only contracted for 6 months, then they are gone. Rinse and repeat, maybe one out of every 3 or 4 attempts somebody actually gets it right and the project doesnt completely fail.

</comment><comment><USER64> > Also, a proper solution for true fault isolation would have been one > microservice per queue per customer, but that would have required over > 10,000 microservices.

I'm a bit confused, they seem to imply that they need a microservice per costumer/destination, but you generically have one instance (aka process) per costumer not an entire separate codebase. The article seem to use the same term for two different concepts. Or i am missing something?

<comment><USER65> Sounds like a job for Erlang

</comment></comment><comment><USER66> there is a related post on Segment's blog : [https://segment.com/blog/goodbye- microservices/](https://segment.com/blog/goodbye-microservices/) which helps to get a bit more details and contexts

update : and related video [https://www.youtube.com/watch?v=lv5o3qnQu5w](https://www.youtube.com/watch?v=lv5o3qnQu5w)

</comment><comment><USER67> That final paragraph is pretty brutal. Are engineers really so reliably obnoxious?

<comment><USER68> In general, yes.

Everyone seems to have their preferred style of coding, and it is an easy defence mechanism, when presented with anyone who tries and finds it wanting, to say that "Well they didn't do it properly".

You find that with Microservices vs Monolith, Strong types vs Weak types, Exception Handling vs Results, Agile vs Waterfall.

People fragment into camps which turn into echo chambers and it's easy to dismiss anyone who doesn't commit to that cult as being unpure and not worthy of being in the cult anyway.

</comment></comment><comment><USER69> The takeaway is about trade-offs. They made a rational decision to improve fault isolation by dividing the app into smaller building blocks managed separately. After working with it for a while, they realized the higher operational overhead made the architecture a bad choice for them. So they went back to a monolith architecture and tried to do fault isolation within the boundaries of that architecture, which might have made fault isolation not as good as in the micro-services architecture but it was acceptable.

It's incredibly tough to know the full effect of a trade-off on your organization until you start going down that path.

We're early in the process of adopting a micro-service architecture. With only a handful of services so far, I can already see how a team of two is going to spend a lot more time with operational issues and debugging.

</comment><comment><USER70> It seems like a lot of the issues were around sharing code and libraries, resulting from isolated codebases per service and the versioning hell of shared libraries.

I work in a org that migrated to microservices over time, but intentionally adopted a monorepo approach as part of it. It works quite well and seems to avoid a lot of the pain points expressed here, while also gaining the benefits of microservices.

There are definitely tradeoffs to the monorepo approach. It makes development on shared libraries more delicate and stressful, however this can be mitigated by more robust cross-service CI, and I definitely think it's a worthwhile tradeoff to the painful cycle of shared lib versions diverging across services and finding issues when some service finally gets around to upgrading its version weeks/months after shared lib changes.

</comment><comment><USER71> Problem with micro-services is that it comes with a huge amount "that's the RIGHT way to do it" and infinite articles talking about what they are and developers fighting over if your services are too big or too small.

That usually results in abandoning the effort to actually map the use case of your particular application, model your services to the size that make sense to your project... Any big enough system will need some services or workers beyond a single monolith, it doesn't matter if they say they follow micro- services, if they follow any other type of SOA or whatever, these silver bullets are killing engineering. Every project needs to take time to be planned, thought it, refactored, analyzed. If you read a bunch of shit on HN and go applying you end up with a random monster.

</comment><comment><USER72> Others have covered most of my thoughts but I haven't seen platform mentioned.

If you already have a platform like Kubernetes/OpenShift (preferably with a service mesh) micro services make a lot more sense to me and can be done well. It gets easy to deploy and scale independently, but still have very low latency communication with a strong security model built in.

If you are deploying everything to completely independent platforms/infrastructure, I get a lot more conservative with "what should be its own service" and what shouldn't. Building a distributed monolith (a bunch of dependent services that aren't reusable/composable) is the worst of all worlds.

</comment><comment><USER73> > "If microservices are implemented incorrectly or used as a band-aid without > addressing some of the root flaws in your system, you'll be unable to do new > product development because you're drowning in the complexity."

Which is nearly verbatim what some of us have been telling you since before the term microservices was coined.

Coupling is the problem. Yes, microservices add friction to coupling, but they don't prevent it. Coupled microservices exist (boy howdy), and they're resource intensive, resistant to evolution, or both.

</comment><comment><USER74> This older "breaking up the monolith" GraphQL talk from Prisma is interesting: [https://invidio.us/watch?v=_MmyTahR9ok](https://invidio.us/watch?v=_MmyTahR9ok)

Especially if you consider RedwoodJS, a new full stack JS framework that's build on Prisma technology (their stuff is an alternative to Rails Active Record ORM). My takeaway is that they provide a similar monolith like experience by acting as a glue between different services.

</comment><comment><USER75> > One of the key takeaways was that spending a few days or weeks to do more > analysis could avoid a situation that takes years to correct.

Exactly my point when I was working on a new project architecture and we had to choose between two authentication methods. Because once all your applications rely on an authentication system you can't just switch to the other one like that... Sadly we did not take the time six month ago and today we are working on a migration which could have been avoided.

<comment><USER76> Why don't you share your insights?

</comment></comment><comment><USER77> I wonder why they are so easily able to work with and integrate external services, but they couldn't work with and integrate with their own internal services. I think it has to do with the fact that the boundaries to the external services are well defined and enforced because there is a physical aspect to it. Perhaps they lack the will to create and enforce those boundaries between their internal services.

</comment><comment> d--b It doesn't seem like their end architecture is anywhere near what they had at the beginning... It's a lot smarter, and it sounds like it's a monolithic distribution system that manages hot-swappable services. So the whole thing seems to fall in the "micro services where we need them" kind of architecture.

They just went from naive monolith, to naive micro services, then to smart coupling of the two...

</comment><comment><USER78> Poor design. Sharing code between microsercices is always a design smell.

You are just building services on top of another monolith...

Sounds like you needed to abstract the work being send to the worker, instead of abstracting the worker around the work.

Meaning don't have many workers for one payload type. Abstract the payload and have a single worker...

That's why most systems become complex and spaghetti. Poor abstraction, so you use shared code to fix it...

</comment><comment><USER79> My org is starting to migrate from a PHP Monolith to Microservices as a way of freeing ourselves from PHP.

Microservices will require more time spent writing interservice APIs, and code execution will be slower since many procedure calls will require data serialization and network requests. But we believe it will be worth the overhead to not be locked into PHP for every new component of the project.

</comment><comment><USER80> Might be a stupid question, it wasn't clear to me in the article.

Did they go back to a monolith service or a monolith repo? It really just sounded like monorepo

</comment><comment><USER81> Microservice is not an organizational problem. Microservices is a design problem. If you implement DDD (Domain Driven Design) first to your application and then start to design the application around your DDD concept, then, it might work.

But, it's extremely hard even to do that. Microservices simply complicates things if any of your domains need to share code with each other. Many DDD paradigms exist to address this, but none are practical. For example, authentication related code. IF one domain sets a cookie and the other one has to rely on that to keep the user (a shared model between the two domains) authenticated, then this means, you need to duplicate code bases across two domains or in the very least put them into some sort of shared helper/library, which DDD is kind of against.

That's why it totally makes sense to go Monolith first and really identify the parts of your application that are slowing you down either development wise, testing wise or performance wise and put them into separate contexts.

Phoenix actually does Microservices right. From all the way to scaffold generation to instructing best practices on keeping your domains properly separated. But even then, I've burnt my finger many a times trying to write simple CMS solutions into mutliple microservices then going back to monoliths again.

</comment><comment><USER82> I think that microservice approach is good when you can share those services between many applications In most cases microservices can be the new premature optimization because you can start to lose the focus on the product itself and think about the the microservice as a product

</comment><comment><USER83> If you're up for a humoristic take on microservices, you could have a look at this video: [https://www.youtube.com/watch?v=y8OnoxKotPQ](https://www.youtube.com/watch?v=y8OnoxKotPQ)

</comment><comment><USER84> Whatever monothlic or microservice, it's all about modularity. Monothlic or microservice is just the implementation on how you achive modularity.

So the problem here, to me, is how you design the modularity for your system, not how you implement it.

</comment><comment><USER85> > There is now a single code repository, and all destination workers use the > same version of the shared library.

What if I told you this is unrelated to microservices? You can keep all of their sources in the same repository, sharing dependencies.

</comment><comment><USER86> Another advantage of monoliths is speed. Running on a Local L1-L3 cache will be orders of a magnitude faster than serializing network round trip and deserializing json in micro service.

Performance per watt/dollar of computing.

</comment><comment><USER87> I don't blame them, microservices requires discipline, careful planning, in depth monitoring and debugging capabilities and whole different mindset. Very few companies can implement it successfully.

</comment><comment><USER88> "Monolith vs Micro-services?" is philosophy at this point. It's a good brain exercise, a great starter for debate, it reveals a lot of insights... all because there is no actual answer.

</comment><comment><USER89> That may sound cynical but if we agree that software development is a show (bet, entertainment, whatever) for investors in 99%, then microservices are good to keep people excited.

</comment><comment><USER90> Is it about process space and programming languages? Or is it about source control and CI architecture?

How is a monolith different from a monorepo?

</comment><comment><USER91> Before going full microservices, try writing your code in a hyper modularized way and see if this doesn't solve your problem first.

</comment><comment><USER92> Having read their articles a few times, the issues that they were attributing to microservice architecture were really CI problems.

</comment><comment><USER30> I think we can have both architectures Monolith and Microservices peacefully co-existing, with their own pros and cons.

</comment><comment><USER93> Monolith vs microservice feels like a higher level case of the expression problem.

</comment><comment><USER90> Note that one big database can decimate productivity as well.

</comment><comment><USER94> I saw a talk with Alexandra from Segment before. When it makes sense to go with monolith it makes sense.

When it makes sense to use microservices, it makes sense.

Doing anything for no reason whatsoever never makes sense.

That is all.

</comment><comment><USER95> I have an idea. Bring the concept of microservices into software!

Within software module interfaces that can only communicate with one another via socket like serial interfaces with no type checking!

Or simply have all your software modules running as forked processes on the same hardware and have them all communicate with one another via sockets or http. That means every software module must be it's own server!

To further imitate microservices, make sure that code in one software module can never ever be moved to another software module. Make it hard to reorganize things. Also make sure teams can only ever work on one section of the code base.

Does the above make any sense to you? If it doesn't make any sense to you it's probably because code organization using microservices doesn't make any sense period because the examples above are literally doing the same thing in software that is done in hardware.

If it does make sense to you, then why are you using microservices to add extra complexity to the code? If you can do the same in software then you'd be doing the exact same thing as the hardware equivalent minus the extra complexity of multiple containers or VMs.

Don't use hardware to organize code, use code to organize code and use hardware to maximize performance.

</comment><comment><USER95> Microservices in terms of code organization was always a redundant concept.

You can organize code with functions and namespaces, why do you need hardware to segregate code? It only makes the segregation permanent but offers nothing else beneficial in terms of code organization.

The underlying reasoning was always that developers tend to move outside of boxed software modules if it wasn't enforced by hardware so the modules will end up not being modules but everything will be blurred into monoliths.

I always figured that if you want really hard lines drawn between software modules you can still do the same stuff in software itself, like why do you need actual silicon or VMs/Containers to do it?

The only real need for different services is performance, otherwise all the benefits and downsides of microservices can be replicated in software.

</comment><comment><USER95> I bombed an interview before because I said microservices can be really bad.

<comment><USER53> Maybe it's because you weren't able to explain why you thought microservices can be really bad, or your explanation didn't hold water. So what was the explanation you gave them, or do you not want to say?

Or perhaps there are other reasons you're simply not recognizing? Like your sarcastic tone?

</comment></comment><comment><USER96> TL;DR: "If microservices are implemented incorrectly ... you're drowning in the complexity."

<comment><USER97> I want to see the microservice architecture where you don't drown in complexity.</comment></comment>


Learning new skills in InfoSec without getting overwhelmed - <AUTHOR> https://azeria-labs.com/paradox-of-choice/ <comment><USER2> Security is about doubting what you're told, productively. Anyone who tells you otherwise should be doubted-- productively.

If you build security tools, verify that they actually work. If you write exploits, doubt every correctness claim until proven. If you make policy, assume ill intent until proven otherwise. These are the iron rules.

Knowing where to look or to care comes with experience. In the meantime, ask questions.

<comment><USER3> > assume ill intent until proven otherwise

What does this even mean? This sounds more like being a roadblock to everyone else because you can say "but what about the security" and everyone has to dance to your tune.

<comment><USER4> Assume the user's search term or form data is going to contain SQL injection attempts. Design to handle that.

Assume the person calling up to reset a password is trying to access someone else's account, so make them prove their identity before resetting a password.

Assume employees will go poking around internal fileshares where they shouldn't have access, and design ACLs to keep them out.

 _and everyone has to dance to your tune._

Good. Developers who don't dance to that tune are responsible for headline after headline after headline of valuable user data being accessed, stolen, or leaked, and should risk having their professional software development licenses revoked and companies being fined by watchdogs.

<comment><USER5> This sort of thing is exhausting if you have to keep it up unsupported. Being the negative nancy that keeps pointing out the holes in other peoples' grand plans earns you no kudos.

<comment><USER6> Depends who you are leaking the holes to. Negative Nancy probably makes a living selling said flaws to foriegn governments these days because no one will listen to them.

<comment><USER2> A team I used to work with closely had a process where if a development team didn't think an issue was important enough to fix and release the security team was encouraged to find other buyers for the vuln. It was actually a pretty great way to both keep people on the same page and keep your security folks tethered to reality.

</comment></comment></comment></comment></comment></comment><comment><USER7> This article isn't about infosec at all. Instead it gives some general advice for avoiding analysis paralysis and choosing from among many options.

</comment><comment><USER8> Yes, you're a pretty little snowflake. Unique among the wold. No other profession or field has to suffer the hardships you have to suffer. Get over yourself. Everyone has has this problem. Many people don't even have a name for what they do. You have conferences, educational curricula including at the university level, professional organizations, etc</comment>


The Washington Post will use robots to write stories about the Rio Olympics - <AUTHOR> http://www.recode.net/2016/8/5/12383340/washington-post-olympics-software <comment><USER2> The BBC has been doing this for the World Cup for a long time. They even use semantic web tech from Bulgaria to do it...

[http://ontotext.com/company/customers/bbc-dynamic- semantic-p...](http://ontotext.com/company/customers/bbc-dynamic-semantic- publishing/)

<comment><AUTHOR> Thanks for another example @PaulHoule, just today Australian ABC News are shutting down a service that could be automated "ABC source dries up as transcripts of news and current affairs shows curtailed" ~ [https://www.theguardian.com/media/2016/aug/05/abc-source- dri...](https://www.theguardian.com/media/2016/aug/05/abc-source-dries-up-as- transcripts-of-news-and-current-affairs-shows-curtailed)</comment></comment>


Why We Do Not Evolve Software? Analysis of Evolutionary Algorithms - <AUTHOR> https://journals.sagepub.com/doi/10.1177/1176934318815906 <comment><USER2> Software does evolve - as we've seen many times in the past projects to do a wholesale replacement of large projects tend to fail spectacularly (the Netscape example being the most prominent).

Large projects do evolve - new parts are added over time, old parts are dropped, and individual parts are rewritten, generally in a piecemeal fashion.

The environment selects for some features (say memory protection, which won - OSs either evolved it independently or died out), and select against others (OSs with builtin adveetising struggle to survive). Having misfeatures doesn't necessarily kill off sufficiently "fit" software (humans have objectively weird/terrible anatomy - sinus construction, that nerve that goes from left to right side of your neck via the heart), etc.

Article should be "artificially evolve" software.

<comment><USER3> This article is talking about Evolutionary Algorithms, a specific practice within the realm of machine learning and emergent behavior.

What you're describing is software engineering.</comment></comment>


Canada must reclaim its role as a world leader - <AUTHOR> http://www.theglobeandmail.com/globe-debate/canada-must-reclaim-its-role-as-a-world-leader/article26337462/ <comment><USER2> Canada never had a "world leader" role in the first place. It is an inconsequential player and that's just as well.</comment>


Jim Lehrer Has Passed Away - <AUTHOR> https://www.pbs.org/newshour/press-releases/from-judy-woodruff-longtime-pbs-newshour-anchor-and-co-founder-jim-lehrer-has-passed-away <comment><USER2> RIP Jim Lehrer. A news anchor from a more civilized era. I really appreciated his matter-of-fact delivery.</comment>


Massive octopus nursery found in deep sea - <AUTHOR> https://www.nationalgeographic.com/animals/2018/10/deep-sea-octopus-nursery-discovered-animals-news/ <comment><USER2> Original source: [https://nautiluslive.org/video/2018/10/24/massive- aggregatio...](https://nautiluslive.org/video/2018/10/24/massive-aggregations- octopus-brooding-near-shimmering-seeps)

Seriously great project, the live feed can make for interesting viewing

</comment><comment><USER3> Every time I see deep sea creatures being filmed with bright white light I always wonder if that leaves them permanently blind.

<comment><USER4> Many times, they will use a wavelength that doesn't bother the animals.

</comment></comment><comment><USER5> Paging Ringo Starr...

<comment><USER6> Paging Raffi :-)

</comment></comment><comment><USER7> Worlds largest, aka the second one found

<comment><USER8> Ok, we've replaced the title with the HTML doc title.

</comment></comment><comment><USER9> This may sound cynical. But, I sigh each time I hear of such natural discoveries. This basically means humans will begin interfering in the name of research.

<comment><USER10> That's optimistic. We'll probably try to eat them.

<comment><USER11> These are not an edible species of octopus, they live at a depth of over 3000 metres, five times deeper than any trawler can reach, and they're small - species in that genus are typically ~10 cm across, so all 1000 of them would make about a tonne, worth <$10k if they somehow became edible. I don't think they're in much danger of being eaten by humans.

<comment><USER12> So a massive nursery of octopus, not a nursery of massive octopus.

P.S. I feel the plural of _octopus_ should be neither _octopi_ nor _octopuses_ , but either: _octopus_ \- same as singular, you know, like 'sheep'; _octopussies_ \- for fun, because they're not as cute as cats.

<comment><USER13> Isn't the plural _octopodes_?

</comment></comment></comment></comment></comment><comment><USER14> I'd like to be under the sea In an octopus' garden in the shade He'd let us in, knows where we've been In his octopus' garden in the shade

<comment><USER15> That's exactly what I just thought, ha.

<comment><USER15> Geez, guess there's no Beatles fans here: [https://en.wikipedia.org/wiki/Octopus%27s_Garden](https://en.wikipedia.org/wiki/Octopus%27s_Garden)

</comment></comment></comment><comment><USER16> It shames me to admit this made me hungry, octopus is delicious.</comment>


Designing your sign up page? Read this. - <AUTHOR> http://www.contrast.ie/blog/designing-your-sign-up-page/

<comment><USER2> It's great -- Unfortunately I run a professional office with woman who work for me, and while they'd probably not care if I sent them an article that says "Suck Balls" at the top, _I_ care enough not to send it.

Couldn't you be just as snarky and instead say "We do everything PayPal does except Suck?"

Just some feedback on an otherwise great post.

<comment><AUTHOR> Hey David, I've updated it to say "Suck". I was correcting a typo so I figured why not :)

Share away now!

Regards, Des

<comment><USER2> awesome. shared!

</comment></comment></comment><comment><USER3> Respectfully, the letter spacing of your main font is so wide and the word spacing so comparatively narrow it's making for slow and difficult reading for me. I can see how you may be going for a kind of spacious theme for the look of the site, but there is probably a way to do it without impeding the visual flow of the text...

None the less, I'm quite eager to get through this very interesting article, I'd just rather do it without using Readability.

(MBP/10.5.8/Safari)

<comment><USER4> I found that using webkit inspector to set the paragraph style to normal letter-spacing, 1.4em line-height helped the readability immensely. Something to consider, perhaps?

</comment></comment><comment><USER5> What is this PerfectPayments company you speak of and where do I sign up?

<comment><USER6> I'm pretty sure it was an allusion to WePay.

Their current website doesn't say _that,_ but it does look similar, and their past antics have certainly snubbed PayPal.

<comment><AUTHOR> Hey Simon,

It wasn't really an allusion. I needed a cover image, I wanted something a bit striking or funny, so I went with a sketchy version of the frame I used in the article. I just picked payment, cause I was giving out about Paypal earlier. Thanks for reading, Des

</comment></comment></comment><comment><USER7> I had the same reaction, so I took a look at the style sheet. Two things struck me: line-height is 35px which is way more than the normal 16px, making it look like double space typescript. Second letter-spacing is 1.6833 which makes it look almost like a mono-spaced font. I would change fonts rather than mess with letter-spacing. The lines can be more open than 16px, but double spacing is too much. Apologies if this is too rantish.

EDIT: that aside, the content is spot on for me.

</comment><comment><USER8> Thanks for the article - it's nice to see a counterpoint to all of the advice that focuses only on a "call to action" on the homepage and trying to get people to take it (yes, like like the rotated angled screenshot that points to the signup button :-)

Seems like the ultimate home page would provide everything a prospect needs to decide before clicking "Signup". Basically no funnel at all - Home Page ->> Signup. Nice.

<comment><USER9> _Seems like the ultimate home page would provide everything a prospect needs to decide before clicking "Signup". Basically no funnel at all - Home Page - >> Signup. Nice._

That's the premise behind the (in)famous "long form", and also why they are so successful (even if generally despised.)

<comment><AUTHOR> Yup - Ogilvys book "On Advertising" talks about this, exclaiming that good content sells better than over-simplifications.

<comment><USER10> Why the confusion? Why all the cramming of information? Yes! Good Content/Service, but also keep it simple. If I have to read a "chapter" on what you do, or play with four widgets to "learn" more, or visit your blog and troll for topics of interest, then I'm already turned off. I want to know if it's a product I can use/sign up for in less than 30 seconds -- speaking of which, there's a good chance you'll snag me if I'm 30 seconds in and still with you.

</comment></comment></comment></comment><comment><USER11> I'll tell you what doesn't work: changing a simple design and burying things in layers. Facebook and now Google Mail is guilty of burying "logoff" in a drop down box. Give me a break, putting it in a drop down box doesn't keep me from logging off, it only annoys me.

Make it obvious and easy to use and resist the urge to f __k with it. I can name many other examples. Central Desktop also comes to mind.

</comment><comment><USER12> Des- this thing was spot on. I just got done reading "Made to Stick" and think their point about being concrete vs. generic directly relates to your ideas. Ex: "Its very easy" vs "It's easier than tying your shoes"

Thanks for this!

</comment><comment><USER13> Great article. It is easy to get caught up in the chasing the best presentation, but if you don't have a great product and the information to support it no one will listen.

</comment><comment><USER14> interesting Des, your presentation on tuesday at DWS was thought provoking

<comment><AUTHOR> Cheers Maxer. Good clean useful thoughts I hope :)

<comment><USER14> always love your facebook slide :)</comment></comment></comment>


Apple Reportedly Increasing Gold iPhone 5s Production in Wake of Launch Demand - <AUTHOR> http://techcrunch.com/2013/09/20/iphone-5s-gold-launch-demand/

<comment><USER2> I'm quite sure they manufactured fewer gold iPhones. It's a win-win. They get to test out the market for a new color before making a ton of them, _and_ this now helps to create the perception of exclusivity around the phone.

</comment><comment><AUTHOR> This is really interesting to me. I was actually asking a lot of my friends which colour they'd buy. No one admitted they'd buy the gold one but I believed most people would buy it because iPhone is status - just like why the Prius had so much success because it stood out and solar panels are installed so other people can see them, not for optimal sunlight.

I think this is the case here.

<comment><USER3> The gold color seems unreasonably ostentatious, I mean, how many times do you see a car painted a golden color? Not often, it's too far out. People would be more embarrassed having a golden-plated car than feel good about it as if it's signaling wealth.

So, I expect the demand for golden iPhones to be high only in very specific subsets of people, not among normal people trying to signal a high status. People also want to signal class in addition to wealth.

<comment><USER4> The gold colour is actually a lot more subtle in real life than I'd expected, it's not really that ostentatious.

<comment><USER5> Wasn't gold originally supposed to be called "champagne" or was that media speculation on a different hue? A champagne color is definitely more muted than straight up "gold".

<comment><USER6> It was probably media speculation. Since "Champagne" is well-known to be trademarked (only sparkling wine made in the Champagne region can technically be called Champagne), Apple likely never intended to call the gold version a champagne color. I never saw any leaked material from Apple referring to it as such, at least. It seems like the media settled on calling it champagne and the Champagne gatekeepers preemptively made noise about it[1], thus adding fuel to the fire.

[1] [https://twitter.com/Champagne/status/370623172166430720](https://twitter.com/Champagne/status/370623172166430720)

<comment><USER7> Well played by Apple. Vaporous controversy = free earned media.

</comment></comment></comment></comment></comment></comment><comment><USER8> It's the only way to easily differentiate the 5s from the 5 at a glance. A lot of people want others to know they have the new one, no matter how tacky it makes them look.

<comment><USER7> No, the silver home button/touchid ring is pretty distinctive as well... but only if you can see the front of the phone.

Plus most folks now have cases, too.

</comment></comment><comment><USER9> They should have offered the gold one at three times the price.

and brought back "I Am Rich" as a 5s gold exclusive

that would really set apart the high rollers

</comment><comment><USER10> I had the golden n64 controller that came with the gold Ocarina of Time cartridge. That was probably the last time I felt special because of a paint job. This is just people wanting to show they have the new and not the old like others have said.

<comment><USER11> Or they like the way it looks.

</comment></comment><comment><USER12> I lined up for a 5S today and they actually had absolutely no golds. So the supply is obviously really low; I had to go with the black version.

(and I was the very first person in line.)

<comment><USER13> Same here. Nashville, Tennessee.

</comment></comment><comment><USER14> This is all supply, not demand. I'm sure demand is huge (it sure sounds like it), but there just aren't very many available. The store where I was this morning had literally ten units available in gold, and that was it. They had hundreds of the others. I've been hearing the same sort of thing from all over. Even if gold was half as popular as the other two 5S choices, it still would have sold out instantaneously.

<comment><USER15> And now there will be extra demand created as word gets out about this one model being scarce.

</comment></comment><comment><USER16> The location I was at (a Verizon store) didn't even have any in gold available.

I just picked up my 5s (silver/white) and it's incredible. My wife has the 5, and the 5s is surprisingly better. I stopped by about 30min before the store opened and there were only about 20 people in line, so figured why not and just caught up on email while I waited, it went pretty fast. The Apple stores are all slammed, but the other retailers aren't too bad.

<comment><USER7> Carriers already whined about 5S supply issues using adverbs like "grotesquely" \- quite telling of their state of mind.

[http://www.ign.com/articles/2013/09/17/iphone-5s-supply- will...](http://www.ign.com/articles/2013/09/17/iphone-5s-supply-will- reportedly-be-grotesquely-low)

</comment></comment><comment><USER17> I bought the gold because it's easiest to distinguish from all the older devices (we have them all for business).

</comment><comment><USER18> Surely people love the C3PO character from Star Wars, and want the same color on their iPhone.

[https://pbs.twimg.com/media/BUA7NhCIcAEHxWO.jpg:large](https://pbs.twimg.com/media/BUA7NhCIcAEHxWO.jpg:large)

</comment><comment><USER19> On the online store the gold ones are the only ones showing that they're now shipping in October rather than 7-10 days. I ordered my space gray one last night and got 1-3 days.

</comment><comment><USER20> It reminds me of when the gold motorola razr came out. It was by far the most desired.

</comment><comment><USER21> Is it real gold, or titanium nitride?

</comment><comment><USER22> Just checked on eBay buy-it-now:

        5s gold:   $786 (just one), every other one is above $914

        5s silver: $816 (just one), every other one is above $915

        5s grey:   $832 (just one), every other one is above $867

Gold doesn't seem very different.</comment>


Cuvva – Car Insurance by the Hour - <AUTHOR> https://cuvva.co/ <comment><USER2> Hey. I built Cuvva. Exciting to see it posted on HN, causing a lot of traffic to our site!

Cuvva is currently UK-only. Here if you drive a friend's car, you're normally only covered for damage to third parties. With Cuvva, it's a full comprehensive policy.

This is not typically relevant for the US. However we are interested in the potential for other products which might work in the US.

<comment><USER3> I think coverage by the mile or the hour for primary drivers would be of interest to a lot of car owners in US urban areas. An existing player, Metro Mile, markets their product as pay-per-mile, but the majority of the cost actually comes from a fixed per-month fee. Of course that might be because what I want makes no actuarial sense...

<comment><USER4> Most US states require that a car be continuously insured as a condition of its registration, so I think a fixed per-month fee is inevitable.

Each US state also regulates insurance completely differently, so trying to break into "the US market" with an insurance company is really trying to break into 50 different legal areas. Very similar to the other article on the HN frontpage today about US "FinTech" / money transmittal companies.

<comment><USER5> > ...require that a car be continuously insured...

This doesn't seem like an issue, maybe I'm missing something. You just model it as continuous insurance based off mileage rather than insurance that's only active when driving. Metromile et al are conventional insurers except that they use actual rather than estimated mileage in their pricing.

I think the real reason that there's a monthly fee is that actuarial risk isn't linear with mileage.

</comment></comment></comment></comment><comment><USER6> This is only really useful in the UK not the US.

 _In most states, comprehensive and collision coverage protects your vehicle regardless of who 's driving it._

[https://www.esurance.com/info/car/myth-car-insurance- follows...](https://www.esurance.com/info/car/myth-car-insurance-follows-the- driver)

Note: There are several edge cases, but it's generally not an issue especially if the person borrowing the car has their own insurance for their own car.

<comment><USER7> What stops someone who is cheap to insure (like a retired person) taking out the insurance policy, then someone who is very expensive to insure (like someone with drink driving convictions) actually using the car all the time?

<comment><USER8> Car insurance in the US is quite odd when you think about it. It's mandatory and if you don't live in a city so is a car so defacto you need to buy it. On top of that that doesn't actually cover your car! At the minimum level only cars you damage are covered so if you get hit by someone without insurance they are covered, you are not. Huge amounts of regulatory capture are involved and in my state (PA) it's almost impossible for someone to start their own car insurance company. I see plenty of room for disruption there.

"In America you can buy a car for hundred dollars, but it costs a thousand to insure it" \- [https://www.youtube.com/watch?v=pMHdGBz2AqI](https://www.youtube.com/watch?v=pMHdGBz2AqI)

<comment><USER9> What's odd about that? It would be weird if driving were a solitary activity, but when you drive you place significant risk on other people. You can easily use a car to cause damage that greatly exceeds what an average person can pay, so to ensure that you don't leave others in the lurch, they require you to carry liability insurance. They don't care one bit if you damage your _own_ car beyond your ability to cover it, because that's your own problem.

There's nothing weird about a $100 car costing $1000 to insure, because the insurance isn't about the $100 car, it's about the $20,000 (or $100,000) car you might crash into.

<comment><USER8> I just always felt a more equitable distribution of funds (as PA is a no fault state) is to have the people who have $100,000 cars to buy the insurance as they are the ones with the disposable income to afford it. The guy making 8 dollars an hour at Walmart with his 900 dollar beater isn't getting much value from that set up.

<comment><USER9> What exactly would you propose should be done to place the burden on the guy with the $100,000 car? If you don't require that the guy with the beater buy insurance, all you're doing is setting him up to be bankrupted of whatever little he does have if he's at fault in an accident. And what if he crashes into another poor slob making $8 an hour at the Wal-Mart who managed to take out a loan to buy a slightly nicer car? Or what if he cripples that person for life, and they require constant medical care for decades? You're setting people up to be screwed a lot more than they need to be.

(Note that the no-fault status of PA doesn't really matter here, because it just means that if both drivers have no-fault insurance, they can't sue the other driver for minor injuries. Property damage and serious injuries are still fair game.)

If you _do_ require insurance, but have the cost of the insurance scale neatly with the cost of the car it covers, then you'll either have a really low limit on liability which brings back all the problems above, or you'll need to subsidize the insurance somehow.

Overall, it seems unfair to say that a person is allowed to put other people at significant risk for property damage and injury without paying for what that risk actually costs. If you don't like that this is more burdensome to the poor than to the rich, I think that more general schemes to alleviate poverty, like government assistance, are the way to go, because car insurance is only one of many, many ways that life is disproportionately expensive when you're poor.

<comment><USER8> four issues:

1:the guy making 8 bucks an hour can only afford minimum coverage, the insurance provides no coverage his car.

2:if we require people get insurance anytime they may potentially put others at risk then why isn't it required when you purchase a firearm? or alcohol? or rat poison? or a pet dog? or a baseball bat? The list goes on forever.

3: The cost of providing the car insurance is far below the cost of providing it when you exclude a few bad actors. (to date for me I've paid in over 20 grand, 0 return) Why is the rest of society providing a subsidy to people who can't drive? it isn't an accident when you change lanes every 30 seconds, tailgate, and drive aggressively. Why is society providing a subsidy for people with expensive cars as well? I've never had a 'accident' and yet I'm expected to pay a company for a product that gives me little value until I die, and while someone who has a 'accident' pays slightly more monthly than me I feel I cover far to much of the burden for them.

4:I drive a datsun in vintage racing events, my skill is far above the average driver. I can tailgate at 90 miles an hour with drum breaks. Why aren't my rates lower? beacuse the free market isn't allowed to do it's work on insurance companies due to a regulatory structure that was put in by insurance companies. Why does my credit score count by my skill does not? regulatory capture.

<comment><USER9> Mandatory liability insurance has actually been proposed for firearms. I imagine the reason it's not actually law is because rabid NRA types see any such requirement as an unconstitutional infringement of their 2nd Amendment rights.

As for the rest, the risk is far lower. How many people are injured in baseball bat accidents each year? How much property damage is caused by dogs? How often does alcohol result in damage to a third party beyond the culprit's ability to pay, outside of alcohol-related car accidents?

Insurance costs vary enormously with your demonstrated risk, which includes your driving record. If you're a safe driver, and demonstrate it with a low accident and traffic citation rate, you pay _much_ less than somebody who crashes and gets pulled over a lot.

Your statement that you can safely tailgate at 90MPH with drum brakes makes me think that you're actually one of those unsafe drivers who merely thinks he's safe because he's gotten lucky so far. No amount of skill is going to save you from human reaction times, or the guy in front of you having superior brakes. If you are in fact skilled, then your skill _does_ count, to the extent that you can demonstrate it in a concrete manner.

<comment><USER8> Not a single accident or moving violation ever, I pay 120 a month for minimum coverage because I have shitty credit. What I do on a racetrack under controlled conditions is entirely different than what I do on the road. I keep my stopping distance between me and the car in front of me.

<comment><USER9> Your credit score counts because credit score is a good predictor of accident risk. Apparently nobody really understands why, but that's how it is.

Insurance is all about trying to predict the future, which is tough. The tools are imperfect. But they're decent, and the data supports using credit score as a factor.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> :D [https://support.cuvva.co/hc/en-gb/articles/201826632-Do-I- ne...](https://support.cuvva.co/hc/en-gb/articles/201826632-Do-I-need-the- owner-s-permission-to-get-cover-)

</comment><comment><USER11> Huh?

Most car insurance policies already cover lending a car to a friend. Am I missing something here?

<comment><USER12> Not always in the UK, and if they do it's only liability and not comprehensive/collision.

<comment><USER11> Seems pretty unique to the UK (maybe Europe?). In North America it is standard to pretty much every insurance policy.

<comment><USER13> The difference, in essence, is that in the US, car insurance covers the _vehicle_ , in the UK insurance covers the _driver_.

</comment></comment></comment></comment><comment><USER14> Could be useful.

But... I'm wondering why this is tech news. Is now every service, but done on a computer, suddenly tech news?

(If so, then I predict this will become a boring place, because there are a gazillion services out there that can be computerized.)

<comment><USER15> It's a startup, they built the stack from scratch which is probably a fair bit more complicated under the hood than it looks, it's a niche which is (probably) currently unserved and it's in a decidedly unglamorous industry.

Not to mention also a service which London-based developers are actually pretty likely to consider using on occasion.

It has a _lot_ more reason to be on here than many other popular links.

</comment></comment><comment><USER16> UK only. This would never fly in the US, where insurance is fairly heavily regulated and on a state-by-state basis.

Europe in general has far more flexibility around insurance products and pricing.

</comment><comment><USER17> Does anyone know a good alternative for rental cars? I figure Cuvva isn't ideal in those situations b/c you need something that lasts multiple days, not just hours.

</comment><comment><USER18> Can I view the terms of the policy before purchasing?

<comment><USER2> Yes - they're available in the app before you pay.

</comment></comment><comment><USER19> This is amazing. Bring it to the US.

</comment><comment><USER20> Great product can't wait to use it

</comment><comment><USER21> Wow, this is a great idea. The fact that you can't drive a friend's car is a crazy aspect of US auto insurance.

Is this a new company? Would it apply to Zipcar / car rentals as well as owned cars? Does anyone know of a US equivalent?

<comment><USER22> > The fact that you can't drive a friend's car is a crazy aspect of US auto > insurance.

That's not true at all. I have State Farm (IL and FL), and as long as my family member, friend, whatever has my permission to drive the vehicle, they're covered under my policy.

<comment><USER23> Indeed. It's apparently a little-known fact that this even applies to rental cars, even though the rental car companies do their best to not tell you this.

The wikipedia page even mentions this: "Although frequently not explicitly stated, US car rental companies are required by respective state law to provide minimal liability coverage, except in California, where the driver is solely responsible."

Of course, _minimal_ liability is only good if you have no assets whatsoever...

<comment><USER9> In addition to that, most personal car insurance policies cover rentals, _and_ every credit card I've ever seen provides insurance for cars rented with that credit card. The average renter is covered by three different policies (the rental company's policy, their personal auto policy, and their credit card, although they don't perfectly overlap and may not cover everything you'd want) before the rental company tries to get them to buy a fourth.

Please do your own research before relying on any of this, of course! Especially since individual insurance and credit cards may vary. But personally I've not seen any reason to purchase the insurance they offer.

<comment><USER23> Credit cards generally only cover the cost of the rental car, I've not seen one that provides liability coverage. So if you don't have a vehicle of your own, which would provide such coverage, the little-known rental car policy is what saves you from having to fork over extra cash for liability coverage.</comment></comment></comment></comment></comment>


[Cringely] What if Microsoft doesn't really hope to buy Yahoo at all? - <AUTHOR> http://www.pbs.org/cringely/pulpit/2008/pulpit_20080215_004309.html

<comment><USER2> They can kill Yahoo as the #2 search engine by leaving them unable to do anything, halfway between being bought out and with employees leaving in droves before they are fired and as Yahoo is floundering take there #2 spot.

Then MS could take pity on Yahoo and buy them for real, breaking them up and selling off any pieces they don't want.</comment>


Show HN: I'm a self-taught developer and this is my first app - <AUTHOR> https://itunes.apple.com/us/app/delphi-simplicity-clarity/id1172671058?mt=8/#_=_ <comment><USER2> This looks well-designed and professional, bravo!

I am looking to do a simple iOS app myself. Could you tell me how long it took you to build and what resources you used to learn,get going ?

<comment><AUTHOR> Thanks! I used Cordova/PhoneGap and it took about 10 days from start to finish.

In terms of resources, Cordova's docs are pretty good.

</comment></comment><comment><USER3> Just tried out the app. Interesting and important goal for the app with an effective clear and simple design. Great work!

Edit: After using the app a bit more, there are a few interface changes that could improve the app. Is there any place to contact you about this? The app store didn't have any contact information.

<comment><USER4> You could try tweeting at him? @danielmeng07

</comment></comment><comment><USER5> Looks excellent.

What was your biggest difficulty/hurdle in this project? How did you solve that problem?

</comment><comment><USER6> Are you trying to be rich and famous?</comment>