Microservices - <AUTHOR> http://basho.com/posts/technical/microservices-please-dont/ <comment><USER2> You need to be this tall to use [micro] services:

* Basic Monitoring, instrumentation, health checks

* Distributed logging, tracing

* Ready to isolate not just code, but whole build+test+package+promote for every service

* Can define upstream/downstream/compile-time/runtime dependencies clearly for each service

* Know how to build, expose and maintain good APIs and contracts

* Ready to honor b/w and f/w compatibility, even if you're the same person consuming this service on the other side

* Good unit testing skills and readiness to do more (as you add more microservices it gets harder to bring everything up, hence more unit/contract/api test driven and lesser e2e driven)

* Aware of [micro] service vs modules vs libraries, distributed monolith, coordinated releases, database-driven integration, etc

* Know infrastructure automation (you'll need more of it)

* Have working CI/CD infrastructure

* Have or ready to invest in development tooling, shared libraries, internal artifact registries, etc

* Have engineering methodologies and process-tools to split down features and develop/track/release them across multiple services (xp, pivotal, scrum, etc)

* A lot more that doesn't come to mind immediately

Thing is - these are all generally good engineering practices.

But with monoliths, you can get away without having to do them. There is the "login to server, clone, run some commands, start a stupid nohup daemon and run ps/top/tail to monitor" way. But with microservices, your average engineering standards have to be really high. Its not enough if you have good developers. You need great engineers.

<comment><USER3> That's pretty spot on. I once made an abbreviated flowchart of the above for my microservices article:

[https://www.stavros.io/posts/microservices-cargo- cult/](https://www.stavros.io/posts/microservices-cargo-cult/)

I urge everyone to use it to decide whether they really need microservices or not.

<comment><USER4> I know it is a bit tongue in cheek but if you're working somewhere where Data Segregation can be solved by "a simple cascading delete" you aren't operating anywhere close to where any service architecture is relevant.

</comment></comment></comment><comment><USER5> Microservices, like nosql databases, and complex deployment systems (docker) are very important solutions to problems a very small percentage of the development community has.

It just so happens that the portion of the community is the one most looked up to by the rest of the community so a sort of cargo cult mentality forms around them.

A differentiator in your productivity as a non-huge-company could well be in not using these tools. There are exceptions, of course, where the problem does call for huge-company solutions, but they're rarer than most people expect.

<comment><USER6> My company built from the ground up with micro-architecture and it is an unmitigated disaster. Totally unnecessary, mind-numbing problems unrelated to end-user features, unpredictability at every step, huge coordination tax, overly-complex deployments, borderline impossible to re-create, >50% energy devoted to "infrastructure", dozens of repos, etc.

The whole thing could be trivially built as a monolith on Rails/Django/Express. But that's not exciting.

<comment><USER7> We were also built from the ground up with microservices and had the exact opposite experience. Faster shipping (more value to end users), more predictability (APIs designed/behaved similarly across functions despite polyglot tech), much less coordination overhead (deployed dozens of times per day with a < 10 dev team, pre-release backends well in advance of the user- facing parts), etc. We had to invest a lot in infrastructure, but that was worth it for many other reasons as well. Dozens of repos is annoying, but not for a technical reason (a lot of SaaS like Bugsnag and GitHub used to charge by project).

The biggest downside is it makes shipping an on-prem version nearly impossible. The infrastructure and the software are so inextricably linked that it is not portable in the least bit.

<comment><USER8> One of the ways we managed to get a microservices system on prem (vSphere or Openstack) is to configure it with a system that can handle the whole provision/build/configure/deploy as a single unit: [http://bosh.io](http://bosh.io) .. though the learning curve was steep.

<comment><USER9> BOSH has a famously steep learning curve: [http://i.imgur.com/4UpbgJm.png](http://i.imgur.com/4UpbgJm.png)

</comment></comment></comment></comment></comment><comment><USER10> I'm a little long in the tooth so aren't as up to date with every new fangled technique to land in IT. Some of you may find this anecdote interesting and somewhat pertinent. _Many_ years ago the electric utility I worked at had a home-grown set of batch-run Pro-C and PL/SQL programs that ran various metrology operations on large volumes of meter data. These things were interdependent, ran single-threaded and created a real "peak-CPU-demand" problem for our compute hardware (the irony was not lost). Our industry was facing an explosion in data due to the switch to smart metering. What to do?

Our apps all depended on an Oracle DB. Oracle had recently introduced Advanced Queuing. So I figured I'd de-batch and decouple these things using AQ. Every program (C++) was broken into "atomic", _stateless_ business tasks. Every task was fed by a "task queue". Tasks would take a work-item off a queue, do their thing and depending on the outcome, would look up a destination queue (destinations could only be "business state" queues; task queues could only be subscribed to state queues (topics)), dropping the task outcome onto the state queue. Being stateless and callback driven by AQ, we could run these things together and ramp them up and down as demand required.

The overall structure and dependency of the various tasks was externalised through the data-driven queue network. The resulting solution was far more maintainable, provided "free" user-exits (by virtue of being able to plumb new tasks to existing "business state" queues), and was eminently horizontally scalable. In hindsight this was definitely not state of the art. But we were a pretty conservative business with a bunch of pretty unworldly C and PL/SQL programmers. None of us had used Java at that point. But with this approach were able to cope with a massive increase in data volume and make use of _all_ our expensive Sun cores most of the time.

No Java, no REST, no HTML, no SOAP. But we called these queue micro services :-)

<comment><USER11> What would happen to the system when Oracle rolled out a new version or patch?

<comment><USER10> Haha. We _wished_ we got patches. This was back in the day when the prevailing mantra was, "Patches? We don't need no stinking patches!" (Though we did convince the DBAs to apply a necessary AQ related patch. Rare.) Having said that, Oracle were pretty good with backwards compatibility regarding their DBs. There was talk of this thing called RAC in the next version. What a dream! That's what I would have gone with to achieve "zero" downtime upgrades. Never got the chance. We used very small patch windows where all boundary processes would stop.

</comment></comment></comment><comment><USER12> From personal experience Microservices enforce a clear interface and isolation pattern. This is achievable many ways, but having discrete deployed code makes it very hard to violate rather than being disciplined.

Licensing costs can go drastically up as most modern licensing is node/core based. As can deployment procedures get more complicated.

I would love to understand how this article believes that the modules in a monolithic system can be scaled horizontally if they are actually a single code base in a single system. Either the system isn't monolithic, or it they have never really done it. Sticking a load balancer in front of a micro service and scaling based on measured load requires tools and technologies, but is very scalable. It also allows you to do rolling deployments of draining/rotate out/update/rotate in that allows you to get near no planned downtime.

Distributed transactions are the devil, but you don't need to do them in a microservice design. It requires design work on the front end to clarify what the system of record is, but if each service has a domain it controls, and all other services treat it as the truth, it's rather simple. I say this having researched doing payment transactions across geographically diverse colo's and we treated that as a sharding/replication/routing issue very successfully.

Ninja edit: Starting with a microservice design is most likely overkill for a lot of systems, but either way, clear interface/boundaries in your system are good and healthy

<comment><USER13> Why would you not be able to scale a monolith? You can apply the same principles to it: "[Stick] a load balancer in front of a micro service and scaling based on measured load"?

A microservices allows you to scale up very particular components of an architecture, but there is nothing stopping a monolith from being horizontally scaled in just the same way. In AWS, I would make the monolith deployed with an AMI in an auto-scaling group with a load balancer in front.

<comment><USER14> Yeah, scaling app code is generally a solved problem: shared nothing with some load balancers. It doesn't work for every problem, but the above has been standard in my circles for at least a decade by now.

Databases are trickier though.

<comment><USER15> Multiple instances with shared nothing is the opposite of a monolith, by definition.

<comment><USER16> Not if they're all running the same code.

<comment><USER15> OK, I looked up the definition of monolithic application in a couple of places, and the meaning is not quite what I thought it was. I thought it was 'one app does everything for everybody' but it's more like 'one app does everything for somebody', so the horizontal scaling applies, and my previous comment is wrong.

It's a shame that a 'monolith' application doesn't just mean a genuine singleton, though, as that would be the perfect name for it. A bank of load- balanced monoliths should be a polylith.

<comment><USER14> I'm not sure if it's a useful distinction, because when you think you're building a 'monolithic' app you might actually be building a 'polylithic' app. Beyond that, in a lot of instances you have to make very few tradeoffs to go from 'monolith' to 'polylith'.

By your definition, most rails/django apps are probably polyliths.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> My approach is to design like microservices and develop like a monolith. Thinking about microservices will force you to define module, their boundary and interfaces. A monolith will simplify deployment, refactoring. Once your code matures, you'll know if any microservice has to be taken out and deployed seperately.

<comment><USER18> My conclusion as well. In my mind, I call them 'air-gapped modules'.

[https://www.linkedin.com/pulse/maintainable-software- archite...](https://www.linkedin.com/pulse/maintainable-software- architectures-microservices-vs-air-liyanage)

</comment></comment><comment><USER19> Instead of microservices, I split my projects in tons of libraries and think of them as products, enforcing a well thought of and consistent api (usage api, not http one). I call that an atomized monolith.

I get the cool things about microservices: properly isolated functionalities, ability to assign a team on it, simplicity of code and considering each feature as important, not just "that thing in the codebase".

But it also have all the good parts of monolith: easy deployment and local setup, aggregation made easy, and ability to run integration tests.

For my rails projects, geminabox was of great use for me to achieve this, as it allowed me to host private gems. Lately, I've done a lot of golang, and was surprised to see how it's a natural pattern with go packages.

Only hurting part for ruby projects: keeping dependencies up to date in all those libs (since they all have their test suite, it means that I at least have to update them for test dependencies). To solve this, I've built some tooling that will update all my project automatically and create merge requests for them, running from a cron task.

<comment><USER20> > Instead of microservices, I split my projects in tons of libraries and think > of them as products, enforcing a well thought of and consistent api (usage > api, not http one). I call that an atomized monolith.

There's already a term for that: modularity.

<comment><USER19> A word which totally not carries lessons learnt from microservices vs monoliths. But using that name is fine by me, I'm not trying to define other people world, just sharing mine.

<comment><USER20> I think it does carry those lessons, but most languages are incredibly bad at enforcing modularity. Monoliths make it far too easy to cheat, and dynamically typed languages are typically worse here.

Microservices then just forces on you the modularity your language should have already given you.

</comment></comment></comment></comment><comment><USER21> A interesting article with some good points. I think the important takeaway is understanding that monoliths are probably better for smaller companies, with less total code, and fewer total engineers. At small scales, the "costs" of microservices (network overhead, distributed transaction management, RPC complexity, dev-environment complexity) outweigh any benefits. A monolith lets you develop quickly, pivot, easily build cross-domain features, and is more efficient up to a point.

That said, I believe there is a point where monoliths begin to break down.

First, It is tough to keep code well structure in a monolith, and eventually things bleed between domains. That means, as mentioned, engineers must understand the entire codebase. This isn't practical for 100k+ LOC codebases. Strict boundaries, in the form of interfaces, limit the scope of code that every engineer must understand. You probably still need gurus who can fathom the entire ecosystem, but a new eng can jump into one service and make changes.

Second, deployment is a mess with any more than a few hundred engineers on a given code base.

Third, it becomes increasingly difficult to incrementally upgrade any part of your tech stack in a monolith. Large monoliths have this tendency to run on 3-year-old releases of everything. This has performance and security implications. It also becomes difficult to changes components within your monolith without versioned interfaces.

Fourth, failure isolation is much harder in a monolith. If any portion of code is re-used between components, thats a single point of failure. If your monolith shares DBs or hardware between components, those are also points of common failure. Circuit-breaking or rate-limiting is less intuitive inside of a monolith then between services.

TLDR; start with a monolith, migrate to micro-services when it becomes too painful.

</comment><comment><USER22> There's some good points here and some I disagree with. One area, though, where I think he misses the point, is:

> Additionally, many of these stories about performance gains are actually > touting the benefits of a new language or technology stack entirely, and not > just the concept of building out code to live in a microservice. Rewriting > an old Ruby on Rails, or Django, or NodeJS app into a language like Scala or > Go (two popular choices for a microservice architecture) is going to have a > lot of performance improvements inherent to the choice of technology itself.

Languages and tech stacks generally have tradeoffs. Considering Rails vs Go, you could consider the (massively over-simplified) tradeoff to be that rails is better for prototyping and iterating quickly, while Go is better for performance. In an ideal world, you'd write your webapp in Rails, but put the performance-intensive stuff in Go. You'd need to communicate between the two by, say, http. Suddenly you have services.

The performance gains of using a new stack aren't orthogonal to services– they're actually one of the key selling points of services: you can use whatever stack is most appropriate for the task at hand without needing to commit the entire project to it. You can use postgres for the 99% of your app that's CRUDy and, I dunno, cassandra for the 1% where it makes sense. It's difficult (although not impossible) to do that cleanly within a monolith.

<comment><USER23> One of the downsides of the blog post was that I adapted it from a lightning talk, so it was meant to be a little content-light, but to put ideas in peoples minds around how to think about the tradeoffs.

For example, your point about Go vs Rails is an apt one - I would only add that I made that comparison because...

A: It was originally a golang meetup where I gave the talk B: Go is increasingly becoming popular as a choice people move to off of Rails, for performance sensitive code (Scala being the other popular choice I see), and also for building "microservices" themselves.

I could have, and maybe should have, gone a little more in depth at that part, but the idea wasn't to be fully exhaustive (for better or worse).

But the main takeaway about the performance gains was that the idea of putting the word "micro" in front of something magically made it more performant without appreciating why. It's a response to folks simply parroting information without understanding it.

Thanks for the feedback.

<comment><USER24> > Go is increasingly becoming popular as a choice people move to off of Rails

If they moved from Rails to Go, these people didn't need Rails at first place given how bare bone Go is. That's the same issue with micro-services, choosing a tech or architecture because hype instead of understanding requirements. Micro-services are something that should be an exception yet it is pushed as a rule by many influential developers, who won't be their to clean up the mess when it becomes obvious it wasn't the right choice.

</comment></comment></comment><comment><USER25> This could be titled "If you do things wrong it won't be good".

A lot of his examples are of people doing things poorly or incorrectly. I could make the same arguments about object oriented programming my saying it's bad because someone makes every function a public function.

For example, microservices are absolutely more scalable if done correctly with bulkheading and proper fallbacks and backoffs, and proper monitoring, altering, and scaling.

But those things are hard to do and hard to get right.

<comment><USER23> Hi, author here - thanks for taking the time read it.

You're not wrong in that this article is meant to point out the pitfalls of the approach, and to advocate for understanding before diving into a particular architecture.

It's meant to give people things to consider before deciding breaking things into "microservices" is the right thing for their engineering org at that time.

I attempted to note several times that my intention was not to say "Microservices are bad", but rather "Please don't dive in before you consider the trade offs". It's not as simple as some folks might have you believe, so I felt it was valuable to have a "lessons learned" type retrospective coming from someone who has been involved in both approaches.

Thanks for the feedback.

<comment><USER25> Got it thanks for clarifying. My suggestion would be to make that a bit more explicit -- I didn't get that impression reading the article.

</comment></comment></comment><comment><USER26> Raises some good points, but I think the title isn't really correct. It's not "don't use microservices" \- it's more about making sure you understand the implications of having a microservice architecture, and making sure it's not an excuse for not writing a monolith (or SOA) properly.

<comment><USER23> I do wish I had titled it better, as most people have (rightfully) dinged me on this one ;)

<comment><USER27> I'll go against the majority (or vocal minority?) and say: I like your title, it's good because it attracts attention (I wouldn't read the post if it were a "understand the implications of having a microservice architecture" or whatever others recommend).

</comment></comment></comment><comment><USER28> "You don't need to introduce a network boundary as an excuse to write better code"

Absolutely this!

microservices is just decoupling by another name.... and you do not need a network-boundary to enforce this.

Monolithic code can also be nicely decoupled too.

<comment><USER29> > microservices is just decoupling by another name.... and you do not need a > network-boundary to enforce this.

If code is decoupled enough that it can be separated into independent processes communicating over a network, that creates additional freedom into how the components can be deployed to (real or virtual) hardware, which is itself a kind of decoupling.

If you have processes communicating by local-only IPC methods or, even moreso, components operating within the same process, there is a form of tighter coupling than exists when the components are separate networked components.

<comment><USER20> > If code is decoupled enough that it can be separated into independent > processes communicating over a network, that creates additional freedom into > how the components can be deployed to (real or virtual) hardware, which is > itself a kind of decoupling.

It also introduces additional failure modes.

</comment></comment></comment><comment><USER30> Cannot agree more with this based on experience of small startup. Let's say, you want to develop a mobile app and REST API for it hosted somewhere in cloud. There's so much hype about it, so you want to do it "right" (it's right indeed, but for some distant future until which your startup needs to survive). So, the possible solution is to take some common stack, like Spring Cloud, and build a number of microservices with service discovery, config server, OAuth and API gateway.

It appears, it's not so easy: 1\. First, documentation as always is not the best, and you'll have to spend time figuring out how to wire together different parts of the system and build various configurations of it for local development, CI build and production. 2\. Then, there's debugging issue. Once you've figured out how to work with Docker (good news, it's really easy today), you may want to do some debugging in IDE, but it becomes really painful to launch everything correctly with attached debugger if the services interact with each other. 3\. Finally, it's production deployment setup and associated costs. Besides the complexity of deployment, do you really want to pay for 14-20 EC2 instances at the time of the launch of your service and burn the money on 0% CPU activity? It will take months, probably years to get user base sufficient for utilizing this power.

The better approach is to develop single server app with future scalability in mind. You can still have separate components for each part of domain, you just wire them together at packaging time. This server app still can scale in cloud, with correctly set up load balancer and database shared between nodes.

Fortunately, we spent not much time on building microservices (about 1m/w to figure out the costs and benefits) and were able to refactor the code to simpler design, but many developers should not care about them at all at early days of their company.

</comment><comment><USER31> I for one, feel the same way when someone tells me they are building "microservices" for a small application that they don't ever plan to scale to that levels. IMO, amongst us, there is a wide-spread issue of "Here's the new cool thing - My application/system has to do it". The other day, a friend was talking on and on about setting up a Hadoop cluster for what I saw as a one- time use batch script.

<comment><USER32> And? Did you try to talk your friend out of it?

<comment><USER31> Yes, I sent him this to read, [http://aadrake.com/command-line-tools-can- be-235x-faster-tha...](http://aadrake.com/command-line-tools-can- be-235x-faster-than-your-hadoop-cluster.html)

and he changed his mind :-)

</comment></comment></comment><comment><USER33> You use microservices when your project expands beyond the monkeysphere number where everyone knows everyone else.

It allows teams to work in their own world without having to coordinate as much with other teams or people.

Microservices are good for large companies. If you're small you don't need them.

<comment><USER34> > You use microservices when your project expands beyond the monkeysphere > number where everyone knows everyone else.

A layered architecture can give you the same.

Microservices, imo, address organizational/industry deficiencies in the design and evolution of domain models. You're basically trading analytical pain for operational pain. As the top comment in this thread (with the excellent list) concludes, you will need "engineers".

> Microservices are good for large companies.

And this has nothing to do with number of developers. It has to do with inherent complexity of a unified domain model for large organizations. As an analogy, consider microservices as scripting to layered architectures compiled language.

<comment><USER33> Microservices also enforce boundaries significantly more strongly. A layered monolith can still easily have random people cut across boundaries without you knowing because there are hundreds of engineers all working in the same system.

Large companies don't have problems throwing more engineers at a problem. But they will always have a problem in coordination costs.

Microservices also allow you to use different tech stacks for different purposes more easily.

Maybe use java for one involving hadoop or some GIS library. Use erlang for some message management service, use golang for some simple API service, use nodejs for some frontend web server, etc.

Overall the advantages of microservices come for social reasons, not for a particular technical reason.

<comment><USER34> Layered systems do not have to be 'monolithical'. Note that we're both at this moment using layered systems to have this conversation.

> A layered monolith can still easily have random people cut across boundaries > without you knowing because there are hundreds of engineers all working in > the same system.

I appreciated your final word regarding "social reasons" and I think we're in strong agreement in that regard.

In the final analysis, it seems accurate to say that Microservices approach permits runtime operational [micro] payments towards organizational and analytical debt [1].

The hypothetical system(/straw man?:) you posit above is indicative of organizational, not architectural, failure/deficiency.

[1]: in the 'technical debt' sense.

</comment></comment></comment></comment><comment><USER35> One suggestion I would make if you are going to use microservices is to consider using gRPC rather than REST. You can save yourself a lot of the hassle involved in the communication that way AND make things quite a bit faster.

</comment><comment><USER36> like all things, one size doesn't fit all. not everything is a nail regardless of how shiny the hammer is. Having said that when the situation is right, microservices are great.

some of us have been through this all before with soa or in my case with com. Each individual component is simpler but the documentation between the components becomes absolutely vital.

we ended up keeping a copies of the interfaces in a central location (with documentation of all changes per version) so that everyone would know how to talk to all the other systems.

and don't think that the interfaces won't change. they will. and often across many systems/components. like a ripple.

</comment><comment><USER37> The problem is to define the scope of each service. And it is still possible to create spaghetti out of how the services interact and how coupled they are with each other.

If done poorly it is like trading one problem with another problem.

<comment><USER13> Yes, I have definitely experienced this. At the company I work at we have dozens of super tiny services that could have been replaced with a single class, or module in a code-base.

Each of the dozens of microservices gets it's very own dedicated AWS load balancer, RDS instance, and Auto-Scaling Group in multiple regions. Just the infrastructure management alone is monumental.

 _Edit: punctuation._

</comment></comment><comment><USER38> As someone working with this setup right now, coming from what is fondly referred to around here as the God-Monolith of our 1.0 version, I couldn't disagree more....

But as always, this is an artform, writing and designing, not laying down pavement.

There's no "right" way, and any blanket statement about anything is false.

Don't use microservices where they don't make sense, make educated decisions, and choose the best option for your situation.

It made sense in our situation, because all our services have very very very specific rules and boundaries and there's no overlap anywhere.

</comment><comment><USER39> > _Fallacy #5: Better for Scalability_

> _However, it's incorrect to say that you can only do this with something > like a microservice. Monolithic applications work with this approach as > well. You can create logical clusters of your monolith which only handle a > certain subset of your traffic. For example, inbound API requests, your > dashboard front end, and your background jobs servers might all share the > same codebase, but you don't need to handle all 3 subsets of work on every > box._

This makes little to no sense to me, and feel like we're bending the definition of "monolith" to mean "microservice" so that we can tick the bullet point. How, exactly, do I achieve this, when my code is mashed together and all running together?

I have a monolithic app today: an internal website, which is so small that it could be served (ignoring that this would make it a SPoF) from a single machine. But it's so closely bound to the rest of the system, it is stuck alongside the main API. So, it gets deployed _everywhere_.

If it were discrete enough that I could run and scale that internal service separately, I wouldn't be calling it a monolith. At that point, they're separate executables, and scalable independently — that's practically the definition of microservice. And I can't do this if (where they need to) they don't talk over the network (one of the earlier bullet points).

<comment><USER40> If you could separate the inbound traffic to either the website or the API, then you could do this. You'd need something in front of the code you're deploying though

My team has a 500k monolith written in java 1.6. I don't really want to invest in fixing it, I'm migrating stuff to the new system. So a way to keep the old one going risk-free is to create three load balancer pools, and have apache send some traffic to the three based on URL pattern

* /users goes to pool one * /dashboards goes to pool two everything else goes to pool three

That guarantees that /users and /dashboards can be kept to certain level of performance - by adding more machines, not by diving into the code and trying to fix stuff.

The benefit is that its the same deployable in all cases, so its very easy to push.

</comment></comment><comment><USER41> Stabby! Several thumbs up to the point #1, that interface boundaries needn't be coincident with service boundaries. In my experience, the benefit of breaking out microservices is the decoupled deployment. A heuristic is, if you have fixes/features that are waiting to be pushed to production until unrelated code is passing/QA'd, you've got a good candidate for a separate service.

</comment><comment><USER42> Nice compilation of fallacies on micro-services, something that we cannot ignore; but after be reading a little about kubernetes I think much of those problems may be resolved using kubernetes and some of common sense.

[https://kubernetesbootcamp.github.io/kubernetes- bootcamp/ind...](https://kubernetesbootcamp.github.io/kubernetes- bootcamp/index.html)

</comment><comment><USER43> @StabbyCutyou, how does Basho's choice of Erlang as the primary language affect it choice. My (naive) understanding is that Erlang forces one to build a single-process system as if it were a multi-process system from Day 1. Does this make the monolith -> microservices switch easier for Erlang systems than it is for others?

<comment><USER44> Erlang doesn't do incapsulation very well. Even if I divide code to clean OTP Applications, each with it's own public interface, nothing in compiler stops people from referencing internal modules and their exported (i.e. "public") functions. The problem is that many OTP behaviors' internal callbacks need to be exported, so they are exposed publicly.

I guess one need to use xref tool, to find all references outside the OTP application.

</comment></comment><comment><USER45> While this and many other writings about microservices are largely concerned with network-based environments, there exists another microservice exemplar specific to the JVM world:

OSGi[0][1]

I mention it mostly to assit those wanting to explore the concept of microservices itself, as opposed to assuming a network transport is always involved. Being JVM specific, "kicking the tires" on it naturally requires that environment. Perhaps, though, some of the writings discussing it would be of benefit to those using other tech stacks.

Of course, OSGi does not preclude distributed processing (and often is employed for such).

0 - [https://www.osgi.org/](https://www.osgi.org/)

1 - [http://www.theserverside.com/news/1363825/OSGi-for- Beginners](http://www.theserverside.com/news/1363825/OSGi-for-Beginners)

</comment><comment> vhost- It doesn't have to be messy. I've worked in monoliths that are a complete disaster. I've worked in micro-architectures that are a complete disaster. It's the same kinds of people and management practices making these disasters.

I will say the only clean systems I've worked in have been microservice oriented. All monolithic systems I've worked on never scaled properly and always had bugs with 1000 function deep stacktraces.

I've talked to people who have worked in excellent monoliths (rails and django). I know they exist.

Moral is: do it right and have good development practices.

</comment><comment><USER46> I heard about microservices about a year ago, and now it said the hype has ended before I even noticed? Admittedly I'm not in the loop, and it's hard to track all the trends from outside.

</comment><comment><USER11> You need almost none of that scary list to start building microservices. Lambda functions can be created in minutes, even in UI console. And they have almost everything from that scary list by default.

Lots of people are still in denial regarding microservices...

</comment><comment><USER47> My thought is that this comparison between a monolithic code base vs a microservices code base is a bit subjective. If you're starting out chances are your code base hasn't even gotten to the level of being monolithic. So those thinking about how they're going to architect their platform may begin to think that a microservice setup could help for future changes to their code. It really depends on each team, their background, and how they want to think about their platform in the future. To list out the pros and cons of both to draw a conclusion that one is better than the other is certainly setting a bias that I believe to be a bit unfair. Just look at Netflix and their container services. It's a platform adopted by a ton of companies including Nike. So for some a microservices approach makes a lot of sense.

<comment><USER23> You have a good point in that in the early stages, any app is likely not to be a "monolith", but it's less about size/LOC and more about the design ethic of the architecture itself.

If you build your codebase internally with service level abstractions in mind, you can gain a lot of benefit without the cost of the network or the additional errors it can introduce.

Thanks for reading!

<comment><USER47> I guess, in my head it's also eliminating a lot of code bloat using microservices. Just my opinion though.

</comment></comment></comment><comment><USER48> Why would someone push for those "5 thruths"? The point of microservices is to ease Ops life, so that deploying is less of a "big bang"-like event and more geared towards incremental and local evolutions.

</comment><comment><USER49> [http://basho.com/posts/technical/riak-quick-start-with- docke...](http://basho.com/posts/technical/riak-quick-start-with-docker/)

</comment><comment><USER50> If your product or project is not or has services, then it does not need such a thing called microservices.

</comment><comment><USER51> If this person used microservices perhaps their site wouldn't be down right now...

</comment><comment><USER52> I swear, the HN front page algorithm is easily gamed, this gets a few points quickly and it rises straight to the front page. I don't know if HN is accounting for vote rings but some penalizing should be implemented.

<comment><USER53> AFAIK the content is curated. So content with a little ammount of upvotes can show up if some admin wants to.

<comment><USER54> It's not curated. People used to be asked to repost their content if the admins thought it was interesting, but this no longer occurs. (This turned into a system that could automatically repost content considered interesting; I've stopped posting stories as often, so I'm not sure if it still happens.)

Furthermore, there are protections against vote rings. If, for example, someone votes directly on the URL for a story, or if the referrer is often the same, those votes are discarded.

However, you're right that the algorithm has evolved over time. Visit [https://news.ycombinator.com/classic](https://news.ycombinator.com/classic) to see the previous algorithm in action.

<comment><USER53> I remember some people complaining about it some time ago, thanks for pointing that out then. :D

</comment></comment></comment></comment><comment><USER55> The title should be "I've never implemented microservices properly, so you should avoid them."

<comment><USER35> I think _a lot_ of people don't implement them properly. And if you're an individual or a small team trying to build something it's usually overkill.

<comment><USER56> Agree 100% with both of these statements. A general rule seems is to divide them on transactional boundaries and business concerns.</comment></comment></comment>


Napoleon's Miniaturized Traveling Library - <AUTHOR> http://www.openculture.com/2017/10/napoleons-kindle-see-the-miniaturized-traveling-library-he-took-on-military-campaigns.html <comment><USER2> \- For religion, he wanted: both (bible) testaments, the Quoran, history of the Church.

\- Epics: Homère, Lucain, Le Tasse, Télémarque, La Henriade

\- Tragedies: some Corneille, Racine (minus a few), Voltaire. No Molière

\- History: some chronology works and works on the Roman Empire

He didn't want many memoires and "useless dissertations".

As a side-note this started in July 1808 and by December he was complaining that there wasn't enough new content.

The full project never got executed, only a subset ever got produced and apparently it wasn't satisfactory.

Source: Les Bibliothèques particulière de L'Empereur Napoléon, Antoine Guillois, 1900 — [http://bibnum.enc.sorbonne.fr/omeka/files/original/0ac1be913...](http://bibnum.enc.sorbonne.fr/omeka/files/original/0ac1be91312051774cb5436fab7eff4d.pdf)

</comment><comment><USER3> Articles like this always make me wonder about the effect of technology, particularly easy access to the Internet at all times, on my generation (born in the late 80s and early 90s).

That discussion -- about attention spans, finding exactly what you're looking for versus having to interpret a static document, etc. -- has been had a million times on this site and elsewhere. But it struck me that he didn't ask for digests or something like that. For Napoleon, evidently there was more to these books than their information content alone.

</comment><comment><USER4> Very cool! Unfortunately, no one seems to have a list of books that were in it. I wonder what Napoleon liked to read...

<comment><USER5> This "Life of Wellington" mentions the capture of Napoleon's traveling library and mentions that they found the works of Homer, the Bible, Ossian (a supposed ancient Scottish poet now thought to be fraudulent), and a work of Voltaire.

[https://books.google.com/books?id=pqv_06RS6XcC&lpg=PA536&ots...](https://books.google.com/books?id=pqv_06RS6XcC&lpg=PA536&ots=XAF9qrRA8Y&pg=PA519)

<comment><USER6> Nabokov has curious things to say about the influence of Ossian (maybe in the introduction to his translation of the Song of Igor's Campaign?).

</comment></comment></comment><comment><USER7> Anti-Adblock or something said

> You value our content ...

Less than you think. Certainly not as much as my time, attention, and security. Bye.

</comment><comment> jay-anderson Does anyone know how small the books were? There's nothing in the picture to give a sense of scale.

<comment><USER8> The article mentions that they were "small 12mo", which is a standard size (also called a duodecimo) corresponding to 5 x 7.375 inches.

</comment></comment><comment><USER9> I heard you like books... so I put a books in your book.

<comment><USER10> Just in case you are wondering why you are being downvoted:

This sort of low-effort joke is generally frowned upon here.

While it might be fun to some, it would ultimately lead to many more comments that do not add to the discussion, and thereby dilute the overall experience.

<comment><USER11> Said account is 5.5 years old, so they know what's what. Furthermore, judging by their comment history, this user as a habit of posting low-quality comments interspersed with enough 'real person' stuff, I suppose, to prevent a ban. I'm not a fan.</comment></comment></comment>


Why Dropbox decided to build its own infrastructure and network - <AUTHOR> https://techcrunch.com/2017/09/15/why-dropbox-decided-to-drop-aws-and-build-its-own-infrastructure-and-network/ <comment><USER2> Dropbox hasn't dropped AWS, they moved things off AWS as it made sense to. The article is talking about two things, the move of file storage and a network backbone. Neither of which were done recently.

The file storage move from S3 to Magic Pocket is detailed in these blog posts:

[https://blogs.dropbox.com/tech/2016/03/magic-pocket- infrastr...](https://blogs.dropbox.com/tech/2016/03/magic-pocket- infrastructure/)

[https://blogs.dropbox.com/tech/2016/05/inside-the-magic- pock...](https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/)

[https://blogs.dropbox.com/tech/2016/07/pocket- watch/](https://blogs.dropbox.com/tech/2016/07/pocket-watch/)

The network backbone is talked about here:

[https://blogs.dropbox.com/tech/2017/09/infrastructure- update...](https://blogs.dropbox.com/tech/2017/09/infrastructure-update- evolution-of-the-dropbox-backbone-network/)

<comment><USER3> The 'value' cloud services provide is purely for experimental services an organization does not want to commit physical assets. Those deluded to believing cloud services provide any value beyond test service deployments are propaganda poster boys for today's tech sucker awards.

<comment><USER4> You are extremely wrong. The Dropbox story shows that if the bulk of your value comes from selling a commodity (storage) then you need to improve your margins by moving away from a provider that also makes the bulk of their money selling the same commodity.

For companies where the value lies in the utility of a service that can't be easily replicated, you have pricing power to make the convenience of AWS worth the expense.

<comment><USER5> Pretty sure that the value of Dropbox doesn't come from the storage. It comes from their software. The magic thing that allows to replicate all your data seamlessly between as many computers and phones as you want, without the need to click any button or understand what the word storage even means.

</comment></comment></comment></comment><comment><USER6> This means their EBITDA now probably shows profitability (servers they own are amortizable [the A in EBITDA], where are AWS is expensed.)

Edit: Amortization/Depreciation can <i>generally</i> be used interchangeably.

<comment><USER7> Maybe but OpEx is tax deductible. CapEx is not.

<comment><USER8> I am pretty sure if they desire they can convert CapEx into OpEx.

<comment><USER9> Sure, if they sell the storage system to someone and arrange a leaseback.

<comment><USER10> That's how Microsoft manages its campus real estate. :-)

<comment><USER11> Nearly every company does that. It's amazing how little many companies actually own. I've worked in offices where everything from building over furniture to even the plants was leased. The company ran with close to zero assets (services business). Pretty sure that this wouldn't make any sense if it wasn't for tax purposes.

</comment></comment></comment></comment></comment></comment><comment><USER12> I'll be 100% honest, I didn't realize Dropbox was still on AWS. You figure at a certain scale it makes more sense to run your own solution.

<comment><USER13> Yeah and that's why it amazes me that Netflix uses Amazon a straight competitor that uses the money it makes from Netflix to compete with Amazon prime.

<comment><USER14> Netflix probably in a better position to do the math than random people on HN. I think they have done their homework and decided that it is overall cheaper to keep ___some___ of their infra on AWS.

<comment><USER15> Yes! Thank you! Every time some piece like this is posted 'experts' arise here and know everything better. Like there is a bunch of idiots working at Netflix who are happily burning money...

</comment></comment></comment></comment><comment><USER16> What does AWS do when someone moves 500PB of storage off of their systems? Do they sit idle till some other big customer comes along or is their growth so phenomenal that a 500PB move off doesn't even slow new (storage) deployments at all and just keep up their pace?

<comment><USER17> with no actual facts, i'll speculate.

let's assume they have 12 buildings at each region with 500 racks in each. 6k racks

Let's assume that S3 is 1/4 of their infrastructure.

Let's assume you can put 60 * 12TB disks in 4U and they use 52 racks. That's 13 _60_ 12TB per cabinet or 9.4PB per rack.

1500 racks at 9.4PB raw capacity with RF=2.1 so best case that's that's 1500 * 9.4 / 2.1 = 6700PB at a site.

So 500PB would be 7% of the space.

They're probably growing 5-15% per month like most hot companies now, so that's 1-2 month's growth.

Of course these numbers are certainly off. But probably in the magnitudinal ballpark.

<comment><USER16> My guess (and it's a terrible guess) is this may be about 4mo growth... and they might take the opp to retire older lower capacity drives and just keeping up the build-out.

But who knows...

<comment><USER18> If you're ordering 500 petabytes you probably sign a contract with Amazon, and have to let them know in advance if you want out.

<comment><USER16> That's a good point --but they didn't start with 500PB, when they first went to AWS they probably only used a few PB to begin with and grew "organically" over the years. But likely still had some contractual stuff for this kind of scenario.

</comment></comment></comment></comment></comment><comment><USER19> I had a chance to compare all of the file services a few years ago in a large corporation and I thought out of all of them, the DropBox engineers were the strongest. Microsoft second.

</comment><comment><USER8> People making sane choice very refreshing. I wish my company would make same decision.

<comment><USER20> Make the sane choice of applying to work for Dropbox =:)

</comment></comment><comment><USER21> I am curious on the hardware build out used for the storage nodes. Some of the major issues I have seen with all the appliances out there for storage are the following:

1\. Network throughput on the appliance is fast but for an Enterprise level the 10gigE cards used become a bottleneck for transactions because of how the software hypervisor scales the data. 2\. Power consumption of the appliances in a rack mount environment are too high and leave needed space that has to stay empty because of the facility power per rack limitations. 3\. The software hypervisor scales the stack vertically and relies on the software to load balance horizontally. The performance in a high transactional environment becomes dependent on the software to scale instead of the natural horizontal distribution that can be setup on the hardware out of the box. Standard multi- purpose storage arrays scale horizontal with very little over head from traditional software storage management. I only found one company whose software does not force the stack to be vertical but they fail to meet a reasonable performance in network/power.

Streaming petabytes of data to keep a dynamic constant (static overall storage requirement that changes it's data life cycle via retention rules) becomes very hard with premade hardware.

Does anyone have any recommendations or has attempted a similar exodus from S3 that they can share?

</comment><comment><USER22> Is 500 petabytes really that epic? 1500 tapes? I could probably stuff that in the back of my minivan.

<comment><USER23> Great comment for 2017. I would like to see the 1987 version and the 2037 version of it.

<comment><USER24> I recently misplaced one of my 32GB SD Micro cards I was using to have an alternate image for my Raspberry Pi. So I just bought another one for the price of a half-decent lunch.

And I did have a flashback to my first hard drive, all 220 glorious megabytes of it, and a great deal more expensive than "a decent lunch", and get a sweet hit of that _yeah... I am in the future, aren 't I?_ feeling.

</comment></comment></comment><comment><USER25> Dropbox is cool and all, but i hate their pricing model. I just want to keep about 50-100 GB there (i don't hoard stuff). I don't wanna pay $10 / month for that. At backblaze i pay less than $0.5 / month. I could pay quadruple, since Dropbox is a lot better service (and quite a different one, too). But not 20 times as much.

I know that Dropbox doesn't care about my tiny dollars and all. But why not let customers pay for what they use? This "constant growth" bullshit is probably the reason they don't care.

<comment><USER26> what bugged me about their pricing model (perhaps it's changed?) is that things people share with me count to my cap. I had a client share stuff with me via dropbox. So I signed up. I get "2 gig free!" Yay. Another client shares stuff with me. Then another. Then another. Then I'm at my 2gig cap with just other peoples' stuff. They all think it's great, because none of them are paying for it. _I_ needed to pay money to support their free use of the service, and I declined. Just one more $x/month service I didn't need to get hooked to. Maybe the pricing model has changed what they count now?

<comment><USER27> How do you have "clients" and you can't afford 10$ a month to not have to think about it?

<comment><USER26> It was largely the principal of the thing. It's _yet another_ $x/month, on top of other services being paid for to help service/manage those clients. Example - PM tools. I'm already paying for project management tools - put the files there, damnit. "oh, but I like dragging to my dropbox!". Oh, and someone else uses box.com, and so on.

Again, what gets my goat is partially the payment, but mostly the "it's free for them, but I end up needing to pay to accommodate their use of the service" angle. If they - other parties - were paying for it first, it would bother me less. But raving about how 'free' a tool is which _I_ have to pay for to work with them (and they're using it precisely because it's free) bugs me.

Getting sucked in to a few more $x/month things here and there 1) dilutes where stuff is supposed to live (wrt files) and 2) just becomes a drag on finances. A handful of services can end up going from several hundred dollars in to the thousands if you don't keep tabs on things.

Maybe I'm an HN failure because saving several hundred dollars per year matters to me? I guess my skills must suck - most people can apparently rustle up $300/hr Rust/Go/Elixir project work simply by starting to formulate the idea that they're considering taking project work. I'm not that skilled.

</comment></comment></comment></comment><comment><USER28> > While cost is always something that we consider, our > goal with our network > expansion was to improve > performance, reliability, flexibility and control > > for our users — which we have succeeded in doing," > a company > spokesperson told TechCrunch.

Who believes this?

It was, easily noticable by everyone, wanting to reduce the cost. Which is fine, everyone does so, so why not admit that it was the primary impetus?

I would not want to outsource control over any larger company that I were to run to other, even bigger companies.

<comment><USER29> Performance and reliability definitely affect subscription rates. Revenue growth is actually more important than decreasing costs (as long as your costs don't start increasing disproportionately).

</comment></comment><comment><USER30> Three data centers "biult" by only the dozen people on the infrastructure team? Not possible. I wish articles like this wouldnt hype small teams where it is obvious that most all of the work was outsourced. It would seem that dropbox here was still operating as customer: ordering rather than physically biulding much of anything. Those dozen dropbox people were not running cables.

<comment><USER31> The article is unfortunately not very specific and I can not find much information with a quick search, but:

It's very possible that they did this all on their own provided they rented cabinets/cages in an existing facility like Equinix, which is extremely common. Then they do not need to manage power/generators, fiber into the building, or any other data center necessities.

It does not take very many people to do the wiring inside a cage, especially considering Dropbox has been doing this over the span of a few years. If you've ever been physically in a data center (personally visited a few Equinix facilities myself), it's mostly one person from a company working in a cage that is wiring everything. I've rarely seen multiple people do wiring, and that same person will come back every day thereafter to continue working until the job is done.

What they are doing sounds entirely feasible. If money is no object in regards to equipment, with 3 data centers, I'd honestly say you only need 4 competent people to get the job done.

3 on-site. 1 remote/office.

From there, the more the merrier. A guy to lift the equipment up as well is nice sometimes - can be pretty heavy!

</comment></comment><comment><USER32> The article is dated today, but isn't this news from a few years ago? [https://www.wired.com/2016/03/epic-story-dropboxs-exodus- ama...](https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud- empire/)

</comment><comment><USER33> They did this a while ago.

Here's an article from last year with a lot more detail - [https://www.wired.com/2016/03/epic-story-dropboxs-exodus- ama...](https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud- empire/)

</comment><comment><USER34> (a) CAF ran out

(b) Without CAF, GC and AWS pricing is ridiculous - somewhat akin to paying $9.99/lb for a Purdue chicken a day before the expiration

(c) Without the scale of AWS and GC one can get vendor prices at Google/Amazon MFN + 10% any day or +5% after schmoozing.

</comment><comment><USER35> I'll save you a click. Cost savings.

</comment><comment><USER36> "We're talking about a company that had 1500 employees, with just around a dozen on the infrastructure team" \- what the rest 99% of the company is doing, marketing?

<comment><USER37> I was also thinking that those numbers are really weird, for a company which does infrastructure(fileservers)-in-the-cloud. Obviously some devs doing software for various platforms, business-side folks and so forth. But 1500:12 is a really weird ratio, given the core function of the business. If they're not spending on infra staff... what staff are they spending on?

<comment><USER38> Seriously, 12 people controlling all of that sounds like a liability. Are they allowed to meet all together?

</comment></comment></comment><comment><USER39> did Dropbox really only have 12 people in the infra team at the time?

</comment><comment><USER40> Can we remove /amp/ from the end of the URL? Some people still use HN on a laptop or a desktop.

Also, as mentioned elsewhere in the comments here, it should be retitled to "Why Dropbox decided to drop AWS"

<comment><USER41> I'm against Google hosted amp but what's the issue with self hosted? This page is _so_ much nicer than the overly busy normal TC site.

<comment><USER42> The problem is you making that decision for me.

<comment><USER43> Wait, but using the non-AMP version is a decision too. Why do _you_ get to make that one instead? Shouldn't the person posting the link get some discretion?

<comment><USER42> Principle of least surprise. techcrunch.com has a default experience, amp is secondary. Since we can't all read each others minds the default behavior should be to use the ...default.

</comment></comment></comment></comment></comment><comment><USER44> Um, what is this crap? When did they actually move? How long did it take?

The title of this HN link leads you to believe this is recent, but the article makes it seem like a multi-year effort, that in fact, could have finished a long time ago.

<comment><USER45> I agree. The title is misleading and should be changed.

<comment><USER44> But the blog itself lacks a lot of information.

The timeline is really, really significant here. Did they initiate this back in 2013? 2015?

My company has seen a remarkable shift in trust for online services change the last two years. Was Dropbox ahead of this curve? Were they someone who brought this change in mentality around?

This blog article is crap because without a basic sense of _when_ it could be that they were riding the coattails of other industry leaders. Or they were leading the charge.

<comment><USER46> It took several years and they finished early last year, much better article on Wired: [https://www.wired.com/2016/03/epic-story-dropboxs-exodus- ama...](https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud- empire/)

TechCrunch is only useful for headlines today, go elsewhere for actual content.

</comment></comment></comment></comment><comment><USER47> The title here isn't quite accurate.

Dropbox has moved user data from S3 to its own colocated data centers over the past few years, and is also doing compute in those data centers too. The compute actually existed for quite a long time - in the past you'd be talking to a Dropbox run server which would connect back to S3 to retrieve data.

Dropbox is definitely still an AWS customer, just not a major S3 or EC2 customer anymore. For example, all transactional email uses SES, and DNS is hosted on Route 53.</comment>