The microservices cargo cult - <AUTHOR> http://www.stavros.io/posts/microservices-cargo-cult/ <comment><USER2> Yes, a thousand times yes! Microservices are yet another tool in the box but they shouldn't be used on everything. And it makes no frigging sense for a startup or any new project to start with microservices...

The main advantage of microservices are in scaling and in reducing complexity of a big system but those advantages only make sense when you have enough traffic so that you have to scale or when your system has become complex enough to warrant microservices.

When first starting development, the most important thing is speed of development to get feedback from users as soon as possible. It's much faster to develop a clean, well optimized monolith than to spend a lot of time developing a whole bunch of micro services. And while thinking in term of microservices will help you to better conceptualize your software architecture, at this stage, you don't have all the informations needed to have a clear idea of what the final architecture will be and you'll often end up with microservices that are divided in suboptimal ways causing a lot of pain.

<comment><USER3> The article didn't even mention a lot of drawbacks :

Changing anything means changing half a dozen programs. Potentially in different languages.

Building any new feature into the program means you get to "walk the dependency tree", making sure no serialized stuff from a new version gets sent to an old version. Good luck with circular dependencies.

Related: deleting a field ... never going to happen. We're talking years of planning. Any field added ... ever ... has to be taken along for the ride for years. Oh and don't even think about deleting the code that interprets and supports the field for the same reason.

Also related: best of luck with the interactions between "oh we're not going to do that after all, sorry about asking you to already push that and update the database" and the "we really need to do this next thing right now" features. And the

Constant serialization overhead. People go overboard with this microservice crap and the result is that 99% of your program's cpu time is used putting objects to json and back (which is very expensive due to constant alloc'ing _), and you have 10-100 times the normal memory overhead.

Microservices should be like optimization : build your program without using them and then figure out where they'd make sense.

_ yes I know, you can sort-of avoid it these days with cap'n proto and flatbuffers

<comment><USER4> > Building any new feature into the program means you get to "walk the > dependency tree"

Same with monoliths, except it is often a worse experience. At least with microservices I _know_ the interface is all I have to worry about. In monoliths of any size inevitably someone has reached into parts of the program they shouldn't have just to get something 'done quickly'. And this is one of the main benefits of microservices - enforcing the interface boundaries.

> Related: deleting a field ... never going to happen. We're talking years of > planning. Any field added ... ever ... has to be taken along for the ride > for years. Oh and don't even think about deleting the code that interprets > and supports the field for the same reason.

That's just poor design and happens just as much in monoliths. The db is the almost always the challenge when removing a field. I could argue that microservices make it easier since the service providing access to that field could remove it from the db and then dummy it out until clients are updated. Also, why wouldn't someone remove the field from all the clients when removing it from the supplier?

With that said, I agree that microservices should be something that happens organically from a monolith. Think about an amoeba that reaches a certain size and only then do parts split off. I also think there is some ambiguity to what constitutes a _micro_ service. I'm sure my idea of proper granularity is different from others.

<comment><USER5> > At least with microservices I know the interface is all I have to worry > about.

You wrap your code inside a reusable library if you want to encapsulate it and expose only a small public interface. You don't start a new service for that.

<comment><USER6> It doesn't ever work that way in practice for larger applications though... once you have multiple teams with many developers working on something, those library boundaries fade, and tbh sometimes they should just be bypassed in favor of more shallow layers (for a monolith).

Once there are more than a handful of people working on anything no one person understands the details of the whole. Smaller, specialized services can at least act as and enforce those boundaries as upstream comments said.

<comment><USER7> > _" Once there are more than a handful of people working on anything no one > person understands the details of the whole. Smaller, specialized services > can at least act as and enforce those boundaries as upstream comments > said."_

Until you have so many services, that you also need someone that understands the details of all the different services interacting.

Really, if you can't control your developers from leaking through separative concerns, then they'll do it in any architecture. Monolithic or microservices- based.

<comment><USER6> I think you can have separated services by concern without getting too deep in terms of complexity though... I wouldn't use the term "micro" though... I also think if you are only doing mutation, you can combine them, or do libraries in a control service... there are lots of ways to skin the issue...

Digging down through 7 dll projects in one solution, then 11 in another to add a small feature that should take 10 minutes taking 4 days isn't so much fun... same would go for 30+ service projects...

I'm also not big on starting with smaller/micro services either.. or ORM, or DI/IoC for that matter... Build the monolith and break pieces off as needed. I'm fine with that... I was saying it's easier to herd cats when they're not in the same space as eachother.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> I'm in the middle of a microservices mess than was forced upon us. I have nothing positive to say. If you're in the SaaS space already and it's not a greenfield project, it's orders of magnitude better to deploy lots of smaller identical monoliths than it is to try and build and deploy lots of services and manage the contracts and complexity between them.

Major problems I've seen are: per transaction performance sucks due to the network or IPC channels, development friction, logical complexity, infrastructure complexity, managing contracts between services, debugging failures, monitoring performance, bootstrapping new staff and the biggest of the lot: headspace.

If you want to succeed, at least in the short term, just keep your monolith tight and fast and without sprawling infrastructure requirements. Single machine, single process, single storage engine (or database), single messaging system. Then scale that to multiple instances. If your site deployment requires at least 20 machines due to sprawl, you're going to be even more screwed when you throw microservices at it, not less. If your application is incredibly complex, it's not going to work either. The problem domain needs to be small and easy to consider as it's difficult to cleanly extract a chunk of your average monolith into a standalone concern.

There are also people with technical authority in many companies who blindly follow the latest fad without real consideration of suitability, risk assessment or accountability. If someone starts waving microservices, AWS and everything else around, they need to fight their position and everyone needs to assume that isn't the default end game.

<comment><USER9>
      > it's orders of magnitude better to deploy lots of smaller identical monoliths than it is to try and build and deploy lots of services and manage the contracts and complexity between them.

This article from 2003 by Martin Fowler is exactly about that: [http://www.drdobbs.com/errant- architectures/184414966](http://www.drdobbs.com/errant- architectures/184414966)

<comment><USER8> Indeed. I remember reading that back then!

</comment></comment></comment><comment><USER9> Microservices are advertised as a means to modularization, but it's what programming language modules are for - they are defined on source code level and can be freely used in different runtime components without network/ops/version-management headaches. When you have your module defined that way, you can think of exposing it as a microservice because this may make sense for your use case.

Imagine that each Python module runs as a microservice. For many modules this would lead to huge performance degradation, for example a regexp module can be called thousands times per second, the running time of a call is usually short and replacing an in-process call with a network call will give 100-1000x slowdown.

But if you take a different use case of the same module - complex regexps running on large texts, potentially causing out-of-memory errors, then packing the module into a microservice can make sense - separate processes can have large caches, an out-of-memory error terminates an instance of a microservice only and not the calling process.

Generally I think the advice should be to always use source code modules in the first place, and create microservices using these modules for specific use cases only involving runtime needs like caching, fault tolerance, scalability.

<comment><USER8> Most of the runtime needs in this space can be managed with circuit breakers without having to provide separate processes or memory spaces, be they temporal or resource based.

For example your regex, could be given 20ms to run and use a maximum of 8Mb of heap before it is interrupted. This can happen on a single thread and drop back to the caller if an exception is thrown. I'd love to see a language feature which defines the maximum stack and heap for a particular scope i.e. in C#:

       Breaker.Heap(8.MiB(), () => {
           Breaker.Time(200.Milliseconds(), () => {
               // risky operation
           });
       });

(we already do the time breaker, but not the heap)

Edit: correct calling convention.

Edit 2: add missing extension method brackets

<comment><USER9> Interesting. Is the circuit breaker a feature of .NET running on Windows?

To my knowledge, it can't be implemented on Linux + Java or Python (a thread can't be terminated from outside, and some syscalls involve a whole process).

<comment><USER8> It's a library I wrote.

The time breaker is actually quite complicated. It is a wrapper that sets up some global parameters on the thread for timeouts on async/await calls and handles the timeout conditions. It integrates with our own async wrappers for external http calls, message delivery, query execution etc. It only enforces that all aggregate async calls will complete or fail by the end of the timeout period. Realistically this is usually around the 500-800ms space as load spikes can break everything otherwise.

</comment></comment></comment></comment><comment><USER10> This leaves open the question of what _are_ microservices. Are they _of necessity_ completely isolated units deployed with Docker and Kubernates on a CoreOS cluster and communicating only via HTTP each with independent databases? Yes this seems a bit much for most projects.

There are midway options though. Even the lowly batch job is a good way to get some of the decoupling without having to go "all-in". I find batch jobs and message queues give me 80% of the benefit of "microservices" with only 5% of the pain.

In fact someone needs to write an article on "levels" of "microserviceness", (which certainly has multiple dimensions and branches) and point out the benefits and drawbacks of each level.

Of course the end game being: "a Docker container for each line of code."

<comment><USER10> > a Docker container for each line of code

Yes, this is the surest way for developers to guarantee the code itself is always bug-free. Everything becomes "just an ops problem".

2017 predictions: buzzy job title: Ops-Dev. Labview-style graphical programming [http://www.ni.com/newsletter/app/largeimage?lang=en&imageurl...](http://www.ni.com/newsletter/app/largeimage?lang=en&imageurl=%2Fcms%2Fimages%2Fdevzone%2Fpub%2Fnrjsxmfm912163998723206173.jpg) becomes the new hotness. Amazon and Google introduce new graphical programming workflow coordination services for lambda. Nothing ever gets finished.

(2018 predictions: google discontinues this service)

(2019 predictions: every machine instruction now belongs to its own docker package; the successor to Kubernates is marketed as a distributed asynchronous virtual CPU. Data centers begin requiring small nuclear stations to generate the electricity to power them. Still nothing gets finished).

(2020 predictions: someone rediscovers this "heroku" thing; things start getting done again. Hot new industry: nuclear waste disposal)

(2119 prediction: nuclear stations powering 2019's data centers are still running at full power but servers are all dormant; nobody wants to turn off a server "just in case").

<comment><USER11> "graphical programming workflow coordination services"

Those things are always sold to CIO types based on the idea that "you won't need developers" and then you end up with some eldritch graphical abomination where being being able to visualise things just makes it worse....

Draw a rough diagram and then write your code - fine, generate a diagram from code - fine, generate code from a diagram - nightmare.

</comment></comment></comment><comment><USER12> I think this article isn't very useful. It's unfortunate that we have this human need to blow things up and then deflate them. I dislike the term "microservices" because for most organizations all it does is plant a flag all the way across the yard from "monolithic." Suddenly the guys at the top have a new buzzword and the engineers are all trying to make their services as small as possible so they're micro enough.

Most of the gotchas the article mentions aren't logical consequences of decomposing into smaller services at all. You don't have to have different data stores for each service. You don't need to "marshal data" between services. If a service needs to call a service it's just a client like any other client, so if we want to call standard http request/response handling "marshaling" I guess it will sound more complex and scary. Breaking a monolithic app into smaller pieces doesn't increase complexity, it reduces it. And to the extent you have more things to monitor that probably means you can now monitor and control things that were more or less invisible outside the log data in the monolithic architecture.

More importantly, decomposing a problem into logically related areas of functionality that can execute separately allows you to make the most efficient use of compute resources, and it is consistent with the idea of favoring multi-processing over multi-threading. In almost every way groups of simpler things collaborating makes much more sense than large complicated things that do all. It's only when we create these Knights in shining armor that people start feeling like they have to be knocked off their horses. Use the tools and techniques that make sense.

</comment><comment><USER13> Some of the weaknesses can be tempered by not using HTTP to communicate between the microservices:

\- "slowdowns on the order of 1000%"

\- " bunch of code necessary to marshal/unmarshal data [...] there are always dragons in there.

And also problems of versioning, data integrity, etc.

I've had those problems in a microservices architecture. That's things that are solved by protobuf[0]. Your servers exchange small, efficient structured data and you get tons of other benefits ({un,}marshaling for free, integrity, versioning, ...).

Potential downside: a language you want to use having no protobuf API.

Finally, I see another downside to the microservices architecture: it may be decided that the smaller, decoupled code bases should be stored in multiple CVS repos. Which turns into a nightmare: a single bugfix may span across multiple repos and there is no clean built-in way to links commits across them, you still should sync the interfaces (e.g. with git submodules), etc. This is a thing I've witnessed firsthand, and proposals to merge the repos were dismissed since "We [were] using a microservices architecture". Yes, it's a mistaken implementation of the microservices paradigm, but it still happens.

edit: I recommend protobuf not by preference over other equivalent solutions, but because it's the only one I know and have used. Alternatives are evoked below.

0: [https://developers.google.com/protocol- buffers/](https://developers.google.com/protocol-buffers/)

<comment><USER14> I really recommend against using protobuffs. There are long standing bugs that Google just refuses to fix in the public version. I can't remember what they are off the top of my head, but I know a semi-prominent YC company that uses them and they pull their hair out all the time.

Just use zerorpc. It's more reliable than zeromq + protobuffs and it comes with a bunch of freebies, like built in heartbeats, streamed responses, etc.

<comment><USER15> How about Cap'n Proto? Their marketing is great, but I don't recall seeing many experiences with it, so I'm asking for feedbacks here.

<comment><USER14> First I'm hearing of it. Looks really promising, but without any experience with it I cannot really say.

<comment><USER16> They wrote it specifically for sandstorm.io, so they're dog-fooding it.

</comment></comment></comment></comment></comment><comment><USER17> So I think there is a parallel to this with the whole "everything must be in tiny modules on npm" movement in the JavaScript community. If you do this, you end up with lots of repositories, a rigid separation of concerns but a network between you and your dependencies that will get hit a lot unless you wield some npm link sorcery, and a bunch of extra complexity.

A modular monolith application is what people have been writing since people thought up the notion of modules. Enforce proper discipline when building your app out and you won't need these physical walls between your functional areas.

I'm currently reading SICP, and the notion of using "block structure" in Lisp to compartmentalize and encapsulate functional areas of code is introduced in _Chapter 1_.

Get the basic stuff right before you start introducing complex systems to split up your software.

<comment><USER18> And there's probably a parallel to "one file per type" with tons of little types. It feels like you're making progress and doing stuff. You've not just done one task, you've deployed changes to 20 services. It also feels like you've made things more abstract than they actually are. It also feels more like Real Serious Programming.

I've used a Twitter API client (works fine) that implements itself across _7_ separate libraries. All same project, they all work together, and they aren't used in any other system. Just separated things out for the fun of it. One library for Twitter client "Factories". One for Twitter "Credentials". Another for Twitter client "Security". Zero benefit to the user or to the project. But it certainly makes things seem more important, eh?

<comment><USER19> It totally makes sense to have separate libraries even for one project if they can be implemented by different people and tested separately. I'd shout for that design any time of the moon.

<comment><USER18> They can't be deployed separately. There's no point in splitting them up. Testing and development aren't affected.

<comment><USER20> Deployment is not the only thing that matters. Development is also important. I am actually working on an interpreter for a web scraping tool, where I'm working on the interpreter and my friend on the IDE. The likelihood of them ever being deployed independently is low but the separation is helping us focus.

</comment></comment></comment></comment></comment><comment><USER21> One thing that is usually overlooked: do microservices fit your team structure? A team who spends all day together can manage a monolith very well, synchronous releases (everything-at-once) are not a problem. If you don't need 24/7, then it's even better.

However, if you're a distributed team (maybe across timezones), quick discussions are difficult and 'costly', then microservices might worth the effort. Managing the deployment and operations is more difficult but sometimes much less coordination is needed when people communicate through APIs and not Skype and Slack.

<comment><USER22> As others have already pointed out, you don't need microservices to achieve a good modularization of your code. Then have each team work on a different module, with clearly defined APIs between them. No need to separate them with HTTP calls.

<comment><USER23> The main benefit of microservices is that each team can _deploy_ their service independently.

This is really useful to reduce the level of required of interaction (and pressure) between teams.

<comment><USER5> It may reduce development interaction, but it'll certainly turn testing into a nightmare.

If you have teams that communicate badly, you'll need people specialized in deployment and assigning issues to them. That's not a good situation anyway, but it's the less worse of them.

</comment></comment></comment></comment><comment><USER24> Having read one of the success stories linked in the beginning ([http://philcalcado.com/2015/09/08/how_we_ended_up_with_micro...](http://philcalcado.com/2015/09/08/how_we_ended_up_with_microservices.html)) I think I am starting to get it. That was a rather candid article describing a team which after having got burned by a sprawling "monorail" architecture decided to split it up into services (nothing micro- there) based on _organizational_ viewpoint (if there is some knowledge about a part of application shared by 3-4 engineers it makes sense to split it into a separate module so it can be developed faster). And as I am inferring from the pictures their services don't call each other much so it is really close to a modular monolith. So if "migrating to microservices architecture" really means "splitting that damned monorail into something more manageable" then it is a really good idea after some point.

</comment><comment><USER25> This phenomenon is just part of human nature. Same thing happened with OOP, design patterns, TDD, etc.

To apply microservices effectively, you should first build the monolith, modularizing at the source code level and adding choking points as needed. Over time, microservices will naturally roll off the monolith not unlike boulders rolling off mountains after rain or earthquake. Don't go dynamiting in anticipation.

<comment><USER26> You don't necessarily need to _build_ the monolith - _design_ it first, for sure, but then pick apart the design and see where the individual components lie.

<comment><USER25> Right. And I'm saying most folks either don't know how to do that or can't because of constraints like time and fluidity. Design one starts and ends up with are more often than not miles apart.

</comment></comment></comment><comment><USER27> I can't help but think that much of the intent behind "cargo-culting" is simply people building up their resumes for future career development.

If you want to work in a sexy new technology, but you need to develop experience in that new stuff to be marketable it is totally understandable to try to build up skills by forcing the implementation of over-sized solutions.

In other words, many employers aren't willing to take on folks if they don't have the requisite experience on some new stack and that compels folks to gain that experience anyway they can, including "cargo-culting" stuff that isn't necessary just for the experience gain.

<comment><USER28> That hasn't been my experience, from the people I've seen. It's mostly just excited engineers wanting to play with a new thing (which is great) but doing that in the most available place, their job (which is bad).

The advice I give them is "do whatever you want in your house (or your side project), but critically evaluate your business needs and only use what makes sense for your business".

Personally, I have a very low-traffic guinea pig side-project that I like working on, and I just try every new thing there.

<comment><USER27> Well, yes, but I would say that the desire to "play with a new thing" is very much tied to career development.

<comment><USER28> The people I know aren't doing it in a conscious way to further their career, they just like playing with new things. The career advancement is incidental.

<comment><USER29> I'm willing to bet that it is a bit of both.

</comment></comment></comment></comment></comment><comment><USER30> The article is a little weak, but well worth the read.

I love the microservices concept, but fair warning: as bad as OO has gotten over the past 20-30 years, microservices promise to be even uglier.

Why? Because not only are you mucking around in the code, _you 're also mucking around in how everything connects to everything else in your cloud_.

Just like we saw vendors come out with click-and-drag ways to create new classes, now we're seeing vendors start to sell "pre-finished" microservices. Get the disk out of the box, boot it up, fill out a couple of forms, and voila! Now you have microservices.

That worries the living crap out of me because microservices _are_ the architecture of the future. You just can't get from here to there using a magic bullet. Learn you some pure FP, make everything composable using the Unix Philosophy, and keep your LOC to a bare minimum. Toss off every damn thing you don't need.

As much as I know they are the way forward, I have a bad feeling that consultants will have plenty of billable time coming up straightening out a lot of messes.

</comment><comment><USER31> Rule of thumb: Divide number of full-time backend engineers by 5 and you get ideal number of microservices :-).

Too many microservices is a complexity mess, too little means you have a monolith that is hard to iterate on.

<comment><USER28> Looks like I can afford half a service, then :P

<comment><USER31> Looks like you can afford to have monolith and don't worry about this microservice thing :-P.

</comment></comment></comment><comment><USER32> I don't agree that scaling up is an unqualified advantage of microservices. In practice you have to co-ordinate communication between your services. Though some of this communication will be asynchronous publish-subscribe (ie: speed unimportant) some communication will necessarily be synchronous.

If one heavy part of your rails app takes up 90% of the processing time, there is nothing wrong with just getting a bigger machine for the whole app. The bigger CPU/memory/whatever will be spent on the heavy part and the rest will be normal.

For most business, scaling is not a problem - they can just get bigger machines. Having to re-implement transactions across your microservice architecture really is a problem. Very often transactions need to cross microservice boundaries and that really requires a lot of thought

<comment><USER19> >> that really requires a lot of thought

Yes, sure, but that's the resources spent right. Not the monolithic spaghetti crap you waste time on trying to figure out what went wrong - 99% businesses everyday activity

<comment><USER33> Spent right? Implementing a protocol to coordinate transactions (if it is possible at all, depending on the various backends you work with) to get what you could have already done in your database?

<comment><USER19> You can keep transactions in one part of the system and the queries in another. CQRS as one example.

</comment></comment></comment></comment><comment><USER34> I see the strengths and weaknesses in the article, and the complaints from all the comments here, but I still find the trade off of microservices worth it. It requires sophisticated ops and well defined deployment tools and dev environments, but we have to handle ten billion requests a month on our stack. The ease at which we handle that scale, and the speed at which engineers get to iterate and deploy makes microservices all the more worth it.

<comment><USER35> If you used those sophisticated ops and well defined deployment tools and dev environments to develop a monolith, wouldn't it be just as easy?

As for scale: are you suggesting that having extra code executing and network calls being made in your system makes it _more_ scalable, rather than less?

Ten billion requsts a month is 3805 requests per second on average; i'd guess that means 10 000 requests per second in the peaks (correct me if i'm wrong!). Is this considered challenging scale today? I'd buy four DL380s and call it done.

<comment><USER34> >If you used those sophisticated ops and well defined deployment tools and dev environments to develop a monolith, wouldn't it be just as easy?

No, it wouldn't allow us to be as fast. Most of our services are under <200LOCs (not a policy, just happens to be the point where people seem to split things out). The idea is that any service can be rewritten completely in a few days.

There are no tie ins to any platform, compiler version, syntax, or language. This might sound like chaos, but it's a huge productivity gain, as I feel full ownership over features I write. Naturally, we aim for good docs and code coverage, and use continuous deployment and integration tools to keep everything green.

As for scale, any microservice can be run across any number of instances without having to scale up the entire platform. This allows us to identify hot areas and deal with them effectively.

We don't use network calls (well, not HTTP or TCP) to communicate between services. Services themselves are pretty transport-independent and work well over tcp, but NATS is the transport of choice at the moment for inter-service communication.

</comment></comment></comment><comment><USER36> > Data segregation: Since all your data now lives in different data stores, > you're responsible for relationships between data. What would be a simple > cascading delete in a monolith is now a complicated symphony of > dependencies, calls and verifications.

IMHO, this is the biggest problem with microservices: "Transactions" are not available in a microservice environment. You'll have to work really hard to get anything that comes close.

<comment><USER37> That sounds more like one example of a suboptimal design than a valid indictment of whole paradigm.

</comment></comment><comment><USER38> I think a cargo cult also has something to do with signaling, sort of like a status symbol ("They can't really be Y if they are not X!" -> "Look at us how X we are!"). It's a self-reinforcing meme that is used as a heuristic for value estimation, but usually fails catastrophically because of its heuristic and self-reinforcing nature.

<comment><USER39> I think that's more a result of the kind of person that adopts a cargo cult mentality.

<comment><USER38> What do you mean exactly? I mean that the status symbol aspect of a cargo cult is a crucial part of its self-reinforcement mechanism.

</comment></comment></comment><comment><USER40> I don't get the trend to split up everything as micro as possible.

Use a proper framework like Symfony (or if, like many people, all you want is a CMS, Drupal) supporting MySQL master-slave or multi-master replication and separation of web frontend and file hosting, host it on AWS (or plain old dedicated servers), put in Cloudflare if you're scared of DDoS kids, and be done. If you need SSO use either provided SSO plugins or an LDAP backend if the SSO is only required for various platforms provided by you.

Said architecture can be built and run on a single server and if you're dealing with spikes you just spin up a couple frontend servers and be done.

<comment><USER41> I don't think you get it... In most cases we are talking about applications beyond the complexity of a configurable CMS. Also assuming that replication solves your (potential) database bottlenecks does not account for write-heavy scaling.

<comment><USER40> Many people (and especially consultants) over-exaggerate the customer's demand for a complex system.

I have seen people shipping SAP to small brick-and-mortar stores with a tiny webshop...

</comment></comment></comment><comment><USER42> Agree with the premise. An excellent example premature optimisation or YAGNI. An alternative is to factor your code by business capability / bounded context as microservices endorses. Factor the code as such but don't deploy the logical partitions as physical ones.

Keep it all in one deployable artefact, in-process for as long as you possibly can. Use an in-proc message bus first, don't dive into Rabbit until you know you need it. As soon as you require that infrastructure cost for http, mq, monitoring a ballooning of boxes / VMs, deployment complications you'll notice the spike in operational expenditure.

Grow you architecture organically.

</comment><comment><USER43> If I learned one thing in Software Engineering it's "modularization matters most". And microservices seem to be the epitome of this concept.

If you have to work with different people, you need a way to minimize dependencies between them.

Also, the more encapsulated things are, the less the starting skill of a person matters. You just need people who get things done. Later you can switch out the bad modules easily. Which is a huge economic factor.

I can't count the hours I spent with fixing horrible monoliths and the years it took to replace them.

But if there is a horrible microservice, you can do this in a fraction of time.

<comment><USER44> Sadly, often the reason you need to work with different people is that you've architected your product to be so complicated that it needs lots of guys working on it.

It's amazing how much a team of one can do if you don't saddle said team with arbitrary complexity such as a microservices architecture. Maybe you'll need to scale to that level one day. But you'll definitely want to ship. One guy and a sane architecture can do that.

</comment></comment><comment><USER45> TL;DR Microservices have their place, and can be useful for certain environments, but they are not a fix-all.

They can be pretty nice for multi-tenanted development environments. Sure, you could use any of the other isolation techniques, but being able to provide an environment that can be started quickly (and somewhat easily depending on the rest of the services required). Not to mention that the popularity of container systems and their ease in understanding (Dockerfile vs RPM spec) means that other people can hack away at the dev environment without having to know the ins and outs of building proper packages (although they should learn).

Now, for a production environment, I would never move to a microservices architecture for the reasons listed in the article and my own dislike for adding overhead and complexity to solve "issues" that can be easily dealt with using tools that have existed for years (proper packaging with dependencies etc..).

</comment><comment><USER46> I agree with the YAGNI-ish approach, but talking about micro services as if they provide modularity is entirely off-base. The decision to use micro services should be driven by _scalability_ , not _modularity_. If you're saying that it's going to be terrible to make changes to your codebase, simply bolting micro services on top of that is going to make things worse.

A well-designed micro service architecture is modular in that each micro service is basically a nice wrapper around either a query or an update. But you can organize your application into an API of queries and updates without micro services.

To be honest, if you don't at least intuitively understand this, you have no business architecting a production system large enough that this matters.

</comment><comment><USER47> Many of the advantages of microservices can be achieved by refactoring your monolith code to be less monolithic.

I would suggest using functional styles wherever possible, plenty of isolated unit testable code, and a hexagonal architecture [http://alistair.cockburn.us/Hexagonal+architecture](http://alistair.cockburn.us/Hexagonal+architecture) that pushes all the I/O, mutation, side effects, etc. to the very boundary of your code. Also see Gary Bernhardt's "Boundaries" talk for more interesting thought in that vein [https://www.youtube.com/watch?v=yTkzNHF6rMs](https://www.youtube.com/watch?v=yTkzNHF6rMs)

</comment><comment><USER48> There are also a distinct lack of tools for debugging co-ordination and scheduling problems in a _micro-service_ (or as they used to call it in my day, Service Oriented Architecture) system.

In an asynchronous RPC scenario, does Microservice A listen for the appropriate response message from Microservice B before continuing work on Request X99? Does it respond to all messages in the appropriate order? What happens in a cascading failure scenario when the back-end system Microservice B relies on is taking too long due to bad hardware/burst traffic/DDOS/resource contention?

Do you have tools that can analyze your program for critical sections where you need explicit locking and ordering mechanisms? Do you have analysis tools that provide guarantees that your fancy distributed architecture is complete/correct?

These are just a sample of the things OpenStack has to think about -- a micro- service architecture for managing, orchestrating, and authenticating access to data-center resources. It's a hard, hard problem and an on-going effort by thousands of well-paid engineers across the globe to get right.

I have no doubt that a small team of talented developers could stand up a system of APIs around their core services to get a system running. However I can guarantee that they will be making huge trade-offs in terms of correctness and reliability.

At least with a _monolith_ (is that a pejorative?) application you do have tools to analyze and debug your code that work well and have been battle- tested for a couple of decades. I suspect you would produce fewer bugs if you were constrained for developer talent and time.

</comment><comment><USER49> Micro service architecture is good to evolve a monolith project who need to scale when dealing with a huge amount of calls. It's great to be able to experiment with different implementations and technologies, do A/B testing. It enforces to have single responsibility modules at architecture level avoiding bad practices if you are dealing with different/remote dev teams.

You have challenges, though. One of them is when implementing micro services you need a cultural change in your business to be able to adapt to the change. You need to deal with more complex architecture, you need to implement your own solution to deal with the architecture, spend time defining a dev ops culture if there is none, ...

Businesses are usually pretty different between others so you can not expect to have the same solution to deal with your problems (For example, using Netflix approach as a silver-bullet solution).

I've heard so many times the concept "micro services" as the goal as same as "big data" as the solution. Again, we should analyze what is our problem and what we want to solve before selling the new shiny thing and making things over complicated.

<comment><USER39> Do you think you can do microservices without devops? I see the correlation you're making but I think a more traditional organizational structure can work fine.

<comment><USER49> I don't think it's impossible but harder. As far as you have great collaboration and communication between teams probably you can achieve the same.

</comment></comment></comment><comment><USER50> Martin Fowler identified a lot of the same tradeoffs in this video: [https://www.youtube.com/watch?v=2yko4TbC8cI](https://www.youtube.com/watch?v=2yko4TbC8cI)

One benefit I haven't seen mentioned yet: microservices are effective at reducing the mental "page size" when working on any particular part of the system.

</comment><comment><USER39> I think some of these points are gross exaggerations.

> You immediately increase the things your servers have to do tenfold.

Really? It's _ten times_ as much work to implement microservices?

> Personally, I've seen slowdowns on the order of 1000% when moving to > microservices (yes, ten times slower).

Then you implemented your microservices _wrong_.

I think that the author's understanding of the goals and purposes of microservices is maybe a bit misguided. Microservices are about front-loading scaling problems, not about having a clean architecture or smaller codebase. If you never need to scale, you don't need microservices (but you're probably wrong).

The flowchart at the end of the post really underscores for me that this author's argument is not genuine. He holds up this shibboleth of a "monolithic" architecture, something that doesn't really exist in 2015.

<comment><USER51> > Really? It's ten times as much work to implement microservices?

No, it says the _servers_ have to do tenfold more work, not _you_ to implement them. Whether that's correct or not is another discussion.

</comment></comment><comment><USER52> The application I work on most of the time is largely monolithic and usually I have no problems with that. Some parts have been extracted to their own codebases and are deployed separately because of performance reasons.

We also separated the main backend/API codebase from the frontend mostly because the frontend devs work prefer to work within the Node ecosystem instead of Python/Django and so that we don't have to think too much about synchronizing deployments. The tests for the backend code take quite long to run as well compared to the frontend tests, so having this separation is nice for the frontend devs that way too.

What I some times would like to have better infrastructure support for though is throwaway prototypes/projects that can live in their own codebases and have access to all the regular databases, blob storage and so on, as well as databases that are private to the prototype that I can do whatever with with no risk of doing something bad to the important databases/storage.

I would also like these prototypes to be able to register themselves with the load balancer to take care of everything under `/halloween-experiment/` for example and have the load balancer add headers like `X-UserEmail`, `X-UserID`, `X-IsEmployee`, and so on so that I don't have to implement authentication/authorization in every prototype.

Today these types of prototypes need to live next to the "important" code so that they can use the same CI pipeline and easily be made public or visible to employees and use real data.

I'm following projects like [https://getkong.org/](https://getkong.org/) with interest, and together with everything happening around Docker such as EC2 container service or Kubernetes, as well as projects for service discover/configuration like etcd or Consul, it feels like we're getting there. There are just so many projects to keep track of, and you need to figure out how to make them all part of your CI pipeline. :)

</comment><comment><USER53> I love his little worflow diagram embedded in the article: [http://media.korokithakis.net/images/microservices-cargo- cul...](http://media.korokithakis.net/images/microservices-cargo- cult/flowchart.png)

<comment><USER54> I wonder if the box on the right should read "You need automated regression tests"

<comment><USER55> It should read "split up your monolith".

Just because one extreme isn't working for you doesn't automatically mean the other extreme is the right solution.

</comment></comment></comment><comment><USER56> Not only that, the microservices is just SUN-RPC, CORBA reborn and we all know how they worked out.

</comment><comment><USER57> Develop as a monolith, deploy as services: engines.

[http://tech.taskrabbit.com/blog/2014/02/11/rails-4-engines/](http://tech.taskrabbit.com/blog/2014/02/11/rails-4-engines/)

</comment><comment><USER58> "As with everything, there are advantages and disadvantages"

The author focuses on microservices, however, I think there is a larger point to be made. It is not that some particular architectural pattern is bad or good, it's that when you don't fully consider the requirements of your application and apply some pattern or technology just because it's the hot item this week you are going to end up with problems. This has less to do with microservices, in my experience, and more to do with less technical managers making decisions for a project when they don't fully understand.

</comment><comment><USER59> Damn! Now what is left as the next big thing?

<comment><USER28> Whatever was the next big thing after SOA! Just look at the Slashdot archives for 1990, we can probably figure it out quickly.

<comment><USER60> A micro-services service-directory micro-service?

Maybe we should call it MDDI?

</comment></comment></comment><comment><USER61> At my job we've avoided microservices thus far. 90% of our deployments are just to Heroku. Every now and then I lament in my mind that we aren't using the coolest new tools (Docker, microservices and all the things that come with), but what we have works _really well_ , and we can easily scale up by 10x and things will still work.

Every time I think of the mess that it will cause to break up things to microservices, I'm glad we aren't doing it- yet. When the time comes, we'll roll out to services as-needed, but that day isn't today.

</comment><comment><USER62> The problem with microservices for us has been the composition of operations. Yeah we use the Rx* observable patterns and it helps but the code is still non intuitive for new developers if the language is pretty much procedural/imperative. Even with languages like Scala it still gets confusing. Even if you have a language where threads are cheap (golang) you still have to compose the operations.

I have been meaning to see if there are microservice like frameworks for Haskell similar to Hystrix (which is what we use).

</comment><comment><USER63> Yet another article that misses a huge aspect of microservices: _Reusability_. (I'm going to borrow from an older comment [1] here.)

Almost all of the swathe of microservices we've developed internally are general-purpose. We've built a dozen or more user-facing apps on top of them. If I wanted to build a new app today, I would typically sit down and write a Node + React app, configure some backends, and I'd be done. I don't need to write a new back end because I can just call our existing services.

If you look at what a modern web app is, most apps these days are actually stupidly similar. They typically need things like:

* User accounts

* Authorization with existing OAuth providers (e.g. Facebook)

* Some kind of database to store and search structured content

* Notifications (email, text, push)

* Storing images or video

* Syncing data from external sources

* Analytics

We have generalized, reusable microservices that do all of this.

Let's say I want to build a HN-type link aggregator with comments. I will use our document store to store the links and the comments in a nice hierarchical structure. I will use our login microservice that mediates between an identity data model and an OAuth account registry. I can use our tiny microservice devoted to recording up-/downvotes. I can use our analytics backend to record high-level events on every UI interaction.

I can write this without a single new line of backend code.

This ability to "pick and mix" functionality you need is the real, largely undiscovered beauty of microservices, in my opinion. It's the same idea that makes AWS attractive to many people; you're building on the foundation of thousands and thousands of work and reusing it.

We just whipped up a new site recently where 95% of the work was purely on the UI, since all the backend parts already existed. The remaining 5% was just code to get data to the system from a third-party source, plus some configuration.

Reusability requires that you plan every microservices to be flexible and multitenant from day one. It's a challenge, but not actually a big one.

Is it possible to do this monolithically? Sure. I would be afraid of touching such a beast. We have very few issues with code devolving into "legacy", for example; the strict shared-nothing APIs ensure that half-baked, client- specific hacks don't sneak into the codebase. If anything messy happens, it happens in the client app, and that's where it should happen. Eventually you'll throw the app away, but the backends remain.

</comment><comment><USER64> Of course microservices are just another tool in the toolbox.

I think what's frustrating is the lack of support in moving from a monolith to a microservice architecture. I haven't built a lot of them myself, but it feels like you're rolling your own framework/architecture whenever you need to make the transition. Is that anyone else's experience, or is it just not possible to codify best practices?

<comment><USER65> Currently in the process of separating several monolithic applications into services. We've found success in identifying individual "resources" that are present and common in each of the monoliths. Once identified, we implement that resource end-point in our service and swap out the existing code with calls to the service. Divide and conquer.

</comment></comment><comment><USER66> The "micro" in microservices is the issue. It reminds me of the "No-SQL" movement. The truth is that EVERYONE has a multi-tiered architecture. The only question is how many tiers you need. It's always more than 1.

<comment><USER67> I think one thing that microservice and some other models correctly point to is that "tiers" of the traditional 2-tier, 3-tier, and n-tier architectures aren't universal, and that their are good reasons for some systems to have components that are arranged in some manner other than stacked, front-to-back layers.

</comment></comment><comment><USER68> If you design your software with a bad architecture you will have problems, if the services and their data are 'cut' in the wrong way you will get performance (and other) problems.

That's valid for functions, state, api and service stores.

</comment><comment><USER69> The thing that nobody addresses and especially not the micro services gurus is how do you know where and what to split into micro services.

When does splitting a service add enough value that it is worth the cost of performance and added complexity?

<comment><USER70> We should grab lunch to chat about this.

Simply put, you split a microservice when you need to split teams.

Microservices aren't a solution to a technical problem, they're a solution to a social/organization problem (described by Conway's law).

</comment></comment><comment><USER71> I think it is fascinating how an idea can emerge with a fuzzy meaning and, in the space of 2 years, become rigidly associated with a narrow set of technologies which will surely be much more temporary than the idea itself, thus forcing people, after 3 or 4 more years, to come up with a new word for roughly the same idea.

In the summer of 2013 I was working at Timeout.com and we were trying to reinvent the architecture of the site. Timeout.com had spent several years using the PHP framework Symfony to build a massive monolithic CMS, and the thing was a disaster. It was shockingly slow. If you ssh'ed to inside the datacenter and then tested the response time of the system, under ideal conditions, from one computer in the data center to another computer in the data center, then the average response time was 10 seconds!

This lead to a long internal debate. I advocated for what I called "An architecture of small apps", because at that time none of us had ever heard the word "microservices". I did not hear that word until March of 2014, when Martin Fowler wrote his essay:

[http://martinfowler.com/articles/microservices.html](http://martinfowler.com/articles/microservices.html)

But back in the summer of 2013, with permission, I published the whole internal debate that we had had at Timeout.com:

[http://www.smashcompany.com/technology/an-architecture-of- sm...](http://www.smashcompany.com/technology/an-architecture-of-small-apps)

You will notice that you don't see the word "Docker" in my essay, nor do you see it in Martin Fowler's essay. And in my essay, I suggest we use ZeroMQ to bind our apps together.

But 2 years after we had our internal debate, I've noticed that more and more people now associate "microservices" with a very specific set of implementation details: Docker, Kubernates, HTTP and Service Discovery.

I acknowledge that these 4 technologies can be combined in very powerful ways. I currently work at the startup incubator run by NYU, and I get to eavesdrop on what the folks at lsq.io doing, since they sit next to me. And I get that Pelly is a frighteningly smart guy doing extremely cutting-edge stuff. I totally admire everything they are doing.

However, I personally feel that I'm following a microservices strategy, and yet what I'm building is still a lot like what I described in my essay of 2013.

July 30th, 2013 [http://www.smashcompany.com/technology/an-architecture-of- sm...](http://www.smashcompany.com/technology/an-architecture-of-small-apps)

</comment><comment><USER72> Example of micro services that just work (tm): Internet.

</comment><comment><USER73> monolithfirst.com

</comment><comment><USER73> monolithfirst.com

fight the future</comment>


A Letter from Dijkstra on APL (1982) - <AUTHOR> http://www.jsoftware.com/papers/Dijkstra_Letter.htm <comment><USER3> The APL family languages (and their users) have always fascinated me somewhat because they seemingly go against the common wisdom of what it means for a programming language or code written in it to be "readable" \--- i.e. instead of the widespread notion that good code should be like English, it turns that concept around completely, treating a programming language like an actual language. APL is as foreign and unreadable to a typical Westerner as Chinese, but just like with natural langauges, you realise that there are many who have learned and are proficient at using them.

In other words, with many programming languages it's "the code is unreadable because it's not sufficiently English-like"; with APL, it's "the code is unreadable because you don't know the language."

<comment><USER4> You see similar cases being made with languages like q, the programming and query language for the kdb database. Arthur Whitney, the q-father, built an apl dialect called a+ at Morgan Stanley, which is still in use.

RTL evaluation semantics aside, the argument I've heard for the terseness of the q language is that the interpreter is tiny so it fits inside a page. Okay, great, so use your words and kill the built-in webserver.

<comment><USER5> .z.ph is extremely convenient. I'd be horrified if he removed it. I'm sure its just a line or two anyway. :)

</comment></comment></comment><comment><USER6> Did you know...

Ken Thompson wrote an APL interpreter for early Research UNIX.

source: [http://www.cs.dartmouth.edu/~doug/reader.pdf](http://www.cs.dartmouth.edu/~doug/reader.pdf)

McIlroy:

"APL influenced the development of pipes. APL did not allow the use of operators with variants, which many utilities had at the time. It only took a willingness to throw in a new separator, the vertical bar. About four years passed, from the time they started talking about developing a new separator, to the time it happened."

source: [https://www.princeton.edu/~hos/frs122/precis/mcilroy.htm](https://www.princeton.edu/~hos/frs122/precis/mcilroy.htm)

</comment><comment><USER7> I think I can kind of grasp why the Ackerman function is a good example of the language, but I feel it falls off a common cliff in programming exercises. It is too far off of things that I will ever actually find myself programming for me to really grok it quickly. Which means often when I thought I understood it, I had a few misunderstandings.

To that end, I'm not sure how to take some of this letter. I don't take the claim that tools shape the users as a criticism directly. Indeed, it seems to be somewhat favorable with the violin example there.

So, the question then, is in what way does APL shape its users? If it is to lament not being able to easily type and share their programs without a specialized terminal, it is hard to argue against the point. Indeed, that I had to be given a glossary to even attempt to read the program is baffling to me.

My guess is APL also helps build the mental muscles to shift things on a stack and compute on them. I recall having that ability from the early HP calculators did seem helpful in ways I couldn't express. Are there other benefits?

<comment><USER8> The fact that APL uses special characters is almost completely arbitrary.

It could trivially be turned into the usual pseudo-English syntax by replacing all the single-character symbols with conventional ASCII keywords. (As in J, etc.)

Of course it wouldn't be as dense then, and the distance between the keywords and the mathematical notation they're based on would be greater.

If that's the only reason EWD didn't like APL, it's a rather silly reason.

It's also wrong. If anything, a couple of centuries of mathematical tradition suggest it's easier to reason about logic and formal correctness with terse arbitrary symbols than with English keywords, and that the terseness makes it easy to write down dense concepts that would be much less elegantly expressed in pseudo-prose.

You _have_ to learn the symbols to do math at that level, and most people don't find the learning all that hard.

Given that you have to learn new keywords in every language anyway, it's not a huge stretch to learn some symbols instead.

And of course after you do that there is absolutely no reason not to write and check code on paper, if you happen to think that's important.

<comment><USER7> This seems somewhat dishonest. It goes out of its way to introduce a new dense language that is just as far from the original math. The Ackerman function is demonstrative of that. I had to consult the wiki page to see what it actually did.

The argument isn't to be against notation. But the most common hurdle to apl that he saw was complaints on lacking a specific terminal. If that is truly needed, maybe a change to not need the terminal would have been a wiser choice?

<comment><USER9> That is of course to implicitly accept that the criticism is in fact valid. It is rebutted later on the page, below the letter, with an argument that, in a modern form, would be something akin to _You would reasonably expect a visiting professor teaching Web design to complain about the lack of a WWW browser_.

</comment></comment></comment></comment><comment> why-el This discussion could be superficially projected onto the Erlang domain, and I think in some sense it shows how wrong it is. Syntax, in my opinion, is the least interesting critique one can mount against a programming system, for the simple reason that it is generally learned the quickest (even if you could argue it takes "longer" to learn). A programmer who understands process isolation the way Erlang meant it is will be far more productive (and correct) than one who doesn't. In comparison, a "few seconds" more to type are nothing.

But I see the point about the internals of a system (I assume this is the crux of Dijkstra's point) leaking into the process of teaching it, hence making it harder to form a coherent high level understanding. I continuously see the same problem with Git, which faces the same problems. It is a great system, but one that does not make it easy to form a generic overview without having to bombard the student with the lower level machinations.

Edit: grammar.

</comment><comment><USER10> I'm constantly amazed there isn't a more standardized "transliteration" of APL into a slightly more verbose form that can be written easily on a conventional QWERTY keyboard (yes, I get it, APL devotees love the information density, but as this letter points out wacky keyboard requirements pose a pretty significant adoption barrier).

Is there some major "modern" language or library that I'm unaware is actually an APL in QWERTY clothing? I'd love to give it a try.

<comment><USER11> > _I'm constantly amazed there isn't a more standardized "transliteration" of > APL into a slightly more verbose form that can be written easily on a > conventional QWERTY keyboard_

You can't be "constantly amazed" unless you have checked for those things -- because J (and other dialects) have been a thing for decades!

<comment><USER12> I am more amazed that there is almost no version that supports English words properly. I get that APL fans are married to the terseness, but 1) it's trivial to do, all the symbols are named anyway, 2) if you're going to abandon the "custom symbols as a tool for thought" you might as well use words instead of line noise, since you have sacrificed the symbol-concept mappping, and 3) it'd be a great way to smooth the learning curve! You already have to remember what a concept is called and what it does - why impose remembering the symbol immediately?

J supposedly supports English verbs as an alternate syntax - there's a library that purports to do this - but I have never managed to make it work in the interpreter at all. It's certainly a second class citizen, if not entirely broken.

The only APL dialect that uses English words I've ever found is QNial. It's a lot of fun to program in as a result (its syntax is also cleverly compatible with s-expressions), but it's hampered primarily by the ancient interpreter and lack of bignum support.

<comment><USER13> I think Q/KDB+ is probably the closest to what you're asking for - <USER2> English words with a direct mapping to the underlying array primitives.

Regarding J specificaly: Some words (sort, inverse, each, every) are in the standard library and loaded by default. What have you tried?

You can define your own cover words for the built-ins (here I've added a cap ([:) to signal a domain error if the word is used with 2 arguments when it expects 1, and vice versa):

        times =: [: : *
        signum =: * : [:
        sum =: +/ : [:
        count =: # : [:

J programmers don't do this for the same reason JavaScript programmers don't do this:

        var ArithmeticOperationsLibraryWhichIsIncludedByDefault = Math;
        var TheLibraryRegardingTheBrowserWindow = window;</comment></comment></comment></comment>


The ugly truth about the ugly produce movement - <AUTHOR> https://newfoodeconomy.org/imperfect-produce-ugly-food-waste-commodification-community-supported-agriculture/ <comment><USER3> _This produce used to be readily available to food banks but now that "ugly" and "imperfect" produce can turn a profit, they are less available to those in need._

So the argument is as follows: Farmers were donating food to the poor because they couldn't sell it. Imperfect Produce figured out how to sell that food, so now farmers are selling it instead of donating it. This is bad because now poor people get less food.

I don't get it. You could argue that the poor need food more than farmers need money. That same argument could be made to any business, though. Why are farmers more obligated to donate to the poor than anyone else?

That is even assuming Imperfect Foods is negatively affecting these food banks. The New Food Economy seems to think so, but Imperfect Foods disagrees.

[https://help.imperfectproduce.com/hc/en- us/articles/11500456...](https://help.imperfectproduce.com/hc/en- us/articles/115004564574-Are-you-taking-produce-that-would-have-gone-to-food- banks-otherwise-Where-did-this-produce-go-before-Imperfect-started-)

I don't know who to believe, but I also don't think it matters. My takeaway is that the food bank model for donating to the poor is less viable now. We'll have to find a better model moving forward.

<comment><USER4> > I don't get it. You could argue that the poor need food more than farmers > need money. That same argument could be made to any business, though. Why > are farmers more obligated to donate to the poor than anyone else?

One of the arguments for subsidies to farmers is that they are the "feeders of people". Sounds to me like they want privileges, why not a few obligations for a change?

<comment><USER5> Instead of adding opacity to the system, I would rather they remove the subsidies instead.

<comment><USER6> You will not have agriculture then. It'll be like not having any industry, only much worse.

<comment><USER7> We had agriculture before the subsidies.

<comment><USER8> Agriculture, at least in the US, has always been subsidized. As a country we used slave labor (free labor) to subsidize agriculture, and then over time we gave away an epic amount of land to white farmers (via the Land Act of 1820, the Homestead Act, and the Morrill Act of 1862).

</comment></comment></comment></comment></comment></comment><comment><USER9> I... see absolutely nothing wrong with this.

Imperfect produce has managed to create a market for food that was previously not able to be sold. Good for them.

Does this mean that we may have to restructure social programs as that food becomes sold rather than donated? Sure might. That's ok though.

I'd also like to clearly point out that the author misquotes their source for the amount of food donated. The California Association of Food Banks states that they have donated 164 million pounds of fruits and vegetables. That's not 164 million PER YEAR. That's 164 million since they were founded in 1995 - <USER2> 7.1 million a year. Additionally, the Imperfect produce comment saying 150,000 lbs specifically calls out "From California farmers". While the food banks numbers are simply totals (they likely purchase and redistribute, as well as take donations from out of state sources). There's probably fudging on both sides, but I find this article fairly disingenuous.

<comment><USER10> I see nothing wrong with: selling something if a paying market is found for that, instead of giving it away for free.

I do see something wrong with: creating that paying market by lying.

I suspect most of those conscientious people who think they are helping farmers and reducing food waste would _not_ be paying for this produce if they knew that it was being taken away from poor people.

<comment><USER11> In that case, you should never buy anything, because anything manufactured might be donated if you hadn't bought it.

<comment><USER12> In the majority of cases, retailers trash their products (sometimes after physically destroying them) that go unsold. Even for used items, most people are too lazy to donate even things in perfect condition

<comment><USER10> Problem with unsold stuff is that people don't want it at any price. Even poor people. Before stuff is unsold, it goes on sale for 50% off, or maybe 75% or more. Poorer people can afford it at those lower price points.

<comment><USER13> Not always true. Some companies will rather destroy product rather than undercut themselves (it's a conscious strategy, especially for luxury items).

[https://theoutline.com/post/2602/clothing-companies-are- tras...](https://theoutline.com/post/2602/clothing-companies-are-trashing- unsold-merchandise-instead-of-donating-it?zd=1&zi=7gyrfcho)

[http://www.slate.com/blogs/xx_factor/2010/01/07/why_hm_destr...](http://www.slate.com/blogs/xx_factor/2010/01/07/why_hm_destroys_unsold_clothes.html)

I guess that's why some of those startups that tried to create an "exclusive" outlet prospered (Gilt Groupe, Vente Privee).

</comment></comment></comment></comment></comment></comment><comment><USER14> "Our venture, and ventures like ours, which rely on redistributing food waste, cannot succeed if people waste less food. So please waste more food."

Talk about putting the cart before the horse. I have no problem with saying, "that food isn't really waste, let's redistribute it if it's going unused." But someone's got their head turned right around if they think that a logical corollary is, "and that food should never be used, because we're redistributing it."

<comment><USER15> I think there's a way to think about this that does actually make sense...

"As long as people are only willing to buy pretty produce, then because actual produce inherently contains only a certain fraction of pretty produce, a demand for pretty produce inherently creates a surplus of ugly produce which can then be redistributed. In effect, every purchase of pretty produce hides an invisible tax which goes to buying ugly produce for the homeless. If paying customers no longer demand pretty produce, then stores will be able to only order as much produce as they need for paying customers, and the ugly-food surplus - that virtual tax on produce customers - will disappear. So, in effect, the ugly produce movement is the equivalent of advocating for a repeal of a tax on produce purchases which is used to buy produce for the poor."

And phrased that way - "do you support or wish to abolish the produce tax and the social program based on it" \- it becomes a political issue which one can reasonably have opinions supporting the tax on the basis that it helps the poor. Especially as this tax and feed-the-poor social program has been created _without government intervention or regulation_ \- they're an emergent product of unforced consumer preferences, and there's no administrative overhead involved in keeping track of the taxes and organizing food purchases because it's just an automatic consequence of spending money to buy produce.

<comment><USER16> This is an interesting way to frame the issue! However, your point about there being no overhead to organize food purchases seems dubious to me. Charities have overhead costs, too. And even if a food bank were run entirely by volunteers at no monetary cost to the public, there is a social cost in terms of the time the volunteers are donating.

Furthermore, I suspect that the improper valuation of "ugly" produce leads to other sources of waste. Sure - some percentage of it makes its way to food banks. But how much of it is left to spoil or used to feed livestock?

</comment></comment></comment><comment><USER17> Imperfect Produce's reply: [https://help.imperfectproduce.com/hc/en- us/articles/11500456...](https://help.imperfectproduce.com/hc/en- us/articles/115004564574-Are-you-taking-produce-that-would-have-gone-to-food- banks-otherwise-Where-did-this-produce-go-before-Imperfect-started-)

<comment><USER18> Beyond the first paragraph which just states that their food would otherwise end up as trash, they don't actually reply to any of the claims in the parent article.

The evidence they provide is broad macro-level statistics about how much food is wasted, but not about where the food they're selling comes from.

The parent article claims specifically that they are getting their food from sources that would otherwise be going to food banks.

The fact that this "reply" repeats talking points with country-level statistics instead of directly addressing the question leaves me feeling suspicious of Imperfect Produce.

<comment><USER19> We source from medium and large scale farms where most of the 20 billion lbs of food waste happens. These farmers either let their ugly produce rot in the field, or sell it for pennies to animal feed. Happy to clear up any other misinformation that this article presents.

</comment></comment></comment><comment><USER20> There is some very disingenous sounding hand-waving going on in this article - and it appears to have been successful! Most of the commenters here seem to have come away with the idea that the problem is that Imperfect Produce is taking food that would have otherwise gone to food banks. While the article _never_ claims thus, I think the misunderstanding is intentional on the part of the author. The key point to me is this:

> Three years later and with a 30-percent drop in customers

and

> We lost customers, a lot of customers.

With a more careful reading, one can see that the "loss" to food banks is because BeetBox is not making as much money now that they have competition with better marketing. In fact the article even says as much:

> subscriptions have fallen so much we've had to cut back on many of the food > justice programs that our CSA proceeds had previously supported.

Sounds a lot less like Imperfect Produce is an evil monster here, and a lot more like they're just a more succesful competitor.

</comment><comment><USER21> I used Imperfect Produce for several months and I wonder how much of it was actually 'imperfect.' Most of it was labeled 'surplus,' as the imperfection, and came in looking identical to the grocery store.

I ended up cancelling my subscription after I did the math and it was slightly more expensive than going to the grocery store, and their selection was narrow and seemed to rotate between the same items. Also almost none of it was local to my region--just like a normal grocery store supply chain.

</comment><comment><USER22> "Imperfect Produce claims it's saving the world by reducing food waste—and helping farmers by buying surplus 'ugly' produce that would have been thrown out. Sounds great. The reality is that this produce would have otherwise gone to food banks, to be redistributed for free."

People are complaining because a hidden subsidy (from government and other food buyers to the food banks) has been removed.

I don't like seeing groups shoehorn welfare into other areas. If you've got an income (or cash welfare) problem, fix that.

<comment><USER23> It's crazy the contortions americans go through to avoid having welfare while still trying to have welfare. But god forbid we just have actual welfare.

Been living here 4 years, still don't get it. Prob never will.

I'm supposed to only buy expensive produce so the cheap stuff can be left for those less fortunate? Why don't you just use some of the exorbitant taxes I pay to help the less fortunate directly? Smh

<comment><USER24> I get the sentiment but...we do have welfare programs. If a food source once available is becoming scarce, then obviously they will need to be revamped where appropriate.

<comment><USER25> I think it's easy to take that cold engineerlike approach when you won't be the one occupying this gap until it is filled

<comment><USER24> Cold? Factual, maybe. Are you saying you buy ugly food?

<comment><USER25> No. I happen to agree that the best thing would be for the market to purchase these items and for food subsidies to adjust. But you say

> they [welfare programs] will need to be revamped where appropriate.

as if this will certainly happen and that it will happen quickly. But that is certainly not the case. And what do people do in the meantime? It doesn't seem tough when you've got $1,000, $500 even $100 in the bank. But losing a small benefit for even a couple years can be brutal if you're in poverty. The gears of the market are a powerful tool for making our society wealthier but they'll grind you up just the same if you're living on the margin as they begin to turn. Ask anyone who has been a resident of a Midwestern town for the past 50 years.

<comment><USER24> I grew up in the Midwest. I think you're mistaking my tone. I'm not saying we shouldn't care, I'm reminding the original parent that we do have welfare programs and they will adjust. It's equally absurd to suggest impoverished people shouldn't have access to the same food others do and be resigned to ugly food because no one else buys it...

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> This seems to be the exact same thing as [https://www.phatbeetsproduce.org/uglyproduce/](https://www.phatbeetsproduce.org/uglyproduce/), discussed before on HN at [https://news.ycombinator.com/item?id=17786894](https://news.ycombinator.com/item?id=17786894).

<comment><USER27> Right, I felt like I was going crazy, because I was _sure_ I had read this line before:

>Soon, Imperfect's single-use cardboard boxes began lining the streets on recycling and garbage days in the East Bay.

but I couldn't find the linked URL elsewhere on HN! It's almost a verbatim quote from the original article:

>Soon, their single use cardboard boxes began lining the streets on recycling and garbage days in the East Bay.

The whole paragraphs are nearly identical:

>Within months of its arrival in the Bay Area, Imperfect Produce fliers were showing up on our car windows, their outreach coordinators were pitching at community meetings, their Facebook advertisements popped up daily in our feeds, and they were edging into community centers we had operated at for years. Their marketing blitz paid off. Soon, their single use cardboard boxes began lining the streets on recycling and garbage days in the East Bay. We lost customers, a lot of customers.

>Within months of its arrival in the Bay Area, Imperfect Produce fliers were showing up on our car windows, its outreach coordinators were pitching at community meetings, its Facebook advertisements popped up daily in our feeds, and it was edging into community centers we had operated at for years. The marketing blitz paid off. Soon, Imperfect's single-use cardboard boxes began lining the streets on recycling and garbage days in the East Bay. We lost customers, a lot of customers.

</comment></comment><comment><USER28> If I'm throwing something in the trash because it is no use to me, it might be a windfall for you if you can snag it and make use of it. But that's what it is, it's a windfall; you shouldn't get dependent on picking my trash and get irate if I stop tossing out the bits you liked to pick.

</comment><comment><USER29> I was an imperfect customer, frankly it figuratively stunk. I was charged $14-$18 per week for around a pound of produce. It was mostly just overstock. Honestly, weird shaped produce was part of the fun for me. It was always on the brink of going very bad, so ironically even though it was just a pound I often threw some away.

After a few weeks I realized that I could get the same value by simply buying some older produce at my grocery store for much cheaper. I can get a pound from any store here in the Chicago area for around $5 - $6.

Also the "carbon savings" their website gave to me seemed pretty suspect. It seemed they were just spitting an average at me and not doing any hard calculations. It's curious that getting a pound of food each week delivered on a truck could actually be a net positive for the environment. I'm sure they can justify it but the premise itself seems kind of absurd.

I was suckered in by the allure this article talks about. No more fancy food startups for me, let me just focus on using all of what I get from a conventional grocery store.

</comment><comment><USER30> > Imperfect Produce claims it's saving the world by reducing food waste—and > helping farmers by buying surplus "ugly" produce that would have been thrown > out. Sounds great. The reality is that this produce would have otherwise > gone to food banks, to be redistributed for free.

So I feel for the social cause, and I think the business ventures should be much more up-front about what they are and not masquerade as something else, but let's be clear that buy buying this less than perfect produce, they are helping growers, including small growers, by allowing them to capitalize on more of their crop.

Ultimately, if you want your social cause to not be a victim of market forces, you either need to work yourself into the market correctly, divorce yourself form it, or find a segregating line that works.

If they were taking monetary donations, or allowing write-offs of the produce at full price (something that conceivably could be supported through targeted legislation), then they might be more competitive here. But since they relied on drowers giving them produce that had little or not value, which now _does_ have some value, they'll see the market respond like anyone else.

Again, this doesn't excuse businesses masquerading as social causes (which is nothing new, even if it is loathsome in the worse cases), but this article isn't exactly presenting the case in a balanced light either.

<comment><USER31> Later on it says that it specifically helps only midsize- and larger growers, because small growers don't produce large enough quantities for it to be profitable to purchase from them:

"'We want to be able to source at least a truckload from these growers each week, so they have to be at least midsize in most cases.' Imperfect Produce is only able to make a profit by working with the larger global agribusinesses, not the picturesque small and mid-sized farms it projects in its marketing campaign."

<comment><USER32> I have yet to see any food industry marketing anywhere that doesn't create a rosy picture of family farming in its marketing, rather than the factory farms and giant agribusiness that actually supply the vast majority of our food.

</comment></comment></comment><comment><USER33> "The company does this by commodifying food that would go to the poor for free"

The article provides 0 evidence towards this case. I'm all for listening to this argument, as it is a good one, but are there any facts to support this other than their shrinking user base?

<comment><USER18> > Within months of its arrival in the Bay Area, Imperfect Produce fliers were > showing up on our car windows, its outreach coordinators were pitching at > community meetings, its Facebook advertisements popped up daily in our > feeds, and it was edging into community centers we had operated at for > years. The marketing blitz paid off. Soon, Imperfect's single-use cardboard > boxes began lining the streets on recycling and garbage days in the East > Bay. We lost customers, a lot of customers.

<comment><USER34> So how were the customers or farmers hurt here? It sounds like they chose to switch services, and the only loser is the author of this post. It's not clear to me how losing customers = food banks suffering.

</comment></comment></comment><comment><USER35> But the total beet supply remains unchanged? Affluent customers are now buying the ugly beets, does that mean more pretty beets go unsold? Can they be donated?

<comment><USER36> >Affluent customers are now buying the ugly beets, does that mean more pretty beets go unsold? Can they be donated?

Realistically they probably end up on grocer shelves, which are themselves a horrific contributor to food waste. Some small portion of those that will otherwise end up in a dumpster will be redirected to other ends (e.g. 'made fresh' foods offered by various grocers) or donated.

<comment><USER37> We waste food because we can. If industrial agriculture and distribution were less effective, there probably would be less food waste. And there would probably be lots more hungry people.

<comment><USER36> You're not wrong, but just because we possess technical capabilities to produce a surplus of goods such that the economy _can_ tolerate high levels of waste doesn't mean that we _should_. This is an area where we should be making incremental improvements, not only to increase the efficiency of how and where we distribute goods, but in the interest of things like public health.

<comment><USER38> Food is wasted because it's the most efficient method of meeting demand. Don't forget that solving the wastage issue _may increase_ the overall resources needed.

In other words, the extra effort to prevent food waste might cost more than the food waste itself.

It doesn't make much sense to spend $1 to prevent the waste of a $0.50 potato.

</comment></comment></comment></comment></comment><comment><USER39> I almost bought into Imperfect Produce, too, what a bad taste in my mouth that would have left. Thanks for the good read.

<comment><USER40> So when I was a kid my family made use of food banks. Now that I'm doing better in life I give a cut of each paycheck to my local food bank (The Alameda County Food Bank). I don't feel too bad about ordering from Imperfect Produce, it's convenient way to get produce delivered to your door, and I still wind up buying some things from the farmers market and co-op. On that note, it is good to know what consequences new business models have on the surrounding community.

<comment><USER41> > I don't feel too bad about ordering from Imperfect Produce, it's convenient > way to get produce delivered to your door

Honest question: are there similarly convent ways to get produce delivered to your door that don't undermine the food banks' fresh food supply? If you can afford to pay for produce, why not pay for the produce that's traditionally been on the market?

<comment><USER36> Local CSAs frequently have a delivery, or convenient nearby pickup option.

<comment><USER42> Bit off topic, but does anyone know of a CSA that delivers in Seattle? I've only been able to find ones that have pickup options.

</comment></comment></comment></comment></comment><comment><USER37> I wonder what's better in the long term for the farmers.

Making sure people have food to eat is something society in the US can easily do, I really wonder if the heroic volunteerism surrounding food banks is a good way to do it.

</comment><comment><USER43> Lol "We donate waste food, so people who reduce waste are bad"

<comment><USER44> They're not reducing waste: before Imperfect Produce, this food went to food banks. It's just that now someone has found a way to make money off of it.

<comment><USER45> Farmers are no more on the hook to give free food to food banks than you and I are.

<comment><USER44> I'm not saying that they are. I'm saying that food that is donated and eaten is not "wasted" just because no one has paid money for it.

</comment></comment></comment></comment><comment><USER46> > This _corporate_ -supported agriculture was avidly commodifying > agribusiness' food "waste" and had little to do with supporting the > community.

I'm not familiar with the publication's audience, but is emphasizing the word "corporate" supposed to prove a point? Is it taken as given that corporations are evil, and therefore Imperfect Foods is evil?

</comment><comment><USER47> This is largely driven by a US government programs and tax laws, which the Op- ed completely ignores: [https://www.usda.gov/oce/foodwaste/resources/donations.htm](https://www.usda.gov/oce/foodwaste/resources/donations.htm)

If those laws need an update, it doesn't seem like a huge deal. There's a lot of support for food assistance programs. I don't see any reason to villainize people trying to reduce food waste.

</comment><comment><USER48> I thought that CSA stood for _Community Supported Agriculture_ :

[https://www.nal.usda.gov/afsic/community-supported- agricultu...](https://www.nal.usda.gov/afsic/community-supported-agriculture)

Is _Community Service Agriculture_ a new social justice version of that, or was there a typo in the article?

CSAs are a great idea for supporting small farms, but adding a social justice mission to an already challenging undertaking seems like biting off too much at one time.

</comment><comment><USER49> Maybe supermarkets will begin selling ugly produce now, too, since Imperfect Produce is raising awareness. If they had aisles of ugly produce at discount prices, would it really undercut their sales of pretty produce? Maybe they should run experiments, to see if they can make higher profit margins on ugly veggies, even sold at discount prices, since they have bargaining power over what they pay the producers.

</comment><comment><USER50> When I did the math + realized the shipping costs, it was a no-brainer.

I know many people who buy into these 'good feelings' products.

<comment><USER36> >I know many people who buy into these 'good feelings' products.

Good, quick talk by Zizek on this phenomenon: [https://www.youtube.com/watch?v=hpAMbpQ8J7g](https://www.youtube.com/watch?v=hpAMbpQ8J7g)

</comment></comment><comment><USER43> Why skirt around the fact that everyone (farmers, consumers, they themselves) is still free to donate all the food they want? Imperfect Produce has not changed that, just given some value to food that used to be "waste", which is a good thing.

</comment><comment><USER51> I like the concept but Sprouts is hard to compete with -> the prices seem to be pretty competitive.

</comment><comment><USER52> It seems quite unlikely that this random startup is actually the source of a 30% drop in business at BeetBox. Both of these companies account for a tiny percent of the produce market. BeetBox would be better off working on making their product better, rather than writing articles criticizing a startup doing something similar.

</comment><comment><USER53> Having a good laugh at the abuse and mutilation of terminology here in the comments section, comedy gold.

</comment><comment><USER54> > Some may claim we have a case of sour grapes. This is capitalism at its > best, they might say.

Based on this biased article I agree.

</comment><comment><USER55> Sounds like they're really mad they didn't partner up with Imperfect Produce when they were given the chance.</comment>


Hackers stole a copy of 'Orange is the New Black' season five - <AUTHOR> https://www.engadget.com/2017/04/29/hackers-stole-a-copy-of-orange-is-the-new-black-season-five/ <comment><USER2> This does still seem to be developing: [https://twitter.com/tdohack3r](https://twitter.com/tdohack3r)</comment>


Ask HN: Would you pay for a web service that lets you manage SSH sessions? - <AUTHOR> I created a cool app called webmux (https://github.com/ronreiter/webmux) which allows you to manage SSH sessions over a web-based terminal. I'm thinking about creating a SaaS out of this so you could login wherever you want to your active sessions.<p>Would you pay for such a service? <comment><USER2> Check out: [https://commando.io](https://commando.io), it is essentialy the SaaS you are describing.

I'd pay for your service provided it had the following improvement over Commando.io: VPN support. Most SSH accounts I manage do require VPN. I don't know if it would be secure/feasible for a web service.

<comment><AUTHOR> Thanks!

</comment></comment><comment><USER3> No, why ? Its just not more then a password manager but with an inbuild ssh terminal and the feature to only store ssh credentials? And ofc im sure there are tons of features not available like authorization with a key etc.

<comment><AUTHOR> Key based authentication is super basic, so of course it will be provided. But thanks for the honest feedback!</comment></comment>