Why's that company so big? I could do that in a weekend - <AUTHOR> http://danluu.com/sounds-easy/ <comment><USER2> I often hear this logic from inexperienced engineers. My answer to them is: Then why don't you do it? <grin>

Recently a junior engineer said this to me about Uber, and I helped him to understand - His logic was "it's just a taxi app".

There were loads of Taxi apps before Uber and loads of them after Uber. Yet none of them were uber?

Sure that a medium complexity CRUD app can do the pickup and scheduling, and spit out some json to the app.

But what if you have 100,000 journeys taking place simultaneously, and you must track their positions so that you can render the journey map on the invoice?

Can you do live traffic planning and re-route around the busy traffic on the fly?

Does your weekend include the full billing system? This is a mandatory requirement for a business.

Can you spot busy periods and increase the fares according to the laws of supply and demand? All online in realtime? Can you continuously A/B test this to optimise profits?

Can you deal with the fact that maps and roads are always changing?

All of these and more are the things that make Uber, Uber, otherwise you're just one of those shitty taxi apps that failed.

Still think you can build it in a weekend?

<comment><USER3> I agree with everything you say, but that response, like the junior engineer, focusses on the technology. So give people six months not a weekend -- there's a good chance they still won't be Uber.

The success of many companies, and probably all of the unicorns, has nothing to do with technology. The tech is necessary, of course, but so are desks and an accounting department.

Internalizing that has been difficult for me as an engineer.

<comment><USER4> At a previous job, we had serious development bandwidth issues. The company was trying to solve that by hiring more developers, and all it was doing was making things worse.

The root cause of the development bandwidth problem was that we wasted vast amounts of time and energy on building misfeatures. And we built so many misfeatures because we had way too much development capacity in relation to our product planning capacity. So the BAs didn't have enough time to fully develop ideas before sending them to engineering, and were instead stuck throwing things over the wall when they were still half baked. All these half- baked ideas took time to build, and then they took time to fix after they bombed on the market, and they slowed everything else down since every new feature is something you have to ensure still works when you make a change in that part of the software. Day in, day out, we were taking on technical debt to cover our design debt bills.

So then we tried Agile, and man oh man did that make things worse. 'Cuz now instead of just lobbing things over the wall, the BAs had to be actively involved in every sprint. There goes any opportunity to sit down and concentrate, because now you've got a nattering horde of 20 developers all trying to get clarification on last week's decisions, which leaves less time to make this week's decisions, which means next sprint things will be even more sketchy and ill-defined.

The correct solution would have been to hire more business analysts, and they key thing we needed from those business analysts wasn't instructions on what to build. It was instructions on what _not_ to build.

<comment><USER5> > The company was trying to solve that by hiring more developers, and all it > was doing was making things worse.

You don't mess with Brook's law!

<comment><USER6> I second this. Additionally, get a grip on your queues!

I have yet to find a better text on how to properly manage software projects than, "The Principles of Product Development Flow: Second Generation Lean Product Development"[0].

[0] [https://www.amazon.com/Principles-Product-Development- Flow-G...](https://www.amazon.com/Principles-Product-Development-Flow- Generation/dp/1935401009)

</comment></comment></comment></comment></comment><comment><USER7> I recently had a discussion about how hard it would be to have a system inside a bus to announce the next stop. Sounds like a weekend project right?

Think a bit further:

\- Hardware needs to be resistant to harsh diesel engine vibrations

\- 3G/4G connectivity

\- Need a mobile data contract with local ISP

\- Software needs to be able to handle network disconnections

\- GPS needs to be able to pin-point at which stop the bus is currently stopped, even with bad GPS coverage in larger cities

\- If the bus skips a stop because of a detour, the software should be able to detect it and announce the next stop

\- The announcement should be bi-lingual for Airport busses

\- The announcement should work for people with hearing aids

\- Server needs to know all bus-stops

\- Server needs to know different stop types, such as bus terminals, intersection stops, regular stops, hand-over stops, virtual stops

\- Server needs to be able both work of a yearly bus schedule and real-time update

\- Server needs to output in an understandable JSON/XML because the government subsidies demand an open-data format

\- Server needs to publish data to an open-data server because of the subsidies

\- Because the bus-company is sponsored by European Union money the control interface should be translated to German/French/English/Spanish

And now this weekend project takes a team of 10 engineers working for a year.

<comment><USER8> All London buses do this. I don't think you need a server, 3G/4G connectivity or even GPS really. You might be over thinking it.

<comment><USER9> On the other hand, the overthinking in question (which is probably true for bare-bones next stop indication) provides the base for my personal bus bug: know if/when a bus is reaching a stop from outside the bus in question, especially with geofencing, in order to know whether the bus I want to take is running early/late/not.

<comment><USER8> They actually do this as well (and many have live updating signs), so the buses do have GPS linked to a server of some kind. It's normally absolutely fine, but on the couple of occasions for me where it broke it seems that the server doesn't conduct the bus in any way and kind of hopes the bus is on the right route.

That's all speculation though.

<comment><USER9> > They actually do this as well (and many have live updating signs)

I don't know about London, I've seen some systems where the stops had "live- updating signs" but only provided the time of the next official passage. And while I wasn't clear, my bug is not just stop signage but phone access as well (that's most useful when e.g. it's raining cats and dogs and you want to avoid waiting in the rain at uncovered bus stops)

<comment><USER10> In London you can go to the TFL website and see live updating times for when the bus will get to your stop: [https://tfl.gov.uk/travel- information/stations-stops-and-pie...](https://tfl.gov.uk/travel- information/stations-stops-and-piers/)

<comment><USER9> OK, that's really cool. Definitely means they have "a server, 3G/4G connectivity [and] GPS" though.

<comment><USER11> Not necessarily. I think some of the systems use radio links to a central location and transponders at fixed points on the route to track the buses.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> This is such a straw man conversation. Of course it is a fallacy to think "I could build that in weekend" and translate that to a claim about the inefficiency of a company. I think that's what is being responded to in this thread. But that's not the interesting debate.

A better consideration (imho) is this:

       For a given business/offering/product, what is a likely optimal number of engineers needed to continuously deliver the product's value efficiently? Then: How far is a given company from that optimal number?

This is a hard analysis to do. It is virtually impossible to quantify or put this kind of evaluation in a laboratory. There are too many counterfactuals that would be at play.

We really just have to consult our experience and consider which of these is the more probable:

(1) A business/offering/product team with lots of money is really good at utilizing its hundreds of staff. OR (2) It's not.

I've seen a few (2)s from the inside. I have never seen a (1). Therefore I lean more probabilistically toward the belief that a company is needlessly big.

What surprises me is that that's not where people on this thread are leaning ---but has anyone on this thread actually ever _seen_ a (1)??? If not, where is this belief coming from? I've never seen a ghost, so I don't believe in ghosts....

<comment><USER13> Consider, for the sake of argument, the possibility that perfect efficiency is _unachievable_. Almost physically impossible. As a company gets larger, certain kinds of inefficiency in hiring and utilization become absolutely inevitable. With that premise in mind, how many employees does a company "need"? If it would need fifty in that universe of perfect inelastic spheres with no friction or similar effects, how many times that does it _actually_ need in the real world of such efficiency-robbing factors?

In my experience, the answer is two _at best_ and more often, around five. The incremental _work product_ from each new hire diminishes more rapidly than many realize, but the incremental _revenue_ might still make such hires worth it. As Dan points out, a few percent increase in efficiency from optimization can be huge. It might take multiple people quite a while to achieve that, and it will still be worth it. There might be another group of several people trying something else that doesn't pan out, but that doesn't mean they were unproductive dead weight either. Bigger companies have to take risks too. They just take them differently, and the cost is more obviously attributed to them instead of the market.

Maybe a _single_ startup could do what Twitter (for example) does, in fairly short order. Even that's unlikely when issues of scale etc. are considered, but that's not even the point. That single startup is not a valid point of comparison, because it doesn't count the cost of failed experiments. Add up _all_ of the startups trying to do what Twitter does, including the failures, and that probably be a better estimate of how big Twitter (Uber, Netflix, whatever) needs to be.

</comment></comment><comment><USER12> There's a naïveté in saying "I could do that in a weekend" as a measure for employee bloat, but that's making a straw man of the question.

It is nearly always more probable that a large company with several hundred or more plus engineers doesn't need that many. That those engineers are there for other reasons -- political reasons or speculating on new products, valuation reasons, etc. -- than for the reason that those engineers are fundamental to the top or bottom line of the company's core product/s.

Uber does not need 2000 engineers. That's a prima facie fact.

I've worked inside several large companies and saw this first hand. Hundreds of engineers across teams doing "stuff", no doubt, but nothing that ultimately or ever tied to the top or bottom line. The only ideas that I could come up with for why these people were there were: perhaps the headcount helped company valuation, perhaps we needed that many benchwarmers for when there was company churn, maybe there is some political reasons a manager wants a sizeable headcount on their team, or maybe it's just leadership ignorance as to how things are getting delivered. But in no way could I see a concrete justification for many teams and many engineers.

What am I missing?

<comment><USER13> Maybe some of those engineers really were superfluous. Maybe some were contributing to the bottom line in ways you don't understand or appreciate. I've seen plenty of engineers take infrastructure for granted, discount the value of research ("speculating on new products" is _important_ ), or even dismiss support as a cost with no revenue upside. Most often, as those people gain experience they also gain a broader view, allowing them to distinguish between complementary roles and truly superfluous personnel.

</comment></comment><comment><USER14> I searched for GTGACCTTGGGCAAGTTACTTAACCTCTCTGTGCCTCAGTTTCCTCATCTGTAAAATGGGGATAATA and found another post by the same author on another website [http://bitfunnel.org/strangeloop/](http://bitfunnel.org/strangeloop/)

I've seen some other posts by Dan before but didn't know what kind of things he works on because I generally don't check that because if I did then I'd end up just checking people instead of reading what people were writing. Anyhow, from the article linked above -- for which a video is available at [https://www.youtube.com/watch?v=80LKF2qph6I](https://www.youtube.com/watch?v=80LKF2qph6I) \-- it turns out that Dan has done quite a bit of work with implementing search related technology, and I think that was interesting. His contributions to BitFunnel can be seen at [https://github.com/BitFunnel/BitFunnel/graphs/contributors](https://github.com/BitFunnel/BitFunnel/graphs/contributors)

<comment><USER15> I thought you had accidentally pasted some of your genetic code from your clipboard. _future internet problems_

</comment></comment><comment><USER16> The other side of the coin is that big companies do move slow.

There is a lot of communication overheads, a lot of bureaucracy, backwards and cross compatibility concerns, conservatism, wasted duplicate effort, wasted pointless effort, some people who aren't contributing much etc. etc.

This is why tiny startups can sometimes dethrone a large leader. However that's often a multi-man-year effort.

So I'd rephrase the statement: "Why's that company so big? I could do that with 20 brilliant engineers and 18 months". I think that order of magnitude can attack pieces of any established software business from a technical perspective. Then there's obviously the business side.

<comment><USER17> That would get you your first launch product maybe. Not an actual enterprise product. Sure, people forgive the small startup for their crashes, their unfinished and unpolished app, their weak customer support, their performance hiccups, etc. As long as they bring something that solves an actual new problem that isn't solved by another product easily available. But an enterprise product, no way you will make that in 18 months with your 20 people while keeping up the maintenance burden you introduced in v1 when you took some shortcuts.

</comment></comment><comment><USER18> Yup. Have never gotten the whole "Airbnb/Uber/<startup>" is just a crud app line of thought.

If they're crud apps then everything else in between is just glue programming.

The author of the post covered some very astute points but one not covered in much detail is the challenge of scale and reliability. Maybe this falls under the bucket of optimization and/or latency but I think this deserves being called out on its own.

Once you have Uber-scale number of users requesting taxis at any point, and have a network of drivers constantly communicating their position with the app, suddenly you no longer have a trivial crud app on your hands...

<comment><USER19> These companies are not "tech" companies in my book any more than the Sears catalog was a "tech" innovation.

They are normal business that happen to be enabled by technology, but the technological challenges are not new or to be frank uncommonly challenging.

Because we're I assume mostly coders, we think they are differentiated by tech, almost everything else (marketing, sales, customer service) is likely more important.

Bad technology could certainly kill a platform, but excellent tech won't save it, and a mediocre implementation is good enough.

<comment><USER20> If you think scaling a service like Uber is not challenging, you're vastly underestimating how large they are.

Uber especially has requirements that make it tough to scale correctly. You need to track each Uber session in near real-time, right? That means you simply cannot afford to drop or interrupt those sessions. It's a completely different problem than scaling Instagram or WhatsApp.

<comment><USER19> Hmmm... nothing like near real-time. Challenging near real-time is sub- nanosecond.

Uber real-time requirement of what < 30s? (for a viable product).

No that doesn't seem like a challenging problem. It seems like a problem that would require almost no throught for a MVP, and then incremental challenges as the problem scales.

It seems like a problem that is so comparatively easy compared to what computers are capable of that I wouldn't even class it as "difficult".

But what's telling about your answer is that your thinking about this technical challenge, this feature as being of primary importance.

Do you think if Uber was 10 times cheaper than a regular taxi real-time tracking would be as significant?

Uber's funding which has enabled them to subsidize their expansion (and rides), advertising and promotion and possibly that they plugged in to a easily accessed communications channel (mobile apps). Were likely all more important than the quality of their implementation.

<comment><USER20> Probably less than 30s between pings during a ride, but I haven't tested it tbh. I usually use the word real-time when talking about systems that operate in the ns time range, and near real-time when talking about software. Sorry for the confusion.

What I was trying to emphasize is that Uber needs to keep track of these sessions and make sure they're not interrupted, or else they lose customers.

Yes, I know building something that fits these requirements and works _most_ of the time is trivial. But we're talking about a system that must basically work as close to always as possible. More importantly, it must do this at scale, maintaining and keeping track of perhaps millions of sessions at once.

In a nutshell, an Uber MVP is probably not difficult, but the transition from MVP to real-world product is non-trivial in my opinion.

I'm actually planning on setting up an Uber-like service for my home country, which is why I've been giving these issues some thought.

<comment><USER21> Even 1M simultaneous users reporting every ~30 seconds could average down to ~30k messages per second. And they'd be extremely trivial to segment between servers. You're right, not a weekend hack, but a few weeks of a small team's time isn't out of the question. The technical side is not the challenge here though really - the business side is.

<comment><USER20> Perhaps you're right when it comes to scaling, but you're assuming that the system was designed correctly in the first place, which would be impressive imo. There is also a lot of failure handling and recovery that needs to be done gracefully, which I think would be complicated for a session-based system.

Again, I'm no expert, just making educated guesses!

</comment></comment></comment></comment></comment></comment></comment><comment><USER22> To me this is a form of the Dunning-Kruger effect[0], where only somewhat informed people estimate the cost of a "0.1" (or POC) release without considering the famous "unknown unknowns" which may involve scaling, billing, 3rd party integrations, etc.

[0]: [https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect](https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect)

<comment><USER23> I find I've had to bar myself from thinking about many of these factors, after I realised how many projects I stop myself from pursuing because I know about the hard bits, and realise how many of my past projects I'd never have taken a shot at if I knew what was coming up and was thinking like that.

It's great to know, but often it's not so great if it leads to inaction...

E.g. if you have a "I could do that in a weekend" impulse, sometimes it's worth following that impulse. If you're right, you've spent a weekend building something potentially great. If you're wrong, you've spent a weekend learning about things you clearly didn't know in advance, and the lessons can often be very different from the objections you'd have throw up if you thought about it without doing, and sometimes what you learn may provide the impulses for another new idea.

But it's _hard_. E.g. my first impulse for anything that requires collecting money is how painful it is to deal with credit card processing and reporting and VAT and accounting, because I've done big complex billing systems with heavy reporting requirements, and compliance complexity etc.. It's very hard to put that impulse aside, and "just do it" and think about how to solve the payments afterwards. But that's how I did things he first few times I needed to handle payments.

</comment></comment><comment><USER24> And yet sometimes a small team really could do it. I spent years complaining about MSN/AOL/Skype etc. and talking about how a small team could build a better messenger in a week, but we didn't really believe it. Then Slack actually took that possibility seriously, and look at the results.

<comment><USER25> Sure, they were a small team in the beginning. Google too... Now they probably have more than 500 employees. (They had 385 in february 2016: [http://uk.businessinsider.com/slack-ceo-worried-about- growin...](http://uk.businessinsider.com/slack-ceo-worried-about-growing- culture-2016-2))

<comment><USER26> Slack also jsut recently hit 3 million users. 100 million people were using AOL Messenger at it's peak, and 12 million were using ICQ when AOL bought Mirabillas.

</comment></comment></comment><comment><USER27> Actually it actually wrong to assume somebody could solve a trivial problem in a weekend, however some companies still have way too many people working at it. Consider twitter with their 3860 people. they could probably do it with half of them. especially the technical people are 40% of the company... way too much. i don't think that they need no people but well...

<comment><USER28> Yeah, this post seems to be addressing an annoying conversational experience, but ignores the other side: the reality, which you describe. I understand that just scaling out their platform is a ton of work that justifies having a lot of engineers. But how does Twitter have so many engineers on staff and yet their users constantly complain that they don't improve even seemingly low hanging fruit? Is ~1,500 people not enough? Is there some lingering technical debt they're dealing with? Or is it just people accumulated over time they value too much to let go?

</comment></comment><comment><USER29> I personally have never seen this complaint leveled against the mighty Goog, nor really any company that's cash positive. When I see people giggle at bloat, it's more often than not twitter and still more often than not, it's not intrinsically because big company = waste, but because company that should be small is made to be big either by VCs or Wall Street, and then can't turn a profit anymore.

No, maybe if twitter was still a tiny company you couldn't use it in Arabic but maybe they also wouldn't constantly be on the edge of going out of business.

Edit: I think this is also a regular criticism of Twitter because Twitter has spent the last several years trying to Facebook itself and strayed far from the original idea. You can argue the pros and cons all day since they're pretty much infinite on both ends, but Twitter as a company has spent vast amounts of money on a lot of features that none of it's original userbase really wanted (and drove a lot of people off in the process).

Me, I only really ever used it as a time-waster so I'm not awfully attached but I listen to a ton of podcasts and I can't count the number of hosts who disliked more or less every new thing Twitter did.

<comment><USER30> Things like Dropbox are striking. I guess it was mostly written originally by Drew Houston over a year or so and was cool.

Then 2011, 50 million users 70 employees. Seems fair enough.

Now about 1500 employees. It worked fine and scaled fine with 70 employees so you wonder if hiring the other 1430 was good business. I guess if VCs are throwing money at you you may as well spend it on something. Though that's probably what Yahoo was thinking when it went to 11,700 employees and that didn't play out so well.

<comment><USER31> I bet its mostly sales. I had no idea about how much sales ppl you need at certain scale.

<comment><USER32> Sales persons are necessary when continuing growth requires efforts that do not scale as effectively. Marketing and advertising campaigns work sufficiently well for b2c with 1-2 people but you hit a wall if you want to selling to, for example, the Pentagon because the way those with the deepest pockets buy is nothing like how most efficient individuals and organizations buy. Groupon's massive needs for salespeople is an example of how even lower margin b2b even needs sales due to how scaling out b2b is so much less cost- effective. But for investors, landing a huge contract is not as much about the revenue as much as an indicator of stability in terms of both revenue and market signal (no Fortune 100 will buy some random fly-by-night vendor's stuff without code escrow even if it's the best thing ever because their sourcing overhead is so poor and big companies are immensely risk-averse by design).

<comment><USER33> Atlassian's 'b2b without salespeople' approach has worked well for them.

<comment><USER32> They definitely have sales roles in more recent years and have a sales ops team [1][2][3]. The Federal channel manager description even acknowledges that there's unique challenges in selling to the government that necessitates someone that knows the different contracting vehicles. What they've done with little turn to sales culture is pretty admirable but at the same time only possible with organic, engineer-to-engineer driven growth Github and Atlassian suites.

Salesforce grew somewhat similarly via rogue accounts on expense accounts, but to actually clinch the biggest growth there was no way to avoid salespeople to get over $500MM in revenue probably.

[1] [https://www.smartrecruiters.com/Atlassian/99086565](https://www.smartrecruiters.com/Atlassian/99086565) [2] [https://www.smartrecruiters.com/Atlassian/95395260](https://www.smartrecruiters.com/Atlassian/95395260) [3] [https://www.smartrecruiters.com/Atlassian/93068238](https://www.smartrecruiters.com/Atlassian/93068238)

</comment></comment></comment></comment></comment></comment><comment><USER19> The sad truth is that in the majority of cases the quality or time taken over the implementation isn't particularly important other factors matter much more:

* building something people actually want. * advertising * engaging with users * aggressively promoting your product. * customer service

It doesn't matter if what you did was hard or not. It matters that you can acquire and retain customers.

</comment><comment><USER34> > _Businesses should keep adding engineers to work on optimization until the > cost of adding an engineering equals the revenue gain plus the cost savings > at the margin. This is often many more engineers than people realize._

I think this is the key realization, and it's actually a variation on Conway's Law[0]. In a nutshell, some part of BigCorp discovers (or believes) that by hiring a software developer for $X, they can make $Y > $X in return. So they do.

As long as some part of BigCorp is able to do that, you get more software engineers. This continues until you either (a) run out of budget, or (b) run out of opportunities.

Because it's not done "top down", it less efficient than it could be to get all of the features in total. But due to _how_ these engineers are funded internally, it's not possible to do the "top down" approach at all. And that's okay, all that really matters is that $Y > $X.

[0] [https://en.wikipedia.org/wiki/Conway%27s_law](https://en.wikipedia.org/wiki/Conway%27s_law) "Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations." In this case, it's not the communication structure that causes the perceived bloat, it's the funding mechanism. The basic idea that software organizations reflect some non-software, human structure in the business, holds.

</comment><comment><USER35> As of 2008, the core search team at Google was just under 100 people.

Then came search customization. Suddenly, no more cached replies.

<comment><USER36> 100 people?! I could do that myself, in a weekend.

</comment></comment><comment><USER13> Everything seems easier when you're not doing it yourself. It's only a minor variation on the grass being greener. Everything gets way more complicated when you're doing it at scale, with high reliability, for real users, with some hope of making money. Practically nobody adds complexity for its own sake. Usually, if you think something is much more complicated than it needs to be, it's because you don't have any idea what the real requirements and constraints are.

</comment><comment><USER37> To me this post addresses the same naivety that David Graeber demonstrates when he claims that everyone not involved in hammering out a sheet of steel to make a horseshoe or pulling a plough is engaged in a bullshit job[1].

I think it's not a coincidence, therefore that many of the people I run into who advocate a universal basic income, arguing tooth and nail against a job guarantee (the YC guys no different) is young, tech savvy and overly optimistic about the scope of automation to subsume all human labour within months.

[1] [http://strikemag.org/bullshit-jobs/](http://strikemag.org/bullshit-jobs/)

<comment><USER38> Graeber's biggest point is that even when he talks to people and asks them if their job is valuable, and _they should know better than anyone else_ , they often say 'no, not really.' I don't think he thinks "anyone not involved in hammering steel" is unnecessary, that's a bit of a strawman. The argument goes that big corporations have a tendency to hire more people than necessary because middle managers are rewarded based on how many people work under them. Or that people looking to keep their jobs, try to make themselves look more busy and valuable than they really are.

People waste a lot of time at offices doing things that aren't work. But our culture demands we be at work 8 hours a day, even if the work only requires 4 hours. Economists in the 1930's thought the world would be so rich by today, that people would only need to work a few hours a week. And the economy has grown vastly since then, and we are much richer. But I don't think most people have the option of working that little, even if they want to.

But that's a different subject entirely from automation. Those jobs can really be valuable and still be vulnerable to automation. AI has improved a lot in the last 5 years. I don't think it's implausible at all to imagine most jobs done now being partially or totally automated.

<comment><USER39> > Graeber's biggest point is that even when he talks to people and asks them > if their job is valuable, and _they should know better than anyone else_ , > they often say 'no, not really.'

I don't think that's really true, especially in a big corporation. Transparency of the value pipeline in such institutions isn't great at any level, and big corps (and similar large institution -- government often has the same problem, for instance) have a tendency toward _not_ making even the concept of the value stream that justifies the position known to the people working in it much of the time. Its quite likely that the people working a position don't understand how its _supposed_ to deliver value (and are in a poor position to see the big picture of whether it does), while the people who do know how it is supposed to deliver value don't see the small picture of what actually goes on (and thus are in a poor position to evaluate whether it _does_ deliver the value it is supposed to.)

There's no particular reason to think the people working in the position are in the _best_ position to evaluate value delivery (or that even the people _in_ the best position, whoever they might be, are better at answering the question than a Magic 8-ball.)

<comment><USER40> There is a great incentive to obfuscate when your "value delivery" is skimming (mutual funds that charge 1.25% management fees for underperforming the S&P), privatization fraud (health insurance in the US, government contractors, and pretty much everything related to real estate since "home appreciation" is now supposed to pay for people's retirements instead of pensions), taking advantage of stupid people (lottery, penny auctions, telemarketing, almost all advertising), education fraud (textbook industry, "student loans now, regret later" private colleges and universities - which since many student loans are federally guaranteed is also a form of privatization fraud), or "intellectual property" racketeering.

</comment></comment></comment></comment><comment><USER41> Same thing happens in software development at the library level:

"Why not use [Example OSS Library]?"

"Nah, I can do it in 15 minutes."

"You know, they have [X] contributors, lots of great documentation, and 6 months of work into it."

"Yeah, I know [but whatever.]"

<comment><USER42> I get the point that you are making, but if something _really_ only takes 15 minutes to do, I wouldn't necessarily use an open source solution just because one exists.

Depending on the situation, the library may have a ton of extra bloat that your application will never use. Also, recklessly deferring to open source libraries can potentially snowball into something like the left-pad fiasco[1].

[1]: [http://blog.npmjs.org/post/141577284765/kik-left-pad-and- npm](http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm)

<comment><USER41> Yeah, and I get your point too. Eight lines of code? Yeah, you don't need a library. But usually when someone says "15 minutes" they mean 1 - 2 days. So things of that complexity is when it's obviously you should choose a good OSS lib.

<comment><USER43> It depends. In many cases you would be right, in many other cases you wouldn't.

It depends on many factors:

\- what is the problem? (some problems are really hard, some problems are trivial and just seem hard)

\- is developer experienced enough to build own solution?

\- are existing solutions good or badly written?(I often refrained from using existing solution when I looked on bloated code in it).

\- is solution suitable to project? ("too much bundle size", "doesn't support X", "not too configurable", "too slow", "too hard to use").

etc.

</comment></comment></comment></comment><comment><USER44> Completely unrelated to the article, but I'm curious what people's thoughts are on full-width lines of text characteristic of unstyled HTML.

I've seen some[1] bemoan its impacts on readability and others[2] complain about websites which artificially restrict themselves to (what can be) a narrow slice of the viewport.

[1]: [http://bettermotherfuckingwebsite.com](http://bettermotherfuckingwebsite.com)

[2]: [https://plus.google.com/+LinusTorvalds/posts/2Ndtw3ZBRX2](https://plus.google.com/+LinusTorvalds/posts/2Ndtw3ZBRX2)

<comment><USER45> On desktop I reflexively zoomed in to %125 and didn't think about it until I read your comment.

It also has a reasonable default font size my phone, so it's easy to read quickly. This is pretty rare since browsers gave up on text reflow.

<comment><USER43> I begin to think that nowadays websites are just designed for "zooming in" in mind (very small fonts, too many space etc.).

So maybe we shouldn't complain. These websites are just meant to be zoomed.

Although some web developers are cruel and block zoom with meta tag (I never understood why some developers did that, maybe for trolling people with poor sight?).

</comment></comment></comment><comment><USER46> >>There's also a wide body of research that's found that decreasing latency has a roughly linear effect on revenue over a pretty wide range of latencies and businesses.

Anyone have pointers to such research?

<comment><USER47> There is no published research I'm aware of. However, you can find an article about every major tech company talking about how reducing their latency increased whatever their core metric was (not necessarily linearly though, sometimes it was and even steeper curve). I know for sure this is true at eBay, Netflix, and reddit because I've seen the numbers personally, and I have friends at Google, Amazon and Facebook who have said the same.

</comment></comment><comment><USER48> Otherwise, indeed, it's a basic evaluation error, as described as long ago as in the The Mythical Man-Month; it just seems that nowadays, the multiplier is not 9, but even bigger.

[https://archive.org/details/mythicalmanmonth00fred](https://archive.org/details/mythicalmanmonth00fred) Figure 1.1, page 5

To go from a program to a programming system (interfaces, system integration), you need to multiply the effort by 3.

To go from a program to a programming product (generalization, testing, documentation, maintainance), you need to multiply the effort by 3.

So to get a programming system product, you need to multiply the effort from the program step, by 9.

Nowadays, we need to compose indeed multipliers for distribution, reliability, availability, localization, targetting multiple UI platforms, etc.

And the original multipliers are also probably too small, when you see the effort required just to integrate testing and CI on some projects.

</comment><comment> FLUX-YOU I have worked with one-man companies that did this and often they are shoddy products. Healthcare just seems to suck at software.

Full-stack developers are already a very fast evolution from the front/back end developers, which were an evolution from a guy doing just HTML/CSS in the 90s. Now you have to include sales, marketing, negotiation, project management, and business management skills on top of your full-stack development skills. The result is that something is going to suffer unless you are a genius, the product is super-simple, or you have 20-30 years to have learned all of this.

There's a much more interesting question in the form of: "Why's that company so big? A tight-knit team could do that in a weekend".

</comment><comment><USER49> The technology of a big company is often a necessary condition, but it's seldom (if ever) a sufficient one.

We technically-minded people are often pretty terrible at accurately picturing reality as it exists outside of our respective spheres. Consider:

"Why is Armani a billion dollar company? I could make a beautiful suit in a weekend," said the talented seamster/seamstress.

</comment><comment><USER50> > Recently, I was curious why an org that's notorious for producing unreliable > services produces so many unreliable services. When I asked around about > why, I found that that upper management were afraid of sending out any sort > of positive message about reliability because they were afraid that people > would use that as an excuse to slip schedules. Upper management changed > their message to include reliability about a year ago, but if you talk to > individual contributors, they still believe that the message is that > features are the #1 priority and slowing down on features to make things > more reliable is bad for your career (and based on who's getting promoted > the individual contributors appear to be right). Maybe in another year, the > org will have really gotten the message through to the people who hand out > promotions, and in another couple of years, enough software will have been > written with reliability in mind that they'll actually have reliable > services. Maybe.

This sounds like it could be Apple?

</comment><comment><USER51> Besides underestimating the scope of the problem, one logical fallacy that programmers fall for is believing that a single developer should be able to juggle lots of tasks easily. If a small team can create a SaaS application, why do large companies need so many engineers for so many specific things?

I think, at a certain scale, it becomes necessary for organisations to ensure that there are certain people keeping specific things running. Let's say, a company doesn't have any specific person assigned for server administration and then one day, is confronted with a DDoS attack. Who is supposed to handle it? Whose responsibility is it? You have few people who know the basics of operating servers but no one who can swiftly respond to it. If no clear role is established, then the company is bound to face more attacks and possibly, lose more customers.

</comment><comment><USER52> Looks like my primary search engine DuckDuckGo tackled this with a relatively small team.

[https://en.wikipedia.org/wiki/DuckDuckGo](https://en.wikipedia.org/wiki/DuckDuckGo)

[https://duckduckgo.com/](https://duckduckgo.com/)

<comment><USER53> At an Alexa rank of 676, that's pretty impressive. I think the key to DDG's success with a small set of engineers is their small scope.

If they ever wanted to expand their product scope, for example, index their own searches instead of aggregation, they would start running into the obstacles discussed in the article. The Lucene comparison in the article doesn't apply to DDG (yet).

</comment></comment><comment><USER54> I was on the other side of this with a product that I was one of the main people on back in the late '90s - I looked at what we were selling and how much we were selling it for, and thought "Why are people buying this? Anyone with a decent IT department should be able to do this in-house."

But really when you sat down and looked at it, what we were selling was a supported and tested product that could be configured and active in less time, with excellent ROI, and at significantly less than the cost of dedicating an in-house developer to do it. What we were selling wasn't cutting-edge technology, it was time and cost savings for a department that was almost certainly already short of both money and staff time.

</comment><comment><USER55> I think a lot of people are willing to "leave cash on the table" if it means a smaller, more effective team and leaner product.

<comment><USER56> Certainly not if you are VC driven.

</comment></comment><comment><USER57> I'm amused the central topic of this post is that google indexes 50M primes in random files in github, which they seem to not. 961748729 is one of the primes just before the 50M'th

Your search - site:github.com 961748729 - did not match any documents.

</comment><comment><USER58> The article raises some good points but there is still a dichotomy here to me. Compare a successful startup with 2 founders with an established company with 2000 employees in the same business. The scale of the difference in employees vs the difference in the product is often shocking. BigCo product might be better, but that much better?

Obviously the quality of the product won't scale linearly with the number of engineers, but still, startups punch above their weight. Maybe it just highlights how much harder things get when the company reaches a certain size.

<comment><USER33> Those 1998 extra staff don't have to make the product 1000x or even 2x better (however you might quantify that). They just have to make it sufficiently better to maintain or gain market share.

</comment></comment><comment><USER59> Weird example, where it's assumed that a single machine can only hold 50GB of document data, which results in the assumption that you'd need 200k machines to store 1T documents...

</comment><comment><USER60> Apps aren't just a pile of features, they're also ecosystems.

In the early years of Twitter, it would have been relatively easy to clone Twitter, but you won't have had the ecosystem.

Poster example: Google+

</comment><comment><USER61> As a counter point, on every product I've worked on, on every time we've wanted to make a 'big leap' we've need to get a small expert team to go make it happen.

<comment><USER62> Dod that small expert team polish the product and maintained it for the next 2 years?

Something about 80%/20% rule...

</comment></comment><comment><USER63> I call this the difference between building a product and a business.

It's relatively easy to build a product and even to ship it.

The real art in building a business to support it.

</comment><comment><USER64> > we want to maintain an artisanal search index of 1B documents. Then our cost > comes down to $12M/yr.

Yeah, no. It doesn't cost that much to maintain an index of that size. Not even remotely close. My roommate coded a search engine back in 2001-02 as part of a class project/hobby and it easily had an index that large, probably larger till he shut it down. The key lies in not crawling it all in a single day. And your index doesn't populate overnight, it takes months of slow crawling. You can maintain an index of practically unlimited size for chump change (well, compared to the millions OP was throwing around). Do people think Excite, Hotbot, Lycos and Yahoo! in the early years had $12 million per year to spend on their indexing? Hell no. Every single one of them would have went bankrupt in a week. Those companies weren't even worth that much back in the mid to late 90s. Your biggest cost is going to be user loads on your servers (CPU/Ram/Bandwidth), not maintaining an index.

</comment><comment><USER65> I doubt most who say this sort of thing mean it literally (although no doubt some naive sorts do, but they usually are self-identifying through the other outrageous things they say, and thus can be ignored). But one counterpoint to keep in mind is that the alternative view is every company is running as efficiently as possible. I doubt anyone really always believes that, either. For example, when a story about Twitter was posted on HN recently, it was widely discussed that they probably don't need anywhere near as many engineers as they actually have, and could perhaps reduce costs to the point where they are at or near profitability just by reducing staff. There might be some truth to that. Point being, it's not that simple, and there are no absolutes.

</comment><comment><USER66> Building an app and building a business aren't the same.

Go out and sell the app to people and find out how easy it is.

</comment><comment><USER67> Always reminds me of: [http://i1.wp.com/beckybendylegs.com/wp- content/uploads/2014/...](http://i1.wp.com/beckybendylegs.com/wp- content/uploads/2014/01/Modern-Art.jpg)

</comment><comment><USER68> From the article:

> There's also a wide body of research that's found that decreasing latency > has a roughly linear effect on revenue over a pretty wide range of latencies > and businesses.

I'd be interested in seeing that citation list, particularly for non- google/amazon businesses.

</comment><comment><USER69> I feel if you have something better to offer, say a better way to look at information, not just a better page rank algo, but a different way to look at information that is scattered across, one that enables a better insight into existing web pages and less overhead of constant 'searching' by typing a query, something that automatically helps to "see" relevant information with less effort, then it can be an alternative to what a Google search does today and can position itself as a competitor in the long run even if it does not exist today.

</comment><comment><USER70> If your problem is small, viable solutions to that problem are vast. If your problem is large (scale), there's usually only one or close to only one way to solve it. So choosing the right path is important. Since you can't predict product market fit (you cannot predict value), you have to figure out how to iterate and test as close to continuously as possible. This last thing is what most people just can't do. Programmers included. So you can see there's a technology side and a culture side and they have to be well-aligned.

</comment><comment><USER71> Even if you're just building an open source app to give away, sure, you can get 99% of it done in a weekend. The last 1% will then consume 99% of your time (if you want anyone to use that app).

</comment><comment><USER72> From 2009, still relevant: [https://blog.codinghorror.com/code-its- trivial/](https://blog.codinghorror.com/code-its-trivial/)

</comment><comment><USER73> The question I was wondering about is: When does an organization realize and say, we have hired enough engineers to support our current systems?

</comment><comment><USER74> Jeff Atwood made a fantastic writeup of a very similar sentiment about StackOverflow / the StackExchange network: [https://blog.codinghorror.com/code-its- trivial/](https://blog.codinghorror.com/code-its-trivial/)

Until reading that article, I often espoused similar views on applications. It was a really eye-opening read.

<comment><USER75> Fog Creek is something like 50 employees right? That seems like a reasonably sized team for their product, but Twitter/Uber are in a different ballpark.

</comment></comment><comment><USER76> I agree and disagree, I've been with companies and seen how scope can expand as you grow, building internal apps, monitoring, reporting, all kinds of stuff. On the flip side i know a guy that managed a team of 5, the 6 of them total managed the snippet editor in Visual Studio. Six people to manage a snippet editor? Really?

</comment><comment><USER77> Reminds me of this great article about the 'this shit is easy' syndrome: [http://steve-yegge.blogspot.com.ar/2009/04/have-you-ever- leg...](http://steve-yegge.blogspot.com.ar/2009/04/have-you-ever-legalized- marijuana.html)

</comment><comment><USER78> Hmm somehow it reminded me of brook's law CEO:

"Give me 9 women and I'll give you a baby in 1 month"

</comment><comment><USER79> Features and scale are huge things.

Maybe you could make, say, a Twitter clone, in a weekend. But you couldn't make one that can handle anything near Twitter's scale nor that is as feature- rich as the actual thing, and besides, nobody will be using it.

</comment><comment><USER80> To this I draw a 2x2 matrix on a whitboard. In each of the 4 quadrants I write:

"Product and Engineering"

"Ops"

"Sales and Marketing"

"Legal"

Enter tasks and goals for each quadrant. Iterate.

This is the minimum effort required to sustain any app. Developers are mostly only aware of the first quadrant.

</comment><comment><USER81> One of my favourite tech jokes used to be "Google? I could build that out of wget, grep and baling wire!". The fact that "I could build that out of Lucene and money!" is not intended as a joke gives me pause.

</comment><comment><USER82> Anyone find the 528 MB C++ file that is referred to in the Twitter screenshot near the end of the article? (With only 57 lines of code but, oh, an array with the first 50M primes hardcoded.) What's that about?

<comment><USER83> I'd guess it is probably for Project Euler.

<comment><USER82> that "probably" makes me wonder why you would say that? is there something about 50 million primes in there?

<comment><USER83> I don't think you really need that many primes for Project Euler, but I can certainly imagine someone tired of including a call to her prime sieve in tons of problems, and of having to waiting for it to run, and thinking, "let's just make a library that has all the primes I could possibly need in a constant array".

</comment></comment></comment></comment><comment><USER84> "I could do that in a weekend"

You could but you didn't so I can't use it...

</comment><comment><USER85> Sometimes it takes a lot of time to reach something that is relatively trivial once you know how to do it.

I can reproduce E=mc² in a matter of minutes. Doesn't mean I could have ever come up with it in the first place.

</comment><comment><USER86> What exactly is the point here? At scale, it's not the tech that got the product to the finish line.

The endurance part of the company has nothing to do with tech or even scaling or optimizing that tech. Not even close.

</comment><comment><USER48> In France, the law (Art. R. 4228-10 du Code du travail) imposes one WC and one urinal per 20 male workers, and two WC per 20 female workers. So basically, it expects that up to 1 worker per each 10 workers will be busy with non-job related stuff.

Another datapoint, I heard that an accountant counted that the actual number was 1 for 50.

In any case, when you're alone or a small startup, this doesn't seem much, but when you have 100,000 employees, this means that between 2,000 and 10,000 equivalent employees are all the time in the WC (and being paid for that, basically).

Of course, there are a lot of other overheard that become very significant when you have a big corporation, but that still exist (and that we easily discount) for small startup and individuals.

<comment><USER87> Ever thought that maybe that 1:20 or 1:10 isn't about handling steady load but about handling pre-/post-lunch peak loads without half hour lines?

(And I've seen the half hour lines, in an office that had two WCs for 100 people. THEN you end up with people spending 10% of their day doing something other than work.)

</comment></comment><comment><USER88> That $12M/yr figure for indexing 1B documents seems way off to me.

<comment><USER89> Too high or too low?

<comment><USER88> Too high. See [http://news.appbase.io/scaling-elasticsearch- writes/](http://news.appbase.io/scaling-elasticsearch-writes/) for example:

> In doing these tests, we indexed over 1 billion documents under just two > hours. The peak merge rate we saw was 129 MB/s, yes our ElasticSearch > cluster on a 24 node setup is gobbling data at this speed.

> Most mind boggling of all: At our peak sustained ingestion rate, we would > have added over 13 billion documents in a single day and this would have > cost us a mere USD 244.

Their test documents are much smaller (100 bytes) than a typical web page but if you multiply that figure by 50 to get to 5kb documents (probably a realistic average web page size minus all markup and assets), you get 12200$ for 13 billion documents or 1220$ for 1.3 billion documents (assuming that scales linearly). I'm probably way off in my estimate myself but I'm convinced that a few thousand dollars is much closer than the proposed 12M$.

</comment></comment></comment><comment><USER90> Glad to see some sanity for once and people that realize building 'trivial' apps like Twitter and Uber are not feasible 'in a weekend'. Just making sure the app is secure takes months, and security becomes more complex as features are added. Pretty much every feature you add to the app can explode the complexity exponentially.

e.g: a sorted timeline if strings is easy, but add the simple feature of 'retweeting' and being able to 'unretweet' for example and things quickly become more complex.

</comment><comment><USER91> Those numbers are whacked. We use ElasticSearch with >10B docs ingested and the cost is like a few thousand dollars a month for hosting.

</comment><comment><USER92> Yeah right, except that you can't. It takes so much time to nurture something and build it in a way that satisfies the user.

</comment><comment><USER93> I really enjoy how this guy's blog refers to research papers. It adds to the level of interest.

</comment><comment><USER94> If you want to build a "boat" you start with a canoe, you don't build a ferry.

</comment><comment><USER95> googles and ubers of the world are not just their end-user facing apps + the backend required to support them.

these are large businesses with end user-facing "IT" side of the house being just the tip of the iceberg.

</comment><comment><USER96> The question is why do someone needs the first 50Mi primes in a C++ file

<comment><USER97> See making crc 4x faster in redis by keeping precommputed values in a lookup table: [https://matt.sh/redis-crcspeed](https://matt.sh/redis-crcspeed)

Maybe something like that ?

</comment></comment><comment><USER98> The code is the easy bit.

</comment><comment><USER99> Generally it is Marketing..

And also "Hofstadter's law"

</comment><comment><USER100> > Businesses that actually care about turning a profit will spend a lot of > time (hence, a lot of engineers) working on optimizing systems

Hm - every time I suggest optimization (of anything), I'm shot down with "premature optimization is the root of all evil!"

</comment><comment><USER101> I'm sorry for the OT, but I feel I should post this: [http://bettermotherfuckingwebsite.com/](http://bettermotherfuckingwebsite.com/)

This is unreadable as is.

<comment><USER102> Resize your browser window?

<comment><USER103> Yeah, but asking readers to resize windows, resize the font, and tweak the CSS to get decent line heights is a bit of cheek when it's trivially easy to implement at least minimally pleasant-to-read typography and layout.

It's a bit shortsighted of a writer to completely ignore the realities of readability. The current layout may conform to some vague idea of purity or minimalism, but that puts form over function in a big way.

<comment><USER104> Resizing a browser window is hardly as involved as tweaking CSS or browser settings.

</comment></comment></comment></comment><comment><USER36> Every variation of this comment/mindset drives me nuts. It's unbelievably thoughtless.</comment>


One Statistics Professor Was Just Banned by Google: Here Is His Story - <AUTHOR> http://www.zerohedge.com/news/2017-08-21/one-statistics-professor-was-just-banned-google-here-his-story <comment><USER2> This is why I just moved on to protonmail and am ditching Google as fast as I can after almost 20 years. I can't control what Google does, but I don't have to help them or be subject to their biases or censorsial behavior. The greatest good in the world is free exchange of ideas and it is not hard to spot stupid ideas (like racism, terrorism, etc.) and simply reject them. The whole Nazi thing seems like an excuse for Google to do what China has been having them do all along. Censorship is the tool of absolutists.

</comment><comment><USER3> Wait, so nobody knows why Google shut down his accounts?

I think we need some digital tenancy laws. These companies need a buffer period before they can digitally "evict" people. If Google disabled my accounts, I'd be totally screwed in so many ways.

<comment><USER4> Absolutely. Letting companies like Google shut down people's accounts unilaterally gives them far too much power. They should be legally required to provide notice and give you a chance to migrate your data.

</comment></comment><comment><USER5> It appears to be that he got filtered by the famed 'AI' terrorist filter.

On a twitter page of his collaborator describing his opponents:

"I don't understand English Middle Class English. What does "You are rude" mean?

1.You are saying the truth 2.Your truth bothers me 3.Can't find an argument against 4.You are rude"

It was probably this and the fact that he touches on "race, 'aryans', antisemitism' in a very provocative manner and has Arabic name made him a sure target for an angry user operations team member who decided to reward him with Internet non-personhood

</comment><comment><USER6> It seems odd that he keeps emphasising what a good person he is, like that will somehow help or even matters if your definition of good differs from who's deciding it.

In any case, thinking of corporations (which are amoral) as a person is silly, although this is a heartbreaking way to learn this lesson.

<comment><USER7> But when it comes to being banned, deplatformed, accused of being in a 'hate group', labelled a Nazi, etc, the problem _is_ the idea that expressing certain opinions indicates that you are a good person.

</comment></comment><comment><USER8> My prior is that there is some piece of the story missing. It appears that Google did a manual review and stated he violated the ToS. However it is certainly possible Google just screwed up, and that alone is pretty scary.

<comment><USER9> This is likely the case.

Accounts can be banned in bulk, but the review and appeal process is fully manual. He must have done something very bad to not get reinstated.

Normally only brand new gmail accounts clearly set up as spam senders are banned.

Probably ran a bot army clicking ads or some kind of billing scam on the play store or Google cloud. It could also be he is on a US government 'do not do business with' list.

<comment><USER10> "likely", "can be", "must have", "normally", "probably", "could also be".

Do you have any information the rest of us don't? You should share it if you do. Otherwise I'll just assume you're making this all up.</comment></comment></comment>


Disqus confirms 2012 hack of its comments tool - <AUTHOR> http://www.zdnet.com/article/disqus-confirms-comments-tool-hacked/ <comment><USER2> Discussion here: [https://news.ycombinator.com/item?id=15421112](https://news.ycombinator.com/item?id=15421112)</comment>


An Open Letter to the Erlang Beginner (2011) - <AUTHOR> http://ferd.ca/an-open-letter-to-the-erlang-beginner-or-onlooker.html <comment><USER2> I've been working in Erlang full time for the last 6 months or so. My background is in I guess what you'd call systems programming in C and C++. I haven't found the Erlang syntax to be a problem at all, but I've had major problems with a few things-

* The ambiguity of the term "application". It has a very specific meaning in the Erlang context but it used very generically elsewhere- it can be very confusing even just reading the docs.

* My background is in GNU Make/autoconf etc. and after a lifetime of building a binary, putting a binary onto a Unix machine, and then running the binary, figuring out how this works in the Erlang context was enormously confusing, and although the docs are very useful once you understand the build and deploy process, I think that some diagrams like those in this post would go a long way toward making it clear to a newbie what the division of labor between the local erlang install, systools, reltool (and possibly rebar since it seems to be nearly ubiquitous), and then once the release is built, what the boot script does, etc. Even little details like, why do I have a file called whatever_sup.erl and another called whatever_app.erl, and why are there a bunch of different functions named stop or start or init? I don't mean to slag on Erlang, as I like it a lot, but I found this part of the learning experience to be painful.

* Everybody talks about the hot code loading as if it just happens automatically, but as evidenced by the existence of Relx, it's not always trivial to work with, and if you don't know what you're doing you can mess things up for yourself pretty easily.

I am not a huge fan of jumping on third-party bandwagons for tools like rebar or relx- I like to try to stay as close to the OEM tools (for lack of a better term) as much as I can, so it'd be nice to see some of these tools become a part of the canonical OTP distro.

BTW the "Learn You Some Erlang..." book, has maybe the clearest and most gentle introduction to the building/releasing stuff that I've seen, but I think it could benefit from some more detailed diagrams.

<comment><AUTHOR> have you looked at 'erlang and otp in action' book ? might be useful...

<comment><USER2> yep that's a good one, it definitely has the most detail of al l the books I've seen so far wrt release/deploy. Also several people have recommended Francesco Cesarini's book as well, and I've seen a few posts about another guy who is writing one specifically on the VM itself, I think that'll be helpful also.

I would like to have one of these books that is more prescriptive than descriptive when it comes to the practices, although I think I could squeeze everything I'm looking for into a chapter-sized doc, might take a stab at it myself!

</comment></comment></comment><comment><USER3> Note that the author of this article also wrote "Learn You Some Erlang for Great Good!"[1], which is a wonderful resource for someone who wants to learn Erlang.

1: [http://learnyousomeerlang.com/content](http://learnyousomeerlang.com/content)

</comment><comment><USER4> And now, you can even have Erlang without the syntax, a la Elixir.

But as a novice, I do not understand people whining and whinging about syntax in language X or Y. Every language makes choices, and lots of those choices are emphasized by focusing/adapting the syntax to doing those things well. In each language that will be different, and that is the beauty of the diversity.

I cannot image Python without Python ranges, yields, and looping constructs. I cannot imagine Ruby without do blocks and the class structure and sigils and how really Rubinesque (this is a joke) code is for DSLs.

I like what the author of the OP did hear: if you complain the language syntax is ugly, perhaps you do not really see the perspective of the language creator quite yet. This explains how I struggled with different languages I studied in school (C++, then Perl, then Python) and even ones I work with which I hate (VBScript, Windows batch), because even those have logic to them when you keep the goals in the forefront of your mind.

<comment><USER5> I've never proved this formally, but one of the big problems with the Erlang syntax is that it has comma, semicolon, and period as distinct separators, despite the fact that as far as I can tell, they are _never_ ambiguous in the grammar. I've never been able to construct a series of tokens in which the meaning changes based on which of those you use, and it's legal either way. If you can, please, by all means show me (no sarcasm, I really would be intrigued to see it), but I bet it won't be commonly found in the wild either.

Beyond that, most of my annoyances are not technically syntax, but I gotta tell you that the above really is a big problem for newbies, and the annoyance never goes away even after you know what you're doing since it always inhibits moving code around.

But as Elixir shows, the annoyances really are mostly with Erlang-the- _language_ , not the implementation. Personally I sort of hope the success of Elixir helps the Erlang team take a fresh look at the Erlang language... there are some fixups one could do to Erlang to make it less annoying, too, and I think it could be all done compatibly. (A syntax to import functions from a module... a "match & rebind" operator... unifying the , . ; operators... something to clean up record manipulation... honestly, it wouldn't take much to really polish up Erlang and make a lot of the recurring klunkiness go away, that list alone would have cleaned up huge swathes of my code. A real string type would help too, even if were _exactly the same_ as a binary but with a different type for matching purposes for all I care, because that would clean up a lot of interop issues I had.)

<comment><USER6>
        > I've never been able to construct a series of tokens in
        > which the meaning changes based on which of those you
        > use, and it's legal either way.

Here's a series of tokens where switching comma to full stop means different things and both are legal:

        1> 1,0.
        0
        2> 1.0.
        1.0

In the first line, the comma is an expression separator. Attempting to use a full stop instead results in a floating point number.

        f(X, Y) when X, Y -> ok.
        g(X, Y) when X; Y -> ok.

In this case, the guard in f is turned into a guard sequence by switching a comma to a semicolon. Both are legal.

Matt

<comment><USER5> Neither of those are valid examples of what I meant. In the first, the . is not the function terminator, it's the floating point separator. Same ASCII character, different semantic token. In the second, the comma is not the statement separator, it's the value separator. Again, same ASCII character, different semantic token.

The question is, can you produce a situation in which ; . , are interchangable and valid in their capacity as statement separators.

My compliments for trying though, as I'm aware this smells like goalpost hiking and I do want to be complimentary about what you did. If you read my post again, though, I think it will become clear that this is what I meant all along, but I agree this superficially looks like I'm trying to cheat.

</comment></comment></comment></comment><comment><USER7> We recently decided to shift some of our nodejs production code to erlang. One of our main fears to shifting was the lack of really good online resources for beginners ("Learn You Some Erlang for Great Good!" is really good but does'nt cover topics like rebar etc).Also the learning curve is really steep compared to say a language like ruby or java

<comment><USER8> Cool - how is it working out so far?

</comment></comment><comment><USER9> I've recently started reading up on Erlang. While I can't comment too much on the syntax or semantics, I'm very much impressed with Erlang's error handling and the fact that it comes with some pretty good tooling out of the box (VM level tracing, process inspection, etc).

</comment><comment><USER10> > Jlouis had great arguments on the syntax with regards to his post: you have > to differentiate semantics from syntax itself. Erlang's semantics are what > should be seen as important, but it's rarely done.

Erlang syntax is small and self-consistent. Yes, it is not familiar. But those are orthogonal. There are languages with curly braces and ; that are terribly complicated, with ambiguous complicated syntax (C++), or ambiguous weak types (JS).

My way to look at it, if syntax is the biggest stumbling block, that is a warning sign that it will probably be hard to handle more difficult things -- thinking in terms of actors, reasoning about distributed systems, figuring out supervision hierarchies and so on.

BUT, as some mentioned here, there is Elixir. You get something like Ruby syntax. And you get variable-reassignment (x=1 then x=2 works). So maybe that helps there too. One still need to learn the semantics of it. You still have applications, supervisors and actors.

</comment><comment><USER11> Hot code reloading is the most sought after feature for me :) Many other technologies are borrowing this idea, namely the unreal engine 4, which comes with a hot code reloading C++.</comment>


Facebook Launches a Craigslist Competitor, Again - <AUTHOR> http://www.bloomberg.com/news/articles/2016-10-03/facebook-launches-a-craigslist-competitor-again <comment><USER2> Haha, good luck. Everyone who's ever tried to beat Craigslist at its own game has failed. Craigslist is an great example of worse-is-better: It does one thing, extremely cheaply, and it works well enough.</comment>


I forgot how to manage a server - <AUTHOR> https://ma.ttias.be/i-forgot-how-to-manage-a-server/ <comment><USER3> The three pitfalls TFA mentions are the reason why I enforce the following rules around operations automation at my company:

1\. Only that should be automated what can be done manually.

Before automating something, a sysadmin must be able to manually carry out the same operation. If they do not understand how it works, without using someone else's scripts, they are ineligible to administrate it.

2\. What has been automated should always be doable manually.

After something has been automated, a new sysadmin should be able to read the automation code and re-do the steps manually. Maybe not at the full scale of repeating actions manually on every server, but enough that if the automation tools are irrecoverably lost they can be rebuilt from knowledge.

This also serves as a test for adequate documentation in the automation code. A new sysadmin should not have trouble figuring out why a certain thing is done a certain way in the scripts.

3\. What happens, even when automated, should not be hidden from the sysadmin

Even in full auto, what's going on is always visible, so a sysadmin can jump in (i.e., go full-manual) at any step. This was one of our earliest rules, because we (me & my teammates) wanted to be able to debug without friction if anything went wrong. Also, it looks cool(er).

\----

As a guiding principle behind how we automate operations, we assert that no amount of automation is replacement for a sysadmin's skills; it replaces only manual, error-prone effort.

<comment><USER4> So, nothing here prevents "forgetting how to manage a server" like the article describes. This is literally how automations generally works. You are doing something manual. You know how to do it manually. You automate it so you don't have to do it manually. You never do it manually again.

Until you have to 10 years later for whatever reason, and suddenly, you forget. And suddenly you have to look things up.

<comment><USER5> Which is why I think there should still be some manually-managed servers remaining (possibly something like testing). What I didn't see mentioned is that when the automation fails for whatever reason, typically it is a five- alarm situation where minutes count. Having to search through man pages or stack exchange is wasting valuable time during one of the few situations where time is super-critical.

<comment><USER3> > Which is why I think there should still be some manually-managed servers > remaining (possibly something like testing).

All operations during development are carried out manually, unless already supported by automation (and until, of course). When automating the new operations, a sysadmin must also conduct the steps manually, and also take care of considerations a developer might not have been aware of.

> What I didn't see mentioned is that when the automation fails for whatever > reason, typically it is a five-alarm situation where minutes count.

Generally, this is easily-handled by redundancy, HA, and snapshot rollbacks. Of course anything that's a potential five-alarm situation has redundancy, right? :P

</comment></comment></comment></comment><comment><USER6> He didn't forget how to manage a web server. He knew his problems, how to look for solutions and how to apply them. That's an important skillset to. Especially since it enables one to keep up with change. You're not a professional sysadmin if you only mindlessly keep using the same configurations that were taught to you.

<comment><USER7> I found it a bit puzzling too. Does someone actually bother memorizing iptables syntax? Sure, if you use it daily for a long time, some of it will stick, but it seems completely pointless to waste time explicitly learning it.

I find it's better to save my brain power for learning things like what I need to take into account when configuring a database for example, or what variables affect network performance and what options I have for tuning, etc.

When I reach for tools to solve problems, the most I usually remember is "I can use X to solve this class of problems" and the how is available from the manual or with a Google search.

<comment><USER8> I'm sure someone does - I mean I have a friend whom I can hand a network diagram with specified cisco devices, and he can write cisco configs by hand, from memory - whereas me, I struggle to recall any of the regex I learned the last time I needed regex.

<comment><USER9> oh for real fuck regex. I need to just sit down for an hour and learn it, but I never do :( it's like Vim..

<comment><USER10> I don't even remember when or why I learned it. Was it 10 years ago? 15? 20? Was it the xkcd.com/208 ? Was it curiosity? All I know is I've been using it ever since and solved so many problems with it and yes, there are places where finding the right expression becomes a problem in itself, but online regex testers largely solve that problem by letting you visually test your regex. (see rubular, pythex)

People have huge reservations about parsing non-regular languages (like xml) with regular expressions because xml could have an attribute with a string with another xml in it or something, but the whole point is moot because regex is mainly used for partial matches and the possibility of having a string inside the html that could match whatever you are looking for is minuscule and will probably show up in tests anyway.

<comment><USER8> if you design your regex for the XML you're using it on, this shouldnt be an issue - I use it frequently to mung config files as needed for work stuff - <USER2> use XML or XML like files for all sorts of stuff.

</comment></comment></comment></comment></comment></comment><comment><USER10> Things he feels bad about are strange. I only remember how to allow root with key auth because I searched for it yesterday. "Not remembering how" is not the problem at all. "Not knowing what" is.

<comment><USER11> _Needing_ to remember is a sign of mismanagement, because if you have to remember it or look it up in third-party docs you might pick a different method than you did last time and create a subtle problem that might be time- consuming to debug. If anyone recalls the old days of having a handful of expensive servers that were configured at different times, often by different people, this was a common source of inconvenience and outages. The right way is to have it automated or at least recorded, for example in his presumably version-controlled config management system.

Of course this creates the pitfalls of automation, but I think it's unavoidable if you desire reliability and efficiency. Bypassing automation and typing in commands from memory or from man files, even if it's a one-off experiment with a fresh server, is going in the wrong direction to address the pitfalls of automation. If you're worried about not understanding the configuration commands your automation uses, a better way would be to call a meeting with some junior engineers (or a rubber duck) and talk through the configuration process line-by-line.

<comment><USER10> We are typing about converting multiple languages of expression (say, iptables rules and sshd config) into one (say, yaml or json) right? The whole point is to homogenize those languages into a single one so the configurations can be managed via a single program. (say chef, ansible)

And in the process of doing this, people stop remembering the original languages and you are now stuck in config language land where you still have to look up syntax on how to do things anyway.

So from your perspective, the problem is not "not remembering how" or "not knowing how" it is "not organising processes" am I getting it correct here?

<comment><USER11> > So from your perspective, the problem is not "not remembering how" or "not > knowing how" it is "not organising processes" am I getting it correct here?

From my perspective, he doesn't have a problem at all. I would even dispute that he doesn't know the "how." If his servers are configured via chef, then chef _is_ the how. Going deeper and looking at the OS-specific mechanisms chef uses is necessary for debugging when the behavior doesn't seem to match the documentation, just like if you're doing it by hand, iptables is the how, and the system calls or library calls it makes are what you'd look at if you couldn't understand its behavior from the man page and other docs.

In other words, you shouldn't worry if you're forgetting something because you never need to use it. When you need to use it, then you will learn it again and count it as an advantage that you did know it at some point in the past.

</comment></comment></comment></comment><comment><USER12> The thing is, memorization itself should -not- be the primary skill of a sysadmin.

A sysadmin should know how to troubleshoot, how to use tools, and (more importantly) how to learn how to use the tools.

I don't want to discount the value of someone who still remembers how to handle seemingly arcane tasks like iptables. Experience has its weight in gold, and memory definitely contributes to its value, but there's more to it than that. It is the capacity and willingness to learn in any situation, however, where a talented sysadmin really shines.

</comment><comment><USER13> I think it's ok to forget syntax and the like as long as you still understand the concepts.

If this guy had to work on a server for a week without any config manager, I'm sure he'd be right back at home.

Opposed to a guy who has never touched iptables. That guy is in it for the long haul.

Maybe a better title is "I forgot the commands(and some directory locations) I use to manager a server."

</comment><comment><USER14> I don't know how chef or puppet or that sort of scripts look like but my administration scripts are littered with: #Here I do A because B didn't work and we need it for C. Or in fstab: # This is the disk I bought together with father. Or in crontab: # This syntax looks weird but otherwise it does not select the correct python env.

(I'm just a home-server admin ;) but that also means significant stuff only happens every 3-5 years at a new LTS or with some major problem, so notes to self are important.)

Such notes to self not only help when things go wrong, they also make for easy repetition of stuff years later. When I reinstall, I first back up the entire old root with all configs and thus all my notes (I could limit myself to /etc probably).

<comment><USER15> I'm primarily a home admin, I keep all `history`, and have a little search script (grep really), that helps a lot.

<comment><USER14> Yeh, I guess the time gaps are so large that you have to. Still, I guess from time to time one also needs to update their orchestration scripts... Tbh, about the ssh, I know its in /etc, the I hit ssh, tab tab / tab tab, ah there is sshd_config, lets search for "root". I also don't really know by heart.

</comment></comment></comment><comment><USER16> I would simply argue that traditional "system administration" is now the responsibility of development teams rather than ops teams (yeah yeah DevOps, but the "ops" part has changed).

Ops today is largely responsible for maintaining build / deploy pipelines, orchestration systems, and ensuring SLAs are met via SRE activities.

Most functioning DevOps teams I've worked with recently have added a more generalist role for a person who is a mile wide and a foot deep. It's more of a hybrid sysadmin / development skill set ranging across base OS and package management, logging, scripting / automation, networking, access control, security, a dozen programming languages and whatever ITIL / EA platforms you have to interface with. These folks are a godsend in issue resolution as they know where the skeletons are buried. They also can pinpoint your top 5 tech debt issues of the top of your head.

The best version of this person also has some BA skills — they work really well as a demand management / intake person because they usually understand the end-to-end architecture — especially the code behind the integration interfaces — better than anyone on the team. They allow developers to focus on code, ops people to focus on production ops, and architects to focus at the right level of abstraction.

</comment><comment><USER17> This seems no different from forgetting how to write assembly by hand because you've been using C for so long.

<comment><USER3> C is a standard, approved by standards bodies, and with multiple implementations on multiple platforms pledged to be supported for ages.

Someone's Puppet and Ansible scripts for some tasks are no comparison to C the standard, and do not come with the breadth, stability, or support of C.

<comment><USER18> What does that have to do with the analogy? Nobody compared C to Puppet.

The point was that it's common to forget about details on a different level of abstraction to the one you normally operate. Doesn't matter if it's an ANSI standard or a hand-rolled DSL for a very niche use case.

<comment><USER3> I meant, C replacing assembly in the skill sets of the vast majority of us is still okay because it's widely supported, unlike many — as you put it — hand- rolled DSLs replacing few standard ways of configuring things.

<comment><USER17> There was never any standard though. sshd.conf is one hand-rolled DSL, smb.conf is another, httpd.conf is another...

<comment><USER3> But there is only one sshd_config with the exact same manual across all the servers across all companies that use it (distro and version differences notwithstanding). If you know what the PasswordAuthentication option in sshd_config means at one company, you know what it means at any other company, unlike each one's way of maintaining the file.

</comment></comment></comment></comment></comment></comment><comment><USER19> I'm not a sysadmin, but I'm remarkably bad at remembering boring things I don't care about. I generally forget things like how to create systemd units or sshd config options a couple minutes after completing the task.

I don't think it's a big problem nowadays. As long as you know how to quickly get the information you need, it's probably not that much more inefficient.

<comment><USER20> Agreed.

90% of stuff I don't know off the top of my head I do like this:

1) connect to a server/repository which already uses that in some form

2) copy/paste relevant parts.

3) add new parts by using existing parts as a reference

4) fill in holes with man page/docs/internet search engine

If I had to start from scratch I'd probably fail the basic syntax with pretty much anything.

The important thing is to know that you forgot implementation details, but know where to look them up.

[https://www.xkcd.com/1168/](https://www.xkcd.com/1168/)

</comment></comment><comment><USER21> "If you cannot already do the machine's job by hand, the machine will outwit you."

Raney Nelson [http://www.daedtoolworks.com/lounge-against-the-machine- daed...](http://www.daedtoolworks.com/lounge-against-the-machine-daedlab- commandments-1-and-2/)

</comment><comment><USER22> Most of us have forgotten how to write machine level code too

<comment><USER23> Fortunately writing home brew for old consoles keeps my wits up about this.

<comment><USER24> To be clear, you mean home brew as in beer, not the package manager, correct?

<comment><USER25> S/He means writing his/her own video games/software for old consoles [1]

[1][https://en.wikipedia.org/wiki/Homebrew_(video_games)](https://en.wikipedia.org/wiki/Homebrew_\(video_games\))

</comment></comment></comment></comment><comment><USER26> I wonder if it is something to do with puppet. I've never used it (I wrote my own thingy before puppet existed), but I gather puppet comes with a whole pile of configuration thingies out of the box. For example, maybe you don't have to know how edit the sudoer's file to prevent password prompting because puppet has a puppet way of doing it that edits the config file for you. I could well imagine if you did things by editing puppet files for a few years you might forget the underlying config file syntax, or even what config file ends up being edited.

If that's true he'll pick it up soon enough. The muscle memory will start flowing again.

If he wants something to panic over try moving to k8s. If effectively includes a puppet like thingy done a completely different way, and worse it builds a very different computing model. Use that for 10 years, then try to re-adjust your thinking back to a single server model that you maintain by editing config files and you will be in for a rough ride.

</comment><comment><USER27> It could be handy to write down some of this stuff you don't do often in a big org file (or plain text) for later reference. I find that some commands for irssi (tui irc client) are hard to remember, so I started writing a local irssiguide.txt that I look at or add to sometimes. I tend to add a new server or channel to autojoin right away, so then if I have a new thing to add 6 months later I can't recall the exact syntax. I've been meaning to follow my own advice in other areas. Taking notes as you learn things so that you can relearn them later, or optionally show them to a friend, seems very useful. I also sometimes get stuck on something and end up giving up and forgetting what the problem was, meaning coming back to it later is that much harder. I'd like to get into the habit of writing down problems I have as well. e.g. figuring out how to mount the storage of an android phone easily

</comment><comment><USER28> I rarely try and remember much more than there being a thing that does a thing, and then skim the reference to get the syntax correct when I need it. Things go in if I am using them regularly and go away again when I don't and I don't worry about it or think that I have forgotten how to do a thing. I also know that learning something again when it has disappeared almost completely, often leads to a deeper understanding of the thing, possibly because new connections have to be made before what is left of the original network gets lit up again, so even complete bewilderment at something I used to do regularly doesn't faze me all that much, I just get back on it in the knowledge that I will discover things I missed the first time round.

</comment><comment><USER29> this is why I don't focus on remembering how to do stuff

I focus on how to quickly learn (or re-learn) how to do it.

</comment><comment><USER30> What it used to mean to say someone was good at math was that they were very good at quickly solving complicated math in their head. One documentary I saw claimed that this is what they looked for at Oxford and Cambridge for their math programs.

As soon as the calculator was invented (okay, and made cheap and portable), this skill was useless. Then you had to set up problems so that the calculator could solve them. (Similarly with the computer).

This is a natural and good progression. All the best practices should be learned and mastered by people building the tool, and you should just be proficient at the tool.

</comment><comment><USER31> This is an honest question for sysadmins. What do you do if you go to a company that doesn't have Puppet or Chef? What do you say in an interview?

<comment><USER32> "I can improve the server architecture by automating a lot of the common maintenance tasks with Puppet or Chef."

</comment></comment><comment><USER33> I've forgotten how to write assembly, but that doesn't mean I can't make the computer do the same things with higher order abstractions.

<comment><USER34> Right. But while I think the important point is about abstractions, it's still valuable to understand-about (even if not "remember in detail how to manipulate") the next lower level abstraction.

Partly this is to fix things when the higher abstraction breaks.

But TFA's point is to manipulate the system without bringing in the cost of the higher abstraction.

</comment></comment><comment><USER35> I've gone through similar scenarios... you hire employees with fantastic experience which includes config management. Everything is optimal until low level problems occur. A good example of this is software raid failing, and the best playbook/manifest writer doesn't know where to begin(without google).

The solution is _really really simple_. Don't hire people without an RHCE, or that haven't had one in the recent past. Feel free to substitute the RHCE for a comparable certification, which are few and far between.

<comment><USER36> RHCEs, etc. don't strike me as adequate signaling. I've worked with RHCEs that didn't have much in the way of RHEL administrative chops. I've worked with people without them (and don't have one myself) who have solid downstack and upstack proficiency.

This is one that I genuinely don't know how to solve; I do not know how you evaluate somebody whose skillset you don't fully understand. But certifications aren't the answer.

<comment><USER35> >> I've worked with RHCEs that didn't have much in the way of RHEL administrative chops

I don't understand how this is possible. Did you validate their RHCE at redhat.com? It's not a multiple choice test, it's real systems(not connected to the internet, no google for you), and you don't pass if you can't configure/repair them and the services required. The requirements aren't toy examples, they tend to be complicated and involving things you want to avoid due to difficulty. Most people(>50%) fail the exam, according to redhat's statistics.

<comment><USER36> I didn't; I didn't hire them. Maybe they were lying. I sorta doubt it--the two I'm thinking about seemed like decent people, just in over their heads. But it's possible.

</comment></comment></comment></comment><comment><USER37> Strange how by "server" author means virtual a Droplet in Digital Ocean, not an actual metal server.

<comment><USER38> I'm not entirely sure why that's strange. It's been a common term for an instance or VM for as long as I can remember.

</comment></comment><comment><USER39> > I had to Google the correct SSH config syntax to allow root logins, but only > via public keys. I had to Google for iptables rule syntax and using ufw to > manage them. I forgot where I had to place the configs for supervisor for > running jobs, let alone how to write the config.

Are these actually things sysadmins know _without_ googling or peeking into manpages?

</comment><comment><USER40> Tools are good and useful, but don't forget the fundamentals.

</comment><comment><USER41> NixOS is much saner than your usual run of the mill distros.

</comment><comment><USER42> Seems a good use case for going serverless (I know about vendor lock in).

<comment><USER43> Serverless email servers! Serverless DNS servers! Serverless web servers!

I believe you should see a problem there.

<comment><USER44> Serverless is a misnomer for this kind of thing; it's "-as-a-service" not long ago. Nobody manages their own email or DNS servers anymore. Webservers, maybe, but that too can be handled by a few hundred different services depending on what is needed.

<comment><USER43> > Nobody manages their own email or DNS servers anymore

Thankfully you are quite wrong there; lots of us keep running the services that were designed to be decentralized.

</comment></comment></comment></comment><comment><USER45> > I had to Google...

This is how I do it too, except replace Google with DuckDuckGo.</comment>


3 reasons I play with my childrens xylophone alone - <AUTHOR> http://theahindle.com/3-reasons-i-play-with-my-childrens-xylophone-by-myself/

<comment><AUTHOR> Hit me up, just to chat!</comment>


Understanding Ruby Blocks, Procs and Lambdas - <AUTHOR> http://www.robertsosinski.com/2008/12/21/understanding-ruby-blocks-procs-and-lambdas/ Learn the subtle differences between Blocks, Procs and Lambdas with Ruby and how to use each effectively in your methods. <comment><USER2> Don't forget the changes in 1.9 <http://railspikes.com/2008/9/8/lambda-in- ruby-1-9>

And the much cooler Fibers, which you can think of as a reentrant block, kind of.

</comment><comment><USER3> Great post, I had the longest time trying to figure out what blocks were. They aren't used that much in PHP and Java and stuff so it was a totally new concept for me.

I still don't understand Procs and Lambdas but my head is too full today, will finish reading your post tomorrow.

As a side note, please enable comments, I'd love to leave you some good feedback on your actual site.

</comment><comment><USER4> "But I have yet to find a good, definitive guide out there."

Really? I thought this was the definitive guide: <http://innig.net/software/ruby/closures-in-ruby.rb>

It seems like a good guide to me.

</comment><comment><USER5> What's the reason for blocks not being real objects (i.e. Procs) themselves (like in JavaScript or Lisp)?

Is it simply to allow for the magic syntax of passing a block to a method so it can be called with yield?

<comment><USER6> Most of the problem is the `return` keyword. In a proc or lambda, it returns from the proc or lambda. In a block, it returns from whatever is enclosing the block, like a proc, lambda, or method.

If you had a block that was an actual object, it could outlive its enclosing environment. Calling return within it would reincarnate its enclosing environment. Calling return twice would fork time.

In other words, if a block was an object, the return keyword would create a _continuation_. This is a very expensive, very heavyweight feature in Ruby, and the whole point of blocks is to provide something cheap and lightweight.

Note that a proc or lambda has no such problem, since a return within it returns to whatever invokes the #call method of the proc or lambda.

Imagine how much simpler things would be if Matz had worked this all out and decided that the return keyword was more trouble than it was worth?

<comment><USER2> I respectfully disagree that the return keyword is more trouble than it is worth.

"return false if simple_question?" ... rest of the body

is a very useful thing to do. granted, you might say it is "programming by GOTO", but it lowers the indentation level, and so keeps me happy.

To the GP: we can also .to_proc on blocks.

<comment><USER6> Sorry I didn't see this when you posted it. The problem is not `return :foo if bar()`, it's something like `pizzles.each { |p| return :foo if p.bar() }`. In Ruby, that returns from the surrounding method, not from the block where it's defined.

You can decide for yourself if this behaviour of return more trouble than it's worth. I didn't say it was, I said that this definition is what forces a block to be something different from a proc, which makes the language more complicated.</comment></comment></comment></comment>